Simon Duff

Ho Ho Ho, time to go

Sat, 22 Dec 2018 00:00:00 +0000

Simon Duff - Ho Ho Ho, Time To Go!

Visualising Service Status with Emoji

Fri, 09 Nov 2018 00:00:00 +0000

Visualising Service Status with Emoji

Splunk dashboards can host a variety of different visualisation types, from maps and charts to 3D globes. Its also possible to use something as simple as Emoji. Inspired by this app, I propose the use of the following mapping of severity to emoji.

Splunk ITSI is a service monitoring tool that can intelligently assign severities to services based on underlying infrastructure or application issues. There are also many other severity and status categorisations, as listed in the references on this page.

Severity	Recommended Emoji	Other Applicable Emoji	Reason for Recommendation
Critical	💀	☠️ 😡🤬🔥💩❌🛑⛔️📛🚫	Significantly different than all other recommended emoji
High	😵	😳🤪😠😠😷🤒🤕🤢🤮⚠️😱😤😢😭😦😧😨😩🤯😬😰	Open mouth and X’d eyes lead to quick identification
Medium	😥	😣😮🤐😯😪😫😖😞😟	Emoji with sweat and frown
Low	🤔	🤨🤨😐😑	Emoji with hand and straight mouth
Normal	😃	😁😀😄😉😊😋🙂😎👍	Big eyes and big smile
Maintenance	🔨	🤞🔧⚒ 🛠⛏	Hammer is significantly different
Informational	ℹ️	🧐 🤓	They made an emoji specifically for Information, let’s use it

Using Emoji in your Splunk query

| rangemap field=load 💀=90-100 😵=80-89 😥=70-79 🤔=50-69 😃=0-49

Example dashboard

<dashboard>
  <label>Emoji-Monitor</label>
  <row>
    <panel>
      <single>
        <search>
          <query>
| makeresults | eval load=random()%100
| rangemap field=load 💀=90-100 😵=80-89 😥=70-79 🤔=50-69 😃=0-49 default=🔨
| fields range
          </query>
          <refresh>1s</refresh>
        </search>
        <option name="underLabel">indexer01</option>
      </single>
    </panel>
  </row>
</dashboard>

References

Now

Mon, 04 Jun 2018 00:00:00 +0000

What I’m Doing Now

Last Updated: 04 June 2018

Location

I’ve returrned to South Korea for a different customer (a large ISP), with which I’m developing machine learninng security use-cases.

Current Focus

Splunk Apps need to be completed!

This page inspired by nownownow.

vim commands for bulk incrementing

Sat, 24 Feb 2018 00:00:00 +0000

`vim` commands for bulk incrementing

One common task required as part of Splunk administration was that of managing the serverclass.conf file¹. This file contains a list of servers that belong to a particular class, and all machines of that class receive a particular set of configuration files.

A particularly frequent and some-what annoying task is adding and removing servers from each class list. All servers in that list need be prefixed with whitelist.X, where X is a consecutive number beginning at 0. When servers are added or removed, potentially many entries require updating, and doing this by hand could be error-prone.

Consider the following stanza; 2 machines have been decommissioned (commented out in the file), and 3 new servers added to the end. At this stage, the list would be rejected, as the entries do not begin at 0, there are gaps, and there are duplicates.

[serverClass:windowsserver]
# whitelist.0 = 192.168.0.1 # Decomisioned !
whitelist.1 = 192.168.0.2
whitelist.2 = 192.168.0.3
# whitelist.3 = 192.168.0.4 # Decomisioned !
whitelist.4 = 192.168.0.5
whitelist.5 = 192.168.0.6
whitelist.6 = 192.168.0.7
whitelist.7 = 192.168.0.8
whitelist.8 = 192.168.0.9
whitelist.9 = 192.168.0.10
whitelist.10 = 192.168.0.11
whitelist.10 = 192.168.0.12 # New !
whitelist.10 = 192.168.0.13 # New !
whitelist.10 = 192.168.0.14 # New !

The following commands will renumber and sort the entries nicely.

let i=0
vipoj:'<,'>g/^whitelist.\d\+/s//\='whitelist.'.i/ | let i=i+1
gv:'<,'>!sort

[serverClass:windowsserver]
# whitelist.0 = 192.168.0.1 # Decomisioned !
# whitelist.3 = 192.168.0.4 # Decomisioned !
whitelist.0 = 192.168.0.2
whitelist.1 = 192.168.0.3
whitelist.10 = 192.168.0.13 # New !
whitelist.11 = 192.168.0.14 # New !
whitelist.2 = 192.168.0.5
whitelist.3 = 192.168.0.6
whitelist.4 = 192.168.0.7
whitelist.5 = 192.168.0.8
whitelist.6 = 192.168.0.9
whitelist.7 = 192.168.0.10
whitelist.8 = 192.168.0.11
whitelist.9 = 192.168.0.12 # New !

Explanation

let i=0 {enter}

Create a new variable i, which will be used to count the valid lines.

vip

Visually select the entire paragraph.

oj

o to move the cursor to the other end of the block (the top), and j to move down once, thus skipping the stanza defining line.

Prepare to enter a command. vim will automatically insert '<,'>, which means to apply the command to the visual selection.

g/^whitelist.\d\+/

Globally perform a search where the start of the line (^) is followed by whitelist. and at least one number (\d\+). Because we look for whitelist at the begining of the line, the search ignores entries that have been commented out with a hash.

s//\='whitelist.'.i/

Substitute what is found by that search with whitelist. followed by the value of i.

| let i=i+1 {enter}

Increment the value of i.

gv

Repeat the previous visual selection.

Prepare to enter a command. Again, vim will automatically insert '<,'>.

:!sort {enter}

Sort the results.

Know a better sort? Please let me know!

References

There’s now a nice GUI for managing this file, but nothing beats getting your hands dirty and editing by hand! ↩

Hammerspoon Configuration

Mon, 30 Oct 2017 00:00:00 +0000

Hammerspoon

Hammerspoon is a OS X utility that provides a high level of customisation to control the working environment. Using it, one can specify configuration where events such as mouse clicks and drags, and keyboard shortcuts can be used to trigger scripts or alter the environment as one prefers.

Caps Lock

For many, the caps lock key is one of the most useless keys on modern keyboards; rarely used, it is one of the largest buttons on a keyboard with little useful functionality.

On the other hand, the new Macbook Pros have eliminated the escape key, one of the most important keys for a vim user, replacing it with the touch bar. While the escape “key” would still be present on the touch bar, there is something reassuring about pressing a physical escape button.

An ideal solution to this is to replace the functionality of the caps lock key with the escape key. Rather than trying to hit the missing escape key, it is possible to map the escape key to the caps lock key, allowing for less hand movement, as well as a experiencing the tactile feedback of an actual key, rather than the touch bar.

Mapping the caps lock key to escape is fairly easy in OS X. Under System Preferences, Keyboard, Modifier Keys, the option to use Escape key as the action to perform when the caps lock key is pressed.

Hammerspoon and Karabiner-Elements take this to a whole new level however, by allowing the caps lock key to function as an escape key when pressed alone, and also acting as a “hyper” key when pressed in conjunction with other keys. This “hyper” key opens up so many more possibilities, for having global or application specific function keys, especially when combined with Hammerspoon.

Karabiner-Elements

The first element (pun intended) in this setup is the configuration for Karabiner-Elements. The following code JSON should be saved in a file in ~/.config/karabiner/karabiner.json. This config remaps the caps lock key to send the escape key when pressed alone, while sending the command, control and option keys when pressed in conjunction with other keys.

{
    "global": {
        "check_for_updates_on_startup": true,
        "show_in_menu_bar": true,
        "show_profile_name_in_menu_bar": false
    },
    "profiles": [
        {
            "complex_modifications": {
                "parameters": {
                    "basic.to_if_alone_timeout_milliseconds": 1000
                },
                "rules": [
                    {
                        "description": "Shift Capslock sends Shift-Hyper, or escape if by itself",
                        "manipulators": [
                            {
                                "from": {
                                    "key_code": "caps_lock",
                                    "modifiers": {
                                        "mandatory": [
                                            "left_shift"
                                        ]
                                    }
                                },
                                "to": [
                                    {
                                        "key_code": "left_shift",
                                        "modifiers": [
                                            "left_command",
                                            "left_control",
                                            "left_option"
                                        ]
                                    }
                                ],
                                "to_if_alone": [
                                    {
                                        "key_code": "escape"
                                    }
                                ],
                                "type": "basic"
                            }
                        ]
                    },
                    {
                        "description": "Capslock sends Hyper, or escape if by itself",
                        "manipulators": [
                            {
                                "from": {
                                    "key_code": "caps_lock"
                                },
                                "to": [
                                    {
                                        "key_code": "left_command",
                                        "modifiers": [
                                            "left_control",
                                            "left_option"
                                        ]
                                    }
                                ],
                                "to_if_alone": [
                                    {
                                        "key_code": "escape"
                                    }
                                ],
                                "type": "basic"
                            }
                        ]
                    }
                ]
            },
            "devices": [],
            "fn_function_keys": [
                {
                    "from": {
                        "key_code": "f1"
                    },
                    "to": {
                        "consumer_key_code": "display_brightness_decrement"
                    }
                },
                {
                    "from": {
                        "key_code": "f2"
                    },
                    "to": {
                        "consumer_key_code": "display_brightness_increment"
                    }
                },
                {
                    "from": {
                        "key_code": "f3"
                    },
                    "to": {
                        "key_code": "mission_control"
                    }
                },
                {
                    "from": {
                        "key_code": "f4"
                    },
                    "to": {
                        "key_code": "launchpad"
                    }
                },
                {
                    "from": {
                        "key_code": "f5"
                    },
                    "to": {
                        "key_code": "illumination_decrement"
                    }
                },
                {
                    "from": {
                        "key_code": "f6"
                    },
                    "to": {
                        "key_code": "illumination_increment"
                    }
                },
                {
                    "from": {
                        "key_code": "f7"
                    },
                    "to": {
                        "consumer_key_code": "rewind"
                    }
                },
                {
                    "from": {
                        "key_code": "f8"
                    },
                    "to": {
                        "consumer_key_code": "play_or_pause"
                    }
                },
                {
                    "from": {
                        "key_code": "f9"
                    },
                    "to": {
                        "consumer_key_code": "fastforward"
                    }
                },
                {
                    "from": {
                        "key_code": "f10"
                    },
                    "to": {
                        "consumer_key_code": "mute"
                    }
                },
                {
                    "from": {
                        "key_code": "f11"
                    },
                    "to": {
                        "consumer_key_code": "volume_decrement"
                    }
                },
                {
                    "from": {
                        "key_code": "f12"
                    },
                    "to": {
                        "consumer_key_code": "volume_increment"
                    }
                }
            ],
            "name": "Default profile",
            "selected": true,
            "simple_modifications": [],
            "virtual_hid_keyboard": {
                "caps_lock_delay_milliseconds": 0,
                "keyboard_type": "ansi"
            }
        }
    ]
}

Hammerspoon Configuration

The following Hammerspoon configuration should be saved to ~/.hammerspoon/init.lua.

It defines hyper as the combination of command, control and option being pressed at the same time, which has been remapped to the caps lock key. shyper is defined as the same combination, in addition with the shift key.

There are various utility functions defined, then the main hyper/shyper key bindings are specified.

For now, there are simple terminal, firefox and finder window launching, as well as a quick shortcut for putting the computer’s display to sleep.

Hammerspoon is not only limited to launching commands triggered by keystrokes. There is a section devoted to window layout, sending windows to various grid locations. If you imagine a screen being split into 4 quadrants, there are various key bindings (in association with the caps lock/hyper remapping) to send windows to occupy various quadrants. This allows a focused window to be quickly mapped to a region of the screen.

There is also a section devoted to using the mouse to control layout of windows. By grabbing the title bar of a window and dragging to either the edge or corner of the screen, the related window will be resized and positioned appropriately.

Concluding the configuration, there are a few miscellaneous items and comments for future plans :)

--[ Defines ]-------------------------------------------------------------

hyper =  {"⌘", "⌥", "⌃"}	-- caps lock held down
shyper = {"⌘", "⌥", "⇧", "⌃"}	-- caps lock held down, with shift
hs.window.animationDuration = 0


--[ Functions ]---------------------------------------------------------
function findOrLaunch(a)
	local app = hs.application.find(a)
	if not app then
		hs.application.launchOrFocus(a)
	end
	return hs.application.find(a)
end

function fancyNotify(t,m)
     hs.notify.new({title=t, informativeText=m}):send():release()
end


--[ Bindings ]---------------------------------------------------------

-- hyper shortcuts
hs.fnutils.each({
	{ key = "a", func = function() 
      myTerm = findOrLaunch("iterm")
      myTerm:selectMenuItem({"Shell","New Window"}) 
      end
  },
  { key = "q", func = function()
      ffox = hs.appfinder.appFromName("Firefox")
      ffox:selectMenuItem({"File","New Window"})
      ffox:activate()
      end
  },
  { key = "z", func = function()
      finder = hs.appfinder.appFromName("Finder")
      finder:selectMenuItem({"File","New Finder Window"})
      finder:activate()
      end
  }
}, function(object) hs.hotkey.bind(hyper, object.key, object.func) end)

-- shyper shoutcuts
hs.fnutils.each({
  { key = "space", func = function()
      os.execute("pmset displaysleepnow")
      end 
  }
}, function(object) hs.hotkey.bind(shyper, object.key, object.func) end)



--[ grid ]---------------------------------------------------------------------
-- home made grid system works with hammerspoon's grid

hs.grid.setGrid('2x2')
hs.grid.setMargins('0x0')

function moveWindow(x, y, w, h)
	local win = hs.window.focusedWindow()
	local f = win:frame()
	local screen = win:screen()
	local max = screen:frame()

	f.x = max.x + (max.w*x)
	f.y = max.y + (max.h*y)
	f.w = max.w*w
	f.h = max.h*h
	win:setFrame(f)
end

-- bind hotkeys to grid 
hs.hotkey.bind(hyper,"y", function() moveWindow(0.0,0.0,0.5,0.5) end)
hs.hotkey.bind(hyper,"u", function() moveWindow(0.0,0.0,1.0,0.5) end)
hs.hotkey.bind(hyper,"i", function() moveWindow(0.5,0.0,0.5,0.5) end)
hs.hotkey.bind(hyper,"h", function() moveWindow(0.0,0.0,0.5,1.0) end)
hs.hotkey.bind(hyper,"j", function() moveWindow(0.0,0.0,1.0,1.0) end)
hs.hotkey.bind(hyper,"k", function() moveWindow(0.5,0.0,0.5,1.0) end)
hs.hotkey.bind(hyper,"n", function() moveWindow(0.0,0.5,0.5,0.5) end)
hs.hotkey.bind(hyper,"m", function() moveWindow(0.0,0.5,1.0,0.5) end)
hs.hotkey.bind(hyper,",", function() moveWindow(0.5,0.5,0.5,0.5) end)

--[ window dragging ]----------------------------------------------------------
dragging_window = nil
click_event = hs.eventtap.new({hs.eventtap.event.types.leftMouseDragged}, function(e)
	if dragging_window == nil  then
		-- check mouse is in titlebar
		local m = hs.mouse.getAbsolutePosition()
		local f = hs.window:focusedWindow():frame()
		local screen = hs.window:focusedWindow():screen()
		local max = screen:frame()
		if m.x > f.x and m.x < (f.x + f.w) then
			if m.y > f.y and m.y < (f.y + 21) then
				dragging_window = hs.window.focusedWindow()
				dragging_window_time = hs.timer.localTime()
			end
		end
	end
end)
unclick_event = hs.eventtap.new({hs.eventtap.event.types.leftMouseUp}, function(e)
	if dragging_window ~= nil then
		local m = hs.mouse.getAbsolutePosition()
		local f = hs.window:focusedWindow():frame()
		local screen = hs.window:focusedWindow():screen()
		local max = screen:frame()
		if m.x < 50 then
			if m.y < 200 then
				moveWindow(0.0,0.0,0.5,0.5)
			elseif m.y > (max.h-200) then
				moveWindow(0.0,0.5,0.5,0.5)
			else
				moveWindow(0.0,0.0,0.5,1.0)
			end
		elseif m.x > (max.w-50) then
			if m.y < 200 then
				moveWindow(0.5,0.0,0.5,0.5)
			elseif m.y > (max.h-200) then
				moveWindow(0.5,0.5,0.5,0.5)
			else
				moveWindow(0.5,0.0,0.5,1.0)
			end
		elseif m.y < 25 and m.x>200 and m.x < (max.w-200) then
				moveWindow(0.0,0.0,1.0,1.0)
		end
	end
	dragging_window = nil
end)
click_event:start()
unclick_event:start()


--[ hourly alarm ]----------------------------------------------------------
hs.timer.doAt("0:00","1h", function() hs.alert("Ding Dong") end)

--[ fan ]-------------------------------------------------------------------
--[[
	my mac has a very noisy fan, this just puts the CPU temperature and fan
	speeds in the title bar
]]--
local menu_bar = hs.menubar.new()
menu_bar:setTitle("¯\\_(ツ)_/¯")
hs.timer.doAt("0:00","1m", function() 
	local d = hs.execute("/Users/sduff/bin/osx-cpu-temp -f"):gsub("[\r\n]","")
	menu_bar:setTitle(d)
end)

--[ hazel-lite ]------------------------------------------------------------
--[[
	Hazel-lite file system cleanup and management
	
	Ideas
	- Downloads/Desktop
		- Tag files
		- _Actions
		- Delete old files, Archive files to monthly folders
		- Move dmg to Trash, then open them
		- Add mp3 files to iTunes (music directory)
		- Drop folder that automatically zips and files away
		- IFTTT download top photo from earthporn/unsplash to dropbox, and then sync dropbox to make wallpapers
		- monitor a folder for images, then create a shrunk version, then move them to website git directory, then add them to the git build. remove the exif info as well, add copyright, etc... Needs Exiftool (https://www.sno.phy.queensu.ca/~phil/exiftool/) or Imagemagick's mogrify (https://stackoverflow.com/questions/229446/how-do-i-add-exif-data-to-an-image)
		- Move Desktop files to Downloads
		- Delete files older than 1 week
		- Archive certain files to Evernote/Dropbox/etc...
		- Save to Evernote drop-folder
		- Move torrent files to VirtualMachine
		- Sync ebooks in dropbox
		- Regularly clear tmp folders
		- Regularly clear Outlook tempfiles (/private/var/folders/y3/g8b0sww1685gtyvx20c7ghpxrdx1c3/T/com.microsoft.Outlook) (maybe symlink ~/Outlook to this folder for future versions)
		- Delete files that were downloaded from localhost
		- Archive downloaded software (app, iso, dmg, deb, exe, pkg, rpm)
		- Move downloaded videos (mov, mp4, avi) to its own directory
		- Move epub,mobi,pdf to an ebooks directory, well named (use mdls to extract pdf metadata, or pdfminer, https://github.com/euske/pdfminer)
		- mdls - https://www.macissues.com/2014/05/12/how-to-look-up-file-metadata-in-os-x/
		- Weekly empty of Trash

	Resources
	- https://github.com/scottcs/dot_hammerspoon/blob/master/.hammerspoon/modules/hazel.lua
	- https://www.reddit.com/r/apple/comments/1wlxtr/do_you_use_hazel_what_are_some_of_your_coolest/

	- http://getawesomeness.herokuapp.com/get/osx	
	- http://awesomeawesome.party/awesome-macOS


	Other efficiency
  - https://msol.io/blog/tech/work-more-efficiently-on-your-mac-for-developers/
  - http://stevelosh.com/blog/2012/10/a-modern-space-cadet/ (The Grand daddy)

  Something about launch or cycle
  - https://github.com/szymonkaliski/Dotfiles/blob/b5a640336efc9fde1e8048c2894529427746076f/Dotfiles/hammerspoon/init.lua#L442-L485

--]]

--[ Huzzah - we're ready! ]----------------------------------------------------
fancyNotify("Hammertime","Captain Hammer!")

Download

My configuration for Karabiner-Elements, Hammerspoon and other utilities can be downloaded from my github config repository.

References

More Hammerspoon sample configurations
A Modern Space Cadet An excellent writeup of one user’s extensive custom configuration

Invisilink

Sun, 29 Oct 2017 00:00:00 +0000

Simon Duff - Invisilink

Google Play and the Google Play logo are trademarks of Google LLC.

Enough vim to be dangerous

Mon, 09 Oct 2017 00:00:00 +0000

Enough `vim` to be dangerous

My occupation means that I’m usually working at a different customer from one week to the next. There is little time to configure the operating environment I’m working, so I have become some-what proficient at using vim using the out-of-the-boxes defaults. Different distributions may ship with a different set of defaults, but it is usually enough to be efficient by relying on the vim basics.

`vim` tips, tricks and shortcuts

vim commands for bulk incrementing

My local `.vimrc`

My .vimrc for my laptop is very minimally configured – again, I’ve avoided the use of plugins and major customisations, lest I develop a habit or muscle memory relying on these.

" vimrc with minimal configuration

" Decoration
syntax on
set background=dark
colorscheme lucario
set ruler
set number numberwidth=4

" Tabbing
set expandtab
set shiftwidth=2 tabstop=2
set smartindent

" Searching
set ignorecase smartcase
set incsearch
set showmatch
set hlsearch

References

Coming soon

Processing Excel XLSX files with Python

Mon, 09 Oct 2017 00:00:00 +0000

Processing Excel XLSX files with Python

Using Python’s ElementTree XML Parser, its possible to quickly parse data within a Microsoft Excel .XLSX file.

An .XLSX file consists of a large number of XML files that are compressed into a single .ZIP file. Some Spreadsheets make large use of sharedStrings, where are strings that are stored in a separate XML file (sharedStrings.xml) which then acts as the source of a lookup for the worksheets. Cells that have a s value in the t attribute for a cell contain a sharedString, whose contents actually exist in another xml fie in the .XLSX. The following script should form the basis of an .XLSX to .CSV convertor.

#!/usr/bin/env python

import xml.etree.ElementTree as ET
import codecs
import sys
import datetime

def excel_date_to_str(d):
  # excel stores a date as days since 01/01/1900
  # by adding 6933594 to the excel value, we can use python's date module to format it correctly
  ret = None
  try:
    ret = datetime.date.fromordinal(int(d) + 693594).strftime("%A, %d %B %Y")
  finally:
    return ret

UTF8Writer = codecs.getwriter('utf8')
sys.stdout = UTF8Writer(sys.stdout)

x = ET.parse(open("xl/sharedStrings.xml"))
sharedStrings = x.getroot()

last_v = None
last_type = None
for event, elem in ET.iterparse("xl/worksheets/sheet1.xml", events=('start', 'end')):
  uri, tag = elem.tag.split("}")

  if event == "start" and tag == "c": # start c tag
      last_v = None
      if "t" in elem.attrib:
        last_type = elem.attrib["t"]
      else:
        last_type = None

  elif event == "end" and tag == "c":   # end c tag
    if "r" in elem.attrib:
      rc = elem.attrib["r"]
      if last_v != None:
        print "RC is ", rc, " = ", last_v

  elif event == "start" and tag == "v": # start v tag

    value = "".join(elem.itertext())
    if last_type == "s":
      last_v = "".join(sharedStrings[int(value)].itertext())
    else:
      last_v = value, "type is ", last_type, excel_date_to_str(value)

References

Simulating the Monty Hall Problem

Sun, 01 Oct 2017 00:00:00 +0000

Monty Hall Problem simulated in Python

In light of the recent news that Monty Hall has passed away, I thought I would get around to posting a simulation I had written a while ago.

The Monty Hall Problem

The Monty Hall Problem is a puzzle that seems to contradict common sense. The problem can be stated as such:

On a game show, there are 3 doors. Behind one door is a car, while the other doors hide goats. The contestant chooses one door, after which, the host opens one door, revealing a goat. The contestant is then offered the choice of either sticking with their current choice, or switching to the other door. What is the best decision?

It is important to note the following conditions are always met:

The host knows what is behind each door, and will always reveal a goat.
The contestant is always offered a chance to switch.

Solution

Switch! At first glance, it appears that the contestant’s chance of wining has increased from 33% to 50%, so switching or not, shouldn’t make any difference. However, it can be shown that switching actually increases the contestant’s chances of wining to 66% !

Python Simulation

I have written the following Python code to simulate the problem, attempting to generalise the solution for any number of doors, along with the host opening any number of doors.

#!/usr/bin/env python

import random

def monty(rounds=1000, doors=3, doors_to_open=1):
    assert (doors_to_open + 2 <= doors)

    initial_wins = 0
    switch_wins = 0

    for r in xrange(1,rounds+1):
        d = ["car"] + ["goat"] * (doors-1)
        random.shuffle(d)

        # player's choice
        c = random.randrange(doors)

        if d[c] == "car":
            initial_wins += 1

        # host will open some number of doors
        opened = 0
        while opened < doors_to_open:
            o = random.randrange(doors)
            if o == c or d[o] == "car" or d[o] == "open":
                continue
            else:
                d[o] = "open"
                opened += 1

        # look for a new door that is closed, and isn't player's current pick
        while True:
            new_door = random.randrange(doors)
            if (new_door != c and d[new_door] != "open"):
                c = new_door
                break
        
        if d[c] == "car":
            switch_wins += 1
    return initial_wins, switch_wins

print("doors,doors_to_open,stick_wins,switch_wins")
for d in range(3,101):
    for dto in range(1,d-1):
        stick, switch = monty(rounds=10000, doors=d, doors_to_open=dto)
        print("%s,%s,%s,%s"%(d,dto,stick,switch))

Operation

This simulation will run many rounds of this scenario with random placement of cars and goats, and count the number of times the contestant randomly chooses the correct door first (winning if they don’t switch), versus having the contestant switch to a randomly selected unopened door after the host reveals a goat.

Results

The results of this simulation can be found in the file, monty_hall_10k.csv. Each row contains the number of doors available, the number of doors opened by the host, then the number of times the contestant wins without switching, and the number of times the contestant wins with switching. Each scenario (number of doors, and number of doors opened by the host) is run 10,000 times.

Observations

Unsurprisingly, there are only a very few cases in this scenario where sticking is the correct course of action – switching almost always results in a higher chance of wining the car. Only when a very small percentage of doors are opened does this appear, and never consistently.

The game is set to the host’s favour until over 95% of the doors are opened. Only then will the contestant have a greater than 50% chance of wining the car, and then, only when he switches.

Sample Splunk Query

source="monty_hall_10k.csv"
| eval perc_doors = round(doors_to_open/doors,2) 
| eval perc_switch = round(switch_wins/10000,2) 
| eval perc_stick = round(stick_wins/10000,2)
| chart avg(perc_switch) as "Switch Wins", avg(perc_stick) as "Stick Win",  over perc_doors
| rename perc_doors as "% Doors opened"

If we describe the problem as there being 100 doors, of which the host opens 98 of them, is it more intuitive that the contestant should switch? They have a 99% chance of wining the car if they do so. Yet it still feels as though they have a 50-50 choice between the door they hold and the remaining door.

Future ideas

Here are a few ideas for altering the game, which may alter the contestant’s best decision.

What if there are more than one car hidden behind the doors?
If there are multiple prizes, what if the number of prizes is not specified? What if there is an upper or lower bound is specified, for example, up to 3 cars, or at least 3 cars are behind the doors.
What happens if the host doesn’t know what’s behind each door?
What happens when there are multiple prizes of different degrees (car, holiday, goat)? Should one switch if the host reveals a car? What are the best rules for a host in chosing what to reveal?

Sitemap pings for GitHub-hosted Jekyll sites

Thu, 28 Sep 2017 00:00:00 +0000

Sitemap pings for GitHub-hosted Jekyll sites

What is a sitemap?

According to a news release by Google, Yahoo and Microsoft, a sitemap is a free and easy way for webmasters to notify search engines about their websites and be indexed more comprehensively and efficiently. Essentially, it is an XML document that describes the various pages of a website, including their relative importance and update frequency. Although these search engines have crawlers that will index websites, pinging them with a sitemap might help guide their attention.

Generating a sitemap with Jekyll

The easiest way to generate a sitemap in Jekyll is to use the Jekyll Sitemap plugin. The plugin is already installed on GitHub, so using it is as simple as including the following stanza in the site’s _config.yaml

plugins:
  - jekyll-sitemap

For my website, I required a little more customisation, so created a file called sitemap.xml in the website root directory.

---
layout: none
sitemap:
  exclude: 'yes'
---
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
{% assign docs = site:docs | reverse docs %}
  {% for page in docs %}
    {% unless page.sitemap.exclude == "yes" %}
    <url>
      <loc>{{ site.url }}{{ page.url | remove: "index.html" }}</loc>
      {% if page.sitemap.lastmod %}
        <lastmod>{{ page.sitemap.lastmod | date: "%Y-%m-%d" }}</lastmod>
      {% elsif page.date %}
        <lastmod>{{ page.date | date_to_xmlschema }}</lastmod>
      {% else %}
        <lastmod>{{ site.time | date_to_xmlschema }}</lastmod>
      {% endif %}
      {% if page.sitemap.changefreq %}
        <changefreq>{{ page.sitemap.changefreq }}</changefreq>
      {% else %}
        <changefreq>monthly</changefreq>
      {% endif %}
      {% if page.sitemap.priority %}
        <priority>{{ page.sitemap.priority }}</priority>
      {% else %}
        <priority>0.3</priority>
      {% endif %}
    </url>
    {% endunless %}
  {% endfor %}
</urlset>

These Jekyll directives generate the relevant XML code for each page in my web site, generating a valid sitemap.xml as the output.

Github webhooks

GitHub allows for webhooks to be attached to actions in a repository. In this case, there should be a push to the various search engines to notify them that the sitemap has been updated. To do so, the following URLs need to be called, either by a GET or POST request.

https://www.google.com/webmasters/tools/ping?sitemap=http://simonduff.net/sitemap.xml
http://www.bing.com/ping?sitemap=http://simonduff.net/sitemap.xml

To configure, go to website repository’s Settings section, and locate the section called Webhooks.

Configure webhooks for Google, Bing and any other search engine with sitemap submission facilities.

After your next website push, you can check the responses of the two pings by going to the individual webhook pages and checking the responses.

Colophon

Thu, 21 Sep 2017 00:00:00 +0000

Colophon

How does this marvelous web site work?

Site Generation and Hosting

This website it hosted on GitHub Pages , which integrates Jekyll, a static site generator. You can find the source for the site at http://github.com/sduff/sduff.github.io.

Previously, I had written a custom site generator, called WASTE, but found I was spending more time working on it than I did writing content for the site. I abandoned it, but the WASTE sources are available, should you wish to see some hacked-together python code.

Theme

I’ve slightly modified an existing Jekyll theme called Pixyll as I wasn’t happy with the readability of my homemade CSS.

For code highlighting, I forked a popular text editor theme called Lucario. This is the same theme I use for my vimrc and terminal.

Build

One particular aspect of this website is its connection to Travis-CI, a continuous integration system. When I push new content to GitHub, Travis will automatically attempt to build the site and verify any URLs are present and correct. I’ll be emailed the results of the build, on success or failure. I can also include an icon on this page showing the build status.

Sitemap pings

WASTE, my homemade site generator, could atomically ping Google and Microsoft Bing to alert them that a new version of the site was available. There are actually several locations where we can hook into the GitHub and Travis build process to perform the same tasks. These mechanisms will be documented at a later stage.

Analytics

I like data, and the site access logs are tracked using Google Analytics. Its probably overkill for my requirements, which is just to reassure myself that someone is actually getting value from my writing.

Affiliate Links

This site may have some affiliate links, which means that when you click on a link that I suggest, it may result in a commission for me. I’ll only link to items I actually use or support, and the possibility of a commission doesn’t sway my opinion of the product.

Seriously, we’re probably talking a few cents here.

Disclaimer

The postings on this site are my own and do not represent the position or opinions of my employer or its affiliates.

All data and information provided on this site is for informational purposes only. I make no representations as to accuracy, completeness, currentness, suitability, or validity of any information on this site and will not be liable for any errors, omissions, or delays in this information or any losses, injuries, or damages arising from its display or use. All information is provided on an as-is basis.

Splunk Alert Script Template

Mon, 17 Jul 2017 00:00:00 +0000

Splunk Alert Script

A Splunk alert is a search that is periodically run, and if there are any results, an alert action can be run. This is a custom script that can be used to iterate over the search results and perform a task with the found fields. This process is documented, but the key takeaways are summarised on this page.

Alert Script Arguments

When Splunk fires an alert action, there are several arguments that are passed to the script. I usually ignore all of them, except for argument 8, which contains a gzip CSV file containing the search results. The process is usually to open this file and iterate over the results.

Arg	Environment Variable	Value
0	SPLUNK_ARG_0	Script name
1	SPLUNK_ARG_1	Number of events returned
2	SPLUNK_ARG_2	Search terms
3	SPLUNK_ARG_3	Fully qualified query string
4	SPLUNK_ARG_4	Name of report
5	SPLUNK_ARG_5	Trigger reason For example, “The number of events was greater than 1.”
6	SPLUNK_ARG_6	Browser URL to view the report.
7	SPLUNK_ARG_7	Not used for historical reasons.
8	SPLUNK_ARG_8	File in which the results for the search are stored. Contains raw results in gzip file format.

Alert Script Code

The following template should be placed in the $SPLUNK_HOME/bin/scripts/ directory.

import gzip
import csv

def openany(p):
	if p.endswith(".gz"):
		return gzip.open(p)
	else:
		return open(p)

results = sys.argv[8]
for row in csv.DictReader(openany(results)):
	# do something with row["field"]

This command can then be selected as a Trigger Action when saving or modifying an alert.

Advanced Alert Actions

The aforementioned script has actually been deprecated in preference of Modular Alerts. These provide much more customisation features, but I often find this overkill.

Splunk also initiated the Adaptive Response Initiative which allows for Splunk to integrate with other (mainly security related) 3rd party products, and allows for bidirectional communication between them. This is a great source for ideas for alert actions.

Alert Script ideas

Ticket Systems (raise a ticket)
EMail (send an email)
Anti Virus (start a scan on a particular machine)
Firewall Rules (isolate a compromised machine)
IOT devices (think physical traffic lights, flags and sirens)
Enrichment (Google a search query, write the results to a file which is indexed by Splunk)
Expect script (ssh to a server and restart a hung process)

Restricting the Splunk Time Range Picker

Thu, 06 Jul 2017 00:00:00 +0000

Restricting the Splunk Time Range Picker

The Splunk time range picker is a convinient widget for selecting froma variety of different timeframes for a search.

There are several ways that this timepicker can be customised, but they are rather heavy-handed, in that they affect the whole app or system to which they are applied. As mentioned in the documentation, the list of Preset time ranges can be modified via editing the times.conf file, either via the CLI or through the WebGUI, at Settings > User Interface > Time Ranges. The other sections of the time range picker cannot be modified in this way however.

Using CSS to hide time range picker elements

An alternative to this approach is to use CSS to hide the time range picker elements that should not be available. A CSS file located in <app>/appserver/static/ can be included in a custom dashboard or form with the line <form stylesheet="filter.css">.

The following CSS provides a template for identifying and filtering the sections and presets that are available by default.

/* Hide whole sections 
 */
div[id^='presets_view'],
div[id^='relative_view'],
div[id^='realtime_view'],
div[id^='daterange_view'],
div[id^='dateandtimerange_view'],
div[id^='advanced_view'] {
	display: none;
}

/* Hide individual items 
 */
a[data-earliest="@d"][data-latest="now"], /* Today */
a[data-earliest="@w0"][data-latest="now"], /* Week to date */
a[data-earliest="@w1"][data-latest="now"], /* Business week to date */
a[data-earliest="@mon"][data-latest="now"], /* Month to date */
a[data-earliest="-1d@d"][data-latest="@d"], /* Yesterday */
a[data-earliest="-7d@w0"][data-latest="@w0"], /* Previous week */
a[data-earliest="-6d@w1"][data-latest="-1d@w6"], /* Previous business week */
a[data-earliest="-1mon@mon"][data-latest="@mon"], /* Previous month */
a[data-earliest="-15m"][data-latest="now"], /* Last 15 minutes */
a[data-earliest="-60m@m"][data-latest="now"], /* Last 60 minutes */
a[data-earliest="-4h@m"][data-latest="now"], /* Last 4 hours */
a[data-earliest="-24h@h"][data-latest="now"], /* Last 24 hours */
a[data-earliest="-7d@h"][data-latest="now"], /* Last 7 days */
a[data-earliest="-30d@d"][data-latest="now"] { /* Last 30 days */
	display: none;
}

The benefit of this approach is that entire sections can be hidden, and this restriction can be done on a per-dashboard level, rather than at a per-app level.

The downsides are that this is only works for custom dashboards and forms, and that these presets are just masked – a crafty user could disable this CSS and select these hidden items. However, Splunk provides sufficient restrictions as part of its RBAC model to limit users from doing long running or all time searches, if required.

You can grab an example app that demonstrates this configuration from https://github.com/sduff/restrict_timepicker.

Splunk

Wed, 21 Jun 2017 00:00:00 +0000

Splunk

I’ve been using Splunk for almost 10 years, after first being introduced to it as a Security tool while I was working at a major Australian bank. So impressed with the product, I joined the company, and have been a Professional Services consultant with them ever since.

What the Splunk?

We Make Machine Data Accessible, Usable and Valuable to Everyone

Splunk provides a mechanism for the collection of machine data information and events, with a robust storage and analysis engine. Sitting on top of this is a web front end that allows rapid exploration and development of dashboards and alerts. The whole framework allows easy integration of machine learning, custom visualisations and automated responses.

We also have the best T-shirts 😉

Splunk Web Resources

Splunk Homepage – Splunk Homepage
Splunk Answers – Questions and Answers!
SplunkBase – Thousands of Splunk apps
/r/Splunk – Splunk’s home on reddit
GoSplunk – Popular Splunk Queries
Big Book Of Splunk – Even more Splunk Queries
Splunk Sizing Calculator – Splunk Sizing Calculator for indexes.conf

Splunk Blogs & Repositories

Splunk Blogs – Official Splunk Blog Posts
Ryan Faircloth
Anthony Tellez
Duane Waddle
Vladimir’s GitHub
David Veuve

Network Connectivity with Python

Wed, 07 Jun 2017 00:00:00 +0000

Using Python for testing network connectivity

More and more Linux distributions have stopped shipping with basic networking tools such as telnet installed by default. ping is still available (sometimes), but to test if a server can communicate with another server on a particular port is now a challenge.

However, I find python is installed more-often-than-not, and ships with networking libraries that one can use for connectivity purposes. The following snippet checks for connectivity, in this case, HTTPS (443) to google.com.

$ python
import socket
socket.socket().connect(("google.com",443))

Run it, and you’ll either get an exception thrown when connectivity fails, or no exception for a successful connection.

N.B. this won’t confirm application firewalls won’t block traffic, but as a basic connectivity test, this works fine.

Here is also a snazzy one liner to make a bash function if you need to run a large number of checks

c() { python <<<"import socket; s=socket.socket(); s.settimeout(5); s.connect(('$1',$2)) ; print 'Connection'"; }

And then use it in the following manner.

$ c google.com 443
$ c google.com 22

If you’re testing an internal network, set the timeout to something lower than 3 seconds, while if you’re testing a WAN or internet server, set it to a higher value.

Connectivity testing with Splunk’s inbuilt Python

I normally need to do connectivity tests as part of installing Splunk, which usually ships with python (although not with the Universal Forwarder). The python interpreter can be accessed from the command line with

/opt/splunk/bin/splunk cmd python

So the one liner becomes

c() { /opt/splunk/bin/splunk cmd python <<<"import socket; s=socket.socket(); s.settimeout(5); s.connect(('$1',$2)) ; print 'Connection'"; }

A simple REST server in Python

Tue, 18 Apr 2017 00:00:00 +0000

A simple REST server in Python

Using Python’s BaseHTTPServer, its easy to build a very simple REST server, very handy for prototyping. In no way should this be used in a production environment!

When visiting, via GET a page that doesn’t exist, a very simple upload page is presented. Revisiting the page will cause that file to be downloaded or viewd. You need to manually (eg, using CURL) POST to the same page to upload a new file to the same endpoint.

All data is stored in memory, and is deleted when the process stops. Like I said, its not for production! If you want some default or consistent data present whenever you start the server, you can include it in the endpoints definition in main.

The reason for the large number of headers is to handle CORS. The settings in the file make the REST endpoint fairly accessible from anywhere.

#!/usr/bin/env python

# Basic REST Server for prototyping

import sys, cgi, BaseHTTPServer, mimetypes

# Port to listen on
PORT = 1024

# List of endpoints
class endpoint:
    mimetype = ""
    content = ""
    def __init__(self,mimetype,content):
        self.mimetype = mimetype
        self.content = content
    
endpoints = {}

# The REST handler
class REST(BaseHTTPServer.BaseHTTPRequestHandler):

    def do_OPTIONS(self):
        self.send_response(200)
        self.send_header("Access-Control-Allow-Origin","*")
        self.send_header("Access-Control-Allow-Methods","*")
        self.send_header("Access-Control-Allow-Headers","*")
        self.end_headers()

    def do_HEAD(self):
        self.send_response(200)
        self.send_header("Access-Control-Allow-Origin","*")
        self.send_header("Access-Control-Allow-Methods","*")
        self.send_header("Access-Control-Allow-Headers","*")
        self.end_headers()

    def do_GET(self):
        self.path = self.path.split("?",1)[0]
        if self.path in endpoints:
            ep = endpoints[self.path]
            self.send_response(200)
            self.send_header("Access-Control-Allow-Origin","*")
            self.send_header("Access-Control-Allow-Methods","*")
            self.send_header("Access-Control-Allow-Headers","*")
            self.send_header("Content-Type", ep.mimetype)
            self.end_headers()
            self.wfile.write(ep.content)

        else:
            self.send_response(200)
            self.send_header("Access-Control-Allow-Origin","*")
            self.send_header("Access-Control-Allow-Methods","*")
            self.send_header("Access-Control-Allow-Headers","*")
            self.send_header("Content-Type", "text/html")
            self.end_headers()
            self.wfile.write("""<html>
<body>
<form method="post" enctype="multipart/form-data" action=""")
            self.wfile.write('"'+self.path+'"')
            self.wfile.write(""">
<input type="file" name="upload"/><input type="submit" value="Submit">
</form>
</body>
</html>""")

    def do_POST(self):
        self.path = self.path.split("?",1)[0]
        try:
            ctype, pdict = cgi.parse_header(self.headers.getheader('content-type'))
            form = cgi.FieldStorage( fp=self.rfile, headers=self.headers, environ={'REQUEST_METHOD':'POST', 'CONTENT_TYPE':self.headers['Content-Type'], })
            content = form['upload'].file.read()
            filename = form['upload'].filename

            mt = mimetypes.guess_type(form["upload"].filename)[0] or "application/octet-stream"
            endpoints[self.path] = endpoint(mt,content)

            self.send_response(200)
            self.send_header("Access-Control-Allow-Origin","*")
            self.send_header("Access-Control-Allow-Methods","*")
            self.send_header("Access-Control-Allow-Headers","*")
            self.send_header("Content-Type",mt)
            self.end_headers()
            self.wfile.write(content)
        except Exception as e:
            print sys.exc_info()
            self.send_response(500)
            self.send_header("Access-Control-Allow-Origin","*")
            self.send_header("Access-Control-Allow-Methods","*")
            self.send_header("Access-Control-Allow-Headers","*")
            self.end_headers()
            self.wfile.write(e)


if __name__ == '__main__':

    endpoints = {   "/": endpoint("text/plain","Hello world!"),
            "/bye.html": endpoint("text/html","<html><body>Bye</body></html>") 
    }

    print repr(endpoints)

    httpd = BaseHTTPServer.HTTPServer(("",PORT), REST)
    try:
        httpd.serve_forever()
    except KeyboardInterrupt:
        pass
    httpd.server_close()

We can use curl for testing:

$ curl -F "upload=@./simon_duff.html" localhost:1024/owner

References

https://www.spock.li/tutorials/rest-api

About Simon Duff

Sun, 01 Jan 2017 00:00:00 +0000

About Simon Duff

Simon was born and raised in Melbourne, Australia. He now spend much of his time traveling between various Australian cities for work, and Singapore, to spend time with his wife and children.

He is a Professional Services Consultant for a leading Big-Data company, which allows him to work with many large enterprises. Each client has their own unique set of use-cases and challenges, which provides ample room for creative and practical solutions.

Simon was inspired to start blogging about his work as an aid to help demystify big-data, both in terms of its capabilities and limitations.

In a former life, Simon was a Computer Security Consultant. Prior to that, he was pursuing research in Artificial Intelligence, leading to him obtaining a Masters in Applied Science.

For more details about Simon’s current activities, see his now page.

You can find Simon elsewhere on the web:

Github
LinkedIn
KeyBase
Facebook
Google+ rarely used
Twitter rarely used

Simon is done talking in the third person 😉

Simon Duff

Ho Ho Ho, time to go

Visualising Service Status with Emoji

Visualising Service Status with Emoji

Using Emoji in your Splunk query

Example dashboard

References

Now

What I’m Doing Now

Location

Current Focus

vim commands for bulk incrementing

vim commands for bulk incrementing

Explanation

Know a better sort? Please let me know!

References

Hammerspoon Configuration

Hammerspoon

Caps Lock

Karabiner-Elements

Hammerspoon Configuration

Download

References

Invisilink

Enough vim to be dangerous

Enough vim to be dangerous

vim tips, tricks and shortcuts

My local .vimrc

References

Processing Excel XLSX files with Python

Processing Excel XLSX files with Python

References

Simulating the Monty Hall Problem

Monty Hall Problem simulated in Python

The Monty Hall Problem

Solution

Python Simulation

Operation

Results

Observations

Future ideas

Sitemap pings for GitHub-hosted Jekyll sites

Sitemap pings for GitHub-hosted Jekyll sites

What is a sitemap?

Generating a sitemap with Jekyll

Github webhooks

Colophon

Colophon

Site Generation and Hosting

Theme

Build

Sitemap pings

Analytics

Affiliate Links

Disclaimer

Splunk Alert Script Template

Splunk Alert Script

Alert Script Arguments

Alert Script Code

Advanced Alert Actions

Alert Script ideas

Restricting the Splunk Time Range Picker

Restricting the Splunk Time Range Picker

Using CSS to hide time range picker elements

Splunk

Splunk

What the Splunk?

Splunk-related pages on this site

Splunk Web Resources

Splunk Blogs & Repositories

Network Connectivity with Python

Using Python for testing network connectivity

Connectivity testing with Splunk’s inbuilt Python

A simple REST server in Python

A simple REST server in Python

References

About Simon Duff

About Simon Duff

`vim` commands for bulk incrementing

Enough `vim` to be dangerous

`vim` tips, tricks and shortcuts

My local `.vimrc`