Data Usability

Buzz words are always overused and Big Data is defiantly one of them. Don’t mind how vendors use it.

The word big has always bothered me as it does not describe what the real intend is.

The author in this article really describes it well. “The basic idea behind the phrase ‘Big Data’ is that everything we do is increasingly leaving a digital trace (or data), which we (and others) can use and analyse. Big Data therefore refers to that data being collected and our ability to make use of it.”

The most important part is the last part, our ability to make use of it irrespective of data source, format etc. The tools, technics, end user enablement, diversity of data, and patterns in Big Data is really what’s new. Which doesn’t mean you throw away what you have but rather make it more usable.

So it’s more usability of data so maybe we should call it Data Usability. Then again we might be introducing more buzz vs. just understanding intent. 

Big Data explained

How to use Scripted Inputs in Splunk with AllSigned Execution Policy

Authors:

Henri van den Bulk, MS, CS, is a Technology Evangelist and Enterprise Architect at Denver Water, where he’s responsible for working with numerous stakeholders and application Scrum teams to evolve IT’s Enterprise Technology and Architecture.  Additionally he’s responsible for Leading their Enterprise Integration Service Team, where he’s optimizing business processes through integration and exposing business and operational data assets for analytics.

Prior to Denver Water he served as President/Enterprise Architect of Actie Consulting where he worked with national and international clients such as NASA, in a broad ranges of verticals.”

Tony Nesavich currently works for Denver Water.  Tony earned his B.S. in Business Management from the University of Puget Sound (1999) and his M.S. in IT Management from Regis University (2004).  Tony has a long and proven track record of solving business and IT problems with MS PowerShell.”


Article:

We’ve found that using scripted inputs helps us to capture specific sources in Windows through the use of PowerShell. Powershell can introduce security complications when execution on machines is not managed. To enhance this security we tend to choose a script execution policy that requires the scripts to be signed.

This article describes how to leverage a more aggressive and secure execution policy setting called AllSigned. Scripts provided by Splunk and others don’t support this policy out of the box, we’ll highlight here what you can do as an organization to address this. Additionally, when developing custom inputs as a developer you’ll have to take this policy into account.

The recipe presented in this article walks through the steps necessary to use PowerShell scripts, as scripted inputs for Splunk, and use the AllSigned execution policy.

In order to use the AllSigned Execution Policy all scripts must be signed with a certificate in the Trusted Publishers.  Common issues that can result if scripts are improperly signed (or not signed at all) are the dreaded ‘AuthorizationManager check failed.‘ message in the powershell.log and many other headaches.

At a high level, regardless of your environment, the essence of the process involves the following elements.

  • Certificates
    • Create and distribute the necessary certificates to the Trusted Publishers for the computer account.  They will be needed / used for validating scripts signed by them on any system running them.
  • Scripted Inputs
    • Sign & deploy PowerShell scripted inputs
  • Splunk Addon for Microsoft PowerShell
    • Sign & deploy the LocalStorage.psm1 script as along with the rest of the Splunk Addon for Microsoft PowerShell to the systems that will run them.

The detailed step-by-step portion in the following sections are intended to serve as an example of the concepts bulleted above.

Certificates

This only needs to be done once (and renewed when expired) and consists of the steps below.

Open Certificate Manager

  • Start –> Type “MMC” –> Enter –> File, Add/Remove Snap-in… –> Certificate –> Add

image001

  • Navigate to Certificates –> Current User –> Personal –> Certificates

image002

  • Create Certificate
    • Right-click an blank area in the right hand column and choose All Tasks –> Request New Certificate…

image003

  • Click Next on the “Before You Begin” Welcome window, –> Accept the defaults (Active Directory Enrollment Policy) and click Next –> Select “PowerShell Code Signing Template 5 Years” and click Enroll

image004

Export & Deploy Certificate

Export certificate as Base64-encoded x509 (.cer) format.
Ref:
http://technet.microsoft.com/en-us/library/cc779668(v=ws.10).aspx
http://msdn.microsoft.com/en-us/library/bb886972(v=bts.10).aspx
http://technet.microsoft.com/en-us/library/cc770735.aspx

Deploy certificate to the Trusted Publishers on the system(s) that will be running them via group policy and or manually

GPO Deployment
Configure GPO to set Execution Policy to AllSigned at the MachinePolicy Scope level on all Laptops, Desktops and Servers expected to run PowerShell Scripts.
http://technet.microsoft.com/en-us/library/cc770315%28v=ws.10%29.aspx

Manually:

Copy the certificate to each system that needs it and change directories to where you copied the cert in the PowerShell ISE or Shell and run the following command:

certmgr.exe -add <certname>.cer -c -s -r localMachine TrustedPublisher

Ref:
http://msdn.microsoft.com/en-us/library/ms172241.aspx

Sign your Scripts

After completing the steps above you will now be able to use the resulting certificate to sign your PS scripts with.  Your new cert should look like the one below (only with your name).  The steps involved in signing your scripts are outlined below.

Open PowerShell ISE & Sign your Scripts

  • Start –> Type “Windows PowerShell ISE” –> Enter –> In the white Script Pane on the top part of the console paste the following:
$cert = @(gci Cert:\CurrentUser\My -CodeSigningCert)[0]
Set-AuthenticodeSignature "*.ps1" $cert -TimestampServer "http://timestamp.comodoca.com/authenticode"
  • In the blue console section change directories to where your scripts are then run the script above to sign the script(s) in the present working directory of the console.
  • The -TimestampServer option is important because it will allow scripts that have been signed with trusted certificates to run after the cert expires if they were signed before the certificate expired.
  • If you receive an error like the one below you will have to type in the information in the code box above:
    • Set-AuthenticodeSignature : Cannot bind parameter ‘Certificate’. Cannot convert value “[Subject]

Scripted Inputs

Scripted inputs are essentially inputs that directly ingest data into Splunk and/or create output that gets ingested into Splunk.  The “netsh” example below is the later but there is a lot of pre-existing documentation on this subject and readers are encouraged to also read the following:
http://docs.splunk.com/Documentation/Splunk/6.2.0/AdvancedDev/ScriptedInputsIntro
http://docs.splunk.com/Documentation/Splunk/6.2.0/AdvancedDev/ScriptSetup

Netsh Inputs

The NetSH example application is developed to capture information about Mobile Broadband Modems that are available in Windows Laptops. This data is useful to understand the versions and communication information. For example, the information provided can allow for identifying communication issues due to roaming or signal strength issues.

This application uses scripted inputs to generate different log files that are then ingested into Splunk. The log files are created by executing the Windows NetSh command through Powershell. This is purely for instructional purposes only as we could have solved this issue in a number of different approaches. The code is available on our BitBucket site.

The following scripts run at different schedules as controlled by the inputs.conf in /%SplunkHome%/Etc/Apps/<Application>/bin.  The contents of these scripts are listed below:

  • Capability.ps1 – Provides the basic information about the capabilities of the Mobile Broadband Modem
# Capability.ps1 --> Run once Daily
netsh mbn show capability interface='Mobile Broadband Connection' > "C:\Windows\Temp\Logs\capability_interface.txt"

Example Output:

Device capability information for interface Mobile Broadband Connection: 
-------------------------------------------------------
    Device type                  : Mobile Broadband device is embedded in the system
    Cellular class               : CDMA
    Voice class                  : No voice
    Sim class                    : SIM is logical or embedded in the device
    Data class                   : 1xRTT, 1xEVDO, 1xEVDO REVA
    CDMA band class              : 800 MHz, 1900 MHz
    SMS capability               : PDU receive, PDU send
    Control capability           : Hardware radio switch functionality available
                                   Configured for mobile IP with simple IP as fallback
  • ConnectionRadio.ps1 – Provides Connection information for the radio
# ConnectionRadio.ps1 --> Run every 5 minutes
netsh mbn show connection interface='Mobile Broadband Connection' > "C:\Windows\Temp\Logs\connection_interface.txt"
netsh mbn show radio interface='Mobile Broadband Connection' > "C:\Windows\Temp\Logs\radio_interface.txt"

Example Output:

Connection information for interface Mobile Broadband Connection: 
-------------------------------------
    Interface State       : Connected
    Register State        : Home
    Register Mode         : Automatic
    Provider Name         : Verizon Wireless
    Provider Id           : 00000
    Provider Data Class   : 1xEVDO REVA
    Number of connections : 1
      Serial number #1
        Connection Id         : 11
        Access Point Name     :
  • Interfaces.ps1
# Interfaces.ps1 --> Run every hour
netsh mbn show interfaces > "C:\Windows\Temp\Logs\interfaces.txt"

Example Output:

There is 1 interface on the system: 
    Name               : Mobile Broadband Connection
    Description        : Qualcomm Gobi 2000 HS-USB Mobile Broadband Device 250F
    GUID               : {90EDEA92-ECD6-46CF-8B53-151A77D40DC2}
    Physical Address   : xx:xx:xx:xx:xx:xx
    State              : Connected
    Device type        : Mobile Broadband device is embedded in the system
    Cellular class     : CDMA
    Device Id          : xxxxxxxxxxxxx
    Manufacturer       : Qualcomm Incorporated
    Model              : Qualcomm Gobi 2000
    Firmware Version   : 09010091
    Provider Name      : Verizon Wireless
    Roaming            : Not roaming
    Signal             : 77%
  • ReadyinfoSignal.ps1
# ReadyinfoSignal.ps1 --> Run every 15 minutes
netsh mbn show readyinfo interface='Mobile Broadband Connection' > "C:\Windows\Temp\Logs\readyinfo_interface.txt"
netsh mbn show signal interface='Mobile Broadband Connection' > "C:\Windows\Temp\Logs\signal_interface.txt"

Example Output:

Ready information for interface Mobile Broadband Connection: 
------------------------------------- 
State : Ready to power up and register 
Emergency mode : Off 
Subscriber Id : xxxxxxx 
SIM ICC Id : xxxxxxx 
Number of telephone numbers : 
1 Telephone #1 : 1720xxxxxx

We then define the scripted inputs and the file monitors.

[powershell://Capability]
 script = . "$SplunkHome\etc\apps\splunk_netsh_inputs\bin\Capability.ps1"
 schedule = 0 * */23 ? * * 
 disabled = false
[monitor://C:\Windows\Temp\Logs\interfaces.txt]
 sourcetype=netsh_interfaces
 index= netsh
 disabled=false
 crcSalt = <SOURCE>


Splunk Add-on for Microsoft PowerShell signing

This is a Splunk Addon that works in conjunction with PowerShell scripted inputs.  Download this from “Splunk Base” here:

https://apps.splunk.com/app/1477/

After downloading this add-on it must be distributed to the systems that are going to run PowerShell scripted inputs.  A crucial step of the deployment process will be to sign the LocalStorage.psm1 found under SA-ModularInput-PowerShell\windows_x86_64\bin\Modules\LocalStorage.  The signing process for this is the same as the example above save for the file extension.

Denver Water Uses Big Data to Improve Efficiency

From time to time I get interviewed for different reasons. I always enjoy sharing my experiences with others in the hope that that might help other grow. in this article my work was featured. We’ve been working hard in improving our efficiencies in many different ways.

Being able to see how users leverage your software is such a valuable capability that many developers forget. We all get excited in implementing the latest wiz bang feature that we get blind sided by how the user really experiences the software. To this I’ve come to believe that being able to understand how this experience is for user is the most important thing to measure.

I’ve been using Splunk to collect, aggregate and correlate how software is being used. It allows me to quickly see trends, hotspots and availability. The later is again a key measure of experience. If your software is not available the experience is not great.

Either way, check out the article and enjoy the reading.

Embedded Code on Sharepoint

If you like me and want to have source code examples or just add snipets of text/code added to a WIKI page then here is a great example of making it look good and make it very shareable.

​Whenever I put documentation together the last thing I want to do is work on lots of formating. Rather use syntax highlighting from an editor or use some great javascript libraries that do the work for you. Now I was working on something where I was leveraging a Sharepoint WIKIs for lots of documentation and it has its limitations in using those capabilities.

The other thing I like to be able to do is share the snippets of code so others can use it. Well there is a great service called Gist from Github. It allows you to share a snippet by posting it on Github and then send someone a link.

I wanted to use this concept with Sharepoint where I post it on Github but then display it on a WIKI in a nice formated way. Well Sharepoint has the ability to add Embedded Code, which is essentialy adding Javascript to a WIKI page. Now Github also provides the ability to generate Embedded Code that will allow you to then paste this into the page. A perfect marriage I must say.

So instead of writing down all the steps look at the video and see how this done.

Embedded Code in Sharepoint

Azure Command Line Interface

Most cloud providers have done a good job at putting together a web ui that allows you to manage the cloud assets such as Virtual Machines (vms). But when trying to manage many vms or just wanting to perform some simple tasks as starting a vm command line interfaces #cli can’t be beat.

#Azure is no exception to this and they have provided a couple of means of manazing your account, assets etc via a #cli. They provide two ways of doing this either through #PowerShell or the cross platform cli based on #nodejs. I choose the later as the prefered mechanism as it’s cross platform and it seems the like the simplest way of using it.

To get started 1st download #nodejs http://nodejs.org/download/ for the platform you’re working on. There are number of prebuild packages but you can build it from source code as well. Next you’ll need to use the node.js package manager #npm to install the azure component:

npm install azure-cli -g

Now that you have installed this part you’re ready to link your account to the cli. You do this by downloading the settings and then importing them. Using the azure command download this will start the download. In which your default browser will be loaded, login to azure with the account you want to link and the download starts.


azure account download

After the file is downloaded you can then import the account by using the import command. Below is the example of the file “Visual Studio U-13-2014-credentials.publishsettings” was downloaded by the browser and using the import command we add it to my profile for the cli.


azure account import "C:\Users\yourtruly\Downloads\Visual Studio U-13-2014-credentials.publishsettings"

Now you’re ready to use the azure commands. Here are some example commands:

List your vms:

azure vms list

Create a Ubuntu vm:

azure vm create

b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-12_04_4-LTS-amd64-server-20140514-en-us-30GB azureuser password –location “West US” -e 22

Check out the references:
http://azure.microsoft.com/en-us/documentation/articles/command-line-tools/#Commands_to_manage_your_Azure_virtual_machines
http://www.hanselman.com/blog/ManagingTheCloudFromTheCommandLine.aspx