Eikonal Blog

2012.06.25

More unix tools

Filed under: scripting, unix — Tags: , , , , , , — sandokan65 @ 13:32
  • Joye’s “moreutils” collection – http://joeyh.name/code/moreutils/. Contains following:

    • chronic: runs a command quietly unless it fails
    • combine: combine the lines in two files using boolean operations
    • ifdata: get network interface info without parsing ifconfig output
    • ifne: run a program if the standard input is not empty
    • isutf8: check if a file or standard input is utf-8
    • lckdo: execute a program with a lock held
    • mispipe: pipe two commands, returning the exit status of the first
    • parallel: run multiple jobs at once
    • pee: tee standard input to pipes
    • sponge: soak up standard input and write to a file
    • ts: timestamp standard input
    • vidir: edit a directory in your text editor
    • vipe: insert a text editor into a pipe
    • zrun: automatically uncompress arguments to command

  • num-utils – http://suso.suso.org/programs/num-utils/. Contains:

    • average: A program for calculating the average of numbers.
    • bound: Finds the boundary numbers (min and max) of input.
    • interval: Shows the numeric intervals between each number in a sequence.
    • normalize: Normalizes a set of numbers between 0 and 1 by default.
    • numgrep: Like normal grep, but for sets of numbers.
    • numprocess: Do mathmatical operations on numbers.
    • numsum: Add up all the numbers.
    • random: Generate a random number from a given expression.
    • range: Generate a set of numbers in a range expression.
    • round: Round each number according to it’s value.

  • Scylla and Charybdis, Tools – http://www.scylla-charybdis.com/tool.php. Contains:

    • checkrun: Program watchdog to terminate a program with starving output)
    • cmpfast: Fast compare two files binary)
    • count: Copy lines, shows progress)
    • dbm: A little tool to access gdbm files from shell.)
    • dirlist: Primitive directory lister, quicker than ls, find and echo *)
    • diskus: Disk geometry checking and repair tool)
    • getrealpath: Print realpath to stdout)
    • histogram: Count bytes in file)
    • kdmktone: Make the console beep)
    • keypressed: Nonblocking, nondestructible test for waiting data on TTYs, sockets and probably pipes)
    • killmem: Protect some memory against paging until you need free memory)
    • lockdir: Create a directory for locking purpose)
    • lockrun: Exclusively run something by placing a file lock)
    • md5chk: Create md5sums for easy shell usage)
    • minicron: This is a program which starts other programs after some time)
    • mvatom: Move files by atomic rename instead of copy.)
    • printansi: Like /bin/echo but ANSI-escapes the output)
    • printargs: Like “hello world” but dumps the argc array)
    • ptybuffer: daemonize interactive tty line driven programs with output history)
    • runningfor: Return true until the given time periode is reached.)
    • slowdown: Slowdown processes or pipes.)
    • socklinger: Execute quick hack shell scripts connected to a socket.)
    • sq: SQLITE3 query tool for shell usage)
    • timeout: Execute a command or pipe only for a given duration.)
    • timestart: Start a program N-M times in parallel by running it it each A-B seconds)
    • tinohtmlparse: Simple HTML parser to extract information from HTML files by shell)
    • tinoseq: An integer seq implementation)
    • udevraw: Dump udev events for bash usage)
    • unbuffered: Copy stdin to stdout and stderr, unbuffered)
    • watcher: A Python2.6 script to watch files, pipes or Unix domain sockets)

2012.02.14

OpenSSL

Filed under: crypto, hashes, infosec, scripting, security testing, tools, VA (Vulnerability Assessment), web security, web tools — Tags: , , , — sandokan65 @ 11:45
  • HTTPS server banner: 

      openssl s_client -connect:IPAddress:443

    after connection is established, type in “HEAD / HTTP/1.0” and press enter.

    Alternative:

      echo -e "HEAD / HTTP/1.0\n\n" | openssl s_client -quiet -connect IPAddress:443

  • NTTPS server banner 

      openssl s_client -connect:IPAddress:563

  • IMAPS server banner: 

      openssl s_client -connect:IPAddress:993

  • POP3S server banner: 

      openssl s_client -connect:IPAddress:995

  • Identifying SSL cyphers: 

      openssl s_client -connect website:443 -cipher EXPORT40
openssl s_client -connect website:443 -cipher NULL
openssl s_client -connect website:443 -cipher HIGH

  • Generating password hash four unix: 

      openssl passwd -1 -salt QIGCa pippo

    output: $1$QIGCa$/ruJs8AvmrkmzKTzM2TYE.

  • Converting a PKCS12-encoded (or .pfx) certificate to PEM format: 

      openssl pkcs12 -in CertFile.p12  -out NewCertFile.pem   -nodes. -cacerts

  • Converting a DER-encoded certificate to PEM format: 

      openssl x509  -in CertFile.crt.  -inform DER  -out NewCertName.pem   -outform PEM

  • Download a proxy’s public certificate: 

      openssl s_client-connect ProxyHostname:port   proxycert.pem

  • Create a key: 

      openssl genrsa -des3 -out server.key 1024

  • Create a CSR (certificate signing request): 

      openssl req -new -key server.key -out server.csr

  • Remove a password from a key: 

      cp server.key server.key.org
openssl rsa -in server.key.org -out server.key

  • Sign the CSR and create the certificate: 

      openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt
cat server.crt server.key > certificate.pem

  • Encrypting a file: 

      cat INFILE | openssl aes-256-ecb -salt -k PASSWORD > INFILE.ssl

  • Decrypting a file: 

      cat INFILE.ssl | openssl aes-256-ecb -d -k PASSWORD > INFILE

2011.09.30

awk

Filed under: scripting, unix — Tags: , — sandokan65 @ 11:03

Passing shell variables to AWK

Thing that works well for me: 

    awk '{print "'"$VARIABLE"'"}' 1 > 2

Related here: Scripting languages – https://eikonal.wordpress.com/2010/06/15/awk-sed/ | Unix tricks – https://eikonal.wordpress.com/2011/02/15/unix-tricks/ | SED tricks – https://eikonal.wordpress.com/2010/10/05/sed-tricks/ | Memory of things disappearing > nmap stuff > getports.awk – https://eikonal.wordpress.com/2010/06/23/memory-of-things-disappearing-nmap-stuff-getports-awk/

2011.02.25

Excel files processing

Filed under: perl, scripting, transformers — Tags: — sandokan65 @ 23:55

Related here: Excel sortIP macro – https://eikonal.wordpress.com/2012/02/07/excel-sortip-macro/ | Excel to text – https://eikonal.wordpress.com/2011/02/14/excel-to-text/ | IT tips pages – https://eikonal.wordpress.com/2010/02/08/it-tips-pages/

2011.02.14

Excel to text

Filed under: perl, scripting, tools, transformers — Tags: , — sandokan65 @ 14:11
  • excel2txt (by Ken Youens-Clark) – http://search.cpan.org/~kclark/excel2txt/excel2txt [Perl] – convert Excel data to delimited text files
      • Example use: 

      > excel2txt PasswdFiles.xls
Processing PasswdFiles.xls
Writing 'passwdfiles-fiapp1.txt'
Writing 'passwdfiles-fiapp2.txt'
Writing 'passwdfiles-fiapp3.txt'
Writing 'passwdfiles-fiapp4.txt'
Writing 'passwdfiles-fiapp6.txt'
Writing 'passwdfiles-fiapp7.txt'
Writing 'passwdfiles-fiapp8.txt'
Writing 'passwdfiles-fiapp9.txt'
Writing 'passwdfiles-fidb1.txt'
Writing 'passwdfiles-fidb2.txt'
Writing 'passwdfiles-fidb3.txt'
Writing 'passwdfiles-fidb4.txt'
Writing 'passwdfiles-fiweb1.txt'
Writing 'passwdfiles-fiweb2.txt'
Writing 'passwdfiles-fiweb3.txt'
Writing 'passwdfiles-fiweb4.txt'
Done, processed 1 Excel file, created 16 data files.

Related here: Excel sortIP macro – https://eikonal.wordpress.com/2012/02/07/excel-sortip-macro/ | Excel files processing – https://eikonal.wordpress.com/2011/02/25/excel-files-processing/ | IT tips pages – https://eikonal.wordpress.com/2010/02/08/it-tips-pages/

2010.10.05

sed tricks

Filed under: scripting, transformers — Tags: , , — sandokan65 @ 15:58

These one-liners are collected from various sites and articles on web – see the list of Sources at the bottom of this posting.

  • Deleting all empty lines from the input file: 
    sed ‘/^$/d’
  • In-place replacement: 
    sed –i ‘/^$/d’ INPUTFILE
  • In-place replacement with backup of original file: 
    sed –ibak ‘/^$/d’ INPUTFILE
  • In-place deletion of all occurences of a string in a file: 
    sed –i ‘/WORDTOBEDELETED/d’
  • How to replace the first occurrence only (of a string match) in a file, using sed 
    sed '0,/THISSTRING/s//TOTHATSTRING/' INPUTFILE
  • Append environment variable PATH with sed: 
    sed -e '/^PATH/s/"$/:\/usr\/lib\/myprog\/bin"/g' -i /etc/environment
  • Remove all whitespace from beinning of lines: 
    sed 's/^[ \t]*//g' foo
  • Deleting the / from all html files contained in current folder: 
    sed -i ‘s/src=”\//src=”/g’ *.html
  • Greedy matching: 
    % echo "foobar" | sed 's///g'
bar
  • Non greedy matching: 
    % echo "foobar" | sed 's/]*>//g'
foobar

Sources:

References

Related here: Command line based text replace – https://eikonal.wordpress.com/2010/07/13/command-line-based-text-replace/.

Related here: Scripting languages – https://eikonal.wordpress.com/2010/06/15/awk-sed/ | Unix tricks – https://eikonal.wordpress.com/2011/02/15/unix-tricks/ | SED tricks – https://eikonal.wordpress.com/2010/10/05/sed-tricks/ | Memory of things disappearing > nmap stuff > getports.awk – https://eikonal.wordpress.com/2010/06/23/memory-of-things-disappearing-nmap-stuff-getports-awk/ | AWK – https://eikonal.wordpress.com/2011/09/30/awk/

2010.08.03

Powershell

Filed under: scripting, windows — Tags: , , , , , — sandokan65 @ 14:32

Portable Powershell:

2010.07.06

googlecl

Filed under: scripting, web tools — Tags: — sandokan65 @ 16:50

A tool that instruments some of the Google services to the unix command line.

Installation:

Examples

Following list is taken verbatim from the project page (http://code.google.com/p/googlecl/wiki/ExampleScripts). Idea is to grow this list as I find further tricks with this toolkit. Here it goes::

  • Blogger:
    • google blogger post –tags “GoogleCL, awesome” –title “Test Post” “I’m posting from the command line”
    • google blogger post blogpost.txt
    • google blogger list title,url-site # List posts
    • google blogger delete –title “Test Post”
    • google delete –title “Silly post number [0-9]*” # Delete posts matching regex
    • google tag –title “Dev post” –tags “Python, software” # label an existing post
  • Calendar:
    • google calendar add “Dinner party with George today at 6pm” # add event to calendar
    • google calendar today # List events for today only.
    • google calendar list –date 2010-06-01,2010-06-30 # List events.
    • google calendar delete –title “Dinner party with George” # Delete an event.
    • google calendar today list –cal .* | egrep ‘\[.*\]’ # List all calendars
  • Contacts:
    • google contacts add “J. Random Hacker, jrandom@example.com”
    • google contacts list name,email –title “J. Random Hacker”
    • google contacts delete –title “J. Random Hacker”
  • Docs:
    • google docs delete –title “Evidence”
    • google docs list title,url-direct –delimiter “: ” # list docs
    • google docs upload the_bobs.csv ~/work/docs_to_share/*

    gdata-python-client >= 1.3.0 ONLY:

    • google docs edit –title “Shopping list” –editor vim
    • google docs get –title “Homework [0-9]*”
  • Picasa:
    • google picasa create –title “Vermont Test” –tags Vermont vermont.jpg
    • google picasa get –title “Vermont Test” /path/to/download/folder
    • google picasa list title,url-direct –query “A tag”
    • google picasa post –title “Vermont Test” ~/old_photos/*.jpg # Add to an album
    • google picasa tag –title “Vermont Test” –tags “places”
    • google picasa delete –title “Vermont Test” # delete entire album
  • Youtube:
    • google youtube post –category Education –devtags GoogleCL killer_robots.avi
    • google youtube delete –title “killer_robots.avi”
    • google youtube list # list my videos
    • google youtube tag -n “.*robot.*” –tags robot

2010.06.15

Scripting languages

Filed under: scripting, unix — Tags: , , , , , , — sandokan65 @ 14:35

Unix shells:

Windows Scripting:

Multi-language pages:

Compiling shell scripts

Can not be done.

Related here: Unix tricks – https://eikonal.wordpress.com/2011/02/15/unix-tricks/ | SED tricks – https://eikonal.wordpress.com/2010/10/05/sed-tricks/ | Memory of things disappearing > nmap stuff > getports.awk – https://eikonal.wordpress.com/2010/06/23/memory-of-things-disappearing-nmap-stuff-getports-awk/ | AWK – https://eikonal.wordpress.com/2011/09/30/awk/

2010.04.02

Regular expressions

Filed under: regexp, scripting, unix — Tags: , , , , , , , , , , , , , , , , — sandokan65 @ 14:52

Sites

Tools

Standalone tools:

Online testers:

Books

Tidbits

Sources: The above links.

  • [abc] – A single character: a, b or c
  • [^abc] – Any single character but a, b, or c
  • [a-z] – Any single character in the range a-z
  • [a-zA-Z] – Any single character in the range a-z or A-Z
  • ^ – Start of line
  • $ – End of line
  • \A – Start of string
  • \z – End of string
  • . – Any single character
  • \s – Any whitespace character
  • \S – Any non-whitespace character
  • \d – Any digit
  • \D – Any non-digit
  • \w – Any word character (letter, number, underscore)
  • \W – Any non-word character
  • \b – Any word boundary character
  • (…) – Capture everything enclosed
  • (a|b) – a or b
  • a? – Zero or one of a
  • a* – Zero or more of a
  • a+ – One or more of a
  • a{3} – Exactly 3 of a
  • a{3,} – 3 or more of a
  • a{3,6} – Between 3 and 6 of a
  • ^\s[ \t]*$ – Match a blank line
  • \d{2}-\d{5} – Validate an ID number consisting of 2 digits, a hyphen, and another 5 digits

Special common strings:

  • Personal Name: ^[\w\.\’]{2,}([\s][\w\.\’]{2,})+$
  • Username: ^[\w\d\_\.]{4,}$
  • Password at least 6 symbols: ^.{6,}$
  • Password or empty input: ^.{6,}$|^$
  • email: ^[\_]*([a-z0-9]+(\.|\_*)?)+@([a-z][a-z0-9\-]+(\.|\-*\.))+[a-z]{2,6}$
  • Email address: \b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b[A-z0-9_.%+-]+@[A-z0-9_.%+-]+\.[A-z]{2,4}
  • US phone: \W?\d{3}\W?\d{3}\W?\d{4}
  • US Phone number: ^\+?[\d\s]{3,}$
  • US Phone with code: ^\+?[\d\s]+\(?[\d\s]{10,}$
  • URL: \W?\d{3}\W?\d{3}\W?\d{4}\b\w+://(\w|-|\.|/)+(/|\b)
  • US Social Security Number (SSN): \d{3}-\d{2}-\d{4}
  • US ZIP: \d{5}(-\d{4})?
  • IP (v4) address: \b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b
  • IP (v4) address: \b(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\b
  • IP (v4) address: ^(\d|[1-9]\d|1\d\d|2[0-4]\d|25[0-5])\.(\d|[1-9]\d|1\d\d|2[0-4]\d|25[0-5]){3}$
  • IP (v4) address: \b(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\b
  • IP (v4) address: \b(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\b
  • IP (v6) address:
  • MAC address: ^([0-9a-fA-F][0-9a-fA-F]:){5}([0-9a-fA-F][0-9a-fA-F])$
  • Positive Integers: ^\d+$
  • Negative Integers: ^-\d+$
  • Integer: ^-{0,1}\d+$
  • Positive Number: ^\d*\.{0,1}\d+$
  • Negative Number: ^-\d*\.{0,1}\d+$
  • Positive Number or Negative Number: ^-{0,1}\d*\.{0,1}\d+$
  • Floating point number: [-+]?([0-9]*\.[0-9]+|[0-9]+)
  • Floating point number: [-+]?(?:\b[0-9]+(?:\.[0-9]*)?|\.[0-9]+\b)(?:[eE][-+]?[0-9]+\b)?
  • Roman number: ^(?i:(?=[MDCLXVI])((M{0,3})((C[DM])|(D?C{0,3}))?((X[LC])|(L?XX{0,2})|L)?((I[VX])|(V?(II{0,2}))|V)?))$
  • Domain Name: ^([a-zA-Z0-9]([a-zA-Z0-9\-]{0,61}[a-zA-Z0-9])?\.)+[a-zA-Z]{2,6}$
  • Domain Name: ^([a-z][a-z0-9\-]+(\.|\-*\.))+[a-z]{2,6}$
  • Windows File Name: (?i)^(?!^(PRN|AUX|CLOCK\$|NUL|CON|COM\d|LPT\d|\..*)(\..+)?$)[^\\\./:\*\?\”\|][^\\/:\*\?\”\|]{0,254}$
  • Date in format yyyy-MM-dd: (19|20)\d\d([- /.])(0[1-9]|1[012])\2(0[1-9]|[12][0-9]|3[01])
  • Date (dd mm yyyy, d/m/yyyy, etc.): ^([1-9]|0[1-9]|[12][0-9]|3[01])\D([1-9]|0[1-9]|1[012])\D(19[0-9][0-9]|20[0-9][0-9])$
  • Year 1900-2099: ^(19|20)[\d]{2,2}$

Related (here at this blog):
Command line based text replace – https://eikonal.wordpress.com/2010/07/13/command-line-based-text-replace/ |
Perl online – https://eikonal.wordpress.com/2010/02/15/perl-online/

Create a free website or blog at WordPress.com.

Design a site like this with WordPress.com
Get started