On Web Security

Compiling WordNet on Windows to use with Emacs

2019-01-24T10:00:00+00:00

As a non-native English speaker who reads, writes and reviews lots of English texts, I frequently look up definitions as well as synonyms of words. Of course there are numerous online sources available to do this, but I like to decrease my online 'footprint' due to privacy reasons. It also takes extra time to switch to a browser window and enter a search query.

Fortunately the fine folks at Princeton University compiled WordNet [1], a large lexical database of English, which can be used offline - together with a tool to search that database. Even better, somebody wrote a package to use WordNet inside my favorite editor Emacs [2]. This means that just by hovering the cursor over a word inside Emacs, the definition as well as synonyms can be shown. The source code [3] is kindly provided by Princeton University.

Compiling WordNet using MSYS2 on/for Windows

As is usually the case, compiling on/for Windows using the MSYS2 subsystem [4] can be done, with a few minor tweaks.

First, start a MSYS2 shell and install the required dependencies (build tools, as well as the programming language Tcl and its widget toolkit Tk ):

pacman -Sy --noconfirm base-devel mingw-w64-x86_64-tcl mingw-w64-x86_64-tk

Then download the source code [3] as well as the latest database files [5] using wget , and unpack both:

wget http://wordnetcode.princeton.edu/3.0/WordNet-3.0.tar.gz
wget http://wordnetcode.princeton.edu/wn3.1.dict.tar.gz
tar -xzvf WordNet-3.0.tar.gz
cd WordNet-3.0
tar -xzvf ../wn3.1.dict.tar.gz

Run configure...

./configure

...and run into the first hurdle:

WARNING: Can't find Tk configuration definitions

The compiler script warns about not being able to find Tk configuration definitions. Looking at the configure script, the file it is looking for is tkConfig.sh . On MSYS2, this file has been is installed under /mingw64/lib/tkConfig.sh and this location can directly be passed as parameter. The same holds true for the Tck configuration definitions:

./configure --with-tcl='/mingw64/lib/' --with-tk='/mingw64/lib/'

Now the configure script succeeds. The next step, compiling the program unfortunately fails at first run:

make

error: 'Tcl_Interp' {aka 'struct Tcl_Interp'} has no member named 'result'

This hints at an incorrect version of Tcl being expected. Fortunately this can be solved [6] by defining a constant (USE_INTERP_RESULT ) while compiling, either in the source code or on the command line. As it's best not to touch the source, I opt for the command line:

CFLAGS="-DUSE_INTERP_RESULT" ./configure --with-tcl='/mingw64/lib/' --with-tk='/mingw64/lib/'

After this the program compiles successfully on MSYS2, running on Windows 10. The next step now strips the binaries and installs it, ready to be used on the system.

make install-strip

After this, when running the executable wn it still complains about not being able to open data and index files:

WordNet library error: Can't open datafile(/mingw64/dict/data.noun)
WordNet library error: Can't open indexfile(/mingw64/dict/index.noun)
WordNet library error: Can't open datafile(/mingw64/dict/data.verb) WordNet
library error: Can't open indexfile(/mingw64/dict/index.verb) WordNet
library error: Can't open datafile(/mingw64/dict/data.adj) WordNet library
error: Can't open indexfile(/mingw64/dict/index.adj) WordNet library error:
Can't open datafile(/mingw64/dict/data.adv) WordNet library error: Can't
open indexfile(/mingw64/dict/index.adv) WordNet library warning: Can't open
verb example sentence file(/mingw64/dict/sents.vrb) WordNet library warning:
Can't open verb example sentence index file(/mingw64/dict/sentidx.vrb) wn:
Fatal error - cannot open WordNet database

The instructions point out that the environment variable WNHOME needs to be set, pointing to the installation directory. After that, the binary seems to work:

export WNHOME=/mingw64
wn successful -over

Using WordNet with Emacs using helm

Next up is tying WordNet to Emacs. This is a matter of installing the packages helm and helm-wordnet , and setting the variable helm-wordnet-wordnet-location to contain the absolute path to the MSYS2 subsystem folder where WordNet installed its dictionary files. This is the same as the 'virtual' path /mingw64/dict .

Personally I use use-package [7] for all Emacs package management: This code snippet from the startup file .init.el expects that package to be installed and will take care of all the heavy lifting.

(use-package helm
  :ensure t
)

(use-package helm-wordnet
  :after helm
  :config (setq helm-wordnet-wordnet-location "/full/path/to/mingw64/lib")
  :ensure t
  :pin "melpa"                           ;; The package is only available at Melpa for the moment
 )

If everything went well, you can execute the function helm-wordnet-suggest and Emacs will happily show synonyms and a definition.

Mission accomplished...

Note

On most Linux variants, installing WordNet is simply a case of using the native package manager, e.g. on Debian sudo apt-get install wordnet

[1]	https://wordnet.princeton.edu

[2]	https://www.gnu.org/software/emacs/

[3]	(1, 2) http://wordnetcode.princeton.edu/3.0/WordNet-3.0.tar.gz

[4]	https://msys2.github.io/

[5]	http://wordnetcode.princeton.edu/wn3.1.dict.tar.gz

[6]	https://www.tcl.tk/man/tcl/TclLib/Interp.htm

[7]	https://github.com/jwiegley/use-package

VirtualBox Does Not Automatically Resize Disk Image

2019-01-14T10:00:00+00:00

I use VirtualBox [1] a lot as (local) virtualization software. It is a full-featured virtualization host, and supports multiple underlying disk image file types for guests.

One of those is VirtualBox' native Virtual Disk Image or VDI file type. An advantage of this type is that one can create a dynamically allocated image. This image will initially be very small and not occupy any space for unused virtual disk sectors, but will grow when a disk sector is written to for the first time. VirtualBox does this by checking for unused sectors.

However, this poses issues for disks with multiple partitions. If the last partition is say a (unused) swap partition, then VirtualBox does not automatically grow the underlying image. Even though the first partition is full, VirtualBox will not grow and therefore the host disk will be full without having reached its full potential.

To solve this issue, the machine needs to be partitioned using one big happy partition. Then VirtualBox will dynamically resize according to expectations.

I use packer [2] to prepare disk images for Debian, together with a preseed [3] file. Using preseeding to partition the disk is limited to what is supported by the partition tool, partman . The standard mode of operation for partitioning is atomic , and according to the documentation, "All files will be in one partition". This is not entirely true, as an additional partition will be created for swap space - and this poses the dynamically resize issue.

So therefore one needs to instruct partman to specifically create one partition only, and not an additional swap partition.

This can be done by adding a recipe to the packer preseed file that will use all available space for one partition only, and by disabling the warning message of not configuring a swap partition.

### Partitioning
d-i partman/choose_partition select finish
d-i partman/confirm boolean true
d-i partman/confirm_nooverwrite boolean true
d-i partman-auto/method string regular
d-i partman-partitioning/confirm_write_new_label boolean true
d-i partman-basicfilesystems/no_swap boolean false
d-i partman-auto/expert_recipe string myroot :: 1000 50 -1 ext4 \
  $primary{ } $bootable{ } method{ format } format{ } use_filesystem{ } \
  filesystem{ ext4 } mountpoint{ / } .
d-i partman-auto/choose_recipe select myroot

The 1000 value in the recipe sets the minimus size for the partition. The 50 value sets the priority, but as there will only be one partition created, this is not used. The -1 is a special value and instructs partman to use all available disk space (unlimited partition size), which is what we're after...

Thanks to an answer on superuser.com [4], which pointed me in the right direction. Sharing the knowledge is the way to go :)

[1]	https://www.virtualbox.org/

[2]	https://www.packer.io

[3]	https://wiki.debian.org/DebianInstaller/Preseed

[4]	https://superuser.com/questions/458672/ubuntu-preseed-use-whole-disk-space-but-no-swap#920957

Setting Up a New Sphinx Documentation Framework

2018-10-24T10:00:00+00:00

When having to write documentation for different formats, I always use the reStructuredText [1] (or reST) format. As this is something that happens quite often, it made sense to put some effort in automating the set up of a new documentation framework, a reusable set up script.

The standard documentation framework that I use consists of Sphinx [2], which takes care of converting source pages written in reST into several formats: For example HTML, but also PDF or something more exotic like ePub files. Note that Sphinx already comes with a setup script, sphinx-quickstart [3] - but this doesn't take care of deploying files.

In order to be able to create a reusable framework, I split the necessary files into three groups:

The Sphinx configuration itself,
version information, and
a LaTeX formatting template.

The Sphinx configuration

This part consists of two different files; A generic Makefile [4] to build the different artifact types - as well as a Sphinx configuration file (conf.py [5]) containing basic information about the project, and plugin details. These files rarely change after having initialized the framework.

Version information

The version information (version, or build number) can change per release, and is therefore contained in a separate text file (VERSION [6]). Its values will be read and included by the build process.

LaTeX template

Thirdly, the LaTeX formatting template latex-styling.tex [7] contains styling information that will be used when building PDF files, and is also stored in a separate file. This allows for easy updating and different re-use across projects.

Setting up the framework

The reusable framework consists of a collection of the files mentioned above, as well as some basic static files.

So all that's left when setting up a new documentation framework is creating some directories, copying the files, and... that's basically it.

In order to take care of this, I wrote a simple Bash installer script, installer.sh [6]. A new feature in Bash version 4 is the support of associative arrays. That functionality makes it easier to write a generic, reusable script which takes care of creating and copying files and directories.

A simple

./installer.sh TARGET

will set up the framework in the directory TARGET . Of course the prerequisites (Sphinx and GNU Make) need to be installed as well. If they are, all that's left to do is write some content in reStructuredText [1], and let Sphinx [2] build it into an artifact of your choosing, for example HTML:

make html

The reusable Sphinx framework can be found at https://github.com/PeterMosmans/setup-sphinx-framework

[1]	(1, 2) http://docutils.sourceforge.net/rst.html

[2]	(1, 2) http://www.sphinx-doc.org

[3]	http://www.sphinx-doc.org/en/master/man/sphinx-quickstart.html

[4]	https://github.com/PeterMosmans/setup-sphinx-framework/blob/master/Makefile

[5]	https://github.com/PeterMosmans/setup-sphinx-framework/blob/master/conf.py

[6]	(1, 2) https://github.com/PeterMosmans/setup-sphinx-framework/blob/master/installer.sh

[7]	https://github.com/PeterMosmans/setup-sphinx-framework/blob/master/latex-styling.tex

Hacker Summer Camp: BSides Las Vegas and DEF CON 2018 review

2018-10-08T10:00:00+00:00

Time flies... It's already been a few months ago that BSides Las Vegas and DEF CON 2018 were held.

BSides Las Vegas was nice, although the overall quality of talks seemed to be a little higher in previous editions. This of course can be completely due to me picking exactly the wrong talks: There is simply too much to see.

DEF CON 2018 was also different than previous editions - mostly, because it was now so spread out (Caesars Palace as well as the Flamingo Las Vegas). In practice, this meant a lot of walking between the two locations. When you were attending a talk in one location, it physically wasn't possible to attend the next one unless it was located in the same or surrounding room.

Fortunately DEF CON is all about learning, doing and networking - and in that aspect it didn't disappoint.

Especially for Hacker Summer Camp, I designed and built my own badge - consisting of an ESP32 [1] microprocessor running MicroPython [2], an e-Ink display, some custom Python code, and a retro cassette case. The display rotates numerous fitting images. The image was visible even when the power was disconnected, thanks to the 3-color e-Ink display.

A follow-up post will detail the badge design, and the (Python) code behind it.

It's the idea to build a more elaborate design for next year, which can communicate with other badges. And yes, build more than one, as the demand seemed to be high. Sharing is caring...

[1]	http://esp32.net

[2]	https://micropython.org/

Customize and theme tmux the easy way

2018-06-11T10:00:00+00:00

Terminal multiplexers allow you to view multiple separate terminal sessions within a single terminal window. Tmux is my terminal multiplexer of choice, as it has more features than the 'original multiplexer' GNU Screen. The default setup gives you some information, but its appearance is, well...

Fortunately you can theme, or customize pretty much everything: From the colors to the information being shown in the status bar.

In order to make it easier to theme tmux, I split the tmux configuration file into two separate files. One file contains the main configuration (~/.tmux.conf ), and another file contains only theming (visual) variables (~/.tmux.THEMENAME.theme ). This setup makes it easier to switch different themes, without changing the main tmux configuration file.

As I wanted to automatically load a theme based on a shell environment variable, I added a small piece of code to the main tmux configuration file. This executes a shell command, which in turn loads the correct theme file.

run-shell "tmux source-file ~/.tmux.\${TMUX_THEME:-default}.theme"

The theme file is loaded dynamically, based on the environment variable $TMUX_THEME . If the environment variable is not set or empty, then the default theme is loaded: ~/.tmux.default.theme .

Loading a different theme is easy to do from within tmux using the source-file :

tmux source-file ~/.tmux.THEMENAME.theme

My main tmux configuration file has a shortcut binding (r ) to quickly reload a new configuration. This means that changes to the current theme can be applied and viewed easily.

bind r source-file ~/.tmux.conf \; display-message "tmux configuration file
reloaded..."

Tmux can be found at https://github.com/tmux/tmux , and my personal tmux configuration and theme files can be found https://github.com/PeterMosmans/dotfiles/blob/master/.tmux.conf and https://github.com/PeterMosmans/dotfiles/blob/master/.tmux.dracula.conf

Improving cross-subsystem git workflow: The different git configuration files

2018-04-24T10:00:00+00:00

Git configuration settings can be stored in three different files: The system configuration file, the global configuration file and the repository's local configuration file. See git on Windows - location of configuration files [1] for their locations.

When you use multiple subsystems on Windows (like MSYS2, Cygwin or any of the the Windows Subsystem for Linux distributions) it can be a chore to keep the git configurations synchronized. In other words: The less configuration files to maintain, the better.

Whether it's git for Windows, or one of the subsystem-specific git binaries:

Each of the git binaries that runs on Windows expands the tilde (~ ) to the home directory, and the path separator is always a slash (/ ).

These features can be used in our advantage in order to simplify the git configuration files between all subsystems.

Re-defining the system

The system configuration file is meant to store all system-specific configuration settings, which will be applied to all users and git repositories on the system.

If you're the only user of your workstation, it makes sense to re-define system as subsystem:

All subsystem-dependent git configuration settings should be set in the system git configuration file.

This means that settings depending on underlying binaries, like tools to diff binary files (see Diff binary files like docx, odt and pdf with git [2]) or subsystem-specific files should be configured in the system git configuration file. Examples of such parameters are

http.sslcainfo
http.sslbackend
sendemail.smtpserver
help.format
diff.pdf.textconv

The system configuration has a fixed path, relative to the installation directory: etc/gitconfig . This configuration file can be read using git config --list --system

Global means universal

Any user-specific setting should be set in the global git configuration file.

Global configuration files can be found in what the binary or environment considers the HOME directory (see git on Windows - location of configuration files [1]).

Let's suppose from Window's point of view, the home directory of MSYS2 is U:\ , and that is where the master .gitconfig file resides. By symlinking that file from a command prompt from the Windows home directory (%USERPROFILE% ) one universal (or global) configuration file can be used - only one that needs to be maintained. From the Windows shell:

mklink %USERPROFILE%\.gitconfig U:\.gitconfig

References to other user dependent files like global gitignore files can be set using the tilde (~ ) notation and slash (/ ) path separator. They need to be relative to the home directory.

git config --global --add core.excludesfile ~/.gitignore_global

Of course you still need to symlink the file .gitignore_global from any subsystem, similar to .gitconfig . From Windows:

mklink %USERPROFILE%\.gitignore_global U:\.gitignore_global

The above workflow allows you to maintain a single personal git configuration file. For each new subsystem all you need to do is create the necessary symlinks, and everything is good to go. If you're using Debian on Windows for example, then the command will look something like:

mklink
%USERPROFILE%\AppData\Local\Packages\TheDebianProject.DebianGNULinux_76v4gfsz19hv4\LocalState\rootfs\home\%USERNAME%\.gitconfig
U:\.gitconfig

The exact path depends on the Windows Subsystem for Linux distribution version, and your username.

Note

Symbolic links and the Windows Subsystem for Linux are currently not entirely working: This is scheduled for next release, see Microsoft's GitHub repository [3].

So, if you're the only user of a system:

subsystem-dependent settings go in the system configuration file, per subsystem.
user-specific settings go in the global configuration file, per user.
always use relative paths, using a tilde (~ ) and slash (/ )

[1]	(1, 2) https://www.onwebsecurity.com/configuration/git-on-windows-location-of-global-configuration-file.html

[2]	https://www.onwebsecurity.com/configuration/diff-binary-files-docx-odt-pdf-with-git.html

[3]	https://github.com/Microsoft/WSL/issues/2995

When to sharpen, and when to cut

2018-03-15T10:00:00+00:00

When performing a task for the first time, I think of whether it's a one-off, or that it will become a recurring thing. Python scripts for example can be developed blazingly fast, and a little bit of automation can go a long way.

However...

...sometimes, while developing an automated solution that looked so simple beforehand, becomes a wild ride from one rabbit hole into the other. Missing dependencies, compile errors, functions that don't lend themselves very well for automation; Everything that can go wrong will go wrong.

That's why I like The Pomodoro Technique [1] so much, where you work in discrete time chunks of say 25, or 30 minutes. You decide upon the maximum cost for the implementation beforehand. Given the expected return, what is a sane investment ? If the time is up, then it's back to the original task at hand.

I have learned the hard way to always budget some time for documenting the (partial) solution, so that at least there's the profit of knowledge gained. Or, another record of a failed attempt...

[1]	https://francescocirillo.com/pages/pomodoro-technique

Rebase OpenSSL 1.0.2-chacha to use TLS 1.3

2018-01-31T10:00:00+00:00

Since its inception in 2014, the OpenSSL 1.0.2-chacha fork [1] has been used as standard OpenSSL distribution for numerous SSL/TLS pentesting tools. It includes default support for ciphers that are deemed insecure, and has extensive starttls support.... in comparison with the vanilla 1.0.2 branch.

However, even though 1.0.2 is deemed a Long Term Supported (LTS) version, no new ciphers or functionality will be added to it.

The initial reason to start the fork was a lack of ChaCha20 / Poly1305 support in the 1.0.2 branch. After that, more and more features and insecure ciphers were added or ported back in from other branches.

As ChaCha20 / Poly1305 support has been added to the 1.1.1 branch, which also contains (preliminary) TLS 1.3 support, it might be time for the insecure OpenSSL version to be rebased onto a new branch. The initial goals will still be the same:

Add as much ciphers and functionality as possible
Keep the source aligned as much as possible to the vanilla version
Keep the patches atomic, transparent and maintainable
Write as little custom code as possible

This will be quite the challenge, as the architecture and underlying code has changed quite a bit from 1.0.2 to 1.1.1.

Differences between 1.0.2 and 1.1.1

Obviously there are a lot of changes in 1.1.1 with regards to 1.0.2-chacha.

Starttls support

The 1.1.1 branch has much more support for starttls protocols (ftp, imap, irc, ldap, lmtp, mysql, nntp, pop3, postgres, sieve, smtp, telnet, xmpp and xmpp_server). Therefore there's no need to 're' implement that.

Protocols

The 1.1.1 branch has (preliminary) TLSv1.3 support, which is great (and one of the reasons to rebase). Unfortunately support for SSLv2 isn't there. Since the number of SSLv2 capable servers is very low, I'm wondering whether it makes sense to 're enable' that in 1.1.1. The SSLv2 code is spread over a large number of files, and therefore difficult to enable.

Authentication algorithms

The 1.1.1 branch lacks weak authentication algorithms, like DH (Diffie-Hellman), DSS and the anonymous ECDH (Elliptic curve Diffie-Hellman). Also missing are the GOST01 and GOST94 algorithms, but they might be re-added by using an engine.

Key exchange algorithms

The 1.1.1 branch lacks Diffie-Hellman with keysizes equal or smaller than 1024 bits (DH(512), DH(1024), RSA(512), RSA(1024), DH/DSS, DH/RSA, and algorithms based on ECDH (ECDH/ECDSA and ECDH/RSA). Also missing is GOST, see above.

Ciphers

The following ciphers are included in 1.0.2-chacha, and missing from 1.1.1-dev:

0x00,0x00,0x00 - NULL-MD5                SSLv2 Kx=RSA(512) Au=RSA  Enc=None      Mac=MD5  export
     0x00,0x03 - EXP-RC4-MD5             SSLv3 Kx=RSA(512) Au=RSA  Enc=RC4(40)   Mac=MD5  export
     0x00,0x06 - EXP-RC2-CBC-MD5         SSLv3 Kx=RSA(512) Au=RSA  Enc=RC2(40)   Mac=MD5  export
     0x00,0x08 - EXP-DES-CBC-SHA         SSLv3 Kx=RSA(512) Au=RSA  Enc=DES(40)   Mac=SHA1 export
     0x00,0x09 - DES-CBC-SHA             SSLv3 Kx=RSA      Au=RSA  Enc=DES(56)   Mac=SHA1
     0x00,0x0B - EXP-DH-DSS-DES-CBC-SHA  SSLv3 Kx=DH/DSS   Au=DH   Enc=DES(40)   Mac=SHA1 export
     0x00,0x0C - DH-DSS-DES-CBC-SHA      SSLv3 Kx=DH/DSS   Au=DH   Enc=DES(56)   Mac=SHA1
     0x00,0x0D - DH-DSS-DES-CBC3-SHA     SSLv3 Kx=DH/DSS   Au=DH   Enc=3DES(168) Mac=SHA1
     0x00,0x0E - EXP-DH-RSA-DES-CBC-SHA  SSLv3 Kx=DH/RSA   Au=DH   Enc=DES(40)   Mac=SHA1 export
     0x00,0x0F - DH-RSA-DES-CBC-SHA      SSLv3 Kx=DH/RSA   Au=DH   Enc=DES(56)   Mac=SHA1
     0x00,0x10 - DH-RSA-DES-CBC3-SHA     SSLv3 Kx=DH/RSA   Au=DH   Enc=3DES(168) Mac=SHA1
     0x00,0x11 - EXP-EDH-DSS-DES-CBC-SHA SSLv3 Kx=DH(512)  Au=DSS  Enc=DES(40)   Mac=SHA1 export
     0x00,0x12 - EDH-DSS-DES-CBC-SHA     SSLv3 Kx=DH       Au=DSS  Enc=DES(56)   Mac=SHA1
     0x00,0x14 - EXP-EDH-RSA-DES-CBC-SHA SSLv3 Kx=DH(512)  Au=RSA  Enc=DES(40)   Mac=SHA1 export
     0x00,0x15 - EDH-RSA-DES-CBC-SHA     SSLv3 Kx=DH       Au=RSA  Enc=DES(56)   Mac=SHA1
     0x00,0x17 - EXP-ADH-RC4-MD5         SSLv3 Kx=DH(512)  Au=None Enc=RC4(40)   Mac=MD5  export
     0x00,0x19 - EXP-ADH-DES-CBC-SHA     SSLv3 Kx=DH(512)  Au=None Enc=DES(40)   Mac=SHA1 export
     0x00,0x1A - ADH-DES-CBC-SHA         SSLv3 Kx=DH       Au=None Enc=DES(56)   Mac=SHA1
     0x00,0x30 - DH-DSS-AES128-SHA       SSLv3 Kx=DH/DSS   Au=DH   Enc=AES(128)  Mac=SHA1
     0x00,0x31 - DH-RSA-AES128-SHA       SSLv3 Kx=DH/RSA   Au=DH   Enc=AES(128)  Mac=SHA1
     0x00,0x36 - DH-DSS-AES256-SHA       SSLv3 Kx=DH/DSS   Au=DH   Enc=AES(256)  Mac=SHA1
     0x00,0x37 - DH-RSA-AES256-SHA       SSLv3 Kx=DH/RSA   Au=DH   Enc=AES(256)  Mac=SHA1
     0x00,0x3E - DH-DSS-AES128-SHA256    TLSv1.2 Kx=DH/DSS   Au=DH   Enc=AES(128)  Mac=SHA256
     0x00,0x3F - DH-RSA-AES128-SHA256    TLSv1.2 Kx=DH/RSA   Au=DH   Enc=AES(128)  Mac=SHA256
     0x00,0x42 - DH-DSS-CAMELLIA128-SHA  SSLv3 Kx=DH/DSS   Au=DH   Enc=Camellia(128) Mac=SHA1
     0x00,0x43 - DH-RSA-CAMELLIA128-SHA  SSLv3 Kx=DH/RSA   Au=DH   Enc=Camellia(128) Mac=SHA1
     0x00,0x62 - EXP1024-DES-CBC-SHA     SSLv3 Kx=RSA(1024) Au=RSA  Enc=DES(56)   Mac=SHA1 export
     0x00,0x63 - EXP1024-DHE-DSS-DES-CBC-SHA SSLv3 Kx=DH(1024) Au=DSS  Enc=DES(56)   Mac=SHA1 export
     0x00,0x64 - EXP1024-RC4-SHA         SSLv3 Kx=RSA(1024) Au=RSA  Enc=RC4(56)   Mac=SHA1 export
     0x00,0x65 - EXP1024-DHE-DSS-RC4-SHA SSLv3 Kx=DH(1024) Au=DSS  Enc=RC4(56)   Mac=SHA1 export
     0x00,0x66 - DHE-DSS-RC4-SHA         SSLv3 Kx=DH       Au=DSS  Enc=RC4(128)  Mac=SHA1
     0x00,0x68 - DH-DSS-AES256-SHA256    TLSv1.2 Kx=DH/DSS   Au=DH   Enc=AES(256)  Mac=SHA256
     0x00,0x69 - DH-RSA-AES256-SHA256    TLSv1.2 Kx=DH/RSA   Au=DH   Enc=AES(256)  Mac=SHA256
     0x00,0x80 - GOST94-GOST89-GOST89    SSLv3 Kx=GOST     Au=GOST94 Enc=GOST89(256) Mac=GOST89
0x05,0x00,0x80 - IDEA-CBC-MD5            SSLv2 Kx=RSA      Au=RSA  Enc=IDEA(128) Mac=MD5
0x03,0x00,0x80 - RC2-CBC-MD5             SSLv2 Kx=RSA      Au=RSA  Enc=RC2(128)  Mac=MD5
0x01,0x00,0x80 - RC4-MD5                 SSLv2 Kx=RSA      Au=RSA  Enc=RC4(128)  Mac=MD5
0x08,0x00,0x80 - RC4-64-MD5              SSLv2 Kx=RSA      Au=RSA  Enc=RC4(64)   Mac=MD5
0x04,0x00,0x80 - EXP-RC2-CBC-MD5         SSLv2 Kx=RSA(512) Au=RSA  Enc=RC2(40)   Mac=MD5  export
0x02,0x00,0x80 - EXP-RC4-MD5             SSLv2 Kx=RSA(512) Au=RSA  Enc=RC4(40)   Mac=MD5  export
     0x00,0x81 - GOST2001-GOST89-GOST89  SSLv3 Kx=GOST     Au=GOST01 Enc=GOST89(256) Mac=GOST89
     0x00,0x82 - GOST94-NULL-GOST94      SSLv3 Kx=GOST     Au=GOST94 Enc=None      Mac=GOST94
     0x00,0x83 - GOST2001-NULL-GOST94    SSLv3 Kx=GOST     Au=GOST01 Enc=None      Mac=GOST94
     0x00,0x85 - DH-DSS-CAMELLIA256-SHA  SSLv3 Kx=DH/DSS   Au=DH   Enc=Camellia(256) Mac=SHA1
     0x00,0x86 - DH-RSA-CAMELLIA256-SHA  SSLv3 Kx=DH/RSA   Au=DH   Enc=Camellia(256) Mac=SHA1
     0x00,0x97 - DH-DSS-SEED-SHA         SSLv3 Kx=DH/DSS   Au=DH   Enc=SEED(128) Mac=SHA1
     0x00,0x98 - DH-RSA-SEED-SHA         SSLv3 Kx=DH/RSA   Au=DH   Enc=SEED(128) Mac=SHA1
     0x00,0xA0 - DH-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=DH/RSA   Au=DH   Enc=AESGCM(128) Mac=AEAD
     0x00,0xA1 - DH-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=DH/RSA   Au=DH   Enc=AESGCM(256) Mac=AEAD
     0x00,0xA4 - DH-DSS-AES128-GCM-SHA256 TLSv1.2 Kx=DH/DSS   Au=DH   Enc=AESGCM(128) Mac=AEAD
     0x00,0xA5 - DH-DSS-AES256-GCM-SHA384 TLSv1.2 Kx=DH/DSS   Au=DH   Enc=AESGCM(256) Mac=AEAD
     0x00,0xBB - DH-DSS-CAMELLIA128-SHA256 TLSv1.2 Kx=DH/DSS   Au=DH   Enc=Camellia(128) Mac=SHA256
     0x00,0xBC - DH-RSA-CAMELLIA128-SHA256 TLSv1.2 Kx=DH/RSA   Au=DH   Enc=Camellia(128) Mac=SHA256
     0x00,0xC1 - DH-DSS-CAMELLIA256-SHA256 TLSv1.2 Kx=DH/DSS   Au=DH   Enc=Camellia(256) Mac=SHA256
     0x00,0xC2 - DH-RSA-CAMELLIA256-SHA256 TLSv1.2 Kx=DH/RSA   Au=DH   Enc=Camellia(256) Mac=SHA256
0x01,0x00,0x80 - RC4-MD5                 SSLv2 Kx=RSA      Au=RSA  Enc=RC4(128)  Mac=MD5
0x02,0x00,0x80 - EXP-RC4-MD5             SSLv2 Kx=RSA(512) Au=RSA  Enc=RC4(40)   Mac=MD5  export
0x03,0x00,0x80 - RC2-CBC-MD5             SSLv2 Kx=RSA      Au=RSA  Enc=RC2(128)  Mac=MD5
0x04,0x00,0x80 - EXP-RC2-CBC-MD5         SSLv2 Kx=RSA(512) Au=RSA  Enc=RC2(40)   Mac=MD5  export
0x05,0x00,0x80 - IDEA-CBC-MD5            SSLv2 Kx=RSA      Au=RSA  Enc=IDEA(128) Mac=MD5
0x06,0x00,0x40 - DES-CBC-MD5             SSLv2 Kx=RSA      Au=RSA  Enc=DES(56)   Mac=MD5
0x07,0x00,0xC0 - DES-CBC3-MD5            SSLv2 Kx=RSA      Au=RSA  Enc=3DES(168) Mac=MD5
0x08,0x00,0x80 - RC4-64-MD5              SSLv2 Kx=RSA      Au=RSA  Enc=RC4(64)   Mac=MD5
     0xC0,0x01 - ECDH-ECDSA-NULL-SHA     SSLv3 Kx=ECDH/ECDSA Au=ECDH Enc=None      Mac=SHA1
     0xC0,0x02 - ECDH-ECDSA-RC4-SHA      SSLv3 Kx=ECDH/ECDSA Au=ECDH Enc=RC4(128)  Mac=SHA1
     0xC0,0x03 - ECDH-ECDSA-DES-CBC3-SHA SSLv3 Kx=ECDH/ECDSA Au=ECDH Enc=3DES(168) Mac=SHA1
     0xC0,0x04 - ECDH-ECDSA-AES128-SHA   SSLv3 Kx=ECDH/ECDSA Au=ECDH Enc=AES(128)  Mac=SHA1
     0xC0,0x05 - ECDH-ECDSA-AES256-SHA   SSLv3 Kx=ECDH/ECDSA Au=ECDH Enc=AES(256)  Mac=SHA1
     0xC0,0x0B - ECDH-RSA-NULL-SHA       SSLv3 Kx=ECDH/RSA Au=ECDH Enc=None      Mac=SHA1
     0xC0,0x0C - ECDH-RSA-RC4-SHA        SSLv3 Kx=ECDH/RSA Au=ECDH Enc=RC4(128)  Mac=SHA1
     0xC0,0x0D - ECDH-RSA-DES-CBC3-SHA   SSLv3 Kx=ECDH/RSA Au=ECDH Enc=3DES(168) Mac=SHA1
     0xC0,0x0E - ECDH-RSA-AES128-SHA     SSLv3 Kx=ECDH/RSA Au=ECDH Enc=AES(128)  Mac=SHA1
     0xC0,0x0F - ECDH-RSA-AES256-SHA     SSLv3 Kx=ECDH/RSA Au=ECDH Enc=AES(256)  Mac=SHA1
     0xC0,0x25 - ECDH-ECDSA-AES128-SHA256 TLSv1.2 Kx=ECDH/ECDSA Au=ECDH Enc=AES(128)  Mac=SHA256
     0xC0,0x26 - ECDH-ECDSA-AES256-SHA384 TLSv1.2 Kx=ECDH/ECDSA Au=ECDH Enc=AES(256)  Mac=SHA384
     0xC0,0x29 - ECDH-RSA-AES128-SHA256  TLSv1.2 Kx=ECDH/RSA Au=ECDH Enc=AES(128)  Mac=SHA256
     0xC0,0x2A - ECDH-RSA-AES256-SHA384  TLSv1.2 Kx=ECDH/RSA Au=ECDH Enc=AES(256)  Mac=SHA384
     0xC0,0x2D - ECDH-ECDSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH/ECDSA Au=ECDH Enc=AESGCM(128) Mac=AEAD
     0xC0,0x2E - ECDH-ECDSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH/ECDSA Au=ECDH Enc=AESGCM(256) Mac=AEAD
     0xC0,0x31 - ECDH-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH/RSA Au=ECDH Enc=AESGCM(128) Mac=AEAD
     0xC0,0x32 - ECDH-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH/RSA Au=ECDH Enc=AESGCM(256) Mac=AEAD
     0xC0,0x74 - ECDH-ECDSA-CAMELLIA128-SHA256 TLSv1.2 Kx=ECDH/ECDSA Au=ECDH Enc=Camellia(128) Mac=SHA256
     0xC0,0x75 - ECDH-ECDSA-CAMELLIA256-SHA384 TLSv1.2 Kx=ECDH/ECDSA Au=ECDH Enc=Camellia(256) Mac=SHA384
     0xC0,0x78 - ECDH-RSA-CAMELLIA128-SHA256 TLSv1.2 Kx=ECDH/RSA Au=ECDH Enc=Camellia(128) Mac=SHA256
     0xC0,0x79 - ECDH-RSA-CAMELLIA256-SHA384 TLSv1.2 Kx=ECDH/RSA Au=ECDH Enc=Camellia(256) Mac=SHA384
     0xCC,0x13 - ECDHE-RSA-CHACHA20-POLY1305 TLSv1.2 Kx=ECDH     Au=RSA  Enc=ChaCha20(256) Mac=AEAD
     0xCC,0x14 - ECDHE-ECDSA-CHACHA20-POLY1305 TLSv1.2 Kx=ECDH     Au=ECDSA Enc=ChaCha20(256) Mac=AEAD
     0xCC,0x15 - DHE-RSA-CHACHA20-POLY1305 TLSv1.2 Kx=DH       Au=RSA  Enc=ChaCha20(256) Mac=AEAD

In short: ciphers based on one of the aforementioned key exchange and authentication protocols, export ciphers and RC4, MD5 or SHA.

Roadmap

The first phase of rebasing 1.0.2-chacha onto master would be to decide on a new name (suggestions anyone ?), and creating a hierarchy of TODO items. This allows me (and hopefully other contributors) to work on the items that are in most 'demand' for this generation.

As soon as a new Long Term Support [2] version will be chosen, it would be time to really rebase on that version, instead of master. If the commits are atomic and 'rebasable', that would be no problem.

I'm looking forward to the Long Term Supported future!

[1]	https://github.com/PeterMosmans/openssl/tree/1.0.2-chacha

[2]	https://github.com/openssl/openssl/issues/5120

Tools for setting, tracking and achieving long term goals

2018-01-18T10:00:00+00:00

Immediately after reading an article on David Allen and his brainchild Getting Things Done, I started with implementing his methodology. I loved it. I still love it - especially the Getting Things Done concepts of inbox ZERO, maintaining lists, and periodic reviews.

Inbox ZERO for me is not so much about having empty email inboxes, as well as making sure that input is collected from multiple locations and stored into one dedicated location. An inbox can also be a notebook, or note taking software like Google Keep.

Electronically stored lists have the benefit of being available on a multitude of devices, the ability to synchronize between them, backups, and their biggest advantage - providing dynamic views.

Both tools that I have been using so far (the open source Java application ThinkingRock [1], and Emacs in Org mode [2]) for maintaining lists of actionable items and projects were great in that perspective. Using those tools for periodic reviews was a different story. After trying numerous configurations I never got the hang of using ThinkingRock and Emacs for that purpose. Items become abstract letters on a screen. Views never fully captured what was important or which project served which goal.

Periodically reviewing projects and actionable lists is in my opinion the single most important thing one needs to do in order to achieve one's long term goals. You need to step out of the operation and look back, in order to plan ahead.

Writing something down with pen on paper is completely different than typing something on a keyboard. Using pen and paper forces you to think more about what exactly it is that you want to write down beforehand. It cannot be changed as easily as with typed words. It also forces you to think about the visual structure: Where to write down what, where to locate it on the paper. Which page, what size, which color ink.

Therefore, in 2017, after a long hiatus, I started using a physical planner again to keep track of projects and actionable items. While looking for a 2018 planner I came across one from Mossery [3], a company I hadn't heard of before. They allow you to customize your planner somewhat, regarding the look as well as the contents.

The planner itself consists of a year section, a monthly overview as well as detailed weekly pages. It has space for recaps, overviews, and highlights of past periods. It's ideal to tick off list items on a daily and weekly base, and naturally nudges you to review each week, each month, and even the year.

The less time I spend looking at screens, the more I can focus on the vision, on more ambitious goals. Where before I tried to automate and tweak as much as possible using applications - the tooling, I'm now focused on the content, the vision.

Mossery is a small company from Kuala Lumpur, Malaysia, and I had the pleasure to meet some of their wonderful team in person for an explanation behind their planner. They're a passionate bunch, and it shows - their planner is chock-full of small details, like a list of personal heroes and inspirational quotes.

Their 2018 planner inspires me to set my future goals even higher - by using old and proven technologies: pen and paper (planner).

[1]	https://www.trgtd.com.au/

[2]	http://orgmode.org/

[3]	https://www.mossery.co/collections/2018-planner

Diff binary files like docx, odt and pdf with git

2017-11-19T10:00:00+00:00

Working with binary file types like the Microsoft Word XML Format Document docx , the OpenDocument Text odt format and the Portable Document Format pdf in combination with git has its difficulties. Out of the box, git only provides diffing for plain text formats. Comparing binary files in textual format is not supported.

With a simple configuration change and some open source, cross-platform tools, git can be adapted to diff those formats as well.

Installing the tools

First, one needs the tools which can convert the binary files to plain text formats. For most formats like docx and odt , the open source tool Pandoc [1] will do the trick. It can even export those files to Markdown format, or (my personal choice) reStructuredText [2]. A markup language like reStructuredText makes it possible to make a detailed comparison between structured documents, for instance when the heading level changed.

For PDF, there's the open source tool pdftotext , which is part of the Poppler [3] utils package and available for (almost) all operating systems. This can convert a PDF file to plain text.

There's a tiny catch with pdftotext , as it has issues using stdout as output, instead of writing to files. This is necessary, as git will expect the output on stdout.

This can be fixed by creating a tiny wrapper named pdftostdout around pdftotext , which will execute the program with the correct parameters. A dash as last parameter will instruct pdftotext to use stdout:

echo "pdftotext $1 -" > /usr/bin/pdftostdout

Of course this wrapper can be stored anywhere, as long as it can be executed and found by git.

Add new text conversion handlers to git

After installing both programs and the wrapper, next git needs to be instructed how to convert the binary file types to text format. This can be accomplished by modifying the global git configuration:

git config --global diff.docx.textconv pandoc --to=rst
git config --global diff.odt.textconv pandoc --to=rst
git config --global diff.pdf.textconv pdftostdout

This creates new diff handlers for each of the file types.

Note

Using the parameter --to=rst specifies pandoc to use the reStructuredText format. This makes comparing hierarchies easier than just using the plain text format.

Instruct git to apply the correct handlers per file type

Finally, git needs to know which conversion handler to use for which file type. That can be accomplished by modifying the global gitattributes [4] file.

The gitattributes file defines attributes per path, or per file. That means that you can specify handlers per file _type_, which will automatically convert the binary format to text format, using the correct tool.

The gitattributes file can be specified locally (per git repository), per system, or globally. Globally is usually the preferred choice, as this means configure once per user, and use everywhere, with each repository. The global gitattributes file can be found under $HOME/.config/git/attributes .

Note

As the global and system git attributes files have the lowest precedence, they can easily be overridden on a local base. This can be done by creating a .gitattributes file in the root of a repository.

The following code-snippet adds the correct conversion handlers per file type to the global git configuration:

echo "*.docx diff=docx" >> ~/.config/git/attributes
echo "*.odt diff=odt" >> ~/.config/git/attributes
echo "*.pdf diff=pdf" >> ~/.config/git/attributes

And that's all there is to it. Now git diff will show all changes in plain text format for the binary file types docx , odt and pdf .

Any binary format can be diffed with git, as long as there's a tool which converts the binary format to plain text. One just needs to add the conversion handlers and attributes in the same way.

[1]	https://pandoc.org/

[2]	http://docutils.sourceforge.net/rst.html

[3]	https://poppler.freedesktop.org/

[4]	https://git-scm.com/docs/gitattributes

Structure research, talks and trainings using mindmaps, reStructuredText and Emacs Org mode

2017-10-18T11:15:00+00:00

It's sometimes difficult to know where to start, or how to begin when preparing talks and trainings, or researching vulnerabilities. This article describes my current workflow and set of tools that help me in that process. From up above the clouds down to the details.

Finding the flow

Mindmaps are great to work with when an idea hasn't fully matured yet: There's a bigger picture, but not yet a structure, a flow. The visual aspect of mindmaps helps with finding patterns, and usually a pattern arises while the mindmap matures (emergent design). It also allows you to quickly reshuffle blobs of text, moving everything around.

For mindmapping I use the open source, cross-platform tool, Freeplane [1]. It stores mindmaps in the open freemind file format [2], an XML format (yes, that's Freemind, which was forked into Freeplane).

Organization or presentation

When a structure emerges, I use either Org mode [3] or reStructuredText [4] to write down the notes in a more detailed format. Why two different formats ?

Both use the underlying plain-text format, and are well suited to keep notes in.

Org mode is excellent for maintaining TODO lists and planning projects. Its format allows you to add various data properties, like tags, custom attributes and even can be used for time registration. It's excellent for organizing data.
reStructuredText is excellent for presentation purposes. It allows seamless exporting to numerous file formats like HTML, PDF or ePub. It gives you a lot of flexibility with regards to presentation.

For example when preparing presentations I choose reStructuredText, and when doing research I use the Org mode format. For trainings I use reStructuredText to prepare handouts, but I use Org mode for my own trainer's notes.

Backward design

When a pattern has emerged, I use backward design to structure trainings:

Identify the desired results, the objectives
Determine how those results can be measured, for example with assessments or challenges.
Plan the training and develop the instructions.

This is also the same way that I write articles or blog posts for example: First write the summary, the objective, and then the actual contents. It's like with test-driven development: The requirements first, then the tests, then the actual code.

Plan for the future

Regardless the tools, I use open file formats. This allows me and others to easily collaborate, to re-use previous work, and of course it provides freedom. Anyone can use any tool, as long as it supports those file formats.

Preferably everything should to be stored in open file formats, which allows re-use of existing materials.

Both Org mode as reStructuredText use the plain-text file format.
When using references to articles, papers or websites I store them in an external file as bibtex [5], a plain-text file format. Bibtex files can be used by a wide range of programs. Both Pelican and Sphinx use reStructuredText, and both allow referencing bibtex files (the reference (Raymond, 2003) for instance is generated from a bibtex file). Org mode also allows including external bibtex files.

As the Unix philosophy states (Raymond, 2003):

Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface.

Therefore either Org mode for organization, or reStructuredText for presentation.

Of course, as they both use the plain-text format, it's easy to convert between the two in a later stage....

[1]	https://www.freeplane.org/wiki/index.php/Home

[2]	http://freemind.sourceforge.net/wiki/index.php/File_format

[3]	http://orgmode.org/

[4]	http://docutils.sourceforge.net/rst.html

[5]	http://www.bibtex.org/Format/

Bibliography

Eric S. Raymond. The Art of UNIX Programming. Addison-Wesley Professional, sep 2003. ↩ ¹ ²

Properly encoding and escaping for the web

2017-09-13T10:15:00+00:00

When processing untrusted user input for (web) applications, filter the input, and encode the output. That is the most widely given advice in order to prevent (server-side) injections. Yet it can be deceivingly difficult to properly encode (user) input. Encoding is dependent on the type of output - which means that for example a string, which will be used in a JavaScript variable, should be treated (encoded) differently than a string which will be used in plain HTML.

When outputting untrusted user input, one should encode or escape, based on the context, the location of the output.

And what's the difference between escaping and encoding ?

Encoding is transforming data from one format into another format.

Escaping is a subset of encoding, where not all characters need to be encoded. Only some characters are encoded (by using an escape character).

There are quite a number of encoding mechanisms, which make this more difficult than it might look at first glance.

URL encoding

URL encoding is a method to encode information in a Uniform Resource Identifier. There's a set of reserved characters, which have special meaning, and unreserved, or safe characters, which are safe to use. If a character is reserved, then the character is encoded using the percent '%' sign, followed by its hexadecimal digits. That's why URL encoding is sometimes referred to as percent encoding.

Note

A Uniform Resource Identifier (URI) identifies an abstract or physical resource. The URI set includes a Uniform Resource Locator (URL), as well as a Uniform Resource Name (URN). A URL includes the 'access mechanism', the scheme. Example of a URN is extract.pw, and a URL is https://extract.pw. Both are URI's, as all URL and URN are. tel:+61-123-123 is a URI, but not a URL.

The current set of reserved characters for a URI are ! * ' ( ) ; : @ & = + $ , / ? # [ ] . See RFC 3986 section 2.2. Note however that the reserved and safe character sets change over time.

An example of URL encoding:

&   ------->   %26
'   ------->   %27
/   ------->   %2F

The following is an example of URL encoding using Python, where only reserved characters are encoded:

% python -c "import urllib;print(urllib.quote_plus('ape&nutz/better '));"

ape%26nutz%2Fbetter+`

The following is an example of URL encoding, where all characters except the access mechanism (protocol) are encoded. This can be used by adversaries to obfuscate valid addresses, while it's still a valid address.

https://%65%78%74%72%61%63%74%2e%70%77

Modern browsers will show you the decoded URL when hovering over these kinds of obfuscated links.

Usually, when people refer to URI encoding, they signify only encoding reserved, non-safe characters. But again, the above example is still valid.

Unicode encoding

Unicode is a standard for encoding, representing and handling characters in most (if not all) languages. Best known is the UTF-8 character encoding standard, which is a variable length encoding (1, 2, 3 or 4 units of 8 bits, hence the name UTF-8). It's very similar to ASCII, as in: the first 128 characters of Unicode are ASCII. That makes it a compact encoding. Other well known Unicode encodings are UTF-16, and Latin-1 (ISO 8859-1). Unicode characters are represented by code points. A character can therefore exist of one or more 8-bit units. Glyphs are representations of characters on the screen.

U+2206 ∆ (Increment)

The code point is U+2206, it's called the Increment character and it is represented by the glyph ∆ on the screen.

Note

Sometimes vulnerabilities can lurk in the part where Unicode characters are decoded, for example by using different word-size encodings, or by using illegal Unicode encodings. Unicode decoding vulnerabilities warrant their own article.

HTML encoding

Also known as HTML entity encoding. The & < > " ' / characters should always be HTML entity encoded before outputting it into content. This is used for example in HTML attributes.

&   ------->   &amp;
'   ------->   &x27;

When outputting untrusted user input in HTML, make sure to properly HTML encode the following characters:

&   ------->   &amp;
<   ------->   &lt;
>   ------->   &gt;
"   ------->   &quot;
'   ------->   &#x27;
/   ------->   &#x2F;

Why ? Because these characters can "break out of the context".

JavaScript encoding

In general, encode all non alphanumeric characters instead of using escape characters. This prevents double encoding / decoding issues, in combination with HTML encoding.

JavaScript encoding is done by prepending the value with \x followed by its hexadecimal digits.

&   ------->   \x26
"   ------->   \x22
'   ------->   \x27

Proper encoding or escaping, based on the context, is really difficult, and easier to get wrong than right. Make sure you know when to use which format.

Hacker Summer Camp: BSides Las Vegas and DEF CON 2017 review

2017-08-10T22:57:00+00:00

The 2017 edition of Hacker Summer Camp is over... Blackhat, BSides and DEF CON: Arguably the best security conferences in the world, being held during a week in Las Vegas. And wow, what an amazing edition it was this time.

I tried to learn, network, enjoy and soak up as much as possible - which unfortunately means not seeing each and every talk, and (probably) missing out on amazing content. That's why I'm so glad that recordings and slidedecks are being released by BSides and DEF CON, so that you can see where you should have been - after the fact.

The biggest draw for me personally to BSides and DEF CON is that you can immerse yourself in fields and interests that are outside of your daily work or routine. Car hacking, lockpicking, the Internet of Things, this year even voting machines: It's all there. You can learn from and play with everything.

As with playing Capture the Flag, it's a great way to touch a lot of surfaces in a short amount of time.

Josh Corman's BSides Las Vegas keynote was amazing - each time I hear him speak, he manages to get everybody even more enthusiastically about cooperation, about personal involvement and about how 'we' can make a change. About how we, security people, should do better. We've got to do better.

The best talk I attended at DEF CON was a talk from Deviant Ollam, "Slides And Jokes And Stage Fright OH MY: How To Prepare A Talk". More often than not, it's not the content that matters, but the presentation. People can talk about the most interesting subject - if the presentation is bad, it's just a bad presentation. If the presentation is great, then the subject can be literally anything. And this presentation matched everything.

A big thank you from the bottom of my heart to all volunteers, organizers and speakers who make BSides Las Vegas and DEF CON possible.

On September 1st 2017, the first edition of BSides Amsterdam will be held in The Netherlands: The latest addition to the BSides family. See you there ?

Generate list of used content tags for Pelican

2017-07-09T20:25:00+00:00

If your Pelican-generated site uses lots of different tags for articles, it can be difficult to remember or use tag names consistently. Therefore I needed a quick method to print (comma separated) unique tags that were stored in text files.

This shell one-liner from within the content directory will sort and show all tags from reStructuredText (*.rst ) files:

grep -h '^:tags:' *.rst | sed -e 's/^:tags:\s*//;s/\s*,\s*/\n/g' | sort -u

First grep will filter on the :tags: property and will only print out the matching line (without filename, thanks to the -h flag).

Then sed will remove the :tags: keyword (and trailing spaces), and all tags will be split using newline characters.

Finally, sort takes care of sorting and only printing unique entries.

Analogous, one can do the same for categories:

grep -h '^:category:' *.rst | sed -e 's/^:category:\s*//' | sort -u

As Pelican only allows one category, this is somewhat simpler.

For maximum readability, tr can convert the newlines into spaces, so that the output is one big line:

grep -h '^:tags:' *.rst | sed -e 's/^:tags:\s*//;s/\s*,\s*/\n/g' | sort -u | tr '\n' ' '; echo

The last echo is meant to end the line correctly, or else it would be a partial line. Shells like zsh can automatically detect (and show) whether a line is correctly terminated, but it's nicer to make sure that the output is correctly formatted.

One-liners to rule 'em all...

Note

The \s character set is a sed GNU extension, so it'll only work on GNU sed. When using non-GNU compatible sed, you can replace \s with [[:space::] , which is the same whitespace character set.

Convert WordPress to static site generator Pelican

2017-06-27T00:00:00+00:00

After a number of years using WordPress as blogging software, I converted the site to a static site generator: Pelican.

Pelican converts reStructuredText into static HTML. No more PHP, no more databases, but straight static HTML.

The process of converting the site was relatively painless. The conversion tool did a great job of converting an XML export of WordPress into reStructuredText pages.

What needed (and still needs) some manual care were/are the code blocks (the biggest reason of the move from WordPress to Pelican) in articles, and the escaping of variables. WordPress gets pretty complex once you're trying to use it for code snippets and console outputs. The reStructuredText is much more flexible and allows you to edit the site using any text editor. There are tools to do that with WordPress and its API, but it always felt like a difficult workaround.

I thought about keeping the URLs as-is: Over the years the number of visitors of the site has steadily risen, as has the level of indexing by search engines. You don't want dead links - but on the other hand, a transition to another content management system would be the perfect moment to 'clean up' the category and tag names.

After having written numerous articles not only the content but also the style of this site has changed. Therefore I decided to generate a

.htaccess

file containing

HTTP 301

permanent redirects for all WordPress articles. This allowed me to retain the links others might have to the articles, while switching to a new structure.

Note

Not having to use WordPress means a reduction of the web server attack surface: No database, no dynamic web interface, and not having to keep (more) server software up-to-date.

Categories versus articles

Pelican works with one category per article, and zero or more tags. That's different than WordPress, where each article can have one or more categories.

Pelican uses the category as type of article, which meant that each page had to be 'recategorized'. A pattern emerged while reading back old posts.

Some posts were experiences, personal configurations or tips; on the use of Emacs, msys, git or other tools, operating systems or architectures.
Some posts were temporary; announcements
Some posts were opinions and personal reviews of security, conferences and meetings.
Most posts were more permanent, specific on security; security assessment and penetration testing, on how to discover (new) vulnerabilities, or the specific use of tools.

That's what I chose for new category names.

The new workflow is much simpler than the WordPress workflow: Open up Emacs, write an article in reStructuredText, and one

make

command later, the article is converted and uploaded to the server. Pelican also comes with a development server, that automatically converts new files, for easy proofviewing.

Amazing open source software, Pelican.

Write once, present anywhere in every format: preparing an interactive secure coding workshop

2017-06-06T08:42:00+00:00

It is always very rewarding teaching people the principles behind secure coding, about the OWASP top 10, and how to test web applications for vulnerabilities. Workshops are an excellent format to convey that knowledge, as participants can immediately practice their knowledge which reinforces the theory.

I recently had a situation, where I could not physically attend the workshop itself - and, when you're supposed to prepare and lead that workshop, it presents a challenge. I like challenges!

Usually I present materials using a slidedeck (either PDF or HTML5 format). As I wanted to deliver all materials to the participants beforehand I felt that the slidedeck format wasn't the best 'fit' for this remote workshop. Yes, a slidedeck format works when you're in the 'classroom', presenting, but to actually structure knowledge it is too rigid. Not each diagram or list of 'best practices' is suited for the screen-optimized slideshow screen ratio of 4:3. Therefore I wanted to use a text format which would not only allow me to export the data to a slidedeck as usual, but also to export it to a HTML website.

Most if not all of my documentation is written in plaintext format - or, to be more precise, the Org mode format. Org mode is amazingly well-suited to organize and structure text. It can be exported to almost any format, like HTML, PDF, OpenOffice, Word or Markdown. The universal open source document converter Pandoc also has support for Org mode.

However, it isn't really well suited to build an interactive website with. I wanted at least to have a good website navigation, search capabilities, automatically include output of programs, and a format which allows easy formatting of images. Initially I wanted to use a wiki (Dokuwiki in particular, as it's really versatile, and stores all pages as plaintext). But that has the disadvantage of using a wiki-specific markup language, which makes it more difficult to export to other formats. Markdown is too fragmented to use consistently, and lacks a lot of formatting options.

Example of Dokuwiki markup to right-align an image in HTML:

{{ wiki:dokuwiki-128.png}}

Example of Org mode to right-align an image (when exported to HTML):

#+ATTR_HTML: style="float:right;"

Then I discovered the reStructuredText format. It's plaintext, and also has Pandoc (and Emacs) support, so it can be converted to 'any other format'. Not only that, but the open source tool Sphinx (and underlying docutils) makes it super easy to convert a number of reStructuredText files into an interactive HTML website. It has built-in interactive search capabilities, uses a Jinja-based templating system, comes with out-of-the-box navigation menu's. Sphinx can also convert the same source into an ePub, or PDF file, a slidedeck, or a number of other formats.

Example of reStructuredText to right-align an image in any format:

.. image:: _static/sunny.svg

   :align: center

It allowed me to build a website specifically for the interactive secure coding workshop in no time, so that I could focus on the content instead of on the presentation. As reStructuredText is plaintext it's really easy to work with, and Emacs has built-in support for it. The content can be exported as ePub and PDF, and optionally used as slidedeck, all without having to alter the content or the formatting.

This is an example of reStructuredText, what it looks like:

And this is part of the resulting HTML, when it's converted using Sphinx:

The secure coding / OWASP top 10 workshop went according to plan, and I'm looking forward to convert more workshops, workshop materials and documentation of projects I'm part of to the reStructuredText format. That allows me to switch easier between slidedecks, interactive HTML websites, ePub and PDF formats - all having the same content.

Write once, export to anything.

zsh shell inside Emacs on Windows

2017-05-19T02:11:00+00:00

The most obvious advantage of running a cross-platform shell (for example Bash or zsh) is that you can use the same syntax and scripts on multiple platforms. Setting up (alternative) shells on Windows can be pretty tricky, but the small investment is well worth the reward.

The MSYS2 subsystem allows you to run shells like Bash or zsh on Windows. An important part of MSYS2 is making sure that the search paths are all pointing to the MSYS2 subsystem: There are a lot of dependencies.

Bash is the default shell once MSYS2 is installed; zsh can be installed using the package manager:

pacman -Sy zsh

Setting zsh as default shell can be done by modifying the /etc/passwd file, for instance:

mkpasswd -c | sed -e 's/bash/zsh/' | tee -a /etc/passwd

This will change the default shell from bash to zsh.

Running zsh under Emacs on Windows can be done by modifying the shell-file-name variable, and pointing it to the zsh binary from the MSYS2 subsystem. The shell binary has to be somewhere in the Emacs exec-path variable.

(setq shell-file-name (executable-find "zsh.exe"))

Don't forget to modify the PATH environment variable for Emacs, as the MSYS2 paths should be resolved before Windows paths. Using the same example, where MSYS2 is installed under

c:\programs\msys2

(setenv "PATH" "C:\\programs\\msys2\\mingw64\\bin;C:\\programs\\msys2\\usr\\local\\bin;C:\\programs\\msys2\\usr\\bin;C:\\Windows\\System32;C:\\Windows")

After setting these two variables in the Emacs configuration file, running

M-x shell

in Emacs should bring up the familiar zsh prompt.

Emacs' terminal settings (eterm) are different than MSYS2' standard terminal settings (xterm-256color). This means that some plugins or themes (prompts) might not work - especially when using oh-my-zsh.

Detecting whether zsh is started under Emacs is easy, using the variable

$INSIDE_EMACS

. This codesnippet in

.zshrc

(which will be sourced for interactive shells) only enables the git plugin when being run in Emacs, and changes the theme

# Disable some plugins while running in Emacs
if [[ -n "$INSIDE_EMACS" ]]; then
  plugins=(git)
  ZSH_THEME="simple"
else
  ZSH_THEME="compact-grey"
fi

By adding the INSIDE_EMACS variable to the local ~/.ssh/config as SendEnv variable...

Host myhost
SendEnv INSIDE_EMACS

... and to a ssh server as AcceptEnv variable in /etc/ssh/sshd_config ...

AcceptEnv LANG LC_* INSIDE_EMACS

... this even works when ssh'ing inside an Emacs shell session to another ssh server, running zsh. When ssh'ing in the zsh shell inside Emacs on Windows, using the parameters -t -t forces pseudo-tty allocation (which is necessary, as Emacs on Windows don't have a true tty).

Cross-platform, open-source goodyness...

The complexity of securing Docker versus Vagrant

2017-05-03T04:40:00+00:00

There are many, many comparisons out there on the Internet of Docker versus Vagrant. Usability, scalability, portability, you-name-it-ability.

This blogpost is about some different security aspects about the current implementation of Docker versus Vagrant. It's a high, high-level comparison of the security impact for the host system.

Can you compare the two ? Not really, they're being used for completely different reasons. Docker is wildly popular for medium to large scale production webserver deployments and microservices, where each service has its own process, or container. Vagrant is more being used during design and development by individual developers, or in teams at companies.

So, having said that, let's compare the security differences in high-level:

Docker

Partial isolation from the host
Shares the kernel with the host.
Resources aren't guaranteed, they're shared.
Uses runC/libcontainer as runtime (before version 1.8 it used linux containers (lxc) under the hood, then libcontainer, and since the inception of the Open Container Initiative it uses runC which abstracts even further away from libcontainer). System resources can be allocated using control groups (cgroups): block devices (blkio), CPU (cpu), CPU accounting (cpuacct), individual CPU and memory nodes (cpuset), devices, task management (freezer), memory usage (memory), network packets (net_cls), network priority (net_prio), namespaces (ns) and performace analysis (perf_event).
Implements network isolation, each Docker container has its own network stack (zero or more network interfaces).
Uses namespaces to isolate processes and therefore containers.
Secures the host, and each of the containers running on the host using apparmor.
Vulnerabilities in the container will not affect the host.
Vulnerabilities in Docker directly affect the host.

Vagrant

Full isolation from the host
Runs its own kernel on top of the virtual machine environment.
Resources are guaranteed in advance.
Vulnerabilities in the host operating system will not affect the host.
Vulnerabilities in the virtual machine environment could affect the host.

Docker seems to give systems administrators a much more fine-grained control set to tune the security. And that's exactly what I personally think still, in 2017, is currently the weak spot: There are simply too much locations that a DevOps engineer or systems administrator needs to think of to properly secure the host from the container(s).

When something is easy to do, and is enabled by default, people will use it. At the moment, configuring and hardening Docker is much easier than it was, but still - not easy. The same reason why (unfortunately) SELinux isn't widely deployed. It's too complex - and complexity is the enemy of security.

Security through obscurity means better operational security

2017-04-16T06:04:00+00:00

What I personally like so much about being a penetration tester, is
that (I'd like to think that) we make the world a safer place. Better
security means better privacy means more democracy.
It's not about telling people what "they" did wrong. On the contrary,
it's a learning process for all of us. No single application, network
or system is the same. Each company has its own risk model, which
means that there often is no one-size-fits-all solution.

It's about "how can we improve the security" for everybody. That's why I think it's so important that penetration testers should lead by example. Apply proper operation security procedures themselves.

Recently my first Pluralsight course was published, operational security for penetration testers. It deals with what opsec is, and how to apply it to your penetration testing workflow. The trailer of the course can be found at https://www.youtube.com/watch?v=DSF6XbCxYGY. The course itself can be found on Pluralsight's site, https://www.pluralsight.com/courses/opsec-penetration-testers

As beautifully stated by the third law of OPSEC: "If you are not protecting it, the adversary wins".

Use Emacs to create OAuth 2.0 UML sequence diagrams

2017-02-23T03:11:00+00:00

It seems that the OAuth 2.0 framework is more and more being used by web (and mobile) applications. Great !

Although the protocol itself is not that complex, there are a number of different use-cases, flows and implementations to choose from. As with most things in life, the devil is in the detail.

When reviewing OAuth 2.0 implementations or writing penetration testing reports I like to draw UML diagrams. That makes it easier to understand what's going on, and to spot potential issues. After all, a picture is worth a thousand words.

This can be done extremely easy using the GPL-licensed open source Emacs editor, in conjunction with the GPL-licensed open source tool PlantUML (and optionally using Eclipse Public Licensed Graphviz).

Emacs is worlds' most versatile editor. In this case, it's being used to edit the text, and automatically convert the text to an image. PlantUML is a tool which allows you to write UML in human readable text and does the actual conversion. Graphviz is visualization software, and optionally - in this case, it's used to show certain images.

Download the compiled PlantUML jar file, Emacs and optionally download and install Graphviz.

Once you have Emacs installed and running, the following Lisp code (actually configuration) in your startup file (~/.emacs.d/init.d ) will

configure org-mode (a mode to organize and edit text files) to use PlantUML
add plantuml to the recognized org-babel languages (which allows you to execute source code from within text files)
allow the execution of PlantUML code as "safe"
automatically show you the resulting image

 ;; tell org-mode where to find the plantuml JAR file (specify the JAR file)
(setq org-plantuml-jar-path (expand-file-name "~/plantuml.jar"))

;; use plantuml as org-babel language
(org-babel-do-load-languages 'org-babel-load-languages '((plantuml . t)))

;; helper function
(defun my-org-confirm-babel-evaluate (lang body)
"Do not ask for confirmation to evaluate code for specified languages."
(member lang '("plantuml")))

;; trust certain code as being safe
(setq org-confirm-babel-evaluate 'my-org-confirm-babel-evaluate)

;; automatically show the resulting image
(add-hook 'org-babel-after-execute-hook 'org-display-inline-images)

If you don't have a startup file yet, add this code to the file ~/.emacs.d/init.el and restart Emacs.

Hint: Control-c Control-f allows you to create/open a (new) file. Control-x Control-s saves a file, and Control-x Control-c exits Emacs.

That's it!

To test the configuration, create/open a file (Control-c Control-f ) with the extension .org , e.g. test.org . This makes sure that Emacs switches to "org-mode" and recognizes the "org-babel" syntax.

Insert the following code, and press Control-c Control-c within the code to test the installation:

#+BEGIN_SRC plantuml :file test.png
@startuml
version
@enduml
#+END_SRC

If everything went well, you'll see an image appearing inside Emacs, below the text.

Note

To quickly insert code snippets like #+BEGIN_SRC and #+END_SRC , you can use the built-in Easy Templates system: Type <s followed by a TAB , and it will automagically insert a template for you.

For a more advanced example, this is the UML source code used to generate the image above:

#+BEGIN_SRC plantuml :file t:/oauth2-abstract-protocol-flow.png
@startuml
hide footbox
title Oauth 2.0 Abstract protocol flow
autonumber
actor user as "resource owner (user)"
box "token stays secure" #FAFAFA
participant client as "client (application)"
participant authorization as "authorization server"
database resource as "resource server"
end box

group user authorizes client
client -> user : request authorization
note left
**grant types**:
# authorization code
# implicit
# password
# client_credentials
end note
user --> client : authorization grant
end

group token is generated
client -> authorization : request token\npresent authorization grant
authorization --> client :var:  access token
note left
**response types**:
# code
# token
end note
end group

group resource can be accessed
client -> resource : request resource\npresent token
resource --> client : resource
end group
@enduml
#+END_SRC

Don't you just love the versatility of Emacs, and open source tools ?

It's about the journey: Compiling 64-bit Unison / GTK2 on Windows

2016-12-24T03:23:00+00:00

The excellent MSYS2 (mingw64 and mingw32) subsytem makes compiling native Windows compilations "as easy as compilation can be". However, as with everything in life, sometimes when trying to do one thing (compile a program), you end up chasing other vaguely related issues (one exotic compile error after another).

For synchronizing files between servers and workstations I use the open source GPLv3 licensed Unison File Synchronizer [1]. Although the text interface version of Unison compiles straight-out-of-the box on mingw64, the GTK2 interface proved to be a bit more cumbersome.

To compile Unison with the GTK2 interface, lablgtk [2] is needed, an OCaml interface to GTK.

So, the journey began with firing up a shell in a fresh mingw64 environment, and installing the build prerequisites:

pacman -Sy --noconfirm base-devel git \
mingw-w64-x86_64-{glib2,gtk2,ocaml,pango,toolchain}

After downloading the latest source (2.18.5 [3]) and trying to compile it (using make ) after running

./configure --prefix=/mingw64 --disable-gtktest

the first error message is shown:

mingw64/include/gtk-2.0/gdk/gdkwin32.h:40:36: fatal error: gdk/win32/gdkwin32keys.h: No such file or directory

It seems that GTK2 version 2.24.31 contains an error [4], and incorrectly still references the file gdkwin32keys.h. This could be solved by removing the reference:

 sed -i "s/#include <gdk\/win32\/gdkwin32keys.h>/\/\/#include <gdk\/win32\/gdkwin32keys.h>/" \
/mingw64/include/gtk-2.0/gdk/gdkwin32.h

After retrying the build process for lablgtk using make , the next error message pops up during the build process:

Error on dynamically loaded library: .\dlllablgtk2.dll: %1 is not a valid Win32 application.

Apparently flexlink 0.34 (which is installed as dependency of OCaml) has an issue [5] with building 64 bit applications on MSYS2 / mingw64. This could be solved by stripping the incorrect library of its debug symbols:

strip src/dlllablgtk2.dll

And indeed, make && make opt works after this. Next up is installing all lablgtk files in the correct locations:

make old-install INSTALLDIR=/mingw64/lib/ocaml/lablgtk2/ BINDIR=/mingw64/bin/ \
DLLDIR=/mingw64/lib/ocaml/stublibs/

After this, the final destination is in sight: cloning, configuring and compiling the git version of Unison for GTK2.

pushd /tmp
git clone --depth=1 https://github.com/bcpierce00/unison
cd unison
make windres
make src OSARCH=win32gnuc

While re-validating these instructions in 2017, I ran into another build issue:

error: 'caml__frame' undeclared (first use in this function)

In file included from ./lwt/lwt_unix_stubs.c:8:0:
./lwt/lwt_unix_stubs.c: In function 'invoke_completion_callback':
mingw64\lib\ocaml/caml/memory.h:236:12: error: 'caml__frame' undeclared (first use in this function)
  (void) caml__frame, \
         ^
mingw64\lib\ocaml/caml/memory.h:236:12: note: in definition of macro 'CAMLxparam2'
  (void) caml__frame, \
         ^~~~~~~~~~~
./lwt/lwt_unix_stubs.c:82:3: note: in expansion of macro 'CAMLlocal2'
CAMLlocal2 (err, name);
^
mingw64\lib\ocaml/caml/memory.h:236:12: note: each undeclared identifier is reported only once for each function it appears in
  (void) caml__frame, \
         ^
mingw64\lib\ocaml/caml/memory.h:236:12: note: in definition of macro 'CAMLxparam2'
  (void) caml__frame, \
         ^~~~~~~~~~~
./lwt/lwt_unix_stubs.c:82:3: note: in expansion of macro 'CAMLlocal2'
CAMLlocal2 (err, name);
^
make[1]: *** [Makefile.OCaml:434: lwt/lwt_unix_stubs.o] Error 2
make[1]: Leaving directory 'unison/src'
make: *** [Makefile:14: src] Error 2

It seems that the current Unison code does not properly follow OCaml's guidelines with regards to interfacing. From the OCaml documentation [6]

Rule 1
    A function that has parameters or local variables of type value must
    begin with a call to one of the CAMLparam macros and return with
    CAMLreturn, CAMLreturn0, or CAMLreturnT. In particular, CAMLlocal and
    CAMLxparam can only be called after CAMLparam.

A simple call using CAMLparam0() fixes that problem.

--- a/src/lwt/lwt_unix_stubs.c       2017-06-30 04:46:47.061185900 +0000
+++ b/src/lwt/lwt_unix_stubs.c       2017-06-30 04:45:42.023470300 +0000
@@ -79,6 +79,7 @@

static void invoke_completion_callback
(long id, long len, long errCode, long action) {
+  CAMLparam0();
   CAMLlocal2 (err, name);
   value args[4];
   err = Val_long(0);

A pull request was sent upstream (https://github.com/bcpierce00/unison/pull/82) which has been merged in October 2017.

And after applying the latest patch...hooray, the binary gets built successfully, and when starting the unison.exe executable under the mingw64 subsytem, the GTK2 interface is shown. Note that the windres command is to make sure that the resource files are 64-bit compatible.

If you want to run the unison.exe executable standalone, on another system or without the MSYS2 / mingw64 subsytem, you'll need the 64-bit GTK2 library files as well as the two MSYS2 libraries libgcc_s_seh-1.dll and libstdc++-6.dll (as flexlink currently doesn't allow static builds). These can found and copied using the following commands:

YOURTARGETDIR=/your/target/search/path
cp
/mingw64/bin/{lib{atk-*,bz2-*,cairo-*2,expat-*,ffi-*,fontconfig-*,freetype-*,gailutil-18,gcc_s_dw2-*,gcc_s_seh-*,gdk*-2.*,gio-2*,glib-2*,gmodule-2*,gobject-2*,gthread-2*,gtk-win32-2*,harfbuzz-*,graphite2,iconv-*,intl-*,pixman-*,pango*-1*,pcre-1,png*-*,stdc++-*,winpthread-*},zlib1}.dll $YOURTARGETDIR
mkdir -p $YOURTARGETDIR/lib/gdk-pixbuf-2.0/2.10.0
cp /mingw64/lib/gdk-pixbuf-2.0/2.10.0/loaders $YOURTARGETDIR/lib/gdk-pixbuf-2.0/2.10.0/

However, after the first try of running unison.exe in a windows shell, with the GTK2 files in a search path, Unison greeted me with the following message:

Fatal error - Uncaught exception Gdk.Error ("Gdk.Pixmap.create_from_xpm_data")

After some searching it became apparent that image (xpm) support is modular, and therefore the Gdk Pixbuf cache (which loads the image modules) needs to be rebuilt. This can be done by copying the file gdk-pixbuf-query-loaders.exe and executing it once on/in the standalone system/path:

gdk-pixbuf-query-loaders.exe --update-cache

This might show some warnings, and will create the file lib/gdk-pixbuf-2.0/2.10.0/loaders.cache. This file contains necessary information for Gdk Pixbuf, so that xpm files can be loaded.

Indeed, this was the final step in running the latest version of Unison standalone on a 64-bit system, with the latest GTK2 64-bit library files.

This was by far a linear path and took me more time than anticipated. However, the satisfaction at the end and the accumulated knowledge makes it worth the journey. Every time.

For debugging purposes I can heartily recommend the Windows version of ldd, ntldd [7]

Summary

To summarize, here are all steps to compile the latest master branch of Unison from scratch on a vanilla mingw64 (MSYS2) installation, using flexdll version 0.34, gcc version 6.2.0, GTK2 version 2.24.31, lablgtk version 2.18.5 and OCaml version 4.02.3:

pacman -Sy --noconfirm base-devel git mingw-w64-x86_64-{glib2,gtk2,ocaml,toolchain}
sed -i "s/#include <gdk\/win32\/gdkwin32keys.h>/\/\/#include
<gdk\/win32\/gdkwin32keys.h>/" /mingw64/include/gtk-2.0/gdk/gdkwin32.h

VERSION=2.18.5
pushd /tmp
wget -c https://forge.ocamlcore.org/frs/download.php/1627/$VERSION.tar.gz
tar -xzvf $VERSION.tar.gz
cd lablgtk-$VERSION
./configure --prefix=/mingw64 --disable-gtktest
make ; strip src/dlllablgtk2.dll
make
make opt
make old-install INSTALLDIR=/mingw64/lib/ocaml/lablgtk2/ \
BINDIR=/mingw64/bin/ DLLDIR=/mingw64/lib/ocaml/stublibs/

pushd /tmp
git clone --depth=1 https://github.com/bcpierce00/unison
cd unison
# Apply patch https://patch-diff.githubusercontent.com/raw/bcpierce00/unison/pull/82.diff
make windres
make src OSARCH=win32gnuc

Pango errors

If you see the message Error: Unbound module Pango , then the OCAMLFIND_CONF path might be incorrect. This can be fixed by setting and exporting the path to the correct value, where findlib.conf lives on your file system. Then, (re)run the make commands.

Example:

export OCAMLFIND_CONF=/mingw64/etc/findlib.conf

lablgtk errors

If you see the message Error: Unbound value Gpointer.region_of_bytes , then the lablgtk version that you're using is older than 2.8.16. You can either upgrade lablgtk, or revert the change from commit 2e7ea9 [8] on 2018-02-26:

git revert --no-commit 2e7ea9

Changelog

2018-05-14: Added lablgtk error message and solution
2017-11-05: Added Pango error message and solution

[1]	https://github.com/bcpierce00/unison

[2]	http://lablgtk.forge.ocamlcore.org/

[3]	https://forge.ocamlcore.org/frs/download.php/1627/2.18.5.tar.gz

[4]	http://osdir.com/ml/commits.gnome/2016-11/msg06962.html

[5]	https://github.com/alainfrisch/flexdll/issues/6

[6]	http://caml.inria.fr/pub/docs/manual-ocaml-4.04/intfc.html

[7]	https://github.com/LRN/ntldd

[8]	https://github.com/bcpierce00/unison/commit/2e7ea9481c

Why Sharing Improves Us

2016-12-16T05:38:00+00:00

If you're a perfectionist, it's difficult to release a product: Whether that's source code, a pentest report or a blogpost. It's always a work in progress, and never finished.

That's why I like open sourcing code for example, releasing it for everybody to see. Knowing beforehand that the code, your work will be read by others (while you're working on it) forces you to think longer, deeper and harder about the variable names, the structures, function names and coding styles.

I'm the lead pentester for a company where we allow the customer to peek over our shoulder while we're working. The customer can see everything that we try, do and find out during the pentest. This improves the relationship with the customer, as s/he sees what we're doing and even can think along with us.

It also improves the customer satisfaction, as they know exactly what they're getting. And, it improves the mutual respect. Instead of becoming a classical us-versus-them pentest (the pentesters versus the developers), it becomes a 'let's improve the overall security together' exercise.

According to all the positive feedback we're receiving, we're onto something here. A win-win.

Sunlight is not only a great disinfectant, it's also a great energy source for growth.

Verify security vulnerabilities: A collection of Bash one-liners

2016-11-25T02:09:00+00:00

There's manual pentesting and writing reports, and there is blindly
copying the output of automated scantools. I am fortunate enough to
write and review a lot of pentest reports, as well as read pentest
reports of a other companies.
Nothing looks as bad as "vulnerabilities" in a report that haven't
been verified as such. This really degrades the quality of the report.

Below is a number of simple one-liners that can help with verifying vulnerabilities. All examples can be run in a basic shell (Bash, zsh), where the TARGET variable contains the hostname of the target that needs to be verified (without protocol).

SSL/TLS: BREACH

for compression in compress deflate exi gzip identity pack200-gzip br
bzip2 lzma peerdist sdch xpress xz; do curl -ksI -H "Accept-Encoding:
$compression" https://$TARGET \| grep -i "content-encoding:
$compression"; done

*Might* be vulnerable when: one or more compression methods are shown.

SSL/TLS: Client-Initiated Secure Renegotiation

echo "R\nQ" | timeout 10 openssl s_client -connect ${TARGET}:443

Vulnerable when: Renegotiation is successful (exit code == 0)

HTTP: TRACE enabled

for proto in http https; do echo testing ${proto}://${TARGET};
curl -qskIX TRACE ${proto}://${TARGET}\|grep -i TRACE; done;

Vulnerable when: the verb TRACE is shown

HTTP: Open (Secure) Redirect

for proto in http https; do curl -sIH "Host: vuln" ${proto}://${TARGET}/ |
grep -i "Location: https\\?://vuln/"; done

Vulnerable when: The Location header is shown

Note that the checks for BREACH and Open Secure Redirect can be found in the latest version of analyze_hosts.py - use the

--http

flag.

Zen provisioning: Bootstrap the installation of Ansible using Vagrant

2016-10-07T04:20:00+00:00

I'm a big fan of the DevOps attitude of "cattle" versus "pets": machines should be built in a repeatable, automated and consistent way. If there's something wrong, don't be afraid to replace a sick "cow" instead of trying to revive your "pet".

This Zen mindset also helps when preparing for demos, trainings and workshops: Usually I need a number of machines, and what better way than create them by using automation ? For that I'm using the tools Ansible, Packer, Vagrant and VirtualBox - they are all Open Source and can be used on a number of platforms (e.g. Windows, Linux and Mac OS X).

Ansible is a tool for managing systems and deploying applications, licensed under the GNU General Public License version 3 (my personal favorite).

Vagrant is a tool for managing virtual machines and is licensed under the MIT license.

VirtualBox is a virtualization environment for local use, licensed under the GNU General Public License version 2.

Packer creates a machine image by installing an operating system to a multitude of local and cloud platforms, for example VMWare, VirtualBox as well as Docker, Amazon EC2 and DigitalOcean. Packer is licensed under the Mozilla Public License Version 2.0.

How to solve the bootstrap problem and use Ansible to provision machines if you haven't got Ansible installed (yet) ?

This post contains the two (!) simple steps on how to set up a Vagrant Box containing an Ansible server, packaged as a Vagrant box. This allows you to use Ansible for provisioning all of your (future) (VirtualBox) installations.

Prerequisites

Make sure that the following tools are installed, up-to-date and can be executed from the command line.

Vagrant : https://www.vagrantup.com (to configure and spin up virtual machines)

VirtualBox : https://www.virtualbox.org/wiki/Downloads (the virtualization environment)
Start your favorite shell (e.g. zsh or Bash), and verify that the tools are working:
```
vagrant version && VBoxManage --version && echo "Hey Ho Let's Go"
```

Clone the repository containing the necessary Ansible and Vagrant files:

git clone https://github.com/PeterMosmans/easy-provisioning && pushd
easy-provisioning

1a: Build and provision Ansible on Debian

Before you start, if you want to add your own user account and public SSH key to the Ansible server, create the file

ansible/group_vars/personal.yml

- there is an example called ansible/group_vars/personal-example.yml in the repository (see for more information the corresponding Ansible Galaxy role, bootstrap).

The following one-liner will spin up a Debian box named

bootstrap

, install Ansible on it, and provision it for use with VirtualBox. It will also compact the box, and shut it down:

pushd vagrant/ansible-server && VAGRANT_VAGRANTFILE=Vagrantfile.bootstrap
vagrant up

What happens ? By using the

VAGRANT_VAGRANTFILE

environment variable, an alternative Vagrantfile is specified,

Vagrantfile.bootstrap

. This file contains parameters that will download a stock Debian installation, installs the pip version of Ansible, installs the Ansible Galaxy roles specified in

ansible/requirements.yml

, runs the playbook

ansible/ansible-server.yml

, and runs a script to compact the box (that was installed by one of the Ansible roles).

1b: Package the Ansible server

Use the following one-liner command to package the Ansible server as a new Vagrantbox, and add it to the local catalog, ready to be used:

VAGRANT_VAGRANTFILE=Vagrantfile.bootstrap vagrant package bootstrap --output
ansible-server.box && vagrant box add ansible-server.box --name
ansible-server

That's all there is to it! Now, you can spin up the Ansible server box anytime using the command

vagrant up

in the

vagrant/ansible-server

directory. The directory

/ansible

in the repository is mapped on the VirtualBox to

/etc/ansible

, so the data persists across Vagrant 'reboots'.

Optionally you can shut down and remove the bootstrap box (as it's already imported) using

VAGRANT_VAGRANTFILE=Vagrantfile.bootstrap vagrant destroy && rm
ansible-server.box

Enjoy the Zen and power of Ansible...

Compile Emacs for Windows using MSYS2 and mingw64

2016-09-19T14:25:00+00:00

Emacs 25.1 was officially released on September 17th, 2016. The excellent MSYS2 subsystem and the open source gcc compiler make it super-easy to build binaries on/for Windows (7, 8, 10). In three easy steps from source to binaries:

1: Install and prepare the MSYS2 subsytem

Download and run the installer at http://repo.msys2.org/distrib/msys2-x86_64-latest.exe After installing, run MSYS2 64bit which drops you in a Bash shell. Update all packages using the following command:

pacman -Syuu

Sometimes updates of the runtime/filesystem can cause update errors. This is no cause for panic - kill and restart the terminal. For building 64-bit Windows binaries, always use mingw64.exe to start the terminal.

Install all packages necessary for building:

pacman -Sy --noconfirm base-devel git \
mingw-w64-x86_64-{giflib,gnutls,jbigkit,lib{jpeg-turbo,png,rsvg,tiff,xml2},toolchain,xpm-nox}

2: Clone the Emacs source

To simplify building, you can define the environment variables BUILDDIR (where the binaries are built), INSTALLDIR (where the binaries will be installed to), and SOURCEDIR (where the source lives, the git repository). Note that since you're in the MSYS2 subsystem, paths are Unix-style, using forward slashes. This command creates SOURCEDIR if it doesn't exist yet, clones the repository in SOURCEDIR , and runs the autogen.sh script to create a configure script.

 SOURCEDIR=c:/var/git/emacs && mkdir -p $SOURCEDIR && cd $SOURCEDIR &&
 git clone git://git.savannah.gnu.org/emacs.git . --depth=1 &&
./autogen.sh

3: Update, build and install Emacs

This updates the repository, empties and/or creates BUILDDIR , and runs make to build a binary. Afterwards it installs a stripped version of Emacs to INSTALLDIR .

SOURCEDIR=c:/var/git/emacs INSTALLDIR=c:/programs/emacs \
BUILDDIR=t:/emacs && pushd $SOURCEDIR && git reset --hard HEAD &&
git fetch && git checkout origin/master && rm -rf $BUILDDIR &>/dev/null
mkdir -p $BUILDDIR; pushd $BUILDDIR && $SOURCEDIR/configure && make &&
make install-strip prefix=$INSTALLDIR

And listo! After having installed MSYS2, only step 3 needs to be repeated to update the binary to the latest git version.

Optional: copy necessary libraries

To run Emacs on a system without MSYS2, one needs to copy a number of libraries to the installation directory.

INSTALLDIR=c:/programs/emacs cp
/mingw64/bin/{zlib\*,lib{winpthread-\*,Xpm-noX\*,gomp-\*,gcc\_s\_seh-\*,glib-\*,intl-\*,iconv-\*,gobject-\*,ffi-\*,gio-\*,gmodule-\*,freetype-\*,pixman-\*,png\*,expat-\*,harfbuzz-\*,gnutls-\*,gnutlsxx-\*,tiff-\*,tiffxx-\*,jpeg-\*,gif-\*,bz2-\*,jbig-\*,gmp-\*,hogweed-\*,nettle-\*,p11-kit-\*,tasn1-\*}}.dll
$INSTALLDIR/bin

Are you a PenTexter ? Now in BSidesLV

2016-07-26T08:15:00+00:00

If you're around in Las Vegas during BlackHat / BSidesLV / DefCon, I'll be presenting at BSidesLV 2016 together with Melanie Rieback on an open source pentest reporting framework.

The talk will announce a new OWASP project: PenText, a fully open-sourced XML-based pentest document automation system. The PenText system is a document automation framework that supports the entire pentesting lifecycle: from the initial inquiry, through pentest scoping, quotations, pentesting, and reporting, through the final invoice.

During this talk, Melanie and I will demonstrate the OWASP PenText system live, in the context of our larger Pentesting ChatOps infrastructure (RocketChat, Hubot, and Gitlab). We will describe the basics of how the OWASP PenText system is architected (XML, XSLT, XSL-FO), and show how the system can be used to manage the entire lifecycle of pentesting data, including the automatic generation of documentation at various points in the process (including quotations, pentest reports, and invoices).

Lots of ChatOps in combination with pentesting and open source.

See https://bsideslv2016.sched.org/event/7azc/are-you-a-pentexter-open-sourcing-pentest-reporting-and-automation

See you there ?

Verifying webserver compression - BREACH attack

2016-06-28T03:21:00+00:00

A few lines of Bash script let you check which compression methods are supported by a SSL/TLS-enabled webserver.

 target=URL-OF-TARGET
for compression in compress deflate exi gzip identity pack200-gzip br
bzip2 lzma peerdist sdch xpress xz; do
curl -ksI -H "Accept-Encoding: ${compression}" https://${target} | grep -i ${compression}
done

If you see any output (and the server supports one of these compression algorithms), the site might be vulnerable to a BREACH attack. Might, because an attacker has to 'inject' content into the output (and have some control over it): This is called a chosen plaintext attack.

By carefully injecting certain content to the page, an attacker is able to deduce (parts) of the page content by merely looking at the response size (speed). An attacker therefore also has to be able to observe the server's response. A third prerequisite is that the secret (that an attacker wants to steal) is contained in the server response's body, and not 'just' in the response's header. Cookies are therefore out of scope.

The easiest mitigation is to disable HTTP compression completely. Other less practical mitigations are adding random content to each page, which changes the compressed size per page request, rate limiting the requests, or changing the secret per request.

See for more information http://breachattack.com/

Automating OAuth 2.0 / JWT token retrieval for pentests

2016-06-21T02:50:00+00:00

Recently I was pentesting a complex API which used the OAuth 2.0 framework for authentication. Each API call needed an Authorization: Bearer header, containing a valid JSON Web Token (JWT).

To access the API I needed a lot of JWT tokens, as the tokens had a very short expiry time. To facilitate the quick generation of tokens I created a basic script that automated the OAuth authorization: It logs on to a domain, requests an authorization code, and converts that token to an authorization token.

One or more of these steps can be circumvented by command line options (e.g. by specifying valid cookies), to speed up the process.

Another feature of the script is that it automatically performs GET, POST, PUTs and DELETEs with valid tokens against a list of API endpoints (URLs). This preloads all API calls into a(n) (attacking) proxy, and helped the pentest speed up tremendously.

JSON Web Tokens

A JSON Web Token (JWT) is basically a string, representing a collection of one or more claims. Claims are name/value pairs which state information about a user or subject. The claims are either signed using a JSON Web Signature (JWS) or encrypted using JSON Web Encryption (JWE). JSON Web Tokens serve as self-contained authorization tokens, represented using JWS or JWE compact serialization.

A token string consists of a header, payload and optional signature, all base-64 encoded and separated by a dot. An example in python using python-jwt is

# The namespace has changed from jwt to python_jwt from version 2.0.1
import python_jwt as jwt
token = jwt.generate_jwt({'name':'value'}, None, 'none')
print(token)

Output:

eyJhbGciOiAibm9uZSIsICJ0eXAiOiAiSldUIn0.eyJpYXQiOiAxNDY2NDY5NDU5LCAianRpIjogIkE2RDVMOGRyV0VjREdhMTNnRW5lU3c9PSIsICJuYW1lIjogInZhbHVlIiwgIm5iZiI6IDE0NjY0Njk0NTl9.

The first part contains the header, declaring that the encoded object is a JWT, and that the payload is a JWT having no encryption/signature (note that I add some base64 padding for null bytes, as the base64 library mandates this):

import base64
header = 'eyJhbGciOiAibm9uZSIsICJ0eXAiOiAiSldUIn0'
padded_header = header + '=' * (-len(header) % 4)
print(base64.b64decode(padded_header))

Output:

{"alg": "none", "typ": "JWT"}

The payload decodes to another JSON object, containing the claims, and some other items (issue time iat , unique identifier jti and not before time nbf ):

import base64
payload = 'eyJpYXQiOiAxNDY2NDY5NDU5LCAianRpIjogIkE2RDVMOGRyV0VjREdhMTNnRW5lU3c9PSIsICJuYW1lIjogInZhbHVlIiwgIm5iZiI6IDE0NjY0Njk0NTl9'
padded_payload = payload + '=' * (-len(payload) % 4)
print(base64.b64decode(padded_payload))

{"iat": 1466469459, "jti": "A6D5L8drWEcDGa13gEneSw==",
"name": "value", "nbf": 1466469459}

In this instance, the claim consists of 'name' = 'value'.

The last part, the signature is empty, as I chose 'none' as signature algorithm, which resulted in an unsecured JWT.

OAuth 2.0

OAuth is a framework used for authentication: Say user U wants to use API A. To use the API a user has to be authorized by domain D, the so-called Authorization Endpoint.

First the user logs onto the Authorization Endpoint using valid credentials (for example a valid username/password), and asking permission to access API A. If the domain authorizes user U to use the API, it can hand out an authorization code to the user. The user is then redirected to the Redirect Endpoint (usually the API).

This again redirects the user to the Token Endpoint (usually again Domain D), after which the authorization server hands out the token, containing a claim that user U is validated by domain D to use API A.

Now when user U wants to access API A, it sends along a Authorization: Bearer HTTP header, accompanied by the token. In this case, the token was a JSON Web Token.

The next automation step for the script could be that valid tokens are automatically generated, used or renewed by the attack proxy (Burp Suite Pro in my case). Due to a lack of time this hasn't been done during that particular pentest: It's always a fine line when automating pentests still makes sense versus 'manual' testing.

You can find the script at https://github.com/radicallyopensecurity/pentesting-scripts/blob/master/apiborker.py.

Feedback, improvements and especially pull requests are always welcome.

See https://tools.ietf.org/html/rfc7519 for more information on JSON Web Tokens, https://tools.ietf.org/html/rfc7797 for more information on JSON Web Signatures (JWS) and http://oauth.net/2/ on OAuth2.

Getting to know your pentesting tools - curl and the HTTP/1.0 protocol

2016-05-27T03:32:00+00:00

When pentesting, it's always handy to have a bunch of automated scanner do the grunt work for you. Using automated tools saves time and can help in spotting potential vulnerabilities. Usually I run analyze_hosts.py, a wrapper around the open source tools droopescan, nmap, nikto, Wappalyzer and WPscan, with a bit of intelligence built in.

Recently I had a pentesting engagement where nikto flagged a IIS server as leaking the internal IP address (see https://support.microsoft.com/en-us/kb/218180 for more information).

This is a very common issue with older, unhardened IIS servers. The issue is triggered when a HTTP 1.0 request is made to the server, without supplying a Host header. The resulting Content-Location header will contain the server's (private) IP address, thereby leaking information which can subsequently be used for other attacks.

Example of a server's partial response:

HTTP/1.1 200 OK Content-Location: http://1.1.1.1/Default.htm

I ran into an interesting observation when I needed to reproduce this using curl. Curl is a great tool do do all kinds of HTTP requests on-the fly, and it's very well suited for scripting. It has flags to specify the protocol (e.g. --http1.0, --http1.1 and --http2).

However, this only changes the protocol version in the Request-Line (the first line of the HTTP protocol). It doesn't change anything to the request itself. That means if you would use curl with the HTTP/1.0 parameter to verify the vulnerability, it would fail. You wouldn't find it using that command. It does something different than you would expect

Although the RFC 1945 doesn't define the request headers Accept, Connection and Host, when you ask curl to use the HTTP/1.0 protocol it still sends those headers. This means that the request is strictly speaking non HTTP/1.0 compliant, not what you might expect when using the --http1.0 flag.

So when verifying the IIS leaking internal vulnerability, one cannot 'simply' use curl by specifying the HTTP/1.0 flag (-0 or --http1.0). Curl doesn't follow the HTTP/1.0 protocol here to the letter...

The command

curl http://extract.pw/ --http1.0

will result in the following request:

GET / HTTP/1.0 Host: extract.pw User-Agent: curl/7.48.0 \
Accept: */* Connection: close

User-Agent is actually the only request header that is defined in the official HTTP/1.0 RFC. About the 'additional' headers the RFC has the following statement:

"Request-Header field names can be extended reliably only in
combination with a change in the protocol version. However, new or
experimental header fields may be given the semantics of request
header fields if all parties in the communication recognize them to
be request header fields. Unrecognized header fields are treated as
Entity-Header fields."

Even though Accept, Connection and Host are not official part of the HTTP/1.0 specification, they are very common. To make a curl request almost completely HTTP/1.0 compliant, one has to overwrite the non-HTTP/1.0 fields (I say almost, as it seems impossible with curl to overwrite/remove the connection header - this field was introduced in HTTP/1.1 (RFC 2068):

curl http://example.com/ --http1.0 --header 'Accept:' --header 'Connection:' --header 'Host:'

This will result in the following request:

GET / HTTP/1.0 User-Agent: curl/7.48.0 Connection: close

And indeed, with this command one can reproduce the earlier mentioned IIS vulnerability.

Lesson learned: spending time on getting to know (the quirks of) your tools is time well spent...

Automating repetitive git / setup tasks

2016-04-14T06:57:00+00:00

Imagine you work on a large number of projects. Each of those projects has its own git repository and accompanying notes file outside of the repo. Each git repository has its own githooks and custom setup.

Imagine having to work with multiple namespaces on different remote servers. Imagine setting up these projects by hand, multiple times a week.

Automation to the rescue ! Where I usually use Bash shell scripts to automate workflows, I'm moving more and more towards Python. It's cross-platform and sometimes easier to work with, as you have a large number of libraries at your disposal.

I wrote a simple Python script that does all of those things more or less 'automated'. Feed the script the name of the repository you want to clone, and optionally a namespace, patchfile and templatefile variable (either command-line or using a configuration file). The script will then:

clone the repository
modify the repository (e.g. apply githooks)
optionally modify the repository based on given variables
create a new notes file from a template
optionally modify the notes file based on given variables

The advantage is that you can use a configuration file containing the location of the remote git repository, the patchfile and the templatefile. If you setup multiple repositories from the same location and use the same repetitive tasks (e.g. adding git hooks for XML validation), this can save you a lot of time (at least it saves me a lot of time...).

Example of cloning a basic repo to the location /var/git/ :

setuprepo.py git-utilities --remote https://github.com --namespace
PeterMosmans --target /var/git

This will perform a basic clone of the repo https://github.com/PeterMosmans/git-utilities to /var/git/ .

If you usually set up some git-specific parameters like the author's email address, and git hooks for XML validation, then you could apply these automatically. First create a patchfile which applies the hooks (see for an example which adds the git hooks for XML validation), and specify that patchfile:

setuprepo.py git-utilities --remote https://github.com --namespace
PeterMosmans --patch hooks.patch

Besides cloning to the current directory (as no target is specified), this will apply the patchfile git.patch . Before patching, it will automatically replace the keywords NAMESPACE , REMOTE , REPO and TARGET with the corresponding variables.

If you create a notes file for each project from a templates file (outside of the repository directory) you could do that like this:

setuprepo.py git-utilities --remote https://github.com --namespace
PeterMosmans --template MY-NOTES --notes /location/of/notes

Again, this will replace the keywords NAMESPACE , REMOTE , REPO and TARGET in MY-NOTES with the corresponding variables.

And that's where the script really shines: You can specify all parameters in a configuration file, and specify that configuration file on the command line. So, you're setting up the remote, the namespace and the locations in the config file once, and you can use it a gazillion times:

setuprepo.py NAME-OF-REPO --config github.yml

The format of the configuration file (in this example github.yml ) is

namespace: PeterMosmans
notes: /location/where/notes/should/be/stored
patchfile: /location/of/patch.file
remote: https://github.com
target: /where/repositories/are/cloned/to
template: /base/template.txt

You can find the script at https://github.com/PeterMosmans/git-utilities/blob/master/setuprepo.py.

Enjoy !

usage: setuprepo.py [-h] [-c CONFIG] [-r REMOTE] [-n NAMESPACE] [--no-clone]
                    [--no-modify] [--no-patch] [--no-template]
                    [--patchfile PATCHFILE] [--prepare] [-t TARGET]
                    [--notes NOTES] [--template TEMPLATE] [-v]
                    repo

setuprepo.py version 0.6 - Clones and configures (patches) a git repository

Copyright (C) 2016 Peter Mosmans [Go Forward]
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.

positional arguments:
  repo                  repository name

optional arguments:
  -h, --help            show this help message and exit
  -c CONFIG, --config CONFIG
                        load config file
  -r REMOTE, --remote REMOTE
                        remote repository address
  -n NAMESPACE, --namespace NAMESPACE
                        namespace of the repository
  --no-clone            do not clone repository
  --no-modify           do not modify patch and/or template
  --no-patch            do not patch repository
  --no-template         do not create template
  --patchfile PATCHFILE
                        patchfile for new repositories
  --prepare             clone the repo, create an original and modified
                        folder, and exit
  -t TARGET, --target TARGET
                        local target for the repository structure
  --notes NOTES         the directory where to create a notes file
  --template TEMPLATE   an empty notes template
  -v, --verbose         Be more verbose

Bash vs Python (dependency hell)

2016-01-31T04:11:00+00:00

For a number of years I maintained a small collection of open source security scripts, written in Bash. The main purpose of these scripts was to act as a wrapper around other open source tools. Why try to remember long and awkward command line parameters, when you can ask a script to do that for you ?

Bash was chosen, as it was distribution-independent. It works almost everywhere (although sometimes OSX support is troublesome, due to outdated Bash versions).

After more and more (requested) features crept in, the

analyze_hosts.sh

Bash script became more and more complex. That's why I decided to port the script to Python. In my experience, it's at-least-as portable, and the usage of third party (pip) packages means that less time is spent on re-inventing the weel, and more on the actual functionality.

Yes, sometimes people talk about the dependency hell of Python, and in
some cases, the usage of third party packages means you have to be
careful of what you're doing.
However, when using virtual environments each Python script and its
dependencies can be safely separated from the 'main' Python. For
example, the following commands create a separate virtual environment
for the security scripts repo:
`` cd security-scripts virtualenv .``

Voila, a safe working environment. As Linux and Windows virtual environments differ a bit, the following commands can be used to switch to the virtual environment and works on either platform.

`` [[ -f bin/activate ]] && source bin/activate [[ -f Scripts/activate ]] && source Scripts/activate``

The following command installs the recommended pip packages (currently
only one):
`` pip install -r requirements.txt``
If you want to switch back to the main environment, use the command
deactivate
.

Although the Python version of

analyze_hosts

script doesn't have all the functionality the Bash version has (yet), it already contains a significant number of other features, and is a bit smarter with regards to working on a large number of hosts.

For instance, when running the script on a subnet, it first creates a queuefile containing all targeted hosts. If a host has been scanned, it will automatically be removed from the queuefile. The script doesn't handle forks and subprocesses (yet), but you can safely execute it in parallel specifying different queue- and output files.

The script can be found on https://github.com/PeterMosmans/security-scripts/.

As always, feedback and pull requests are appreciated :smile:

Open secure redirect

2015-12-08T07:33:00+00:00

Aren't those RFC docs amazing ? Reading up on standards ?

I needed plenty of time for them, as I encountered some interesting issues. As it turned out, some websites / loadbalancers are overly optimistic in encrypting all the things - actually, in redirecting all the things.

TL;DR

Never trust HTTP(s) clients, and be careful when setting up redirection rules. A non-RFC compliant client can trigger a (difficult to exploit) open redirect vulnerability, due to a non-RFC compliant server.

This vulnerability can be tested using

analyze_hosts.py --http TARGET

See https://github.com/PeterMosmans/security-scripts/ for the latest version of analyze_hosts.py

Be warned, long post ahead: A while ago I came across some servers that, when being sent insecure requests, responded with a redirect to the secure version.

Request:

% curl -sI http://VICTIM/

Response:

HTTP/1.1 301 Moved Permanently
Connection: close
Location: https://VICTIM/

So far so good, nothing fancy going on here. In fact, this is excellent behaviour. Insecure requests are immediately upgraded to secure requests.

However, the server seemed to be overly happy in redirecting, as it listened to the client-supplied Host parameter:

% curl -s -I -H "Host: MALICIOUS" http://VICTIM/

And the server responded by

HTTP/1.1 301 Moved Permanently
Connection: close
Location: https://MALICIOUS/

Vulnerability

An attacker is able to perform malicious open redirects, if he is able to forge the client's HTTP header.

Truth be told, this is quite difficult to exploit, as this is currently not possible with e.g. XMLHttpRequest (see http://www.w3.org/TR/XMLHttpRequest/#the-setrequestheader%28%29-method).

Still, an attacker could use this attack to redirect unknowing users, or try to poison caches.

What does the standard say ? Semantic-wise (in RFC speak) for the last request, the target URI equals the absolute-URI and is http://VICTIM/. The authority is VICTIM, the request-target is /, and the uri-host is MALICIOUS.

The latest HTTP/1.1 standard RFC 7230, paragraph 5.4, states the following:

"If the target URI includes an authority component, then a client MUST send a field-value for Host that is identical to that authority component".

So, the client isn't compliant... but as an attacker, we don't have to be compliant. We can do whatever we want.

Next, it states our issue:

"Since the Host header field acts as an application-level routing mechanism, it is a frequent target for malware seeking to poison a shared cache or redirect a request to an unintended server."

Exactly what is happening here. But, should it happen ? According to the RFC (bear with me, lots of theory here):

"If the server's configuration (or outbound gateway) provides a fixed URI authority component, that authority is used for the effective request URI.".

Translation: if the server only serves VICTIM, then it should use that to serve the request URI from.

"If not, then if the request-target is in authority-form, the effective request URI's authority component is the same as the request-target."

Translation: if the request contains a host, then the server should 'overrule' the request URI. This is mostly meant for proxies, for example:

% curl -s -H "Host: TARGET" http://VICTIM/ -x http://PROXY:80

Here, the PROXY will receive a request target of HTTP//VICTIM/, and it should use that as authority, instead of PROXY.

"If not, then if a Host header field is supplied with a non-empty field-value, the authority component is the same as the Host field-value.".

Translation: If the Host header is supplied (TARGET), it should be the same as the authority. But in our 'attack' it's not, authority is VICTIM. So, let's move to the next line.

"Otherwise, the authority component is assigned the default name configured for the server and, if the connection's incoming TCP port number differs from the default port for the effective request URI's scheme, then a colon (":") and the incoming port number (in decimal form) are appended to the authority component.".

And this is where the server VICTIM fails to follow the RFC guidelines. The authority component should be specified by VICTIM, and not by the client.

Most of the times this issue is the result of wrong redirection rules for servers and loadbalancers. The rules incorrectly (ab)use the Host field as authority.

How to test

$ curl -sIH "Host: vuln" http://$VICTIM/|grep -q "Location: https\?://vuln/" \
&& echo "Server is vulnerable to open secure redirect"

The

analyze_hosts.py

script on https://github.com/PeterMosmans/security-scripts supports it now as well, with the

--http

option.

Mitigation / solution

Make sure that you use the correct authority, as specified (or served) by the server, and do not blindly redirect to the Host 'authority', as specified by the client.

Stay safe!

automatic XML validation when using git

2015-09-29T13:51:00+00:00

Recently I worked on a project which involved manually editing a bunch of XML files. Emacs is my favorite ~operating system~ editor, and it has XML validation built in (using the nXML mode). It highlights validation errors while-you-type. Unfortunately, even with Emacs showing potential issues in RED COLOR, I managed to commit a number of broken XML files to my local git repository. Subsequently when I pushed my errors to the remote 'origin' git repository, the errors broke builds.

Of course this can be completely prevented by locally using pre-commit hooks. If your local git repository validates XML files before you can commit them, and denies invalid XML files, then one part of the problem is solved.

A pre-receive hook on the receiving server side can do the same as a pre-commit hook locally: Validate XML files before letting somebody push a commit which can break the build process.

I looked around the Internet but couldn't find a lightweight quick script to do only and exactly that. That's the reason I whipped up a basic pre-commit and pre-receive hook, written in Python.

You can find the very basic and rough code at https://github.com/PeterMosmans/git-utilities. By changing the variable

reject

in the hook scripts the behaviour can be changed to reject commits and updates on validation errors, or not.

(Of course, the most obvious solution would be that I pay more attention to RED COLORED ERRORS...)

Preparing your team for a CTF competition - Defcon style

2015-08-15T14:35:00+00:00

Playing Capture The Flag with a team on location is something completely different than performing penetration tests, security assessments or even trying to solve CTF challenges over the Internet.

At Defcon 23 I joined a team of really knowledgeable, nice and friendly people for the OpenCTF competition. It was an exhilarating ride from setting up all equipment to the glorious finish. Playing Capture The Flag on Defcon was educational but foremost fun, fun and fun.

So why would you spend a good chunk of 48 hours sitting in a chair behind a screen while there is so much more to see and experience at Defcon ? In one word: The undescribable exciting atmosphere of playing during a conference, of competing against all these bright people from all over the world, desperately trying to solve the challenges.

Here are some of my personal notes on how to get the most out of competing in an OpenCTF competition with a team:

Allow plenty of time before the competition to set up (and harden - don't be a fool like me) your machine. Make sure you have all necessary tools and notes.
Make sure beforehand that all team members have one communication channel (eg. IRC) readily available, and ways of sharing files (eg. USB sticks, pastebin or dropbox). Share the knowledge. Our team first met up at Defcon 23, so this was important for us.
Read the instructions of the challenge. Read them again. And preferably again. Look at the challenge. This was for me an eye opener, as I got the feeling that didn't allow myself enough time reading into each challenge.
Don't forget to make notes continuously while playing. If you're stuck, if there aren't challenges to your liking open or if the competition floor is closed, you can always do write ups (please don't hold it against me, but I really like doing write ups).
Make sure you're comfortable with setting up networking, routing and, most important (local) firewalls. Several challenges needed network tweaks like VLAN settings. You could also go on the offense and attack other teams, so be prepared.
Bring hardware. Lots of it. Patch cables, USB sticks, chargers, foreign adaptersmonitors, you-name-it. One of our team members brought a switch, and it made him the instant uncrowned king of our team.
The advantage of being in one location is that you can easily communicate with each other. I really enjoyed walking around to other team members, seeing what they were up to and hearing their ideas. Bouncing theories around is so extremely valuable. If you have pair programming, why not pair hacking ? And Extreme Hacking is born...
The helpful organizers Team Vand were walking around all day. It seemed like they really enjoyed it when you asked them about the challenges. Why not walk up to them more often when you're stuck ?
With regards to technical skills, I felt that it was an immense advantage if you're pretty confident with a glue language like e.g. Python. Writing (simple) file parsers, doing basic byte conversions and other file operations is a plus.
Remember to have fun, it's a social thing.

The learning experience was my most important goal of joining a CTF team and the competition. In that regard Defcon 23 was a huge success. Not only that, but our team also ended up at a rewarding 12th place out of the 61 teams. Something that I'm really proud of.

We will be competing again at Defcon 24, and you should too. See you next year ?

Thanks to the organizers "Team Vand" and last but not least all of our team members. I learned so much from you - You know who you are!

Defcon 23 was great - people are great

2015-08-09T21:51:00+00:00

For quite a while now, I work in the security industry. One of the things I do is providing security advice for companies on all sorts of guidelines, policies and hardening stuff. Web penetration tests is also something I do very regularly. In other words, a disclaimer before you read on: I should have known better...

VirtualBox, Packer, Vagrant and Ansible are tools that I use a lot.
These four tools make virtualizing and provisioning really easy. You
can create new machines, experiment with them and test different
setups in a repeatable and automated way.
As I sometimes organize pentesting workshops, I have several virtual
machines with Kali (a penetration testing distribution) installed on
them readily availabe.
So, I connected my laptop to the network of the 23rd Defcon conference
in Las Vegas, when one of these standard Kali virtual machines was
(still) running as guest on my machine. Not only was Kali running, the
guest was also configured to run in bridged networking mode. This
means that Kali got it's own network IP address assigned.
What I hadn't changed on that machine was Kali's default root
password. To make matters worse, what I had changed was the ssh server
(which doesn't run by default) that was running now.
Once again, I should have known better. What you should do when you
visit the world's biggest hacker's conference is hardening your
machine, and bring a burner (throwaway) laptop.
What you shouldn't do is exposing the ssh server of a standard Kali
distribution to the network, with the default root password.
Yet, that's exactly what I did...

What surprised me though is how quickly the box was owned. Literally within a minute someone on the network was so kind enough to change the root password for me. Within a minute! I expected hostile traffic, but this turnaround time really was impressive. So helpful!

I immediately shut down the box and will do some offline forensic research on it, just for fun. Thankfully, when you have prepared virtual machines you can be up and running within almost the same time as it took for someone else to hack the box.

Exactly because of my beginner's mistake I liked Defcon so much. It made me feel like a rookie (and rightfully so!). There were so many knowledgeable people, so many clever talks, demonstrations, workshops and games going on. It made me humble to the core. Everybody was so friendly and helpful, I learned a lot.

One of the things I always tell people is that pentesting taught me that people are generally good. You always find vulnerabilities in (web) applications, but people haven't exploited those vulnerabilities yet. People must be good by nature.

So a big thank you to the one owning my virtual machine within a minute. Even though I'm deeply ashamed about my beginner's mistake, thank you for reminding me of how important consistent operational security is. Thank you for reminding me of what the world could look like if people would be evil. But they aren't. People are generally good. Hackers even more so.

Another post will be on the Defcon OpenCTF competition, which I was fortunate enough to compete in.

The future is here: HTTP/2

2015-07-31T22:26:00+00:00

Last month I held a number of presentations on the latest and greatest HTTP/2 protocol. It's an area where there's currently a lot of demand for knowledge and practical tips. Most people are surprised to find out that the're actually already using it on a daily base.

If you're interested you could check out an Ansible role which installs a number of client-side and server-side tools all HTTP/2 enabled:

curl - A data transferring tool with HTTP/2 support
h2load - A benchmarking tool for HTTP/2 and SPDY servers
nghttp - A HTTP/2 client with SPDY support
nghttpd - A HTTP/2 server with SPDY support
nghttpx - A transparent HTTP/2 proxy with SPDY support
openssl - A cryptographic library with ALPN support (1.0.2-chacha)

The following libraries will be installed:

libcrypto - OpenSSL
libcurl - CURL library
libnghttp2 - A HTTP/2 and HPACK C library
libspdylay - A SPDY library
libssl - OpenSSL

You can find the role at https://github.com/PeterMosmans/ansible-role-http2

Safely storing Ansible playbook secrets

2015-06-23T06:30:00+00:00

More and more organizations use dedicated software to safely handle the creation and management of secrets (for example SSL certificate keys, private variables and passwords). Three 'well known' solutions are Square's Keywhiz, Hashicorp's Vault and crypt in combination with etcd or consul.

As with all security solutions the roll-out can be quite cumbersome. The correct implementation (think key management, think audit trails, think key recovery) of any one of these solutions is difficult. And difficult means that most people won't use it, at least not right away (remember SELinux ?).

There are a number of tools available to encrypt secrets within (Ansible) repositories. One of them for instance is Ansible Vault (look here for a more in-depth review). Although the idea of selectively encrypting data is a good one, text-oriented version control systems like git or Subversion aren't meant to store binary blobs of encrypted data. Moreover you still run the risk of accidentally uploading or sharing unencrypted files. Mitigations like adding filenames of unencrypted secrets to a .gitignore file are error-prone.

How to facilitate developers and system operators to store secrets in a safe place, outside the repositories where Ansible playbooks and configuration files are kept ?

This article describes a possible workflow for Ansible users who want to safely store secrets outside of the Ansible repository for use within their playbooks. It assumes that current operators already have access to the correct secrets: Developers have access to the development environment secrets for example, and system operators have access to production environment secrets. How you store these secrets is outside the scope of this article - this is about structuring (Ansible) variables in such a way to facilitate the safe storage of secrets.

The underlying goals are:

Make it as easy as possible for humans to use secrets and store them outside the repository (don't disrupt the current workflow)
Have different security levels depending on the environment (Development, Test, Acceptance or Production)

File structure

This is an example of a file structure, where /ansible is a git repository, and /protected/storage is a protected storage file system. This can be anything. The protected development storage can be for instance a private git repository shared amongst developers (low security clearance), and the production storage could be an encrypted network mount that is only accessible by operators (high security clearance). They don't have to share the same filesystem (see goal 2).

|-ansible
|---/roles
|-----/vars
|-------public_vars
|-----/files
|-------public_file
|-protected
|---/storage
|-----/development
|-------secret_file
|-------secret_vars.yml
|-----/production
|-------secret_file
|-------secret_vars.yml

Ansible server works great from Vagrant for example, where the various Vagrant guest directories are mapped to host directories.

Ansible setup

The design pattern for Ansible is to create an inventory file which groups the hosts into environments having different (security) restrictions. This can be one inventory file or separate ones:

# file: inventory [development] development.domain.com [test] test.domain.com [acceptance] acceptance.domain.com [production] production.domain.com

This means that Ansible will look for variables for the development environment in the current directory under group_vars/development and in /etc/ansible/group_vars/development. Regardless where you store these files (in the current directory or on the ansible server in /etc/ansible), now you can include other files based on the environment. See for example the file group_vars/development...

--- # file: group_vars/development
protected_storage: /protected/storage/development

....and group_vars/production

 --- # file: group_vars/production
protected_storage: /protected/storage/production

The variable protected_storage is set according to the environment. This means that by utilizing this pattern you can store any secret outside of the current Ansible repository, and the secrets will be loaded based on the environment. This can be used not only for variables, but files as well. By using the name protected_storage developers of Ansible playbooks are 'forced' to store secret files inside of that directory. Below is a configuration example to deploy SSL keys:

- name: Deploy private keys
  copy:
  src={{ protected_storage }}/{{ item }}
  dest=/etc/ssl/private/{{ item }}
  owner=root
  group=root
  mode=600
  with_items: secret_file

This deploys the correct SSL keys based on the environment from the protected storage to the server. You can also include files containing secret variables in playbooks:

# file: webservers.yml
- hosts: all
  roles:
  - role: webtier
    tags: webtier
    tasks:
    - include: roles/webtier/tasks/php.yml
      tags: php
      - include: roles/webtier/tasks/wordpress.yml
        tags: wordpress
        vars_files:
        - "{{ protected_storage }}/wordpress.yml"``

When you run this playbook for the development environment, the correct secrets will be included from /protected/storage/development

--- # file: protected-storage/development/wordpress.yml
wp_sites:
- name: www.domain.com
  db_name: database_name
  db_user: database_user
  db_password: secret_password
  plugins:
  - akismet

This structure is relatively easy to implement, doesn't interfere with the current workflow, and separates secrets from the public variables and files. Hopefully this will last until your company does a complete roll-out of Keywhiz, Vault or crypt...

OWASP AppSecEU 2015 review - more and more DevOps

2015-06-15T06:32:00+00:00

This year, the European edition of OWASP AppSec conference was held in
Amsterdam, The Netherlands.
One of the things I really like about OWASP conferences is the
atmosphere. Usually it consists of a nice blend of IT people from
literally all over the world, and this conference didn't disappoint.
One of the added values of visiting such a conference is that you hear
stories from the trenches from peers and likeminded people. It makes
it easier to (try to) spot trends in the security world.

Some observations:

DevOps

I'm a big fan of the DevOps movement, and what it means for security. More cooperation plus more automated testing means more secure systems. Thankfully there were a lot of presentations that focused on how to integrate automated security testing into the continuous deployment pipeline. As the O from OWASP stands for open, mainly open source testing tools were covered, like OWASP ZAP, Arachni and the Gauntlt framework. Some tools still need quite some tweaking to be successful, but the landscape surely is promising.

Dev is running faster than Ops

I'm still under the impression that the DevOps movement is mainly led by developers. The tools that are improving faster are the ones that specifically target a website or application. Although there are a number of automated open source tools for infrastructural testing, like Seccubus or OpenVAS, the changes seem to come from the development side.

Go

Python was the glue-language-of-choice for a lot of operations people. Not anymore, the new go-to language is Go. More and more (DevOps) people seem to choose that nowadays. More minimalistic and faster.

Mind you, these observations are personal and not backed by any scientific measure.

Overall it was a well-balanced conference, with content for builders, breakers and defenders. It seems that OWASP gains more and more traction besides the 'usual security incrowd' - which is a good thing. Learning about breaking security is one thing, but learning how to protect your assets against attackers is more important in my opinion. The more developers join the crusade for a safer Internet the better.

A big thanks to the OWASP AppSec 2015 team for a great conference.

OpenSSL the Ansible vault... using PBKDF2

2015-04-25T12:38:00+00:00

Ansible is a popular open-source software platform for configuring and managing computers. It helps sysadmins to provision new servers in a reliable and repeatable way, and helps developers who want to push their code as fast as possible. It takes scripts (playbooks) as input, which a lot of people can and do share with each other. The beauty of open source. Playbooks can contain sensitive data like passwords and SSL keys - stuff that you don't want to share, or incidentally upload to GitHub.

Last year Ansible added a tool to its arsenal to easily encrypt structured datafiles (containing sensitive data), called Ansible Vault. You can specify a key or keyfile when running a playbook, which decrypts the data on-the-fly. Encrypted data can still be edited

I love it when people make it easier to use encryption. The easier it becomes, the more people will use it, the safer everybody will be.

Another beauty of open source is that you can inspect the code. And modify it! I wanted to be able to encrypt and decrypt the data where/when you cannot use Ansible vault, by using other tools and languages like OpenSSL and Bash script.

Under the hood Ansible vault uses open and known encryption standards, through the Python PyCrypto library. All information is encrypted using the AES cipher in counter (CTR) mode with a key size of 256 bits [1].

A user enters a password, which, together with a salt, serves as input for PBKDF2, a well-known keystretching function. Here, SHA 256 is being used as hash-based pseudorandom function for PBKDF2.

After 10000 iterations PBKDF2 spews out three 'keys': One 256 bit key to encrypt the actual data, one 256 bit key to create a keyed hash (again using SHA 256) over the data, and one 128 bit nonce (initialization vector) [2].

The 10000 iterations is a nice big number to make it harder for attackers to bruteforce the key.

When you use Ansible vault on a plaintext file the contents are overwritten with the encrypted contents. The first line contains a header of three fields separated by colons: A header containing the magic string $ANSIBLE_VAULT, the Vault version number (as of this writing 1.1), and the cipher being used (AES256). The next huge string (truncated on 80 columns) contains a hexadecimal representation of three variables. Each of those variables is also hexadecimal encoded [3]. The three values are a salt, the HMAC and the encrypted data.

To be able to decrypt this data using OpenSSL you first need to stretch a password using PBKDF2 using 10000 iterations and SHA 256. As the stock version of OpenSSL doesn't have this functionality exposed (yet) in the application I created a patch (loosely inspired on an earlier rejected OpenSSL patch). It lets you choose the number of iterations, the pseudo random function and the number of keys. Using this patch you can compile openssl, and use it to create two 256 bit keys and a 128 bit initialization vector from a given salt and password. That's almost all that's needed to convert Ansible vault encrypted files into plaintext, and vice versa: AES-256-CTR and SHA 256 are already supported.

Using the following command, three keys can be derived from password "password" and salt "salt":

openssl enc -pbkdf2 -pass pass:password -S 73616c74 -md sha256 -c 10000 -dklen 80

Almost... all that's missing is a Bash script wrapping everything nicely up. Until next time!

You can find the PBKDF2 patch for openssl on my GitHub repository.

[1]	AES CTR mode is a stream cipher, so padding the data using the PKCS7 standard seems an unnecessary step.

[2]	Since 256 + 256 + 128 bits of output are needed, SHA 512 would probably be a better fit than SHA 256.

[3]	The hexadecimal-encoding-of-hexadecimal-encoded data explains the weird looking file with lots of 3's in it.

Replacing ChaCha20/Poly1305: a new owner

2015-04-07T02:27:00+00:00

A post back I wrote about the 'design goals' of the 1.0.2-chacha fork of OpenSSL - see https://www.onwebsecurity.com/openssl/the-work-flow-of-the-full-featured-openssl-fork-chacha20poly1305.

The ChaCha20 / Poly1305 code in the 1.0.2-chacha fork is originally from the OpenSSL repository, but has since been abandoned there. BoringSSL became its new home, where it's actively being maintained by Google (primarily Adam Langley and David Benjamin). Over time I applied several patches that BoringSSL applied to the ChaCha20 / Poly1305 code, to keep it as up to date as possible.

The issue now is that BoringSSL diverges more and more from the OpenSSL code, which makes it more difficult to maintain (error-prone), and, more important, makes the fork itself diverge too much from OpenSSL.

That's why it's my intention to replace the current ChaCha20 / Poly1305 code from 1.0.2-chacha with more recent attributions that align better with the official OpenSSL code. As far as I understood the official OpenSSL distribution will add ChaCha20 / Poly1305 at some time in the future, which of course would be the best possible outcome. Official support.

Until that time I will do my best to maintain the 1.0.2-chacha branch.

The workflow of the Full-Featured openssl Fork (ChaCha20/Poly1305) 1.0.2-chacha

2015-03-13T08:49:00+00:00

As you might know I maintain a fork of OpenSSL at https://github.com/PeterMosmans/openssl The 1.0.2-chacha fork started out of adding the ChaCha20/Poly1305 ciphers to the official fork, and slowly more and more ciphers and features were added.

The main goals of the fork are

add as much ciphers and (test)functionality as possible
to keep the source as aligned to the original as possible
keep the patches transparent (easily applicable to the original source)
keep the patches maintainable
write as little custom/new code as possible

For 2 (to keep the source as aligned to the original as possible) I try to merge and test the code as often as I can, so that the fork is never too far behind the official repository.

As it was my first idea to start a feature branch I used no-fast forwarding git merges. This kept it transparent when I merged the code, and what the history of the commits was. However, since I'm probably going to maintain this fork besides the official fork I'm going to use fast-forwarding merges from now on (March 2014) whenever possible. I think this will keep the commit history cleaner - see point 3 (keep the patches transparent).

Goal 4 (keep the patches maintainable) is somewhat more difficult. The ChaCha20/Poly1305 source code is from a now-defunct AEAD branch of the official openssl source. Since then, the code has kind of moved to Google's BoringSSL fork, where it is being actively maintained.

In the near future I will try to replace the ChaCha20/Poly1305 with BoringSSL's active codebase. I am aware of a recent patch from Intel (see https://rt.openssl.org/Ticket/Display.html?id=3615) but that conflicts with goal 3 (keep the patches maintainable). Patches are usually a one-off thing, and as you might know security is a dynamic thing. What is secure code today can become insecure code tomorrow.

For complete transparency please find an alphabetical list here of the differences between the 1.0.2-chacha fork and the official OpenSSL_1_0_2-stable fork:

.gitignore
.travis.yml
CHANGES
Configure
Makefile.org
Makefile.shared
apps/s_client.c
apps/speed.c
crypto/chacha/Makefile
crypto/chacha/chacha.h
crypto/chacha/chacha_enc.c
crypto/chacha/chacha_vec.c
crypto/chacha/chachatest.c
crypto/evp/Makefile
crypto/evp/aeadtest.c
crypto/evp/aes_128_gcm_tests.txt
crypto/evp/aes_256_gcm_tests.txt
crypto/evp/chacha20_poly1305_tests.txt
crypto/evp/e_aes.c
crypto/evp/e_chacha20_poly1305.c
crypto/evp/evp.h
crypto/evp/e_aead.c
crypto/evp/evp_err.c
crypto/evp/evp_locl.h
crypto/opensslv.h
crypto/poly1305/Makefile
crypto/poly1305/poly1305.c
crypto/poly1305/poly1305.h
crypto/poly1305/poly1305_arm.c
crypto/poly1305/poly1305_vec.c
crypto/poly1305/poly1305test.c
doc/apps/ciphers.pod
doc/apps/s_client.pod
doc/crypto/EVP_AEAD_CTX_init.pod
engines/Makefile
engines/ccgost/Makefile
ssl/s2_clnt.c
ssl/s2_enc.c
ssl/s2_lib.c
ssl/s2_srvr.c
ssl/s3_clnt.c
ssl/s3_enc.c
ssl/s3_lib.c
ssl/s3_srvr.c
ssl/ssl.h
ssl/ssl3.h
ssl/ssl_ciph.c
ssl/ssl_lib.c
ssl/ssl_locl.h
ssl/ssl_txt.c
ssl/ssltest.c
ssl/t1_enc.c
ssl/tls1.h
test/Makefile
util/libeay.num
util/mkbuildinf.pl
util/mkdef.pl

This list can be created by adding the official openssl repo as upstream for the 1.0.2-chacha branch, and by comparing (diffing) the branches:

git remote add upstream https://github.com/openssl/openssl.git
git diff 1.0.2-chacha upstream/OpenSSL_1_0_2-stable

Please enjoy!

FREAK!

2015-03-04T11:40:00+00:00

As you probably read somewhere else, and on another place, and another... on March 3rd 2015, another attack on SSL/TLS was published. Following the tradition of BEAST, CRIME, Heartbleed, LUCKY13 and POODLE this one also has a catchy name: FREAK (Factoring RSA Export Keys).

It's a man-in-the-middle attack where a man in the middle can decrypt a SSL/TLS connection between a client and a server.

Vulnerable *servers* are servers that accept export-grade ciphers (RSA-EXPORT). Checking whether a server is vulnerable can be done in many ways.

Take for example analyze_hosts.py, a Python wrapper around several tools:

analyze_hosts --ssl HOST

If you see any EXPort ciphers, the server is vulnerable.

Another way is by using cipherscan

cipherscan HOST:443

If you see any EXPort ciphers, the server is vulnerable.

Yet another way is by using nmap:

nmap --script ssl-enum-ciphers -p433 HOST

If you see any EXPort ciphers, the server is vulnerable.

You get the idea...

Mitigate this vulnerability server-side by making sure that your server doesn't allow export ciphers in the OpenSSL configuration: add the following expression

!EXP

There are also vulnerable clients...

Clients using OpenSSL are not vulnerable if they were built after CVE-2015-0204 was published.

The current/latest version of 1.0.2-chacha, the OpenSSL fork including ChaCha20 and Poly1305 is not vulnerable. The fork is being kept up to date on a regular base. See https://www.onwebsecurity.com/cryptography/openssl for the latest Windows binaries, and more information on the source.

Note

This attack method is possible due to the NSA deliberately weakening crypto. Read more on this fascinating story at https://blog.cryptographyengineering.com/2015/03/attack-of-week-freak-or-factoring-nsa.html

To paraphrase cryptographer Matthew Green: "Encryption backdoors will always turn around and bite you in the ass. They are never worth it."

Us versus them: CrikeyCon 2015 review

2015-02-26T11:27:00+00:00

I had the chance to visit CrikeyCon February 2015, which was held in Brisbane

It was the second time this event was held, but it already got the looks and feel of a professional organization behind it. The program was really diverse, from social engineering and awkward hugs to iOS runtime hacking, and everything in between.

Takeaway ? Well, it surprised me to hear that there's still a general feeling of us versus them. We the security gods who lay bare all the faults and stupid mistakes the others make.

As a security professional, especially as a pentester, it's your job to find vulnerabilities and weaknesses. It's your job to hunt for other people's mistakes, lack of knowledge, or constrained security budgets. Security falls into the quality assurance department.

This means that most of the time you're telling other people what's wrong with an application. Unfortunately it's not your job to tell them how awesome their web application is, how well it scales, or the nifty features it has.

One of the more challenging issues when for instance presenting a pentest report to a group of developers is to get everybody on board, to get everybody to work together. And if we are only telling them what they did wrong, well, personally I don't think that's the best way to get everybody behind the same goal of improving the security.

Security needs more cooperation. Let's see what we can do better...

OpenSSL 1.0.2 - now with less whitespace!

2015-02-08T15:34:00+00:00

On January 22 2015, version 1.0.2 of OpenSSL was released. Besides
some new bugfixes and features, the biggest change under the hood was
a complete
reformatting
of the source code. An official coding style
document was
published, and as a result primarily buckets and lots of tabs and
newline characters have been converted into whitespaces.
Personally I hope that this action, which affected the majority of
lines(!) of code, will help the project for the best and will make it
easier to maintain the project in the future.
One disadvantage of reformatting code for instance however is that it
makes it a lot harder to spot differences in code between certain
versions, as almost all files have most of their lines changed.
Another disadvantage is that merging additional patches (like the
ChaCha20 and Poly1305 ciphers) back into OpenSSL took a great deal of
extra time. Unnecessary time, one might say.

The OpenSSL 1.0.2 fork including the ChaCha20 and Poly1305 ciphers has been pushed to the github repo at https://github.com/PeterMosmans/openssl/

As always, you can find compiled Windows 32 and 64 bit binaries at https://www.onwebsecurity.com/cryptography/openssl.

February 2015 update: read more here about the whole reformatting process at OpenSSL

Enjoy!

CVSSv2 rating of new vulnerabilities patched in OpenSSL

2015-01-10T04:45:00+00:00

On January 8th, 2015, the OpenSSL team published an OpenSSL Security Advisory containing 8 previously unknown vulnerabilities in OpenSSL.

Unfortunately, as with most large software suppliers/vendors nowadays, OpenSSL uses its own severity classification system for vulnerabilities. There are many classifications systems out there, which results in less transparent patching policies and procedures for system administrators and end users. Fortunately NIST publishes the CVSSv2 Severity Base Score of most, if not all vulnerabilities with a CVE entry. This makes it easier to classify.

Two of the eight vulnerabilities (CVE-2014-3571 and CVE-2015-0205) have the OpenSSL vulnerability rating 'moderate'. This corresponds to a CVSSv2 base score of 5.0 (MEDIUM) for CVE-2014-2571 as well as CVE-2015-0205 Both of these vulnerabilities could be exploited for a Denial of Service attack of the OpenSSL service. The remaining six vulnerabilities have a lower rating.

The 1.0.2-chacha and 1.0.1-chacha branches of the ChaCha20 - Poly1305 fork of OpenSSL have been patched for all of the published vulnerabilities.

As always, see https://www.onwebsecurity.com/cryptography/openssl for the Windows 32 and 64 bit binaries, and more information.

MSYS2 - successful successor of MSYS ?

2014-12-05T09:34:00+00:00

MSYS can best be described as a Bash shell and some GNU tools, which facilitate compiling sources under and for a Windows environment. It's the environment to use when compiling for instance OpenSSL or Emacs on Windows.

MSYS2 is the 'new and improved' version of MSYS.

One of its biggest advantages is better package management. MSYS2 has pacman, a package manager from Arch Linux. Upgrading can finally be done from within the shell session itself, with only a few basic commands.

# download package descriptions from the remote repositories
pacman -Sy
# upgrade MSYS2 core components and the shell itself
pacman --needed -S bash pacman msys2-runtime
# restart MSYS2 if any package needed updating, then update the rest
pacman -Su

This was 'somewhat more difficult' under MSYS.

Another advantage is the Bash version - currently at 4.3.30 versus 3.1.17 on MSYS. Bash 4 means support for functions like associative arrays and fancier redirections:

# redirect stdout and stderr at the same time
command &> output

# same command in Bash 3 syntax
command > output 2>&1

# pipe stdout and stderr at the same time
command |& someothercommand

# same command in Bash 3 syntax
command 2>&1 \| someothercommand

The third big plus is that it's easier to switch between a 32 bit and 64 bit environment.

With MSYS you had to change the mapping in /etc/fstab of /mingw . That meant that files that you installed in /mingw in one environment weren't available in the other.

MSYS2 keeps the same environments accessible at the same time - /mingw32 for 32 bit, and /mingw64 for 64 bit. You can install packages for one environment while you're in the other.

All OpenSSL binaries that are distributed through this site will be built using MSYS2 from now on.

MSYS2 can be found at https://sourceforge.net/projects/msys2/

More on pacman at https://wiki.archlinux.org/index.php/pacman

OpenSSL: Fatal SSL alert number 47 (Illegal Parameter)

2014-11-25T06:33:00+00:00

As a pentester, I regularly test the configuration of SSL servers. For this purpose I use my customized OpenSSL fork which contains a lot more ciphers than the official version, and wrapper scripts (easier than remembering command line options).

Last month I ran into an issue with servers behind a SSL terminator from a well-known network equipment supplier. As soon as the SSL Client Hello offered 128 or more ciphers to the server and the tls1_2 protocol was specified, the handshake was aborted with the following error message

9304:error:14094417:SSL routines:ssl3_read_bytes:sslv3 alert illegal parameter:s3_pkt.c:1481:SSL alert number 47 9304:error:1409E0E5:SSL routines:ssl3_write_bytes:ssl handshake failure:s3_pkt.c:636:

The supplier hadn't heard of this bug yet - I suspect that not that many browsers or generic SSL clients offer 128 or more ciphers. A bugreport has been filed.

To facilitate the testing of SSL/TLS handshakes I created a script, which can be found at GitHub. Currently 3 handshake bugs are identified.

Of course you can test for this bug using a version of OpenSSL with enough (128 or more) ciphers, and the command

openssl s_client -connect host:port -tls1_2

Black Hat Europe 2014 review

2014-11-11T12:56:00+00:00

Black Hat Europe was held in Amsterdam in October, 2014. The so called Briefings (tech talks) were extremely versatile, ranging from Android pentesting to SCADA hacks. It was difficult to choose which talk to attend - there were around 50 or so, each title vying for your attention ("Hack Your ATM with Friend's Raspberry.Py" [sic], or "Endrun - Secure Digital Communications for Our Modern Dystopia").

I really enjoyed the technical depth of some talks, and it was great to hear someone like Adi Shamir, one of the inventors of the famous RSA algorithm, talk about his current research.

Because of the sheer size I couldn't find one generic takeaway, or see the, or a current bigger picture in information security land. Even all of the vendors' offerings looked similar.

Now that I come to think of it, that actually IS the key takeaway: Genericity.

Creating and verifying digital signatures of files

2014-10-28T06:51:00+00:00

Digital signatures can be used to establish the authenticity and integrity of a (binary) file. These signatures can also be used for non-repudiation purposes, but that's usually not the intention when you're distributing or receiving files. (Note: non-repudiation means impossible to reject; to make sure beyond a doubt that the signer's key has been used to create that signature).

The easiest and most secure way of creating and verifying digital signatures is by using PGP. The following commands assume that you have downloaded and configured GPG, the free and complete implementation of the OpenPGP standard.

Create a digital signature of FILENAME

gpg --armor --detach-sig --output FILENAME.sig FILENAME

`--armor`	make sure that the file is ASCII armored (Radix-64 encoded)
`--detach-sig`	create a separate signature file
`--output`	the name of the signature file

Paranoid options

`--no-version`	don't show which software version has been used to create the signature
`--comment`	don't show which software has been used to create the signature

Verify a digital signature

gpg --verify FILENAME.sig

This command assumes that the original file is FILENAME and resides in the same location as the signature file FILENAME.sig. To verify a signature you also need the signer's public key. If you don't have that key you can add it to your local keyring.

Add a public key to your keyring

gpg --recv-keys KEYID

You also can use --keyserver KEYSERVER to specify a keyserver.

Patched openssl SSLv3 downgrade attack (POODLE) with ChaCha20 and Poly1305 support

2014-10-25T07:35:00+00:00

The OpenSSL team published a security advisory on October 15th 2014,
see the OpenSSL
site for more
information.
In short, SSLv3 using Cipher Block Chaining mode (CBC) has a weakness,
which can be exploited using the POODLE attack having CVE entry
CVE-2014-3566.
The POODLE attack depends on SSLv3 and tries to downgrade a connection
to that specific, really old protocol. This downgrade can be mitigated
by using the signaling cipher suite value (SCSV) TLS_FALLBACK_SCSV,
which is implemented in the/this latest version of openssl. Please be
advised that not only the server, but the client itself also has to
support this relatively new method.

All vulnerabilities in the advisory have been patched in the latest versions of OpenSSL 1.0.2-chacha. Moreover, the new binaries/source are aligned with the latest beta release (3).

SRTP Memory Leak (CVE-2014-3513)
Session Ticket Memory Leak (CVE-2014-3567)
SSL 3.0 Fallback protection
Build option no-ssl3 is incomplete (CVE-2014-3568)

As always, check https://onwebsecurity.com/cryptography/openssl for the latest Windows 32 and 64 bit binaries, and https://github.com/PeterMosmans/openssl for the latest sources.

Patched vulnerabilities in OpenSSL with ChaCha20 and Poly1305 support

2014-08-07T14:49:00+00:00

The OpenSSL team published a security advisory on August 6th 2014, see the OpenSSL site for more information. All vulnerabilities in that advisory have been patched in the latest versions of OpenSSL 1.0.1-chacha and 1.0.2-chacha:

Information leak in pretty printing functions (CVE-2014-3508)
Crash with SRP ciphersuite in Server Hello message (CVE-2014-5139)
Race condition in ssl_parse_serverhello_tlsext (CVE-2014-3509)
Double Free when processing DTLS packets (CVE-2014-3505)
DTLS memory exhaustion (CVE-2014-3506)
DTLS memory leak from zero-length fragments (CVE-2014-3507)
OpenSSL DTLS anonymous EC(DH) denial of service (CVE-2014-3510)
OpenSSL TLS protocol downgrade attack (CVE-2014-3511)
SRP buffer overrun (CVE-2014-3512)

As always, check https://onwebsecurity.com/cryptography/openssl for the latest Windows 32 and 64 bit binaries, and https://github.com/PeterMosmans/openssl for the latest sources.

DevOps 2014 Brisbane and security

2014-07-28T13:10:00+00:00

DevOps is a worldwide phenomenon, which is reflected by the global popularity of its major event, the DevOps Days.

I was fortunate enough to attend the DevOps Days 2014 in Brisbane.

The keynote speaker was Sidney Dekker, a Dutchman who has extensive experience on human factors and safety. He argued that a lot of major incidents don't have any precursor events.

You can have a clean track record with regards to security and still suffer a huge incident. Do I agree ? Not completely, but nonetheless thought provoking.

Personally I think that the inverse will always hold true: There is a higher chance on a major security incidents after a number of several minor security incidents. Cluttered desks mean cluttered minds after all.

Some buzzwords and issues that were (frequently) discussed:

Docker - A lightweight virtualization platform (can it live up to its sky-high expectations ?)
Microservices - Build small, independently deployable services
ChaosMonkey
Terminate random virtual machines to test (and improve) resiliency
Edwards Deming - The godfather of Devops ?

For me the key takeaway was that DevOps doesn't really changes your (level of operational) security. Whether system administrators deploy code built by developers or developers push their own code to an environment - in both instances the implicit level of trust doesn't change. Yes, procedures change, quite profound sometimes.

Still I was left with the overall impression that there are more developers interested in moving towards DevOps than system administrators....

Should you disable RC4 in SSL/TLS ?

2014-07-23T11:22:00+00:00

I'm by no means a crypto expert. Still I'm frequently getting (and answering) questions regarding the use of RC4 in SSL/TLS. Should you disable it? Or keep it enabled?

March 2015 update - A 'new' attack method (Bar Mitsvah Attack) using a previously known RC4 vulnerability was presented, thereby reducing the RC4 security even more.

February 2015 update - RFC 7456 has been published, which effectively prohibits the use of RC4 in TLS.

This document requires that Transport Layer Security (TLS) clients

and servers never negotiate the use of RC4 cipher suites when they

establish connections. This applies to all TLS versions.

See http://tools.ietf.org/html/rfc7465

Here is my reasoning to disable all ciphersuites using RC4:

RC4 is a stream cipher that has been around since 1987. The number and quality of attacks on RC4 (in SSL/TLS) increases. Fact: Attacks on encryption algorithms only get better, they never get worse.
A lot of sites still enable RC4 in their ciphers, to support a wide browser base. Fact: Even Internet Explorer on Windows XP supports DES-CBC3-SHA (an alternative to one of the RC4 ciphers)
RC4 is one of the few ciphers that is resistant to the BEAST attack Fact: BEAST is a client-side attack, announced in 2011. Since then, almost all clients (openssl, NSS, SChannel) have mitigations implemented. BEAST can also be mitigated by using TLS 1.1 and TLS 1.2
A lot of companies can only use Federal Information Processing Standard FIPS PUB 140-2 approved algorithms. Fact: RC4 is not a FIPS-approved algorithm

What do the crypto experts say ?

Bruce Schneier: Move away from RC4 to something like AES (March 2013)
Jacob Appelbaum: RC4 is broken in realtime by the NSA. Stop using it (November 2013)
Microsoft: Disable RC4 (November 2013)

Some form of encryption is always better than no encryption - however you don't want to give visitors of your site a false sense of security. Currently there are better cipher alternatives to RC4.

Disadvantages

RC4 is a fast algorithm; 3DES is slower and more CPU intensive

How to disable it
When using openssl, add the following line in your cipher string:
`` !RC4``

Example of an Apache SSL/TLS configuration which disables RC4 among other ciphers and protocols

`` SSLProtocol All -SSLv2 -SSLv3 SSLHonorCipherOrder On SSLCipherSuite HIGH:!IDEA:!RC4:!MD5:!ADH:!AECDH:!aNULL:!eNULL:@STRENGTH``

Useful links

git on Windows - location of configuration files

2014-07-09T09:10:00+00:00

Git is used as distributed version control system for the majority of projects I work on. On Windows I use the official Git for Windows version, as well as the 'native' mingw/MSYS2 git binary when using the MSYS2 shell.

The location of the system and global gitconfig configuration files varies, depending on which environment (native Windows command, Windows shell or MSYS2 shell) you're using, and depending on which binary (Git for Windows versus native git). There's a logic to it, but it can be hard to figure out...

Git version 2 introduced a much easier method of finding where the git configuration files are stored, the --show-origin flag. This parameter tells you exactly where each of the configuration files can be found.

Retrieve the locations (and name value pairs) of all git configuration files:

git config --list --show-origin

Retrieve the location (and name value pairs) of the system git configuration file:

git config --list --system --show-origin

Retrieve the unique locations of all git configuration files:

git config --list --show-origin | awk '{print $1}' | uniq

Local

Regardless from where you use git on Windows, the repository (local) configuration always resides at the same location, in the root directory of your repository: .git\config

You can check this configuration using

git config --list --local

System

The system configuration has also a fixed path on Windows, relative to the installation directory: etc\gitconfig

For msysGit: %ProgramFiles(x86)%\Git\etc\gitconfig
For Git for Windows (64 bit): %ProgramFiles%\Git\mingw64\etc\gitconfig
For MSYS2-native git: [MSYS2-install-path]\etc\gitconfig

This configuration file can be read using

git config --list --system

Global

This is the tricky one. Basically, it depends on what the binary or environment considers the HOME directory.

When using the Windows command shell, batch scripts or Windows programs, HOME is %USERPROFILE% . The global config file will be read from %USERPROFILE%\.gitconfig

However, when you're using a (Bash) shell under MSYS2 or Cygwin, HOME under that shell is %HOME% . The global config file will be read from $HOME/.gitconfig

The global configuration file can be found on yet another location, for Windows programs that use their own HOME environment. Take Emacs (which uses magit) for example: When Emacs is started from a Windows command shell, it sets the HOME variable to %APPDATA% , which means that .gitconfig will reside in %APPDATA%\.gitconfig

This configuration file can be read using

git config --list --global

So to recap, the global git configuration file can be found under what that application considers to be the HOME directory, in .gitconfig .

Workflow setup: Linking the global configuration

In my workflow I symlink the various versions of the global configuration. The system configration stays independent. The file at %USERPROFILE%\.gitconfig is considered the master global file where you make all your changes.

Run this command from within a Windows command shell to create a symbolic link for the system and global file. Be careful, as this command deletes the current gitconfig file in your MSYS2 home directory, as well as the one in %APPDATA% . Moreover mklink doesn't expand variables, so you'll have to type in the expanded path yourself.

mklink /h %APPDATA%\.gitconfig %USERPROFILE%\.gitconfig
mklink /h [MSYS2-install-path]\home\[username]\.gitconfig
%USERPROFILE%\.gitconfig

You also might be interested in other git-related articles articles, for instance on how to diff binary files with git.

Changelog

2018-04-19: Added better explanation about HOME directory, links
2017-10-19: Formatting
2017-09-17: Formatting
2016-12-03: Added new --show-location option, changed git binary, edits.

OpenSSL 1.0.2-chacha

2014-06-26T14:04:00+00:00

Note: see `http://www.onwebsecurity.com/cryptography/openssl <https://www.onwebsecurity.com/cryptography/openssl>`__ for the latest binary. The version below is obsoleted by newer builds

Windows 64-bit binary build from a 26-06-2014 snapshot of https://github.com/PeterMosmans/openssl/tree/1.0.2-chacha. This is the official 1.0.2 branch (OpenSSL_1_0_2_stable), merged with support for the ChaCha20 and Poly1305 ciphers. Some minor build patches for Windows compatibility were applied. See the git repo for the full source.

Build commands:

mingw64 shared experimental-jpake enable-md2 enable-rc5 enable-rfc3779 enable-ec_nistp_64_gcc_128 enable-static-engine --openssldir=c\:/programs/openssl -DOPENSSL_NO_HEARTBEATS
make depend
make util/libeay.num
make util/ssleay.num
make
make report

Compiler used:

gcc version 4.9.0 (x86\_64-posix-seh-rev1, Built by MinGW-W64 project)

All tests passed

OpenSSL 1.0.1-chacha

2014-06-18T05:54:00+00:00

A Windows 64-bit binary build from the 1.0.1 branch of OpenSSL
(OpenSSL_1_0_1-stable), including (assembly code for) ChaCha20,
Poly1305, J-PAKE, NIST P-224, NIST P-256 and the relatively unsafe
ciphers MD2 and RC5 and broken protocol SSLv2. All available engines
are provided as separate DLLs.
If you're using this in a production environment, don't forget to
explicitly enable only ciphers that are considered safe.
The code for this build can be found at
https://github.com/PeterMosmans/openssl/tree/1.0.1-chacha

Example openssl.cnf cipher string:

`` HIGH:!SSLv2:!IDEA:!RC4:!MD5:!ADH:!aNULL:!eNULL``

Build commands:

Configure mingw64 shared experimental-jpake enable-md2 enable-rc5 enable-rfc3779 enable-ec_nistp_64_gcc_128 enable-static-engine --openssldir=c\:/programs/openssl -DOPENSSL_NO_HEARTBEATS
make depend
make util/libeay.num
make util/ssleay.num
make
make report (all tests passed)

http://www.onwebsecurity.com/files/1.0.1-chacha-latest.zip

md5sum: d890de1ab4eba13c7d39139c5726144f

Compiler used:

gcc version 4.9.0 (x86_64-posix-seh-rev1, Built by MinGW-W64 project)

OpenSSL 1.0.2 (10-06-2014)

2014-06-10T07:37:00+00:00

Note: see http://www.onwebsecurity.com/cryptography/openssl for the latest binary. The version below is obsoleted by newer builds

A Windows 64-bit binary build from a 10-06-2014 snapshot of the official 1.0.2 branch (OpenSSL_1_0_2_stable). This means that 'the latest OpenSSL vulnerabilities' that were disclosed on June 5th 2014 are fixed - see https://www.openssl.org/news/secadv_20140605.txt for more information.

I applied some minor patches for Windows compatibility and changed the version string.

Build commands:

Configure mingw64 shared experimental-jpake enable-md2 enable-rc5 \
enable-rfc3779 enable-ssl-trace enable-ec_nistp_64_gcc_128 \
enable-static-engine --openssldir=c:/tools -DOPENSSL_NO_HEARTBEATS \
-mtune=native
make depend
make util/libeay.num
make util/ssleay.num
make
make report

Compiler used:

gcc version 4.9.0 (x86_64-posix-seh-rev1, Built by MinGW-W64 project)

OpenSSL on 64 bit Windows with ChaCha and Poly1305 support

2014-05-19T09:09:00+00:00

Note: see https://www.onwebsecurity.com/cryptography/openssl for the latest binary. The version below is obsoleted by newer builds

The main development branch of OpenSSL doesn't have support yet for the (relatively new) ChaCha 20 and Poly1305 ciphers. These can be found however on the 1.0.2-aead branch.

By slightly modifying some makefiles the source can be compiled for 64-bit Windows using mingw64 and msys.

Please find a binary build from a 27-05-2014 snapshot of the source code (1.0.2-aead branch) with assembly code enabled (imported from the 1.0.2 stable branch), and a lot of insecure, new and experimental ciphers enabled. I added the GOST engine gosteay32.dll as well.

The source code for this build can be found at https://github.com/PeterMosmans/openssl

Build commands:

Configure mingw64 shared experimental-jpake enable-md2 enable-rc5 enable-rfc3779 enable-ssl-trace enable-ec_nistp_64_gcc_128 enable-static-engine --openssldir=c:/tools
make depend
make util/libeay.num
make util/ssleay.num
make
make test

Enabled ciphers:

ECDHE-RSA-AES256-GCM-SHA384
ECDHE-ECDSA-AES256-GCM-SHA384
ECDHE-RSA-AES256-SHA384
ECDHE-ECDSA-AES256-SHA384
ECDHE-RSA-AES256-SHA
ECDHE-ECDSA-AES256-SHA
SRP-DSS-AES-256-CBC-SHA
SRP-RSA-AES-256-CBC-SHA
DH-DSS-AES256-GCM-SHA384
DHE-DSS-AES256-GCM-SHA384
DH-RSA-AES256-GCM-SHA384
DHE-RSA-AES256-GCM-SHA384
DHE-RSA-AES256-SHA256
DHE-DSS-AES256-SHA256
DH-RSA-AES256-SHA256
DH-DSS-AES256-SHA256
DHE-RSA-AES256-SHA
DHE-DSS-AES256-SHA
DH-RSA-AES256-SHA
DH-DSS-AES256-SHA
ECDHE-ECDSA-CHACHA20-POLY1305
ECDHE-RSA-CHACHA20-POLY1305
DHE-RSA-CHACHA20-POLY1305
DHE-RSA-CAMELLIA256-SHA
DHE-DSS-CAMELLIA256-SHA
DH-RSA-CAMELLIA256-SHA
DH-DSS-CAMELLIA256-SHA
GOST2001-GOST89-GOST89
GOST94-GOST89-GOST89
AECDH-AES256-SHA
SRP-AES-256-CBC-SHA
ADH-AES256-GCM-SHA384
ADH-AES256-SHA256
ADH-AES256-SHA
ADH-CAMELLIA256-SHA
ECDH-RSA-AES256-GCM-SHA384
ECDH-ECDSA-AES256-GCM-SHA384
ECDH-RSA-AES256-SHA384
ECDH-ECDSA-AES256-SHA384
ECDH-RSA-AES256-SHA
ECDH-ECDSA-AES256-SHA
AES256-GCM-SHA384
AES256-SHA256
AES256-SHA
CAMELLIA256-SHA
PSK-AES256-CBC-SHA
ECDHE-RSA-DES-CBC3-SHA
ECDHE-ECDSA-DES-CBC3-SHA
SRP-DSS-3DES-EDE-CBC-SHA
SRP-RSA-3DES-EDE-CBC-SHA
EDH-RSA-DES-CBC3-SHA
EDH-DSS-DES-CBC3-SHA
DH-RSA-DES-CBC3-SHA
DH-DSS-DES-CBC3-SHA
AECDH-DES-CBC3-SHA
SRP-3DES-EDE-CBC-SHA
ADH-DES-CBC3-SHA
ECDH-RSA-DES-CBC3-SHA
ECDH-ECDSA-DES-CBC3-SHA
DES-CBC3-SHA
DES-CBC3-MD5
PSK-3DES-EDE-CBC-SHA
ECDHE-RSA-AES128-GCM-SHA256
ECDHE-ECDSA-AES128-GCM-SHA256
ECDHE-RSA-AES128-SHA256
ECDHE-ECDSA-AES128-SHA256
ECDHE-RSA-AES128-SHA
ECDHE-ECDSA-AES128-SHA
SRP-DSS-AES-128-CBC-SHA
SRP-RSA-AES-128-CBC-SHA
DH-DSS-AES128-GCM-SHA256
DHE-DSS-AES128-GCM-SHA256
DH-RSA-AES128-GCM-SHA256
DHE-RSA-AES128-GCM-SHA256
DHE-RSA-AES128-SHA256
DHE-DSS-AES128-SHA256
DH-RSA-AES128-SHA256
DH-DSS-AES128-SHA256
DHE-RSA-AES128-SHA
DHE-DSS-AES128-SHA
DH-RSA-AES128-SHA
DH-DSS-AES128-SHA
DHE-RSA-SEED-SHA
DHE-DSS-SEED-SHA
DH-RSA-SEED-SHA
DH-DSS-SEED-SHA
DHE-RSA-CAMELLIA128-SHA
DHE-DSS-CAMELLIA128-SHA
DH-RSA-CAMELLIA128-SHA
DH-DSS-CAMELLIA128-SHA
AECDH-AES128-SHA
SRP-AES-128-CBC-SHA
ADH-AES128-GCM-SHA256
ADH-AES128-SHA256
ADH-AES128-SHA
ADH-SEED-SHA
ADH-CAMELLIA128-SHA
ECDH-RSA-AES128-GCM-SHA256
ECDH-ECDSA-AES128-GCM-SHA256
ECDH-RSA-AES128-SHA256
ECDH-ECDSA-AES128-SHA256
ECDH-RSA-AES128-SHA
ECDH-ECDSA-AES128-SHA
AES128-GCM-SHA256
AES128-SHA256
AES128-SHA
SEED-SHA
CAMELLIA128-SHA
IDEA-CBC-SHA
IDEA-CBC-MD5
RC2-CBC-MD5
PSK-AES128-CBC-SHA
ECDHE-RSA-RC4-SHA
ECDHE-ECDSA-RC4-SHA
AECDH-RC4-SHA
ADH-RC4-MD5
ECDH-RSA-RC4-SHA
ECDH-ECDSA-RC4-SHA
RC4-SHA
RC4-MD5
PSK-RC4-SHA
EDH-RSA-DES-CBC-SHA
EDH-DSS-DES-CBC-SHA
DH-RSA-DES-CBC-SHA
DH-DSS-DES-CBC-SHA
ADH-DES-CBC-SHA
DES-CBC-SHA
DES-CBC-MD5
EXP-EDH-RSA-DES-CBC-SHA
EXP-EDH-DSS-DES-CBC-SHA
EXP-DH-RSA-DES-CBC-SHA
EXP-DH-DSS-DES-CBC-SHA
EXP-ADH-DES-CBC-SHA
EXP-DES-CBC-SHA
EXP-RC2-CBC-MD5
EXP-ADH-RC4-MD5
EXP-RC4-MD5

Enjoy!

test_bn fails for OpenSSL on Windows

2014-05-06T02:16:00+00:00

Compiling OpenSSL on Windows using MSYS and mingw64 is pretty straightforward. However, one of the tests (test_bn) to verify OpenSSL fails: The temporary file that test_bncreates contains Windows newline characters (\r\n) instead of the Unix type newline charater (\n).

The original regular expression checks for a zero (0) at the beginning of a line, and a newline character (\n).

(!/^0$$/)

A change to the regular expression that test_bn uses fixes this problem, and can be used on Unix as well as Windows environments. This makes the Makefile more cross-platform friendly. The modified regular expression checks for a zero (0) at the beginning of a line, an optional Windows newline character (\r) and a newline character (\n).

(!/^0\r?$$/)

File: openssl-test-Makefile.patch

md5sum: 1032dff7f957c4d1cdfa96af305c152b

Here's the patchfile (can be applied in the source directory using patch -Np1)

--- openssl-1.0.1g/test/Makefile 2014-04-07 16:55:44 +0000 +++ patched/test/Makefile 2014-05-06 00:07:20 +0000 @@ -227,7 +227,7 @@ @../util/shlib_wrap.sh ./$(BNTEST) >tmp.bntest @echo quit >>tmp.bntest @echo "running bc" - @) {if (/^test (.*)/) {print STDERR "\nverify $$1";} elsif (!/^0$$/) {die "\nFailed! bc: $$_";} else {print STDERR "."; $$i++;}} print STDERR "\n$$i tests passed\n"' + @) {if (/^test (.*)/) {print STDERR "\nverify $$1";} elsif (!/^0\r?$$/) {die "\nFailed! bc: $$_";} else {print STDERR "."; $$i++;}} print STDERR "\n$$i tests passed\n"' @echo 'test a^b%c implementations' ../util/shlib_wrap.sh ./$(EXPTEST)

As of May 28, 2014 this patch has been applied to the main branch of openssl: see http://git.openssl.org/gitweb/?p=openssl.git;a=commit;h=028bac0670c167f154438742eb4d0fbed73df209

If you're interested in cherry-picking:

git cherry-pick 028bac0670c167f154438742eb4d0fbed73df209

analyze_hosts

2014-01-23T03:57:00+00:00

If you're like me, you don't want to spend your precious memory on remembering awkward command line parameters. However, lots of tools require exactly that: awkward command line parameters.

To simplify scanning of hosts for network vulnerabilities I wrote a simple wrapper script around several open source security tools. The script lets you analyze one or several hosts for common misconfiguration vulnerabilities and weaknesses.

My main objective in writing the script was to make it as easy as possible to perform generic security tests, without any heavy prerequisites, make the script as informative as possible, and make use of open source tools.

Note that the latest version is the Python version - please use that one.

How to install

Clone the git archive using the command

git clone https://github.com/PeterMosmans/security-scripts.git

Needed

Linux, and nmap

Optional

```
curl
```
for fingerprinting and to test for TRACE
```
dig
```
to test for recursive DNS servers
```
git
```
to update the script
```
nikto
```
for webscanning
```
testssl.sh
```
to check the SSL configuration

Usage

Oh irony - the command line parameters for the tool:

usage: analyze_hosts.sh [OPTION]... [HOST]

Scanning options:
 -a, --all perform all basic scans
 --max perform all advanced scans (more thorough)
 -b, --basic perform basic scans (fingerprint, ssl, trace)
 --filter=FILTER only proceed with scan of HOST if WHOIS
 results of HOST matches regexp FILTER
 --dns test for recursive query
 -f perform web fingerprinting (all webports)
 --fingerprint perform all web fingerprinting methods
 -h, --header show webserver headers (all webports)
 -n, --nikto nikto webscan (all webports)
 -p nmap portscan (top 1000 ports)
 --ports nmap portscan (all ports)
 -s check SSL configuration
 --ssl perform all SSL configuration checks
 --timeout=SECONDS change timeout for sslscan (default=30)
 --ssh perform SSH configuration checks
 -t check webserver for HTTP TRACE method
 --trace perform all HTTP TRACE method checks
 -w, --whois perform WHOIS lookup for the IP address
 -W confirm WHOIS results before continuing scan

Port selection (comma separated list):
 --webports=PORTS use PORTS for web scans (default 80,443)
 --sslports=PORTS use PORTS for ssl scans (default 443,993,995)

Logging and input file:
 -d, --directory=DIR location of temporary files (default /tmp)
 -i, --inputfile=FILE use a file containing hostnames
 -l, --log log each scan in a separate logfile
 --nocolor don't use fancy colors in screen output
 -o, --output=FILE concatenate all results into FILE
 -q, --quiet quiet
 -v, --verbose show server responses

-u update this script (if it's a cloned repository)
 --update force update (overwrite all local modifications)
 --version print version information and exit

 BLUE: status messages
 GREEN: secure settings
 RED: possible vulnerabilities

[HOST] can be a single (IP) address, an IP range, eg. 127.0.0.1-255
 or multiple comma-separated addresses, eg 127.0.0.1,127.0.0.2

example: /usr/local/sbin/analyze_hosts -a --filter Amazon www.google.com

securing AMFPHP

2013-05-30T09:13:00+00:00

I regulary run into Flash applications when I perform a web application penetration test. One of the most widely used server frameworks for communicating with a Flash object is AMFPHP.

Unfortunately the default installation of AMFPHP is insecure. A system administrator or developer actively has to secure the installation, which is often forgotten.

There are some tips lying around the Internet how to secure an AMFPHP installation. The summary:

In the root of your AMFPHP deployment,

delete the DiscoveryService.php file
Delete the browser folder and its contents
Edit gateway.php and set the PRODUCTION_SERVER property to true

Of course it's at least as important to write secure code, harden your server and implement proper patch and maintenance procedures.

unsafe HTTP methods

2013-03-20T09:01:00+00:00

Vulnerability name: Unsafe HTTP methods

Aliases

Web server HTTP Trace/Track method support
Cross-site tracing vulnerability
Dangerous HTTP methods

Scope

Although this is a server configuration issue, the client is at risk here

Remediation

Disable TRACE and/or TRACK and/or DEBUG methods

Verification

Using curl , one can employ one of the methods by hand:

curl -sIX TRACE $TARGET | awk 'NR==1 {print $2}'

Vulnerable when: the result is 200

One should expect (not vulnerable) 405 (Method Not Allowed) or 501 (Not Implemented) results.

This executes the TRACE method against $TARGET , and prints out the HTTP status code using awk . The -I parameter fetches the head only, -s stands for silent mode, and -X specifies the method.

The easiest way to test whether a server is vulnerable is by using the script analyze_hosts.py [1].

This script uses curl as well as nmap to perform multiple tests.

analyze_hosts.py --trace http://www.target.com

Note

When an OPTIONS method is issued, the webserver should return the supported methods. Some web servers have a habit of replying with methods that are in fact not supported - which does not combine nicely with inferior security scanners (and pentesters, I might add) that relying on the OPTIONS command. Always verify the method itself, and do not rely on the OPTIONS method (and the results of a security scanner).

Explanation

There are a number of official (standards compliant) HTTP methods:

OPTIONS, HEAD, GET, POST, PUT, DELETE, TRACE, CONNECT

An ordinary web server supports the HEAD, GET and POST methods to retrieve static and dynamic content (enabling WebDAV on a web server will add support for the PUT and DELETE methods).

TRACE and TRACK are methods which can be used for debugging purposes. It repeats the content of a request, and an attacker could steal credentials by using a client-side attack.

These HTTP methods should not be supported on public web servers, as they increase the attack surface.

Solution/remediation

Apache

Add the following to your Apache httpd.conf file:

TraceEnable Off

IIS7 and higher

appcmd.exe set config /section:requestfiltering /+verbs.[verb='TRACE',allowed='false']

IIS6

REGEDIT4 [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\W3SVC\Parameters] "EnableTraceMethod"=dword:00000000

IIS5 and lower

Install URLscan [2]

Note

Microsoft's IIS 5 supports the non-RFC compliant methode TRACK, which is basically the same as the TRACE method. This should be disabled as well (by installing URLScan)

[1]	https://github.com/PeterMosmans/security-scripts/

[2]	https://technet.microsoft.com/en-us/security/cc242650.aspx