Toon Verwerft - VeeWee

Taming the XML beast!

2021-02-07T00:00:00+01:00

Imagine a world ... in which you didn't have to deal with all PHP's strange XML quirks. The people following me on Twitter might have noticed that I don't like the XML tools that PHP currently provides. My main frustrations with the built-in tools can be divided in 2 big items: The lack of error handling and the bloated APIs.

Lack of error handling

Most of the time, you’ll be working with XML files of which you know they contain valid XML syntax. In some situations however, you might receive invalid XML or XML that is not valid according to an XSD schema. At this point, lib-xml gives you following:

<?php

$doc = new DOMDocument();
$result = $doc->loadXML('<invalid>');

var_dump($result);

Resulting in:

Warning: DOMDocument::loadXML():
  Premature end of data in tag invalid line 1 in Entity,
  line: 1 in /in/0I8Or on line 4
  
bool(false)

Yes, that is right… It returns false and it triggers … a PHP warning … This means that code execution is not being stopped and a shitload of other warnings will be triggered if you do not check the false case of the loadXML function.

If you want to cover up the warning and display the actual XML issue, you’ll have to write code that looks like this.

<?php

$doc = new DOMDocument();

$previousErrorReporting = libxml_use_internal_errors(true);
libxml_clear_errors();

$result = $doc->loadXML('<invalid>');

$errors = libxml_get_errors();
libxml_clear_errors();
libxml_use_internal_errors($previousErrorReporting);

if (!$result) {
    // Do something with the errors and break execution
}

Do note that most XML functions work in a similar way: they return false and trigger a warning. So if you want to reliably work with XML, you need to wrap them all with your own error handling.

The code above can be found in so many packages out there. It’s like everybody is OK with this approach. At least, I am not!

Bloated API

The XML extensions provided by PHP contain functions for everything you want to do with XML. The DOMDocument class contains about 50 functions and 20 properties (estimated values) which all have their own purpose. This is only the main entry point of your XML, besides that you can find DOMElements, DOMNodes, DOMAttributes, … which all have a similar amount of functions.

Don’t get me wrong, I like this structure: It is created to do whatever you want to do with XML. You want to load XML, fine! Or do you rather manipulate something, fine as well! Do whatever you please! The downside of this all, is that you need to know a lot about the low level API!

To give you an example for day-to-day loading of XML:

<?php

$document = new DOMDocument();
$document->encoding = 'UTF-8';
$document->preserveWhiteSpace = false;
$document->formatOutput = true;

// wrap error handling here ...
$result = $document->loadXML('<root />');

if (!$result) {
    // Custom error handling
}

With error handling, this is about 15 lines of code for just loading an XML file! Not to mention that I needed to look up every property in there …

The same goes for basically any manipulation you want to do. Did you ever try to create an XML document with DOMDocument? It’s aweful - because it is all this ultra flexible.

What if we were able to split all these functions and different XML types per use case? That way we could create a smaller Document class that you can use to perform specific types of actions:

Loading with configurable settings.
Validating based on whatever schema you please.
Building XML
Manipulating XML
Locating items inside an XML
…

Other components

I don’t want to convert this blog post into a big rant, so I am not going to focus on all other XML components at this moment. So instead of complaining, let’s deal with these problems and let’s create something that covers these problems instead.

XML without worries

In order to tame the XML beast, I decided to create yet another package. It’s not that there isn’t any XML library out there, but most focus on 1 specific problem or are bloated wrappers around DOMDocument. With this package I want to improve ALL XML extensions that are available in PHP.

The goal of the package is to provide all tools for dealing with XML in PHP without worries. You will find a type-safe, declarative API that deals with errors for you!

Currently the package provides DOM, XSD, XSLT, Error-Handling and Reader components. In the future I will also focus on additional components like a memory-safe Writer and focus a bit on “simple”Xml. There is a lot of stuff to discover, so let me give you a couple of highlights which I personally really like!

Locating data inside an XML document

Loading the XML document

You can load an XML document with configurable functions to determine how it will load. These configurable functions can contain whatever you want:

Specify encoding
Specify output
Validate internal XSD schema’s whilst loading
You can create custom configurators that e.g. replace import statements with their eventual XML (flattening)
…

<?php

use VeeWee\XML\DOM\Configurator;
use VeeWee\XML\DOM\Document;
use VeeWee\XML\DOM\Loader;
use VeeWee\XML\DOM\Validator;
use VeeWee\Xml\ErrorHandling\Issue\Level;

$doc = Document::configure(
    Configurator\utf8(),
    Configurator\trim_spaces(),
    Configurator\Loader(
      Loader\xml_file_loader('data.xml')
    ),
    Configurator\validator(
        Validator\internal_xsd_validator(),
        Level::warning()
    )
);

XPath improvements

Next up is the actual querying. You start of by configuring how you want to create your XPath object. In this case, you can configure it with lookup namespaces:

<?php

use Psl\Type;
use VeeWee\XML\DOM\Xpath\Configurator;

$xpath = $doc->xpath(
    Configurator\namespaces([
        'acme' => 'http://acme.com',
    ])
);

$currentNode = $xpath->querySingle('//acme:products');
$count = $xpath->evaluate('count(.//item)', Type\int(), $currentNode);

After creating the XPath object, you can see a new function querySingle. I found myself frequently looking for one element and wanted to make that possible. Another addition is typed evaluation. By specifying a type, you can make sure that the $count variable actually contains an integer or throws an exception instead.

If you write an error in your XPath query, you will receive a meaningfull error message explaining you what is wrong with your query!

Memory-safe Reading

If you have a very big XML file, you want to lookup nodes in a memory-safe way. The improved version of the XMLReader looks like this:

<?php

use VeeWee\Xml\Dom\Document;
use VeeWee\Xml\Reader\Configurator;
use VeeWee\Xml\Reader\Reader;
use VeeWee\Xml\Reader\Matcher;

$reader = Reader::fromXmlFile(
    'large-data.xml',
    Configurator\xsd_schema('schema.xsd')
);

/** @var \Generator<string> $provider */
$provider = $reader->provide(
    Matcher\all(
        Matcher\node_name('item'),
        Matcher\node_attribute('locale', 'nl-BE')
    )
);

foreach ($provider as $nlItem) {
    $dom = Document::fromXmlString($nlItem);
    // Do something with it
}

You can trust this reader to handle the boring node traversal for you. By specifying matcher functions, you are in control of what XML strings are yielded. From that point on, you can e.g. use a regular Document to deal with the smaller XML chunks.

The reader is able to detect errors or invalid schema implementations whilst iterating over the dataset lazily. You will receive a meaningful exception at all time!

Writing

Reading is a bug chunk of your day-to-day XML usage, but in some cases you mind want to create or manipulate an XML. In the XML package, you will find some tools that make building XML quite intuitive. You basically specify what the XML should look like:

<?php

use VeeWee\Xml\Dom\Document;
use function VeeWee\Xml\Dom\Builder\attribute;
use function VeeWee\Xml\Dom\Builder\children;
use function VeeWee\Xml\Dom\Builder\element;
use function VeeWee\Xml\Dom\Builder\namespaced_element;
use function VeeWee\Xml\Dom\Builder\value;
use function VeeWee\Xml\Dom\Manipulator\append;

$doc = Document::empty();
$doc->manipulate(
    append(...$doc->build(
        element('root', children(
            element('foo',
                attribute('bar', 'baz'),
                value('hello')
            ),
            namespaced_element('http://namespace', 'foo',
                attribute('bar', 'baz'),
                children(
                    element('hello', value('world'))
                )
            )
        ))
    ))
);

$xml = $doc->toXmlString();

In closing

The snippets above are only some highlights of the package. There is a lot more in there for you to discover!

At this moment, the package already contains a lot of components that you can use on a day-to-day basis. Since XML is very broad, there are still missing functions. It might contain some rough corners and will surely grow in time.

It has to start somewhere, it has to start sometime What better place than here, what better time than now?!

Introducing veewee/xml!

Feel free to drop me any feedback.

Hope you love it!

Run composer tasks in parallel

2020-12-02T00:00:00+01:00

Composer has a built-in way of running multiple composer scripts by declaring a script alias. This alias runs all specified scripts one by one and fails if one of the scripts fails. To save you some time, I've written the veewee/composer-run-parallel plugin that can run multiple composer scripts in parallel and won't stop before all tasks have been executed:

Why I created this package?

As the creator of GrumPHP, I am used that all tasks run in parallel. Some of the open-source project that I work on don't have a GrumPHP configuration (yet). Most of those projects run Quality Assurance tools crafted out of composer scripts. Even on those projects, I want to make sure to commit good code before pushing it to a Pull Request. Since I got bored of waiting for the composer scripts to run one by one, I decided to dive into the composer internals in order to make composer scripts run in parallel as well.

Shut up and tell me how to use it!

Configuring parallel scripts is like configuring any regular composer script. This package provides a `@parallel` script that takes other composer script aliases as arguments:

{
  "scripts": {
  
      "ci": "@parallel cs tests",
  
      "cs": "php-cs-fixer fix --dry-run",
      "tests": "phpunit"
  }
}

The code above adds a `ci` composer script to your project, that runs the coding styles and unit tests in parallel. You can run the script like this:

composer run ci

Most of the times, you don't want to introduce dependencies to projects you don't own. Therefore, this plugin also provides a `composer parallel` command. You can install this package globally, meaning you can unleash the power of parallel task execution even on packages that don't have this plugin installed:

composer parallel cs tests

In closing

No more waiting! This veewee/composer-run-parallel plugin can be quickly added to any project and is fun to use. It contains a few more options. Head over to the readme on GitHub to learn more.

Whilst writing the plugin, I was pleasantly surprised by what is available in the composer codebase by default. Writing the package was a matter of linking the various existing components together. Big-ups to the composer team!

This plugin comes with one big limitation: Since it is only setting composer as a dependency, it is limited to the `Loop` and `ProcessExecutor` that is provided by composer. To display the result of a script, both the stderr and stdout are printed out to the end user. This could result in non-sequential output if the script you are executing mixes stdout and stderr messages.

Let GrumPHP fix your code!

2020-06-07T00:00:00+02:00

One of the most requested features is the ability to automatically fix broken code. Since previously we could not do this in a controlled an safe way, we decided that this won't be a GrumPHP feature. In the new GrumPHP version (0.19.0), we rewrote the task runner system. Now it is possible to do these kind of code manipulations. Let's see how it works...

How do I get this madness to work?

No worries: Fixing is enabled by default! Currently we added fixers for `phpcs` and `PHP-CS-Fixer`. This list might be extended in the future and you can of course also provide fixers for your own custom tasks!

How does GrumPHP knows what to fix?

GrumPHP will first run the configured tasks as it always did. When a task fails, we already provided the commandline command to fix the code. This resulted in manual copy/pasting the code, which is a stupid repeatable task. With this new feature, the task can now determine if it can autofix the code by e.g. running the console command it displayed. In a last phase before finishing the task runner, the autofixer process is initialized. It will look for fixable tasks and after asking for your consent, it will run the fix command that is provided from the task. GrumPHP will not add any new files to GIT or won't automatically commit these changes.

You are in full control!

In previous versions, we only displayed the command line tool that you can use to fix your code. We never wanted GrumPHP to change code during a pre-commit or inside a CI cycle. Therefore, we decided to fix the code but don't stage the changes to git. This way, you can always review the changes that were made before recommitting the code.

Don't touch my code!

I can imagine that you don't want GrumPHP to touch your code. Luckily, the fixer is configurable: You can choose to completely disable it or you can choose to change the default consent answer.

# grumphp.yaml

grumphp:
    fixer:
        enabled: true
        fix_by_default: false

Note: The `fix_by_defaut` parameter will also be used in situations where CLI input is not supported. Depending on your CLI, this could be e.g. during pre-commit.

Enjoy and let me know what you think of it!

Running GrumPHP tasks in parallel

2020-06-06T00:00:00+02:00

Last week, we launched a new version of GrumPHP (0.19.0). One of the main new features inside this release is the ability to run your tasks in parallel. This feature will save you a lot of waiting, both during pre-commit and on the CI environments!

How do I get this madness to work?

Actually, you don't have to do anything! Parallel task execution is enabled by default. All tasks will run in parallel, unless you specify a different priority on a task. This way, you can specify parallel groups inside your configuration. Imagine following setup:

# grumphp.yaml

grumphp:
  tasks:
    phpcs:
      standard: PSR2
    phpunit: ~
    clover_coverage:
      clover_file: 'test/clover.xml'
      metadata:
        priority: -10

This configuration will run both `phpunit` and `phpcs` in parallel since they are both in priority group 0. When both tasks are ready, the `clover_coverage` task is started

Enjoy and let me know what you think of it!

Optimizing PHP performance by using fully-qualified function calls

2016-12-21T00:00:00+01:00

Today, a little conversation on Twitter escalated rather quickly. Apparently PHP runs function calls differently depending on namespaced or non namespaced context. When calling functions in a namespaced context, additional actions are triggered in PHP which result in slower execution. In this article, I'll explain what happens and how you can speed up your application.

Wondering how much #PHP I could break by removing the local namespace lookup semantics for namespaced vs. core functions...
— Reviewed, BLYATIFUL! (@Ocramius) December 21, 2016

The conversation started with the tweet above. To understand the difference between global and namespaced function calls better, I'll explain what is going on under the hood.

Calling functions in the global namespace

Function calls in the global namespace look like this:

<?php
// global.php

function foo() {
    echo 'bar';
}

call_user_func('foo');

After parsing this script, the opcodes look like this:

$ php -d vld.active=1 -d vld.execute=0 global.php

...
line     #* E I O op                           fetch          ext  return  operands
-------------------------------------------------------------------------------------
   4     0  E >   EXT_STMT
         1        NOP
   8     2        EXT_STMT
         3        EXT_FCALL_BEGIN
         4        SEND_VAL                                                 'foo'
         5        DO_FCALL                                      1          'call_user_func'
         6        EXT_FCALL_END
         7      > RETURN                                                   1

...

As you can see, this is a simple EXT_FCALL sequence of opcodes.

Calling functions in a namespace

Function calls in a namespace look like this:

<?php
// namespaced.php

namespace baz;

function foo() {
    echo 'bar';
}

call_user_func('foo');

After parsing this script, the opcodes look like this:

$ php -d vld.active=1 -d vld.execute=0 global.php

...
line     #* E I O op                           fetch          ext  return  operands
-------------------------------------------------------------------------------------
   4     0  E >   NOP
   6     1        EXT_STMT
         2        NOP
  10     3        EXT_STMT
         4        INIT_NS_FCALL_BY_NAME
         5        EXT_FCALL_BEGIN
         6        SEND_VAL                                                 'foo'
         7        DO_FCALL_BY_NAME                              1
         8        EXT_FCALL_END
         9      > RETURN                                                   1
...

The opcodes look pretty much the same as in the global namespace. However, there is one additional opcode added: INIT_NS_FCALL_BY_NAME. When PHP runs over this opcode, it will check if the function call_user_func() is found in the namespace. If the function exists in the namespace, PHP will run this one. When the function does not exist in current namespace, PHP will check if it exists in the global namespace and execute that one.

This handy “feature” is frequently (ab)used during testing. A good example for this (ab)use is overwriting the functions to read from or write to the filesystem. In the source files, you can for example use the fopen() function. During the tests, you can mock this function by placing it in the namespace of the class that you are testing. An example of this can be found in the local adapter of flysystem.

Benchmarks

One of the next tweets stated that a performance gain of 4.5% was made by using fully qualified function calls. Of course, this is just a non-proven number and depends on the project you are working on. To make sure that I am not writing nonsense, I made a little benchmark in PHP 7.1:

$ php -v

PHP 7.1.0 (cli) (built: Dec  2 2016 11:29:13) ( NTS )
Copyright (c) 1997-2016 The PHP Group
Zend Engine v3.1.0-dev, Copyright (c) 1998-2016 Zend Technologies

Next, I wrote code that can be run with the phpbench tool. There are 4 cases I covered:

Run a global function in a fully qualified way.
Run a global function in a non-fully qualified way.
Run an overridden function that exists globally and in the namespace.
Run a namespaced function.

I’ve chosen a rather big amount of revs and iterations to make sure the results are accurate. The code in the benchmark looks like this:

<?php

namespace {
    function a(){}
    function b() {}
}

namespace foo {

    function b() {}
    function c() {}

    /**
     * @Revs(10000)
     * @Iterations(100)
     */
    class MixedBench
    {
        public function benchFqGlobalFunction()
        {
            \a();
        }

        public function benchGlobalFunction()
        {
            a();
        }

        public function benchOverriddenFunction()
        {
            b();
        }

        public function benchNamespacedFunction()
        {
            c();
        }
    }
}

This is an overview of the results:

$ phpbench run bench

\foo\MixedBench

    benchFqGlobalFunction         I99 P0 	[μ Mo]/r: 0.145 0.141 (μs) 	[μSD μRSD]/r: 0.018μs 12.59%
    benchGlobalFunction           I99 P0 	[μ Mo]/r: 0.148 0.145 (μs) 	[μSD μRSD]/r: 0.021μs 14.38%
    benchOverriddenFunction       I99 P0 	[μ Mo]/r: 0.157 0.157 (μs) 	[μSD μRSD]/r: 0.022μs 13.82%
    benchNamespacedFunction       I99 P0 	[μ Mo]/r: 0.157 0.159 (μs) 	[μSD μRSD]/r: 0.019μs 12.38%

4 subjects, 400 iterations, 40,000 revs, 0 rejects
(best [mean mode] worst) = 0.124 [0.152 0.151] 0.225 (μs)
⅀T: 60.773μs μSD/r 0.020μs μRSD/r: 13.293%
suite: 133a2c5566a4e9fb57b0251cebfd189bc150f104, date: 2016-12-21, stime: 22:06:03

+-------------------------+-------+-----+---------+--------+
| subject                 | revs  | its | mean    | diff   |
+-------------------------+-------+-----+---------+--------+
| benchFqGlobalFunction   | 10000 | 100 | 0.145μs | 0.00%  |
| benchGlobalFunction     | 10000 | 100 | 0.148μs | +2.26% |
| benchNamespacedFunction | 10000 | 100 | 0.157μs | +8.55% |
| benchOverriddenFunction | 10000 | 100 | 0.157μs | +8.38% |
+-------------------------+-------+-----+---------+--------+

As expected, the fully qualified global function call is the fastest one. This is because PHP does not need to go through the INIT_NS_FCALL_BY_NAME opcode. When calling the global function in a non-fully qualified way, it is slower. Running functions inside a namespace are always slower then running global functions.

Of course, this is not a big overhead in this simple benchmark. It could be a big overhead if you think about the amount of function calls per run. PHP is not able to optimize this since it is possible that functions get defined during runtime.

Speeding up your application

Currently the only way to speed up the function calls, is by making sure that the global functions are called in a fully qualified way. This is a tedious manual action. You could do this in one of these 2 ways:

<?php
// solution 1:
namespace baz;
\foo();

// solution 2:
namespace baz;
use function foo;
foo();

Luckily for us, the community is very creative and alert when it comes to performance. Maybe one of the following (future) solutions is less boring to implement:

Add the php_backslasher package as a git hook.
Add a declare(no_dynamic_functions=1) on top of the PHP file or maybe in a future namespace_scoped_declares method.
Autocompletion to FQ function names in PHPStorm.
Another awesome package by Ocramius?
A CLI tool that checks your files for FQ function calls.
A GrumPHP task.

Conclusion

As you can see, performance killers can hide in small corners. I'm glad to see how this little tweet can get this much feedback from the community in no time. Let's hope that a good solution for this performance problem will be added to PHP soon so that we can easily speed up our applications even more. Now that the secret behind this optimization is revealed, I am looking forward to discover more of these little performance killers.

Writing your first NodeJS Lambda function

2016-07-30T00:00:00+02:00

Last week I was playing around with writing some custom Slack slashcommands and cronjobs. Originally, I started writing these integrations with PHP. Since slack requires a publically available endpoint and I did not want to set up a server for these scripts, I decided to take a look at AWS Lambda. This Amazon service makes it possible to run scripts serverless. The scripts are triggered based on triggers inside of AWS. In this artile you can find out how to get started with writing lambda functions.

Selecting the right tools

AWS Lambda can run Python, Java and NodeJS scripts. Since I've got some experience writing NodeJS, I've choosen to go for this technology. There are many tools available for writing Lambda functions in NodeJS. After investigating these tools, I've decided to go with lambda-tools. This tool makes it easy to develop and deploy multiple lambda functions inside one codebase. It comes with a Yeoman generator named generator-lambda-tools.

Getting started

The first thing we'll want to do is scafold the application:

$ yo lambda-tools

? Service name: your-lambda-name
? Service description: Some Description
? License (API): MIT
? Author email your-email
? Author name your-name
? Default Lambda runtime Node.js 4.3
? Install dependencies lambda-tools, lambda-foundation

When the command is finished, you'll find a project structure like this:

.
├── .lambda-tools-rc.json
├── .yo-rc.json
├── api.json
├── cf.json
├── lambdas/
└── node_modules/

There are 2 hidden configuration files named `.lambda-tools-rc.json` and `.yo-rc.json`. These are used to configure the yeoman generator and the lambda-tools command. The `api.json` is used for the AWS API Gateway service. This service will make it possible to create a public endpoint that triggers the Lambda method we'll create next.
The `cf.json` is used for the AWS CloudFormation service. This service will take care of orchestrating the related AWS services. It will be used to deploy your lambda functions and let all the AWS services talk to each other.
The `lambdas` directory is currently empty, but will contain all the lambda functions you are going to write.
Finally there is the `node_modules` directory which contains all your project dependencies.

Writing your first lambda

$ yo lambda-tools:lambda

? Lambda function name: mylambda
? Create event.json: Yes

The command above will create a folder named `mylambda` inside the `lambdas` directory. In this folder there are 2 files: `event.json` and `index.js`. The event.json contains the data that will be used as input for the Lambda function. Feel free to put it in `.gitignore` since you can only use this for testing purpose. The `index.js` file contains a `handler()` method and looks like this:

'use strict';

exports.handler = function(event, context) {
    context.succeed('Hello!');
};

The callback has 2 parameters: `event` which will be populated by AWS Lambda. During development this parameter can be linked to the `event.json` file. `context` which will make it possible to `succeed()` or `fail()` the lambda method. The parameters added to these 2 functions will be returned as result of the lambda:

$ node node_modules/lambda-tools/bin/lambda.js execute ./lambdas/mylambda/index.js -e lambdas/mylambda/event.json -v

Executing: lambdas/mylambda/index.js
	--
	With event:
	{}
	--


	--
	Result 'Hello!'

Creating a cronjob

$ yo lambda-tools:periodic-lambda      

? Lambda function name cronjob
? Create event.json Yes
? Schedule/Rate expression cron(0 10 * * ? *)
? CloudWatch rule name (leave empty for automatic name) ten-o-clock
? CloudWatch rule logical ID (used in CloudFormation template) TenOClockEvent
? CloudWatch rule description (leave empty to skip)
? CloudWatch rule static input (JSON string, leave empty to skip)
? CloudWatch rule input path (JSONPath string, leave empty to skip)
? Is CloudWatch rule ENABLED? Yes

In AWS CloudWatch, it is possible to create RATE or CRON rules. These rules can be the trigger for a Lambda function to execute. This means it is possible to create a lambda function that executes periodically. The generated code looks and behaves exactly the same as the `mylambda` code above. In the `cf.json` file, a event rule is added to cloudwatch and the event is linked to the lambda function.

Creating an API endpoint

$ yo lambda-tools:endpoint   

? Path for the endpoint /lambdas/apilambda
? HTTP Method POST
? Lambda function name lambdas-apilambda-post
? Map request body to event property (leave blank to skip) body
? Map HTTP headers? No

By adding an API endpoint, you can create a publically available HTTPS endpoint that triggers a Lambda function. The generated code again looks the same as the `mylambda` code. The `api.json` file is altered and now describes the new endpoint:

{
    "paths": {
        "/lambdas/apilambda": {
            "post": {
                "responses": {
                    "200": {
                        "description": "Default response",
                        "schema": {
                            "type": "object",
                            "properties": {},
                            "additionalProperties": true
                        }
                    }
                },
                "parameters": [],
                "x-amazon-apigateway-auth": {
                    "type": "none"
                },
                "x-amazon-apigateway-integration": {
                    "type": "aws",
                    "uri": "$lLambdasApilambdaPost",
                    "credentials": "$IamRoleArnApiGateway",
                    "httpMethod": "POST",
                    "requestParameters": {},
                    "requestTemplates": {
                        "application/json": "{\"path\":\"$context.resourcePath\",\"body\":$input.json('$')}"
                    },
                    "responses": {
                        "default": {
                            "statusCode": "200",
                            "responseTemplates": {
                                "application/json": "$input.json('$')"
                            }
                        }
                    }
                }
            }
        }
    }
}

The added path describes the endpoint. As you can see it only handles POST requests with a content-type of `application/json`. The JSON body of the request is added as the "body" parameter of the Lambda event. This is done internally through the `x-amazon-apigateway-integration` header. In the response you'll have to add the responseTemplate to make sure that the result of the lambda is returned as JSON. You can test the API endpoint with following commands:

# This command will start a server on port 3000:
$ node node_modules/lambda-tools/bin/lambda.js run -a api.json

# Send a curl request:
$ curl -X POST -H "Content-Type: application/json" -d '{request: "will be added to event as body param"}' "http://localhost:3000/lambdas/apilambda"

This HTTP request will be handled as followed:

  <-- POST /lambdas/apilambda
	<-- Configuring Lambda function
	--> Configuring Lambda function 1ms

	<-- Creating integration
	{
		"context": {
			"apiId": "local-lambda",
			"httpMethod": "POST",
			"identity": {},
			"requestId": "98e626ab-bbc7-4be2-a94c-b6f622b1d140",
			"resourceId": "POST /lambdas/apilambda",
			"resourcePath": "/lambdas/apilambda",
			"stage": "dev"
		},
		"event": {
			"path": "/lambdas/apilambda",
			"body": {
				"request": "will be added to event as body param"
			}
		}
	}
	--> Creating integration 14ms

	<-- Executing Lambda (lambdas-apilambda-post/index.handler)


	--> Executing Lambda function 151ms

	<-- Creating response
	{
		"status": 200,
		"headers": {},
		"type": "application/json",
		"body": "hello!"
	}
	--> Creating response 1ms

  --> POST /lambdas/apilambda 200 200ms 87b

Configuring AWS

Now that we've created our lambda methods, the next step is to deploy them to AWS. This step is not documented very good, so I'll try to describe it as good as possible.

First you'll need to create a user with the IAM-permissions described on the lambda-tools page. When you've created this user, you'll have to make sure that the user has an Access Key so that it can be used to automate stuff. Now that you've got both the Access Id and Secret Access Key, you'll have to generate an API token:

$ aws configure
$ aws sts get-session-token

Now that we have all the AWS parameters, we'll need to create a `.env` file so that the tool knows how to deploy the application:

AWS_ENVIRONMENT=dev
AWS_ACCESS_KEY_ID=your-key-id
AWS_SECRET_ACCESS_KEY=your-key-secret
AWS_SESSION_TOKEN=your-key-token
AWS_REGION=eu-west-1
AWS_RUNTIME=nodejs4.3
EXCLUDE_GLOBS="event.json *.env *.dist"
PACKAGE_DIRECTORY=build

Finally we need to configure a AWS S3 bucket in which the lambda functions will be stored. This needs to be a unique index which can be configured in the `.lambda-tools-rc.json` file. Add following `tools` to the root element:

{
    "tools": {
        "resources": {
            "s3Bucket": "your-unique-s3-bucket-name"
        }
    }
}

Now that we've configured the project source code to run with AWS, we'll have to set up everything in AWS. Luckilly this can be done with one simple command that only needs to run once:

$ node node_modules/lambda-tools/bin/lambda.js setup -r eu-west-1

Deploying your lambdas to AWS

$ node node_modules/lambda-tools/bin/lambda.js deploy -r eu-west-1

By running the command above, the lambdas are build into some archive files. These archive files are stored in your AWS S3 Bucket. When this is done, the `cf.json` file is added to AWS CloudFormation which will start the orchestration of the installation:

A CloudFormation stack will be created or updated based on the `cf.json` file.
The source code of the lambdas are stored in your S3 Bucket
IAM permissions will be added
A new scheduling rule will be added to CloudWatch.
New logging groups will be added to CloudWatch.
The API gateway will be configured with the `api.json` file.
The lambdas will be created and configured.

Adding configurable variables to the lambdas

Adding custom configuration to a lambda function can be done based on environment variables. These variables can be added with the `-e` option during a deploy. When you've got a lot of configurable variables, another option is to add a .env file to every lambda directory and use the env-loader. You'll have to create an asset to make sure the .env file is added to your lambda source code. This can be done by adding a new `cf.json` file to every lambda directory that looks like:

{
  "Assets": {
    ".env": ".env"
  },
}

Conclusion

If you are writing your first AWS project, getting started is pretty complex. The documentation is rather big and you'll have to find your way in all the different AWS services. Once you found your way and got your project set-up and configured, writing Lambda functions is pretty fun. They are easy to understand and there are a lot of NPM packages available for Rapid Application Development. I am definitely looking forward in writing more Lambda integrations!

GrumPHP approves Bitbucket Pipelines!

2016-06-15T00:00:00+02:00

Today, my invitation for Bitbucket Pipelines Beta got accepted! For those who haven't heard about it: Pipelines is a Continuous Delivery tool integrated in Bitbucket. When you push your changes to your Bitbucket repository, it will spin up a Docker container and run the actions you specify. I was quite exited about the concept and immediately started to set up an automated test scenario with GrumPHP.

Configuration

The configuration of Pipelines is similar to existing services like e.g. Travis. It all starts by adding a bitbucket-pipelines.yml file to the root of your project.

bitbucket-pipelines.yml:

image: php:7.0-cli

pipelines:
  default:
    - step:
        script:
          - apt-get update && apt-get install -y git
          - php -r "readfile('https://getcomposer.org/installer');" | php
          - php composer.phar global require hirak/prestissimo
          - php composer.phar install --prefer-dist
          - ./vendor/bin/grumphp run

After every commit, the pipelines file will be loaded. Since we've added a default pipeline, it will spin up a basic PHP 7.0 container. Because it's a basic container, we'll have to install git and composer to make sure the test scenario works. The installation of these packages will happen on every commit, so it is better to create your own container which contains all the tools you need to test your code. Finally, the composer dependencies are installed and the GrumPHP command is being triggered.

Analysing the results

As you can see, the Pipelines overview page is very clean and focuses on the output of your tasks. Every commit will trigger a new run and a new results overview page. This makes it very easy to spot problems before they got merged in!

Pros

It is a unique service that integrates completely on your git server!
The interface is very simple and focuses on the errors.
Pipelines is integrated in every screen in Bitbucket. You get feedback about the committed code in a wink of an eye.
It is possible to let Pipelines handle your Continuous Delivery flow!
You can specify different steps per git branch.
It works perfectly fine with GrumPHP!
It is FREE!

Cons

The docker image is loaded from DockerHub which takes a while.
You can only specify 1 image, which means you cannot test every commit against multiple PHP versions.
It is not possible to cache the composer packages. All dependencies are downloaded on every commit.

Conclusion

Bitbucket has done a really great job creating ther unique Pipelines service! Currently it is still in Beta which means they are still working on making it even better. I hope to see some additional features like running tests on multiple images, caching volumes, ... in a near future. In a next phase, I will be playing around with the Continuous Delivery options and maybe link Pipelines with an Ansible script to automatically deploy the changes to a development server.

Authenticating with X.509 client certificates

2015-12-07T00:00:00+01:00

Last week, I was diving in different authentication systems for API's. One of the better ways of authentication is through X.509 client certificates. This one is a bit is harder to set-up, but sure is secure, manageable and powerful. While searching for documentation on the subject, I was surprised there weren't a lot of good articles. In this article, I will try to explain every step as easy as possible.

Why should I use X.509 authentication?

The main advantage is that the client is not sending a username or password to the server. This means that a man-in-the-middle attack is nearly impossible. It is much easier to steal a username/password login, for example by bruteforcing, then stealing a certificate. Because the certificate is signed, it is only possible to connect to the real server. It is possible to revoke and manage these certificates in an easy way.

Configuring the server

Client Certificate authentication can only be done while running HTTPS. So first of all, make sure the server is running HTTPS. This can be done with a self-signed or a signed certificate. Your apache VHost configuration should look more or less like this:

SSLEngine on
SSLCertificateFile      "/etc/ssl/certs/server.pem"
SSLCertificateKeyFile   "/etc/ssl/private/server.key"
SSLProtocol             TLSv1 TLSv1.1 TLSv1.2
SSLCipherSuite          "......."
 
# When using signed certificates:
SSLCertificateChainFile "/etc/ssl/yourDomainName.ca-bundle"

Now that you got HTTPS up and running, you should be able to browse to your application through HTTPS.

Creating a certification authority

A certification authority (CA) hands out a digital certificate in which the CA says that a public key in the certificate, belongs to the person, organization, server or entity that is mentioned in the certificate. In our example, this will be done based on the e-mail address that is provided in the certificate. The task of the CA is to control the identity of the issuer, so that the client that is using the certificates from the CA can be trusted.

First we will need to create the CA private key and certificate:

cd /etc/ssl/certs/
openssl genrsa -out ca.key 4096
openssl req -new -x509 -days 365 -key ca.key -out ca.crt

Note: Make sure to set the CN to a valid server domain. Otherwise this might result in exceptions!

Now that we have the CA key and certificate, we can register it to Apache to validate the client certificates:

SSLCACertificateFile "/etc/ssl/certs/ca.crt"
SSLVerifyClient optional
SSLVerifyDepth 1
SSLOptions +StdEnvVars

As you can see the client certificate verification is optional. This will make it possible to add another type of authentication like basic authentication when there is nog client certificate. The verify depth is set to 1 so that it only accepts certificates signed by the configured CA. Finally, the StdEnvVars are registered so that the additional SSL server variables are available in PHP.

Configuring the client

Now that we got or server fully configured, it is time to create a client certificate. Following commands should be ran on the client machine to create a Certificate Signing Request (CSR):

openssl genrsa -out client.key 4096
openssl req -new -key client.key -out client.csr

Note: Make sure to set the CN to a valid server domain. Otherwise this might result in exceptions!
Note: The email field will be used to authenticate the user.

The CSR file is useless when it is not signed by the CA. So the logical next step is to transfer the certificate to the server and sign it with the CA.

openssl x509 -req -days 365 -in client.csr -CA ca.crt -CAkey ca.key -set_serial 01 -out client.crt

The command above results in a useful client certificate. To make it easier to use the certificate, we will pack the client private key and the certificate in one file. This action should run on the client machine:

cat client.crt client.key > client.pem

Note: it's important to place the crt before the key in the pem file!

In above example the CSR was created on the client, to make it clear that the certificate + key should only be known by the client. However, it is perfectly possible to run all these commands on the server and send the pem file to the client who will be using the certificate. This means that the creation of the client certificate can be automated. You could create your own user interface to make the keys manageable per user of the application. This can for example be done with the built-in openssl extension of PHP. When the certification file is generated on the server, you should transfer this certificate in a trusted way. For example a download over HTTPS in the back-end of your application.

Requesting resources with a client certificate

Ok, we configured our server and requested a client certificate. Now how do I use this certificate to get my resources? The easiest way is the add the "-cert" attribute to the curl command. Since we are using Guzzle for HTTP requests, the client configuration will look like this:

<?php
$client = new Guzzle\Http\Client($baseUrl, [
    'ssl.certificate_authority' => 'system',
    'request.options' => [
        'cert' => 'client.pem',
    ]
]);

As you can see it is possible to specify the certificate in the request.options part of the configuration. Another option is the ssl.certificate_authority. This one can be used to specify which CA that should be used. By default the built-in CA file is being used. You can choose to disable ssl verification or add your own ca file. For example, when using self-signed certificates, you can run following command:

openssl s_client -showcerts -connect my.host.com:443 > ca-file.pem

This ca-file.pem file will contain the certificate of the certification authority and mark it as trusted. Now it can be used in the PHP configuration as followed:

<?php
$client = new Guzzle\Http\Client($baseUrl, [
    'ssl.certificate_authority' => 'ca-file.pem',
    'request.options' => [
        'cert' => 'client.pem',
    ]
]);

X.509 authentication in Symfony

One of the less known features of Symfony is X.509 authentication. It can easily be configured with the default x509 authentication security adapter. The configuration is pretty easy:

security:
    providers:
        client_certificate:
            memory:
                users:
                    email@certification.file:
                      roles: ROLE_SUPER_ADMIN
 
    firewalls:
        main:
            x509:
                provider: client_certificate
 
    access_control:
            - { path: ^/, roles: ROLE_SUPER_ADMIN, requires_channel: https }

Note: This type of authentication only works with HTTPS. You might want to enforce HTTPS!

As you can see HTTPS is enforced and authentication will try X.509 authentication by default. When a valid client certificate is found, Symfony will try to match the email that is configured inside the certificate with a user in the client_certificate user provider. In this case we are using an in-memory provider that links an e-mail to a security role.

X.509 authentication in PHP

That was easy! But how does it work? By adding the "SSLOptions +StdEnvVars" configuration in Apache, there are some additional "SSL_" environment variables available. These variables contain the email in the client certificate. For example:

<?php
$email = $_SERVER['SSL_CLIENT_S_DN_Email'];
if (!$email && preg_match('#/emailAddress=(.+\@.+\..+)(/|$)#', $_SERVER['SSL_CLIENT_S_DN'], $matches)) {
    $email = $matches[1]
}

Now that we have the e-mail, it is possible to search this email in the list of configured client_certificates. Symfony will throw an exception when the email can't be found.

Conclusion

Even though it is a lot of work to get this type of authentication running, it sure is a powerful type of authentication. The users of your application will be able to connect to your application in an improved and secure way. It also comes in very handy for server to server communication in which you don't always want to hardcode user credentials. Your customers will surely thank you for the extra effort you made!

Hijacking your code with streams

2015-10-26T00:00:00+01:00

Lately, you might have noticed that streams are getting more important in the PHP scene. They are for example widely used in PSR-7 and implemented in Diactoros. Yesterday I was scrolling down the issues list of Phpspec and bumped in on this pull request. This pull request shows that it is possible to convert code in-memory and later require the modified content to be executed. Wow, that is awesome! Let's take a look at it.

The concept

You probably already used some built-in PHP streams like regular files, `php://memory` or `data://...` in combination with the `fopen()` method or the `StdFileObject` class. Beside those built-in streams, it is possible to configure your own stream wrappers. For this post I will be creating a blacklist stream that will remove blacklisted methods from your code. The blacklist stream wrapper will read a PHP file and replaces blacklisted methods from the code. When the file is required through the blacklist wrapper, all blacklisted methods will be removed from the code.

Show me some code

<?php

class StreamWrapper
{

    private $realPath;
    private $fileResource;
    private static $blacklistedMethods = [];

    public static function blacklistMethod($method)
    {
        self::$blacklistedMethods[] = $method;
    }

    public function stream_open($path, $mode, $options, &$opened_path)
    {
        $this->realPath = preg_replace('|^blacklist://|', '', $path);
        if (!file_exists($this->realPath)) {
            return false;
        }

        $content = file_get_contents($this->realPath);
        foreach (self::$blacklistedMethods as $blacklistedMethod) {
            $content = preg_replace('/^' . $blacklistedMethod . '\([^\)]*\);$/im', '', $content);
        }

        $this->fileResource = fopen('php://memory', 'w+');
        fwrite($this->fileResource, $content);
        rewind($this->fileResource);

        $opened_path = $this->realPath;
        return true;
    }

    public function stream_stat()
    {
        return stat($this->realPath);
    }

    public function stream_read($count)
    {
        return fread($this->fileResource, $count);
    }

    public function stream_eof()
    {
        return feof($this->fileResource);
    }
}

Usage

To use the stream wrapper, you have to register it with the name blacklist. Next you can require the file with the new stream URL. Internally PHP will open the file with your custom stream reader. This means that you can alter the content of the PHP file in an in-memory stream which will be used by PHP. Here is what the file looks like.

<?php
// run.php

require('StreamWrapper.php');
StreamWrapper::blacklistMethod('die');
stream_wrapper_register('blacklist', 'StreamWrapper');

require 'blacklist://test.php';

The above code needs a file test.php. This file looks like this:

<?php
// test.php

die('now you see me ...');
echo('now you dont!');

When executing the run.php you will see the result:

$ php run.php
now you dont!

What the hell did I just read…

I know, most projects won't need in-memory code adjustments. The blacklist wrapper in this post will probably never be needed, but it is used to show you what is possible. Streams are pretty cool and will probably become more important in the future.

Replacing service dependencies with proxies

2014-10-11T00:00:00+02:00

When using the Zend Framework 2 service manager, it is possible to create shared services that will be loaded only once. In some situations however, it is very hard to switch the already injected dependencies in this service. You could mark the service as unshared even if this is often unnecessary. Another solution is to wrap the service with a proxy object and use the proxy instead of the service.

Real-life Scenario

In an application with multiple tenants, there is one database per tenant. When browsing the application the tenant's database connection is always known. This connection doesn't ever change when browsing the application. Therefor the Doctrine DocumentManager is injected and the services are marked as shared.

On the server, there are some long-running processes to handle async commands. These processes run globally and are not dependant on a specific tenant. This means that the processes need to switch the DocumentManager based on the tenant they are working for at the moment.

Off course, the registered services have to work with the current DocumentManager, even if they are marked as shared.

Delegators

One of the cool features of ZF2 are service delegators. These delegators make it possible to attach or wrap custom functionality to an instance. In this case we will use the delegator to wrap the actual service in a proxy. The instance of the service will still be created in the service manager like before, but instead of returning this instance we will return a proxy object.

Proxies

In this post, I already talked a lot about using proxies. But what are those proxies?

The short answer: Proxies are objects that serve as a gateway to the actual object.

The long answer: There are many different types of proxies. Here you can find a good overview of all proxy patterns in PHP.

In this specific case, a Virtual Proxy is the right proxy to use. It extends the actual base class and has exactly the same API. An over-simplified proxy of a Doctrine DocumentManager could look like this:

<?php

class DocumentManagerProxy
    extends DocumentManager
{

    protected $instance;

    public function __construct(DocumentManager $instance)
    {
        $this->instance = $instance;
    }
    
    public function find($className, $id)
    {
        return $this->instance->find($className, $id);
    }
    
    // ... All other methods of the DocumentManager ...

}

Because this kind of objects will result in a lot of effort in maintaining, it is better to use a library that automatically generates the proxy objects. One of those libraries is ProxyManager by Ocramius.

Putting it all together

ServiceManager configuration

<?php

return [
    'factories' => [
        'documentmanager' => new \DoctrineMongoODMModule\Service\DocumentManagerFactory('manager_key'),
        'Application\DocumentManagerProxyDelegator' => 'Application\Factory\DocumentManagerProxyDelegatorFactory',
        'Application\DynamicDocumentManager' => 'Application\Factory\DynamicDocumentManagerFactory',
    ],
    'delegators' => [
        'documentmanager' => ['Application\DocumentManagerProxyDelegator'],
    ]
];

Delegator

The delegator is responsible for initializing the actual DocumentManager and creating the proxy object. Another service is added that is responsible for maintaining and loading the different DocumentManagers. Make sure that the proxy initializer returns false, so that the proxy is initialized with the current DocumentManager on every method call.

<?php

namespace Application;

use ProxyManager\Factory\LazyLoadingValueHolderFactory;
use Zend\ServiceManager\DelegatorFactoryInterface;
use Zend\ServiceManager\ServiceLocatorInterface;

class DocumentManagerProxyDelegator implements DelegatorFactoryInterface
{

    /**
     * @var LazyLoadingValueHolderFactory
     */
    protected $proxyFactory;

    /**
     * @var DynamicDocumentManager
     */
    protected $dynamicDocumentManager;

    /**
     * @param $dynamicDocumentManager
     * @param $proxyFactory
     */
    public function __construct($dynamicDocumentManager, $proxyFactory)
    {
        $this->dynamicDocumentManager = $dynamicDocumentManager;
        $this->proxyFactory = $proxyFactory;
    }

    /**
     * {@inheritdoc}
     */
    public function createDelegatorWithName(ServiceLocatorInterface $serviceLocator, $name, $requestedName, $callback)
    {
        $dynamicDocumentManager = $this->dynamicDocumentManager;
        $currentDocumentManager = $callback();
        $dynamicDocumentManager->setCurrentManager($currentDocumentManager);

        $proxy = $this->proxyFactory->createProxy(
            'Doctrine\ODM\MongoDB\DocumentManager',
            function (& $wrappedObject) use ($dynamicDocumentManager) {
                $wrappedObject = $dynamicDocumentManager->getCurrentManager();
                return false;
            }
        );

        return $proxy;
    }

}

Delegator Factory

The delegator factory creates an instance of the factory. For this example It is very straight forward. If you are planning to use it in production, you might add some extra caching configuration to the proxy factory. This configuration can be based on `Zend\ServiceManager\Proxy\LazyServiceFactoryFactory`.

<?php

namespace Application;

use Application\DocumentManagerProxyDelegator;
use ProxyManager\Factory\LazyLoadingValueHolderFactory;
use Zend\ServiceManager\FactoryInterface;
use Zend\ServiceManager\ServiceLocatorInterface;

class DocumentManagerProxyDelegatorFactory implements FactoryInterface
{

    public function createService(ServiceLocatorInterface $serviceLocator)
    {
        $dynamicDocumentManager = $serviceLocator->get('Application\DynamicDocumentManager');
        $proxyFactory = new LazyLoadingValueHolderFactory();
        return new DocumentManagerProxyDelegator($dynamicDocumentManager, $proxyFactory);
    }
    
}

Dynamic DocumentManager

The dynamic DocumentManager is responsible for loading and managing the DocumentManagers. While creating an instance for the DocumentManager, the real instance is injected in the DynamicDocumentManager. An implementation might look like this:

<?php

interface DynamicDocumentManagerInterface
{

    public function getCurrentManager();
    public function setCurrentManager(DocumentManager $documentManager);
    public function loadForTenant(Tenant $tenant);

}

The `setCurrentManager()` is called by the delegator to set the current manager.

The `getCurrentManager()` is called while initializing the proxy. This way, the current manager will always be used when using the `documentmanager` in a shared service.

Before executing a command in the long-running process, the `loadForTenant()` method is being called. This method will close the old manager and reset the service manager keys for all Doctrine services. Finally it will recreate the `documentmanager`, which will trigger the `setCurrentManager()` method again.

Final thoughts

This technique might look a little bit `hacky`, but is actually a very neat trick to make it easier to change instances on the fly.

You don't have to mark all services that rely on the DocumentManager as unshared. In the codebase, you can still use the actual instance type in the annotations.

You might think of some other good use-cases to implement this powerful technique yourself.

Generating ZF2 template maps on the fly

2014-08-19T00:00:00+02:00

One of the performance boosters in Zend Framework 2 is using `template_map` instead of the `template_path_stack` in the view manager. While developing, it is easier to use the stack. This is because you don't have to add a mapping every time you create a new view. At the end of the development you might forget to transform the stack into a template map. This is why it is useful to never use the stack and automate the generation of the template_map array instead.

There is already a great tool called `templatemap_generator.php` included in the bin folder of zf2. It can be used to generate a template map for every module in your application. In the next step, we want to make sure that this tool is called every time a view file is created. This kind of automation can be done with a task manager like `Grunt`.

Sample configuration

grunt.initConfig({

    phplint: {
      applicationViews: ['module/Application/**/*.phtml'],
    }

    watch: {
      applicationViews: {
        files: ['module/Application/view/**/*.phtml'],
        tasks: ['phplint:applicationViews', 'exec:generate_application_template_map']
      }
    },

    exec: {
      generate_application_template_map: {
        cmd: '../../vendor/bin/templatemap_generator.php',
        cwd: 'module/Application'
      }
    }

});

grunt.registerTask('generate-template-maps', ['exec:template_map_application']);

By using this configuration, Grunt will watch for file changes in the `view` folder of the `Application` module. When a file has changed, PHPLint will check for syntax errors and finally a new template map file will be generated. There is also a command named `generate-template-maps` which you can run manually.

Note: Make sure to include the npm packages grunt-exec and grunt-phplint.

Conclusion

By automating all of the boring tasks, you can gain a lot of time during development. If you want to automate some more tasks during PHP development, you can check an earlier blog called `Validating your PHP code on the fly`.

Writing your first Grunt plugin

2014-04-27T00:00:00+02:00

For one of my projects, I needed a tool to list all used translations keys from a PHP project. Because I want to keep the translation database clean, I decided to create a Grunt plugin. There was one big problem: How do you create this plugin? So I started to read the documentation, some blog posts, etc. and noticed that the documentation was pretty confusing. Let me take you on a tour through my first plugin.

Where to start?

Create the plugin skeleton

The Grunt team has created a tool called grunt-init. It’s purpose is to quickly scaffold your project with some basic templates. The tool does not include any templates, so you have to download them manually. There is a tempalte for creating a grunt-plugin and another one to add a Gruntfile to your project. When you run the tool, a wizard is launched to personalize the template.

$ grunt-init gruntplugin

After running this command, all the template files are being copied to your folder. The structure is pretty straight forward. There is a tasks folder which contains the Grunt tasks. Another folder is the test folder. This is the place where you write some node tests for your plugin. Finally, there is also a Gruntfile which will help you with some basic tasks like testing the plugin.

Configure NPM

Once you got through the installation wizard, you should personalize the NPM file. My plugin is project specific, so I decided to make it a private repository. I also like using Lo-Dash for handling arrays, so I added this one as a dependency of the plugin. All these parameters can be configured in the package.json file.

  "private": true,
  "dependencies": {
    "lodash": "~2.4.1"
  },

When the configuration is saved, the next step is to install the dependencies. This can be done by running the command:

$ npm install

Writing the task

Task structure

You can add one or more tasks to your plugin. By default there is one multi-task in the tasks folder. It looks like this:

'use strict';

module.exports = function(grunt) {
  grunt.registerMultiTask('findTranslations', 'Find all translations in your project.', function() {

    // The task code comes here

  };
};

Initialize dependencies

Grunt has some standard tools to handle file actions and logging. When you want to use custom libraries, you can include them through require.

    var _ = require('lodash');

Options

If you ever worked with Grunt before, you know that all commands are configurable. Some of the options will have a default value. In the plugin, you can add custom options as followed:

    var options = this.options({
      format: 'json',
    });

The options in the object are the default options and can be overwritten in the Gruntfile.

Files

Next to the options, there is a basic configuration parameter to specify source and destinations files. The configuration looks like:

  files: [
    {
      src: [
        'test/fixtures/php-controller-plugin',
      ],
      dest: 'tmp/translations-json'
  }
],

As you can see, it is possible to add multiple src / dest collections. The source can consist out of multiple files and the names can contain regular expressions like wildcards. In the plugin, it is very easy to loop through the files:

    this.files.forEach(function (f) {

      var translations = [];
      f.src.forEach(function(filePath) {
        var content = grunt.file.read(filePath);
        // Todo: Find translation keys
      });

      grunt.file.write(f.dest, translations.join("\n");
    });

Locating translation keys

To find the translation keys in a file, I needed to write some regular expressions to find the key. Luckily for me, I use a very basic translation method. In my project I have 2 different ways of retrieving translations: one for the controller and one for the view.

    var translationKeyRegex = "[\s]?.*[\\\'\\\"]{1}([a-z0-9\\-\\_]*)[\\\'\\\"]{1}[\s]?.*";
    var regexs = {
      'phpControllerPlugin': "translator->get\\(" + translationKeyRegex + "\\)",
      'phpViewHelper': "_t->get\\(" + translationKeyRegex + "\\)"
    };

This may look a bit hard at first, but it is actually very easy. There are only 2 patterns:

In the controller: $translator->get('KEY_123');
In the view: _t->get('KEY_123');

After the regular expressions were written, I created a function that returns the translation keys:

    var findTranslationKeys = function(content, regex) {
      var found = [];
      var match;

      while ((match = regex.exec(content)) !== null) {
        found.push(match[1]);
      }

      return found;
    };

Okay, the hard part of the plugin is written. Now the only thing left is adding this functionality to the part where we read the file. I want to execute all regular expressions patterns on every file. The resulting array should only contain unique entries. This can easily be done with the Lo-Dash forEach and union method:

  var found, regex;
  _.forEach(regexs, function(expression) {
    regex = new RegExp(expression, 'gi');
    found = findTranslationKeys(content, regex);

    // Only use unique translation keys:
    translations = _.union(translations, found);
  });

Formatting the output

As you can see in the files part, at the moment we only output the result as newline-separated text. Because I want to clean up my database through PHP, I decided to make JSON the default format. To specify which format will be used I added one other function:

    var formatTranslations = function(translations, format) {
      var formatted = '';
      switch(format) {
        case 'newlines':
          formatted = translations.join("\n");
          break;
        case 'json':
          formatted = JSON.stringify(translations, null, 4);
          break;
        default:
          formatted = 'Invalid export format';
          break;
      }
      return formatted;
    };

The translations parameter is the array of found translation keys and the format is configured in the options.format variable.

Logging

At the end of the command, I find it useful to log the results of the task. In this case, I only print an indication of the amount of found items.

    grunt.log.writeln('Saved ' + translations.length + ' translation keys to file ' + f.dest);

Testing

For now, I only talked about writing the plugin. Of course it is also recommended to write some test for your plugin. As far as I can see in other plugins, there are only functional tests. In the test directory there are 2 folders. The fixtures folder is where you add the files where you want to find translation keys in. In the expected folder you place the output file that you expect the plugin to generate.

Now the first thing that you need to do, is to configure your task in the Gruntfile. When you run the test command, your configured tasks will run first. So in the tasks you can configure to run the tasks with the fixtures file and save it in a tmp folder. When you know which files are being read and written, it is easy to create your test:

  json_format: function(test) {
    test.expect(1);

    var actual = grunt.file.read('tmp/translations-json');
    var expected = grunt.file.read('test/expected/json_format');
    test.equal(actual, expected, 'should return translations in json.');

    test.done();
  },

Now the only command you need to run to test your code is:

$ grunt test

Deployment

Once your plugin is ready, you want to place it in GIT. Because this is a private repository, I placed it on a private repository on BitBucket. Now the next problem arose: How can I load my custom plugin through NPM in another project? In the latest versions of NPM, it is possible to add packages that are not managed on NPM. The real problem is that it is a private repository. After some searching, I found out that the only good way the require the package is to use the SSH url of the package. This is what I placed in the package.json file of the main project.

  "devDependencies": {
    "grunt-find-translations": "git+ssh://git@bitbucket.org:organization/package.git"
  }

Note: Don’t forget to npm install

In the Gruntfile of the project, I configured the task to search for translations in all .php and .phtml files. The result of the tasks will be saved in the logs folder.

    findTranslations: {
      json_format: {
        options: {
            format: 'json',
        },
        files: [
          {
            src: [
              '{,*/}*.php',
              '{,*/}*.phtml',
            ],
            dest: 'logs/translation-keys.json'
          }
        ],
      },
    },

Finally my task was ready to use in my project:

$ grunt findTranslations

>> Running "findTranslations:json_format" (findTranslations) task
>> Saved 250 translation keys to file logs/translation-keys.json

Three-way data binding in AngularJS

2014-04-06T00:00:00+02:00

One of the cool features in AngularJS is the two-way data binding. When you change a property of an object, the value is instantly changed in every view or service that is using this property. A while ago, I saw a video about the combination of AngularJS and Firebase. This combination makes it possible to create three-way data binding in AngularJS.

What is Firebase?

Firebase is an online service that provides an API to store and sync data in realtime. This means that when the data is saved, all connected clients will retrieve the latest data without reloading the page. You can see Firebase as a database service with some extra features like authentication. The cool thing about this service is that you don’t need to create a database schema. You can create collections just by specifying a path to the database in the URL. An additional package ‘AngularFire’ is available for integration with AngularJS.

The world’s easiest guestbook

Getting started

A good way to start a test project is by using Yeoman. Both AngularJs and AngularFire have a generator that will create your basic application in no time. Just run following commands and follow the wizard:

$ yo angular
$ yo angularfire
$ grunt serve

The view

<div class="jumbotron">
  <h1>'Allo, 'Allo!</h1>
  <p class="lead">
    Please leave a message if your like my page!
  </p>

  <div class="form-group">
      <input ng-model="input.name" placeholder="Name" class="form-control" />
  </div>
  <div class="form-group">
      <textarea ng-model="input.message" class="form-control" placeholder="Message"></textarea>
  </div>
  <p><a class="btn btn-lg btn-success" data-ng-click="addMessage()">Splendid!</a></p>
</div>

<div class="messages" ng-cloack="">
    <div class="message" id="message-{{ message.$id }}" ng-repeat="message in messages | orderByPriority | reverse">
        <div class="name">{{ message.name }}</div>
        <div class="content">
            {{ message.message }}
        </div>
    </div>
</div>

Initializing your firebase collection

The AngularFire package provides some ways to interact with Firebase. The easiest way to connecto to a collection is by using the syncData service. We can create the connection with one line of code:

angular.module('app')
  .controller('MainCtrl', function ($scope, syncData) {

    $scope.messages = syncData('messages');

  });

Add a message

Now that we set up the connection, we want to add messages to the collection. This part is very easy: you just pass a JSON object to the collection:

$scope.input = {
  message: '',
  name: ''
};

$scope.addMessage = function()
{
    $scope.messages.$add({
      name: $scope.input.name,
      message: $scope.input.message,
      timestamp: Date.now()
    });

    $scope.input.message = '';
}

Sort messages in reverse order

When you want to display the messages in reverse order you should create one extra filer called reverse. This filter is allready written in one of the example AngularFire projects. You can find the code over here.

A quick way to create a custom filter is by using the Yeoman angular:filter generator:

$ yo angular:filter reverse

Hosting

It is also possible to use Firebase hosting. A custom CLI tool is provided an can be installed through NPM:

$ npm install -g firebase-tools

The next step is to create a firebase configuration file. This can be done with the command:

$ firebase init

Note: The destination folder should be dist

The next step is to deploy the application. This is also very easy:

$ grunt build
$ firebase deploy

The sample application is now available at: Firebase App.

PHPBenelux 2014

2014-01-27T00:00:00+01:00

Last weekend, the annual PHPBenlux conference took place. With a small delegation of the Phpro crew, I went to Kontich for 2 days filled with talks, discussions and pleasure. In this article you can find a small summary of the talks that inspired me.

Day 1:

Keynote: Mentoring developers

It won't be a surprise if I told you that the conference opened with a motivational keynote. Elizabeth Marie Smith took us back to her childhood and told us why and how she started with programming. It reminded me of the time when I wrote my first super flashy website and of my first steps in PHP development. When you start learning something new, you NEED somebody to look up to. This will help you grow so much faster. In a next stadium it is your turn to give something back. Help the people around you that need your help, but DON'T spoon-feed the n00b.

What hackers do once they get passed your code

In this first session, Mattias Geniar showed us what hackers can do once they get passed your code. A presentation filled with clear examples on how hackers will try to gain and remain access to your server. Most of the time, hackers will use automated tools to find a weak point in your code. Once they have found this weak spot, they will try to compromise your website. This can be done by injecting obfuscated 'eval' scripts into your website. Or worse: Injecting a complete backdoor shell on which hackers can fully control your hosting environment.

Refactoring to Design Patters

In this talk, Benjamin Eberlei gave us a hands-on demo on refactoring. One of his first quotes was an unquestionable statement: "Neglecting Object-Oriented design leads to under-engineering, but focusing soley on Design-Pattern leads to over-engineering". Next he started with his demo, in which he made small changes to a legacy controller. By making these small changes, it was very clear were the Factory, Facade and Strategy patterns were needed. Some tips to keep in mind:

Get in control on how objects are created and centralize object creation.
Refactor step by step, otherwise you will be debugging all the time.

PHP Performance Under the hood

The last talk of the day was brought to us by Davey Shafik. It was a rather heavy subject to close the day with, but it gave a good oversight of what happens in PHP internally. Nice to see how the PHP code gets tokenized, transformed into opcodes which are interpreted into memory. What I wrote down for my day to day tasks is to make sure Zend OPcache is enabled on production environments.

Day 2:

Models and Service Layers; Hemoglobin and Hobgoblins

It was still early in the morning and apparently nobody had it's first coffee yet. Anyway, the first talk of the day, with a rather strange title, was given to us by a rather strange headed guy named Ross Tuck. He prepared some great slides for us, in which he warned us that he had an average of 1 slide per 14 seconds. The talk itself turned out to be about Domain Design Patterns. By showing some easy examples, he explained why you should be using fat model / skinny services, CQRS and Event Sourcing. This was a very enlightening talk for the project I am currently working on.

Moving Away from Legacy code with BDD

This was one of the talks I really looked forward to, given by Konstantin Kudryashov (aka everzet). As a BDD evangelist, he explained why it is better to promise less features, but more business value to a customer. This can be done with a BDD-Pipeline in which we handle some stages before starting refactoring the code:

Impact Mapping
Feature mapping
Order the tasks by role and benefit
Describe features into example scenarios
Describe, Implement, Design

The last 2 steps is where Behat and Phpspec come into play. Some really nice testing tools, to describe and validate how your application and models should behave.

Maximising Performance in ZF2

Another talk on my TODO list was this one, brought to us by Gary Hockin. In the first part he told us how he benchmarks and profiles his ZF2 applications. This was mainly with the tools Siege and Xdebug. Next he gave us some very helpful tips on how to increase performance in ZF2:

Use EdpSuperLuminal on production
Turn on Zend Opcode caching on production.
Do not place closures in config file, place them in the Module.php file.
Place popular routes in the and of your configuration. The router is a LIFO stack. Note: Take care of module order.
Use View template maps in production instead of the template_view_stack. This can be done with the default templatemap_generator.php, shipped with zend.
Minimize the use of the shared event manager.
Cache big datasets.
Add default files like favicon.ico, robots.txt, ... to make sure ZF2 does not get bootstrapped on request.

Uncon: Feature branches & PR’s

This year, there was also an Uncon going on at PHPBenelux. The first session I followed, was a small talk about PhlyBox by Pascal de Vink. At this moment, the tool doesn't do much. It is just a PoC for something that might become powerfull. It is designed to make it easier to deploy feature branches to a specific environment. I have thought about the concept myself, but don't think this is the way to go. Basically, the tool spawns new vagrant boxes with a custom branch as source. I think it would be easier to use a tool like grunt-git to clone a branch to a specific directory and create a new VirtualHost that will redirect to the branch. Then the only thing that remains a problem, is running database migrations. So, I guess there is still a lot of stuff to think through.

Uncon: Queue your work; about job queues and abstraction layers

The next part of the Uncon was a talk about Queueing and Messaging systems in PHP, brought to us by Jurian Sluiman. All the actions that users don't need to be aware of, should be done asynchronous. This way, you can deliver the content to the user in no time and let the server do the other jobs when there are resources to do the task. A good way to do this kind of asynchronous data handling, is by using queue deamons like Gearman or Beanstalkd. A Job Server will remember what tasks that need to be done and a Job Worker will handle the actions and give a response to the Task Server. When you need more control over the Messaging, you can use a Messaging Queues like ZeroMQ or RabbitMQ. This was a really interesting topic. I can't wait to add this asynchronous queues into a next project.

Extract Till You Drop

The last talk was about extracting legacy code. Mathias Verraes gave us a live demo, a bit similar to the talk of Benjamin Eberlei. It was great to watch how Mathias used all the build-in PhpStorm refactoring macro's. He also had a very interesting way in naming stuff. Exactly what you would expect from a DDD evangelist! The tests ran in PHPUnit, but had a BDD like structure. It looked like a mixture of PhpUnit, PhpSpec And Behat.

Conclusion

PHPBenlux was, again, a great conference. It's amazing to see how the sessions and socials improve every year. To all the organizers: keep up the good work! I hope to be there again next year.

Validating your PHP code on the fly

2014-01-21T00:00:00+01:00

It's amazing how much time you can loose while running PHP tests. First you alter your code or test. Next you start your test and wait for it to finish. Finally you find out you messed up and start the process all over again. In this article I am going to show you how to automate PHP testing, so you can focus on adding business value to your application.

For this article I created a simple Zend Framework 2 application with Phpspec tests. Even though this article focuses on these 2 packages, it is possible to create this kind of automation for every possible package.

Application structure

First, let's take a look at the directory structure. In the representation below, you can see a default ZF2 skeleton. The root folder contains the directories config, module, public and vendor. The `config` folder provides the application configuration. In the `module` folder you can see 2 modules that should be tested: Export and Import. Next you can find the public directory which will bootstrap the application. And finally there is the vendor folder that will be filled with dependencies by Composer.

.
|-- config
|   |-- application.config.php
|-- module
|   |-- Export
|   |   |-- config
|   |   |-- spec
|   |   |-- src
|   |   |  `-- Export
|   |   |       |-- Service
|   |   |       `-- Module.php
|   |   |-- view
|   |   `-- Module.php
|   `-- Import
|       |-- config
|       |-- spec
|       |-- src
|       |  `-- Import
|       |       |-- Service
|       |       `-- Module.php
|       |-- view
|       `-- Module.php
|-- public
|-- vendor

Configuring composer

Because nobody wants to download all the dependencies manually, we will be using composer to get the required packages. We also want to configure composer to automatically load our custom modules: Export and Import. This way, no additional autoloading is required and Phpspec will be able to find all classes by itself.

composer.json

{
    "name": "veewee/validating-on-the-fly",
    "require": {
        "zendframework/zendframework": "2.*",
    },
    "require-dev": {
        "phpspec/phpspec": "dev-master",
        "fabpot/PHP-CS-Fixer": "*",
    },
    "minimum-stability": "dev",
    "prefer-stable": true,
    "autoload": {
        "psr-0": {
          "Export": "module/Export/src/",
          "Import": "module/Import/src/"
        }
    }
}

As you can see, composer will download Zend Framework 2 to the vendor directory. When you add the option `--dev` during the installation, Phpspec and PHP-CS-Fixer will also be downloaded. The last part of this configuration file will provide autoloading for the custom Export and Import module. To download and install all the dependencies, one command is being used:

~$ composer install --dev --prefer-dist

Configuring phpspec

As you can see in the directory overview, there is already a spec folder available. The Phpspec tests will be placed in this directory. This way, every module has it's own scope and can be tested separately.

Now the tricky part: let's tell Phpspec where to find the custom source and spec files. Because we need to test multiple modules on multiple locations, we need to add multiple test suites to Phpspec. This is done by creating a phpspec.yml file in the root folder of the application. A test suite describes where the source and spec files of a custom namespace are located. Let's take a look at the configuration file for our application:

phpspec.yml

formatter.name: dot
suites:
  Export:
    namespace: Export
    src_path: 'module/Export/src/'
    spec_path: 'module/Export/'

  Import:
    namespace: Import
    src_path: 'module/Import/src/'
    spec_path: 'module/Import/'

One of the downsides of Phpspec is that, at the moment, you can not run suites based on suite name. To test this configuration, you can run following commands:

~$ ./vendor/bin/phpspec run module/Export/src
~$ ./vendor/bin/phpspec run module/Import/src

Configuring php-cs-fixer

Everybody wants clean, readable and maintainable code. That is why the code of the modules should be validated against the desired PHP standards. In this case, we want to check our classes against the PSR2 standard. Per module, you should add a file called .php_cs in the root directory of the module. In this file you can tell the Php-CS-Fixer which paths to check and add some default configuration settings.

.php_cs

<?php
$finder = Symfony\CS\Finder\DefaultFinder::create()
    ->exclude('language')
    ->exclude('view')
    ->exclude('config')
    ->in(__DIR__ . '/src')
    ->in(__DIR__ . '/spec');

$config = Symfony\CS\Config\Config::create();
$config->fixers(Symfony\CS\FixerInterface::PSR2_LEVEL);
$config->finder($finder);
return $config;

To test this configuration, you can run following commands:

~$ ./vendor/bin/php-cs-fixer fix module/Export --dry-run
~$ ./vendor/bin/php-cs-fixer fix module/Import --dry-run

Automate the testing process

Before the automation, every time you change a file in your module, you need to run Phpspec and occasionally the Php-CS-Fixer to validate your code. Because you have to define which module to check every time you run the command, you will spend some time searching for the right command in your console. A better way is to automate this process, so that a file change will trigger the command to test your code. This will save you some time and will give you an indication of when and where you messed up. What was that tool again to automate tasks? Exactly: Grunt!

There are some grunt plugins that can be used to automate PHP validation. In this article we are using Phpspec, but there are also plugins for other testing libraries like Phpunit and Behat. In the next part, I will tell you how to test your code on syntax errors, coding standard errors and bad functionality.

Grunt-Phpspec

For validating bad functionality we use grunt-phpspec. This tool will run Phpspec on our custom modules. For every PHP module we need to check, an entry in the configuration is required. Here is the sample configuration for the current project:

phpspec: {
  options: {
    prefix: './vendor/bin/'
  },
  export: {
    specs: 'module/Export/src'
  },
  import: {
    specs: 'module/Import/src'
  }
}

To run spec tests, you can use following commands:

~$ grunt phpspec
~$ grunt phpspec:export
~$ grunt phpspec:import

Grunt-php-cs-fixer

The tool for validating PHP coding standards is grunt-php-cs-fixer. As the name suggests: it does not only check for issues, it also fixes them. Therefore it must be possible to validate and fix errors with Grunt. This can be done by adding the parameter `--fixcs`. Here is the sample configuration for the current project:

// Place this parameter before grunt.initConfig() method:
var fixCs = grunt.option('fixcs') || false;

// Place this configuration in grunt.initConfig() method:
phpcsfixer: {
  options: {
    bin: 'vendor/bin/php-cs-fixer',
    verbose: true,
    dryRun: !fixCs,
    ignoreExitCode: fixCs,
    level: 'psr2',
    standard: 'Zend'
  },
  export: {
    dir: 'module/Export'
  },
  import: {
    dir: 'module/Import'
  }
}

To run the cs fixer, you can use following commands:

# Validating:
~$ grunt phpcsfixer
~$ grunt phpcsfixer:export
~$ grunt phpcsfixer:import

# Fixing:
~$ grunt phpcsfixer --fixcs
~$ grunt phpcsfixer:export --fixcs
~$ grunt phpcsfixer:import --fixcs

Grunt-Phplint

To validate PHP files for syntax errors you can use the commnd `php -l`. For automation, there is also a Grunt plugin available named grunt-phplint. This tool is added to the test suite to make sure the CS-fixer and spec tests only run when there is no syntax error. Here is the sample configuration for the current project:

phplint: {
  export: ['module/Export/**/*.php'],
  import: ['module/Import/**/*.php']
}

To run PHP lint, you can use following commands:

~$ grunt phplint
~$ grunt phplint:export
~$ grunt phplint:import

Grunt-watch

Now that we configured all tasks to validate the PHP code, we want the tests to run every time we change a class or spec test in our module. This is where grunt-contrib-watch comes in to play. This task will listen for file changes and will configured tasks for these files. When one of the tasks fail, the other tasks will not run. This is why I decided to run the tests in the order: lint, csfixer, phpspec. Here is the sample configuration for the current project:

watch: {
  export: {
    files: ['module/Export/**/*.php'],
    tasks: ['phplint:export', 'phpcsfixer:export', 'phpspec:export']
  },
  import: {
    files: ['module/Import/**/*.php'],
    tasks: ['phplint:import', 'phpcsfixer:import', 'phpspec:import']
  }
},

To start the watcher and automate the tests, you only have to use one command:

~$ grunt watch

Run all tests in one

For Continious Integration it would be nice if all tests could be run in one command. Well off course this is possible! It is just one line of configuration:

grunt.registerTask('test', ['phplint', 'phpcsfixer', 'phpspec']);

To start the complete test, you can use following command:

~$ grunt test

Conclusion

As you can see there are some tools to configure. Once the tools are configured, this set-up will save you some valuable time while testing your code. I highly recommend you to use a similar set-up because it is very easy to use yet very powerful. You won't regret!

Grand Opening!

2014-01-14T00:00:00+01:00

With a new year, come some new resolutions. I was already thinking of creating a blog for quite some time, but the new year forced me to finally get this site up and running. So why did I start this blog? Not to tell you who I am, but to inform you about some cool stuff I encounter during the development of web applications.

Just another blog you think? Ok that's true ... Except ...
Last year I found myself digging in the latest subjects of Web Development. Some of these subjects are poor documented and are not handled by many blog articles. My goal is to help you fix the issues that I encountered while using open-source modules. That, and off course to tell you more about some cool tools / projects I found.

Let's not let my first blog article go to waste and add some technical information about this blog.

Choose the right platform

The first problem while creating this blog, was deciding which platform to use. At first there were the obvious platforms like Wordpress or some well-known hosted services like Blogger. Because I rather didn't want to use these platforms, I went on a blog-platform-quest. After a while I bumped into Jekyll. This Ruby based static site generator is easy to use and is perfect for building something as simple as this blog. Extra bonus: you can deploy to github pages instead of wasting your own web space!

Off course, there were dozens of people who already worked with Jekyll before me. That is Why I searched for some good articles about building a blog for GitHub pages. One of the results I found was the "How to build a blog in one day" article by Eric Jones. This was a good place to start, but after some boring repeated tasks like the configuration of tools, I found a great tool to build this site:

Installation

By installing the jekyllrb generator for yeoman, all the work was done for me. Following items were configured:

Files structure
Grunt configuration
Compass
File watcher - which is much better than the default Jekyll watcher!
LiveReload
Jekyll configuration options
Bower: Twitter Bootstrap / jQuery / ...

All that it took to let me focus on the lay-out and content, were 2 commands:

~$ npm install -g generator-jekyllrb
~$ yo jekyllrb

Development

The real power of this generator, is the integration with grunt. Several handy tasks are pre-configured for you to use. For example:

~$ grunt serve

This command will launch the Jekyll server and will rebuild the static website on file changes. When the static site is build, the livereload task will refresh your browser, so you can validate your changes right away. Also your compass files will be automatically converted to usable CSS files.

Publishing

So, you finished your blog and want to place it on github.io? I forgot to tell you there is one little problem left. Github.io uses the root of the master branch to find your index.html file. Because yeoman creates your site in a subfolder called "app", github.io will not find your files. That is why we need to separate our development code from the final site. The distribution version will be placed on the 'master' branch and the main code will be placed on a separate branch named 'develop'. In GIT, this can be done by creating an 'orphan' branch, in this case named 'develop'. When you finished your blog, the final site should be committed to 'master' and the source code should be committed to 'develop'. I hear you say: So much work to deploy my blog? Fortunately there is a grunt tool named gh-pages which does the job. I used following configuration which will commit my 'dist' folder of the 'develop' branch to the 'master' branch in git.

'gh-pages': {
  options: {
    base: 'dist',
    branch: 'master',
    message: 'Auto-generated build',
  },
  src: '**/*'
}

grunt.registerTask('github-deploy', [
    'default',
    'gh-pages'
]);

Now, when you want to publish your site, it's just one simple command:

~$ grunt github-deploy

This will first clean up all temporary files and validate your Jekyll site, javascript and compass files. When everything is good to go, it will build the static version of your website for distribution. The files in the distribution folder will be minimized and optimized for quick access. After all actions are done, the optimized version will be pushed to your master branch, using the gh-pages tool in grunt.

Conclusion

With these easy to use, but incredibly powerful tools, I hope to focus myself on writing articles about some interesting topics. If you want to take a look at my configuration or want to re-use it for your own project, feel free to fork veewee.github.io from my GitHub account.