Tokenizer: String Tokenization

Tokenizer is a simple tool that uses regular expressions to split a given string into tokens. This library is no longer being developed.

Installation:

composer require nette/tokenizer

String Tokenization

Let's create a simple tokenizer that separates strings into numbers, whitespaces, and letters.

$tokenizer = new Nette\Tokenizer\Tokenizer([
	T_DNUMBER => '\d+',
	T_WHITESPACE => '\s+',
	T_STRING => '\w+',
]);

The constructor optionally accepts a second argument $flags with regular expression modifiers, such as 'u' for UTF-8 mode.

Hint: In case you are wondering where the T_ constants come from, they are internal types used for parsing code. They cover most of the common token names we usually need. Keep in mind their value is not guaranteed so don't use numbers for comparison.

Now when we give it a string, it will return a stream of tokens.

$stream = $tokenizer->tokenize("say \n123");

The resulting array of tokens $stream->tokens would look like this.

[
	new Token('say', T_STRING, 0),
	new Token(" \n", T_WHITESPACE, 3),
	new Token('123', T_DNUMBER, 5),
]

Also, you can access the individual properties of the token:

$firstToken = $stream->tokens[0];
$firstToken->value; // say
$firstToken->type; // value of T_STRING
$firstToken->offset; // position in string: 0

Simple, isn't it?

Processing the Tokens

Now we know how to create tokens from a string. Let's effectively process them using Stream. It has a lot of really awesome methods if you need to traverse tokens!

Let's try to parse a simple annotation from PHPDoc and create an object from it. What regular expressions do we need for tokens? All the annotations start with @, then there is a name, whitespace and its value.

• @ for the annotation start
• \s+ for whitespaces
• \w+ for strings

Never use capturing subpatterns in Tokenizer's regular expressions like '(ab)+c', use only non-capturing ones '(?:ab)+c'.

This should work on simple annotations, right? Now let's show the input string that we will try to parse.

$input = '
	@author David Grudl
	@package Nette
';

Let's create a Parser class that will accept the string and return an array of pairs [name, value]. It will be very naive and simple.

use Nette\Tokenizer\Tokenizer;
use Nette\Tokenizer\Stream;

class Parser
{
	const T_AT = 1;
	const T_WHITESPACE = 2;
	const T_STRING = 3;

	private Tokenizer $tokenizer;
	private Stream $stream;

	public function __construct()
	{
		$this->tokenizer = new Tokenizer([
			self::T_AT => '@',
			self::T_WHITESPACE => '\s+',
			self::T_STRING => '\w+',
		]);
	}

	public function parse(string $input): array
	{
		$this->stream = $this->tokenizer->tokenize($input);

		$result = [];
		while ($this->stream->nextToken()) {
			if ($this->stream->isCurrent(self::T_AT)) {
				$result[] = $this->parseAnnotation();
			}
		}

		return $result;
	}

	private function parseAnnotation(): array
	{
		$name = $this->stream->joinUntil(self::T_WHITESPACE);
		$this->stream->nextUntil(self::T_STRING);
		$content = $this->stream->joinUntil(self::T_AT);

		return [$name, trim($content)];
	}
}

$parser = new Parser;
$annotations = $parser->parse($input);

So what does the parse() method do? It iterates over the tokens and searches for @ which is the symbol annotations start with. Calling nextToken() moves the cursor to the next token. The isCurrent() method checks if the current token at the cursor is of the given type. Then, if the @ is found, the parse() method calls parseAnnotation() which expects the annotations to be in a very specific format.

First, using the method joinUntil(), the stream keeps moving the cursor and appending the values of the tokens to the buffer until it finds a token of the required type, then stops and returns the buffer output. Because there is only one token of type T_STRING at that given position and it's 'author', there will be the value 'author' in the variable $name.

The nextUntil() method is similar to joinUntil() but it has no buffer. It only moves the cursor until it finds the token. So this call simply skips all the whitespaces after the annotation name.

And then, there is another joinUntil(), that searches for the next @. This specific call will return "David Grudl\n ".

And there we go, we've parsed one whole annotation! The $content probably ends with whitespaces, so we have to trim it. Now we can return this specific annotation as pair [$name, $content].

Try copypasting the code and running it. If you dump the $annotations variable it should return some similar output.

array (2)
   0 => array (2)
   |  0 => 'author'
   |  1 => 'David Grudl'
   1 => array (2)
   |  0 => 'package'
   |  1 => 'Nette'

Stream Methods

The stream can return the current token using the currentToken() method, or only its value using currentValue().

nextToken() moves the cursor and returns the token. If you give it no arguments, it simply returns the next token.

nextValue() is just like nextToken() but it only returns the token value.

consumeToken() works like nextToken(), but if it doesn't find a matching token it throws an Exception instead of returning null. consumeValue() is its counterpart returning the token value.

Most of the methods also accept multiple arguments so you can search for multiple types at once.

// iterate until a string or a whitespace is found, then return the following token
$token = $stream->nextToken(T_STRING, T_WHITESPACE);

// give me next token
$token = $stream->nextToken();

You can also search by the token value.

// move the cursor until you find token containing only '@', then stop and return it
$token = $stream->nextToken('@');

nextUntil() moves the cursor and returns an array of all the tokens it sees until it finds the desired token, but it stops before the token. It can accept multiple arguments.

joinUntil() is similar to nextUntil(), but concatenates all the tokens it passed through and returns a string.

joinAll() simply concatenates all the remaining token values and returns them. It moves the cursor to the end of the token stream.

nextAll() is just like joinAll(), but it returns an array of the tokens.

isCurrent() checks if the current token or the current token's value is equal to one of the given arguments.

// is the current token '@' or type of T_AT?
$stream->isCurrent(T_AT, '@');

isNext() is just like isCurrent() but it checks the next token.

isPrev() is just like isCurrent() but it checks the previous token.

If you set the $ignored property to an array of token types, the search methods will skip over such tokens instead of stopping on them.

And the last method reset() resets the cursor, so you can iterate the token stream again.