This release has a few new (minor) features and a couple of bug fixes. The most important change is not in code, but in the license. With this release, I will no longer offer paid licenses. Instead, the code is released under the Apache Public License 2.0, and free for use in closed, commercial applications.

I have never worked full-time on upscaledb (and its predecessor hamsterdb), but the amount of time I have invested was often more than 40 hours per week. Over the last year I realized that I do no longer have the resources and the grit to do this. Spending time with the family is just way more valuable.

Still, it was a difficult decision. For many years I had the vision to write database code for a living. Working on my own database, my own project, felt like a dream coming true. Commercially, the dream turned out to be not sustainable. It did not feel good to see that others had the same problems, but it helped nevertheless.

What now? I will continue working on upscaledb. I will invest my limited time where it makes most sense (for me). Things that I do not enjoy or that consume too much time will be neglected. That's mostly related to providing Win32 builds (if you look at the download page then you will see that they are missing). Bugs will be fixed as soon as possible, patches are more than welcome.

From the release notes:

New Features

• Added a new API function for bulk operations (ups_db_bulk_operations in ups/upscaledb_int.h)

Bugfixes

• Fixed compiler error related to inline assembly on gcc 4.8.x
• Fixed bug when ups_cursor_overwrite would overwrite a transactional record instead of the (correct) btree record
• Fixed several bugs in the duplicate key consolidation
• issue #80: fixed streamvbyte compilation for c++11
• issue #79: fixed crc32 failure when reusing deleted blobs spanning multiple pages
• Fixed a bug when recovering duplicates that were inserted with one of the UPS_DUPLICATE_INSERT_* flags
• Minor improvements for the journalling performance
• Fixed compilation issues w/ gcc 6.2.1 (Thanks, Roel Brook)

Other Changes

• License is now Apache License 2.0
• Performance improvements when appending keys at the end of the database
• The flags UPS_HINT_APPEND and UPS_HINT_PREPEND are now deprecated
• Removed the libuv dependency; switched to boost::asio instead
• Performance improvements when using many duplicate keys (with a duplicate table spanning multiple pages)
• Committed transactions are now batched before they are flushed to disk
• The integer compression codecs UPS_COMPRESSOR_UINT32_GROUPVARINT and UPS_COMPRESSOR_UINT32_STREAMVBYTE are now deprecated
• The integer compression codec UPS_COMPRESSOR_UINT32_MASKEDVBYTE is now a synonym for UPS_COMPRESSOR_UINT32_VARBYTE, but uses the MaskedVbyte library under the hood.
• Added Mingw compiler support (thanks, topilski)

All files are available on the download page.

After several months of hard work, I'm proud to release the first (beta) version of upscaledb-mysql, a MySQL storage engine based on upscaledb.

MySQL uses "storage engines" to persist and access the actual table data. The most prominent storage engines are InnoDB and MyISAM. The upscaledb storage engine tries to be 100% compatible to InnoDB, while offering higher performance and an even faster NoSQL interface to the table data. If MySQL hits a performance wall you can use the regular upscaledb API, connect (via the remote interface) to the MySQL process and directly query or modify the data.

Performance

I have tested performance with several benchmark programs.

Sysbench 1.0

Sysbench’s OLAP test uses transactions which are not yet implemented [Scroll down for the TODO list]. To avoid race conditions, they were run with one client connection only. The tests were run for 60 seconds, the test data consists of 1 million rows. Tests were run with BEGIN/COMMIT transaction envelopes and with LOCK TABLE. The graphs show the total number of queries.

For such simple operations %%upscledb outperforms InnoDB.

Benchmark results of sysbench 1.0 with and without transactions (1 mio rows)

Wordpress 4.3

I have run two benchmarks using siege, a http benchmark tool. The first one retrieves the start page of a Wordpress blog with five posts and several comments. The second benchmark searches the blog by accessing the “Search” page. Both tests were run for 30 seconds, with 30 concurrent connections.

The results are slightly in favour of InnoDB. Upscaledb leaves some performance on the table, mostly because it requires two database accesses when reading or writing a secondary index. In addition, InnoDB implements a MySQL feature called “Index Condition Pushdown”, which optimizes range lookups via secondary index. Both optimizations are on the TODO list.

Benchmark results of wordpress 4.3

Remote NoSQL interface

Upscaledb does not yet optimize the COUNT(*) queries, therefore InnoDB is currently much faster. But upscaledb’s built-in query interface (“UQI”) is still the winner. This query interface can be extended through plugins, and any type of full-table scan can thus be implemented. In addition, you can use all other upscaledb API functions to read or manipulate the table data. The graphs show the duration of the SELECT COUNT(*) query (in seconds).

Benchmark results for a COUNT(*) query over 6mio rows

You can find detailled information about the benchmarks in a GitHub Gist.

Current status

The current version (0.0.1) is not yet ready for production. I'm not aware of critical bugs, but a few things are still on my TODO list (in no particular order):

• Support transactions (BEGIN/ABORT/COMMIT)
• Optimize count(*) and make it as fast as in InnoDB
• Make secondary indices a first-class citizen; currently, the index table stores the primary key, instead of directly referring to the blob with the row’s data
• Support Integer compression for indices
• Several internal storage engine functions are not yet provided
• Duplicate key performance (for non-unique indices) does not scale well
• Improve concurrency; upscaledb is single-threaded
• The MySQL feature "Index Condition Pushdown" is not yet supported
• The generated files sometimes are slightly larger than those of InnoDB; this needs to improve
• Charsets which are case-insensitive are not fully supported
• The installation procedure is very complex (If you can help w/ packaging please send me an email)

Installation and configuration

Both are described in the Wiki: Installation, Configuration.

Found a bug?

File an issue at the GitHub issue tracker. Please provide the following information:

• The versions of MySQL and upscaledb-mysql you are using
• The sequence of SQL commands which can be used to reproduce the bug
• The expected vs. the actual outcome

Frequently Asked Questions

Q: Is this ready for production?
A: No, unless your definition of "ready for production" is significantly different from mine.

Q: How can I improve the performance?
A: You can increase the cache size and tune various other performance options. See this page for documentation. Also, keep in mind:

• Small keys are faster than large keys
• Fixed-length keys are faster than variable-length keys
• Unique indices are faster than non-unique indices

Q: Does upscaledb use the same cache size as InnoDB?
A: Like InnoDB, upscaledb's default cache size is 128mb. However, this cache size is *per table*, and not set globally! See this page for details, and make sure you keep this in mind when comparing the performance of InnoDB and upscaledb.

Q: Is the NoSQL API security critical?
A: Yes, you basically open a backdoor to your data and avoid all of MySQL's security layers and authentication features. By default, this interface is therefore switched off.

Q: How can I use the NoSQL API to access the various columns?
A: You have to de-serialize the keys and records that are stored. I'll cover this in another blog post.

Q: Does the upscaledb storage handler support compression?
A: Yes, you can compress the row data with zlib, snappy or lzf. See this page for more information.

Q: Does the upscaledb storage handler run with MariaDB?
A: I don’t know, I did not yet have time to test. But I would expect so.

Follow me on Twitter!

This is the third (and last) summary of a research paper by Daniel Lemire and me about the use of 32bit integer compression for keys in a B+-tree index. We evaluate a wide range of compression schemes, some of which leverage the SIMD instructions available in modern processors. To our knowledge, such a comparison has never been attempted before.

If you missed the previous blogs, you can find them here and here.

upscaledb is a parameterized B+tree, and the memory layout and algorithms of the B+tree nodes differ depending on the parameters the user chose when the database was created. If integer compression is enabled, then a specific "Compressed Integer Blocks" implementation is selected, which is further parameterized for each of the supported codecs.

However, adding integer compression (or ANY key compression) does not just affect the B+tree nodes. Other more generic algorithms (those that are independent from the parameterization) also have to be changed, e.g. how you insert or delete keys from the B+tree. In this article I want to describe those changes and how upscaledb's B+tree differs from the textbook algorithm.

When we started adding key compression to upscaledb, we encountered a few major problems that we had to solve.

Leaf node capacity as storage space

A textbook B+tree (or Btree, or any other derivate) defines its capacity as the number of keys that a node can store. Usually, a constant "k" is defined, and a node must store between k and 2k-1 items (otherwise it's underfilled and will be merged, or it overflows and will be split). (See Wikipedia for more information.)

But as soon as you add compression, there is no more "k" that you can use to describe the capacity. If you have a set of compressed keys, and you add a new one, you do not know in advance whether the key will fit into the available space or not. This depends on the codec, and on the "compressability" of the keys. When inserting a new key in a node with compressed keys, we found it therefore best to TRY the insertion. The codec would then decide whether the currently available space would be sufficient to store the new key, or not. If not, then the caller will attempt to reorganize the node in order to free up space. Only if this is impossible then the node is split.

This affects not just the B+tree node, but also the generic "insert" function (which is independent from the actual node format). It will simply attempt to insert the key, and split the node if insertion fails (and not if an arbitrary 2k-1 limit is reached).

Deleting keys can grow the storage

If you recall the codecs from part 1, we implemented one codec which is not "delete stable": BP128 (SimdComp). It compresses integers by bit-packing their deltas. Imagine the following sequence: 1, 2, 3, 4, 5, 6, 7, 8 (this is typical for an auto-incremented primary key). After encoding, all deltas can be stored in a single byte with all bits set (11111111). Now delete 4 and the delta of 3 and 5 will jump to 2. Each delta now requires 2 bits, and our storage grows to two bytes (the last two bits are unused): 01010110 01010100. In effect, our storage doubled although we deleted a key!

As a consequence, a B+tree node can run out of space if a key is deleted, and will require a node split.

This is such a weird scenario and so contrary to the textbook implementation that our first reaction was to avoid all codecs which behaved this way. Indeed, other researchers (like IBM for DB2) also had "delete stability" as a codec requirement. But after a while we discovered that it's possible to handle such cases - and changing the generic delete algorithm was fairly simple because it was very similar to the insert algorithm (see below).

Local balancing

Even before adding compression, upscaledb did not always follow the standard B+tree algorithm. Its insertion split the nodes on the way "down", when descending the tree, and not on the way "up", when returning from the recursion. This approach was first described by Guibas and Sedgewick. It's a much simpler approach. And deleting keys works similar: nodes are merged when "going down", not when going back up. Keys are never shifted. Underfilled nodes are accepted, and nodes are only merged when they are nearly empty, and if merging the node only has a local effect. If a merge would have to be propagated to more levels than just the parent, then the node will be left empty. This sounds wasteful, but is neglectable in practice.

The insert and delete operations are now very similar. In both cases the tree is descended, and internal nodes are merged or split if necessary. Only when the leaf is reached the operations diverge, and the new key is either inserted, or the old key is deleted.

If a split is necessary, a new node is allocated and the parent is updated. The split is never propagated because the parent node definitely has space for one more key (we made sure of that when descending the tree). The split only has a local effect.

Conclusion

It was definitely worth investing additional time to support BP128 (SimdComp) as a codec. BP128 offers the best compression ratio, especially for dense key ranges like auto-incremented primary keys or dense time stamps. Especially for analytical queries with upscaledb's query interface UQI.

Follow me on Twitter and get immediately notified when there are new announcements (like this one)!

This blog post continues with the summary of a research paper by Daniel Lemire and me about the use of 32bit integer compression for keys in a B+-tree index. We evaluate a wide range of compression schemes, some of which leverage the SIMD instructions available in modern processors. To our knowledge, such a comparison has never been attempted before.

You can read part one here. This part describes the integration of integer compression in upscaledb and presents benchmark results.

The upscaledb B+-tree

upscaledb’s B+-tree node stores keys and values separately from each other. Each node has a header structure of 32 bytes containing flags, a key counter, pointers to the left and right siblings and to the child node. This header is followed by the KeyList (where we store the key data) and the RecordList (where we store the value’s data). Their layout depends on the index configuration and data types.

The RecordList of an internal node stores 64-bit pointers to child nodes, whereas the RecordList of a leaf node stores values or 64-bit pointers to external blobs if the values are too large.

Fixed-length keys are always stored sequentially and without overhead. They are implemented as plain C arrays of the key type, i.e., uint32 t keys[] for a database of 32-bit integers. Variable-length keys use a small in-node index to manage the keys. Long keys are stored in separate blobs; the B+-tree node then points to this blob.

Memory layout for variable-length (top) and fixed-length keys (bottom)

This in-memory representation of fixed-length keys makes applying compression easy - at least in theory. The keys are already stored in sorted order, therefore applying differential encoding does not require a change in the memory layout. Since keys are stored sequentially in memory, SIMD instructions can be used efficiently if needed.

When using integer compression, the keys are split into blocks of up to 256 integers per block. We chose this to offer faster random access (by skipping blocks). Also, inserting or deleting values will only affect a small block instead of a huge one.

Codec requirements

I already described the various codecs in the previous part. In order to integrate a codec into upscaledb, it had to implement a few functions.

uncompress block

A full block is not decoded during CRUD (Create/Read/Update/Delete) operations, but it is decoded frequently during analytical queries (see below).

compress block

Used i.e. after a page split or in the "vacuumize" operation, which periodically compacts a node in order to save space (see below).

insert

Inserts a new integer value in a compressed block. A few codecs (Varbyte, MaskedVByte and VarIntGB) can do this without uncompressing the block. All other codecs uncompress the block, modify it and then re-compress the block.

append

Appending an index key at the “end” of the database is a very common operation, therefore we optimized it. Most codecs are now able to append extremely fast with O(1), and without decoding the whole block. Appending an integer key to a compressed block would otherwise be costly, because all codecs (except FOR and SIMD FOR) have to reconstruct all values because of the differential encoding.

search

Performs a lower-bound search in a compressed block, without decompressing it.

delete

Not a priority since we did not use this in our benchmark. Implemented for Varbyte (and therefore also for MaskedVByte) directly on the compressed data. All other codecs decode the block, delete the value and then re-encode the block again.

vacuumize

Performs a compaction by removing gaps between blocks. Gaps occur when keys are deleted and blocks therefore shrink, or blocks require more space and are moved to a different location in the node. SIMD FOR and BP128 periodically also decode all keys and re-encode them again.

(select)

Was originally used for cursor operations. But then we found out that cursors often access the same block. We decided to decode and cache the last used block. The select function is therefore no longer used.

Benchmarks

All our experiments are executed on an Intel Core i7-4770 CPU (Haswell) with 32 GB memory (DDR3-1600 with double-channel). The CPU has 4 cores of 3.40 GHz each, and 8 MB of L3 cache. Turbo Boost is disabled on the test machine, and the processor is set to run at its highest clock speed. The computer runs Linux Ubuntu 14.04. We report wall-clock time.

Insert

This benchmark creates a new database for 32bit integer keys and inserts various numbers of keys. We should expect a compressed database to be slower for such applications, as insertions may require complete recompression of some blocks, at least in the worst case. Among the compressed formats, the best insertion performance is offered by the FOR, SIMD FOR and Masked VByte codecs, followed by BP128 and VarIntGB. VByte is slower than all other codecs. If one uses FOR, SIMD FOR and Masked VByte, insertions in a compressed database are only 2.5× slower than insertions in an uncompressed database.

Relative insert timings

Look-ups

This benchmark opens an existing database and performs point look-ups of all inserted keys. Each look-up requires a B+-tree traversal to the leaf node. The node then performs a linear search through the block index and locates the block which stores the requested key. The codec then searches the block for the key.

The benchmarks show that integer compression does not cause a significant performance hit for lookup operations. We get the best results with SIMD FOR, VarIntGB and FOR, and the worst results with VByte and BP128 (with a penalty of up to 60 %). Actually this was a positive surprise; we expected worse!

Relative look-up timings

Cursors

This benchmark opens an existing database and creates a cursor to traverse from the first to the last key. To position itself on the first key, the cursor traverses down the B+-tree at the left-most path down to the leaf, then visits each leaf. Since all leaf nodes are linked, no further traversal is required.

In our original implementation, the cursor then used the select method to retrieve the key directly from the compressed block. But since cursors are usually used for sequential access, and therefore frequently access the same block, we decided to decode the block and cache the decoded values. Our tests showed a significant performance improvement compared to the previous implementation based on select.

Relative cursor timings

Sum

This benchmark performs a “SUM” operation on all keys. It is equivalent to a SELECT SUM(column) operation of a SQL database, where the specified column is an index of unique 32bit integer keys. For such operations, upscaledb does not use a cursor to traverse the B+-tree, but performs the operation directly on the B+-tree’s data, without copying the keys into the application’s memory. SUM performance is impacted by database size: the bigger the database, the more compression is beneficial, with BP128 and SIMD FOR offering the best performance.

Relative SUM timings

File sizes

Finally, here are the compression results with the various database sizes, using the default block sizes (128 for BP128 and 256 for other codecs). BP128 is the clear winner. It is able to compress the database by a factor of ten compared to an uncompressed B+-tree. The compression ratios offered by the other codecs are similar (compression ratio of 2 or 3), with SIMD FOR compressing slightly less and VByte compressing slightly better.

Relative database sizes

Conclusion

We have shown that fast key compression could improve the single-threaded performance of a key-value store - if these compression techniques are accelerated with SIMD instructions. We get the best performance for SIMD codecs (BP128 and SIMD FOR). Unlike other codecs, they show an ability to noticeably improve query performance in all our tests (from small to large databases) on the analytic “SUM” benchmark.

Another conclusion would be: if your application mostly appends keys (instead of inserting them at random positions), and you perform many analytical queries or full-table scans, then performance will improve (in the best case) or at least not deteriorate significantly (in the worst case). But you will save lots of RAM.

One of our codecs, BP128 ("SimdComp"), comes with an interesting limitation: it is not "delete stable" and often grows when keys are deleted. To our knowledge, upscaledb is the only database which is able to deal with such codecs. In the next blog I'll describe how we did this, a few other problems we faced when we integrated compression, and how upscaledb's B+-tree differs from the textbook B-tree algorithm. Sign up to our newsletter to get notified when the next blog is published.

Follow me on Twitter!

This blog post shows how to use the new "UQI" query interface of upscaledb 2.2.0. It's the follow-up to the series "Building an embedded column store database with upscaledb" (part one, part two and part three).

In the previous parts I demonstrated how to use upscaledb to create multiple indexes on one or several columns, and how to perform range queries with cursors. This part will show how to run queries with the UQI query interface.

More analytical functions: "UQI"

upscaledb has a lot to offer regarding analytics: it has a full set of APIs dedicated to analytical calculations. These APIs are called "UQI" (Upscaledb Query Interface) and declared in the header file `ups/upscaledb_uqi.h`. These functions were rewritten for version 2.2.0 and now support an SQL-ish query language which can be extended through custom plugins.

This query interface can run full-table scans over a single database and apply aggregation functions and filter predicates. Functions and predicates can be extended through user-supplied plugins. Joins are not supported.

Internally, the UQI functions can operate directly on the B+-tree node data. B+-trees store keys and records very compact and efficiently. While you could also use cursors to implement this functionality, a cursor requires the key- and record data to be copied from the B+-tree to the application. The new UQI interface does the opposite: it performs the query in the B+-tree and does not copy any data. This makes the UQI queries much faster.

Here is a list of the most important APIs:

uqi_select

Performs a "select" query. The query syntax is described below. Results are returned in result. This is a shortcut for uqi_select_range.

UPS_EXPORT ups_status_t UPS_CALLCONV
uqi_select(ups_env_t *env, const char *query, uqi_result_t **result);

uqi_select_range

Performs a "select" query. If begin is not NULL then the query starts at the specified position, otherwise it starts at the beginning of the database. Afterwards, the begin cursor is advanced to the next element after end. If end is not NULL then the query stops at the specified position, otherwise it stops at the end of the database. This API can be used for pagination.

UPS_EXPORT ups_status_t UPS_CALLCONV
uqi_select_range(ups_env_t *env, const char *query, ups_cursor_t *begin,
                            const ups_cursor_t *end, uqi_result_t **result);

uqi_result_get_row_count

Returns the number of rows from a result object.

UPS_EXPORT uint32_t UPS_CALLCONV
uqi_result_get_row_count(uqi_result_t *result);

uqi_result_get_key_type

Returns the key type from a result object. Key types are UPS_TYPE_UINT8, UPS_TYPE_UINT32, UPS_TYPE_BINARY etc.

UPS_EXPORT uint32_t UPS_CALLCONV
uqi_result_get_key_type(uqi_result_t *result);

uqi_result_get_record_type

Returns the record type from a result object. Record types are UPS_TYPE_UINT8, UPS_TYPE_UINT32, UPS_TYPE_BINARY etc.

UPS_EXPORT uint32_t UPS_CALLCONV
uqi_result_get_record_type(uqi_result_t *result);

uqi_result_get_key

Returns a key from the specified row of an result object. I.e. uqi_result_get_key(&result, 0, &key); returns the first key from the query results.

UPS_EXPORT void UPS_CALLCONV
uqi_result_get_key(uqi_result_t *result, uint32_t row, ups_key_t *key);

uqi_result_get_record

Returns a record from the specified row of an result object. I.e. uqi_result_get_record(&result, 0, &record); returns the first record from the query results.

UPS_EXPORT void UPS_CALLCONV
uqi_result_get_record(uqi_result_t *result, uint32_t row, ups_record_t *record);

uqi_result_get_key_data

Returns the serialized key data. If the key type has fixed length (i.e. UPS_TYPE_UINT32) then the returned pointer can be casted directly into a uint32_t * pointer. The size returns the size of the integer array.

UPS_EXPORT void *UPS_CALLCONV
uqi_result_get_key_data(uqi_result_t *result, uint32_t *size);

uqi_result_get_record_data

Returns the serialized record data. If the record type has fixed length (i.e. UPS_TYPE_UINT32) then the returned pointer can be casted directly into a uint32_t * pointer. The size returns the size of the integer array.

UPS_EXPORT void *UPS_CALLCONV
uqi_result_get_record_data(uqi_result_t *result, uint32_t *size);

UQI examples

The query format is simple and mostly self-explanatory. Key words are not case sensitive. Here are a few examples:

Counting things

SUM($key) from database 1

count($key) from database 13

distinct count($key) from database 13

distinct count($key) from database 13 where even($record)

Summing up

Note that you can only SUM up records (or keys) if their type is numeric!

SUM($key) from database 1

SUM($record) from database 1

TOP-k and BOTTOM-k

The TOP and BOTTOM queries are the only queries which accept the LIMIT keyword. If you specify a LIMIT for any other function the the UQI parser will currently fail with an error. You can also query TOP/BOTTOM queries over the keys, but since keys are already sorted it would be much cheaper to use a cursor instead (with the BOTTOM elements starting at the beginning of the database, and the TOP elements at the end).

TOP($record) from database 1 LIMIT 20

BOTTOM($record) from database 1 LIMIT 10

Min and Max

Again you can calculate the minimum and maximum from keys as well (by using MIN($keys) from database 1), but since keys are already sorted it would be much more efficient to use a cursor and retrieve the first (or last) element in the database.

MIN($record) from database 1

MAX($record) from database 1

Using predicates

A predicate can be supplied with the WHERE clause. In release 2.2.0 there are no predicates supplied, but they can be written as a plugin. A predicate can also be loaded from an external (dynamic) library.

distinct count($key) from database 13 where "even@plugin.so"($record)

The "distinct" keyword ignores duplicate keys and skips them when processing the input data. Its meaning is therefore different from DISTINCT in an SQL database, where it filters duplicate results.

Creating custom aggregation and predicate functions

For performance reasons, plugins have to be written in C or C++. With some knowledge in C/C++, writing a plugin is not difficult, but it extends the scope of this blog post a bit. We've created a sample which demonstrates how an application can supply its own plugins for aggregating and filtering data: uqi2.c.

The UQI interface is brand new, and fairly unique. If you have any questions then please do not hesitate to write!

The new release comes with significant changes!

A new query interface (uqi_select and uqi_select_range) with an SQL-ish query language. The query engine can be extended through custom plugins, which can also be loaded from an external library. You can therefore run fast queries even when using the remote API or wrapper APIs like java, .NET, Python or Erlang.

Example queries are:

Just like the keys, records can now also be "typed", i.e. for uint32, uint64 and all other currently supported types.

In addition there were a few (mostly minor) changes in the API and bug fixes.

All files are available on the download page.

This blog post is a summary of a research paper by Daniel Lemire and me about the use of 32bit integer compression for keys in a B+-tree index. We evaluate a wide range of compression schemes, some of which leverage the SIMD instructions available in modern processors. To our knowledge, such a comparison has never been attempted before.

The paper was submitted to a journal and is currently in review. We started writing in August 2014 and finished the first version in December 2015.

Our paper evaluated implementations of the most common codecs. First we showed micro benchmarks of the codecs, then added them to upscaledb to compress the keys. In upscaledb, a database key is "typed". Integer keys are stored in a plain integer array (uint32_t[]). The keys can therefore be optimally compressed, and it's even possible to use SIMD instructions.

Since the keys are sorted, most codecs also use differential encoding. Instead of storing the original values, only the differences between the values is stored. The sequence N1, N2, N3, N4, ... Nn is then stored as N1, N2 - N1, N3 - N2, N4 - N3, ... Nn - Nn-1. Differential encoding stores smaller values and therefore has more efficient compression.

We implemented additional operations directly on the compressed data, without decompression. Such operations include lower-bound searches and selecting values at a specific position; some codecs even can insert values without decompressing the sequence.

This is the first blog post, focussing only on the codecs. Another blog post will follow where I will present the results from the upscaledb integration.

The Codecs

VByte

VByte (or var-byte, varint, variable-byte) is one of the most popular and established integer compression techniques. It is used, for example, in Google's Protocol Buffer and Apache Lucene. Small numbers (< 127) are stored in a single byte. If the number is larger, a "continuation bit" is set, and an additional byte is used. Here are a few examples:

Decimal     Binary                Variable Byte encoded
    127     00000000 01111111     01111111
    128     00000000 10000000     10000000 00000000
  16384     01000000 00000000     10000000 10000000 00000001
                

Our VByte implementation uses standard C code. It is very efficient if many integers fit in a single byte, and since we use delta compression this is often the case. One can then decode over a billion integers per second on commodity superscalar desktop processors. However, the performance can be lower when integers fit in various numbers of bytes, as the cost of branch mispredictions increases.

VByte is a very simple format, and it is possible to implement fast select and sequential searches without first decompressing all integers. It is even possible to insert keys directly into the compressed data.

VarIntGB (or Group Varint)

To overcome the performance problems of VByte, Google created the VarIntGB format (here is Jeff Dean's presentation). It can encode and decode blocks of four integers instead of one integer at a time. It works by storing the sizes of up to four integers in a single byte (two bits per size).

01-00-00-01 | 0x0004 | 0x0c | 0x0a | 0x0200
Encoded bytes corresponding to integers 1024, 12, 10, 512.
The result occupies seven bytes.  
                

Like VByte, implementing a fast select or a fast sequential search over VarIntGB is straightforward. However, a fast insertion is more difficult. When inserting a value at index i, the previous values do not have to be rewritten. But all the remaining values have to be recoded. We found it more appropriate to decompress the remaining values, and then recompress them with the new value added.

Masked VByte

VByte decoders process one input byte at a time. Masked VByte is a very fast SIMD-based decoder which is able to decode many integers at a time. The Masked VByte decoder first collects all the most significant bits using the ‘pmovmskb’ instruction. These bits are then used as an index in a precomputed lookup table, which stores masks for the ‘pshufb’ instruction. pshufb permutes the bytes of a register. Finally, we need to mask or shift the permuted bytes to arrive at the decoded integers.

Masked VByte can be three times faster than a conventional VByte decoder when the data is sufficiently compressible. Masked VByte also benefits from an integrated SIMD-accelerated differential coding.

We implemented accelerated select and sequential search functions that are similar to the VByte functions. Selection is SIMD-accelerated: we decode in registers the first 4 i/4 values and return the value at the proper index. Sequential search is handled similarly. For the insertion, we use the same function as for VByte: this is possible because the underlying data format is identical.

Binary Packing (BP128/SimdComp)

Binary Packing fetches the maximum m of an integer sequence and then stores all integers of this sequence in log2(m+1) bits (in plain English: in as many bits as required to store the maximum value). If differential coding is used, the maximum can be relatively low, and sometimes only a few bits are required.

Our BP128 implementation uses SIMD instructions to efficiently decode a block of 128 integers. It uses differential coding, and we implemented a fast select function and a sequential search function. Insertions require decoding the entire array.

Frame of Reference

Frame of Reference (or FOR) is the only technique listed here that does not use differential coding. The downside of differential coding is that it is difficult to skip to a random index, because each value has to be reconstructed from the previous delta. FOR offers good compression and has fast random access. Its select function is very fast, and we implemented binary searches which are faster than the linear searches of the other codecs.

In FOR, the minimum value of an integer sequence is used as a reference for all other values. The sequence N1, N2, N3, N4 is encoded as N1, N2 - N1, N3 - N1, N4 - N1 etc. To decode these values, we need the minimum and then compute the sums. Since our arrays are sorted, the minimum is always at the beginning.

We evaluated two FOR libraries: the scalar libfor implementation, and SIMD-FOR, using SIMD instructions similar to BP128. Both are not compatible. Both implement select and search functions without decompressing the data. Inserting a new value can be done by uncompressing the block, inserting the value in the uncompressed data and then recompressing it.

Micro Benchmarks

Our micro benchmarks were run without database interaction. We compile our C++ benchmarking software using the GNU GCC 4.8.2 compiler with the -O3 flag. Our implementation is freely available under an open-source license.

All our experiments are executed on an Intel Core i7-4770 CPU (Haswell) with 32 GB memory (DDR3-1600 with double-channel). The CPU has 4 cores of 3.40 GHz each, and 8 MB of L3 cache. Turbo Boost is disabled on the test machine, and the processor is set to run at its highest clock speed. The computer runs Linux Ubuntu 14.04. We report wall-clock time.

Compression ratios

Given a bit width b ≤ 24, we first generate an array of 256 sorted 32-bit integers. We see that BP128 offers the best compression whereas FOR and SIMD-FOR offer poorer compression compared to other codecs.

Bits per integer - compression ratios

Decompression

Decompression speed is plotted in billions of integers per second. SIMD-FOR is twice as fast as the next scheme, BP128, which is itself much faster than most other alternatives (up to twice as fast). VByte is the only codec that is limited by a best speed of only about 1 billion integers per second. In contrast, SIMD-FOR can decompress data at a rate of over seven billion integers per second – or about two integers decoded per clock cycle.

Decompression speed

Search

We benchmark the search function by randomly seeking a value in range. VByte is significantly slower than the other codecs. VarIntGB offers the best performance in this case. FOR and SIMD-FOR differ from the other schemes in that they use a binary search (as a sequential search proved slower) whereas all other codecs shown here rely on a sequential search. If the block size was much larger, FOR and SIMD-FOR could be expected to perform better, but we are not interested in that case.

Search speed

Select

We randomly select the value at one of the indexes. We present the data in millions of operations per second with a logarithmic scale. We see that FOR and SIMD-FOR are an order of magnitude faster at this task because they do not rely on differential coding. BP128 is the next fastest codec while VByte is the slowest.

Select speed

Conclusion

Overall, our results suggest that on speed and compression ratios, BP128 offers good performance. If faster random access is necessary, and the compression ratio is not an issue, then FOR and SIMD-FOR might be preferable.

In part two I will write about integrating key compression in the upscaledb B+-tree, describe the challenges we faced and show more benchmarks. Follow me on Twitter!

In the first part of this series I have described the ideas behind "row oriented" and "column oriented" databases. The second part went into more depth. This part will introduce some code and implement the user database from part 1 of this blog series.

Implementing a user database

Imagine a web application where users can register; each user has a unique id, a name, a password and an email address. In addition we will store the birthday. This will be our scenario for this chapter.

We also need to specify the indexes. We want to have fast lookups by user id, obviously; also we will add indexes for the email address and a compound index of birthday and username.

Creating indexes

In upscaledb an index is actually called a "Database", and all Databases are stored in an "Environment". First step is therefore to create such an Environment.

The code samples here are written in C++ using the C API. upscaledb supports other APIs as well (Java, Python, .NET, Erlang...). All those APIs are modelled closely after the C API and their use is very similar. If there are problems or questions then just add a comment to this blog or send me an email.

The following code creates an Environment. I enabled Transactions in order to roll back insertions in case of an error.

static ups_env_t *
create_environment()
{
  ups_env_t *env;
  ups_status_t st = ups_env_create(&env, "tutorial.db",
                  UPS_ENABLE_TRANSACTIONS, 0664, 0);
  if (st != UPS_SUCCESS)
    handle_error("ups_env_create", st);
  return env;
}

As you can see, no surprises here. The Environment is created with the default settings (no flags or extra parameters) and is stored in a file called "tutorial.db" in the current working directory. See the API reference for more options, i.e. to enable transactions, recovery, increase the cache size or to run purely in memory.

Next we need a Database with the primary key (the user id) as the index, and the full row as the record. Our C structures would look like this:

struct UserInformation {
  char username[64];
  char password[64];
  char email[64];
  uint32 day_of_birth; // time_t: seconds since 1.1.1970
};

As mentioned in part 2 you should store the data as compact as possible. Here I am completely failing to follow my own advice: fixed length arrays are wasteful. A more efficient solution would be to use variable-length strings or zero-terminated C strings (const char *) which are then serialized into a byte array. For sake of brevity I have used the less efficient, but simpler representation.

One thing to keep in mind is that C and C++ compilers often add padding to the structures. When serializing a structure to disk then padding should be disabled. Check your compiler documentation how to do this.

Let's finally create a Database with the unique user id as a primary key. We use a 32bit "Record number" Database which automatically assigns the primary key, just like MySQL's AUTO_INCREMENT column. upscaledb identifies each Database with a numerical id (here: "1").

static ups_db_t *
create_primary_database(ups_env_t *env)
{
  ups_db_t *db;
  ups_status_t st = ups_env_create_db(env, &db, 1, UPS_RECORD_NUMBER32, 0);
  if (st != UPS_SUCCESS)
    handle_error("ups_env_create_db", st);
  return db;
}

Now we're done with the boilerplate code and we can finally insert a new record:

UserInformation ui = {0};
::strncpy(ui.username, username, sizeof(ui.username));
::strncpy(ui.password, password, sizeof(ui.password));
::strncpy(ui.email, email, sizeof(ui.email));
ui.day_of_birth = day_of_birth;

/* ommitted transaction-handling code for clarity */

ups_key_t key = {0};
ups_record_t record = ups_make_record(&ui, sizeof(ui));
st = ups_db_insert(db_primary, 0, &key, &record, 0);
if (st != UPS_SUCCESS) {
  ups_txn_abort(txn, 0);
  handle_error("ups_db_insert (primary)", st);
}

The use of ::strncpy() is unsafe because it might not write the terminating zero byte at the end of the buffer if the allowed size is overwritten. I have avoided all necessary sanity checks to keep the code snippets short.

So what's the user id of the new record? It can now be retrieved from the key structure:

uint32_t user_id = *(uint32_t *)key.data;

Voila, the first record is inserted. If we move all that code into a single function which returns the user id then we have successfully replaced an SQL statement like INSERT INTO users VALUES ('tom', 's3cr3t', 'batman@host.com')

It is important to always close the handles before the program is terminated; the corresponding calls are ups_db_close and ups_env_close Note that the C API has a flag UPS_AUTO_CLEANUP which automatically closes all open Databases, Cursors and other resources that are part of this Environment. Other APIs (Erlang, Python) do not have this flag and you need to close all resources manually.

st = ups_env_close(env, UPS_AUTO_CLEANUP);
if (st != UPS_SUCCESS)
  handle_error(st);

Creating unique indexes

We want to create an additional index for the username field. In SQL databases we would define this index as "unique", because each username should only exist once. In upscaledb, uniqueness is verified when inserting the value in a Database (with ups_db_insert as demonstrated above). By default, the insert operation will fail if the key already exist, which is exactly the behaviour that we require.

This Database will store the primary key as records; since the primary key is a 32bit number we can therefore set the record size to 4 (4 bytes = 32 bit).

static ups_db_t *
create_username_index(ups_env_t *env)
{
  ups_db_t *db;
  ups_parameter_t parameters[] = {
    {UPS_PARAM_KEY_SIZE, 64},
    {UPS_PARAM_RECORD_SIZE, 4},
    {0, 0}
  };
  ups_status_t st = ups_env_create_db(env, &db, 2, 0, &parameters[0]);
  if (st != UPS_SUCCESS)
    handle_error("ups_env_create_db", st);
  return db;
}

We can now use this Database to verify that a username is unique by performing a simple lookup:

static bool
is_username_unique(const char *username)
{
  char keyname[64] = {0};
  ::strncpy(keyname, username, sizeof(keyname));

  ups_key_t key = ups_make_key(keyname, sizeof(keyname));
  ups_record_t record = {0};
  ups_status_t st = ups_db_find(dbi_username, 0, &key, &record, 0);
  if (st == UPS_SUCCESS)
    return false;
  if (st != UPS_KEY_NOT_FOUND)
    handle_error("ups_db_find", st);
  return true;
}

Whenever a new user signs up, this index has to be updated. This is handled by the following code:

ups_key_t kusername = ups_make_key(ui.username, sizeof(ui.username));
ups_record_t rusername = ups_make_record(&user_id, sizeof(user_id));
st = ups_db_insert(dbi_username, 0, &kusername, &rusername, 0);
if (st != UPS_SUCCESS) {
  ups_txn_abort(txn, 0);
  handle_error("ups_db_insert (username index)", st);
}

Creating numeric indexes

Our webpage should reward users which log in on their birthday (or maybe some parts of the webpage are restricted for minors?) and an index is required to get a list of users sorted by birthday.

The upscaledb database for this index therefore stores the birth date (with 32bit keys), and again it has the 4 byte primary key as record. Also we need to allow duplicate keys here, because several users could be born on the same day. The flag UPS_ENABLE_DUPLICATES will do the trick.

static ups_db_t *
create_dayofbirth_index(ups_env_t *env)
{
  ups_db_t *db;
  ups_parameter_t parameters[] = {
    {UPS_PARAM_KEY_TYPE, UPS_TYPE_UINT32},
    {UPS_PARAM_RECORD_SIZE, 4},
    {0, 0}
  };
  ups_status_t st = ups_env_create_db(env, &db, 3, UPS_ENABLE_DUPLICATES,
                  &parameters[0]);
  if (st != UPS_SUCCESS)
    handle_error("ups_env_create_db", st);
  return db;
}

Inserting values into this database is analog to the previous insert statements, just with a difference: the call will never fail if the key already exists, because duplicate keys are now allowed.

Creating compound indexes

After a few months our database grows and grows, and for additional functionality we want to run queries that retrieve an alphabetical list of user names grouped by their birthday. In other words: retrieve all users whose birthday is on 10.10.1977, and sort the result alphabetically.

Such a query can be implemented with nested loops. We could use our "day of birth" index to retrieve all user ids whose birthday is on 10.10.1977. Then we could retrieve the names of those users and sort the list. This would work fine, but it might become a bottleneck if the number of users keeps growing.

A much better solution would be to create a compound index, one which combines several attributes. In our case we combine the username and the birthday.

Such a compound index is possible with upscaledb, but unlike the previous indexes we now have to provide additional information regarding the sort order. For custom sort orders a new key type UPS_TYPE_CUSTOM and a compare function must be specified. Since our key size is always static, we also specify the key size.

This structure will be used for the keys (again take care of padding issues, as mentioned earlier):

struct CompoundIndex {
  uint32 day_of_birth; // time_t: seconds since 1.1.1970
  char username[64];
};

If we would store the 32bit user id instead of the user name, our key size would shrink dramatically. But we would lose the ability to retrieve the user names in alphabetically sorted order, which was our requirement (see above).

A callback function will sort the CompoundIndex structures first by day_of_birth, then alphabetically by username. It returns -1 if the first element (lhs - "left hand side") is smaller, or +1 if it's larger than the second element (rhs - "right hand side"), and 0 if both are equal:

static int
compare_compound_index(ups_db_t *db,
                const uint8_t *lhs_data, uint32_t lhs_size,
                const uint8_t *rhs_data, uint32_t rhs_size)
{
  assert(lhs_size == sizeof(CompoundIndex));
  assert(rhs_size == sizeof(CompoundIndex));
  const CompoundIndex *lhs = (const CompoundIndex *)lhs_data;
  const CompoundIndex *rhs = (const CompoundIndex *)rhs_data;
  if (lhs->day_of_birth < rhs->day_of_birth)
    return -1;
  if (lhs->day_of_birth > rhs->day_of_birth)
    return +1;
  // both are equal - compare usernames
  return ::strncmp(lhs->username, rhs->username, sizeof(lhs->username));
}

static ups_db_t *
create_compound_index(ups_env_t *env)
{
  ups_db_t *db;
  ups_parameter_t parameters[] = {
    {UPS_PARAM_KEY_TYPE, UPS_TYPE_CUSTOM},
    {UPS_PARAM_KEY_SIZE, sizeof(CompoundIndex)},
    {UPS_PARAM_CUSTOM_COMPARE_NAME, (uint64_t)"CompoundIndex-compare"},
    {UPS_PARAM_RECORD_SIZE, 4},
    {0, 0}
  };

  /* register the compare function */
  ups_status_t st = ups_register_compare("CompoundIndex-compare",
                  compare_compound_index);
  if (st != UPS_SUCCESS)
    handle_error("ups_register_compare", st);

  st = ups_env_create_db(env, &db, 4, 0, &parameters[0]);
  if (st != UPS_SUCCESS)
    handle_error("ups_env_create_db", st);
  return db;
}

Inserting and looking up values is nearly identical to the code for the other indexes, but the CompoundStructure key must be initialized properly. In the next part we will then see how to perform range queries on those indexes.

Download the code.

From the release notes:

This release fixes a few bugs. This is the last stable release before 2.2.0.

The most significant fixes are:

• #47: fixed bad assert in ups_cursor_move; this caused crashes in debug mode
• #66: return with error if a database is modified by a committed transaction which was not yet flushed to disk; this caused crashes when closing an Environment (ups_env_close)
• #64: fixed segfault when recovering a database with a custom compare function; deprecated ups_db_set_compare_func, use ups_register_compare/UPS_PARAM_CUSTOM_COMPARE_NAME instead. Also, the .NET/C# delegate for custom compare functions has been changed, the delegate has to perform the marshalling now (see unittests for a sample)
• issue #60: files are now truncated after a database was deleted with ups_env_erase_db

All files are available on the download page.

In the first part of this series I have described the ideas behind "row oriented" and "column oriented" databases. This part will delve even deeper.

OLAP vs OLTP, or: Column Oriented vs Row Oriented

You might have seen the terms OLAP and OLTP before. OLAP is an abbreviation for OnLine Analytical Processing, and this basically means what was described in part 1: full-table scans with analytical calculations. An example would be to calculate the average salary of all employees, the monthly sum of all incoming orders for a dashboard, the 10 top scorers of your favourite sports league etc.

OLTP, in contrast, is about OnLine Transactional Processing: short read/write operations. A typical example is the shopping cart of an online warehouse. If a visitor "adds" an item to her shopping cart then a short transaction with a few write operations will update the database.

As mentioned previously, upscaledb users have the choice between a column oriented and a row oriented data representation. Of course you can also build hybrid databases where data is stored row oriented, with a few additional indexes for running analytical queries or fast lookup operations.

Optimizing for OLTP

Short read and write operations are fastest if as few indexes as possible have to be updated. In best case, there is only one index (which is as small as possible), and the record data.

In order to make inserts and lookups as fast as possible, it is really essential to reduce your data size as much as possible! It will make a huge difference if you store a timestamp as a 32bit number (number of seconds since 1.1.1970), or as a string in YYYYMMDDhhmmss format, especially if this timestamp is used as the index key.

If records are very small then upscaledb will store those records in the Btree leaf; otherwise it will allocate a "blob" somewhere in the file and store the record there. Just how small do those records have to be in order to get stored in the Btree leaf? That's up to upscaledb to decide, and it also depends on the page size and the key size. You can use `ups_bench` to run a simulation, then use `ups_info` on the generated database and it will print whether records are stored as blobs or not.

Optimizing for OLAP

Long queries and full table scans require lots of I/O, therefore it is important to reduce the size of the data as much as possible. As outlined in part 1 of this series, column store databases reduce the data size that is required for a scan by storing only a single attribute in a Btree node, and many offer additional compression of these nodes.

There are many compression algorithms optimized for the different data types, i.e. bitmap compression, prefix compression, heavy weight compression (using compression algorithms like LZO, LZF), dictionary compression etc. upscaledb implements some of these.

In order to optimize for analytical queries, you should create indexes for all those attributes that you want to query. Make sure that your databases are configured for the data type of this attribute, i.e. specify a key type (`UPS_KEY_UINT32`, `UPS_KEY_UINT64` etc). Since all those indexes just offer an indirection to the primary key, you should make sure that the primary key is as small as possible, in order to have upscaledb store it in the Btree leaf.

If you do not have a natural primary key, then upscaledb offers a nifty feature called "Record number": It will automagically assign a 32bit (or 64bit) numeric key to each new record, just like MySQL's AUTO_INCREMENT column.

Data Modelling in NoSQL databases

Ultimately, you will often end up with a "primary index", which contains the data for your rows (similar to a "row oriented" storage), and secondary indexes for querying other attributes (as known from the "column oriented" storage). A graphical representation looks like this:

(C) wikipedia, CC BY-SA 3.0

Looks like a star, right? That's why this schema is called "Star Schema". It's a simple way to model your data, especially if the database lacks functionality for more sophisticated models (i.e. enforcing integrity). NoSQL databases work well with the star schema, and upscaledb is no exception. The web has a lot of information about the star schema.

Next time I will show how to implement an improved user database from part 1 with upscaledb, including indexes and range queries.

Writing your own SQL database with upscaledb is not difficult. Start of a new series.

This post is the first in a longer series of posts describing how to implement SQL-ish scenarios in a key/value store. In the first article I'll explain the basics, then will continue to describe OLAP vs OLTP and how embedded databases (specifically upscaledb) can be optimized for both usages. The third post will show simple recipes for translating SQL databases to upscaledb and for running simple queries. Afterwards I will go in depth and describe analytical operations on complex real-live data.

What’s the point in using an embedded key/value store to implement SQL-style database functions? Wouldn’t it be much easier to just use an SQL database instead? Most likely it would indeed be “easier”, but there are some compelling reasons why an embedded key/value store is more attractive. One is performance; most key/value stores beat SQL databases hands down. Another one is the user experience. SQL databases often require the installation (and maintenance) of a database server, which is time-consuming and requires lots of support. If you follow this blog series then you will see that it’s not difficult to work with an embedded database. Let me show you how you can build your own column store database tailored for your application.

What is a column store database?

The blob title mentioned a "column store database" - one of the buzzwords in the DBMS market. But what is this exactly?

Traditionally, databases stored the data of a single row sequentially; they are "row oriented". The row's data is indexed by its primary key. Imagine a table with user information, here described in pseudo-SQL:

CREATE TABLE users (
  USERID NUMERIC PRIMARY KEY,
  USERNAME TEXT,
  PASSWORD TEXT,
  EMAIL TEXT
) 

(Storing a password as plain text is a security problem - let’s pretend that we store password hashes instead.)

We fill this table with some dummy data:

1	tom	s3cr3t	batman@host.com
2	amy	AS302.x	amy@server.com
3	rick	kcir	the_rick@email.net

In order to store this data, it has to be serialized to disk. Each row information is indexed by its primary key (the user id). I am using the arrow (->) to describe the "is indexing" relation.

1 -> tom|s3cr3t|batman@host.com / 2 -> amy|AS302.x|amy@server.com / 3 -> rick|kcir|the_rick@email.net

(Note that databases could also assign their own internal id instead of using the primary key as an index.)

This schema is simple and works well. When looking up data, the USERID column serves as an index and returns a blob with a compact representation of the row's full data. Reading and writing of a single row is very fast.

This schema also has a disadvantage: Database indexes are usually implemented as B-tree structures. A B-tree consists of nodes, and each node will fill up quickly if the row data becomes large. The B-tree will have a high fanout, and performing full table scans requires lots of slow disk access.

In contrast, a column store database (or "column oriented database") stores each column as a separate index. In our example, the data could be serialized like this:

1 -> tom / 2 -> amy / 3 -> rick
amy -> 2 / tom -> 1 / rick -> 3
amy@server.com -> 2 / batman@host.com -> 1 / the_rick@email.net -> 3
AS302.x -> 2 / kcir -> 3 / s3cr3t -> 1

In real life, a column store DBMS might automatically choose which kind of indexes it wants to create. It might even create compound indexes which combine multiple rows.

In contrast to the row oriented serialization, a column oriented index is much smaller because it only stores attributes of the same type. In addition, these attributes are sorted and therefore can be compressed heavily. Running table scans therefore requires less I/O, and is much faster as long as only one index is scanned. These table scans are typical for analytical queries, i.e. to calculate AVERAGE or SUM over a full table, or MIN and MAX values.

On the other hand, inserting a new row requires several write operations, because each index must be updated. Therefore column store databases are often faster when reading data, while traditional, row oriented databases are faster when writing data.

How can upscaledb help?

As a key/value store, upscaledb specializes in providing a small subset of the functionality that a full DBMS offers. This subset enables you to store indexed data and create multiple indexes. upscaledb knows about the "type" of your data, i.e. whether it's numeric or a binary blob. Its representation is therefore very compact. In addition, upscaledb has built-in API functions for analytical calculations (i.e. COUNT, COUNT DISTINCT, SUM, AVERAGE).

As an application developer you have the full freedom (or responsibility?) to create those indexes that you require, and to optimize the database for the specific needs of your application. You can decide whether your database is optimized for short read/writes or for long analytical scans, or even have a mix of both worlds.

What's the meaning of "embedded"?

upscaledb is an embedded database, but the word "embedded" is a bit ambiguous. upscaledb will run fine on embedded platforms (smart phones, tablets, Raspberry PI etc). But more importantly, upscaledb is a library which runs embedded into your application. End users are not exposed to the database. They do not have to maintain it, and your application does not need to deploy or install a separate database. In addition, your application does not require expensive inter-process communication (IPC) when talking to the database. Not relying on IPC makes the whole interaction between application and database much faster.

In the next part of this series I will write more about row oriented vs column oriented database design, and show how upscaledb can be used to implement these concepts.

Switching from hamsterdb to upscaledb can be performed with simple search&replace commands.

The new header file is now <ups/upscaledb.h> and the library name is libupscaledb.a (or upscaledb.dll). The API functions, structures and macros were renamed. Previously, they started with “ham_” (or “HAM_”), now they start with “ups_” (and “UPS_”), e.g., “ham_env_create” is now “ups_env_create”.

There are only a few additional changes:

• The remote URL identifier "ham://" was changed to "ups://"
• The function ham_set_errhandler was renamed to ups_set_error_handler
• The flag UPS_FIND_NEAR_MATCH was removed
• The flag HAM_FIND_EXACT_MATCH was renamed to UPS_FIND_EQ_MATCH
• The functions ham_db_get_error, ham_is_pro, ham_is_pro_evaluation were removed

Please see the Changelog of upscaledb-2.1.12 for more information.

And not just the name has changed.

upscaledb is a very fast key/value store with a new focus on blazingly fast built-in analytical functions.

All closed source PRO features are now freely available:

• Built-in Analytics
http://upscaledb.com/about01.html#analytics
• Bleeding edge integer key compression and record compression
http://upscaledb.com/about01.html#compression
• CRC32 verification and AES encryption
http://upscaledb.com/about03.html#compliancy

upscaledb is free for non-commercial use. Commercial licenses are available.

The documentation was rewritten, and the about page explains the architecture and the unique features of upscaledb.

From the article: While many facets of the Internet of Things (IoT) are falling into place, some hurdles still exist for the databases that will be used to manage IoT sensor data. In this roundtable with Christoph Rupp of hamsterdb, Sasan Montaseri of ITTIA, Steve Graves of McObject, and ScaleDB’s Mike Hogan we explore the factors currently limiting embedded databases, scaling and securing IoT databases, and the available tools and techniques for managing and analyzing sensor inputs from a sea of connected embedded devices.

Read the full article here: http://embedded-computing.com/articles/the-ins-and-outs-of-embedded-databases-for-the-iot/.

A blog post originally published on http://highscalability.com/blog/2014/8/13/hamsterdb-an-analytical-embedded-key-value-store.html with an introduction to hamsterdb, an embedded analytical key-value database library similar to Google's leveldb and Oracle's BerkeleyDB.

In this post, I’d like to introduce you to hamsterdb, an Apache 2-licensed, embedded analytical key-value database library similar to Google’s leveldb and Oracle’s BerkeleyDB.

hamsterdb is not a new contender in this niche. In fact, hamsterdb has been around for over 9 years. In this time, it has dramatically grown, and the focus has shifted from a pure key-value store to an analytical database offering functionality similar to a column store database.

hamsterdb is single-threaded and non-distributed, and users usually link it directly into their applications. hamsterdb offers a unique (at least, as far as I know) implementation of Transactions, as well as other unique features similar to column store databases, making it a natural fit for analytical workloads. It can be used natively from C/C++ and has bindings for Erlang, Python, Java, .NET, and even Ada. It is used in embedded devices and on-premise applications with millions of deployments, as well as serving in cloud instances for caching and indexing.

hamsterdb has a unique feature in the key-value niche: it understands schema information. While most databases do not know or care what kind of keys are inserted, hamsterdb supports key types for binary keys (fixed length or variable length) or numerical keys (i.e. uint32, uint64, real32, real64).

hamsterdb’s databases are Btree indexes that can be stored either in a file or in memory. The Btree implementation is what makes hamsterdb so analytical. Btree indexes are implemented with C++ templates, with the template parameters depending on the configuration of the database. More specifically, the template parameters depend on the key type, the record’s size (fixed length vs. variable length), and whether duplicate keys are enabled or not. Each Btree node is therefore highly optimized for its workload. For fixed length keys, there is zero overhead for each key, and the keys are serialized as a simple arrays. On the lowest level, a database with uint64 keys is implemented as a simple C array of type uint64_t[].

This representation is extremely compact, reduces I/O and has the added benefit of exploiting CPU caches much better. Modern CPUs are optimized for processing sequential memory. hamsterdb takes advantage this. For example, when searching through leaf nodes, binary search will be skipped when the remaining range has reached a certain threshold, and linear search is used. Also, hamsterdb has APIs equivalent to the SQL commands COUNT, COUNT DISTINCT, SUM and AVERAGE, all of which are extremely fast when working on fixed length keys thanks to running directly in the Btree.

hamsterdb also supports variable length keys. For this, each Btree node has a very small index upfront pointing into the node’s payload. This can lead to defragmentation if existing keys are resized or deleted, so the node has to be periodically “vacuumized” to avoid wasting space. The vacuumize operation turned out to be a performance killer, and it was a big challenge to make it fast and call it as rarely as possible.

hamsterdb also supports duplicate keys, meaning a given key can have several records assigned to it. All records of a key are grouped together. These groups are managed with the same upfront index structure as variable length keys. (If a single key has too many duplicate records, they are removed from the Btree leaf and stored in a separate overflow area.)

hamsterdb supports ACID Transactions with a read-committed isolation level. Transactional updates (that are not yet flushed to disk) are stored as delta-operations in memory. Each Database has its own Transaction index as an “overlay”, and updates in that Transaction index have higher priority than those in the Btree. Aborting a Transaction simply means dropping the transaction’s updates from the Transaction index, and a commit flushes the transaction’s updates to the Btree.

This unique design choice has a few significant advantages. Transactional updates stay in RAM instead of requiring I/O. Also, no undo logic is required because aborted Transactions are never persisted. Redo logic (in case of a crash) is performed with a simple logical journal. But there is also a major challenge: at runtime, both trees have to be consolidated. This turned out to be very complex; imagine using a database cursor (hamsterdb has very fast cursors) to perform a full scan. Some keys are in the Btree, while others are in the Transaction tree. Keys in the Btree can be overwritten or even deleted in the Transaction tree, or another Transaction could currently modify them. Things get even harder when duplicate keys are involved.

hamsterdb’s driving feature, however, is testability. The essential criteria for a database – even more important than performance! – is that it should not lose data. Critical bugs are forbidden. In addition, one side effect of a 9+ year development history is that there are those occasional bad design decisions I’d like to remove in order to keep the technical debt low, remain agile (with a lower case ‘a’) and quickly react to user’s wishes or new ideas. I’m constantly rewriting parts of the code or trying out new ideas. Having high test coverage is essential to give me the confidence that those changes do not break anything.

Focusing on testability and a high level of automation enables me to do all this. At the lowest level, hamsterdb’s debug build is cluttered with asserts and many integrity checks. There are about 1,800 unit tests and about 35,000 acceptance tests (many of which are scripted, others of which are randomly generated). Those acceptance tests run in dozens of different configurations and are executed in parallel against BerkeleyDB. Both databases are permanently checked for equality, so any introduced bug will immediately show up. In addition, each test spits out a long list of metrics, including memory consumption, number of heap allocations, number of allocated pages, blobs, and btree splits and merges, among others.

Some of those tests are executed in valgrind. Some tests are executed, and their performance results are compared to those of the previous release. This way I can quickly spot and correct performance regressions.

Then there’s another set of tests simulating crashes to test the recoverability features of hamsterdb. And last but not least, I use QuviQ‘s QuickCheck, a property based testing tool for Erlang. QuickCheck lets you describe properties of your software, then executes fancy pseudo-randomized instructions, always verifying the integrity of your properties.

Static code analysis is used with Coverity’s open source program and clang’s scan-build. Both have found a few minor issues.

All tests are fully automated and performed before each release. A full release cycle takes a few days, and they kill a hard drive every couple of months.

If I’ve learned any lessons I can offer, it’s that writing tests really can be fun if you see the benefits! And iterative development without reliable tests is simply impossible.

Let me also mention hamsterdb pro, the commercial twin. hamsterdb pro offers heavy-weight compression (zlib, snappy, lzf, and lzo) for keys, records and the journal, and AES encryption and SIMD optimizations for leaf node lookups. More compression algorithms (bitmap compression, prefix compression) are either planned or in progress right now. The webpage has more information.

So far so good, but what about hamsterdb’s performance? I have run hamsterdb 2.1.8 against Google’s leveldb 1.15 in their own benchmark. Compression was disabled (hamsterdb does not offer compression, but hamsterdb pro does). Fsyncs were also disabled, as was hamsterdb’s recovery feature (implemented with a Write-Ahead Log). I used test sizes ranging from small keys/small records to medium sized keys with larger records, and inserted data ranging from 100k to 100m items. In addition, I ran two analytical functions of hamsterdb (sum and count w/ predicate), and implemented the same for leveldb. All tests ran with cache sizes of 4 MB and 1 GB, on machines equipped with a HDD and an SSD.

hamsterdb’s configuration is always set to fixed length keys, and for the 8 byte keys hamsterdb stores uint64 numbers. This is a bit to hamsterdb’s advantage, since leveldb requires the number to be converted to a string.

I have also added tests with small records (size 8), because small records are typically used for secondary indexes, when they contain the primary key. I ran each test on both a machine with a HDD (Core i7 i7-950 with 8 cores and 8 MB cache), and a machine with an SSD (Core i5-3550 with 4 cores and 8 MB cache). I am not showing all the benchmark results here; if you are interested you can download them here.

Sequential writes; key size: 16, record size: 100 (HDD, 1 GB Cache)

Sequential reads; key size: 16, record size: 100 (HDD, 1 GB Cache)

Random writes; key size: 16, record size: 100 (HDD, 1 GB Cache)

Random reads; key size: 16, record size: 100 (HDD, 1 GB Cache)

Calculating the sum of all keys (HDD, 4 MB Cache)

Counting all keys which end with “77″ (SSD, 1 GB Cache)

For random reads, hamsterdb performs better than leveldb. For random writes, hamsterdb is faster as long as the data size is not too large. Starting from 10 million keys and above, hamsterdb suffers from the typical problems of a BTree database: lots of non-sequential I/O with high disk seek latencies.

That being said, the tests nicely demonstrate the analytical powers of hamsterdb. In particular, both the sum and the count calculations are scaling quite nicely. hamsterdb also shines when inserting and scanning sequentially, and stays extremely fast regardless of the data size.

The benchmarks show my homework for the coming months: optimize random read/write operations by making hamsterdb concurrent. This is a big one I’m working on, and I’ve already sketched out a design and begun refactoring a few releases ago.

I hope this post has given you a good introduction to hamsterdb. If you’re interested, go have a look at the webpage (http://hamsterdb.com) and the software. The source code is available on github. I appreciate all kinds of feedback!