Implement read-snapshots

[rvagg]

options.snapshot = NULL; on an iterator will give a consistent snapshot view of the data while reading. db->GetSnapshot(); will create a snapshot and return a version that can be set against options.snapshot. The drawback of the latter is that you have to explicitly db->ReleaseSnapshot() to clean up versions you've created.

At least the NULL case would be handy to start with.

[Raynos]

So currently no snapshot is used whilst reading and you get an inconsistent view (i.e. puts after you start reading reflect in your results)

[rvagg]

actually, sorry, it's NULL by default so any of the ReadStreams will be over a consistent view of the database created by an implicit snapshot.

[dominictarr]

I think this is off by default,
I have a test where a readStream starts in the first tick,
and then later that tick I do some puts into that read's range.

The puts come out in the read stream.
I think when it says "implicit snapshot" it's just referring to an implementation detail.
It's not providing a consistent view.

[rvagg]

OK, well we need tests for snapshots anyway so a good start would be a test for the current implementation to see what it's actually doing. I'll try and get to that soon.

[rvagg]

See: Level/levelup@40dbde6#diff-3

You'll need to update your slow-stream in your devDeps to make this one run properly.

Basically the test adds in a bunch of data, then creates a readStream() and then goes ahead and overwrites all the original data with new cruft. The stream is piped through a SlowStream to ensure that it's throttled to be slower than it takes the overwrite to happen.

The test demonstrates that an "implicit snapshot" indeed means that a snapshot is being created to handle the iterator even without being told to do so. You can verify this further with some debug prints on stream 'data' and the return from the second batch() which is done with {sync:true} to be absolutely sure. You can also increase the rate of the SlowStream to something even higher, say 50ms between 'data' events and it should still pass. (And yes, SlowStream does apply back-pressure, I even did some debugging from the C++ layer to ensure that the iterator was being called much slower than it took the batch to happen).

Got any better ideas for a test case on this?

Also on that commit you'll see the basics of some snapshot work on the C++ layer, it should be fairly simple to implement but I need some time to make a nice JS interface and some decent tests for it.

P.S. if you ever get failed tests, then run again and get passed tests, it's probably a database not being deleted properly from the test/ directory. I really need to do a rimraf() on the dir before it's used again just to make sure..

[luk-]

I'm +1 on snapshots exposed in levelup

[sethyuan]

Still no support for snapshots?

[rvagg]

see #138
marked as "help wanted", it's mostly specced out if someone wants to tackle it but for now I think this should be mostly kept to leveldown rather than being exposed here in levelup

[vweevers]

Tracked in #118.