cyberhoboing with dominic tarr

fairly tale cryptography 2: hashes

Wed, 21 Dec 2016 17:24:05 +0000

In “cinderella” a young woman living in unfortunate circumstances who’s situation suddenly changes, when a magical entity appears and gives her a make-over, allowing her to attend a fancy ball at where the local prince falls in love with her – but must hastily depart before the magic wears off… and looses a shoe, which happens to be made of glass.

The prince doesn’t write to the “missed connections” column, because it’s medieval times and Cinderella is most likely illiterate. Instead he sends out his people to test everyone and find the person whom the shoe fits.

In this story, the shoe behaves exactly like a hash. A hash is a small value that reflects the uniqueness of an file, But does not give you any other clues about the file.

A hash might usually looks something like this: ebd8581d6cea19558023b851fbdf599a5044b446 Just a seemingly random sequence of numbers and letters. There is nothing obviously indicative of a particular file in the output, yet if the file is changed by even one letter, it will have a completely different hash.

If you have that file (“Cinderella”) you can create that hash (a glass shoe that fits her) but you cannot go back the other way (you cannot create a Cinderella from the shoe!). Possessing the shoe also does not tell you where Cinderella is Possesing the shoe does not allow the prince to home in on her location - it does not give clues like “warmer” or “colder”

But what the shoe does do, is it allows the prince to know when he has found her. The prince can send someone out to take the shoe to every woman in the kingdom, and test if each one fits the shoe. A hash works exactly like this! If you know the hash, you can check whether any file produces the same hash. But if you only know the hash, and are looking for the file, it may be very hard to find that file.

What happens if the shoe fits someone else? In modern times, shoes are made in factories to standard sizes, but in medieval times, your shoes would have been handmade to fit your feet, and then worn in to your feet on top of that. If you imagine it from that perspective, it seems more reasonable that the shoe uniquely fits Cinderella, but indeed it is certainly possible that there is different person with the same feet as cinderella, it’s just extremely unlikely.

Again, this is exactly like a hash. The since the hash is often smaller than the file, there are obviously more possible files than hashes, so it’s entirely possible that there are two possible files that have the same hash, but there are already so many possible hashes (given 40+characters) that the chance of ever finding two files with the same hash is so low that it will basically never happen. Not before you and everyone you know is dead, or maybe not before the sun swallows the earth. Basically, with a well designed hash, we have much bigger things to worry about than hashes colliding, and if we are worried, we can just use a longer hash.

So there you go, hashes explained via a fairy tale you already know. Also see part 1: rumplestiltskin as bitcoin

Video

Tue, 29 Nov 2016 15:14:53 +0000

Video

Tue, 29 Nov 2016 15:14:28 +0000

Design Challenges of Decentralized Systems (at Data Terra Nemo)

Tue, 29 Nov 2016 15:10:41 +0000

Design Challenges of Decentralized Systems (at Data Terra Nemo)

pull streams pull-streams are a very simple streaming primitive, but can do everything that streams...

Sun, 21 Aug 2016 03:02:26 +0100

pull streams

pull-streams are a very simple streaming primitive, but can do everything that streams need to do, from back pressure to propagating errors (which node streams do not support!)

You can use them to write any application or system you might use node for, but they are also great for simple realtime data processing - like Array.map but now you can easily do it on async – realtime – sequences of data. Things that would be cumbersome to do with node streams.

sources

The first pull stream you’ll meet is values, it creates a pull-stream from an array.

function values(array) {
  var i = 0
  return function (abort, cb) {
    if(abort) return cb(abort)
    return cb(i >= array.length ? true : null, array[i++])
  }
}

That’s it. values is a source pull-stream. A source stream is just an async function that you may call repeatedly. If you call with the first argument as true or an error, that tells the stream to abort, but we’ll come back to that later.

Otherwise, each time you call the stream, it calls back with the next item in the array. When all the items have been returned, the first argument will be true.

this is because err and end both are the terminal state of the stream: they both mean that nothing more is coming

sinks

data comes out of sources and goes into sinks. Where a source is a function you call repeatedly, a sink is a function you pass a source to.

Here is a simple sink that dumps a pull-stream to the console.

function log () {
  return function (read) {
    read(null, function next (end, data) {
      if(end == true) return
      else if(end) throw end //error
      console.log(data)
      read(null, next) //loop again.
    })
  }
}

log creates a function that accepts a source - i.e. a read function. it then calls that read function, and if it doesn’t give an “end” or “error” signal, then it reads again.

Other stream apis such as node.js streams have the concept of a “writable” stream. A writable is a passive object that accepts data, like a couch-potato watching the TV. They must actively use the TV remote to send an explicit signal to the TV when they want to stop or slow down. A pull-stream sink is a reader. it is like a book-worm reading a book - they are active and can read faster or slower. They don’t need to send an explicit signal to slow down, they just act slower.

you could combine these two streams like this:

log() (values([1,2,3]))

output:

1
2
3

Normally, we are used to reading code from left to right, so seeing log before values is a bit weird, when the data is coming out of values, but we’ll come back to that.

Since pull-streams use an async call, we get two way back pressure. The source can slow the stream down by calling back slower. And the sink can slow the stream down by waiting longer until they call read again.

Slowing down is imporant, because that is how you save resource. software performance is like loosing weight, not building muscles. To go faster you must do less. Streams are about doing less, and about not doing things you don’t need to do yet.

throughs/transforms

Often we want to transform data, we might split a file into lines, or parse objects out of json, or look for matches.

for that we need a stream which is both a source and a sink. here is a through stream that takes applies a function to the data in the stream.

function map (fn) {
  return function (read) {
    return function (abort, cb) {
      read(abort, function (end, data) {
        if(end) cb(end)
        else cb(null, fn(data))
      })
    }
  }
}

A through stream is just a sink stream that returns a source stream. You can think of it as transforming that stream.

if we put these streams to gether it would look like this.

var vs = values([2,4,6])
var ms = map(function (e) { return e*10 })
var ls = log()

ls(ms(vs))

output:

  20
  40
  60

A through stream is both a source and a sink but also neither. To the source, the through seems like a sink, but it’s really just proxying the sink behind it. To the sink, the through is a source, but it’s really just transforming the values that the source gives it. This means through streams are very cheap. They don’t need to add their own back pressure, but they allow the source/sink back pressure to flow through them.

Now, we don’t like reversing our thinking from left to right to right to left. lets fix that.

function pull() {
  var stream = arguments[0]
  //detect a sink (or through)
  if(stream.length === 1) {
    var args = [].slice.call(arguments)
    return function (read) {
      return pull.apply(null, [read].concat(args))
    }
  }
  for(var i = 1; i 0) read(abort, cb)
      else      read(true, cb)
    }
  }
}

take doesn’t change the incoming data, but after you have called it n times (n is decremented until it’s zero) and then on the next call it calls read with true as the first argument. If you check back at values, if the first argument is true, then it just calls back immediately.

map doesn’t need to handle this case specially, it just passes it through.

that means we can do things like:

pull(
  values([1,2,3,4,5,6,7]),
  map(function (e) { return e*e })
  take(4),
  log()
)

output:

when take decides to stop reading, it passes that on to the source end of the stream and they can stop too. If there was work they where gonna do, they wont need to do it now. This property is known as lazyness. pull-streams are very lazy.

Because lazyness means we don’t do any work until we know we’ll need to, we can commit to work that we know we’ll never do! like, we can have an infinite stream.

function random () {
  return function (abort, cb) {
    cb(abort, abort || Math.random())
  }
}

random creats and infinite stream of random values.

pull(random(), take(5), log())

the output will look like:

0.16838359273970127
0.10774739156477153
0.6401788892690092
0.24491786980070174
0.7873906309250742

Aborting an infinite stream is fun, but in real life there are lots of reasons for aborting a stream. maybe you are uploading a file, and the wifi goes down (now there is no where to send that file)

maybe you are running a server with many clients downloading files, if some of those clients break, you need to clean up after them otherwise you will get a resource leak.

Sometimes a stream needs to stop in the middle, maybe a parser has found an error.

expanding or shrinking streams

sometimes we have a stream of big chunks that we want to break into small chunks (say, pages into lines) this means few items become many. On the other hand, we may want to filter out lines that don’t match some pattern or property - here we turn many into few.

flatmap takes a function that returns an array, and outputs a stream of the items in returned arrays. (note, the array could have zero or more items!)

function flatmap (fn) {
  var queue = []
  return function (read) {
    return function again (abort, cb) {
      if(abort)        return read(abort, cb)
      if(queue.length) return cb(null, queue.shift())

      read(null, function (err, data) {
        if(err) return cb(err)
        queue = fn(data)
        again(null, cb) //cb or read again if queue is empty.
      })
    }
  }
}

This gives us expanding or contracting streams! The trick is to maintain a queue of the current state, and on the next read, read from that instead of asking the source. If fn returns an empty stream, the oppositie happens, the source is possibly consulted many times without passing an answer through to the sink.

Notice that we have not installed any dependencies, or even imported any libraries. We have acomplished a lot, and without writing a function longer than 20 lines! I’m not pulling your leg, the code we have written is completely valid pull-streams, without bugs.

one small lie: log() will get a stack overflow if you use it on a long synchronous source stream, but will be fine with an async stream. use drain in the pull-stream module to get around this.

pull-streams are just a pattern. There are many modules that implement particular pull streams, but you can write a useful pull-stream without using them. many pull-stream modules don’t depend on other pull-streams, except maybe for testing.

That is how pull-streams work (except we havn’t gone into any detail about duplex stream - which are distinct from transform streams) but the best thing about pull-streams is that they are very easy to compose and it is very easy to write modules for and reuse those modules and so there are many pull-stream modules.

The most useful pull-streams are being curated into the pull-stream github org.

source for thes post as git repo

History Of Streams

Mon, 30 May 2016 04:37:07 +0100

The history of streams

prehistory (streams before a stream api)
streams 1
summer of streams, userland vs core
streams2 (one step forward, 2 steps back)
streams3
conclusion

prehistory

in node.js’s early days back in 2010, the node.js website featured an example of a tcp echo server.

on feb 9th, 2010 the tcp stream’s events were named “receive” and “eof”. Shorly later, by feb 18th, 2010 those events had been renamed to their current “data” and “end”. There was not yet a Stream class, just an EventEmitter, but it was recognised that files and tcp sockets, etc were basically the same and a convention had developed.

This is the birth of node’s most important api, the Stream. But things were just beginning to take form.

Although it never appeared on the website, in nodejs@0.2 the sys module (now the util module) featured a pump method. by the time of the first node conference there was a base class for Stream and sys.pump(readStream, writeStream, callback) had become readStream.pipe(writeStream)

By node@0.4 stream was now well cemented in the node.js api, but it wasn’t until pipe becomes chainable that it feels like the node.js streams as we know it.

Around this time the streams was beginning to get popular. When I discovered streams and realized they would be useful, I wrote event-stream which applied streams to a use similar to Functional Reactive Programming

I convinced substack and Max Ogden how useful streams were, and then they convinced everyone else.

Streams had been created by the core node developers, but they were then adopted by userspace - by people who maybe didn’t contribute much to node’s core, but did contribute greatly to node’s ecosystem.

These two groups had slightly different interpretations of what a stream was. Mainly, the core developers intended streams to only be applied to raw data. The core interpretation was that stream could be buffers or strings - but the userland interpretation was that a stream could be anything that is serializeable - anything that could become buffers or strings.

My argument was that you needed all the features of streams, passing items one at a time, possibly with an eventual end or error wether it was a sequence of buffers, bytes, strings or objects. Why not use the same api? If javascript was strictly typed, we never would have had this argument, but javascript didn’t care and the streams code at that time was agnostic - it worked just the same with objects as it did with strings.

problems with early node streams.

Streams were recognised as a powerful abstraction, but also difficult to use. One main difficulty was that data started flowing through streams as soon as you created them. Once you had a stream, had to pipe it somewhere before you did anything async.

This problem was exacerbated by the fact that the pause method was only advisory, it did not mean that data would stop flowing immediately.

There were also generally just too many things you had to get right for streams to work - not only did you have to implement write end and events 'data' 'end' and then there was the (still controversial) destroy methods and 'close' events.

I had written pause-stream to handle the second case, and more importanly through. Through represented about 11 months worth of myself using streams in userland, and I had finally figured out the pattern that was most general and easiest to use - from a userland persective (stream authors near core were more likely to implement streams that read from or wrote to some device - low level interfaces - but userland streams are mostly transformations)

I had called the module “through” because I figured that input, output… throughput. it was later pointed out by Isaac that “through” was a common word and “transform” stream was better name. In hindsight, I agree. through finally made creating streams easy, you just had to provide two functions, and all the edge cases were handled for you. through currently has 1601 dependant modules, and it’s heir, through2 (same api but uses streams2) has 5341 dependents. (check with npm-dependents)

Interestingly, I started this a few weeks before isaac began developing streams2 - streams2 addressed the problems in streams1, but made a much more radical departure from the api.

streams2 was paused until you called pipe, and was implemented around a read(n) function. read took an optional number of bytes, and would return a buffer with that many bytes in it. This was intended to be useful for implementing parsers, but to use it you had to have a handle on the stream directly, because pipe didn’t give you a way to specify a number of bytes.

streams2 was much more complicated than streams1, but worse, it was backwards compatible with streams1 (although streams1 was considered “legacy”)

streams1 was a readable 100 lines or so, and I read it many times, but by the time streams2 was ready to go into node@0.10, the main class: Readable, was over 800 lines

During streams2 development the are-objects-data argument was settled with streams of objects officially sanctioned in the api (which would have otherwise excluded their possibility)

problems with streams2

All these changes in streams were causing their own problems, but one good move was developing streams2 in it’s own repository. This made it possible for those interested to depend on streams2 and test it out without having to use a fork of node.

Indeed, if you use through2 (and you very likey do use it, maybe because a module you use uses it) then you depend on that readable-stream repo. Although it’s built into node, the version in the repo is more cross version compatible. Modules that use readable-stream still work in old versions of node that did may not have as modern a version of streams!

Once the initial difficulties of piping new streams and getting friendlier apis for stream creation were surmounted, new problems emerged. There is a growing awareness that errors need to be propagated somehow. In this (open) issue, it’s shown that a naive http file server will have a file descriptor leak

To prevent that, you need to have a handle on both the source fs stream, and the destination stream that errored, not to mention a detailed understanding of how node streams function.

This is an open issue, and not even the iojs revolution could fix it. This would most surely be a breaking change, and so changing how core streams worked would be very difficult.

There is an approach in userland (that harkens back to sys.pump!)

There was one serious proposal to add error propagation to node streams. But the difficulty of providing backwards compatibilty made it untenable.

streams 3

The version of streams now in contemporary node@5 is considered streams3. it’s a significant refactor from streams2, but, is still backwards compatible. While it still supports streams2#read(n) that behavior is not activated until you trigger it by using it in a streams2 style. If you just use streams normally, via source.pipe(dest) then it works basically like streams1 streams3.pipe

If streams1 is classic streams and streams2 is new streams then streams3 is vintage streams. It’s old, but it’s new. But the sad thing, is that now it’s still too complex, because of features picked up during the streams2 detour.

stream 4? no.

If node streams teach us anything, it’s that it’s very difficult to develop something as fundamental as streams inside a “core” you can’t change core without breaking things, because things simply assume core and never declare what aspects of core they depend on. Hence a very strong incentive occurs to simply make core always be backwards compatible, and to focus only on performance improvements. This is still a pretty good thing, except sometimes decisions get inadvertently made that have negative implications, but that isn’t apparent until it’s too late. In this situation, a clean break is necessary - node.js itself is a great example of this. node created a IO performance improvement over previous dynamic languages (perl, ruby, python) because node was able to abandon threads and blocking, and write an entire stack from scratch.

But, usability problems with streams1 stopped us from seeing the need for error propagation until it was too late, and now node’s success has created it’s own obstacle for further improvement.

Streams are one of the best things about node, but they are still quite difficult to use, and they still have room for improvement. But that improvement cannot be made while they are a part of node.js core.

to be continued in part two.

Video

Tue, 09 Feb 2016 23:24:01 +0000

Which JS crypto library should I use?

Fri, 13 Nov 2015 03:29:45 +0000

A friend recently asked me “which js crypto library should I use?” Since this is a question that many people have, I have answered with a blog post.

There are basically two schools of crypto library.

There is the openssl school - here you get a library that was built the same crypto as TSL, the most accessable one of these is node.js’s crypto module.

If you are using browserify, then it also works in the browser.

https://www.npmjs.com/package/crypto-browserify

This was a project that I started, but it was brought to completion by other contributors. It has a massive download count because it’s bundled in browserify - I have no idea how many people are actually using it, but probably lots.

That is probably what you should use if you are trying to be backwards compatible with a legacy system. But the problem is that all the standards in TLS have bugs, and you really need to understand those bugs to not use that in an insecure way. (hint: if the crypto standard has a 3 letter acronym, it has a bug you need to know about ;)

The other school is NACL. This is what I’d recommend if you are building something new, or just learning. NACL is carefully designed so that it’s primitves are simple and easy to use and don’t have the surprising bugs that openssl school librarys do. Where as TSL has a veritible buffet of options, NACL has only the best option in each category (signing, DH, hashing, authenticated encryption). This is better because you don’t have to risk not selecting the best thing. nacl was written by Daniel Bernstien (“djb”) who is widely revered among crypto hackers. Sometimes a little too revered - he is certainly an influential academic crypto expert - there are quite a few, but I suspect the reason he is so revered is that he actually produces code, and not just papers, making him a lot more useful to the hacker community.

nacl links: http://nacl.cr.yp.to/ there is a node binding, http://npm.im/sodium and in the browser you can use https://www.npmjs.com/package/libsodium-wrappers which has been compiled to javascript, and https://www.npmjs.com/package/tweetnacl which has been handwritten. I maintain https://npm.im/chloride which brings together all of these modules with the same api. You can use it to fallback to the small handwritten javascript version, or the large but fast js.

There is actually one more option: modern browsers now have built in crypto support - the WebCrypto api. https://developer.mozilla.org/en-US/docs/Web/API/SubtleCrypto This is also a reasonable option for the openssl school. Here is probably the most contraversal part of the advice I’m gonna give here: I don’t really advise using this. It has a weird api, it uses promises for everything - but the only operation that is so slow it needs to be async is generating RSA keys, or using scrypt or pbkdf2 to strengthen user passwords. But worst, the political process of adding new browser apis is very slow and does not generally result in the best designs. Although WebCrypto is modern, it’s already stuck in legacy because it just exposes the openssl style primitives that browsers use to implement TLS. The one argument for this is that it’s faster than JS.

However, what I am more optimistic about is another developing browser technology - ASM.js and web assembly. This allows you to, end of the day, run C within JS efficiently. This will mean that you can just use nacl in the browser and it will be about as fast as webcrypto. But, there are a lot more interests who will benefit from C in JS (games etc) vs a tiny crypto community who is pushing for WebCrypto. My prediction is that the forces of C get us good browser crypto before the WebCrypto committee modernizes itself.

I’ve talked a lot about speed here - this may or may not be a problem for your application. Encrypting a few messages or verifying a few signatures is not gonna be slow enough to really be noticable - unless you have to do it to many messages in realtime. (exception: performance is critical for scrypt/pbkdf2)

That said and done, the most important question is: What is your motivation? security requires a different sort of thinking than ordinary web development. In short, it requires paranoia. But, a healthy paranoia tempered by a rational evaluation of the actual theats you system faces - more like a survialist, and less like a conspiracy theorist.

Choosing the right library is merely the first step - but understanding why to make that choice is the most important. To do this you find crypto experts and study why they make the decisions they do.

Of course, you can find this every where - and the blogs of things like https://whispersystems.org http://www.tor-project.org/ bitcoin etc are all worth studying, as well as classic papers, such as New Directions in Cryptography http://citeseer.ist.psu.edu/viewdoc/summary?doi=10.1.1.37.9720 (which is much more readable than you might expect! before this paper, cryptography is very different, so they had to explain it simply!)

Asymmetric Cryptography: Works Like Magic

Mon, 29 Dec 2014 09:12:00 +0000

It’s a common complaint that cryptography is too hard for regular people to understand - and that all our current cryptographically secure applications are designed for cyborgs and not humans. While the latter charge may well be correct, I argue that the former most certainly isn’t, because we have been teaching children the basic security principles behind asymmetric cryptography for probably thousands of years.

What am I talking about? A fairly tail called Rumplestiltskin, which is actually about bitcoin!

You probably heard this fairly tale as a child - but let me refresh your memory.

There is a miller, who drunkenly brags that is daughter can spin straw into gold.

probably, he was posting about his half baked cryptocurrency ideas on bitcointalk, and creating money “gold” from pointless work “spinning straw” sounds A LOT like bitcoin mining.

Anyway, the king is very impressed with his story.

the king is a venture capitalist?

And wants to see a demonstration, oh and if it doesn’t work he will cut off both their heads.

I have not heard about venture capitalists being quite this evil, but it seems some of them are into this medieval stuff

Of course, the miller and his daughter don’t actually have the ability to create gold by magic, so they are in big trouble! but just then a magic imp appears.

a hacker, who understands cryptography

The imp says he can spin straw into gold, but for a price: the daughter’s first born child.

in the modern version he wants her naked selfies

It’s a terrible deal, but the alternative is death, so they reluctantly accept. The imp spins straw into gold in 3 increasingly dramatic episodes.

The kind is satisified, and marries the daughter, making her queen.

their startup is aquired

One year later, the first child is born. The imp returns demanding his prize. Because they love their baby, the King and Queen pleads with the imp to get out of the deal. They offer him all their riches, but the imp is not interested! Desperately, they ask is there any other way? any at all? The imp replies, “Of course not! not unless you can guess my True Name”

the true name is actually his private key. If they can guess that, the hacker looses his magical power over them

“Okay I will try and guess your name” says the Queen. The imp just laughs! “you’ll never guess it!” “but I’ll give you three days to try!”

The imp skips off into the forrest, and the queen trys to think of his name for 3 days… but can’t figure it out.

The queen trys to brute force his private key. but there is not enough compute in the entire kingdom!

But then, the a messenger is travelling through the forrest, and he happens past a strange little man, dancing around a camp fire, singing:

ha ha ha!
te he he!
they’ll never guess my private key!
just three days! not enough to begin,
to guess my name is rumplestiltskin!

Being a messenger, he had a good memory for things he heard. When he arrived back at the castle, he mentioned the curious story to the queen.

the hacker had been careless with his private key

When the imp arrived in the morning, the queen greeted him by name. He was furious! He stamped his foot so hard the ground split open and then he fell into the gaping hole, never to be seen again. The king, queen, baby lived happily ever after, etc, etc.

they stole all his bitcoin

The simularities between this fairly tale and cryptography is uncanny. It has proof of work, it has private keys, it has an attempted brute force attack, and a successful (if accidental) end point attack. The essential point about your private key is captured successfully: the source of your magic is just a hard to guess secret, and that it’s easy to have a hard to guess name, but what gets you in the end is some work around when they steal your key some other way. This is the most important thing.

It’s not a talisman that can be physically protected, or an inate power you are born with - it’s just a name, but it must be an ungessable name, so the weirder the better.

“rumplestiltskin” is the german name for this story, which became wildly known in english after the brothers grim published their collection of folktales in the early 19th century, but according to wikipedia there are versions of this story throughout the europe, and the concept that knowing the true name of a magical creature give one power over it is common in mythology around the world.

How did the ancients come up with a children’s story that quite accurately (and amusingly) explains some of the important things about asymettric cryptography, and yet we moderns did not figure out the math that makes this possible this until the 1970’s?

Since the villian of the story is magical, really they have chosen any mechanism for the imps magic, why his name? Is this just a coincidence, or was there inspiration?

The astute reader has probably already guessed, but I think the simplest (and most fun) explaination is the best: extraterrestials with advanced cryptosystems visited earth during prehistory, and early humans didn’t really understand how their “magic” worked, but got the basic idea

To be continued in PART 2…

Social Networks and the Legal Hack

Fri, 26 Sep 2014 12:58:00 +0100

There have been a bit of fuss on my twitters about ello.co a hip new social network that is gonna change everything. They are gonna do things right, no ads, and not sell out their users.

They have a manifesto!

People are certainly trying it out, but the general vibe amongst my friends is that while they are ready for change, they are not yet convinced that we are gonna be treated better this time.

But here is an idea.

Burn their bridges to advertising as a business model, so that they couldn’t take advertising even if they wanted to.

How would you do that? Well, the simplest way would be with a terms of service! A terms of service is a contract with your users. Normally, terms of service are designed to protect the rights of the service provider. ToS are likely to include phrases such as “terms may change without notice”.

But it doesn’t have to be this way. The open source movement, for example, has performed a legal hack where by they used copyright law to undo copyrights. The right to prevent copying becomes the right for anyone to copy. Instead of reserving rights, open source gives them away!

An open source license might say that you are free to redistribute the code, or even to modify and sell it. It might say that the license must not be changed, or it might say that you can relicense the code.

This is not a thing that only opensource can do, services like ello.co and other social networks can also create a contract with their users that protects the users instead of the service.

just include a paragraph something like this

The user maintains ownership of the data they put into ello, including but not limited to content they post, replies to other users, and also metadata such as logs created by interacting with the ello user interface. Access to this material is granted to ello for the purposes of running the site, and to other users that have explicitly solicited that information by “following” in the user interface. Users have the right to refuse information “block” other users that have solicited their information. The data may not be passed on or sold to third parties. ello must honestly represent the information solicited by via the user interface, and must not inject spurious information that the user has not specifically solicited, this includes but is not limited to advertising. ello cannot change the terms of service without explicit consent of the users.

(Obviously get a real lawyer to write your ToS)

Something like that. It lays out clearly what ello is allowed to do with your data and metadata, and it also lays out information ello is to present to you - material created by users you have solicited content from (i.e. followed), and what not to show you, (i.e. advertising)

It may be too late for ello.co to do this now, since they may already have obligations to their investors that giving up a potential business model would conflict with.

But another social network could certainly do something like this. To my knowledge, such a legal hack has not been executed for a social network, so you might be the first!

Why Software is Important

Tue, 07 Jan 2014 10:36:00 +0000

A Hypothesis:

The demand for software increases exponentially, each piece of software enables more software. As we get more software, we want more and more.

Before I learned to program, all I wanted to create was a few games. Now that I am much more experienced, and have written a lot of code, I now have so many ideas that I’ll probably never be able to implement them all. Even if I never have another idea again… As I implement new things I learn new skills and thus get even more ideas.

Here is a short list of some things I’d like to exist:

A role playing game where you cast magic spells my typing “magic words”. Except the “magic words” are actually a programming language, but you won’t realize you have learned to program until you have gotten good at the game.
video chat that uses facial recognition to animate a virtual puppet with your expressions. If you had eye tracking, you could also animate eye contact. When another participant looks directly at your avatar, you should see them make eye contact, but another participant should see them looking to the side (just like in physical space)
An operating system where every application is downloaded over the network and runs inside a secure sandbox. Like tabs in a web browser, except native, and with fully configurable permissions. Also, like an app-store, but without a central authority that rejects applications they dislike.
An open-source webmail that is as good as gmail.
A system of polyfils so that it’s possible to pipe data from the command line directly into widgets in a web browser. I want to be able to pipe a csv into a graphing application in a browser, even if it’s on another computer. I want to be able to write an HTML file, and push deploy it to a particular mobile device. I want to be able to tail -f the status of a check box, and see each update as lines of true and false in my terminal.
A modular database where you plug together primitive parts that are optimized for a particular type of data, (say, mutable data, append only data, graph data, relational data) we have begun to explore this idea with level-db. What if you could have a database that profiled itself and switched out different underlying data stores depending on how you use it, like how v8 optimizes javascript dynamically.
A framework for creating arbitrary applications that are highly secure and are p2p, without a central server.
An HTML5 map application that works offline (that can replicate areas of the map)
A 2d drafting application that where you select shapes and can use those shapes to describe new shapes, say, two points can describe a line, or a circle. and a point and a circle can be used to describe a line that passes through the point and is tangential to the circle.
a secure p2p package manager, like npm (used in the os mentioned above)
A text editor that runs in the terminal, but has all the same keyboard shortcuts as notepad (so user interfaces are consistent, and you don’t end up like vim users, accidentally hitting :wq to send an email)
A system to describe the behavior of FSMs with a regular-expression-like-DSL (regular expressions describe pretty much the same layer of the Chomsky hierarchy as finite state machines, but are much more familiar to application developers) This could be used to verify the correctness of systems that are both async and stateful (something I’ve found very difficult to test)
A pure js implementation of node.js’s crypto module.

Some of these I have started on already. some of these are less ambitious than the others, but all of these would comprise of a considerable number of sub problems, and would also lead to new possibilities, and give me new ideas for new programs to write.

Software does not have a talent drought. The demands for software is increasing faster than new humans are learning to code. Even in just a few decades, most parts of the structure of society have become computer programs. Between the point that photons hit the camera’s detectors to when the jpeg is rendered in your browser, the news is all software. The banking system is all software. The music you listen to is recorded with software, if not made entirely with software.

Yet, despite all this, the quality of most software leads a lot to be desired. If you have had the misfortune to be exposed to the enterprise legacy systems you’ll know what I mean.

The biggest problems facing humanity are really software problems.

Even politics is a software problem. Politicians use spin and scare tactics, instead of making decisions based on objectively measurable data - because there is no software to facilitate the right kind of data gathering and decision making.

A lack of education is a problem, because there we the right kind of educational software.

Currently, humanity stands at a crossroads like never before. Great power and great changes are looming on the horizon - genetic engineering, nano technology. Yet also great dangers - Climate Change, peak oil, and the fact that western governments are getting stupider and more corrupt. We need better software to solve these problems.

The good news is that we have already figured out how to reliable create high quality software. Open Source. Open source has already created all the best Operating Systems, Databases, Programming Languages, and Application Distribution (web browser & virtual machines). The problem is that open source is not yet wildly distributed. Apart from developer tools, most consumer facing or enterprise software is generally not open source.

I think the problem is that there has been no way for consumers to pay for open source. Open source has the illusion of being free - but it’s not. You pay for open source by contributing to open source and by writing new software. The open source community is what anthropologists call a “gift culture”. This happens often when there is an abundance of resources (because of the ever increasing demand for software). Instead of gaining power and respect for what resources you control (such as in a commercial culture), in a gift culture individuals gain respect and status by what they give away. Often, in a gift culture, the most respected individuals are by no means the wealthiest. This is certainly the case in open source!

What we need is a way to turn open source inside out, or at least expose a way for people who do not have technical skills to meaningfully contribute to it. Paying for the software in the traditional way is out of the question. To be able to sell something, you need to have the power to withhold it until you have received the payment. This is impractical with software because it’s so easy to copy. If you want to sell software in a traditional way, you have to add copy protection features, which actually make the software worse! On the app store you can purchase software in single units - but to be able to do that, apple controls the entire ecosystem all the way down to the physical hardware!

But there is one way you can reliably withhold software, and thus effectively demand payment: There is no easy someone non-technical to get software that has not been written yet.

You can’t sell software, but you can sell a promise to write software.

Before software is written is hangs there only as a lingering possibility. It must be mined out of the ether by highly trained monkeys in a process where they drink coffee, stare at screens, and hammer away at some very fancy typewriters. There is pretty much no way to get software that to feed these super-monkeys, or to be one. Being one is a curse as well as a blessing because the rate at which primates can create software is really rather low, not more than a few hundred lines of tested, working code a day on average. It’s totally feasible for one person to grow all the vegetables you need to live, but you could not in your lifetime create all the software one would like to implement.

Fortunately, it’s possible for these monkeys to work in teams. In fact, it’s really quite simple, because they actually work better if they are just let to themselves and given only the most gentle direction. If you try and dictate what they do they’ll only resent you. All that is needed is the right sort of guide to encourage them to communicate in the most effective way, and make it more rewarding for them to work together than to work on their own.

This is essentially what git, github and npm accomplish. Over the years, we’ve made pretty good progress in this area, but this has only increased the insatiable demand for software, and raised our expectations for software quality and functionality.

We have such need for software that we really need to look for a force multiplier, that increases our ability to create software. We don’t need to find another low hanging fruit, what we need is a ladder.

I have an idea for a ladder. Will be announcing it shortly. Watch this space!

(here it is: feedopensource)

Some Thoughts on the Economics of Software Development

Thu, 02 Jan 2014 12:21:00 +0000

The problem with software development, is that the ways used to fund it are either more suited to physical products, or are otherwise out of line with the interests of the users of the software. Here are my thoughts on the ways this fails, and also an idea for another approach.

A Summary of the Current Methods

There are currently only a couple of ways of funding software:

VC-funded free service
proprietary software licensed for a fee
open source, funded by charity, or by the creative drive of it’s author.

Each of these methods create a different kind of software. Essentially, different sources of funding mean different motivations, and these result in different decisions by the developers.

The “Free” Service

VC-funded software seems very good at first, but, they aren’t really doing it for free.

The investors are looking to make money, and there is often a “business model” question hanging over the software. This question is nearly always resolved by changing the software from something that enables users, to something that manipulates the user. If this is the sort of software you would like to use (and be used by), then you are already well catered to! Your Welcome!

Private Source

Someone builds the software, then you buy it from them.

What could be simpler? This works well with physical goods, because the exchange is easy to enforce. For example, I have made a bicycle, and you want to buy it. You give me some pieces of paper with pictures of George Washington, or Chairman Mao or whatever, and I give you a bicycle. This is naturally in line with the physics of physical items, but software is totally different. Instead of giving you the bike, I copy the bike to you. Now there are two copies. What’s more, you can also copy the bike to someone else. Now there are a lot of people riding my bike, without me being able to recover the value that I think I deserve for creating it.

So, to make the private source model work, you have to add extra copy protection “features” to make the software behave more like a physical good. Like turnstiles on a subway, these copy-protection features prevent you from accessing the natural physics of software. This takes extra work, and so you end up paying not just for the product itself, but also for the work to cripple the product so that it’s more difficult to copy.

If the value of the software is high enough (enterprise), then they can just use lawyers instead of copy protection. I’m not sure which is worse.

This is widely used for games, or enterprise applications. Apple’s App Store is a great example of an elaborate system for making software difficult to copy.

You purchase a crippled phone that can only install software that apple likes, and then you can pay for software, and also 30% to apple for the service of stopping you from being able to install anything else.

Open Source

Instead of going to the hassle of charging for software, open source just gives it away.

This eliminates the costs of erecting turnstiles. It also reduces the cost of evaluating software. You don’t have to pay for something to try it out. Open Source has created all the best operating systems, programming languages, developer tools, and databases.

But the scope of Open Source is pretty much limited to this. Open Source is created by enthusiasts, and it does get paid for, but intangibly.

A project usually starts because someone thinks it might be fun to implement. But then, if that project becomes useful, it becomes in the interest of other well funded parties. Any large project requires an extraordinary attention to detail and refinements.

It would not be possible to pursue such a project at hobby scale. though, still, there is a huge amount of valuable software essentially created by hobbyists. Tragically, these hobbyists may work a day job writing “free” services or private software.

Moving Forward

There have recently been some attempts to move beyond the hobby economics of open source. Each of these are interesting in their own way.

gittip

gittip.com enables you to anonymously donate to someone. This is the digital equivalent of busking.

You can easily give a small amount of money as encouragement to an artist who’s creating things you enjoy. But this doesn’t really influence them in any way. This is an interesting model, but not many people are earning enough to survive like this (and thus devote themselves fully to their art).

By the way, if you enjoy my creations, please feel free to offer me a token of your appreciation: gittip.com/dominictarr

Kickstarter, Indiegogo, etc.

Another approach is crowd-funding the development of a project.

The developer makes a bunch of promises of what they want to deliver, and how much they think it will cost. People either fund it or not. And then, development begins. If you have been following any software projects like this recently it won’t be hard to think of examples of one that hasn’t really gone according to plan.

All the crowd-funding platforms and geared towards a single, large, funding round. So, all promises and plans must be made up front, well in advance.

The problem with promises made well in advance is that they are hard to keep. Not necessarily because you didn’t mean them, but just because the situation changes, and when you start coding you learn things you didn’t know when you started.

Like representational democracy, you don’t know if the party you elect (fund) will keep their promises, or even are capable of achieving them.

Essentially, Kickstarter forces you to use the waterfall model. The waterfall model is widely understood to result in low quality software that is delivered late and over-budget. Quality Software must be built iteratively, changing the plan as your learn things, but the economics of crowd-funding platforms prevent this.

Bountysource

Bountysource takes the opposite approach of Kickstarter.

Instead of funding a whole project all at once, you can fund a specific issue, or feature that you want. It’s really too early to say what will happen with Bountysource, but my feeling is that Bountysource is too specific where Kickstarter is too vague. The features of a software project are not isolated. You can’t just buy a feature or bug fix. On the one hand, casual users may not understand which issues are really important, and on the other there may be a refactor that simplifies multiple features, or eliminates multiple bugs. At best, this may be suitable for the developer tools that open source is already good at.

If open source is to go truly mainstream, it needs be able to create software that is used by people who are not themselves developers, or even particularity competent with computers.

One More Way!

There is one more way software is funded.

If you have a software problem that is particular to you, or just want your own thing, you can hire a consulting company and get them to make you a bespoke system.

This can actually work pretty well, provided that communication between the developers and the clients is good, and an iterative process of building a prototype, testing it against client expectations, and refining/rebuilding the design. This process is often known as Agile Development.

The developers and the clients both have power in their relationship. The developers have the technical expertise to write the software, but the clients have the checkbook. If a project is going badly, they can potentially abandon it and find new consultants.

A lot of software is built this way, but it’s not scalable. The weakness is that is that it must provide sufficient value to a single party, who must be willing to foot the bill entirely. It doesn’t support the case where a small amount of value is provided to a large number of people.

But what if it could?

My idea

What we need is something that aligns the interests of the developers and with the interests of users. Something that works with the natural physics of computers and data, not against it. Something that scales, so that you can use to fund end-user software - not just developer tools.

Something halfway between Kickstarter, and Bountysource, but with the best parts of Agile development. Development and funding must be iterative. The developers should not have to make promises that will be hard to keep.

Instead, developers should build trust with the clients by making steady, incremental progress. If the progress is in the wrong direction, or the developers turn out to be incompetent, then clients can abandon the project without having over-committed.

If a project goes well, the clients can continue to fund the project. Since the uncertainty decreases as the developers build the clients trust, they could potentially fund much more than they would with a single round of funding, where they have to balance their desire for the promised features with their estimation of the projects eventual success.

I am currently working on a prototype of this idea, Watch this space!

Mad Science Architectures for Totally Distributed Applications.

Tue, 22 Oct 2013 20:29:06 +0100

Mad Science Architectures for Totally Distributed Applications.

What is scalability, anyway?

Thu, 10 Oct 2013 16:06:41 +0100

What is scalability, anyway?

leveldb - followed by part2...

Thu, 10 Oct 2013 16:02:28 +0100

leveldb - followed by part2 http://www.youtube.com/watch?v=a8rTAIWjIpw

data, events, streams at webrebels 2013

Thu, 10 Oct 2013 16:00:11 +0100

data, events, streams at webrebels 2013

watch live demos, and event the presentation software...

Thu, 10 Oct 2013 15:56:27 +0100

watch live demos, and event the presentation software fail (based on https://github.com/jed/weenote) in the end I save the day by making a joke about java.

the database of the future

Thu, 18 Jul 2013 07:16:30 +0100

the database of the future

wizard talk.

Thu, 18 Jul 2013 07:09:42 +0100

wizard talk.

distributed systems for realtime applications.

Thu, 18 Jul 2013 07:09:15 +0100

distributed systems for realtime applications.

cyberhoboing with dominic tarr

fairly tale cryptography 2: hashes

Video

Video

Design Challenges of Decentralized Systems (at Data Terra Nemo)

pull streams pull-streams are a very simple streaming primitive, but can do everything that streams...

pull streams

sources

sinks

throughs/transforms

expanding or shrinking streams

History Of Streams

The history of streams

contents

prehistory

problems with early node streams.

problems with streams2

streams 3

stream 4? no.

Video

Which JS crypto library should I use?

Asymmetric Cryptography: Works Like Magic

Social Networks and the Legal Hack

Why Software is Important

Some Thoughts on the Economics of Software Development

A Summary of the Current Methods

The “Free” Service

Private Source

Open Source

Moving Forward

gittip

Kickstarter, Indiegogo, etc.

Bountysource

One More Way!

My idea

Mad Science Architectures for Totally Distributed Applications.

What is scalability, anyway?

leveldb - followed by part2...

data, events, streams at webrebels 2013

watch live demos, and event the presentation software...

the database of the future

wizard talk.

distributed systems for realtime applications.