Mitar's Point

Slovenian TDM exceptions and content available online

Wed, 24 Sep 2025 08:33:42 +0000

Does lawful access under the Slovenian TDM exceptions also include content that is freely available online?

To answer that question, I submitted a complaint to the European Commission in June 2024 alleging that the Republic of Slovenia had not properly implement the Directive (EU) 2019/790 on copyright and related rights in the Digital Single Market (hereinafter: the CDSM Directive). The complaint argued that Articles 57.a and 57.b of the Slovenian Copyright and Related Rights Act (hereinafter: ZASP), which regulate the exceptions for text and data mining (hereinafter: TDM exceptions), fail to explicitly provide that the notion of “lawful access” includes content that is freely available online. The European Commission did not initiate infringement proceedings against Slovenia, reasoning that although the Slovenian implementation does not explicitly mention freely available online content as part of the notion of lawful access, it also does not expressly exclude it. Consequently, it may be interpreted that freely available online content forms part of lawful access under the TDM exceptions in ZASP.

For researchers and others building artificial intelligence models in Slovenia, it is of significant importance whether the TDM exceptions in Articles 57.a and 57.b of ZASP permit text and data mining on freely available online content, even though the Slovenian legislator did not expressly include such content when defining “lawful access” in ZASP.

Indeed, the Slovenian legislator has defined the notion of lawful access in ZASP more narrowly than the European legislator in Recital 14 of the CDSM Directive. Under ZASP, lawful access is understood to mean access based on free and open licenses, contracts, or other legal grounds such as copyright limitations and exceptions, or special legislation (e.g. the Legal Deposit Act). The wording does not explicitly extend to content that is freely available online. Nevertheless, it remains possible that, in the event of litigation, Slovenian courts would interpret ZASP more broadly and, taking into account Recital 14 of the CDSM Directive, conclude that lawful access also encompasses freely available online content.

To enhance legal certainty for researchers, I raised the question of whether “lawful access” under Articles 57.a and 57.b of ZASP includes freely available online content in the complaint (Ref. CPLT(2024)01829) of 30 June 2024 against the Republic of Slovenia for alleged improper transposition of the CDSM Directive. The European Commission concluded the procedure on 2 April 2025 and issued its position on the interpretation of “lawful access” in Slovenian law. The Commission decided not to initiate infringement proceedings against Slovenia, concluding that although the Slovenian implementation does not expressly mention freely available online content as part of lawful access, neither does it explicitly exclude it. It therefore considered that the CDSM Directive had been implemented into ZASP in accordance with its purpose as reflected in the recitals.

As a result, it shall be inferred that the relevant provisions of ZASP should be interpreted in light of Recital 14 of the CDSM Directive, which suggests that lawful access under ZASP may indeed include access to content that is freely available online.

dinit

Sun, 09 Jul 2023 18:49:53 +0000

I made an opinionated init specially for Docker containers: dinit. It supports running and managing multiple programs, their stdout and stderr, and handling signals, all in the way more suitable for Docker containers. Using it means no more resource exhaustion from zombie processes, data loss by uncleanly terminated containers, stray daemonized processes that keep running, or hidden issues because failed processes inside a container gets restarted instead of taking the whole container with them.

regex2json

Wed, 14 Jun 2023 09:35:07 +0000

I made a simple tool regex2json in Go to convert traditional text-based (and line-based) logs to JSON for programs which do not support JSON logs themselves. But the tool is more general and can enable any workflow where you prefer operating on JSON instead of text. It works especially great when combined with jq. I was inspired by this blog post.

Nonenumerable database

Thu, 23 Jun 2022 22:29:17 +0000

Sometimes it is useful to first define an abstraction. A good abstraction can reveal that some problems could be solved by the same solution. A good abstraction also makes it easier for us to communicate about the solution and understand its properties.

One such abstraction which I think should be defined and named is nonenumerable database. A database where you can query a value by a key, but you cannot enumerate all keys (nor all values). Even a database operator/administrator is not be able to do so.

At the same time, some problems are more intuitive than others and can help us better understand or even inform the abstraction. We will look at three examples where nonenumerable database can help: DNS/DNSSEC, gun ownership registry, and voting.

Not just a key-value database

A nonenumerable database is not just a key-value database and does not just provide an efficient and durable way of storing and retrieving the value given a key, but it also prevents one to learn about all keys stored in the database. Without knowing a key it is also not possible to obtain the value or really any value from the database. All this holds even for a database operator/administrator (or cloud provider). This last requirement should hold even when the traffic to the database is observed. Many key-value databases still support various forms of querying, but a nonenumerable database by design does not.

DNS, DNSSEC and enumeration of names in a zone

Domain Name System Security Extensions (DNSSEC) adds cryptographic authentication and integrity to DNS, preventing various attacks when resolving a domain name to its IP. The issue is that because DNSSEC also has to cryptographically prove that a name does not resolve (when it does not), it opens a door to domain walking: enumerating all names available.

Could DNS then be seen as an example of a nonenumerable database? Given interest in addressing domain walking issue introduced by DNSSEC it seems many would like to see it as such, but it is not quite: longer names might not be enumerable, but for short names one can enumerate them simply by trying them (e.g., aa.com, bb.com, cc.com). Note as well that, in DNS, author of the value (DNS record) picks the key (name) as well: this is probably a realistic requirement for a nonenumerable database despite allowing the author to subvert security guarantees otherwise afforded by the database. This informs us that for our nonenumerable database abstraction to be practically secure we have to put some guidelines/limits on the minimal key size/entropy and/or maximal allowed query rate.

DNSSEC made domain walking easier than just trying names but it also raises a question: what should a nonenumerable database return when a key does not exist in the database? DNSSEC requires an authoritative (negative) answer, but our definition of nonenumerable database is silent on that. Moreover, it does not even require authenticated answers. Should it?

DNSSEC does not provide confidentiality and large DNS providers (e.g., Google at 8.8.8.8 and CloudFlare at 1.1.1.1) probably know all names stored in DNS, without having to enumerate them themselves. Users did that for them. A nonenumerable database could prevent that.

Because DNS is also a distributed database it might be hard to guarantee all properties of a nonenumerable database, but future implementations of a nonenumerable database might still inform future versions of DNSSEC.

Gun ownership registry

In USA, there is no searchable database of gun owners, no national gun registry. Tracing a gun owner from the serial number of the gun is all done by hand, searching through boxes and boxes of forms. The gun lobby worries that a searchable registry would give the government a tool to confiscate guns.

A nonenumerable database addresses both the gun lobby’s concerns and allow fast tracing of a gun owner when trace is required. In fact, if the worry of the gun lobby is that the government could confiscate guns of everyone, then current approach with boxes full of paper forms does not really prevent that: the government can still go box by box and confiscate guns from everyone found in the box. It would be not very efficient (each box probably contains forms from people living in different places) but it is doable. With a nonenumerable database even that is not be possible.

On the other hand, once a gun is found and tracing the owner is required, lookup by gun’s serial number would be very fast. Moreover, having database online could allow sellers to directly input ownership information immediately after the sale, which would help in cases when a crime happens soon after selling the gun. This could be a good use case for a blockchain solution, where records could also be updated in a decentralized way as guns are resold.

To make enumeration harder by simply trying serial numbers in order, guns’ serial numbers should probably be changed to something closer to UUID, e.g., a random 128-bit number.

Verifiable voting

Online voting is hard: it is hard to ensure anonymity of votes, it is hard to prevent vote-buying and coercion, it is hard to prevent compromised identities and computers, it is hard to prevent phishing, it is hard to prevent compromised servers and manipulating the vote tally, etc. One piece of a solution might be a nonenumerable database.

Simply storing votes into a nonenumerable database is useless by design: one cannot tally such votes. Instead, you could store receipts for cast votes and proofs that they have been included in the tally, keyed by a ballot number.

Only knowing the ballot number one could check the cast vote. Based on the guarantees of the nonenumerable database, just checking the vote using the ballot number does not reveal your identity. If the vote does not match what you cast or if the proof is invalid, you can make the ballot number public and others can check as well that there was something wrong during the vote.

A downside is that it makes vote-buying easier: a voter can give their ballot number to the buyer for them to verify the (bought) vote. Because it is hard to obtain a ballot number of someone else, the buyer can have high confidence they are spending their money well.

Possible design

One possible design of a nonenumerable database is to use (a hash of) an encryption key as a database key and the corresponding value is then encrypted with that key. Without knowing the (encryption) key, you cannot obtain the corresponding value. You cannot enumerate values as well.

As noted above, we should also allow a mapping between an arbitrary key and the corresponding encryption key. This would allow storing a DNS name which maps to an encryption key which can decrypt the corresponding IP address stored as a value. But how do we store this mapping so that it cannot be used to enumerate the database?

Moreover, our definition of a nonenumerable database requires that even when the traffic to the database is observed, its guarantees should still hold. Both this and the mapping are the core design issues of a nonenumerable database. Maybe they can be addressed by using a trusted execution environment.

When interoperability is needed between multiple parties using a nonenumerable database, a blockchain-based solution might be reasonable. Especially if querying such blockchain-based nonenumerable database requires a non-free transaction: then there are both technical and economical limits on query rate which helps prevent enumeration by trying keys.

PeerDoc – Scaling real-time text editing

Tue, 12 Oct 2021 20:27:30 +0000

PeerDoc is a collaborative real-time rich-text editor with undo/redo, cursor tracking, inline comments, permissions/sharing control over documents, a change history. Things you would expect from a modern collaborative editor on the web.

But the main difference is that it combines two types of collaboration:

real-time collaboration between collaborators on the draft of the document (push-based collaboration)
fork and merge request style of collaboration with others, allowing collaboration to scale beyond a small group of collaborators (pull-based collaboration)

Push-based and pull-based collaboration

Push-based collaboration is collaboration where a change can be made directly (pushed). Google Docs and Wikipedia (and wikis in general), for example, work this way by default. If the change is deemed problematic by others, it can be reverted or further changes can be made to fix the problems.

Pull-based collaboration instead works by offering a change that must be approved before it is integrated (pulled). GitHub has popularized this model through its pull-requests (which GitLab calls merge-requests because integrating a change is called merging).

Both types of collaboration are widely used. Both offer some advantages and disadvantages. Push-based collaboration:

Works best with a small group of collaborators (Google Docs limits collaborators to 100) who generally trust each other (Wikipedia needs editors who patrol recent changes to detect vandalism).
It has less procedural overhead and is generally easier for a novice to understand and do.
When done in real-time, potential conflicts (due to changes to the same section of the document) can be resolved more easily and quickly, often without having to coordinate the resolution at all.
Each individual change is made by one user and there is no collaboration on the change itself. This can be less friendly to novice collaborators, who might prefer to have some help with their change. If their change is then further changed or even reverted, they may feel hurt.

Pull-based collaboration:

Can scale up to many collaborators who can independently offer changes.
Often, changes can be made, discussed, and iterated upon by collaborators, before they are integrated.
Each change requires a review process before it is integrated (or rejected), which can become a bottleneck. This process must be learned by novice collaborators before they can contribute.
Multiple independent changes may be made to the same section of the document and resolving conflicts is difficult to coordinate (since each change is usually part of a separate review process) and lengthy (can be repetitive).
Collaborators need only trust those reviewing and approving the changes.
There is usually a time and place to discuss the changes next to the changes themselves. These discussions can remain available for future reference.

Combining push- and pull-based collaboration

PeerDoc supports real-time editing for those who have permission to edit the document directly (given to them by the initial editor of the document). Others (if they have permission to see the document) can fork the document and edit the fork (the user who made the fork can also grant permission to other users to edit the fork). Later, an editor of the fork can suggest that all changes from the fork be merged back into the parent document. If an editor of the parent document approves, the merge is performed. A fork itself can also be further forked. This can soon lead to a tree of documents like this:

PeerDoc uses the great ProseMirror library for its editor, which already includes features for real-time collaboration. For a single document this works great, but making things work across a tree of documents requires a precise cross-document sharing of changes.

When a change x in Document A is made, it is made after all existing changes a:

Change x then has to propagate to Fork B and Fork C, integrating with other additional changes already there, changes b and changes c, respectively. This is done by rebasing those additional changes on top of changes from Document A. So all changes from Document A are always the oldest in the history, with forks adding changes on top of that history. Rebasing changes b on top of change x makes changes b’, and rebasing changes c on top of change x makes changes c’. After rebasing, changes b’ and changes c’ follow after changes a and change x.

Integrating changes from Document A into Fork C changes Fork C. Because of that, those changes (change x and changes c’) have to propagate in turn to Fork D. This is where rebasing becomes more complicated. Rebasing requires branches to have a shared base, but changes d have base a + c while to-be-rebased changes d’ have base a + x + c’. To get to a shared base we start with a + c + d and then invert (undo) changes c and changes d and add them to get a + c + d + (-d) + (-c). Then we add new change x from Document A to get a + c + d + (-d) + (-c) + x. With that we rebase changes c into changes c’ (which should produce same c’ as we got for Fork C) and now we can rebase changes d into changes d’. This gives us a + c + d + (-d) + (-c) + x + c’ + d’. We now add changes d’ as rebased changes to a + x + c’ to get fork’s changes a + x + c’ + d’ and forget those temporary transformations we made.

Merging a fork into the parent document is easier. First, we have to make sure all parent changes are propagated to forks. Then, merging is simply adding additional changes to the parent document. For example, merging Fork B into Document A would mean that we add changes b’ to changes a + x, making Document A have the same content as Fork B (until another change to Document A). Merging changes Document A, so now we have to propagate changes b’ to other forks, i.e., Fork C (and in turn Fork D).

This explains what is happening on the server-side. Client-side also has to adapt to these changes. For example, an editor which has Fork D open and displayed it is showing changes a + c + d + y, where changes y are local changes the editor might have and have not yet been send to the server-side. When server-side propagates change x and then sends updated Fork D to the client-side, the editor has to update showing changes a + c + d + y to changes a + x + c’ + d’ + y’ while preserving any other editor state. PeerDoc does not yet implement this and just resets the editor.

Discussion

As you can see, cross-document sharing of changes can be very active. Changes can be small (e.g., the user adds an extra character to Document A in real-time, which then propagates to the forks) or large (e.g., the user merges a large set of changes b’ into Document A, which then propagate to the forks). All of this not only puts a burden on the server, but can also confuse users, since large chunks of content can be changed while they are editing the document. Moreover, the interaction with existing changes (e.g., the interaction of changes b’ with changes c’ after changes b’ have been merged into Document A and propagated to Fork B) can sometimes be surprising. All conflicts between changes to the same section of the document are resolved mechanically, without regard to how a human understands the content. When editing in real-time, this is fine because editors can usually observe and adjust to the results of conflict resolution in real-time (e.g., “Oh, I got pushed into a bulleted list by another user, let me get out before I type more.”). Furthermore, conflicts tend to be small anyway. However, when mechanically resolving conflicts for larger chunks of content, this can lead to surprises and require followup fixes.

When we think about collaboration, we also need to think about the process and the goal of collaboration. Who can edit and when, and in the context of PeerDoc, who can edit and when in real-time, and who can edit and when using forks and merges. One approach is to decide this based on trust between users, and edit in real-time if they have high trust, and edit through forks and merges if they do not. But if the collaboration is part of a wider community, we may need more control over when something can happen. For example, should any editor of the parent document really be able to approve the merge request, or should there be a process for that? After the merge, can the parent document editors continue to edit the document directly, possibly modifying the just merged (and community-approved) change?

PeerDoc supports being embedded inside another tool, which can then control the collaborative process more precisely. Moreover, it can operate in two different modes. In one mode, documents can be forked and merged at any time, and editors can still edit documents directly. In the other mode, documents can be marked as published and only then can they be forked and merged, but they can no longer be edited directly. The idea is that editors can directly edit the document only while it is not yet published (it is a draft). Then they publish the document and others can see the document, fork it, edit the fork in real-time, and then suggest changes that can be approved or rejected by the editors of the document.

When a fork is merged, it can no longer be edited and changes from the parent document are not propagated to it. It remains frozen and archived. If fork is not merged, it can be published instead. What this means for the community depends on the community, but it could mean that the fork editors want to make it visible to the community as an alternative to the parent document (rather than retracting the proposed changes).

Deployment

PeerDoc has been used by The Federal University of São Paulo, Brazil, to write their 5 years strategic plan allowing everyone at the university to collaborate and contribute. It has been used integrated with AppCivist which guided the community through multiple phases of participation.

Collaboration is not easy

PeerDoc does not have all the answers (yet) about how exactly to best use a combination of these two types of collaboration. Are forks created by working groups that meet and collaborate and then propose changes to the main document? How are the proposed changes approved? Does an editor of the parent document make a decision or is there some other decision-making process, e.g., voting? What happens if a change is not approved? Does the community vote to choose between the parent document and a fork? Or is the fork published as an alternative document? Or is it simply forgotten? Is PeerDoc used as-is, or is it integrated with another application that can help organize the community and coordinate the process? PeerDoc currently rebases all changes and maintains a linear history of all documents, but would a branched history without rebasing be better when trying to understand how documents came about? Can one effectively proceed in followup fixes of poorly resolved conflicts without having access to both versions (parent and fork) of the content prior to the merge, or is it sufficient to see only the (poorly) merged result? More experimentation is needed to answer these questions. Try it out and please report your experiences.

JSON decoding in node is fast

Sun, 09 May 2021 08:34:27 +0000

I made a benchmark comparing how long it takes to decode JSON in node.js and Go. As I have noticed in the past, JSON decoding in node.js is really fast.

I used node v16.0.0 and Go v1.16. For Go, I compared both the structure of the data (providing a struct type) and not knowing it (using map[string]interface{} interface). Moreover, I compared standard encoding/json and jsoniter. I measured on a range of JSON files from 10 regular and 10 array fields (~23 KB) to 200 regular and 200 array fields (~1.3 MB). I repeated each run 10 times and averaged.

Results

	node	go interface std	go interface jsoniter	go struct std	go struct jsoniter
10	0.00008	0.00016	0.00011	0.00013	0.00060
20	0.00016	0.00036	0.00022	0.00025	0.00063
30	0.00022	0.00058	0.00035	0.00040	0.00067
40	0.00042	0.00088	0.00055	0.00057	0.00072
50	0.00058	0.00119	0.00075	0.00079	0.00080
60	0.00066	0.00154	0.00104	0.00101	0.00088
70	0.00059	0.00200	0.00135	0.00129	0.00092
80	0.00073	0.00239	0.00167	0.00158	0.00100
90	0.00100	0.00289	0.00205	0.00189	0.00111
100	0.00193	0.00333	0.00243	0.00224	0.00120
110	0.00324	0.00388	0.00282	0.00262	0.00133
120	0.00312	0.00445	0.00323	0.00300	0.00143
130	0.00356	0.00563	0.00403	0.00351	0.00153
140	0.00349	0.00594	0.00449	0.00388	0.00169
150	0.00366	0.00673	0.00576	0.00432	0.00183
160	0.00411	0.00779	0.00665	0.00483	0.00200
170	0.00411	0.00906	0.00708	0.00540	0.00217
180	0.00438	0.01020	0.00787	0.00597	0.00227
190	0.00493	0.01057	0.00832	0.00652	0.00245
200	0.00528	0.01143	0.00905	0.00711	0.00267

Parsing JSON in node is really fast and beats Go when no structure about JSON is provided (which is how parsing in node is done). Only with information about the structure, the non-standard jsoniter library is faster.

Google search across languages

Fri, 08 Jan 2021 07:54:22 +0000

In 2013, I wrote how crowdsourcing could help break out of the language bubble we all find ourselves in when we search online. How our search results only come from the language we write our search keywords in. But since then, machine translation between languages has greatly improved. So why are we still in our language bubbles?

There are some rays of hope. Google is reportedly able to return you results from other languages when there are no good hits in your primary language. I have to say, though, that I have never experienced this. Maybe because I usually search in English. I would guess that when you search in smaller languages, you sometimes get English results added.

But I want the opposite. I want to see ideas and thoughts and solutions that might be available in other languages, the languages I do not speak, when I search in English. I want diversity. I think all the pieces to build this are available. Why is this not already available? Am I overlooking any obstacle to this?

Store IDs in MongoDB as binary or as string?

Sat, 07 Nov 2020 07:51:12 +0000

I was curious if MongoDB compression can efficiently store IDs if they are represented as string instead in a more compact binary form. So I made a benchmark and measure compression performance of three available compressors: zlib, snappy, and zstd.

Results for MongoDB 4.4.1 of storing 128 bit random values (e.g., UUIDs) as binary (16 bytes) or as base-58 encoded (22 characters):

	Binary none	String none	Binary snappy	String snappy	Binary zlib	String zlib	Binary zstd	String zstd
size	3100000	3697229	3100000	3697196	3100000	3697150	3100000	3697196
count	100000	100000	100000	100000	100000	100000	100000	100000
avgObjSize	31	36	31	36	31	36	31	36
storageSize	3645440	4243456	2404352	3022848	2142208	2203648	1892352	2080768
totalIndexSize	2523136	3325952	2519040	3330048	2519040	3334144	2531328	3330048
totalSize	6168576	7569408	4923392	6352896	4661248	5537792	4423680	5410816

zstd compression looks really good. Moreover, it is clear that storing values as binary is more efficient than as string, even with compression, because compression can compress also binary representation despite values being random. The most compressed string size (zstd, 5410816 B) is still larger than the least compressed binary size (snappy, 4923392 B). Do note though that zlib compressed string (5537792 B) and zstd compressed string (5410816 B) are smaller than uncompressed binary (6168576 B), meaning that those compression algorithms can recover storage lost in string representation. But given that they can compress binary values even more, it seems there are still things to improve in those algorithms.

Note: Compression algorithms generally perform poorly on small data and here we had very small object sizes. This means that insights here cannot be generalized to performance with larger amounts of binary (or string) data stored in MongoDB. (MongoDB does combine objects into blocks to compress to alleviate this issue.)

Can humans grow up in zero gravity?

Sun, 25 Oct 2020 06:54:33 +0000

Gravity seems to be our first and most important teacher. Patient, consistent and always present. Children can learn cause and effect through it. What happens if I lift an object and let go? Over and over again. On repeat. Always the same thing.

When we think about life in space. Can we raise children there? Can children grow up in zero gravity? Can they develop their mental abilities without having this teacher around them? Or will they develop in other ways? A different kind of logic?

Database-abstraction APIs should not exist

Sun, 04 Oct 2020 23:44:26 -0700

Database-abstraction APIs where you write a query using the host programming language should not exist. Or more precisely, should not have to exist. For example, in Django you can query the database using the following Python code:

Entry.objects.filter(is_draft=True)

Which Django translates (roughly) into the following SQL:

SELECT *
  FROM blog_entry
  WHERE is_draft = true;

But why we cannot write SQL query directly as an SQL query, while retaining all other features Django offers through its database-abstraction API (database agnostic code, inputs to queries and outputs from queries being Python objects, etc.)? I claim there is no reason anymore for that.

Babel

I have confidence in this claim because of the success of Babel JavaScript transpiler and the ecosystem around it. Babel can take a newer version of JavaScript as input and converts it so that it can run in older JavaScript runtimes. It maps new features to old ones. It allows one to use syntax of new or even just proposed features of the JavaScript language. It allows experimentation around the language.

The approach it takes for this is that Babel first parses the input code into abstract syntax tree (AST), an internal representation of the code, then converts the tree so that it represents the code which can run in older JavaScript runtimes, and then renders AST back to textual representation of the code.

Babel is successful because it has an ecosystem of plugins that make it even more powerful, allowing integration with various other libraries and programming language patterns, so that you can tailor programing in JavaScript to your needs and preferences.

Why we have database-abstraction APIs?

Such high-level database-abstraction APIs exist in almost all programming languages and are often paired with object-relation mapping (ORM) capabilities to present data from the database as objects in the host language.

What features do we currently get (or are at least promised) from these database-abstraction APIs?¹

Database engine independence: the Django query above can be converted to SQL dialects for many popular database engines without having to change Python code.
Queries are part of the rest of the code, which allows tight integration with the host language. For example, type checking of the code can type check queries as well, which is not possible with opaque SQL queries represented as strings in the host language. Moreover, other types of query validation can be done before the query hits the database engine.
It is easier to organize and compose such code. E.g., often there is support for various hooks to register code to be called before and after queries. Such features allow better modularity of code based on concerns (a module for logging all queries, a module for checking permissions, etc.). Such hooks are also written in the host language itself and not some database engine specific language.
Support for custom data types which are transparently converted to and from the underlying data types supported by a database engine.
Can have a more user friendly and familiar syntax than SQL, especially if you do not know SQL.

But in practice there are many limits:

Database engine independence means that one can use only common/shared features between different database engines and if one wants to use a particular special feature, they have to use a “raw query as string” escape hatch. Which of course makes the code depend on a particular database engine and you loose many other features (like query validation/type checking). Database-abstraction API can add high-level support for such special feature and map it to different database engines, but wouldn’t it be better if you could just write the raw SQL query without loosing other features?
Because queries can be very different and inputs and outputs to queries even more so, high-level database-abstraction API functions are generally very broadly typed, allowing inputs with types which do not even match the query, failing only at runtime (e.g., *args, **kwargs in Python).
Hooks in the host language are nice, but because they are run outside of the database engine an issue of recovering from failures arises. What if a database transaction completes but a hook fails? What if a database transaction fails but a hook succeeds? In general synchronizing between such external hooks and a database engine can be tricky.²

Parsing SQL into AST

If you look at the Python code example above, it uses Python to generate AST and then renders this AST to SQL. But why not use SQL itself to generate this AST? And then use plugins to transform AST to implement various features we want?

First, it is important to note that there are two types of SQL statements: data definition statements and queries. Parsing data definitions statements (e.g., CREATE TABLE) allows us to extract typing information about data. We can use this typing information to type check queries as well. Moreover, we can use that information to know how to convert query results to host language objects. Depending on the integration with the host language, type checking and validation of both queries and argument values might be possible at the compile time, or at least at runtime before even hitting the database engine.

Examples

Having plugins which can operate on AST allows developer to write simple queries which are then transformed further. This achieves decoupling of concerns where original query deals with what data is needed at that point in the program while how is that data obtained, queried, structured in the database, or if query is even allowed, can all be delegated to plugins.

Permission checks

If original simple query looks like:

SELECT *
  FROM blog_entry
  WHERE is_draft = true;

But only the owner of a blog entry should be able to see it while it is draft. A plugin for row-level permission checks could transform this query automatically into:

SELECT *
  FROM blog_entry
  WHERE is_draft = true
    AND owner_id = current_user_id();

Furthermore, the plugin could change definition of blog_entry table to include necessary owner_id column.

Doing such checks manually is tedious and error prone, especially when you have nested queries or joins and you have to make sure you add check to everywhere.

We can expand this example further if we introduce groups of users and allow multiple users to own a blog entry:

SELECT DISTINCT id, title, body, pub_date, is_draft, internal_comment
  FROM blog_entry
    LEFT JOIN blog_entry_owner_users
      ON blog_entry.id = blog_entry_owner_users.entry_id
    LEFT JOIN blog_entry_owner_groups
      ON blog_entry.id = blog_entry_owner_groups.entry_id
    LEFT JOIN groups_to_users
      ON blog_entry_owner_groups.group_id = groups_to_users.group_id
  WHERE is_draft = true
    AND (
      blog_entry_owner_users.user_id = current_user_id()
      OR groups_to_users.user_id = current_user_id()
    );

The plugin has information about blog_entry table and can enumerate all columns when necessary. Here it makes sure that returned columns match the original query.

Enjoy writing this again and again. This is why people prefer doing permission checks in application code.

Plugin could also support permissions on columns. For example, internal_comment field should be visible only to blog entry owners.

SELECT *
  FROM blog_entry 
  WHERE id = 42;

Could get transformed into:

SELECT DISTINCT id, title, body, pub_date, is_draft,
    CASE blog_entry_owner_users.user_id = current_user_id()
        OR groups_to_users.user_id = current_user_id()
      WHEN true THEN internal_comment
    END AS internal_comment
  FROM blog_entry
    LEFT JOIN blog_entry_owner_users
      ON blog_entry.id = blog_entry_owner_users.entry_id
    LEFT JOIN blog_entry_owner_groups
      ON blog_entry.id = blog_entry_owner_groups.entry_id
    LEFT JOIN groups_to_users
      ON blog_entry_owner_groups.group_id = groups_to_users.group_id
  WHERE id = 42
    AND (
      is_draft = false
      OR (
        is_draft = true
        AND (
          blog_entry_owner_users.user_id = current_user_id()
          OR groups_to_users.user_id = current_user_id()
        )
      )
    );

Keeping historical versions

If you want to keep all historical versions of all data, you can setup a database view which modifies UPDATE queries into INSERT queries and for SELECT queries only returns the latest version. But managing views and their migration can add additional complexity, so we can instead use a plugin which transforms queries in this way before they hit the database engine.

The plugin could automatically add version field to all tables and then a query like:

UPDATE blog_entry
  SET title = "Hello world"
  WHERE id = 42;

Gets transformed into:

INSERT INTO blog_entry
  SELECT b1.id, b1.version + 1, "Hello world",
      b1.body, b1.pub_date, b1.is_draft, b1.internal_comment
    FROM blog_entry b1
    WHERE b1.id = 42
      AND b1.version = (
        SELECT MAX(version) FROM blog_entry b2 WHERE b2.id = b1.id
      );

The query:

SELECT *
  FROM blog_entry 
  WHERE id = 42;

Gets transformed into:

SELECT *
  FROM blog_entry b1
  WHERE b1.id = 42
    AND b1.version = (
      SELECT MAX(version) FROM blog_entry b2 WHERE b2.id = b1.id
    );

Such plugin can also help with database engines which do not support views with INSTEAD OF triggers.

Do note that storing all history in the same table can be inefficient if only the latest version is almost always queried.

More ideas

A wide range of plugins could be made:

SQL is case insensitive. A plugin could automatically quote all identifiers to make queries case sensitive instead.
Generating query documentation from query placeholders and their expected types.
Reordering table fields to optimize the table.
Pretty printing of queries.
Static analyzer. Instead of rewriting the query, just validate all queries for some properties (e.g., that they do correct permission checks, or type check them). Or perform some other linting checks.

Queries can be made from the client side (e.g., browser) or on the server side as prepared queries. The latter could even be exposed as JSON RPC endpoints automatically.

Such approach (and general enough AST) could work also for parsing various SQL dialects into the same AST. But we do not even have to use SQL to get this AST. Other query languages like MongoDB queries and GraphQL could be parsed into AST as well?³ Or SQL-like query language PartiQL, too. One could even see REST as just a simple query language you can map to AST and then have a chain of plugins to add permission checks and everything else before querying the database, getting quickly a REST server implementation. JSON Schema could also be used to describe types. Many JSON query languages could be mapped to this AST as well.

Moreover, this AST does not have to be converted to just SQL at the end, but could in fact be converted into any query language for other database engines, or even just to query static files like JSON or CSV. You could run same query on the client side against an in-memory state and on the server side against an SQL-based database engine.

In a way all those technologies exist just to describe data structure and queries in different ways, but underlying principles are the same. Why then have those technologies so separated? Can we simply translate between them automatically as needed?

Have I missed any major feature? Leave a comment if there is an important feature missing. ↩︎
These days many database engines support triggers and writing them in a wide range of languages, e.g., PL/Python, PLV8. ↩︎
I would even claim that their popularity is because they do not require parsing and are already in JSON-compatible representation. ↩︎

Towards Automatic Machine Learning Pipeline Design

Sun, 01 Sep 2019 22:40:14 -0700

I recently finished my PhD thesis and is now available online. Most of the code related to the thesis is available in this repository.

Abstract

The rapid increase in the amount of data collected is quickly shifting the bottleneck of making informed decisions from a lack of data to a lack of data scientists to help analyze the collected data. Moreover, the publishing rate of new potential solutions and approaches for data analysis has surpassed what a human data scientist can follow. At the same time, we observe that many tasks a data scientist performs during analysis could be automated. Automatic machine learning (AutoML) research and solutions attempt to automate portions or even the entire data analysis process.

We address two challenges in AutoML research: first, how to represent ML programs suitably for metalearning; and second, how to improve evaluations of AutoML systems to be able to compare approaches, not just predictions.

To this end, we have designed and implemented a framework for ML programs which provides all the components needed to describe ML programs in a standard way. The framework is extensible and framework’s components are decoupled from each other, e.g., the framework can be used to describe ML programs which use neural networks. We provide reference tooling for execution of programs described in the framework. We have also designed and implemented a service, a metalearning database, that stores information about executed ML programs generated by different AutoML systems.

We evaluate our framework by measuring the computational overhead of using the framework as compared to executing ML programs which directly call underlying libraries. We observe that the framework’s ML program execution time is an order of magnitude slower and its memory usage is twice that of ML programs which do not use this framework.

We demonstrate our framework’s ability to evaluate AutoML systems by comparing 10 different AutoML systems that use our framework. The results show that the framework can be used both to describe a diverse set of ML programs and to determine unambiguously which AutoML system produced the best ML programs. In many cases, the produced ML programs outperformed ML programs made by human experts.

In node.js, always query in JSON from PostgreSQL

Thu, 10 Jan 2019 01:51:10 -0700

Recently I was exploring the use of PostgreSQL as a replacement for MongoDB. PostgreSQL has in recent versions great support for JSON. You can store JSON values and you can even make indices on JSON fields. When combined with node.js and its driver things look almost magical. You read from PostgreSQL and you get automatically a JavaScript object, JSON fields automatically embedded. But can we also use JSON for transporting results of queries themselves, especially joins? In MongoDB the idea is to embed such related documents. In PostgreSQL we could also embed them instead of joining them, but would that be faster? I made a benchmark to get answers.

Source code and results are available, for JS driver and for native driver. First I populated the database with two tables, posts and comments, where each comment has a related post. I generated 10000 posts, each with 100 comments. I tested these queries.

	js	native
raw	json	jsonb	raw	json	jsonb
posts	48.6	24.1	28.2	45.5	33.3	40.2
comments	4208.5	2957.2	3752.9	4457.8	4636.8	5214.7
JOIN	7387.2	4661.5	5747.4	8509.2	7137.4	8165.3
JOIN + array_agg	2828.6	2967.4	3919.1	2956.3	4015.4	5091.2
JOIN + to_json(array_agg)	2847.6	2822.4		4305.4	4278.5
JOIN + to_jsonb(array_agg)	3640		4246.4	5015.8		5824.2
SUBQUERY + array_agg	2696.4	2726.3	3530.5	3030.1	4125.5	4513.3
SUBQUERY + to_json(array_agg)	2690.4	2729.9		3751.2	3765.6
SUBQUERY + to_jsonb(array_agg)	3525.8		4776.6	4828.2		6047.6

Green color marks combinations I would recommend. Red color shows JSONB should be avoided.

My insights from the benchmark are:

JavaScript driver is surprisingly faster than a native driver. Even just for a simple SELECT * FROM comments. So I will focus just on JavaScript driver and would not recommend using native driver at all.
We can immediately see that converting query results to JSON in the database and then sending them over is generally much faster then sending over raw PostgreSQL fields. I believe this is because JSON parsing is so much faster in JavaScript than parsing of PostgreSQL field types. We can see that using native driver there is no such boost.
Doing a traditional JOIN without aggregation is really slow. I attribute this to the fact that traditional join repeats all column values in the main table. So for 100 comments, related post ’s row is repeated 100 times. Doing aggregation of comments into an array seems to really improve things. MongoDB’s idea of embedding is really powerful and you can see benefits even here.
While using JSONB for storing JSON in PostgreSQL and even having indices on JSON fields is very cool, but using it to transport results to the client is not a good idea. Again, probably because JavaScript’s JSON parsing cannot be used.
Subquery is faster than JOIN. A surprise because the word is that subqueries can be at best as fast as joins, but sometimes they will be worse. Here, there are better.
It seems that the best approach is to use subqueries to get related documents, aggregate them into an array, and convert the array to JSON. Or, you can be lazy and just simply always convert the whole result (with aggregated arrays) to JSON.
Important to note is that while it looks that using a subquery and just aggregation is the fastest, the results you get on the client are not really parsed into a JavaScript object, because the driver does not parse embedded records. Probably parsing them properly would add quite a bit.

Conclusion. If you are using PostgreSQL fields which can be reasonably converted to JSON and you are using node.js, it seems you should simply always convert results to JSON before sending them over to the client. If you want to fetch related documents, do not do JOIN but do a subquery and aggregate results into an array. If you are always converting results to JSON, then this is it. If not, then at least convert that aggregated array to JSON.

Reactive queries in PostgreSQL

Mon, 07 Jan 2019 01:50:10 -0700

I am a big fan of the application architecture promoted by Meteor. I like declarative programming. You describe what you want and not how and the system does the rest. Reactive programming is very similar. You define how outputs should be computed from inputs, but when is this computed and how it is composed with other computations is left to the system. So you can define what is read from the database and send to the client. And how it is read on the client and transformed and send to the UI library. And then UI library can render this data. And every time something changes, the rest gets automatically recomputed, refreshed, re-rendered.

Meteor is tightly linked with MongoDB. They developed a complex piece of technology to provide reactive queries. Reactive queries are queries which after providing initial results they also continue providing any changes to those results as input data used in queries change. While I like MongoDB, I still prefer consistency tools provided by traditional SQL databases: transactions, foreign keys, joins and triggers. They are close to declarative programming as well. You define relations between data once and then the system makes sure data is consistent. I had to implement many of those features on top of MongoDB, like my package PeerDB.

This is why I made reactive-postgres node.js package. It provides exactly such reactive queries, but for PostgreSQL open source database. Its API is simple, on purpose, and because it should be. You provide a query, you get initial data, and then you get all changes. Try it out.

Proof of luck consensus protocol and Luckychain blockchain

Tue, 21 Mar 2017 05:52:10 -0700

Proof of work consensus protocol used in modern cryptocurrencies like Bitcoin and Ethereum consumes a lot of energy and requires participants to use their CPUs for mining instead of other useful work. But exactly this cost is why it is works to prevent Sybil attacks. One cannot participate in the selection of the next block without paying this cost, which makes the issue of puppet participants trying to influence block selection irrelevant, because they also have to do the work, and pay the cost.

In recent Intel CPUs a new set of instructions is available, SGX, which allows one to run code inside a special environment where even operating system cannot change its execution. In the paper we published (arXiv, Cryptology ePrint Archive) we explore consensus protocol designs using the Intel SGX technology, with the goal of making blockchain participation energy efficient, with low CPU usage, and to democratize mining so that participants can participate again with their general purpose computers (with Intel CPUs) instead of only with specialized ASICs.

Proof of luck is an example of such a Intel SGX based consensus protocol where all participants select a random number (luck) and one with the highest number wins (luckiest). The luckiest block is then used as the next block in the blockchain. Because random number selection happens inside an SGX environment one cannot fake it. Each CPU can choose only one random number per block.

We made a blockchain implementation based on the proof of luck. Instead of reimplementing lower layers needed for a blockchain we built it on top of IPFS. It is written in JavaScript and uses a NPM module to run JavaScript inside SGX environment. Because it is build on top of IPFS, its transactions can reference any other object stored in IPFS, whole files, large or small, and other linked data structures.

Two main properties we want from a blockchain is liveness and persistence. Liveness means that transactions submitted into a blockchain cannot be prevented from being added to a blockchain. One could ignore a particular transaction in their block, but because somebody else’s block might win instead, they cannot really prevent that winning block to include the transaction. And even if their block wins this time, the next winning block might include the transaction instead.

Persistence is the one we commonly associate with a blockchain. Once a transaction is committed into a blockchain block, an attacker should not be able make the transaction or the block be removed later on from the blockchain. Effectively this means that the attacker should not be able to convince all other participants to switch to a blockchain version with that block removed (or changed to not include the transaction). In our blockchain participants switch to a new blockchain when the new blockchain is luckier as a whole (sum of all blocks’ numbers) than the old blockchain. But an attacker would have to create an alternative chain with many blocks luckier than those in the original chain, and this is harder and harder as the chain is getting longer, because all other participants are choosing the luckiest block among all of their block candidates, and the attacker would have to surpass them all.

All this is currently at a prototype stage and not everything is yet properly implemented, or even designed, but try it out and contribute to its development. Moreover, you can now use SGX in your node.js app, too.

Cookie consent fiasco

Wed, 07 Dec 2016 17:13:10 -0700

In May 2011 a EU directive was adopted with the goal of empowering web users with control over their exposure to cookies. The main issue is that 3rd party cookies allow users to be tracked across websites. The issue is that websites are often a mash-up of content coming from various services, each providing their own set of cookies. A service (a 3rd party) can thus track users across all websites using it.

In Slovenia I have participated in the process of adopting this directive into a local law which come into the effect in 2013. During this process I believed that the goal is good, and the law is reasonable. I thought that it handles technology well and with understanding, defining cookies broadly enough to be applicable to various tracking techniques and not just literally only cookies.

Maybe because of my participation and hearing all the arguments and perspectives I had a biased view, because once the law got into the effect a public outcry followed. At approximately the same time it got into the effect also in other EU countries which just reinforced public reception. Developers did not like that they had to do extra work and web frameworks they were using were not really helping them. It was unclear who will pay for that, especially because those changes were not planned and budgeted, especially for sites already made. To my surprise even developers who are otherwise outspoken about users’ privacy disliked the requirement of asking users for consent about cookies.

As a consequence, developers did not try at all to imagine how to address the spirit of the law and allow users to minimize their tracking around the web, but searched for a way to satisfy the letter of the law with the least amount of effort. Plugins have been developed which made it easy to turn site into a “compliant” site by showing users a banner where they were informed about cookies. Developers used them and move on.

Looking now back we can observe multiple issues with the whole process. First, different EU countries adopted the directive differently, but that was not really part of the public discourse, nor big international websites really payed much attention to it. This meant that if you read international information about the cookie requirements you got different information than if you would really read a local law itself. Even local news organizations got confused about it, sourcing information from their international partners. If you used a plugin developed by someone in a different country you might got a plugin which did not really made your site compliant. But because everyone wanted to do the least effort and least amount of work required by users, soon everyone converged to use the least demanding solutions.

For example, in Slovenia we required that users have to opt-in. It is not enough to just display a banner saying that by continuing using the website you agree with cookies. User had to click on a button to confirm. Some other countries focused more only on educating users about cookies and informing them that the website is using them, but not requiring them to do any action about it.

If you do provide a banner requesting from user to confirm, the question is what happens if they do not. Most developers I have seen decided to simply prevent user from using the website by redirecting them away. Or sometimes they redirected you back to the same website, displaying a banner again, forcing you to stay in this loop.

This made users learn that they should not really care what cookies are or whether they like them or not, but that they simply have to confirm the banner to be able to use the website at all. Developers managed to get users to confirm, but of course users were not really empowered in any way. Developers themselves understood that such banners do not really do anything meaningful. And this made part of a negative cycle which further diminished the perception of the law, and why would anyone spend time working on a good solution for a bad law?

Moreover, users became habituated to this banner and that they just have to confirm it and this is it. Even if a website would allow operation without cookies, users learned from other websites that this is not really an option to even try.

Furthermore, most banners are simply claiming that those cookies are required for site operation. But this is simply not true. Most 3rd party cookies are not. When using web with disabled 3rd party cookies as an experiment I noticed an issue only with commenting platform Disqus: I had to make an exception to be able to login to comment. But cookies for website analytics is definitely not needed from the perspective of the user.

It is interesting to note that in this case developers’ perception of the law guided their response which was then coded into the code and became interpretation of that same law, effectively became the law. The normative technology became the law. While original intent of the law was slightly different, at least how it was discussed during the process of adopting the EU directive, the code implementing it became the real law. Became what users see what the law is, and what other public see how reasonable and useful the law is. And because it is clear that banners are not really useful, the law itself is perceived as such as well.

In big international companies it seems that dealing with this regulatory change was done mostly by lawyers who are not really in a position to imagine technically changes needed to their website stack to innovate in a way which empower users and at the same time satisfy regulatory requirements. The easiest thing is to come up with a text to put on a banner and request from developers to display this banner to users.

The issue underlying this law is of course also the issue of enforcement. How to really enforce such requirements when web is global? If different EU countries have different requirements, how can a website know what to do for a particular user? Even before user really interacted in any meaningful way with the website. The website can use their IP to try to guess users’ geographical location, but this is not always precise. Who is liable when they make a mistake?

Moreover, having multiple versions for multiple countries is hard to develop and maintain. So the question posses itself: should websites implement the strictest requirements and thus satisfy everyone? But what if authors of a website believe that this degrades users’ experience of the website. Then we want to minimize the number of users who have to experience this degraded version of the website.

But it is really so hard to address legal requirements in a way which empowers users? When we were debating the law before it passed it was straightforward to us how it should work. Users should not get any 3rd party cookies by default and then only when they would request some feature of a website which requires them, they would be informed that cookies are needed for this particular feature and to proceed with loading/enabling it. This can work great for Facebook like and share buttons, YouTube and Google Maps embeds and many others. But it cannot work well for ads and analytics. Which is probably exactly those which the law wanted to protect against.

Social Share Privacy is an example of a plugin which behaves in this way. It uses a two-click approach, where the first click loads the 3rd party content and the second click then interacts with that content. It is a great and simple idea, which does not drastically worsens user experience, while at the same time empowers users.

Sadly, I have not seen other web frameworks or plugins providing such feature out of the box. If this would be something which would just work it would be much easier to get adoption. Similarly to how Cross-Site Request Forgery (CSRF) protection became widely used once web frameworks provided it automatically. Then even developers would not see legal requirements as unreasonable. But privacy is not yet seen as important as security itself.

Existing web frameworks even use cookies in an anti-pattern in the respect to cookie laws, setting them immediately when user visits a website, and not when user requests a feature. For example, session cookie is automatically set, even if a website does not even use sessions or even if user has not yet logged in.

In some way it is thus understandable that developers have chosen an easier path. Law tried to force this change in an architecture of web applications but it did not happen. Maybe this effort was not coordinated enough and core developers of web applications did not feel enough personal motivation to coordinate better. So what could be a better way?

Maybe instead of requiring websites to provide such protection for users, law could first require it from web frameworks and service providers. Similar to how YouTube provides a nocookie version of its service. Maybe government could simply develop plugins for popular web frameworks. All that so that when later on developers are required to change websites, tools and services are ready.

Or was it a problem that the law allowed for multiple ways to ask for users’ consent. So developers chose the easiest one. Or is issue that in general developers are wary of Internet being regulated and was this a form of a protest as well? Would it be better if this would be done through a standardization body with many stakeholders? Including browser vendors. For example, the law has provisions for users to configure their browsers to inform websites about their cookie preferences. But no browser really provides such feature, and there is no standard yet to inform the website about cookie preferences. And it is unclear if anyone from the government even worked with browser vendors to implement something like this.

Pay-it-forward cryptocurrency

Sun, 09 Oct 2016 03:12:10 -0700

Bitcoin, Ethereum, and blockchain in particular are often claimed as revolutionary as the Internet itself. They will decentralize the Internet again, change how we make apps, empower end-users, and remove intermediaries. But are they really so revolutionary? Even ignoring the technical limitations of scaling and power consumption, we can hardly imagine such wide influence on our society as we observed for Internet. Internet connected people globally, provided means of immediate communication and access to knowledge and information. It changed many aspects of our lives and how we as a species operate. But blockchain, does it really have this potential?

Take Bitcoin for example. Its technical innovations are undeniable, but as a currency itself, it lacks innovation. It is still a one-dimensional numeric-like value assigned to each transaction. If we are comparing it with the Internet, then it is the Internet in the form where people would take physical books, scan them as images, and make them available online. No interactivity, no multimedia, no searching or social capabilities, no additional value besides digitizing the content in the most simple way. Bitcoin does the same. It just digitizes existing concept of money. It does this in an interesting and innovative way, but it is still just a digital image of money.

How much can then Bitcoin as a technology really influence the society, if it just copying and digitizing existing practices?

To answer this question I asked myself another question: how would a cryptocurrency which would not be just digitizing existing practices look like? How would we design a new cryptocurrency which would by its design influence and change the behavior of its users?

To be able to answer these questions I needed an additional insight about current cryptocurrencies: it is not true that they provide just a one-dimensional numeric-like value. As a side-effect of their public nature they in fact provide a vector for each transaction, direction and a magnitude of the value transfer.

Using this idea we can design an anti-cryptocurrency. If our current economy operates on debt, we can design a cryptocurrency for an economy which operates on giving. If currently you lend money to somebody, you want to record this fact to assure that they return the money to you eventually. In our cryptocurrency, you would record this fact to assure that they give the money further and do not keep it for themselves. We know the direction of money, so we can make sure that it never flows back. That one cannot return the money it received. If currently money transfer is instantaneous, independent from your behavior and existing wealth, in our cryptocurrency it would be a process depending on your behavior and existing wealth.

In short, our cryptocurrency would be a pay-it-forward cryptocurrency.

Its properties would be:

Everyone can have only a non-negative amount of money.
When somebody wants to send you money, they direct a flow of that amount of money towards you. Less money you have, quicker can that money flow towards you.
Moreover, looking at the network of your past transactions, less money you sent in the general direction of the sender, quicker can that sender send you money.
Once the sender directs a flow, they cannot cancel it anymore.
The money once directed (even if not yet transferred completely) is reserved for that flow.

The less money you have, quicker and more money others can send you. You have incentive to send the money forward as soon as possible, because only this is how you can get even more money. If you do not have any concrete reason to send money, you can always donate it. If you do not know of anyone to donate to, you can always send it to a special address for universal basic income. In this way you get rid of your money, and at regular intervals then all money from that address is redistributed back to all users of the cryptocurrency. (How one would assure unique users is another problem.)

One would still be able to pay in a store with this cryptocurrency. Only that store would have to have by itself little money in its account to quickly receive your payment. A store can donate as well if needed.

There are some interesting details. Like if I direct a flow towards somebody with a lot of money, they are receiving my money slowly, but then also I cannot receive money from others, because I still have money. So I should be selecting recipients who do not have much money.

There are also some issues to be ironed out. Such cryptocurrency might incentivize wealthy people to hire other people to be their wallets. They could distribute their wealth among those people-wallets. So some form of regulation might be needed to forbid this. Or laws which would allow people-wallets to claim the money as theirs, if wanted.

Such cryptocurrency seems to be aligned with the idea of improving our economy by maximizing the velocity of money.

Decentralized governance and four fallacies

Sun, 18 Sep 2016 01:40:10 -0700

Together with popularization of blockchain we can notice revived calls for decentralization of national and international governments, their reboots, or even their dissolution. But such calls lack fundamental understanding of how our governments operate, their role in our global society, and what all in fact regulate and control our existence beyond just governments.

Jo Freeman in her essay The Tyranny of Structurelessness reminds us that humans tried to remove centers of power and organize themselves in a horizontal way in the past as well, and presents to us issues with power relations she observed in a 1960s women’s liberation group.

What she observed was that once you removed explicit centers of power, implicit and hidden centers of power emerge. I see this is as a normal social phenomenon. More people we have to coordinate with, more effort and time coordination requires from us. Especially as the community scales up and more people join. We start interacting more with those we interact easily, and cliques emerge. Through time some of those gain more power than others because they do not distribute their power outside the clique in the same way as inside. So power concentrates.

The difference is that this power is implicit, hidden. While in traditional communities with centralized structures we established institutions for checks and balances, like transparency and accountability, those do not exist for implicit centers of powers. The whole concept of a “public service” is based around public being able to oversee the service. But once we remove those public services and replace them with private services we loose mechanisms to oversee their behavior. We are lucky if we can even detect that such concentrations of power exist at all.

Thus, I would name the first fallacy of decentralization: a fallacy of structurelessness. It is better to have clear and explicit centers of powers with transparency and accountability than pretend that we have decentralization, just for those centers of power to emerge hidden and unchecked.

Moreover, just having a decentralized underlying technology like blockchain does not mean that the community using it is decentralized. Or that using a centralized technology means that the community is centralized. Wikipedia and many open source projects can be seen as communities with centralized technologies which help the community operate together, but the community itself is decentralized and global. On the other hand, Bitcoin mining power has concentrated in few mining pools, despite technology being decentralized. This leads to the second fallacy: a fallacy of equating decentralization of power with decentralization of structure.

Those building such new decentralized systems to replace existing governments and institutions often become victims of a fallacy of beautiful models and designs. Our minds are limited. To be able to think about complex processes we create simplified models using many assumptions and reductions. We love such models. Simplicity is beautiful.

Those tools are important to help us give insight into those complex processes, but we have to be wary of starting to believe that this is all there is to those processes. That we can then just create code which encodes this simplified models and require humans to adapt their behavior to the code. We can observe that in modern urbanism where we segment cities into clear and predictable areas of different uses. This makes it easier to reason about cities, but it establishes artificial limits on possible interactions between people living in those cities. We get commuter parts of a city where people just sleep over the night. And parts of a city like financial districts where people work just over the day, but over the night nobody is around. This makes those cities and communities fragile. So different from old cities like Jerusalem.

Such models are maybe locally useful, but globally restricting. It is simply not possible to make one model, one design, which will serve everybody. Nobody likes to make programs with a long list of exceptions, but I would claim that this is what we should start doing. We should start making programs which can be messy. Be easily adapted, changed, modified. Web is like that. Wikis are like that. Programs should also be like that.

Arthur Brock talks about this by describing how cryptocurrencies have crises of governance because they have removed people from the design. It is easier to design things without messy people, but then do not be surprised when design collapses in practice.

Lawrence Lessig discusses the idea that there are four modalities which regulate us: the law, social norms, the market, and architecture/structure. Of course they also interact with each other and are dynamic, changing. On the Internet, we can see architecture/structure in a form of the code which governs cyber spaces we inhabit. Lessig talks how the government has multiple ways to regulate, not just by passing laws. And not just the government, but everyone who controls cyber spaces. Moreover, he talks also about the danger of moving from direct and explicit regulation by laws to implicit and indirect. For example, by changing the architecture/structure. Instead of using laws to legalize segregation one can use highways without easy crossings and railroad tracks to divide communities. Such urban changes still regulate, but we lose on transparency and oversight. It is a similar issue to the previously described fallacy of structurelessness. Those issues exist online as well. We replaced regulation of the cyberspace by law with the code operated by private companies. Who oversees that code? Does public have any say in it? Do you know what Facebook removes from your feed and which videos are deleted from YouTube?

But for this discussion much more important is realization how many things regulate us and how are they all intertwined, besides just law. We developed through centuries so many social institutions for all bugs and issues we discovered in our social and economic processes. We might think those institutions are rigid and slow, unneeded, and obsolete. But this is often a consequence of a fallacy of beautiful models and designs. We just do not realize all the reasons for some things to exist. Yes, we could often improve them and modernize them. But it is a fallacy to believe we can reboot a living system from scratch.

Through that we lose all the knowledge embedded inside those institutions. Most of them we are not even realizing we have. Many of them are there to protect us against misuses and mistakes which can happen. Inefficiencies are sometimes a feature to allow for a human to detect a problem.

Every founder of a community can tell you how many small things are necessary to establish for a community to really live, and especially to make it pass the first generation. Even in software we are often unable to rewrite the code from scratch successfully. And we would want to do this for such a complicated ecosystem as our society and economy is?

Current system is a mess, but how can we know that without it we would be better of? We already had unrestricted capitalism in 19th century and learned that it has shortcomings for those who do not own the capital and means of production. How we know that by removing safety belts embedded inside current institutions the old oligarchies will not reappear? That we will not have again the wild wild west. Just this time, instead of six barrel manual guns we will have assault weapons and cyber attacks at a disposal of every individual.

The DAO hack was an example of such realization. We are unable to write perfect contracts. This is why we created the judicial system. It is not perfect, but it is still better than what was at the disposal to the community to do in the case of the DAO attack. At the end, effectively rebooting the system to try again.

The main danger is when these fallacies combine. They reinforce each other. They make us think that we can reboot the society because we designed a beautiful, decentralized, simple, and generalized model, which we encoded into a protocol, code. But by doing that we forgot about humans and what makes us humans. The chaos of all exceptions and special cases. Concentrations of power because we have friends. Because we like some people more than other. Because we do trust some people. It is not simple. It never is simple. But that is OK.

Sybil attacks and shell corporations

Mon, 25 Jul 2016 18:47:10 -0700

In computer security an important type of an attack in a decentralized system like Internet is a Sybil attack. The core of the attack is that many protocols we have developed depend on the assumption that each entity participating in a protocol participates only once and that it cannot create an arbitrarily number of additional “puppet” entities which it can control. Because on the Internet it is easy to present yourself with multiple identities, decentralized systems with open membership are often susceptible to this type of an attack.

Why are we designing such protocols? Maybe it is because so many protocols are based on existing processes we find between humans? Or maybe there is some fundamental issue of open, decentralized systems and identities.

But even more interesting is to observe that we have similar protocols in our existing society, with the same assumption. Moreover, we also allow people to create additional identities as needed. We call them corporations. Many our protocols in our society were designed when only humans were persons. Governments make sure that humans have unique identities and we have passports to allow governments to trust other governments about this validation. Creating corporations is on the other hand much less controlled. You have many countries with less strict laws which allow one to create shell (“puppet”) corporations. Traversing multiple jurisdictions through interactions between such corporations can hide many traces of linked identities. In a way, we allow arbitrary number of corporations to be created, without really requiring passports for them to be able to work with other corporations across borders. A passport which would link corporations to their unique identities. Furthermore, the issue is even more complicated because there can be multiple people behind corporations, and also other corporations.

So a real question is not why we are designing such protocols on the Internet, but why we are having ways to compromise such protocols outside the Internet. When we know that they can be misused and used to launch attacks. We already see such attacks in practice through pervasive tax evasions and other financial maneuvers.

Wanted: precise terminology about democracy

Fri, 22 Jul 2016 15:48:10 -0700

In the previous blog post I presented one example of a confusion when talking about democracy: we use democracy for both “one person, one vote” and “one dollar, one vote” approaches to voting. But the issue is much broader. Saying that something is democratic does not really tell much, because it can mean anything from a majority voting, consensus (unanimity), voting based on shares, a system with representatives and one where we vote directly. Democracy is used to wage wars, topple dictators, but also topple democratically elected people. We use democracy to say “you cannot argue with it”. And we use it to position ourselves as morally superior. As such, the term democracy became almost useless.

We need to start finding more precise terminology for all aspects of democracy. What does it mean that a cooperative is democratically run? That workers can elect board members? That they do not have votes based on shares? Or that they can directly influence business decisions through a democratic process? Which process exactly? Does it matter? Are all the same? I do not think so.

Let’s start building terminology. Collectively.

One person, one vote or one dollar, one vote and blockchain

Wed, 20 Jul 2016 01:52:10 -0700

We live in times of a hidden war between “one person, one vote” and “one dollar, one vote” ideologies. The reason why it is hidden is because we use the same terms for both: democracy, voting, consensus, etc. We govern our governments each having one vote, but in our companies shareholders commonly hold votes proportional to their share. Some people are claiming that the latter is a better approach and everything should be decided through markets and power. I believe that using power (physical or monetary) to make decisions is barbaric and that our civilization progress was to introduce a more true democracy, one person, one vote. But I do not believe even that is the end of our developments in this respect and we should continue developing our collective governance. Moreover, I do not believe that these two positions are necessary the only possibilities, and some combinations might also exist. In some way we might even already have that: using “one person, one vote” to decide the rules under which we operate, but using “one dollar, one vote” to decide how to split the profits.

Anyway, all this could be a topic of some other longer blog post. Here I wanted to explain this existing tension between these two ideologies to present how they have existed in decentralized technologies as well and why Bitcoin’s blockchain is so innovative.

Traditional perspective on decentralized systems is that you have agents which communicate with each other to form a system. Many decentralized protocols use some form of voting to decide on common decisions, e.g., to which new state they should all transition. What the majority of agents vote for is seen as the correct new state for the whole system (so called consensus). This means that there can be up to half of malicious agents and the decentralized system will still operate correctly.

But sadly it is not so easy. The issue is that if membership of agents in the system is open, then a malicious agent can join the system multiple times with “puppet” agents, and because each agent gets one vote, can then enforce its view of system’s state. This is know as the Sybil attack. In centralized systems this is not a problem because you can have a centralized entity controlling membership.

The ingenious solution made by Satoshi Nakamoto for Bitcoin’s blockchain is to replace this “one person, one vote” voting with “one dollar, one vote” voting. In Bitcoin’s case you vote with CPU cycles, not dollars. Even if you create multiple “puppet” agents they all have to share the same amount of CPU cycles you have at your disposal so you do not really gain any advantage over others.

This innovation revived the interest in decentralized technologies. A hope emerged that we could completely decentralize all aspects of our society and remove intermediaries who require fees we do not like, or can abuse power they have. We have seen many projects embark to achieve this.

But while the solution is ingenious and is an innovative shift in a perspective, it has not solved the problem at a fundamental level. As such it has issues we have to be aware of.

“One dollar, one vote” voting is the most suitable for use cases where we are already used to such voting: markets and companies with shareholders. But we have to be wary of attempts of everything being transitioned to this type of voting because we might lose true democracy of “one person, one vote” without even noticing. This is often not conscious and is just a consequence of uncritically applying blockchain technology to the problem. Especially if blockchain is already used for other aspects of a project.

While I can understand that some people like transition to “one dollar, one vote” for all aspects of a decentralized society, I would like to warn especially people who otherwise do not align with this ideology to be careful when embracing blockchain as a technology of choice for their projects. For example, projects around the platform cooperativism idea.

Another issue is that “one dollar, one vote” environment is susceptible to concentrations of power, like any other environment which is built around power. While initially it looks like such voting is democratic as well, through time an oligarchy forms. This can be seen in Bitcoin as well, where all mining is now effectively done only by a very few large mining pools.

We can see that while “one dollar, one vote” voting solution addressed one technical challenge, it brought new (but known elsewhere) challenges to the technical realm.

A consequence is that we have not really solved the problem of a decentralized true democracy where each user person would have only one vote and where we would have an open membership. We still do not know how to do achieve both at the same time. This can be seen with all governance issues of Bitcoin and Ethereum communities. One could say that those communities are not interested in true democracy, but the real issue is that they do not really have a choice at the moment. We do not yet know a necessary solution (and are we even working on it?). So we have to decide what to do in meantime: use centralized technologies which allow true democracy, or use decentralized technologies which do not?

Moreover, in meantime, when somebody talks about voting, consensus, and democratic decentralized technologies, listen very carefully if they are talking about “one person, one vote” or “one dollar, one vote”.

Blockchain as an Internet of things savior?

Tue, 19 Jul 2016 15:50:10 -0700

Internet of things (IoT) has some big issues. Interoperability is one. And potential vendor lock-in and data silos are another.

But I do not buy that blockchain is the solution to those problems.

I have heard so many times the idea of combining blockchain and IoT, but I do not get it. IoT will be producing potentially much more data that anything we had before, and a lot of that data will be privacy sensitive. How can a public and decentralized technology with scaling issues help here?

It is true that data on a blockchain is public and interoperability seems easier if everyone agrees on one blockchain. But those properties are not the fundamental properties of a blockchain. Any common technology, if parties agree to use it, improves interoperability. But nobody assures that data stored on a blockchain would be in open standard. And what when people will encrypt data stored on a blockchain to address the privacy concerns with IoT data? How does this prevent data silos?

There are many other ways to address issues of interoperability and data access. One I like is that gateways for IoT should not be black boxes but something end-users can control. You can control your home Internet router, and you should be able to control your home IoT router. If you can do that, you can prevent data silos and you can assure interoperability with those you want. Furthermore, we could enforce it through regulation (require open standards for IoT), instead of trying to get IoT companies to adopt a new technology which is not even yet ready for this use case. Why would those companies do that?

The issues here are to me easier solved through regulation than through technology. And if we want a technological solution, then open IoT gateways are a better approach. What we need is a net neutrality movement for IoT and not a blockchain.

Free as in … ? My LibrePlanet 2016 talk

Tue, 28 Jun 2016 10:30:10 -0700

Interesting talk by Luis Villa pointing out that freedoms are not enough, but we have to aim for empowering people to have capabilities to exercise those freedoms.

An example of this can be found in free software where just a freedom to run a software is not always enough because it does not address that many people cannot do much with such an abstract freedom. We should build software which gives people capability to run them.

Bruce Sterling on startups' role in helping the global rich get richer

Mon, 27 Jun 2016 11:30:10 -0700

Bruce Sterling’s speech from NEXT Berlin is a blast of cold air on the themes of startup life, disruption, and global collapse. Bruce excoriates the startup world for its complicity with the conspiracy of the global investor class to vastly increase the wealth of a tiny minority, and describes the role that “design fiction” has in changing this.

via Boing Boing

Learn how to walk

Sat, 25 Jun 2016 10:30:10 -0700

Learn how to walk. Literally. When I observe people how they walk I notice that everyone walks differently. This is great, it allows one to recognize friends from far away. But it is also bad: some types of walking and posture is worse for you than other. Especially because you are probably walking the same way your whole life.

We are never really taught how to walk. We learn on our own. It is one of first things we learn. And everyone around us is so happy that they forget to help us improve our walk. And we, based on our first steps we did by chance, extrapolate to walking and get used to it. For long our body tolerates any way we are using it, but through years we might suddenly discover that it cannot anymore. But then it is too late.

While others help us with other things we learn (for example, talking, you will get feedback if you talk incomprehensibly, or too loudly), for walking we have to do it ourselves. So stop just repeating steps you did as a child and start walking your grown-up walk.

Speech recognition

Fri, 24 Jun 2016 10:30:10 -0700

I do not know much about speech recognition. I do not know what is the state of the art. But years ago I was playing a bit with it and I would like to throw an idea our there, maybe somebody picks it up, maybe it turns out useful, or maybe it is already being used. Please tell me. It can be used for not just speech recognition, but any general audio pattern recognition, or any signal pattern recognition.

The basic idea is to observe that human hearing works by first cochlea doing physically a frequency transform. Hairs of different lengths resonate to frequencies in the audio input. Stronger a particular frequency is in the input, stronger will be a signal for that hair. A stronger signal in neurons does not mean a larger amplitude of the action potential, but more of them. So a stronger signal for a particular frequency means that more impulses will go over that neuron. More impulses mean a higher frequency of those impulses. So brain has to learn not directly from the input audio, but from changes of frequency of the signal for each frequency in the input audio. If brain is recognizing patterns from that, we should too.

A common way to analyze signals is to do a frequency transform and try to learn about properties of the signal from there. But the issue is that most signals change through time, including speech, but a frequency transform tells us just an average presence of frequencies at the interval we are looking at. There are other ways how are people trying to address this and they might be compatible with my idea, but to keep it simple we will just use a basic frequency transform as a primitive.

We can take a small interval window (which window function exactly is another parameter here) and slide it through the input signal and compute frequencies as we are sliding. We can visualize the result for an audio of few spoken vowels into the following spectrogram.

But we can also move a window very slowly, sample by sample in the original input audio. Then we get for each frequency a new signal corresponding to how this particular frequency has been changing on average inside this window (I used a 30 ms long rectangular window). An example of such new signal is depicted with green border on the spectrogram above. Now, to use my idea, we run a frequency transform on these signals for each frequency again. As a result we get for each frequency in the input audio what are frequencies of changes of that frequency. To me this is similar to a derivative of a derivative, a second derivative.

As na experiment, I ran this on recordings of individual speech sounds. In this case I simply computed the second frequency transform over the whole new signal for each input frequency. But we could be doing windowing and sliding again, to determine how sounds are changing through time.

When doing a series of such transforms we hit against a problem of frequency resolution vs. time resolution. Shorter the window is, higher time resolution we have, but less frequency resolution we have. For speech we are interested only in a short frequency band inside which humans speak, so we could use different ways to get good frequency resolution at that band, but I simply used recordings made at a 192 kHz sample rate which provided me with more samples so I was able to have a 30 ms window but still reasonable frequency resolution.

For each of multiple samples per sound I made a double transform and visualized results below. On Y axis we have various frequencies from the input (oriented the same as in the spectrogram above, but just band around human speech shown), and on X axis we have power of each frequency of frequency, starting from the left. It seems changes to frequencies are itself at a very low frequency so I am showing only low frequencies. Each image continues far to the right, but does not contain anything visible there.

Visually, they look pretty similar to each other when comparing the same sound, and nicely different when comparing across sounds. This is great! This gives us 555 values (37x15) we can feed into a pattern recognition algorithm. Deep learning or something similar. I will leave to somebody else to test deep learning on this and to determine how to upgrade this one sound recognition to speech recognition when sounds are changing through time. Good luck!

Attributions: vowel spectrogram

Hacking capitalism with capped returns

Thu, 23 Jun 2016 10:30:10 -0700

A post by Joshua Vial.

Do Publicly Owned, Planned Economies Work?

Wed, 22 Jun 2016 10:30:10 -0700

A post by Stephen Gowans.

What we need to progress as a human civilization is more diversity in how we are organizing ourselves, in how we run our economies, and in which values we pursue. We need a global understanding that such diversity is important and we should establish ways for countries to try and experiment with different paths. Even if we do not believe that their path is correct we should be proud that they are walking it and we should help them if needed. We will all learn as a result.

If we really believe that our system is better, do we really need to actively try to undermine the other system? We could even help it, if they need anything, and once it will collapse (if, but our system is better and the only with a future!), with all circumstances being positive for it, including us helping, we will know for sure that it is really a worse system. We can even then help those brave explorers who hit the floor. But, until then, let us have some diversity and encouragement for each other when trying something new.

The fact that Spartacus lost a war does not mean that slavery is a great system. How many more years of exploration of various systems were needed before we realized that?

Filter to extract all clean water from soda and other drinks

Tue, 21 Jun 2016 10:30:10 -0700

Can somebody create a filter which extracts only clean water from soda and other drinks? No sugar, no colors, no additives. Just pure water. It seems bottled water is more expensive than soda drinks, so, let’s just use a filter.

Towards layered re-decentralized web

Mon, 20 Jun 2016 15:45:10 -0700

Re-decentralizing web and Internet-based technologies is gaining momentum. Just recently there was a Decentralized Web Summit where many projects were represented. But while people contemplate importance of decentralized technologies and are building alternatives, to me it seems we are forgetting one important lecture from the past: we should be building layered technological stacks and not vertically integrated ones. This is how you can encourage diverse implementations (critical for stability of decentralized technologies) and experimentation at each layer without having to reimplement a whole stack. I can understand why this is happening. It is already hard enough to develop decentralized technologies. So it feels easier to control the stack at least vertically. But we should not be building decentralized technologies because it is easy. But because it is hard.

Decentralized and permanent web

From the perspective of a human civilization current world wide web is broken. It has evolved from serving static content into dynamic app-based services. And while static content is based on open standards which can be easily preserved for future generations, dynamic content requires service providers to stay online forever. There is no way to archive services and preserve them for current or future generations. Architectures of those services simply do not support that. And for success of any civilization it is vitally important to be able to preserve its history.

Web as we know it today, as explained by the Mozilla’s Mitchell Baker at the Decentralized Web Summit, is:

Immediate. Safe instant access to content accessible via a universal address without the need for install.
Open. Anyone can publish any content without permission or barrier and provide access as they see fit.
Universal. Content runs on any device on any platform.
Agency. User agent can choose how to interpret content provided by a service offering.

But users are not really in control. Service providers today can decide and deny a service to the user, they can terminate their service, change it without user’s consent. User’s rely on service providers for their security. A compromised service provider also compromises all their users.

Even if a developer wants to build a web app where user would be in power, it is not possible to do so with currently widely available technologies. The main reason is that such solution has to be done by multiple parties. Like web itself does not really bring much advantage if there is only one web server in existence, permanent web works when there are multiple developers using it for their apps and services, providing to each other necessary infrastructure to assure apps stay available forever. For permanent world wide web applications, once content, an app, or a service is deployed it should stay available while there is at least one user using it (maybe just a librarian). And not only while a service provider cares about an app.

There are many existing projects working in this space: IPFS, WebTorrent, BigchainDB, Ethereum, Matrix, Tendermint, Blockstack, SwellRT, Scuttlebot, and many others. But while all these are all great projects to explore the design space of modern decentralized technologies, the power of decentralization is not in the fact that one system manages to decentralize itself, or that we create one decentralized architecture to rule them all, but that we have decentralized layers which empower each other and mix use cases to make a robust ecosystem.

For example, Bitcoin uses its own P2P data-exchange network at the lowest layer. This means that any attack on the P2P network is an attack only on Bitcoin and has no other side effects an attacker might dislike. Moreover, the only reason and incentive users have to participate in this P2P network is to participate in the cryptocurrency, so the networks userbase is limited only to those users.

On the other hand, users of Tor network are diverse. Different people are using the network for different reasons. Even governments are using Tor network. Trying to limit one use of the network interferes with other uses. Trying to limit some users limits other users as well. Diversity in users and use cases makes Tor strong. This makes Internet and web strong. But we lack this in our decentralized technologies we are building.

Decentralized and layered

We should be building our technologies according to the following design principles, which were used also for Internet itself:

The end-to-end principle.
Optimize for flexibility.
No barriers to entry. No gatekeepers. Low investment to join or start participating.
Make it easy to try. Fault tolerance.
Build an ecosystem. Distributed coordination.

The architecture should be decentralized but also layered. For example, we could try to organize new technologies mentioned above into a stack like:

Those layers could be split into even smaller layers (IPFS itself is, for example, split into many smaller layers).

Such architecture is of course harder to build. You have to design interfaces between layers well. You have to talk to and coordinate with other projects. You have to establish standards. But it allows more innovation: one can just innovate in one layer and reuse others. I am not talking here about code reuse. We can have layers nicely each in its own code module and then you can change code of only one layer to experiment. This is nice, a requirement, but this is not it. We want to have other layers running as they are, deployed. When I create a new web application, I do not have to re-deploy the whole Internet stack and get users to use my new stack (even if it happens to share code with other stacks based on the same codebase). Other layers simply keep running and supporting my new app as well. This allows me to innovate only inside one layer and reuse existence and adoption of other layers directly.

Besides improving potential for innovation such architecture improves robustness of the ecosystem as well. Let us look at smart contracts as a target application in the pyramid above, as an example. Instead of developing smart contracts in its own stack, we could be reusing layers running for other purposes. decentralized data layer would not be used only for storing blocks of transactions, but also for distributing files, static web content, archiving the web, distributing scientific data sets and so on. Decentralized computations could run any type of computations, a background worker for a decentralized web application, a CGI rendering for a new movie, protein folding, etc. Some databases on top of those computations might not care about consensus of the data. They might just want to store all data and allow queries on top. But some might care about consensus. Some in a permissioned setting where you exactly know who the users are and can use BFT protocols. Some in permissionless setting using some type of proof of work to resolve the consensus. An interesting thing is that by using a general data layer, you can commit to the new block locally, for example, add it to your local IPFS instance, and if your block wins, only then everyone asks for it and the block gets distributed among all participants in the consensus layer.

Smart contracts can then be seen just a special case of those generic computations on top of a consensus-based decentralized database. Which can access also all other layers as needed, not just transactions, but for example a rendered movie, to detect if a payment has to be made for a finished rendering service. It can leverage other layers and capabilities there as well.

This makes the whole ecosystem much more robust and diverse. A specialized attack on one use case can be mitigated by other use cases. A general attack has potential side-effects you are not willing to do. For example, if incentivization issue is discovered for smart contracts which makes users not willing to run smart contracts of others, users who are participating at the decentralized computation layer and are not part of smart contracts users would still continue running them, maybe just to gain their own “CPU credits” inside the decentralized computation layer. It would be much harder to game the system and find disruptive strategy because you would have actors in the system with various backgrounds, values, and incentives.

On the other hand, if somebody would like to disrupt the data sharing layer because they want to censor some content there, they might discover that this disrupts the whole financial sector as well, build on top of smart contracts on top of this stack. This might make them less willing to attack the data layer.

There is strength and power in diversity. We should be building towards that.

Decentralized computations layer

The missing layer seems to be this decentralized computation layers for general computations. An ultimate virtual machine, if you wish. But necessary pieces might already be there. First, we just have to store code for computations in some serialized way on the data layer. This might be simply reusing Docker images, or even just a JavaScript code string to run inside node.js or something. Or all of those, supporting various languages and binaries. Probably we would require that computations are pure and deterministic, not allowing to call any system calls. On top of those serialized computations a decentralized set of participating workers would run those serialized computations every time new data input is available for them, storing the result. How data input is made available depends on a particular design, but we could imagine some pub/sub approach which can take data from any other layer, and then store the results into a similar output publication.

Both inputs and outputs would need some validity schemes, which would verify that inputs are really meant for this computation (for example, they are signed by a key embedded in a computation). Trickier is to validate that outputs are really correct outputs for a given computation. One approach is to have multiple workers run the same computation and compare the outputs, similar to what BOINC is doing, or even have much more complicated output validity schemes with more workers participating, in the extreme, all workers computing all computations and then comparing outputs.

Another approach is to use new SGX capabilities of new Intel CPUs which allow one to run code in a trusted enclave and have a verifiable statement that a particular code has run inside that enclave, producing a particular result. Verifying such statement is cheaper than running the computation multiple times. Moreover, code running inside an enclave is protected and nobody, even the operating system itself, cannot access data inside the enclave. This could allow decentralized computations where computations can operate on sensitive data as well, without a need to use homomorphic encryption. Furthermore, it allows also non-deterministic outputs because an output does not have to match outputs of other runs to be considered valid. A downside is that currently verification of statements is done in a centralized way by Intel.

Which workers execute which computations can depend on various policies as implemented by multiple computations schedulers inside the decentralized computations layer. There can be multiple computation schedulers listening for new computations and assigning them to workers based on their policies. For example, one could have a set of computers connected to the system and would execute only their own computations. Alternatively, workers could participate in a CPU sharing scheme and run computations for each other. Or one could simply be selling CPU cycles on a market on top of this stack.

Computations scheduler would take output validation schemes into the account when scheduling computations (for example, SGX-based scheme might need only one worker for a given computation, while some other scheme would require that all workers run all computations).

Analogy

To easier understand the proposed stack, you can imagine that serialized computations are something like IP packets, with payload, and source and destination addresses, and computations scheduling is something like routing layer for those packets, and a CPU sharing scheme is something like Bittorrent sharing protocol which incentivizes people to participate. Or users can just buy resources on the market.

Conclusion

To conclude. If we are serious about decentralized technologies, we should be building them as independent layers where we all reuse them and have common users. This improves how innovative we can be and how robust our technologies can be. The only missing piece is a decentralized computations layer for general computations. Instead of focusing on specialized computations (like smart contracts) we should design it for general computations and have smart contracts be just a special case.

Fixing a bearer token vulnerability

Wed, 04 May 2016 16:20:10 -0700

A security bug I discovered in npm has been finally fixed.

Building free market from scratch

Fri, 08 Apr 2016 23:48:10 -0700

Interesting that those who like free market prefer changing existing economies which are build on public spending and public infrastructure, over building their own infrastructure from scratch somewhere in the middle of an ocean, and then having free market there. I am still waiting for that. Also, please do not dump your trash into that ocean. Ocean is commons. Make it work inside a closed system of your free market, without counting on future generations to clean after you.

Taxation and rent

Fri, 08 Apr 2016 23:47:10 -0700

Those who claim that taxation is theft, why is it any different from rent? Just because one has an implicit contract, and another an explicit? If you do not like the contract, you can always leave for another country. Oh, you cannot? Maybe you are just not trying enough, like those who cannot afford your rents. So see, taxation is simply a rent for you to be in a country.

nodewatcher: A Substrate for Growing Your own Community Network

Fri, 15 Jan 2016 16:26:10 -0700

A paper presenting nodewatcher.

Presentation and release of nodewatcher 3.0

Thu, 19 Nov 2015 10:30:10 -0700

Presentation and release of nodewatcher 3.0 at Battlemesh v8. Nodewatcher is an open source network planning, deployment, monitoring and maintenance platform with emphasis on community. Version 3.0 is a complete rewrite bringing modularity and extensibility.

A panel on how to build community mesh networks

Thu, 19 Nov 2015 02:23:10 -0700

A panel on how to build community mesh networks at Open Source Bridge 2014 where we discussed both technical and community aspects of such networks.

Why programming is like writing poems?

Wed, 04 Nov 2015 01:38:10 -0700

When friends ask me how they could improve their programming skills I often reply that they should imagine that they are writing poems. How do poets improve their skill and quality of their poems? They read many more poems than they write. Reading what masters produced in the past can give you insights into ideas, patterns, and structure. When was the last time you went and read how something is implemented?

People often say that they do not like reading the code made by others because it is hard to understand. But this is similar to reading poetry as well. Sometimes it is hard to understand. But poetry done by masters gives you some understanding at any level of reading. Read great code made by masters. More you will read, easier it will be also to read code made by everyone. Do you have an issue that the project you want to use has lacking documentation? No problem, with your skill of code reading you can understand how to use the project by just reading the code. Code is documentation as well and the source of truth.

Personally, I first noticed what a difference it makes when reading Django’s code. Despite its large size and complexity it was always clear what a particular small piece of the code does and how it fits a larger structure. Reading (and thus contributing) was really easy and I learned many new things. It is also very useful to use a smart IDE like PyCharm which allows you to easily click through the code to definitions of functions. It is like reading literature with footnotes which gives you relevant context and references to other parts to easier understand what is happening in a given segment.

Moreover, there are some other similarities between programs and poems, or programs and literature works in general. One great and smart line does not make a poem. It has to fit the whole poem, its structure. It has to be clear how you get to it and how you get to the next line. Do not try to make smart one-liners. Make great poems.

Larger software is also similar to larger works. Split into chapters, sections, paragraphs or stanzas. There is some overarching story, idea.

If you look at functions, good functions have a start, then body, and the end. They do not have multiple bodies, doing different things.

Also, after you publish one book, you might have to write the next one, which should be compatible with the previous one. You should not change the history of your characters in the next book.

People might adopt your work to different media, your story should work there as well.

Futarchy: Vote Values, But Bet Beliefs

Mon, 07 Sep 2015 13:28:10 -0700

An interesting idea how to aggregate knowledge for decision making.

Sharing economy and its non-sharing truth

Sat, 01 Aug 2015 14:25:10 -0700

Term “sharing economy” was coopted. It has little to do with real sharing. It has as much with sharing as my landlord “shares” the apartment to me. This is not sharing. Sharing economy became another term for renting, lending, outsourcing, only that the units of “sharing” are now smaller because information technology backing sharing economy allow less overhead in coordination and transactions. Oh, and of course, almost no regulation and oversight. So less paperwork. No contracts are signed, no responsibility or liability, no labor rights, no insurance. In real sharing you also have none of those, so is maybe this the only thing which sharing economy shares with real sharing?

It is interesting to notice that a web service is not seen as sharing economy if it just facilitates real sharing between people, without having an intermediary in between which is collecting a commission every time somebody shares something. You need that intermediary charging a commission for a service to become sharing economy.

Wikipedia is not called sharing economy despite being a web service where people share knowledge with each other.

On the other hand, Amazon Mechanical Turk was probably too early to be put into the sharing economy basket, but it should be there. You use your own computer to do micro tasks for others. No contracts, no labor rights, and nobody cares about tools you use.

It is interesting that services can be technically very similar, but results can be very different. Uber connects drivers with riders. Prevoz, a simple Slovenian ridesharing website, is doing exactly the same. The main difference is that in the latter case website just connects people together, but does not inject itself into the whole riding and payment processes later on, or collects any commission.

A similar example we can see is if we compare AirBnB and CouchSurfing. AirBnB connects and takes commission, CouchSurfing just connects, and even forbids any monetary award for hosting a couch surfer.

Maybe sharing economy is real sharing, just with those intermediaries trying to find a new business model to keep the sharing going? While Wikipedia lives from donations, CouchSurfing has not figured a good way to keep itself running. So maybe putting yourself between all transactions is just a new approach to this problem of sustainability for the service?

Maybe. But it also changes the dynamics and interactions with the service if every interaction has to be payed for, so that commission can be extracted. Contrast this with volunteer donations of Wikipedia. Imagine that you would have to pay a fee, even if it is a very small fee, every time you would want to read a Wikipedia page. Would you still use it the same? Would still all people be able to use it the same? Decoupling consuming from a payment and even not requiring a payment is an important property of real sharing.

Maybe the main difference is not in the ways you make a service sustainable, with commission or not, but what you do with the income. Wikipedia is open and transparent about their income and expenses and they are putting all the income back into the service itself, its operation, and its and community’s growth. But what happens when the commission is higher than just to keep the service running? When it has also satisfy investors’ appetite for profit?

Those little but important details are what differentiates real sharing supported by information technology from a simple capitalism driven to the extreme using the same information technology. In sharing economy workers are individualized. They work on micro tasks. They have no continuity of employment. They all compete against each other so their wages are minimized. They even have to use their own means of production (cars, apartments, etc.).

On the other hand, the companies, the intermediaries, keep all traditional powers (and issues arising from those powers). They control who can work or not (you can only accept terms of use and cannot negotiate them). They control how much workers are compensated (Uber decides the level of commission and prices of rides themselves). They decide about the product or service and how it will evolve (drivers at Uber have no direct influence on the development of apps and services provided). They control which values will be embedded in the company, product, service. Workers are there just to do the work and this is it. Sharing economy is presented as full of amazing innovations, but maybe it is simply just a regression back to the 19th century capitalism, just hidden under the veil of technology.

To address this discrepancy in power the solution back in the 19th century was for workers to unionize. They discovered that if each worker operates on their own, they collectively get into a worse of situation. Without organizing they cannot coordinate their actions, so they are at a disadvantage. They can make decisions only about themselves (work for a given compensation or not), while employer can make decisions about all of them. Thus, employer can play workers against each other, while workers cannot do anything against that. Their own personal decision might be the best for them personally, but for them overall it might be a bad decision, especially in the long term.

This is one of the main issues of capitalism and free market anyway: the assumption that if all actors behave according to what is the best for them, this will be the best for all. But this does not take into the account differences in information one has, and power to react based on that information. Even if one worker has the best possible information, they cannot just decide to make all workers respond based on this information. On the other hand, a company can. A company can react as a whole, coordinate its activities, and have a persistent influence on how things are evolving.

The unionization solution to such capitalistic exploitation was so effective in the past that it had to be demonized. Unionization was not perfect, it brought some new issues (power struggles inside unions, corruption, and other issues common in hierarchical organization structures), but it was effective against the main issue it was trying to address: lack of coordination and information flow between workers while companies have a natural benefit of both.

So if sharing economy is the old traditional capitalism using modern technologies to coordinate their workers, maybe workers should start unionizing using modern technologies as well, removing the other issues unions brought with them. Maybe it is time for a simple online platform where workers could share how much you earn with each other, your benefits, you could coordinate actions, help each other, engage in solidarity with each other, educate and learn from each other, you could decide to not work more than 8 hours.

We can learn from sharing economy. We can learn that using information technology to improve how people coordinate small units of their time, work, property, effort, money, is really powerful, but do we really need intermediaries taking commission to have this information technology to help us?

Limitations of Bitcoin revolution

Wed, 22 Jul 2015 13:02:10 -0700

I really like how Bitcoin brought up so much new energy into the cryptocurrencies world. It combined great ideas of P2P networks, proof of work, append-only databases, and others together into a working and popular system and spawn new ideas and new innovation. It started a new cycle of decentralization of technologies.

But as a currency itself, it lacks innovation. It is still an one-dimensional numeric-like value used to valuate everything humans do in the same traditional way. Bitcoin is often compared to the Internet, how it brings a new era. But if we are comparing it with the Internet, then it is the Internet at the stage where people would take physical books, scan them as images, and make them available online. No interactivity, no multimedia, no searching or social capabilities, no additional value besides digitizing the content in the most simple way. Bitcoin does the same. It just digitizes existing concept of money. It does this in an interesting and innovative way, but it is still just a digital image of money.

It is important to notice that traditional money was already a form of the Internet. A global information exchange network exchanging numerical values of transactions. In fact, it was already digital (numeric), with various transport layers (gold, coins, banknotes, and recently Internet itself). But it is not so fancy and is full of rules and conditions if you want to participate in this global network. So a reboot was necessary.

Now we got a different implementation, decentralized one, using modern computer science techniques, but the content is still the same. Despite being modern, transactions still record only a numerical value/valuation of the transaction itself. It is still a great reduction of information of what exactly the transaction was about. Such reduction was maybe reasonable for traditional money, where technology allowed passing around only one number, but today we should move on.

Bitcoin innovation is continuing, with all new projects building on ideas, trying new ideas, expanding and diversifying the ecosystem. Ethereum is for example building a general purpose computing platform on top of these ideas. All this is beautiful. But we should also innovate what money itself is, and do we even need it in such form? Bitcoin revolution should not be just a technological innovation. It should start asking questions about the nature of money itself.

Let's simplify taxes: tax all transactions

Mon, 20 Jul 2015 10:30:10 -0700

Have you ever looked into laws governing taxes? The tax code? I cannot believe that anyone can really adhere to all the rules found there. I can understand that as a living organism they evolved through time and new and new rules were added to patch holes from before, to support new technologies, new economic practices, etc. But maybe it is a time for a reboot? If laws are too complex to comprehend and follow, do they still hold their social contract?

Moreover, the situation additionally reinforces existing power relations in a society. Those with more power and more knowledge have easier time (or can obtain help) figuring out how to navigate all those rules to their own benefit. This is why big corporations and wealthy individuals are paying so little taxes and know how to take the advantage of tax havens.

We could simplify tax code. Simply, every monetary transaction would be taxed. No matter if it is buying or selling food, apartments, or companies. Each year, for everyone, we would compute tax on the accumulated absolute value of all their incoming and outgoing transactions. Each transaction contributes to the yearly accumulated sum of both parties in the transaction.

In this way it would not matter in which way you achieved your wealth. If you want to use it, you have to pay tax on it. If you do not have a lot, you pay less, if you have more, you pay more. If you wire money out of the country, you still pay taxes, so there is less incentive for wiring it out (or you spend it in small transactions in the country, or you make one large to get the money out of the country, it is the same). If you receive money, you pay taxes even if you have not spend the money. And you pay again when you spend it.

Many taxes already work like that so many mechanisms to record such transactions are already in place. For example, when you buy things in stores. This would be just generalization and simplification.

Then we just have to decide on the exact function used to compute the tax from the value of all your transactions. How much we want to tax the wealth. Should it be a progressive tax? Do we want to have any minimal limit under which you do not have to pay any taxes? These are then political questions.

Debt and moral obligations

Sun, 05 Jul 2015 05:58:10 -0700

Those lazy Greeks. They should pay their debts! If you take a loan you are obliged to pay it back. But are you? Let’s assume that it is true, if you lend something then you should return it someday. But there is a big difference between returning a loan and paying also the interest on that loan. Are you obliged to pay also the interest? And how much of the interest should you pay?

One could argue that it does not matter. If you accepted the terms of the loan, you should respect them. But these days loans are made with interest which grows exponentially through time. The idea, I hear, is that you are not paying back for the service of being given a loan, but you are paying for the potential profit that money could bring if it was used differently, instead of it being lent to you. Longer you are returning the loan, longer could that money be used differently, so the interest grows, exponentially with time. But here an important assumption is hidden. That money (or capital) can produce an exponential growth of profits. This is of course a wet dream of investors, but in practice it is simply impossible. Maybe you can see a temporary exponential growth, but in the longer term you will experience or crash or devaluation. It is simply impossible to have an exponential growth with limited resources, space, and time available on this planet. Should we then judge debtors by the standard which is in fact impossible to reproduce otherwise?

Where does this lead us concerning the debt? If you offer a contract with conditions which are in general impossible to satisfy (except for few lucky ones which do manage to experience a temporary exponential growth and were able to pay the loan back in time, before it got out of hands), and you trick somebody to accept them, who is to blame? The person (or country) who needed a loan or you, who wanted not just a reasonable award for the service you offered, but were greedy and wanted an exponentially growing award? And now you are complaining that a reality struck you? You were simply greedy.

But what would be a reasonable award for giving loans? Luckily, there are already alternatives. For example, there are already interest-free banks in existence. Or should we calculate how much real profits would money produce if invested otherwise? Then in times of recession those profits might even be negative, meaning that debtors would have to return even less than they took out.

How to get people to work more than 8 hours a day for you?

Thu, 18 Jun 2015 05:28:10 -0700

If I approached you, the most talented person around, you with great knowledge and superior mind, you who can make any concept into a reality, and I told you that I want you to work for me and develop my idea. I want you to work a lot more than 8 hours a day, let’s say 12, 15? I do not want that you do anything else. You should have have no life, you should more or less live in the office. You should think about the project every moment of your life. Breath the project. Would you accept it? I would pay you as little as possible, just so that you can get through in this expensive city. I will maybe give you 10% of the profits of the company, if we grow really blazingly fast, but if we do not grow fast enough, I will cancel the project and fire you. I would even ask you to invest your own money, and require you to ask your parents, friends, to do the same. Would you accept? You who could work part-time and still earn enough to live a comfortable live because you are well educated and you do not have to search for jobs, because you are getting job offers daily. You can be picky. Would you still choose to work for me instead?

You would. You do. And I will tell you why.

We know that ideas are cheap. For each given idea there are different groups of people working on that particular idea. Your idea is not special. And you are also not special. You are maybe good, but there also others who are good. You maybe have a special set of skills, but others have another special set of skills.

Today in this globalized and interconnected world this means that if I have an idea I would want to make a reality, instead of me creating a company, trying to hire people, and face the reality that they will not work for me under conditions I described above, a much easier way is to announce to people that they should come to me with their ideas. I will listen to them, waiting for a group to come and tell me that they are working on my idea. Of course, they will believe that it is their idea, but they do not understand that they are not special. I will help them create a company, give them a bit of money, request just 30% of possible profit, and let them work whole days on “their” idea. I will let them fight for me. If they will grow fast, I will provide more money so that they can continue working for me, requesting next portion of the profit in exchange. And because their growth will not be blazingly fast (I admit, my idea was not the best) and the project will take even more time (normally, they overestimated how good they are and simply could not know all the issues they will be facing; any project has issues, you might not know which those will be, but you can be assured that there will be issues), so I will allow them to work even more and take the rest of the profit, maybe leaving few percents to so called “founders”. To those people who were so happy when I decided to allow them to work for me. Oh, what silly people. They were so happy. Sharing the success with their friends. Success that they can now work for me for more than 8 hours a day. I can at any point decide that things are not going well, that maybe my idea was even worse than I though, and I can pull out, effectively firing them. Maybe they get somebody else to hire them again, for another share of their profits, if they have anything left. Maybe they still succeed, in this case profits are still mine. They do not understand that me deciding not to fund them in the next round means effectively that I fired them. And me funding them means that I am just continuing paying their salaries. Why would I get a bigger share just for continuing allowing them to work for me? But they do not understand. They are happy. And I am happy. Win-win.

Imagine instead that we would use the globalized and interconnected world to get all these people to work together instead on that one idea they share? Instead of making them compete and under the pressure, they could just collaborate. They could take time to do things properly, thoughtfully and with a long-term vision. Hey, maybe you should consider also the environmental impact of your idea. We would not require that the only thing about the project is the growth of the userbase. Some projects are too important for that. But we as a society do not know how to approach their realization. Or even how to recognize them. We use the same hammer for all nails.

Why vacations and 8 hour workday matter

Mon, 15 Jun 2015 03:23:10 -0700

Vacations and the 8 hour workday is a great achievement of a human society. It might not be seen as something revolutionary, but it is. It is a gentlemen’s agreement. Despite some of us being able to work more, at least sometimes, during some periods of our life, we collectively agree that we will not. This is not because we are lazy. But because we want to protect those who cannot afford to work more. 8 hours is an arbitrary decision (we could work less), but it is an important one.

I hear the argument that why would anyone work less if you can work more? Why limit yourself? The main answer to this is that you are not always able to work more. Maybe currently you do not have a family. Maybe you do not have any other interests. Maybe you do not want to volunteer for some cause. Maybe you do not read or continue learning. But maybe that is just at this moment. Things change. So when we as a society put a limit on how many hours to work and we all respect that limit, we allow all of us more time for other things. Maybe you do not see the benefit of that yet, but there will be time when you will be glad for it. And because limit is for everyone, we can still compete the same, if we like to compete. We can still determine who can do more in those 8 hours and who less. The only difference is that we also have 16 hours for other things. Like raising our children. Enjoying company of loved ones. Sleeping. Reading. Creating things which are not connected to our vocation. I would even claim that having those 16 hours help us do better at those 8 hours of work.

But we are forgetting that this achievement is something we as a society created and like any other social contract can work only if we believe in it. But today people are proud that they can work more. When they take work home. When they try to minimize time they spend eating every day. And yes, there are times when some of us can do that. When we can work more than somebody else. But this is a very short-sighted approach. Of course you can do more. We know that. But you should not. And you do not know that.

Meteor Blaze Components

Wed, 08 Apr 2015 15:38:10 -0700

Blaze Components for Meteor are a system for easily developing complex UI elements that need to be reused around your Meteor app. Live tutorial.

Trust and decentralization

Mon, 24 Nov 2014 12:12:10 -0700

With recent NSA surveillance revelations many in the tech community started asking again for more decentralization and encryption of our Internet tools and services. We should build technologies which allow us to minimize trust and dependencies on others. Tools which allow each of us to self-protect ourselves. A new wave of such tools is emerging and while I agree that it is good that we have such tools, the question is if we want to live in such a society. In which society you would like to live? In society which is build around not trusting anyone. Or in society where you can trust your fellow beings and where we handle rare exceptions, misuses of trust?

I am not proposing any changes in direction of technological development of such decentralized tools and services, or our adoption of them. But I think we should rethink our perspective on the issue and solutions. We should change our mindset when we are developing these tools.

Instead of thinking how we want to minimize trust of others, or even how to create zero-trust systems, we should know that we can trust most other users on the Internet. That we can rely on them and that it is good that we have a community where we cooperate based on the trust. The issue is that we do not know who we can trust and how to handle cases of misuses of the trust. But this is then a good direction to think about. Let us develop ways to handle rare exceptions, develop ways to detect exceptions in advance, maybe even prevent them. Instead of saying that because we cannot trust few, let us not trust anybody, we should be saying let us trust everybody, but know how to handle well those few who misuse the trust.

Decentralization can be viewed as means to combine/coordinate efforts of people towards a common goal.

For few years now I have been involved in open wireless network in Slovenija, wlan slovenija, build around open source mesh technology. Mesh networks are now again becoming popular because they are decentralized. But for me their decentralized nature is not important because it would give us protection against NSA (in fact security properties of current mesh networks are pretty bad), but because it allows community to come together with little coordination overhead and build a network together. A network where none might be before, providing Internet connectivity to everyone.

Wikipedia is an encyclopedia anyone can edit. It is build around the principle of assuming good faith of fellow editors. Of course this does not mean that there are no misuses and vandalism. But it is build around trust, understanding that vandalism is a rare exception compared to all the edits made on Wikipedia. And it works. Its editing is decentralized to anyone in the world with little coordination between them.

Bitcoin is build in a decentralized manner in a way that each participant is trusted minimally. They work and vote together and if more than 50% agree on something, this is taken as true. The idea is that it is highly improbable that more than half of users will be corrupted or compromised. In this respect it is similar to Wikipedia, where we also assume that most of the users will not be malicious. But the perspective is different. We do not assume good faith in fellow users of Bitcoin.

So while basic ideas of distributed trust might be similar, the perspective is one which makes the difference.

There is research done based on the observation that you can often trust people in your social circles more. One can use information about your friends from Facebook to build decentralized systems which are even more resistant to coordinated attacks of malicious users. But this is not the type of trusting others I am talking about here. Trusting your friends more is nothing special, of course you know them. But that you can trust a stranger, that is what makes society strong. That you can trust even when you do not have any other information about a new person. The rest of society promises to handle a misuse, if anything happens. Why are not we building technical systems which would work the same?

Sadly, in our current society it is not true that we always handle misuse of the trust properly. But we should then strive for it. Instead of giving up on the society and collaboration and trying to build technological fixes, we should work on fixing what we already have. Laws with holes and situations where those with power escape consequences of misusing the power.

Presenting some ideas of what is missing in open access and open science community

Sat, 22 Nov 2014 03:09:10 -0700

Presenting some ideas of what is missing in open access and open science community at OpenCon 2014 in Washington, DC. Slides.

My presentation of PeerLibrary at OpenCon 2014

Sat, 22 Nov 2014 02:59:10 -0700

My presentation of PeerLibrary at OpenCon 2014, the student and early career researcher conference on Open Access, Open Education, and Open Data. Slides.

My talk about how we can create real digital libraries

Mon, 27 Oct 2014 17:30:10 -0700

My talk about how we can create real digital libraries. Those which not just archive publications, but provide social interactions around books as well. In short: involve the community to build tools for various social interactions around digital publications and for that to be possible publishers should offer APIs.

Etage – A general data-flow framework featuring nondeterminism, laziness and neurological pseudo-terminology

Fri, 03 Oct 2014 22:56:10 -0700

Etage is a data-flow based programming language. It is build upon Haskell and provides nondeterminism and laziness. It is text based and through few basic commands it allows defining operations (neurons) and connections between operations (nerves) along which data can flow. Moreover, program can form a network with loops and is not limited to only tree-like structure.

Introduction

Current mainstream programming languages are based on the control-flow concept. They are series of instructions to compute, modify memory, do IO, which more or less abstract another similar series of instructions running on a CPU. While there are many variations and ways of abstraction of control-flow programs, there is another way how to specify a program and this is by defining how data flows in the program, instead of defining how control flows. Such programs are data-flow programs.

In the past there were attempts at data-flow computer architecture, like Manchester Dataflow Machine, but have not managed to outperform control-flow processors and architecture and gain wide acceptance. Nevertheless, we can still design a data-flow programming language which is then compiled and executed on a control-flow architecture.

Motivations for expressing a program in a data-flow manner are multiple. Sometimes a problem itself is easier described in a data-flow manner. Sometimes we can discover and gain performance advantages by explicitly defining and isolating interactions between data in the program, which helps automatically parallelizing programs to multi-core architecture. Our motivation was to be able to easier model complex behaviors of high-level neural and signal networks found in nature. If you observe how signals travel from senses to brain and back to muscles in an animal, you notice that behavior is closer to data-flow than control-flow.

We implemented our data-flow programming language Etage as a framework on top of Haskell programming language. Haskell is a lazy functional programming language with many bleeding-edge concepts and serves well as a host. Main features we find useful are support for lightweight threads called sparks which can be automatically parallelized and a powerful type system which allows checking if our data-flow program has correctly connected flows. The downside is that syntax itself is directed and limited by that of Haskell, so programs are sometimes more verbose than they would have to be with specific syntax. We could address some of these issues with a preprocessor stage. Etage is open source and is available as a Haskell package and it comes with extensive documentation. In this paper we present its design.

Example

To easier understand how our programming language works, see the following example program, a program which outputs series of Fibonacci numbers, as a control-flow and data-flow programs.

a := 0
b := 1
repeat:
    print b
    c := b
    b := b + a
    a := c

Data-flow program is shown graphically for easier understanding. It consist of neurons (also known as operators) and directional nerves (connections between operators). Neurons can be arbitrary computations which receive zero or more inputs and have zero or more outputs.

Observe that in our case data-flow programs can contain loops. See that the data-flow program first prints $1$ because print neuron is the only one which has data on all its inputs, and at the same time that $1$ is send back to inputs of + neuron, once to each input. Now + neuron has both $0$ and $1$ as inputs (including one more $1$ waiting after $0$) and outputs new $1$. Data flows again and next time it outputs $2$ and so on, the print neuron printing those outputs as they come. With correct arrival order of data to + neuron inputs we achieved summation of previous and current value.

incubate $ do
  nerveDump <- (growNeuron :: NerveOnlyFor DumpNeuron) defaultOptions
  nerveSum <- (growNeuron :: NerveBoth (FunctionNeuron IIntegerList IInteger))
                (\o -> o { function = fibSum })

  liftIO $ do
    t <- getCurrentImpulseTime
    sendFromNeuron nerveSum $ IValue t 0

  nerveSum' <- liftIO $ branchNerveFrom nerveSum
  nerveFused <- [TranslatableFrom nerveSum, TranslatableFrom nerveSum']
                  `fuseWith` listFuser

  nerveSum `attachTo` [TranslatableFor nerveDump]
  nerveFused `attachTo` [TranslatableFor nerveSum]

  liftIO $ do
    t <- getCurrentImpulseTime
    sendFromNeuron nerveSum $ IValue t 1

The same program in our programming language Etage. While before we shown a graphical representation of the program, programming language Etage is text based and you describe neurons and nerves as a series of command to create them and connect them together.

First we invoke our monad which encapsulates all logic of growing neurons and connecting them together, and waiting for them to finish. Then we create (grow) two neurons. We determine which neurons we create by specify the return type of the growNeuron function. Additionally, we have to specify if nerve attached to the neuron should serve only as input, only as output, or both, some neurons can work in different configurations. By explicitly stating this, type system can help us verify if we are using neurons as expected. DumpNeuron has only one input and it prints any input to the console. FunctionNeuron has both input and output. It is defined as a general neuron which takes a function to run on inputs to get next output. In our case our function fibSum receives a list of integers (two in our case) as input and returns one integer, sum of input integers. (Definition is omitted for brevity.)

Then we prepopulate input of FunctionNeuron with $0$, after which we branch nerveSum and fuse it together back. This makes nerveFused in fact be two nerves, in a similar way that our graphical representation shows how each output goes back to FunctionNeuron twice. We then connect output to DumpNeuron, and duplicated one back as inputs to FunctionNeuron. We start everything by pushing $1$ as next input.

This example looks a bit complicated and this is truthfully so because it tries to showcase multiple features of our programming language at the same time. In reality, we would not program like this and we would simply use DelayNeuron to delay input for one value, allowing easy summation of previous and current value.

Type system checks if types of inputs match types of outputs, or at least that there is translation defined between output and input types. For this we are using an existentially quantified types TranslatableFrom and TranslatableFor encompassing all nerves which can be translated to and from the same data type, respectively. Translators between data types can be defined through Haskell’s type class system which allows great modularity.

There are many production systems build upon data-flow programming model, but most of them are visual: PureData, Max, LabVIEV, or abstract systems: MillWheel¹. But all those data-flow implementations are deterministic. We decided to design out programming language without any guarantees on determinism to allow easier parallelization of its execution. If needed, determinism can be build upon our programming language with concepts like token-passing.

In addition, our data-flow programming language is lazy with an interesting property that information about data traverses nerves in an eager manner which invokes neurons to run immediately when data is available on their inputs, but data value is computed only if needed in a lazy manner which means that computation itself in a way back-propagates to previous neurons to compute their computation necessary for computing resulting value.

While other data-flow systems forces operations to be assembled in a tree-like structure, out programs can be arbitrary networks with loops if necessary. Programming language does not provide any guarantees about program termination and bounds on memory consumed by data accumulating in nerves if neurons are consuming data slower than other neurons are producing data. In theory nerves are unbounded, in practice they are limited by memory available. Correct programs should probably never have a nerve growing in size constantly.

Design

class (Typeable n, Impulse (NeuronFromImpulse n),
       Impulse (NeuronForImpulse n), Typeable (NeuronFromImpulse n),
       Typeable (NeuronForImpulse n)) => Neuron n where
  type NeuronFromImpulse n
  type NeuronForImpulse n
  data NeuronOptions n

  mkDefaultOptions :: IO (NeuronOptions n)

  getNeuronMapCapability :: NeuronOptions n -> NeuronMapCapability

  grow :: NeuronOptions n -> IO n
  live :: Nerve (NeuronFromImpulse n) fromConductivity (NeuronForImpulse n)
    forConductivity -> n -> IO ()
  dissolve :: n -> IO ()

  attach :: (NeuronOptions n -> NeuronOptions n) ->
    Nerve (NeuronFromImpulse n) fromConductivity (NeuronForImpulse n)

Design of a programming language is based on two basic concepts: neurons and nerves.

You define neurons with input and output types and computation which is run when input data is available. Neurons can keep arbitrary internal state. Neurons can also do IO operations. IO operations can in this way be easily encapsulated without other neurons having to know about the nature of IO operation. Computation done by a neuron can be programmed in Haskell or another data-flow program. Neurons are defined as Haskell class types where one has to define mainly grow, dissolve and live methods for three stages in a life cycle of a neuron.

growNeuron :: (Neuron n,
  GrowAxon (Axon (NeuronFromImpulse n) fromConductivity),
  GrowAxon (Axon (NeuronForImpulse n) forConductivity)) =>
  (NeuronOptions n -> NeuronOptions n) ->
    Incubation (Nerve (NeuronFromImpulse n) fromConductivity
                      (NeuronForImpulse n) forConductivity)

attachTo :: forall from for forConductivity. (Impulse from, Impulse for) =>
  Nerve from AxonConductive for forConductivity -> [TranslatableFor from] ->
  Incubation ()

fuseWith :: forall i j. (Impulse i, Impulse j) =>
  [TranslatableFrom i] -> (ImpulseTime -> [i] -> [j]) ->
  Incubation (Nerve (FuseFromImpulse i j) AxonConductive

Nerves are predefined and serve to connect neurons together. You can attach them to nerves, branch them and fuse them together to create arbitrary networks of nerves between neurons. Nerves operate in FIFO manner and have guaranteed order.

Data types send around the program can be extended through type class. Various neurons can support various data types and to assure compatibility between various neurons, explicit translator class type instance can be defined. In this way Haskell knows how to automatically translate between various data types as needed.

Conclusion

While presented programming language promises an alternative approach to programing data-flow programs through text based programming and offers features like nondeterminism and laziness, it has to our knowledge not yet been used in a larger program to really determine usefulness of these features. Moreover, it has not yet been used to model high-level neural and signal networks, what was our initial motivation. On the other side, design seems possible to implement and implementation was successfully stress tested. Extensive use of Haskell type system allows compile time verification of data-flow programs and assurance that inputs and outputs are correctly connected.

Tyler Akidau, Alex Balikov, Kaya Bekiroglu, Slava Chernyak, Josh Haberman, Reuven Lax, Sam McVeety, Daniel Mills, Paul Nordstrom, Sam Whittle. MillWheel: Fault-Tolerant Stream Processing at Internet Scale. 2013. ↩︎

PeerLibrary is calling for collaboration on open dataset of all academic publications

Sun, 27 Jul 2014 15:34:10 -0700

via peerlibrary

xml4js – XML to JavaScript parser using XML Schema to guide conversion

Sun, 22 Jun 2014 04:17:10 -0700

XML to JavaScript parser which uses XML Schema to guide conversion. Main motivation is that instead of guessing the structure of an XML document from the document itself, structure is created automatically from the XML Schema which makes output consistent for all XML documents, not just one at hand. For example, arrays are always arrays even when there is only one element in the particular XML document, and if there is only one element allowed, then there will be no one-element array wrapped around. This makes programmatically traversing the structure much easier, because structure is consistent and predictable.

Ideal of a economy in the society

Thu, 29 May 2014 11:01:10 -0700

I proposed the new global economic system which allows gradual transition from the current system. But to what would such a system lead? What is the ideal organization of economy in the society?

I do not know, but I like to imagine the following metaphor. Imagine a village with a big warehouse in the middle. Whatever you need you go that warehouse, find it, and take it. Whatever you produce you take to the warehouse and leave it there. You do not get any money for leaving things there, you do not spend any money to take things.

In front of the warehouse there is a bulletin board where people can request things so that you know what people would like for somebody to produce. People work on anything they want, or nothing. In the era of abundance it is not important if we all work or how much we work. It is not seen anymore that it is unfair if somebody works and somebody else does not, because anyone works only if they want to anyway. And what exactly is work in such a society anyway? You live, socialize, play, create, and if you decide to share what you created, it is easy. And this holds for all.

Gradual transition to the new global economic system

Sun, 25 May 2014 02:41:10 -0700

I wrote about the new global economic system in the past. One property I like is that it provides a simple and gradual transition to it.

Imagine a hairdresser. The hairdresser could start participating in the new economic system by starting offering few haircuts through it. Other people participating in the new economic system would apply for those haircuts and the hairdresser would based on their past transactions inside the new economic system determine for which of all applied would do haircuts. For each of those haircuts, both the hairdresser and a person receiving a haircut would leave each other a reference, recording and evaluating a transaction. Such a record can be highly subjective, personal, elaborate, and even objective, all at the same time through various ways transactions could be described. For example, simply as free form description, optionally expressing general sentiment of a transaction by choosing from a predefined set of values. Important is to notice that both a person performing a service (hairdresser) and a person receiving it get a reference and feedback, not just the hairdresser. In this way it is not just important how you performed your service, but also how you received it. The hairdresser could be very entertaining while performing a haircut and people could like that and describe that recommendations. But at the same time also a particular person receiving a haircut could have many interesting stories to share, making hairdresser’s service more pleasant. With these new references and a recorded transaction both of them can in the future then engage in the new economic system with others.

In addition, the hairdresser has benefits even in the current economic system. References received can be used to encourage customers to buy hairdresser’s haircuts. References and transactions can serve in a way as a alternative to Yelp.

Cargo Cult Science

Sat, 17 May 2014 18:05:10 -0700

Feynman’s 1974 Caltech commencement address.

We made a new video talking about the emerging community of PeerLibrary

Sun, 27 Apr 2014 15:25:10 -0700

We made a new video talking about the emerging community of PeerLibrary, as well as the project’s origins in the open access movement.

via peerlibrary

Presentation of PeerLibrary at iAnnotate 2014 conference

Thu, 17 Apr 2014 10:45:10 -0700

Presentation of PeerLibrary at iAnnotate 2014 conference in San Francisco. Including the demo of current version in development, 0.2.

via peerlibrary

Are corporations a new form of slavery?

Sat, 29 Mar 2014 07:30:10 -0700

It is interesting to think about the whole concept of shareholding and corporations. It seems like a modern form of slavery. Because you cannot own a person anymore we developed a concept of corporate personhood, which we can own, sell, resell, split, merge, while it is doing work for you, its owner.

Instead of directly owning people we created an abstraction layer in between. This abstraction layer, a corporate person, does work for you, you own it, you can benefit from it without having to really work, just by mere fact of owning it, you are getting benefits, interests. You exploit the corporation. Sometimes bad owners push it too far, want too much profit, so the corporation cannot sustain itself anymore, it bankrupts, it dies.

You can see an example of this in factories taken over by workers in Argentina. Owners were not getting enough profit from factories so they decided to close them. But for workers factories were producing enough income to cover their wages. Once factories become free of owners they could live, before they would die.

Corporations are the victim here. And through corporations, all the people working in them. Corporations are a tool to own people.

Who is an owner? Who is a shareholder? In contrast with workers and worker-owners they do not really work in the corporation, they do not contribute to corporation’s business through personal work. They have control over the corporation. Owners pay other owners when they buy a (share of a) corporation, they do not pay workers, they do not pay the corporation.

Shareholding is an interesting concept as well. Instead of one owning a whole slave, you own just s small small portion of it. Then we can maybe say that nobody is really a slave owner, because that small portion is insignificant, but all together, collectively, we are still slave owners? Are we then collectively responsible? Or are we more diffusing our collective responsibility in this way? We pretend that when we buy a share, we are not really buying part of a real person who will now work for us? Because now almost anybody can be a slave owner, own at least one share, we cannot admit to ourselves what we are? We talk about how our share appreciate through time, just by itself, without us having to do any work. We just had to invest. Is there a big difference between past slave markets and current stock exchanges?

If we look how wealth moved after slavery was officially abolished, same privileged people who owned slaves directly before, could own corporations instead, which then employed workers. Power relations did not change much. Even if workers are now employed, their income is still small in comparison with owners. Their income more or less just covers life expenses, similar to how slaves were given food to survive. It would be interesting to compare how much food and housing slaves were given in comparison to their owners, and how much food and housing workers in corporations are given in comparison to corporations’ owners.

To be precise, through corporations we do not own other people directly, just their work. (Which is arguably the main reason why we owned people in the first place.) We can then sell and resell the ownership of work on stock exchanges. We introduced a layer of abstraction between work and its ownership. And if needed, we can add more layers of abstractions by introducing more corporations owning other corporations. To be precise, workers in corporations can theoretically freely decide to move to another corporation, so they are not stuck with their owners. But this freedom is quite theoretical if you know how hard is to get a job this days. And if you can freely chose between your owners, but have to stay in the system, have to stay under a corporation owned by somebody, is this really a freedom? Of course you could always buy yourself real freedom, but where would you get money for that?

I do not believe corporations developed this way consciously. They developed as a way to limit liability, but once we added ways to sell them and since our economy is based on debt and workers in most cases work to pay off a debt, we have slavery again.

I do recognize that real slaves were and are in much worse situation than many current workers in corporations and I do not want to minimize their suffering and issues, but on the other hand I want to show that we maybe officially replaced slavery with something more human and civilized, but maybe conceptually not much different.

Owning a corporation allows owning a person. Or at least person’s work.

Data-flow society

Fri, 28 Mar 2014 07:30:10 -0700

I have proposed a new global economic system which, instead of trying to reduce transactions between people to one numerical value (money), stores all information about the transaction itself so that then later on everybody can assign a personalized value to the transaction.

An interesting observation is that we can see people as nodes in a network and transactions between them as connections (edges) in the network. The question is how we define these edges. Do we assign an edge a precomputed simple value, a number, which is made at the moment the edge itself is made. (This is what we currently do.) Or do assign context of the edge to the edge, so that anybody can evaluate the edge by themselves. (This is what I am proposing.)

I think this could be generalized to many different networks found in a society. We can see relations as interactions between peers in a network. Do we then assign to edges values as “objective” facts, independent from who is observing/evaluating those relations. Or do we embed into edges contexts and stories so that everyone listening to them can evaluate them for themselves. Do we remember interactions between people as a simple value, what we gained or lost from the interaction, or do we remember interactions through stories, experiences.

By observing this we can see that a general tendency in our society today is to reduce interactions between people into simple values, evaluate them immediately, in short term, and then remember them forever as such. When we argue who is right we list facts and try to determine a conclusion. Once we find a conclusion, we assume it will hold forever. We rarely go back and observe that environment or context changed. We can see most of our laws are defined in this manner.

But it is not necessary to be so. Traditionally we can see that stories and story-telling, anecdotes and personal experiences, had been an important way for people to pass on memories, tell each other interactions. Each such story tried to encode the context of the event, but then listener was able to interpret it by themself, adding their own values and experiences. Today we are reducing interactions to simple “likes”, a very simple value. But we are losing that expression another person leaves on you when you tell them your story.

Maybe it is not enough to reimagine our economic system, maybe we have to reimagine also how we remember other our interactions. Let’s remember stories. Let’s write down laws together with stories which prompted a rule. So that we can later on reevaluate if stories are still applicable to a changed world. Maybe the rule will not work anymore directly, but having stories would allow us to still apply it to a new situation.

Thanks to Derek Razo for triggering this line of thoughts.

Proactive federation of commons is better than just federation

Thu, 27 Mar 2014 07:30:10 -0700

For PeerLibrary project we had discussions on how much distributed or centralized should we make it. It is a cloud service and centralizing (unifying) user base and content has clear benefits for both end users and developers. End users have better user experience and ease of use, all content is available quickly and easily, and all other users are there, making social experience better. For developers, code can be much simpler and maintenance of only one instance makes it easier to deploy new versions and push security fixes quickly. It is easier to collect statistics and do A/B testing on a large sample. On the other hand, having multiple instances of PeerLibrary distributed around the world makes whole system more robust, specialized instances could be offered, privacy of users increased. Having PeerLibrary distributed would make forkability easier, encouraging more community control of both the project and the content (commons), preventing corruption of the main instance or core project.

I argue that what if we want a distributed system, what we in fact want is what I call “proactive federation” and not simple federation.

By simple federation I have in mind a project that supports running multiple instances which communicate to each other and sync between themselves. This gives a promise of data not being siloed, a promise of data being open and for everyone to use. But I believe we can do better. It is not so important that data is stored in different locations, but that data is used for different purposes and processed in different systems and simple federation does not addresses this.

With proactive federation I have in mind a platform which proactively pushes its data to other, different, platforms. Other platforms which have different use cases and do different things with data. Proactively means that platform itself takes care of distributing data to other platforms. Important part is that they are different platforms, different codebases, different organizations, different reasons for their existence. This makes data really useful: it finds use in use cases not imagined or possible by original project. It makes sure that data is assured longevity because different projects take care of it. It makes sure that it is combined with other data. If we compare to nature, instead of trying to prevent extinction of a particular gene by cloning, we should mix the gene with genes of other organisms. This improves whole ecosystem.

Technically, it also improves access to data. If data is available only through API for a project A, then any other project has to implement project A API. If project A pushes to project B, then and other project which already uses project B API benefits from that. We get a network effect.

As data crosses borders between projects we might say that data will lose some of its original properties. Maybe. But does translation between languages always just loose or it sometimes also adds? Adds new culture, new context? In the long term I believe this will enrich commons. We should not be afraid of imperfect copies. This is how progress is made.

So instead of using time and resources on how to make federation between instances of PeerLibrary I am more interested in how to proactively push data to other projects. To push metadata about academic publications. To push annotations of academic publications. To push relations between academic publications. Let’s share, proactively share. Because you never know how will data available in some other project spark a new idea and use.

Universal basic Bitcoin income

Wed, 26 Mar 2014 07:30:10 -0700

I have a proposal for a double social experiment, merging two disruptive ideas. One is that Bitcoin technology is removing government from the equation of controlling and taxing money flows, which is arguably better because it allows people to freely decide how they want their money to be spend and not be forced by others. The other is universal basic income, a system of social security where everybody regularly receives an unconditional sum of money. Let’s create a way for all Bitcoin users to voluntarily add Bitcoins to a common wallet from which all users would then unconditionally get a regular and equal payment.

Argument is that if we would not force people to contribute to social welfare through taxes they would still philanthropically donate enough. That we just have to allow everyone to decide on their own how much to contribute and to which socially important goal or service. Would we have enough Bitcoins donated to the common wallet for payments to be of any significant value? Above minimal salary? On the other side, would getting a regular and equal payment really improve the economical situation of everyone, giving them more freedom to do with their lives whatever they would want to? Or would those payments just offset a basic cost of everything for exactly the same amount, not really changing anything? Rents on housing would just increase for exactly that amount, making those payments automatically flow back into the hands of those who own housing?

(An open question is how to make it so that only real physical persons receive a regular payment, because one could otherwise create multiple fake Bitcoin users to receive payments multiple times.)

Wikiprogramming – editing a program from the program itself

Tue, 25 Mar 2014 16:58:10 -0700

I am still waiting for somebody to create a programming framework where user could edit the program itself while it is running. Each user interface element would contain an “edit” button and by clicking it you would get source code editor for all code related to that element. Edit it, try it temporary, click save, and you have a new version of the program.

This is even more interesting if we imagine a webapp, a cloud service, which would be built upon such a framework. Users could immediately edit the webapp, without having to establish a development environment somewhere else. You would use the webapp, see something to improve, click edit, try it, save it, and everyone else would get this new version of the program. All changes would be stored so somebody else could reverse back, or build upon your change. This would make programming more accessible and approachable to novices. Develop faster. Community of users would also not be so locked into what developers imagine a webapp should be and do.

What about security you say? What if somebody adds some malicious code? But is this really so different than what happens on Wikipedia? If you read something misleading there, or if you run something misleading here. It is vandalism in both cases and similar measures could be employed. Or even some new ones, like not applying the new code to everybody immediately, but A/B test it on a random subset of users. Or ask a random subset of users to review it. This would be a nice addition to Wikipedia as well. So some ideas how to improve Wikipedia itself could be applied here as well.

Of course you could also run two instances of the webapp, one for collaborative development, one for users to use. With some process of code review in between.

In gift economies exchanges do not have the impersonal qualities of the capitalist marketplace

Tue, 25 Mar 2014 07:30:10 -0700

In gift economies, Mauss argued, exchanges do not have the impersonal qualities of the capitalist marketplace: In fact, even when objects of great value change hands, what really matters is the relations between the people; exchange is about creating friendships, or working out rivalries, or obligations, and only incidentally about moving around valuable goods.

As a result everything becomes personally charged, even property: In gift economies, the most famous objects of wealth - heirloom necklaces, weapons, feather cloaks – always seem to develop personalities of their own.

David Graeber on MAUSS.

Local cooperation benefits global competitiveness

Thu, 20 Mar 2014 10:07:10 -0700

The basic biological answer is that local cooperation benefits global competitiveness. But that depends on the definition of local, either actual or perceived!

As the Chinese say, in times of small trouble go to the city. In times of big trouble go to the countryside. The rest is commentary!

Answer by Robert Dudley to my question which is more evolutionary “natural” for humans to do, cooperate or compete.

Net neutrality and traffic shaping

Tue, 11 Mar 2014 03:00:10 -0700

Net neutrality is an important feature of the Internet, but it is at risk. Net neutrality means that Internet service provides must not discriminate and must transport all traffic with same priority and quality. ISPs would like to be able to discriminate because this opens new profit sources for ISPs. For example, they can start asking various Internet services to pay them, or their traffic will have lower priority than some other competing web service.

Profit is not the only reason why would somebody like to discriminate traffic on the Internet. One more reasonable reason is that sometimes to assure good quality of service, it is better for ISP to prioritize, for example, latency sensitive VoIP protocol over bulk download of the movie. This means that you can still use high quality voice communication over the Internet, while at the same time consume your whole link for movie download.

But one can look at the issue from the other perspective as well. The issue of net neutrality is the issue of a power relation. Because ISP has power to decide on the traffic shaping policy, this power can be misused or at least used against end-user’s wishes. The solution is thus simple. End-user ought to be the one deciding on priorities of their traffic. ISPs ought to provide a protocol through which end-user can request how ISP ought to shape incoming traffic (outgoing traffic end-user can shape by themself). ISP can provide a default (high priority for VoIP), but end-user should be able to change those rules and add their own rules. If traffic shaping is done with the consent of the user then this is not discrimination anymore. But a feature, and it still serves the purpose of higher quality of service for the end-user and better user experience with ISP’s service.

Evolutionary reasons for music and dance

Sat, 22 Feb 2014 17:12:10 -0700

Isn’t it interesting how music and dance are so widespread among human cultures? How music makes us move, how we feel that we should respond to music? But why we feel like that? Do other species have music and dancing as well? Not one which is hard-coded in their genes, but one where they can improvise, be creative, go crazy? Do monkeys sing? Is maybe music the reason why we shaped our language and intellect the way we have? To be able to sign together, to be able to create new music not yet made in the past and impress our peers? To move our body in new ways not really necessary for our survival, in a way maybe none of our ancestors ever moved?

Interesting presentation of DDOSing

Mon, 03 Feb 2014 17:47:10 -0700

Interesting presentation of DDOSing as an online collective action. Is it ethical? Activism? What are important aspects of user interface which allows participation?

A new global economic system

Sun, 02 Feb 2014 17:21:10 -0700

I am envisioning a new global economic system¹ where you do not provide service or goods to another person because they pay you, but because you evaluate how much, which and how they themselves have provided services and goods to others in the past. You announce your availability of a given service or goods to others, interested apply, and based on your values, beliefs, and information about their past contributions, you decide to whom you then provide given service or goods. Such economy promotes balance, is not impersonal and can properly award cultural, artistic, and sustainable services and goods.

If you know Couchsurfing, then this will be very familiar to you. In essence it is a generalization of Couchsurfing model, while thinking about necessary components to make it scalable and distributed.

It was written few years ago, before Bitcoin, which gives even more ideas how such economic system could be technically implemented.

Such economic system can be achieved only by using cutting edge information technologies. Current economic systems based on money come from times of no global and omnipresent interconnectivity towards which we are steadily progressing. At that times, money as a medium of conveying information about the value of goods and services exchanged and creation of markets paved paths to great technological and cultural advances. Money streamlined goods and services exchanges and allowed temporal and geographical division between providing goods and services on the one side and using them on the other.

But we can do much better! Economic systems based on money have some severe limitations. They assume that all goods and services can be numerically and one-dimensionally valuated. This is inherently not true for services and goods of subjective value (such as found in culture or arts), alternative or uncommon nature (how much is teaching a new cooking recipe worth?), or long-lasting or late results (how much is using sustainable technology now worth for people which will live hundreds of years in the future?). Valuation in a western world is more and more indifferent to people involved and their personal opinions, but is left to impersonal market to determine prices, influenced by demand and supply and not by personal values and beliefs people involved might cherish. Or if influenced by values, it is just through an aggregate of those values and without considering each one of the people involved.

Furthermore, when was the last time you bargained for a price for your apples at a store of a global international food-market chain? And when was the last time you bargained to pay more than was the price of a book on sale at a book store because you believed that it is worth more? Economic systems we have currently only award the buyer for lowering the price and not for rising it - how can we then say that current market can valuate services and goods properly?

Because in current systems we valuate goods and services with simple numerical value, we are assuming that algebraical properties of numbers hold also for these valuations. But is this necessary so? Maybe it holds for apples, valuation of hundred apples is the same as hundred times valuation of one apple, but for many things it is not so. Cutting one tree in the forest have completely other valuation than cutting the whole forest and definitely is not just sum of valuations of individual trees.

Such limitations are reasonable for technologies, tools, and methods used at the time. But this days, with all information technologies at our disposal, we can do much better. We can now connect all sellers and buyers together into instantaneous communication. We can collect, store and process vast amounts of data. We can display information tailored to each individual, personalized. We do not need to abstract information about each transaction into one-dimensional value. We can do much more involved computations than just adding or subtracting them together. We have seen all this technologies being used in other fields of human activities, like social interactions, why we would not use them also for powering our economic system itself?

For the transition to the new economic system to be successful, it should be a gradual and voluntary process. This can be done by building a parallel economic system on top of the decentralized nature of the Internet by inviting existing Internet members which are offering services and/or goods to join and participate in it. More of them will join, more diverse and thus applicable to different life situations the parallel system will be, slowly replacing current systems.

At the beginning, the new economic system itself will in practice not be much more than just a standardized way of exchanging information about transactions, which will existing Internet services be able to integrate. Together with few helper services. Later on, more higher-level services, for example centralized or specialized markets, will emerge.

Only in such decentralized and decoupled way it is possible to cover all different cultural and other differences between all participants. In this aspect it is similar to current economic systems based on money. It is not really important which currency exactly you are using and how exactly are transactions being done, just that some basic properties of the system are uphold (for example, that you cannot spend the same money twice). Of course, for the economic system build upon current information technologies, such properties are of a different nature.

To easier illustrate the concept we can compare it with the OpenID standard. But while OpenID deals with distributed authentication, our system will deal with transactions. Each time a transaction (of service or goods) in virtual or physical world will take place both parties will record this information through one of participating providers (Internet services which are in the new system), each giving their feedback on the transaction, both in few quantitative values and a free form comment. In this way each involved party will leave to each other a reference, linked to their identities.

To help people finding goods and services to request, existing and new virtual and physical markets will be used. Often markets themselves will be also providers recording transactions themselves.

Once a person will find something of interested offered, he or she will contact the person offering it and request for it. In theory, the person offering it will get multiple such requests and decide to whom to give what he or she is offering based on past references (recorded transactions) those people will have. He or she can decide that based on personal values and beliefs, content, quantity and/or quality of those references. Of course very soon the number of those references each person has will explode, but smart tools, evaluators, can be employed to display only to each person relevant sample of all transactions, or to summarize them in some manner. In a similar way how Google is providing relevant search results or aggregated news on Google News. Or how Facebook limits displayed posts from your friends.

Summary of components:

standard for exchange of transactions
providers, participating Internet services which are integrating the standard to record transactions (current similar service: PayPal)²
markets, spaces (virtual or physical) where people can find what others are offering and can request it (current similar service: eBay)
evaluators, tools which help evaluate those offering services or goods, and to those offering services or goods, help evaluate and choose among interested for it; here we will probably see a vast diversity of approaches created by global community of users (currently there is nothing like that)

Providers, markets and evaluators will often be combined into one, probably around a community; often specialized in some type of service or goods, making bigger emphasis on those aspects of both what is offered on the market and how transactions are evaluated and presented.

It is important to understand that everybody can become/deploy a provider, market, or evaluator service. In a similar way to how currently everybody can use PayPal to record their (money based) transactions on their website. In the proposed system you will in theory be able to be your own provider, but in practice you will in a similar manner to PayPal outsource some existing provider. Or provide your own market for many users, or just a specialized webshop for yourself. If you will be interesting in how to evaluate transactions, maybe you will develop a mobile application visualizing transactions in a way to help others to decide to whom to give service of goods they are offering.

Furthermore, the new economic system is not limited only to services and goods provided over the Internet, it can be equally used for traditional and physical services and goods. Only a record of the transaction is stored in the Internet, and for that many providers will be available.

In fact I really dislike use of word “economic” and “economy” here because the proposed system is in fact a money-less and exchange-less system. You do not use any type of money (even any of many alternative currencies) and you do not engage in a “I give you this in exchange for that” exchange, but just mutually record that you provided, gave, or did something to somebody and your personal evaluation of this deed, and this is it. ↩︎
Today Bitcoin would be another similar but decentralized system. ↩︎

PeerLibrary 0.1 released

Sat, 01 Feb 2014 15:28:10 -0700

We released a 0.1 version of PeerLibrary. Check it out live. It is a really very beta version, with many planned features missing, but so that you can get a better taste of what is coming. Please, give us feedback.

via peerlibrary

Peer-to-Peer voting scheme

Mon, 20 Jan 2014 12:51:10 -0700

The following was a class report done together with Valkyrie Savage where we compared my idea for voting described in the previous post with others similar ideas we found around. This idea is envisioned as a backend for the massive online collaborative decision making system already described. Please comment below if we missed something or understood something wrong.

Abstract

We designed a voting scheme which allows a group of people to better decide on a common opinion about an issue. Currently, the most used approach is to simply count number of votes against and for, while not taking into account people who do not cast a vote. Our approach is to have each person define delegates whose votes will be counted when he/she does not vote him/herself. In this way we get a social network, a trust network, between users which can be used to transitively compute missing votes. We believe such a result better represents the will of the group.

Introduction

Currently the most common way of a group decision-making is voting with simple vote counting, determining the result by majority. Such a scheme is easy to understand and implement, but the question is if it does the most important thing in the best possible way. Namely, the main purpose of a group decision-making scheme is to find the result which would satisfy the whole group the most. In practice, issues arise because only a small part of the whole group participate in the decision making, namely by casting a vote. Decisions are thus based on opinions of this smaller part and despite others not participating in the decision-making they still might have an opinion or a general preference based on their values, even if they do not know exactly which voting option best satisfies their values. The reason for this is often lack of the time or knowledge necessary to make an informed decision.

In this paper we address this issue with a novel group decision-making scheme which, when determining the outcome of the decision-making process, takes into account the social network of the group and relationships between members. In short, it allows members to delegate their votes to one or more other members. These delegations are used to infer the missing vote when the member does not vote him- or herself. Such delegations are transitive and form a kind of social network between members.

The idea falls into the more general idea of proxy voting or voting with delegation. In the following section we analyze existing proposals inside this space. After that, we present our proposal in more detail, comparing it with existing proposals. We analyze its runtime and data structures. Finally, we conclude with suggestions for future work.

Smartocracy

Smartocracy¹ is a system implemented by a group of researchers and tested in a beta period with several users from their institutions. They implemented a trust-driven social network for decision making along with three different algorithms for determining which problem solutions were the most desirable by the collective: these three algorithms were direct democracy, dynamically distributed democracy (DDD), and proxy voting. Direct democracy is exactly as it sounds; a person’s vote is thrown away if she does not cast a ballot. DDD allows users to designate a single other person who is given the power to use their voting influence in the case that they do not cast a ballot. The proxy voting algorithm redistributes voting influence based on incoming edges at each node at the beginning of each vote. That is, at step 0 everyone receives an equal amount of voting power, then at step 1 voters who are trusted by other voters are given additional voting influence equal to all the voting influence of the people who trust them.

Vote delegation is transitive such that if Alice does not vote and has delegated her vote to Bob, and Bob does not vote but has delegated his vote to Cathy, then Cathy has the power to vote with the influence commanded by Alice and Bob. Delegation is done in general rather than by domain, such that a person who you “trust” is implicitly trusted in all voting situations.

Under all algorithms, solutions voted on by participants can be weighted: e.g. Alice can delegate 40% of her vote to solution 1 and 60% to solution 2. The solution with the most vote weight at the end of the vote wins. There is only a single vote, and no consideration is given to a situation in which there is a cycle of non-voters who trust each other.

In dynamically distributed democracy, voters are assigned additional voting power at the beginning of a vote, based on how many people trust them.

Delegative Democracy

Delegative Democracy² is an idea, rather than an implemented system, proposed by Bryan Ford. This system offers voters the choice of being an active delegate or a passive delegator. In the first case, they exercise their voting power on their own, along with any delegated to them. In the second case, they select one other person to whom they delegate their full voting power. The system emphasizes exercise of voting strength rather than the current paradigm of winning or losing a seat. Each person is initially given the same voting weight, and the final distribution of weights for members of the population depends upon the trust network. It is understood that the privacy of non-voting (delegating) individuals is protected by this system so that they cannot be coerced into giving their vote to particular delegates and that the privacy of voting (delegated) individuals is waived such that they are held accountable to those whose voting power they hold.

This system offers significant flexibility in delegation. It is possible to assign votes to different people depending on which forum the vote is taking place: e.g. votes on environmental policy can be delegated from Alice to Bob, while votes on space program funding can be delegated from Alice to Cathy. Delegation is transitive as it propagates through the network. Cycles in delegation are resolved through delegation ranking. Alice can state that her votes should go to Bob, but if that is not possible for some reason (e.g. Bob has delegated his votes to Alice and no one else), Alice can state that her votes go to Cathy instead. If ranking does not resolve a cycle, all the voting power of the delegates in the cycle are discarded. After delegation resolution, a single majority vote based on voting strength determines the outcome.

Delegative democracy allows voters to assign delegates of their votes (black lines) and secondary delegates (grey lines) in case of a delegation cycle in which no one casts a vote.

Toward Delegated Democracy

The Toward Delegated Democracy paper³ describes two novel methods of determining the outcome of a vote based on a trust-based social network. Voters create trust connections to other voters in a multiple transitive delegation system in which voters delegate their vote to one or many others in general rather than by domain. The transitive nature of this voting graph is described by a falloff: i.e. if Alice delegates her vote to Bob and Bob delegates his vote to Cathy, Cathy receives 1 unit of vote from Bob plus $1 \cdot (1-\mathrm{falloff})$ units of vote from Alice. This is an attempt to quantify the non-transitive nature of value systems and trust connections.

All voters are initialized with equal voting power which is distributed through the network based on the trust edges. Voting is expected to be performed cyclically; i.e. when a proposal is made, voters vote on it, and as discussion about the proposal continues, they are free to change their votes. This is a departure from the traditional voting process and the process described in all the previous papers.

The real contributions of this paper lie in their outcome estimation algorithms. Based on the votes of a few voters and the created trust graph, they can estimate the outcome of the vote. They also propose a system in which voters who have the most trust-based voting power and the weakest delegation relations (e.g., if Bob and Cathy have the most voting power of anyone else, we do not select them both if Bob delegates his power to Cathy) are selected from the social graph and asked to vote: based on the selection mechanism, they are supposed to be the best representatives of the whole voter community.

The Towards Delegated Democracy paper describes the non-transitive nature of trust, in which only $1-\epsilon$ of the blue vote is transferred from the red delegate to the yellow delegate.

A Voting System for Internet-Based Democracy (Liquid Democracy)

Liquid Democracy⁴ focuses on an iterative voting system in which voters can change their votes at any time in order to show their approval or disapproval of an idea or policy. This system is based upon a social game (i.e., the prisoner’s dilemma) in which voters identify themselves as part of the group or not. Voting is performed with transitive delegation (which is domain-insensitive), but is resolved as a series of voting vectors. A person has a total of $+1$ voting power, but can assign $-1$ or $+1$ to all measures being voted upon (such that the sum of all votes totals to $+1$), and the voting is arbitrated in order to ensure maximum happiness for the group based upon all voting vectors. Since voting is continuous, voters can “vote together” to improve the strength of their combined vectors and achieve a more preferred outcome. This is conducted per the Internet Voting System where voters' voting vectors are represented in n-dimensional space and match up with points on an n-sphere. The vectors are added in the usual fashion and the outcomes are determined by their sum.

The main advantage of this system is that it was implemented into multiple software solutions and it is used actively by various activist groups and the most prominently by the Pirate Party in Germany. The informal feedback we gathered from their users is that while theory is interesting, user experience of using the software is bad because of too complex user interface which exposes many of underlying principles but makes it hard to use for their non-technical users. This shows that such delegation schemes are in general more complex and harder to understand and simplicity is something we must have in mind when designing one.

Liquid Feedback allows voters to cast their vote as a vector on a sphere, and the result (black) is calculated based on all the voters’ vectors.

Trust-based Recommendation Systems: An Axiomatic Approach

The trust-based recommendations paper⁵ deals with the theory of recommendation systems, i.e. systems in which the end goal is to recommend new items for a user given her preferences and information about whose preferences she trusts. The authors describe five axioms which may be desirable in trust-based recommendation systems:

Symmetry (isomoorphic graphs result in isomorphic recommendations)
Positive response (if a node with a netural recommendation is attached to a + voter, the recommendation moves from neutral to +)
Independence of Irrelevant Stuff (a node’s recommendation does not depend on nodes it cannot reach in the trust web)
Neighborhood Consensus (if a nonvoters neighbors are all +, if that node votes + its neighbors recommendations do not change)
Transitivity (the “trusts more” relationship is transitive)

And they go on to prove that not all 5 are satisfiable within a single recommendation system, but that each subset of 4 uniquely leads to a recommendation system. They suggest relaxations and replacements for axiom 5 which also lead to unique recommendation systems. The authors offer a theoretical framework for the continued study of systems in which a user’s preferences are inferred through explicit or implict trust relationships.

Peer-to-peer voting scheme

In designing our scheme we approached the issue from the other direction. We see voting as expressing the opinions of people. When not everybody votes, the question is what are the opinions of non-voters and how can we include these opinions in the final result. Currently, in commonly-used voting schemes their opinions are simply discarded. We approached the issue from a machine learning perspective, seeing this as a prediction problem. We have a set of known values (votes) and would like to infer the unknown values (non-voter opinions) from them. When deciding which data to use we decided to use social network and trust relationships between people. This is based on our anecdotal observations that people tend to ask their friends how to vote when they themselves do not have a firm opinion on an issue. In our scheme, we formalize this and make it explicit, thus simplifying and streamlining the process, making it scalable and less time-consuming.

We decided to design a theoretical framework of a general process for inferring opinions from an entire population for those who have not voted. How votes and inferred votes are tallied is not defined by our scheme, but for the sake of discussion we assume that all votes are of equal weight and the final result is computed in the same manner as when only cast votes are tallied.

In our scheme each member can delegate to an arbitrary number of other members, assigning each delegate an arbitrary fraction of her total voting strength. Our scheme is orthogonal to the way in which members cast their vote. It can be a simple yes/no or for/against decision, multiple option decision, or a ranking of options. All that is required is that votes have a definite way to combine themselves from multiple delegates based on their ratios.

Our scheme is a two-stage process. In the first stage every member of a group chooses zero or more other members whom she trusts and would delegate her voting decision to in the case she does not cast a vote herself. If she chooses nobody and does not vote, her vote is discarded. If she chooses one or more delegates and does not vote, her vote is inferred from her delegates in the chosen ratios. She can also declare that she wants part of her vote inferred from her delegates, even if she does vote herself.

We present some examples. We have three members, Alice, Bob and Cathy. Alice can decide to delegate like this:

$$(\mathrm{Alice}, *)$$

This is a default which means her vote counts only if she casts a vote.

$$(\mathrm{Alice}, *), (\mathrm{Bob}, 0.4), (\mathrm{Cathy}, 0.6)$$

If Alice does not cast a vote, her vote is inferred $0.4$ from Bob and $0.6$ from Cathy. If she casts a vote, only her vote counts.

$$(\mathrm{Alice}, 0.9), (\mathrm{Bob}, 0.04), (\mathrm{Cathy}, 0.06)$$

If she casts a vote then $0.9$ of her vote is counted, but still $0.04$ and $0.06$ is inferred from Bob and Cathy, respectively. If she does not cast a vote, then her vote is inferred from Bob and Cathy in $0.4$ and $0.6$ shares.

These weighted delegation edges define a (social) network between members which is a kind of “web of trust” or “trust network”. We can see it as a directed graph between (hopefully) everybody. We call this a “delegation network”.

When a decision is needed, votes are cast. This is the second stage of the process. This is done in any manner settled on by the members and can be the same as in traditional voting. But, it is not required that everybody casts a vote and missing votes are not simply discarded. For those who do not cast a vote, their vote is inferred. This is done transitively. So in the final example above, if Bob does not cast a vote, then Bob’s $0.4$ share of Alice’s vote is inferred from Bob’s own delegations. In the case that none of Bob’s delegates (transitively) casts a vote, then Alice’s vote is wholly inferred from Cathy’s vote.

This last possibility is an unfortunate one as it means we have lost the votes of Bob and all Bob’s delegates who have not voted, transitively. This is a very unlikely event (based on the six degrees of separation idea) and possible only if we allow members to not choose any delegates (we could change the default delegation or make it compulsory for participants to delegate in the first phase).

Our voting scheme allows precise definition of voting weight between delegates, and votes are counted recursively.

Analysis

Our scheme’s definition is simple and general. From the point of view of the member she just has to (once) select delegates and ratios between them, and then the scheme takes care of the rest. Depending on the use case of our scheme, members might have to define multiple sets of delegates for various domains they are voting in.

“Casting a vote” itself is not defined on purpose. Our scheme deals only with a question of how to infer missing votes, thus augmenting existing vote casting schemes.

In social choice theory, Arrow’s impossibility theorem⁶ states limits on how ranked preferences of individuals are converted into a community-wide (complete and transitive) ranking and this begs the question: how does this relate to our scheme? We can observe that our scheme is orthogonal to this issue as it does not deal directly with the question of aggregating the votes of individuals into a common decision, but just infers missing votes enabling any approach of aggregation to work arguably better. Ideally unsolvable conflicts such as those described by Arrow (in which $A > B, B > C, C > A$) could be resolved through the addition of iteration to our voting scheme, as in Liquid Democracy.

The additional data we gather in the first stage, the delegation network, allows us to infer missing votes in the second stage. We implemented it as a graph-walking algorithm in Python. Code is available in the appendix.

Our scheme is inherently non-anonymous. We must know whose vote is whose so that uncast votes can be inferred from the non-anonymous delegation network. The votes do not need to be public, but they still have to be stored non-anonymously. (To assist voters in understanding why the loser lost and the winner won, the votes could be made public. This allows voters to recalculate the results.) Moreover, a member’s social network is revealed. This is a very private sort of social network – the network of influence between people. Such is additional limitation of our proposed scheme with respect to privacy.

Future work

Future research can address these privacy issues. For example, it would be possible to build a distributed system in which only people one-hop away from the voter know how she cast her vote, but in further hops only an aggregate and anonymized vote is seen. Another option is to develop some new cryptographic primitives which allow privacy on the one hand and verifiability on the other. We do not really need to have public votes, just computability/verifiability of the results.

Additionally, an analysis of possible possible misuses of the scheme or issues with it should be performed. Can it be used by a small number of members? Is it stable, or how much, in general, does the result change when one person changes her vote?

As we described, we see this is a general prediction problem of inferring of missing votes. It would be interesting to try other existing (novel and traditional) approaches to predict missing data and use possible other sources of relevant data, not just a trust network.

Someday, we would also like to convince Facebook that we are geniuses and that they should use their social network for the power of good voting. It may be possible to infer voters’ trust networks and link strengths based on their real social networks and the interactions performed via applications like Facebook. This would allow skipping the first stage of our scheme (in which voters explicitly delegate their votes) while not endangering the voting power of any non-voters.

Source code

Example implementation of peer-to-peer voting calculation is available on GitHub.

Marko A. Rodriguez, Daniel J. Steinbock, Jennifer H. Watkins, Carlos Gershenson, Johan Bollen, Victor Grey, Brad Degraf. Smartocracy: Social Networks for Collective Decision Making. 2007. ↩︎
Bryan Ford. Delegative Democracy. 2002. ↩︎
Hiroshi Yamakawa, Michiko Yoshida, Motohiro Tsuchiya. Toward delegated democracy: Vote by yourself, or trust your network. 2007. ↩︎
Stefan Dirnstorfer. A Voting System for Internet Based Democracy. 2010. ↩︎
Reid Andersen, Christian Borgs, Jennifer Chayes, Uriel Feige, Abraham Flaxman, Adam Kalai, Vahab Mirrokni, Moshe Tennenholtz. Trust-based recommendation systems: An axiomatic approach. 2007. ↩︎
Kenneth J. Arrow. A Difficulty in the Concept of Social Welfare. 1950. ↩︎

Towards a better group decision-making

Mon, 20 Jan 2014 01:14:10 -0700

One more old document explaining technical background of the voting system presented in the previous post. Historically it was made few years before the ideas of the previous post, so it does not describe all the properties needed for it. But it does explain how to achieve that just a smaller number of people have to vote and you can still have some understanding what is the opinion of the whole population of a group. This is a necessary property when you want to vote often and on many different topics. We cannot assume that all members of the group will have time and/or knowledge to decide on all those questions, but we believe that they at least know who in their social network is capable of making such a decision and at the same time share their values. So they can delegate voting on this particular questions to them.

Finding out what the whole group wants

Currently the most common way of a group decision-making is voting with simple vote counting, determining the result by the majority. Such a scheme is easy to understand and implement, but the question is if it does the most important thing in the best possible way. Namely, the main purpose of a group decision-making scheme is to give the result which would satisfy the whole group the most.

Theory sounds good, everybody in the group expresses her opinion and by majority you can determine the result which would satisfy the group the most. But in practice it is quite different, group members can have time, knowledge, information, motivational and/or other constraints which prevent them from expressing their opinion in an optimal way. This is even more important if we would like to use such scheme on a regular basis on a variety of subjects, for example, in a horizontally organized open source project, an association or even at a state level.

If voting is used regularly, not everybody can participate every time so it is important to be able to infer their votes, for them not to be at a disadvantage. If used on a broad spectrum of decision topics, not everybody has necessary knowledge and information to draw an opinion, but at the same time consequences of decision-making still affect them. So it is important that the scheme would allow such group members not to vote while still taking them into the account when determining the results, not simply discarding them out of the equation in the way simple vote counting is doing. And if used in a group where voting participation is generally low, good voting scheme should still determine the results which would satisfy the whole group the most.

One approach to the problem of determining the results is to see it as a prediction problem: we have a subset of votes and we would like to infer how would the whole population vote if it would be able to.

In this text I present a proposal for a group decision-making scheme aimed at solving the above-mentioned issues. It uses additional information of a social/trust network in the group to infer votes for members who have not cast them themselves.

This scheme is targeting electronic voting, which is the form of voting that can be done on a regular basis, where additional information can be collected, and complex algorithms can be employed to determine the results.

I will first present the proposed scheme, suitable for general decision-making in groups, and then show some ideas how to use it in different contexts.

The scheme

The scheme is a two-stage process. In the first stage every member of a group chooses zero or more other members who she trusts and would delegate her voting decision to in the case she does not cast a vote herself. In the case she chooses nobody, her vote is discarded if she does not cast a vote. In the case of more than one delegates, her one vote is inferred from delegates in chosen ratios, summing together to her one vote.

The first stage can be done only once but members could also change their delegates at will later on, probably with some limitations on how often in a given time span this can be done. The important thing is that this stage can be done less frequently than the second stage.

In fact, this can be generalized somehow in a way that a member chooses one or more other members as delegates for her one vote with chosen ratios with an addition of selecting whether her own vote casting has a precedence (in the case that she casts a vote) or not. Default if somebody does not do the first stage could be that a member has only a precedence delegation on herself. Although maybe the first stage should simply be compulsory for everybody.

Examples on how could a member $A$ delegate her voting decision in the first stage. We have three members, $A$, $B$ and $C$. $A$ can decide to delegate like this:

$$(A, *), (B, 0.4), (C, 0.6)$$

$A$ has a precedence vote for herself but in the case she does not cast a vote, her vote is inferred $0.4$ from $B$ and $0.6$ from $C$. If she casts a vote, only her vote counts.

$$(A, *)$$

This is a default which means her vote counts only if she casts a vote.

$$(A, 0.9), (B, 0.04), (C, 0.06)$$

This is without a precedence vote which means that if she casts a vote then $0.9$ of her vote is counted, but still $0.04$ and $0.06$ is inferred from $B$ and $C$, respectively. If she does not cast a vote, then her vote is inferred from $B$ and $C$ in $0.4$ and $0.6$ shares.

There could be also another possibility how to interpret these delegations: in the third example, in the case that $A$ would not cast a vote, only $0.04$ and $0.06$ of a vote would be inferred – no normalization to the whole vote would be done. But I see this as problematic as it would dilute the main difference between current voting schemes and the proposed scheme: it would make a vote (or part of it) to be lost if a member does not vote. This losing of votes is also the problem with default delegation mentioned above (when a member does not vote, her vote is lost), but such default is proposed to ease transition. Later on I propose some other default delegations.

Even more possibilities would make the scheme even more complicated. On the one hand this would allow people to have more freedom to choose the combination they like. But on the other hand this would also make scheme so complicated that it would be hard to understand all possibilities and (especially important) to reason about them and how different possibilities influence the whole scheme. So maybe it is better to chose only these few (which are already numerous) possibilities, but to choose them well and to reason about their implications well. I see chosen possibilities as a good compromise for initial research on the subject, but of course also other possibilities could be researched.

So now we have all this delegations. This defines a (social) network between members which is a kind of “web of trust” or “trust network”. We can see it as a directed graph between (hopefully) everybody. We call this a “delegation network”.

When a decision is needed, votes are cast. It is not required that everybody casts a vote. For those who do not cast a vote, their vote is inferred. But even more than that. This is done transitively. So in the last example above, if also $B$ (next to $A$) does not cast a vote, then $B$’s $0.4$ share is inferred from $B$’s own delegations. In the case nobody of $B$’s delegates (transitively) casts a vote, then $A$’s vote is wholly inferred just from $C$’s vote.

This last possibility is an unfortunate one as it means we have lost votes: of $B$ and all $B$’s delegates who have not voted, transitively. This is probably a very unlikely event (based on the six degrees of separation idea) and possible only if we allow members to not choose any delegates (we could change default delegation or make it compulsory to delegate in the first phase).

This is it. Inferred votes are calculated transitively and votes, real and inferred, are counted.

Discussion and variations

In this way everybody participates in the decision making even if they do not vote in a particular instance of vote casting.

The key insight is that many people currently cast votes based on opinions and decisions of their friends. So we can generalize and formalize this informal practice, make it a part of the scheme, making it not just a legitimate practice, but the integral part of the scheme.

There are different variations of the idea, based on the scale of its use. On a smaller scale, for example, on a scale of associations or other NGOs who might use such voting scheme to do decision making in their organization, the proposed scheme could be used directly. So also members who might not be able to attend a given organization’s meeting could still indirectly influence the results through members who are attending the meeting.

At a state level it would probably be better to define a number of delegation networks for every area of state’s conduct. Of course the question is then which areas should be defined and to which of these areas should a given issue be assigned and who would assign this. This could open a window to manipulation: you categorize an issue to an area where delegation network is better suited to your own agenda.

To solve this problem, we can again use the proposed scheme. First we have a voting round on to which areas the issue belongs and then in the next voting round we calculate the result based on the first round ratios between determined areas. We do not use an absolute winner of the first round but just weight accordingly to the second round’s results of different areas and their delegation networks.

There are also two kinds of this voting scheme whether we deal with a limited (known) population or unlimited population. An example of the first would be a state which has a known index of all eligible voters. An example of the second is an open community on the Internet where everybody can join (and thus gets a vote). Because of this there is no limit on number of votes so everybody can steer the results into her own direction if she gets enough of her friends to join.

This is why there exist two perspectives on a decision. A group perspective is used when population is limited and the decision represents the will of the whole population. An individual perspective is used when population is unlimited and the decision represents a decision important mostly just to one individual and is probably different for somebody else.

Maybe an example is due to present these two ideas better. For the first one making decisions on state affairs is a good example. And for the second one a collaborative moderation tool could be an example: users vote if a website comment is abusive or not. You do not really care what is the real “truth”, you just care that what you get moderated is something you would say that it is abusive and that what you do not get moderated is something you would not say it is abusive. You delegate this decision to friends you believe have a similar taste of what is abusive and what is not and have read a given comment before you. It does not matter if some abuser gets millions of her friends to vote, as you and your friends and their friends… do not infer their votes from those millions of abuser’s friends. And even if somebody would do so, it would mostly be just a rare exception.

The other such system with individual perspective, already deployed, is a PGP/GPG web of trust approach to users' certificates validation.

As I wrote before there are also different possibilities what to choose for a delegation default. What I have proposed above is based on the currently pervasive voting scheme, where if you do not cast a vote, your vote does not count in any way. But other possibilities are also possible. For example, that the default is that your parents have equal share of your vote in the case you do not vote, or person’s husband, or children, or some other social person/group you identify with.

Different number of votes (numerical maximum, not just 1.0) could be given to members. Based, for example, on values of a given group this voting scheme is applied in. My proposal is based on current values found in modern societies: everybody gets one equally valued vote. But next to this, the voting scheme allows also that members decide (for example in a company) that number of votes depends on the stock share, or in some (dystopian) society on IQ or some other metric.

Furthermore, in the case multiple options are given for a decision, we could, instead of one preferred choice, rank choices in preference and then use this as an input to the delegation network.

One problem of proposed voting scheme is that it is not easily understandable. Traditional voting is easy to reason about and people understand why somebody lost and why somebody won. Here the loser would be able to stir this complexity for her own agenda and disagree with the results (or argue about corruption). This could be mitigated partially by using open source technologies and public information on the systems employed to run the voting scheme.

The scheme is inherently non-anonymous. For every vote it is required that it is known from whom it is so that inferred votes can be calculated. The votes do not need to be public, but they still have to be stored non-anonymously. But to really mitigate the issue of understanding why somebody lost and somebody won also the votes should be made public so that people would be able to recalculate results by themselves. It would be an interesting research subject to find some solutions to this problem of privacy. For example, some way of distributed computing where only people one-hop away from the voter would know how she cast a vote, but in further hops only a combined vote would be seen. The other way is to develop some new cryptographic primitives which would allow privacy on the one hand and verifiability on the other. We do not really need to have public votes, just computability/verifiability of the results.

A members’ social networks are also revealed. Especially the very important type of a social network – the network of influence between people. This is an additional hit on their privacy by the proposed scheme.

The voting scheme is still a majority rule and allows tyranny of the majority without proper (legal) preventions.

Vote trading is even easier done as person needs just to delegate as instructed (and payed). But also such “influences” are then visible and recorded in the delegation network itself, so probably could be discovered if done systematically.

There are many things to research. Does limitations prevail over advantages? Would the proposed scheme be better than current schemes? What are possible other misuses of the scheme or issues with it? How to improve privacy? Is it stable (how much, in general, does the result change when one person changes her vote)? Can we employ even better algorithms to analyze such delegation networks, or even use some other concept altogether to solve the presented decision problem. All this are the questions which I would like to research, possibly with both modeling them mathematically and real-life experiments.

A followup with comparison with other similar ideas and the analysis is available in this post.

Massive online collaborative decision-making system

Mon, 20 Jan 2014 00:01:10 -0700

I am publishing a longer document describing one my idea of a massive online collaborative decision-making system. I worked on this idea last year, but for now I put it on hold focusing on other projects. Still, I think there is some value in sharing it.

Introduction

Doing group decisions can on the Internet be a much more common occurrence than in traditional contexts of, for example, government elections. Many Internet communities, web portals and services regularly use some sort of user feedback to make decisions, select or moderate the content, etc. But those systems tend not to leverage its online context but are just adapting traditional offline approaches.

In the offline context group decision-making is done rarely, commences with fixed and limited in number options to choose among (legacy of paper-based voting) and have an in-advance defined timespan (legacy of manually counting and organizational limitations). Additionally, inherent assumptions in current democracy processes are that everybody knows how and has time to make informed decisions.

In the online context all this is not necessary true anymore. Because decision-making feedback can be in real-time, it is much more reasonable to want to do changes, propose new options or variations, while the decision-making is in progress. Thus, options to choose among can be numerous, new options constantly being added and removed. Communities can be both tightly or loosely knit, users more or less engaged, informed, with more or less time to participate, especially in prolonged processes. It is hard to decide on the best timespan for the decision-making process, especially when users are spread over multiple timezones with various commitment levels to the community, and issues are of various importance to the community.

Current online systems concentrate on security/privacy (Helios, Civitas), experiment with more complicated voting schemes (CIVS), use delegation between users (LiquidFeedback), or try to engage users in deliberative interaction (Deliberative Polling). But none is a generalization and combination of those efforts which would really leverage the Internet medium.

We want to design a reasonable model of an online voting system and implement a proof-of-concept application – a massive online collaborative decision-making system where users can collaboratively author texts on controversial topics in a democratic and deliberative manner.

Motivation

We believe that group decision making should be done while in the stage of collaborative text authoring on which to make decision to support or oppose it. That if community is democratically involved in authoring of a text then the decision making for accepting the text is just a formality, community already agreeing on its content.

Wikipedia is currently the most widely used collaborative text authoring system. But while it is suitable for an encyclopedia and use cases where people more or less just want to collect and combine information, it fails miserably when used on a controversial topic. Its main tool to determine which version of the text is available to others is persistence of users – the user with more persistence to edit the text will have his or her version prevail over others’. The more persistent/stronger wins. This is mitigated by community-appointed authority figures who patrol Wikipedia and make sure such disputes are peacefully discussed and resolved. In the encyclopedia context this is maybe reasonable as arguably it can be determined which version is true and which is not. But in more controversial topics or topics where the truth is not really known and is a matter of public opinion, Wikipedia style of editing fails.

Furthermore, Wikipedia style breaks when there are large number of concurrent users wanting to contribute. Moreover, it does not support different ways of contribution: some might want to contribute the content, changes to the text, but others would just like to say whether current or proposed text is something they agree on or not, to direct its evolution.

User-facing interface

Design principles:

user friendly, easy to use, intuitive
real-time interaction and feedback
collaborative, users should be able to directly interact with other users, “see & feel the community”
minimization of “operation mode” switching, e.g., discussions and voting should not be separated from the content; if necessary, switching should be done automatically and in a transitive and intuitive manner
interaction and voting should be made pervasive, something users can easily do while browsing around the portal (similar to “liking” on Facebook)
no need to understand the technical background of the underlying voting system to be able to use the interface (but of course they can, the system is open source, algorithms used available, documented and published, and also some statistics and visualizations are available through the interface itself)

Based on this we are proposing a new kind of interface for collaborative text authoring. It consists of the current version of the text in question and proposed changes to be included into future versions. Users can vote on those proposed changes, discuss them, and once a change gets enough support it is applied.

On the left we can see the current version of the text. On the right we initially have a list of proposed changes ordered by how community currently votes on them. Those to the top are those which will sooner be applied. Uses can see avatars of who all is participating in the change authoring (each change itself can be authored by multiple users), current score for the change, what is the current vote for the user computed from his or her delegates in the case of delegation, and four buttons, upvote, abstain, delegate, downvote. Default can be abstain or delegate, based on site/user preferences. Hovering over each change shows in the panel on the left what the change adds and removes, with that text emphasized.

In current online group decision making we found out that too much time is spent on discussing without concrete results. The idea of our interface is to encourage concrete improvements to the topic at hand. Instead of people just discussing and discussing what should be done, what should be changed, they can just propose a change directly. So that once through decision process (both discussion and voting) some support is reached, there is an immediate effect on the common result. In this way a collaborative text is also automatically summary of all discussions and users who later on start participating can immediately see current status and continue from there, while still be able to traverse the history (under “History” option) and see why some text part is as it is.

To facilitate all this, users can easily start editing the text directly in the left panel. They can decide to do this in private, invite others to collaborate, or do it publicly for everyone to participate. Collaboration is in real-time, similar to Google Docs. Once a user or users are satisfied with the change, they can submit it to the vote. Those changes can be listed under “My changes”.

Clicking on a change listed in the right panel, discussion about the change is displayed below the change in the right panel. Users can vote on the comments in the discussion as well in the same manner as on changes themselves. Those votes are used to select relevant and community-supported comments.

User can open the list of all his or her votes (“My votes”) and refine the order of them. The list is split into two sections, upvotes above and downvotes below. With drag and drop user can select the preference among them. Initially, preference is based on his or her delegates. Optionally, user can also see a list of changes from which he or she abstained.

Backend voting system

The user-interface shows the needed requirements for the underlying voting system, but at the same time it is just one use case of our Internet-centered voting system. The voting system is general and can be used in many scenarios.

The voting system supports the following:

vote for or against an option, abstain from the option, or delegate the vote
ranking of options
adding and removing options at any time
support for transitive delegation for votes and ranking
real-time intermediate results
various strategies for determining when and how voting is concluded, including indefinitely open voting

The voting system allows not just voting for or against an option, but also to abstain from the vote or to delegate a vote to your network of delegates. The main idea of the voting system is to allow users to explicitly declare why they are not for or against the option. Is this because they object to the option as a whole, or is this because they know they do not know how to decide, but they still want some say in the decision. In our voting system, users can in the latter case delegate their vote to their delegation network.

Through this delegation the voting system can take into account not just the will of those who explicitly voted for or against the option, but also those who delegated the vote. They can delegate the vote to multiple other users, i.e., their friends or other people they trust, possibly share their values, but have more knowledge about the option at hand. Instead of simply discarding the users who do not vote directly, our voting system having such delegation network can, we believe, better achieve satisfaction of users with the results, and at worst be at par with current voting schemes. Voting can be now done more frequently and in a loosely knit or engaged communities where voter turn-up is low. In our use case of collaborative text authoring, the voting system allows that not all users read all proposed changes (which can be quite numerous) and that the system can be used even for determining which comments in discussions the community finds relevant, without users even noticing that there is voting happening for each of them.

A more detailed description of delegation in the voting system can be found in the following post with a followup with comparison with other similar ideas and the analysis.

To simplify, users can specify delegates in advance, so that the voting system can compute results in real-time and without a need for users intervention. Users can change their position on the option at any time, moving from explicit vote to delegation and back, and in the case of the delegation, see in real-time how their delegated vote is being used.

Such an online and real-time voting system allows novel ways on how decisions get into effect. Instead of having a predefined and fixed timespan of how long a voting lasts, voting can be done in a sliding-window manner. Voting is open until for, e.g., a week a majority has been uninterruptedly for or against. That is, while support fluctuates between for and against the voting is open, once it settles for one, for or against, and does not cross the 50% threshold for a week, the decision is made. Other similar schemes are possible as well. For example, another approach would be to use hysteresis and say that decision is made when it gets 60% of the support. But that it is in effect until support fails under 40%. If it fails under, decision is retracted. In this sense voting is open forever and users can later on change the outcome, retract the decision.

There are multiple ways to map our general voting system to common use cases. For example, a traditional voting “for” or “against” can be done through providing only one option “we accept/support”, without any ranking. Voting for public offices can be done by having users to vote for one or more options (candidates), optionally ranking them by preference. Or, they could just be offered to rank them.

Miscellaneous

In discussions users can reference other:

users with @username
changes with %id and %id.comment
Wikipedia articles with #wikipedia_article_name
other existing documents &id

Future work

Security/logging/history (verifiable and unforgeable results)
Voting model properties analysis
Privacy of voters and votes
Decentralization and/or federation

Acknowledgements

I would like to thank Jure Judež for great mockups of the user interface.

Crowdsourcing and ethics

Mon, 21 Oct 2013 22:01:10 -0700

Crowdsourcing definitely has some issues we are not talking about enough. They are workers as any other workers. They are precarious as many others today. Who is thinking about their rights, worker rights? Who is making sure that they are payed and are not working in sweatshops or even slavery conditions? We really do need a 21st century concept of worker unions, for globalized and precarious world. All crowdsourcing workers, unite!

Rising sea level will save us!

Fri, 18 Oct 2013 07:30:10 -0700

I do not understand why people have issues with sea level rising. I think we should just take the opportunity. Cheaply build tidal power plants on what is currently easy to build on land and then just wait for sea to come and power them up. Cheap energy. For whole world. No more wars.

Travelling across social divides

Tue, 15 Oct 2013 07:30:10 -0700

Few days ago I was taking a BART and saw a guy going into the BART without the ticket and the officer there complaining and not allowing him to go in. He was not lying or trying to make excuses. He said that he does not have money but needs a ride.

What I was thinking was that we have so many divides between classes. More money you have, more you can do. Less you have, less you can do. How can one then move towards more money, if you are limited? Limited in your mobility, in your Internet access, your education access … Maybe just because I have all those resources at my disposal, my idea will succeed but some others’ idea will not. Because I can travel, I can present the idea, I can network with other people. But then, maybe somebody else has a better idea, better concept, but he or she cannot even move from one part of the San Francisco to another. Maybe even cannot go to work.

You have people who have money and they can invest it. They can give this person with an idea money to be able to travel and able to go to Internet and be able to make his or her idea come true. But then most of the money the idea gets back goes to the person funding, makes him or her even richer. So it all depends on which side of the divide you start. Crossing it, that is hard. Getting more and more apart, this is easy.

Plato's Idealism by J. B. S. Haldane

Mon, 14 Oct 2013 07:31:10 -0700

And just as there is a best size for every animal, so the same is true for every human institution. In the Greek type of democracy all the citizens could listen to a series of orators and vote directly on questions of legislation. Hence their philosophers held that a small city was the largest possible democratic state. The English invention of representative government made a democratic nation possible, and the possibility was first realized in the United States, and later elsewhere. With the development of broadcasting it has once more become possible for every citizen to listen to the political views of representative orators, and the future may perhaps see the return of the national state to the Greek form of democracy. Even the referendum has been made possible only by the institution of daily newspapers.

An essay by J. B. S. Haldane from 1928.

Screencast previewing the PeerLibrary project

Tue, 08 Oct 2013 04:25:10 -0700

via peerlibrary

Thank you USA for making universal health care all around the world possible

Sat, 05 Oct 2013 22:41:10 -0700

Today, health care related products and services markets are global. So thank you USA for having so ridiculously high costs of health care because this allows all other countries to negotiate lower costs for their universal health care, while corporations still having enough income to research and develop new cures and equipment. Without you, USA, we would all have to pay higher prices, all around the world. Thank you for deciding to carry this burden yourself.

Seats on airplanes

Tue, 24 Sep 2013 04:26:10 -0700

Why, when choosing an airplane seat, you cannot say that you do not care where you seat, just that you seat next to somebody interesting (listing your interests)? Sitting next to somebody for 10 hours could be a great opportunity to exchange ideas, broaden horizons, and argue about the future of the planet.

Presentation of PeerLibrary at OKCon 2013

Tue, 24 Sep 2013 03:48:10 -0700

Presentation of PeerLibrary at OKCon 2013.

via peerlibrary

The document that officially put the World Wide Web into the public domain

Wed, 18 Sep 2013 07:30:10 -0700

The document that officially put the World Wide Web into the public domain on 30 April 1993.

Transparent touch-screen on the laptop

Tue, 17 Sep 2013 07:31:10 -0700

I am still waiting for the laptop which would have an on demand transparent (see through) screen. This would be useful while working. But even more interesting would be that you could close the laptop down and then use the other side of the screen as a touch-screen based device. No flipping of the screen around necessary.

Open source as a consequence of the lack of programmers?

Mon, 16 Sep 2013 09:43:10 -0700

Is open source possible because of the general lack of programmers so we do not even need an union to be able to require from our employers perks? Employers do not have much space to negotiate if they want to do their businesses. As workers we are then so lazy that we rather cooperate with others from other companies then (re)do all their work ourselves. In which other industry you can say to the boss that you do not feel to do the mundane task which was already done by somebody else and you would rather just work together with that colleague from the other company and let the boss imagine some other business model instead?

Internet has to go!

Sun, 15 Sep 2013 15:41:10 -0700

Internet has to go! Perfect explanation why and how!

Two thousand years ago, Plato wrote the collected dialogues

Wed, 11 Sep 2013 15:31:10 -0700

Two thousand years ago, Plato wrote the collected dialogues in which he presented the views of Socrates on the important issues of those times. Socrates, Plato tells us, argued that books would destroy thought. How could this be? After all, books, reading, and writing are considered to be the very essence of the educated, intellectual citizen. How could one of the foremost thinkers of civilization deny their importance?

Socrates is famous for his dialogues between teacher and student in which each questions and examines the thoughts of the other. Questioning and examination are the tools of reflection: Hear an idea, ponder it, question it, modify it, explore its limitations. When the idea is presented by a person, the audience can interrupt, ask questions, probe to get at the underlying assumptions. But the author doesn’t come along with a book, so how could the book be questioned if it couldn’t answer back? This is what bothered Socrates.

Socrates was concerned with reflective thought: the ability to think deeply about things, to question and examine every statement. He thought that reading was experiential, that it would not lead to reflection.

SOCRATES: Then anyone who leaves behind him a written manual, and likewise anyone who takes it over from him, on the supposition that such writing will provide something reliable and permanent, must be exceedingly simple-minded; he must really be ignorant of Ammon’s utterance, if he imagines that written words can do anything more than remind one who knows that which the writing is concerned with.

PHAEDRUS: Very true.

SOCRATES: You know, Phaedrus, that’s the strange thing about writing, which makes it truly analogous to painting. The painter’s products stand before us as if they were alive, but if you question them, they maintain a most majestic silence. It is the same with written words; they seem to talk to you as if they were intelligent, but if you ask them anything about what they say, from a desire to be instructed, they go on telling you just the same thing forever. And once a thing is put in writing, the composition, whatever it might be, drifts all over the place, getting into the hands not only of those who understand it, but equally of those who have no business with it; it doesn’t know how to address the right people, and not to address the wrong. And when it is ill-treated and unfairly abused it always needs its parent to come to its help, being unable to defend or help itself.

PHAEDRUS: Once again you are perfectly right.

Socrates was an intellectual, and to him thinking was reflection or nothing. He didn’t go for this experiential stuff. The worst kind of writing for people like Socrates would be novels, storytelling. A story engages the mind in an experiential mode, capturing the reader in the flow of events. All such experiential modes – music, drama, and novels – were considered to be the entertainment of the masses, not worthy of serious respect. Socrates worried that reading would be too passive, an acceptance of the thoughts of the writer without the chance to question them seriously.

In the Middle Ages, just the opposite was true. Reading was generally done aloud, often to an audience. It was an active process, so active that Susan Noakes, in her analysis of medieval reading, points out “that it had been recommended by physicians, since classical times, as a mild form of exercise, like walking.”

Moreover, Noakes observes that the characteristics of a good novel today were unheard of in earlier times: “Today, many readers take as the hallmark of the good novel the way it propels them to read it continuously, without putting it down, from beginning to end. Readers of many late medieval books would have been forced, on the other hand, to read dis- continuously, stopping to puzzle over the relationship between complement and text.” (The term complement refers to the dialogue provided through the illustrations and marginal comments – illuminations and glosses – sometimes put in by the author, sometimes by the copyist, sometimes by other readers.)

During the Middle Ages, readers were taught the rules of rhetoric and were implored to employ them with each sentence: mnemonics, to memorize and learn the material; allegory, to find the multiple levels of meaning hidden beneath the literal text; typology, to think in historical parallels. No text was thought to be complete without mental elaboration in the mind of the individual reader or debates within the social group that might be listening to the read-aloud text.

Readers in the latter part of the Middle Ages did with books exactly what Socrates had claimed was impossible: They questioned and debated each idea. True, the author wasn’t around, but in many ways that made the job more challenging, more interesting. Read a sentence, question it. Read a page, criticize it. No authors to object. No authors to refute your arguments with the force of their rhetoric. Readers were free to develop their own objections and opinions without interference from meddling authors. Today we may have regressed to match the fears of Socrates: We read too quickly, without questioning or debating the thoughts of the author. But the fault does not lie with the book, the fault lies with the reader.

Cognitive artifacts are tools, cognitive tools. But how they interact with the mind and what results they deliver depend upon how they are used. A book is a cognitive tool only for those who know how to read, but even then, what kind of tool it is depends upon how the reader employs it. A book cannot serve reflective thought unless the reader knows how to reason, to reflect upon the material.

From Things that Make Us Smart: Defending Human Attributes in the Age of the Machine by Donald A. Norman.

via peerlibrary

Presentation of PeerLibrary at Creative Commons Global Summit 2013

Thu, 29 Aug 2013 09:31:10 -0700

Presentation of PeerLibrary and open access tools panel at Creative Commons Global Summit 2013 in Buenos Aires. Slides.

via peerlibrary

UC Berkeley students lose their health care coverage if they participate in riots

Tue, 27 Aug 2013 13:11:10 -0700

From UC Berkeley health care plan:

This Plan does not cover nor provide benefits for:

Expense incurred as a result of injury due to participation in a riot. “Participation in a riot” means taking part in a riot in any way; including inciting the riot or conspiring to incite it. It does not include actions taken in self-defense; so long as they are not taken against persons who are trying to restore law and order.

How can participation in legal or legitimate demonstrations (which is an expression of free speech) which turn violent suppress your health insurance? One more nail in the apathy coffin.

The meeting and panel on open access

Tue, 20 Aug 2013 07:19:10 -0700

The meeting and panel on open access organized by Open Access Initiative at Berkeley on May 8 2013 at UC Berkeley, South Hall (School of Information), answering many questions about open access.

The panel includes Mike Eisen (co-founder of PLoS), Molly Von Houweling (Boalt Law School, former director of Creative Commons), and Laine Farley (Executive Director of California Digital Library). Moderated by me.

A great talk by Bret Victor about the future of programming

Sun, 04 Aug 2013 01:57:10 -0700

A great talk by Bret Victor about the future (or past?) of programming.

Effective democracy

Sat, 03 Aug 2013 02:13:10 -0700

What exactly is effective democracy? What exactly is effective participation in democracy? Some people complain that other people do not participate in democracy enough. Not enough people vote. Not enough people sign petitions. Not enough people go to riots. Why?

I believe the answer is simple: people do not see such their civic participation as effective. Effective means two things:

it is not (too) time consuming
it is has a (visible) impact

It is really easy. People do not have time to engage in democracy. That is why you appoint your representatives or delegates, so that you can work your double-shifts. While your representatives or delegates work for you. On the other hand, you want to feel that their actions do have some impact. That it matters how you vote. Does current democracy delivers that? No.

We should start working on an effective democracy. Democracy, where it is easy to see how your actions influence changes. What is the chain of influences between you, your vote or some other civic action and those changes, which representatives and which their actions. And to have this presented in an easy, intuitive and not time consuming manner. Actions should be easy to do, not time consuming, and with visible results. We should create 21st century versions of votes, petitions and riots. We could build technological tools to make actions less time consuming and to display nice visualizations of changes you influenced, but this is not really enough. We need changes in legislation which should make possible that those actions have real impact. So a solution might be:

technological tools to make 21 century civic engagement time efficient and results of your civic actions evident
rethink civic actions for 21 century
legislation which makes such 21st century civic engagement impactful, for real changes

But who wants changes?

PeerLibrary is participating at August 3rd Hacktivation

Fri, 02 Aug 2013 06:49:10 -0700

Hacktivations bring together developers, designers and storytellers to come together and hack for good on software projects that are creating social good through their work. PeerLibrary will be one among 8 projects to hack on this time. Come!

via peerlibrary

Is There Anything Good About Men?

Sat, 27 Jul 2013 07:31:10 -0700

I am not sure if I agree with everything in this essay by Roy F. Baumeister, but it contains some very interesting ideas on sources of differences between men and women in the society.

The Awesomest 7-Year Postdoc

Tue, 23 Jul 2013 17:20:10 -0700

The Awesomest 7-Year Postdoc or: How I Learned to Stop Worrying and Love the Tenure-Track Faculty Life. How to have fun in academia. Or in life in general. I believe all those ideas and suggestions apply to the life in general as well. To the meaningful and fun life. Be the best “whole” person you can.

Kids are the future

Fri, 19 Jul 2013 01:35:10 -0700

Beauty of the nature, in its many forms of beauty

Sat, 13 Jul 2013 13:30:10 -0700

Beauty of the nature, in its many forms of beauty. Not just an aesthetic one. There is always something beautiful in everything and everybody.

Jane Elliott's "brown eyes, blue eyes" experiment

Thu, 11 Jul 2013 07:31:10 -0700

Jane Elliott’s “brown eyes, blue eyes” experiment in 1970 showing racism through a game of racism based on eyes color.

PeerLibrary won the Highest Impact award at Meteor Hackathon 2013

Wed, 10 Jul 2013 13:57:10 -0700

PeerLibrary, a project I am currently working on, won the award for Highest Impact at Meteor Hackathon 2013. This is how they describe the project:

Peer Library is like Rap Genius for academia that adds crowdsourced notetaking to academic papers. Users import academic papers in PDF format, view papers in-browser from the library, and make inline comments on selected text in each paper. Users can share those comments and view other people’s notes in real time. They also implemented searching for papers by title or journal. Team: Mitar, Gheric Speiginer, Tony Chen.

What's Wrong with Technological Fixes?

Wed, 10 Jul 2013 07:31:10 -0700

It is always very important to ask yourself about the meaning of some solution. The reason for it. Engineers too often like to solve some problem because they can. But is the problem really needed to be solved? Should there really be an app for it?

Morozov characterizes this impulse to fix everything as “solutionism,” and offers two broad challenges to the solutionist sensibility. First, solutionists often turn public problems into more bite-sized private ones. Instead of addressing obesity by regulating the content of food, for example, they offer apps that will ’nudge’ people into better personal choices. Second, solutionists overlook the positive value in the ‘vices’ they seek to ‘cure.’

I like the thought that sometimes inefficiencies in the system have a purpose. Probably we should really be careful which inefficiencies we tackle and which we allow. Today we have means to remove many. But this does not mean that we should remove them all.

Gov.uk: how geeks opened up government

Tue, 09 Jul 2013 07:31:10 -0700

A nice example how to approach open data and open government and how to build tools for public.

Parrots the Universe and Everything

Mon, 08 Jul 2013 06:30:10 -0700

Parrots the Universe and Everything is a beautiful set of stories by Douglas Adams which show an interesting perspective of relation between human animal and nature.

The Danger in Demonizing Male Sexuality

Sun, 07 Jul 2013 06:30:10 -0700

Alyssa Royse explains how our current predator/prey model of sexual relationships is harmful to both men and women.

Crowdfunding or funding the crowds: a new model for the distribution of wealth?

Mon, 24 Jun 2013 06:30:10 -0700

An interesting critique of crowdfunding.

If one actually paid a decent wage to all of the people involved in helping with the campaign, one would be loosing much of the money raised. The production costs of organizing a €10000 campaign will have already used up some of the funds for the project that is to be executed when funded. The actual costs of organizing, raising money and carrying out the campaign are therefore not funded. It is the people who actually do most of the work (the campaigner) who are not paid enough (if anything at all) and then only if the project is successful, i.e. if the amount of funding requested has been raised.

Crowdsourcing the language bubble breakout

Sun, 23 Jun 2013 21:50:10 -0700

When searching in English you are reaching only English content on the web. But we could crowdsource search in other languages to native speakers. Where is crowdsearching when you need it?

Much have already been said about a filter bubble. Search engines like Google and other information feeds like Facebook personalize results to you so that you find easier information which is relevant to you. This is both good and bad. Good is because often it works very well and you faster find information you are searching for. Bad is, arguably, because it is locking you in the bubble of information you already know or is close to what you already know. Your opinions and worldviews are thus reinforced as more and more information you are exposed to aligns with existing opinions and worldviews.

Personally, I don’t worry so much about this. I know how to increase search results per page on Google so that I get 100 results displayed and can very fast skim over all of them without the need to click through pages. I get information from many different sources and especially by following links more than just one level deep. So even if you get initially personalized link, further links are not personalized anymore, or at least not personalized in the same way. Have you ever opened one Wikipedia page to end up having dozens of tabs open? Lastly, many service providers are well aware of this and there are many ways they can help – like adding some random or not personalized results to your results set, or even known opposites for you.

At the end it all sums to your personal approach of dealing with conflicting information (and ensuing cognitive dissonance). Do you embrace it, try to understand it and include it into your understanding of the world, or do you just ignore it, or even ignore the signs of it? It was very similar in the past. If you felt that there might be something more to the information at hand, you could just ignore this feeling or you could go to libraries and try to find books which would tell you more, confirming or opposing the information you already have. But it was still your decision how deep you wanted to search.

I am much more worried about the language bubble. We search for information only with keywords in languages we know and we get results mostly in those languages as well. Even if we search in English, we are still reaching only a bit more than 50% of the content on the Internet. So half of the content is out of the reach. Half of human ideas, concepts, solutions, stories, knowledge.

This is not something new. For centuries knowledge was passing from country to country, from civilization to civilization by translations done by scholars. But that took ages. Literally.

What I am missing now is a web portal where people could ask other people to help them find information in other languages. Do you want to find what are current best open government practices in South America? What open source projects are people in China developing? What open source tools are farmers in Africa using to collaborate, if any? It is not really enough just to translate those keywords. Even when searching in English it is often not really enough just to type some keywords to search on, but you have to see what results you are getting, skim them, iterate few times, repeat, and then you maybe find an useful link. When searching in other languages it takes too much time even with automatic tools helping you translate results. If you know that one page is relevant, than automatic translation can give you some help to understand the page. But automatic translation is still too imprecise to navigate around fast while searching.

Furthermore, by publicly asking native speakers for help when searching, multiple similar search queries could be aggregated together. So most searched for queries could be done first. Even more, if there is no useful content already available, native speakers could be even willing to prepare such content. For example, in Slovenia we have since the beginning of 2013 net neutrality required by law. There is no official translation (yet), so when we noticed that people are searching for the law and its translation to help them advocate for a similar law in their countries, we made an unofficial translation.

Such crowdsearching would help you find relevant similar information you might not know you are searching for. When you search in a language you understand you often find additional resources of interest. But this works mostly by having some background knowledge and understanding by which you can determine fast that some link could contain useful information. Native speakers often have a better understanding of such useful cultural backgrounds and common knowledge which can help improve quality of results – even those you have not asked for.

My presentation of nodewatcher

Fri, 21 Jun 2013 20:24:10 -0700

My presentation of nodewatcher, an open source planning, deployment, monitoring and maintenance platform for community wireless networks such as wlan slovenija. At International Summit for Community Wireless Networks 2010 in Vienna.

Slovenia in 21st century

Wed, 19 Jun 2013 02:20:10 -0700

When I look at Slovenia I see a country without a vision. Without imagination what would make the country something special. What would put it on the map. I see many great individuals with great ideas doing great things. But as a country? A small country lost somewhere in the middle of the Europe, without its own path, just following and responding to internal and external pressures. Not really in control of its own future.

Maybe it is because our politicians are corrupt, but maybe it is also because we don’t really want to change things. We don’t aim for some bright future, but would just like that it is “good enough”. We just want to survive. We don’t see the country as something which is ours and over which we have some control and which we can improve and improve until it shines.

So how do you want to see Slovenia in 21st century? For what it should be known? What should be those ways which would bring prosperity to it? We should do new things for that, as a country, we cannot just replicate others.

My main idea is that laws are those which define the country. If the country has all laws copied from other countries, what advantage it can have over them? It can be bigger, it can have more natural resources, it can have colonies or in general some kind of global power, it can have better education and academia it worked on for centuries, it can be more connected, more cooperative, more streamlined. But what about young small country without natural resources and with traditional internal conflicts like Slovenia?

It can have laws which make it special in some way. Which can make for its citizens easier to innovate, build, and provide some products and services to others. Some countries provide lower taxes for companies, less protection for workers, and in general less government services and regulation. But what if we want to keep social and caring nature of Slovenia? If we want to keep universal healthcare, free education for everybody, and workers’ rights? We cannot really lower taxes then.

Some countries provide laws kind to financial sector, tax havens, banking friendly laws. But market here is saturated. One more tax haven? No thank you.

But what we could do is recognize that we are in 21st century. We could embrace Internet and all its sharing and P2P culture, its crowdsourced and crowdfunded creativity, and its data and networks centric lifestyle. We could innovate in laws concerning intellectual rights, data protection, privacy and anonymity. We could innovate in ways laws govern how things can be funded and how things can be build in a collaborative fashion. The country could stand behind people publishing creativity, information, and knowledge, in the same way countries traditionally stand behind their banks.

For example, Slovenia could pass the law which would give foreigners sovereignty over their data which is stored in datacenters in Slovenia. Like embassies in a given country have sovereignty. Why not also data? So than Google could have datacenters in Slovenia, but they would still be susceptible only to USA legislation (because it is founded there) and Slovenia would not have any right to seize or access that data. This would allow big Internet companies to have their servers in the center of Europe, close to their European customers, but could still be assured that nobody would interfere with their operations.

On the other hand, Slovenia could change laws to provide asylum to data. Foreigners could then store data in datacenters in Slovenia and request its protection from their own government. Or the protection could simply be automatic.

Slovenia with its central geoposition in Europe could become a country known by its safe datacenters, protecting both commercial and humanitarian interests. It would be clean, ecological and something to be proud of. Something to build upon other innovative law changes and resulting innovative products and services.

Oh, yes, with our central location in Europe we could become also the transport hub of Europe. I even believe that this is our current strategy and we are building highways for that. But is this really the best we can do? Have high prices for road tax vignettes for all trucks and tourists going to Croatia (so that they really remember their short but expensive moment in Slovenia)? Have polluted air? To be known with our only two lanes highways as a constipation of Europe? Isn’t it better to become the data hub of Europe?

If corporations have rights of Fourteenth Amendment, then they have rights of Thirteenth Amendment as well

Tue, 04 Jun 2013 19:57:10 -0700

If corporations are people and have their rights, then they should not be enslaved, owned and traded as well, no?

Copyright and patents in the time of crowdfunding

Sun, 26 May 2013 02:25:10 -0700

Many things were already said about how Internet makes distribution faster and cheaper and how this is putting under the question rationale for copyright and patents. But Internet through facilitation of crowdfunding is changing also the money flow around the production itself, changing the nature of the investment which arguably has to have a protection of exclusivity to return on the investment. Crowdfunding is changing this because it allows creators to skip the investment-return cycle and directly fund the creation from its users. As such, copyright and patents are obsolete.

In practice, the reason why copyright and patents¹ exist is that often production and distribution costs of some creative work are so high that a large upfront investment is required. This investment is often too large for creators themselves, so they partner with some sort of a publisher organization. This organization wants (higher) assurance that its investment will be returned and an established way is through copyright and patents which creators pass over to the organization for the organization to exclusively monetize the creation. The organization gets for this service an award, share of the profits from the monetization. To be truthful, those organizations do often provide additional services, like promotion and many other needed services which creators might not be too familiar with themselves.

Proponents of copyright and patents argue that those rights do exist primarily to incentivize creators to create, but in practice we see that this is not really true. We can see people everywhere who create because they love to create, because they want to tell or give something to other people. But yes, creators do have to life from something.

It is important to remember what are downsides of copyright and patents. All creations are based on past creations by other people. More creations are available for everybody to build upon, more new creations there can be. This is why copyright and patents had to historically find the right balance between being long enough for investments to return and profits to grow, but as short as possible to allow other people to build upon as soon as possible. Sadly, power to exclusively monetize has proven too attractive for organizations to give up, so in 20th century we have seen extensions of this time period which now extend even after the death of original creators. This should not be too surprising though, as we have just to remember again that creators pass over the rights to organizations, which then collect the profits.

While Internet is making distribution of creations cheaper and it is turning business models around, costs of the production can still be too high for creators themselves. But crowdfunding is turning even this around. For example, on Kickstarter, currently the most popular crowdfunding website, creators propose their projects and required budgets to realize them. Those budgets can include everything, from material costs to awards creators want for themselves. People (“the crowd”) can then decide whether they like the project and back (fund) the project if it reaches the given budget before the deadline. For the (co)funding of the project, backers are often promised something in exchange, some result of the project. Once the project reaches the budget and the funding deadline, creators get the funding, can create the project and backers get the results.

Such model of funding of production effectively puts copyright and patents under the question. Creators in advance decide what is the incentive they require to create the project, together with all other costs. Once the project’s budget has been meet they can realize the project and results can enter into the public domain, available for others to build upon and propose other projects. In this way we can all achieve better and faster innovation while still allowing expensive productions to occur.

Trademarks, moral rights, and some other intellectual property rights have arguably still some reason to exist. Especially as a defensive measures. ↩︎

In Praise of Idleness by Bertrand Russell

Fri, 17 May 2013 07:01:10 -0700

Suppose that, at a given moment, a certain number of people are engaged in the manufacture of pins. They make as many pins as the world needs, working (say) eight hours a day. Someone makes an invention by which the same number of men can make twice as many pins: pins are already so cheap that hardly any more will be bought at a lower price. In a sensible world, everybody concerned in the manufacturing of pins would take to working four hours instead of eight, and everything else would go on as before. But in the actual world this would be thought demoralizing. The men still work eight hours, there are too many pins, some employers go bankrupt, and half the men previously concerned in making pins are thrown out of work. There is, in the end, just as much leisure as on the other plan, but half the men are totally idle while half are still overworked. In this way, it is insured that the unavoidable leisure shall cause misery all round instead of being a universal source of happiness. Can anything more insane be imagined?

An essay by Bertrand Russell from 1932.

Read-write science

Tue, 14 May 2013 07:02:10 -0700

There is a notion of a read-write culture, a culture where we are not just consumers of the culture, but also producers. Prosumers. But we are missing a read-write science culture!

I see move from read-only to read-write culture as a democratization of the culture. Creation of the culture became much more inclusive, participatory and collaborative. There are less hierarchical structures, less authorities. In the past, all this was needed because cost of communication was high, cost of production was high. You needed people to aggregate, digest, and select to keep costs down and communication efficient. But this changed with the Internet and now we see changes in all spheres of human culture.

In all spheres? Really? Science, scientific research and scientific communication, is resisting any changes quite well. Research is still primarily done in closed and well-defined groups. Communication still happens through publishing papers in journals, a model useful for past centuries where it was a good way to communicate with many other researchers at the same time. But now we know better.

In science, we are still missing collaboration and inclusive approach to the research becoming something of a standard. We have beautiful examples of opening research. For example, BOINC and Foldit to general public, and PolyMath to fellow mathematicians. Citizen science is slowly taking off, despite most of the papers still being closed behind paywalls.

We are missing a generalization of these individual cases and while involving general public is beautiful, an easy and standard way for researches themselves to collaborate is still missing. If we are capable of making an encyclopedia together, why not science as well? Do we really believe that from more than million papers which are published each year none of those authors could rather work together on one paper, globally, collaboratively?

What is a paper? It is a log by a researcher (or researchers) of what has been done, what were results and what are (possible) conclusions. It is a testimony that this is true, signed by the authors (sometimes I have a feeling that in pursuit of being listed as an author, researchers forget that by having their name there it means they are also standing behind the paper). Why we then not move this log keeping to the Internet?

Instead of publishing the log once a year in the form of a paper, we could have it online publicly on a common web platform and update it regularly. I get an idea to research, I open a new log, describe the idea. I start thinking how to define a hypothesis. How to design an experiment. How do implement the idea. I write all this down as it goes. I try it out, I get some data, I upload data. I then analyze this data, I get some conclusions. I publish those conclusions. Maybe during all this I am distracted by some other idea. Maybe I never finish. Maybe somebody else joins in. Maybe somebody else decides to do the same experiment on the other side of the world, adding to my data and improving the conclusion? Maybe somebody can discover an error in reasoning even before I get to the stage of doing an experiment? Maybe somebody else things some other experiment would be better, he or she forks the log, does it, gets the data. We can then compare, merge, get a combined better understanding. The web platform could find automatically cases of possible collaboration. Or contradiction: this two hypotheses are stating the opposite. Scientific community can be involved from the very beginning. Helping with ideas, discussing and giving feedback. Other researches could vote which ideas look more reasonable to research first. They could even define dependencies: I need this research to finish for me to be able to finish mine. They could pitch in. Or they could just propose ideas for research they themselves do not have time to work on, but maybe some young researcher searching for an idea could adopt it? The community itself being a mentor to him or her, following the execution, teaching the proper research in a hands-on manner? Somebody could join in and make a summary suitable for public in innovative ways.

Then we might even remove the distinction between general public and researchers, citizen science and science. Everything would be science, everybody could contribute, no matter his or her position in the society. Each contribution, an idea, comment, data sample, or a beautiful drawing of a concept or the result, being evaluated just by its (scientific) merit.

My presentation of the PeerLibrary at Meteor Devshop 3

Wed, 08 May 2013 11:13:10 -0700

My presentation of the PeerLibrary at Meteor Devshop 3. PeerLibrary aims to enrich the experience of open access scholarly literature by combining real-time discussions with scientific publications. GitHub.

Governments and wars

Fri, 19 Apr 2013 17:59:10 -0700

I believe that any government which is involved in the war (or any armed conflict, or in fact any conflict where human rights are violated, or, maybe, even any act where human rights are violated) has failed its people and all other people from all around the world. The government is doing something wrong and should be helped to find a way to stop the war peacefully or be replaced with one which can do that. How can people make this change happen is often not obvious, but definitely it must not be done violently, because this would defeat the main idea behind.

The change should come from people governed and not from the outside. The people themselves have the means (and arguably the right) over their own government. People themselves should oppose their government’s decision not because they have been attacked but because they are attacking. They should come into the defense of the people their government has decided to attack. They should cooperate with people from the other side, the attacked. The other side should be doing the same with their government. Because sooner or later there is no difference between attackers and defenders, they are both attackers and they both should want their governments to stop attacking the other.

You can say: “They attacking us, we have to defend!” Yes, maybe. I am saying that the people of the country who is attacking you should pressure their government. You can help those people by informing them, explaining to them, discussing with them, working with them, giving them reasons why they should help you, you can provide them with ideas how. World is now global, you can meet outside involved countries and discuss in-person and find a common solution, people to people. World is now virtual, you can use Internet to communicate. You should at the same time pressure your own government, because it might be that from the viewpoint of others, you are the attacking country, even if you believe that you are just defending. You should do this with even larger vigor as you would want the people of the other country to do for their government.

Many times governments involved in wars are democratically elected and even have an ongoing support by people. If you want to create a change, you have to influence this support. You have to expose the reasons for the attack, inform and educate the people, show the mistruths, false representations. I cannot believe that people from any side would want war, but I can understand that they individually do not have a feeling that they can change anything. I can understand that they believe that their government is doing the best for them in a given situation. That there is no other way. But this is not true. They are not alone, they can work with other people. Moreover, the image of the situation can be skewed, it might just look like it is the only way to deal with the situation. It is really simple, if we have not found a solution how to not go into a war, we have not tried enough.

This is why we should become better investigators, better at exposing, better at telling what’s wrong and why. Better at being creative and imagine creative solutions. Better at discussing and better at listening. Especially to the people we do not agree with or we just believe we do not agree with. We should not wait for somebody to lead us, for the government to find a way to the peace, because it might be we will never see it.

I am not saying that people should overthrow the government involved in a war. The best is if they manage to change things from inside the system, possibly in an innovative way which plays the system against itself. Create a new party? Launch a new radio station? Organize demonstrations? Prepare and disseminate publications? Deploy Internet discussion forums involving both sides. Organize mutual cultural events with artists from all sides? Maybe current government is simply not capable of finding a better solution, presenting such a solution and helping the government could already be enough.

People have advantages. They are many. They have bigger imagination. They are diverse. They can create ideas from all possible walks of life. They can imagine completely new ways to change things and life peacefully with a other countries, especially with other people. But they have to first want to make a change.

It is important to remember that governments might declare wars and invoke conflicts, but people wage them. If governments are unable to stop wars and prevent conflicts, people can do that. In the past, there were not many ways for people from different countries to directly work together and decide on a peace. Their only possibility of communication was through their respective governments. But today this is not true anymore. People on both sides can decide for a mutual peace no matter what their governments decide. They can communicate and coordinate. They can make a change. They can learn about each other, understand each other, help each other.

Learn from the Balkans

Thu, 18 Apr 2013 14:56:10 -0700

Yesterday I was following the ASUC Senate (student body) discussion whether UC Berkeley should divest $14 million from companies affiliated with Israel’s military in opposition to human rights abuses happening in the Israeli-Palestinian conflict. It was an interesting experience to listen and I believe UC Berkeley has really some great future politicians. But it still made me sad because we all lost once again. We would lost no matter how the decision would be made. I will explain you why.

I do not know much about situation there, I have never been there, I am not really following or reading much about it, it is not really my tea of cup. But I know one other story. The story of the Balkans, or more precisely, of Yugoslavia. I know the story of brothers and sisters living for half a century in a common country, working together, growing together, caring together. Not everything was perfect, far from it, and we will probably never be able to describe relationships between all the people there. A kind of love-hate relationships where you wish for your neighbor to break his leg, but if somebody dare to touch him, he will have to speak with you. Because it is your neighbor. Relationships where you beat each other to a pulp over which football team is better during the day but then drink together in the evening until you cannot stand on your feet anymore and the person you fought before drags you to your home. Or invites you over to sleep on his sofa.

But all this changed in 1991 with split of the federation. Federal states now became countries. Or wanted to become countries. It is interesting how fast brothers and sisters became arch-enemies willing to kill, rape, torture, humiliate each other over the question of borders. This hill, this river, this village, it is ours! We have the right to it! You? You should just disappear from the world. We do not care if this is your house, your farming land, if many generations of your family lived here. Go! Disappear! Be cleansed or erased!

What changed? How so that people who just a moment ago lived next to each other now have problems with land? Did land just disappear? Did many new people move to live there and they were just struggling to find a place under the sun for themselves? No. Physically, everything stayed the same. Same people, same land, same hills, same rivers, same villages.

In 1980 Tito died. A benevolent dictator managed to establish brotherhood and sisterhood both internally and externally. But after his death, power struggles started. Yugoslavia became a hunting ground open for everybody. Wannabe politicians, religious leaders, private profiteers, and external interests. Everybody wanted a piece of cake and the best way to achieve this is divide and conquer. Exactly this happened and small low ranking politicians on a federal level now became top ranking politicians of new countries. Dream come true. Not just that they moved to the top of the food chain, there was less competing politicians now, especially when you take the opportunity and remove some. Ah, and of course, arms dealing to all sides.

How is this possible? How could those politicians achieve this? People are for sure not so stupid! They lived together, they knew each other, they speak more or less the same language, they were able to understand each other. But it seems it is possible. It is quite easy in fact. You appeal to ethnicity, to nationalism, to religion. Or better, to differences in those. (Catholics, Orthodox Christians, and Muslims, all lived in Yugoslavia.) You tell stories to people and assure them that they are the only one who have the right to that hill over there. Because this is where the past hero from long forgotten times fought his war. Or that river over there, because “our” people were first to cross it (forgetting, conveniently, that of course there were some other people living there before “your” people). People start believing this. Despite the fact that they lived next to the person of other religion for many years, they now discover that it is simply impossible to live like that, that differences are too big, that the other one is barbaric, non-human. Despite the hill being a nice hiking spot for many people of various backgrounds, meeting happily on the top, now it becomes to small for all and only one has to rule it.

As a consequence, we had wars. Bloody wars. Sad wars. Wars full of tears and pain. Suffering and hate. Yugoslavia is no more and all those private internal and external interests got what they wanted. They won. People, all people, lost. Lost brotherhood and sisterhood, lost lives, lost property. For what? For that hill over there which they were already free to climb?

This is why I became sad, seeing all this people discussing for hours and hours into the night with so much energy and heart and hope. Because all this energy was directed into widening the gap between people. It will not move a bit governments of both Israeli and Palestinian people, it will not move a bit their religious leaders. We will have a feeling that we did the right thing, that we are fighting the human rights violations and now we will be able to sleep better. Yes? How so? We did not do anything expect increased the pain of people. Yes, divestment is a powerful message, powerful statement. But to whom? To the governments and religious leaders? Or to the people? Governments and religious leaders will now just have one story more to tell to their people. See, how we are attacked from all sides, we have to attack back ever more! See, how we are supported internationally, we have to attack even more, because now we will for sure win!

Win? Who will win? We will all just lose. People will lose. Leaders will win. There is no conflict and no war without leaders keeping them going. People do not want conflicts. People do not want to be killed. But they will stay in conflicts and they will keep killing and being killed because nobody is helping them to get out that and it seems they are unable to get out of it themselves? Or are they? Maybe they are unable to solve the issue themselves because external interests are too big? (And yes, divestment could be seen as a step to decrease those external interests.)

Win? Some students now have a feeling that they won. Some students have the feeling that they lost. That they were invaded, violated. This is sad. Because this is not a question of victory but of not yet getting the right (part of the) solution to the issue we want to address. I dream of a consensus, of both sides agreeing wholeheartedly on a solution. Then we would be able to say that we made a change. That we spun the world into the other direction. This would not just be a solution to the issue but would also be an example of cooperation between all. It would be a double win!

We could. All this brains, all this energy, we could instead for 10 hours discuss a strategy how to enable Israeli and Palestinian people to empower themselves so that they could create a change for themselves. We are lucky here, we do not have bombs falling around and we can think in peace. If we cannot get Israeli and Palestinian people here, removed from the harsh reality of the conflict, to work together towards a common peace, how can we believe we can change anything there, there where it really matters? But now just imagine that all the Israeli and Palestinian people here would come to agree on a common plan how to live, love and build together a place under the sun? How to recognize, analyze and counteract all internal and external influences keeping this conflict going? How to not seek separation but cooperation. They inhibit the same land, they are the same people. They are not? They are not the same people? They are. There are in fact no Palestinians and no Israelis. They are all just humans.

We need an out-of-the-box solutions to the conflict. Where if not here we could create such solutions? But first we have to recognize what or who we are fighting. Leaders who are keeping this conflict going. Yes, we can fight profiteers as well, but this is just a side story. If we really want to make a change we should start thinking differently. First, let us get all Israeli and Palestinian people on the campus to work together. Why they could not sign a peace act by themselves? Create it in the way they believe it should be done by their governments. Put in everything they believe it has to contain. Then seek to get more and more Israeli and Palestinian people from all around the world do join it, sign it. Israeli and Palestinian people should take into their own hands the peace their governments seem not be able or willing to create. All this can be started here and now. Would not be that much better message and statement to make?

In the Balkans people are again learning how to live together. Wondering, what frenzy got over them in ’90s – how was that possible? It is possible. But it is possible to get out of it as well.

Questioning democracy

Tue, 16 Apr 2013 16:13:10 -0700

Why it is not allowed to question democracy? Why this became a taboo? An idea you do not question? A human artifact you do not want to improve? We wage wars in the name of the democracy. It is a stop word for any discussion: “This is not democratic.” Yes? Maybe? It is good to recognize this, but not to stop discussion because of that.

How many of you can count how many times democracy went wrong? How many of you can count how many times a dictatorship turned good? Do you even know about benevolent dictators from our past? Do you believe that there have been none? Or you just have not learned about them? I agree that we have come far in our search for a good system to govern ourselves, but we must not stop our search here. We should be able to look in all directions for clues to guide us to the system which would really have people at its core. Not just in the name.

(I will not even go into an issue that there many different forms of democracy and many different forms of dictatorship and everything in between.)

Game of Thrones recreated in Minecraft

Mon, 01 Apr 2013 00:36:10 -0700

If this is not a prime example of how Internet can help people collaborate at scales previously not imaginable. So, if we can create virtual words together, can we also govern our own physical world together?

Spletni portal, ki bo demonstracije še približal ljudem

Fri, 29 Mar 2013 00:46:10 -0700

Article about the Exposer project in Dnevnik, Slovenian daily newspaper. It is described as a web portal which brings demonstrations closer to the people, as a way for those who are otherwise not using social networks to see what is happening there.

Vse demonstracije na enem mestu

Fri, 29 Mar 2013 00:32:10 -0700

Presentation of the Exposer project in Mladina, a weekly Slovenian newspaper, describing it as a place to learn everything about demonstrations in Slovenia.

Exposer

Fri, 29 Mar 2013 00:25:10 -0700

Exposer collects all social posts from Facebook and Twitter containing keywords and displays them aggregated and in real-time. Useful to follow everything around those keywords. See live instance of all posts about demonstrations in Slovenia. And another collecting tweet testimonials of sexism.

Ekskluziven intervju z organizatorjem protestov

Fri, 29 Mar 2013 00:19:10 -0700

Interview with me about ongoing demonstrations against corrupt government officials in Slovenia in most-read and quite a bit yellowish Slovenian daily newspaper.

PiplMesh

Thu, 28 Mar 2013 23:44:10 -0700

PiplMesh is a social info portal for wireless networks combining social interactions between users, wireless, mobile and locality-sensitive nature of their connectivity (the reach of what they share is limited by the reach of the wireless signal), and relevant GIS-based information.

nodewatcher

Thu, 28 Mar 2013 23:39:10 -0700

nodewatcher is a platform for organic (bottom-up) growing of open networks, a network planning, deployment, monitoring, and maintenance platform.

wlan slovenija

Thu, 28 Mar 2013 23:37:10 -0700

wlan slovenija is an open wireless network initiative, building an open, common and free network in Slovenia by encouraging people to share their existing Internet connectivity with others, providing participants guidance, knowledge and technology. For this effect, we have developed and used many open source technologies (GitHub).

(Formerly wlan ljubljana.)

django-pushserver

Thu, 28 Mar 2013 23:32:10 -0700

This Django application provides a push server for Django based on Leo Ponomarev’s Basic HTTP Push Relay Protocol. Useful especially while locally developing Django applications using Nginx HTTP push module.

django-tastypie-mongoengine

Thu, 28 Mar 2013 23:22:10 -0700

This Django application provides MongoEngine support for django-tastypie.

django-mongo-auth

Thu, 28 Mar 2013 23:17:10 -0700

Django authentication based on an extensible MongoEngine user class.

Along with expected support for common external authentication providers and a traditional on-site registration workflow with e-mail address confirmation, anonymous users are given a temporary account instance which can then be converted to an authenticated one.

I believe this is the best way for community oriented websites where you want to allow even anonymous users to edit the content or in some other way interact with the website. Instead of displaying their IPs (like Wikipedia does) or having all anonymous users have the same name displayed, a new account with random username is generated for each anonymous user and all content he or she creates is linked to it. Anonymous user can in this way be completely equal to registered users and programming logic can be thus simplified, without a special case for anonymous users. The only difference is that when user session terminates, this anonymous account gets orphaned. If the user wants to keep control over it, he or she can convert it to a registered account. In fact, the only difference between anonymous and registered accounts is that registered accounts have some authentication mechanism linked to them.

django-hashlink

Thu, 28 Mar 2013 22:19:10 -0700

Django app to store and retrieve URL hash (fragment) based webpage state in MongoDB through Django.

It is an experimental idea to store state of GUI of modern dynamic web applications on the server, identified by a string which is then used as hash (fragment) in URL. In this way URL really represents the state of the application so if it is shared and opened independently, visitor gets exactly the same application GUI state (opened windows, location on the page, etc.). On every GUI state change web application sends new state and retrieves string ID for it, updating the URL.

Additionally, this helps application developers to better understand how users are using the application as it allows precise tracking of their interaction with the application. django-hashlink is designed so that it does not store any user identifiable information along its data.

django-missing

Thu, 28 Mar 2013 22:09:10 -0700

This Django application bundles some common and useful features which have not (yet) found a way into Django itself.

cmsplugin-markup-tracwiki

Thu, 28 Mar 2013 22:02:10 -0700

A plugin for cmsplugin-markup which adds Trac wiki engine support to Django CMS. This means you can have content in Trac wiki syntax with all the power of it – Trac plugins and macros.

cmsplugin-markup

Thu, 28 Mar 2013 21:57:10 -0700

An extendable markup content plugin for Django CMS. It allows you to use various markup languages for content.

Plugins for Trac

Sat, 23 Mar 2013 02:58:10 -0700

I made, forked and improved, or contributed to many Trac plugins with emphasis on improving workflow of collaborative processes in projects using Trac to coordinate their efforts.

jQuery Visage

Sat, 23 Mar 2013 02:56:10 -0700

jQuery Visage provides a stable (no partially displayed elements), robust (it closes when you click close), styleable (CSS), customizable (you can reprogram many aspects of its behavior) and clean (being an jQuery plugin) way to display a series of images in a lightbox-like way.

Growl notifications add-on for Thunderbird

Sat, 23 Mar 2013 02:54:10 -0700

Growl New Message Notification add-on for Thunderbird is an alternative way to have Growl notifications when new e-mails arrive.

Lego Mindstorms NXT interface for Haskell

Sat, 23 Mar 2013 02:51:10 -0700

A Haskell interface to Lego Mindstorms NXT over Bluetoooth. It supports direct commands, messages and many sensors (also unofficial). It has also support for a simple message-based control of a NXT brick via remotely executed program (basic NXC code included).

Etage

Sat, 23 Mar 2013 02:45:10 -0700

Etage is a general data-flow framework for Haskell featuring nondeterminism, laziness and neurological pseudo-terminology. It can be used for example for data-flow computations or event propagation networks.

There is also an additional package with data-flow based graph algorithms.

Solar powered WiFi node

Sat, 23 Mar 2013 02:39:10 -0700

Solar powered WiFi node (in Slovene).

wlan slovenija open wireless network

Sat, 23 Mar 2013 02:37:10 -0700

wlan slovenija open wireless network at International Summit for Community Wireless Networks 2010.

wlan slovenija (part 2)

Sat, 23 Mar 2013 02:32:10 -0700

wlan slovenija open wireless network radio interview at local Radio Student (in Slovene). Part 2.

wlan slovenija (part 1)

Sat, 23 Mar 2013 02:24:10 -0700

wlan slovenija open wireless network radio interview at local Radio Student (in Slovene). Part 1.

Mitar's Point

Slovenian TDM exceptions and content available online

dinit

regex2json

Nonenumerable database

Not just a key-value database

DNS, DNSSEC and enumeration of names in a zone

Gun ownership registry

Verifiable voting

Possible design

PeerDoc – Scaling real-time text editing

Push-based and pull-based collaboration

Combining push- and pull-based collaboration

Discussion

Deployment

Collaboration is not easy

JSON decoding in node is fast

Results

Google search across languages

Store IDs in MongoDB as binary or as string?

Can humans grow up in zero gravity?

Database-abstraction APIs should not exist

Babel

Why we have database-abstraction APIs?

Parsing SQL into AST

Examples

Permission checks

Keeping historical versions

More ideas

Towards Automatic Machine Learning Pipeline Design

Abstract

In node.js, always query in JSON from PostgreSQL

Reactive queries in PostgreSQL

Proof of luck consensus protocol and Luckychain blockchain

Cookie consent fiasco

Pay-it-forward cryptocurrency

Decentralized governance and four fallacies

Sybil attacks and shell corporations

Wanted: precise terminology about democracy

One person, one vote or one dollar, one vote and blockchain

Blockchain as an Internet of things savior?

Free as in … ? My LibrePlanet 2016 talk

Bruce Sterling on startups' role in helping the global rich get richer

Learn how to walk

Speech recognition

Hacking capitalism with capped returns

Do Publicly Owned, Planned Economies Work?

Filter to extract all clean water from soda and other drinks

Towards layered re-decentralized web

Decentralized and permanent web

Decentralized and layered

Decentralized computations layer

Analogy

Conclusion

Fixing a bearer token vulnerability

Building free market from scratch

Taxation and rent

nodewatcher: A Substrate for Growing Your own Community Network

Presentation and release of nodewatcher 3.0

A panel on how to build community mesh networks

Why programming is like writing poems?

Futarchy: Vote Values, But Bet Beliefs

Sharing economy and its non-sharing truth

Limitations of Bitcoin revolution

Let's simplify taxes: tax all transactions

Debt and moral obligations

How to get people to work more than 8 hours a day for you?

Why vacations and 8 hour workday matter

Meteor Blaze Components

Trust and decentralization

Presenting some ideas of what is missing in open access and open science community

My presentation of PeerLibrary at OpenCon 2014

My talk about how we can create real digital libraries

Etage – A general data-flow framework featuring nondeterminism, laziness and neurological pseudo-terminology

Introduction

Example

Related work

Design

Conclusion

PeerLibrary is calling for collaboration on open dataset of all academic publications