cat /dev/random

2015 - A year in review

Fri, 01 Jan 2016 12:15:24 +0530

It’s that time of year when people are writing about what they accomplished in the year, sites are publishing countless the-best-of lists and the only thing I’m left wondering seeing YouTube Rewind is - “How come I’ve never heard any of this?”

Since last year, I too have started this ritual of writing down key highlights of my year so that I can look back a few years from now and have a chuckle at my naiveté.

Open Source

2015 was a great year for me on Github. I did quite a bunch of work across various projects, different technologies and overall had a blast compulsively typing the innocuous looking incantation - git push

Yay! Lots of green!

The highlight of the year, however, was my selection in Google Summer of Code 2015 and the project (called Surveyman) that I worked on during the summer. Having a great mentor, an exciting problem and a chance to work on React gave me enough motivation to spend late-nights after my day-job to hack on the project for approximately 4 months non-stop!

The first chunk of grey blocks visible around the month of March are due to a mini-burnout that I suffered. Coupled with college admissions announcements and a major release at my workplace I had no energy to hack after coming back from work. In order to recuperate, I decided to not write code on 2 days of the week - which is what you see in bottom gray boxes in April. In hindsight, it worked out well as the weekends were spent playing squash and finishing a couple of video games that were pending on my list for quite a while. Note to self: Do try this again!

New Tech - React, Go and Clojure

If 2014 was the year of Nodejs, 2015 was the year of React. Although I jumped on the React bandwagon pretty late, I got an opportunity to work on a quite a few fun projects. From boilerplates to chrome extensions - React till now has scaled to all my needs. It feels great to have an awesome tool in my arsenal that I can wield whenever I need something done on the front-end.

Towards the end of last year I promised myself to learn a Lisp and that’s how I stumbled onto Clojure. The sales pitch of offering the trifecta of expressiveness (macros), practicality (JVM) and concurrency sounded like a compelling alternative to the languages I was using for most of my day-to-day work. So roughly 365 days ago, I tried a small experiment wherein I ported some Python code over to Clojure and came away pretty impressed. In the ensuing months, after spending a good chunk of my time working on 4clojure and a mini-course, I eventually started writing (but unfortunately didn’t finish) a guide for others. To be honest, the only reason I decided to learn React was to get started with Om.

Long story short, little did I know that the language that I had become so fond of during the year, will eventually help me land an internship!

I’m super excited to share that I’ll be interning this summer in the beautiful city of LA with Factual where I’ll be building distributed systems in Clojure!

My affair with Go is still in its very early stages. Go is an ideal candidate for building easy-to-deploy concurrent programs and that’s what I used it for. Started initially with building a worker that consumes messages from a queue, I eventually ended up writing a crawler for bekanjoos in Go as well. Overall, I think Go is a language very easy to get productive with. It is simple, has easy-to-grok concurrency primitives and produces statically compiled artifacts which are an absolute pleasure to deploy on servers. At the risk of sounding controversial - I’d recommend that every Python / Node.js developer should give Go a try for their next backend project.

Courses and Papers

On the academic front, this year I fared better than last year. Outside of grad school, I completed two courses - Intro to Big Data in Spark and Scalable Machine Learning. All wasn’t as rosy though, I started with a couple more but had to drop out as I was unable to cope up. I plan to give these courses a second attempt in this year - let’s hope I am more successful this time.

One department that I drastically need to improve in is reading technical papers. I’m still excruciatingly slow when it comes to reading papers and even more so when I need to blog about it. My grandiose plans of blogging about every distributed systems paper I read collapsed as the semester picked up pace and I started falling behind in classes. Clearly spending 20 hours on reading one paper, making notes and then writing about is not scalable. Writing paper summaries on the other hand has been a bit easier as you expect the reader to have an idea of the paper already. The only positive aspect was that I met my goal of reading more than 15 papers spanning across cloud computing, distributed systems and information retrieval.

Personal

As far as relationships go, 2015 was just fantastic. The most exciting part of joining graduate school was meeting more like-minded nerds! I made a bunch of new friends, met school friends after eons and even played host to a complete stranger for an afternoon!

Last year’s post ends with me promising to get more active. Thankfully, having a room overlooking the picturesque Riverside Park meant that I didn’t have to work hard to fight my lizard brain to stay indoors. As a result, I started logging my runs and eventually ended up running my first 10k around Central Park!

Plans for 2016

Build a non-trivial mobile app using React Native
Write more Clojure(script). Publish something useful on Clojars
Blog about at least 5 more technical papers.
Run at least 150 miles over the course of the entire year

Here’s to an equally productive and fun 2016! 🍻

Understanding RPCs - Part II

Thu, 24 Sep 2015 20:33:29 +0300

In the previous post, we introduced remote procedure calls and their use within a distributed system. In this post, we are going to dive deep into the concept of semantics - first in the context of distributed systems in general and then discuss various RPC semantics.

Semantics

We’re now going to briefly talk about one of the most important aspects in a distributed systems - semantics. What do you mean by semantics? Before we get to that, let’s talk about failures. As we all know, failures are the norm rather than an exception in a distributed system. Failures be of many forms - network, machine or even datacenters. Hence, one of the goals in designing a distributed system is for it to be fault tolerant. We would ideally want our systems to hide the failures as much as possible and be available to our users.

However, as we’ll learn later, all kinds of failures cannot be completely hidden from the user. When one of the replica goes down, you can hide that from the user but when the majority go down it can be very hard to keep the system functioning. So what happens when your system cannot handle a failure mode or has some other limitation? The idea is to expose those semantics very clearly in your APIs to your users so that they can then in turn plan for that scenario.

One such example can be found in RethinkDB docs under the write acknowledgements section which is a configuration setting that can be set by the user.

The default is majority, meaning writes will be acknowledged when a majority of (voting) replicas have confirmed their writes. The other possible option is single, meaning writes will be acknowledged when a single replica acknowledges it.

By giving this option to the user, the creators of the RethinkDB allow the users to make a tradeoff between safety and performance.

RethinkDB’s default settings prioritize safety over performance, except in one case: read_mode defaults to single rather than majority. The majority read mode requires sending a query to all of the replicas and waiting for a majority to reply, which significantly degrades performance.

Hence, by clearly laying out semantics designers of distributed systems can keep their users informed.

RPCs Semantics

With that out of the way, let’s see how what kind of semantics are exposed by RPCs. One very important point to consider is that since RPCs are the building blocks of distributed systems, the semantics dictated by RPCs bubble up to higher layers and affect the design of a distributed system.

In the previous post we took at look at one of the key design goals of an RPC system.

Make the process of executing code on a remote machine as simple and straight-forward as calling a local function.

Surely this is a very worthwhile goal to pursue. Having a model that works similar to the local environment helps programmers in reasoning easily about the program and makes it easier to build distributed systems application. But the question remains that how far do RPCs systems generally come in achieveing this goal. How similar are they to LPCs (Local Procedure Calls) and to what extent does it the free the programmer from the mental overhead of thinking about communication in a distributed system?

Let’s say you are writing a program and decide to make a local procedure call. In this case two things can happen -

The call returns which means that either procedure was successful or it met with an exception.
The call didn’t return which means that it got stuck in an infinite loop and it didn’t complete. In this case, the call was not successful.

In both these cases, the final state of the system is clearly known and is easy to reason about. In our day-to-day work we deal with each scenarios multiple times and by now have become well equipped to deal with them.

When communicating over a network, things are not this straightforward. Consider the case when the programmer makes a remote procedure call. When the call returns, you know that the call succeeded - it’s all fine and dandy. However, when the call fails then are multiple scenarios that could’ve happened -

The call didn’t reach the server.
The call was executed but mid-way the server crashed.
The call was executed, the results were returned but the return message did not reach the client.

All of a sudden, something which seemed quite trivial has become sufficiently complex. To some extent, some of these problems can be dealt with repeated ACKs between the server and client. This is the approach as taken by the paper. The challenge with these approaches is that more messages put a stronger bound on the performance of the system.

In the face of these difficulties, there are three semantics that have been proposed for RPC systems -

At-least-once

At-least-once are the easiest to implement. As you can guess, the promise here is that the call would be executed one times or more. The way this works that the client keeps making the call to the server as long as it doesn’t recieve an ACK. In case the call is executed on the remote service but the ACK got lost, the client re-sends the call and server executes it. Quite clearly, one can see that the call might be executed multiple times - this is precisely where the at-least-once semantics comes from. The psuedo-code for this can look like the following -

method_executed = False
while not method_executed:
    ack = send_RPC_and_wait_for_ack(timeout=T)
    if ack and ack.SUCCESS:
        method_executed = True

When would this be useful? These semantics are useful when the operation being requested by the client is idempodent i.e repeated executions of the same call do not affect the state. An example of an idempodent operation is the setting a bit to 1 when it is zero and not doing anything otherwise. No matter how many times you call this operation, once the bit has been set, subsequent calls do not change the state.

At-most-once

In at-most-once the server either executes the call once or not at all. The client makes a call, the server executes it and sends an ACK. Whether or not the ACK is received by the client, the server does not run the procedure again.

request_sent = False
while not request_sent: # only care about msg delivery
    ack = send_RPC_and_wait_for_ack(timeout=T)
    request_sent = True
    if ack and ack.FAILURE:
        request_sent = True

Exactly-once

Exactly-once semantics are the gold standard of RPC semantics. It guarantees that each call is executed just once thereby mimicing a local procedure call. However, exactly-once semantics are extremely hard to build given the complexities in a distributed system.

To understand where the difficulty comes from let’s think about storing some state on the server to keep track of what all calls have been completed. This way when a new call comes in, depending on whether it has been executed or not we can take an appropriate action.

# On the server side
state = {}                       # to manage the state
if not state[client][method]:    # if not executed previously
    resp = handle(method)        # (1)
    state[client][method] = resp # (2) 
    return resp
else:                            # has been executed previously
    return state[client][method] # return cached response

Though the code above simple there are two big assumptions that it rests on. For starters, consider what happens when the server fails after executing (1) but before (2). In this case, the server hasn’t actually recorded the fact that the execution is complete. When the server is brought back up, and the client sends the request again, the server ends up executing the call again. Hence, the exactly-once semantics are broken.

One way out then is to find a way to make (1) and (2) atomic - ensuring either both of these commands execute or none of these commands execute. If you’ve done a course on databases, you know that this itself is a hard problem to solve.

Other limitations

Aside from the semantics, there are two other aspects of RPCs which are worth talking about -

Address Space: Suppose you’re making a local call and you need to pass in a big datastruture (e.g. linked list with 1000 nodes) to the method. How do you do that? You pass a reference to the object. The reason why this works is that in local call, the caller and the callee share the same address space. This is not true in an RPC. In such a scenario, you have to think about creative ways of wiring over a big datastructure efficiently over the network.
Performance: Performance in a RPC system is order of magnitude slower than the equivalent local call. Apart from the overhead of marshalling etc. the very fact is that the Round-trip-time puts a lower bound on performance - which in itself is substantial. Hence, typically RPC systems addtionally have an asynchronous version of their API so that the main thread on the caller is not blocked while it waits for the results.

Conclusion

In conclusion what we’ve seen is the mere fact of adding an unreliable network makes the goals of building a LPC-like abstraction so elusive. Although most of these problems have been solved to a large extent thanks to latest advances & the research but the point remains that the RPCs are much more complex than they seem to be. Hopefully with these set of blog posts, I’ve whetted your appetite enough to pursue these topics further.

Till next time!

Understanding RPCs - Part I

Wed, 23 Sep 2015 20:33:29 +0300

Having looked at a distributed system i.e the domain name system in the previous post, lets turn our attention to something more fundamental. In this blog post, we are going to start diving deeper into one of the basic blocks of Distributed Systems - Remote Procedure Calls or RPCs.

The paper that we’re going to be looking at today is authored by the duo of Nelson and Birell who were the first set of people to build an RPC implementation for their work at Xerox PARC. Nelson incidentally is also credited for the coining the term!

Why RPCs?

In the last post, we defined a distributed system as below -

A distributed system is a set of independent machines that coordinate over a network to achieve a common goal.

One of the important keywords in the above definition is coordinate. However, in order to coordinate, systems first need a way to communicate. Given that we have a messaging layer (the network) what kind of a communication abstraction we can build that can be helpful to programmers while building distributed systems?

RPCs are one such abstraction borrowed from programming languages that have a simple goal:

Make the process of executing code on a remote machine as simple and straight-forward as calling a local function.

But why did we specifically choose procedures? Let’s hear what the authors have to say - RPCs are based on the observation that procedure calls are a well known and well understood mechanism for transfer of control and data within a program running on a single computer. Therefore, it is proposed that this same mechanism be extended to provide for transfer of control and data across a communication network.

Components

Components involved in a RPC. Image courtesy: Birell and Nelson's RPC paper

The underlying mechanism of a RPC is simple - the caller machine makes a procedure call over a network to the callee machine which then executes the procedure locally, collects the results and sends it over to the caller. The figure above lays out the key components in an RPC system, which at a high-level has two main pieces: a stub generator and the run-time library.

The stubs are responsible for placing a specification and packing / unpacking the arguments falling that specification into the message. This message is then forwarded to the runtime so that it can be wired to the callee (in case of the user-stub). The process of packing the arguments is usually called marshalling or serialization of the message. Likewise, unmarshalling or deserialization involves extracting information received into something which the system can understand.

When writing a distributed application, a programmer first writes an interface module in which she specifies the procedure names, the types of the arguments it takes and finally the return types. The compiler then uses this definition and generates the user-stub and the client-stub.

// An example of an interface for generating stubs
service FacebookService {
  // Returns a descriptive name of the service
  string getName(),

  // Returns the version of the service
  string getVersion(),
    
  // Gets an option
  string getOption(1: string key),

  // Gets all options
  map<string, string> getOptions()
}

The runtime is responsible for retransmissions, acknowledgments, packet routing, and encryption. It handles much of the heavy lifting in the RPC system. One of the key challenges faced by the runtime is locating the remote service. Once the service is located, the RPC package then binds the importer of the interface to an exporter of the interface.

Naming

The problem of naming is a common one in distributed systems. In a cluster how do we know the names and addresses of each of the machines? How do we maintain this list and do we keep this dynamic? We’ll look at these problems much later when we talk about directory services and service discovery.

In the paper the authors use Grapevine, a distributed service that provides DNS service, resource locating service, authentication and mail service.

The major attraction of using Grapevine is that it is widely and reliably available. Grapevine is distributed across multiple servers strategically located in our internet topology, and is configured to maintain at least three copies of each database entry. Since the Grapevine servers themselves are highly reliable and the data is replicated, it is extremely rare for us to be unable to look up a database entry.

For more info, checkout the paper on Grapevine.

Protocol

Once the caller knows which callee it needs to communicate with, the next question is which transport level protocol to use? The authors had an option of going ahead with a PUP byte stream protocol but decided against it as the protocol was ideal for sending bigger data packets. In the case of RPC, the design goal was different -

One aim we emphasized in our protocol design was minimizing the elapsed real-time between initiating a call and getting results. With protocols for bulk data transfer this is not important: most of the time is spent actually transferring the data.

In the modern world, we have an option of using UDP (an unreliable protocol) or TCP (a reliable protocol). While the choice might seem clear, the caveat here is that building RPC on top of a reliable communication protocol can lead to a severe drop in performance. Hence, many RPC packages are built on top of unreliable communication layers (e.g. UDP) and then bake in the extra logic for dealing with ACKs, retries etc.

Now would be a good time to talk about Thrift - the RPC framework developed at Facebook. Thrift has its own interface language called IDL that is used to generate the stubs. The communication protocol uses protocol buffers as its data format under the hood. Unlike XML/JSON protocol buffers are a binary format and hence is much smaller, less ambiguous and faster than a plain-text data format.

Apache Thrift allows you to define data types and service interfaces in a simple definition file. Taking that file as input, the compiler generates code to be used to easily build RPC clients and servers that communicate seamlessly across programming languages.

Nowadays Thrift is being used more and more amongst heterogenous services to talk amongst each other - for example, you could write a user authentication service in Java, but call it from your Ruby web application.

Conclusion

In this post we went over the overall idea behind RPCs and took a deep dive into the components which form a RPC system. In the next post we are going to continue our discussion about RPCs by talking about semantics and some limitations of RPCs.

The Domain Name System

Fri, 18 Sep 2015 20:33:29 +0300

One of the classes that I’ve taken this semester is Advanced Distributed Systems. The course is primarily a research seminar wherein the students are required to read a couple of research topics for each class and participate in a two hour discussion which is guided by the professor. My goal in these series of blog posts is to summarize the papers in an approchable manner primarily to test my own understanding of the topic.

The Domain Name System

The first paper discussion for this class was a paper on the Developement of the DNS. This paper examines the ideas behind the initial design of the DNS in 1983 and discusses the evolution of these ideas into a working implementation. Why is this paper important in a distributed systems class? Before we answer that, lets try to define a distributed system.

A distributed system is a set of independent machines that coordinate over a network to achieve a common goal.

If you look at the definition (particuarly the emphasized words) above carefully, you’ll realize that the DNS indeed behaves as a distributed system. Infact the DNS is one of earliest examples of a distributed system deployed at scale.

The paper

The hierarchical structure of DNS. Image courtesy: Wikibooks

Before the DNS, the HOSTS.TXT file was used for publishing the mapping between host names and addresses. Eventually as the number of users and workstations grew it became harder and harder to transmit the file due to its increasing size.
An early design goal of the DNS system was, aside from feature-parity with HOSTS.TXT, was the ability for the system to be independent of network topology and to be capable of encapsulating other names spaces.
The initial DNS design assumed the necessity of striking a balance between a very lean service and a completely general distributed database. The leaner and more general the service, the easier it would be to extend and replace the existing implementation. Similarly, although the service offers lookup and persistence like a database it would not provide other features such as atomic guarantees around updates.
The active components of the DNS are of two major types: name servers and resolvers.
- Name servers are repositories of information, and answer queries using whatever information they possess.
- Resolvers interface to client programs, and embody the algorithms necessary to find a name server that has the information sought by the client.
Early on it was decided that the DNS would exhibit a distributed system of control and its design (as a tree with organizations being in control of their subtree) grew as a natural consequence of this choice.
The domain name of a node is the concatenation of all labels on the path from the node to the root of the tree. Hence, the domain www.google.com is a node in the tree and has www, google, com and . as its ancestors. This has been demonstrated more clearly in the DNS resolution section below.
The DNS provides two major mechanisms for transferring data from its ultimate source to ultimate destination: zones and caching.
- Zones are sections of the system-wide database which are controlled by a specific organization. The organization controlling a zone is responsible for distributing current copies of the zones to multiple servers which make the zones available to clients throughout the Internet.
- Caching is a mechanism whereby data acquired in response to a client’s request can be locally stored against future requests by the same or other client. The mechanism for controlling caching is a time-to-live (TTL) field. However, all components of the DNS prefer authoritative information to cached information when both are available locally.

The rest of the paper talks about the issues with implementing and gradually migrating the servers to use the new servers. Their experience with performance, for example, is quite indicative of how hard it is to benchmark systems that have multiple components all of which keep changing quickly.

A related surprise was the difficulty in making reasonable measurements of DNS performance. We had planned to measure the performance of DNS components in order to estimate costs for future enhancement and growth, and to guide tuning of existing retransmission intervals, but the measurements were often swamped by unrelated effects due to gateway changes, new DNS software releases, and the like.

Negative Caching

A negative cache is a cache that also stores “negative” responses, i.e. failures. This means that a program remembers the result indicating a failure even after the cause has been corrected. In DNS, negative caching is a feature and the authors make a strong case for it in the paper.

Our conclusion is that any naming system that relies on caching for performance may need caching for negative results as well. Such a mechanism has been added to the DNS as an optional feature, with impressive performance gains in cases where it is supported in both the involved name servers and resolvers.

The primary reason for maintaining a negative cache is performance. As seen by the authors, typically one out of every four DNS queries were for negative results i.e asking resolvers for hosts or data that did not exist. To ensure that the volume of queries do not impact the system, the servers and resolvers would cache these negative responses with its own TTL.

Root Servers

The most interesting part in the paper for me was the idea of root servers - the servers which form the apex of the domain name system. In absence of a caching mechanism every DNS query would have to flow via one of these root servers. How many servers are there? Let’s run dig to find out.

$ dig NS +noadditional +noquestion +nocomments +nocmd +nostats . @8.8.8.8

	NS	a.root-servers.net
	NS	b.root-servers.net
	NS	c.root-servers.net
	NS	d.root-servers.net
	NS	e.root-servers.net
	NS	f.root-servers.net
	NS	g.root-servers.net
	NS	h.root-servers.net
	NS	i.root-servers.net
	NS	j.root-servers.net
	NS	k.root-servers.net
	NS	l.root-servers.net
	NS	m.root-servers.net

So there are just 13 servers responsible for the whole internet? Well no, this does not mean there are 13 physical servers; each operator uses multiple servers distributed geographically to service the requests.

Who operates these servers? These servers are collectively operated by universities, companies and government bodies. To know more about who operates which server see the entry on Wikipedia.

DNS Resolution

Now that we know the overall architecture of the DNS lets see if we can figure out how a typical DNS resolution happens. Although the full cycle spans multiple steps, in practice heavy levels of caching at each step ensures that no typical name-server is inundated with requests. However, let’s assume that in this example all our caches are purged and hence our query goes via the root.

Suppose the DNS query is for: www.google.com

[subdomain] [domain]  [TLD] [root (implicit)]
    www    . google .  com   .

The DNS lookup starts from right to left.
When the . is encountered, the client queries the root name server asking for what all name servers are available to service its request. The root server responds with the list of 13 name servers.
The client then picks a name server (lets assume d.root-servers.net) and asks the name server about the .com TLD (top-level domain). The root server responds with all the name-servers that serve this TLD (which interestingly is also 13)
Following along, the client picks up a name server and asks it if it knows about the google domain. The server responds with the name servers that manage that namespace. In Google’s case, the company itself manages its name servers but for usual websites this typically resolves to the DNS provider (e.g. Godaddy / Namecheap etc.) with which the domain name is registered.
In the last step the query for the subdomain (www / mail / maps etc.) is made and the name server accordingly maps to the IP of the server as specified in the A (Address) record. This typically for large websites is the IP of a load balancer that is responsible for forwarding requests to the correct webserver.
After the DNS resolution is complete, the browser caches this mapping and uses it to directly for subsequent requests (e.g images / css / javascript etc.)

Conclusion

Overall the paper give an interesting historical perspective on why the DNS came to be and the issues that came to light during its implementation and deployement. The paper, surprisingly, is also littered with good advice about software engineering and the human-side that forms the essential aspect of building software. I will end this writeup with my favorite sentence from the paper -

Documentation should always be written with the assumption that only the examples are read.

Till next time!

Scalable Machine Learning

Fri, 31 Jul 2015 20:33:29 +0300

A couple of days ago I finally finished the last assignment of the Scalable Machine Learning course. The last few days in Kuwait were a bit tough as I had tons of stuff to do before leaving the country. It turns out leaving Kuwait is even harder than getting there in the first place. Additionally I had set a personal deadline of finishing the course a week before the actual deadline so that I when I reach home I could focus on other work.

Enough of rambling - let’s get on with the review. Scalable Machine Learning or CS 190.1x is the second installment in the Big Data series at EdX. The course picks up where the previous one - CS 100.1x left off and dives deeper into using Apache Spark’s MLlib to solve machine learning problems.

If you’re interested to know more about CS100.1x, go ahead and read the course review for it here.

Since MLlib runs on top of Apache Spark it comes with the same benefits as Spark i.e performance, scalability and easy-to-use APIs in Python, Scala and Java. The library implements a host of popular machine learning algorithms out of the box such SVM, random forests and k-means clustering. In this course, however, the assignments ended up requiring only logisitic regression and linear regression with L₁ and L₂-regularization.

Pedagogy

The coursework spans five weeks (six if you include the week for setup) thereby making it almost as long as CS 100.1x. The course starts off with an introduction to types of machine learning algorithms - supervised learning & unsupervised learning. The following week is same as the second week of CS 100.1x and hence can be skipped entirely.

The three subsequent weeks cover three ML algorithms with one in each the lecture series and the accompanying labs -

Linear Regression
Logistic Regression
Principal Component Analysis

Each week there is roughly an hour long video content to be watched. At the end, lectures are followed by a quiz. Like the previous course, there are no final exams that need to be taken.

In contrast to a typical ML class, the lectures are pretty light on the math which make it more approachable to newcomers. The course actually focuses more on the practical side of ML and less on theory. I wouldn’t call it a rigorous discussion of machine learning algorithms, however, they do teach the amount of theory a person will need for developing an intuition for the algorithm and solving the programming assignments. If you’ve done the ML course by Andrew Ng, then the pedagogy of this course will seem quite different.

Programming Assignments

The programming assignments like the ones in CS 100.1x are the highlight of the course. The usage of IPython for exploratory data analysis is stellar. The notebooks follow the same incremental approach like the last course and are interspersed with beautiful graphs all of which help you in making sense of the data.

The assignments deal with practical problems and thus are quite interesting. There’s one problem directly from a Kaggle competition and an assignment on PCA that deals with identifying patterns of brain activity in a larval zebrafish. The end result generates a heat-map like figure which seems very gratifying after the laborious programming assignment.

Oh look - I generated similar temporal regions in a larval zebrafish via principal component analysis. #SparkMLcourse pic.twitter.com/tARm66Igrl
— Prakhar Srivastav (@prakharsriv9) July 26, 2015

On the negative side, the datasets used in the assignments are rather small and in few assignments the model is only run against the test and validation data set to test for accuracy i.e. the model is not used to make predictions on new data (a minor but important point).

Scalable?

How does this course teach the scalable aspect of machine learning? For one, the fact that you’re using Spark RDDs to run on your workers itself gives you ability to run your code on a cluster for free of cost. The other aspect is that the professor sets aside a couple of videos each week to discuss cases where both N(the number of data points) and D(the number of features) are large - a key characteristic of big data.

Unfortunately, none of these techniques are used in the programming assignments so it is quite possible that these lessons might not stick with you for long. Having said that, the treatment of these topics in the lectures is relatively interesting and sufficiently detailed.

Conclusion

In summary, CS 190.1x is a reasonably good introduction to Machine Learning. If you’ve never done any ML before, then this course will surely whet your appetite and encourage you to dive deeper into other popular algorithms. If you’re reasonably proficient with ML, then this course will serve to be a primer to Spark’s MLlib library - although the low coverage might leave you a bit disappointed.

The instructors indicated that they might come out with a second version of course that will cover more algorithms in detail. I, for one, will surely be waiting for more info on that one. Overall, in my opinion, time spent doing this course is time well spent.

Week 10 - Surveyman

Wed, 01 Jul 2015 15:50:49 +0300

After a fun and productive two weeks at home, I have now rejoined office back in Kuwait. As all good things in life, the vacation has come to an end and it’s now time to head back to the usual humdrum of 9 to 5. What also didn’t help was the fact that since most of easy (low hanging fruits, if you will) features had already been built, I had no option but to start work on the feature which I most dreaded - implementing sortable on blocks / questions.

The primary idea behind this feature is to allow users to reorder questions and blocks in the survey. Instead of manipulating the survey directly (which can be arbitrarily long) the user interacts with the minimap (or the treeview) and the changes are reflected in the survey when the drag-and-drop is over.

Since this feature was particularly hard to tackle I decide to chalk out a rough plan on what are the sub-problems that I need to solve in order to ship this feature. The github issue lays this out quite clearly. To begin with, I thought long and hard on what this feature and why this is required. I then broke this down into tasks or todos which I could tackle one by one. This was done primarily so that I stop feeling overwhelmed and just concentrate on the task at hand.

I’ve been tagging commits very religiously in this project so the best way to understand the code would be to checkout the commits tagged in the issue above. Piece by piece it’ll start to make sense. In the remaining part of this blog post I quickly touch upon two key things that I struggled with and how I found a solution.

The first issue that I spent quite a long time breaking my head had to do with drag-and-drop. The initial drag and drop that I implemented was not behaving at all like the version done by Dan Abramov. However, after comparing the code and spending hours seeing the DOM behavior the issue seemed to be with non-index keys passed to the React component because of which React was unable to reconcile the virtual DOM.

// changing the value of key from index i to
<BlockNode key={i} id={block.get('id')} >

// below
<BlockNode key={block.get('id')} id={block.get('id')} >

The second issue was with implementing the state propagation where the idea was to allow the treeview to be manipulated and when the drag-and-drop operation finished then propogate and update the actual survey. Since the treeview recieved the json survey as a prop I had to create a replica of it in the local state and let the component re-render when the props changed. After looking around the documentation I found that there’s indeed a lifecycle method that takes care of this -

var TreeView = React.createClass({
    propTypes: {
        survey: React.PropTypes.instanceOf(List)
    },
    getInitialState() {
        return {
            survey: this.props.survey,
            finalIndex: -1
        }
    },
    componentWillReceiveProps(nextProps) {
        this.setState({
            survey: nextProps.survey,
            finalIndex: -1
        });
    },
...

The finalIndex above is used to keep track of the final location of where the item is dropped. This value is then sent to the store so that it can make the necessary changes. Below is the code for reordering Blocks for example -

reorderBlock(draggedBlockId, overBlockId) {
    var survey = this.state.survey;
    var draggedBlockIndex = survey.findIndex(b => b.get('id') === draggedBlockId);
    var overBlockIndex = survey.findIndex(b => b.get('id') === overBlockId);

    var block = survey.get(draggedBlockIndex);
    var newSurvey = survey.delete(draggedBlockIndex).splice(overBlockIndex, 0, block);
    this.setState({
        survey: newSurvey,
        finalIndex: overBlockIndex
    });
}

So with that feature shipped the drag-and-drop sortable is now complete. Go and try it out!

Until next time!

Intro to Big Data with Apache Spark

Sat, 27 Jun 2015 20:33:29 +0300

Last night I finished the final assignment for the new course that I had been working on in the past week called Intro to Big Data with Apache Spark or CS100.1 x. With the course over, I decided to write down a quick review in the hope that it will help others get an idea of what they can expect by enrolling in this popular MOOC by UC Berkeley.

I heard about the course after stumbling upon a laudatory review by an existing student on Hacker News. After learning that the course involves programming assignments that teach you the Spark API, I couldn’t help signing up.

TL;DR - If you’re a programmer who wants to explore what the hype on Spark is all about and considers solving code katas a great way to spend your evenings, then take this course - you will not be disappointed.

Apache Spark

So what is Apache Spark? Spark is a fast and general-purpose cluster computing system that is used for large-scale data processing. The project was developed by the AMPlab at UC Berkeley and has now evolved into a standalone company - Databricks that, among other things, provides enterprise consulting and training for Apache Spark.

Why should you care? Well, if you’re working as a data scientist or a data engineer you’re no stranger to the monumental growth of Spark in the bigdata field. Spark has been taking the big-data world by storm after showing consistent benchmarks that boast of it being upto 100x faster than Hadoop. Late last year, Spark also made waves by setting the world record for large-scale on-disk sorting.

In order to evangelize Spark and help developers / data scientists learn about it, Databricks and UC Berkeley teamed up to start a Big Data Xseries on EdX that consists of two MOOCs - CS 100.1x followed by Scalable Machine Learning.

The Big Data XSeries, created in partnership with Databricks, will articulate the expected output of Data Scientists and then teach students how to use PySpark (part of Apache Spark) to deliver against these expectations. Students will gain hands-on experience applying big data principles using Apache Spark, a cluster computing system well-suited for large-scale machine learning tasks.

Pedagogy

With the elevator pitch out of the way, let’s get back to the review. The entire course spans 5 weeks (including week 0 which dealt only with the setup) with the material consisting of lecture videos, short quizzes and programming assignments. Unlike the typical technical MOOCs that I’ve done in the past, there are no exams and surprisingly the last week has no lecture videos to watch.

The lecture videos, in my opinion, were pretty light in content. Even on a lethargic evening, it took me roughly somewhere around 30 mins to breeze through the videos and answer the quizzes. The first week started off with talking about Apache Spark and how it is different from Hadoop. The two subsequent weeks focused on issues with data management and talked about how a typical big data processing pipeline looks like. The penultimate video in week four spent roughly 5 mins talking about Collaborative Filtering and Spark MLib which somehow seemed enough to the teaching staff to justify a programming assignment on the topic.

As far as forums and class interactions go, the instructors are quite active on the forums and the staff is quick to sort out any hiccups with the autograder. I got stuck a couple of times but was quickly able to find a way by searching for previous threads in the forums.

Programming Assignments

Coming to my initial motivation for taking this course, if are you wondering whether this course delivered on its promise on teaching Spark then the answer is a resounding yes! The programming assignments were exceptionally well thought-out and interesting. Each week’s lab exercise was presented as an IPython notebook which provided the ideal environment for interacting with and exploring the Spark’s Python API. (As an aside, if you haven’t checked out IPython notebooks yet, please do yourselves a favor and give it a go.)

What the course lacked in the lectures was more than compensated for in the extremely detailed and well presented notebooks. The instructors leveraged the entire set of IPython capabilities wonderfully as the notebooks were interspersed with diagrams, LaTeX equations and even plots courtesy of Matplotlib.

The typical structure followed in the programming assignments was

Introduce a topic with theory and examples
Give a small coding exercise on solving the problem
Run your code against the test cases provided

Rather than having a big problem statement for the students to solve in one go, the bite-sized problems and test cases provided instant feedback and the necessary encouragement to keep forging ahead. Lastly, much to the project’s credit the Spark documentation was well written which made the process of finding a function that does X to using it quite gratifying.

Even though Spark supports variety of languages R, Spark SQL, Java, Python and Scala, the instructors choice to go ahead with Python and IPython clearly shows how powerful of a teaching tool it can be.

Setup

Before jumping on to the first assignment, I was reasonably certain that setting up a cluster computing system was going to be a big pain - even on a single machine. The courseware’s long setup document and set of 5 videos also corroborated my fears. What turned out to be delight though that it took me less than 10mins to get the entire setup running.

Showing great wisdom on their part, the course designers made the process extremely straightforward by providing a custom Vagrant image which had everything installed. Since I use Vagrant on a daily basis, going from never running Spark to a functioning Spark environment was just a matter of few commands.

$ wget <course_materails_url>
$ cd bigdata_course
$ vagrant up
# 5 mins later ... IPython is running on 8001

The VM also came bundled with all the required IPython notebooks and data associated with the labs which meant that you needed to setup everything just once.

Conclusion

CS 100.1x is probably the course where I’ve spent the least amount of time and had the most amount of fun (Steve Huffman’s web dev course comes a close second). I started a week later and within 3 weekends I was able to finish the material. While that could be attributed to my prior experience with machine learning and functional programming, from what I could see in the forums students were able to finish the assignments within reasonable time.

What this goes to show is that the material is light, focused and can be done with quickly if you find it interesting - much like any other crash course. For someone who’s curious to understand how Spark works and wants to dip their toes in the water, I doubt there can be a anything more fruitful than spending a couple of weekends working through the programming assignments.

Overall, I had a great time learning about the Spark computation model, its API and how it can be efficiently used to process large amounts of data. Although the path from completing this course to setting up a data processing pipeline on a cluster for your company is a tough one, I feel that the material provided me enough tools to go and explore further.

Week 9 - Surveyman

Sun, 21 Jun 2015 15:50:49 +0300

This is my last week at home (India) and the fact that I had a pretty productive week shows that I’m ensconced within the comforts of my home. After the hectic last week which involved traveling, meeting relatives and getting documentation ready for the Visa interview this week was just me sitting at home working on this project without any distractions.

So there were three key features that I tackled this week, namely - TreeView, Option Lists and React transitions. Cumulatively, these changed the UI considerably. The app has finally started to look much more polished.

TreeView

TreeView and OptionList in action

TreeView makes it easy for users to navigate the survey easily. The idea is inspired by Sublime Text’s minimap that allows you to conveniently navigate the file. Having the ability to collapse and expand blocks and questions will be useful in getting a particular item in focus just by clicking on it.

To get started with the feature, I surveyed the landscape to check if there already existing a customizable TreeView component. After checking out a few examples on react.rocks I didn’t find anything that met my requirements so I decided to build my own. More importantly, since the structure of the JSON was simple building a tree view didn’t seem too time consuming.

Before I started writing any code, I read through Cheng Lou’s React Treeview to get an idea of how to approach the problem. So at the end, I just ended up creating two new components - one called TreeNode and another called TreeView. This is how the render function of TreeView looked like.

var TreeView = React.createClass({
  render: function() {
    return (
        <TreeNode key={i}
              label={blockLabel}
              type={ItemTypes.BLOCK}
              handleClick={self.focusOnItem.bind(this, block.get('id'))}>

                 {questions.map((ques, j) => {
                     return (
                         <TreeNode key={j}
                             label={label}
                             type={ItemTypes.QUESTION}
                             handleClick={self.focusOnItem.bind(this, ques.get('id'))} />
                     )})}
        </TreeNode>
      )
    }
  });

To add the additional functionality of bringing the object in focus just required a simple change to window.location like so -

// inside the TreeView component
focusOnItem(id) {
       window.location.hash = id;
}

Option Lists

If you’ve ever built a survey before, you know that most of the questions will end up having the same set of options. For example likert scales are very common, especially in market research surveys. Having to type all the options multiple times can get frustrating and hence providing an easy way to reuse a common set of options would be much better.

That is where OptionLists come in. Options List make it very convenient to re-use an existing set of common options or add your own. In order to use them, you just to drag and drop them on your question.

Technically, option lists were quite simple to build. At the Store I maintain an array of option lists which can later be persisted in LocalStorage. When the user selects an option list from the dropdown I remember that state in the Store. Lastly, when an DraggableOptionList is dropped on a Question I read the index from the Store and add the options to the questions. It’s just that simple.

Transitions

The last improvement that had been on my mind for a while was to add transitions in the application. If you interact with the items for example - try deleting an item you’ll notice that it just disappears out of the survey. The result can feel quite jarring. What I wanted to do was find a way to add subtle animations so that the UI feels much more pleasant.

Unsurprisingly, for transitions I used ReactCSSTransitionGroup which is explained quite clearly in the documentation.

ReactCSSTransitionGroup is based on ReactTransitionGroup and is an easy way to perform CSS transitions and animations when a React component enters or leaves the DOM. It’s inspired by the excellent ng-animate library.

Basically, you just wrap your container in ReactCSSTransitionGroup and provide the CSS class that you want to apply on entry and exit of any child component.

<ReactCSSTransitionGroup transitionName="itemTransition" transitionEnter={false}>
    { blocks }
</ReactCSSTransitionGroup>

The best part of building this was when I learnt that there exists a CSS pseudo class called target.

The :target pseudo-class represents the unique element, if any, with an id matching the fragment identifier of the URI of the document.

So you want to animate a subtle effect when the browser focuses on an item? Try this -

:target {
  animation: highlight 1s ease-in;
}
@keyframes highlight {
  0% { background-color: #fcf1bb; }
  100% { background-color: white; }
}

Wicked cool, right?

Week 8 - Surveyman

Mon, 15 Jun 2015 15:50:49 +0300

If you’ve read the previous blog post, you would be aware that I’m right now back to my home country on a vacation. This week has been quite hectic and filled with anxiety owing to the visa interview that I had scheduled during the start of the week. Since this was my first time appearing for a US Visa interview, I was quite in awe of how the smooth and planned the entire process was, especially compared to the other interview processes I’d been in (I’m looking at you Kuwait). Thankfully, my visa got approved and now I just can’t wait for grad school to start!

So circling back to Surveyman, this week, I deployed a relatively straight forward but important feature that allows users to clone or duplicate existing blocks / questions. While interacting with the survey I constantly felt that need to build similar blocks. For example, if you have a block having 3 questions and the next block that you want to add has the same number of questions in a similar configuration, building it incrementally can be exhausting.

In cases like these, which I am hoping would be quite common, a simple click that clones the entire block or question would be helpful. That was the primary motivation for building this feature. As far as writing the code goes, this feature was quite simple (at least in theory). The idea was simply deep clone the question or block object and update the survey JSON when required. What you would also need to ensure is to keep the two maps i.e. OptionMap and QuestionMap aware of the new items so that the mappings are in sync with the app state

The most fun I had while building this feature was working with the Immutable API. Having a brief stint with ClojureScript before, the idea of returning new collections after every mutable operation and subsequently then chaining operations was not new to me. This is how, for example, the code for cloning the Question looked like -

/**
 * Returns a clone of Question with new ID of itself
 * and all options.
 * @param question - type of Immutable.Map. The question to be cloned
 */
cloneQuestion(question) {
    var self = this;
    return question
            .set('id', self.getNewId(ItemTypes.QUESTION)) // generate new ID
            .update('options', (list) =>                  // do the same for all options
                        list.map(
                            o => Immutable.Map({
                                id: self.getNewId(ItemTypes.OPTION),
                                otext: o.get('otext')
                            })
                        )
                    );
},

After getting the new clonedQuestion, I just needed to append it to the parent container and update the relevant OptionMap and QuestionMap.

let newQuestion = this.cloneQuestion(question);
let newSurvey = survey.updateIn([blockIndex, 'questions'], list =>
    list.splice(questionIndex + 1, 0, newQuestion)
);
// update the maps
_questionMap = _questionMap.set(qId, block.get('id'));
newQuestion.get('options').forEach( o => {
    _optionMap = _optionMap.set(o.get('id'), qId)
});

Similarly, the code for adding blocks was also pretty simple. However, since it was one level up the chain what required was all child questions should be also be updated with the mappings. If you’re looking at diving deeper, feel free to browse through the commits tagged with this issue.

Until next time!

Week 7 - Surveyman

Mon, 08 Jun 2015 15:50:49 +0300

Guess what, I’m writing this blog post from my home in Delhi! I’ll be in India for full 2 weeks where apart from visiting my family, cousins and friends I’ll also be giving my visa interview - which is the main reason why I’ve taken a vacation.

As I had been traveling the progress this week has been rather minimal. Keeping the travel schedule in mind, I had kept the goal to be reasonable which was upgrading the react-surveyman project to use React-DnD’s latest 1.x version.

If you’ve been following the progress of the project closely then you’re probably aware the that React-DnD is the drag-and-drop library for React that I use in my projects. Authored by the super productive and super smart, Dan Abramov, React DND abstracts out the various HTML5 drag-and-drop events and gives you a nice way to customize the drag and drop behavior of your React components.

Earlier last month, Dan published the 1.0 release of the library where he made considerable changes in the API. Keeping in the mind the best practices in the React world, the most significant change was replacing mixins with decorators. To understand the motivation behind this change, you can read the blog post written by the man himself.

To be clear, mixins is an escape hatch to work around reusability limitations in the system. It’s not idiomatic React. Making composition easier is a higher priority than making arbitrary mixins work. I’ll focus on making composition easier so we can get rid of mixins. - Sebastian Markbage (React core team)

All in all, the change was quite straightforward, the credit for which goes to the excellent documentation on the website. Dan has worked really hard to make sure that his library is approachable and it clearly shows. The tutorial itself, is so well explained that I’d go to the extent to say that it’s one of the best React tutorials out there.

Attention library authors: @dan_abramov's tutorial on React-DND is the new gold standard for API documentation. http://t.co/AeLugT5Iov
— Prakhar Srivastav (@prakharsriv9) June 3, 2015

So after working through the tutorial and the browsing thorough the examples, I was able to successfully port both react-surveyman and the react-tags project to use the latest React-DnD library.

Week 6 - Surveyman

Mon, 01 Jun 2015 15:50:49 +0300

So I started this week with the primary goal of re-writing my stores and migrating them to use Immutable collections. In the last week’s post, I laid out the motivation on why this is a good idea and how it will speed up the React application.

While the benefits were clear, I was still not sure on how I could implement and exploit the collections provided by Immutable to my advantage. After numerous posts on forums, Github issues and IRC discussions I realized that I was looking at it from the wrong direction.

The key idea was explained to by Matt Greer whose extremely long and clear email helped me really understand the issue. With that out of the way, slowly and steadily I was able to migrate the store piece by piece to Immutable.

Initially it was a bit frustrating to keep updating nested structures using set but after finding the updateIn method, things got really good.

// on question dropped
var index = this.getBlockIndex(questionObj.parentID);
var newSurvey = this.data.surveyData.updateIn([index, 'questions'], list =>
    list.push(newQuestion)
);
this.updateSurveyData(newSurvey, true);

Throughout the store, I also keep track of two maps namely, _optionMap and _questionMap which are basically mappings from the option ID to the question ID and from the question ID to the block ID respectively.

// on question dropped - update question map with new question
var block = this.data.surveyData.get(index);
_questionMap = _questionMap.set(newQuestion.get('id'), block.get('id'));

Adding Undo

After finishing up integrating Immutable and adding PureRenderMixin in all my components, my next goal was to quickly add a undo feature that would allow users to undo a destructive action.

The idea here is simple - at each stage cache the application state in a history object and use that to go back when the user clicks on undo

// mananging history
var _history = [];

// in the store
updateSurveyData(data, cache = false) {
    if (cache) {
        _history.push({
            data        : this.data.surveyData,
            optionMap   : _optionMap.toJS(),
            questionMap : _questionMap.toJS()
        });
    }
    this.data.surveyData = data
    this.trigger(this.data);
},

The other side benefit that this allowed me was to add an undo feature which made sure users are allowed to go back any destructive action they carry out in the UI. To make sure only destructive actions are cached, the updateSurveyData takes an extra cache param which when true pushes the data into the history. After this, the undo operation is simple to implement.

onUndoSurvey() {
    // retrieve cached data
    var { data, optionMap, questionMap } = _history.pop();
    _questionMap = Immutable.Map(questionMap);
    _optionMap = Immutable.Map(optionMap);
    this.updateSurveyData(data);
}

That finally sums up the work that went in this week to finish two key features. With the architecture now stable, I feel much more confident in adding more features without impacting the performance. However, now that the codebase has grown to roughly ~1500 lines of code, it is high time to add a test suite so that refactoring becomes more managable and less scary.

The plan for this week, then, is to add tests and make the codebase more robust. Till next time.

Week 5 - Surveyman

Sun, 24 May 2015 15:50:49 +0300

I signed off the last post mentioning that I’ll be integrating the React tags component into Surveyman which did not turn out to be difficult. As the API of the component was simple, it just meant adding the component and providing the necessary event handlers.

The suggestions are loaded as soon as a Question component is dropped, which after careful thought, sounds like a practical tradeoff. Suggestions are maintained as a set and whenever is new Option is added by the user, it gets appended to the set.

// In the store:
var _optionsSet = Immutable.OrderedSet();

var SurveyStore = Reflux.createStore({
    onOptionAdded(otext) {
        // ..
        _optionsSet = _optionsSet.add(otext);
    },
    getOptionsSet() {
        return _optionSet;
    }
});

On the component side, the react tags live in an overlying Options component.

var Options = React.createClass({
    getInitialState() {
        return {
            suggestions: []
        }
    },
    componentDidMount() {
        var optionSet = SurveyStore.getOptionsSet();
        this.setState({ suggestions: Array.from(optionSet) });
    },
    render() {
        return (
            <ReactTags tags={options}
                suggestions={this.state.suggestions}
                handleAddition={this.handleAddition}
                handleDelete={this.handleDeletion}
                handleDrag={this.handleDrag}
                labelField={'otext'}
                placeholder={"Add new option"} />
        )
    }
    // ...
});

You can view the final code here.

After finishing up this by mid week, I was deciding on what issue to work on next. Although I should have run tests to prove this but it felt that rendering the entire application whenever a new option was added or removed was leading to performance bottlenecks. Since most of the data remainded unchanged most of the re-renders were useless and could be pre-emptively cancelled by the handly lifecycle function - shouldComponentUpdate.

On reading up further, I found that React already provides a mixin called PureRenderMixin that does exactly does.

Under the hood, the mixin implements shouldComponentUpdate, in which it compares the current props and state with the next ones and returns false if the equalities pass.

This pattern is indeed a common way to improve performance of your React applications. The only catch though was that for this to work correctly, we needed some way to quickly do shallow checks in order to determine equality. Since all our data i.e. questions and blocks is nested this meant we could not use the mixin as it is. As recommended by the docs, this is where using Immutable collections with a library like Immutable comes in handy.

So for next week, my goal will be convert all my collections into Immutable so that I can harness the power of PureRenderMixin in my app! Stay tuned for next week.

Week 4 - Surveyman

Sun, 17 May 2015 15:50:49 +0300

I closed last week’s update giving an indication that I started work on a standalone React component. After working on the component this week, I was finally able to finish it and publish it on NPM.

So before I go into what React-tag-input does, I’ll try and explain the motivation for building it. If you’ve been following closely, the primary way of building the survey in Surveyman is via Drag and Drop. On the plus side, this makes it easy for non-technical users to interact but on the negative side drag and drop can become cumbersome especially if you’re relying it on to repeatedly add items.

Last week, after adding an alternative way to add a question and block I was in the lookout for making a convenient way for adding Options. Luckily, after multiple conversations with a friend and some inspiration I decided to use a tag like component where users could add options in one go. Since, the order of options is important, I also wanted the options to be order able. Lastly, since most of options in a typical survey would be re-used, especially the ones that are a part of Likert scale, the ability to autosuggest would keep the errors to a minimum and save on typing.

After failing to find a React component that could satisfy the above requirement, I decided that it would be a good idea to build one myself and then open-source in the community so that others could also benefit. Apart from deriving satisfaction from dog-fooding a component in my project, the idea of publishing a complete, useable react component in the wild sounded too interesting to let go.

Overall the entire exercise was a lot of fun and I definitely learnt a lot. The fact that I was able to build a working prototype is less than 150 lines of code (initially) speaks volumes about how powerful React is. The paradigm of thinking in terms of re-rendering the entire DOM makes it extremely easy for a developer to reason about the UI and the enclosed state. It is definitely one of the most exciting JS libraries I’ve ever worked with.

Not all is rosy though. When you are using React, you are inundated with lots of jargon and tools that form a vital part of the React ecosystem such as ES6, Webpack, Immutability etc. Initially it does feel that its a lot to take in. Hence, the toughest part for me was following the best practices on how to build a React component that could be used easily via CommonJS and also the browser (via a simple script tag). Thankfully, libraries like ReactDND helped me a lot and I was finally able to a come up with a satisfactory dev and build system in Webpack.

A quick demo of the component can be viewed here and the code is available on Github for you to fork and point out issues!

My plan for this week is to integrate this component with Surveyman and finish out the CRUD functionalities. Unfortunately, a key project at my workplace is sucking in all my time resulting in drowsy days and sleep deprived nights so I’ll have to balance my time really well to stay productive on open-source this week. Let’s hope I am able to meet this milestone!

Week 3 - Surveyman

Sun, 10 May 2015 15:50:49 +0300

This update is coming in a bit late, especially after the week 1 and week 2 updates posted earlier. After progressing a bit on the initial setup and features I decided to take a week off Surveyman and instead wait for the GSOC results to arrive so that I could get feedback from Emma, my mentor for this project and see if I were on the right track.

Much to my delight I got selected in GSOC to work for Surveyman! To top it all, I was the first ever student to represent Kuwait in Google Summer of Code!

Kuwait's first time in GSOC!

Starting last week, I decided to organize tasks as Github issues so it’s easier to track progress. Having specific issues for each milestone helps me write down ideas to revisit later - this could be blog posts, github repos or other random tidbits that strike my head. Secondly, it allows me to earmark commits for that specific issue which would be useful later on. See the issue on Immutability for example.

With that out of the way, let me quickly summarize the progress I made on the project. Specifically, I knocked out issues #14, #13 and #6 which primarily deal with adding the update and delete operations for all types. What that means is that Blocks and Questions can now be edited and deleted once they are created.

The only challenging part, I felt, was factoring in all these controls in the UI without adding any visual clutter. Filling up the blocks / questions area with various icons didn’t feel like a good solution. As a result I spent some time talking to friends and looking for inspiration in other applications. Although I did go ahead with a simple Trash icon for delete, the solution for editing question text turned out to be much more interesting.

In-place editing and deletion for Questions!

I started off working on HTML5 ContentEditable implementation but after diving deeper and looking at some of the issues I decided to just replace the <span> tag with an input when clicked. With some React awesomeness, I was able to build this in just a few lines of code -

// file - js/components/Question.js

render() {
    // ...
    var input = <input type="text"
            defaultValue={this.props.qtext}
            style={{width: 200}}
            onBlur={this.handleEdit}
            onKeyPress={this.handleEdit} />;
            
    <div className="qtext-area">
        <span className="qtext" onClick={this.toggleInput}>
            { this.state.editing ? input : this.props.qtext }
        </span>
        <i className="ion-trash-b" onClick={this.handleDelete}></i>
    </div>
}, 
toggleInput() {
    this.setState({ editing: true });
},
handleEdit(e) {
    if (e.type === "blur" || 
       (e.type === "keypress" && e.key === "Enter")) {

        this.setState({ editing: false });

        // if the new value is blank do nothing
        if (e.target.value.trim() === "") {
            return; 
        }
        SurveyActions.saveEditText(e.target.value, this.props.id);
    }
}

For a more complete breakdown, feel free to the browse the issues linked above and view the source code.

If you’ve been following closely, then you might have noticed that Options are still not editable. Well, the short answer is that while looking around for ways to factor in edit / delete in the small-sized element I came across a neat idea which eventually lead into me building a standalone react component. For the long answer, wait till next week!

Week 2 - Surveyman

Mon, 13 Apr 2015 06:58:24 +0300

After the drop targets and drag sources were configured last week, this week my primary motive was to develop a better UI for adding Question types. After mulling over the UI a bit, I decided to replace the existing Javascript prompt with a modal which would allow text entry and multiple radio buttons / checkboxes to configure various parts of a Question type.

Adding a Bootstrap Modal

UI

The key idea behind Surveyman is that by giving survey authors a way to write their surveys that steers them away from unnecessary ordering constraints, we can apply static analysis, randomization, and statistical dynamic analysis to locate survey errors and ensure the quality of responses.

The extra configuration parameters allow survey authors to modify the behavior of Surveyman. For example, if you are adding a question where you don’t want your options to be displayed in a random order, checking the ordered checkbox ensures that Surveyman shows the options in the order in which they were added.

The other half of this is displaying these parameters in a clear way to the users. For this, I decided on using the combo of meaningful icons and tooltips to communicate this information.

Icons for showing configuration

Architecture

To build this, I started off with migrating from Skeleton to Bootstrap so that I could use the React-Bootstrap library. The motivation here was to save time by using a set of prebuilt components that can be used right off the shelf. Apart from the modal, I knew that I’d be needing an Alert and Tooltip very soon.

With that done, the next part was laying down the architecture for Modals. The key idea here was to setup an action that would be called when the question is dropped on a Block. This would then cause the store to change a piece of data, which in our case would be the isOpen property on the modal. The triggering of this change would then finally re-render the views.

All this sounds simple, especially once you have a look at the sample code for custom triggering a modal. But things start to get interesting once you start building the form inside the modal. How do you tag DOM elements so that you can retrieve their value when an event is called? Use refs silly!

<div className="form-group">
   <label htmlFor="qtext">Question Text</label>
   <input type="text" placeholder="What is value of 4 + 5?" 
      className="form-control" id="qtext" ref="qtext" />
</div>

So I built a few form elements and tagged with them using refs and when I ran the application I got a cryptic error

Uncaught Error: Invariant Violation: addComponentAsRefTo(...): Only a ReactOwner can have refs.

You can go dive deeper into the issue but the main idea here was with the way renderOverlay was rendering the DOM in a different DOM Tree. As a fix, I had to create a BaseModal component that would then rendered inside a renderOverlay.

The final component looked like this -

var BaseModal = React.createClass({
   // custom handler code
   render() {
      return <Modal />
   }
});

var QuestionModal = React.createClass({
   mixins: [OverlayMixin],
   render() {
      return <div className="static-modal"></div>
   },
   renderOverlay() {
      return (
         this.props.open ? <BaseModal parentId={this.props.parentID} /> 
                         : <div></div>
      )
   }
});

The last step was simply rendering the QuestionModal in the Application component.

// in Application component
render() {
   <QuestionModal
   isOpen={modalState.question}
   parentID={this.state.dropTargetID}/>
}

The other parts of the application were quite straightforward and got done rather quickly. I did get stuck with building a self-dismissing alert box but the React IRC channel helped out quite a bit.

If you’ve read this far, go ahead and take the app for a spin. For the complete code, feel free to browse the Github repository.

This week my plan is to finish off in-place editing Block, Options and Questions. Stay tuned for the next post!