Inspiration
The inspiration for Pipes came from a staple feature of the Unix terminal -- the ability to pipe output from one command to the input of another. The piping infrastructure allows tedious and complex tasks to be expressed in a way that is simple, logical, and very powerful. We set out to apply the same concept to the web and its diverse set of APIs.
What it does
Pipes condenses complex actions into simple, yet robust, building blocks from which larger "flows" can be assembled. Acting on some kind of event from an input node or static text node, the data travels through an unlimited number of transform streams to an eventual output node or "sink." Unlike most Unix shells, piping allows for types other than strings to be passed from node to node and for multiple input and output streams, each with a particular identifier. Integrated with a custom language processing engine on top of Google's Natural Language APIs, Pipes can be used by beginners and experts alike in an intuitive manner that distances itself from its Unix shell inspiration. Even so, above all, we wanted to design Pipes for the extensibility and customizability people familiar with Unix-like operating systems have grown accustomed to. Input, transform, and output nodes (AKA plugins) can be composed simply in just a few lines of TypeScript or JavaScript.
How we built it
We built Pipes in TypeScript on the Node.js platform. Nedb served as the embedded database that we used to keep track of instantiated nodes and their instance variables and save and retrieve prerecorded graph actions. Google's Cloud Natural Language API was used as the basis for our language processing engine. The codebase is highly modular in nature with a core that handles the interconnections between nodes and plugins that provide the nodes, actions, and the necessary information about how they interconnect. With this structure, Pipes can be adapted easily to add, update, or remove APIs with just a simple restart of the server.
Challenges we ran into
Creating an interface that was accessible yet highly flexible proved a great challenge at first. Initially, all actions were configured in pure JSON that was difficult even for us to keep track of. We searched for a while to find a library suitable of displaying and updating the generated node graphs in the way that we wanted before finally settling on vis.js.
Accomplishments that we're proud of
We're really proud of the integrated natural language features that generates the node graphs on the fly in near-real time as the command is changed in the input box. As a feature that made it into our final product fairly late in the process, it really helps to make Pipes more accessible to all while retaining the power of the pipeline metaphor.
What we learned
Michael learned some TypeScript
What's next for Pipes
In the future we hope to add or improve upon the following features
- Drag and dropping nodes
- Voice to text integration into our NLP
- User authentication (currently the system is designed to run on the end-user's machine)
- Extend Pipes' features for real-world use at our upcoming HackGT hackathon in October and beyond
Log in or sign up for Devpost to join the conversation.