Inspiration
As a team, we were inspired to challenge the foundations of our understanding regarding log-based analysis, and aspired to consolidate our approach & skills concerning this field. We were aware of the valuable information that can be extracted from logs, including key data than can aid troubleshooting & optimisation. Thus, we decided to develop our own environment for data analysis, implementing techniques and a pipeline that piqued our curiosity.
What it does
Our product (aptly named 'Bingle'), is software designed to streamline customisable data analysis, an interactive canvas hosted via a web browser that serves as host for any number of scripts / plugins that an end-user may need for data extraction. This canvas supplies an easily digestible interface, allowing for the systematic creation of additional modules and dependencies, and contains parallel processing to support a robust job environment.
How we built it
Using a combination of a traditional REST API and a webSocketServer, we created a network of connections between our back-end server and our clients' devices that enables each to visualise their own real time stream of data. Furthermore, in order to allow for parallel processing, we implemented 'web workers' (akin to threading), in tandem with a queue data structure for storing jobs. These workers were given event listeners such that when a job was completed, the newly-available worker would proceed to perform the next job in the queue. Last but not least, we incorporated a proprietary JavaScript utility to help style our environment.
Challenges we ran into
The primary issue faced by the team was the difficulty regarding networking; debugging connectional faults proved to be a frustrating task while also being a significant bottleneck for the rest of our development, temporarily stunting progress.
Accomplishments that we're proud of
We feel proud that we were able to ultimately accomplish the goals set up by our sponsor challenge, while providing a framework that presents the opportunity for the project to be expounded upon beyond the hackathon. Our system, while requiring a long & arduous development, is modular and has an infinity of applications.
What we learned
The team gained a plethora of experience in JavaScript & node.js, while also consolidating our knowledge on parallel processing. Furthermore, we acquired fundamental experience when working with a team containing skewed levels of experience, improving communication and work delegation.
What's next for Bingle
Due to the modular nature of the project, the framework for Bingle already represents a robust data analysis solution. In future, we hope to clean up the UI and implement a node-based programming system as opposed to requiring users to write JavaScript code directly into the website.
Log in or sign up for Devpost to join the conversation.