TABs & TAGS

As engineering students, we are on the computer 24/7. We are always busy, and there are many tedious tasks that we need to do every day that could be automated to save us time. Because of this, we wanted to create an application or software that allows us to easily automate menial tasks without requiring much computer science knowledge. We have really outdone ourselves here. Our solution to simple and understandable custom computer automation is the use of block-like modules, similar to Scratch. However, instead of running computer code, it runs simple and universal actions on the computer such as moving the cursor, clicking, typing, opening URLs, and more. It also contains essential coding blocks such as loops. Later on, we integrated Gemini into our user interface so the process of creating custom macros for tasks is even simpler, as Gemini essentially turns the prompt you give into blocks that you can run directly. Because of the time constraints, we did not have enough time to create a GUI from Java from scratch, so instead we used Blockly, which provides ready-to-use programmable blocks—exactly what we needed. We also utilized the Java AWT Robot library to simulate user keystrokes and mouse input. We ran into a major challenge where Blockly exports its instructions in a pseudo-code format using JSON, and in order for us to use this output, we needed to parse and interpret the pseudo-code into Java. We solved this issue by creating a digital “wire” between the locally hosted webpage and our Java backend. After receiving the pseudo-code in the correct order, we parsed it to match fundamental user actions and queued those actions, and once the user chooses to execute the block code, the stored instructions in Java are executed to achieve the desired macro. We later embedded a Gemini chat directly into the website, and using the Gemini API, we are able to take user prompts directly from our locally hosted website and break them down into a sequence of fundamental user inputs in JSON format so they can be run by our Java program, essentially skipping the need for manually placing code blocks. We are very happy with how all the pieces came together and with our ability to overcome constraints and navigate the obstacles we encountered, and we learned an immense amount about software development as a whole as well as the libraries and resources available to us. The future for TABs and TAGs is limitless, and even with the limited functionalities we implemented, we are already able to automate many tasks easily, such as taking attendance, generating messages, or even annoying your friends by spamming them, and additional functionalities can be added and tailored to specific needs, such as Excel data entry capabilities.

Built With

• css
• html
• java
• javascript