-
-
Us!
Inspiration
LLMs writing abilities have been widely publicized for almost 2 years now. For many peoples its use-cases haven't looked further than writing examples, editing, or image generation. This inspired us to look beyond and explore an LLMs basic nature by testing its internal world knowledge and ability to predict the next token. And, what world is better for this than Minecraft!
What it does
BlockBot is a Minecraft bot/agent that is programmed with the bare commands that we think are necessary to progress in the game. It is then able to take a complex task input that a user can feed in and break it into multiple sections in multiple simple tasks. For example, the task for mining diamonds is broken down into getting a pickaxe, finding, traveling, and then mining; while getting the pickaxe is further broken into getting the resources to build that.
How we built it
BlockBot is an LLM powered agent. It sends the complex user request to an OpenAI LLM which is responsible for breaking up the task into multiple components. The overall LLM is layered so that at the final layer, the task is now only consistent of the simple commands that the Minecraft bot itself can understand and perform.
Challenges we ran into
In doing this task, we had programming language differences as the Minecraft bot works best in javascript while the LLM that we program will be in Python. To address this we made it so that all communication between the LLM and the bot is done through REST API calls for the functions. This took a little learning but was overall a learning experience.
Further, another issue we had was considering what were the most fundemental commands that we wanted the LLM to break tasks into. This was due to the fact that we had really precise control over the bot with a robust collection of commands and information. However, to make a simple command like collect stone resources, it involves the combination of a few of these precise commands like path navigation, start mining and stop mining. On the other hand, even if we only address obvious commands like mining, placing, attacking, there are commands that are more subtle like with crafting the need to locate, path find, and then finally craft to do something that seems very simple.
Accomplishments that we're proud of
I am proud to have built a bot that could understand arbitrary commands even with some limited understandability cases. For most bots, people get frustrated as commands need to follow a strict set of guidelines so that the bot can perform simple tasks. This could include adding specific characters, spelling words correctly, or following a specific command format to make the bot do a certain command. Sure this was necessary to debug during the bot's construction but it was much more relieving to be able to start being more loose with the structure of how to communicate with the bot.
What we learned
This project was a great learning experience across a variety of technical problems. Here are a few:
- Porting python logic into javascript commands used by the agent with REST.api
- Learning javascript features as well as getting comfortable with the Mineflayer package
- Determine a base set of instructions is not easy and requires serious Minecraft dedication and problem solving.
- Going into this project we both had limited experience tuning LLMs and dealing with their hallucinations.
What's next for BlockBot
We didn't have enough time to fulfill our original goal of a task being subdivided into multiple Minecraft agents. We were hoping that we could make the controls intuitive enough so that the LLM could subdivide an even more complicated task into parallelizable parts where different agents could perform different parts of the same task according to the LLM. For example collecting more wood and stone to build the house could be a separation of collect house resources. Next, we would do a few quick bug fixes for some instances of the bot component so that it performs more robustly to simple commands and train the LLM to work with multiple agents.
Log in or sign up for Devpost to join the conversation.