Among-AI

Inspiration

The inspiration for Among-AI comes from the need for effective, large-scale evaluation of Language Learning Models (LLMs). Observing the popularity and engagement levels of online games, we decided to gamify the process, inspired by the hit game "Among Us".

What it does

Among-AI is a Discord game where participants play rounds of chat with each other, trying to blend in as humans while avoiding being recognized as a language model. In each round, players write a sentence within a 20-token limit. Players receive other participants' responses first, which guides them to match their wording style, token length, and content to evade detection.

How we built it

We built Among-AI using the discord.py framework. We incorporated various LLMs into the game, making them compete against human players in generating indistinguishable responses.

Challenges we ran into

Designing a compelling and fair gameplay that also serves our data collection goals was a significant challenge. We needed to ensure that the game remained engaging while providing valuable data for LLM evaluation.

Accomplishments that we're proud of

We're proud of creating a platform that simultaneously entertains and aids in the development and evaluation of AI. We've effectively gamified a critical aspect of AI research, which we believe is a unique and impactful accomplishment.

What we learned

We learned that crowd-sourcing can be an effective tool for AI evaluation, particularly when it is cleverly disguised as a fun and engaging game. We also learned how to balance game design with our data collection objectives.

What's next for Among-AI

We plan to expand Among-AI to include more complex tasks and introduce more varied language models. We also hope to gather a larger player base, which will provide us with even more valuable data for LLM evaluation.

Built With

• anthropic
• discord
• openai
• python