Dread Director Devpost

Inspiration

Most games are tuned for an average player who doesn't really exist. Difficulty settings and adaptive AI react to how well you play, but not to how you feel while you play. You see it in any experience meant to move you: a moment that grips one person does nothing for the next, and the designer is left guessing where the emotional beats should land.

At the same time, reading someone's physical state got a lot easier. You can now estimate heart rate, heart rate variability, and breathing from a normal webcam, with no wearable. The catch is that this only gives you numbers. There was no simple way for a developer to go from "the player's heart rate is 92" to "so change what the game is doing." We wanted to build that missing piece and package it so any developer can drop it in.

What it does

Dread Developer lets a game sense how a player is feeling and respond in real time. A small sensor unit, a Raspberry Pi with a camera and a contact pulse sensor, reads the player. Our system turns the noisy raw signals into three values that are easy to work with: noise, sustained stress, and composure. A live "director" then decides what should happen next, whether that is building tension, easing off when the player is overwhelmed, or giving them room to recover, narrated with ElevenLabs and written with GeminiAPI as a response to the user in real time. Players will not be tortured! Not to worry, their perserverence will be rewarded via Solana on Unifold in the form of credits.

Developers never touch raw biometrics or signal processing. They receive high-level cues (escalate, panic, recover) and connect their own reactions to them, such as enemy behavior, music, lighting, or pacing.

Our reference build, Night Watch, is a demo where a stalking creature hunts harder when you stay calm and backs off when you panic, with a survival contract that rewards keeping your composure. It runs the full loop from camera to gameplay. It is only one example, though. Nothing in the framework is tied to that style of game, or to games at all.

The same three signals and the same sense, decide, and adapt loop work anywhere software could use a read on how someone feels. A few concrete examples:

Sales and client calls: a live read on how engaged or tense the other person is, so the rep knows when to ease off or dig in. Best as a consented, both-sides tool.
Coaching and telehealth: it flags when a speaker, or a patient on a video visit, is getting overwhelmed, even when they say they are fine.
Learning and tutoring: the lesson slows down or re-explains when a student is clearly confused or checked out.

How we built it

We split the system into tiers so the sensitive data stays isolated:

Sensor unit (Raspberry Pi with a camera and a contact pulse sensor): the camera estimates breathing and heart rate from the face, and the pulse sensor reads the heartbeat directly. Reading both on one small board keeps all the raw data local and gives us a clean reference to check the camera against.
Decision core (portable, deterministic C++17 "Director"): fuses the camera and pulse data, gates it by confidence, turns it into normalized cues, and makes the pacing calls. It has no engine, vendor, or OS-specific headers, so it can be reused elsewhere.
Narrator (Gemini API, ElevenLabs Text to Speech): using a microphone module, the user's verbal reactions to the game are recorded and responded to to increase the atmosphere of the game.
CV to gain information about the person (Presage Tools): Assessing the user's breathing, blinking, and expressions to give them levels of fear.
Game (Unity 6, URP): receives only high-level JSON events over local UDP and turns them into gameplay.
Interactive weapon (ESP32 + MPU6050): which the user can use to hit the monster with your own hands.

Our basic stack was building most of our features on the computer (e.g. CV, Gemini) as separate tests and slowly integrating them into the game as the Unity app was developed. This allowed us to also slowly add some hardware to the game in the form of the weapon, pulse sensor, and camera.

Challenges we ran into

As nobody in our group had ever used Unity before, this was our first experience with game development. There was a lot of work put into learning how to model the player, the functions, and the environment of the game.
Using the Raspberry Pi with QNX OS was an extreme learning curve for us as we did not have much experience with Raspberry Pi before. Using a different OS required us to problem-solve a LOT and constantly adapt to these solutions.
Maintaining the github and accessing different parts of the code across the team. Managing our dependencies and our pushes across the team would have prevented the loss of a lot of our game through pushes.
Raw biosignals are noisy and slow to trust. HRV is not reliable for about the first 60 seconds, breathing needs around 30, etc. Turning that into something a developer can use without a physiology background, while staying honest about when we do not know yet was difficult to pace in the horror environment.
Handling different hardware. Cameras, drivers, and lighting all behave differently, so a lot of the effort went into making the output consistent no matter the setup underneath.
Tuning the feel. As we learned through many cringy dialogues, prompt engineering with Gemini requires extremely specific commands. the AI does not know the definition of scary. Balancing the monster, the pacing, and even the lighting so the experience stays tense but readable took a lot of iteration.

Accomplishments that we're proud of

Developing an interactive Unity game (interacting with both hardware and other features) with no initial experience.
Problem-solving over time in a lot of our hardware challenges. Even though QNX was hard to integrate, it was still solved through a step-by-step process that we increased despite being novices in hardware.
Being able to implement the vast majority of the features we planned by segmenting the work. The weapon was an afterthought initially.

What we learned

To ALWAYS have a precise plan in hardware hacks and then have a backup plan for those plans. Sometimes you won't get a specific module. Sometimes it will break (both happened). You need to have a pivot point!
Maintain your github precisely! Merge conflicts and general architecture matters. Do not just work on different branches and plan to integrate at the very end (we lost a lot of game this way :( )
An insane amount about Raspberry Pi architecture and how to develop with a different OS.
An insane amount about game development through Unity. The use of 3D assets, player control, and game functions.
Hardware to software real-time interaction and programming this.

What's next for Dread Developer

To become more immersive through AR and sound. Interacting with the human environment and it affecting your game would place you exactly where you need to be in the game.
Expanding to other sectors (education, health, sales. etc) with these tools to create better user experiences for everyone.