Inspiration
Finding opportunities shouldn't feel like a full-time research job.
Hackathons, fellowships, scholarships, grants, competitions, and internships are scattered across the web. Platforms like Jobright, Scholly, and Devpost each focus on a particular category, meaning you often have to search multiple platforms separately and repeat the same process every time.
The problem isn't that opportunities don't exist. The problem is finding the opportunities that are actually relevant to you.
We wanted to build something that felt less like another directory and more like having a knowledgeable friend search on your behalf — someone who knows your background, understands what you're looking for, searches beyond the websites you already know, and comes back with a small list of opportunities with an actual reason for each recommendation.
That became ScoutDeck.
What it does
You start by creating a short profile with your skills, education, interests, location, remote preferences, and the types of opportunities you're interested in.
ScoutDeck then:
- Searches the live web for relevant opportunities instead of relying entirely on a static database.
- Scrapes the pages it finds and extracts the useful information from otherwise messy web pages.
- Uses AI to structure the information, extracting eligibility, required skills, deadlines, location, stipends, and other important details.
- Compares candidates against your profile and ranks them based on overall fit.
- Streams the process live so you can see the system searching, processing, and evaluating opportunities instead of waiting behind a generic loading screen.
- Returns up to 5 strong matches, each with a match score and a specific explanation of why it fits your profile.
We intentionally don't pad the results. If there are only three opportunities that genuinely fit, ScoutDeck shows three rather than pretending five are good matches.
If live search doesn't produce enough strong candidates, a small pre-vetted fallback pool helps prevent an empty result while keeping the experience consistent.
The goal is simple: help people spend less time searching and more time applying to opportunities that are actually relevant to them.
How we built it
Stack: Next.js, TypeScript, Supabase/Postgres, Tavily, Firecrawl, Zod, and multiple AI providers.
The core pipeline looks like this:
Profile
↓
Search query generation
↓
Tavily — live web search
↓
Firecrawl — page extraction
↓
AI — structured opportunity extraction
↓
AI — comparative ranking
↓
Server-Sent Events — live progress
↓
Supabase — persistence
One of our important technical decisions was treating AI output as untrusted data.
We use Zod to validate both user input and AI-generated structured data before it is used by the application. This helped us avoid assuming that an LLM would always return exactly the structure we expected.
We also designed the AI layer with fallbacks across providers. Groq handles the primary workloads, with Gemini and OpenRouter available as fallbacks when a provider becomes unavailable or rate-limited.
Challenges we ran into
The most valuable part of building ScoutDeck was discovering that getting a prototype to work locally is very different from making a multi-step application reliable.
1. Results were silently disappearing
At one point, ScoutDeck would sometimes return only one or two opportunities even though multiple candidates had successfully gone through earlier stages.
The problem was a candidate-matching bug. We were using full URLs as identifiers and expecting the AI to reproduce them exactly. Small changes to a URL could cause an otherwise valid result to be discarded.
We solved this by giving candidates short internal identifiers such as c0, c1, and c2, then resolving those IDs back to the original candidates ourselves.
This taught us an important lesson: when working with AI, don't make the model responsible for reproducing data that your application can identify deterministically.
2. API rate limits
Our first approach sent scraping and AI requests too aggressively.
That worked during small tests but quickly ran into provider rate limits. We had to learn about concurrency, request pacing, and why "send everything at once" isn't necessarily faster when external services have limits.
We added concurrency control and deliberate request pacing to make the pipeline more predictable.
3. Fallback errors were hiding the real problem
Our AI fallback initially wasn't giving us enough information about why a provider failed.
A Gemini failure could end up looking like a Groq failure because of how errors were being propagated.
We improved provider-specific logging so that when something failed, we could identify which provider failed and why instead of debugging from a generic error.
4. Local vs. production behavior
The pipeline could take significantly longer than expected when all of the search, scraping, extraction, and ranking steps were combined.
Locally, this was easy to overlook. In production, serverless execution limits made it a real constraint.
This forced us to start thinking about execution time as part of the architecture rather than something to optimize after everything else was finished.
5. Debugging through the pipeline
One of our biggest improvements was adding stage-by-stage logging.
Instead of asking "why isn't ScoutDeck working?", we could ask:
How many search results did we get?
↓
How many pages scraped successfully?
↓
How many candidates were extracted?
↓
How many survived validation?
↓
How many were returned by ranking?
That changed debugging from guessing into investigation.
Accomplishments we're proud of
- Built a working end-to-end opportunity discovery pipeline using live web search.
- Connected search, scraping, AI extraction, ranking, streaming, and persistence into one workflow.
- Added structured validation for AI-generated data using Zod.
- Implemented AI provider fallbacks for greater resilience.
- Added concurrency control and request pacing to work within real API limits.
- Built live progress updates using Server-Sent Events.
- Designed ScoutDeck to return fewer results rather than padding recommendations with poor matches.
- Created a visual identity around the idea of an opportunity atlas rather than a generic SaaS dashboard.
What we learned
The biggest lesson was that building an AI application is much more than calling an AI API.
We learned how easily silent failures can happen when multiple services are connected together, and how important structured validation and logging become once there are several stages in a pipeline.
We also learned that something working on a local machine doesn't mean it will behave the same way in production. Rate limits, execution limits, network issues, and external API failures all become part of the engineering problem.
Most importantly, we learned to debug systematically.
Instead of immediately changing code when something broke, we started measuring each stage, identifying exactly where the expected data disappeared, and then fixing the underlying problem.
That was probably the biggest improvement in our engineering process during the project.
What's next for ScoutDeck
We want ScoutDeck to become more useful beyond the initial discovery step.
Our next ideas include:
- Application tracking and deadline reminders so users don't lose opportunities after discovering them.
- Personalized gap analysis showing what skills or experience could make someone more competitive for an opportunity.
- Preparation roadmaps for saved opportunities.
- Crowdsourced opportunity submissions to surface opportunities that automated search may miss.
- Stronger explanations that can explain not only why an opportunity matches, but why it may be a better choice than similar options.
Ultimately, we want ScoutDeck to make opportunity discovery less dependent on already knowing the right websites, having a large network, or spending hours searching.
Built With
- gemini-api
- groq
- nextjs
- openrouter
- supabase
- supabse-auth
- tailwindcss
Log in or sign up for Devpost to join the conversation.