×

86
87

ShowcaseI built a civic transparency platform with FastAPI that aggregates 40+ government APIs (self.Python)

submitted by Prestigious-Wrap2341

What My Project Does:

WeThePeople is a FastAPI application that pulls data from 40+ public government APIs to track corporate lobbying, government contracts, congressional stock trades, enforcement actions, and campaign donations across 9 economic sectors. It serves 3 web frontends and a mobile app from a single backend.

Target Audience:

Journalists, researchers, and citizens who want to understand corporate influence on government. Also useful as a reference for anyone building a multi-connector API aggregation platform in Python.

How Python Relates:

The entire backend is Python. FastAPI, SQLAlchemy, and 36 API connectors that each wrap a different government data source.

The dialect compatibility layer (utils/db_compat.py) abstracts SQLite, PostgreSQL, and Oracle differences behind helper functions for date arithmetic, string aggregation, and pagination. The same queries run on all three without changes.

The circuit breaker (services/circuit_breaker.py) is a thread-safe implementation that auto-disables failing external APIs after N consecutive failures, with half-open probe recovery.

The job scheduler uses file-lock based execution to prevent SQLite write conflicts across 35+ automated sync jobs running on different intervals (24h, 48h, 72h, weekly).

All 36 API connectors follow the same pattern. Each wraps a government API (Senate LDA, USASpending, FEC, Congress.gov, SEC EDGAR, Federal Register, OpenFDA, EPA, FARA, and more) with retry logic, caching, and circuit breaker integration.

The claims verification pipeline extracts assertions from text and matches them against 9 data sources using a multi-matcher architecture.

Runs on a $4 monthly Hetzner ARM server. 4.1GB SQLite database in WAL mode. Let's Encrypt TLS via certbot.

Source code: github.com/Obelus-Labs-LLC/WeThePeople

Live: wethepeopleforus.com

all 28 comments
sorted by:
best
topnewcontroversialoldq&a

[–]AutoModerator[M] [score hidden] stickied comment (0 children)

Hi there, from the /r/Python mods.

We want to emphasize that while security-centric programs are fun project spaces to explore we do not recommend that they be treated as a security solution unless they’ve been audited by a third party, security professional and the audit is visible for review.

Security is not easy. And making project to learn how to manage it is a great idea to learn about the complexity of this world. That said, there’s a difference between exploring and learning about a topic space, and trusting that a product is secure for sensitive materials in the face of adversaries.

We hope you enjoy projects like these from a safety conscious perspective.

Warm regards and all the best for your future Pythoneering,

/r/Python moderator team

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[–]datadidit 7 points8 points  (3 children)

Pretty cool! What you end up using for the frontend 

[–]JoeHillsBonesIt works on my machine 5 points6 points  (1 child)

How do you deal with all the different APIs? Are they standardized in some way?

More comments

More from r/Python

  Hide

Comments, continued...

[–]No_Lingonberry1201pip needs updating 4 points5 points  (2 children)

Pretty cool stuff! Are you using Hetzner because it's not a US company?

[–]Prestigious-Wrap2341[S] 2 points3 points  (1 child)

No, not just that. I mainly use it because it's cheap. I'm only spending $4 for an ARM server that can do everything I need. I couldn't really find any US providers that had anything close to that price. Oracle Cloud did have free ARM instances, but the capacity was really hard to get, and Hertzner just worked on the first try. My server is in Nuremberg, so there's some latency to US users, but Vercel's edge CDN handles the front end, so the only cross-Atlantic hop is API calls, and those are cached. So it all kind of just worked out for me.

[–]No_Lingonberry1201pip needs updating 0 points1 point  (0 children)

That sounds awesome. I'm also with Hetzer, I like that they are low-key and cheap.

[–]swift-sentinel 1 point2 points  (0 children)

Very cool! I'll take a look as see if I can help. Perhaps this can expand to state governments.

[–]gwood113 1 point2 points  (3 children)

Your live version incorrectly identifies South Carolina zip codes as a Georgian one.

[–]Prestigious-Wrap2341[S] 2 points3 points  (0 children)

I’m gonna fix that right now, thank you!

[–]Prestigious-Wrap2341[S] 1 point2 points  (1 child)

I fixed it, thanks again!

[–]gwood113 1 point2 points  (0 children)

Quick work! Just an aside, house.gov lets you use look up your zip to see which representative is yours based on where you live.

The url is https://ziplook.house.gov/htbin/findrep_house?ZIP= I haven't tried curling it but it seems to have a standard format that I believe would lend itself to parsing.

That would let you tag reps the way you tag senators.

Love the app overall!

[–]Busy_Network_7167 3 points4 points  (1 child)

This is brilliant work - running 40+ government APIs through a single FastAPI backend on just 4 quid a month is proper impressive. Love seeing SQLite getting used for something this substantial, especially with WAL mode handling all those concurrent sync jobs

Your circuit breaker implementation caught my eye since I've been dealing with flaky external APIs at work lately. Having it auto-disable failing endpoints with probe recovery is exactly what I need to steal for my own projects

The dialect compatibility layer is clever too - being able to swap between SQLite, Postgres and Oracle without touching queries saves so much headache down the road. How's performance been with that 4.1GB database on ARM?

[–]Prestigious-Wrap2341[S] 3 points4 points  (0 children)

Honestly, the $4 server is overkill for the current traffic that it gets. The database sits on NVMe, so it reads pretty fast. WAL mode handles the concurrent reads and writes without issue. I haven't had a problem there. The biggest bottleneck is that the sync jobs hit 40+ external APIs, not the database itself. The SQLite single writer limitation hasn't been a problem, though, because the scheduler runs jobs sequentially through a file lock.

[–]double 0 points1 point  (1 child)

Why did you move the LLM prompts to env vars? Interesting choice.

Which ai tools are you using?

[–]Prestigious-Wrap2341[S] 1 point2 points  (0 children)

The prompts were originally inline in the source code. I moved them to env vars partly for security (keeping proprietary logic out of the public repo) and partly for operational flexibility. If I need to tune a prompt, I update the env on the server without redeploying code. The platform uses Claude Haiku for story generation, claim extraction, and AI summaries. It's cheap enough for bulk work, about $0.003 per story, running 20+ stories a day.

[–]Actual__Wizard 0 points1 point  (0 children)

Neat project!

[–]lordbrocktree1 0 points1 point  (5 children)

Asking for contributions while not being open source is an interesting choice. Any reason to not do FOSS or even an open source tier with premium features? If you are asking for community contributions seems like a fair trade but idk

[–]Prestigious-Wrap2341[S] 0 points1 point  (4 children)

It is essentially 99% open source. The only things I did was remove my prompts to an env and I put the extended use of my verification pipeline behind an enterprise tier. The rest of the platform is free for anybody to use without signing in.

And the only reason I moved the prompts were because to be quite honest I’m deathly afraid of somebody else taking this doing it better, monetizing it and leaving me with nothing

[–]lordbrocktree1 0 points1 point  (3 children)

Recommend changing the licensing on your readme to reflect that then! Makes a lot of sense!

[–]Prestigious-Wrap2341[S] 1 point2 points  (2 children)

I’m at my actual job right now but as soon as I get home, that’ll be the first thing I change. Thank you for that.

[–]lordbrocktree1 0 points1 point  (1 child)

Awesome! I actually do some similar work, and am considering contributing!

[–]Prestigious-Wrap2341[S] 0 points1 point  (0 children)

I’ve never had contributors before, in fact, I’ve never had anybody help me with any of this! This is my first open source project or at least my main one I kind of built a few of them in parallel. Anyway, you can contribute or help. I’d appreciate it even if it’s just testing endpoints or looking for bugs and errors, everything helps!

[–]Otherwise_Manner9148 0 points1 point  (0 children)

you should do this on a granular level for local governments down to the county and city level