DataDoor

Opening the door to accessible data, four datasets at a time.

Comment

Inspiration

Michelle publishes data on the California Open Data portal as part of her work. She realized that there was a lot of great data on the data portal, but it was hard for people to find out about this data to explore it. There is a need for a tool that empowers people to discover data from a variety of sources.

What it does

DataDoor is a search engine that compiles government data from the municipal, county, state, and federal level that displays the most relevant databases in a user-friendly UI.

How we built it

Michelle worked on the frontend in Adobe XD, web development using Flask, search algorithm, and zipcode geocoding. Abrar worked on the backend webscraping using Scrapy. All the code is in Python.

Challenges we ran into

Search engines could be slow, different websites had different HTML code and structure that hindered generalizability and automation of webscraping, and some websites were not suitable for webscraping using Scrapy.

Accomplishments that we're proud of

We are proud that we learned many new tools, libraries, and algorithms for HackDavis and developed a functional end product in our first hackathon.

What we learned

We learned a lot. As neither of us are CS majors (civil engineer and biomed engineer), basically everything we did, from the web scraping, to the web development, is brand new to us! Thank you to the mentors for all your help throughout the weekend.

What's next for DataDoor

DataDoor can improve performance by developing a general Scrapy code and more powerful search algorithm to produce faster, more relevant results. We can implement more of the functionalities of the prototype, including the ability to filter by year and query more results.

Built With

Share this project:

Updates