As you know by now, our Open Data Portal is hard at work keeping San Diego’s data accessible and up-to-date. As we strive to keep San Diego at the forefront of SmartCity technology, more citizens and city officials continue to recognize the portal’s benefits.

Our site and team are even turning heads beyond city boundaries! Netlify, an online platform that easily allows users to deploy websites, recently crowned our Open Data Portal the “Netlify Site of the Week.” By using the hassle-free services Netlify offers, we saved the city from spending money on expensive website builders, and saved ourselves from hours of tedious work. We are proud to epitomize “innovation, creativity, and utility,” and join the other sites in Netlify’s hall of fame.

Check out Netlify’s article and the Open Data Portal to read more about how our team aims to make San Diego more efficient and informed.

Netlify Site of the Week was originally published in DataSD on Medium, where people are continuing the conversation by highlighting and responding to this story.

We recently completed a small sprint and made a few fixes and improvements.

We love listening to our users, and one of the most requested features has been a way to type in an address and see what streets have been or will be repaired around where you live or work.

We implemented an autocompleting search using Mapzen’s awesome search api. Now you can search for a place (like Balboa Park), or an address and it will zoom right in for you:

We also made a few performance tweaks by moving hosting from Github Pages to Amazon S3 and CloudFront to align with our overall infrastructure and deployment strategy. Hopefully sometime soon we’ll be able to make a few tweaks like we did on data.sandiego.gov to significantly increase the loading speed.

Finally, there is one more tweak we made, but we’re not ready to talk about it yet, because it needs a bigger piece before it goes into production. However, you can probably find it by looking at the open source code driving StreetsSD.

As always, we learn, iterate and continuously improve our products. We Love To Hear and Implement your suggestions!!

But we love pull requests even more!

A few updates to StreetsSD was originally published in DataSD on Medium, where people are continuing the conversation by highlighting and responding to this story.

The Energy & Sustainability Division within the Environmental Services Department has been upgrading City streetlights to SMART adaptive control LED lighting systems.

Our systems include adaptive control meters, which provide the City with real-time data, energy savings, and asset management and improve lighting throughout the City.

The map below displays each retrofitted streetlight in the City. Orange dots represent retrofitted induction lights; green dots represent LED lights with adaptive controls; and blue dots represent all other LED lights.

See the map below!

https://cityofsandiego.carto.com/u/andrellbower/viz/b0c3ff3f-4a61-4da7-84e3-0e7a134ef067/embed_map

City Streetlights Get Smart was originally published in DataSD on Medium, where people are continuing the conversation by highlighting and responding to this story.

The Data & Analytics program aims to empower both officials and residents with easy access to San Diego’s data. But if people don’t know about the portal, they won’t be able to build apps, hold the City accountable, or make decisions based on data.

That’s where I came in. As a summer intern focused on community engagement, I took on the task of bringing more attention to the portal. I’m excited to share my last project: a video championing the Open Data Portal. Using concrete examples and minimal technical jargon, I hope to make the portal both comprehensible and intriguing.

San Diego’s Open Data Portal Video was originally published in DataSD on Medium, where people are continuing the conversation by highlighting and responding to this story.

The 2017 GE Minds & Machines conference featured an “Appathon” where teams were challenged to come up with a working application in a day and a half. The challenge theme was all about decarbonization, decentralization, and digitization in the global energy sector. Each team was provided with a Predix Kit that included an Intel Nook and various sensors, sample data from power grid feeders in Ireland, and access to the CityIQ API for accessing the intelligent street lamps in the city of San Diego.

Our team took inspiration from Google’s “Project Sunroof,” which analyzes satellite imagery and 3D renderings of rooftops in various cities to calculate potential savings from solar energy. Our idea was to take this data for solar potential, subtract the existing installations extrapolated from the permit dataset on data.sandiego.gov, and use the city budget and other financial information to provide city planners with strategic information around solar subsidies.

We also built a regression model using Predix analytics and services to calculate the best way to balance residential, commercial, and industrial solar installations to arrive at the city’s goal of carbon neutrality by 2035. Our basic finding was that, even though commercial and industrial buildings generally have larger rooftop footprints, the most economical and scalable solution would be to target widespread residential installations. We then used income data for households in the area to estimate the relative affordability and required subsidies for various income brackets in San Diego.

Check it out!

• Devpost
• Running Application
• Github

Originally published at https://data.sandiego.gov/stories/solar-permits-prediction/.

Solar Permits Prediction was originally published in DataSD on Medium, where people are continuing the conversation by highlighting and responding to this story.

Arnaud was the City of San Diego’s first data scientist. He joined the Data & Analytics team in October 2016, and right away he was speaking at events, teaching data science and Python at meetups, and spending his free time with community nerds fiddling around with tech gadgets like LoRa. And, obviously, solving problems in the City using data and technology.

One time, he did this:

Arnaud Vedy on Twitter

NYC on my desk using #Google #ARCore and #pixelphone. #lunchtimeprojects #ar https://t.co/q3ud8TVTxf

We lost Arnaud in a motorcycle accident in early November. I was not at all prepared for how devastating it is to lose a co-worker, especially one you work closely with and is also a friend. He was smart and talented, but not condescending. I could, and did, ask many dumb questions about Python, GIS, gadgets, France, artisan pizza, and coffee. He answered my questions patiently and never made me feel stupid, or like I was wasting his time. We ate many lunches together, along with our boss Maksim, and chatted a lot about code, and life.

Data & Analytics is a 3-person team within a larger department of only 13, that we affectionately refer to as PandA (Performance & Analytics). Arnaud’s death was a blow to the entire department, and to people in various other departments who had worked with him, not just the two of us left in Data & Analytics. The City Council even adjourned a meeting in Arnaud’s memory.

Council member Chris Ward read about Arnaud’s life and his accomplishments with the City. At this point, I was done crying about it all the time, and it made me really happy to hear him remembered and have him memorialized in the official record of the City of San Diego. Well, I was almost done crying at that point.

Anyway, here’s the clip:

We also have this certificate hanging in the hall leading to our wing of the 8th floor of the City Administration Building.

You’ll see some posts on this blog that I published under my account after his death. He absolutely wanted you guys to learn all kinds of geeky stuff about GIS and data science, so I had to get these out into the world and share a bit of his passion and knowledge.

We talked all the time about how both of us started off in other disciplines (myself in journalism, him in Geography) and found our way to programming, data analysis, and data visualization. Our code was a little sloppy sometimes, but we were both excited to be doing something that we discovered such a passion for that we had spent our free time teaching ourselves the skills.

So, if you really want to channel Arnaud, wear this shirt. I proudly display a magnet version of that shirt in my not-at-all-dull cubicle.

Even better, build yourself a crazy powerful Linux computer that can handle Lidar data, spend your Saturdays coding, and always make maps with dark backgrounds (just kidding, channel me and put some thought into what your color choices communicate).

Remembering Arnaud Vedy was originally published in DataSD on Medium, where people are continuing the conversation by highlighting and responding to this story.

The question

How many times did someone report a pothole to the City through the Get It Done applications in 2017 for the Downtown area?

The tool

Although we could use various GIS tools to answer this question, here we are using Mapshaper. Why? Because you will be able to reproduce the workflow without having to install anything, and you’ll be surprised that an online tool can handle this sort of analysis.

The steps

1. Download data

To start with, download the following two datasets from our open data portal:

• Get It Done requests
• Community Planning District boundaries

2. Import data into Mapshaper

Go to Mapshaper and select/import the Get It Done dataset. First thing’s first, let’s rename the dataset to ‘gid’ for easier processing by opening the console (top-right button) and typing the following command: -rename-layers gid

At this point, this dataset is not interpreted as spatial data, and all you can see is a grid of points. To solve this issue, we are going to tell Mapshaper which columns of the .csv file contain spatial data by typing the following command into the console: -points x=long y=lat

If you’re curious about all the commands you can use in Mapshaper, check out this reference.

3. Filter data

Two elements of the question we are attempting to answer can be addressed by filtering the Get It Done data:

• ‘In 2017’ with the following command: -filter ‘requested_datetime > “2016–12–31” && requested_datetime < “2018–01–01”’
• ‘Requests for potholes’ with the following command: -filter “service_name == ‘Pothole’”

Now your screen should look like this:

4. Import more data into Mapshaper

To answer the last element of the question, ‘in the Downtown area’, we need to perform a spatial join between the neighborhoods from the Community Planning shapefile and the filtered Get It Done data.

To do so, first import the Community shapefile by clicking on ‘gid’ (top-center), then ‘add a file’, then ‘import’. Rename the layer to ‘neighborhoods’: -rename-layers neighborhoods target=cmty_plan_datasd

One last thing — reprojection. If you type -info in the console, you will notice that both spatial layers are using different coordinates systems. All you need to do here is to put both layers in the same system with the following command: -proj wgs84

5. Perform a spatial join

You are now all set to operate a spatial join between the two layers. With a spatial join, we will end up with a column that contains the neighborhood name from the ‘neighborhoods’ layer for each row of the ‘gid’ layer.

To perform the join, select the ‘gid’ layer from the layer switcher (top-center), and type the following line: -join neighborhoods fields=‘cpname’

If you now explore the gid data, you will notice an extra field at the bottom with the neighborhood information for each pothole:

6. Answer the question

You can now answer the initial question with a last command: -calc ‘count()’ where=“cpname == ‘Downtown’”

The findings

At the time of this writing, 814 potholes were reported with the Get It Done applications for the downtown area in 2017. Put that in your report or slide deck!

Do you have more questions? I did.

What about Uptown (1059), or Golden Hill (194), or La Jolla (725)? What is Uptown, anyway (Mission Hills + Hillcrest?)? Have the number of potholes reported per month citywide gone down (yes! See the table below).

For that last one, I had to run multiple calculations and use requested_datetime in the where clause. There might be a better way to do that, but I don’t know it.

What I do know, is that spatial data has all the answers.

Arnaud Vedy contributed to this article.

Using spatial data and Mapshaper to answer questions was originally published in DataSD on Medium, where people are continuing the conversation by highlighting and responding to this story.

The online way

The quickest way to open and visualize the content of a shapefile is to use a tool called Mapshaper.

Simply drag the freshly downloaded .zip file from our portal into the web interface, click ‘Import’ and visualize your data! By clicking on the ‘i’ at the top-right of the interface, you will be able to view the attributes of each feature as you hover it.

More adventurous / advanced users are encouraged to read the project’s wiki. Indeed, Mapshaper can do way more than open shapefiles. You can join, dissolve, reproject, simplify, and export as well as perform other essential GIS vector operations.

The desktop way

If you do not mind installing software on your computer, QGIS is the way to go. Thanks to its numerous plugins and toolboxes, this free and open source software (available for Windows / Mac / Linux) will cover most of your GIS needs. Opening one of our shapefiles is as easy as selecting it in the file explorer (left bar). Wondering what to do next, now that you have opened a file? Check out the QGIS training manual and QGIS user guide!

The Python way

If you are a developer or a data scientist, you are probably dealing with tabular data using Python. When it comes to spatial data, you can take advantage of plenty of Python packages to open, edit, and process shapefiles.

Here is a non-exhaustive list of packages:

• pyshp
• GDAL
• Shapely
• Fiona
• GeoPandas

This article was originally written by Arnaud Vedy

What to do with GIS files was originally published in DataSD on Medium, where people are continuing the conversation by highlighting and responding to this story.

Current geo data

It means you now have access to the most current geographic data at all times. Some data sources are updated every month. For others, this process occurs on a daily basis. No matter the schedule of the dataset, our portal will now automatically update along with it. This ensures that as a user of the portal, you will be retrieving the most up-to-date data available.

New formats

Until recently, we were sharing geo data exclusively as shapefiles — one of the most commonly used format for geo data. Nevertheless, there are a few reasons why it should not be the only format of geo data offered on an open data portal:

• It is not an open standard format.
• It is actually multiple files. These files need to be zipped, making it inconvenient to share.
• It is not a practical format for web developers. Few web mapping javascript libraries support shapefiles natively, which usually leads to data conversion.
• It is not the most practical format for non-technical users. It requires software installation to be visualized / used for data analysis.

For these reasons, we have decided to provide you with additional geo data formats. For now (and we will be adding more) they are: GeoJSON, TopoJSON, and Geobuf.

• GeoJSON: This is the most common alternative to shapefiles. GeoJSON is an open standard format based on the JSON specification. It is supported by most desktop and web based mapping/GIS solutions. Geo data in this format is also human-readable. This means it can be opened with a text editor, which make it very easy to use. The only drawback with GeoJSON when compared to shapefile is size. This is a consequence of GeoJSON’s lack of topology support.
• TopoJSON: This format is an extension of GeoJSON that encodes topology. As a result, TopoJSON files that contain line and polygon geometries are significantly smaller than both GeoJSON and shapefiles.
• Geobuf: Geobuf is a relatively new format introduced by Mapbox. Geobuf provides lossless compression of GeoJSON data into protocol buffers. It is fast to decode and very light, making it attractive for developers who want to use large geo datasets in their web applications. Here is a comparative table to demonstrate the compression power of the Geobuf specification on our data:

So, you now have access to the most current geo data in multiple formats and we are pretty excited about it. What will you do with it? Let us know at data@sandiego.gov and stay tuned for more announcements!

Originally written by Arnaud Vedy and published at https://data.sandiego.gov

Geo data: up-to-date and in multiple formats was originally published in DataSD on Medium, where people are continuing the conversation by highlighting and responding to this story.

The portal has a fresh look, but more importantly, we rebuilt the technology behind the portal and upgraded our workflow for keeping data up-to-date.

Just yesterday

We initially launched our data portal by buying a ready-made portal product from a company. As our team was newer, this made a lot of sense, and the portal the vendor made for us was exactly what we needed at that time.

But we’ve always had a long-term vision for open data that relied on high-tech strategies. We never wanted to publish data that was manually updated. We want City employees to spend their time using data to serve residents, not trying to remember to update data on an open data portal. We also want our data to be put to use helping the City perform better. This means the data needs to be accessible not only for simple downloads, but also for alerting, prediction, and other advanced analytics.

Ready-made portal products are more geared toward a manual update process, so when we started working to automate data updates late last year, our automation solution didn’t work as well with our portal product as we had hoped.

Normally, a client uploads a dataset to its cloud, and then that dataset is available for download, for preview in the browser, and through its data API.

But we were not hosting our data in the cloud service our portal provider offered. We uploaded to a different cloud service and then linked the files to the portal. Not using a direct upload limited our ability to show a preview in the browser or make the data API available without manual intervention. We also couldn’t trigger a change in the extremely important metadata field of “Date Updated” without a manual upload.

Although a ready-made portal product is convenient, it is also expensive. Adding functionality and features as we sought continuous improvement in our program would end up costing taxpayers money that could be better spent.

Taking all of these factors together, we decided to take San Diego’s open data program to another level and build a portal ourselves. We have a team with the technical expertise to run our own data portal, so we went for it!

Today

At a minimum, a data portal is just a website. All it really needs is a page with a list of datasets (the catalog) and individual pages for each dataset that contain links to download the data. A ready-made portal product is a website with an admin interface that makes it easy for just about anybody to manually update data.

The reality is that some open data programs do depend on employees to upload data manually. This is because open data became a thing long after government and technology were invented. In fact, doing open data well is very, very, very complex (did we say very?). Not all departments and divisions of a government use the same database technology. Government information is housed in hundreds of different systems or even on spreadsheets in shared drives. Some data even comes from third-parties that are contracted to provide services on behalf of a government, and therefore maintain data on their own servers.

Our team spent a significant amount of time writing code to build our automation process that pulls in data from our City’s various systems, transforms it, and puts it into the our cloud. We could have probably scripted a solution that would copy files from our cloud to the portal cloud. But it would have taken some work, and the effort would be hampered by the fact that we don’t have control over a product made by an outside company. As they upgraded their product, our script might have broken over, and over again.

A portal we created that worked perfectly with our automation process was the obvious solution. However, we can’t take ALL the credit for this new portal. We based our new portal on an open-source project by Tim Wisniewski, Chief Data Officer for the City of Philadelphia. The open-source portal he created, called JKAN, was instrumental in getting us up and running with our own version. (Thanks, Tim!!!)

Tomorrow+

Moving forward, this data portal is the new hub for all open data and analytics projects for the Performance & Analytics Department. We used to maintain a separate informational site, datasd.org, that we no longer need since we have full control over this one. Keep checking back for the latest news, how-tos, data stories, visualizations, and of course, the most up-to-date City data.

If you are intrigued by the work we’ve done, our portal is also open-source! Check out our code, copy it, and make your own data portal, aka website with a catalog. We use Jekyll, a static site generator, to manage all the files and pages of our portal.

You also need to check out our automation process. We are using Airflow, another open-source tool, originally developed to improve workflow for employees of companies that crunch data.

Basically, each little task that we need to complete when we extract and transform data is handled with bits of Python code. Examples of these tasks include connecting to a database, pulling a file from one of the City’s shared drives, detecting the current date and using that to grab the correct time interval of data, etc.

Airflow keeps all these bits organized into a hierarchy of jobs, and we get alerts if anything fails. We can also use Airflow to schedule automatic kick-off of jobs, which is one of the major headaches of automation. Keep an eye on this blog for more in-depth and technical explanations of how we are using Airflow.

If you are a person who was using the data API from our previous portal, you will instead have to work with the link to the csv. We also had to change how we stored data in our cloud, so please plan on updating any previous links to csv files by the end of March. You can get updated links by checking the dataset pages on this new portal.

We hope you love this new portal, and we especially hope you love having automatically updated data! Let us know what you think.

A better open data portal was originally published in DataSD on Medium, where people are continuing the conversation by highlighting and responding to this story.