Today we are very pleased to announce the availability of deck.gl v8.6, which alongside a whole host of new features, adds full support for the WebGL Overlay View feature of the Google Maps JavaScript API.

Google Maps Platform revolutionized the web mapping space when it was launched back in 2005 and has become the most popular framework for developing map applications on the web.

As the leading open source framework for data visualization, we recognize the importance for deck.gl to have strong support for Google Maps and in 2019 we kickstarted development to ensure the two could work seamlessly together. This made it possible to use Google Maps as a raster basemap, but with this year’s addition of support for WebGL overlays we have taken it even further.

Luckily Google thinks the same and has been working hard alongside the CARTO team to add much stronger deck.gl support for Google Maps by utilizing these new WebGL features.

Read the latest blogpost from Google Maps Platform to check out the great visualizations they have created to showcase deck.gl and Google Maps integration.

What’s new?

This new release adds support for the vector basemap in interleaved mode. This means that with deck.gl, Google’s vector map can now be mixed with data layers, providing a pixel perfect composition where labels, 3D, and other content are respected and rendered perfectly with depth and occlusion.

Full vector rendering support with backwards compatibility

While implementing support for vector rendering, we worked hard to maintain compatibility with the existing GoogleMapsOverlay class. So you can continue using it as before but with the following enhancements:

• Shared 3D space: objects drawn by the GoogleMapsOverlay class appear inside the Google Maps scene, correctly intersecting with 3D buildings and behind the contextual labels drawn by Google Maps.
• Tilting and rotating the view is supported.
• Rendering uses the same WebGL context as Google Maps, improving performance.

Getting started with deck.gl and Google Maps Platform

The basics are simple. Create a deck.gl Layer and add it to a GoogleMapsOverlay object and finally add that overlay to your Google Maps Map object.

You start by loading deck.gl 8.6 and the Google Maps library. The WebGL Overlay View of Google Maps is currently only available on the beta channel. You need to specify your API key and a Map ID with vector maps enabled. For more details check out the official documentation.

<script src=”https://unpkg.com/deck.gl@8.6.0/dist.min.js"></script><script src=”https://maps.googleapis.com/maps/api/js?key=xx&v=beta&map_ids=xxx"></script>

The next thing is to create the Google Map and a deck.gl Layer.

// Create a Google Map
const map = new google.maps.Map(document.getElementById("map"), {
  center: {lat: 50, lng: 14},
  tilt: 30,
  gestureHandling: 'greedy',
  mapId: 'fae05836df2dc8bb',
  zoom: 3
});

// Create a deck.gl visualization layer
const flightsLayer = new deck.ArcLayer({
  id: 'flights',
  data: AIR_PORTS,
  dataTransform: d => d.features.filter(f => f.properties.scalerank < 4),
  getSourcePosition: f => [14.42076, 50.08804], // Prague
  getTargetPosition: f => f.geometry.coordinates,
  getSourceColor: [0, 128, 200],
  getTargetColor: [0, 0, 80],
  getWidth: 1
});

And finally create a GoogleMapsOverlay that will be added to the map.

const overlay = new deck.GoogleMapsOverlay({
  layers: [flightsLayer]
});

overlay.setMap(map);

Live demo in JSfiddle

To learn more and see a full set of example visualizations check out this link: http://jsfiddle.net/user/felixp/fiddles/

Other improvements coming on deck.gl 8.6

There are many other improvements available in this version of deck.gl. Here are some of the highlights:

• CartoLayer adds new geoColumn and columns props, enables more granular data fetching from CARTO backend.
• ColumnLayer and GridCellLayer add radiusUnits prop.
• H3HexagonLayer now supports manually forcing low-precision, high-performance rendering with highPrecision: false.
• HeatmapLayer adds weightsTextureSize and debounceTimeout props for fine-tuning performance.
• MVTLayer now defaults to handling geometries in binary.
• Scatterplot and GeoJsonLayer add option to turn off antialiasing to avoid artifacts in depth oclusion.
• TileLayer no longer purges its cache when data changes, resulting in a smoother “reload” experience

Thank you very much to all of the developers who have contributed to this latest release.

Welcome Google Maps developers!

The vision of deck.gl is to provide the best open source, open governance visualization library for large-scale datasets. CARTO and Google Maps have agreed to maintain and support this connection between deck.gl and Google Maps for the years to come.

Many organizations are now collaborating to make this project a success and that’s why we believe our open governance model is so important.

So stay tuned for more layers and features, and please share with us what you build with deck.gl 8.6.

We can’t wait to see what you build!

deck.gl v8.6 now available with deeper Google Maps support was originally published in vis.gl on Medium, where people are continuing the conversation by highlighting and responding to this story.

This blog post shares some thoughts about why the move to open governance was so important, and gives some glimpses of the exciting public roadmap that was presented.

Why Open Governance?

Many great open source software projects, while free to use and open to contribution, are still controlled by one company or a small group of people. This is often fine, as the owners are also typically the maintainers and tend to work hard (usually for free) to support their communities.

But once an open software project gets to the point where multiple companies depend on it for their own commercial products, and the project starts to receive major development contributions from those companies, the introduction of an open governance model can make a big difference in removing barriers to adoption.

The goal for moving vis.gl to open governance was to create an even more inviting playing field for anyone who wants to use or contribute to deck.gl. Contributors can now become involved in the decision making process, and those that make significant contributions can even get seats on the steering committee and get a say in all vis.gl-related program matters.

An Open Planning Process

A key aspect of open governance is to let the community take part in the planning of upcoming software releases, and the first community planning meeting for the deck.gl 8.2 release was held in early May 2020.

The various vis.gl tech leads took turns in sharing the plans for the upcoming deck.gl 8.2 and 8.3 releases, as well as longer term roadmaps.

The planning meeting was well attended, with representatives from a number of leading geospatial software companies, and the feedback from the audience was very positive.

Highlights from the community planning meeting

Advanced TileLayer development is a major focus area for deck.gl 8.x

Support for tiled data has seen a lot of development over the last year, and the deck.gl + loaders.gl combination now provides a comprehensive support for tiled data layers including 2D tiles (geospatial and non-geospatial), 3D tiles (both the 3D Tiles and I3S OGC standards are supported), as well as terrain from tiled elevation data sources etc.

Also many “tricky corner cases” that arise when visualizing tiled data, such as cross-tile highlighting, working with high pitch views, request throttling when loading multiple tiled data sources etc are now handled correctly by deck.gl.

Longer Term Roadmaps (deck.gl v9)

The open governance meeting also gave the tech leads a good opportunity to provide the community with a glimpse of features that are being considered for the next major releases of the frameworks, such as:

Improved support for non-JavaScript programming languages

• pydeck
• deck.gl-native (C++ port)
• Swiftdeck, Javadeck for mobile
• Language independent styling specification, cross-language transport protocol

Core Features

• Globe projections
• Advanced Tile Layer Features
• Typescript, WebGPU, ES Modules

These are just a few highlights, for those interested in the details, the slides from the presentation are linked at the end of this post.

Also check out the deck.gl 8.2 Whats’ New page as it details the outcome of the first community planned release.

Join the vis.gl movement!

A goal of the moving deck.gl and the other vis.gl frameworks to an open governance model is of course to open the doors to additional participation. We want to grow the vis.gl community, and if you are interested don’t hesitate to reach out on the channels below. We look forward to hearing from you!

• vis.gl blog (Medium)
• deck.gl slack room
• deck.gl GitHub issues
• vis.gl community planning meeting slides
• join vis.gl announcement mailing list (low volume)
• vis.gl technical charter
• deck.gl website

deck.gl 8.2 moves to Open Governance was originally published in vis.gl on Medium, where people are continuing the conversation by highlighting and responding to this story.

Today, we’re excited to announce that we’ll be bringing deck.gl’s declarative syntax, the ease of use, and ability to render massive data to Python with the beta release of pydeck, a set of Python bindings for deck.gl. In this post, we’ll cover pydeck’s unique features, how it works beneath the hood, and how you can get started using it today.

The mission of pydeck is to let Python users create deck.gl maps without having to know a lot of Javascript. At its core, pydeck focuses on data analytics use cases, so it works best in environments used by data professionals, like Jupyter Notebooks. With pydeck, users can embed visualizations interactively in a Jupyter Notebook or simply export them to a standalone HTML file.

A few mapmaking libraries exist within the Python ecosystem, so it’s worth highlighting the unique features that pydeck provides which differentiate it:

• The availability of the full deck.gl layer catalog in Python
• Support for large-scale updates, like color changes or data modification, to hundreds of thousands of visualized data points
• Two-way communication, where data selected in a visualization can be passed back to a Jupyter Notebook’s kernel
• The ability to map hundreds of thousands of data points in 2D and 3D via a Python API

Under the hood, pydeck converts its Python objects to JSON and passes that JSON to the new @deck.gl/json API. This library interprets JSON objects into deck.gl layers, letting users render a visualization written in Javascript without having to know Javascript itself. Pydeck augments this by also providing components for Jupyter Notebook integration and convenience functions for data processing.

You can get started using pydeck with a simple copy and paste in your terminal:

pip install pydeck

To see pydeck in action, you can copy and paste this Python code into a Jupyter Notebook or an IPython terminal:

You can also check out the hosted examples on mybinder.org, where you can use pydeck in your browser without installing it. For further details while coding, review the documentation home page here.

You can file bugs and submit feature requests with our team on GitHub. Contributions are both welcome and encouraged, not only to improve pydeck but also to bindings for other languages via the @deck.gl/json library.

pydeck: Unlocking deck.gl for use in Python was originally published in vis.gl on Medium, where people are continuing the conversation by highlighting and responding to this story.

Co-authors: Gabriel Durkin, Sina Kashuk

kepler.gl is an advanced geospatial visualization tool open sourced by Uber’s visualization team in 2018 and contributed to the Urban Computing Foundation in early 2019.

At Uber, kepler.gl is the de facto tool for geospatial data analysis. In a previous article, we introduced kepler.gl for Jupyter Notebook. In this article, we want to showcase how data scientists at Uber use kepler.gl to understand massive amounts of aggregated geospatial data and derive insights that improve our business. All the analysis presented in this blog post is based on data aggregated by H3, Uber’s open source geospatial indexing system, with an aperture equal to 12, for the locations with a minimum of 100 trips counts using at least 6 months of data.

Uber’s platform leverages digital solutions to tackle transportation problems in the physical world, such as ridesharing and meal delivery. Gabriel Durkin and Sina Kashuk, data scientists from Uber’s Rider Geospatial Intelligence team, leverage kepler.gl to analyze trip data, specifically to understand the real-world challenge of driver-partners and riders locating each other for pick-up in a complex cityscape. Figure 2, above, illustrates how the projection of the pick-up data at high resolution can create a map of the city of Toronto based entirely on usage of the Uber app, without leveraging a single base map.

The pick-up process of an Uber ride or Uber Eats meal is one of Sina and Gabriel’s biggest data science pain points. There are many geospatial challenges associated with the pick-up process, and our teams frequently use kepler.gl to inform the development of geospatial solutions to improve this part of the ridesharing and delivery experiences. A common visualization rendered for this type of problem solving is a time lapse animation, as depicted in Figure 3, above, to identify temporal trends and areas with a higher concentration of trip requests, which may be correlated with suboptimal pick-up experiences.

Sina and Gabriel also created a ‘success’ metric that identifies places associated with quality pick-up experiences, i.e., spots with a minimum of cancellations, pick-up location errors or other types of defects. These were projected onto city maps as hexagonal spatial units visualized in kepler.gl using the H3 layer (Figure 4), and ingested by our pick-up spot recommendation engine, so that the future suggested pick-up locations are far from ‘low success’ areas.

This success metric can equally be mapped to the rider device request location (likely inside a building) rather than the “in the street” vehicle pick-up location. Two complementary maps are created, as depicted in Figure 5, for understanding the pick-up experience: (top) one depicting hexagonal heatmaps of Uber metrics and (lower) one depicting the physical landscape of buildings and city-block polygons for the same locations. The finalized maps, like those in Figure 5, are saved to an interactive HTML file that is shared with regional Uber Operations teams to flag and address complex or problematic pick-up areas. Numerical labels are applied to annotate the densest areas, and geofences are created around them (as depicted by the polygons in the lower plot). When requesting a trip in these ‘enhanced’ pick-up zones, riders will be given additional instructions to guide them through the process.

Another kepler.gl metric Gabriel and Sina assessed when working on ways to improve the Uber pick-up experience was the estimated time of arrival (ETA) error. When users request a ride on the Uber platform, they expect the ETA of their driver to map as accurately as possible to our in-app calculations. ETA errors will result in poor user experiences. The two data scientists created maps to identify areas with higher ETA errors, allowing teams to better understand where the algorithms that produce the estimates are performing well and where they need to improve.

For example, in the very early morning, there seems to be a longer ETA in the Northeast, evidencing sparsity of vehicle supply (Figure 6). The lack of clustering and low density of requests throughout suburban neighborhoods clearly presents a challenge for dispatching new trips at that time of day.

The dual map in Figure 6, above, uses kepler.gl to discover correlations between arrival times and the volume of Uber requests in San Francisco. Both maps show places of highest request volume but the left map is colored by ETA (long ETAs in red, short ETAs in blue), and the right map is colored by request volume (high volume in yellow).

Gabriel and Sina are working with product managers on the Rider Team to build solutions that have spatial and temporal context-awareness. One possibility would be a notification sent to the rider at rush-hour, e.g. “Hey, you are in a busy area. You should request a few minutes in advance to avoid longer delays.” Insights derived via the lens of kepler.gl will drive the design of potential new Uber rider app geo-contextual features.

kepler.gl also provides a lens to bring clarity to the understanding of the movements and travel behaviors of people in cities. Figure 7 shows three different map layers of kepler.gl for Manhattan. The left image projects place-heterogeneity — places with a higher ratio of riders to trips are indicated in yellow (Fig.7 LEFT). The heterogeneity, or public/private metric, usually indicates whether an area is more public and populated by visitors and tourists. For instance, this metric is close to 1.0 at airports because each rider typically takes a single trip from the airport, compared with < 0.6 at residential neighborhoods where the same rider may request multiple trips, e.g., every weekday morning to work. In Figure 7, these regions appear dark red. Challenging pick-ups can result from a rider’s unfamiliarity with a new place, rather than any intrinsic property of the place itself. The heterogeneity metric projected via kepler.gl highlights these cases geospatially.

The middle image of Manhattan represents ETA errors (Fig.7 MIDDLE) and suggests that more isolated locations (near the waters’ edge) lead to greater uncertainty in the estimated time of arrival of the dispatched cars.

By projecting ‘completed trip distance’ onto the geo-location of requests in kepler.gl (Fig.7 RIGHT) additional geographical constraints imposed by the waterways surrounding Manhattan are apparent. For example, trips beginning at its southernmost tip tend to be longer, in part because riders must travel North to get anywhere.

The Jupyter Notebook platform combined with kepler.gl (Figure 8) allows data scientists like Sina and Gabriel to identify and develop an understanding of the geospatial nature of Uber’s data — that data patterns and clustering is apparent via kepler.gl maps is proof that the ‘prior art’ of collecting data in tables and spreadsheets alone ignores the confounding influence of its spatial context.

Using kepler.gl, Gabriel and Sina were able to gain new insights, consistently increase pick-up quality, and improve rider experience across all the cities and continents where Uber operates.

Now that kepler.gl for Jupyter is open sourced, we are eager to see how it is adopted by other data scientists working in the geospatial domain and learn about inventive new uses cases developed by the community in the coming months.

Exploring Geospatial data with kepler.gl was originally published in vis.gl on Medium, where people are continuing the conversation by highlighting and responding to this story.

Two excellent reasons to launch our second Vis Hackathon, in which 11 teams from across Uber Vis participated. We had two themes this time:

• Using the new speed datasets from movement.uber.com,
• Showcasing new features from our frameworks like deck.gl, loaders.gl, luma.gl, etc.

Without further ado, here’s what we did!

Urban Symphony

Jon Sadka

Close your eyes for a moment and imagine what it’s like sitting in a quickly accelerating car. Chances are that part of this imagination includes sound.

Sound is a natural byproduct of movement and speed, yet, by their nature, most visualizations about speed only express data visually. Urban Symphony is an experiment to see what happens when we used speed data to generate a melody for a given path of road segments.

Sparrow

Wesam Manassra, Ed Barwani, Adam Kidder, Ib Green
Best use of city data prize

Apache Arrow is a columnar in-memory data format that is interoperable across different languages, and makes it efficient to transport and scan large datasets without the need for serialization/deserialization on servers/clients.

We unlocked the powers of Arrow by building Sparrow, a tool to visualize a year’s worth of hourly speed data in the browser dynamically and efficiently (around 130 million speed readings). We built a lightweight Python server to read the dataset partitioned by day and stream it over HTTP. We can then visualize hourly speed data in the browser as soon as it arrives from the server. As the user scrubs to a certain date, data is loaded and flushed dynamically to maintain an upper bound on the total memory used by the browser.

This approach allows us to visualize very large data sets with low latency. As a future exploration, we’d like to stream this data directly to the GPU to enable even better performance.

Traffic Flow

Javid Hsueh

This project simplifies the road network and visualizes the traffic flow of the city. The junctions will move toward the major traffic direction at each hour.

The two main challenges of this project were the road network simplification and using transform feedback in a custom deck.gl layer to animate the flow on the edges.

ML Speeds Limit

David Schnurr, Ben Kramer

Basemaps like OSM often have inconsistent availability of speed limit data. Can we build a supervised machine learning model that utilizes Movement Speeds data to predict speed limits on roads where it’s not available?

Our team had limited ML experience, so a lot of time was spent learning about model selection (we settled on a decision tree regressor) and feature engineering.

The accuracy of the final model was actually relatively decent, and we think given extra time its accuracy could be improved even further with better feature engineering, hyper-parameter tuning, and testing deep learning approaches.

SpeedsUp

Bryant Luong, Lezhi Li

In this project, we tried to discover daily speed patterns of various streets in a city using machine learning on Movement Speeds data. On the right, we represent streets as time series of speed over the course of the day. We can then cluster these time series by similarity — the streets which are always fast, always slow, fast and slow at similar times, etc.

We used Apache Arrow to load a very large dataset in the browser and TensorFlow.js for efficient in-browser machine-learning.

Really Fast Rectangles

Brian Ford

Can aggregation and drawing functions based on the incremental lambda calculus be a practical way to speed up data visualization programs? To find out, we wrote one data visualization using d3.js and another one using incremental functions, and then benchmarked the two approaches.

While our intuition is correct and that lambda calculus can yield drastic performance improvement, we also found that writing aggregations in terms of changes is surprisingly difficult, and that not all types of aggregation benefit from this approach.

How to Animate Anything

Shan He
Grand prize

In this project, we wanted to experiment with deck.gl’s attributes transition in kepler.gl to visualize speed changes over time.

Animation is, at its core, the transition of visual attributes (position, color, radius, thickness) based on time. In kepler.gl, layer visual attributes can be encoded by selected field values. However, there is currently no way to animate these attributes based on time: e.g., you can draw road segments and color them by speed, but you can’t animate the colors based on speed changes over time.

The way we allow the animation of visual attributes (position, color, radius, etc) is based on an additional time table. The user can select a dataset to draw the shape (road segments), and another dataset that contains a time table (speed per segments per hour). They can then join these two data tables and enable animation for visual attributes that are based on value (speed, # of cars, # of passengers, etc).

Cinematic kepler.gl

Chris Gervang, Chun Jiang, Alison Lee

This project provides the ability to sequence animations and generate videos in kepler.gl using an easy magic keyframe UI. It integrates a WIP video library, hubble.gl, into kepler.gl delivering instant videos, right to your downloads! It demonstrates the new luma.gl Timeline and KeyFrames classes.

We were challenged with striking a balance between simplicity and functionality in the UI design, by writing new UI components into kepler.gl, and by figuring out how to integrate the necessary parts of luma.gl, deck.gl, kepler.gl, and hubble.gl together.

Citymorph

Xiaoji Chen, Jerome Cukier
Best use of frameworks prize

From the Movement Speeds dataset, we can deduce how much time it takes, on average, to traverse street segments at various times of the day. We can use that data to morph the maps of cities and from one given point, reposition all street intersections according to how much time it would take to go there.

One interesting aspect of this project is that we did the required graph traversal directly in the GPU.

(The interactive version of this project will be coming shortly to the deck.gl showcase.)

Hex Hedgehogs

Nick Rabinowitz, Ib Green

The Hex Flow project was intended to prototype visualization approaches for city-level trip flow patterns, using the H3 library and deck.gl. Using several weeks of anonymized routes from Jump Bike data, we aggregated flow patterns by hour of week for every H3 cell in the city, then built an interface that could display flow as an animated vector field.

The primary challenge was to aggregate tens of thousands of routes into a format that was both performant and visually comprehensible. Our final version included both offline and runtime aggregation, including multi-pass smoothing, to generate an H3-based vector field for each time slice and interpolate between them.

Validating the data processing and visualization would have been very difficult without knowledge of the city in question. We knew we were on the right track when we could see the clear bike commute patterns up and down Market Street at appropriate times!

Map Collage

Yang Wang, Jian Huang, Ravi Akkenapally

Map Collage is like photo collage, but (wait for it) for maps.

This consolidates multiple sub-maps into one canvas with dynamic boundaries generated by the underlying data. It frees you from constant zooming in and out when examining multiple places-of-interest with limited screen space (e.g. on mobile devices or in a UI widget).

The hardest part of the project was to interactively and dynamically generate the boundaries of each sub-map.

Vis Hackathon 2019 was originally published in vis.gl on Medium, where people are continuing the conversation by highlighting and responding to this story.

I previously posted two articles where I described some of the application features. Both articles are the following:

• Animating 40 years of California earthquakes
• Kepler.gl + Dropbox = Save and share your maps

In this article we are going to describe the export map feature in Kepler.gl.

Let’s create a simple map

In order to export our map, you first need to create a map. Kepler.gl website has some predefined maps:

• visit Kepler.gl earthquake demo

The new map will look like the screenshot below, showing all California earthquakes over the past 40 years.

Once you are happy with the map and all the layers you can create on it (user guide), let’s move on to exporting the current visualization by clicking on the share button (top left corner),

upon clicking the Share button, you will be provided with a list of options as shown below.

Click on Export Map.

After clicking on Export Map, Kepler.gl will show a modal dialog where you can select the format of your exported map. As shown in the following screenshot

Kepler.gl provides two different options:

• HTML: create a single HTML file of your map with data and configuration.
• JSON: create a JSON file with your current map data and config. You can later load it back to kepler.gl

Export Map as HTML

The default export map option is HTML, see HTML export modal dialog image. As part of HTML export options, you can provide a Mapbox token to be saved in the HTML file.

If you don’t provide a Mapbox token, Kepler.gl will use a temporary one which can expire later. We recommend to use your own Mapbox token.

If you don’t have an access token when you export the map, you can always create a new token and update the generated HTML file later one when necessary.

Data and configuration will be inlined in the single HTML file. You can use it as a static page and share it with other people.

The generate HTML file can be pretty big since it contains all map data.

Export Map as JSON

JSON is the second available option to export your map. Once you click on JSON button, Kepler.gl will show the following modal window.

The lower part of the modal dialog shows the current map configuration.

The map configuration is coupled with loaded datasets; each part of the configuration will contain a dataId which will refer to one of the loaded datasets.

Once you click on Export, Kepler.gl will create a JSON file containing both map configuration and data. The newly generated file can be used to recreate our original map. See below

The generate JSON file can be pretty big since it contains all map data.

Conclusions

In this post we reviewed the Kepler.gl export map functionality to create an HTML or JSON file.

HTML export can be really useful if you want a static representation of our map to include on your website, share it with other users or visualizing it offline on our computer.

JSON is a lean representation of our map and the file can be stored in a CDN, share with other people and load it later to reproduce the exact same map.

Exporting map and data is part of the core abilities Kepler.gl provides and we are continuing working on improving and adding new features.

If you want to know more about Kepler.gl export features, check the user guides.

Kepler.gl: Export Map was originally published in vis.gl on Medium, where people are continuing the conversation by highlighting and responding to this story.

kepler.gl is an advanced geospatial visualization tool open sourced by Uber’s Visualization team in 2018 and contributed to the Urban Computing Foundation earlier this year. At Uber, kepler.gl is the defacto tool for geospatial data analysis.

In order to help data scientists work more effectively, we integrated kepler.gl into many widely used data analysis platforms, now including Jupyter Notebook. Jupyter Notebook is a popular open source web application used to create and share documents that contain live code, equations, visualizations, and text, commonly used among data scientists to conduct data analysis and share results. At Uber, data scientists have utilized this integration to analyze multitudes of geospatial data collected through the app, in order to better understand how people use Uber, and how to improve their trip experience. Now, everyone can leverage kepler.gl within Jupyter Notebook.

We integrated kepler.gl as a Jupyter Widget. It loads kepler.gl inside a notebook cell, allowing users to quickly plot maps with simple python commands and interact with the UI to customize the visualization (Figure 1). It provides a seamless analysis workflow, combining data querying, transformation, analysis, and visualization — all inside Jupyter Notebook.

How to use the kepler.gl widget on Jupyter Notebook

You can find the complete user guide and demo notebooks in the kepler.gl repo documentation folder.

First, install keplergl with pip.

$ pip install keplergl

Launch Jupyter Notebook either on your local machine or on the server.

$ jupyter notebook

Load the kepler.gl widget with the command below, and an empty kepler.gl map will be loaded below the cell (Figure 2.). You can use the `height` parameter to define window size.

Now, let’s add data to the map. Like the kepler.gl app, the kepler.gl widget supports CSV and GeoJSON. In addition, it also supports Pandas DataFrame and GeoPandas GeoDataFrame. For more on supported data formats, go to the Data Format section of the user guide. Call add_data To add data to map. Dataset is required to have a name. Name is the id of the dataset in kepler.gl config, and will be used to link layers and filters to it.

After the data is loaded into the map, you can use the side panel to edit the layers, filters and base map style just like how you would with kepler.gl (Figure 3).

When you are happy with the final result, print out the current map configuration using .config and save it for later to be used as a template (Figure 4).

When calling keplergl.KeplerGl(), you can pass in config and data values to initialize a predefined map. Pay attention to the name of the data and dataId saved in layer and filter config. They need to match each other for the config to be applied. Read more about match config with data.

kepler.gl also supports saving the map as an interactive HTML document with save_to_html.

kepler.gl map data and config are saved to iPython widget state. This allows the notebook file to be rendered with rendered map. Unfortunately, widget state is not automatically saved when the kernel shuts down. Which means in order to load the map after restarting the kernel, you need to manually save notebook widget state before shutting down the kernel (Figure 5).

That’s it! This is all you need to know to use kepler.gl inside Jupyter Notebook. You can submit bug and feature request using our GitHub template. Make sure to share your maps and notebooks built with kepler.gl widget on Twitter with #keplergl or tag @heshan_cheri.

Introducing kepler.gl for Jupyter was originally published in vis.gl on Medium, where people are continuing the conversation by highlighting and responding to this story.

Authored by Georgios Karnas, Ib Green, Travis Gorkin and Xintong Xia

In 2016, the Uber Visualization team released an open source version of deck.gl and luma.gl, two Khronos Group WebGL™-powered frameworks for visualizing and exploring huge geospatial data sets on maps. Since then, the technology has flourished into a full-fledged suite of over a dozen open source WebGL and GPGPU data visualization libraries and tools, known collectively as vis.gl. loaders.gl, the newest addition to the vis.gl family, adds support for loading and rendering glTF™ assets across the tech stack. This unlocks the ability to include rich 3D content within data visualization applications built using luma.gl and deck.gl, enabling a variety of interesting new use cases. In this post, we’ll show some applications and walk through how you can use deck.gl and glTF, Khronos’ open standard 3D file format, to quickly create a geospatial data visualization that renders tens of thousands of 3D models.

The Case for glTF

Prior to glTF, it would not have been practical to add support for high-quality 3D models to a visualization framework like deck.gl. Being able to mix 3D models into visualizations is, of course, appealing, but the work involved would have been too big and difficult to prioritize, as data visualizations are typically more focused on abstract visualization types like scatterplots, lines, and extruded geometric primitives.

However, glTF has quickly risen to become a major format for 3D assets in the WebGL world: glTF is supported by many major companies and products; online model marketplaces support downloading models in glTF; many popular tools support importing and exporting models in glTF, including the open-source Blender. Additionally, glTF models can now be loaded into a growing number of WebGL frameworks, as well as opened in the Windows Explorer and dropped right into in the Facebook feed. glTF is not only a format but an ecosystem, and we wanted to let deck.gl users leverage the benefits of that ecosystem.

A major upside of the wide adoption of glTF is that the same models can now be loaded into most major WebGL frameworks. Thanks to a well-defined, modern Physically Based Rendering (PBR) material and lighting model, glTF assets render identically (and look spectacular) regardless of where they are loaded.

By following the clear and concise glTF specification, building the Khronos reference PBR shader implementation, and consulting many invaluable open source resources, deck.gl was able to relatively quickly implement support for importing and rendering high-quality 3D models at a quality which is at parity with the best gaming focused WebGL frameworks.

To us, this is a testament to the power of glTF and a major example of the benefits of open, royalty-free, and thoughtfully-defined standards within the web and computer graphics industries.

glTF for Data Visualization

Earlier this year, the Uber Visualization team, in partnership with Uber’s Advanced Technology Group (ATG), released the open source AVS.auto system. AVS.auto leverages the vis.gl tech stack for visualizing and exploring autonomous vehicle data, such as point clouds from LIDAR sensors and predicted vehicle trajectories. AVS.auto and deck.gl are being adopted across a wide range of autonomous companies to build tools that render increasingly realistic scenes containing real-world objects, like vehicles, bikes, pedestrians, and other street/city features.

In our own tools, we use the visual difference between realistic vs. abstract visual elements as part of our design vocabulary. We use more realistic designs (such as glTF models) for real world objects and contrast that with abstract “Tron”-like graphics that show paths, trajectories, and point clouds to represent what the autonomous system is seeing, predicting, or planning to do. In addition, the geospatial features of deck.gl enable glTF models to be correctly positioned on top of base maps and visually interleaved with deck.gl’s standard abstract 3D visual elements.

Another interesting use case involves replacing abstract visual elements (such as points, lines, or arcs) in deck.gl visualizations with instances of glTF models that represent something concrete about the data.

deck.gl now offers a ScenegraphLayer that accepts a glTF scenegraph. Like all deck.gl layers, the ScenegraphLayer also accepts a table of data, and each row in the supplied data table is mapped to one instance of the glTF model during rendering, with offset, rotation, and scale extracted from each table row. The ScenegraphLayer uses instanced rendering on the GPU (the same technique that is used to performantly render large numbers of similar soldiers, vegetation elements, etc. in games), which enables remarkably performant rendering of thousands of identical models.

How to Use glTF in luma.gl, deck.gl and nebula.gl

It is very easy to create a glTF layer in deck.gl; all you have to do is import ScenegraphLayer and pass the proper parameters. Here is an example that allows you to place glTF Meshes in NYC: in our case, avocados. This could represent places where you can buy good produce or fruits.

This example leverages nebula.gl, another member of the vis.gl family that provides high performance, interactive editing of deck.gl layers. The result demonstrates a geospatial glTF editor example in less than a page of code:

glTF-nebula.gl - CodeSandbox

You can move the map or rotate it using Option+Drag. Notice that you can have a 360-degree view of the loaded model as it rotates with the map!

The Future of glTF and vis.gl

Our frameworks have benefitted from the work done by many others on the glTF standard, and we want to give something back to the community. As a first step, we have open-sourced the framework-independent glTF and Draco loaders that we developed to support glTF in deck.gl.

These loaders became foundational building blocks for our new companion framework loaders.gl. This framework provides a growing suite of loaders for important 3D, geospatial, and visualization focused formats, including glTF and Draco.

The GLTFLoader in loaders.gl loads all the Khronos reference models and supports all incarnations of the glTF formats (binary GLB format, base-64 encoded JSON, and JSON with linked binary files) and also supports decoding of Draco compressed meshes. Support for additional extensions (like KHR_lights_punctual) is in progress.

Khronos, EGL, glTF, NNEF, OpenVG, OpenVX, OpenXR, SPIR, SPIR-V, SYCL, Vulkan and WebGL are trademarks or registered trademarks of The Khronos Group Inc. OpenCL is a trademark of Apple Inc. and OpenGL and OpenML are registered trademarks and the OpenGL ES and OpenGL SC logos are trademarks of Hewlett Packard Enterprise used under license by Khronos. All other product names, trademarks, and/or company names are used solely for identification and belong to their respective owners.

Uber’s vis.gl brings glTF to geospatial data visualization was originally published in vis.gl on Medium, where people are continuing the conversation by highlighting and responding to this story.

Live demo

deck.gl was originally created to work with Mapbox GL JS. All of the examples on its website still use Mapbox as the base map. Yet, since we first open-sourced the framework, we’ve been asked on GitHub: how do you use it with Google Maps?

Both map solutions use Web Mercator Projection, so the same camera settings should (for the most part) yield pixel-matching views. In practice, we found there was more to it than that, so we worked with the Google Maps Platform engineering team to make it happen.

So what was the issue?

The largest issue that delayed adding Google Maps support to deck.gl was camera synchronization. For example, when a user zooms in and out of a Google map with the scroll wheel, Google Maps always settles on an integer zoom level, and applies a smooth transition between the current and new zoom levels. For deck.gl’s visualization layers to look as a seamless part of the Google map, deck.gl must redraw its canvas on every transition frame to match itself with the base map.

To achieve this, we had two options:

• Match the deck.gl camera with that of the base map. This requires us to listen to a camera event of the map component and update the deck.gl view state on every change. But there’s a big problem with this approach: in Google Maps, the zoom_changed event is fired only once when zoom happens, and the map.getZoom method only ever returns the target (integer) zoom level. This means there’s no way for deck.gl to stay synchronized with Google’s transition animation between zoom levels.
• Match the base map’s camera with that of deck.gl. This approach would require disabling the default interaction of Google Maps, and use deck.gl’s map controller to drive all camera changes. Bad news: we can’t do that either because the map.setZoom function of the Google Maps JS API doesn’t accept fractional values, meaning there is no way for external code to plug in a smooth transition logic.

So neither of these options was viable. From the API perspective, Google Maps JS API doesn’t offer the fine-grain callbacks, getters and/or setters that deck.gl normally uses for camera synchronization. Additionally, we had no visibility into the animation curve used to control the camera in Google Map, so we couldn’t hack our end to produce the same look. It started to look like Google Maps support wouldn’t be possible.

The solution

While working on deck.gl v7.0, we partnered with developers from Google to restart the investigation into how deck.gl can work seamlessly with Google Maps. Their familiarity with the Maps API gave us the eventual breakthrough. The solution involved extending the OverlayView API and reverse-engineering the projection matrix as well as some CSS. If you are curious, here is how we did it.

We figured you probably don’t need to know the details, so we packaged it up and offered it as a new npm module @deck.gl/google-maps. To use this module, you construct a GoogleMapsOverlay instance:

import {GoogleMapsOverlay} from '@deck.gl/google-maps';

const overlay = new GoogleMapsOverlay({
  // Deck props
});

And then simply use it as any other Maps overlay:

overlay.setMap(map);

Here is what the code would look like:

Live demo

All deck.gl layers and picking functionalities (e.g. onHover and onClick) work with the Google Maps integration.

The Google Maps JavaScript library does not support free rotation of the camera, so sadly the 3D layers of deck.gl may not work very well. deck.gl does not currently support zoom levels below zero, so it will hide itself when the user zooms all the way out.

What’s next

Because we don’t use Google Maps for internal apps at Uber, this new module might not be as battle-tested as the rest of the framework. We ask external developers to kindly provide us feedback if anything can be improved.

This is not the end of our work with Google Maps. The Google Maps API team plan on offering more of their WebGL infrastructure through the JS API and are committed to work closely with us as they bring their 3D rendering to their external developers. We love the richness of Google’s map data and its beautiful visuals. I would personally die to put my data in the same 3D space as its building models and street view.

Big thanks to everyone who worked on this feature: Pascal Gula, Ib Green, Don McCurdy, and Travis McPhail.

Using deck.gl with Google Maps was originally published in vis.gl on Medium, where people are continuing the conversation by highlighting and responding to this story.

Since deck.gl’s debut as an open-source framework designed for geospatial analytics, we have continued to build boundary-pushing WebGL-powered applications at Uber, such as autonomy visualization, map editing, and machine learning. We have also seen a tremendous amount of enthusiasm from our open-source community, and learned about the unique use cases and challenges that each of our users face.

This week, we are excited to announce a new deck.gl milestone: v7.0. This is a major step forward for the framework, with more powerful visuals, a more extensible architecture, better support for industry standards, and an easier learning curve. The internal architecture has been substantially rewritten to support a multitude of new features planned for the months to come.

A Growing Layer Catalog

In this release, we have added 11 new layers to deck.gl’s official layer catalog to make common scenarios easier to implement.

TileLayer

First introduced in deck.gl v6.3, the TileLayer has graduated from experimental status. This layer loads and renders data within the current viewport using the OSM tile indexing system. The data source, format and rendering can all be customized. It demonstrates how large datasets can be spatially-indexed and interactively loaded based on the camera. You can find examples that render raster map tiles and vector map tiles on the website.

BitmapLayer

The BitmapLayer places a rectangular image in world space, with options for simple image filtering effects. It is especially useful when composing vector data with satellite imagery or raster scans of the environment. Check out the weather system example for how to use this layer.

ColumnLayer

The ColumnLayer is a generalized replacement of the HexagonCellLayer and GridCellLayer from deck.gl v6. It renders a cylinder at each given position with specified height and color, useful for constructing 3D bar-charts that visualize density.

TripsLayer

The TripsLayer, as can be seen on the deck.gl home page, has also graduated from experimental status. Support for customizable line width, a much-requested feature, has been restored.

S2Layer and H3 Layers

Popular spatial indexing systems such as S2 and H3 divide the surface of the earth into linearly indexed grid cells. Geospatial data can be efficiently aggregated, filtered and queried within these systems. The new S2Layer and H3 layers make it easy to visualize datasets that use the S2 or H3 indexes.

There are more! Check out our what’s new page for a full list.

Loaders.gl and Mesh Layers

In v7.0, we added a new category of layers: the mesh layers, which render an arbitrary 3D object at each given position with a specified size, orientation, and color.

To facilitate the loading of 3D models into these layers, we have released a new library — loaders.gl, a new companion to deck.gl. Loaders.gl is an open-source collection of framework-agnostic loaders for industry-standard 3D file formats such as OBJ, PLY and glTF. All loaders in this suite output to the same consistent, standardized format.

The SimpleMeshLayer is great for rendering simple 3D models in formats such as OBJ and PLY (example).

Most notably, we are adding initial integration with the popular glTF™ asset format. glTF is a royalty-free specification for the efficient transmission and loading of 3D assets, with a rich ecosystem of tools and extensions. All variants of glTF 2.0 are supported, including binary .glb files as well as JSON .gltf files with binary assets in base64 encoding or in separate files.

A full scene graph can be rendered with the ScenegraphLayer — a more advanced version of the SimpleMeshLayer — with experimental support for PBR (physically based rendering) materials, lights and animations.

Lighting Effect

A new effects system has been rewritten from the ground up for v7.0. This will act as a foundation for many exciting visual effect features down the road. The first feature to take advantage of this new system is the LightingEffect — an easier, more comprehensive way to control the lighting for your layers.

Light sources are now specified at a global level. All 3D layers will share the same light sources. Each layer now supports a new prop `material`, which defines how it reacts to lighting. The following light sources are supported as of 7.0:

• AmbientLight
• PointLight
• DirectionalLight
• CameraLight — a variation of the PointLight, always positioned at the camera.
• SunLight — a variation of the DirectionalLight, automatically set based on a UTC time and the current viewport.

See our developer guide for usage examples.

Google Maps Integration

deck.gl has added experimental support for using Google Maps as the base map with the @deck.gl/google-maps module.

Why Upgrade?

In addition to the features highlighted above, we have also made dozens of incremental improvements to the existing API, including:

• First-class binary data support
• Support for switching between pixel sizes and meter sizes
• Support for toggling billboard mode in IconLayer and TextLayer
• Projection and interaction bug fixes of OrthographicView and OrbitView for info-vis use cases
• Improved picking performance
• Improved React performance and synchronization with the base map
• Improved dependency management to reduced version conflicts and bundle size

Make sure to check out the upgrade guide page for deprecations and breaking changes.

Introducing deck.gl v7.0 was originally published in vis.gl on Medium, where people are continuing the conversation by highlighting and responding to this story.