tag:speakerdeck.com,2005:/tomnicholas tag:speakerdeck.com,2005:Talk/1542991 2026-05-18T10:26:53-04:00 2026-05-18T10:29:05-04:00 Talk given to the ESIP Cloud Computing Cluster Tom Nicholas (@tomnicholas) tag:speakerdeck.com,2005:Talk/1526829 2026-04-08T10:14:39-04:00 2026-04-08T10:16:56-04:00 Talk given at AMS 2026 in Houston, Texas Tom Nicholas (@tomnicholas) tag:speakerdeck.com,2005:Talk/1416776 2025-07-30T10:08:10-04:00 2025-07-30T10:16:07-04:00 Talk given to a group of Oxford University Research Software Engineers. Tom Nicholas (@tomnicholas) tag:speakerdeck.com,2005:Talk/1401795 2025-07-11T10:22:17-04:00 2025-07-11T10:25:54-04:00 Tom Nicholas (@tomnicholas) tag:speakerdeck.com,2005:Talk/1399396 2025-07-09T16:59:54-04:00 2025-07-09T17:01:04-04:00 https://github.com/cubed-dev/cubed Tom Nicholas (@tomnicholas) tag:speakerdeck.com,2005:Talk/1365392 2025-05-06T12:44:10-04:00 2025-05-06T12:48:15-04:00 Talk given at the Cloud-Native Geospatial Forum 2025 Tom Nicholas (@tomnicholas) tag:speakerdeck.com,2005:Talk/1324206 2025-02-12T20:59:04-05:00 2025-02-12T21:03:30-05:00 Talk slides from Pangeo Showcase February 12th 2025 https://discourse.pangeo.io/t/pangeo-showcase-frost-federated-registry-of-scientific-things-feb-12-2025/4861 Blog post mentioned: https://hackmd.io/@TomNicholas/H1KzoYrPJe Abstract: Context: The easiest way to store and provide access to big scientific datasets is via ARCO data in S3-compatible cloud object storage. We now have scalable cloud-optimised formats that are version-controlled at rest in object storage (particularly Icechunk for arrays and Iceberg for tables). This is huge, as even dynamically-updated datasets can now be distributed via raw S3, with no other server needed. All the data providers who are paying attention are about to put their data in these formats, but then they will try to advertise the S3 URLs to the world via ad-hoc data catalogs. Problem: Everyone’s catalogs are disconnected from everyone else’s. This means: - No cross-org discoverability (e.g. NASA catalog users won’t see NOAA datasets or vice versa). - No cross-org tracking of updates (e.g. NOAA datasets derived directly from NASA datasets won’t automatically know if the NASA datasets have been updated upstream). - Risk of “catalog wars” where platform services compete to make more and more comprehensive “meta-catalogs” which merely track (outdated) links to other orgs’ data. - Risk that if one platform does win everyone might feel locked in to it via the social network effect. Solution: Federated catalog protocol with cross-org publish-subcribe model. - Cross-org discoverability enabled via displaying the contents of the dataset entries being broadcast, - Cross-org tracking of updates to datasets enabled the same way, - No need to compete to make a better catalog, as anyone can easily consume and display the entire global catalog, including updates, - Federated trust model allows proliferation of high-quality centralized services, whilst also guarding against platform lock-in. How do we build it?: Not sure exactly, but the problem is analogous to creating Federated alternatives to centralized social media (i.e. Bluesky/Mastodon vs Twitter). Perhaps we can piggyback off of Bluesky’s ATproto or Mastodon’s ActivityPub? Tom Nicholas (@tomnicholas) tag:speakerdeck.com,2005:Talk/1282153 2024-11-21T10:35:26-05:00 2024-11-21T10:38:12-05:00 VirtualiZarr talk with section on Icechunk integration, given to the MET Office Architecture Guild. Tom Nicholas (@tomnicholas) tag:speakerdeck.com,2005:Talk/1183322 2024-05-15T13:59:41-04:00 2024-05-15T14:00:50-04:00 Long-form talk on the VirtualiZarr package as an alternative to kerchunk for creating virtual Zarr stores which point to archival data (e.g. many netCDF files). Recording will be posted here (https://discourse.pangeo.io/t/pangeo-showcase-virtualizarr-create-virtual-zarr-stores-using-xarray-syntax/4127) See the VirtualiZarr repository for more details (https://github.com/TomNicholas/VirtualiZarr) Tom Nicholas (@tomnicholas) tag:speakerdeck.com,2005:Talk/1118426 2023-12-08T11:10:13-05:00 2023-12-08T11:11:11-05:00 Talk given at the Pangeo Showcase on 6th December 2023. Intended to start a community discussion, which was then recorded here: https://discourse.pangeo.io/t/pangeo-showcase-whats-next-for-pangeo/3870 Tom Nicholas (@tomnicholas) tag:speakerdeck.com,2005:Talk/1106805 2023-11-15T13:32:07-05:00 2023-11-15T13:49:48-05:00 Long-form talk on using the Cubed package as an alternative to dask.array for processing large datasets in Xarray. Recording will be posted here (https://discourse.pangeo.io/t/pangeo-showcase-cubed-bounded-memory-serverless-array-processing-in-xarray/3836) See this blog post for more details (https://xarray.dev/blog/cubed-xarray) Tom Nicholas (@tomnicholas) tag:speakerdeck.com,2005:Talk/1052064 2023-07-17T13:32:03-04:00 2023-07-17T13:32:50-04:00 Short 4-minute talk on using the Cubed package as an alternative to dask.array for processing large datasets in Xarray. Given as a lightning talk at the SciPy Conference 2023 in Austin, TX. See this blog post for more details (https://xarray.dev/blog/cubed-xarray) Tom Nicholas (@tomnicholas) tag:speakerdeck.com,2005:Talk/975513 2023-01-12T01:23:56-05:00 2023-01-12T01:24:45-05:00 Presentation at the BOUT++ workshop 2023 at LLNL, on why the fusion plasma physics analysis community should learn lessons from the success of the Pangeo model in the geosciences community. Tom Nicholas (@tomnicholas) tag:speakerdeck.com,2005:Talk/975204 2023-01-11T13:23:08-05:00 2023-12-11T10:50:14-05:00 (This talk was given at the AMS conference in Denver 2023) Staggered grids such as Arakawa grids are ubiquitous in climate models. Analysis and post-processing tools using finite-volume operators must respect staggering to get correct results. This presents a challenge for users of General Circulation Model (GCM) data, whose analysis routines must follow the idiosyncrasies of particular GCM grids. The xGCM package [1] is designed to solve this problem, by extending xarray’s data model with information about the grid. It encodes the positions of variables along each axis of the grid, so that operations can respect differences in staggering between variables. It also encodes the topology of the grid, understanding how the spherical Earth is divided into different regions in various models. XGCM has recently been upgraded by introducing the concept of “Grid Ufuncs”, which are analogous to numpy ufuncs but grid-aware. Users can define their own grid ufuncs, then apply them to their data. We hope that this extensible model can be built upon by scientists who work post-processing all types of climate models. We present an overview of xGCM and its capabilities, before showing an example of using it on a multi-TB scale oceanographic dataset. [1] https://github.com/xgcm/xgcm Tom Nicholas (@tomnicholas) tag:speakerdeck.com,2005:Talk/974167 2023-01-09T16:12:40-05:00 2023-01-09T16:13:10-05:00 Talk from the AMS 2023 conference. Abstract: Real scientific workflows often require working with many heterogeneous but related datasets. Examples in geoscience include: (1) scenario simulations by many different climate models in the same intercomparison project, (2) simulation data at multiple resolutions from a convergence scan or sub-grid-scale study, and (3) observational + simulation data of the same region. There is a need for a general high-level data structure which can organize such data in an accessible way, whilst still being flexible enough to adapt to the user’s mental model of their data. It should also be intuitive, so that simple operations such as calculating average climatologies are still simple to express. It should also serialize to a commonly-used data format, so as not to create backwards compatibility problems. The new xarray-datatree [1] package solves these problems, by providing a tree-like hierarchical data structure that is general enough to be useful in a wide variety of cases. Datatree extends xarray - generalizing xarray.Dataset to build upon an interface that many geoscientists are already familiar with. Analysis operations can be mapped over a whole tree, allowing simple operations to be expressed intuitively, even over complex heterogeneous datasets. Datatree is inspired by netCDF: Xarray’s highest-level object is currently an xarray.Dataset, which stores collections of arrays with a shared coordinate system and corresponds to a single group in a netCDF file. A DataTree object is instead a structured hierarchical collection of Datasets, and would map to multiple netCDF groups. Therefore serialization to and from netCDF files is possible with datatree, so backwards compatibility is maintained. We will explain the model of datatree, its relation to netCDF & Zarr, and how to use the data structure to simplify your own work. We will also give examples of using datatree with real geoscience datasets, such as CMIP6 model data. [2] [1] https://github.com/xarray-contrib/datatree [2] https://medium.com/pangeo/easy-ipcc-part-1-multi-model-datatree-469b87cf9114 Tom Nicholas (@tomnicholas) tag:speakerdeck.com,2005:Talk/905477 2022-08-08T09:41:40-04:00 2022-08-08T11:06:36-04:00 Tom Nicholas (@tomnicholas) 2026-05-18T10:26:53-04:00