OMGIMERG, analyzing forecast rollouts cont., NVIDIA Earth-2

Who's the acronym artist at NASA responsible for this tomfoolery? Yes, IMERG is constructed from Integrated Multi-satellitE (!!!) Retrievals for GPM. Next level.

NASA IMERG Early and Late are live

IMERG combines observations from a constellation of satellites into a global precipitation analysis. We now have two live-updating IMERG datasets in the catalog:

NASA IMERG Early is available about 4 hours after observation time.
NASA IMERG Late incorporates more observations and both forward and backward propagation, with a latency of roughly 12 to 18 hours.

Both are global, 0.1° (~10 km), half-hourly, and extend back to 1998. They include precipitation rate and a precipitation quality index.

NASA IMERG Early

We are releasing the Early and Late runs of IMERG V07. When NASA releases V08, we plan to add the Final run as well.

STAC + Icechunk

Every catalog page now includes two ways to open a dataset: with dynamical-catalog, or directly from our STAC catalog with Icechunk.

The examples use pystac because it keeps the code concise, but any HTTP client can read the same STAC metadata. The important part is that the STAC is the source of truth for the current Icechunk asset:

import icechunk
import pystac
import xarray as xr

catalog = pystac.Catalog.from_file("https://stac.dynamical.org/catalog.json")
collection = catalog.get_child("noaa-gfs-forecast")
asset = collection.assets["icechunk-https"]

repo = icechunk.Repository.open(icechunk.http_storage(asset.href))
session = repo.readonly_session("main")
ds = xr.open_zarr(session.store, chunks=None)

One request: don't copy the asset URL out of STAC and hard-code it into your application. We may move or version stores in the future. Resolve the current location from STAC when you open the dataset, or let dynamical-catalog do that for you.

This is a good time to remind everyone still accessing data products via data.dynamical.org: that endpoint is deprecated and will be discontinued at the end of August.

When is a forecast late?

"What then is time? If no one asks me, I know; if I wish to explain it, I do not."

You all made Saint Augustine proud with your questions, and prompted a deeper dive into when a forecast should be considered "delayed" (among other explorations).

The short of it:

The delay threshold has been switched p95 + max(p95 - p50, 15 minutes) which does a decent job across all sources whether they have a really tight distribution (HRRR) or a wide one (ECMWF AIFS ENS)
We analyzed whether delays of early lead groups can detect the full run being delayed, or if they often recover
We loaded NOAA and ECMWF dissemination advisories (see next section) to test if we could predict them through observed file arrivals.

The longer version, including what we learned from replaying roughly a year of arrivals, is in our living lab notes.

The findings resulted in improvements to our pipeline status.

Upstream dissemination advisories

We now ingest dissemination advisories from NOAA and ECMWF, map them to the products we monitor, and show them alongside observed delays. We backfilled the advisory archive to early 2022 and now track new and resolved notices in real time.

These are useful as two separate signals: our arrival monitoring tells us what is happening in the files, while an agency advisory can tell us why. Advisories are available on the pipeline status page, in the machine readable status feed, and as a webhook event.

New variables

Thank you for your requests! We added a few requested variables to existing datasets:

GFS analysis and forecast: 80 m wind components, temperature, and pressure
GEFS analysis and 35-day forecast: 80 m wind components, temperature, and pressure
HRRR analysis and materialized 48-hour forecast: visible-beam downward solar flux, or direct/beam surface irradiance

The exact variable names and coverage are in each catalog entry.

dynamical.org in NVIDIA Earth2Studio

NVIDIA Earth2Studio now includes native dynamical.org data sources. Earth2Studio resolves collections through our public STAC catalog and reads the anonymous Icechunk repositories directly.

Thanks to Duncan Anderson (who made an excellent book recommendation), Nick Geneva (who has not yet discussed books with me -- yet --, but was very pleasant to chat with in the PR!) and the NVIDIA Earth2Studio team for suggesting and working through the integration with us.

Enjoy the weather, ^ MM