dynamical.org

Knowing the moment a forecast is ready

2026-07-07T00:00:00Z

Forecast production is a factory assembly line, a fulfillment center, and a delivery route all in one. Lewis Fry Richardson's Weather Forecasting Factory was not too far off.

“Weather Forecasting Factory” by Stephen Conlin, 1986. Based on the description in Weather Prediction by Numerical Process, by L.F. Richardson, Cambridge University Press, 1922, and on advice from Prof. John Byrne, Trinity College Dublin. Image: ink and water colour, c. 50 x 38.5 cm. © Stephen Conlin 1986. All Rights Reserved ´. (Courtesy: Hendrik Hoffmann, School of Mathematics & Statistics, University College Dublin. Source)

Every dataset in the dynamical.org catalog (so far) is downstream of a model run that someone else produces on their own cadence. And the initialization is just the beginning (literally and philosophically). Then the files start landing, one by one, eventually trickling in over tens of minutes (or even hours) rather than appearing all at once. If your pipeline depends on that data, you have two bad options: pretend you can divine a cron schedule that will "always work," or poll blindly.

And to add to that, we had questions like:

How often is GEFS full lead time completed "late"?
What does "late" even mean? What is the spread of the min to the max latencies from init?
What does the rollout of a forecast look like, file by file, minute by minute?
How do different delivery routes (read: file destinations) impact latency?

These questions and many more we sought to understand deeply so that the dynamical.org catalog was resilient, low-latency, and designed with minute details about the upstream sources in mind.

So, we built a tool called wxopticon to remove that guesswork (I pronounce it "waxopticon", and I say it in a slightly mischievous voice and picture Saruman reaching for the Palantir -- no not THAT Palantir. Oh never mind). It watches upstream weather sources and dynamical.org's own catalog stores, and it answers two operational questions:

When can I expect lead-group X dataset Y? — a next-run countdown learned from observed arrival history.
Is this run on time? — per-init status measured against the distribution of prior arrivals for that product, so "late" means late relative to how this feed usually behaves.

You can see all of this on the pipeline status page.

The second component is a system that enables consumers to create subscriptions (via webhooks, Slack notifications, etc) to meaningful events (e.g. "notify me when IFS ENS progress:f024 is complete" or "notify me when GEFS on AWS is delayed").

What "ready" actually means

wxopticon models each run's progress as a sequence of readiness boundaries. Rather than a single "done" flag, a run crosses named milestones as its lead hours become available:

kind	fires when
`started`	the first file of a run lands (any lead time)
`progress`	every lead ≤ a lead-group horizon is available (e.g. `progress:f240`)
`complete`	the full run is available — you don't need to know group names
`delayed`	the run is still in flight a minute past its expected completion time (p95)

I went back and forth on the correct threshold for "delayed." Raw p95 turned out to be too harsh: for a very consistent feed the distribution is so tight that p95 sits only slightly above the median, so firing exactly at p95 would page on roughly 1 in 20 perfectly normal runs. So we nudge the trigger just past it — p95 + 1 minute — which keeps ordinary run-to-run variation quiet and lets only genuinely stalled runs cross the line. I still think it could be an area where further tweaks are needed. As we roll out our own SLA, we will treat delayed for dynamical.org as a commitment rather than being driven by historical stats.

How it works, briefly

wxopticon is a set of stateless functions over a single append-only event log in object storage. There is no database — the log is the source of truth, and everything else (the dashboard, on-timedness, the readiness milestones) is a pure replay of it.

A lean detection scan runs every two minutes: it replays the log to find the runs still expected, probes their upstream locations, appends any new state transitions, and fans each new milestone out to subscribers. A separate summarize pass runs every five minutes to refresh the status feed and seed "delayed" signals. For products with a notification stream (e.g. AWS SNS), a continuous listener catches arrivals within seconds instead of waiting for the next scan.

Because the derived state is always a replay of the log, the same milestone can never be reported inconsistently across the webhook feed, the polling feed, and the dashboard — they are three views of one computation.

What a year of arrivals actually looks like

Because every arrival is in the log, we can replay a whole year of it. Over the last 365 days wxopticon recorded roughly 1.8 million file arrivals across about 8,700 runs of thirteen upstream feeds.

A run arrives over time, not in an instant. The moment a run starts and the moment it's complete can be hours apart, and the shape of that arrival looks different for every model.

Every file found over the last year, by forecast hour and how long after init time it landed; the dark line is the per-lead median. GFS trickles its 16-day run in over about two hours; HRRR's 48 hours land in a tight climb between roughly 50 and 110 minutes; GEFS races out to day 16, pauses, then delivers its 35-day tail in a burst almost a day later; AIFS drops its entire 15-day run in a single ~1-hour window. (A few files with rewritten upstream timestamps are clipped from view.)

This is why "ready" is a set of boundaries rather than one flag — a short-range consumer can start the instant the early leads land, hours before the archival tail shows up.

The feeds are punctualish! Measured from init time to the last file of the run, the median completion runs about 1h47m for HRRR, 3h37m for DWD's ICON-EU, 5h15m for AIFS, 5h22m for GFS, and a full ~26h for GEFS's 35-day run.

Time from init time to a complete run: median (filled) through the 95th to the 99th percentile (open circle). The striking part is how narrow most of these ranges are.

GFS completes within a 13-minute band from its median to its 99th percentile, run after run. Regularity is exactly what makes a learned next-run expectation meaningful. A "late" signal is useful because on-time is so consistent.

When a run is slow, it's the rare exception. The clearest case is ECMWF's AIFS-ENS: its median run finishes in 5h57m, but its slowest one percent of runs stretch past 13 hours. The value isn't the common run, which is boringly regular; it's catching the handful each year that stall, and the four runs — out of roughly 8,700, a 99.95% completion rate — that never completed.

One last thing the log settles: how far the cloud copy lags. Every NOAA model is disseminated through both NOAA's NOMADS server and a range of cloud providers (S3, GCS, Azure, and others, via NOAA's Open Data Dissemination program), and a consumer might read whichever it sees first (we, for example, blend our reads across sources in an attempt to optimize and roll with NOMADS rate limits).

For GFS, NOMADS is always first. Across roughly 75,000 files carrying the same forecast hour, the S3 copy trailed NOMADS by a median of about a minute and a half and never once led it — the cost of the extra ingest hop into the cloud. So the earliest a run is actually obtainable is its NOMADS timestamp, and that's the baseline wxopticon measures arrival against.

Subscribing: signed webhooks

If you can expose an inbound HTTP endpoint, webhooks are the lowish-latency path: seconds after arrival for NODD push products, at most one cycle for everything else. wxopticon POSTs you a signed JSON body the moment a run crosses a boundary you've subscribed to:

{
  "event_id": "noaa-gfs/external-noaa-gfs-aws/2026-06-10T06:00Z/complete",
  "group_id": "noaa-gfs",
  "product_id": "external-noaa-gfs-aws",
  "product_label": "NOAA GFS forecast (AWS)",
  "init_time": "2026-06-10T06:00:00Z",
  "kind": "complete",
  "occurred_at": "2026-06-10T11:12:04Z"
}

Every event carries a human-readable product_label (with the AWS/NOMADS source badge baked in), and progress/complete/delayed events add a lead_group and lead_group_label naming the horizon they concern — so a payload reads on its own without a lookup table.

Every delivery is signed (X-Wxopticon-Signature), retried with backoff on failure, and stable per boundary. Subscriptions are managed at status.dynamical.org/webhooks; access is currently allowlisted, so get in touch if you'd like to try it. You can even attach a small sandboxed Python function that runs against the just-arrived dataset and shapes the payload or filters out deliveries you don't want.

wxopticon also supports Slack-style incoming webhooks, so boundaries can land straight in a channel.

Prefer polling? The status feed

Not every consumer can accept inbound requests. For those of you who hear the soft footfall of the IT team plodding imperceptibly, but threateningly, in the distance -- coming closer, ever closer at the mention of webhooks, wxopticon publishes the same events as a single JSON file you fetch on your own schedule, with no subscription or auth:

https://assets.dynamical.org/wxopticon/feed.json

It's a product-keyed snapshot. Each product carries its most recent runs, and every run nests the same discrete events a webhook would deliver:

{
  "generated_at": "2026-06-10T09:45:00+00:00",
  "products": {
    "external-noaa-gfs-aws": {
      "label": "NOAA GFS (AWS)",
      "runs": [
        { "init_time": "2026-06-10T06:00:00+00:00", "status": "processing",
          "completion_pct": 0.62,
          "events": [
            { "event_id": "noaa-gfs/external-noaa-gfs-aws/2026-06-10T06:00Z/progress/f240",
              "group_id": "noaa-gfs", "product_id": "external-noaa-gfs-aws",
              "product_label": "NOAA GFS forecast (AWS)",
              "kind": "progress", "lead_group": "f240", "lead_group_label": "10d",
              "occurred_at": "2026-06-10T09:41:03Z" }
          ] }
      ]
    }
  }
}

The events are identical to what a webhook carries (minus the subscription id), so the client logic is the same: select the products you care about, and dedupe on event_id across polls. The file is refreshed each five-minute cycle and served with Cache-Control: max-age=5, stale-while-revalidate=10, so feel free to slam it.

Where to go from here

Watch the pipeline live: status.dynamical.org/pipeline
Poll the feed: assets.dynamical.org/wxopticon/feed.json
Manage webhook subscriptions: status.dynamical.org/webhooks

wxopticon is a living, but experimental piece of our infrastructure. If there's a source you'd like us to watch, or a boundary you wish you could subscribe to, let us know.

AIFS ENS and validation reports

2026-06-01T00:00:00Z

I was tempted, but alas Gefs (short for Gefsfry) did not make the cut for the name of my newborn son. He's here and everyone is healthy - so I'm getting back to it. Some catch-up:

ECMWF AIFS ENS forecast

The ensemble sibling to AIFS Single is now in the catalog. ECMWF AIFS ENS: 15-day global forecasts at 0.25°, 6-hourly steps, 51 members, updating every 6 hours. Archive starts 2025-07-02.

Validation reports

Did you know Alden has knuckle tats? They spell

 ┌─┬─┬─┬─┐   ┌─┬─┬─┬─┐
 │R│I│G│O│   │R│O│U│S│
 └┬┴┬┴┬┴┬┘   └┬┴┬┴┬┴┬┘
  │ │ │ │     │ │ │ │
══╧═╧═╧═╧══ ══╧═╧═╧═╧══

To live up to the knuckles, we are adding in-depth validation reports to each catalog entry. For each variable, you get three views: a null-fraction trace over time (so you can see any gaps in the archive at a glance), side-by-side spatial maps comparing our dataset against analysis at a sample timestamp, and time-series comparisons at two random geographic points. The spatial plots also include a value-distribution histogram so you can see at a glance whether the numbers look right.

Find it linked from each catalog page, or go directly — e.g. ECMWF AIFS ENS forecast validation.

We are in the process of backfilling these for existing data products and we would LOVE your feedback!

These are one layer of a broader validation process that runs automatically as each dataset updates. The goal is to give us (and you) enough visibility into the data that you can trust it before you build on it.

On the AWS blog

If you've ever wondered why we started with the data products that make up our catalog, check out my blog post on the AWS Public Sector Blog: What if swapping your weather model was boring?

AWS Open Data has been an incredible partner as we've gotten dynamical.org up and running!

Enjoy the weather,

Time 2 Chunk (2.0), DWD ICON-EU, status for weather

2026-04-24T00:00:00Z

Listen, my second child is arriving any minute now. This transmission will be brief. I haven't decided if I will name him Poseidon, Gefs, or Turbulence.

Zarr v3 deprecation

We're all-in on Icechunk. Our existing Zarr v3 datasets will continue updating for 90 days, at which point we will sunset them. Consider them deprecated and update to use our Icechunk Zarrs by July 23, 2026.

Speaking of which

The new access pattern:

Via dynamical-catalog

import dynamical_catalog

# Open a dataset as an xarray Dataset via its Icechunk repository
ds = dynamical_catalog.open("noaa-gfs-forecast")

To more easily ensure the right set of dependencies and to make an increasingly variable set of opening incantations more accessible, we made a python library to list and open datasets.

import dynamical_catalog

# List all available datasets
dynamical_catalog.list()

# Get the underlying Zarr store if you want even more control
store = dynamical_catalog.get_store("noaa-gfs-forecast")
ds = xr.open_zarr(store)

Via STAC

We are STAC-maxxers now. https://stac.dynamical.org/catalog.json contains ALL of the data used to create our documentation and drive the dynamical-catalog library. Point your robots at it for all the documentation they need to get going. STAC-as-skill.

ICON-EU

Deutscher Wetterdienst's ICON-EU is now a cloud-optimized, live-updating Icechunk 2.0 Zarr. High-resolution 5-day forecasts over Europe, nested from DWD's global ICON model, at the 00/06/12/18 UTC cycles.

What makes this one extra fun: as far as we can tell, no other public archive of ICON-EU exists. DWD only keeps a short rolling window on their own servers, so we'll keep growing the archive from here.

Huge thanks to:

Jack Kelly from Open Climate Fix for getting this off the ground and hoarding forecasts to kickstart the archive and to the Patrick J McGovern Foundation for supporting their work.
The DWD Open Data program
Source Cooperative for hosting the upstream grib archive
AWS Open Data for the Icechunk storage

Find it at:

Enjoy the weather,

dynamical.org - icechunk 2.0 upgrade cometh

2026-04-17T00:00:00Z

Short one today — a quick housekeeping note for anyone pulling from our icechunk data products.

Icechunk 2 is out! If you're using our datasets via Icechunk, please update your Python library before Thursday April 24:

pip install icechunk --upgrade

uv add icechunk --upgrade

Note that icechunk 2 requires Python 3.12 or higher. If you're on an older runtime, now's the time to bump it.

We'll be targeting the v2 format going forward.

Enjoy the weather,

Rain, radar, and reckless optimism.

2026-04-01T00:00:00Z

Salutations, wxers. To balance out all the April 1 corporate tomfoolery I'm going to be less of a goofball today. Let's see if I can keep it STRICTLY BUSINESS.

Lots to report since my last update. The dynamical(.org) core is a-running.

ECMWF AIFS Single forecast

The AI weather model from the European Centre has entered the catalog. ECMWF's AIFS Single — 15-day global forecasts at 0.25°, 6-hourly steps, live updating. This is the single-trace (deterministic-ish?) run of ECMWF's machine learning forecast system.

And of course, don't miss the release trailer: Youtube | LinkedIn

NOAA MRMS

New CONUS RADAR dataset! MRMS is Multi-Radar Multi-Sensor — NOAA's merged radar and gauge precipitation analysis over CONUS, hourly, back to late 2014. If you've ever wanted to answer "how much did it actually rain here, at this exact hour" without downloading 47 files from 3 different FTP servers... this is for you.

Release trailer: Youtube | LinkedIn

CHILL

We passed the 100M monthly request mark. And then we passed the 200M monthly request mark. Which is either a sign that the data is useful or that three of you are running massive autoresearch.md deployments.

Either way, it's exciting and we appreciate everyone who's sent us what you're building!

dynamical-catalog python library

We're testing a simple Python library: dynamical-catalog. The idea is to make discovery and loading even simpler — browse the catalog, pick a dataset, and get an xarray Dataset or Store back without copy-pasting URLs. We'll handle picking the best Store backend. It's early days. If you try it, let us know what breaks.

Enjoy the weather,

Warmer temps, datasets heating up.

2026-03-10T00:00:00Z

It's Spring and unlike hobbits (I'm reading LOTR), who would be thinking about gardens and second breakfast and whether Farmer Maggot's mushroom crop will come in early, we are staring at grib files. Just kidding, we are walking outside enjoying the warm weather ... while talking on the phone about grib files.

We've been busy with a whole load of updates.

GFS analysis is live

Our old noaa-gfs-analysis-hourly was our "Hello World" — 4 variables, static, Zarr v2.

It has been given a lovely retirement party and replaced with the new GFS analysis.

Live! (not static)
21 variables (up from 4)
Available in Zarr v3 and Icechunk


Spatial domain	Global
Spatial resolution	0.25°
Time resolution	1 hour

HRRR analysis back to 2014

Four more sweet years. High Resolution Rapid Refresh has been retroactively retained through repeated re-retrieval - and now, re-released for a more uh... robust retrospective.

It previously started in 2018 and now reaches back to 2014. The early years (HRRR v1/v2 era) have more missing source files — NaN values where data are unavailable — but that's the archaeology tax.

Icechunk note: we've updated to v0.2.0. Please point your code at noaa-hrrr-analysis/v0.2.0.icechunk/. We'll be stopping updates to v0.1.0 on 2026-03-19 and removing the archive after that.

More variables

By popular demand we've been adding variables to existing datasets. ECMWF, GEFS, and HRRR have all received new variables recently. We drew the line at will_the_sports_be_cancelled_2m and ideal_dog_walk_surface but almost everything else is in there. Check the catalog pages for the full list.

Experimental ASOS GeoParquet

I tried (and tried [and tried]) to find a good [citation needed] open source of obs once ISD was decomissioned and GHCNh made me want to cry enough tears to register on the precip gauge. Alas.

Those tireless little weather stations at airports across the US are available as a GeoParquet dataset. 1940ish to present. This is our first non-model dataset and it's 100% experimental, but it updates every hour and it's what the scorecard is built on.

DWD ICON-EU grib archive

As a first step towards cloud-optimized ICON-EU Zarrs, we've started archiving DWD's gribs on Source Coop — DWD only maintains a short rolling window on their servers, so someone had to hoard them. Data hoarding is our North Star!

Big thanks to Jack Kelly of Open Climate Fix for leading this effort and to the Patrick J McGovern Foundation for supporting the work.

Enjoy the weather

January cornucopia: AMS | HRRR analysis | more

2026-01-23T00:00:00Z

What say the pantheon of forecasts? Shall AMS travel be a disaster?

As the flakes that fall thick upon a winter's day, when Zeus is minded to snow and to display these his arrows to humankind - he lulls the wind to rest, and snows hour after hour till he has buried the tops of the high mountains, the headlands that jut into the sea, the grassy plains, and the tilled fields of men...

Find Alden @ AMS

"What Happens When Weather Data is Easy to Use?". Join Alden during the Environmental Data Access and Its Future: Open Data for Open Science Session on Monday, January 26 at 2:15 PM CT to explore how accessible weather data transforms research, operations, and decision-making across the atmospheric sciences community. Shoot him an email if you want to meet up and chat!

HRRR analysis

HRRR analysis (2018-09-16 to present) is now available in the catalog, including the Icechunk preview.

Launch trailer


Spatial domain	Continental United States
Spatial resolution	3 km
Time domain	2018-09-16 00:00:00 UTC to Present
Time resolution	1 hour

Icechunk catalog parity party

Icechunk previews for our datasets are now listed on the Registry of Open Data on AWS! Check out the Icechunk example usage in our docs for IFS ENS, GEFS, GFS, and HRRR forecasts to get started. With the addition of GEFS forecast and analysis we now have an identical, live updating Icechunk Zarr for all live updating standard Zarr v3 datasets in our catalog!

Earthmover Marketplace

The "Union of Organizations Concerned That Earth Data Is Generally Too Hard To Work With" (still workshopping this title, but it has a good ring to it) just became more powerful - Earthmover launched their Marketplace. (Congrats team)

And the aforementioned dynamical.org Icechunk catalog is within!

Steering committee minutes

Shrouded in mystery and intrigue, the steering committee guides the helm from the shadows. To lift the veil is to challenge authority itself, and... hm? What's that? The meeting notes are public?! Oh.

Shout out to all the contributors to cf_xarray which both a) now works great with our datasets and b) is part of our test suite to ensure all new datasets are CF compliant from the start. Thanks to our steering committee for guiding us towards more interoperable standards.

Stay warm, safe travels.

Preview of dynamical.org Icechunk Zarrs are now listed on the Registry of Open Data on AWS!

2026-01-08T00:00:00Z

Check out the Icechunk example usage in our docs for IFS ENS, GFS, and HRRR forecasts to get started or browse these datasets on the Registry of Open Data on AWS.

Why Icechunk?

Consistent reads every time. Updating a standard Zarr while users are reading it requires extreme care. Icechunk handles atomic updates correctly, eliminating rare race conditions that can occur during standard Zarr updates.
Reliability and throughput. Icechunk is developing features to offer long-term stable dataset URLs. This provides direct access to underlying chunk data for high-throughput reads while allowing us to provide a permanent URL. Note: This feature will be part of the upcoming Icechunk 2.0, so when released dynamical.org will require a change to the access URL.
Performance with default Python tooling. While the storage formats are similar, the Icechunk client is currently faster than default libraries used for traditional Zarrs in python. You get maximum efficiency (comparable to specialized libraries like Obstore) by default.

When Icechunk 2 is released we will make two small breaking changes (and notify you about them here):

URLs will be updated to stable, long-term URLs
Icechunk client version 2 will be required

Special thanks to the AWS Open Data Sponsorship Program for supporting the storage and distribution of these datasets.

🟢 ECMWF IFS ENS live and updating

2025-11-20T00:00:00Z

Colder temps may be here (where I live, at least), but ensembles are so hot right now. Can we agree on that? No? Ha ha get it!?

As of this week The European Centre's IFS ENS forecast is live and updating in our catalog. See the catalog for init times, spatiotemporal domain, and the variable list.

I suppose with that HRRR "release trailer" I've signed myself up for something completely unsustainable -- but until that time comes, when the catalog entries cometh faster than I can splice together sci fi video clips... -- ENS has a release trailer too.

Icechunk

Icechunk versions are available to select groups for testing (and will likely be our default in the future). If you want to try them out please respond.

I've actually been a little surprised at the lack of Icechunk inquiries. Are y'all even drinking the cutting edge data format koolaide?!

Sending from an airship above the clouds. Enjoy the weather. End transmission.

Wake up, babe. A new zarr just dropped.

2025-09-29T00:00:00Z

The last time I wrote you all, the summer sweltered, determinism was still alive (now it is dead, and only 11 AI weather papers had been published in Nature (now the total is up to 188,732).

If I keep it up y'all are never going to take me seriously. But lo, some seriously new stuff!

HRRR go brrr

HRRR forecasts are live and updating! Check out the release trailer (lol) here.

Give it a try! It'll be like that time in high school when you played too much Elder Scrolls Oblivion during a long weekend, and when you finally stepped outside into nature you thought to yourself "woah, these graphics are incredible."

A massive thanks to Alden, Tony, Alex and Sam for getting this one out there!

Icechunk support

We've also launched ~~Earthchunk~~ Icechunk versions of the GFS and HRRR forecasts. It's internal only until we sort out a few final details, but if you'd feel the need for speed and are prerelease-tolerant let us know and we can hook you up.

weathering

In case you missed it we have a podcast now! We so far have covered:

The limits of predictability
End-to-end data-driven weather prediction (Aardvark Weather)
The Hybrid Approach (NeuralGCN)
Efficient AI ensemble forecasts (ECMWF AIFS-CRPS, DeepMind FGN)

5 papers down, 188,727 to go! No new AI weather papers allowed.

Scorecard

Hey, wouldn't you know it. Having stuff in this here catalog makes it easier to build things. I was tired of stats against reanalysis as "truth" so Alden and I hacked out a very simple scorecard. When I say simple, I mean simple. For now it's just RMSE for 2m temp and precip. Of course, a few weeks after this went live, ISD-lite was killed in favor of GHCN so I gotta flip that over to keep things live-updating! URGH.

Enjoy the weather.

🟢 GFS is live

2025-07-02T00:00:00Z

GFS forecasts are live

GFS is GEFS' less ambiguous older sibling. The archive is processed and the updater is chugging: 4 years of 16 day forecasts, hourly step, released 4 times a day. GFS might be wrong, but at least it's confident!

A note from Alden, Chef de cuisine

Tonight, I've prepared for you a dataset I think you will enjoy. Check out the example notebook linked in the docs to make your own interactive animation of an atmospheric river with buttery smooth hourly resolution. The way atmospheric rivers move reminds me of how rivers move on land, only on a time scale of hours not years. Served as a semifreddo with a peach compote.

Steering committee

The dynamical.org steering committee met and talked about [1] Data and [2] Things About Data Beyond My Comprehension. Read the summary notes

podcast

Did you catch episode number two of weathering? This time we talked about Aardvark Weather. As Marta put it: "Where does the end begin? Where does the end end? And what the HELL is happening in the middle?"

Check it out. Like (in your heart) and subscribe (to our obsessions).

introducing a new pod; GFS forecasts incoming

2025-06-14T00:00:00Z

Not even my proprietary UniverseCast™ Reality Ensemble Forecast⁺ could have predicted this. The most unlikely of scenarios: A podcast? Yes, a podcast. And it's called Weathering.

We are trying to make sense of the current wave of progress occurring in weather forecasting.

Each episode, we focus on a paper or group of papers and try to take a intertextual approach -- clicking on mental hyperlinks to help us understand and put into context. The format is inspired by the only two podcasts I regularly listen to (Future of Coding and Jed Sundwall's Techs on Texts).

Episode 001: The limits of predictability
Our maiden voyage dives into a fresh-off-the-press paper from the University of Washington: Testing the Limit of Atmospheric Predictability With a Machine Learning Weather Model.

Check it out, and let us know if there's a paper you think we should cover next.

In normal dynamical.org news, operational GFS forecasts are landing in the catalog imminently. We previously did an analysis as our first demo, so this will be a nice upgrade to tie the bow on the GEFS/GFS family.

After that, the catalog will be expanding rapidly! Stay tuned.

Have a good weekend,
MM

⁺ Fictional? Or extremely limited access to drive scarcity? You decide.

GEFS analysis is live!

2025-04-29T00:00:00Z

Well, technically it's been live for a few days. Check out our new GEFS analysis, which is derived from the GEFS forecast catalog item.

Both are live updating, and the analysis was designed to work easily with the forecast dataset. For example, you can develop your algorithm/model/sentient AI television weatherperson on the analysis, and then deploy it live with the forecast dataset. COOL!

25+ years of data
Live updating
Performance optimized for time series queries

Striving to create weather data experiences as satisfying as when you peel citrus in one contiguous piece. You know what I mean :)

Enjoy the weather,

1 Million Requests!

2025-03-20T00:00:00Z

While modest on the scale of Computers, 1M is not bad for our baby’s first week in the world. The gigabytes are flying and we’ve received requests from 51 countries over the last few days. No single person has downloaded the whole dataset, so thank you but also WHAT ARE YOU WAITING FOR?!

A reminder that the GEFS dataset is Zarr v3, so don’t forget to update your Xarray>=2025.1.2 which includes v3 support!

Let’s make Phase 1 happen (and how you can help)

With GEFS complete, we are already working on our next slate of catalog additions. Our Phase 1 will include the following datasets:

Forecasts

NOAA GEFS, 35 day, 1x day (we did it)
NOAA HRRR (Tony is on it!)
NOAA GFS (Dan is on it!)
NOAA GEFS, 10 day, 4x day
ECMWF AIFS
ECMWF ENS, 15 day, 1x day
ECCC HRDPS

Analyses

NOAA GEFS (Alden is on it!)
ECMWF AIFS
ECMWF ENS
NOAA HRRR
NOAA GFS
ECCC HRDPS

Three ways you, dear reader, can help:

[1] Financial support:

We estimate that making Phase 1 operational & publicly accessible will run around $100,000 per year with the cost growing by approximately 30% year over year as the datasets get inevitably larger. Pennies compared to the value, in my humble opinion!

So… we are seeking our first cohort of sponsors to make these forecasts and analyses a reality. If you belong to an organization that benefits from, and is able to support our work, we’d love to hear from you. Reply to this email to learn more about our plans for fiscal sponsorship.

[2] Publicize these datasets as they’re released! Tell your coworkers, post on whatever social media we are using these days, tell your parents over dinner, teach your toddler about scones and pancakes.

[3] Write code? Not quite yet but soon. We are testing to see if our open source code so far is sufficiently clear to have other folks pitch in and write reformatters for new datasets. If all goes well we may solicit help from your big brain.

Help us out, and especially [1] so that there are NO BLOCKERS, ONLY WEATHER.

Stay dry,

About GEFS-fing Time & Introducing Our Steering Committee

2025-03-12T00:00:00Z

Weatherlings, the time has come for our first non-demo release. Weighing in at 115TB compressed and 815TB (!) uncompressed, GEFS ain’t messing around.

Last year, we launched dynamical.org into the world and watched as hundreds of weather data nerds emerged (sublimated?) from the woodwork. Our community has been growing steadily, united by a shared mission to make weather data Spark Joy.

🌎 Introducing NOAA's Global Ensemble Forecast System (GEFS) 35-Day Forecasts!

I am thrilled to announce the latest addition to Dynamical’s catalog: NOAA's Global Ensemble Forecast System (GEFS) 35-day forecasts!

📦 What's in the box?

The Specs:

Parameter	Details
Spatial Domain	Global
Spatial Resolution	0-240 hours: 0.25 degrees (~20km), 243-840 hours: 0.5 degrees (~40km)
Time Domain	Forecasts initialized 2020-10-01 00:00:00 UTC to Present
Time Resolution	Forecasts initialized every 24 hours
Forecast Domain	Forecast lead time 0-840 hours (0-35 days) ahead
Forecast Resolution	Forecast step 0-240 hours: 3-hourly, 243-840 hours: 6-hourly

Best of all, it's operationally updated with minimal latency and available in the Zarr v3 format that plants crave:

import xarray as xr

ds = xr.open_dataset("https://data.dynamical.org/noaa/gefs/forecast-35-day/latest.zarr?email=optional@email.com")

We are already using this new dataset for R&D at Upstream Tech, and we can’t wait to hear how you use it! We've included documentation and example notebooks in the catalog entry to get you started. If you hit snags or have feedback, drop us a line at feedback@dynamical.org.

We’ll be holding our first dynamical.org office hours Thursday Mar 20, 2025 at 10-11am PDT / 1-2pm EDT / 5-6pm UTC. Join this Google Meet to ask questions or give feedback.

Introducing Our Steering Committee!

We're excited to announce the formation of the dynamical.org Steering Committee! This dream team will:

Advise on project priorities (Which datasets to zarr-ify next?)
Bridge the gap between our open-source code and real-world applications
Shape the future of how we access and analyze weather and climate data

Current Members:

Alden Keefe Sampson – dynamical.org & Upstream Tech
Jack Kelly – Open Climate Fix
Jake Zwart – United States Geological Survey
Joe Hamman – Earthmover
Stephan Hoyer – Xarray

By bringing together this collective expertise, we're ensuring that Dynamical.org remains relevant, impactful, and true to its mission.

Shine on (lol),

Alden @ AMS; GEFS zarrs?!

2025-01-10T00:00:00Z

CALLING ALL WEATHER DWEEBS! Bracing for a winter storm here in Nashville, where I moved to from Boston in order to escape winter storms. 3 to 7 inches they say - good thing I brought the XC skis!

dynamical.org @ AMS

From the look of it the vast majority of our subscribers will be at AMS. We are excited to see you there!

Alden will be presenting at the 105th Annual Meeting of the American Meteorological Society (AMS) along with our first fiscal sponsor at Hydro-Québec! Their talk, titled “Fast, Easy, Open: Sustaining Live Updating, Analysis Ready, Cloud Optimized Weather Data Archives,” will highlight the vision behind dynamical.org and the drive to create and sustain ARCO weather datasets.

🗓️ When: Tuesday, January 14, 9:15 AM
📍 Where: 355 (New Orleans Ernest N. Morial Convention Center)

If you’re at AMS, find Alden and Upstream Tech's Phil Butcher.

Catalog Updates: GEFS is Almost Here!

We’ve made some big strides toward releasing a complete archive and live-updating GEFS dataset to the catalog:

The GEFS reformatter can now create an archive of years of data super quickly, thanks to tons of parallelism.
The dataset is not only comprehensive but also a dream to work with:

>>> ds.u100
<xarray.DataArray 'u100' (init_time: 1, ensemble_member: 4, lead_time: 5,
                          latitude: 721, longitude: 1440)> Size: 83MB
dask.array<open_dataset-u100, shape=(1, 4, 5, 721, 1440), dtype=float32, chunksize=(1, 4, 5, 73, 72), chunktype=numpy.ndarray>
Coordinates:
  * ensemble_member           (ensemble_member) uint16 8B 0 1 2 3
    expected_forecast_length  (init_time) timedelta64[ns] 8B 35 days
    ingested_forecast_length  (init_time) timedelta64[ns] 8B 35 days
  * init_time                 (init_time) datetime64[ns] 8B 2025-01-08
  * latitude                  (latitude) float64 6kB 90.0 89.75 ... -89.75 -90.0
  * lead_time                 (lead_time) timedelta64[ns] 40B 0 days 00:00:00...
  * longitude                 (longitude) float64 12kB -180.0 -179.8 ... 179.8
    valid_time                (init_time, lead_time) datetime64[ns] 40B 2025-...
Attributes:
    long_name:      100 metre U wind component
    short_name:     u100
    standard_name:  eastward_wind
    step_type:      instant
    units:          m/s

Woah nelly, check out those attributes!

We’re testing out operational updates in our compute cluster THIS WEEK. Once that’s complete, we’ll be ready to push live, real-time updates to the catalog. So close, we can almost taste the churros/pancakes/scones (ha ha ha get it).

What We Are Reading

Zarr Python 3 released! We’re especially excited about the included chunk sharding extension, which will enable a much wider range of efficient read patterns with a single, optimized dataset. Read more here
Earthmover debuted Icechunk, a new open storage engine for Zarr datasets. It supports simple and correct updates to datasets that are being read from and written to simultaneously (e.g., live-updated weather), along with great features like dataset versioning and improved read performance. We’re already experimenting with it and considering support once the storage format stabilizes. Check it out here
China is planning to become a weather superpower by investing in advanced meteorological technologies and infrastructure to enhance its weather forecasting capabilities. Bloomberg

Finally - we'll be launching our steering committee this quarter. Thanks to all those who responded with interest in helping us achieve our mission.

Catch you in the clouds.

September update

2024-09-26T00:00:00Z

Ahoy weatherheads! It’s been a busy few months for us as we shift gears from dreaming to building.

Catalog updates

Our dataset for NOAA GFS analysis has getting some awesome usage! Thanks to everyone who has tested it out.
We are prepping to launch NOAA GEFS forecasts as the second entry to dynamical.org/catalog. It will include a large historical forecast record and update in real-time. Dealing with GEFS in its standard format has, in the past, broken my spirit; and I suppose it’s part of why we are so passionate about making weather data easier to work with! So… thank you GEFS? Excited for this one. GEFS ensemble average downward long-wave radiation flux.
We plan to fast-follow GEFS with GFS (forecasts vs our current analysis) and HRRR from NOAA as well as GDPS and HRDPS from ECCC (Environment and Climate Change Canada). As part of these releases we will be announcing our first catalog sponsor!

GFS analysis feedback

A hearty thank you to everyone who filled out our GFS survey. Common themes included the value of live updating and forecast data, and a clear preference for reading data with python + xarray. That aligned with our expectations (wonderful) and is reflected in our roadmap above. We did see more desire for long history (re)analysis data than we had anticipated -- we'll be moving up a true reanalysis dataset in priority thanks to this feedback. Reply to this email if there's a specific reanalysis you'd like!

Assembling our Steering Committee

We are officially launching our steering committee, which will be comprised of industry users and experts in the technical and scientific domain. The committee will meet quarterly to evaluate and influence the catalog and open-source roadmap and advise potential collaborations and partnerships.

If you’re interested in joining or nominating someone else, please respond to this email or email marshall@dynamical.org!

What we’ve been reading

My souls, how the wind did scream along! And every second or two there’d come a glare that lit up the white-caps for a half a mile around, and you’d see the islands looking dusty through the rain, and the trees thrashing around in the wind; then comes a h-whack!—bum! bum! bumble-umble-um-bum-bum-bum-bum—and the thunder would go rumbling and grumbling away, and quit—and then rip comes another flash and another sockdolager.

I’m reading Adventures of Huckleberry Finn, which has some stellar meteorological descriptions. Should we petition NOAA to make their detailed forecasts use “h-wack!-bum! bum!” and “sockdolager?” I think yes.

We enjoyed Dr. Jack Kelly’s post on the vision this community shares, where he outlined a complimentary route to solving major pain points. Thanks for the dynamical.org shoutouts, Jack! Definitely check it out here: https://openclimatefix.org/post/lazy-loading-making-it-easier-to-access-vast-datasets-of-weather-satellite-data

We were thrilled to see our pals at Brightband announce themselves to the world. The field of AI meteorological forecasts is progressing at quite the clip - three cheers for more open source models! We also enjoyed Zeus AI's cool work synthesizing multiple, sparse data sources into a single gap filled analysis state.

Enjoy the weather!

Let there be zarr: GFS hourly analysis is live

2024-07-09T00:00:00Z

Last week, we announced dynamical.org to the world. The response was unexpected*: in just seven days, over a hundred nerds registered themselves as testers with many more following along for the ride.

Apparently, we are not alone in our frustration with yesterday's weather data. Together we can create a future of well-organized data about our most beloved chaotic system!

Today, we are excited to announce the first entry into dynamical’s catalog: an hourly GFS analysis from 2015-01-15 to present.

Testers! Fire up your rigs - in the catalog entry above, we've included documentation and an example notebook. If you run into any issues, email us at feedback@dynamical.org. As our very first release, we’re expecting there will be lots of additions and refinement based on community input.

Our pals at Source Cooperative are generously providing storage for this first catalog entry. Their team is wonderful and their mission is badass - check them and the broader Radiant Earth organization out.

Enjoy the weather^!

* or expected, depending on how long you've been working with yesterday's weather data

^ courtesy of Alex M.

Introducing dynamical.org

2024-06-25T00:00:00Z

Introducing dynamical!

Thank you for being the first to follow our journey making weather and climate data more accessible. We are humbled by the response. Our mission clearly strikes a chord - weather data today is a pain to work with.

Over one hundred people (!?) signed up to test the first versions of the data we endeavor to release. And that’s without having spilled the beans on the best parts 😉

So what’s the plan? First, check out Alden’s fantastic introduction to dynamical.org (video link). Second, read about how we are designing dynamical to be a long-term steward and rock-solid foundation on which to build.

We are aiming to release our first public test next week in collaboration with Source Cooperative: a GFS analysis with a long historical record. If you are interested in testing this out and haven’t already told us, reply to this email. We plan to fast-follow with documentation and a data catalog with our roadmap.

Enjoy the weather!

MM + AKS

P.S. Please forward this email to folks working with weather data and consider amplifying our announcement on linkedin / fedi!