🟢 GFS is live

GFS forecasts are live

GFS is GEFS' less ambiguous older sibling. The archive is processed and the updater is chugging: 4 years of 16 day forecasts, hourly step, released 4 times a day. GFS might be wrong, but at least it's confident!

A note from Alden, Chef de cuisine

Tonight, I've prepared for you a dataset I think you will enjoy. Check out the example notebook linked in the docs to make your own interactive animation of an atmospheric river with buttery smooth hourly resolution. The way atmospheric rivers move reminds me of how rivers move on land, only on a time scale of hours not years. Served as a semifreddo with a peach compote.

Steering committee

The dynamical.org steering committee met and talked about [1] Data and [2] Things About Data Beyond My Comprehension. Read the summary notes

podcast

Did you catch episode number two of weathering? This time we talked about Aardvark Weather. As Marta put it: "Where does the end begin? Where does the end end? And what the HELL is happening in the middle?"

Check it out. Like (in your heart) and subscribe (to our obsessions).

introducing a new pod; GFS forecasts incoming

Not even my proprietary UniverseCast™ Reality Ensemble Forecast⁺ could have predicted this. The most unlikely of scenarios: A podcast? Yes, a podcast. And it's called Weathering.

We are trying to make sense of the current wave of progress occurring in weather forecasting.

Each episode, we focus on a paper or group of papers and try to take a intertextual approach -- clicking on mental hyperlinks to help us understand and put into context. The format is inspired by the only two podcasts I regularly listen to (Future of Coding and Jed Sundwall's Techs on Texts).

Episode 001: The limits of predictability
Our maiden voyage dives into a fresh-off-the-press paper from the University of Washington: Testing the Limit of Atmospheric Predictability With a Machine Learning Weather Model.

Check it out, and let us know if there's a paper you think we should cover next.

In normal dynamical.org news, operational GFS forecasts are landing in the catalog imminently. We previously did an analysis as our first demo, so this will be a nice upgrade to tie the bow on the GEFS/GFS family.

After that, the catalog will be expanding rapidly! Stay tuned.

Have a good weekend,
MM

⁺ Fictional? Or extremely limited access to drive scarcity? You decide.

GEFS analysis is live!

Well, technically it's been live for a few days. Check out our new GEFS analysis, which is derived from the GEFS forecast catalog item.

Both are live updating, and the analysis was designed to work easily with the forecast dataset. For example, you can develop your algorithm/model/sentient AI television weatherperson on the analysis, and then deploy it live with the forecast dataset. COOL!

• 25+ years of data
• Live updating
• Performance optimized for time series queries

Striving to create weather data experiences as satisfying as when you peel citrus in one contiguous piece. You know what I mean :)

Enjoy the weather,

MM

1 Million Requests!

While modest on the scale of Computers, 1M is not bad for our baby’s first week in the world. The gigabytes are flying and we’ve received requests from 51 countries over the last few days. No single person has downloaded the whole dataset, so thank you but also WHAT ARE YOU WAITING FOR?!

A reminder that the GEFS dataset is Zarr v3, so don’t forget to update your Xarray>=2025.1.2 which includes v3 support!

Let’s make Phase 1 happen (and how you can help)

With GEFS complete, we are already working on our next slate of catalog additions. Our Phase 1 will include the following datasets:

Forecasts

• NOAA GEFS, 35 day, 1x day (we did it)
• NOAA HRRR (Tony is on it!)
• NOAA GFS (Dan is on it!)
• NOAA GEFS, 10 day, 4x day
• ECMWF AIFS
• ECMWF ENS, 15 day, 1x day
• ECCC HRDPS

Analyses

• NOAA GEFS (Alden is on it!)
• ECMWF AIFS
• ECMWF ENS
• NOAA HRRR
• NOAA GFS
• ECCC HRDPS

Three ways you, dear reader, can help:

[1] Financial support:

We estimate that making Phase 1 operational & publicly accessible will run around $100,000 per year with the cost growing by approximately 30% year over year as the datasets get inevitably larger. Pennies compared to the value, in my humble opinion!

So… we are seeking our first cohort of sponsors to make these forecasts and analyses a reality. If you belong to an organization that benefits from, and is able to support our work, we’d love to hear from you. Reply to this email to learn more about our plans for fiscal sponsorship.

[2] Publicize these datasets as they’re released! Tell your coworkers, post on whatever social media we are using these days, tell your parents over dinner, teach your toddler about scones and pancakes.

[3] Write code? Not quite yet but soon. We are testing to see if our open source code so far is sufficiently clear to have other folks pitch in and write reformatters for new datasets. If all goes well we may solicit help from your big brain.

Help us out, and especially [1] so that there are NO BLOCKERS, ONLY WEATHER.

Stay dry,

MM

About GEFS-fing Time & Introducing Our Steering Committee

Weatherlings, the time has come for our first non-demo release. Weighing in at 115TB compressed and 815TB (!) uncompressed, GEFS ain’t messing around.

Last year, we launched dynamical.org into the world and watched as hundreds of weather data nerds emerged (sublimated?) from the woodwork. Our community has been growing steadily, united by a shared mission to make weather data Spark Joy.

🌎 Introducing NOAA's Global Ensemble Forecast System (GEFS) 35-Day Forecasts!

I am thrilled to announce the latest addition to Dynamical’s catalog: NOAA's Global Ensemble Forecast System (GEFS) 35-day forecasts!

📦 What's in the box?

The Specs:

Best of all, it's operationally updated with minimal latency and available in the Zarr v3 format that plants crave:

import xarray as xr

ds = xr.open_dataset("https://data.dynamical.org/noaa/gefs/forecast-35-day/[email protected]")

We are already using this new dataset for R&D at Upstream Tech, and we can’t wait to hear how you use it! We've included documentation and example notebooks in the catalog entry to get you started. If you hit snags or have feedback, drop us a line at [email protected].

We’ll be holding our first dynamical.org office hours Thursday Mar 20, 2025 at 10-11am PDT / 1-2pm EDT / 5-6pm UTC. Join this Google Meet to ask questions or give feedback.

Introducing Our Steering Committee!

We're excited to announce the formation of the dynamical.org Steering Committee! This dream team will:

• Advise on project priorities (Which datasets to zarr-ify next?)
• Bridge the gap between our open-source code and real-world applications
• Shape the future of how we access and analyze weather and climate data

Current Members:

• Alden Keefe Sampson – dynamical.org & Upstream Tech
• Jack Kelly – Open Climate Fix
• Jake Zwart – United States Geological Survey
• Joe Hamman – Earthmover
• Stephan Hoyer – Xarray

By bringing together this collective expertise, we're ensuring that Dynamical.org remains relevant, impactful, and true to its mission.

Shine on (lol),

MM

Alden @ AMS; GEFS zarrs?!

CALLING ALL WEATHER DWEEBS! Bracing for a winter storm here in Nashville, where I moved to from Boston in order to escape winter storms. 3 to 7 inches they say - good thing I brought the XC skis!

dynamical.org @ AMS

From the look of it the vast majority of our subscribers will be at AMS. We are excited to see you there!

Alden will be presenting at the 105th Annual Meeting of the American Meteorological Society (AMS) along with our first fiscal sponsor at Hydro-Québec! Their talk, titled “Fast, Easy, Open: Sustaining Live Updating, Analysis Ready, Cloud Optimized Weather Data Archives,” will highlight the vision behind dynamical.org and the drive to create and sustain ARCO weather datasets.

• 🗓️ When: Tuesday, January 14, 9:15 AM
• 📍 Where: 355 (New Orleans Ernest N. Morial Convention Center)

If you’re at AMS, find Alden and Upstream Tech's Phil Butcher.

Catalog Updates: GEFS is Almost Here!

We’ve made some big strides toward releasing a complete archive and live-updating GEFS dataset to the catalog:

• The GEFS reformatter can now create an archive of years of data super quickly, thanks to tons of parallelism.
• The dataset is not only comprehensive but also a dream to work with:

>>> ds.u100
<xarray.DataArray 'u100' (init_time: 1, ensemble_member: 4, lead_time: 5,
                          latitude: 721, longitude: 1440)> Size: 83MB
dask.array<open_dataset-u100, shape=(1, 4, 5, 721, 1440), dtype=float32, chunksize=(1, 4, 5, 73, 72), chunktype=numpy.ndarray>
Coordinates:
  * ensemble_member           (ensemble_member) uint16 8B 0 1 2 3
    expected_forecast_length  (init_time) timedelta64[ns] 8B 35 days
    ingested_forecast_length  (init_time) timedelta64[ns] 8B 35 days
  * init_time                 (init_time) datetime64[ns] 8B 2025-01-08
  * latitude                  (latitude) float64 6kB 90.0 89.75 ... -89.75 -90.0
  * lead_time                 (lead_time) timedelta64[ns] 40B 0 days 00:00:00...
  * longitude                 (longitude) float64 12kB -180.0 -179.8 ... 179.8
    valid_time                (init_time, lead_time) datetime64[ns] 40B 2025-...
Attributes:
    long_name:      100 metre U wind component
    short_name:     u100
    standard_name:  eastward_wind
    step_type:      instant
    units:          m/s

Woah nelly, check out those attributes!

We’re testing out operational updates in our compute cluster THIS WEEK. Once that’s complete, we’ll be ready to push live, real-time updates to the catalog. So close, we can almost taste the churros/pancakes/scones (ha ha ha get it).

What We Are Reading

• Zarr Python 3 released! We’re especially excited about the included chunk sharding extension, which will enable a much wider range of efficient read patterns with a single, optimized dataset. Read more here
• Earthmover debuted Icechunk, a new open storage engine for Zarr datasets. It supports simple and correct updates to datasets that are being read from and written to simultaneously (e.g., live-updated weather), along with great features like dataset versioning and improved read performance. We’re already experimenting with it and considering support once the storage format stabilizes. Check it out here
• China is planning to become a weather superpower by investing in advanced meteorological technologies and infrastructure to enhance its weather forecasting capabilities. Bloomberg

Finally - we'll be launching our steering committee this quarter. Thanks to all those who responded with interest in helping us achieve our mission.

Catch you in the clouds.

MM

September update

Ahoy weatherheads! It’s been a busy few months for us as we shift gears from dreaming to building.

Catalog updates

• Our dataset for NOAA GFS analysis has getting some awesome usage! Thanks to everyone who has tested it out.
• We are prepping to launch NOAA GEFS forecasts as the second entry to dynamical.org/catalog. It will include a large historical forecast record and update in real-time. Dealing with GEFS in its standard format has, in the past, broken my spirit; and I suppose it’s part of why we are so passionate about making weather data easier to work with! So… thank you GEFS? Excited for this one. GEFS ensemble average downward long-wave radiation flux.
• We plan to fast-follow GEFS with GFS (forecasts vs our current analysis) and HRRR from NOAA as well as GDPS and HRDPS from ECCC (Environment and Climate Change Canada). As part of these releases we will be announcing our first catalog sponsor!

GFS analysis feedback

A hearty thank you to everyone who filled out our GFS survey. Common themes included the value of live updating and forecast data, and a clear preference for reading data with python + xarray. That aligned with our expectations (wonderful) and is reflected in our roadmap above. We did see more desire for long history (re)analysis data than we had anticipated -- we'll be moving up a true reanalysis dataset in priority thanks to this feedback. Reply to this email if there's a specific reanalysis you'd like!

Assembling our Steering Committee

We are officially launching our steering committee, which will be comprised of industry users and experts in the technical and scientific domain. The committee will meet quarterly to evaluate and influence the catalog and open-source roadmap and advise potential collaborations and partnerships.

If you’re interested in joining or nominating someone else, please respond to this email or email [email protected]!

What we’ve been reading

My souls, how the wind did scream along! And every second or two there’d come a glare that lit up the white-caps for a half a mile around, and you’d see the islands looking dusty through the rain, and the trees thrashing around in the wind; then comes a h-whack!—bum! bum! bumble-umble-um-bum-bum-bum-bum—and the thunder would go rumbling and grumbling away, and quit—and then rip comes another flash and another sockdolager.

I’m reading Adventures of Huckleberry Finn, which has some stellar meteorological descriptions. Should we petition NOAA to make their detailed forecasts use “h-wack!-bum! bum!” and “sockdolager?” I think yes.

We enjoyed Dr. Jack Kelly’s post on the vision this community shares, where he outlined a complimentary route to solving major pain points. Thanks for the dynamical.org shoutouts, Jack! Definitely check it out here: https://openclimatefix.org/post/lazy-loading-making-it-easier-to-access-vast-datasets-of-weather-satellite-data

We were thrilled to see our pals at Brightband announce themselves to the world. The field of AI meteorological forecasts is progressing at quite the clip - three cheers for more open source models! We also enjoyed Zeus AI's cool work synthesizing multiple, sparse data sources into a single gap filled analysis state.

Enjoy the weather!

Let there be zarr: GFS hourly analysis is live

Last week, we announced dynamical.org to the world. The response was unexpected*: in just seven days, over a hundred nerds registered themselves as testers with many more following along for the ride.

Apparently, we are not alone in our frustration with yesterday's weather data. Together we can create a future of well-organized data about our most beloved chaotic system!

Today, we are excited to announce the first entry into dynamical’s catalog: an hourly GFS analysis from 2015-01-15 to present.

Testers! Fire up your rigs - in the catalog entry above, we've included documentation and an example notebook. If you run into any issues, email us at [email protected]. As our very first release, we’re expecting there will be lots of additions and refinement based on community input.

Our pals at Source Cooperative are generously providing storage for this first catalog entry. Their team is wonderful and their mission is badass - check them and the broader Radiant Earth organization out.

Enjoy the weather^!

MM

* or expected, depending on how long you've been working with yesterday's weather data

^ courtesy of Alex M.

Introducing dynamical.org

Introducing dynamical!

Thank you for being the first to follow our journey making weather and climate data more accessible. We are humbled by the response. Our mission clearly strikes a chord - weather data today is a pain to work with.

Over one hundred people (!?) signed up to test the first versions of the data we endeavor to release. And that’s without having spilled the beans on the best parts 😉

So what’s the plan? First, check out Alden’s fantastic introduction to dynamical.org (video link). Second, read about how we are designing dynamical to be a long-term steward and rock-solid foundation on which to build.

We are aiming to release our first public test next week in collaboration with Source Cooperative: a GFS analysis with a long historical record. If you are interested in testing this out and haven’t already told us, reply to this email. We plan to fast-follow with documentation and a data catalog with our roadmap.

Enjoy the weather!

MM + AKS

P.S. Please forward this email to folks working with weather data and consider amplifying our announcement on linkedin / fedi!