Alden @ AMS; GEFS zarrs?!

CALLING ALL WEATHER DWEEBS! Bracing for a winter storm here in Nashville, where I moved to from Boston in order to escape winter storms. 3 to 7 inches they say - good thing I brought the XC skis!

dynamical.org @ AMS

From the look of it the vast majority of our subscribers will be at AMS. We are excited to see you there!

Alden will be presenting at the 105th Annual Meeting of the American Meteorological Society (AMS) along with our first fiscal sponsor at Hydro-Québec! Their talk, titled “Fast, Easy, Open: Sustaining Live Updating, Analysis Ready, Cloud Optimized Weather Data Archives,” will highlight the vision behind dynamical.org and the drive to create and sustain ARCO weather datasets.

🗓️ When: Tuesday, January 14, 9:15 AM
📍 Where: 355 (New Orleans Ernest N. Morial Convention Center)

If you’re at AMS, find Alden and Upstream Tech's Phil Butcher.

Catalog Updates: GEFS is Almost Here!

We’ve made some big strides toward releasing a complete archive and live-updating GEFS dataset to the catalog:

The GEFS reformatter can now create an archive of years of data super quickly, thanks to tons of parallelism.
The dataset is not only comprehensive but also a dream to work with:

>>> ds.u100
<xarray.DataArray 'u100' (init_time: 1, ensemble_member: 4, lead_time: 5,
                          latitude: 721, longitude: 1440)> Size: 83MB
dask.array<open_dataset-u100, shape=(1, 4, 5, 721, 1440), dtype=float32, chunksize=(1, 4, 5, 73, 72), chunktype=numpy.ndarray>
Coordinates:
  * ensemble_member           (ensemble_member) uint16 8B 0 1 2 3
    expected_forecast_length  (init_time) timedelta64[ns] 8B 35 days
    ingested_forecast_length  (init_time) timedelta64[ns] 8B 35 days
  * init_time                 (init_time) datetime64[ns] 8B 2025-01-08
  * latitude                  (latitude) float64 6kB 90.0 89.75 ... -89.75 -90.0
  * lead_time                 (lead_time) timedelta64[ns] 40B 0 days 00:00:00...
  * longitude                 (longitude) float64 12kB -180.0 -179.8 ... 179.8
    valid_time                (init_time, lead_time) datetime64[ns] 40B 2025-...
Attributes:
    long_name:      100 metre U wind component
    short_name:     u100
    standard_name:  eastward_wind
    step_type:      instant
    units:          m/s

Woah nelly, check out those attributes!

We’re testing out operational updates in our compute cluster THIS WEEK. Once that’s complete, we’ll be ready to push live, real-time updates to the catalog. So close, we can almost taste the churros/pancakes/scones (ha ha ha get it).

What We Are Reading

Zarr Python 3 released! We’re especially excited about the included chunk sharding extension, which will enable a much wider range of efficient read patterns with a single, optimized dataset. Read more here
Earthmover debuted Icechunk, a new open storage engine for Zarr datasets. It supports simple and correct updates to datasets that are being read from and written to simultaneously (e.g., live-updated weather), along with great features like dataset versioning and improved read performance. We’re already experimenting with it and considering support once the storage format stabilizes. Check it out here
China is planning to become a weather superpower by investing in advanced meteorological technologies and infrastructure to enhance its weather forecasting capabilities. Bloomberg

Finally - we'll be launching our steering committee this quarter. Thanks to all those who responded with interest in helping us achieve our mission.

Catch you in the clouds.