Steering Committee Meeting, Q4 2025

Steering Committee Meeting, Q4 2025

Date: December 3, 2025

Members present:

• Jacob Zwart
• Joe Hamman
• Jack Kelly
• Stephan Hoyer
• Alden Keefe Sampson

Community engagement

Steering committee members discussed expanding our outreach to help new weather data user communities become aware of our work. We’ve begun this outreach work based on Steering committee suggestions and conferences such as AGU and AMS have been helpful as well, albeit US focused. If you know a community who you think might find our data useful, please email us to let us know! We’d love to work with you to make examples relevant to new users.

Usage statistics

November (the last complete month) was our highest request month yet at 80M at ~50TB served. We continue to see a skewed distribution in requested variables, with temperature, precipitation and wind leading across datasets. Usage has become more global, while still led by access from the US and Europe.

Steering Committee Meeting, Q3 2025

Steering Committee Meeting, Q3 2025

Date: September 3, 2025

Members present:

• Joe Hamman
• Jack Kelly
• Stephan Hoyer
• Alden Keefe Sampson

Our September steering committee meeting focused on dataset format and metadata conventions.

Dataset format

Zarr is a fantastic format for gridded weather data. But updating a Zarr dataset while others are reading from it at the same time is hard to do exactly right. Icechunk is a storage engine for Zarr that addresses this issue, among other benefits. Steering committee members weighed adopting Icechunk for dynamical datasets and decided on the following plan: Dynamical will start offering Icechunk versions of our datasets alongside standard Zarr datasets to gather feedback and iron out issues [We reached this point as of Dec 2025]. When we’re confident the user experience using our Icechunk datasets is equal or better than our existing Zarrs we will start publishing new datasets solely in Icechunk. Following significant heads up through our newsletter we will cease updating existing Zarr datasets and eventually remove them. We believe this change will set our datasets up well for the long-term from a user experience, correctness and performance perspective.

Metadata conventions

The more consistent and interoperable our datasets are the better. Adopting CF Conventions in our datasets was the unanimous suggestion of steering committee members and is now a priority for our dataset development. We discussed how AI coding tools can help expedite this process and the cf_xarray library can help validate conformance.

Steering Committee Meeting, Q2 2025

Steering Committee Meeting, Q2 2025

Date: June 4, 2025

Members present:

• Jacob Zwart
• Joe Hamman
• Jack Kelly
• Alden Keefe Sampson

Dataset Usage Statistics

Usage has been growing steadily in recent months from over 10M requests/month in March to almost 40M in May. Roughly 45TB of data was served in May.

• GFS and GEFS analysis datasets see the most requests.
• Temperature, wind, and precipitation are by far the most accessed variables.
• Usage is global, with the majority coming from North America and Europe.

AWS Open Data Program Acceptance

dynamical.org was accepted into AWS’s Open Data Program, which provides:

• Cost-free S3 storage and hosting for a significant portion of dynamical’s datasets
• Support duration of at least two years

This will serve as a bridge to publish more datasets as dynamical secures fiscal support.

Data Proxy Considerations

Discussion focused on the current data.dynamical.org proxy:

Benefits:

• Allows switching to lowest cost/most available storage with no user impact
• Enables usage analytics

Downsides:

• May not scale well with growing usage

A potential improvement:

• Metadata requests go to data.dynamical.org
• Heavy data requests go directly to cloud object storage
• This approach will be enabled via Icechunk

External Contributor Strategy

Key considerations for managing external contributors:

• QA/QC process will require close involvement from dynamical.org engineers at first
• Introduce controls and staging areas to manage contributions
• Begin with a small-scale experiment (trusted contributor + single dataset — in progress)
• Use automated quality checks to compare new datasets against existing ones

Evaluating Icechunk

Icechunk is a new format built on Zarr. Committee discussed whether dynamical should adopt it.

Benefits:

• Ensures datasets being read from and written to simultaneously remain correct
• Avoids edge cases and operational issues in current Zarr workarounds

Transition Plan:

• Maintain continuity by publishing in both formats temporarily
• Begin with double-write tests