Low-latency HRRR, with all variables and levels, now in the catalog
Are we living inside a Bostrom-esque simulation? Does that simulation exist inside another simulation?
There's only one way to find out: let's add increasing layers of derivative abstraction on already-digital signals that inadequately capture what the sheeple call "reality." Yeah?! Yeah.
If anyone has Morpheus' phone number let him know I'm ready to be picked up on the Nebuchadnezzar.
noaa-hrrr-forecast-48-hour-virtual is now available
It is our first virtual dataset. It's low-latency, optimized for spatial access (i.e. good for maps) and has all variables and vertical levels. Try it out.
We now have the machinery for rigorous, high-performance virtual data products. You will see many more of these, and they will be as thoughtfully designed as our materialized data products.
You can read more about our design process in Alden's lab note: Having it all: virtual and materialized data products.
Gettin' serious. status.dynamical.org is live
We stood up status.dynamical.org to give insight into both:
- dynamical.org system uptime and
- dynamical.org and source agency forecast production pipeline
The latter initially focuses on upstream observability (e.g. GEFS dissemination on NOAA NOMADS) and our data products (materialized and virtual) will land shortly. We just wanted to make sure they were heckin' fast first ;)
We later added webhooks for subscribers to be notified on dissemination milestones or delays. We're still tuning what counts as "delayed", but overall it's been an invaluable tool for our team (and others) to learn more about the dissemination process.
You can read the lab notes here: Knowing the moment a forecast is ready.
This level of rigor is helping us prepare to offer an SLA to organizations who need additional support and latency/uptime assurance. Don't worry, free public access stays free. Email me at [email protected] if you're interested in learning more about our SLA and/or supporting the sustainable operations & growth of dynamical.org <3
Validation reports
The per-dataset validation reports we started in June got richer. I mean just look at this absolutely ridiculous one for the noaa-hrrr-forecast-48-hour-virtual data product.
A cleaner catalog and site
- Catalog pages now document chunk and shard sizes per dataset — element counts, coordinate spans, and uncompressed sizes — so you (or your robot) can align your reads to the grid.
- We reorganized the site around research areas and projects, refreshed the homepage, and made the whole thing legible to search engines and LLMs (
llms.txt, structured metadata, sitemap). Lab notes inspired by the inimitable Ink & Switch.