The National Weather Service's Data Storage Problem

The Public Sector Takes on AI Forecasting

Public sector officials are enthusiastic about using AI to track weather patterns and predict climate events. Alan Gerard, a former NOAA and NWS scientist who now runs the Substack newsletter Balanced Weather, describes the early work in this field as “extremely promising.” 

Much of the momentum, though, has come from the private sector. About two years ago, public agencies began to view AI  “as a challenge to the way they had been doing things,” Morgan says. Tech giants like Google had developed AI models that could match — and, in some cases, surpass — NOAA’s physics-based models.

Even more enticing, these models ran faster on far less computing power. “They could run these models that [usually] take hundreds of processors a few hours … [in] under a minute, on a single processor,” Morgan explains. It can be expensive to train AI on the front end, but Morgan says this reduced computational footprint can save incredible amounts of money over time.

Inspired by these advances, NOAA and other government agencies developed their own AI-fueled projects. Earlier this year, NOAA launched Project EAGLE, an AI forecast system designed to help scientists test and refine AI models for global ensemble forecasting. Early projects include HRRR-Cast and WoFSCast — AI versions of NOAA flagship systems for predicting regional rainfall and severe weather, respectively.

Gerard says HRRR-Cast has shown real promise west of the Rocky Mountains, where radar coverage isn’t as robust. By blending NOAA’s physics-based precipitation estimation with AI techniques, it provides “crucial” insight for water management and flood forecasting, Gerard explains.

WoFSCast, built on Google DeepMind’s GraphCast framework and NOAA’s Warn on Forecast (WoFS), targets severe weather at the local level. Like WoFS, it runs an ensemble forecast — meaning it consists of a collection of models operating together. According to Morgan, WoFS has issued alerts further in advance, but WoFSCast has the potential to amplify the ensemble. It could work even faster, with a smaller computing footprint, allowing more storms to be tracked at once. 

“That just has massive implications for what we could potentially do,” Gerard says. “We could be running ensemble systems that have thousands of members very quickly — and really improve forecasts at all timescales.”

Both HRRR-Cast and WoFSCast aim to solve a longstanding AI weakness: local forecasting. While physics-based models excel at analyzing fine details, AI models have historically shown the most promise with global forecasting projects. This is one theory for why most AI models failed to predict the severity of the recent Texas floods. But, according to Morgan, “WoFSCast is one place where actually, globally, NOAA is leading in this effort.”

Those efforts, however, now face new threats. The Trump administration’s proposed budget cuts for the agency recommend closing NOAA’s National Severe Storms Laboratory (NSSL) and the Office of Ocean Atmospheric Research (OAR) labs, both of which are developing AI projects. While there’s been significant pushback on these proposals from Congress, Morgan says if the funding cuts move forward, projects like WoFSCast would be “lost, or slowed considerably.” 

Can Weather Forecasting Be Privatized?

The federal funding cuts initially caught public attention this spring when the Department of Commerce threatened to cancel an Amazon Web Services contract that would have caused several NOAA agency websites to go dark. As Alejandra Borunda reported for NPR: “Farmers looking for seasonal drought forecasts would encounter a dead link. Coastal managers looking for ways to protect their communities from high-tide flooding wouldn’t find mapping tools to help them figure out where to focus their efforts.”

The crisis was temporarily averted when the contract was renewed, but a new proposed budget continues to threaten public-private partnerships like the AWS deal. These contracts play a vital role in ensuring innovative AI weather forecasting research continues both in the public and private sector.

Many in the industry believe proposed federal cuts represent an attempt to privatize weather forecasting. Some media outlets have even argued that Trump appointees might financially benefit from this shift. But the private and public sectors have much to gain from working together. While the private sector is currently more advanced than the public sector when it comes to AI forecasting, many private AI companies use publicly available data from NOAA to model their forecasts.

In fact, that open data is how many private sector companies first started creating AI weather forecasting models. In 2018, the European Centre for Medium-Range Weather Forecasts (ECMWF) released ERA5, a set of analyses constructed from all the hourly weather observations available since 1940. The purpose of the ERA5 was to make the data publicly available for scientific research. But private-sector researchers quickly realized it was also perfect training data for AI models, says Morgan.

Since then, most private companies have continued to rely on public resources, such as NOAA and NWS weather modeling, data and other products. For example, WindBorne — a Palo Alto-based start-up that uses AI to better predict events like heat waves further in advance — has been using data sets released by public weather agencies to train its deep learning software.

That reliance may shift, especially as the cost of some technologies fall and more organizations choose to build their own infrastructure. “The private sector is making a lot of progress on developing their own resources, particularly when it comes to [AI] modeling,” Gerard says, citing companies ClimaVision and Tomorrow.io, which are beginning to deploy their own radar systems and satellite observational platforms, respectively. Gerard points out that Google and Amazon already have far more computing resources than the federal government will likely ever develop.

Still, Gerard worries about the equity ramifications if weather forecasting is fully privatized, and he’s not alone. Pedro David Espinoza, founder of PDE Ventures and Pan Peru USA, is excited about the potential of AI for climate risk mapping and weather tracking in the private sector but emphasizes the need for accessibility. “The high cost of proprietary AI tools can definitely take a toll when it comes to SMBs, or small communities of color,” he says. If we're intentional about democratizing access, he adds, public-private partnerships could improve outcomes for both rural and under-resourced communities.

Fighting for the Cloud

While the White House’s proposed NOAA budget doesn’t reduce the money allocated for forecasting directly, it cuts funding for satellite and radar systems that collect this data and threatens contracts like the AWS deal, which help make public data readily available. Gerard, who utilized the AWS contract at NSSL, says he thinks the proposed budget cuts could “jeopardize” future agreements like these.

Morgan agrees: “My understanding is that if people want to get anything over $100K approved, it has to go through the Commerce Secretary’s office.” Others say NOAA’s research infrastructure is necessary for these sorts of agreements to move forward at all.

The AWS contract offers NOAA some obvious benefits. Namely, it’s been helping the agency improve daily operations for their physics-based models, such as HAFS — allowing for more simultaneous regional predictions at very high resolution in an era of increasing flooding events and heavy rainfall. “Frankly, NOAA lacks significant on-premises compute to do the job that they have to do,” says Morgan. “Being able to do high-resolution models in real time may even [help] people protect some of their property.”

Critically, the AWS contract ensures continued collaboration between the private and public sector beyond the agreement itself — by continuing to make NWS data available for private companies to iterate on.

Remember the ERA5? That data was perfect for training AI in perhaps all ways but one: It wasn’t cloud-friendly, so companies had to waste time downloading and creating their own versions. Since ERA5’s release in 2018, agreements like NOAA’s Open Data Dissemination partnership with Microsoft, AWS and Google make it possible to download this data easily and quickly for free. Agencies like NOAA get better data storage, while private companies like Google get access to a valuable data set they can use for training and developing all kinds of weather products, both AI-driven and otherwise.

It’s a type of public-private collaboration that many in the industry believe will drive innovation. “Having access to the cloud, having reliable contracts, and [having] the data out there I think only encourages more people in the community to help support this effort,” says Morgan. “If you lock this data up and it’s not accessible again, that innovation is just going to be wiped out.” He also points out that the standard of products could fall if a respected authority like NOAA is no longer involved.

Morgan’s not the only one putting faith into these partnerships. Gerard says the very existence of some of these agreements mitigates some of his equity concerns, because it would allow the public sector to continue providing services to more under-resourced communities, while the private sector can continue to innovate. “It’s a win-win,” he says.

So what happens if the data disappears? Experts in the sector are torn. While Morgan and Gerard worry innovation may falter, equity gaps may deepen, and the standard of the products developed could worsen, Espinoza thinks it could be an opportunity for growth. “Crises are the best times for people, enterprises, groups or corporations to innovate,” he says.

Espinoza notes that the proposed NOAA budget raises concerns, especially given the impact of climate change, but says he remains hopeful: “I tend to be an optimist in the sense that…there's going to be more motivation for climate tech startups to take the lead.”

And that may be the crux of the issue: As demand for accurate, real-time forecasts surges, the future of weather prediction depends on whether public agencies and private innovators can work together. If they fail, the risks will only grow. But if they succeed, forecasting could become faster, cheaper and more reliable — giving communities the information they need to prepare for a changing climate.