Using past and forecast data from multiple numerical weather models can be combined using ML to achieve better forecast skill than any individual model. Because each model is physically bound, the resulting ML model should be stable.
So not "the weather on 25 December 2022 was such and such" but rather "on 20 December 2022 the forecast for 25 December 2022 was such and such"
It was super easy and the responses are very fast.
Based on historical data!
I just hit the daily limit on the second request at https://climate-api.open-meteo.com/v1/climate
I see the limit for non-commercial use should be "less than 10.000 daily API calls". Technically 2 is less than 10.000, I know, but still I decided to drop you a comment. :)
or 1 request every ~9 seconds.
Maybe you just didn't space them enough.
It's a pleasure being able to use it in https://weathergraph.app
Enjoy the data directly from the source producing them.
American weather agency: https://www.nco.ncep.noaa.gov/pmb/products/gfs/
European weather agency: https://www.ecmwf.int/en/forecasts/datasets/open-data
The data’s not necessarily east to work with, but it’s all there, and you get all the forecast ensembles (potential forecasted weather paths) too
Does anyone have a compare this API with the latest API we have here?
With Open-Meteo, I'm working to integrate more weather models, offering access not only to current forecasts but also past data. For Europe and South-East Asia, high-resolution models from 7 different weather services improve forecast accuracy compared to global models. The data covers not only common weather variables like temperature, wind, and precipitation but also includes information on wind at higher altitudes, solar radiation forecasts, and soil properties.
Using custom compression methods, large historical weather datasets like ERA5 are compressed from 20 TB to 4 TB, making them accessible through a time-series API. All data is stored in local files; no database set-up required. If you're interested in creating your own weather API, Docker images are provided, and you can download open data from NOAA GFS or other weather models.
I am exploiting on the homogeneity of gridded data. In a 2D field, calculating the data position for a graphical coordinate is straightforward. Once you add time as a third dimension, you can pick any timestamp at any point on earth. To optimize read speed, all time steps are stored sequentially on disk in a rotated/transposed OLAP cube.
Although the data now consists of millions of floating-point values without accompanying attributes like timestamps or geographical coordinates, the storage requirements are still high. Open-Meteo chunks data into small portions, each covering 10 locations and 2 weeks of data. Each block is individually compressed using an optimized compression scheme.
While this process isn't groundbreaking and is supported by file systems like NetCDF, Zarr, or HDF5, the challenge lies in efficiently working with multiple weather models and updating data with each new weather model run every few hours.
You can find more information here: https://openmeteo.substack.com/i/64601201/how-data-are-store...
I'm used to working with different weather stations, e.g. seeing different snowfall prediction at the bottom of a mountain, halfway up, and at the top, where the coordinates are quite similar.
Open-Meteo focuses on providing access to weather data for single locations or small areas. If you look at data for coastal areas, forecast and past weather data will show severe winds. Storm tracks or maps are not available, but might be implemented in the future.
> For inputs, GraphCast requires just two sets of data: the state of the weather 6 hours ago, and the current state of the weather. The model then predicts the weather 6 hours in the future. This process can then be rolled forward in 6-hour increments to provide state-of-the-art forecasts up to 10 days in advance.
Initializing directly from, say, geostationary and LEO satellite data with complementary surface station observations - skipping the assimilation step entirely - is clearly where this revolution is headed, but it's very important to explicitly note that we're not there yet (even in a research capacity).
Furthermore, a predictive model is not working with a complete picture of the weather, but rather some limited-resolution measurements. So, even ignoring non-weather, there may be local weather phenomena detected at time t0, escaping detection at time t1, but still affecting weather at time t2.
In any case, GraphCast is a 10-day global model, whereas MetNet is a 24-hour regional model, among other differences.
Some of these forecasts are also downloadable as data, but I don't know whether GraphCast is. Alternatively, if forecasts have a big economic value to you, loading latest ERA5 and the model code, and running it yourself should be relatively trivial? (I'm no expert on this, but I think that is ECMWF's aim, to distribute some of the models and initial states as easily runnable.)
> ... with the current version being the largest we can practically fit under current engineering constraints, but which have potential to scale much further in the future with greater compute resources and higher resolution data.
I can't wait to see how far other people take this.
It is a kind of iterative refinement on the data that supercomputers produce — it doesn’t supplant supercomputers. In fact the paper calls out that it has a hard dependency on the output produced by supercomputers.
Dask-jobqueue https://jobqueue.dask.org/ :
> provides cluster managers for PBS, SLURM, LSF, SGE and other [HPC supercomputer] resource managers
Helpful tools for this work: Dask-labextension, DaskML, CuPY, SymPy's lambdify(), Parquet, Arrow
GFS: Global Forecast System: https://en.wikipedia.org/wiki/Global_Forecast_System
TIL about Raspberry-NOAA and pywws in researching and summarizing for a comment on "Nrsc5: Receive NRSC-5 digital radio stations using an RTL-SDR dongle" (2023) https://news.ycombinator.com/item?id=38158091
I didn't read the paper but the linked post seems to say otherwise? It mentions it used the supercomputer output to impute data during training. But for prediction it just needs:
> For inputs, GraphCast requires just two sets of data: the state of the weather 6 hours ago, and the current state of the weather. The model then predicts the weather 6 hours in the future. This process can then be rolled forward in 6-hour increments to provide state-of-the-art forecasts up to 10 days in advance.
Full disclosure: I think DeepMind often publish these bombastic headlines about their models which often don't live up to their hype, or at least that was my personal experience. They have a good PR team, anyway.
There are really convenient apps that show an animated map with radar data of rain, historical data + prediction (typically).
The prediction is always completely bonkers.
You can eyeball it better.
No wonder "AI" can improve that. Even linear extrapolation is better.
Yes, local rain prediction is a different thing from global forecasting.
[1] https://www.buienradar.nl [2] https://www.meteoschweiz.admin.ch/service-und-publikationen/...
Weather forecasting has been moving focus towards ensembles to account for uncertainty in forecasts. I see a future of large ensembles of ML models being ran hourly incorporating the latest measurements
Another very cool application could incorporate generative modeling. Inject a bit of uncertainty in a some observations and study how the manifold of forecast outputs changes... ultimately, you could tackle things like studying the sensitivity of forecast uncertainty for, say, a tropical cyclone or nor'easter relative to targeted observations. Imagine a tool where you could optimize where a Global Hawk should drop rawindsondes over the Pacific Ocean to maximally decrease forecast uncertainty for a big winter storm impacting New England...
We may not be able to engineer the weather anytime soon, but in the next few years we may have a new type of crystal ball for anticipating its nuances with far more fidelity than ever before.
But it is exciting that they are able to recognize patterns in multi year and produce medium term forecasts.
Some comments here suggest this replaces supercomputers models. This would a wrong conclusion.It does not (the paper explicitly states this). It uses their output as input data.
https://www.ecmwf.int/en/forecasts/documentation-and-support
You could numerically render a 4k scene at 120FPS at extreme cost, or you could render a 2k scene at 60FPS, then feed that to DLSS to get a close-enough approximation of the former at enormous energy and hardware savings.
So there's radial-coordinate voxels that represent a low resolution of the physical state of the entire atmosphere.
I'm all for better weather data, it's quite critical up in the mountains, so that's why my question about how reliable it is in life&death situations.
Seems like it's better at predicting extreme weather events
The global models tend to consistently miss in places that have local weather "quirks" - which is why local forecasters tend to do better than, say, accuweather, where it just posts what the models say.
Local forecasters might have learned over time that, in early Autumn, the models tend to overpredict rain, and so when they give their forecasts, they'll tweak the predictions based on the model tendencies.
National weather institutions sometimes do this, since they don't have the resources to run a massive supercomputer model.
Apple moved from using The Weather Channel to their own forecasting a year ago [1].
Using AI to produce better weather forecasts is exactly the kind of thing that is right up Google's alley -- I'm very happy to see this, and can't wait for this to get built into our weather apps.
1: https://www.theverge.com/2020/3/31/21201666/apple-acquires-w...
AFAIK they don't have their own forecasting models, they use same data sources as everyone else: https://support.apple.com/en-us/HT211777
My experience in this space is that I was first employee at Solcast building a live 'nowcast' system for 4+ years (left ~2021) targeting solar radiation and cloud opacity initially, but expanding into all aspects of weather, focusing on the use of the newer generation of satellites, but also heavily using NWP models like ECMWF. Last I knew,nowcasts were made in minutes on a decent size cluster of systems, and has been shown in various studies and comparisons to produce extremely accurate data (This article claims 'the best' without links which is weird..), be interesting on how many TPUsv4 were used to produce these forecasts and how quickly? Solcast used ML as a part of their systems, but when it comes down to it, there is a lot more operationally to producing accurate and reliable forecasts, eg it would be arrogant to say the least to switch from something like ECMWF to this black box anytime soon.
Something I said as just before I left Solcast was that their biggest competition would come from Amazon/Google/Microsoft and not other incumbent weather companies. They have some really smart modelers, but its hard to compete with big tech resources. I believe Amazon has been acquiring power usage IoT related companies over the past few years, I can see AI heavily moving into that space as well.. for better or worse.
Its like a shotgun shooting a wall. You can pretty accurately predict the area of the shot, but its incredibly hard to place where exactly each shot in that area will land.
All the fjords and mountains and lakes in norway really make it hard, to precicesly model it. And I think they strongly and chaotically influence the weather in sweden as well.
Also, there are way more people living in central europe, so probably more effort is spend on them.
Ten years ago, the weather forecast was so unreliable that I just assumed anything could happen on a given day, no matter the season. Frequently it would be unable to even tell you whether it was currently raining, and my heuristic for next day forecast instead was to just assume the weather would be the same as today.
Nowadays I find the next day forecasts are nearly always accurate and hourly precipitation forecasts are good enough that I can plan my cycles and walks around them.
(A friend of mine who moved to the Boston area about ten years after the event once told me that she had never seen a northern city in which so many people headed home from work if they saw so much as a snowflake.)
There are a limited number of weather stations producing measurements, and a limited "cell size" for being able to calculate forecasts quickly enough, and geographical factors that aren't perfectly accounted for in models.
AI is able to help substantially with all of these -- from interpolation to computational complexity to geography effects.
I would imagine we probably have a solid mathematical model of how weather behaves, so given enough resources to measure and calculate, could you, in theory, predict the daily weather going 10 years into the future? Or is there something inherently “random” there?
But this isn't a "weather forecast." Weather forecasting is an initial value problem - you care a great deal about how the weather will evolve from the current atmospheric conditions. Precisely because weather is a result of what happens in this complex, 3D fluid atmosphere surrounding the Earth, it happens that small changes in those initial conditions can have a very big impact on the forecast on relatively short time-periods - as little as 6-12 hours. Small perturbations grow into larger ones and feedback across spatial scales. Ultimately, by day ~3-7, you wind up with a very different atmospheric state than what you'd have if you undid those small changes in the initial conditions.
This is the essence of what "chaos" means in the context of weather prediction; we can't perfectly know the initial conditions we feed into the model, so over some relatively short time, the "model world" will start to look very different than the "real world." Even if we had perfect models - capable of representing all the physics in the atmosphere - we'd still have this issue as long as we had to imperfectly sample the atmosphere for our initial conditions.
So weather isn't inherently "unpredictable." And in fact, by running lots of weather models simultaneously with slightly perturbed initial conditions, we can suss out this uncertainty and improve our estimate of the forecast weather. In fact, this is what's so exciting to meteorologists about the new AI models - they're so much cheaper to run that we can much more effectively explore this uncertainty in initial conditions, which will indirectly lead to improved forecasts.
Say you had a massive array of billions of perfect sensors in different locations, and had all the computing power to process this data, would an N year daily forecast then be a solved problem?
For the sake of the argument I’m ignoring ”external” factors that could affect the weather (e.g meteors hitting earth, changes in man-made pollution, etc)
> Present understanding is that this chaotic behavior limits accurate forecasts to about 14 days even with accurate input data and a flawless model. In addition, the partial differential equations used in the model need to be supplemented with parameterizations for solar radiation, moist processes (clouds and precipitation), heat exchange, soil, vegetation, surface water, and the effects of terrain.
I think there is a hope that DL models wont have this problem.
But the real problem is chaos - which says that even with perfect data, unless you also have computations with infinite precision and time/spatial/temperature/pressure/etc resolution, eventually you wind up far from reality.
The use of ensembles reduces the effect of chaos a bit, although they tend to smooth it out - so your broad pattern 12 days out might be more accurately forecast than without them, but the weather at your house may not be.
Iterative DL models tend to smooth it faster, according to a recent paper.
The Allen Institute has worked on it for a while, and has hired quite a few PhDs (https://allenai.org/climate-modeling).
Similar to the DeepMind effort, the ACE ML model that AI2+others developed is really just looking for parity with physical models at this stage. It looks like they've almost achieved this, with similar massive improvements in compute time + resource needs.
https://techcrunch.com/2023/11/14/courtesy-of-ai-weather-for...
They're also good at prioritizing outcomes, rather than other stuff.
Unknown what licensing options ECMWF offers for Era5, but to use this model in any live fashion, I think one is probably going to need a small fortune. Maybe some other dataset can be adapted (likely at great pain)...
To use the data in live fashion I think you would need to get license from ECMWF...
I think that only some variables from the HRES are free, but not 100% sure.
Any pointers?
the resolution, while seemingly impressive, is very imprecise compared to the SOTA in the theoretical modelling side.
this discredits the computational claims made by the paper for me. i understand that the current simulations can go down to meter scale, but i wonder what the compuational requirements are when you calculate for this resolution.
If it is possible, then I will try using the sensor to measure my velocity at some place where I live, and I can run the model and see how the results look like. I don't know if it's going to accurately predict the future or within a 10% error bar range.
To the best of my knowledge, poor weather (especially wind shear/microbursts) are one of the most dangerous things possible in aviation. Is there any chance, or plans, to implement this in the current weather radars in planes?
Any way to run this at even higher resolution, like 1 km? Could this resolve terrain forced effects like lenticular clouds on mountain tops?
ECMWF publishes a tool that can help bootstrap simple inference runs with different AI models [1] (they have plugins for several). You could write a tool that re-maps a GDAS analysis to "look like" ERA-5 or IFS analysis, and then try feeding it into GraphCast. But YMMV if the integration is stable or not - models like PanguWx do not work off-the-shelf with this approach.
Is it inevitable that all market alpha gets mined by AI?
Does GraphCast come close to them?
You'd think predicting the weather is mostly a matter of fast computation. The physical rules are well understood, so to get a better estimate use a finer mesh in your finite element computation and use a smaller time scale in estimating your differential equations.
Neural networks are notoriously bad at exact approximation. I mean you can never beat a calculator when the issue is doing calculations.
So apparently the AI found some shortcut for doing the actual computational work. That is also surprising as weather is a chaotic system. Shortcuts should not exist.
Long story short, I don't get what's going on here.
[1] http://www.incompleteideas.net/IncIdeas/BitterLesson.html
You mean by remembering piles and piles of example data and interpolating between it.
Nope. They're constantly updating these models with really finnicky things like cloud nucleation rates that differ depending on which tree species's pollen is in the air. They've gotten a lot better (~2 day to ~7 day hi-res forecasts) but they're still wrong a lot of the time. The reason is the chaos as you say, however, chaos is deterministic, so, that a deterministic method can approximate a deterministic system is really not the surprising part.
You don't get what's going on here because your baseline understanding is a lot worse than you think it is.
What they're doing is skipping literal numerical simulation in favor of graph- (attention-) based approaches. Typical weather models simulate pretty fine resolution and return hourly forecasts. Google's new approach is learning an approximate Markov model at 6 hours resolution directly so they don't need to run on massive supercomputers.
And it turns out to be better?
That's so counter-intuitive I'm kinda amazed anyone even bothered to research it, let alone that it worked.
Uh..... now do horse racing.
The speed difference is a side effect of completely different implementations. One is a step-by-step simulator, the other is an input/output pattern matcher.
This translates especially well to games like Go, where computing all moves is not even pragmatically possible the classic way. But AI beats the best Go players.
Raw models are excellent for establishing the theory, and for training the AI. But... the AI is better at figuring out more effective, precise, and efficient model within itself, based on both synthetic (based on models) and real data (actual weather patterns).
EDIT: And just to point out, this is not just an AI phenomenon. You are a neural network. And "intuition" is the sense of predicting outcomes you develop, without knowing how and why precisely. This is why I frown upon people with academic knowledge who dismiss people with say engineering or other practical experience in a field. A farmer may not tell you why doing things a weird way results in amazing crop yields, but he gets the gains, and when theory doesn't correlate with reality, it's not reality that's wrong, but the theory.
To recap, nothing beats "learning by example". And AI learns by example. Of course, the formal theoretic models that we can collectively share, explain, and evolve over time have their own strong benefits and have allowed us to grow as a civilization. Computers are in effect "formal computation machines". I don't think we'll run AI for long on digital circuits and it's a clumsy workaround. Computers will have analog processing units for AI and digital processing units for cold, hard logic and data. And the combination is the most powerful approach of all.
Weather forecasting is two separate problems. The first of these is physics - given the state of the atmosphere right now, what will it do. And this is hard, because there are so many different effects, combined with the fact that our computational models have a limited resolution. There's a huge amount of work that goes into making the simulation behave like a real atmosphere does, and a lot of that is faking what is going on at a smaller scale than the model grid.
The second part is to work out what the current state of the atmosphere is. This is what takes vast amounts of computing power. We don't have an observation station at every grid point and at every altitude in the atmospheric model, so we need to find some other way to infer what the atmospheric state is from the observations that we can from it. Many of these observations are limited in locality, like weather stations, or are a complex function of the atmospheric state, like satellite imagery. The light reaching a satellite has been affected by all the layers of the atmosphere it passes through, and sometimes in a highly nonlinear way. In order to calculate the atmospheric state, we need to take the previous forecast of the current atmospheric state, compare it to the observations, then find the first derivative (as in calculus) of the observation function so that we can adjust the atmospheric state estimate to the new best estimate. This is then complicated by the fact that the observations were not all taken at a single time snapshot - for instance polar orbiting satellites will be taking observations spread out in time. So, we need to use the physics model to wind the atmospheric state back in time to when the observation was taken, find the first derivative of that too, and use it to reconcile the observations with the atmospheric state.
It's a massive minimisation/optimisation problem with millions of free variables, and in some cases we need the second derivative of all these functions too in order to make the whole thing converge correctly and within a reasonable amount of time. It takes a reasonable number of iterations of the minimisation algorithm to get it settle on a solution. The problem is that these minimisation methods often assume that the function being minimised is reasonably linear, which certain atmospheric phenomena are not (such as clouds), so certain observations have to be left out of the analysis to avoid the whole thing blowing up.
My doctorate was looking to see if the nonlinearity involved in a cloud forming as air was moving upwards could be used to translate a time-series of satellite infra-red observations into a measurement of vertical air velocity. The answer was that this single form of nonlinearity made the whole minimisation process fairly dire. I implemented a fairly simple not-quite-machine-learning approach, and it was able to find a solution that was almost as accurate but much more reliable than the traditional minimisation method.
Also, to answer the dead sibling comment asking whether weather is really a chaotic system - yes it is. The definition of a chaotic system is that a small change in current state results in a very large change in outcome, and that's definitely the case. The improvements in weather forecasting over the last few decades have been due to improvements in solving both of the above problems - the physics has been pinned down better, but we're also better as working out the current atmospheric state fairly accurately, and that has added something like a day of forecasting accuracy each decade we have been working on it.
What's your take on GraphCast - do you see it as a step forward?
Neural networks can compute pretty much anything. There's no reason, given the same inputs and with enough trainining data that it shouldn't be able to discover the same physical laws that were hard-coded previously.
Why do you say that shortcuts should not exist? Even very basic statements like "falling pressure and increasing humidity indicate a storm is coming" are generally valid. I've done a little bit of storm-chasing and I'm able to point out areas that are likely to experience severe thunderstorms based on a few values (CAPE, dew point, wind shear, etc). I'm sure forecast meteorologists have even better skills. Are those not shortcuts?
Imagine another physical problem. Simulating a sand grain and how it bounces off other sand grains or lodges against them. If you wanted to simulate a sand mountain, you could use a massive amount of compute and predict the location and behaviour of every single grain.
Or, you could take a bunch of well-known shortcuts and just know that sand sits in a heap at the angle-of-repose. That angle decides how steep the mountain will be. any steeper and it will tumble till it's at that angle.
Suddenly, the computation is dramatically reduced, and you get pretty much the same result.
Where this falls apart is that error accumulates over time and not just for one heap of sand but for many such heaps of sand that also interact with other heaps of sand.
Predicting weather for the next hour is trivial. Aviation runs on the fact that you can forecast fairly accurately into the next hour most of the time.
The difficulty scales superlinearly over time due to the error accumulation over predictions
The example you gave does not really explain anything.
OpenAI has become one of the fastest growing companies of all time. And much of it is based on Google's "Attention is all you need" and other papers.
Since Microsoft added the Dall-E 3 image creator to Bing, Bing saw a huge inflow of new users. Dall-E is also a technology rooted in Google papers.
I wonder how Google thinks about this internally.
Why? Other AI studios seem to work on gimmicks while DeepMind seems to work on genuinely useful AI applications [0].
Thanks for the good work!
[0] Not to say that Chat GPT & Midjourney are not useful, I just find DeepMinds quality of research more interesting.
Aren't existing weather forecasting models, already a form of "AI"?
I'm no AI/ML expert, but isn't the real story here is that a new model (like GPT-4.0) is better than the previous/existing model (GPT-3.5).
It's just grabs way more attention calling the new model "AI" (vs not referring to the old as such).
> GraphCast utilizes what researchers call a "graph neural network" machine-learning architecture, trained on over four decades of ECMWF's historical weather data. It processes the current and six-hour-old global atmospheric states, generating a 10-day forecast in about a minute on a Google TPU v4 cloud computer. Google's machine learning method contrasts with conventional numerical weather prediction methods that rely on supercomputers to process equations based on atmospheric physics, consuming significantly more time and energy.
The ML algorithm doesn't care about the science, the agendas, the theories, nothing. It just looks for patterns in the data. Instead of an exact calculation it's more akin to numerical analysis. Turns out that looking at the whole in this case, is better than the sum of the parts.
It's the problem that's hard.
That was a very different beast. It relied on using Google searches to infer the prevalence of various Influenza Like Illnesses in real time, while the CDC reports data with a 2-week lag. Notably, some of the queries they found to be correlated were... strange... like NBA results.
Not unsurprisingly (in hindsight, at least) [2], this eventually broke down when epidemics and flu symptoms got in the news and completely changed what people were searching for.
On top of this, surely the actual observations that feed into the model are terrible -- they come from weather stations, sounding rockets, balloons, radar, etc, none of which seem likely to be especially accurate in all locations. Except that, where a weather station exists, the output of that station is the observation that people care about -- unless you're in an airplane, you don't personally care about the geopotential, but you do care about how windy it is, what the temperature and humidity are, and how much precipitation there is.
ISTM these dynamics ought to be better captured by learning them from actual observations than from trying to map physics both ways onto the rather limited datasets that are available. And a trained model could also learn about the idiosyncrasies of the observation and the extra bits of forcing (buildings, etc) that simply are not captured by the inputs.
(Heck, my personal in-my-head neural network can learn a mapping from NWS forecasts to NWS observations later in the same day that seems better than what the NWS itself produces. Surely someone could train a very simple model that takes NWS forecasts as inputs and produces its estimates of NWS observations during the forecast period as outputs, thus handling things like "the NWS consistently underestimates the daily high temperature at such-and-such location during a summer heat wave.")
It also seems like some of your facts differ from theirs, may I ask how far you read into the paper?
