How we use 500TB of weather forecasts: A paradigm shift to array-native infrastructure

The transition to renewables relies heavily on accurate weather forecasting. Wind farms and solar parks produce energy when the wind blows and the sun shines, and unforeseen weather patterns often lead to volatile power prices and high balancing costs.

The good news: Numerical weather prediction (NWP) models have seen a ‘quiet revolution’ – significant, steady progress rarely headlined in the news. Following many advances such as launching satellites, building new weather stations, and improving data infrastructure, we can nowadays do amazing things with atmospheric simulations.

The bad news: The incredible performance of NWPs also leads to an immense amount of data to store and manage. Weather data is high-volume, and storage has many limitations, mainly because modern data storage systems are not designed to meet specific requirements.

In this article, we map out the challenges of storing large volumes of weather data, share insights into the custom solution we developed at Dexter Energy, and explain how this approach benefits our customers. Ultimately, we propose a shift to array-native storage, a tailored infrastructure designed to handle weather data complexities efficiently.

Weather data is big data

Let’s start by understanding what makes weather data unique – and high-volume. NWP models simulate the atmosphere and oceans by solving physical equations (e.g., fluid dynamics) and return predictions for hundreds of meteorological variables, from the ocean surface to the upper atmosphere.

The image below provides an example output from the ECMWF high-resolution model. Picture for a second the type and amount of data that lies behind it. Note that, for every ‘piece’ of the atmosphere, there is a cell holding a type of information across:

• The position on Earth, i.e., spatial dimensions in latitude, longitude, and height
• The precise timestep
• Different meteorological variables, e.g., irradiation, wind speed, temperature, pressure, and humidity

The amount of output data is amplified by the resolution. The finer the spatial resolution and the higher the temporal frequency, the more data.

Example output from the ECMWF high-resolution model

This variety of measurements taken at different locations and times accounts for the multidimensional nature of weather information. Consequently, it results in a staggering volume of data, with modern high-resolution NWP models covering millions of data points.

At Dexter, we are currently ingesting all major weather data providers daily, from large global institutes to local providers, both private and commercial. We also store two, three, and sometimes five years of historical forecasts, all accessible to our data scientists at a moment’s notice. This enables us to develop our machine learning products and to offer our customers the most reliable backtests.

What does it amount to? Our historical NWP forecast data totals more than 500 terabytes (half a petabyte), and we store more than 1 terabyte of new forecasts daily. If you showed these forecasts on a high-definition TV, you’d have 2,500 hours of footage – every day.

Tables versus multidimensional arrays

Most modern big data infrastructures are not ideal for storing large amounts of weather data, primarily because these technologies evolved to address the specific needs and challenges of the business domain.

The ever-growing volume of structured business data, such as customer records or financial transactions, is well-suited for representation in tabular formats. Thus, we’ve seen developments in relational databases and SQL-based querying tailored to this type of information.

But unlike business data, weather information, as outlined, is inherently multidimensional, encompassing spatial and temporal dimensions and numerous meteorological variables. While it too can be represented in tabular form, a storage format that natively ‘understands’ this multidimensional nature is more effective, allowing proper access to the data across its different dimensions.

Therefore, our paradigm shift involves moving away from the conventional tabular mindset and organizing weather data as dense multidimensional arrays. Imagine going from a spreadsheet to a sophisticated 3D model where each layer holds information about specific meteorological variables.

This shift is well understood by the geospatial community, and we use many open-source packages for data analysis and visualization, such as the excellent xarray package.

However, there is no perfect and readily available database technology for efficient large-scale storage and retrieval of this data. At least not one that meets our need to retrieve years of data in seconds for our research and production workloads.

An array-native infrastructure

To address the limitations of existing big data systems and leverage the benefits of dense multidimensional arrays, we developed an in-house array-native storage solution built on a selected storage format. This custom infrastructure, likely unique in our industry, further aligns with cloud usage and benefits our customers.

The approach

We started by using a simple, big-data-optimized database with a tabular structure and concluded that it is not optimal for storing terabytes upon terabytes of weather data. As we transitioned to array databases, we explored several file formats before settling on an embedded, cloud-optimized database that fully decouples the storage and compute layers.

There were three decisive criteria in our selection. The storage format had to:

• Support parallel saving of data safely. In our work, we want to process and save raw weather data for different days without risking data corruption. With most array data formats, writing concurrently poses the threat of overwriting the same file.
• Permit incremental writes but efficient queries over extended time periods. This is important because we write meteorological data per day, but we need to query it for very long periods, since patterns and trends over time are often of interest. Using consolidation features, we can efficiently query large arrays.
• Work well with cloud object (blob) storage. This modern form of storage is several times cheaper and much more scalable than disk (block) storage but offers fewer guarantees.

Once we selected our storage format, we built a layer on top to translate to and from weather data. It was necessary because there are many different geographical projections, resolutions, time-filtering methods, and other peculiarities that require translation.

The benefits

This approach offers several advantages, including enhanced analysis, better compression, faster retrieval, and improved compatibility with mathematical and statistical operations. Overall, it is a much easier format to work with, allowing for very powerful postprocessing.

But beyond making our lives easier, we are able to offer our customers better forecasting products because we can:

• Better leverage AI. AI models rely on extensive historical training data, coupled with the ability to perform extensive feature engineering, and our weather forecast storage paradigm supports this.
• Make precise backtests and tune the models just right.
• Quickly process the large volume of weather forecasts coming in continuously and make them available for forecasting.

The result

Our data pipelines run 24/7 to extract all necessary information to create the best forecasts. Some of our AI models use a single three-year time series for a single point, while others use gridded forecasts for thousands of points across an entire continent. Additionally, our data scientists conduct research, and our meteorologists plot weather forecasts to gain insights and inform better modeling of phenomena.

Below is a simple visualization of some forecast data: a time series from the high-resolution ICON-D2 forecast model and a corresponding 2D image.  Due to the tile-based division, we can quickly query in both the time and space dimensions, satisfying the hunger of our AI models and data scientists for data.

The footprint

Last but not least, since our company’s reason for existing is to have a positive impact on the climate, we cannot overlook the carbon emissions from computationally intensive data operations. Our decoupling of storage and computing, together with the efficiency in compressing array data, contributes to a reduced carbon footprint.

Closing thoughts

Weather forecast data storage and processing are constant challenges, mirroring the chaotic interactions in the atmosphere they represent. In this article, we aimed to match the problem with a solution. The main takeaways I’d like to leave our readers with are:

1. Weather data is high volume due to its inherently multidimensional nature.
2. The tabular format in most modern big data infrastructures is not ideal for weather data.
3. Organizing weather data as dense, tiled multidimensional arrays allows for more powerful postprocessing.

The approach we presented here is the result of a three-year journey at Dexter Energy. This transition to array-native storage has empowered us to store extensive historical forecasts and offer on-the-fly querying for our machine learning products. Ultimately, it has allowed us to offer better AI-powered forecasts and reliable backtests.

Interested in a demo of our Power Forecast? Let us know.