Cleaner data, sharper forecasts: Automation and human judgment in wind and solar power forecasting

One plus one is sometimes more than two. In power forecasting, combining automated systems with human expertise leads to better outcomes than either alone.

While computers excel at processing large volumes of data at high speeds, humans bring complementary depth of understanding and adaptability. This is a simple definition of ‘human-in-the-loop’ (HITL), and it applies directly to the work of short-term power traders and quantitative analysts who rely on accurate generation forecasts every day.

In this article, we explain how automated filtering and HITL work together in our Wind and Solar Power Forecasts, and why this directly impacts forecast accuracy and trading results. We look at real-world scenarios where per-asset nuance matters, and review both the benefits and challenges of this approach.

Data quality, front and center

The underlying philosophy at Dexter Energy centers on meticulous attention to data quality. As we develop our machine learning products for renewable energy companies, we integrate multiple external data sources, including weather models, market data, and historical production data from asset owners. The latter is what most power forecast models are trained on. However, it is rarely clean data.

Real-world signals reflect the complex operation of a wind or solar park, which can include malfunctions, maintenance, curtailment, re-dispatching, or extreme weather events. If production data contains inaccurate or unrepresentative measurements, the model will not make the most precise predictions.

Rather than directly using the historical production data to train our models, we take the time to understand its intricacies. We do so by combining data science capabilities with domain knowledge: a good grasp of statistics and an understanding of the physical processes underlying the data, such as meteorological phenomena or the engineering of wind turbines and solar panels.

Then, we filter this data automatically where possible, and manually where needed.

Automatic data filtering

We start by building automated pipelines that detect and remove periods where observed production does not reflect what an asset was actually capable of under those conditions.

Automation allows us to apply consistent logic across all assets and continuously adapt as new data comes in. It handles the bulk of unrepresentative data without manual intervention.

Although automation handles the bulk of data processing, it is insufficient in removing all noise from the data. Figuring out the underlying reasons for some of these scenarios and acting in ways that benefit our customers also requires the discerning eye of a human.

Asset tuning with a human-in-the-loop

From years of experience improving our Power Forecast, we learned that per-asset tuning makes a real difference for our customers. Here are common scenarios.

Asset-level changes

In the image below, we see the production level of a fictitious wind park significantly change halfway through the analyzed period:

A model learning from these combined signals would forecast a middle level between the two periods, which is a naive result. A human reviewing this graph would intuitively recognize that the wind park likely shut down some of its turbines permanently or for maintenance.

What the data alone cannot answer: should we forecast the current (possibly temporary) state, or the previous one? A forecast engineer will reach out to the asset owner to find out. Generalizing to larger portfolios, powerful models can flag anomalies across thousands of assets. But it’s a human who works with the renewable energy company to decide on the right path forward for those specific turbines.

Faulty data

Data from an asset owner might contain measurement errors. A visually striking fourfold discrepancy like the one below is rare but could occur if a measurement for a specific time step is submitted in kW rather than kWh.

Automated filters are useful in cases like this, but they won’t detect all the possible issues. It’s worthwhile to have a human review suggestions for plausibly faulty data and decide what to remove and what to keep.

The onboarding phase for a new customer is a natural opportunity for this. During onboarding, we perform automated checks on production data and review and flag any issues our algorithms detect. Clear communication during onboarding is key to a good start, and we’re glad to see our customers appreciate it

Curtailment, a challenge of its own

Curtailment is one of the most important and complex filtering challenges. For both wind and solar, periods of curtailment introduce a mismatch between weather conditions and observed production, making it harder for the model to learn the true relationship between input and output.

Wind curtailment

In the image below, production is reduced or turned off during high-wind periods (highlighted in dark teal), which is common when balancing prices are negative. Although wind conditions would support higher output, the observed signal is suppressed. If used as-is, the model starts to drift and under-forecast.

To address this, we go beyond historical production data. We analyze time-series signals around a park’s availability to determine whether turbines are down for maintenance or turned off due to market conditions.

Using these signals to improve the training data used in our models requires tuning. The right handling differs between a small wind park, where availability rarely drops below 100%, and a large offshore park, where a few turbines are often offline.

Solar curtailment

The same challenge applies to solar assets. Production can show sudden drops or caps even when irradiance would allow for higher output. As shown in the examples below, both full and partial curtailment can be detected and filtered out before training.

Full curtailment creates clear zero or flatline periods:

Note that partial curtailment is more subtle; production remains consistently below potential without appearing obviously abnormal:

Unrepresentative data can also persist over longer periods. Extended outages or degraded availability can shift the model’s baseline understanding of the asset if not handled correctly.

By combining automated detection with asset-level interpretation, we can correct for historical curtailments, preventing under-forecasting and the resulting lower trade profits.

Scaling the approach

Scaling a HITL approach is not straightforward, especially when dealing with thousands of wind parks and solar assets across several countries, with new customers continuously onboarding.

To address this, we built internal tooling that supports both automated filtering and efficient human review. We mapped out the required jobs, identified the ones that took the most time, and turned them into a forecast-tuning tool. In this way, we ‘freed’ humans to focus on higher-value tasks rather than tedious ones.

This method is robust because our operations staff have strong technical acumen, allowing them to actively contribute to and drive improvements to our tooling. This speeds up software development and lets us work with large datasets in a structured way, find issues, and work with our customers to fix them.

The customer benefits

Optimizing our models through human intervention and automated pipelines allows us to maintain high-quality data: clean, complete, and consistent. When errors or fundamental changes occur in the signal we are trying to predict, we can detect them and adjust more quickly.

Filtering directly impacts model performance. Models are trained on a cleaner signal, leading to a more accurate representation of the asset’s behavior under normal operating conditions. These improvements translate into lower forecast error, measured, for example, through a decrease in w-NMAE.

Of course, filtering is one step in improving data quality, but not the only one. We continue to invest in methods that reduce models’ dependence on imperfect operational data.

The bottom line

Despite claims that total automation might be the endgame for AI in energy trading, we know that we’ll always need and count on human skills, augmented by intelligent tools. Our human-in-the-loop approach ensures that models are trained on data that reflects reality, not just what is recorded. Combined with automated filtering, this leads to more accurate and reliable forecasts.

Could your short-term trading operation use a reliable Wind or Solar Power Forecast? Book a demo and start saving on balancing costs!