Home/Field guides/How to tell whether a forecast helps
Forecasts & uncertainty / FIELD GUIDE

How to tell whether a forecast helps

Evaluation needs many forecasts, matching observations and a suitable reference. One memorable success or failure cannot rank a system.

NowLaterThreshold
Illustration of a related weather process.

What to notice

A forecast looks accurate for common dry days but misses the rare heavy-rain events that matter to the reader.

What is happening?

Forecast verification compares predictions with observations for a defined variable, location, period and lead time. Accuracy describes agreement; skill usually concerns improvement over a reference such as climatology or persistence. The reference matters. Predicting no rain can appear accurate in a very dry region while adding little useful information beyond knowing that rain is rare.

Different metrics emphasise different qualities. Temperature errors can be summarised in degrees; event forecasts can be assessed through hits, misses and false alarms; probabilistic forecasts require checking how predicted chances relate to observed frequencies. A system can improve one measure while worsening another. Evaluation should match the actual question the user is trying to answer.

Basis: ECMWF [1] · ECMWF [2] · ECMWF [3]

Read the observation

  • Check the sample size, region, season and forecast lead time.
  • Identify the reference forecast used for a skill score.
  • Compare probabilities with event frequencies across many cases.
A USEFUL DISTINCTION

A single failed high-probability forecast does not establish poor calibration, and a good average error does not describe every kind of event.

Go a little deeper

Reliability means that events assigned a particular probability occur at approximately that frequency over a sufficient sample. Resolution concerns the ability to distinguish situations with different outcomes. Always predicting a climatological probability can be reliable while failing to separate wetter from drier situations. Conversely, sharp confident probabilities can be poorly calibrated. Both properties matter.

For a simple binary event, the Brier score averages (p − o)², where p is the forecast probability from 0 to 1 and o is 1 if the event occurs and 0 otherwise. A 0.7 forecast scores 0.09 when it occurs and 0.49 when it does not. Lower is better, but interpretation still needs a reference and many cases. These worked numbers demonstrate the calculation. The probability lab counts hypothetical ensemble exceedances; it does not calculate a Brier score or establish the measured skill of an operational forecast service.

Basis: ECMWF [1] · ECMWF [2] · ECMWF [3]

FOLLOW THE EVIDENCE

Sources & provenance

  1. Separating the signal from the noise ECMWF · 2026-03-25Reference only; original paraphrase; no source media reproduced.
  2. Fact sheet: Ensemble weather forecasting ECMWF · Publication date not statedReference only; original paraphrase; no source media reproduced.
  3. Quantifying forecast uncertainty ECMWF · Publication date not statedReference only; original paraphrase; no source media reproduced.

Explanations and diagrams are original educational interpretations. Source institutions have not endorsed or reviewed this publication. How we distinguish evidence from models.