Earlier this year an Instagram reel about college kids scraping temperature readings from commercial airplanes to trade Polymarket weather contracts caught my attention. It doesn't work, for many reasons but not least because commercial airplanes tend to be in the air which is not a good proxy for the ground temperature. In any case, this led to a conversation over drinks with a friend about what it would actually take to beat the market.

For context, the condition of these particular contracts is predicting the daily high in 1°C increments. The contracts resolve based on the highest daily METAR reported by NOAA at a number of airports. For this project I selected 14 cities, primarily in Asia and Europe but mostly based on total trading volume. Most of the trading is focused on predicting the exact bin during the peak phase of the day; this segment competes on information quality, data access speed, and trade timing. The high-frequency, intra-day trading felt like a losing battle for a model that was going to run on a Macbook Air, so instead I shifted my focus to trying to predict in the lead-up to the market's target date. My goal became to model the temperature forecast as a calibrated distribution using freely available weather forecast data.

Predicting the Weather

Most weather apps rely on government backed Numerical Weather Prediction (NWP) models to predict the weather. These models segment the planet into a grid and solve the equations of fluid dynamics forward in time on supercomputers to predict the weather in each cell. The difference between a rough forecast and an accurate prediction comes from the post-processing step. The classic approach is known as Model Output Statistics (MOS). MOS fits a linear regression to the historical observations to map the model's output to a point forecast.

Whereas weather apps only need to produce a single point forecast, trading on weather markets requires a full probability distribution - specifically, a CDF over temperatures discretized into 1°C bins. To get there, we use Ensemble Model Output Statistics (EMOS), which combines the outputs of multiple NWP models into a calibrated predictive distribution by learning a mean and variance from their forecasts. The foundation of the model was built on the following six models from ECMWF and NOAA:

• ECMWF IFS Deterministic
  via ECMWF Open Data
• ECMWF AIFS Deterministic
  via ECMWF Open Data
• ECMWF IFS Ensemble
  via ECMWF Open Data
• ECMWF AIFS Ensemble
  via ECMWF Open Data
• GFS Seamless
  via dynamical.org
• GEFS Ensemble
  via NOAA S3

After about three months of collecting daily predictions, ICON Seamless (via Open-Meteo) was added as a seventh predictor. Most of these sources offer ~2-6 weeks of previous forecasts depending on where you query them, and in my findings it took about two months of data to get a stable contribution to a short term EMOS prediction.

As the names suggest, these models fall into one of two categories: deterministic or ensemble. Deterministic models give you one prediction; ensemble models run the simulation many times with slightly perturbed initializations to see how much the answer varies. ECMWF ensemble models each have 51 members and GEFS has 31. Intuitively, when ensemble members disagree more it suggests that the forecast is genuinely more uncertain. Generally NWP models produced by governments will be optimized to forecast locally; you can usually find specific models that offer higher resolution predictions in limited regions like those offered by Japan's JMA or Canada's ECCC. The models I opted for are all global and generally perform well world-wide, but each has their strengths and weaknesses.

With these models as inputs, we try to learn a predictive distribution. My initial approach used a Gaussian distribution but after testing with a Logistic and Student-t distribution I found Student-t produced the best results. The Logistic distribution tended to overestimate the probability of the most likely bin, and the Gaussian distribution was too light on the tails. The heavier (and tunable) tails of Student-t captured the frequency of outliers best in the end. The model learns an appropriate number of degrees of freedom, but it's generally settled around 5.6 to produce the best forecasts.

The mean is calculated as a blend of each model's forecast, \(b_m\) controls how much weight model \(m\) gets, and \(\delta_s\) is a per-station bias correction. One of the biggest improvements the model saw was statically debiasing \(f_m\) when I was able to find large datasets of past forecasts for a given model (GFS, GEFS, AIFS and IFS were all possible). The bias parameter blends historical data over a 15 day window centered on the target date to capture and account for any seasonal biases within the model. The variance is a function of two spread signals: how much the NWP models disagree with each other, and how much each ensemble's own members disagree internally. This lets the model widen its distribution when forecasts are uncertain and tighten it when they agree.

Scoring & Calibration

All parameters are learned by minimizing the Continuous Ranked Probability Score (CRPS). CRPS measures how well a predicted CDF matches the observed outcome; it can be thought of as the error of our model's CDF for a given forecast. If you imagine the pricing across bins as a CDF for the markets prediction, a lower CRPS would imply our forecast is better calibrated than the market's. For a predicted CDF \(F\) and observed value \(y\), CRPS is defined as^[1]:

CRPS defines a step function that steps at the observed value \(y\), and measures the area of the squared difference between it and the predicted CDF. For the Student-t distribution, CRPS has a closed-form expression that avoids numerical integration^[2]. The result combines three terms:

def crps_student_t(mu, sigma, nu, y):
    z = (y - mu) / sigma  # standardize observation
    pdf_z = t.pdf(z, nu)
    cdf_z = t.cdf(z, nu)
    beta_ratio = (
        beta_func(0.5, nu - 0.5)
        / (beta_func(0.5, 0.5 * nu) ** 2)
    )
    return sigma * (
        z * (2 * cdf_z - 1)                        # displacement from center
        + 2 * pdf_z * (nu + z**2) / (nu - 1)       # spread penalty
        - 2 * np.sqrt(nu) * beta_ratio / (nu - 1)  # tail weight baseline
    )

A model that minimizes CRPS will tend to produce better-calibrated forecasts. The simplest way to frame calibration is to ask, "if the model predicts an outcome 30% likelihood, how often does it actually occur?" With a perfectly calibrated model, 30% of the time. A useful diagnostic tool is the Probability Integral Transform (PIT) histogram. For each forecast, evaluate the CDF at the observed outcome \(F(\hat{y})\). If the model is calibrated these PIT values will be uniformly distributed.

The PIT histogram ultimately informed the distribution choice. Gaussian produced a U-shaped PIT histogram - implying too much probability was left in the tails. Logistic improved tail coverage but overconcentrated around the peak bin - meaning the outcomes were underdispersed. Student-t's tunable \(\nu\) let the model learn the correct degrees of freedom to smooth the PIT histogram.

The model is fit hierarchically in three stages: first, global parameters are learned across all stations and lead times (L0). Next, variance parameters are learned at several lead times (L1). Finally, the per-station intercepts are fitted to capture any remaining biases (L2). Each level is validated via leave-one-out cross-validation before being adopted. The optimizer is scipy's implementation of L-BFGS which does well with smooth, low-dimensional problems like this.

CRPS is not easily interpretable in terms of exact prediction error, but is useful for comparison between models. The following are a sample of training CRPS values achieved by the best model, an equal-weight mixture of the seven predictors, and a model consisting of a single ensemble model (GEFS). CRPS values are split by lead-time to forecast date

For posterity, these are the CRPS numbers using a test set of a 14-day window after the model was trained. The EMOS with learned weights performs much better on real data suggesting some overfitting in the equal weights model - likely extreme station intercept values.

Results

CRPS is great for measuring calibration, but the question was whether or not this model could beat the market. The answer is - kind of. To evaluate model vs. market we look to the Brier Score. Brier Score measures the accuracy of probabilistic predictions. For all bins over a given time period (technically, we filter to bins with >5% likelihood since a large number of bins receive 0% weight it skews the numbers) we compute the squared error of our predicted likelihood of an event vs. whether it actually occurred or not. We can repeat this for the market's predicted likelihood (the price of the option) and compare these two numbers directly. If our number is smaller, we have a tradable edge.

Of the 14 cities selected for this project, the results were generally on-par with the market. Seasonal and dynamic weather would lead to the model outperforming some weeks and underperforming the next. Through it all, there were a few standout stations worth flagging.

Starting with the strongest performers, Seoul and Tokyo. With Brier scores consistently lower than the market across several months it's safe to say the model has an edge over the market at these stations. It's hard to say exactly where the edge comes from, but my running hypothesis has been that during the summer months turbulent afternoon weather was modeled well by the physical simulations and traders relying on point forecasts were blind to the possible variance in the actual outcome.

And two weaker performers. In London's case, some weeks the forecasting was dead on and other weeks off by multiple degrees. I haven't been able to nail down the source of this but my goal is to roll in additional predictors from the UK's MET office that may have better modeling for the UK's climate. In Madrid, the market often predicts the peak temperature with >40% likelihood, on average higher than any other station. It's likely that stable weather patterns over the Spanish plateau make it generally easier to forecast for the city. My model on the other hand is not so locally tuned and is punished for letting the variance get so low, often assigning too much weight to the tail bins of this city.

Will it Trade?

I ended up giving the bot $100 and setting it loose to see what would happen. The short version: after about a month of ups and downs it busted. I learned a great deal about algorithmic trading and why a +EV doesn't necessarily equate to +money, but I'll save that for a future article. For now, I'll keep running my data scraper and hopefully seeing those calibration improve.

Trevor McCann