FACTOST-V2 / TEMPORAL MODELS
Universal temporal pretraining
How can pretraining capture temporal patterns that transfer across domains while supporting different context and forecast lengths?
Designed and pretrained UTP, and ran cross-domain forecasting evaluations and component ablations. October 2025 – January 2026.
Model design and pretraining
UTP uses an encoder-only temporal backbone. Input sequences are normalized and split into non-overlapping patches; randomly masked history prefixes expose the model to varying amounts of context. The pretraining configuration supports up to 2,048 input steps and 256 forecast steps.
A learnable register token separates context from future placeholders. Partial RoPE and gated attention form the backbone; a multi-quantile head uses pinball loss for probabilistic forecasts. This stage focuses on temporal patterns, with downstream spatial adaptation adding domain information in the full framework.
Evaluating cross-domain transfer
We evaluated fixed-weight zero-shot forecasts on benchmarks excluded from pretraining, using 12→12 and 96→96 context/forecast settings to test transfer to unseen datasets.
Ablation: the role of history masking
PEMS08, with 96-step context and forecast horizons. Metric: MAE (lower is better). These are the results shown in the introduction deck and Tables IX–X of the paper.
| Variant | MAE |
|---|---|
| Full UTP | 91.09 |
| Vanilla RoPE | 91.65 |
| No gated attention | 94.71 |
| No quantile loss | 97.48 |
| No prefix masking | 107.22 |
Adding prefix masking reduces MAE from 107.22 to 91.09, a relative reduction of about 15.0%. Conversely, removing it increases error by about 17.7% relative to full UTP; the percentages use different denominators.
This ablation tests masking under the stated evaluation setting. Broader transfer conclusions depend on the full datasets and protocols reported in the paper.