gallery unit · data core
Point-in-time correctness guardrail
Your new model scores 0.999 offline. That is not skill: the training table quietly joined in features from the future, and the metric you trust is rewarding the bug.
scenario sandbox
You are the ML engineer building a training table. Same 8 labels, same feature history, live in your browser; the only choice is how they get joined.
pick the join
clean pass
0 of 8 rows use a future feature. Every feature was available when its label was recorded.
| label | recorded at | feature from | value | point-in-time |
|---|---|---|---|---|
| #0 · y=0 | t=32 | t=30 | 48.9 | ok |
| #1 · y=1 | t=50 | t=50 | 64.7 | ok |
| #2 · y=0 | t=41 | t=40 | 50.6 | ok |
| #3 · y=1 | t=31 | t=30 | 53.2 | ok |
| #4 · y=1 | t=57 | t=50 | 56.9 | ok |
| #5 · y=1 | t=66 | t=60 | 60.8 | ok |
| #6 · y=1 | t=68 | t=60 | 61.7 | ok |
| #7 · y=1 | t=62 | t=60 | 53.1 | ok |
The join and guardrail are live. The model impact, leaked AUC 0.999 vs point-in-time-correct 0.771, is the measured figure from the Python run, not recomputed here.
what you are looking at
01 · data in
Eight labels, each with its own feature history, generated deterministically in your browser.
02 · check runs
The join you pick builds the training table. The guardrail then checks one rule: no feature may be timestamped after its label.
03 · verdict out
Leaked rows are marked line by line, and the banner totals them in plain words.
Go deeper
Watch the walkthrough (45 s, silent)
The leaky join scored against the as-of join · recorded from this page, unedited.
The measured result: AUC 0.999 vs 0.771
The one bug where offline accuracy goes up. So accuracy can’t catch it, and a schema check can’t see it.
Same data, joined two ways
1200/1200 rows leaked · guardrail ~0.1 ms
| Check | Leaky join | Point-in-time | Verdict |
|---|---|---|---|
| Point-in-time guardrail | FLAGGED | passes | catches it |
| Schema / type contract | passes | passes | misses it |
| Offline AUC (5-fold) | 0.999 | 0.771 | rewards it |
A working 0.771 model; the leak inflates it to a too-good-to-be-true 0.999. Pick a pipeline by accuracy and you pick the bug. The schema contract passes it. Only the guardrail catches it.
Where the problem statement comes from
“…to prevent data leakage, which occurs when you use feature values for model training that were not available at the time the label was recorded.”
Databricks feature-store docs · Point-in-time feature joins · 2026-06-12
Why accuracy can't catch this
- Schema validation checks the shape. A leaky training table and a correct one have identical columns and types; the leak is in which value got joined, not the schema. It passes.
- Offline accuracy is actively misleading. The leak raises the score, so the metric you use to judge the pipeline gives the broken one the better mark.
Point-in-time correctness is a property of the join. The right join is as-of: for each label, the most recent feature at or before its time. The guardrail is one line of intent: no feature may be timestamped after its label, checked independently of any score.
leaky join : label at t=40 ─▶ feature at t=95 (35 units of future) ✗ as-of join : label at t=40 ─▶ feature at t=40 (present) ✓ guardrail: flag rows where feature_ts > label_ts
Evidence + how to reproduce
The same data joined two ways, scored by the guardrail, a real schema validator, and a model’s offline AUC. The AUC line is the sharp part: the metric people trust to catch problems instead rewards this one. Synthetic data (leakage needs controlled feature timing); stated on the page, not hidden.
make setup && make test && make run # $0, laptop, no GPU, no network
A leak inside the anti-leak join
The first “correct” as-of join was itself leaky at the boundary, a feature timestamped exactly at the label instant was a post-outcome value, and the join picked it up for any label landing on a sample step. The honest model scored AUC 0.40, below chance, which is impossible for a real signal, and exactly the tell that the clean table wasn’t clean. Fixing the boundary restored 0.77. Only reading a number that didn’t make sense surfaced it.