gallery unit · data core · bottleneck class B
Point-in-time correctness guardrail
“…to prevent data leakage, which occurs when you use feature values for model training that were not available at the time the label was recorded.”
Databricks feature-store docs · Point-in-time feature joins · 2026-06-12
The one bug where offline accuracy goes up. So accuracy can’t catch it, and a schema check can’t see it.
Same data, joined two ways
1200/1200 rows leaked · guardrail ~0.1 ms
| Check | Leaky join | Point-in-time | |
|---|---|---|---|
| Point-in-time guardrail | FLAGGED | passes | catches it |
| Schema / type contract | passes | passes | misses it |
| Offline AUC (5-fold) | 0.999 | 0.771 | rewards it |
A working 0.771 model; the leak inflates it to a too-good-to-be-true 0.999. Pick a pipeline by accuracy and you pick the bug. The schema contract passes it. Only the guardrail catches it.
run it yourself
Same 8 labels, joined to their feature two ways — live in your browser. Toggle the join and watch the guardrail flag every row whose feature is timestamped after its label.
| label | recorded at | feature from | value | point-in-time |
|---|---|---|---|---|
| #0 · y=0 | t=32 | t=30 | 48.9 | ok |
| #1 · y=1 | t=50 | t=50 | 64.7 | ok |
| #2 · y=0 | t=41 | t=40 | 50.6 | ok |
| #3 · y=1 | t=31 | t=30 | 53.2 | ok |
| #4 · y=1 | t=57 | t=50 | 56.9 | ok |
| #5 · y=1 | t=66 | t=60 | 60.8 | ok |
| #6 · y=1 | t=68 | t=60 | 61.7 | ok |
| #7 · y=1 | t=62 | t=60 | 53.1 | ok |
0/8 rows use a future feature. Every feature was available when its label was recorded.
The join and guardrail are live. The model impact — leaked AUC 0.999 vs point-in-time-correct 0.771 — is the measured figure from the Python run above, not recomputed here.
Why the usual safeguards fail
- Schema validation checks the shape. A leaky training table and a correct one have identical columns and types — the leak is in which value got joined, not the schema. It passes.
- Offline accuracy is actively misleading. The leak raises the score, so the metric you use to judge the pipeline gives the broken one the better mark.
Point-in-time correctness is a property of the join. The right join is as-of: for each label, the most recent feature at or before its time. The guardrail is one line of intent — no feature may be timestamped after its label — checked independently of any score.
leaky join : label at t=40 ─▶ feature at t=95 (35 units of future) ✗ as-of join : label at t=40 ─▶ feature at t=40 (present) ✓ guardrail: flag rows where feature_ts > label_ts
Evidence
Tier 4 — the same data joined two ways, scored by the guardrail, a real schema validator, and a model’s offline AUC. The AUC line is the sharp part: the metric people trust to catch problems instead rewards this one. Synthetic data (leakage needs controlled feature timing); stated on the page, not hidden.
make setup && make test && make run # $0, laptop, no GPU, no network
A leak inside the anti-leak join
The first “correct” as-of join was itself leaky at the boundary — a feature timestamped exactly at the label instant was a post-outcome value, and the join picked it up for any label landing on a sample step. The honest model scored AUC 0.40 — below chance, which is impossible for a real signal, and exactly the tell that the clean table wasn’t clean. Fixing the boundary restored 0.77. Only reading a number that didn’t make sense surfaced it.