Why it matters
Ryan Ahern demonstrates historical web retrieval for an acquisition predictor. A winning entry memorizes the visible company list, exposing the need for held-out tests. Exa’s documentation adds a key limit: snapshot content predates the cutoff, but candidate discovery uses current retrieval signals. Missing snapshots, model memory, and test-set tuning can still bias a backtest; the workshop does not establish predictive performance.
My takeaway: Record the cutoff, retrieved content, missing pages, model version, and feature definitions. Set snapshotAsOf consistently on search and content requests, and prevent later live retrieval. Keep companies and later periods out of tuning, then measure misses and false alarms. Check access limits and possible leakage through current retrieval signals or model training before interpreting a historical score.