Why it matters
Bridgewater’s Vault talk describes building a small, manually labeled search benchmark with relevant documents and plausible but unhelpful results. It measures top-ten precision and examines how date, source and asset filters affect retrieval. Human search and agent workloads need different load tests. Private benchmark scores and investment outcomes are not independently verified.
My takeaway: Collect representative queries and label relevant results alongside hard negatives. Measure precision at ten, recall and latency under the date, source and permission filters your users need. Test human query rates and agent bursts separately. Preserve a held-out query set and inspect missed documents before changing retrieval or ranking.