Needle In A Haystack (niah)

Use it when

• You need to measure whether a model actually retrieves facts placed at specific depths as context length grows, instead of trusting the advertised context window
• You want a reproducible sweep comparing long-context recall across OpenAI, Anthropic, and Cohere models, including multi-hop UUID-chain reasoning, with per-cell cost and score logged to JSONL

What it solves

Not the fit when

• Built-in providers are OpenAI, Anthropic, and Cohere; other providers require writing a small plugin
• Real-model sweeps consume paid API tokens; only the fake provider is free
• Pre-v2 result files in original_results/ do not load with the current tooling
• MIT license requires attribution to the original author and project
• context-window-overflow
• embedding-recall-poor
• no-regression-signal
• cost-per-call-too-high

Install

pip install needlehaystack

Invoke

niah demo --fake to verify the install with no API key, then niah run my-run.yaml with a YAML run config referencing a model config; niah reconstruct results.jsonl --row N rebuilds the exact prompt a cell saw

Alternatives

No reviewed alternatives recorded yet.