Search Tool Benchmark Methodology
The Searches.com Search Tool Benchmark is a reproducible framework for evaluating search products by the job they are supposed to perform. Version 1.0 defines the protocol before publishing performance claims, reducing the temptation to invent a score after seeing a result.
Benchmark dimensions
| Dimension | What we measure | Why it matters |
|---|---|---|
| Task fit | Whether the result solves the intended search job | A strong general tool can still be weak for a specialist task |
| Retrieval transparency | Ability to inspect result sources, links or index context | Supports verification and follow-up research |
| Freshness | Handling of recently changed information | Critical for news, products, prices and current facts |
| Control | Filters, operators, domain controls and query refinement | Determines how effectively expert users can steer retrieval |
| Failure behavior | Uncertainty, unsupported answers, dead ends and recoverability | Reliable systems should fail visibly rather than fabricate certainty |
Query-set design
Tests are organized by task family rather than using one universal query set. Planned families include citation-first AI research, academic discovery, privacy-first web search, reverse-image/source discovery and code search. Queries are frozen before each benchmark run and versioned so later results can be compared without silently changing the test.
Scoring discipline
- Define pass/fail or ordinal rules before running the query.
- Record the exact date because indexes, models and interfaces change.
- Separate factual feature checks from subjective workflow preference.
- Do not extrapolate a small task-specific test into a universal “best search engine” claim.
- Publish material limitations, account tier and geography when they could affect the result.
Capability baseline v1
Before controlled performance testing, Searches.com maintains a capability baseline from first-party documentation. This is structured comparison data—not a performance leaderboard.
| Tool | Primary job | Source visibility | Specialized index / retrieval | Access model |
|---|---|---|---|---|
| Perplexity | AI synthesis | Inline citations | Web retrieval + AI synthesis | Freemium |
| Kagi | Premium web search | Result links | Search ranking + lenses/domain controls | Paid subscription |
| Google Scholar | Scholarly discovery | Publication/citation records | Scholarly literature | Free |
| PubMed | Biomedical literature | Citation/abstract records | MEDLINE + related NLM records | Free |
| TinEye | Reverse-image matching | Matched image pages | Image fingerprint matching | Free web use / commercial APIs |
| GitHub Search | Repository/code discovery | Repository/file context | GitHub-hosted code and metadata | Free account features + platform plans |
Baseline reviewed August 23, 2026. Capability labels summarize documented product behavior and are not controlled performance scores.
Search Tool Capability Baseline v1.0 provides the normalized 18-tool dataset in HTML, CSV and JSON.