
How We Measure Improvements in Intelligence Search

Every engineering team believes the latest change in its search tool is an improvement. What separates belief from knowledge is measurement.
Measuring the efficacy of company search is challenging for two reasons:
- First, whether acompany is “relevant” to a query is partly a judgment call.
- Second, a "good" list of results means different things depending on the task at hand. If you're an analyst trying to verify a company name, you need the exact right company to appear first. If you're exploring a market, you scan the top results. If you're sizing a market, you need the whole list, down to the last page, to be on target.
The quality of the search results also has to be consistent across the app, the agent, and external AI assistants.
Below, we break down how the Grata Intelligence team measures improvements in each release of Intelligence Search to ensure the highest quality for our users.
A Benchmark for Every Key Question
We run a family of offline benchmarks, each built to answer a specific question. This ensures that each user need has a specific metric.

Judging Relevance at Scale
For the main search benchmark, we test ranking quality across a broad set of natural language queries. An LLM judges whether each returned company is relevant to its query. From those judgments, we compute the following ranking measures:
• Precision@50 asks how many of the top 50 results are on target.
• NDCG@50 asks how close the ranking is to ideal, with credit forputting the best matches first.
• MRR asks how high the first good result appears.
We use the judge to compare runs against each other, andpair it with deterministic checks like location and with human review.
Four Releases, One Steady Climb
Because the benchmark is repeatable, we can chart how quality changes over time. Each release of Intelligence Search was run against the same suite with Agentic Search, our earlier approach, as a reference.

Quality from the First Result to the Last Page
Analysts building market maps do not only look at the first page. They care about how big the market is and whether the long tail is real. So the market sizing benchmark scores precision at the head, at the quartiles, and at the very last page, then combines them into one estimated precision per query.
.png)
Intelligence Search holds high precision across the whole list, and it does so with a far more focused result set. For the same queries, the median number of companies returned fell from 173,674 to 16,774.
A separate set of 68 hand-picked queries tested qualitatively showed the same shape. Estimated precision rose from 0.215 to0.446, and Intelligence Search improved 53 of 68 queries (78%). Onthe spot check of known companies, the share found in the top 50 rose from14.6% to 34.1%.
Looking at Every Query, Not Just the Average
When we evaluated re-ranking the top candidates bymeaning, we looked beyond the mean to paired, per-query results. For each of1,000 queries, did the change help? 673 improved on NDCG@50, and the overallgain was +11%. Broad gains like that, spread across many kinds of queries, arewhat give us confidence to ship a change.
Measurement That Runs by Itself
A benchmark that runs only when someone remembers drifts. We automated the two main suites in two ways.

The weekly run builds the trend line over time. The PR run showsthe effect of one change before it merges.
Engineers see the effect of a change with a single label,and the whole team reads one consistent trend line.
Evaluating Agents, Not Just Search
Search quality is the foundation for agent quality. For the Grata Intelligence agent, product squads keep a set of golden questions with expected answers in a plain spreadsheet. When someone adds a question or fixes an expected answer, the evaluation re-runs and reports how the agent performed, with no coding required.
The people who understand the business problem define what "good" means.
From Numbers to a Release Decision
Metrics inform decisions, and people make them. Before releasing Intelligence Search we reviewed the detailed market sizing report, then a product owner ran qualitative testing to assess release readiness. Findings were resolved and re-confirmed, and we agreed the release plan and follow-up improvements. The numbers got us to the right conversation, and people finished it.
What We Learned
• Match the benchmark tothe question. Lookup, ranking, marketsize and geography each have their own measure.
• Evaluate the whole list. Quality is tracked from the first result to the lastpage.
• Look at distributions. Per-query comparisons show that gains are broad.
• Automate it. Routine measurement beats heroic measurement.
• Put domain experts incharge. The people closest to usersdefine what good looks like.
What's Next
We are extending evaluation to new entity types and to cross-entity retrieval for agents, so the same quality standard applies wherever Intelligence Search is used.
Want to try Intelligence Search for yourself? Schedule a demo here.

Subscribe to our newsletter
Monthly tactics from Grata’s team and operators.







.png)