How We Measure Improvements in Intelligence Search

Understand how Grata Intelligence's engineering team evaluates benchmarks in precision and quality for Intelligence Search.
Search
AI & Automation
How We Measure Improvements in Intelligence Search

Every engineering team believes the latest change in its search tool is an improvement. What separates belief from knowledge is measurement.

Measuring the efficacy of company search is challenging for two reasons:

  • First, whether acompany is “relevant” to a query is partly a judgment call.
  • Second, a "good" list of results means different things depending on the task at hand. If you're an analyst trying to verify a company name, you need the exact right company to appear first. If you're exploring a market, you scan the top results. If you're sizing a market, you need the whole list, down to the last page, to be on target.

The quality of the search results also has to be consistent across the app, the agent, and external AI assistants.

Below, we break down how the Grata Intelligence team measures improvements in each release of Intelligence Search to ensure the highest quality for our users.

A Benchmark for Every Key Question

We run a family of offline benchmarks, each built to answer a specific question. This ensures that each user need has a specific metric.

Judging Relevance at Scale

For the main search benchmark, we test ranking quality across a broad set of natural language queries. An LLM judges whether each returned company is relevant to its query. From those judgments, we compute the following ranking measures:

•     Precision@50 asks how many of the top 50 results are on target.

•     NDCG@50 asks how close the ranking is to ideal, with credit forputting the best matches first.

•     MRR asks how high the first good result appears.

We use the judge to compare runs against each other, andpair it with deterministic checks like location and with human review.

Four Releases, One Steady Climb

Because the benchmark is repeatable, we can chart how quality changes over time. Each release of Intelligence Search was run against the same suite with Agentic Search, our earlier approach, as a reference.

Quality from the First Result to the Last Page

Analysts building market maps do not only look at the first page. They care about how big the market is and whether the long tail is real. So the market sizing benchmark scores precision at the head, at the quartiles, and at the very last page, then combines them into one estimated precision per query.

Intelligence Search holds high precision across the whole list, and it does so with a far more focused result set. For the same queries, the median number of companies returned fell from 173,674 to 16,774.

A separate set of 68 hand-picked queries tested qualitatively showed the same shape. Estimated precision rose from 0.215 to0.446, and Intelligence Search improved 53 of 68 queries (78%). Onthe spot check of known companies, the share found in the top 50 rose from14.6% to 34.1%.

Looking at Every Query, Not Just the Average

When we evaluated re-ranking the top candidates bymeaning, we looked beyond the mean to paired, per-query results. For each of1,000 queries, did the change help? 673 improved on NDCG@50, and the overallgain was +11%. Broad gains like that, spread across many kinds of queries, arewhat give us confidence to ship a change.

Measurement That Runs by Itself

A benchmark that runs only when someone remembers drifts. We automated the two main suites in two ways.

The weekly run builds the trend line over time. The PR run showsthe effect of one change before it merges.

Engineers see the effect of a change with a single label,and the whole team reads one consistent trend line.

Evaluating Agents, Not Just Search

Search quality is the foundation for agent quality. For the Grata Intelligence agent, product squads keep a set of golden questions with expected answers in a plain spreadsheet. When someone adds a question or fixes an expected answer, the evaluation re-runs and reports how the agent performed, with no coding required.

The people who understand the business problem define what "good" means.

From Numbers to a Release Decision

Metrics inform decisions, and people make them. Before releasing Intelligence Search we reviewed the detailed market sizing report, then a product owner ran qualitative testing to assess release readiness. Findings were resolved and re-confirmed, and we agreed the release plan and follow-up improvements. The numbers got us to the right conversation, and people finished it.

What We Learned

•     Match the benchmark tothe question. Lookup, ranking, marketsize and geography each have their own measure.

•     Evaluate the whole list. Quality is tracked from the first result to the lastpage.

•     Look at distributions. Per-query comparisons show that gains are broad.

•     Automate it. Routine measurement beats heroic measurement.

•     Put domain experts incharge. The people closest to usersdefine what good looks like.

What's Next

We are extending evaluation to new entity types and to cross-entity retrieval for agents, so the same quality standard applies wherever Intelligence Search is used.

Want to try Intelligence Search for yourself? Schedule a demo here.

Subscribe to our newsletter

Monthly tactics from Grata’s team and operators.

Related
Articles

From market trends to career advice, Grata’s content fuels smarter decisions.

3 Content Creation Workflows to Try with Grata Intelligence
AI & Automation

3 Content Creation Workflows to Try with Grata Intelligence

Discover 3 ways Blueflame in Grata Intelligence helps dealmakers create IC memos, Excel market maps, and LBO models with AI-powered workflows.
How Dealmakers Win Now That AI Is Table Stakes
AI & Automation

How Dealmakers Win Now That AI Is Table Stakes

Research from Financial Times Longitude and Datasite shows AI is becoming standard across M&A workflows. Learn why the next competitive advantage won't be the technology itself — but the quality and accessibility of the private market intelligence powering it.
The PE Playbook: Landscaping & Property Maintenance
Industrials
Private Equity

The PE Playbook: Landscaping & Property Maintenance

Discover the need-to-know investment trends for private middle market dealmakers looking to make moves in the landscaping and property maintenance industry.
What's New in Grata Intelligence: Richer EU Data, Sharper Search & 50,000+ Dealmaker Profiles
No items found.

What's New in Grata Intelligence: Richer EU Data, Sharper Search & 50,000+ Dealmaker Profiles

Explore what's new in Grata Intelligence: richer European company data, faster semantic search, similar company discovery, and 50,000+ banker profiles.
From Filters to Intent: How Grata Intelligence's Search Evolved to Understand Users' Questions
Search
Workflows

From Filters to Intent: How Grata Intelligence's Search Evolved to Understand Users' Questions

Understand how Grata Intelligence's Search feature has evolved to provide the most relevant results for dealmakers, faster.
The Economics of Agentic AI in M&A: Tokens, Costs, and ROI
AI & Automation
Workflows

The Economics of Agentic AI in M&A: Tokens, Costs, and ROI

Dig into how token efficiency impacts costs and ROI for agentic AI in M&A, and how Grata Intelligence helps firms maximize value.
The PE Playbook: Death Services
Private Equity
Market Trends
North America

The PE Playbook: Death Services

Understand the key private market trends shaping the death services industry - growth segments, M&A, and top roll-up strategies.
Blue backgound