FinSearch-Eval: Evaluating Financial Search on Live, Billion-Document Corpora

November 13, 2026

Search, Evaluation, Research/30 minutes read

Information Retrieval (IR) has a rigorous evaluation tradition. From the Cranfield paradigm to TREC to ranking metrics such as nDCG and large-scale benchmarks such as MS MARCO and BEIR, the field has spent decades learning how to measure search quality (Cleverdon, 1967; Voorhees and Harman, 2005; Järvelin and Kekäläinen, 2002; Bajaj et al., 2016; Thakur et al., 2021). Those methods remain essential. But financial Retrieval-Augmented Generation (RAG) raises the bar for retrieval quality. It is no longer enough to return documents that are broadly about the right topic and likely useful to a human reader. In financial AI systems, the retriever must return evidence that an automated model can safely use: the right issuer, the right period, the right source, the right document subtype, and the exact passage that can ground the answer. And when that evidence does not exist in the intended scope, the correct behavior is abstention. That requirement is stricter than broad semantic similarity, and it becomes even harder when the index is not a fixed academic corpus but a live, continuously updated database of billions of financial documents. In that setting, several assumptions behind standard closed-corpus evaluation begin to break:

You cannot exhaustively label every relevant document.
Many queries have multiple valid documents or answer-bearing passages.
Users often omit key constraints, forcing the retriever to infer implicit intent.
Query meaning is time-dependent: “latest earnings call,” “current guidance,” or even “Q2” only make sense relative to when the query was asked.
Some queries are genuinely unsupported in scope.

A useful financial benchmark therefore has to evaluate more than broad relevance on a frozen corpus. That is why we built FinSearch-Eval, a time-aware benchmark for evaluating financial search systems at production scale. Importantly, the results in this paper are a point-in-time snapshot, not a claim that financial retrieval can be captured by a single frozen dataset. FinSearch-Eval is designed for a live corpus and is continuously refreshed over time. Every query carries a formulation timestamp so that relative asks such as “latest earnings call,” “current debt maturity schedule,” or “this quarter” are interpreted as of when the query would actually have been asked. We also include a dedicated class of no-answer queries to test whether systems abstain when no valid evidence exists in scope, an important but still under-measured capability in financial RAG. FinSearch-Eval combines:

deterministic rule-based scoring on structured constraints such as reporting entity, mentioned entity, reporting period, document type, document subtype, and source;
pointwise LLM-as-a-judge scoring to measure whether retrieved chunks are actually useful for answering the query;
explicit evaluation of implicit intent resolution;
time-aware query generation and evaluation on a dynamic corpus;
no-answer / abstention evaluation.

This paper explains why standard closed-corpus benchmarks are insufficient for financial RAG, how we generate realistic evaluation data at scale, how we compare search systems and what a current FinSearch-Eval snapshot shows across major AI search providers.

1. Retrieval did not disappear when context windows got bigger

A recurring claim in the AI ecosystem is that long-context LLMs reduce the need for retrieval. If a model can ingest hundreds of thousands or even a million tokens, why not place the relevant documents directly into context? That argument misses the real bottleneck. In financial search, the hard problem is not whether a model can read more text once the right evidence has already been selected. The hard problem is selecting the right evidence from a live corpus of billions of documents. Longer context windows help only after that selection problem has been solved. Retrieval-augmented methods remain central to knowledge-intensive NLP because they improve access to fresh, tail, and attributable evidence beyond what can be reliably stored in parametric memory alone (Karpukhin et al., 2020; Guu et al., 2020; Lewis et al., 2020; Izacard et al., 2022). Long-context models also remain vulnerable to distractors and position effects, so a larger context window is not the same thing as reliable evidence use (Liu et al., 2023; Hsieh et al., 2024). Language models can encode and recall factual knowledge, and research on parametric memory continues to improve these capabilities (Back, S., et al. 2026). However, factual recall alone does not establish that an answer reflects the correct source, reporting period, or information available at a particular time. For auditable financial research, claims must remain verifiable against authoritative, versioned external records. LLMs can help select and interpret that evidence; model memory alone should not be treated as the authoritative record. Long-context models can reduce the need for passage selection within a bounded document collection and may outperform particular RAG pipelines on some tasks. They do not remove the need to discover and access current evidence across a live, billion-document corpus. In this setting, retrieval quality determines which sources reach the model, while context length affects how much of that evidence it can examine. The appropriate balance depends on the task, evidence coverage, cost, and latency. This is especially true in finance, where pulling data from the wrong quarter, the wrong issuer, or the wrong source can produce an answer that sounds highly plausible but is materially incorrect. Large context windows do not solve:

Selection: Which documents deserve to be included in the first place.
Freshness: The answer may depend on the latest filing or transcript released minutes ago. In finance, freshness is not a UX detail; it is part of correctness.
Authority: The best source may be a primary SEC filing, not a news summary.
Cost and latency: Blindly stuffing more context is computationally expensive and slow.
Grounding: More text is not the same as better evidence.
Abstention: If no in-scope evidence exists, the system should abstain.

Retrieval precision may also matter for downstream robustness. Hägele et al. (2026) find that longer reasoning and action trajectories are associated with more incoherent failures. Although this does not directly establish the effect of retrieval noise, it motivates a related hypothesis: semantically adjacent but unsupported chunks increase the evidence-selection burden and may make answer synthesis less stable. It is very plausible to think that high-precision retrieval and appropriate abstention improve stability.

2. Why traditional IR evaluation is necessary, but not sufficient

IR evaluation has a long and successful history. Benchmarks such as TREC and MS MARCO have been enormously useful for advancing retrieval systems, and we still rely on many of their underlying concepts. But financial RAG exposes several limitations of the standard “closed eval” setup. The issue is not that classical IR stopped mattering. It is that live financial retrieval imposes additional requirements: issuer- and period-level correctness, source authority, passage-level answerability, time-aware interpretation, and calibrated abstention. FinSearch-Eval is designed to measure those requirements directly. Section 2.5 situates it relative to prior IR, RAG, and finance-specific benchmarks.

2.1 Fixed-corpus labels do not map cleanly to live corpora

Classic benchmarks often assume a fixed document corpus, a stable set of relevance judgments, and a manageable evaluation environment. Financial search systems do not operate that way. They retrieve over live corpora that change continuously: SEC filings, earnings call transcripts, press releases, annual reports, investor presentations, news, podcasts, expert content, and other semi-structured sources. At that scale, exhaustive labeling is impossible. A search system can return a perfectly valid primary-source document that was never part of a pre-labeled pool. In a fixed benchmark, that can look like a false negative even when the result is genuinely better. Incomplete relevance judgments are a known challenge even in classical pooled IR evaluation (Buckley and Voorhees, 2004). Live financial corpora amplify this problem because valid documents continue to arrive, and strong results may fall outside any fixed judged pool. That is why FinSearch-Eval does not define retrieval correctness as matching a single canonical gold document ID. Instead, it evaluates whether the system surfaced acceptable evidence under the right financial constraints.

2.2 In finance, relevance is not enough

A result can be topically related and still be entirely unusable because it is about the wrong company, the wrong reporting period, the wrong source class, the wrong filing type, the wrong document subtype, or simply a passage that does not contain answer-bearing evidence. That is why we do not evaluate financial search purely as “did we retrieve relevant text?” We also evaluate it as “did we retrieve the correct and authoritative evidence for this financial intent?” In finance, the most dangerous retrieval failures are often not obviously irrelevant pages, but authoritative-looking near-misses: real filings, transcripts, or newswire items that still fail the query on issuer, period, source, or answer-bearing content. This is exactly why financial retrieval must be evaluated with explicit issuer-, period-, source-, and document-aware constraints rather than broad topical similarity alone.

2.3 Downstream answer grading alone is not enough

Evaluating search through downstream answer quality is useful, but insufficient on its own. If the final answer is wrong, was it because the retriever missed the right document, the model failed to select the right evidence, or the generation step used it incorrectly? A useful financial benchmark must isolate the retriever as much as possible. Otherwise, retrieval failures get mixed together with generation failures, making the system much harder to diagnose. A production RAG system includes retrieval, pruning, compression, memory, and generation. Later stages cannot recover missing evidence without another search or tool call. FinSearch-Eval evaluates retrieval before those downstream steps. That asymmetry is why FinSearch-Eval focuses on retrieval rather than treating it as a hidden subroutine inside end-to-end answer grading. Recent work on deep-research agents reinforces this separation. BrowseComp-Plus, for example, highlights that evaluations built on opaque live web APIs make it difficult to disentangle agent reasoning from underlying retrieval quality and to run controlled, reproducible experiments on the retrieval layer (Chen et al., 2025). If retrieval is not isolated, it becomes much harder to tell whether a failure came from missing evidence or from downstream synthesis.

2.4 Static financial benchmarks decay over time

One under-appreciated problem in live domains is benchmark decay. If a benchmark keeps asking only about older filings and transcripts, it silently gets easier over time. Those documents become better indexed, more widely summarized, more heavily cited on the web, and more likely to have leaked into general web corpora or model pretraining data. Worse, without explicit query timestamps, a system can appear to answer an older query correctly using a newer document that did not exist when the query would actually have been asked. In a time-sensitive domain like finance, that is a measurement error, not a success. FinSearch-Eval addresses this by attaching a formulation timestamp to every query and evaluating results relative to that timestamp rather than the run date. Section 4.7 describes the time-aware generation and rolling refresh process.

2.5 Why this is not just another RAG eval

FinSearch-Eval sits at the intersection of classical IR, RAG evaluation, and finance-specific QA benchmarks, but it asks a narrower and earlier question in the stack: before generation begins, did the search system retrieve the correct issuer-, period-, source-, document-, and passage-level evidence from a live financial corpus? Classical IR benchmarks such as TREC, MS MARCO, BEIR, and KILT remain indispensable for measuring retrieval quality and evidence provenance (Voorhees and Harman, 2005; Bajaj et al., 2016; Thakur et al., 2021; Petroni et al., 2021). However, they generally operate on fixed corpora. Financial search must work over live, heterogeneous, and continuously changing document universes. RAG evaluation frameworks such as RAGAS and ARES assess retrieved-context relevance alongside downstream answer faithfulness and relevance (Es et al., 2024; Saad-Falcon et al., 2024). FinSearch-Eval adopts a narrower evaluation boundary: it assesses the evidence returned by the search system before downstream answer generation, without allowing a generator’s ability to compensate for missing or unsuitable evidence to influence the retrieval score. Its contribution combines this retrieval-only focus with time-aware financial constraints, provenance-sensitive scoring, and bounded-scope abstention evaluation over a live corpus. Finance-specific QA benchmarks such as FinQA, TAT-QA, and ConvFinQA are valuable for testing numerical reasoning over financial text and tables (Chen et al., 2021; Zhu et al., 2021; Chen et al., 2022). Yet they typically assume that the relevant evidence has already been selected. FinSearch-Eval focuses on the retrieval step that precedes that reasoning. More recently, deep-research and finance-agent benchmarks, evaluate complex, multi-step search-and-reasoning workflows rather than retrieval in isolation. This includes FinSearchComp (Hu et al., 2025), Finance Agent Benchmark (Bigeard et al., 2025), FinDeepResearch (Zhu et al., 2025), BrowseComp and BrowseComp-Plus for general web browsing (Wei et al., 2025; Chen et al., 2025), Search Arena for search-augmented LLMs (Miroyan et al., 2025), SAGE for scientific literature retrieval (Hu et al., 2026), and DeepResearchEval for report-level agent evaluation (Wang et al., 2026). FinSearch-Eval complements this line of work by isolating the retrieval layer itself in a domain where issuer resolution, period correctness, source authority, provenance, and abstention are all first-class requirements. Taken together, FinSearch-Eval should be read as complementary to prior IR, RAG, and finance-benchmark work. More broadly, we view it as aligned with efforts to treat retrieval quality as a first-class evaluation target rather than a hidden subroutine inside answer generation (Petroni et al., 2021; Es et al., 2024; Saad-Falcon et al., 2024). Its contribution is not to replace those benchmarks, but to add time-aware query resolution, structured financial constraints, provenance-sensitive scoring, and abstention evaluation for retrieval on a live corpus.

3. What FinSearch-Eval measures

FinSearch-Eval is a retrieval benchmark, not an LLM reasoning benchmark. It asks a simple question:

Given a raw financial query, how well does a search system interpret the query and retrieve the right evidence from a massive live corpus?

Returning entire filings can increase context-processing cost and introduce distracting material, although broader context can also preserve qualifications and relationships that short chunks omit. Even if modern LLMs can ingest hundreds of thousands or millions of tokens, handing a downstream model entire filings and transcripts turns the model into a brute-force filter. In finance, that increases latency, compute cost, and hallucination risk by forcing the model to sift through irrelevant or weakly related material to find a small amount of usable evidence. A model-facing search system should push as much of that burden as possible into retrieval itself. Its goal is not simply to return documents that are broadly relevant, but to surface compact, attributable, answer-bearing evidence units (sections, spans, paragraphs, tables, or Q&A exchanges) that a downstream model can use directly. Concretely, FinSearch-Eval measures several capabilities at once:

Query understanding: Can the system infer what the user is asking for?
Entity resolution: Can it distinguish the reporting company from other entities mentioned in the query?
Temporal normalization: Can it correctly interpret periods such as “Q3 2025,” “latest,” or “annual report”?
Time consistency: Can it retrieve evidence that is valid as of when the query was asked, rather than using future or stale documents incorrectly?
Source selection: Can it prefer primary disclosures when the task requires them?
Document-type understanding: Can it distinguish filings, transcripts, news, podcasts, and other source families?
Document-subtype understanding: Can it distinguish 10-Ks, 10-Qs, 8-Ks, 20-Fs, earnings calls, and other subclasses?
Passage-level answerability: Once the correct document is found, does the returned chunk actually contain evidence that answers the query?
Abstention behaviour: When no valid evidence exists in scope, can the system abstain rather than return misleading near-matches?

Seen another way, FinSearch-Eval separates two axes that are often conflated in retrieval evaluation: semantic usefulness and structural acceptability. A chunk can be structurally acceptable (from the correct issuer, period, source, and document class) yet still fail to answer the query. Conversely, a chunk can look semantically useful while remaining unacceptable because it comes from the wrong quarter, wrong issuer, or wrong source class. A production financial benchmark needs to score both, and it needs to do so point-in-time: evidence must be valid as of the query timestamp, not merely topically related in hindsight.

Key benchmark fields

Because some of our schema terms are not standard outside our system, we define them here:

Document: The parent source artifact, such as a filing, transcript, or news article.
Chunk: The retrievable evidence unit returned to the model, ideally a self-contained passage or structured region of a source document. A good chunk preserves provenance while minimizing irrelevant surrounding text.
Result: The ranked item returned by a search system after normalization. A result may correspond to a chunk, a document-level snippet, or another retrievable evidence unit.
Document type: Broad source family, such as filings, transcripts, news, or podcasts.
Document subtype: A more specific class within a type, such as 10-K, 10-Q, 8-K, 20-F, or earnings call.
Reporting entity: The primary company whose document or event is being searched. Example: in “Find what was said about Home Depot in Stanley Black & Decker’s 10-K,” the reporting entity is Stanley Black & Decker.
Mentioned entity: Another company, product, country, or entity referenced inside the reporting entity’s document. In the same example, the mentioned entity is Home Depot.
Reporting period: The relevant quarter, fiscal year, or date window.
Source: The specific publisher or outlet when relevant, such as Reuters, Bloomberg, or WSJ.
Query timestamp: The time when the query is considered to have been formulated. Relative temporal expressions such as “latest,” “current,” or “this quarter” are resolved against this timestamp, not the evaluation run date.
Expected behavior: Whether the query is expected to retrieve evidence or abstain.

Explicit vs. implicit intent

One of the most important distinctions in financial search is the difference between explicit and implicit intent.

Explicit intent means the user states the constraint directly. Example: “in their 10-K”
Implicit intent means the constraint is necessary for a correct answer, but not literally stated. Example: “What’s Uber’s current debt maturity schedule?”

A strong system should infer that the answer to an implicit query likely lives in recent primary filings, not in generic news. We view measuring this capability as an important gap in many current RAG benchmarks. Recent work on deep-research agents reinforces the broader point that raw queries are often under-specified. The AgentIR framework, for example, improves retrieval by conditioning search on the agent’s intermediate reasoning trace rather than the bare query alone (Chen et al., 2026). FinSearch-Eval asks a complementary question: can a financial search system resolve these implicit constraints from the raw query itself, without depending on hidden intermediate reasoning supplied by the caller? This is a deliberate design choice. Passing conversation history or reasoning traces to the retriever can be highly effective, but benchmarking that setting introduces substantial variability because the quality and structure of the added context depend on the surrounding agent architecture, prompting, tool use, and interaction history. Rather than engineering highly subjective synthetic context scenarios, FinSearch-Eval deliberately isolates the retrieval tool to measure its baseline zero-context capability. If a search system can reliably resolve implicit financial intent from a raw query alone, it provides a stronger foundation for a richer, context-aware agent loop.

4. Building the evaluation dataset: from templates to realistic financial queries

Before evaluating search, you need queries. In finance, that is not trivial. Even though we operate systems that process large volumes of queries, we cannot use or inspect them directly: privacy and confidentiality constraints make that off-limits. Instead, we ground dataset construction in a combination of client-informed use cases, domain expertise and systematic query-generation methods. This allows us to approximate realistic information needs without relying on sensitive production data.

4.1 Why manual datasets do not scale

You can hand-write 50 or 100 good queries. But that quickly breaks down if you want coverage across:

many industries and geographies,
multiple document types and subtypes,
temporal variation,
dual-entity queries,
primary-source preferences,
time-relative asks such as “latest” or “current,”
and realistic linguistic variation.

Manual datasets also introduce human bias: authors overuse their preferred language, certain companies get overrepresented, and rare but important query forms are missed.

4.2 Our hybrid generation strategy

To achieve scale without giving up control, we use a human + programmatic + LLM pipeline. Domain experts define the underlying financial intents, constraints, and QA rules; structured slot filling produces deterministic query labels; LLMs generate natural-language variation; and QA filters out invalid or low-quality examples. This lets us scale coverage without sacrificing control over labeling or difficulty.

4.3 Human-authored intent templates

The starting point is a library of financial research templates created by domain experts, covering workflows such as Guidance & Outlook, Risk & Litigation, Strategic Initiatives, Management Changes, Segment Performance, Capital Structure, Supply Chain, and Competitive Mentions. Examples:

Template	Implies
”What did [COMPANY]‘s CEO say about [TOPIC] on the earnings call?“	transcripts, earnings call
”What management changes did [COMPANY] announce in their 8-K?“	filings, 8-K
”Find mentions of [MENTIONED_ENTITY] in [COMPANY]‘s 10-K.”	filings, 10-K, dual-entity constraint

4.4 Slot filling with a financial knowledge graph

We populate these templates with structured slot values drawn from curated datasets and our internal knowledge graph, including thousands of global companies, other entities such as people or places, macro topics, sources, and temporal expressions. Which companies fill the slots is itself refreshed for every run (see Section 4.7). Because companies and entities are tied to canonical IDs, the generated queries inherit deterministic intent labels. Example generated query: “Find references to Home Depot in Stanley Black & Decker’s 10-K.” Attached deterministic labels:

document_type    = filings
document_subtype = 10-K
reporting_entity = CE1002 (Stanley Black & Decker)
mentioned_entity = ACDF88 (Home Depot)

When we say “deterministic ground truth” we mean deterministic query-intent labels, not an exhaustive list of every relevant document in the world.

4.5 LLM-driven language variation

Raw templates are too clean. To make the eval realistic, we use LLMs to rewrite filled templates into natural-language variants with analyst shorthand, alternate word order, abbreviations, and conversational phrasing. For example:

“Did JPM talk about delinquencies on the Q3 ‘25 call?”
“Anything from Bloomberg on Uber’s regulatory issues?”
“How did Domino’s frame reshoring on the Q2 call?”

This step is important because retrieval systems may fail not on the underlying intent, but on how naturally or ambiguously the query is phrased.

4.6 Validation, balancing, and QA

A good benchmark needs data-quality controls. Our pipeline includes template-slot compatibility checks, canonicalization checks, alias-to-ID validation, temporal validity checks, and some human spot reviews (not exhaustive). We also curate slices for difficult cases such as:

dual-entity queries,
adjacent-quarter confusion,
source-specific asks,
section-specific requests,
implicit-intent queries,
time-relative “latest/current” queries,
and no-answer queries.

Grounding query generation in company identities, reporting periods, document types, and sources helps address common ambiguities in financial search. Historical queries require particular care: company names and legal structures can change, and mapping a past question to today’s entity metadata can silently distort its meaning. Other important distinctions include fiscal versus calendar periods, point-in-time versus period-flow quantities, GAAP versus non-GAAP measures, units and currencies, share classes, and market conventions such as onshore versus offshore FX rates. These considerations inform our approach to financially meaningful queries. We also view retrieval complexity through three common analyst workflows: current-information retrieval, historical lookup, and multi-document evidence gathering. These emphasize freshness and recency resolution; issuer-, period-, and filing-level correctness; and evidence coverage across multiple periods or sources, respectively.

4.7 Time-aware query generation and rolling refresh

The results in this paper are a snapshot, but the underlying system is not static. Finance is highly time-sensitive. Queries such as “latest earnings call,” “current guidance,” or “Q2 results” only make sense relative to a specific moment. A benchmark that keeps asking only about older filings and transcripts becomes stale and artificially easier over time as those documents become better indexed, more widely summarized, and more heavily discussed elsewhere. Time-dependent relevance has long been recognized in temporal information retrieval (e.g., Campos et al., 2014). Financial search inherits that problem, but with higher stakes. In finance, freshness is a component of correctness. A result can be perfectly readable and topically related, yet still ground a materially wrong answer if it reflects a superseded guidance range, an older capital structure, or the wrong reporting period. For that reason, every query in FinSearch-Eval carries a formulation timestamp. Relative temporal expressions such as “latest quarter” are normalized against that timestamp, and the associated deterministic labels are resolved as of that moment. This lets us version-control the benchmark, preserve historical queries, and continuously generate fresh evaluation sets as new filings and transcripts arrive. In that sense, FinSearch-Eval functions more like a rolling regression suite for financial retrieval than a one-time leaderboard. This time-awareness also enables checks for temporal leakage, often referred to in finance as look-ahead bias. Without explicit query timestamps, a search system can appear to answer a historical query correctly by using a document that did not yet exist when the query would have been asked. That risk is especially important in offline evaluation. When iterating on search models or comparing providers against historical query sets, query timestamps must be paired with timestamped source content, as provided by bigdata.com, to check whether retrieved evidence was available when each query was asked. Preventing future information from leaking into historical evaluations is therefore a requirement for fair and reproducible measurement.

Choosing which companies to ask about: spotlight and anchor sets

Templates decide what kind of question is asked; the entity set decides about whom. A fixed company list drifts away from what analysts are researching this month, and it under-tests freshness, because the documents that matter most (this week’s earnings calls, filings and breaking news) are exactly the ones a static list rarely points to. Each refresh therefore draws its companies from two equally sized groups. Spotlight companies are companies with an unusual burst of activity just before the run. For each candidate we compare its document volume in the most recent week with its own baseline over the previous twelve weeks, separately for earnings-call transcripts, filings and news. Comparing a company with itself rather than with the market avoids picking the largest names every month: a mid-cap that just reported, or a company at the center of a news story, outranks a mega-cap having a normal week. Candidates come from our curated universe plus companies discovered by co-mention analysis of that week’s documents, so emerging names can enter the set. A few rules keep the signal meaningful:

Calls and filings count only documents the company itself filed or hosted, so a company is not “spotlight” because a fund’s holdings report or a peer’s call mentions it.
A minimum level of baseline news coverage filters out noisy signals.
Discovered companies must be listed operating companies; publishers and newswires are excluded.
Spotlight slots are split across the three document types, so the set isn’t dominated by news.

Anchor companies are a fixed, deterministic sample of the curated universe, reused in every refresh. They act as a control group: when scores move, anchors show whether the search system changed or this month’s questions were simply harder. Spotlight companies are not just dropped into generic templates. Each one is paired only with templates of the document type that made it spotlight, and the generation model receives the titles and dates of that company’s latest relevant documents, with instructions to ask about one of those events: the quarter of the call it just held, the filing it just submitted, or the subject of the headline. The result is questions like “What did PepsiCo say about pricing on its Q3 2026 earnings call?” in the week of that call, rather than questions about periods that are months old.

4.8 No-answer queries and abstention

Financial search quality is not only about finding the right document when it exists. It is also about correctly recognizing when the requested evidence does not exist. We therefore generate a dedicated class of no-answer queries whose correct behavior is empty retrieval. The importance of abstention is well established in QA and evidence-based evaluation settings. SQuAD 2.0 introduced unanswerable questions as a first-class benchmark component, and FEVER made “not enough information” an explicit evaluation outcome (Rajpurkar et al., 2018; Thorne et al., 2018). FinSearch-Eval brings the same principle down to the retrieval layer for live financial search. These are not random nonsense prompts. They are realistic counterfactuals built from valid financial intents, but instantiated in ways that should not produce evidence-bearing results. For example:

“Show me mentions of Spain in Tesla’s Q4 2025 earnings call”

If Spain was not actually mentioned in that transcript, then the correct retrieval behavior is not a near miss, such as a generic passage about Tesla’s international operations. This matters because many search systems are optimized to avoid empty result pages. That bias may be acceptable in consumer web search, but it is dangerous in enterprise RAG. Near-miss chunks consume context-window budget, dilute precision, and give downstream LLMs more opportunities to synthesize unsupported claims. In this sense, the preferred behavior is fail-closed retrieval: if the system cannot find fresh, verifiable, in-scope evidence, it should abstain rather than approximate. For rigor, our no-answer queries are concentrated in scopes where absence can be validated with high confidence, such as a specific transcript or filing, a specific source, or a specific reporting period or entity mention.

5. An open benchmark for financial search

Financial search needs two complementary evaluation modes:

Timestamped queries and annotations for consistent evaluation across systems.
Rolling, time-aware evals for realistic monitoring on a live corpus.

The public release contains 638 queries generated on 26 October 2026, with timestamps and structured labels. We will expand the same dataset with new timestamped queries, retaining earlier queries and annotations for comparison. Internally, the rolling framework continues to refresh queries and preserve future-period holdouts for ongoing regression testing. One advantage of a live benchmark is that future financial periods create natural hidden holdouts. Search providers can tune on earlier queries, but they cannot memorize next quarter’s earnings calls before those calls exist. Rolling evaluation is therefore useful not only for monitoring retrieval quality over time, but also for testing whether systems continue to generalize as the corpus evolves.

6. Scoring methodology: deterministic where possible, LLM-based where necessary

No single scoring method is enough for this problem. We therefore use multiple layers:

Rule-based scoring for fields where correctness is structural.
LLM-as-a-judge scoring for semantic answerability.
Abstention metrics for queries whose correct outcome is empty retrieval.
A composite score that combines the previous metrics.

In a live financial corpus, many queries also have more than one acceptable supporting document. A question may be answered by a 10-Q, an 8-K furnishing the earnings release, or the earnings call transcript. For that reason, FinSearch-Eval does not reduce correctness to a single labeled document ID. Instead, it scores whether retrieved evidence satisfies the required financial constraints and whether the returned chunk is actually answer-bearing. Because live financial corpora rarely admit exhaustive relevance labels, and because many queries have multiple acceptable supporting documents, we do not score against a single canonical gold document or closed-world recall. Instead, our deterministic evaluation emphasizes structured correctness within the top-k returned results, paired with semantic scoring for answerability.

6.1 Tier 1: Rule-based field precision

For labelled retrieval queries, we inspect up to k returned chunks. Each field score is the number of matching chunks divided by the number evaluated. Our main reported metrics are:

Document Type Precision@k
Document Subtype Precision@k
Reporting Entity Precision@k
Reporting Period Precision@k
Mentioned Entity Precision@k
Source Precision@k

Why foreground precision-style reporting so heavily? In many consumer search settings, it is acceptable if the right result appears somewhere on the page and the human user performs the final selection. In financial RAG, the retrieved set is itself the model’s evidence budget. Every wrong-entity, wrong-period, or wrong-source chunk directly competes with the correct evidence for attention and context budget. Field scores are averaged across labelled retrieval queries that constrain that field; unconstrained fields are excluded. A Reporting Entity Precision@20 of 90% therefore means that the average query-level score for reporting-entity correctness is 90%. A chunk that is too ambiguous to verify does not receive partial credit. If the evaluation pipeline cannot recover or infer the required metadata, that result fails the corresponding rule-based check.

6.2 Why Top-20?

Top-k is only a proxy for what downstream systems actually care about. Context windows are budgeted in tokens, not result counts. Two providers can both return 20 results, yet one may return long, noisy excerpts while another returns compact, self-contained evidence spans. For model-facing search, that distinction matters: finer-grained retrieval lets the downstream system fit more distinct evidence into the same context budget. That is one reason we treat passage-level retrieval as a first-class problem rather than assuming document-level ranking is sufficient. This emphasis on compact, passage-level evidence is also consistent with passage retrieval and late-interaction retrieval work, which treats fine-grained matching at passage granularity as a first-class IR problem (Karpukhin et al., 2020; Khattab and Zaharia, 2020). We use Top-20 as a common result budget across providers. bigdata.com supports higher result limits, with native metadata filters that enforce the selected constraints across all returned results. This helps preserve structural precision as the result set grows, making Top-20 an evaluation choice rather than a product limit.

6.3 Tier 2: Pointwise LLM-as-a-judge for answerability

Correct document metadata is necessary, but not sufficient. A chunk can come from the correct filing and still fail to answer the query. We therefore score each (query, chunk) pair with a pointwise LLM judge using a simple rubric:

Score	Meaning
4	Directly answers the query
3	Highly useful context, but incomplete
2	Weakly relevant or tangential
1	Not relevant

This follows broader work on rubric-based LLM evaluation and LLM-as-a-judge methods (Liu et al., 2023; Zheng et al., 2023; Hashemi et al., 2024). We use a deliberately narrow pointwise setup because it is easier to audit, more scalable, and less sensitive to long-context effects than open-ended listwise judging. For time-relative queries, the judge receives both the query timestamp and the result timestamp, so expressions such as “latest” or “current” are interpreted relative to the correct moment. Scoring is intentionally conservative: a chunk receives a 4 only if it can largely stand on its own as direct evidence, not merely because it is topically adjacent. As in broader LLM-as-a-judge work, we treat the judge as a measurement instrument rather than an oracle. Judge outputs can vary with rubric design, prompt wording, and repeated evaluation, which is why prompt versioning, spot audits, and calibration checks are part of our process (Hashemi et al., 2024; Haldar and Hockenmaier, 2025).

6.4 Measuring abstention accuracy

For queries where no qualifying evidence is expected within scope as of the query timestamp, we measure abstention accuracy. Each query receives:

1 if the system returns no results.
0 if the system returns any results.

Abstention accuracy is the average of these binary scores across the abstention-query slice. It is independent of the Top-k budget: returning even one result counts as a failure to abstain. This matters because search APIs designed to always return results may supply plausible-looking near-matches when no qualifying evidence exists. Abstention accuracy distinguishes that behavior from correctly returning nothing. In financial RAG, “something close” can be worse than nothing: unsupported results consume context budget, increase the model’s evidence-selection burden, and may make downstream synthesis less stable.

6.5 Reading failures by failure type

One reason we isolate retrieval is that “bad search” is not one failure mode. In practice, we see at least four major error types:

Coverage failure: The relevant content is not in the search system’s corpus. In black-box comparisons this is not always cleanly separable from the failure types below, because one rarely has full visibility into each system’s corpus coverage.
Constraint failure: The result is about the wrong entity, period, source, document type, or subtype.
Passage failure: The right constraints were satisfied, but the returned chunk is not answer-bearing.
Abstention failure: No valid answer exists in scope, but the system returns near-misses anyway.

This taxonomy is useful both for debugging retrieval systems and for interpreting benchmark results more honestly. In finance, several concrete manifestations of these failures recur frequently: stale or mis-timestamped evidence, report-calendar misalignment (for example, confusing fiscal and calendar periods), reliance on secondary summaries instead of primary disclosures, and overly shallow search that never reaches the relevant source.

6.6 Composite score

We measure a query-level composite@k score that combines structural correctness (Tier 1), semantic usefulness (Tier 2), and abstention behavior. For labelled retrieval queries, we evaluate up to k returned chunks using the following rules:

A chunk receives its LLM-judge score only if all Tier 1 constraints are satisfied.
If any Tier 1 constraint fails, the chunk is assigned a score of 1, regardless of the judge’s rating.

The query-level score averages these adjusted chunk scores. For no-answer queries, empty retrieval receives 4 and any returned evidence receives 1. The overall composite averages all query-level scores equally. Let n_q be the number of returned results evaluated for query q, capped at k. For each chunk i, J_qi is its judge score on the 1–4 scale, and s_qi is its adjusted score after applying all required structural constraints. The query-level composite is C_q@k, and Q_eval denotes the set of evaluated queries.

s_qi equals J_qi when all applicable structural constraints pass, and 1 otherwise. C_q@k equals the mean of s_qi over the n_q results for a labelled retrieval query with n_q greater than 0, 4 for a no-answer query with n_q equal to 0, and 1 for a no-answer query with n_q greater than 0. Composite@k is the mean of C_q@k over all evaluated queries Q_eval. — The adjusted chunk score, the query-level composite and the overall composite.

7. The metadata gap: normalizing heterogeneous provider outputs

Different search systems do not return the same kinds of objects. To compare them fairly, we normalize every result into a common evaluation schema before scoring. Generalist search systems are optimized for breadth across the public web. Financial search can instead optimize for a narrower but more demanding objective: issuer resolution, period correctness, source authority, document identity, and provenance. Strong general web search performance therefore does not automatically translate into evidence-grade financial retrieval. This normalization step is also where the biggest architectural difference between bigdata.com and generic search APIs becomes visible.

The bigdata.com advantage: native structured metadata

When bigdata.com Search returns a result, it comes with rich, deterministic metadata natively attached in the index. bigdata.com search returns indexed metadata that can be compared directly with the benchmark’s intent labels. We know what the document_type, document_subtype, reporting_entity, reporting_period, source are, and which other entities are mentioned, mapped directly to canonical internal IDs. Evaluating these results is therefore a straightforward deterministic match against the query’s ground truth.

Evaluating generalist search APIs: the extraction burden

Generic search APIs typically do not provide this metadata. In most cases, the returned object is some combination of URL, title, timestamp, and snippet text. To evaluate whether a generic search API actually satisfied the constraints of a financial query, for example, whether a snippet is truly from a 10-Q, we pass its outputs through a dedicated normalization pipeline. This pipeline uses an LLM to infer missing metadata from URL structure, title patterns, source domains, temporal cues, and the snippet context itself, then maps the inferred attributes into the common benchmark schema. This reveals an important product gap: missing metadata can be as problematic as incorrect metadata. If a provider returns a result whose document type, reporting entity, or other required field cannot be resolved with sufficient confidence, that field is marked as Unknown. In our scoring, Unknown is an automatic failure for the corresponding rule-based check. That is an intentionally strict standard. In enterprise finance, “probably a 10-Q” is not enough. A retrieved result must be attributable enough to support downstream filtering, ranking, and verification. This extraction layer is not merely a benchmarking inconvenience. It is a real burden pushed onto downstream RAG developers when the retrieval system does not expose the metadata needed to verify or safely use its results. bigdata.com solves that problem natively at the retrieval layer. While this normalization pipeline usually determines pass/fail with high confidence, there are inevitably cases where a provider’s snippet simply lacks enough context for reliable verification. In financial RAG, ambiguity is itself a failure mode. Even if the underlying document might technically satisfy the query, a result that cannot be verified cannot safely ground an answer. In regulated and enterprise workflows, provenance is not only a transparency feature; it is part of auditability. A retrieved chunk must be attributable enough for an analyst, model, or reviewer to reopen the source and verify what was said, when, and by whom. This is not only an evaluation issue. Recent financial tool-use work finds that exposing finance-specific attributes at selection time improves constraint alignment and reduces redundant or mismatched actions (Lu et al., 2026). We see a closely related effect in retrieval: native financial metadata is not only easier to evaluate, it is easier for downstream systems to use safely. Structured fields such as issuer, period, source, and document subtype make conservative filtering and ranking possible before irrelevant or risky evidence enters the context window.

8. Illustrative retrieval failure modes

The following examples illustrate how topically relevant results can fail the financial constraints of a query. Example 1: primary-source extraction Query: “What is Moderna’s revenue breakdown by product in their 10-Q?” Typical failure mode: source-class mismatch. A news summary may sound useful, but it does not satisfy the requirement for the underlying filing. Example 2: temporal + transcript intent Query: “Did JPMorgan mention credit card delinquency trends in their Q3 2025 earnings call?” Typical failure mode: constraint failure. The system retrieves the right topic, but the wrong issuer, wrong period, or wrong document class. Example 3: correct abstention on a no-answer query Query: “Show me mentions of Spain in Tesla’s Q4 2025 earnings call.” Correct behavior: If the specified transcript contains no qualifying mention of Spain, the correct behavior is abstention. These examples distinguish topical relevance from usable evidence: a result can fail through the wrong source, issuer, period, or an unsupported near-match.

9. Benchmark results: bigdata.com vs. generalist AI search APIs

Using the methodology above, we compared bigdata.com Search against several generalist AI search providers on the public FinSearch-Eval dataset generated on 26 October 2026. Results reflect the provider endpoints and configurations used in this evaluation.

Top-k field precision and judge scores

All metrics are computed over the top 20 results. The first six rows are field precision@20.

Metric @20	bigdata.com	P1	P2	P3	P4	P5
Doc subtype	90.9%	34.0%	25.1%	34.9%	22.0%	35.8%
Doc type	87.6%	46.4%	38.3%	44.7%	38.1%	49.6%
Mentioned entity	80.9%	69.4%	76.8%	76.9%	72.3%	75.7%
Reporting entity	94.1%	56.9%	53.6%	62.8%	63.9%	66.0%
Reporting period	88.5%	48.2%	54.0%	64.0%	50.5%	66.4%
Source	55.0%	31.5%	15.3%	26.3%	30.0%	37.3%
Abstention	99%	0%	0%	0%	0%	0%
Judge score (1–4)	2.5	2.8	2.5	2.9	2.4	2.9
Composite	2.2	1.49	1.34	1.59	1.40	1.61

Results as of 26th of October, 2026. The list of providers include, in a random order: Perplexity, Tavily, You.com, Parallel AI and Exa.AI, used in the default Search modes. The gap is clear when looking at the structured financial intent resolution: issuer, period, source, and document-level correctness under a constrained evidence budget. Those are precisely the dimensions where broad semantic search often underperforms in financial settings. The LLM-judge does not show this gap: this is because semantically adjacent chunks can still look useful to a pointwise judge even when it fails issuer, period, source, or document constraints. That is why we pair semantic judging with strict structural scoring. When reported, the composite score makes that difference more explicit. Under limited RAG context budgets, quality near the top of the ranking is particularly important. We observe the same qualitative pattern on internal slices for time-relative and no-answer queries, where systems optimized to always return something are especially vulnerable.

10. Evaluation platform and run reproducibility

Our evaluation platform supports query generation, result capture, normalization, scoring, and cross-provider comparison. Each run records query metadata, raw provider responses, normalized results, capture timestamps, scoring outputs, and the evaluation configuration. These captured traces support auditing and rescoring without repeating live searches. Query timestamps and bigdata.com’s timestamped content support evaluation against the query’s temporal constraints. However, the searchable corpus evolves, including through source retention policies. A later search may therefore return different evidence; captured runs preserve the basis for interpreting the original results.

11. What this benchmark does not measure and future work

FinSearch-Eval, in its current form, does not measure final answer style or fluency, multi-hop synthesis across many documents, table-structure extraction fidelity, or full end-to-end reasoning quality. It is a system-level black-box benchmark, so results reflect not only ranking quality, but also corpus coverage, freshness, parsing, chunking, and metadata availability. The object of evaluation is the search system itself, not the downstream reasoning agent. At the same time, we do not claim to prove global non-existence across the open web for unconstrained no-answer queries. Our no-answer slice is deliberately concentrated in bounded scopes where absence can be validated with high confidence. Strict abstention is partly constrained by provider API behavior; some APIs may be designed to always return something. A search system can score highly on retrieval and still feed evidence into a weak generator that produces a poor answer. Conversely, a strong generator can sometimes mask retrieval defects in end-to-end demos. That is precisely why retrieval deserves separate evaluation. No benchmark is complete. One limitation of FinSearch-Eval is that its queries are synthetically generated rather than drawn directly from real user traffic. This gives us scale, coverage, and deterministic labels, but it does not fully reproduce the ambiguity, noise, and distributional quirks of production usage. We mitigate that by combining templates, knowledge-graph grounding, LLM-based variation, and QA filters, but it remains an approximation rather than a direct sample of live traffic. Important next steps include:

expanding the public benchmark size and query coverage;
creating a submission service for hidden holdout splits;
incorporating section-specific and table-specific retrieval evaluation;
measuring freshness latency more explicitly, i.e., how quickly a new filing or transcript becomes retrievable.

FinSearch-Eval today focuses on document retrieval: finding relevant passages in news, filings, transcripts, and other textual sources. Some financially important questions, however, cannot be answered from documents alone. They require computation over structured or real-time data, such as market prices, financial statements, and continuously updated datasets. Questions about current valuations, price moves, or derived metrics depend on stateful data, not just text. Evaluating those cases requires a different benchmark design, with different assumptions about freshness, data access, and computation. A natural extension is to evaluate retrieval under a fixed token budget rather than only a fixed result count. Top-k is convenient and comparable across providers, but it does not capture a key difference between model-facing search systems: the granularity and compactness of the returned evidence. A provider that returns shorter, self-contained, answer-bearing spans may be materially more useful than a provider that returns longer snippets, even if both return the same number of results. More broadly, this points toward retrieval under explicit context budgets and selective retention policies rather than fixed result counts alone, echoing work on hierarchical or virtual context management for long-horizon language-model tasks (Packer et al., 2023). We therefore view token-normalized utility and support-under-budget as important future evaluation directions. A further extension would be to study downstream answer variance as a function of retrieval set size and retrieval noise. Recent work suggests that longer reasoning trajectories can produce more incoherent failures (Hägele et al., 2026). An important open question for financial RAG is how much noisier retrieval sets increase answer instability even when average relevance appears acceptable. FinSearch-Eval focuses on the retrieval layer itself, but it would be valuable to connect future benchmark variants to cost/latency/quality trade-offs in downstream systems. Recent controlled benchmarks such as BrowseComp-Plus freeze a curated corpus to isolate retrieval more cleanly (Chen et al., 2025). That is valuable for controlled experiments. FinSearch-Eval makes a different trade-off by evaluating retrieval against a live, continuously updating financial corpus so that freshness, temporal correctness, and real-world corpus dynamics remain part of what is being measured.

12. Conclusion

FinSearch-Eval is designed to evaluate the retrieval layer under the constraints that matter in financial AI: issuer and period correctness, source authority, passage-level answerability, time-aware evaluation, and abstention when no valid evidence exists. We view the current release as the first public set of queries and annotations from a broader rolling benchmark. By progressively publishing these alongside methodology and scoring logic, we aim to make financial retrieval evaluation more transparent, comparable, and auditable. The first public release of FinSearch-Eval contains the queries and annotations used in this evaluation. We plan to expand it with additional queries over time.

Appendix: Benchmark Card and Dataset Schema

The FinSearch-Eval Benchmark Card

Intended use: Evaluating financial search on structured constraint satisfaction, passage-level answerability, and abstention, before downstream answer generation.
Dataset: 638 queries generated on 26 October 2026: 565 labelled retrieval queries and 73 abstention queries. The results in this paper use this public release.
Contents: Queries, formulation timestamps, and structured intent labels. Source documents and captured search results are not included.
Provenance: Claude Sonnet generated queries from financial templates and company reporting periods. Company and source names were resolved through the Bigdata.com Knowledge Graph.
Temporal scope: Relative expressions and evidence eligibility are evaluated against each query’s timestamp. Query timestamps must be paired with timestamped source content, as provided by bigdata.com. Searchable coverage evolves, including through retention policies, so later runs may return different evidence.
Scoring: Structured labels support field-level correctness checks. Passage-level answerability is assessed separately, as described in Section 6. Abstention queries expect empty retrieval within the specified scope as of the query timestamp.
Limitations: This release was generated automatically without extensive manual review. Labels and entity mappings may contain errors.
Out of scope: Downstream generation quality, agent orchestration, and reconstruction of historical search-index states.

Dataset Schema

Field	Description
`query`	Financial search query.
`query_timestamp`	Query generation time in UTC; the reference for temporal interpretation.
`abstention`	Whether empty retrieval is expected within scope as of the query timestamp.
`document_type`	Acceptable document types.
`document_subtype`	Acceptable document subtypes.
`reporting_entities`	Canonical names of companies that issued, filed, or hosted the expected documents.
`reporting_period`	Acceptable reporting-period labels; any listed value may match.
`reporting_period_all`	Reporting-period labels that must all be present.
`entities`	Canonical names of companies that qualifying chunks must mention.
`source`	Required publisher names, where specified.

Label fields are lists; an empty list indicates an unconstrained attribute. Abstention rows carry no structured intent labels. Public dataset: RavenPack/finsearch-eval

References

Back, S., et al. (2026). Understanding LoRA as Knowledge Memory: An Empirical Analysis. ICML 2026.
Bajaj, P., et al. (2016). MS MARCO: A Human Generated MAchine Reading COmprehension Dataset.
Bigeard, A., Nashold, L., Krishnan, R., & Wu, S. (2025). Finance Agent Benchmark: Benchmarking LLMs on Real-world Financial Research Tasks.
Buckley, C., & Voorhees, E. M. (2004). Retrieval Evaluation with Incomplete Information. Proceedings of SIGIR 2004.
Campos, R., Dias, G., Jorge, A., & Jatowt, A. (2014). Survey of Temporal Information Retrieval and Related Applications. ACM Computing Surveys.
Chen, Z., et al. (2021). FinQA: A Dataset of Numerical Reasoning over Financial Data. EMNLP 2021.
Chen, Z., et al. (2022). ConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answering. EMNLP 2022.
Chen, Z., et al. (2025). BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agent.
Chen, Z., et al. (2026). AgentIR: Reasoning-Aware Retrieval for Deep Research Agents. arXiv:2603.04384.
Cleverdon, C. W. (1967). The Cranfield Tests on Index Language Devices.
Es, S., James, J., Espinosa-Anke, L., & Schockaert, S. (2024). Ragas: Automated Evaluation of Retrieval Augmented Generation.
Guu, K., et al. (2020). REALM: Retrieval-Augmented Language Model Pre-Training. ICML 2020.
Hägele, A., Gema, A. P., Sleight, H., Perez, E., & Sohl-Dickstein, J. (2026). The Hot Mess of AI: How Does Misalignment Scale With Model Intelligence and Task Complexity?
Haldar, R., & Hockenmaier, J. (2025). Rating Roulette: Self-Inconsistency in LLM-As-A-Judge Frameworks. Findings of EMNLP 2025.
Hashemi, H., Eisner, J., Rosset, C., Van Durme, B., & Kedzie, C. (2024). LLM-Rubric: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Texts. ACL 2024.
Hsieh, C.-P., et al. (2024). RULER: What’s the Real Context Size of Your Long-Context Language Models?
Hu, L., et al. (2025). FinSearchComp: Towards a Realistic, Expert-Level Evaluation of Financial Search and Reasoning.
Hu, T., et al. (2026). SAGE: Benchmarking and Improving Retrieval for Deep Research Agents.
Izacard, G., et al. (2022). Atlas: Few-shot Learning with Retrieval Augmented Language Models.
Järvelin, K., & Kekäläinen, J. (2002). Cumulated Gain-Based Evaluation of IR Techniques. ACM Transactions on Information Systems, 20(4), 422–446.
Karpukhin, V., et al. (2020). Dense Passage Retrieval for Open-Domain Question Answering.
Khattab, O., & Zaharia, M. (2020). ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT.
Lewis, P., et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. NeurIPS 2020.
Liu, N. F., et al. (2023). Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics, 12, 157–173.
Liu, Y., et al. (2023). G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment.
Lu, J., et al. (2026). FinToolBench: Evaluating LLM Agents for Real-World Financial Tool Use.
Miroyan, M., et al. (2025). Search Arena: Analyzing Search-Augmented LLMs.
Packer, C., et al. (2023). MemGPT: Towards LLMs as Operating Systems.
Petroni, F., et al. (2021). KILT: A Benchmark for Knowledge Intensive Language Tasks. NAACL 2021.
Rajpurkar, P., Jia, R., & Liang, P. (2018). Know What You Don’t Know: Unanswerable Questions for SQuAD. ACL 2018.
Saad-Falcon, J., Khattab, O., Potts, C., & Zaharia, M. (2024). ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems.
Tan, J., et al. (2025). HierSearch: A Hierarchical Enterprise Deep Search Framework Integrating Local and Web Searches.
Thakur, N., et al. (2021). BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models. NeurIPS Datasets and Benchmarks 2021.
Thorne, J., Vlachos, A., Christodoulopoulos, C., & Mittal, A. (2018). FEVER: a Large-scale Dataset for Fact Extraction and VERification. NAACL 2018.
Voorhees, E. M., & Harman, D. K. (Eds.). (2005). TREC: Experiment and Evaluation in Information Retrieval. MIT Press.
Wang, Y., et al. (2026). DeepResearchEval: An Automated Framework for Deep Research Task Construction and Agentic Evaluation.
Wei, J., et al. (2025). BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents.
Zheng, L., et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS 2023.
Zhu, F., et al. (2021). TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Finance. ACL-IJCNLP 2021.
Zhu, F., et al. (2025). FinDeepResearch: Evaluating Deep Research Agents in Rigorous Financial Analysis.

Ricard Matas Navarro

Director of Search & Retrieval (SVP)

Hugo Jiménez Muñoz

Senior Machine Learning Engineer

Sandra Bullón

Senior Product Manager