Skip to main content

April 25, 2026

Search, API, Screening/8 minutes read
When teams screen a large universe for a theme, the first reliable pattern is usually a full grid: split the universe into manageable company groups, split the time range into windows, run Search across every slice, and collect the results. It is comprehensive, auditable, and easy to reason about. The challenge is allocation. A full grid gives every company-time slice the same request budget even though coverage is rarely distributed evenly. Some names and periods carry most of the relevant evidence. Others have little or no coverage for the theme being screened. When the universe has thousands of names, uniform allocation can turn a straightforward research question into a long-running retrieval job. Smart Batching is a planning layer for Bigdata.com Search that makes this allocation adaptive. It estimates volume before execution, groups entities with similar expected coverage, splits high-volume periods more carefully, and then runs Search with a proportional budget. The underlying retrieval remains Bigdata.com Search; Smart Batching decides where to spend the calls. In benchmarks across three universe-scale screens, this plan-then-execute pattern cut runtime from several minutes to seconds: 10.6x to 21.6x faster at the 100% planned budget, and 14.6x to 44.3x faster at the 1% budget. The highest-relevance bins were strongly preserved in the tested scenarios, while lower budget settings concentrated the output into a smaller, more focused result set.

​
The Problem: Uniform Budgets

Imagine a six-month screen for companies affected by US import tariffs against China. Some semiconductor, retail, industrial, and logistics names will have meaningful coverage. Many other companies will have little direct exposure in the news flow for that topic. A full-grid search handles this by being exhaustive: each company group and time window gets queried. That is often the right baseline when completeness is the priority. But if the workflow is a recurring monitor, alert generator, LLM verification queue, or dashboard refresh, you may want a way to preserve the evidence-rich parts of the universe without spending the same budget on every quiet slice. Smart Batching addresses that operational problem. It is not a replacement for Search quality. It is a planner that makes large Search jobs more aware of the distribution of expected content.

​
How Smart Batching Works

Smart Batching has two phases. The planning phase has two distinct substeps.
Schematic comparing full grid search with Smart Batching planning and execution
Step 1a: Group entities into volume baskets. The system queries the Bigdata.com Search Volume API to estimate how many chunks each entity will produce for a given topic over the full time range. It then groups entities into baskets based on expected volume, merging low-volume entities together and isolating high-volume entities when needed. This preserves the cross-sectional distribution of the universe: companies are not flattened into a single bucket, and the names that dominate coverage do not crowd out the long tail. Step 1b: Break each basket down by time. For each basket, the planner pulls the expected-volume time series and partitions the full date range into batch periods. High-volume periods get shorter windows; low-volume periods can use longer windows. This preserves temporal structure the same way Step 1a preserves cross-sectional structure: periods of heavy coverage get their own batches, and quiet periods are consolidated so the executor does not spend unnecessary requests on sparse windows. Step 2: Execute. The executor turns the plan into Search calls. Each basket receives a max_chunks allocation proportional to its expected volume and the selected chunk_percentage. A 10,000-chunk basket at chunk_percentage=0.01 requests about 100 chunks. A 50-chunk basket still receives a minimum budget so it remains represented. That chunk_percentage setting is the main control:
  • 1.0 runs the full planned retrieval budget.
  • 0.1 is a balanced setting for research workflows and dashboards.
  • 0.01 is a signal-first setting for alerts, triage, and LLM verification.
Together, Step 1a and Step 1b produce an upper-bound estimate of how many chunks the execution phase will retrieve before the larger Search calls are made. That makes latency and downstream processing costs easier to manage.

​
Benchmark Design

We tested Smart Batching against a full-grid baseline on three screens over the historical US top-company universe:
  • Tariffs China 2025: “The company is affected by US import tariffs against China,” January to June 2025.
  • Leadership 2023: “The company has faced leadership changes and executive appointments,” September to December 2023.
  • Confidence Decline 2021: “Decline in customer confidence in the company,” January to June 2021.
The baseline split date ranges into windows and queried batches of 10 entities. Smart Batching used volume-based planning and was tested at 100%, 10%, and 1% planned chunk budgets. All methods used the same Bigdata.com API environment for the comparison.

​
Speed: minutes to seconds

Benchmark execution time comparison for full grid and Smart Batching
The timing results are the simplest way to see the effect. Tariffs China 2025 went from 452.2 seconds with full grid to 42.5 seconds with Smart Batching at 100%, 32.0 seconds at 10%, and 30.9 seconds at 1%. That is a 10.6x to 14.6x speedup. Leadership 2023 went from 323.2 seconds to 20.7 seconds at 100%, 10.2 seconds at 10%, and 9.4 seconds at 1%. That is a 15.6x to 34.4x speedup. Confidence Decline 2021 went from 451.6 seconds to 20.9 seconds at 100%, 11.0 seconds at 10%, and 10.2 seconds at 1%. That is a 21.6x to 44.3x speedup. For a team running daily or weekly screens, this changes the shape of the workflow. A retrieval step that used to block a pipeline for several minutes becomes a sub-minute component that can feed monitoring, dashboarding, or verification layers.

​
What happens to relevance?

Runtime alone is not enough. The real question is what kind of evidence remains in the result set. The scatter plots below show per-chunk relevance over time for each benchmark. The important pattern is that lower chunk_percentage settings reduce the volume of returned chunks and concentrate the output into higher-ranked material. At the same time, Smart Batching still samples through the planned baskets, so the result is not simply one global top-N list.
Tariffs China 2025 date versus relevance scatter plots
For Tariffs China 2025, full grid returned 81,037 chunks with average relevance of 0.049. Smart Batching returned 60,135 chunks at 100%, 10,825 at 10%, and 1,188 at 1%. Average relevance rose to 0.130 at 10% and 0.331 at 1%.
Leadership 2023 date versus relevance scatter plots
For Leadership 2023, full grid returned 9,033 chunks with average relevance of 0.086. Smart Batching returned 7,591 chunks at 100%, 1,070 at 10%, and 100 at 1%. Average relevance rose to 0.109 at 10% and 0.273 at 1%.
Confidence Decline 2021 date versus relevance scatter plots
For Confidence Decline 2021, full grid returned 18,634 chunks with average relevance of 0.036. Smart Batching returned 16,923 chunks at 100%, 2,503 at 10%, and 295 at 1%. Average relevance rose to 0.046 at 10% and 0.111 at 1%. One nuance matters: the 100% Smart Batching runs can have lower average relevance than the full-grid reference because the allocation structure is different. The goal of 100% Smart Batching is efficient planned coverage, not necessarily a higher average relevance score. The higher-concentration effect becomes clearest when the budget is intentionally reduced.

​
Coverage: What You Keep

The coverage charts compare Smart Batching against the full-grid reference by relevance bin. They show how much of each reference bin appears in the Smart Batching output.
Tariffs China 2025 coverage by relevance range
In the tariff screen, Smart Batching preserved nearly all chunks in the highest reference bins at 10% and 1%, while aggressively reducing the lowest-relevance bin at smaller budgets.
Leadership 2023 coverage by relevance range
In the leadership screen, the upper relevance bins were also strongly retained. The 1% setting returned only 100 chunks, but still preserved the highest reference bin in the benchmark.
Confidence Decline 2021 coverage by relevance range
In the confidence-decline screen, which was the sparsest topic, the 10% setting preserved the mid and upper reference bins in the chart while reducing the lowest bin sharply. The 1% setting was much more selective, which is useful for triage but may be too aggressive for workflows that need broad moderate-relevance coverage.

​
Choosing a budget

The best chunk_percentage depends on what happens downstream. Use 1% when the output feeds a human triage queue, alerting workflow, or LLM verification step and the goal is to start with the highest-conviction evidence. Use 10% when you are building entity-level scores, dashboards, or time-series views and want a stronger balance between speed, relevance, and coverage. Use 100% when completeness matters and you still want the operational benefit of planned grouping and volume-aware splitting.

​
Get Started

The full implementation is available in the Smart Batching folder of the Bigdata.com cookbook repository, including the technical notebook. The reusable Python library is published on PyPI as bigdata-smart-batching. The notebook walks through environment setup, planning, execution, deduplication, plan persistence, and DataFrame conversion using the bigdata-smart-batching Python package. If you are integrating it into your own project, install it with uv add bigdata-smart-batching. For teams already running large Search workflows on Bigdata.com, Smart Batching is a drop-in pattern for making those workflows more predictable: plan first, choose a budget, execute only what the research task requires, and keep the result grounded in Bigdata.com Search. This content is for informational purposes only and does not constitute investment advice.
Alberto Martinez

Alberto Martinez

Senior Data Engineer

Francisco Gomez

Francisco Gomez

Manager of Applied Intelligence (VP)