The True Scale of Market Intelligence in the AI Era

While OpenAI’s Deep Research makes headlines with its ability to scan hundreds of sources in just 10 minutes, the Bigdata API operates at a fundamentally different scale - searching across billions of documents and delivering up to 1 million relevant results for comprehensive analysis.

In a world where OpenAI’s solution might skim the surface with limited document retrieval, true market intelligence requires industrial-strength pipelines capable of processing orders of magnitude more data with deeper reasoning capabilities. The difference isn’t just quantitative - it’s qualitative. When your competitors are making decisions based on thousands of documents, you’ll be identifying patterns, anomalies, and opportunities across millions.

We demonstrate how to harness the Bigdata.com API to retrieve and process 1 million documents for Russell 1000 companies in minutes transforming raw data into actionable intelligence at a scale that traditional search paradigms simply cannot match. Whether you’re performing sentiment analysis, tracking emerging market trends, or building sophisticated entity relationship networks, the ability to process data at this magnitude represents the new frontier of competitive advantage.

We’ll show you how to: - Process documents across the Russell 1000 companies focusing on Trump tariff impacts - Analyze sentiment by sector to provide comprehensive macro color - Visualize which sectors have the highest percentage of companies negatively impacted

Welcome to data processing at true enterprise scale.

​

Step 0: Prerequisites

We need to import the Bigdata client library with the supporting modules:

import base64
import time
from io import BytesIO
from datetime import timedelta


import matplotlib.pyplot as plt
import matplotlib.colors as mcolors
import pandas as pd
from IPython.display import display, HTML
from bigdata_client import Bigdata
from bigdata_client.query import Similarity, Entity
from bigdata_research_tools.search import run_search

​

Step 1: Initialization

We begin by initializing the Bigdata client. The authentication is handled through environment variables or can be passed directly:

# Initialize the Bigdata client
# Make sure BIGDATA_USERNAME and BIGDATA_PASSWORD are set in the environment
# Alternatively, you can pass your credentials directly to the Bigdata class
bigdata = Bigdata()

​

Step 2: Creating Queries for Russell 1000 Companies

This is where the magic begins. We’ll create queries for every company in the Russell 1000 index. Each query combines an entity search with a similarity search for relevant content:

portfolio_entities = bigdata.knowledge_graph.get_entities(
    RUSSELL_1000_ENTITIES)

# Create a combined query for each entity
queries = [Entity(entity_id) & Similarity('Trump 2.0 tariffs impact')
           for entity_id in RUSSELL_1000_ENTITIES]
print(f'Number of queries: {len(queries)}')

Number of queries: 1000

​

Step 3: Execute the Search

Now for the heavy lifting. We’ll use concurrent processing to execute all queries efficiently:

def execute_search_concurrent(queries, document_limit=10):
    results = run_search(
        queries,
        limit=document_limit,  # Max number of documents per search query
        only_results=False  # Return query to results mapping
    )
    return results

tic = time.perf_counter()
results = execute_search_concurrent(queries)
toc = time.perf_counter()
elapsed = timedelta(seconds=toc - tic)
print(f"Time taken: {elapsed}")

Querying Bigdata...: 100%|██████████| 1000/1000 [03:44<00:00,  4.46bit/s]

Time taken: 0:03:44.918372

That’s right - we just processed 1 million documents (1,000 queries × 1,000 documents per query) in just minutes. This is the industrial-strength pipeline that drinks from the firehose of market data without choking.

​

Why a Million Documents Matter

Processing vast amounts of data unlocks critical advantages:

1. Comprehensive Market Coverage — Capture signals from every corner of the market.
2. Statistical Significance — Identify trends with greater confidence.
3. Rare Event Detection — Catch the 0.1% of documents that could make or break your strategy.
4. Real-Time Insights — Process breaking news across the entire market in minutes, not days.

​

Bonus: Macro-level Analysis of the Results

With a million documents at your disposal, you can build sophisticated sector-wide sentiment analysis. Thanks to Bigdata API, we can compute the sentiment at the text chunk-level:

company_sentiments = {}
for entity, documents in zip(portfolio_entities, results.values()):
    # Calculate the sentiment ––at the chunk level–– for each queried company
    weighted_sum = sum(chunk.sentiment * chunk.relevance
                       for doc in documents
                       for chunk in doc.chunks)
    sum_of_weights = sum(chunk.relevance
                         for doc in documents
                         for chunk in doc.chunks)
    try:
        company_sentiments[entity.id] = weighted_sum / sum_of_weights
    except ZeroDivisionError:
        company_sentiments[entity.id] = 0

Now we build the DataFrame to further our analysis:

df = pd.DataFrame(vars(pe) for pe in portfolio_entities)
df = df.dropna(subset=['sector', 'industry_group', 'industry'])
df['sentiment'] = df['id'].map(company_sentiments)

# Total number of companies per sector
total_by_sector = df.groupby('sector').size()
# Count companies with a [meaningful] negative sentiment
negative_by_sector = df[df['sentiment'] < -0.45].groupby('sector').size()
# Compute percentage of negatively impacted companies
percentage = (negative_by_sector / total_by_sector * 100).fillna(0)
# Sort the percentages descending (largest percentage first)
percentage = percentage.sort_values(ascending=False)
# Define a color map from the dark red to pure red
custom_red = mcolors.LinearSegmentedColormap.from_list('custom_red',
                                                       [(0.5, 0, 0),
                                                        (1, 0, 0)])
# Normalize the percentage values so they map to [0, 1]
norm = mcolors.Normalize(vmin=percentage.min(), vmax=percentage.max())
# Map each percentage value to a color using the custom colormap
colors = [custom_red(norm(val)) for val in percentage]

Below we just prepare to show the visualization, feel free to skip code if you just want to see the visualization:

Based on this bar char, the Telecom, Consumer goods, and Tech sectors appear to be the most negatively impacted by the tariffs introduced during the second administration of President Trump.

​

Practical Applications

This kind of enterprise-strength data processing opens up possibilities that simply aren’t available when you’re limited to a few thousand documents:

1. Comprehensive Market Sentiment: Track sentiment across the entire Russell 1000 in near real-time.
2. Supply Chain Monitoring: Detect early warnings across global supply networks.
3. Competitive Intelligence: Monitor every competitor and adjacent industry simultaneously.
4. Regulatory Impact Assessment: Analyze how policy changes affect every sector at once.

​

Conclusion

In the pursuit of alpha---or the next business breakthrough---you need unmatched power and agility. This example illustrates Bigdata.com’s capability to search billions of news articles, corporate filings, and transcripts at an extraordinary scale.

The ability to process millions of searches and documents in minutes isn’t just a technical milestone---it’s a fundamental shift in how financial analysts, researchers, and decision-makers engage with market data.

Ready to process your first million documents? Visit Bigdata.com to get started with our API today.

Happy data processing! 🚀