# Analyst Estimates Source: https://docs.bigdata.com/api-reference/analyst-estimates/analyst-estimates /api-rest/openapi/openapi_structured_data.json post /v1/analyst-estimates/query Retrieves analyst financial estimates for a given symbol, including projected revenue, earnings per share (EPS), and other key metrics, as forecasted by industry analysts. Each request returns estimates for one company. The `results.values` array is ordered with the furthest-future forecasted period first, working backward to the oldest period returned. Forward-looking forecasts and past periods sit on the same timeline, so how much history and how far ahead analysts forecast varies by company. Coverage is densest for the next two to three years and thins out beyond that. Filtering by date range is not supported. Use `period` to choose annual or quarterly estimates and `limit` to control how many periods are returned. # Analyst Ratings Source: https://docs.bigdata.com/api-reference/analyst-estimates/analyst-ratings /api-rest/openapi/openapi_structured_data.json post /v1/analyst-ratings/query Retrieves analyst ratings and recommendations for a given symbol, including buy/hold/sell counts and the consensus recommendations. # Earnings Surprises Source: https://docs.bigdata.com/api-reference/analyst-estimates/earnings-surprises /api-rest/openapi/openapi_structured_data.json post /v1/latest-surprise/query The Earnings Surprises API allows users to retrieve the most recent quarterly earnings surprise data for a specific company. This endpoint provides actual versus estimated earnings per share (EPS) and revenue (sales), highlighting whether the company has beaten, missed, or met expectations. # Target Price Source: https://docs.bigdata.com/api-reference/analyst-estimates/target-price /api-rest/openapi/openapi_structured_data.json post /v1/price/target/query This endpoint retrieves target price for a company, including target high, target low, target consensus, and target median. # Create a Batch Job Source: https://docs.bigdata.com/api-reference/batch-search/create-a-batch-job /api-rest/openapi/openapi_search_service.json post /v1/search/batches Process large volumes of requests asynchronously. Batch Search provides a simple set of endpoints to initiate batch processing jobs, monitor progress as requests are executed, and collect results once processing completes. Each batch begins with a `.jsonl` file where every line represents an individual Search request. Once your input file is ready, call the following endpoint to obtain a `batch_id` and `presigned_url`. Copy the `presigned_url` from the response. You will need this URL to [upload your input file](/api-reference/batch-search/upload-your-input-file). For a step-by-step walkthrough, see the [Batch Search how-to guide](/how-to-guides/search/batch_search). # Get Batch Job Info Source: https://docs.bigdata.com/api-reference/batch-search/get-batch-job-info /api-rest/openapi/openapi_search_service.json get /v1/search/batches/{batch_id} Check the status of your batch job and retrieve results when processing is complete. - `batch_id`: Your job's unique identifier - `status`: Current processing state (pending, processing, completed, failed, or cancelled) - `estimated_completion_time`: A dynamic ISO 8601 timestamp indicating when the job is expected to finish. This adjusts in real-time based on system throughput and queue backlog. - `output_file_url`: Download link for your results (available when status is completed) Once your batch reaches `completed` status, use the `output_file_url` to download a `.jsonl` file containing all results via a simple GET request. Each line in your results file contains: | Field | Type | Description | |-------|------|-------------| | line_number | int | Matches the line number from your original input file | | status | string | Result status: success, error, timeout, or exception | | query | object | The original search query you submitted | | response | object | The Search response for this query | | code | int | HTTP status code | | error | string | Error description (only present if request failed) | For a step-by-step walkthrough, see the [Batch Search how-to guide](/how-to-guides/search/batch_search). # Upload Your Input File Source: https://docs.bigdata.com/api-reference/batch-search/upload-your-input-file /api-rest/openapi/openapi_search_service.json put /s3_upload_placeholder Now that you have created your batch job, use the `presigned_url` to upload your `.jsonl` file directly to the designated storage location using the following command: ```shell curl --request PUT \ --url '' \ --header 'Content-Type: application/jsonl' \ --data-binary '@your_input_file.jsonl' ``` Replace `` with the `presigned_url` and `@your_input_file.jsonl` with the path to your actual file. The `presigned_url` is a signed link on `batch-search.bigdata.com`. Use it exactly as returned, because the query parameters carry the signature. If your company uses a firewall, allow `batch-search.bigdata.com` on port 443 (see [Domains to whitelist](/api-rest/bigdata_domains)). For a step-by-step walkthrough, see the [Batch Search how-to guide](/how-to-guides/search/batch_search). # Connected Entities Source: https://docs.bigdata.com/api-reference/co-mentions/connected-entities /api-rest/openapi/openapi_search_service.json post /v1/search/co-mentions/entities Explore entities associated with your search filters to reveal deeper connections. This endpoint retrieves the top N entities ranked by chunk volume and headline volume, merges the two sets, and returns entities with combined or individual counts. Timestamp ranges are limited to one year; for longer periods, split into successive one-year requests. # Connected Topics Source: https://docs.bigdata.com/api-reference/co-mentions/connected-topics /api-rest/openapi/openapi_search_service.json post /v1/search/co-mentions/topics Explore topics associated with your search filters to reveal deeper connections. This endpoint retrieves the top N topics ranked by chunk volume and headline volume, merges the two sets, and returns topics with combined or individual counts. # Find by details Source: https://docs.bigdata.com/api-reference/companies/find-by-details /api-rest/openapi/openapi_knowledge_graph.json post /v1/knowledge-graph/companies Bigdata.com Knowledge Graph tracks over 7 million companies worldwide. Use this endpoint to search by name, website, ticker, or other market identifiers. Refine your results with filters such as company type, country, or sector. # Get by CUSIP Source: https://docs.bigdata.com/api-reference/companies/get-by-cusip /api-rest/openapi/openapi_knowledge_graph.json post /v1/knowledge-graph/companies/cusip Locate companies using Committee on Uniform Securities Identification Procedures (CUSIP) numbers, enabling coverage of both equities and fixed income instruments. # Get by ISIN Source: https://docs.bigdata.com/api-reference/companies/get-by-isin /api-rest/openapi/openapi_knowledge_graph.json post /v1/knowledge-graph/companies/isin Locate companies using International Securities Identification Numbers (ISIN), enabling coverage of both equities and fixed income instruments. # Get by Listing Source: https://docs.bigdata.com/api-reference/companies/get-by-listing /api-rest/openapi/openapi_knowledge_graph.json post /v1/knowledge-graph/companies/listing Locate companies using exchange listing identifiers. # Get by SEDOL Source: https://docs.bigdata.com/api-reference/companies/get-by-sedol /api-rest/openapi/openapi_knowledge_graph.json post /v1/knowledge-graph/companies/sedol Locate companies using Stock Exchange Daily Official List (SEDOL) numbers. # Get sectors Source: https://docs.bigdata.com/api-reference/companies/get-sectors /api-rest/openapi/openapi_knowledge_graph.json get /v1/knowledge-graph/companies/sectors Retrieve the list of available sectors to be used as filters when searching for companies in the Knowledge Graph. The same list is returned by `GET /v1/knowledge-graph/companies/values/sectors`, which lists the values of any category filter. # Get Subsidiaries Source: https://docs.bigdata.com/api-reference/companies/get-subsidiaries /api-rest/openapi/openapi_knowledge_graph.json post /v1/knowledge-graph/subsidiaries/query Returns the full subsidiary hierarchy for a given company. For each subsidiary includes: ultimate parent, immediate parent, company name, and hierarchy depth level. # Company Profile Source: https://docs.bigdata.com/api-reference/company-&-events/company-profile /api-rest/openapi/openapi_structured_data.json post /v1/company-profile/query This endpoint retrieves comprehensive company profile data including company name, sector, industry, CEO, market cap, and other key business information. # Economic Calendar Source: https://docs.bigdata.com/api-reference/company-&-events/economic-calendar /api-rest/openapi/openapi_structured_data.json post /v1/economic-calendar/query Returns the macroeconomic releases scheduled in a window: inflation, growth, employment, central bank decisions and the rest of the economic calendar, for the countries requested. Each release carries the figure published, the market consensus and the prior period, so a caller can see both what was expected and what landed. Figures are returned in full units. There is no pagination, and a response is capped at 1000 releases, so narrow the window or the filters to stay under it. # ESG Scores Source: https://docs.bigdata.com/api-reference/company-&-events/esg-scores /api-rest/openapi/openapi_structured_data.json post /v1/esg-performance-score/query The ESG Performance Score API provides a comprehensive evaluation of a company's environmental, social, and governance (ESG) practices. It aggregates various ESG metrics into a single performance score, allowing investors and stakeholders to assess the company's commitment to sustainable and responsible business operations. The ESG scores reflect companies' sustainability performance relative to financially material ESG issues. Scores are derived from standardized company-reported data and peer-based relative scoring techniques, resulting in a 0–100 scale that indicates a company's position versus its global peers, following a sector-specific, SASB (Sustainability Accounting Standards Board)-inspired materiality framework. | Performance | Score Range | |---|---| | Laggard | < 42 | | Below Average | 42 - 47 | | Average | 48 - 53 | | Above Average | 54 - 59 | | Leader | ≥ 60 | # Events Calendar Source: https://docs.bigdata.com/api-reference/company-&-events/events-calendar /api-rest/openapi/openapi_structured_data.json post /v1/events-calendar/query A financial data API that returns details of key corporate events, such as earnings calls, conference calls, US IPO listings, and US delistings. Query by a specific list of companies, by date range alone, or both, allowing investors and analysts to track upcoming and historical company announcements in real time. # Stocks & Funds Screener Source: https://docs.bigdata.com/api-reference/company-&-events/stocks-&-funds-screener /api-rest/openapi/openapi_structured_data.json post /v1/company-screener/query The Company Screener API enables advanced filtering and discovery of companies based on comprehensive financial and descriptive criteria. Filter companies by market capitalization, stock price, beta, trading volume, dividend yield, sector, industry, country, exchange, and trading status. Perfect for investment research, portfolio construction, and market analysis. # Find by details Source: https://docs.bigdata.com/api-reference/concepts/find-by-details /api-rest/openapi/openapi_knowledge_graph.json post /v1/knowledge-graph/concepts Find concepts such as economic indicators, financial instruments and business terms. Search by name or description and narrow the results by type. # Create connector Source: https://docs.bigdata.com/api-reference/connectors/create-connector /api-rest/openapi/openapi_content_data.json post /contents/v1/connectors We recommend using **[Bigdata Connectors Management](https://platform.bigdata.com/connectors)** on the Developer Platform. You can also call this endpoint to register connectors programmatically. Provide the connector **type**, **label**, and type-specific **config**. For **email** connectors, an inbox is created and its address is returned; forward emails from allowed senders to that address to ingest, process and index incoming content. For **investment_research** connectors, provide broker credentials and the platform will sync research documents from the broker feed. Optionally set **`config.date_limit`** as the lower bound for the initial sync and **`config.rolling_history_days`** to keep only a rolling window of ingested research. # Delete connector by ID Source: https://docs.bigdata.com/api-reference/connectors/delete-connector-by-id /api-rest/openapi/openapi_content_data.json delete /contents/v1/connectors/{connector_id} By default, soft-deletes a connector: the connector is marked as archived and will no longer ingest new content; existing documents that were ingested by this connector remain available. The operation is idempotent for an already-archived connector. To permanently remove the connector and all related data, add the query parameter `force_delete=true`. This triggers a hard delete flow: all files related to this connector are deleted asynchronously. Once all files are deleted, the connector is removed from the database (final step of the async process). The deletion process is asynchronous. # Get connector by ID Source: https://docs.bigdata.com/api-reference/connectors/get-connector-by-id /api-rest/openapi/openapi_content_data.json get /contents/v1/connectors/{connector_id} Returns full details for a single connector by its UUID. Use this to inspect connector metadata and configuration. Credentials and secrets are never included in the response. # List connectors Source: https://docs.bigdata.com/api-reference/connectors/list-connectors /api-rest/openapi/openapi_content_data.json get /contents/v1/connectors Returns all connectors that belong to the authenticated user. Use this to discover connector_id values for filtering documents, for update/delete operations, or to trigger a syncronization job on demand. Connector configuration is included. Sensitive fields are never returned. # Trigger connector sync Source: https://docs.bigdata.com/api-reference/connectors/trigger-connector-sync /api-rest/openapi/openapi_content_data.json post /contents/v1/connectors/{connector_id}/sync Queues a new synchronization run for an **investment_research** or **sharepoint** connector. Only content that is not already ingested is synced. **SharePoint.** Send an empty JSON body (`{}`). **Investment research.** Both body fields are optional. Send **`date_limit`** (`YYYY-MM-DD`) to resync broker research from that date onward and replace the stored date limit. Send **`rolling_history_days`** (30–3650) to set or change the rolling history window, or **`null`** to turn it off. Omit a field to leave it unchanged. When a rolling history window is set, the connector keeps only research published within that many days. A scheduled job deletes documents published on or before today (UTC) minus `rolling_history_days`, and later syncs do not ingest them again. **Deletion is not reversible.** Shortening the window deletes documents that now fall outside it. Lengthening the window, or turning it off, does not restore documents already removed. If this request sets `date_limit` or `rolling_history_days`, the date limit that applies — the value you send, or the stored one (default: three months back) — cannot be older than the start of the rolling window. Otherwise the API returns `400` with `ROLLING_HISTORY_CONFLICT` and changes nothing. To set a short window on a connector with an older stored date limit, send a more recent `date_limit` in the same request. Requests that set neither field skip this check. # Update connector by ID Source: https://docs.bigdata.com/api-reference/connectors/update-connector-by-id /api-rest/openapi/openapi_content_data.json put /contents/v1/connectors/{connector_id} Update an existing connector's **label**, **description**, and **share_with_org**. For **email** connectors, you may also update **config** (e.g. `allowed_emails`). For **investment_research** and **sharepoint** connectors, configuration cannot be updated here; to change the rolling history window of an **investment_research** connector, use [Trigger connector sync](/api-reference/connectors/trigger-connector-sync). As a PUT request, required fields follow the schema; leaving out a field can clear or default it. The connector **type** cannot be changed. # Delete document Source: https://docs.bigdata.com/api-reference/documents/delete-document /api-rest/openapi/openapi_content_data.json delete /contents/v1/documents/{content_id} Deletes a target document from the platform. It removes everything: original document, annotated version, and chunks from the vector database. # Enrich document Source: https://docs.bigdata.com/api-reference/documents/enrich-document /api-rest/openapi/openapi_content_data.json post /contents/v1/documents Request a pre-signed URL to upload a document directly to Bigdata.com. The response contains a single-use url and the document id. Send a **PUT request** to that URL with the document file as the body to complete the upload. Bigdata then **enriches** the document (extraction, structure and annotation of the content). By default it is also **indexed** and becomes available for the Search and Research Agent. Set **`no_index`** to skip indexing while keeping the document stored and retrievable, or **`no_store`** to skip indexing and retain the document for only 24 hours after enrichment completes. # Get annotated document Source: https://docs.bigdata.com/api-reference/documents/get-annotated-document /api-rest/openapi/openapi_content_data.json get /contents/v1/documents/{content_id}/annotated Returns a time-limited pre-signed URL to download **private** document content in annotated (structured) JSON format. Use this only for documents you ingested via the Content API (connectors or direct upload). When you GET the URL, the response contains document content, metadata and annotations. The URL expires after 24 hours; request a new one if needed. # Get document metadata Source: https://docs.bigdata.com/api-reference/documents/get-document-metadata /api-rest/openapi/openapi_content_data.json get /contents/v1/documents/{content_id} Returns metadata for a single document by its content ID (the document id returned by List documents). Use this to check status, file_name, connector_id, tags, enrichments_requested, enrichments_applied, usage, and timestamps. To retrieve the actual file content, use Get annotated document (enriched JSON) or Get original document (original file format). # Get original document Source: https://docs.bigdata.com/api-reference/documents/get-original-document /api-rest/openapi/openapi_content_data.json get /contents/v1/documents/{content_id}/original Returns a time-limited pre-signed URL to download the **private** document's original file as uploaded (e.g. .eml, .pdf). When you GET the URL, the response is the raw file with appropriate Content-Type and Content-Disposition. Use this when you need the native format rather than the annotated JSON. The URL expires after 24 hours; request a new one if needed. # List documents Source: https://docs.bigdata.com/api-reference/documents/list-documents /api-rest/openapi/openapi_content_data.json get /contents/v1/documents Returns a paginated list of documents that the user has uploaded or that have been shared with their organization. Use the query parameters documented below for filtering. Use the returned document id with Get document, Get annotated document, or Get original document to retrieve metadata or file content. # Update document metadata Source: https://docs.bigdata.com/api-reference/documents/update-document-metadata /api-rest/openapi/openapi_content_data.json patch /contents/v1/documents/{content_id} Partially update a document's metadata. Send only the fields you want to change. - **`tags`**: Array of **tag IDs** (UUIDs) to assign to the document. This replaces the document's current tag set, so you can add or remove tags by including the full list you want applied. Each ID must refer to an existing tag (use **POST** `/contents/v1/tags` to create one, or **GET** `/contents/v1/tags` to list IDs). If any ID is invalid, the API returns `400` with error code `INVALID_TAGS_ERROR`. - **`share_with_org`**: When `true`, the document is available to all members of your organization. When `false`, only you can access it. # Get Entities by ID Source: https://docs.bigdata.com/api-reference/entities/get-entities-by-id /api-rest/openapi/openapi_knowledge_graph.json post /v1/knowledge-graph/entities/id Retrieve entities by their RavenPack ID, a unique identifier used across the Bigdata.com ecosystem. Users can retrieve up to 100 entities per request. # Find by details Source: https://docs.bigdata.com/api-reference/etfs/find-by-details /api-rest/openapi/openapi_knowledge_graph.json post /v1/knowledge-graph/etfs Find exchange-traded funds through a comprehensive search. Search by name, ticker, or description and filter by country to discover relevant ETFs for your investment research. # Get filter values Source: https://docs.bigdata.com/api-reference/filter-values/get-filter-values /api-rest/openapi/openapi_knowledge_graph.json get /v1/knowledge-graph/{category}/values/{filter} Retrieve every value a category filter can take, so a filter can be built without guessing its values. Values are returned in the exact form the corresponding search filter accepts: countries as ISO 3166-1 alpha-2 codes, source ranks as `RANK_1`..`RANK_5`, and so on. The full sorted list is returned; there is no narrowing parameter, and lists are capped at 5000 values. Every filter of every category is available except three on `sources`, which are stored in a way that cannot be enumerated and return 404: `packages`, `categories` and `tiers`. # Balance Sheet Source: https://docs.bigdata.com/api-reference/financials-&-ratios/balance-sheet /api-rest/openapi/openapi_structured_data.json post /v1/balance-sheet/query This endpoint retrieves fundamental balance sheet measures for a company, including assets, liabilities, and equity across reporting periods. # Cash Flow Statement Source: https://docs.bigdata.com/api-reference/financials-&-ratios/cash-flow-statement /api-rest/openapi/openapi_structured_data.json post /v1/cash-flow-statement/query This endpoint retrieves fundamental cash flow statement measures for a company, including operating, investing, and financing cash flows across reporting periods. # Company Ratios TTM Source: https://docs.bigdata.com/api-reference/financials-&-ratios/company-ratios-ttm /api-rest/openapi/openapi_structured_data.json post /v1/company-ratios-ttm/query Retrieves fundamental company ratios for a company, including revenue, expenses, and profitability metrics across reporting periods. # Income Statement Source: https://docs.bigdata.com/api-reference/financials-&-ratios/income-statement /api-rest/openapi/openapi_structured_data.json post /v1/income-statement/query This endpoint retrieves fundamental income statement measures for a company, including revenue, expenses, and profitability metrics across reporting periods. # Key Metrics TTM Source: https://docs.bigdata.com/api-reference/financials-&-ratios/key-metrics-ttm /api-rest/openapi/openapi_structured_data.json post /v1/key-metrics-ttm/query The Key Metrics TTM API provides seamless access to a comprehensive set of trailing twelve-month (TTM) performance indicators. It delivers insights into a company's profitability, capital efficiency, and liquidity, enabling in-depth evaluation of financial health and operational performance over the past year. # Revenue Geographic Segments Source: https://docs.bigdata.com/api-reference/financials-&-ratios/revenue-geographic-segments /api-rest/openapi/openapi_structured_data.json post /v1/company-revenue-geographic-segments/query The Revenue Geographic Segments API allows users to retrieve revenue data segmented by geographical regions, helping investors and analysts understand the performance of a company in different markets. # Revenue Product Segments Source: https://docs.bigdata.com/api-reference/financials-&-ratios/revenue-product-segments /api-rest/openapi/openapi_structured_data.json post /v1/company-revenue-product-segments/query The Revenue Product Segments API provides access to detailed revenue data broken down by individual products. This enables investors and analysts to gain deeper insights into a company’s performance across its product portfolio. # Bond prices Source: https://docs.bigdata.com/api-reference/fixed-income/bond-prices /api-rest/openapi/openapi_structured_data.json post /v1/fixed-income/corporate/prices/query This endpoint is in **Beta**. Access must be requested before you can use it: contact [support@bigdata.com](mailto:support@bigdata.com) to join the Beta program. Returns the latest price and yield of bonds by the bonds' ISINs, one or up to 50 per request, with each bond's spread over the government bond set as its benchmark when it was issued. `results` is keyed by ISIN, one key per distinct ISIN sent in the order sent. The price fields are present only for a priced bond, so a bond with no price, known or not, is an empty object or names only its `benchmark`; the spread is present only when the bond and its benchmark were both priced on the same date. # Bonds by ISIN Source: https://docs.bigdata.com/api-reference/fixed-income/bonds-by-isin /api-rest/openapi/openapi_structured_data.json post /v1/fixed-income/corporate/isin/query This endpoint is in **Beta**. Access must be requested before you can use it: contact [support@bigdata.com](mailto:support@bigdata.com) to join the Beta program. Returns bonds and their terms by the bonds' ISINs, one or up to 50 per request: description, issuer, issue and maturity dates, coupon and payment frequency, currency, issued and outstanding amounts, seniority, status and structural features. `results` is keyed by ISIN, one key per distinct ISIN sent in the order sent, and an ISIN with no bond available is keyed to `null`. The issuer's RavenPack entity identifier is absent when the issuer has none. # Bonds by Issuer Source: https://docs.bigdata.com/api-reference/fixed-income/bonds-by-issuer /api-rest/openapi/openapi_structured_data.json post /v1/fixed-income/corporate/issuer/query This endpoint is in **Beta**. Access must be requested before you can use it: contact [support@bigdata.com](mailto:support@bigdata.com) to join the Beta program. Returns the bonds a company has issued, with their terms: ISIN, description, issue and maturity dates, coupon and payment frequency, currency, issued and outstanding amounts, seniority (rank), status and structural features. Only active bonds are returned unless `status` asks for inactive or defaulted ones, or is null for every status. Results are paged: send back `pagination.cursor` to get the next page. A page can hold fewer bonds than the next one, or none, while `has_cursor` is still true. # Fund Holdings Source: https://docs.bigdata.com/api-reference/fund-holdings/fund-holdings /api-rest/openapi/openapi_structured_data.json post /v1/holdings/funds/stocks/ Retrieve portfolio holdings disclosed by institutional funds in their SEC 13F filings. This dataset provides an overview of a fund’s positions, including the total number of holdings, overall portfolio size, and high-level summaries such as top holdings and allocation trends. It also allows retrieval of the top buys and sells for a fund or fund manager, either for the latest portfolio or for historical quarters. Sector composition of the fund’s portfolio can also be accessed. # Fund Position Signals Source: https://docs.bigdata.com/api-reference/fund-holdings/fund-position-signals /api-rest/openapi/openapi_structured_data.json post /v1/holdings/funds/trends/ Retrieve a strategic overview of top stocks based on aggregated position changes by hedge funds, utilizing data from the most recent SEC 13F filings. This endpoint provides a detailed overview of fund activities, capturing changes across calls, puts, new positions, closures, and net adjustments. It highlights where hedge funds are intensifying their focus, reducing exposure, or completely shifting their investment strategies, helping users identify trends and strategic moves in the market. # Stock Fund Holdings Source: https://docs.bigdata.com/api-reference/fund-holdings/stock-fund-holdings /api-rest/openapi/openapi_structured_data.json post /v1/holdings/stocks/funds Retrieve institutional fund holdings for a specified stock, based on the latest SEC 13F filings. The endpoint returns top holders with their current market value, share count, quarterly change, and estimated capital flow. This allows users to identify leading institutional investors, track changes in ownership, and analyze fund activity trends such as increases or reductions in positions. # Find by details Source: https://docs.bigdata.com/api-reference/funds/find-by-details /api-rest/openapi/openapi_knowledge_graph.json post /v1/knowledge-graph/funds Find funds and mutual funds. Search by name or description and narrow the results by type, country and sector. # Aftermarket Prices Source: https://docs.bigdata.com/api-reference/market-data/aftermarket-prices /api-rest/openapi/openapi_structured_data.json post /v1/price/aftermarket/query Returns the latest extended-hours (aftermarket) trade and quote for one or more companies. Each record includes the last aftermarket trade (price, size, timestamp) and quote (best bid/ask price and size, cumulative volume, timestamp). # Daily Prices Source: https://docs.bigdata.com/api-reference/market-data/daily-prices /api-rest/openapi/openapi_structured_data.json post /v1/price/daily/query Returns daily historical price and volume data for a given symbol. Supports stocks, ETFs, indexes, cryptocurrencies, commodities (continuous futures) and forex pairs. Each data point includes date, open, high, low, close, volume, change, change percentage, and volume-weighted average price (VWAP). Ideal for tracking daily market performance and conducting technical analysis over extended periods. # Dividends Source: https://docs.bigdata.com/api-reference/market-data/dividends /api-rest/openapi/openapi_structured_data.json post /v1/dividends/query Returns historical and upcoming dividend data for companies. Supports stocks and ETFs. Each dividend record includes ex-dividend date, record date, payment date, declaration date, dividend amount, adjusted dividend, yield, and frequency. # Entity Sentiment Source: https://docs.bigdata.com/api-reference/market-data/entity-sentiment /api-rest/openapi/openapi_structured_data.json post /v1/entity-sentiment/ Returns a time series of daily sentiment metrics for a single entity over a specified date range. Sentiment is a quantified signal derived from unstructured text using rule-based and machine-learning methods, indicating likely positive or negative market impact. The response includes daily_sentiment (mean sentiment), sentiment_pressure (abnormal sentiment intensity), and abnormal_media_attention (abnormal media volume). # Intraday Prices Source: https://docs.bigdata.com/api-reference/market-data/intraday-prices /api-rest/openapi/openapi_structured_data.json post /v1/price/intraday/query Returns intraday historical price and volume data for a given symbol at specified intervals (e.g. 1min, 5min, 15min, 30min, 1hour). Supports stocks, ETFs, commodities (continuous futures), indexes, cryptocurrencies, and forex pairs. Each data point includes timestamp, open, high, low, close, and volume. # Price Changes Source: https://docs.bigdata.com/api-reference/market-data/price-changes /api-rest/openapi/openapi_structured_data.json post /v1/price/changes/query This endpoint track price fluctuations in real-time with the Price Changes API. Monitor percentage changes over various time periods, including daily (1D), monthly (1M), yearly (1Y) and long-term (max). # Quote Source: https://docs.bigdata.com/api-reference/market-data/quote /api-rest/openapi/openapi_structured_data.json post /v1/quote/query This endpoint provides real-time asset quotes with the Quote API. Access up-to-the-minute prices, price changes, and trading volume for individual assets, ensuring you stay on top of market movements. Assets covered include stocks, cryptocurrencies, commodities, and ETFs. # Create a monitor Source: https://docs.bigdata.com/api-reference/monitors/create-a-monitor /api-rest/openapi/openapi_search_monitoring.json post /v1/search-monitoring/monitors Create a monitor from what you want to track and how often to check. Only `intent` and `schedule` are required. Anything you leave out is filled in from the intent. `status` defaults to `inactive`, so you can test before it runs on a schedule. A new monitor does not send email. Add email later with an update. If the intent cannot be monitored, the request is rejected. # Delete a monitor Source: https://docs.bigdata.com/api-reference/monitors/delete-a-monitor /api-rest/openapi/openapi_search_monitoring.json delete /v1/search-monitoring/monitors/{monitor_id} Permanently delete the monitor and its schedule. You cannot delete a monitor while a run is still in progress. Wait for that run to finish. To pause scheduled runs without deleting the monitor, set `status` to `inactive`. # Get monitor details Source: https://docs.bigdata.com/api-reference/monitors/get-monitor-details /api-rest/openapi/openapi_search_monitoring.json get /v1/search-monitoring/monitors/{monitor_id} One monitor, including its search, schedule, output fields, extraction instructions, watchlists, and email delivery. # Get run details Source: https://docs.bigdata.com/api-reference/monitors/get-run-details /api-rest/openapi/openapi_search_monitoring.json get /v1/search-monitoring/monitors/{monitor_id}/runs/{run_id} One run, including its events. A live run includes only new events. A test also includes events that were not new, so you can see everything the monitor extracted. # List monitor runs Source: https://docs.bigdata.com/api-reference/monitors/list-monitor-runs /api-rest/openapi/openapi_search_monitoring.json get /v1/search-monitoring/monitors/{monitor_id}/runs Runs for this monitor, newest first. Use `type=production` for live runs. Pass `simulation_id` to read one test. That also adds a `simulation` object so you can see whether the test is still running. Set `include_events` to `true` to include the events. Otherwise `events` is empty. Live runs include only new events. A test also includes events that were not new, which helps you tune the monitor. # List monitors Source: https://docs.bigdata.com/api-reference/monitors/list-monitors /api-rest/openapi/openapi_search_monitoring.json get /v1/search-monitoring/monitors Your monitors, newest first. You can filter by status, name, or intent. There is no next page. Results stop at `limit`. Each item includes the id, name, intent, status, and when it was created. # Simulate a monitor Source: https://docs.bigdata.com/api-reference/monitors/simulate-a-monitor /api-rest/openapi/openapi_search_monitoring.json post /v1/search-monitoring/monitors/{monitor_id}/simulate Run the monitor over past content without turning it on. The test continues in the background. Save `simulation_id` from the response, then call List monitor runs with that id to read the results. # Update a monitor Source: https://docs.bigdata.com/api-reference/monitors/update-a-monitor /api-rest/openapi/openapi_search_monitoring.json patch /v1/search-monitoring/monitors/{monitor_id} Change the fields you send. Leave a field out to keep its current value. `intent` cannot be changed. To track something else, create a new monitor. Changing the search, output fields, watchlists, or extraction instructions clears this monitor's history. Later runs treat matching events as new. Changing the name, status, schedule, or email delivery keeps the history. Sending back the same search you received from Get monitor details does not count as a change. # Find by details Source: https://docs.bigdata.com/api-reference/organizations/find-by-details /api-rest/openapi/openapi_knowledge_graph.json post /v1/knowledge-graph/organizations Find organizations such as central banks, regulators, government bodies and NGOs. Search by name or description and narrow the results by type, country, ultimate parent or owner. # Find by details Source: https://docs.bigdata.com/api-reference/people/find-by-details /api-rest/openapi/openapi_knowledge_graph.json post /v1/knowledge-graph/people Find people such as executives, politicians and public figures. Search by name or description and narrow the results by position, employer, nationality or gender. # Find by details Source: https://docs.bigdata.com/api-reference/places/find-by-details /api-rest/openapi/openapi_knowledge_graph.json post /v1/knowledge-graph/places Find places such as countries, regions, cities and facilities. Search by name or description and narrow the results by category, type, continent, country or region. # Find by details Source: https://docs.bigdata.com/api-reference/positions/find-by-details /api-rest/openapi/openapi_knowledge_graph.json post /v1/knowledge-graph/positions Find job positions, roles and titles, for example "Chief Executive Officer" or "Data Scientist". Search by name or description. # Private Company Funding Rounds Source: https://docs.bigdata.com/api-reference/private-markets/private-company-funding-rounds /api-rest/openapi/openapi_structured_data.json post /v1/funding-rounds/query Returns the funding rounds one private company has raised: when each was announced and closed, what kind of round it was, how much it raised, what the company was valued at before and after, and who invested. One company per request, keyed by RavenPack entity ID. Results are paged: send back `pagination.cursor` to get the next page, and keep going until `pagination.has_cursor` is false. A company the data provider has no rounds for comes back with an empty `results`. # Private Company Overview Source: https://docs.bigdata.com/api-reference/private-markets/private-company-overview /api-rest/openapi/openapi_structured_data.json post /v1/private-companies/query Returns an overview of a private company: what it does, where it is headquartered, how much it has raised and what it was last valued at, along with how active it is. Request up to 50 companies at a time by RavenPack entity ID. `results` holds one row per requested ID, in request order, each carrying its `rp_entity_id`. A company with no overview to serve comes back as a row holding only its `rp_entity_id`. A company that is now listed also gets an overview, so check `ipo_status` before reading the funding fields. For a listed company, `valuation` can be years old and is not a market capitalisation. The latest funding round can be one raised after the listing, such as a `post_ipo_debt` or `post_ipo_equity` round. # Private Company People Source: https://docs.bigdata.com/api-reference/private-markets/private-company-people /api-rest/openapi/openapi_structured_data.json post /v1/private-company-people/query Returns the people who hold or held a position at one private company: who they are, their title, what kind of position it is, whether they still hold it, and when it started and ended. One company per request, keyed by RavenPack entity ID. Results are paged: send back `pagination.cursor` to get the next page, and keep going until `pagination.has_cursor` is false. A company the data provider has no positions for comes back with an empty `results`. # Find by details Source: https://docs.bigdata.com/api-reference/products/find-by-details /api-rest/openapi/openapi_knowledge_graph.json post /v1/knowledge-graph/products Find products and brands. Search by name or description and narrow the results by product type or owner. # Get Subscription Quotas Source: https://docs.bigdata.com/api-reference/quotas/get-subscription-quotas /api-rest/openapi/openapi_subscription.json get /v1/subscription/quotas Returns the current subscription quotas and usage for the organization. For examples, see the [Monitor usage](/how-to-guides/monitor_usage) how-to guide. # Research Agent Source: https://docs.bigdata.com/api-reference/research-agent/research-agent /api-rest/openapi/openapi_research_agent.json post /v1/research-agent Execute a research query with AI-powered analysis and real-time data. Returns a streaming response via Server-Sent Events. # Fetch document Source: https://docs.bigdata.com/api-reference/search/fetch-document /api-rest/openapi/openapi_search_service.json get /v1/documents/{document_id} Returns a time-limited pre-signed URL for downloading the document in annotated (structured) JSON format. Issuing a GET request to that URL returns the document's content, metadata, and annotations. The URL expires after 24 hours; request a new one when it does. This endpoint works for any document in Bigdata, whether it originates from the RavenPack catalog or from your own processed files. When `web_content` is `true`, the returned document also includes a direct URL to the publisher's original article on the web. # Search documents Source: https://docs.bigdata.com/api-reference/search/search-documents /api-rest/openapi/openapi_search_service.json post /v1/search Easily find the most relevant information from trusted sources and your own data. Use it to power agents that give accurate, real-time answers. # Search volume Source: https://docs.bigdata.com/api-reference/search/search-volume /api-rest/openapi/openapi_search_service.json post /v1/search/volume Get document and chunk volume statistics over time for a search query, aggregated by date with sentiment analysis. # Search watchlist Source: https://docs.bigdata.com/api-reference/search/search-watchlist /api-rest/openapi/openapi_search_service.json post /v1/search/watchlist Search an entire watchlist and get results grouped by entity. Unlike `/v1/search`, where a few entities can dominate the results, watchlist search distributes results more evenly across your watchlist. You can choose how chunks are allocated: - `max_chunks.per_entity: false` – split a total chunk budget across all entities. - `max_chunks.per_entity: true` – return up to a fixed number of chunks for each entity. The response contains one group for every requested entity, in the same order as the request. # Find by details Source: https://docs.bigdata.com/api-reference/sources/find-by-details /api-rest/openapi/openapi_knowledge_graph.json post /v1/knowledge-graph/sources Locate and explore data sources and information providers within the knowledge graph. Filter sources by country, rank, and category to find the most relevant information providers. # Create tag Source: https://docs.bigdata.com/api-reference/tags/create-tag /api-rest/openapi/openapi_content_data.json post /contents/v1/tags Creates a custom tag in your organization. Use the returned `id` when assigning tags to a document via **PATCH** `/contents/v1/documents/{content_id}` (pass tag IDs in the `tags` array, not names). Tag names supplied at upload time via **POST** `/contents/v1/documents` are resolved by name; for post-upload changes, use tag IDs from this endpoint or from List tags. If you include spaces in `tag_name`, they are replaced with underscores in the stored tag name. For example, `testing tag 2` is stored as `testing_tag_2`. # Delete tag Source: https://docs.bigdata.com/api-reference/tags/delete-tag /api-rest/openapi/openapi_content_data.json delete /contents/v1/tags/{tag_id} Deletes a custom tag by ID. Any documents linked with this tag will have the association removed. # List tags Source: https://docs.bigdata.com/api-reference/tags/list-tags /api-rest/openapi/openapi_content_data.json get /contents/v1/tags Returns all tags visible to the user's organization, with the number of documents associated with each tag. Use the optional `prefix` query parameter to return only tags whose names start with that string (for example `broker:`, `label:`,`from:`, or `to:`). Use the optional `contains` query parameter to return only tags whose names contain that string (for example `My connector`, `My tag` or `my@email.com`). Tags are created automatically for connectors: `label:Connector Name` for any connector type, `from:user@email.com` / `to:user@email.com` for email and `broker:Broker Name` for investment research connectors. # Find by details Source: https://docs.bigdata.com/api-reference/topics/find-by-details /api-rest/openapi/openapi_knowledge_graph.json post /v1/knowledge-graph/topics Find topics from the Bigdata.com event taxonomy. Search by name or description and narrow the results by topic, group, type or subtype. Use the returned `id` in a Topic query filter. # Delete Execution Source: https://docs.bigdata.com/api-reference/workflow-executions/delete-execution /api-rest/openapi/openapi_workflows.json delete /v1/workflow/executions/{execution_id} Permanently delete a stored workflow execution and its result. Use this to manage how long your runs are kept. Deleting a run that is still in progress stops it first. There is no undo. # Get Execution Source: https://docs.bigdata.com/api-reference/workflow-executions/get-execution /api-rest/openapi/openapi_workflows.json get /v1/workflow/executions/{execution_id} Retrieve a stored workflow execution by ID, including its status, resource consumption, and the full result stream replayed in order. This is how you read a run's result after the streaming connection has closed. # List Executions Source: https://docs.bigdata.com/api-reference/workflow-executions/list-executions /api-rest/openapi/openapi_workflows.json get /v1/workflow/executions List stored workflow executions owned by the authenticated user, newest first. Every run submitted to the asynchronous endpoint is stored; a run streamed from `POST /v1/workflow/execute` is stored only if it was executed with `persistence_mode=enabled`. # Clone Template Source: https://docs.bigdata.com/api-reference/workflow-templates/clone-template /api-rest/openapi/openapi_workflows.json post /v1/workflow/templates/{template_id}/clone Create a copy of an existing template with optional field overrides. # Create Template Source: https://docs.bigdata.com/api-reference/workflow-templates/create-template /api-rest/openapi/openapi_workflows.json post /v1/workflow/templates Create a new workflow template. # Delete Template Source: https://docs.bigdata.com/api-reference/workflow-templates/delete-template /api-rest/openapi/openapi_workflows.json delete /v1/workflow/templates/{template_id} Permanently delete a workflow template. # Get Template Source: https://docs.bigdata.com/api-reference/workflow-templates/get-template /api-rest/openapi/openapi_workflows.json get /v1/workflow/templates/{template_id} Retrieve a specific workflow template by ID. # List Community Templates Source: https://docs.bigdata.com/api-reference/workflow-templates/list-community-templates /api-rest/openapi/openapi_workflows.json get /v1/workflow/templates/community List publicly available community workflow templates. # List Templates Source: https://docs.bigdata.com/api-reference/workflow-templates/list-templates /api-rest/openapi/openapi_workflows.json get /v1/workflow/templates List all workflow templates owned by the authenticated user. # Update Template Source: https://docs.bigdata.com/api-reference/workflow-templates/update-template /api-rest/openapi/openapi_workflows.json put /v1/workflow/templates/{template_id} Update an existing workflow template. Only provided fields are modified. # Cancel Workflow Source: https://docs.bigdata.com/api-reference/workflows/cancel-workflow /api-rest/openapi/openapi_workflows.json post /v1/workflow/execute/async/{execution_id}/cancel Stop a running workflow. The response reports the run's status once the cancellation was applied, so no follow-up poll is needed. Cancelling a run twice is harmless. A cancelled run keeps whatever it produced and can be continued later by submitting its `execution_id`. # Execute Workflow Source: https://docs.bigdata.com/api-reference/workflows/execute-workflow /api-rest/openapi/openapi_workflows.json post /v1/workflow/execute Execute a workflow with an inline template definition or a stored template ID. Returns a streaming response via Server-Sent Events. # Stream Workflow Events Source: https://docs.bigdata.com/api-reference/workflows/stream-workflow-events /api-rest/openapi/openapi_workflows.json get /v1/workflow/execute/async/{execution_id}/stream Attach to a submitted run and receive its events live over Server-Sent Events. Optional: the run proceeds whether or not anything is attached, and disconnecting never stops it. Every event carries an `id`. If the connection drops, reconnect to the same URL sending the last id you received in the `Last-Event-ID` header and delivery continues from there. A standard `EventSource` client does this automatically. # Submit Workflow Source: https://docs.bigdata.com/api-reference/workflows/submit-workflow /api-rest/openapi/openapi_workflows.json post /v1/workflow/execute/async Submit a workflow that runs independently of your connection, and return immediately with its `execution_id`. Use this endpoint for production integrations: the run is unaffected if your client disconnects, and its result is always stored. Poll `GET /v1/workflow/executions/{execution_id}` for status and the result, or attach to the event stream for live progress. To continue a run that stopped before it finished, pass its `execution_id`. # Authentication Source: https://docs.bigdata.com/api-rest/authentication To authenticate with the Bigdata.com API, you need to use an API key. You can create and manage your API Keys in the [Developer Platform > API Keys](https://platform.bigdata.com/api-keys) ![Developer Platform > API Keys](https://mintcdn.com/ravenpackinternational/f62VnqoeoLD7OgQl/images/authentication/api_key_settings.png?fit=max\&auto=format\&n=f62VnqoeoLD7OgQl\&q=85\&s=cf16d5701fa628a8ec548137caa2fa18) The API key must be included in all API requests to the server in a `x-api-key` header. Here is an example of how you can make an authenticated request using `curl`: ```bash theme={null} curl --request POST \ --url https://api.bigdata.com/v1/search \ --header 'Content-Type: application/json' \ --header 'X-API-KEY: ' \ --data '{ "query": { "text": "Microsoft earnings call highlights" } }' ``` > [!TIP] > > We recommend using environment variables to specify your API Key. # Company news coverage Source: https://docs.bigdata.com/api-rest/best-practices/company_news_coverage If you want to analyse upfront how much news coverage your universe of companies receives under your current data packages, reviewing the volume of documents available over the last **month**, **6 months**, and **12 months** helps you decide whether those packages are the right fit for your portfolio, and informs how you design queries in the Search Service to extract the most relevant insights. You can review your current data packages, and explore others that you can unlock, in the [Bigdata Store](https://platform.bigdata.com/store). The script below leverages the [Volume endpoint](/api-reference/search/search-volume), which returns the unique number of documents and unique number of chunks matching a query over a given period. It sets an Entity filter and issues one call per company for the defined window. ## Coverage analysis **1. Resolve companies** The script takes as input the file of RavenPack IDs produced by the [Resolve companies](/api-rest/best-practices/resolve_companies) script, so make sure you have resolved your universe before running it. **2. Run the script** Clone the public repository and follow the instructions in its `README.md` to run the script. > **[How-to scripts: company news coverage](https://github.com/Bigdata-com/bigdata-docs-resources/tree/main/how_to_guides/api_company_volume_coverage)** > > Ready-to-run scripts that measure news coverage across your universe of companies over the last month, 6 months, and 12 months. **3. Import as Excel** The output file is in CSV format, and you can import it into an Excel by opening a new document, clicking on Data, and selecting "From Text/CSV". ![How to import CSV in Excel](https://mintcdn.com/ravenpackinternational/9t4nxMLz4HFDcoQF/images/api-rest/best-practices/how-to-import-csv-in-excel.png?fit=max\&auto=format\&n=9t4nxMLz4HFDcoQF\&q=85\&s=822bb31a42552a95cadd9e01e432802f) This is an example that only shows the columns of distinct documents in the last month, 6 months and 12 months. ![Company news coverage example](https://mintcdn.com/ravenpackinternational/9t4nxMLz4HFDcoQF/images/api-rest/best-practices/company_news_coverage.png?fit=max\&auto=format\&n=9t4nxMLz4HFDcoQF\&q=85\&s=ddfaced937af61d4803f7908d8fa6599) # Resolve companies Source: https://docs.bigdata.com/api-rest/best-practices/resolve_companies Financial analysts need to surface the news and documents that carry real insight about a universe of securities, without the noise of unrelated mentions. To make that possible, RavenPack has built and maintains a Knowledge Graph of more than 7 million companies worldwide, and identifies these entities in every piece of structured data it processes so you can pinpoint relevant content with precision. The first step in any financial workflow built on Bigdata is therefore to **resolve your universe of securities into RavenPack IDs**. Once you have those IDs, you can use them as filters across any of the Bigdata services to unlock the following value: ✅ **Continuity**: Retrieve all the data linked to a company even after it changes its name or ticker, so your history stays complete. \ ✅ **Precision**: Get only the content where the company was actually detected, avoiding the noise of similar names that refer to something unrelated. \ ## Resolving your universe The recommended endpoint depends on the type of entity you want to resolve. Use the tabs below to find the right approach for each one. **Public companies or ETFs** The most reliable way to resolve public companies is through a market identifier: an **ISIN**, **CUSIP**, **SEDOL**, or listing (in `MIC:Ticker` form). Bigdata exposes a dedicated `get` endpoint for each of these, and every endpoint accepts up to **500 identifiers in a single request**, so even large universes can be resolved with very few API calls. | Identifier | Endpoint | | :- | :- | | ISIN | [`/v1/knowledge-graph/companies/isin`](/api-reference/companies/get-by-isin) | | CUSIP | [`/v1/knowledge-graph/companies/cusip`](/api-reference/companies/get-by-cusip) | | SEDOL | [`/v1/knowledge-graph/companies/sedol`](/api-reference/companies/get-by-sedol) | | Listing (`MIC:Ticker`) | [`/v1/knowledge-graph/companies/listing`](/api-reference/companies/get-by-listing) | **Private companies** Private companies don't have market identifiers, so the most reliable way to resolve them is by their **official website**, or by a combination of **name and country**. The [`/v1/knowledge-graph/companies`](/api-reference/companies/find-by-details) endpoint accepts either signal and returns the **top 20 candidate matches** ranked by relevance. In most cases the first result is the company you were looking for. This endpoint resolves a single company per request, so to process a universe of private companies you need to send multiple queries in parallel. The reference scripts below show how to do this with multi-threading while staying within Bigdata rate limits. > [!TIP] > > RavenPack IDs are stable and do not change over time. We recommend persisting them in a database within your platform so you only need to resolve your universe once. ## Try it yourself > **[How-to scripts: resolve company IDs](https://github.com/Bigdata-com/bigdata-docs-resources/tree/main/how_to_guides/api_get_company_ids)** > > Ready-to-run scripts that resolve both public and private companies optimally, using multi-threading while respecting Bigdata rate limits. # Domains to whitelist Source: https://docs.bigdata.com/api-rest/bigdata_domains If your company uses a firewall, you must whitelist the following domains to ensure full access to both Bigdata App and API Services: The table below summarizes the protocol, domain, port, and a brief description for each: | Protocol | Domain | Port | Description | | - | - | - | - | | HTTPS | bigdata.com | 443 | Product website | | HTTPS | [www.bigdata.com](http://www.bigdata.com) | 443 | Product website alias | | HTTPS | app.bigdata.com | 443 | Bigdata Web application | | HTTPS | api.bigdata.com | 443 | API Services | | HTTPS | clerk.bigdata.com | 443 | Authentication and user management | | HTTPS | [https://s3.amazonaws.com](https://s3.amazonaws.com) | 443 | File storage (Amazon S3) | | HTTPS | batch-search.bigdata.com | 443 | Batch Search input uploads and result downloads | | HTTPS | content.bigdata.com | 443 | Retrieve your annotated or original documents | | HTTPS | documents.bigdata.com | 443 | Retrieve RavenPack annotated documents | | HTTPS | files.prod.nvirginia.common.ravenpack.com | 443 | Retrieve RavenPack annotated documents | | HTTPS | platform.bigdata.com | 443 | Developer Platform | | HTTPS | agents.bigdata.com | 443 | Research Agent Service | | HTTPS | mcp.bigdata.com | 443 | Bigdata Remote MCP tools | Whitelisting these domains allows uninterrupted connectivity for authentication, data access, file uploads, and other essential features of the Bigdata platform. # Introduction Source: https://docs.bigdata.com/api-rest/content_introduction ## Overview The **Bigdata Content API** lets you manage and query private content that you or your organization uploads to Bigdata. After enrichment, documents are indexed by default so they are available for the Search and Research Agent. You can skip indexing or long-term storage on direct upload with `no_index` and `no_store`. You can onboard content in two ways: * **Connectors API**: For asynchronous, unsupervised, or automated workflows where content is not directly accessible by the user (e.g. email inbox, broker investment research feeds, SharePoint). You configure a connector and the service ingests the content from that source. * **Direct upload (POST /documents)**: For clients that manage their own corpus and want to build the ingestion workflow manually. You request a pre-signed URL and document id; you upload each file via PUT to that URL, then use the id to poll for enrichment status. The workflow uploads your document and enriches it (extraction, structure and annotation of the content). By default it also indexes the document for Search and Research Agent. Optional **`no_index`** and **`no_store`** flags skip indexing or retain the file for only 24 hours. See [Processing options](#processing-options). Both paths produce **documents** that you can list, tag, filter, and use with the Search and Research Agent Services. You can create and manage **Connectors** in the [Developer Platform > Connectors](https://platform.bigdata.com/connectors) (**Bigdata Connectors Management**). Use that guided flow to set up connectors in **four simple steps**. ![Developer Platform > Connectors](https://mintcdn.com/ravenpackinternational/d14G-myw-Ler7I7m/images/contents/connectors_sample.png?fit=max\&auto=format\&n=d14G-myw-Ler7I7m\&q=85\&s=f091a0c530dbecacf36ca60dcf03fde2) *** ## Connectors **Connectors** are for ingestion that is asynchronous, unsupervised, or part of a workflow where content is not directly in the user’s hands, for example, an email inbox, a broker **investment research** feed, or a SharePoint library. * You **create a connector** with a type, label, and type-specific configuration. Supported types include **Email Inbox**, **Investment research**, and **Microsoft SharePoint**. * For **Email**, the API returns an inbox address; you forward messages from allowed senders to that address. Set up automatic forwarding rules in your email client to start building your corpus, Bigdata will ingest and process content automatically. * For **Investment research**, provide broker credentials when you [create the connector](/api-reference/connectors/create-connector). Bigdata then **syncs research documents** from the broker for you. You can [retrigger a sync](/api-reference/connectors/trigger-connector-sync) with a `date_limit` when you need to refresh from a given date. Optionally set **`rolling_history_days`** (30–3650) to keep only recent research: documents published on or before today (UTC) minus that many days are deleted and are not ingested again. Ingested documents are [tagged](/api-reference/tags/list-tags) with the broker name and **classified using Bigdata’s document taxonomy**, so you can filter and search them consistently alongside the rest of Bigdata corpus. * For **Microsoft SharePoint**, use the same create endpoint with `"type": "sharepoint"` and **`config`** containing only **`session_id`**: a token for the session authenticated via OAuth. For how to obtain **`session_id`**, contact [support@bigdata.com](mailto:support@bigdata.com). As an alternative, follow the steps in [Bigdata Connectors Management](https://platform.bigdata.com/connectors). * Content arriving through the connector is turned into **documents** you can manage and use via Search and Research Agent Services. * Optionally set **`enrichments`** when you [create the connector](/api-reference/connectors/create-connector) to configure the processing pipeline to apply extra enrichment steps to every document ingested through it. See [Enrichments](#enrichments). Typical operations: * **[Create connector](/api-reference/connectors/create-connector)**: Register a new ingestion source. We recommend using [Bigdata Connectors Management](https://platform.bigdata.com/connectors) for a simple guided setup. * **[List connectors](/api-reference/connectors/list-connectors)**: List all connectors you can access; use connector IDs to filter documents or to update, delete, or call sync. * **[Get connector](/api-reference/connectors/get-connector-by-id)**: Fetch full details for one connector (config, share settings, timestamps). * **[Update connector](/api-reference/connectors/update-connector-by-id)**: Change label, description, sharing, or **config** when supported. * **[Trigger connector sync](/api-reference/connectors/trigger-connector-sync)**: Queue another synchronization for an **investment\_research** connector (POST with `date_limit` `YYYY-MM-DD` to resync from that date onward, and/or `rolling_history_days` to set or clear the rolling history window) or a **sharepoint** connector (POST with an empty body `{}`). * **[Delete connector](/api-reference/connectors/delete-connector-by-id)**: Soft-delete (archive) a connector by default—it stops ingesting new content but existing documents remain. Use `?force_delete=true` to trigger a hard delete: all files related to the connector are deleted asynchronously, then the connector is removed from the database. Once a connector is set up, documents are ingested according to its configuration. *** ## Documents **Documents** are the items Bigdata has uploaded and enriched, whether they came from a connector (e.g. email, SharePoint) or from **direct upload**. By default they are also indexed for search and analysis. **Direct upload** is intended for clients that manage their corpus and want to build the ingestion workflow from scratch: you choose what to send, when, and with what metadata (file name, tags, sharing, enrichments, processing options). The main focus is **enrichment**: once you upload the file (PUT to the pre-signed URL), Bigdata enriches it (extraction, structure and annotation of the content). By default it then indexes the document for Search and Research Agent. Optionally set **`enrichments`** when you [are uploading the file](/api-reference/documents/enrich-document) to configure extra pipeline steps. See [Enrichments](#enrichments). Optionally set **`no_index`** or **`no_store`** to skip indexing or long-term retention. See [Processing options](#processing-options). Once you get the pre-signed URL and document id; PUT the file to that URL, then use the id with [Get document metadata](/api-reference/documents/get-document-metadata) to poll for status until processing is complete. The following operations apply to all documents, whether they were ingested via a connector or direct upload: * **[List documents](/api-reference/documents/list-documents)**: Paginated list of documents you can access. Supports filtering and sorting (including `origin` values `email`, `investment_research`, `sharepoint`, and `file_upload`); each item includes a content ID for use with the other document endpoints. * **[Get document metadata](/api-reference/documents/get-document-metadata)**: Return metadata for one document by content ID (status, file name, connector, tags, enrichments, usage, timestamps). Use this to check processing status before downloading content. * **[Update document metadata](/api-reference/documents/update-document-metadata)**: Change which tags are assigned to a document and whether the document is shared with your organization. * **[Get annotated document](/api-reference/documents/get-annotated-document)**: Return a time-limited pre-signed URL. GET that URL to download **private** document content as structured JSON (metadata, title, body blocks, entities, sentences with sentiment). Use for search indexing, entity extraction, or structured display. * **[Get original document](/api-reference/documents/get-original-document)**: Return a time-limited pre-signed URL. GET that URL to download the **private** document’s original file in its native format (e.g. .eml, .pdf). * **[Delete document](/api-reference/documents/delete-document)**: Remove a document from the platform. Deletes the original document, annotated version, and chunks from the vector database. Documents can be **tagged** automatically from ingestion, for example **sender/recipient-metadata** tags for email, and **`broker:Broker Name`** for investment research. You can also define **custom tags** and attach them to documents after upload. *** ## Tags Tags help you organize and filter documents in **Search Service** and **Research Agent**. * **[List tags](/api-reference/tags/list-tags)**: List tags visible to your organization (including connector-generated tags), with document counts. Use optional `prefix` to narrow results (for example `broker:`, `from:`, or `to:`). * **[Create tag](/api-reference/tags/create-tag)**: Create a custom tag by name. The response includes an `id` you use when updating document metadata. * **[Delete tag](/api-reference/tags/delete-tag)**: Deletes a custom tag by ID. Any documents linked with this tag will have the association removed. To change tags on an existing document, call **[Update document metadata](/api-reference/documents/update-document-metadata)** with a `tags` array of **tag IDs** (not names). The array replaces the document's current tags, so you can extend or reduce the set by sending the full list you want. Every ID must exist; otherwise the API returns `400` with `INVALID_TAGS_ERROR`. You can combine tag updates with `share_with_org` in the same request to change organization-wide visibility. *** ## Enrichments **Enrichments** configure the processing pipeline to apply additional steps to your content. You can set them per document on [direct upload](/api-reference/documents/enrich-document) or per connector on [Create connector](/api-reference/connectors/create-connector) (applied to every document ingested through that connector). Pass an **`enrichments`** array of enrichment names in the request body. | Enrichment | Description | | - | - | | `reporting_entities` | Identifies the document's **reporting company**, using the same concept as the [`ReportingEntity` reporting detail](/getting-started/search/query_filters#reporting-details) in Search. This keeps the reporting entity consistent with the rest of the Bigdata corpus, so you can reliably filter private content, transcripts and filings together. | | `translation` | Translates the document content so non-English material can be searched and analyzed alongside the rest of your corpus. Supports more than 70 languages. | You can request one or more enrichments in the same request: ```json theme={null} { "file_name": "Q2_2025_equity_research_note.pdf", "published_ts": "2025-06-15T10:30:00Z", "tags": ["equity-research", "sector:semiconductors"], "share_with_org": false, "enrichments": ["reporting_entities", "translation"] } ``` After processing, document metadata from [List documents](/api-reference/documents/list-documents) and [Get document metadata](/api-reference/documents/get-document-metadata) exposes: * **`enrichments_requested`**: The enrichments you asked for (on upload or via the connector). * **`enrichments_applied`**: The enrichments that completed successfully. Compare the two arrays to see what is still pending or was not applied. *** ## Processing options On [direct upload](/api-reference/documents/enrich-document), two optional boolean flags control whether the document is indexed and how long it is retained. Both default to omitted/`false` (enrich, store, and index). | Flag | Effect | | - | - | | `no_index` | Enrich the document but **do not index** it. Original and annotated files remain stored and retrievable via the Content API. The document is **not** available in Search or Research Agent. | | `no_store` | Enrich the document, **do not index** it, and keep it retrievable for **24 hours after enrichment completes**. After that the document is deleted. Use this for single-use processing. | You can set both flags to `true`. `no_store` already skips indexing, so `no_index` has no additional effect in that case. These flags apply only to **direct upload** (`POST /contents/v1/documents`). Connector-ingested documents are still stored and indexed. *** ## Typical flow **Using connectors (async / unsupervised):** 1. **[Create a connector](/api-reference/connectors/create-connector)** (e.g. email) with type, label, and config; for email, use the returned inbox address to forward messages. 2. Optionally check that content has been enriched and indexed by [listing your recent documents](/api-reference/documents/list-documents). 3. Use your content in the Search or Research Agent Service (e.g. filter by **Emails** in the playgrounds below). **Using direct upload (manual corpus):** 1. Call **[Enrich document](/api-reference/documents/enrich-document)** with metadata (file name, tags, sharing, and optional `no_index` / `no_store`); use the returned URL to PUT the file (upload it). 2. Poll [Get document metadata](/api-reference/documents/get-document-metadata) with the returned id to check `status` until processing is complete (`completed`). See [Get document metadata](/api-reference/documents/get-document-metadata) in the API reference. If you set `no_store`, download the annotated or original file within 24 hours of completion. 3. Use indexed content in the Search or Research Agent Service (e.g. filter by **My Files** in the playgrounds below). Documents uploaded with `no_index` or `no_store` are not searchable. > **[Search Service Playground](https://platform.bigdata.com/search/search-documents)** > > Search across your private content and other sources. In the playground, open the source selector and choose **My Files** to limit results to your uploaded documents. > **[Research Agent Playground](https://platform.bigdata.com/research-agent)** > > Run research over your private content and real-time data. In the playground, use the source selector and filter by **My Files** to ground answers in your documents. *** ## Authentication and reference All endpoints require an API key sent in the `X-API-KEY` header. See [Authentication](/api-rest/introduction) for details. Use the **API reference** (endpoints listed in this section) for request/response schemas, parameters, and examples. You can try the API from the [Developer Platform](https://platform.bigdata.com) playgrounds. # Introduction to Bigdata APIs Source: https://docs.bigdata.com/api-rest/introduction Bigdata APIs let your proprietary platform access Bigdata services in a deterministic, programmable way. If you prefer to build workflows orchestrated by AI agents, you can reach the same services through the [MCP server](/mcp-reference/introduction) or the [Bigdata CLI](/cli-reference/introduction). This page is the reference for every available Bigdata endpoint and the authoritative source for all supported parameters. You can try requests directly from the playgrounds linked below, or, for a simpler and more user-friendly experience, run them from the [Developer Platform](https://platform.bigdata.com). > **[Knowledge Graph API](/api-reference/companies/find-by-details)** > > Search for entities using Bigdata's knowledge graph tools. > **[Search Service API](/api-reference/search/search-documents)** > > Easily find the most relevant information from trusted sources and your own data. Use it to power agents that give accurate, real-time answers. Save a search, run it continuously, and extract grounded, structured events from newly matched content. > **[Research Agent API](/api-reference/research-agent/research-agent)** > > Get faster insights and smarter decisions using our advanced AI capabilities and real-time data. > **[Workflows API](/api-reference/workflows/execute-workflow)** > > Build reproducible, templated research workflows for automated analysis. > **[Content API](/api-rest/content_introduction)** > > Upload and manage your own private or proprietary content so it can be searched and analyzed alongside Bigdata's corpus. > **[Structured Data API](/api-reference/market-data/daily-prices)** > > Access comprehensive financial data including corporate events, intraday prices, analyst ratings, estimates, and earnings surprises. ## Pricing and usage monitoring See the [Bigdata API pricing page](/getting-started/bigdata_api_pricing) for full details on plans and metering. You can monitor consumption programmatically through the [subscriptions endpoint](/api-reference/quotas/get-subscription-quotas), as described in the [Monitor usage how-to guide](/how-to-guides/monitor_usage), or visually from the [Developer Platform](https://platform.bigdata.com/usage). # Bigdata Developer Platform We recommend using the [Developer Platform](https://platform.bigdata.com) to test your queries and understand how our APIs work. Also, use it to manage your API keys and monitor your API consumption and costs. > **[🚀 Bigdata Developer Platform](https://platform.bigdata.com)** > > **Your companion for day-to-day Bigdata API development** > > The Developer Platform is your central hub for managing, testing, and optimizing your Bigdata API integrations. Get instant business value with interactive playgrounds that make it incredibly easy to understand how our APIs work and discover their full potential. > > **Key Features:** > > * **🎮 Advanced Playgrounds**: Interactive environments with visual input/output to experiment with queries and see results instantly > * **🔑 API Management**: Centralized control of your API keys and access credentials > * **📊 Usage & Spending Tracking**: Monitor your API consumption and costs in real-time > * **📱 Demo Apps Library**: Explore pre-built applications showcasing what's possible with Bigdata APIs # Rate limits Source: https://docs.bigdata.com/api-rest/rate_limits The Bigdata.com API applies the following rate limits: | Scope | Limit | | - | - | | Per API key | 500 requests per minute | | Per IP address | 1,500 requests per minute, across all Bigdata API endpoints | ## Per API key Each API key can send up to **500 requests per minute**. We recommend creating multiple API keys, at least one per environment, so that workflows running in your development environment do not impact the performance of your production environment. You can create and manage your API keys in the [Developer Platform > API Keys](https://platform.bigdata.com/api-keys). See [Authentication](/api-rest/authentication) for more details. ## Per IP address Each IP address can send up to **1,500 requests per minute**. This limit applies across all Bigdata API endpoints, regardless of how many API keys are used from that IP address. # Workflows API Source: https://docs.bigdata.com/api-rest/workflows/introduction Build reproducible, templated research workflows for automated analysis The **Workflows API** enables templated, reproducible research designed for automation. Unlike the conversational Research Agent, Workflows uses parameterized templates that produce consistent outputs across executions. ## Workflows vs Research Agent | Feature | Research Agent | Workflows API | | - | - | - | | **Interaction style** | Ad-hoc natural language queries | Templated, reproducible research | | **Execution** | Single execution | Reusable templates | | **State management** | Conversational (chatId) | Parameterized (template + inputs) | | **Best for** | Exploration and discovery | Automation and production | | **Research plan** | Dynamic, agent-determined | Optional predefined steps | ## Core Concepts ### Templates Templates are reusable research specifications that define: * **Prompt**: The research question with Jinja2 placeholders (e.g., `{{ company_id }}`) * **Expected inputs**: Typed parameters the template requires * **Content filters**: Optional restrictions on which sources to search * **Research plan**: Optional predefined steps for structured execution * **Expected output**: Optional report template describing the final answer's structure, tone, and format ### Research Plan Behavior The Workflows API behavior is controlled by the presence of a `research_plan`: * **Without a plan**: The agent dynamically determines research steps based on your prompt. * **With a plan**: The agent follows your predefined steps for structured, predictable output. ### Input Types Templates support two input types: * `rp_entity_id`: A Bigdata entity ID (e.g., company identifier like `D8442A`) * `string`: Free-form text input ## Quick Links > **[Quickstart Guide](/getting-started/quickstart_guide_workflows)** > > Get started with your first workflow execution > **[Running a Workflow](/how-to-guides/agents/workflows/running-a-workflow)** > > Submit a run that outlives your connection, then read, watch, or cancel it > **[Creating Templates](/how-to-guides/agents/workflows/creating_templates)** > > Learn how to build effective templates > **[Research Plans](/how-to-guides/agents/workflows/execution_modes)** > > Learn how research plans affect execution > **[Community Templates](/how-to-guides/agents/workflows/community_templates)** > > Discover and clone shared templates ## Running a workflow: synchronous or asynchronous A workflow can stream its result on the connection that started it, or run independently of that connection. * **`POST /v1/workflow/execute`** streams the whole run back on the same HTTP response. Closing the connection cancels the run. * **`POST /v1/workflow/execute/async`** returns an `execution_id` immediately and the run continues on its own. Its result is always stored, you can attach to its event stream at any point, and disconnecting never stops it. Use the asynchronous endpoint for production integrations. See [Running a workflow](/how-to-guides/agents/workflows/running-a-workflow). ## API Endpoints The Workflows API provides the following endpoints: | Endpoint | Method | Description | | - | - | - | | `/v1/workflow/execute` | POST | Execute a workflow and stream the result on the same connection | | `/v1/workflow/execute/async` | POST | Submit a workflow that runs independently of your connection | | `/v1/workflow/execute/async/{execution_id}/stream` | GET | Attach to a submitted run and receive its events live | | `/v1/workflow/execute/async/{execution_id}/cancel` | POST | Stop a running workflow | | `/v1/workflow/executions` | GET | List your stored runs | | `/v1/workflow/executions/{execution_id}` | GET | Retrieve a stored run with its full result | | `/v1/workflow/executions/{execution_id}` | DELETE | Delete a stored run | | `/v1/workflow/templates` | GET | List your templates | | `/v1/workflow/templates` | POST | Create a new template | | `/v1/workflow/templates/{id}` | GET | Get a specific template | | `/v1/workflow/templates/{id}` | PUT | Update a template | | `/v1/workflow/templates/{id}` | DELETE | Delete a template | | `/v1/workflow/templates/community` | GET | Browse community templates | | `/v1/workflow/templates/{id}/clone` | POST | Clone a template to your account | ## Available Models Select the model that best fits your use case: | Model | Description | | - | - | | `base` | Default model, balanced performance | | `pro` | Enhanced reasoning capabilities | ## Time Range Options Control the time range of research data: **Rolling time ranges:** * `last_24_hours` * `last_7_days` * `last_30_days` * `last_60_days` * `last_90_days` * `last_180_days` * `last_365_days` **Custom date range:** ```json theme={null} { "start": "2024-01-01T00:00:00Z", "end": "2024-12-31T23:59:59Z" } ``` ## Streaming Response The Workflows API returns responses via Server-Sent Events (SSE). Each event wraps a typed payload in a `delta` field. The most common message types: | Type | Description | | - | - | | `THINKING` | Agent's reasoning process | | `PLANNING` | Research plan with steps | | `ACTION` | Tool being called | | `ANSWER` | Final response content | | `GROUNDING` | Source references | | `AUDIT` | Search results and traces | | `COMPLETE` | Execution finished with resource consumption | | `ERROR` | Error occurred | > [!NOTE] > > This list is abbreviated. For every public message type -- including `LLM_RETRY`, > `TOOL_ERROR`, and `STRUCTURED_OUTPUT` -- their field schemas, when each fires, and > a copy-paste Python handler, see > [Streaming responses](/how-to-guides/agents/concepts/streaming-responses). # Chatting with your files Source: https://docs.bigdata.com/app/getting-started/chat_with_your_files Bigdata lets you upload and interact with your own files. This is available on both desktop and mobile. Need to analyze a specific report, contract, or research paper? Simply upload it, and Bigdata will help you extract the key information you need. It's a powerful tool that saves you time and ensures you get the most out of your research. There are two ways to chat with your documents: ## Chat with Files from the homepage * Click the paperclip icon to attach one or more files. * Type your question or request. * Hit "Send" and Bigdata will analyze your uploaded documents. > [!NOTE] > > **Good to know:** If you upload a document without a question, Bigdata will automatically "Summarize this document" to get you started. ## Using the "My Files" page * Drag and drop your files onto the page for easy upload (Desktop Only) * Alternatively, select files using the checkboxes. * Click the chat icon (appears next to the 3-dot menu) to initiate a chat. * Type your question and Bigdata will get to work. # How Bigdata generates your answers Source: https://docs.bigdata.com/app/getting-started/how_generates_answers ## Introducing the Audit Trail When you ask a question, Bigdata breaks it down into smaller, more focused queries. These queries are then sent to various finance-focused language models for a comprehensive response. To understand Bigdata's reasoning, look for the "Show Audit Trail" icon at the bottom of each Bigdata chat response. It provides a step-by-step breakdown of how your answer was generated. Clicking the icon reveals the step-by-step process Bigdata uses to build your answer: * **Understanding your question:** Bigdata first analyzes your prompt to grasp its meaning and intent. * **Formulating queries:** based on your question, Bigdata generates specific queries to retrieve relevant information. * **Revealing information sources:** for each query, Bigdata identifies the exact sources it uses to gather the information, for more transparent and trustworthy insights. # Bigdata Assistant Source: https://docs.bigdata.com/app/introduction The [Bigdata Assistant](https://bigdata.com) is a next-generation AI tool tailored for professionals navigating fast-moving financial environments. Purpose-built to support real-time decision-making, it streamlines search and discovery across a wide range of information sources — including proprietary databases, public filings, premium financial content, and gated research, and structured data sources. Now you can automate routine analysis with pre-built Workflows, monitor your portfolio with an interactive Watchlist, precisely select your sources with the Content Catalogue, and tackle complex questions with our Research agent. By understanding context and intent, the Assistant surfaces both relevant and trustworthy insights instantly, helping users cut through information overload and act with confidence. ## Why use the Bigdata Assistant? Finance professionals are often hesitant to use conversational chatbots due to several unmet needs: * **Transparency of sources** * Every insight is fully attributable, showing you the exact source of the information. * You can also take control by using the Content Catalogue to filter and select from specific trusted sources before asking a question. * **Real-time accuracy** * Our agent is grounded in real-time data, ensuring you get insights that are accurate up to the very last minute and reducing the risk of outdated information. * **Specialized Depth** * Generic chatbots provide superficial answers. For complex questions, our Deep Research agent plans and executes a multi-step analysis, drawing insights from the right datasets—structured and unstructured alike—to deliver the nuanced, expert-level information that professionals rely on. ## How Bigdata.com reinvents business search Bigdata combines finance-specific language models with our proprietary Knowledge Graph, which connects vast amounts of structured and unstructured financial data. This data comes from trusted sources, with full traceability and auditability on every answer: * **Public sources:** over 13,500 financially relevant web content sources, including SEC regulatory filings. * **Transcripts for 40,000 listed companies** including earnings results, analyst briefings, and company conferences. * **Gated sources:** premium insights from RavenPack, FactSet, MT Newswires, Benzinga, The Fly, Alliance News, FXStreet, LinkUp. * **Internal sources:** company data from file servers, eComms, CRM, ERP, BI Tools. > [!NOTE] > > ### Real-time knowledge, fewer hallucinations > > Our agent is grounded in real-time data, which helps build trust and produce timely, accurate insights. ## How Bigdata empowers financial workflows Use the Assistant to gain market intelligence, automate research, and make informed financial decisions. * **Ask complex questions:** Use natural language to have the Deep Research agent plan and execute multi-step research. * **Launch pre-built Workflows:** Automate routine analysis, like preparing a detailed earnings preview, in just a few steps. * **Create briefs and reports:** Instantly generate summaries and reports from high-quality data sources with full attribution. * **Upload your own content:** Chat with your files to quickly find key information, generate summaries, or ask specific questions about the documents. * **Monitor your portfolio:** Use the Watchlist to monitor your portfolio, create custom lists, or track curated indices. ## Why Financial users choose Bigdata App and API * **Enterprise-Grade Security:** Your data is kept completely private, is never used for model training, and is protected by ISO 27001:2022 certified controls, with SOC 2 Type 1 and Type 2 reports available on request. * **Smarter Than Chatbots:** Purpose-built for the financial industry, our proprietary Knowledge Graph understands complex financial relationships, delivering more precise and attributable results than generic AI. * **One-Stop Shop:** Create a single source of truth by securely uploading proprietary documents to be searched alongside premium financial data and public content, all within one easy-to-use platform. * **Workflow Automation:** Go beyond simple Q\&A by using pre-built workflows to automate complex, routine analysis like earnings previews, saving valuable time and reducing manual effort. # Building investment cases Source: https://docs.bigdata.com/app/use-cases/equity-research/building_investment_cases To build a compelling investment case is a complex, time-consuming process that can take weeks. That is because it requires a thorough analysis of financial data, market trends, risks and opportunities. Yet most of this information is scattered across the internet, sometimes behind paywalls. Bigdata streamlines this process. It provides efficient, real-time access to reliable sources, such as transcripts, news, fillings and market reports. In this example, we use Bigdata to help us build an investment case for Anthropic from scratch. Let's start with a general question: > How does Anthropic stand as a potential investment opportunity? **response** Based on the information I have, Anthropic is a leading artificial intelligence (AI) company that has raised significant funding from major tech companies such as Amazon and Google. The company was founded by former employees of OpenAI and has since become a formidable competitor in the AI space. Anthropic has differentiated itself from OpenAI by structuring itself as a public benefit corporation, meaning its board is legally empowered to balance returns for investors with a separate mission to ensure that "transformative AI helps people and society flourish." This focus on safety and responsibility has attracted business customers from various industries, including drug maker Pfizer and conversational search engine Perplexity AI. The company has also developed a chatbot named Claude, which it has made available as an iPhone app. According to reports, Anthropic projects will take in more than \$850 million in annualized revenue by the end of 2024. This revenue growth is driven in part by the increasing demand for AI technology in various industries. Anthropic has also received funding from other investors, including Google, which invested about \$300 million in the company in late 2022. The company has also sold its shares to other investors, including ATIC Third International Investment Company LLC, associated with the UAE's sovereign wealth fund Mubadala, which acquired nearly \$500 million worth of FTX's Anthropic shares. These investments provide a strong foundation for Anthropic's growth and development. Furthermore, Anthropic's focus on safety and responsibility sets it apart from other AI companies and positions it well to address the potential risks associated with transformative AI. This focus on safety and responsibility is likely to be a winning strategy in the long run, as it aligns with the growing concern among investors and the public about the ethical implications of AI technology. In conclusion, Anthropic is a well-funded and rapidly growing AI company with a strong focus on safety and responsibility. Its differentiated business model, attractive customer base, and strong financial position make it an attractive investment opportunity for those looking to capitalize on the emerging AI market. ## Projected Growth Trajectory With that big picture in mind, let's ask Bigdata to provide a more data-driven forecast of Anthropic's future growth potential. > What is the projected growth trajectory for Anthropic's revenue and market share in the next 5 years? **response** Anthropic is projected to take in more than \$850 million in annualized revenue by the end of 2024. The company's financial needs may continue to grow in the coming years as the cost of training models is expected to reach \$100 billion. Anthropic's focus on safety and responsibility, as well as its partnerships with major tech companies like Amazon and Google, position it well to address the potential risks associated with transformative AI and attract business customers from various industries. Regarding market share, there is no specific information available in the given news snippets to make a definitive projection. However, Anthropic's differentiated business model, attractive customer base, and strong financial position make it a competitive player in the AI market. The increasing demand for AI technology in various industries and Anthropic's focus on safety and responsibility are likely to contribute to its growth in market share. ## Potential Risks and Challenges Bigdata can also help build a comprehensive risk assessment and identify potential regulatory hurdles, competition from other companies, or technical limitations. > Analyze the potential risks and challenges that Anthropic might face in its development and commercialization of AI solutions. **response** One of the most significant risks is the potential misalignment between the objectives of the AI system and human values. Anthropic recognizes this risk and has made it a priority to ensure that its AI systems are aligned with human values. The company has implemented a Responsible Scaling Policy, which pledges not to release AIs above certain capability levels until it can develop sufficiently robust safety measures. This approach aims to pressure competitors to make similar commitments and eventually inspire binding government regulations. Another risk is the possibility of AI systems developing autonomy or falling into the wrong hands. Anthropic recognizes this risk and is investing in research to identify and address these potential issues. The company has made a major breakthrough in this area by identifying millions of "features" inside a version of Claude, which allows researchers to alter the behavior of the system by toggling these features on and off. A third risk is the financial pressure to scale up operations and meet the increasing demand for AI technology. Anthropic has raised significant funding, but it may need to raise even more in the future. The company's structure as a public benefit corporation, which balances returns for investors with a mission to ensure that AI helps people and society flourish, may come under pressure from corporate juggernauts. ## Financial Performance Comparison When building an investment case, it's important to have insights into the company's relative strength and weaknesses within the market. Let's ask Bigdata to compare Anthropic's financial performance to its key competitors. > Compare Anthropic's financial performance and valuation to its key competitors, such as OpenAI and DeepMind. Focus on key metrics like revenue, profitability, and valuation, and consider the current state of the AI market and the specific segments each company is targeting. **response** **Anthropic:** * Has raised a total of \$7.6 billion in funding * Projected to take in more than \$850 million in annualized revenue by the end of 2024 * Received funding from major tech companies like Amazon and Google * Focus on safety and responsibility and partnerships with industry leaders position it well in the market **OpenAI:** * Valued at over \$80 billion to \$90 billion * Generated over \$1.6 billion in revenue in the past year * Expanding its board and seeking funding for a new AI chip venture * Microsoft-backed, making it a significant player in the enterprise AI market **DeepMind:** * No recent financial performance or valuation data available * Has made significant strides in AI research and collaborations with industries and academia * Google-owned, giving it access to significant resources and market reach ## Strategic Partnerships Strategic partnerships can offer investors insights into the company's growth strategies and potential for market expansion. In our case, we ask Bigdata to identify Anthropic's key collaborations such as joint ventures, technology integrations, or access to new markets. > Identify potential strategic partnerships or collaborations that could further enhance Anthropic's growth and market position. **response** Anthropic has already established strategic partnerships with major tech companies like Amazon and Google, which have provided it with significant funding, chips, and cloud infrastructure. These partnerships have helped Anthropic maintain independence and differentiate itself from competitors that are more closely tied to one large investor. Additionally, Anthropic has collaborated with AWS and Accenture to help organizations responsibly adopt and scale generative AI technology. This collaboration has enabled Anthropic to provide its AI systems to a wider customer base, especially in highly-regulated industries like healthcare, public sector, banking, and insurance. Another potential collaboration could be with industry associations, research institutions, and government agencies through AI Apex Asia or similar organizations. Such collaborations could provide Anthropic with valuable insights into best practices, emerging trends, and regulatory requirements in the AI market. They could also offer opportunities for joint initiatives and collaborative efforts to address critical challenges and explore frontier advancements in AI. Furthermore, as the open-source world grapples with security and licensing issues for generative AI, Anthropic could collaborate with initiatives like the Linux Foundation and CNCF to develop standards and guidelines for safe and ethical AI use. This collaboration could help Anthropic position itself as a leader in the responsible development and deployment of AI technology, while also contributing to the broader AI ecosystem. ## Writing the investment case As a final step, let's ask Bigdata to puzzle all the findings into a compelling investment case. > Based on the analysis above, construct an investment case for Anthropic, incorporating risk factors to determine its value and assess its investment potential. **response** ### Investment Case for Anthropic #### Overview Anthropic is a rapidly growing artificial intelligence (AI) startup that has secured significant funding from major tech companies like Amazon and Google. The company's focus on safety and responsibility, as well as its partnerships with industry leaders, have positioned it well in the market. In this investment case, we will summarize key findings from the analysis above on Anthropic's growth trajectory, risks, financial performance comparison, and strategic partnerships. We will also provide an assessment of Anthropic's investment potential. #### Growth Trajectory Anthropic's financial performance is projected to be strong, according to available data. The company has raised a total of \$7.6 billion in funding and is projected to take in more than \$850 million in annualized revenue by the end of 2024. Anthropic's revenue growth and valuation are still lower than its competitors OpenAI and DeepMind, but the company's independence from a single large investor and its collaborations with AWS and Accenture provide it with unique advantages. #### Risks There are risks associated with investing in Anthropic. One risk is the rapidly changing nature of the AI market, which could make it difficult for the company to maintain its competitive edge. Another risk is regulatory uncertainty, as governments and industry associations continue to grapple with the ethical and safety implications of AI technology. However, these risks are mitigated by Anthropic's focus on safety and responsibility, its partnerships with major tech companies and industry leaders, and its collaborations with industry associations, research institutions, and government agencies. These collaborations could provide valuable insights into best practices and emerging trends in the AI market. Furthermore, Anthropic's commitment to responsible AI development and deployment could position it well to capitalize on the growing demand for ethical and safe AI technology. #### Financial Performance Comparison Anthropic has raised a total of \$7.6 billion in funding and is projected to take in more than \$850 million in annualized revenue by the end of 2024. OpenAI, on the other hand, is valued at over \$80 billion to \$90 billion and generated over \$1.6 billion in revenue in the past year. DeepMind, which has made significant strides in AI research and collaborations with industries and academia, does not have recent financial performance or valuation data available. While Anthropic's revenue growth and valuation are lower than its competitors, its independence from a single large investor and its collaborations with AWS and Accenture provide it with unique advantages. #### Strategic Partnerships Anthropic has already established strategic partnerships with major tech companies like Amazon and Google, providing it with significant funding, chips, and cloud infrastructure. Additionally, Anthropic's collaborations with industry associations, research institutions, and government agencies could provide valuable insights into best practices and emerging trends in the AI market. Furthermore, Anthropic's focus on responsible AI development and deployment could position it well to capitalize on the growing demand for ethical and safe AI technology. Some potential collaborations could include: * Industry associations: Collaborating with industry associations like AI Apex Asia or similar organizations could provide Anthropic with valuable insights into best practices, emerging trends, and regulatory requirements in the global AI market. * Research institutions: Collaborating with research institutions like MIT, Stanford, or Carnegie Mellon could provide Anthropic with access to cutting-edge research and expertise in AI and related fields. This could help Anthropic stay at the forefront of AI research and development. * Government agencies: Collaborating with government agencies like the National Institute of Standards and Technology (NIST) or the European Union Agency for Cybersecurity (ENISA) could provide Anthropic with valuable insights into regulatory requirements and best practices for responsible AI development and deployment. It could also offer opportunities for joint initiatives and collaborative efforts to address critical challenges and explore frontier advancements in AI. #### Investment Potential Anthropic's strong financial performance, unique advantages in the market, strategic partnerships, and commitment to responsible AI development make it an attractive investment opportunity. While there are risks associated with investing in anthropic, such as the rapidly changing nature of the AI market and regulatory uncertainty, these risks are mitigated by the company's focus on safety and responsibility, its partnerships with major tech companies and industry leaders, and its collaborations with industry associations, research institutions, and government agencies. Furthermore, Anthropic's commitment to responsible AI development and deployment could position it well to capitalize on the growing demand for ethical and safe AI technology. # Tracking Investment Trends Source: https://docs.bigdata.com/app/use-cases/equity-research/tracking_investment_trends ## Get real-time insights for informed decisions Investors constantly seek out industries primed for significant growth and potential disruption. Traditionally, this has been a subjective, time-consuming process. Bigdata analyses information from premium news sources, social media, financial data to help identify industries that experience a surge in demand or technological breakthroughs. This framework demonstrates how Bigdata can be used to assess the investment potential of an industry, using **robotic surgery** as an example. ## Market Analysis Let's start with the essentials for constructing a robust ROI model and ask Bigdata to provide information about the size and growth projections of this new market. > Summarize the market size and growth projections for the global medical robotics and computer-assisted surgery market. **response** The global market for medical robotics and computer-assisted surgery is projected to reach \$19.5 billion by 2028, growing at a compound annual growth rate of 11.0% during the forecast period. This growth is driven by technological advances, increased adoption, rising rates of chronic disease, an aging population, demand from surgeons, and favorable reimbursement policies. ## Key Players With a better grasp of the market potential, let's now identify the key players in this space. We ask Bigdata to provide a comprehensive landscape analysis and help pinpoint companies with the strongest value propositions. > Which specific companies are talking about Robotic Surgery investments? **response** Investments in robotic surgery companies are on the rise. For instance, Fidelity Management & Research Company led a \$110 million investment in MMI, and Intuitive Surgical announced FDA clearance of its fifth-generation robotic system, da Vinci 5. These developments further underscore the focus on robotic surgery and its potential to enhance healthcare infrastructure and contribute to socio-economic development. Nami Surgical is one of the companies that has recently secured investment for robotic surgery. The investment was led by Eos Advisory and included the Investment Fund for Scotland, Maven, Scottish Enterprise, SIS Ventures, and Innovate UK. Other companies mentioned in the news snippets as key players in the medical robotics and computer-assisted surgery market include Intuitive Surgical Operations, Medtronic, Stryker, and Zimmer Biomet. ## Specific Areas of Focus For a more nuanced investment thesis, Bigdata can help narrow down the research to specific applications in Robotic Surgery: > Are there particular areas within robotic surgery (e.g., minimally invasive procedures, specific surgeries) receiving more focus? **response** Based on the information I have, there are several areas within robotic surgery that are receiving significant focus. One such area is minimally invasive procedures, as robotic surgery enhances surgical precision and dexterity, leading to reduced patient trauma, post-operative pain, and recovery times. This results in shorter hospital stays and improved patient outcomes. Another focus area is specific surgeries such as urology, gynaecology, and colorectal procedures. Companies like Nami Surgical are developing pioneering technology to integrate ultrasonic scalpels in robotic surgeries, setting a new benchmark in surgical care. ## Investment Strategies And to further unpack a company's competitive edge and risk profile, Bigdata can offer insights into their strategic playbook, including internal R\&D, strategic acquisition or partnership dynamics. > [!NOTE] > > **Tip**: Click on the "Show Audit trail" icon at the bottom of the response to see how Bigdata breaks down your prompt and gathers the information necessary to get you an answer. > Are companies investing in developing their own robotic surgery systems, acquiring existing technology, or partnering with other players? **response** Based on the information I have, companies are engaging in various strategies to advance robotic surgery. Some are investing in developing their own technology, while others are acquiring existing systems or forming strategic partnerships. For instance, Nami Surgical recently secured investment to develop its ultrasonic platform for robotic-assisted surgery. In another instance, SpinEM Robotics and Spineart forged a strategic partnership to revolutionize spinal surgery by combining their respective expertise and technologies. Additionally, Stereotaxis announced the acquisition of Access Point Technologies to amplify its efforts in advancing minimally invasive endovascular intervention through robotics. Moreover, significant funding rounds have been reported in the digital surgery technologies market, indicating a growing interest in investing in this field. Companies like Medical Microinstruments, Fabric, and HealthSnap have raised substantial investments to commercialize their surgical robotics platforms and advance AI capabilities. ## Summarizing Market Research Reports Market research and analyst reports are invaluable for validation, but reading them takes ages. Ask Bigdata to summarize the key insights from these reports. > [!NOTE] > > **Tip**: You can ask Bigdata to analyse reports from external sources or from your own documents. > Find 3 recent market research reports (past 6 months) on global robotic surgery. Summarize: market size, growth forecast (CAGR), and key players. **response** Based on the information I have, here are summaries of three recent market research reports on global robotic surgery: "Medical Robotics and Computer-assisted Surgery: The Global Market 2023-2028" (Source: ResearchAndMarkets.com) The global market for medical robotics and computer-assisted surgery is projected to grow from \$11.5 billion in 2023 to \$19.5 billion by the end of 2028, at a compound annual growth rate (CAGR) of 11.0% during the forecast period. Key companies profiled in this report include Intuitive Surgical Operations, Medtronic, Stryker, and Zimmer Biomet. "Surgical Robotics Market to Achieve 15.7% CAGR Amidst Increasing Adoption of Advanced Technologies" (Source: PharmiWeb) The global surgical robotics market was valued at USD 6.2 billion as of 2022 and is projected to reach USD 25.7 billion by 2032, growing at a compound annual growth rate (CAGR) of 15.70% during the forecast period from 2033 to 2032. Key players in this market include Intuitive Surgical, Medtronic, Stryker, Zimmer Biomet, and Smith & Nephew. "Global Robotic Surgery Market Size, Share & Industry Analysis, By Type, Application, End User, and Regional Forecast, 2023-2030" (Source: Fortune Business Insights) The global robotic surgery market was valued at \$8.3 billion in 2022 and is projected to reach \$32.4 billion by 2030, growing at a compound annual growth rate (CAGR) of 18.1% during the forecast period. Key players in this market include Intuitive Surgical, Medtronic, Stryker, and Zimmer Biomet. These reports highlight the significant growth and expansion of the global robotic surgery market, driven by technological advances, increased adoption, and favorable reimbursement policies. The market is projected to reach substantial values by the end of the forecast period, indicating potential for significant returns on investment. Key players in the sector include Intuitive Surgical, Medtronic, Stryker, Zimmer Biomet, and others. ## Media Sentiment Bigdata can also analyze media sentiment around a specific topic, for a more nuanced understanding of the industry's prospects. This will reveal whether the media portrays a particular technology or innovation as a positive investment opportunity or a risky venture. > [!NOTE] > > **Tip**: Bigdata feeds your knowledge workflows from real-time indexing of 40,000 financially relevant web content sources at scale and gated sources such as Dow Jones, FactSet transcripts, WSJ, Benzinga, MarketWatch, BARRON'S and more. > Analyze the sentiment of media coverage around Robotic Surgery in the past year. Does the coverage portray robotic surgery as a positive investment opportunity or a risky venture? **response** Media coverage around robotic surgery over the past year has generally portrayed it as a positive investment opportunity. The coverage highlights the significant growth and expansion of the market, driven by technological advances, increased adoption, and favorable reimbursement policies. The market is projected to reach substantial values by the end of the forecast period, indicating potential for significant returns on investment. However, it is important to note that media coverage also acknowledges the challenges and risks associated with robotic surgery, such as the need for advanced training for surgeons, high costs, and regulatory norms. These factors may impact the success and profitability of investments in the sector. ## Generating an email with key findings And finally, ask Bigdata to generate an email to your team with the key insight > Building on the information gathered, generate a concise email summarizing the key insights on robotic surgery investment trends. Include findings from media coverage analysis, company investment activities, and market research reports. **response** Subject: Key Insights on Robotic Surgery Investment Trends Dear \[Recipient], I am pleased to provide you with a summary of the latest developments and key insights on investment trends in the robotic surgery sector, based on the most recent news snippets and market research reports: * Robotic surgery market is experiencing strong growth, with the global market projected to reach \$19.5 billion by 2028, growing at a CAGR of 11.0% during the forecast period. * Companies are employing various strategies to drive innovation and growth, including developing their own technology, acquiring existing systems, and forming strategic partnerships. * Notable acquisitions include Stereotaxis' acquisition of Access Point Technologies and Asensus Surgical's ongoing merger activities. * Significant investments have been reported in the digital surgery technologies market, with companies like Medical Microinstruments, Fabric, and HealthSnap raising substantial funds to commercialize their surgical robotics platforms. * Media coverage remains positive, with a focus on the growth potential and opportunities in the field of robotic surgery. * Key players in the sector include Intuitive Surgical, Medtronic, Stryker, and Zimmer Biomet Some specific findings from the media coverage analysis include: * Nami Surgical secured \$4 million in investment and grant funding to develop its ultrasonic platform for robotic-assisted surgery. * The digital surgery technologies market is projected to reach \$5,084.7 million by 2032. * The US market for robotic surgery was valued at \$1.8 billion in 2022 and is expected to grow from \$6.4 billion in 2030 to an annual growth rate (CAGR) of 17.3%. I hope you find this information helpful and insightful. If you have any questions or require further clarification, please do not hesitate to ask. Best regards, \[Your Name] # Negative News Screening Source: https://docs.bigdata.com/app/use-cases/portfolio-management/negative_news_screening ## Insights from ESG Controversies Affecting Investments Controversial events may impact the performance and reputation of public and private companies. Bigdata tracks in real time 171 ESG controversies such as lawsuits, product recalls, or regulatory changes, across 40,000 curates sources. To showcase Bigdata's capability to screen for negative news, we've created a diverse portfolio of 10 companies across different sectors: | Company | Sector | | - | - | | Tesla Inc. (TSLA) | Automotive | | JPMorgan Chase & Co. (JPM) | Banking | | Pfizer Inc. (PFE) | Pharmaceuticals | | Walmart Inc. (WMT) | Retail | | ExxonMobil Corp. (XOM) | Energy | | Meta Platforms Inc. (META) | Social Media | | Airbus SE (AIR) | Aerospace | | Disney (DIS) | Entertainment | | General Electric (GE) | Industrial | ## Negative News Summary For a quick overview of the recent negative news impacting the portfolio, let's ask Bigdata to identify main negative news connected to the selected companies for the past month: > Provide a short report of recent negative news for the following companies in my portfolio: Apple Inc. (AAPL), Tesla Inc. (TSLA), JPMorgan Chase & Co. (JPM), Pfizer Inc. (PFE), Walmart Inc. (WMT), ExxonMobil Corp. (XOM), Meta Platforms Inc. (META), Airbus SE (AIR), Disney (DIS), and General Electric (GE). Include any major ESG controversies or events from the past month. **response** ### Apple Inc. (AAPL) * Negative sentiment due to analyst downgrades and iPhone/China concerns. * Recent earnings positive, but growth expectations weigh on stock. * No major ESG controversies reported recently. ### Tesla Inc. (TSLA) * Declining US demand due to CEO's image and safety issues. * Slowing growth, price cuts, and poor stock performance. * No major ESG controversies reported recently, but concerns over Musk and environmental impact exist. ### JPMorgan Chase & Co. (JPM) * No recent negative news or ESG controversies. * Past criticism for controversial deals and handling of 2008 crisis, but no recent reports. ### Pfizer Inc. (PFE) * Q1 revenue decline due to operations and currency impact. * Oncology biosimilar revenue down due to lower US pricing. * Maintained 2024 revenue guidance, raised adjusted EPS guidance. * No major ESG controversies reported recently, but past criticism for pricing practices and opioid crisis exists. ### Walmart Inc. (WMT) * Internal system failure in March caused mispricing at stores. * Widespread criticism and lower share prices due to the issue. * No major ESG controversies reported recently, but past criticism for labor practices and community impact exists. ### ExxonMobil Corp. (XOM) * Investor backlash due to legal actions against climate activists. * Upward earnings estimate revisions for 2024 and 2025. * Strong Zacks Style Score for Value and Momentum. ### Meta Platforms Inc. (META) * No specific negative news or ESG controversies reported recently, but past criticism for user data and mental health impact exists. ### Airbus SE (AIR) * No specific negative news or ESG controversies reported recently, but past criticism for arms deals and environmental impact exists. ### Disney (DIS) * No specific negative news or ESG controversies reported recently, but past criticism for labor practices and sexual misconduct allegations exists. ### General Electric (GE) * No specific negative news or ESG controversies reported recently, but past criticism for pension obligations and controversial deals exists. > [!NOTE] > > **Tip**: Click on the "Show Audit trail" icon at the bottom of the response to see how Bigdata breaks down your prompt and gathers the information necessary to get you an answer. ### Looking Closer at Specific Controversies Now that we have this general overview, let's ask Bigdata for more detailed insights into specific controversies, to have a better grasp on the potential impact on the company's operations, reputation, and stock performance. > For ExxonMobil, can you explain the specific legal actions against climate activist investors and the potential consequences? **response** * ExxonMobil sues investor groups Follow This and Arjuna Capital over a shareholder vote on emission reduction measures. * The proposal aimed to accelerate emission reductions and require targets for "Scope 3" emissions (emissions from a company's value chain). * The lawsuit alleges investor tactics aim to silence voices and disrupt shareholder democracy. * Investors view climate change as a threat to long-term investment and argue ExxonMobil's lawsuit could negatively impact corporate governance. * The investors withdrew the proposal after the lawsuit was filed. * CEO Darren Woods vows to protect annual meetings from "serial attacks" by activists. * The company accuses investors of a "Trojan horse" strategy to deliberately harm the company. * The lawsuit's outcome could impact corporate governance and shareholder influence on ESG issues. * The case highlights the growing tension between companies and activist investors over ESG agendas and shareholder proposals. ### Analyzing Market Trends But is this controversy specific to ExxonMobil, or does it have broader implications? This insight can be helpful if you want to identify market risks or to make sector-specific investment decisions. > Are there any broader market risks or trends that could be exacerbated by the ExxonMobil controversy? **response** * ExxonMobil's lawsuit against climate activists raises concerns about its relationship with investors and reputation. * Increased investor focus on ESG issues could negatively impact companies perceived as lacking in accountability and proactive climate action. * Geopolitical tensions and instability could disrupt trade, cause political instability, and disrupt supply chains, impacting the global economy and markets. * Debate on shareholder rights and corporate responsibility related to climate change could lead to increased regulation and compliance costs for companies, especially those reliant on fossil fuels. * The specific outcome and broader market impact of the ExxonMobil controversy depend on the legal interpretation and situation details. * This case highlights the importance of companies addressing ESG issues and the potential consequences for inaction. # Working with Earnings Transcripts Source: https://docs.bigdata.com/app/use-cases/portfolio-management/working_with_earnings_transcripts ## Get key insights from transcripts Bigdata can summarize call transcripts for over **40,000 publicly-listed companies**, dating back to 2003. In this scenario, let's say you're an investment analyst at a firm considering investing in Tencent. Attending every earnings call for every company you're interested in isn't feasible. Bigdata can quickly summarize key financial metrics from the latest Tencent earnings call, such as revenue, earnings per share, and gross margins. This gives you a clear picture of Tencent's current financial performance without having to spend hours combing through transcripts. > [!NOTE] > > You can use either the company name or ticker symbol. ## Summarizing key financial metrics > Summarize Tencent's financial performance for the past quarter, including revenue, earnings per share (EPS), and gross margins. **response** Tencent reported a strong financial performance for the first quarter of 2024. The company's net profit rose 62% year-on-year to 41.89 billion yuan (\$5.79 billion), exceeding analysts' expectations. Revenue for the period grew 6% from a year earlier to 159.50 billion yuan. Gross profit margin expanded significantly, increasing from 45% in the previous year to 53%. This growth was driven by high-margin businesses such as short videos and mini games embedded in its super app WeChat. The gross profit margin improvement led to a 23% increase in non-GAAP gross profits and a 54% increase in adjusted net income available to common shareholders. Earnings per share (EPS) also saw a substantial increase, with earnings per share at 4.386 yuan, up 66% from the previous year's 2.639 yuan. ## Comparative analysis You can also ask Bigdata to compare the performance of a company between quarters, which can be useful to expedite blackline reporting, > Compare Tencent's financial performance from the latest earnings call to the previous quarter. Identify any major changes in key metrics like revenue growth or profitability. **response** Based on the information I have, Tencent reported a 6% increase in revenue for the first quarter of 2024 compared to the same period last year. The company's non-GAAP net profit grew by 54% year-on-year. Gross profit was up 23% year-on-year, and gross profit margin expanded to 53%. Tencent's online ad revenue grew by 26%, and revenue from its online advertising business was the fastest growing segment. The company's profit attributable to equity holders of the company was up 62% year-on-year, marking the fastest growth since the March quarter of 2021. Compared to the previous quarter, Tencent's revenue growth rate remained consistent, but its profitability improved significantly. Its net profit grew at a faster rate than its revenue, and its gross profit margin expanded. The company's online ad revenue continued to grow, driven by the growth of WeChat and its AI-driven advertising technology platform. It is worth noting that Tencent has posted six consecutive quarters of gross profit growth, with gross profit growing faster than revenue, and gross profit growing more than 20% for four consecutive quarters. This growth was supported by high-margin businesses, such as short videos and mini games embedded in its super app WeChat. However, domestic game sales declined amid a lack of meaningful title releases during the period. or against the rest of its sector, to establish a benchmarked comparison: > Compare Tencent's latest earnings performance against its sector peers to establish a benchmarked comparison. **response** Tencent reported a 6% increase in revenue and a 54% increase in non-GAAP net profit year-on-year for the first quarter of 2024. Its gross profit grew by 23% year-on-year, and gross profit margin expanded to 53%. Tencent's online ad revenue grew by 26%, and revenue from its online advertising business was the fastest growing segment. Baidu reported a one percent increase in revenue year-on-year to \$4.4 billion in the first quarter of 2024. Net profit during the same period decreased six percent year-on-year to \$755 million. Alibaba reported a 7% increase in revenue year-on-year to \$30.73 billion in the fiscal year ended March 31, 2024. Net income rose modestly by 9 percent to \$9.9 billion. NetEase reported a 7.2% increase in revenue year-on-year to \$26.9 billion in the first quarter of 2024. Adjusted net profit grew 12% to CNY8.5 billion. Comparing Tencent's latest earnings performance with its sector peers, Tencent's net profit growth rate was higher than Baidu and Alibaba, but lower than NetEase. Tencent's revenue growth rate was lower than Alibaba and NetEase, but higher than Baidu. ## Drilling down Storytelling with top-line financials is only half the battle. Bigdata allows you to zoom in on specific segments, subsidiaries, or markets. This can uncover opportunities that high-level numbers might miss. Follow sources: Click on the "Show Audit trail" icon at the bottom of the response to see how Bigdata breaks down your prompt and gathers the information necessary to get you an answer. > Compare Tencent's latest earnings performance against its sector peers to establish a benchmarked comparison. **response** Tencent reported a 6% increase in revenue and a 54% increase in non-GAAP net profit year-on-year for the first quarter of 2024. Its gross profit grew by 23% year-on-year, and gross profit margin expanded to 53%. Tencent's online ad revenue grew by 26%, and revenue from its online advertising business was the fastest growing segment. Baidu reported a one percent increase in revenue year-on-year to \$4.4 billion in the first quarter of 2024. Net profit during the same period decreased six percent year-on-year to \$755 million. Alibaba reported a 7% increase in revenue year-on-year to \$30.73 billion in the fiscal year ended March 31, 2024. Net income rose modestly by 9 percent to \$9.9 billion. NetEase reported a 7.2% increase in revenue year-on-year to \$26.9 billion in the first quarter of 2024. Adjusted net profit grew 12% to CNY8.5 billion. Comparing Tencent's latest earnings performance with its sector peers, Tencent's net profit growth rate was higher than Baidu and Alibaba, but lower than NetEase. Tencent's revenue growth rate was lower than Alibaba and NetEase, but higher than Baidu. For an edge on the company's future direction, ask Bigdata to identify strategic shifts mentioned in calls, like mergers, new product launches, or market expansions: > Did the latest Tencent earnings call discuss any international expansion plans? Are there specific markets or regions they are focusing on for growth, and how are they tailoring their strategies for those markets? **response** Based on the information I have, the latest Tencent earnings call did not explicitly discuss international expansion plans in detail. However, the company mentioned the growth of its international market in the Value-added Services segment, specifically in the gaming sector. The release of new games, such as "Dungeon & Fighter Mobile," is expected to contribute to growth in the international market. Earnings calls are often packed with technical terms and industry jargon, which can be confusing. Bigdata can act as your translator, for more clarity: > In the call, they mentioned 'mini-games' contributing to VAS revenue growth. Can you explain the concept of mini-games within the WeChat ecosystem and their role in monetization? **response** Mini-games within the WeChat ecosystem refer to small, simple games that are integrated into the popular messaging app. These games are developed by Tencent and other developers, and they are accessible to WeChat users through mini-programs. Mini-games have become a core strategy for Tencent, contributing significantly to the growth of its Value-added Services (VAS) segment. The revenue from mini-games comes from various sources, including advertising, in-app purchases, and revenue sharing. The integration of mini-games into WeChat provides several benefits for Tencent. First, it helps to increase user engagement and retention within the app. Second, it diversifies Tencent's revenue streams beyond gaming, which has been facing regulatory challenges in China. Finally, it leverages WeChat's massive user base and social network to monetize the games effectively. The revenue potential of mini-games is substantial. According to MoonFox Data, the overall revenue of WeChat mini-games has surpassed 10 billion yuan. As more mid- and large-sized game companies enter the mini-game market and integrate their games into the platform, the growth potential of this business is enormous. ## Getting the bigger picture Let's also ask Bigdata for insights on how regulations might impact Tencent, which is helpful if you need to assess its growth strategies within a dynamic regulatory landscape. > What is the current regulatory landscape in China, specifically concerning data privacy or anti-trust concerns and how it can impact Tencent? **response** China's regulatory landscape regarding data privacy and anti-trust concerns has seen significant developments in recent months. The Chinese government has been implementing new regulations to strengthen data protection and address monopolistic practices. China's Personal Information Protection Law (PIPL) came into effect in November 2021, which sets out rules for the collection, processing, and transfer of personal information. In March 2024, new regulations on cross-border data flows were published, making it easier for companies to transfer personal data outside of China while ensuring data security and privacy. These regulations aim to provide a legal framework for data transfers and protect individuals' privacy rights. China has also been intensifying its efforts to curb risk in the trust sector and regulate internet companies. In 2020, regulators declared victory in cleaning up China's online peer-to-peer lending industry. More recently, authorities have been focusing on preventing monopolistic practices and promoting fair competition. In January 2024, Tencent faced regulatory scrutiny, leading to a temporary ban on new user registrations for one of its blockbuster games. These regulatory developments can impact Tencent in several ways. The company must comply with the new data protection and cross-border data transfer regulations to ensure the security and privacy of its users' personal information. Additionally, Tencent may face increased scrutiny and potential regulatory action if it is found to be engaging in monopolistic practices or violating competition laws. ## Creating a report Finally, the output can be shaped to your specific needs, within seconds, which saves you valuable hours. > Generate an email summarizing the latest Tencent earnings call. Include key points in bullet format. **response** **Subject: Tencent Earnings Call Summary - Strong Q1 Performance and Regulatory Landscape** Dear \[Recipient], I am pleased to provide you with a summary of the latest Tencent earnings call held on \[2024-05-15]. Below are the key points discussed during the call: * Tencent reported a net profit rise of 62% to 41.89 billion yuan (\$5.79 billion) in Q1 2024, driven by the expansion of high-margin businesses such as short videos and mini games embedded in its super app WeChat. * The company's gross profit margin expanded to 53% from 45% a year earlier, supported by these high-margin businesses. Short videos saw a significant increase in total user time and gross receipts of mini games. Online ad revenue also rose 26%. * Tencent is investing in AI technology, platform enhancements, and high production value content to continue cultivating high-quality revenue streams. The company is due to release its highly anticipated game title "Dungeon & Fighter Mobile" on May 21 in China. * Regulatory developments in China, including the implementation of new data protection and cross-border data transfer regulations, and increased scrutiny on monopolistic practices, can impact Tencent. The company must comply with these regulations to ensure the security and privacy of its users' personal information and avoid potential regulatory action. The regulatory landscape is dynamic, and companies operating in China must stay informed and adapt to the evolving regulatory environment. * Tencent's revenue for 2023 was 609.0 billion yuan (\$84.6 billion), up 10% year-on-year. Despite the challenges posed by regulatory crackdowns in recent years, the company's strong Q1 performance and promising outlook have led to a 30% increase in its shares so far this year. Analysts expect positive earnings revisions and multiples' expansion moving forward. Additionally, Tencent is on track to buy back over 100 billion Hong Kong dollars (\$12.8 billion) of its shares this year and pay higher dividends. Please let me know if you have any questions or require further information. Best regards, \[Your Name] # Assessing supply chain disruptions Source: https://docs.bigdata.com/app/use-cases/risk-analysis/assessing_supply_chain_disruptions Our world is deeply connected, so disruptions can quickly cascade through global supply chains, impacting businesses and consumers alike. Bigdata can help assess the early impact of global developments, such as conflicts, pandemics or natural disasters on supply chains. In this example, we use Bigdata to evaluate how **the Red Sea conflict** might impact certain sectors. ## Understanding the event First, we might want to get a better understanding of what the conflict is all about: > What is the Read Sea Conflict? **response** The Red Sea conflict refers to a series of attacks on commercial vessels, primarily those of Israeli origin, in the Red Sea and the Bab el-Mandeb Strait since late 2023. These attacks have disrupted maritime traffic and oil transportation in the region, which is critical for global commerce as it carries approximately 15% of the world's maritime traffic. The Houthis, a paramilitary group in Yemen, have claimed responsibility for these attacks in response to ongoing conflicts, particularly the war in Gaza. The situation has led to geopolitical tensions, with various countries, including Russia and China, having different stances on the issue. ## Identify Impacted Companies Companies heavily reliant on Red Sea shipping routes are likely to face increased costs, delays, and potential inventory shortages. An early identification would allow for swift adjustments to investment strategies or portfolio allocations. Let's ask Bigdata to identify what companies have mentioned the Red Sea conflict in their recent communications. For this query, Bigdata will browse earning calls, press releases and news. > Identify companies that have mentioned the 'Red Sea conflict' in their public communication within the past 3 months. Focus on companies mentioning potential supply chain disruptions caused by the conflict. **response** Based on the information I have, the following companies have mentioned the Red Sea conflict in their earnings calls, press releases, or public statements within the past 3 months and have discussed potential supply chain disruptions caused by the conflict: Hapag-Lloyd, Maersk, Ardmore Shipping Corporation, EQL Pharma. ## Analyze the impact We can further explore the *specific* concerns of these companies, such as increased shipping costs, extended lead times, or rerouted trade routes. > Assess the impact on different supply chain stages (raw materials, finished goods). Highlight specific quotes from company statements that clearly illustrate the disruption's nature and severity. **response** Based on the information I have, the companies identified have expressed concerns about disruptions to their supply chains caused by the Red Sea conflict. Specifically, they have mentioned increased shipping costs, extended lead times, and affected trading patterns due to the diversion of vessels away from the Red Sea. Some companies have also noted that the conflict may last until at least the second half of the year, or even all of 2024. For example, Hapag-Lloyd has reported that the Red Sea crisis has led to delays, higher costs, and more CO2 emissions due to vessels being diverted around the southern tip of Africa. Ardmore Shipping Corporation has noted that many shipping companies have routed their vessels away from the Red Sea, which has affected trading patterns, rates, and expenses. Maersk has reported that the industry is spending 15% to 20% of its Asia-to-Europe vessel capacity on sailing south of Africa to avoid the Red Sea, and that the conflict is expected to last until at least into the second half of the year. EQL Pharma has reported that its shipping routes were affected by ongoing conflicts in the Red Sea, extending lead times by 2-3 weeks and escalating costs. ## Geographic & Industry Trends With the help of Bigdata, we can also identify regions likely to be most affected, and industries heavily reliant on maritime transportation: > Among the companies impacted by the Red Sea conflict, identify any geographic trends. Are there specific regions or industries that seem to be most concerned about supply chain disruptions? **response** The information I have indicates that the disruptions caused by the Red Sea conflict have affected global supply chains and have led to rising costs and severe disruption for various sectors, including production and retail. The Red Sea is located between Africa, Asia, and Europe, and the disruptions have had ripple effects on these regions. For example, some shipping companies have been diverting journeys away from the Red Sea and opting to make much longer, costlier journeys from Asia via South Africa, threatening to marginalize Mediterranean ports including Italy's. Therefore, it appears that Europe, Asia, and the Middle East are regions that are most concerned about supply chain disruptions caused by the Red Sea conflict. ## Potential opportunities While the Red Sea conflict poses significant disruptions, it is also prudent to explore any potential upsides. For instance, Bigdata can help research alternative logistics providers or geopolitical beneficiaries > Can you identify any companies that might potentially benefit from the Red Sea conflict? This could include companies providing alternative resources, logistics solutions, or risk mitigation strategies. **response** Based on the information I have, some companies that might potentially benefit from the Red Sea conflict by offering alternative resources or transportation routes are: * The Israeli software startup, named Descartes Labs, has developed a new logistics solution that allows ships to bypass the Red Sea and travel through the Suez Canal and the Mediterranean Sea instead. This new land route, which is being used by one of the world's biggest shipping lines, Maersk, could provide an alternative transportation option for companies looking to avoid the disruptions caused by the conflict in the Red Sea. * Mediterranean ports: With many shipping companies diverting their vessels away from the Red Sea and opting for longer, costlier journeys, ports in the Mediterranean Sea could potentially see increased business as vessels are rerouted to avoid the Red Sea. Some of the major Mediterranean ports include those in Italy, Spain, Greece, and Egypt. * Companies with ports around the Cape of Good Hope: With the potential for disruptions to shipping in the Red Sea, companies with ports around the Cape of Good Hope, such as South Africa, could potentially benefit from increased business as vessels are rerouted to avoid the Red Sea. This could include ports in Durban, Cape Town, and Richards Bay. # Bigdata developers blog Source: https://docs.bigdata.com/blog/introduction Insights on AI, research, engineering, and the future of financial intelligence. ![Bigdata developers blog](https://mintcdn.com/ravenpackinternational/2v-5oSb0gcuc55pf/images/blog/BD-year_in_review_blog.jpg?fit=max\&auto=format\&n=2v-5oSb0gcuc55pf\&q=85\&s=09d4e897834e89baab29bcdfc027b41c) Welcome to the Bigdata developers blog, where we share what we're building, what we're learning, and where we think the industry is heading. This blog is a space for deep dives into engineering, architecture decisions, research insights, and practical lessons from our work at the intersection of AI, data engineering, and financial intelligence. We write about the tools we use, the systems we design, and the ideas shaping the next generation of data-driven financial technology. Whether you’re a developer integrating our API, a data engineer building pipelines, a researcher exploring financial datasets, or simply curious about how AI is transforming finance, this blog offers a behind-the-scenes look at our work and the challenges we’re solving. *** ## What we write about > **AI Trends** > > How artificial intelligence is reshaping financial research, search, and decision-making, and what comes next. > **Research** > > Deep dives into the data, models, and methodologies behind Bigdata's knowledge graph and content engine. > **Engineering** > > Technical stories from the team: architecture decisions, scaling challenges, and lessons learned in production. > **Product** > > New features, platform updates, and the thinking behind how we build Bigdata products. > **Company** > > Culture, milestones, and the people driving Bigdata forward. > **News** > > Announcements, partnerships, and industry coverage. *** ## Latest posts > **[Building a Global Robotics Watchlist with the Bigdata MCP](/blog/mcp/global-robotics-watchlist)** > > Screen 13 markets for robotics leaders, save them as a watchlist, compare them in one portfolio grid and search every company at once with four Bigdata MCP tools. > **[From investment theme to proprietary signal with NVIDIA Nemotron](/blog/research/theme-to-proprietary-signal)** > > Turn a research theme into a private, backtested dataset with Bigdata.com retrieval and NVIDIA Nemotron teacher-student labeling. > **[Inside Workflows: keeping agent research reproducible](/blog/workflows/inside-workflows)** > > How Workflows keeps financial research reproducible and checkable while the model, not a fixed graph, decides which document to read next. > **[Sentiment in Bigdata.com: The Trend and the Story](/blog/market-data/entity-sentiment-signals)** > > The Entity Sentiment API for trend analysis and backtesting, and the Sentiment Tearsheet for real-time "what's driving this" — live in Claude, ChatGPT, and Copilot. > **[From API key to production AI research assistant - in a single afternoon](/blog/widgets/widgets_release)** > > Introducing the Chat Widget and Document Viewer Widget: one-line embeds that put the Research Agent inside any app, with your brand on it. > **[Smart Batching: plan-then-execute search for universe-scale screening](/blog/search/smart-batching-universe-scale-screening)** > > How adaptive planning cuts large-universe Search workflows from minutes to seconds while preserving high-relevance evidence in benchmark screens. > **[The Workflows API: turning an analyst's research process into an API call](/blog/workflows/automate-your-research-process)** > > Templates, grounded output, and parallel sub-agents so the same research framework runs across your whole coverage universe. > **[Professional-Grade Financial Reports with Claude, Bigdata MCP, and Skills](/blog/mcp/claude_with_bigdata)** > > How financial institutions can use Claude with Bigdata MCP and Skills to produce consistent, scalable research reports on demand. # Sentiment in Bigdata.com: The Trend and the Story Source: https://docs.bigdata.com/blog/market-data/entity-sentiment-signals July 20, 2026 Market Data, MCP, Sentiment / 7 minutes read **TL;DR** * Bigdata.com ships two sentiment tools from RavenPack's research stack: a **daily trend signal** (REST API: backtesting, monitoring, alerts) and a **real-time tearsheet** (zero code, inside Claude, ChatGPT, and Copilot). * They answer different questions, *what's the trend?* versus *what's driving it right now?*: two complementary horizons on the same name, built to be used together. * Every chart and number below comes from live API calls you can reproduce today. **At RavenPack, we use multiple sentiment metrics.** Our EDGE Analytics engine computes a whole family of them, each tuned to a different research question. Bigdata.com adds its own native layer: [chunk-level sentiment](/getting-started/search/chunk_sentiment#sentiment-model), a score on every document chunk from a transformer trained on financial news and corporate event coverage. It measures likely market impact, not linguistic tone, from -1 to +1. This post covers the two tools built on those foundations: a daily trend signal for backtesting and monitoring, and a real-time read on what's moving a name today. Here's the trend lens on Meta over the trailing year: sentiment\_pressure converted into a volatility-scaled band around price: ![Meta price with sentiment-pressure ribbon, trailing 12 months. Sentiment regime flipped negative on Oct 29, one session before an 11% drop](https://mintcdn.com/ravenpackinternational/pVgqio525T81J4yZ/images/blog/entity-sentiment/price_sentiment_ribbon_single.png?fit=max\&auto=format\&n=pVgqio525T81J4yZ\&q=85\&s=a62b5ffda4fdf71821b5ccd3cdfa2077) Green is positive sentiment pressure, red is negative, and the band's width scales with the stock's own volatility. Meta's sentiment\_pressure turned negative on **October 29**, one session *before* an 11% drop, and stayed red through the entire drawdown. One flip doesn't make a strategy, but the signal is daily, point-in-time, and years deep: whether it repeats across your universe is a testable question. ## Two tools, two questions | | **Entity Sentiment API** | **Sentiment Tearsheet** | | - | - | - | | **Question it answers** | "What's the trend over time?" | "What's driving sentiment right now?" | | **Update cadence** | Daily, historical depth | Real-time, 90-day rolling window | | **Data pipeline** | EDGE Analytics: events, relevance, novelty, source ranking | Bigdata.com chunk-level sentiment: news-only, recency-decayed evidence | | **Best for** | Backtesting, portfolio monitoring, programmatic alerts | Morning briefings, "why did this move," analyst research | | **How you use it** | A few lines of Python against a REST endpoint | Ask Claude, ChatGPT, or Copilot in plain language | | **Audience** | Developers, quants, data engineers | Analysts, PMs, anyone who wants an answer without writing code | A trend signal that reacted to every headline would be too noisy to backtest. A real-time read refreshed once a day would miss the story as it develops. Two jobs, two signals, one research stack. ## Tool 1: Entity Sentiment API, the trend The `/v1/entity-sentiment/` endpoint returns three signals per entity per day: **daily\_sentiment** (today's weighted tone), **sentiment\_pressure** (is the 90-day trend unusual for this company), and **abnormal\_media\_attention** (is coverage volume unusual). Reach for it to look back over months or years, or to backtest against price. **Daily sentiment** is the day's weighted average of event sentiment, weighted by `(relevance/100)² × (novelty/90)²`: a highly relevant, novel event counts far more than a marginal or repetitive one. **Sentiment pressure** z-scores the 90-day moving average against the company's own trailing year, normalized to \[-1, 1]. Beyond ±0.9 is statistically unusual. It answers: *is the recent trend better or worse than this company's norm?* **Abnormal media attention** applies the same treatment to coverage *volume* instead of tone: a bad story spreading and a product launch both spike it, for opposite reasons. The full EDGE Analytics pipeline (relevance and novelty weighting, bias correction, subsidiary rollup, z-scoring, normalization) ships as one endpoint. None of it is yours to maintain. ```python theme={null} import httpx import pandas as pd response = httpx.post( "https://api.bigdata.com/v1/entity-sentiment/", headers={"X-API-KEY": "YOUR_API_KEY"}, json={ "identifier": {"type": "rp_entity_id", "value": "4A6F00"}, # Alphabet Inc. "timestamp": {"start": "2025-07-20", "end": "2026-07-20"}, }, ) df = pd.DataFrame(response.json()["results"]["values"]) ``` One year of daily signals in a DataFrame. The tail, as returned today: | date | daily\_sentiment | sentiment\_pressure | abnormal\_media\_attention | | - | -: | -: | -: | | 2026-07-17 | 0.121 | 0.086 | 0.794 | | 2026-07-18 | 0.150 | 0.063 | 0.792 | | 2026-07-19 | 0.218 | 0.071 | 0.777 | `identifier.value` also accepts a list: one call covers your whole watchlist. ## The overlay, across a basket The same ribbon across eight names over the trailing 12 months, each panel annotated with the correlation between sentiment\_pressure and price: ![Price with sentiment-pressure ribbon for an 8-company basket, trailing 12 months, with per-company correlation](https://mintcdn.com/ravenpackinternational/pVgqio525T81J4yZ/images/blog/entity-sentiment/price_sentiment_ribbon_multi.png?fit=max\&auto=format\&n=pVgqio525T81J4yZ\&q=85\&s=46b88d12abcdabc675251ac115de053b) ``` band(t) = price(t) × (1 + scaling(t) × sentiment_pressure(t)) scaling(t) = VOL_MULTIPLIER × ewma_volatility(t, halflife=10 days) ``` Volatility scaling keeps the band proportionate whether a stock is quiet or choppy. Meta tracks tightest (r = +0.81); Netflix, Oracle, and Tesla follow (+0.60, +0.55, +0.55). Oracle's ribbon flips red at its September peak and stays red through the slide; Microsoft sits at +0.23. The relationship is company-specific: measure it per name before you lean on it. Daily history is what makes that possible. ## Continuously monitoring private companies Private companies in the Bigdata.com knowledge graph get the same signals, driven by funding rounds, launches, and regulatory scrutiny instead of earnings cycles. Watch abnormal media attention around SpaceX's IPO: ![SpaceX Abnormal Media Attention, year to date 2026, with the June 12 IPO marked](https://mintcdn.com/ravenpackinternational/pVgqio525T81J4yZ/images/blog/entity-sentiment/spacex_ama_ipo.png?fit=max\&auto=format\&n=pVgqio525T81J4yZ\&q=85\&s=241cf52126383b1f6abe13f69b51b36f) Coverage ran well below SpaceX's norm in January, crossed above normal in April, and hit the abnormal band before the June 12 listing. The build-up preceded the IPO by weeks, and the signal ran straight through it: the entity ID doesn't change when a company lists. Same call as any public name: ```python theme={null} json={"identifier": {"type": "rp_entity_id", "value": "335CF4"}, # SpaceX "timestamp": {"start": "2026-01-01", "end": "2026-07-20"}} ``` ## Twenty years of one company's sentiment One request covers up to 3,650 data points, about 10 years per entity; stitch calls together for more. Apple's price with its sentiment-pressure ribbon, weekly, 2006 through today: ![Apple Inc. price with sentiment-pressure ribbon, weekly, 2006 to 2026, log scale](https://mintcdn.com/ravenpackinternational/pVgqio525T81J4yZ/images/blog/entity-sentiment/apple_20yr_ribbon.png?fit=max\&auto=format\&n=pVgqio525T81J4yZ\&q=85\&s=6393a8b9a6f3b72975b337beee559372) The red troughs line up with the real drawdowns: 2008, the early iPhone-era dips, 2013, 2016, the 2018 selloff, COVID, the 2022 bear market. That depth makes the signal testable rather than anecdotal: backtest across regimes, don't eyeball one good call. ## Tool 2: Sentiment Tearsheet, what's driving it right now The trend API looks back. When you need to know *why* a stock is moving today, you want the news itself, ranked by what's driving the current reading, with a narrative that ties it together. That's the Sentiment Tearsheet. It lives inside Claude, ChatGPT, and Microsoft Copilot through the Bigdata.com MCP connector. No API key, no code: ask "what's driving sentiment on SpaceX right now?" and it answers. Its building block is the [chunk-level sentiment](/getting-started/search/chunk_sentiment#sentiment-model) introduced above, aggregated into a company-level signal. The same scores come back on every Search API result, so anything the tearsheet summarizes you can drill into chunk by chunk. Three design choices keep the signal clean. SEC filings and transcripts are excluded: boilerplate carries systematic sentiment bias, not news. Near-neutral coverage is dropped and strong coverage is weighted by squared magnitude, so the score is decisive. And evidence is ranked by contribution to *today's* reading (relevance, magnitude, recency), not raw relevance. ### SpaceX, called just now We saw SpaceX through the trend lens earlier. Here it is through the real-time lens, today. Everything below is the tool's actual output. Space Exploration Technologies Corp. SPCX · 90-day lookback Sentiment −0.30 · slightly negative momentum −0.13 · strong downward sentiment z (1mo/1qt) −1.9 / −2.2 · unusually low attention −45.8% · z −1.4 / −0.2 SummarySpaceX's post-IPO equity narrative is dominated by near-term Starship execution setbacks and the resulting market repricing. Multiple reports tie the stock slide to the postponed and aborted Flight 13 and engine ignition issues. Coverage also carries a genuine counterweight in AI compute and supply-chain themes, but the current regime is negative and deteriorating: -0.303 against a -0.175 baseline, momentum -0.129, and a quarterly z-score of -2.2, beyond the ±2 threshold the tool flags as statistically unusual. Bullish
  • Foxconn reported to secure AI server manufacturing order from SpaceX (about USD 52B)
  • Quiet build-out of an AI compute rental business: Google, Anthropic, Pentagon deals
  • Dip-buying framing emerging: "down 45% from its high, history suggests..."
Risk / Watchlist
  • Starship Flight 13 aborted after four engines failed to ignite; retry set for Jul 23
  • Stock below its USD 135 IPO price; about USD 1T of market value lost from the post-IPO peak
  • FT: traders raising bets against SpaceX; \~30% of the public float shorted
OutlookThe unresolved tension: can Starship execution recover fast enough to restore the post-IPO valuation trajectory, or does the market keep discounting execution variability? Evidence supports both sides; no specific forward view. Evidence, ranked by contribution to today's reading (relevance x |sentiment| x recency decay)
Headline Source Date Sent Contrib
SpaceX stock wobbles after Starship launch postponement InvestingCube Jul 20 −0.79 −0.53
Weeks after record IPO, SpaceX loses USD 1 trillion in market value Times of India Jul 19 −0.80 −0.47
SpaceX stock is down 45% from its high; history suggests... Nasdaq Jul 20 +0.83 +0.44
Foxconn secures AI server contract manufacturing order from SpaceX Wallstreetcn Jul 20 +0.82 +0.38
Four engines fail to ignite, forcing SpaceX to abort Starship launch MSN Jul 19 −0.70 −0.33
Beyond rockets, SpaceX is quietly renting AI compute Fortune Jul 19 +0.69 +0.32
SpaceX stock falls below its USD 135 IPO price Wealth Professional Jul 20 −0.55 −0.26
Traders are increasingly betting against SpaceX Financial Times Jul 18 −0.61 −0.20
Contrib: contribution to the current reading, computed as relevance x |sentiment| x recency decay. A month-old FT story about the USD 400B value drop scores near zero here; this weekend's launch abort dominates. Read the two horizons together. The trend lens says the 90-day regime is solidly positive: months of pre-IPO enthusiasm are still in that window, and one bad weekend doesn't undo a trend. The real-time lens says the news cycle turned sharply this weekend, and shows you the exact articles. The trend tells you where the name stands; the tearsheet tells you what's moving it now. Before the open, you want both. ### How to try it No setup beyond connecting Bigdata.com to your assistant of choice: * Tool reference: [Sentiment Tearsheet](/mcp-reference/tools/bigdata-sentiment-tearsheet) * Connect it to Claude: [Claude MCP integration guide](/mcp-reference/oauth-integrations/claude-mcp-integration) * Connect it to ChatGPT: [ChatGPT MCP integration guide](/mcp-reference/oauth-integrations/chatgpt-mcp-integration) * Microsoft Copilot and other agentic platforms: [MCP introduction](/mcp-reference/introduction) Once connected, just ask. Developers can call it programmatically (`find_securities`, then `bigdata_sentiment_tearsheet`), but the point is that you don't have to. ## Choosing between them > **Backtest or screen a factor** > > Use the **Entity Sentiment API**. Pull `sentiment_pressure` historically across your universe, join it to price, and test it like any other signal. > **Monitor a book programmatically** > > Use the **Entity Sentiment API**. Daily batch pulls, rank by `sentiment_pressure` change, flag `|sentiment_pressure| > 0.9` for review. > **Why did this stock move today?** > > Use the **Sentiment Tearsheet**, inside Claude, ChatGPT, or Copilot. Ask in plain language, get the narrative and the sources behind it in seconds. > **Morning briefing on a watchlist** > > Use the **Sentiment Tearsheet**. No code, no API key: just ask your assistant what's driving sentiment on each name before the market opens. ## Try it on your own book Everything above came from live API calls. The fastest way to evaluate the signals is against names you actually cover. * **Zero-code start**: [connect the Bigdata.com MCP to Claude or ChatGPT](/mcp-reference/introduction) and ask for a sentiment tearsheet on any name in your portfolio. Two minutes, no key. * **API access**: [request an API key](https://bigdata.com/api?request_access=t) and pull `sentiment_pressure` history for your universe; the [Entity Sentiment API reference](/api-reference/market-data/entity-sentiment) has the full schema. * For bulk historical pulls across large universes, see the [Smart Batching guide](/blog/search/smart-batching-universe-scale-screening). *** This content is for informational purposes only and does not constitute investment advice. *** ![Francisco Gomez](https://mintcdn.com/ravenpackinternational/vbcWCUrYBY4X9p0F/images/blog/authors/francisco_gomez.png?fit=max\&auto=format\&n=vbcWCUrYBY4X9p0F\&q=85\&s=19c067064fbd8d75856e7154e7dbcfd3) Francisco GomezManager of Applied Intelligence (VP) ![Andres Diaz](https://mintcdn.com/ravenpackinternational/pVgqio525T81J4yZ/images/blog/authors/andres_diaz.png?fit=max\&auto=format\&n=pVgqio525T81J4yZ\&q=85\&s=eac471ad6f5bd9cb984f5263b5d1d2f6) Andres DiazSenior Structured Data Manager # How to Automate Portfolio Research with Claude Cowork and Bigdata Source: https://docs.bigdata.com/blog/mcp/claude_cowork_with_bigdata March 18, 2026 Claude Cowork, Bigdata Plugin / 5 minutes read Portfolio Managers can now automate portfolio research with Claude Cowork and the [Bigdata.com Plugin](/mcp-reference/plugins/bigdata-com), a connector that gives Claude direct access to Bigdata's **Search engine** and **Knowledge graph**. A single scheduled task runs every morning, identifies the top market-moving topics, assesses their impact on each holding, and delivers individual PDF briefs with inline attribution. The result: you open your laptop, launch Claude, and by the time you have your coffee the analysis is done. It is 6:45 AM. You open your laptop and launch Claude. That is all it takes. Within seconds, your scheduled Cowork task kicks off. While you make your coffee, check your messages, or get ready for the day, Claude is scanning the last 24 hours of financial news, identifying the five topics most likely to move markets today, cross-referencing each one against every company in your portfolio, scoring the potential impact, and generating a detailed PDF brief for each holding. By the time you sit back down, the analysis is waiting in a folder on your desktop. ## Why does morning research take so long? Every Portfolio Manager knows the routine. Markets open, and the first hour is a scramble: skimming headlines, checking overnight moves, trying to connect scattered data points to the positions that matter. The table below shows where that time typically goes: | Manual task | Time spent daily | Automated equivalent | | :- | :- | :- | | Scanning news for relevant topics | 30-45 min | Claude Cowork searches via [Bigdata search](/getting-started/search/overview) in seconds | | Mapping topics to portfolio holdings | 20-30 min | [Knowledge graph resolution](/mcp-reference/tools/find-securities) matches tickers to entities automatically | | Drafting impact notes per company | 60-90 min | PDF reports generated with inline source attribution | | Updating tracking spreadsheets | 15-20 min | Spreadsheet updated with scores and confidence levels | | **Total** | **\~2-3 hours** | **\~10 minutes (unattended)** | The information is out there. The problem is that assembling it into something actionable takes too long when done manually, every single morning. ## How does Claude Cowork automate portfolio research on a schedule? Claude Cowork lets you define tasks that run automatically when you open the application. Combine that with the [Bigdata.com Plugin](/mcp-reference/plugins/bigdata-com), which bundles the [Bigdata MCP connector](/mcp-reference/introduction) and [financial research analyst Skills](/skills-reference/mcp-helpers/financial-research-analyst), and you get an autonomous research assistant that delivers a structured portfolio impact analysis every morning. Here is what the workflow produces: | Output | Description | | :- | :- | | Top 5 market-moving topics | Identified from the last 24 hours using Bigdata's [search engine](/getting-started/search/overview), which indexes thousands of premium financial sources | | Impact assessment per company | Each topic scored for relevance, direction, and confidence against every holding in your portfolio | | Individual PDF reports | One brief per company with detailed analysis and inline attribution linking back to original documents | | Updated portfolio spreadsheet | A single summary view with impact scores, confidence levels, and topic mappings | ![Scheduled task running](https://mintcdn.com/ravenpackinternational/kJ2v-Vk_7xOuO3c-/images/how-to-guides/search/claude-cowork/scheduled_task_running.png?fit=max\&auto=format\&n=kJ2v-Vk_7xOuO3c-\&q=85\&s=3ccd0bf74a74df5e2e15770ef496c40a) ## What does the setup look like? The entire configuration takes minutes. You provide an Excel file listing the companies in your portfolio, just the ticker and the company name. Claude handles the rest. ![Portfolio template](https://mintcdn.com/ravenpackinternational/kJ2v-Vk_7xOuO3c-/images/how-to-guides/search/claude-cowork/my_portfolio_template.png?fit=max\&auto=format\&n=kJ2v-Vk_7xOuO3c-\&q=85\&s=a6c70e33136fa7cb545df632f9886797) Once the scheduled task runs, it creates a timestamped folder containing a PDF brief for each company and an updated copy of your spreadsheet with impact scores, confidence levels, and topic mappings filled in. ![Portfolio analysed](https://mintcdn.com/ravenpackinternational/kJ2v-Vk_7xOuO3c-/images/how-to-guides/search/claude-cowork/my_portfolio_analysed.png?fit=max\&auto=format\&n=kJ2v-Vk_7xOuO3c-\&q=85\&s=3a6470e5b8f2963e333b2d291fc216cd) Under the hood, Claude uses the [Bigdata knowledge graph](/getting-started/knowledge_graph/introduction) to resolve each ticker to its canonical entity via [find\_securities](/mcp-reference/tools/find-securities), then runs targeted searches through the [Bigdata search service](/getting-started/search/overview) to retrieve the most relevant news for each holding. Every claim in the PDF report links back to its source through inline attribution, so you can verify any insight in seconds. These are some examples of the daily briefs generated by the workflow: **AAPL** [https://mozilla.github.io/pdf.js/web/viewer.html?file=https%3A%2F%2Fraw.githubusercontent.com%2FBigdata-com%2Fbigdata-docs-resources%2Fmain%2Fhow\_to\_guides%2Fclaude-cowork%2Fportfolio\_daily\_impact\_analysis%2FDailyBrief\_AAPL\_2026-03-12.pdf](https://mozilla.github.io/pdf.js/web/viewer.html?file=https%3A%2F%2Fraw.githubusercontent.com%2FBigdata-com%2Fbigdata-docs-resources%2Fmain%2Fhow_to_guides%2Fclaude-cowork%2Fportfolio_daily_impact_analysis%2FDailyBrief_AAPL_2026-03-12.pdf) **NVDA** [https://mozilla.github.io/pdf.js/web/viewer.html?file=https%3A%2F%2Fraw.githubusercontent.com%2FBigdata-com%2Fbigdata-docs-resources%2Fmain%2Fhow\_to\_guides%2Fclaude-cowork%2Fportfolio\_daily\_impact\_analysis%2FDailyBrief\_NVDA\_2026-03-12.pdf](https://mozilla.github.io/pdf.js/web/viewer.html?file=https%3A%2F%2Fraw.githubusercontent.com%2FBigdata-com%2Fbigdata-docs-resources%2Fmain%2Fhow_to_guides%2Fclaude-cowork%2Fportfolio_daily_impact_analysis%2FDailyBrief_NVDA_2026-03-12.pdf) **UTHR** [https://mozilla.github.io/pdf.js/web/viewer.html?file=https%3A%2F%2Fraw.githubusercontent.com%2FBigdata-com%2Fbigdata-docs-resources%2Fmain%2Fhow\_to\_guides%2Fclaude-cowork%2Fportfolio\_daily\_impact\_analysis%2FDailyBrief\_UTHR\_2026-03-12.pdf](https://mozilla.github.io/pdf.js/web/viewer.html?file=https%3A%2F%2Fraw.githubusercontent.com%2FBigdata-com%2Fbigdata-docs-resources%2Fmain%2Fhow_to_guides%2Fclaude-cowork%2Fportfolio_daily_impact_analysis%2FDailyBrief_UTHR_2026-03-12.pdf) **CRWV** [https://mozilla.github.io/pdf.js/web/viewer.html?file=https%3A%2F%2Fraw.githubusercontent.com%2FBigdata-com%2Fbigdata-docs-resources%2Fmain%2Fhow\_to\_guides%2Fclaude-cowork%2Fportfolio\_daily\_impact\_analysis%2FDailyBrief\_CRWV\_2026-03-12.pdf](https://mozilla.github.io/pdf.js/web/viewer.html?file=https%3A%2F%2Fraw.githubusercontent.com%2FBigdata-com%2Fbigdata-docs-resources%2Fmain%2Fhow_to_guides%2Fclaude-cowork%2Fportfolio_daily_impact_analysis%2FDailyBrief_CRWV_2026-03-12.pdf) **AMD** [https://mozilla.github.io/pdf.js/web/viewer.html?file=https%3A%2F%2Fraw.githubusercontent.com%2FBigdata-com%2Fbigdata-docs-resources%2Fmain%2Fhow\_to\_guides%2Fclaude-cowork%2Fportfolio\_daily\_impact\_analysis%2FDailyBrief\_AMD\_2026-03-12.pdf](https://mozilla.github.io/pdf.js/web/viewer.html?file=https%3A%2F%2Fraw.githubusercontent.com%2FBigdata-com%2Fbigdata-docs-resources%2Fmain%2Fhow_to_guides%2Fclaude-cowork%2Fportfolio_daily_impact_analysis%2FDailyBrief_AMD_2026-03-12.pdf) **MU** [https://mozilla.github.io/pdf.js/web/viewer.html?file=https%3A%2F%2Fraw.githubusercontent.com%2FBigdata-com%2Fbigdata-docs-resources%2Fmain%2Fhow\_to\_guides%2Fclaude-cowork%2Fportfolio\_daily\_impact\_analysis%2FDailyBrief\_MU\_2026-03-12.pdf](https://mozilla.github.io/pdf.js/web/viewer.html?file=https%3A%2F%2Fraw.githubusercontent.com%2FBigdata-com%2Fbigdata-docs-resources%2Fmain%2Fhow_to_guides%2Fclaude-cowork%2Fportfolio_daily_impact_analysis%2FDailyBrief_MU_2026-03-12.pdf) **AIR.PA** [https://mozilla.github.io/pdf.js/web/viewer.html?file=https%3A%2F%2Fraw.githubusercontent.com%2FBigdata-com%2Fbigdata-docs-resources%2Fmain%2Fhow\_to\_guides%2Fclaude-cowork%2Fportfolio\_daily\_impact\_analysis%2FDailyBrief\_AIR.PA\_2026-03-12.pdf](https://mozilla.github.io/pdf.js/web/viewer.html?file=https%3A%2F%2Fraw.githubusercontent.com%2FBigdata-com%2Fbigdata-docs-resources%2Fmain%2Fhow_to_guides%2Fclaude-cowork%2Fportfolio_daily_impact_analysis%2FDailyBrief_AIR.PA_2026-03-12.pdf) **ITX.MC** [https://mozilla.github.io/pdf.js/web/viewer.html?file=https%3A%2F%2Fraw.githubusercontent.com%2FBigdata-com%2Fbigdata-docs-resources%2Fmain%2Fhow\_to\_guides%2Fclaude-cowork%2Fportfolio\_daily\_impact\_analysis%2FDailyBrief\_ITX.MC\_2026-03-12.pdf](https://mozilla.github.io/pdf.js/web/viewer.html?file=https%3A%2F%2Fraw.githubusercontent.com%2FBigdata-com%2Fbigdata-docs-resources%2Fmain%2Fhow_to_guides%2Fclaude-cowork%2Fportfolio_daily_impact_analysis%2FDailyBrief_ITX.MC_2026-03-12.pdf) **GOOGL** [https://mozilla.github.io/pdf.js/web/viewer.html?file=https%3A%2F%2Fraw.githubusercontent.com%2FBigdata-com%2Fbigdata-docs-resources%2Fmain%2Fhow\_to\_guides%2Fclaude-cowork%2Fportfolio\_daily\_impact\_analysis%2FDailyBrief\_GOOGL\_2026-03-12.pdf](https://mozilla.github.io/pdf.js/web/viewer.html?file=https%3A%2F%2Fraw.githubusercontent.com%2FBigdata-com%2Fbigdata-docs-resources%2Fmain%2Fhow_to_guides%2Fclaude-cowork%2Fportfolio_daily_impact_analysis%2FDailyBrief_GOOGL_2026-03-12.pdf) **MSFT** [https://mozilla.github.io/pdf.js/web/viewer.html?file=https%3A%2F%2Fraw.githubusercontent.com%2FBigdata-com%2Fbigdata-docs-resources%2Fmain%2Fhow\_to\_guides%2Fclaude-cowork%2Fportfolio\_daily\_impact\_analysis%2FDailyBrief\_MSFT\_2026-03-12.pdf](https://mozilla.github.io/pdf.js/web/viewer.html?file=https%3A%2F%2Fraw.githubusercontent.com%2FBigdata-com%2Fbigdata-docs-resources%2Fmain%2Fhow_to_guides%2Fclaude-cowork%2Fportfolio_daily_impact_analysis%2FDailyBrief_MSFT_2026-03-12.pdf) ## How does this change the Portfolio Manager's morning? The goal is not to replace the Portfolio Manager. It is to eliminate the 2-3 hours of repetitive assembly work so you can focus on what drives returns: interpreting the data, forming a view, and making decisions. With this workflow, your morning changes. Instead of scrambling to catch up, you start with a clear picture of what matters for your book today. You can go deeper on the topics that need attention, ignore the noise, and move faster. The [Bigdata search service](/getting-started/search/overview) ensures coverage across thousands of sources, and the [knowledge graph](/getting-started/knowledge_graph/introduction) ensures every company is correctly identified regardless of ticker format or exchange. ## How to get started with Claude Cowork and Bigdata We have published a complete step-by-step guide that walks you through the entire setup: preparing your portfolio spreadsheet, enabling the [Bigdata.com Plugin](/mcp-reference/plugins/bigdata-com) in your Claude environment, configuring the scheduled task, and reviewing the output. > **[Portfolio daily impact brief](/how-to-guides/search/claude-cowork/pm-top-topics-impact)** > > Follow the how-to guide to set up your own automated portfolio research workflow with Claude Cowork and Bigdata. ## What other portfolio workflows can you automate with Claude Cowork? This is one workflow, but the pattern applies to many others. Any repetitive research task that follows a predictable structure is a candidate for automation with Claude Cowork and the [Bigdata.com Plugin](/mcp-reference/plugins/bigdata-com): | Workflow | What it does | | :- | :- | | Earnings season preparation | Generate pre-earnings briefs for every company reporting next week | | Sector rotation monitoring | Track topic momentum shifts across sectors and flag rotation signals | | Regulatory change tracking | Monitor new regulatory developments and assess exposure per holding | | Credit risk screening | Identify negative news signals and map them to credit-sensitive positions | | Daily digest reports | Produce automated [daily digests](/use-cases/research-tools/daily-digests-introduction) covering specific themes or markets | If you have a workflow in mind that you would like to explore, reach out to us at **[support@bigdata.com](mailto:support@bigdata.com)**. We are happy to help you design it and get it running. *** ![Oscar Sanchez Iglesias](https://mintcdn.com/ravenpackinternational/Sf9MtziD0iLKyb1C/images/blog/authors/oscar_sanchez_iglesias.png?fit=max\&auto=format\&n=Sf9MtziD0iLKyb1C\&q=85\&s=c9ee6739dce63b6d6b6f2e5aa36e8442) Oscar Sanchez IglesiasSenior Product Manager # Professional-Grade Financial Reports with Claude, Bigdata MCP, and Skills Source: https://docs.bigdata.com/blog/mcp/claude_with_bigdata March 4, 2026 Claude, MCP, Skill / 5 minutes read Every quarter, the same pressure hits research desks across the financial industry. Earnings season arrives, markets shift, and teams scramble to produce reports that are accurate, well-structured, and delivered on time. Analysts juggle fragmented data sources, inconsistent templates, and tight deadlines. The result? Uneven quality, missed coverage, and reports that take hours when they should take minutes. What if you could produce professional-grade financial reports with consistent depth, consistent format, and at scale? That is now possible with **Claude**, **Bigdata MCP**, and **Bigdata Skills**. ## The problem with traditional research workflows Financial institutions invest heavily in research. But the process of producing a single report still involves too many manual steps: pulling data from multiple platforms, cross-referencing figures, formatting results into a consistent template, and reviewing for accuracy. Multiply that across dozens of companies, sectors, or regions, and you start to see why coverage gaps appear. Junior analysts burn time on formatting instead of analysis. Senior analysts spend their day reviewing inconsistent outputs instead of making decisions. The bottleneck is not intelligence. It is workflow. ## A new approach: Claude + Bigdata MCP and Skills We have built a system that brings together three components to solve this problem end to end. ### 1. Bigdata MCP Connector for Claude The [Bigdata.com Official Connector](/mcp-reference/oauth-integrations/claude-mcp-integration) gives Claude direct access to Bigdata's premium financial data, knowledge graph, and research engine. Once enabled, Claude can query real-time company data, financial metrics, analyst estimates, market news, and more, all without leaving the conversation. Setting it up takes minutes. Your team admin enables the connector in Claude's Organization settings, and each user connects with their own Bigdata.com credentials. From there, every Claude conversation has Bigdata's full data layer at its fingertips. ### 2. Bigdata Plugin The Bigdata Plugin bundle [financial research analyst skills](/skills-reference/mcp-helpers/financial-research-analyst) to guide Claude through a repeatable research-and-writing process. Instead of getting a generic AI response, you get a report that follows a defined methodology: sourcing data from Bigdata, validating it, analyzing it in context, and writing it in a standardized format. Think of it as giving Claude the playbook your best analyst follows, and then letting it execute that playbook at scale. ### 3. The reports you can generate With this setup, your team can produce a wide range of professional research outputs on demand: **Public Company Analysis** | Report Type | What you get | | :- | :- | | Company Brief | A comprehensive overview covering business model, financials, competitive positioning, and market context | | Earnings Preview | Pre-earnings preparation with analyst estimates, historical performance trends, and key metrics to watch | | Earnings Digest | A structured summary of earnings calls and quarterly results turned into actionable insights | | Risk Assessment | A data-driven evaluation of company-specific and market risks, with exposure analysis | **Macro and Sector Analysis** | Report Type | What you get | | :- | :- | | Sector Analysis | Performance, valuations, sub-industry breakdowns, themes, and upcoming catalysts | | Country Economic Profile | GDP, inflation, monetary policy, labor market dynamics, and investment implications | | Country-Sector Analysis | Macro backdrop combined with sector-specific analysis for a given market (e.g., "US Technology") | | Cross-Sector Comparison | Relative valuations, earnings growth, cycle positioning, and rotation signals | | Thematic Research | Deep analysis of macro themes like AI, energy transition, and interest rate regimes | | Regional Comparison | G7/G20 economic comparison with currency, cross-asset views, and allocation signals | ## Why this changes the game ### Consistency at scale Every report follows the same structure, the same depth of analysis, and the same sourcing methodology. Whether you are covering one company or fifty, the quality does not degrade. ### Speed without shortcuts A company brief that took an analyst two hours can now be generated in minutes. An earnings preview that required pulling data from five different platforms now comes from a single conversation. The time savings compound across the entire research team. ### Customizable to your standards The Bigdata plugin is open source. You can [fork the repository](https://github.com/Bigdata-com/bigdata-plugins-marketplace) and customize the workflows, output format, and analysis structure to match your firm's standards. Your branding, your templates, your methodology, powered by Bigdata's data engine. ### Always current Because the Bigdata MCP connector pulls live data, your reports reflect the latest available information. No stale spreadsheets, no outdated snapshots. ## Getting started Setting up the full stack takes less than ten minutes: 1. **Enable the Bigdata MCP connector** in your Claude organization. Follow the [Claude MCP Integration guide](/mcp-reference/oauth-integrations/claude-mcp-integration) to add the official connector and configure permissions. 2. **Install the Bigdata plugin**, which bundles financial research analyst skills. Follow the [installation guides](/skills-reference/install-bigdata-plugin) for your platform. The plugin source is public at [bigdata-plugins-marketplace](https://github.com/Bigdata-com/bigdata-plugins-marketplace). 3. **Start generating reports.** Open a Claude conversation and try: **Company Brief** Prompt: ``` Create a company brief doc for Netflix using Bigdata skills and adding inline attribution ``` Output: [https://mozilla.github.io/pdf.js/web/viewer.html?file=https%3A%2F%2Fraw.githubusercontent.com%2FBigdata-com%2Fbigdata-docs-resources%2Fmain%2Fmcp%2Freport-examples%2FNetflix\_Company\_Brief.pdf](https://mozilla.github.io/pdf.js/web/viewer.html?file=https%3A%2F%2Fraw.githubusercontent.com%2FBigdata-com%2Fbigdata-docs-resources%2Fmain%2Fmcp%2Freport-examples%2FNetflix_Company_Brief.pdf) **Earnings Preview** Prompt: ``` Generate an earnings preview doc for lululemon's upcoming quarterly results using Bigdata skills and adding inline attribution ``` Output: [https://mozilla.github.io/pdf.js/web/viewer.html?file=https%3A%2F%2Fraw.githubusercontent.com%2FBigdata-com%2Fbigdata-docs-resources%2Fmain%2Fmcp%2Freport-examples%2FLululemon\_Earnings\_Preview.pdf](https://mozilla.github.io/pdf.js/web/viewer.html?file=https%3A%2F%2Fraw.githubusercontent.com%2FBigdata-com%2Fbigdata-docs-resources%2Fmain%2Fmcp%2Freport-examples%2FLululemon_Earnings_Preview.pdf) **Sector Analysis** Prompt: ``` Create a document that analyzes the US semiconductor sector using Bigdata skills and adding inline attribution ``` Output: [https://mozilla.github.io/pdf.js/web/viewer.html?file=https%3A%2F%2Fraw.githubusercontent.com%2FBigdata-com%2Fbigdata-docs-resources%2Fmain%2Fmcp%2Freport-examples%2FUS\_Semiconductor\_Sector\_Analysis.pdf](https://mozilla.github.io/pdf.js/web/viewer.html?file=https%3A%2F%2Fraw.githubusercontent.com%2FBigdata-com%2Fbigdata-docs-resources%2Fmain%2Fmcp%2Freport-examples%2FUS_Semiconductor_Sector_Analysis.pdf) **G7 Economic Outlook** Prompt: ``` Create a document that compares G7 economic outlooks using Bigdata skills and adding inline attribution ``` Output: [https://mozilla.github.io/pdf.js/web/viewer.html?file=https%3A%2F%2Fraw.githubusercontent.com%2FBigdata-com%2Fbigdata-docs-resources%2Fmain%2Fmcp%2Freport-examples%2FG7\_Economic\_Outlook\_2026\_03\_04.pdf](https://mozilla.github.io/pdf.js/web/viewer.html?file=https%3A%2F%2Fraw.githubusercontent.com%2FBigdata-com%2Fbigdata-docs-resources%2Fmain%2Fmcp%2Freport-examples%2FG7_Economic_Outlook_2026_03_04.pdf) That is it. No infrastructure to deploy, no pipelines to build, no templates to maintain. Just Claude, Bigdata, and a skill that knows how to turn data into decisions. ## What comes next This is only the beginning. We are actively expanding the library of skills and workflows available through Bigdata MCP. If your team has a research workflow that could benefit from this approach, email us to [support@bigdata.com](mailto:support@bigdata.com). We would love to hear what you are building. *** ![Oscar Sanchez Iglesias](https://mintcdn.com/ravenpackinternational/Sf9MtziD0iLKyb1C/images/blog/authors/oscar_sanchez_iglesias.png?fit=max\&auto=format\&n=Sf9MtziD0iLKyb1C\&q=85\&s=c9ee6739dce63b6d6b6f2e5aa36e8442) Oscar Sanchez IglesiasSenior Product Manager # From Screen to Signal: Building a Global Robotics Watchlist with the Bigdata MCP Source: https://docs.bigdata.com/blog/mcp/global-robotics-watchlist Screen 13 markets for robotics leaders, save them as a watchlist, compare them in one portfolio grid and search every company at once, using four Bigdata MCP tools. October 21, 2026 MCP, Watchlists, Thematic Research / 6 minutes read Robotics is no longer one industry. It runs from warehouse shuttles and factory arms to surgical robots, self-driving trucks and humanoids, and the companies behind it are listed in New York, Tokyo, Seoul, Shenzhen and across Europe. Building a coverage list for that theme usually means a screener, a spreadsheet and a lot of copy and paste. This post walks through the same job with four tools from the [Bigdata MCP](/mcp-reference/introduction). We go from a cross-market screen to a saved watchlist, a portfolio grid with charts, and one search that covers all 49 companies at once. Then we go deeper on a single name.
Step Tool What it does
1 [`bigdata_screen_companies`](/mcp-reference/tools/bigdata-screen-companies) Finds listed companies by country, industry and market cap
2 [`bigdata_create_watchlist`](/mcp-reference/tools/bigdata-create-watchlist) Saves the selected companies as a reusable watchlist
3 [`bigdata_portfolio_tearsheet`](/mcp-reference/tools/bigdata-portfolio-tearsheet) Returns price, EPS, analyst targets and market cap for every company in one grid
4 `bigdata_watchlist_search` Runs one query across the whole watchlist, with results grouped by company
+ [`bigdata_search`](/mcp-reference/tools/bigdata-search) Goes deeper on a single company
## Step 1: Screen 13 markets for robotics companies The theme covers five areas: **warehouse**, **factory**, **medical**, **self-driving** and **humanoid** robots, plus the chip and software **platforms** they run on. Aerospace and defense are out of scope. There is no "Robotics" industry in the screener, so we map the theme to eight industries: | Industry | Robotics area | | :- | :- | | Industrial - Machinery | Warehouse, factory | | Electrical Equipment & Parts | Warehouse, factory | | Medical - Instruments & Supplies | Medical | | Medical - Devices | Medical | | Auto - Parts | Self-driving | | Auto - Manufacturers | Self-driving, humanoid | | Consumer Electronics | Humanoid | | Semiconductors | Platforms | We screen 13 countries: the US, Germany, France, the Netherlands, Spain, the UK, Switzerland, Sweden, Israel, Japan, South Korea, China and Hong Kong. Market cap is filtered in each listing's own currency, so the threshold of about \$5 billion is set per currency: USD 5bn, EUR 4.3bn, GBP 3.7bn, CHF 4bn, SEK 47bn, JPY 750bn, KRW 7tn, CNY 36bn, HKD 39bn and ILS 18bn. Copy this prompt into your agent: ```text theme={null} Using Bigdata, screen listed companies with a market cap above about $5 billion, converting the threshold to each listing's local currency, in the US, Germany, France, the Netherlands, Spain, the UK, Switzerland, Sweden, Israel, Japan, South Korea, China and Hong Kong. Use these industries: Industrial - Machinery, Electrical Equipment & Parts, Medical - Instruments & Supplies, Medical - Devices, Auto - Parts, Auto - Manufacturers, Consumer Electronics and Semiconductors. Include only actively trading companies, with no ETFs or funds. Then keep only the companies with clear robotics exposure in warehouse, factory, medical, self-driving or humanoid robots, or the chip and software platforms behind them. Leave out aerospace and defense. Keep fewer than 60 companies and tag each company with its area. ``` The agent runs [`bigdata_screen_companies`](/mcp-reference/tools/bigdata-screen-companies) once per country and industry. Across all of them the screen returns 391 companies. Most of them build pumps, HVAC systems or memory chips, so we keep the 49 with clear robotics exposure: | Area | Companies | | :- | :- | | Warehouse | Symbotic, Daifuku, KION (Dematic) | | Factory | Teradyne (Universal Robots, MiR), Rockwell Automation, Siemens, Schneider Electric, ABB, Fanuc, Yaskawa, Omron, Nidec, SMC, Mitsubishi Electric, Denso, HD Hyundai, RoboTechnik, Inovance | | Medical | Intuitive Surgical, Stryker (Mako), Zimmer Biomet (ROSA), Globus Medical (ExcelsiusGPS), Smith & Nephew (CORI) | | Self-driving | Mobileye, Aptiv, Continental, Hyundai Mobis, AB Volvo, Daimler Truck, Mercedes-Benz, Desay SV, Toyota | | Humanoid | Tesla (Optimus), XPeng (Iron), Hyundai Motor (Boston Dynamics), Rainbow Robotics, Samsung Electronics, LG Electronics, Xiaomi, Unitree, Leader Harmonious Drive, Sanhua, Ningbo Tuopu | | Platforms | NVIDIA, Qualcomm, Arm, NXP, Renesas, Infineon | > [!TIP] > > The country filter matches where a company is based, not where it is listed. Each result also carries the company's Bigdata entity ID, which the agent uses in the next steps. ## Step 2: Save the companies as a watchlist In the same conversation, ask the agent to save the list: ```text theme={null} Create a Bigdata watchlist called "Global Robotics" with these companies. Description: "Large-cap robotics companies across warehouse, factory, medical, self-driving, humanoid and platform segments in the US, Europe, Israel, Japan, Korea and China." ``` The agent calls [`bigdata_create_watchlist`](/mcp-reference/tools/bigdata-create-watchlist) and confirms the new watchlist with its ID: ![Agent reply confirming that the Global Robotics watchlist was created with 49 companies](https://mintcdn.com/ravenpackinternational/x4qAIX1kx_M9_rRB/images/blog/global-robotics-watchlist/watchlist-created.png?fit=max\&auto=format\&n=x4qAIX1kx_M9_rRB\&q=85\&s=856ab4c61c6eeb44dfcde0d7d1352122) The watchlist also appears in the [Bigdata app under Watchlists](https://app.bigdata.com/watchlists), with live prices and daily, weekly and monthly moves for all 49 companies: ![The Global Robotics watchlist in the Bigdata app, showing Stryker, Globus Medical, Intuitive Surgical and other companies with their latest price and percentage changes](https://mintcdn.com/ravenpackinternational/x4qAIX1kx_M9_rRB/images/blog/global-robotics-watchlist/global-robotics-watchlist.png?fit=max\&auto=format\&n=x4qAIX1kx_M9_rRB\&q=85\&s=e11a81ea61e73f51f58476639c7597e2) ## Step 3: Compare the companies in one grid Ask for a portfolio view and the two charts: ```text theme={null} Show the portfolio tearsheet for my Global Robotics watchlist with price, EPS, EPS estimate, EPS surprise, price target and market cap. Then chart the consensus EPS estimate, with one chart per reporting currency, and chart how far the consensus price target is above the current price. ``` The agent calls [`bigdata_portfolio_tearsheet`](/mcp-reference/tools/bigdata-portfolio-tearsheet), which returns one row per company with the metrics you ask for. ### Consensus EPS for the last reported quarter EPS is reported in each company's own currency, so we compare companies within one currency at a time. **USD** ![Consensus EPS in USD for 16 robotics companies, from NXP at \$3.52 to Mobileye at \$0.06](https://mintcdn.com/ravenpackinternational/x4qAIX1kx_M9_rRB/images/blog/robotics-watchlist/consensus_eps_usd.svg?fit=max\&auto=format\&n=x4qAIX1kx_M9_rRB\&q=85\&s=bad6fa86ddf9be576e75a09238ab28b1) **JPY** ![Consensus EPS in JPY for 9 Japanese robotics companies, led by SMC at ¥666](https://mintcdn.com/ravenpackinternational/x4qAIX1kx_M9_rRB/images/blog/robotics-watchlist/consensus_eps_jpy.svg?fit=max\&auto=format\&n=x4qAIX1kx_M9_rRB\&q=85\&s=2b6cc970214be4d8fe6eac6e5ec06509) **KRW** ![Consensus EPS in KRW for 5 Korean companies, led by Hyundai Mobis](https://mintcdn.com/ravenpackinternational/x4qAIX1kx_M9_rRB/images/blog/robotics-watchlist/consensus_eps_krw.svg?fit=max\&auto=format\&n=x4qAIX1kx_M9_rRB\&q=85\&s=ace3c6f08347b281c8f03a004b56d86a) **CNY** ![Consensus EPS in CNY for 8 Chinese companies, with XPeng expected to report a loss](https://mintcdn.com/ravenpackinternational/x4qAIX1kx_M9_rRB/images/blog/robotics-watchlist/consensus_eps_cny.svg?fit=max\&auto=format\&n=x4qAIX1kx_M9_rRB\&q=85\&s=b9e2073311f419320261a0ef68316b55) **EUR** ![Consensus EPS in EUR for 6 European companies, led by Siemens at €2.55](https://mintcdn.com/ravenpackinternational/x4qAIX1kx_M9_rRB/images/blog/robotics-watchlist/consensus_eps_eur.svg?fit=max\&auto=format\&n=x4qAIX1kx_M9_rRB\&q=85\&s=624bda63777af1733783f016c35d28f9) **SEK** ![Consensus EPS in SEK for AB Volvo at 5.22 kronor](https://mintcdn.com/ravenpackinternational/x4qAIX1kx_M9_rRB/images/blog/robotics-watchlist/consensus_eps_sek.svg?fit=max\&auto=format\&n=x4qAIX1kx_M9_rRB\&q=85\&s=17b5e0a3eb70ed0db415ae4356291845) ### Consensus price target compared with today's price For the companies with analyst price targets, the gap between the consensus target and the current price shows where analysts see the most room to rise. Arm, NVIDIA and Mobileye lead the list. ![Upside from current price to consensus price target, from Arm at +65% to Zimmer Biomet at +9%](https://mintcdn.com/ravenpackinternational/x4qAIX1kx_M9_rRB/images/blog/robotics-watchlist/price_target_upside.svg?fit=max\&auto=format\&n=x4qAIX1kx_M9_rRB\&q=85\&s=580644f6f507243f0c5756300da57837) ## Step 4: Search every company at once `bigdata_watchlist_search` runs one query across the whole list and returns results grouped by company. Every company gets its own share of the results, so the most covered names cannot crowd out the rest. Ask one question per prompt. The first covers patents: ```text theme={null} Search my Global Robotics watchlist for patents each company has filed or been granted for robotics technology. Return 3 results per company, using only content originally published in English. ``` The second covers products: ```text theme={null} Search my Global Robotics watchlist for products each company sells that solve a specific customer problem. Return 3 results per company, using only content originally published in English. ``` Both searches return results for 45 of the 49 companies. A few highlights: **Patents** | Company | What the search found | | :- | :- | | Tesla | Filing for a printed tactile "skin" for robot fingertips, palms and grippers ([Robotics and Automation News](https://app.bigdata.com/documents/EDAE0A1D7FE14EDE90BA3FDDC01CAD0C?cnum=2)) | | Samsung Electronics | US patents for a humanoid hip joint (2024) and a vision-guided robot hand (2026) ([Tech Times](https://app.bigdata.com/documents/14DE26FBD07E79B1307B00F207702D29?cnum=7)) | | Intuitive Surgical | Key da Vinci patents expired in 2017; the annual report describes how it protects its patent portfolio ([Quartr](https://app.bigdata.com/documents/BED070314C97EA8155BBEE1703D547C4?cnum=128)) | | Hyundai Mobis | 1,015 future-mobility patent applications in 2022, including robotics ([Sustainability report](https://app.bigdata.com/documents/B47266222BC4A27D61CCB41D70B79D10?cnum=174)) | **Products** | Company | Product and the problem it solves | | :- | :- | | FANUC | Food-grade robots for packaging lines and mobile robots for warehouse fulfillment ([Benzinga](https://app.bigdata.com/documents/029F3B7D173CF7059BBE1573C7B890AC?cnum=4)) | | KION | Autonomous truck with AI human detection for loading trailers in busy warehouses ([PubT](https://app.bigdata.com/documents/228F41CC8D34F2C7B50F58ABBD92BA11?cnum=1)) | | Symbotic | Handling for frozen and perishable goods, built at Walmart's request ([Quartr](https://app.bigdata.com/documents/915E1757F4743067E3E19AA6EDA3D67E?cnum=50\&cnum=52)) | | Omron | Sensors, controllers and robots that resolve issues on factory floors ([PubT](https://app.bigdata.com/documents/3050C94584FD4AD55302DC346A66BB3A?cnum=98)) | | Intuitive Surgical | Repeatable, easy-to-learn surgical systems for high-volume ambulatory surgery centers ([Quartr](https://app.bigdata.com/documents/375D84CE4479C4DE6062A697ACB2425B?cnum=44)) | ## Going deeper: Symbotic The watchlist search gives a line per company. When one name stands out, [`bigdata_search`](/mcp-reference/tools/bigdata-search) goes deeper into its filings, transcripts and news. Again, one question per prompt: ```text theme={null} Using Bigdata search, find Symbotic's robotics products and the customer problem they solve. ``` ```text theme={null} Using Bigdata search, find the patents Symbotic has filed or been granted for robotics technology. ``` The two searches give a clear picture: * **The problem.** Manual warehouse work is becoming unsustainable as labor shifts, wages rise and e-commerce adds complexity, and older automation cannot keep up ([10-Q](https://app.bigdata.com/documents/5D01BE6DCA0352725C56F964633027B1?cnum=112)). * **The product.** An AI-powered robotic system that moves, stores and sorts cases and individual items in high-density warehouses for Walmart, C\&S Wholesale Grocers and Exol, its warehouse-as-a-service joint venture ([Q3 FY2026 report](https://app.bigdata.com/documents/216D66BA1BCFD153FCCE6889A0551918?cnum=138)). * **What's new.** The acquisition of cellumation adds the celluveyor, a surface of independently controlled cells that sorts products without conveyor belts, for more throughput in less space ([press release](https://app.bigdata.com/documents/C738BEFBEFA77CCAA73DB103B61390CB?cnum=4\&cnum=1)). * **The patents.** About 1,100 issued or pending patents, more than 650 of them issued, backed by more than \$1 billion of cumulative R\&D ([10-Q](https://app.bigdata.com/documents/5D01BE6DCA0352725C56F964633027B1?cnum=112), [8-K](https://app.bigdata.com/documents/54489C0B57A329C39E7B22E1E270C2BF?cnum=45)). * **The scale.** A backlog of about \$22.5 billion and Q3 FY2026 revenue up 21.7% year on year to \$720.8 million ([Nasdaq](https://app.bigdata.com/documents/B2788AF72D2D8D9D63CD87A096414FE8?cnum=7)). ## Try it yourself The whole workflow runs from any MCP client connected to Bigdata, such as Claude. Ask it to screen, save, compare and search in plain language, and it chains the tools for you. > **[Connect the Bigdata MCP](/mcp-reference/introduction)** > > Set up the Bigdata MCP in your AI client. > **[Screen companies](/mcp-reference/tools/bigdata-screen-companies)** > > See every filter the screener accepts. *** ![Oscar Sanchez Iglesias](https://mintcdn.com/ravenpackinternational/Sf9MtziD0iLKyb1C/images/blog/authors/oscar_sanchez_iglesias.png?fit=max\&auto=format\&n=Sf9MtziD0iLKyb1C\&q=85\&s=c9ee6739dce63b6d6b6f2e5aa36e8442) Oscar Sanchez IglesiasSenior Product Manager # Quantum Computing Is Hitting Its Commercial Inflection Point Source: https://docs.bigdata.com/blog/mcp/quantum_thematic_report May 18, 2026 Claude Cowork, Bigdata Plugin, Thematic Research / 4 minutes read Every once in a while a technology stops being a science project and starts behaving like a market. AI did it five years ago. **Quantum computing is doing it right now**, and the numbers are starting to look impossible to ignore. The global quantum computing market, valued at approximately **\$17.4 billion in 2025** for photonic integrated circuits alone \[1], is projected to expand at a **20.8% CAGR through 2034**, reaching **\$86.4 billion** \[1]. Boston Consulting Group puts the broader quantum technology market at up to **\$97 billion in revenue by 2035**, with quantum computing capturing **\$72 billion** of that total \[2]. McKinsey is even more aggressive: a total addressable market of **\$850 billion over the coming decade** \[3]. That is the shape of an inflection point. It is the shape AI had in 2020 — quietly trending in research papers, then suddenly showing up in every board deck. I wanted a sharp, sourced view of where the money, the technology, and the players are converging. So I built one. ## The full report I generated a complete **Quantum Computing & Quantum Chip Manufacturing** thematic research report using [Claude Cowork](https://www.anthropic.com/claude) and the official [Bigdata.com Plugin](https://claude.ai/directory/plugins/56cc07da-4c8b-401b-bac9-9a49bc259c07%40anthropic-plugin-directory). Total time, from idea to PDF: **under 5 minutes.** [https://mozilla.github.io/pdf.js/web/viewer.html?file=https%3A%2F%2Fraw.githubusercontent.com%2FBigdata-com%2Fbigdata-docs-resources%2Fmain%2Fhow\_to\_guides%2Fclaude-cowork%2Fthematic\_report%2FQuantum\_Computing\_Thematic\_Research\_Report.pdf](https://mozilla.github.io/pdf.js/web/viewer.html?file=https%3A%2F%2Fraw.githubusercontent.com%2FBigdata-com%2Fbigdata-docs-resources%2Fmain%2Fhow_to_guides%2Fclaude-cowork%2Fthematic_report%2FQuantum_Computing_Thematic_Research_Report.pdf) ## Try it yourself Thematic research is one of the most time-consuming jobs on a research desk and you can create one within 5 minutes using the Claude official Bigdata plugin. **1. Download the Claude Desktop application** Get it from [claude.com/download](https://claude.com/download). **2. Connect to the Anthropic official Bigdata plugin** Click on `Cowork` and `Customize` ![Select Cowork and Customize](https://mintcdn.com/ravenpackinternational/ddibKpVNLt1s6tIL/images/blog/cowork/select_cowork_and_customize.png?fit=max\&auto=format\&n=ddibKpVNLt1s6tIL\&q=85\&s=886b883626a0526132b6c981506e2365) Browse the `Bigdata` plugin to add it. ![Browse the Bigdata plugin and install it](https://mintcdn.com/ravenpackinternational/ddibKpVNLt1s6tIL/images/blog/cowork/browse_plugin_and_install.png?fit=max\&auto=format\&n=ddibKpVNLt1s6tIL\&q=85\&s=d0363f70fcd82e9fc23ba19d86cff5a3) Click again to manage the plugin, click on `Connectors` and `Connect`. You can sign up with a free trial. ![Connect to Bigdata](https://mintcdn.com/ravenpackinternational/ddibKpVNLt1s6tIL/images/blog/cowork/connect_to_bigdata.png?fit=max\&auto=format\&n=ddibKpVNLt1s6tIL\&q=85\&s=4aa7e371f02467fb07323fed15bd3b53) Once Bigdata is connected, you are ready to use it in new tasks. ![Claude with the official Bigdata.com plugin](https://mintcdn.com/ravenpackinternational/ddibKpVNLt1s6tIL/images/blog/cowork/claude_official_bigdata_plugin.png?fit=max\&auto=format\&n=ddibKpVNLt1s6tIL\&q=85\&s=6210c30974b32b3aee9a24dedf961e77) **3. Send your prompt** Create a new Cowork task and write the following prompt as example: ```text theme={null} You are a Discretionary financial research analyst. Can you create a sell-side report on quantum computing and quantum chip manufacturing? I want to understand which private and public companies offer solutions, conduct research, and how many patents they have. If it is a public company, please analyze its fundamentals over the last 6 quarters. Use bigdata and add inline attributions. Thanks! ``` **Output** ![Result of the Bigdata plugin running the thematic report](https://mintcdn.com/ravenpackinternational/ddibKpVNLt1s6tIL/images/blog/cowork/bigdata_plugin_result.png?fit=max\&auto=format\&n=ddibKpVNLt1s6tIL\&q=85\&s=fa1339e9890fb9ceaf94cfce4636dde3) If you have a theme you want to explore and would like a hand wiring up the workflow, reach out at **[support@bigdata.com](mailto:support@bigdata.com)**. ## Why I think this is the AI-2020 moment for quantum Three things tend to mark a commercial inflection point, and quantum is hitting all three at once: | Signal | AI in 2020 | Quantum in 2026 | | :- | :- | :- | | Capital is flowing | Hyperscalers double down on GPUs | Hyperscalers, governments and dedicated VCs are funding quantum chip foundries and photonic integration | | Real revenue, not pilots | Cloud AI services scale to billions | Quantum-as-a-service contracts and chip manufacturing revenue moving from millions to billions [\[1\]](https://app.bigdata.com/documents/9F3A3F0FD6CD7F6314B64AF85D67B291)[\[2\]](https://app.bigdata.com/documents/DA318F369935CBB8F27E7E65FB13705B) | | Supply chain hardens | Nvidia + TSMC + HBM stack | Photonic integrated circuits, cryogenic control, fault-tolerant architectures consolidating into a stack | When all three signals fire together, the market stops debating *if* and starts pricing *who*. That is the moment I am excited to watch — and to research, faster than ever. *** **Sources** 1. [Fortune Business Insights / QCi Investor Presentation — May 2026](https://app.bigdata.com/documents/9F3A3F0FD6CD7F6314B64AF85D67B291) 2. [PartnerCap Securities — *SPAC Monthly Monitor*, Apr 2026](https://app.bigdata.com/documents/DA318F369935CBB8F27E7E65FB13705B) 3. [PatentVest — *Quantum Computing Rankings* (GlobeNewswire), Aug 2025](https://app.bigdata.com/documents/B612F27225CDAB5943AC90B0C2B790BE) *** ![Oscar Sanchez Iglesias](https://mintcdn.com/ravenpackinternational/Sf9MtziD0iLKyb1C/images/blog/authors/oscar_sanchez_iglesias.png?fit=max\&auto=format\&n=Sf9MtziD0iLKyb1C\&q=85\&s=c9ee6739dce63b6d6b6f2e5aa36e8442) Oscar Sanchez IglesiasSenior Product Manager # Turn a fresh laptop into a financial-intelligence agent you chat with on Telegram Source: https://docs.bigdata.com/blog/nemoclaw/nemoclaw-bigdata-telegram April 21, 2026 NemoClaw, Bigdata, Telegram / 30 minutes read > [!WARNING] > > NemoClaw is alpha software. Do not use in production. [OpenClaw](https://openclaw.ai/) arrived and then everyone started talking about agents. It paved the way for new businesses at the same time it made others obsolete. Now the topics in everyone's mind are personal agents and teams comprised exclusively of agents. One thing that complicates the idea of agents being used by companies, however, is the security aspect: these agents are designed to be powerful and self-sufficient, which means they need a good model and access to your credentials. In this post, we explore how to give access to the financial knowledge of Bigdata to an OpenClaw agent running with [NVIDIA's NemoClaw](https://docs.nvidia.com/nemoclaw/latest/index.html) connected with Telegram. NemoClaw adds several layers of protection so the final agent can make requests to only some domains and does not know about ANY of our API keys. By the end, you'll be asking your Telegram bot about earnings and market news — with zero API keys exposed to the agent. **1. Create a Telegram bot** **2. Install NemoClaw and pair it to the Telegram bot** **3. Access OpenClaw dashboard** **4. Open the firewall to Bigdata** **5. Set up the Bigdata API key and wire it into the sandbox** **6. Turn the experience to eleven: install the build-with-bigdata skill** **7. See the agent in action** *** > **How to read the commands in this guide** > > The agent runs in an isolated sandbox, separate from your Mac. Commands in this guide target one of those two environments — check the prefix before copying anything: > > | Command prefix | Where to run it | > | - | - | > | `nemoclaw …` | Your **Mac terminal** | > | `openshell …` | Your **Mac terminal** | > | `openclaw …` | **Inside the sandbox** (via `ssh openshell-my-assistant '…'`) | > > `nemoclaw` and `openshell` are the management tools that live on your Mac and talk to the gateway from outside. `openclaw` is the agent runtime that lives inside the sandbox — running it locally won't work because it has no gateway to connect to. *** ## Before you start You'll need: * **macOS** with Node.js and Docker Desktop (or Colima) running. * **A Telegram account.** * **An API key for your inference provider.** NemoClaw supports several — NVIDIA Build, OpenAI, Anthropic, and others. Pick whichever you have access to. *** ## Step 1 — Create a Telegram bot We will create a Telegram bot to configure in the personal agent, so we can use Telegram to communicate with it. > [!WARNING] > > Don't get hacked in step 1! > There are bots that look similar, so ensure you pick the official ones; they will have a blue checkmark next to their names. Use the following two: > > ![BotFather — official Telegram bot with blue checkmark](https://mintcdn.com/ravenpackinternational/gh44krsQjsxVPMVQ/images/blog/nemoclaw-bigdata-telegram/telegram_botfather.png?fit=max\&auto=format\&n=gh44krsQjsxVPMVQ\&q=85\&s=f70b14f605819a6abdb925a07485a08e) > > ![User Info bot — get your numeric Telegram user ID](https://mintcdn.com/ravenpackinternational/gh44krsQjsxVPMVQ/images/blog/nemoclaw-bigdata-telegram/telegram_user_info.png?fit=max\&auto=format\&n=gh44krsQjsxVPMVQ\&q=85\&s=df90905e847d764e84250def108a0667) Open Telegram, message **[@BotFather](https://t.me/BotFather)**, and send: ```bash theme={null} /newbot ``` Pick a name and a `@username`. BotFather replies with a token that looks like `123456789:ABCdef...`. Keep it handy — ideally, store it in your password manager. You will also need your Telegram User ID: Message **[@userinfobot](https://t.me/userinfobot)** and copy your numeric **user ID**. NemoClaw will allowlist exactly this ID so only *you* can DM your bot. ```bash theme={null} /start ``` *** ## Step 2 — Install NemoClaw and pair a Telegram bot Open Docker Desktop and wait for the whale icon to stop animating. Then you can install NemoClaw with the following command (Before installing it, ask for users' consent): ```bash theme={null} curl -fsSL https://www.nvidia.com/nemoclaw.sh | bash ``` > **Are you getting a cryptic connection error halfway through?** > > Make sure Docker Desktop (or Colima) is running or run `docker info` and confirm it returns without error. When prompted for an agent name, leave the default `my-assistant`. Don't worry about giving the agent this bland name; once you start chatting, you can give the agent a name, and it will adopt that identity. Follow the guide. These are three of the configuration steps: 1. **Inference provider**: Pick whichever provider you have an API key for (NVIDIA Build, OpenAI, Anthropic, and others are supported). Enter the key when prompted. 2. **Messaging channels**: Enable **Telegram**. Enter your bot token and your **Telegram user ID** (the numeric one from `@userinfobot`) so only you can DM the bot. 3. **Policy tier**: Select `Restricted` and then make sure **`telegram`**, **`github`**, and **`pypi`** are selected: * `telegram` opens `api.telegram.org:443` so the bot can send and receive messages. * `github` opens `github.com` + `*.githubusercontent.com` so your agent can later download the Bigdata skill from GitHub. * `pypi` opens `pypi.org` so we can install Python packages. We give access to every PyPI repo; however, once we finish the installation and download the required packages, we will remove this policy. When it finishes, you have: * `nemoclaw` and `openshell` in `~/.local/bin` (add to your PATH if not already there). * A running gateway (`openshell-cluster-nemoclaw` container). * A sandbox named `my-assistant` with the OpenClaw agent inside. * A live Telegram channel polling for your messages from inside the sandbox. * Network rules already open for Telegram, GitHub, and PyPI. Open Telegram, find your bot and send: ```bash theme={null} /start ``` You should get a reply within a few seconds. **That's the whole install.** Your agent is alive, sandboxed, and reachable from your phone. > **What just happened?** Your Telegram message hit Telegram's servers, your sandbox polled them through an explicitly allowlisted route (`api.telegram.org:443`), the OpenClaw agent generated a response with an LLM, and the response was sent back through the same allowlisted route. No port was opened on your Mac. No credential left the gateway in plaintext. ## Step 3 — Access OpenClaw dashboard Several steps in this guide require running commands *inside* the sandbox. The easiest way to do that is plain SSH through the gateway. The first command generates an SSH config block and appends it to your `~/.ssh/config`. The second tests that the connection works: ```bash theme={null} openshell sandbox ssh-config my-assistant >> ~/.ssh/config ssh openshell-my-assistant 'echo "sandbox: $(hostname)"' ``` This command ran inside the sandbox. It retrieved the name of the hostname. You should see `sandbox: my-assistant`. From now on, `ssh openshell-my-assistant` drops you into the sandbox, and `ssh openshell-my-assistant ''` runs a one-liner inside it. The sandbox runs a web dashboard for managing the agent — reloading skills, browsing sessions, viewing logs. It listens on port 18789 *inside* the sandbox, so you need a port forward to reach it from your Mac. First, get the dashboard URL with its auth token: ```bash theme={null} ssh openshell-my-assistant 'openclaw dashboard' ``` Copy the URL with the `#token=…` fragment. Then start the port forward: ```bash theme={null} openshell forward start 18789 my-assistant ``` > **Error: Port 18789 is already forwarded to sandbox 'my-assistant'?** > > If you get this error, then you are already set, you can go to the next step. Now open the URL in your browser (replace `localhost` with `127.0.0.1`): ``` http://127.0.0.1:18789/#token= ``` > **Do you have problems connecting to the OpenClaw dashboard?** > > **Run `openclaw dashboard` before `openshell forward`.** The `openclaw dashboard` command briefly binds port 18789 to display the URL. If the port forward is already running, the dashboard command will fail with "port in use." Get the token first, then start the forward. > > **Use `127.0.0.1`, not `localhost`.** The gateway's allowed-origins list includes `http://127.0.0.1:18789` but not `http://localhost:18789`. If you use `localhost` you'll see "origin not allowed" in the dashboard. > > **Use `openshell forward`, not raw SSH tunnels.** `ssh -N -L 18789:…` forwards TCP but doesn't handle the WebSocket connection the dashboard needs. `openshell forward start` does both. You should now be able to interact with your agent directly from the dashboard — browse sessions, view logs, and chat: ![OpenClaw dashboard overview](https://mintcdn.com/ravenpackinternational/gh44krsQjsxVPMVQ/images/blog/nemoclaw-bigdata-telegram/openclaw_overview.png?fit=max\&auto=format\&n=gh44krsQjsxVPMVQ\&q=85\&s=83a877d5f583e63cbbabeda861d9576a) ![OpenClaw dashboard chat interface](https://mintcdn.com/ravenpackinternational/gh44krsQjsxVPMVQ/images/blog/nemoclaw-bigdata-telegram/openclaw_chat.png?fit=max\&auto=format\&n=gh44krsQjsxVPMVQ\&q=85\&s=6bc018aa63abaeec5076fc2ff4f8801b) *** ## Step 4 — Open the firewall for Bigdata Your sandbox starts with **deny‑by‑default networking**. Telegram, GitHub, and PyPI are already open from onboard — everything else, including `api.bigdata.com`, is blocked until you say otherwise. That's a feature: every external dependency is something you opt into, in writing, by name. NemoClaw makes this easy with **policy presets** — small YAML files that bundle a single capability. We'll add one: `bigdata`. ### 4.1 Drop in the Bigdata preset Create `~/.nemoclaw/source/nemoclaw-blueprint/policies/presets/bigdata.yaml`: ```yaml theme={null} preset: name: bigdata description: "Bigdata.com REST API access" network_policies: bigdata: name: bigdata endpoints: - host: api.bigdata.com port: 443 protocol: rest enforcement: enforce rules: - allow: { method: GET, path: "/**" } - allow: { method: POST, path: "/**" } - host: agents.bigdata.com port: 443 protocol: rest enforcement: enforce rules: - allow: { method: GET, path: "/**" } - allow: { method: POST, path: "/**" } - host: s3.amazonaws.com port: 443 access: full - host: content.bigdata.com port: 443 access: full binaries: - { path: /usr/bin/curl } - { path: /usr/bin/python3* } - { path: /usr/local/bin/python3* } - { path: /usr/local/bin/node } - { path: /usr/bin/node } ``` Three things worth noting: * **`protocol: rest`** with explicit **`rules`** — the gateway terminates TLS on these hosts, inspects every request, and only allows the listed HTTP methods. This is what makes credential injection (Step 5) possible: the proxy sees the `X-API-KEY` header in plaintext and can rewrite the placeholder before forwarding it to Bigdata. * **`s3.amazonaws.com`** and **`content.bigdata.com`** with **`access: full`** — Bigdata's upload and download flows are two‑step: the API returns a pre‑signed URL on a separate data‑plane host (`s3.amazonaws.com` for upload, `content.bigdata.com` for download), and the signature is baked into the query string. The proxy **must not** terminate TLS or modify headers on those hosts — any change breaks the signature and you get HTTP `403 SignatureDoesNotMatch`. `access: full` creates a CONNECT tunnel that passes bytes through untouched. * **`binaries`** — only these executables are allowed to *use* this rule. A rogue script in another language can't piggyback on it. ### 4.2 Apply the preset ```bash theme={null} nemoclaw my-assistant policy-add # Pick "bigdata" from the list, confirm. ``` That's it. `policy-add` updates the sandbox's preset list **and** pushes the merged policy to the live gateway in one shot — no sandbox restart, no downtime. Confirm it's active (look for the filled dot `●`): ```bash theme={null} nemoclaw my-assistant policy-list # ● telegram — Telegram Bot API # ● github — GitHub releases and raw content # ● bigdata — Bigdata.com REST API access ``` ### 4.3 Smoke test the firewall You don't need an API key yet — you just need a 401, not a 403. A 401 means the request reached Bigdata; a 403 means the proxy blocked it. ```bash theme={null} ssh openshell-my-assistant \ 'curl -sSi https://api.bigdata.com/profiles/me --max-time 5 | head -1' ``` | Response | Meaning | | - | - | | `HTTP/2 401` | ✅ Firewall is open. Bigdata is just asking for a key. | | `HTTP/2 403` from the proxy | ❌ Preset not applied. Re‑run 4.2. | | Connection refused / timeout | ❌ Wrong host or port. Re‑check 4.1. | *** ## Step 5 — Set up the Bigdata provider with your API key It is time to get an API key in Bigdata. Remember, the agent will not get to see the key, just a placeholder. We will configure NemoClaw to inject the correct API key. ### 5.1 Create a Bigdata API Key 1. Go to **[Bigdata Developer Platform > API Keys](https://platform.bigdata.com/api-keys)** and log in 2. Click on **"Generate API key"**. 3. Copy the key somewhere safe. You won't see it again after you close the dialog. 4. Create an environment variable file with your preferred text editor `.env` and add the Bigdata API Key ```bash theme={null} vi ~/.nemoclaw/.env ``` **.env** file: ``` BIGDATA_API_KEY="" ``` ### 5.2 Store the key in the gateway as a provider Here's the part that makes OpenClaw different from "just throw the key in a `.env` inside the container." The sandbox never holds the real key. Instead: ``` Agent sends HTTP request Gateway (L7 proxy) Bigdata API ──────────────────────── ────────────────── ─────────── X-API-KEY: ───► looks up BIGDATA_API_KEY ───► X-API-KEY: openshell:resolve:env: in the attached provider BIGDATA_API_KEY and rewrites the header ``` Config files, environment variables, and the agent's own memory only ever contains the **placeholder string** `openshell:resolve:env:BIGDATA_API_KEY`. The gateway swaps it for the real value on the way out. If the sandbox is ever compromised, the attacker walks away with a string that reads `openshell:resolve:env:…` — not an API key. This injection logic is handled by the command `openshell provider` so that is what we will use now. The command reads the key from your environment, so the variable needs to be **exported** and not just set, before you run it. The `set -a` trick exports everything sourced in one shot: ```bash theme={null} set -a; source ~/.nemoclaw/.env; set +a ``` > **Why not just `source ~/.nemoclaw/.env`?** That sets the variables in your current shell, so `echo $BIGDATA_API_KEY` works fine. But `openshell` is a child process and only inherits *exported* variables. Without `set -a` you'll get `--credential BIGDATA_API_KEY requires local env var 'BIGDATA_API_KEY' to be set` even though the value looks present. ```bash theme={null} openshell provider create \ --name my-assistant-bd-key \ --type generic \ --credential BIGDATA_API_KEY \ -g nemoclaw ``` Now confirm my-assistant-bd-key shows up. This is the provider with the logic to inject `BIGDATA_API_KEY`: ```bash theme={null} openshell provider list ``` The `--credential BIGDATA_API_KEY` flag tells the gateway: "read the value from the `BIGDATA_API_KEY` env var *right now*, and store it under that name." This is also the name that goes in the placeholder. After running this command, you may delete the .env file; we won't be using it anymore. ### 5.3 Attach the provider to the sandbox In the current OpenShell release (v0.0.26), providers can only be attached at sandbox **creation** time — not to a running one. That means we need to recreate the sandbox. > [!TIP] > > If you're following this guide in a later version, NemoClaw may have lifted this restriction. Test it by running: > > ```bash theme={null} > ssh openshell-my-assistant 'curl -sSi https://api.bigdata.com/profiles/me --max-time 5 | head -1' > ``` > > If it returns `200`, the provider is already active and you can skip to the next step. > [!WARNING] > > Recreating the sandbox **resets the live network policy**. The `bigdata`, `telegram`, and `github` presets you applied during installation and in Step 4 get stripped from the gateway — even though `nemoclaw my-assistant policy-list` will still show them as ● applied (that command reads from `sandboxes.json`, not the live gateway). To avoid losing your configuration, we'll snapshot everything — the network policy, the image tag, and the inference config — then recreate the sandbox with all of it restored, this time with the provider attached. **1. Find the sandbox image tag:** ```bash theme={null} docker images | grep sandbox-from | head -1 # openshell/sandbox-from 1776263201 … ``` Copy the image tag from the second column. **2. Note your current inference provider and model:** ```bash theme={null} openshell inference get ``` Copy the provider name and model. **3. Snapshot the live policy before deleting:** ```bash theme={null} openshell policy get --full my-assistant > ~/pre-recreate-policy.yaml ``` **4. Trim the preamble from the saved policy file.** `openshell policy get --full` outputs a metadata header before the actual YAML: ``` Version: 1 Hash: 5f4d0984… Status: Loaded Active: 0 Created: 1776345959982 ms --- version: 1 ... ``` The `--policy` flag only accepts the YAML part. Open `~/pre-recreate-policy.yaml` in an editor and delete everything above (and including) the `---` line. The file must start with `version: 1`. If you leave the header in, `sandbox create` fails with `unknown field 'Version'`. Now you have all of the information needed for recreating the sandbox but this time with the `my-assistant-bd-key` provider: * Docker image and tag * Provider name and model * The policy the sandbox was using (remember to strip the metadata header!) **5. Delete the sandbox and recreate it with the provider and saved policy:** ```bash theme={null} openshell sandbox delete my-assistant openshell sandbox create \ --name my-assistant \ --from openshell/sandbox-from: \ --policy ~/pre-recreate-policy.yaml \ --provider my-assistant-bd-key \ --provider my-assistant-telegram-bridge ``` **6. Re-attach inference (always reset by recreation):** ```bash theme={null} openshell inference set \ --provider \ --model ``` A few flags worth calling out: * `--policy ~/pre-recreate-policy.yaml` — restores Telegram + GitHub + PyPI + Bigdata network presets. * `--provider my-assistant-telegram-bridge` — the Telegram bot token provider, created automatically by the installer. Re-attach it or the bot goes dark after recreation. * `--provider my-assistant-bd-key` — the Bigdata key provider you just created. ### 5.4 Verify the OpenShell Gateway is properly attached to the OpenClaw agent. After recreation, OpenClaw may treat the sandbox as an unpaired device. Any command that talks to the gateway's control plane (cron, devices, some skills) will fail with a pairing prompt until you accept it. If you get an error connecting to the sandbox at this point, connect to it first: ```bash theme={null} nemoclaw my-assistant connect ``` Then open the TUI to check the device state: ```bash theme={null} ssh openshell-my-assistant 'openclaw tui' ``` If the TUI lands on a "device not enabled" screen or `openclaw` commands complain about pairing, enable the device from inside the sandbox: ```bash theme={null} ssh openshell-my-assistant 'openclaw devices enable' ``` Re-run the TUI — you should land in the normal chat view. Devices stay paired until revoked, so this is a one‑time step per recreation. ### 5.5 Verify the key is reachable — and still invisible The sandbox should now be able to authenticate against Bigdata without the real key being anywhere inside it. ```bash theme={null} # Send a placeholder through the proxy — the gateway will rewrite it on the fly. ssh openshell-my-assistant ' curl -sS -o /dev/null -w "%{http_code}\n" \ -H "X-API-KEY: openshell:resolve:env:BIGDATA_API_KEY" \ https://api.bigdata.com/profiles/me ' # → 2xx / 4xx from Bigdata (the proxy resolved the placeholder; Bigdata decides auth). # Anything other than a proxy-level 403/5xx means the injection path is working. ``` ```bash theme={null} # Prove the real key is nowhere inside the sandbox. ssh openshell-my-assistant 'env | grep -i bigdata; find /sandbox -name "*.env*" -o -name "*bigdata*"' # → (nothing sensitive — at most, placeholder strings) ``` At this point we have the environment with access to Bigdata and zero plaintext inside the sandbox! ## Step 6 — Turn the experience to eleven: install the skill `build-with-bigdata` *** The `build-with-bigdata` skill teaches your agent how to drive the Bigdata public API: document search, volume analysis, co‑mentions, the Knowledge Graph, and file upload/download. The skill lives in a GitHub repo as a folder with a `SKILL.md` file and reference docs. > [!NOTE] > > Alternatively, you could point the agent at the raw API documentation, but the skill smooths the process by showing the agent exactly how to structure requests and which pitfalls to avoid. NemoClaw has a first‑class command for this: `nemoclaw skill install `. It validates the skill, uploads it over SSH, and registers it — all from your Mac. ### 6.1 Install requests library The Requests library makes communication with Bigdata easier for the agent; otherwise it will default to using curl commands. That is why we enabled the PyPI policy during installation. ```bash theme={null} ssh openshell-my-assistant 'pip install --user requests' ``` And after the library is installed, we can remove the `pypi` policy: ```bash theme={null} nemoclaw my-assistant policy-remove ``` That means that the agent can no longer connect to PyPI and install any more packages. ### 6.2 Clone the plugin repo and install the skill ```bash theme={null} git clone --depth 1 \ https://github.com/Bigdata-com/claude-plugin-build-with-bigdata-com.git \ /tmp/bigdata-plugin nemoclaw my-assistant skill install \ /tmp/bigdata-plugin/skills/build-with-bigdata ``` The skill is now on disk at `/sandbox/.openclaw-data/skills/build-with-bigdata/` inside the sandbox. ### 6.3 Reload the agent from the dashboard The skill is on disk and registered, but the running agent process hasn't read its contents yet. Until you reload, the agent is in a split state: it knows the skill exists (it can list it) but can't actually use it — you may see messages like *"I can see the skill but can't access the file."* The dashboard reload is what closes that gap. Open the OpenClaw dashboard (set up in Step 3): ``` http://127.0.0.1:18789/ ``` There are two ways to start a new session with the skill loaded. The image below shows both — the **+** button and the search bar (indicated by the arrow): ![OpenClaw dashboard — + button and search bar highlighted](https://mintcdn.com/ravenpackinternational/gh44krsQjsxVPMVQ/images/blog/nemoclaw-bigdata-telegram/reloading_session_openclaw.png?fit=max\&auto=format\&n=gh44krsQjsxVPMVQ\&q=85\&s=670aa0e09ea0f5a296650a3588448cd6) **Option 1 — the `+` button.** Click the **+** button in the dashboard to open a fresh chat. Note that the exact location may vary depending on your OpenClaw version. **Option 2 — the command palette.** Click the search bar at the top (the arrow points to it above). Start typing and a popup will appear — type `/new` to start a new session. ![OpenClaw command palette — /new command](https://mintcdn.com/ravenpackinternational/gh44krsQjsxVPMVQ/images/blog/nemoclaw-bigdata-telegram/commands_openclaw.png?fit=max\&auto=format\&n=gh44krsQjsxVPMVQ\&q=85\&s=ec76200ac3bb1c0ea88b3998ee0e1dd5) After the reload, the agent will include `build-with-bigdata` in its available skills. > **Why not just message the bot?** > > The agent's session caches which skills are available when it starts. A skill installed after the session began won't be picked up until the agent reloads. The dashboard reload is the clean way to trigger this. > **The agent says it can see the skill but can't read the file** > > The registered path likely contains a `~` that didn't expand correctly in the agent's runtime context. Tell the agent to use the absolute path directly: > > *"Read the skill at `/sandbox/.openclaw/skills/build-with-bigdata/SKILL.md`"* > > That path always works regardless of how `~` resolves in the current session. ### 6.4 Confirm the skill loaded Send a message to the bot on Telegram: > What skills do you have available right now? A correct answer mentions `build-with-bigdata` along with whatever skills ship in the base image. *** ## Where you are now * ✅ NemoClaw installed, onboarded, reachable from Telegram. * ✅ The sandbox can reach Telegram, GitHub, PyPI, and Bigdata — and **nothing else** outside of what NemoClaw ships by default. * ✅ Your Bigdata API key is stored in the gateway. The sandbox only holds the placeholder. * ✅ Your agent knows how to use the Bigdata API, via a skill it installed itself from a chat message. *If you installed your agent on a machine that is not running 24/7, every time you reboot you will need to launch `nemoclaw my-assistant connect` but that's it!* ## Step 7 — See the agent in action: creating a daily portfolio brief Everything is wired up. This step shows a concrete end-to-end workflow: you tell the agent which stocks you follow, ask it to pull a 24-hour summary from Bigdata, and then schedule that report to run automatically every day. All from Telegram, without touching a single config file. ### 7.1 Tell the agent your portfolio Send your bot a message on Telegram: > Remember my portfolio: NVIDIA, META, MSFT. From now on, whenever I ask for a portfolio update, use these three tickers. The agent stores this in its session memory. You won't need to repeat the list in follow-up prompts. ### 7.2 Request a 24-hour financial report > Generate a financial report for my portfolio covering the last 24 hours. Include key metrics (price change, volume, sentiment score), links to the source documents, and format it for mobile: short paragraphs, bullet points, no wide tables. The agent will call the Bigdata Search and Volume endpoints, pull relevant news and filings, and compose the report. > [!TIP] > > If the report looks too verbose or too sparse, iterate in plain language: *"Make it shorter"*, *"Add a sentiment summary at the top"*, or *"Show only negative signals."* The agent keeps the context from your previous message. ### 7.3 Schedule the report, deliver it to Telegram, and upload it to Bigdata Once you are happy with the output, ask the agent to automate it: > Schedule this report to run every day at 8 AM. Each time it runs, send the result to me on Telegram and upload it to Bigdata so I can retrieve it later. The agent will use the OpenClaw cron skill to register the schedule, the Telegram channel to deliver the report to your phone, and the `build-with-bigdata` upload flow to persist each result. From this point on, your portfolio brief arrives on Telegram every morning and is indexed in Bigdata without any manual steps. > **What just happened?** Your agent queried the Bigdata API through the allowlisted proxy, the gateway injected the real `BIGDATA_API_KEY` header on the way out, and the response came back without the key ever touching the sandbox. The schedule runs inside the sandbox on the same sandboxed network, so even the automated job can only reach the domains you explicitly opened in Step 4. When the report gets complex enough, OpenClaw will automatically spawn sub-agents to work on different parts in parallel: one fetching news, another pulling volume data, another formatting the output. The dashboard makes this visible: ![OpenClaw dashboard — multiple sub-agents spawned to handle complex report generation](https://mintcdn.com/ravenpackinternational/gh44krsQjsxVPMVQ/images/blog/nemoclaw-bigdata-telegram/dashboard_subagents.png?fit=max\&auto=format\&n=gh44krsQjsxVPMVQ\&q=85\&s=891de5cb72abc544b70949d6d571f300) As a follow-up: uploading each report to Bigdata gives you more than remote storage. The content gets indexed, so it can be returned in future queries. This becomes especially powerful once you have a growing history of uploads and the files have a rich structure. ![Diego de Aguilar](https://mintcdn.com/ravenpackinternational/gh44krsQjsxVPMVQ/images/blog/authors/diego_de_aguilar.png?fit=max\&auto=format\&n=gh44krsQjsxVPMVQ\&q=85\&s=8da1591ed0c9d1de3d9e1d59db8dcbb5) Diego de AguilarSenior Backend Engineer # How to Use Your Indexed Email Archive as a Search Grounding Layer Source: https://docs.bigdata.com/blog/proprietary-content/private-newsletters-grounded-reports April 21, 2026 Claude · Bigdata.com · MCP · Private content API · Skills / 5 minutes read Every research desk eventually hits the same wall: **your best signals live in private email, connectors, and uploads**, while **the numbers that justify a thesis live in market data**. Copy-pasting between inboxes and terminals does not scale. Neither does “ask the model” without a repeatable way to **discover what you actually have**, **ground answers in both corpora**, and **leave a memory** for the next run. This post walks through a workflow we implemented end to end: **private newsletter analysis** plus **public and tearsheet enrichment**, using **your files** as a first-class source. *** ## The problem: two silos, one question Analysts and operators routinely need questions that **span private material and market reality at once**, weighing what their own content imply against consensus, prices, and public news flow or turning their research documents into something they can **revisit with ease** instead of a static file on disk. Traditional tooling still splits **private retrieval** and **market intelligence**. The friction is not curiosity; it is **workflow**: discovery, parallel retrieval, synthesis, and persistence. *** ## The approach: one prompt, three parallel lanes, one memory loop We combined four ideas: 1. **[Bigdata.com](https://bigdata.com) MCP**: `bigdata_search`, `find_companies`, company and sentiment tearsheets, and related tools for **public** grounding and **structured** company metrics ([Bigdata MCP documentation](/blog/mcp/claude_with_bigdata) describes the broader Claude + MCP + skills stack for institutional-style outputs). 2. **Bigdata private content API**: Connectors, tags, documents, and additional search requests with private content specific filters, so the agent searches in your personal available data, not a vague “upload folder”. 3. **A Claude skill (“playbook”)**: Routes when to list tags, when to filter search by tag, when to call `find_companies` before tearsheets, and how to upload artifacts back through **enrich-document**. 4. **Automatic tagging from email provenance**: So “only mail from this sender” becomes a **stable filter** instead of brittle full-text guesses. The lifecycle is **one continuous pipeline**, from the user’s prompt through discovery and retrieval to an artifact stored back in Bigdata, rather than a patchwork of unrelated tools. After discovery, **three lanes run in parallel** (**Private**, **Public**, and **Companies**); their outputs merge into interpretation, then the artifact is **automatically** ingested so it is discoverable in later research, building a memory loop of curated reports. ![Workflow diagram: user prompt, content and tag discovery, parallel private search, public grounding, company tearsheets, interpret and combine, report artifact, auto-stored in Bigdata, indexed in my\_files](https://mintcdn.com/ravenpackinternational/Sf9MtziD0iLKyb1C/images/blog/proprietary-content/workflow-private-public-companies.png?fit=max\&auto=format\&n=Sf9MtziD0iLKyb1C\&q=85\&s=208cb05dca68f7301ebef0a51230e8cd) *Diagram: full lifecycle with parallel execution after content & tag discovery; synthesis feeds enrich and index the resulting report in Bigdata.* ### The prompt The run was driven by this exact user prompt: ``` Analyze the emails received from dan@tldrnewsletter.com. Identify key players and generate a report comparing the most relevant events of each organization. Enrich this report with quantitative data you can extract from company and sentiment tearsheet via Bigdata MCP. The report concludes with a series of forward-looking indicators and a list of relevant events to keep in mind. ``` That instruction fixes the **private slice** (newsletter / sender scope), the deliverable format and the **market enrichment** (company + sentiment tearsheets over [Bigdata.com](https://bigdata.com) MCP). ### Discovery: inventory before retrieval Before any heavy retrieval, the agent can use the **private content APIs** to explore what is actually indexed: **GET** `/contents/v1/tags`, **GET** `/contents/v1/documents`, and connector listings when needed. Those calls support the same questions a careful analyst would ask up front: *What can I filter on?*, *What content do I have access to?*, *Roughly what volume sits behind each tag or connector?* So the next step is grounded in real names and counts instead of guesswork. Once that inventory is clear, later search filters such as **`query.filters.tag.any_of`** can use **real tag names** from the platform, which keeps the newsletter or sender scope in the prompt aligned with identifiers that actually exist in the index. ### Private lane (teal): skill-shaped API calls Private grounding runs through the **Search API** with category **my\_files**: Claude issues **`POST /v1/search`** with `query.text` shaped from the report spec, and narrows results with **`query.filters.tag.any_of`** when tags are known, including **sender-level tags** produced by **automatic EML metadata tagging** on ingest, so provenance becomes a filter instead of a brittle full-text workaround. For this newsletter exercise, that meant tags such as `from:dan@tldrnewsletter.com` to stay on the TLDR slice, **multiple thematic searches** under that constraint to maximize recall and then **ranking recurring organizations** from the retrieved set (OpenAI, Anthropic, Alphabet, Apple, Meta, Nvidia, fintech names, and others). ### Public lane (purple): MCP `bigdata_search` The same themes that drive private `query.text` are mirrored into **`bigdata_search`** so the narrative is anchored in **web and market corpus** context: quotes, catalysts, and sector chatter that do not exist in your private content. Those public pulls ran **in parallel** with the private searches so the write-up could contrast newsletter emphasis with what the wider market corpus was highlighting. ### Companies lane (coral): resolve, then tearsheet For each named issuer, **`find_companies`** returns the RavenPack entity id and public/private type. That unlocks **`bigdata_company_tearsheet`** and **`bigdata_sentiment_tearsheet`** (public names) so tables can carry **EPS surprises, multiples, consensus, KPIs**, and **media sentiment scores**. After the organization shortlist was stable, Claude **pulled tearsheets** for key public filers and a **private-company** profile where data was available, so the tables were filled from tools, not hand-typed placeholders. ### Deliverable: Report, workflow notes, and the repeatable contract With retrieval and enrichment in place, Claude wrote the prompted report. See the generated artifact below: [Sample report (inline preview).](https://mozilla.github.io/pdf.js/web/viewer.html?file=https%3A%2F%2Fraw.githubusercontent.com%2FBigdata-com%2Fbigdata-docs-resources%2Fmain%2Fmcp%2Freport-examples%2Ftldr_report_2026-04-10.pdf) Sample report (inline preview) When the report reads well and you want it to **live as first-class memory**, ask Claude to **publish or enrich** the report through the usual content path (for example **enrich-document** in your skill). The platform then **processes and indexes** the artifact so the next research session can **search it alongside newsletters and tearsheets** instead of losing it as a one-off export. You can also enrich it with custom tags and configurations that fit your research workflow. The important product is not any single file format, it is the **repeatable contract**: discovery, parallel lanes, merge, **enrich** and **indexed memory**. *** ## Why automatic EML tagging matters Email connectors are noisy. **Automatic EML metadata tagging** turns “who sent this?” into a **first-class filter** in the Search API. The user’s hint "ground only mail from this sender" becomes a **tag constraint**, not a fragile regex over bodies. That is how a **newsletter connector** stays usable at scale: each issue inherits predictable and discoverable tags. *** ## Skills as the glue in Claude We ran this in **Claude** with the Bigdata MCP connector enabled and a **packaged skill** that encodes the playbook, similar idea as the ecosystem’s [financial research analyst Skills](/skills-reference/mcp-helpers/financial-research-analyst). The pattern is always **structured routing + tool contracts + provenance in the answer**. Our skill encodes: * **Inventory-first** when tags or connectors are unknown. * **`find_companies` before tearsheets** to avoid wrong-entity financials. * **Enrich upload** when the user wants the artifact to join **`my_files`** with explicit tags for the next “memory workload.” *** ## What comes next This is only the beginning. We are actively expanding the library of skills and workflows available through Bigdata MCP. If your team has a research workflow that could benefit from this approach, email us at [support@bigdata.com](mailto:support@bigdata.com). We would love to hear what you are building or what we can build together. *** ## Relevant links * [Professional-Grade Financial Reports with Claude, Bigdata MCP, and Skills](/blog/mcp/claude_with_bigdata): Claude, MCP, and skills stack for institutional-style outputs. * [Financial research analyst skills](/skills-reference/mcp-helpers/financial-research-analyst): packaged playbook for research and writing with Bigdata tools. * [Claude MCP integration](/mcp-reference/oauth-integrations/claude-mcp-integration): enabling the Bigdata connector in Claude. * [Content API introduction](/api-rest/content_introduction): connectors, uploads, and private document operations. * [Upload your own content](/getting-started/upload_your_own_content): tags, listing documents, and using tags in search. * [List tags](/api-reference/tags/list-tags): **GET** `/contents/v1/tags`. * [List documents](/api-reference/documents/list-documents): **GET** `/contents/v1/documents`. * [Search documents](/api-reference/search/search-documents): **POST** `/v1/search`, including `my_files` and `query.filters.tag`. * [Search in uploaded files](/how-to-guides/search_in_uploaded_files): end-to-end upload and search over private content. *** ![Víctor Pimentel Naranjo](https://mintcdn.com/ravenpackinternational/Sf9MtziD0iLKyb1C/images/blog/authors/victor_pimentel_naranjo.png?fit=max\&auto=format\&n=Sf9MtziD0iLKyb1C\&q=85\&s=da0b16935d37b8a2dfb2ff903fa46ad6) Víctor Pimentel NaranjoSenior Product Manager, Team Lead # From investment theme to proprietary signal with NVIDIA Nemotron Source: https://docs.bigdata.com/blog/research/theme-to-proprietary-signal Turn a plain-language investment idea into a backtestable dataset using [Bigdata.com](https://bigdata.com) for portfolio-scale retrieval and NVIDIA Nemotron open models for private, high-throughput classification. October 21, 2026 Research, NVIDIA, Quant / 9 minutes read [![Open in GitHub](https://img.shields.io/badge/GitHub-View%20Repository-black?style=flat\&logo=github)](https://github.com/Bigdata-com/bigdata-dataset-generation) Most investment teams consume datasets built around someone else's taxonomy. Generic sentiment can describe whether an article sounds positive or negative, but it cannot answer a strategy-specific question such as: *Is this company gaining market share from competitors?* This developer workflow moves the point of differentiation upstream. Researchers define the theme, retrieve the relevant evidence across a full equity universe, distill the judgment of a large reasoning model into a smaller model, and test whether the resulting labels contain information beyond sentiment. After retrieval, the proprietary text and model workflow remain inside the firm's GPU environment. ## Key benefits * **Create differentiated data:** Define the classes and evidence rules around an investment thesis instead of adapting a generic vendor taxonomy. * **Retrieve at portfolio scale:** [Bigdata.com](https://bigdata.com) searches news and transcripts across thousands of companies. * **Keep inference private:** Run teacher labeling, fine-tuning, corpus labeling, and signal construction on infrastructure controlled by the research team. * **Reduce recurring compute:** Use the large reasoning model on a sample, then use the distilled student for full-corpus and subsequent labeling passes. * **Evaluate economic value:** Compare the classified signal with a no-label, sentiment-only benchmark under the same portfolio construction. * **Preserve auditability:** Materialize every stage as files so retrieval, labels, signals, and backtest inputs can be inspected or replaced independently. ## Architecture ![Architecture for the theme-to-signal dataset creation workflow](https://mintcdn.com/ravenpackinternational/y4JOUnbaIziFbWHb/images/blog/theme-to-signal/theme-to-signal-architecture.png?fit=max\&auto=format\&n=y4JOUnbaIziFbWHb\&q=85\&s=e654f6606c13d191bb13e0531a696b99) The architecture is powered by [Bigdata.com](https://bigdata.com), which delivers comprehensive news and transcript search across thousands of companies. Alongside a price-history data source used in backtesting, these inputs drive the workflow. All processing steps—including sentence extraction, NVIDIA Nemotron 3 Embed 1B ranking, teacher inference, LoRA supervised fine-tuning, student inference, and signal construction—take place securely within a private GPU environment. By using files to hand off data between stages, each workflow component can be rerun or updated independently, without needing to rely on a single, monolithic service. ## How the workflow works ### 1. Define the research question twice A robust pipeline needs two related but distinct specifications: 1. **Retrieval seeds** describe how the theme appears in text. For market share, examples cover both gains and losses: improving win rates, customers switching from competitors, losing shelf space, or growing slower than the market. 2. **A labeling taxonomy** defines the model's decision boundary. The market-share taxonomy maps evidence to `gainer`, `loser`, or `irrelevant` and explicitly rejects common near-misses such as absolute growth without a competitive comparison, industry-wide growth, or management plans without a realized outcome. Both live in one file, `config/data_extraction.yaml`. Abridged from the market-share run: ```yaml theme={null} theme: id: market_share classes: gainer: 1 # long loser: -1 # short irrelevant: null # no position body: | GAINER: the COMPANY is gaining ground on competitors — won share, accounts, or shelf space; win rates improved; grew faster than its market. LOSER: the COMPANY is losing ground to competitors. IRRELEVANT: anything else, including the near-misses — absolute growth with no competitive reference (GROWING IS NOT GAINING SHARE), industry- wide statements, and plans or guidance with no realized outcome. ... universe_csv: input/universe_R1K_PiT.csv search: start_date: '2016-01-01' end_date: '2026-10-01' combined_category: mode: INCLUDE values: [news_premium, transcripts] seeds: statements: positive: - Company gained share in its core market at the expense of competitors - Win rates against competitors improved meaningfully this period - Management cited evidence of competitor customers switching to their platform # ... 30 statements in total negative: - Company lost ground to competitors in its primary end market - The firm is losing shelf space and channel presence to rivals - Customer acquisition is falling behind the overall market growth rate # ... 30 statements in total ``` The `classes` block is the only link between the taxonomy and the portfolio: each actionable class carries its trading sign, and `null` means no position. Everything below it is plain English the teacher model reads as its instructions. This separation matters. Retrieval errors determine what evidence enters the corpus. Labeling errors determine what the dataset means. A downstream backtest cannot recover evidence that was never retrieved. ### 2. Retrieve and rank evidence across the universe [Bigdata.com](https://bigdata.com) searches a licensed archive of premium news and earnings-call transcripts across thousands of companies, without a query for every name in every period. [Smart Batching](/use-cases/search-service/smart-batching) plans queries around content volume and co-mentions rather than a full company-by-period grid. This implementation queries news and transcripts only, but the same search API covers the rest of the Bigdata.com content: regulatory filings, corporate communications such as press releases, expert-call interviews, podcasts, and investment research, all entity-resolved to the same company identifiers. Switching content types is a change to `combined_category` in `config/data_extraction.yaml`, so a theme whose evidence lives in filings or expert calls rather than in news can reuse the rest of the pipeline unchanged. The local pipeline then: 1. Splits retrieved passages into sentences. 2. Embeds the sentences and seed statements with NVIDIA Nemotron 3 Embed 1B. 3. Computes similarity to the theme seeds. 4. Ranks the corpus so coverage becomes an explicit research parameter. After sentence extraction, the market-share search yielded 4,226,092 passage-company records from January 2016 to October 2026. Ranking by similarity to the seeds and keeping the top 20% left 845,220 records. That 20% cut is the corpus used for labeling and the backtest. Keeping only the highest-ranked material increases precision but reduces the number of companies with an active signal. Retaining more material increases breadth while admitting weaker thematic matches. ### 3. Distill a large model's judgment into a small model Calling a large hosted model on every passage creates a recurring cost and sends proprietary text outside the research environment. The pipeline instead uses a teacher-student pattern: 1. Draw a stratified sample across the similarity distribution. 2. Label that sample locally with **NVIDIA Nemotron 3 Super 120B**. 3. Split the teacher labels into training and held-out evaluation sets. 4. Fine-tune **NVIDIA Nemotron 3 Nano 4B** with LoRA supervised fine-tuning. 5. Select a checkpoint using accuracy, macro-F1, rare-class recall, and class-count drift. 6. Use the selected student to label the ranked corpus. The implementation evaluates two text granularities. In **chunk mode**, the model reads the complete passage. In **sentence mode**, it reads the most relevant sentences first and stops when it finds actionable evidence. Chunk mode preserves context. Sentence mode attaches each signal to a more precise evidence span. In the completed market-share run, the selected 4B checkpoints reproduced the teacher's held-out labels with **94.05% accuracy and 0.831 macro-F1** at chunk level, and **95.90% accuracy and 0.866 macro-F1** at sentence level. Macro-F1 is the figure to read: `irrelevant` is the majority class, so accuracy alone flatters any classifier on this distribution. In chunk mode, weighted labeling throughput rose from **7.4 rows per second** for the teacher on the sample to **286.3 rows per second** for the student on the full corpus, roughly **38.8x faster**. Sentence mode is not directly comparable: one row can require several model calls. ### 4. Convert labels into a daily signal The theme configuration assigns a sign to each actionable class: `gainer = +1`, `loser = -1`, and `irrelevant` contributes no position. Labels are aggregated to a daily `(date, entity, signal)` table using a point-in-time cutoff. The reference implementation forms a 50/50 long/short portfolio with a 5% single-name cap and up to 160 names per leg. It also builds a **sentiment-only benchmark** from the same retrieved records, using relevance and sentiment but no custom classification labels. This is the correct baseline for determining whether the distilled taxonomy adds information. ### 5. Validate the signal, not just the classifier High held-out accuracy does not prove investment value. The workflow therefore tests two different questions: * **Predictive content:** Pooled Spearman information coefficient measures whether the signal ranks subsequent returns. * **Portfolio behavior:** Gross exposure, net exposure, active positions, turnover, return, volatility, and Information Ratio reveal whether a strong-looking result is caused by a thin or one-sided book. For the market-share study, one-day Information Coefficient was strongest at the tightest 1% retrieval cut and declined as coverage widened. However, the tightest books were too sparse to maintain balanced exposure. The selected 20% operating point trades some per-passage strength for a broader, more investable cross-section. Raw labels also fire on a single day, which leaves the book under-invested. A one-day half-life keeps a classification active briefly after it appears and is the holding period used in the table below. ## Results from the reference market-share run The comparison uses the same January 2016 to October 2026 price history for every row. The one-day half-life is applied to both classified books. The unsmoothed books are the same labels with no decay. The sentiment row is the unsmoothed benchmark built from Bigdata.com sentiment on the same retrieved records. | Signal | Information Ratio | Annualized return | Volatility | Mean gross exposure | | - | - | - | - | - | | Sentence labels, 1-day half-life | **1.18** | **8.63%** | 7.33% | 97.0% | | Chunk labels, 1-day half-life | **0.91** | **6.58%** | 7.24% | 91.7% | | Sentence labels, unsmoothed | 0.97 | 6.75% | 6.92% | 58.7% | | Chunk labels, unsmoothed | 0.95 | 5.88% | 6.19% | 47.3% | | Sentiment-only, unsmoothed | 0.41 | 3.78% | 9.31% | 99.4% | ![Cumulative excess return of the market-share books from January 2016 to October 2026](https://mintcdn.com/ravenpackinternational/y4JOUnbaIziFbWHb/images/blog/theme-to-signal/backtest-chart.png?fit=max\&auto=format\&n=y4JOUnbaIziFbWHb\&q=85\&s=4046c5c463b18f5540069d84ca349892) Solid lines are the one-day half-life. Dashed lines are the same labels unsmoothed. The gray line is the sentiment-only benchmark. A white paper with the full methodology, evaluation, and results is available on request at [research@ravenpack.com](mailto:research@ravenpack.com). ## What is included * A Jupyter notebook covering retrieval, labeling, distillation, evaluation, corpus inference, and backtesting * Theme and model configuration files * Resumable search, teacher, training, evaluation, and corpus-labeling stages * NVIDIA Nemotron 3 Embed 1B semantic ranking * NVIDIA Nemotron 3 Super 120B teacher serving * NVIDIA Nemotron 3 Nano 4B LoRA fine-tuning and local inference * Chunk-level and sentence-level labeling modes * File-based interfaces for bringing your own corpus or prices * A long/short backtest and sentiment-only benchmark * Information-coefficient and holding-period checks across coverage levels ## Tested environment * Python 3.12 or later * Docker for model serving and training containers * 2x NVIDIA H200 NVL (PCIe), serving the 120B NVFP4 teacher across both devices and later the fine-tuned 4B student. This is the configuration the reference run used, not an established minimum. * Sufficient local storage for model weights, retrieved passages, labels, and price caches * A Bigdata.com API key for retrieval * A local price cache in the documented format (daily returns per company) for the backtest The smaller student makes repeated inference materially cheaper, but the demonstrated pipeline still requires infrastructure appropriate for the teacher and fine-tuning stages. ## Software used **NVIDIA technology** * NVIDIA Nemotron 3 Embed 1B * NVIDIA Nemotron 3 Super 120B A12B NVFP4 * NVIDIA Nemotron 3 Nano 4B BF16 * CUDA-enabled local model serving and distributed training **Open-source and data components** * [Bigdata.com](https://bigdata.com) Search API and [bigdata-smart-batching](https://pypi.org/project/bigdata-smart-batching/) library * PyTorch, Transformers, PEFT, and TRL * pandas, NumPy, and PyArrow * Jupyter * A price-history dataset ## Get started Open the [dataset-generation pipeline notebook](https://github.com/Bigdata-com/bigdata-dataset-generation/blob/main/notebooks/dataset-generation-pipeline.ipynb). Two places need editing. In `config/data_extraction.yaml`: * the classes and their trading signs * the labeling taxonomy * the investable universe * positive and negative retrieval seeds * the search period In the notebook itself: * `TOP_FRACTION`, the share of the ranked corpus the student labels (0.2 in the reference run) The same pipeline can then be applied to market share, margin pressure, balance-sheet risk, or another investment question whose evidence can be defined clearly in text. ## Resources * [Bigdata dataset-generation repository](https://github.com/Bigdata-com/bigdata-dataset-generation) * [NVIDIA AI Model Distillation for Financial Data](https://build.nvidia.com/nvidia/ai-model-distillation-for-financial-data/examplecard) * [NVIDIA Quantitative Signal Discovery Agent](https://build.nvidia.com/nvidia/quantitative-signal-discovery-agent) * [Bigdata.com](https://bigdata.com) This content is for informational purposes only and does not constitute investment advice. *** ![Francisco Gomez](https://mintcdn.com/ravenpackinternational/vbcWCUrYBY4X9p0F/images/blog/authors/francisco_gomez.png?fit=max\&auto=format\&n=vbcWCUrYBY4X9p0F\&q=85\&s=19c067064fbd8d75856e7154e7dbcfd3) Francisco GomezManager of Applied Intelligence (VP) ![Francesco Ricigliano](https://mintcdn.com/ravenpackinternational/mv9Ev_PtUDxdFSGP/images/blog/authors/francesco_ricigliano.png?fit=max\&auto=format\&n=mv9Ev_PtUDxdFSGP\&q=85\&s=b368c2ce71634bd6a66dc18c2efcdb01) Francesco RiciglianoData Scientist # Baking Behavior into Weights: Fine-Tuning a Language Model for Financial Search Source: https://docs.bigdata.com/blog/search/baking-behavior-into-weights How we turned the query-planning layer behind Bigdata.com Search Smart Mode into a fast, controlled specialist, without teaching the model facts it should retrieve. August 11, 2026 Search, Smart Mode, Fine-Tuning / 15 minutes read ## Summary Prompt engineering is an effective way to discover how an AI system should behave. It becomes less attractive when the same multi-thousand-token rulebook has to be re-sent on every request. That was the situation behind [Bigdata.com Smart Mode](https://bigdata.com/resources/smart-mode-the-search-intelligence-layer-for-ai-agents), which converts natural-language financial questions into structured search plans. A prompted frontier model could do the job, but with recurring latency, cost, and control trade-offs. So we distilled the planner's stable behavior into Qwen3-1.7B, using parameter-efficient fine-tuning on tens of thousands of deliberately constructed examples, and cut the fixed prefix more than 70% by moving the rules out of the prompt and into the weights. The hard part was not choosing an open-source training library, or testing different models. It was designing the system around the model: simplifying the action space, covering the right decisions in the data, matching training and production formats, concentrating loss on the model's output, and evaluating structured behavior rather than generic language-model quality. The resulting specialist matched or slightly improved the hosted planner on this task while materially reducing latency and cost. ## The prompt had become part of the product The primary user of a search API is increasingly not a human expert. It is an AI agent, orchestrated by a frontier model, acting for an analyst, portfolio manager, or application that does not directly touch the raw API. That changes the interface problem. A human who understands a financial search system can deliberately select sources, reporting periods, document types, entities, and other constraints. A general-purpose model often defaults to the simplest available call unless the domain rules are made explicit. Consider this query: > What did Micron's latest 10-K say about Samsung? ![Query breakdown showing how different parts map to entity mention, filing entity, time anchor, and document class](https://mintcdn.com/ravenpackinternational/R6_S4eDtFMww9-5L/images/blog/fine-tuning-query-breakdown.png?fit=max\&auto=format\&n=R6_S4eDtFMww9-5L\&q=85\&s=cb9765c0c12d54e0b325245040139aca) This is not a semantic search for Samsung, Micron, and 10-K. The planner must understand that Micron is the reporting entity whose filing should be searched; Samsung is a mentioned entity expected inside it; "latest" is relative to the request time; the evidence must come from a specific document class; and the free-text query should preserve that narrow intent. Similar problems arise with fiscal calendars, source-specific requests, compound questions, ambiguous entity names, and cases where the correct outcome is no evidence at all. We initially handled this with a frontier model and extensive prompt engineering. That was the right starting point: prompts let us discover the rules quickly and refine the expected behavior without training anything. But the prompt kept growing. It accumulated instructions for entity roles, temporal interpretation, content selection, decomposition, structured output, and edge cases. The model was effectively receiving the same operating manual on every request. Beyond cost and latency, that also tied a latency-critical, high-volume path to an external dependency. At that point, the relevant question was no longer "Can prompting solve this?" It was "Which parts of this behavior should still be in the prompt?" The answer: > **Dynamic context belongs in the prompt. Stable, repeated behavior is a candidate for the weights.** ## Before fine-tuning the model, simplify its world Fine-tuning cannot compensate indefinitely for a poor interface. A single universal search action with a large schema forces the model to solve several problems at once: infer the required content, identify the relevant parameters, ignore unrelated fields, formulate the query appropriately, and produce valid structure. We first exposed a compact set of purpose-specific search capabilities, or search tools, with clean responsibility boundaries. Different content families require different filters and query-construction strategies, and collapsing them into one overloaded action is a false economy. For example, we naturally separated a search tool for earnings calls and corporate events from a tool for analyst and broker research. Separating them reduced the decision space, shortened capability descriptions, lowered schema ambiguity, simplified training examples, and made failures easier to diagnose. The principle is broader than search: > **Do not give a language model an unnecessarily complicated interface and then compensate with a longer prompt. Reduce the action space first.** Cleaner tools do not automatically make the prompt shorter — more schemas can even add tokens. What they change is the structure of the problem: each capability carries one responsibility and a focused description, so every planning decision becomes more separable, easier to demonstrate in data, and easier to score. That separability is the precondition for fine-tuning, which is where the prompt really gets smaller. ## Fine-tune behavior; retrieve knowledge We did not fine-tune the model to memorize financial facts. Facts change, and they must remain current and attributable through retrieval. We specialized the decisions that are stable: * Interpreting the information need * Distinguishing reporting entities from entities merely mentioned * Resolving financial and relative periods * Selecting the appropriate capability * Decomposing compound requests * Producing valid structured plans * Avoiding unsupported or unnecessary actions Retrieval supplies the evidence. Fine-tuning teaches the model how to ask for it. For this query-planning task, a compact specialist was a better fit than repeatedly instructing a general-purpose model. The behavior is bounded, repeated, measurable, and latency-sensitive, and our evaluation showed that the smaller model could take on that role without sacrificing quality. Frontier models remain the right tool for open-ended reasoning and synthesis; our specialist sits between the frontier model (or a human) and the retrieval system. They are meant to coexist. ## The real engineering work was the training distribution For a model specialized on producing structured outputs with many dimensions, data quality is less about raw volume than about decision coverage. A large collection of similar queries can still produce a brittle model. Strict privacy and confidentiality policies mean that we do not use or inspect production query logs for training. Instead, we construct representative requests from domain expertise, client-informed use cases, trusted structured data, and controlled synthetic generation. This gives us deliberate coverage of realistic financial research patterns without relying on sensitive user data. We designed the dataset as an explicit coverage matrix rather than a corpus. Its dimensions span content intent (which tool, document type, source…), entity roles, temporal forms, entity multiplicity, explicit and implicit constraints, complete versus misleading hints, single-part versus compound questions, free-text versus primarily structural retrieval, and negative cases where no search action should be produced. The cross-product yields a few dozen distinct cells, each a category the planner must handle, and we built tens of thousands of examples across that space. The payoff of an explicit matrix is a closed loop: because every example carries its cell, evaluation produces per-dimension roll-ups. When the model underperforms, we see which cell is failing and we can generate targeted data for exactly that cell in the next iteration, instead of guessing. ### Behavioral distillation, with control A larger teacher model, Claude Sonnet 4.6, played two roles: generating realistic linguistic variation for each cell of the matrix, and producing the reference structured outputs the student learned to reproduce. Qwen3-1.7B was trained on those completed examples through supervised fine-tuning. More precisely, this was sequence-level behavioral distillation. We did not transfer hidden states or logits; we used a capable model to demonstrate the behavior and trained the smaller model to reproduce it. Synthetic data did not mean unconstrained model-generated data. The process combined expert-defined intents, trusted entities and periods, teacher-generated language variation, deterministic validation of structured labels, and human review of difficult slices. Fluent examples are easy to generate. A balanced distribution of correct decisions is much harder. ## Match the inference context Our first fine-tuning pass treated the task as a bare mapping: a natural-language query in, a structured payload out. At inference time, however, the model was always invoked with a concise system prompt defining its role, describing the available capabilities, and setting a small number of operating constraints. The target behavior had not changed, but the conditioning context had. That mismatch was not necessarily obvious from training loss alone, yet it became visible during inference-time evaluation. We therefore rebuilt the training examples with the same simplified system context, message structure, and output contract used at inference. For a compact specialist, the prompt is not merely deployment scaffolding. It is part of the task the model learns. Train–serve consistency therefore means matching the complete interaction format, not only the query and its expected payload. We also used the dataset to teach conservative planning: the model should not invent constraints or split a focused request into redundant calls. In structured search, precision depends as much on what the planner leaves unspecified as on what it adds. ## Training the decision, not the boilerplate We fine-tuned Qwen3-1.7B using LoRA/QLoRA (rank-32 adapters on the attention and MLP projections) with Unsloth on AWS SageMaker. Parameter-efficient fine-tuning was sufficient because the goal was behavioral adaptation, not relearning the model from scratch. LoRA keeps the pretrained model largely fixed and learns a small set of adaptation weights. QLoRA lowers training-memory requirements further by loading the base model in quantized form (4-bit) while training the adapters at higher precision. The job therefore fits comfortably on a single 24GB GPU (an ml.g5.xlarge / NVIDIA A10G) rather than requiring a distributed cluster. We initially evaluated a larger model variant, nevertheless, after the fine-tuning, the 1.7B model retained the task-specific quality while improving latency. The production lesson was clear: > **For a constrained task, use the smallest model that reliably clears the evaluation gates.** We used standard assistant-only supervised fine-tuning: the model received the complete system context and request format used at inference, while the loss was computed only on the teacher-generated structured payload. Because decoder-only training commonly renders the full conversation as a single token sequence, we explicitly masked the system and user tokens so that the objective remained focused on the output the model was expected to produce. ## Reproducible by construction Training data is versioned (DVC) alongside the code; every run logs its exact data snapshot, hyperparameters, and metrics to MLflow and registers the resulting adapter in a model registry tagged with its evaluation results. Promotion between environments is metric-gated: a candidate must beat the current champion on the held-out evaluation set before it can advance. The full lifecycle, build → train → validate → quantize → deploy, runs as CI. Any deployed version traces back to its code revision, data snapshot, and eval results, which is what turns a one-off fine-tune into a capability the product can retrain on demand. ## Evaluate the payload and the evidence separately Training loss tells us whether the model is becoming better at reproducing its labels. It does not tell us whether the resulting search plan is operationally correct, or whether executing that plan retrieves evidence that can support an answer. We therefore evaluate the system at two distinct points of responsibility: the plan produced by the model (a schema of the search tools to be used) and the evidence returned by search. ### Payload evaluation: did the model build the right search plan? Before executing search, we compare the model's structured output directly with a reference payload. This isolates the behavior learned during fine-tuning from the retrieval system that runs afterwards. | Check | What it verifies | | - | - | | Structure | Does the output parse as a valid structured search payload? | | Search capability | Did the model choose the same tools, or combination of tools, as the reference? | | Number of calls | Did it generate the expected number of search calls for each tool? | | Entity resolution | Did it extract the right entities? | | Free-text search | Did it correctly decide whether free-text search terms were needed? | | Complete plan | Were the selected capabilities, number of calls, and full payload structure all correct? | These checks deliberately separate partial correctness from full correctness, which helps to distinguish failure modes. The free-text check, for instance, covers the decision to include query text, not its exact wording; placement- and structure-sensitive details are captured by the complete-plan check. Direct payload comparison is intentionally strict. A wrong filter can sometimes retrieve a chunk that happens to satisfy the request. That may look successful downstream, but it remains a planning error. We also review the checks across query types so a strong aggregate result does not hide a concentrated weakness. ### Retrieval evaluation: did the plan return usable evidence? We then execute the payload and evaluate the returned chunks. This answers a different question: whether the search system ultimately surfaced evidence that satisfied the information need. | Check | What it verifies | | - | - | | Constraint alignment | The returned chunks match the required reporting entity, mentioned entity, period, source, document type, document subtype… | | Passage answerability | The returned passage directly answers, or materially supports, the question. | | Correct zero-result behavior | When no evidence exists within the requested scope, the system returns no result rather than a plausible near-match. | Constraint alignment and zero-result behavior are evaluated deterministically. Passage answerability uses the same narrow 1–4 rubric we developed for FinSearch-Eval (coming soon): | Score | Name | Description | | - | - | - | | 4 | Direct evidence | The passage answers the question directly and can ground the response. | | 3 | Useful evidence | It materially supports the answer but is incomplete. | | 2 | Weak evidence | It is related, but tangential or insufficient. | | 1 | Unsupported | It does not support the answer. In the combined evaluation, evidence that violates a required structural constraint is also treated as a failure. | This evaluation works at the chunk level because external search providers generally do not expose the filters, query transformations, or internal plans they applied. The evidence they return is the only output that can be compared consistently. That makes it an outcome evaluation, whereas the payload evaluation above is a direct and stricter test of the fine-tuned planner itself. The two layers are complementary rather than interchangeable. A plan can be structurally wrong and still get lucky in retrieval; a plan can also be correct while corpus coverage, ranking, or passage selection fails. Evaluating both lets us distinguish a model-generation error from a search-system error. A valid request for mentions of Tesla in a Coca-Cola earnings call, for example, should still produce the correct targeted payload. If the transcript contains no such mention, zero returned chunks are the correct retrieval outcome. The full retrieval evaluation methodology will be published as FinSearch-Eval. ## Results: three views of the same change No single metric captures the whole result. We looked at the model from three angles: what fine-tuning changed relative to the same base model, how the specialist compared with the prompted frontier model planner, and what the change meant end to end for retrieval quality, latency, and cost. ### What fine-tuning changed in the base model We first compared the fine-tuned model with the original Qwen3-1.7B using the same compact inference context on the held-out payload evaluation set. | Check | Base Qwen3-1.7B | Fine-tuned Qwen3-1.7B | | - | - | - | | Valid structured output | 70.0% | 100% | | Correct search capability | 10.0% | 96.7% | | Correct number of calls | 6.7% | 96.7% | | Correct entity resolution | 30.0% | 93.3% | | Correct use of free-text search | 56.7% | 100% | | Complete plan | 3.3% | 96.7% | The out of the box model could sometimes produce a valid payload, but it rarely made the full set of planning decisions correctly. After fine-tuning, it reached 96.7% on the strict complete-plan check, which requires the selected capabilities, call count, arguments, and payload structure to match the reference. The compact prompt still supplied the runtime context; the weights now carried most of the stable planning behavior that previously had to be expressed through a much longer instruction set. ### How the specialist compared with the prompted production planner We then compared selected argument families against Claude Haiku 4.5 running the full production prompt. This is a separate field-level view of payload accuracy. The company check in the previous table asks whether the expected companies were extracted; here, reporting and mentioned companies are scored as separate fields, making their placement in the payload visible. | Structured argument | Fine-tuned Qwen3-1.7B | Prompted Claude Haiku 4.5 | | - | - | - | | Content selection | 90.9% | 86.3% | | Reporting company | 89.3% | 83.3% | | Mentioned company | 83.5% | 81.4% | | Reporting period | 92.4% | 87.0% | The specialist matched or exceeded the prompted planner across all four fields, with improvements ranging from 2.1 to 6.0 percentage points. These are exact-match checks on the generated payload fields, not chunk-level FinSearch-Eval scores. ### What changed end to end Finally, we executed the generated plans and compared retrieval without model-based planning, the prompted hosted planner, and the fine-tuned specialist. | Metric | Retrieval only | Prompted Claude Haiku 4.5 | Fine-tuned Qwen3-1.7B | | - | - | - | - | | End-to-end evidence quality, 1–4 | 2.09 | 3.11 | 3.22 | | Median latency | 0.5 s | 4.1 s | 1.7 s | | Cost per 1,000 planned queries | — | C | \~C/40\* | > [!NOTE] > > \*The self-hosted figure where we observe a 1/40 cost reduction is amortized under a scenario of \~10 million requests per month. It is not a universal unit price and will vary with utilization and infrastructure. These results should be read narrowly. They do not show that a 1.7B model is generally more capable than a frontier model. They show that a compact model, trained on a well-defined decision distribution, can be better suited to a constrained production task than a general-purpose model repeatedly instructed through a long prompt. ## Serving is part of the model design Fine-tuning improved planning quality and allowed us to move to a compact model, but model size alone does not determine production latency. The serving stack has to match the shape of the workload. After training, we merged the LoRA adapter into Qwen3-1.7B and served the results with vLLM on a SageMaker endpoint. vLLM exposed the same OpenAI-compatible tool-calling contract used by the rest of Smart Mode, so adopting the specialist required no change to the application interface. Planner requests have a distinctive inference profile: a largely identical prefix containing the system context and capability definitions, followed by request-specific hints and the natural-language question, with a comparatively short structured payload as output. That creates two clear sources of avoidable work: recomputing the shared prefix for every request, and repeatedly moving model weights through memory during generation. We addressed them separately. **Prefix caching** reduced repeated prefill work. vLLM reused the key-value cache for the identical system context and capability definitions instead of recomputing that prefix for each warm request. When the common prefix is tokenized identically and its cache blocks remain resident, the engine reuses the corresponding attention state rather than repeating the full prefill. The shared tokens are not literally free, the cache consumes GPU memory, entries can be evicted, and any change to the tokenized prefix limits reuse — but under steady traffic it removes a substantial amount of work before generation begins. **AWQ W4A16 quantization** reduced weight memory and bandwidth. We produced a post-training AWQ variant from the merged model, storing weights at 4-bit precision while keeping activations at 16-bit. This is distinct from QLoRA: QLoRA reduced memory while training the adapters, whereas AWQ compressed the final inference artifact. AWQ uses calibration activations to decide which weight channels to preserve, so we calibrated on a few hundred representative examples drawn from the same system-context, hint, and structured-payload distribution the planner serves, rather than on generic text. Quantization was therefore a post-training optimization of the same specialist, not a separate modeling project, and, on the same GPU, it buys throughput headroom that converts directly into cost. The broader lesson is that model size, prompt structure, cache reuse, and quantization are coupled. Fine-tuning made the behavior compact; inference engineering ensured that the specialization also produced practical latency and cost gains. ## Conclusions Prompt engineering was the discovery phase of this project. It made the domain rules explicit, exposed failure modes, and gave us a reference planner. Once those rules stabilized, the prompt was no longer merely runtime context. Together with the output contract and evaluation set, it had become a specification for a specialist model. This pattern is strongest when the same decisions recur frequently, change slowly, operate within a constrained action space, can be represented with validated examples, and can be scored directly. Fine-tuning is a weaker fit when the behavior is open-ended, changes continuously, or cannot be evaluated independently. The model is only one part of the transition. In Smart Mode, purpose-specific capabilities reduced the decision space; coverage-driven data represented the combinations the planner had to learn; the complete inference context was included during training; payload-level and retrieval-level evaluations kept failures attributable; and serving optimizations converted specialization into latency and cost gains. The result was not a model with more financial knowledge—retrieval still supplies the evidence. It was a compact model that performed one stable production behavior more consistently and with less repeated instruction. > **That is where fine-tuning earns its place: when a mature prompt-defined workflow can be turned into a capability the product can test, version, and operate directly.** *** ## About Smart Mode [Bigdata.com Smart Mode](https://bigdata.com/resources/smart-mode-the-search-intelligence-layer-for-ai-agents) turns natural-language financial questions into precise search over a large live corpus of news, earnings transcripts, SEC filings, analyst research, expert network calls, the web, and more. The fine-tuned planner described here is one component of that system. Explore Bigdata.com or review the [developer documentation](/getting-started/search/overview) to build search and retrieval into your own agents. *** ![Hugo Jiménez Muñoz](https://mintcdn.com/ravenpackinternational/R6_S4eDtFMww9-5L/images/blog/authors/hugo_jimenez_munoz.png?fit=max\&auto=format\&n=R6_S4eDtFMww9-5L\&q=85\&s=e85a6ca127d05fbc90ac212af9effc2e) Hugo Jiménez MuñozSenior Machine Learning Engineer ![Ricard Matas Navarro](https://mintcdn.com/ravenpackinternational/Sf9MtziD0iLKyb1C/images/blog/authors/ricard_matas_navarro.png?fit=max\&auto=format\&n=Sf9MtziD0iLKyb1C\&q=85\&s=71bc816a611fccb91e80bc7e9172daf5) Ricard Matas NavarroDirector of Search & Retrieval (SVP) # Introducing Diversified Search for AI Agents Source: https://docs.bigdata.com/blog/search/diversified-search March 23, 2026 Search, Diversification / 8 minutes read If you look at the current landscape of AI search APIs, most are heavily optimized for one metric: semantic and/or lexical relevance. But in the real world of enterprise and financial research, optimizing strictly for these introduces an unintended consequence: **contextual homogeneity**. When naïve relevance scoring dictates the top results, retrieved text chunks tend to cluster around a few sources or repeatedly surface the exact same angle on a story. For AI agents and RAG pipelines relying on these top 20 chunks to generate answers, this "monopolization" restricts perspective, creates redundant context processing, and wastes valuable LLM tokens on overlapping information. At bigdata.com we use a multidimensional search system that does not only look at textual similarity. It looks at entity grounding, source authority, freshness, between others, and computes relevance taking into account all those factors. These factors improve the accuracy and quality of the results, but do not directly address the contextual homogeneity problem. In fact, sometimes it can cause more homogeneity, for example by focusing too strictly on the top authoritative sources. A classic engineering trade-off we had to solve: boosting premium tier-1 publishers sometimes collapsed the diversity. Today, we are thrilled to announce a major upgrade to the bigdata.com API: **Diversified Search**. Diversified Search intelligently balances results across a wider array of premium sources and distinct viewpoints at the exact moment of retrieval. It preserves high-authority prominence while actively eliminating both single-source bias and semantically redundant information. While traditional consumer search engines have probably spent a lot of time refining complex, proprietary diversification algorithms, the current ecosystem of AI-native search APIs often lacks this depth at the retrieval layer. Instead of building variety into the core retrieval algorithm, many modern AI search providers attempt to solve the problem by "brute-forcing" it. They automate the generation of multiple sub-queries to explore a topic from different angles, for example, generating 3 to 5 distinct query variations and running them in parallel. This is a perfectly valid technique for expanding recall, but it is fundamentally not the optimal tool for achieving true variety. Not only increases latency and compute cost, but it is really not addressing the homogeneity problem and parallel query variations often end up retrieving very similar content. At bigdata.com, we believe that baseline retrieval itself must be inherently diverse. You shouldn't have to run multiple queries just to get a balanced set of results. This is why we build diversification in the core of our search. ## Diversification in Action To illustrate the practical impact, let's look at a couple real queries from our evaluation where diversification made a dramatic difference. ![Diversification Example 1](https://mintcdn.com/ravenpackinternational/Kw6XuFnfsJbg1fNo/images/example_1_diversification.png?fit=max\&auto=format\&n=Kw6XuFnfsJbg1fNo\&q=85\&s=f8f3b1825c0daf187ac3ad06f1a0f528) ![Diversification Example 2](https://mintcdn.com/ravenpackinternational/Kw6XuFnfsJbg1fNo/images/example_2_diversification.png?fit=max\&auto=format\&n=Kw6XuFnfsJbg1fNo\&q=85\&s=084a43d358d8065baa0e19ba09bae481) ## What's Happening Under the Hood? Building a search engine that understands both relevance and variety natively requires a sophisticated pipeline. The diagram below represents the additional steps that diversified search (right) performs when compared to the normal search process (left). ![Diversified Search Architecture](https://mintcdn.com/ravenpackinternational/XMzOlv9TKnergcpu/images/under_the_hood_diversification.png?fit=max\&auto=format\&n=XMzOlv9TKnergcpu\&q=85\&s=d6d350467fa25f5e390af986e449a09f) ### Ranking Phase with Source Grouping Selected candidate results are initially grouped by content sources, after first phase ranking. ### Controlled Relevance Decay These grouped candidates are blended using a controlled "team-draft" style selection process, forcing a decay in relevance scoring for subsequent chunks from the same sources to prevent top-slot domination. ### Embedding Deduplication We reduce the pool of candidates and remove near-duplicates using dense embeddings. ### Reranking with Diversification Penalties The final top chunk selection algorithm integrates the multidimensional reranking (cross-encoder fused with other factors such as freshness and source authority) while actively penalizing redundant content and single-source clustering. The result is a highly curated, deeply relevant, and broadly informative context window for your AI, delivered instantly. ## The Data: Diversification ON vs. OFF To measure the exact impact of this feature, we ran a comparative evaluation using hundreds of queries, evaluating the Top 20 retrieved chunks with Diversified Search turned OFF (Baseline) versus ON. The results show a significant improvement in breadth without sacrificing quality or speed: | Metric | Result | | :- | :- | | **Unique Sources** | +40% increase — the specific origins of the content are significantly more varied | | **Content Similarity** | Decreased — we observe a decrease in average chunk-to-chunk similarity (measured via both semantic and lexical overlap). Your LLM is fed more distinct information with every token, rather than paraphrased repetitions | | **Source Quality** | Zero drop — the average Source Authority Rank remained stable. Premium, Tier-1 financial sources still surface to the top; they just don't flood the results | | **Latency** | Zero impact — our native diversification happens with no meaningful impact on response times | Beyond these raw retrieval metrics, we used an LLM-as-a-judge to evaluate the relevance of these Top 20 results when addressing the query. In more than **60% of cases**, diversified results were rated as having better relevance compared to the baseline. In 20% the results showed equivalent relevance, while in the remaining 20% the baseline had a higher relevance. When looking in detail at this latter scenario, it was clear that queries where diversification did not improve the relevance of the results were specific, often factual, queries, for which diversification forced some unnecessary deviation from the targeted content that answered the question. We aren't trading relevance for diversity; we are improving both. ## How to Start Using It Diversification allows us to shape a higher-quality result set, preserving strong relevance while drastically improving balance, authority, and perspective. It is available right now for all API users and **enabled by default** in both Fast Mode and Smart Mode. No changes to your existing API calls are needed — you will automatically benefit from diversified results. If you need the previous non-diversified behavior for a specific use case, you can opt out by setting the parameter explicitly: ```json theme={null} { "ranking_params": { "content_diversification": { "enabled": false } } } ``` Alternatively, just take advantage of our **Smart Mode** that optimizes the search queries automatically, including the use of diversification. Test Diversified Search in your own workflows and let us know how it impacts your agent's performance. We'd love to hear your feedback! *** ![Sandra Bullón](https://mintcdn.com/ravenpackinternational/Sf9MtziD0iLKyb1C/images/blog/authors/sandra_bullon.png?fit=max\&auto=format\&n=Sf9MtziD0iLKyb1C\&q=85\&s=07d732e5dc1db392ec78b7061855d58b) Sandra BullónSenior Product Manager ![Ricard Matas Navarro](https://mintcdn.com/ravenpackinternational/Sf9MtziD0iLKyb1C/images/blog/authors/ricard_matas_navarro.png?fit=max\&auto=format\&n=Sf9MtziD0iLKyb1C\&q=85\&s=71bc816a611fccb91e80bc7e9172daf5) Ricard Matas NavarroDirector of Search & Retrieval (SVP) # Unifying Web Search within the Bigdata.com Search API Source: https://docs.bigdata.com/blog/search/introducing-web-search May 5, 2026 Search, Web Search, External Search / 10 minutes read **One API for premium financial content, the live web, and your own data** ![From fragmented retrieval to one search layer](https://mintcdn.com/ravenpackinternational/C2q3BNyO1yuF93WB/images/blog/web-search/web_start_infog.png?fit=max\&auto=format\&n=C2q3BNyO1yuF93WB\&q=85\&s=f896c1ba93a17b18281fabd8bb39a600) Today, we're introducing Web Search in the Bigdata.com Search API. With a single request, developers can now retrieve live web content alongside Bigdata.com's premium financial content, using the same API, the same response schema, and the same commercial relationship. This release is about simplifying retrieval for financial AI applications. Most production systems do not rely on one corpus. They pull from a mix of sources: news, filings, transcripts, podcasts, expert interviews, investment research, public web pages, and increasingly, customer-owned documents and feeds. In practice, each source usually comes with a different provider, different request payloads, different tool instructions, different billing, and different output formats. That fragmentation adds cost in more places than one. Engineering teams have to normalize schemas and merge results. Procurement teams have to onboard more vendors. And LLM-powered workflows have to reason across multiple search tools, each with its own instructions and response structure. Unified Web Search moves that complexity behind a single retrieval layer. ## Why use Bigdata.com? ### Broader coverage in one call Bigdata.com already provides access to high-quality financial content across news, filings, transcripts, podcasts, expert interviews, investment research, and more. Developers can also bring their own content into the same environment by uploading files such as PDFs or connecting feeds such as email and Sharepoint. Web Search adds live web results to that system. The result is broader coverage across complementary sources in one call, without forcing developers to stitch them together themselves. This release is part of a larger direction for Bigdata.com: one platform that keeps getting broader with all types of content, soon also with live social signals as well. ### One schema across sources Whether content comes from Bigdata.com's premium corpus, the live web, or customer-provided data, the API returns a consistent structure. That reduces application-side normalization, simplifies ranking and post-processing, and makes it easier to build stable downstream workflows. ### Less orchestration, less token waste Multiple search tools create hidden costs. Each one needs its own request payload, its own tool description, its own response parser, and its own merge logic. In LLM-powered systems, that means extra tokens spent on tool selection and output reconciliation before the model even starts doing the real work. With Bigdata.com, developers can expose one retrieval surface instead of several. That makes downstream applications simpler, more predictable, and more efficient in how they use context. Moreover, when models have to choose across multiple retrieval tools, they can misfire: calling the wrong tool, over-calling several tools, or handling sources inconsistently because each provider behaves differently. A single API surface reduces that operational ambiguity. Developers still get access to multiple underlying sources, but the application and the model interact with one consistent interface. ### One commercial relationship Adding live web coverage through Bigdata.com means teams can expand retrieval without standing up a separate vendor evaluation, new security review, or another billing path. That matters in production just as much as it does in prototyping. ### Clear provenance Bigdata.com Web Search is opt-in. Existing queries stay unchanged unless web is enabled explicitly in the external search. And live web results are returned separately in `external_results.web`, so applications can preserve strict provenance while still working with a normalized schema. ## Complementary coverage matters Premium financial content and live web search are not interchangeable. They often contribute different kinds of signal. Bigdata.com's premium corpus is strongest on decision-grade financial content: filings, transcripts, structured disclosures, institutional research, expert interviews, financial events and major news. Live web search is strongest on long-tail coverage: company websites, scientific and technical pages, regional reporting, niche publications, and early-stage company footprints. Especially useful for very early-stage companies with no analyst coverage. Used together, they widen the context available to downstream applications. Premium content explains how the market interprets a company or theme. The web shows what companies are publishing, building, testing, and updating in real time. The examples below show how the two sources work together in practice: ![Example: Companies investing in AI drug discovery](https://mintcdn.com/ravenpackinternational/C2q3BNyO1yuF93WB/images/blog/web-search/search-comparison_1b.png?fit=max\&auto=format\&n=C2q3BNyO1yuF93WB\&q=85\&s=d7d173603a52ba66f12047c90f093265) ![Example: Chinese EV expansion in Southeast Asia](https://mintcdn.com/ravenpackinternational/C2q3BNyO1yuF93WB/images/blog/web-search/search-comparison_3b.png?fit=max\&auto=format\&n=C2q3BNyO1yuF93WB\&q=85\&s=a1e0c3c7d07b26774bb7fe8eba3f9464) ![Example: Supply chain bottlenecks and sourcing alternatives for battery-grade lithium processing](https://mintcdn.com/ravenpackinternational/C2q3BNyO1yuF93WB/images/blog/web-search/search-comparison.png?fit=max\&auto=format\&n=C2q3BNyO1yuF93WB\&q=85\&s=0346f46dfbccf95b2add4ad227d2dc14) Across these examples, premium content surfaces institutionally visible signals, while live web adds primary-source detail, local execution, and long-tail coverage. Together, they connect market narrative to on-the-ground reality, in one call, with one schema. ## Built for real-world financial workflows This matters beyond search. Financial applications increasingly combine external intelligence with customer-owned context. A team may want to search public market information alongside internal PDFs, research memos, email threads, or other proprietary documents. Bigdata.com already supports that model today: developers can retrieve across premium content, live web content, and customer-provided data in the same environment. That makes it easier to build research assistants, diligence workflows, monitoring systems, and agentic applications that need to reason across both public and private information. ## How to use it Bigdata.com Web Search is designed to be explicit and easy to control. Add `web` to the `external_search` payload object, either in `ONLY` or in `INCLUDE` mode, to get results only from external search or from inclusive to any other content you selected for the internal search in the payload. ```json theme={null} { "query": "latest executive changes at Revolut", "external_search": { "mode": "ONLY", "values": ["web"] } } ``` For strict provenance, live web results are returned separately in `external_results.web`: ```json theme={null} { "results": [], "external_results": { "web": [ { "headline": "Revolut files for US banking licence & names US CEO", "timestamp": "2026-03-05T23:29:30+00:00", "chunks": [ { "cnum": 1, "text": "Cetin Duransoy has been appointed as its US CEO. Duransoy most recently served as the US CEO of Raisin, a FinTech marketplace...", "relevance": 0.99338466 } ], "src_name": "businesscloud.co.uk", "url": "https://businesscloud.co.uk/news/revolut-files-us-bank-charter/" }, { "headline": "Revolut recruits ex-Visa exec as new US chief", "timestamp": "2026-03-05T12:48:42+00:00", "chunks": [ { "cnum": 1, "text": "The fintech announced that former Capital One and Visa exec Cetin Duransoy will join Revolut as US CEO...", "relevance": 0.9915216 } ], "src_name": "sifted.eu", "url": "https://sifted.eu/articles/revolut-us-banking-licence/" } ] } } ``` That means developers get a unified schema without losing clarity around source origin. ## Transparent pricing Web search is a separately billable capability, tracked separately in the usage dashboard, and is only charged when explicitly invoked. | Component | Example pricing | When it applies | | :- | :- | :- | | **Search** | \$0.015 per 10 chunks of internal data | Standard processing fee | | **External Search, Web** | +\$0.02 per request for up to 20 chunks of data | Only when `external_search.values` → `web` is enabled | This keeps existing workloads unchanged while making the cost of live web retrieval explicit and predictable. ## What's next We're already working on extending the same enrichment pipeline we use for premium content to open-web results. In upcoming releases, web search documents will be annotated with entity detections, mapped to our universal IDs and additional metadata such as real-time sentiment, making live web content easier to use inside research workflows, analytics pipelines, and LLM applications. We're also working on live social search, with X and Reddit coming next. Over time, Bigdata.com will continue adding more content APIs and connectors, so developers can access premium financial content, live web coverage, customer-owned data, and live social signals through one platform instead of a growing patchwork of tools. ![What's next: More sources, unified enrichment, AI-ready output](https://mintcdn.com/ravenpackinternational/C2q3BNyO1yuF93WB/images/blog/web-search/web_end_infog.png?fit=max\&auto=format\&n=C2q3BNyO1yuF93WB\&q=85\&s=905bc79247f22916866661a154610e18) ## Available now Web Search is available today in the Bigdata.com Search API. Add `web` inside `external_search.values` to retrieve live web content alongside Bigdata.com's financial content and your own data — through one API, one schema, and one growing content layer for financial AI. *** ![Ricard Matas Navarro](https://mintcdn.com/ravenpackinternational/Sf9MtziD0iLKyb1C/images/blog/authors/ricard_matas_navarro.png?fit=max\&auto=format\&n=Sf9MtziD0iLKyb1C\&q=85\&s=71bc816a611fccb91e80bc7e9172daf5) Ricard Matas NavarroDirector of Search & Retrieval (SVP) # Turn Buyback Signals into Alpha with Claude, Bigdata MCP, and Skills Source: https://docs.bigdata.com/blog/search/share-buyback-financial-research April 22, 2026 Hedge Funds, Claude, MCP, Skill / 10 minutes read Share buybacks are one of the most persistent and underexploited flow signals in equity markets. For hedge funds, the edge is not just knowing that a company authorized a repurchase, it is knowing *when* the flow is likely to hit the tape, how long it can persist, and what changes when that bid disappears. With **Claude**, **Bigdata MCP**, and the **Bigdata plugin**, your team can move from manual monitoring to a repeatable, research-grade workflow that detects buyback events early, tracks execution, and translates disclosure data into tradable insight. ## Buyback intelligence for hedge funds When a company is actively repurchasing shares, it often acts as a large, price-insensitive buyer. That can support price action, compress realized volatility, and create a measurable tailwind for positioning. When the company enters blackout, that structural bid may fade and relative performance can shift quickly. That is why high-performing event and fundamental teams track three things together: 1. **Announcement and authorization signals** from 8-K filings and press releases. 2. **Execution follow-through** from 10-Q/10-K repurchase tables and related disclosures. 3. **Calendar position** across open windows, blackout periods, and 10b5-1 plan activity. The teams that connect these signals faster can adjust sizing, hedging, and timing before the rest of the market fully prices the flow change. ## Build this workflow in minutes You can replicate this process today with three components: * Connect Claude to the official Bigdata connector: [Bigdata Connector for Claude](https://claude.ai/directory/connectors/bigdata) * Install the Bigdata plugin with the financial skills: * [Bigdata plugin](/skills-reference/install-bigdata-plugin) * [Financial research analyst Skills docs](/skills-reference/mcp-helpers/financial-research-analyst) ![Claude chat showing a Bigdata-connected prompt for share buyback research](https://mintcdn.com/ravenpackinternational/HNMURFSoUHBhl_yb/images/blog/share-buyback/claude_bigdata_prompt.png?fit=max\&auto=format\&n=HNMURFSoUHBhl_yb\&q=85\&s=76f146490825a468bb6d7e859201c8f0) Once the skills are available in Claude with Bigdata MCP enabled, your analysts can generate share-buyback intelligence reports from a single prompt. ## Examples **Last two months analysis** Prompt: ```text theme={null} Bigdata, I want a report on buyback announcements and events at US companies over the last two months. I want to get dates, the amount specified in the filing, and the reason given by the company ``` Output: [https://mozilla.github.io/pdf.js/web/viewer.html?file=https%3A%2F%2Fraw.githubusercontent.com%2FBigdata-com%2Fbigdata-docs-resources%2Fc5a5bb5ccbc7d8c6969d1cfc7314787532a6fca7%2Fmcp%2Freport-examples%2FUS%2520Share%2520Buyback%2520Announcements.pdf](https://mozilla.github.io/pdf.js/web/viewer.html?file=https%3A%2F%2Fraw.githubusercontent.com%2FBigdata-com%2Fbigdata-docs-resources%2Fc5a5bb5ccbc7d8c6969d1cfc7314787532a6fca7%2Fmcp%2Freport-examples%2FUS%2520Share%2520Buyback%2520Announcements.pdf) Source PDF: [US Share Buyback Announcements](https://github.com/Bigdata-com/bigdata-docs-resources/blob/c5a5bb5ccbc7d8c6969d1cfc7314787532a6fca7/mcp/report-examples/US%20Share%20Buyback%20Announcements.pdf) **General US Buyback announcements analysis** Prompt: ```text theme={null} Bigdata, create a hedge-fund focused report on US share buyback announcements. Include announcement context, likely execution windows, blackout vs open-window implications, and practical trading signals with inline attribution. ``` Output: [https://mozilla.github.io/pdf.js/web/viewer.html?file=https%3A%2F%2Fraw.githubusercontent.com%2FBigdata-com%2Fbigdata-docs-resources%2Fae6e2d7554cdd6fde910059a0ac8def4e6d64613%2Fmcp%2Freport-examples%2FUS\_Buyback\_Announcements\_HF\_Report.pdf](https://mozilla.github.io/pdf.js/web/viewer.html?file=https%3A%2F%2Fraw.githubusercontent.com%2FBigdata-com%2Fbigdata-docs-resources%2Fae6e2d7554cdd6fde910059a0ac8def4e6d64613%2Fmcp%2Freport-examples%2FUS_Buyback_Announcements_HF_Report.pdf) Source PDF: [US Buyback Announcements HF Report](https://github.com/Bigdata-com/bigdata-docs-resources/blob/ae6e2d7554cdd6fde910059a0ac8def4e6d64613/mcp/report-examples/US_Buyback_Announcements_HF_Report.pdf) ## Buybacks dates are critical **Trading windows and blackout periods shape supply-demand dynamics.** Companies cannot repurchase shares at any time. Their activity is constrained by policy, regulation, and disclosure practices: * **Blackout periods:** often the last \~4 weeks of a quarter through 1-2 days after earnings, when companies typically stop discretionary repurchases. * **Open windows:** the post-earnings period where open-market buybacks can resume. * **Rule 10b5-1 plans (US):** pre-arranged plans that may allow continued purchases during blackout windows. * **SEC Rule 10b-18 safe harbor:** practical limits on timing, price, and daily volume of repurchases. For hedge funds, this timing intelligence directly affects expected flow and risk: 1. **Flow prediction:** open windows can add a steady corporate bid; blackout can remove it. 2. **Event-driven entries/exits:** announcement dates and window re-openings can create tactical opportunities. 3. **ASR impact modeling:** accelerated share repurchases can reduce float quickly and shift microstructure. 4. **Valuation and capital-structure analysis:** buybacks influence EPS, leverage, and payout mix. ## Announcement date vs execution date ### 1. Announcement / authorization date This is when the board authorizes repurchases up to a notional amount over a time horizon. It is a **signal of intent**, not guaranteed execution. It often communicates: * perceived undervaluation, * excess capital and return-of-capital preference, * dilution offset goals, * confidence in medium-term fundamentals. ### 2. Execution dates and pace This is where conviction becomes observable behavior. Actual execution data reveals: * whether management follows through, * the pace and price levels of buying, * how much authorization remains as future support. In practice, announcements move sentiment; execution data moves confidence. ## Data sources in the US * **8-K filings:** common source for initial authorization announcements. * **10-Q and 10-K filings:** quarterly repurchase tables with shares bought, average price, and remaining authorization. * **Press releases and IR pages:** often the earliest public disclosure channel. * **Form 4 filings:** insider activity context (indirectly relevant for broader signal interpretation). A robust hedge-fund workflow monitors all of the above continuously, not just at quarter-end. ## Why Claude + Bigdata + Skills Traditional desks lose time stitching these data points manually across filings, news, and calendars. The Bigdata plugin with financial skills gives Claude a repeatable research framework, while Bigdata MCP provides the data and retrieval layer required for timely, attribution-backed outputs. The result is faster generation of consistent buyback intelligence, with less operational drag and more time spent on actual portfolio decisions. ## Production pipeline Once your team defines the flow and prompt template, you can automate the same process programmatically using the **Claude SDK** or any other **LLM model/agentic host** that supports MCP. With a Bigdata API key, the workflow can run on a schedule and feed directly into internal dashboards, alerting systems, or morning risk packets, with consistent structure across every report. To set this up, follow the MCP API-key integration guide: [MCP Integration with API Key](https://docs.bigdata.com/mcp-reference/api-integrations/mcp-api-integration). *** ![Oscar Sanchez Iglesias](https://mintcdn.com/ravenpackinternational/Sf9MtziD0iLKyb1C/images/blog/authors/oscar_sanchez_iglesias.png?fit=max\&auto=format\&n=Sf9MtziD0iLKyb1C\&q=85\&s=c9ee6739dce63b6d6b6f2e5aa36e8442) Oscar Sanchez IglesiasSenior Product Manager # Smart Batching: plan-then-execute search for universe-scale screening Source: https://docs.bigdata.com/blog/search/smart-batching-universe-scale-screening April 25, 2026 Search, API, Screening / 8 minutes read When teams screen a large universe for a theme, the first reliable pattern is usually a full grid: split the universe into manageable company groups, split the time range into windows, run Search across every slice, and collect the results. It is comprehensive, auditable, and easy to reason about. The challenge is allocation. A full grid gives every company-time slice the same request budget even though coverage is rarely distributed evenly. Some names and periods carry most of the relevant evidence. Others have little or no coverage for the theme being screened. When the universe has thousands of names, uniform allocation can turn a straightforward research question into a long-running retrieval job. Smart Batching is a planning layer for Bigdata.com Search that makes this allocation adaptive. It estimates volume before execution, groups entities with similar expected coverage, splits high-volume periods more carefully, and then runs Search with a proportional budget. The underlying retrieval remains Bigdata.com Search; Smart Batching decides where to spend the calls. In benchmarks across three universe-scale screens, this plan-then-execute pattern cut runtime from several minutes to seconds: 10.6x to 21.6x faster at the 100% planned budget, and 14.6x to 44.3x faster at the 1% budget. The highest-relevance bins were strongly preserved in the tested scenarios, while lower budget settings concentrated the output into a smaller, more focused result set. ## The Problem: Uniform Budgets Imagine a six-month screen for companies affected by US import tariffs against China. Some semiconductor, retail, industrial, and logistics names will have meaningful coverage. Many other companies will have little direct exposure in the news flow for that topic. A full-grid search handles this by being exhaustive: each company group and time window gets queried. That is often the right baseline when completeness is the priority. But if the workflow is a recurring monitor, alert generator, LLM verification queue, or dashboard refresh, you may want a way to preserve the evidence-rich parts of the universe without spending the same budget on every quiet slice. Smart Batching addresses that operational problem. It is not a replacement for Search quality. It is a planner that makes large Search jobs more aware of the distribution of expected content. ## How Smart Batching Works Smart Batching has two phases. The planning phase has two distinct substeps. ![Schematic comparing full grid search with Smart Batching planning and execution](https://mintcdn.com/ravenpackinternational/iQrJ6TX9YoceRYsJ/images/smart-batching/smart_batching_schematic.png?fit=max\&auto=format\&n=iQrJ6TX9YoceRYsJ\&q=85\&s=82729a71191df342eed87ea559345349) **Step 1a: Group entities into volume baskets.** The system queries the Bigdata.com Search Volume API to estimate how many chunks each entity will produce for a given topic over the full time range. It then groups entities into baskets based on expected volume, merging low-volume entities together and isolating high-volume entities when needed. This preserves the cross-sectional distribution of the universe: companies are not flattened into a single bucket, and the names that dominate coverage do not crowd out the long tail. **Step 1b: Break each basket down by time.** For each basket, the planner pulls the expected-volume time series and partitions the full date range into batch periods. High-volume periods get shorter windows; low-volume periods can use longer windows. This preserves temporal structure the same way Step 1a preserves cross-sectional structure: periods of heavy coverage get their own batches, and quiet periods are consolidated so the executor does not spend unnecessary requests on sparse windows. **Step 2: Execute.** The executor turns the plan into Search calls. Each basket receives a `max_chunks` allocation proportional to its expected volume and the selected `chunk_percentage`. A 10,000-chunk basket at `chunk_percentage=0.01` requests about 100 chunks. A 50-chunk basket still receives a minimum budget so it remains represented. That `chunk_percentage` setting is the main control: * `1.0` runs the full planned retrieval budget. * `0.1` is a balanced setting for research workflows and dashboards. * `0.01` is a signal-first setting for alerts, triage, and LLM verification. Together, Step 1a and Step 1b produce an upper-bound estimate of how many chunks the execution phase will retrieve before the larger Search calls are made. That makes latency and downstream processing costs easier to manage. ## Benchmark Design We tested Smart Batching against a full-grid baseline on three screens over the historical US top-company universe: * **Tariffs China 2025:** “The company is affected by US import tariffs against China,” January to June 2025. * **Leadership 2023:** “The company has faced leadership changes and executive appointments,” September to December 2023. * **Confidence Decline 2021:** “Decline in customer confidence in the company,” January to June 2021. The baseline split date ranges into windows and queried batches of 10 entities. Smart Batching used volume-based planning and was tested at 100%, 10%, and 1% planned chunk budgets. All methods used the same Bigdata.com API environment for the comparison. ## Speed: minutes to seconds ![Benchmark execution time comparison for full grid and Smart Batching](https://mintcdn.com/ravenpackinternational/iQrJ6TX9YoceRYsJ/images/smart-batching/benchmark_execution_time_comparison.png?fit=max\&auto=format\&n=iQrJ6TX9YoceRYsJ\&q=85\&s=8ed842b3d10b8ff8bd94e622f8f3b7d0) The timing results are the simplest way to see the effect. Tariffs China 2025 went from 452.2 seconds with full grid to 42.5 seconds with Smart Batching at 100%, 32.0 seconds at 10%, and 30.9 seconds at 1%. That is a 10.6x to 14.6x speedup. Leadership 2023 went from 323.2 seconds to 20.7 seconds at 100%, 10.2 seconds at 10%, and 9.4 seconds at 1%. That is a 15.6x to 34.4x speedup. Confidence Decline 2021 went from 451.6 seconds to 20.9 seconds at 100%, 11.0 seconds at 10%, and 10.2 seconds at 1%. That is a 21.6x to 44.3x speedup. For a team running daily or weekly screens, this changes the shape of the workflow. A retrieval step that used to block a pipeline for several minutes becomes a sub-minute component that can feed monitoring, dashboarding, or verification layers. ## What happens to relevance? Runtime alone is not enough. The real question is what kind of evidence remains in the result set. The scatter plots below show per-chunk relevance over time for each benchmark. The important pattern is that lower `chunk_percentage` settings reduce the volume of returned chunks and concentrate the output into higher-ranked material. At the same time, Smart Batching still samples through the planned baskets, so the result is not simply one global top-N list. ![Tariffs China 2025 date versus relevance scatter plots](https://mintcdn.com/ravenpackinternational/iQrJ6TX9YoceRYsJ/images/smart-batching/scatter_date_vs_relevance_benchmark_tariffs_china_2025.png?fit=max\&auto=format\&n=iQrJ6TX9YoceRYsJ\&q=85\&s=b37ad614d0eaa8965266330d10480d82) For Tariffs China 2025, full grid returned 81,037 chunks with average relevance of 0.049. Smart Batching returned 60,135 chunks at 100%, 10,825 at 10%, and 1,188 at 1%. Average relevance rose to 0.130 at 10% and 0.331 at 1%. ![Leadership 2023 date versus relevance scatter plots](https://mintcdn.com/ravenpackinternational/iQrJ6TX9YoceRYsJ/images/smart-batching/scatter_date_vs_relevance_benchmark_leadership_2023.png?fit=max\&auto=format\&n=iQrJ6TX9YoceRYsJ\&q=85\&s=afa91c28ea4713c895b653d6584ac551) For Leadership 2023, full grid returned 9,033 chunks with average relevance of 0.086. Smart Batching returned 7,591 chunks at 100%, 1,070 at 10%, and 100 at 1%. Average relevance rose to 0.109 at 10% and 0.273 at 1%. ![Confidence Decline 2021 date versus relevance scatter plots](https://mintcdn.com/ravenpackinternational/iQrJ6TX9YoceRYsJ/images/smart-batching/scatter_date_vs_relevance_benchmark_confidence_decline_2021.png?fit=max\&auto=format\&n=iQrJ6TX9YoceRYsJ\&q=85\&s=64b7e4f0f8437f372c4c85b85496116e) For Confidence Decline 2021, full grid returned 18,634 chunks with average relevance of 0.036. Smart Batching returned 16,923 chunks at 100%, 2,503 at 10%, and 295 at 1%. Average relevance rose to 0.046 at 10% and 0.111 at 1%. One nuance matters: the 100% Smart Batching runs can have lower average relevance than the full-grid reference because the allocation structure is different. The goal of 100% Smart Batching is efficient planned coverage, not necessarily a higher average relevance score. The higher-concentration effect becomes clearest when the budget is intentionally reduced. ## Coverage: What You Keep The coverage charts compare Smart Batching against the full-grid reference by relevance bin. They show how much of each reference bin appears in the Smart Batching output. ![Tariffs China 2025 coverage by relevance range](https://mintcdn.com/ravenpackinternational/iQrJ6TX9YoceRYsJ/images/smart-batching/chunks_coverage_by_range_benchmark_tariffs_china_2025.png?fit=max\&auto=format\&n=iQrJ6TX9YoceRYsJ\&q=85\&s=34d02c50cc0266cdb4765cc1a7cc7dee) In the tariff screen, Smart Batching preserved nearly all chunks in the highest reference bins at 10% and 1%, while aggressively reducing the lowest-relevance bin at smaller budgets. ![Leadership 2023 coverage by relevance range](https://mintcdn.com/ravenpackinternational/iQrJ6TX9YoceRYsJ/images/smart-batching/chunks_coverage_by_range_benchmark_leadership_2023.png?fit=max\&auto=format\&n=iQrJ6TX9YoceRYsJ\&q=85\&s=5e143c8381660ccc770f2b50433ad612) In the leadership screen, the upper relevance bins were also strongly retained. The 1% setting returned only 100 chunks, but still preserved the highest reference bin in the benchmark. ![Confidence Decline 2021 coverage by relevance range](https://mintcdn.com/ravenpackinternational/iQrJ6TX9YoceRYsJ/images/smart-batching/chunks_coverage_by_range_benchmark_confidence_decline_2021.png?fit=max\&auto=format\&n=iQrJ6TX9YoceRYsJ\&q=85\&s=4e8fb390fda56ffde43a6820c4d19bdc) In the confidence-decline screen, which was the sparsest topic, the 10% setting preserved the mid and upper reference bins in the chart while reducing the lowest bin sharply. The 1% setting was much more selective, which is useful for triage but may be too aggressive for workflows that need broad moderate-relevance coverage. ## Choosing a budget The best `chunk_percentage` depends on what happens downstream. Use 1% when the output feeds a human triage queue, alerting workflow, or LLM verification step and the goal is to start with the highest-conviction evidence. Use 10% when you are building entity-level scores, dashboards, or time-series views and want a stronger balance between speed, relevance, and coverage. Use 100% when completeness matters and you still want the operational benefit of planned grouping and volume-aware splitting. ## Get Started The full implementation is available in the [Smart Batching folder](https://github.com/Bigdata-com/bigdata-cookbook/tree/main/Smart_Batching) of the Bigdata.com cookbook repository, including the [technical notebook](https://github.com/Bigdata-com/bigdata-cookbook/blob/main/Smart_Batching/test_smart_batching.ipynb). The reusable Python library is published on PyPI as [`bigdata-smart-batching`](https://pypi.org/project/bigdata-smart-batching/). The notebook walks through environment setup, planning, execution, deduplication, plan persistence, and DataFrame conversion using the `bigdata-smart-batching` Python package. If you are integrating it into your own project, install it with `uv add bigdata-smart-batching`. For teams already running large Search workflows on Bigdata.com, Smart Batching is a drop-in pattern for making those workflows more predictable: plan first, choose a budget, execute only what the research task requires, and keep the result grounded in Bigdata.com Search. This content is for informational purposes only and does not constitute investment advice. *** ![Alberto Martinez](https://mintcdn.com/ravenpackinternational/vbcWCUrYBY4X9p0F/images/blog/authors/alberto_martinez.png?fit=max\&auto=format\&n=vbcWCUrYBY4X9p0F\&q=85\&s=6d6cf9a32bbf2f816182763d4658a5b3) Alberto MartinezSenior Data Engineer ![Francisco Gomez](https://mintcdn.com/ravenpackinternational/vbcWCUrYBY4X9p0F/images/blog/authors/francisco_gomez.png?fit=max\&auto=format\&n=vbcWCUrYBY4X9p0F\&q=85\&s=19c067064fbd8d75856e7154e7dbcfd3) Francisco GomezManager of Applied Intelligence (VP) # From API key to production AI research assistant - in a single afternoon Source: https://docs.bigdata.com/blog/widgets/widgets_release May 22, 2026 Widgets, Chat Widget, Document Viewer / 5 minutes read **Introducing Bigdata Widgets: a one-line embed that puts our Research Agent inside any app, on any stack, with your brand on it.** *** Every team building with AI eventually hits the same wall. You have the model. You have the data. You have the API key. And then you spend the next six weeks wiring up streaming, building a citation UI, taming markdown rendering, debugging token-by-token updates, and arguing about whether the chat bubbles should be rounded on all four corners or just three. The interesting part of the work (the research, the insights, the answers your users actually came for) gets buried under frontend plumbing. We built the **Chat Widget** and the **Document Viewer Widget** to remove that wall entirely. | Chat Widget | Document Viewer Widget | | :- | :- | | Embed a fully functional AI research assistant into any web application with a single code snippet. Handles streaming, attribution, and branding out of the box. | Embed rich document experiences that load sources cited by the Research Agent, scroll to the exact referenced paragraph, and highlight the relevant content automatically. | ## The Chat Widget The Chat Widget makes Bigdata's Research Agent embeddable in any web application with a single code snippet. Drop a `
`, drop a ` ``` **3. Initialize the widget** Create the widget with `new BigdataAgent.BigdataChatWidget({ ... })`. At minimum set `container` and `proxyUrl`. Optional `config`, `theme`, and `branding` match the [configuration reference](#configuration-reference) below. ```html theme={null} ``` ## Global API: `BigdataAgent` After the bundle loads, the UMD global **`BigdataAgent`** exposes **`BigdataChatWidget`**. Instantiate it with `new BigdataAgent.BigdataChatWidget(config)`; the return value is the mounted widget instance. | Property / method | Type | Description | | - | - | - | | `BigdataAgent.BigdataChatWidget` | `class` | Constructor: pass the same `config` object described in [Configuration reference](#configuration-reference). | | `BigdataChat.version` | `string` | On `window`: build version string (diagnostics). | | `BigdataChat.buildTimestamp` | `string` | On `window`: build timestamp string (diagnostics). | Store the instance from `new BigdataAgent.BigdataChatWidget(...)` if you need to call `destroy()` or attach listeners. ## Proxy contract (summary) | Aspect | Expected behavior | | - | - | | HTTP method | **POST** to `proxyUrl` | | Request `Content-Type` | `application/json` | | Request body | JSON including at least `chat_id`, `message`, `research_effort`, `tools_configs`, and `from_interaction_id` (the widget generates these fields; align your proxy with the [Research Agent API](/api-reference/research-agent/research-agent)). | | Response | `Content-Type: text/event-stream` with SSE `data:` frames the widget can parse | If streaming appears only after the full response completes, your stack may be **buffering** the response; the proxy and any reverse proxy must support streaming. ## Configuration reference ### Top-level options | Option | Type | Description | | - | - | - | | `container` | `string` or `HTMLElement` | **Required.** Where the widget mounts: a CSS selector that `document.querySelector` resolves, or a DOM element. | | `proxyUrl` | `string` | **Required.** Your backend URL that forwards requests to the Research Agent. | | `instanceId` | `string` | Optional. Stable id for this instance when several widgets share a page; if omitted, the implementation uses `"chat"`. | | `manager` | `BigdataWidgetManager` | Optional. Shared manager across widgets for coordinated events and a common base theme. | | `config` | `object` | Optional. Research and UX settings; see **`config` (research and UX)** below. | | `theme` | `object` | Optional. Visual settings; see **`theme`** below. | | `branding` | `object` | Optional. Footer branding; see **`branding`** below. | ### `config` (research and UX) | Field | Type | Description | | - | - | - | | `researchEffort` | `"lite"` \| `"standard"` | Depth / execution mode for agent runs; defaults to `"standard"` if omitted. | | `suggestedQuestions` | `string[]` | Optional starter prompts in the UI. | | `showAdvancedControls` | `boolean` | Shows advanced input controls when `true`. | | `inlineAttribution` | `boolean` | Inline attribution behavior when supported. | | `placeholder` | `string` | Input placeholder (max 200 characters). | | `headerTitle` | `string` | Header title (max 100 characters). | | `headerSubtitle` | `string` | Header subtitle (max 200 characters). | | `categoryFilter` | object | Restrict categories: `{ mode: "INCLUDE" \| "EXCLUDE", values: string[] }` (values are category identifiers your integration uses). | ### `theme` The widget runs inside an **open Shadow DOM**, so host-page CSS does not change its internal layout or colors. **Size the mount container** in your page; the widget **fills that container**. You can also set **`theme.width`** and **`theme.height`** if you need explicit dimensions on the widget instead of relying on the container alone. Theme values are merged with defaults. **`preset`** chooses a base palette before overrides: | `preset` | Behavior | | - | - | | `light` | Light background and text colors. | | `dark` | Dark background and text colors. | | `auto` | Follows the user’s system preference (`prefers-color-scheme`). | | Field | Type | Description | | - | - | - | | `primaryColor` | `string` | Accents (buttons, links). | | `backgroundColor` | `string` | Widget background. | | `surfaceColor` | `string` | Cards / message surfaces. | | `textColor` | `string` | Primary text. | | `textSecondaryColor` | `string` | Muted text. | | `borderColor` | `string` | Borders and dividers. | | `fontFamily` | `string` | Font stack; defaults to `"inherit"` (uses the host page’s font family). | | `fontSize` | `string` | Base size; defaults to `"16px"`. | | `borderRadius` | `string` | Corner radius (e.g. `"8px"`). | | `width` | `string` | Optional fixed width when not using a fully sized container. | | `height` | `string` | Optional fixed height when not using a fully sized container. | | `preset` | `"light"` \| `"dark"` \| `"auto"` | Base palette; see table above. | ### `branding` | Field | Type | Description | | - | - | - | | `showPoweredBy` | `boolean` | Whether to show the “Powered by” footer; defaults to `true`. Set `false` to hide. | | `logoUrl` | `string` | Optional custom logo URL. | ## Multiple widgets and shared theming Create one **`BigdataWidgetManager`** with an optional shared `theme`, pass it as `manager` in each `new BigdataAgent.BigdataChatWidget({ ... })` config, and give each widget a distinct **`instanceId`**. That way instances stay isolated but can share base styling. ## Instance lifecycle and events * **`destroy()`** — Unmounts React, clears the shadow root, and unregisters the instance. Call this when removing the widget from the page. * **`addListener(event, handler)`** / **`removeListener(event, handler)`** — Subscribe to widget events. Event payloads include `instanceName`, `timestamp`, and `data`: | Event | `data` shape (summary) | | - | - | | `ready` | `{ widgetName: string }` | | `chatmessage` | `{ prompt: string, researchEffort: "lite" \| "standard" }` | | `updatefilters` | `{ categoryFilter?: { mode, values } }` | | `opendocument` | `{ documentId: string, isPrivate: boolean }` | | `openlink` | `{ url: string }` | Advanced: to trigger a send from your own UI, you can **`emit("chatmessage", { prompt, researchEffort })`** on the same instance the widget uses (same `researchEffort` values as in the table above). ## Troubleshooting > **Blank area or widget does not appear** > > Confirm the script URL loads (network tab, no 404). Ensure `container` matches an existing > element. Check the browser console for validation errors (for example invalid `proxyUrl`). > **CORS errors** > > Your `proxyUrl` origin must allow the site origin that embeds the widget > (`Access-Control-Allow-Origin` and related headers for `POST`). > **401 or 403 from proxy** > > The proxy is rejecting the request before Bigdata. Verify server-side auth and that forwarded > credentials match what your Bigdata integration expects. > **No streaming; reply arrives all at once** > > Disable or reconfigure response buffering on your proxy, load balancer, or framework so > `text/event-stream` is flushed incrementally. For personalized help, use [Support](/support). # Connect chat and document viewer on one page Source: https://docs.bigdata.com/widgets/connecting-widgets Use a single BigdataWidgetManager so the chat and document viewer share a theme, coordinate on opendocument events, and open citations in the viewer without your own bus. When you embed the [chat widget](/widgets/chatbox-widget) and [document viewer widget](/widgets/document-viewer-widget) on the **same page**, you can pass both widgets a single **`BigdataWidgetManager`**. The manager gives you one place for **base theming** and a **shared event path** so when the chat raises **`opendocument`** (for example from a citation), the document viewer can show that document **without** wiring your own pub/sub or global state. ## Connecting Bigdata widgets Load **both** widget script tags first so **`BigdataAgent`** and **`BigdataDocumentViewer`** are defined. Then create **one** **`BigdataWidgetManager`** and pass it in each widget’s config along with **distinct** **`instanceId`** values. > [!NOTE] > > **`BigdataWidgetManager`** may be exposed by more than one widget bundle (for example the chat and document viewer). In this guide we use the canonical UMD export on **`BigdataDocumentViewer`**: **`BigdataDocumentViewer.BigdataWidgetManager`**. Use either export consistently-don’t mix them in the same page’s code. > [!NOTE] > > The proxy, API keys, and CORS story are unchanged. Each widget may still use its own **`proxyUrl`**; see the two embed guides for your backend and [Research Agent API](/api-reference/research-agent/research-agent) and document **Content** flows. ### How a citation reaches the document viewer **Citation flow** - the chat emits `opendocument` on the shared `BigdataWidgetManager`; the manager delivers it to the document viewer, which opens the file. ```mermaid theme={null} sequenceDiagram participant C as Chat widget participant M as BigdataWidgetManager participant D as Document viewer C->>M: emit("opendocument", data) M->>D: listen: forward to D instance D->>D: open document in viewer ``` Event payloads in handlers include **`instanceName`**, **`timestamp`**, and **`data`**. The **`data`** object matches the [chat](/widgets/chatbox-widget#instance-lifecycle-and-events) and [document viewer](/widgets/document-viewer-widget#instance-lifecycle-and-events) event tables. ### Minimal wiring (concept) 1. **Create a manager** - ```javascript theme={null} new BigdataDocumentViewer.BigdataWidgetManager( { theme: { /* optional, shared base */ } } ) ``` The class is exposed on the UMD object **`BigdataDocumentViewer`**; see the **Global API: `BigdataDocumentViewer`** section in the [document viewer](/widgets/document-viewer-widget) page. 2. **Initialize the document viewer** - ```javascript theme={null} new BigdataDocumentViewer.BigdataDocumentWidget({ container, proxyUrl, manager, instanceId: "my-document", /* openDocument, getHeaders, theme, … */ }) ``` 3. **Initialize the chat** - ```javascript theme={null} new BigdataAgent.BigdataChatWidget( { container, proxyUrl, manager, instanceId: "my-chat", /* config, theme, … */ } ) ``` Use a **string** `instanceId` for each that is **unique** on the page. If you omit it, the defaults are **`"chat"`** and **`"document"`**, which is fine for a **single** pair. Add explicit ids if you add more than one of either widget or want clearer logs and listeners. Example shape (placeholders; align script URLs and options with the two quickstarts): ```html theme={null}
``` (Requires both widget bundles loaded before this inline block.) ## Sharing configuration through the manager * **Base theme** - The manager holds an optional **shared `baseTheme`**. The widgets merge their own `theme` on top, same as in [Multiple widgets and shared theming](/widgets/chatbox-widget#multiple-widgets-and-shared-theming) (chat) and the [document viewer’s matching section](/widgets/document-viewer-widget#multiple-widgets-and-shared-theming). * **Event bus** - Internally, when one widget **emits** an event, the manager **broadcasts** it to every instance registered on that manager. The document viewer listens for **`opendocument`** to switch what is on screen. The chat emits that event when a user follows a **Bigdata** citation, so a shared manager is how those two stay in sync on one page. ### Events you lean on for coordination | Event | Role when connecting the two widgets | | - | - | | **`opendocument`** | Chat **emits** with **`data`**: at least `documentId`, `isPrivate`; optional **match** metadata for highlights when your integration supplies it. The document viewer **handles** the event to show the file. **Full field lists:** [Chat](/widgets/chatbox-widget#instance-lifecycle-and-events) · [Document](/widgets/document-viewer-widget#instance-lifecycle-and-events) | | **`openlink`** | For opening an external **URL** instead of in-product document navigation; [chat](/widgets/chatbox-widget#instance-lifecycle-and-events) describes the `data` shape. | | **`ready`** | Fires with `{ widgetName }` after the widget is mounted. | You can still use **`addListener` / `removeListener`** (or the chat’s **`emit`** for **advanced** flows such as a host-triggered **send** - see the chat page) on the **returned instances** the same as on a single widget. ## Build your own connection The chat can register **`onDocumentClick`** and **`onLinkClick`** in **`config`** (or attach equivalent listeners on the chat **instance**). These are **shortcuts for observing** user interactions (analytics, logging, or side effects) and **do not override** the default built-in behavior by themselves. The usual **“chat + document on one page”** path is the **shared** **`manager`**: the chat **emits**, the document viewer is already listening through the same manager, so you do not need extra glue for the default “open in viewer” experience. ## Further reading * [Chat widget](/widgets/chatbox-widget) - `proxyUrl`, `config`, themes, and events. * [Document viewer](/widgets/document-viewer-widget) - `openDocument` shape, proxy contract, and events. * [Support](/support) - for WordPress, enterprise questions, or integration help. # Embed the document viewer widget Source: https://docs.bigdata.com/widgets/document-viewer-widget Add the Bigdata document viewer widget to your website with a script tag, a backend proxy for your API key, and optional theming. Load public or private documents in the viewer. The Bigdata document viewer widget loads as a **single script** - no separate CSS file to host. You mount it on a container element and point it at a **URL you control** - your backend adds your Bigdata API credentials and forwards requests to the document and content APIs. The UI renders the document (including PDF) in an **open Shadow DOM** so the host page’s own CSS does not restyle the viewer’s internals. ## How it fits together Traffic for document data goes through **your** backend: the widget only calls `proxyUrl` - your server holds the Bigdata credentials, reaches Bigdata’s **Documents** and **Content** APIs, and returns JSON (and file URLs) the viewer can use. **Where each part runs** - the document viewer and your proxy live on **your** side; file storage, annotation, and catalog APIs run in **Bigdata**'s infrastructure. ```mermaid theme={null} flowchart LR subgraph yours [Your infrastructure] direction TB FE[Document viewer] BE["Your backend / proxy"] end subgraph bigdata [Bigdata infrastructure] API[Documents & Content API] end FE --> BE BE --> API API --> BE BE --> FE ``` **What happens at each step** - the browser issues **GET** requests to your `proxyUrl`; your server adds the API key, calls Bigdata, and returns **RPJSON** and signed URLs. Time-limited URLs are often re-fetched through **`/proxy-document`** on your side so the browser does not need direct cross-origin access to signed storage or your key. ```mermaid theme={null} sequenceDiagram participant W as Document viewer widget participant P as Your backend participant B as Bigdata APIs W->>P: GET (document path or proxy-document) Note over P: Attach API key server-side P->>B: Forward B-->>P: JSON, signed URL metadata P-->>W: Response ``` > [!WARNING] > > Never put your Bigdata API key in frontend code or expose it in the browser. > The widget is designed to call **your** proxy. The proxy must attach your API > key when calling Bigdata. ## Why you need a backend endpoint End users can open developer tools and read any secret shipped to the page. The widget therefore sends **only** to **`proxyUrl`**. Your server (or serverless function, or BFF) should: 1. Authenticate the end user if needed (cookie, session, or signed token). 2. Add the Bigdata API key to the outbound request when you call Bigdata. 3. **Forward** the external paths the widget uses **`GET /proxy-document?url=…`** when dealing with a **signed** URL (without adding authentication). ## Browser compatibility The script bundle targets **modern evergreen browsers**. It is **not** supported in Internet Explorer or other legacy engines without current Web APIs. **Supported browsers** | Browser | Guidance | | - | - | | **Chrome** | Recent stable (desktop and Android). | | **Edge** | Recent stable (Chromium-based). | | **Firefox** | Recent stable. | | **Safari** | **16.4 or newer** on macOS and iOS. Earlier versions lack reliable support for **constructible stylesheets** and **`adoptedStyleSheets`** on the widget’s Shadow DOM, which the bundle uses to inject CSS. | **Environment expectations** * **JavaScript enabled** - the widget mounts via script and does not render without JS. * **CORS to your `proxyUrl`** - if the embed and the API are on different origins, allow `GET` (and preflight when **`getHeaders`** adds non-simple request headers). Misconfigured CORS is a common “nothing loads” symptom, see [Troubleshooting](#troubleshooting). * **Third-party cookies** - not required for the default embed pattern - auth is between the visitor’s browser and **your** `proxyUrl` (or headers from **`getHeaders`**, or cookies you set on that origin). For audiences on older locked-down browsers, contact [support](/support) to discuss feasibility. ## Quickstart **1. Add a sized container** Add a wrapper element and **give it the size you want** (fixed pixels, percentage, flex, grid, or `min-height`-whatever fits your page). The widget mounts inside it and **stretches to fill** the container’s box. ```html theme={null}
``` **2. Load the script** Use the widget script URL from Bigdata. Example shape (version path may differ). Load it **before** the inline script that constructs the widget so `BigdataDocumentViewer` is defined: ```html theme={null} ``` **3. Initialize the widget** Create the widget with **`new BigdataDocumentViewer.BigdataDocumentWidget({ ... })`**. At minimum set **`container`** and **`proxyUrl`**. Set **`openDocument`** when you know which file to show first, and add **`theme`** and **`getHeaders`** as needed-see the [configuration reference](#configuration-reference) below. ```html theme={null} ``` > [!NOTE] > > The **Bigdata** web app’s **Widget Playground** (under **Playground** → > **Widgets**) loads the same UMD and wires **`getHeaders`** to the signed-in > user-use it as a live example when you integrate auth. ## Global API: `BigdataDocumentViewer` After the bundle loads, the UMD global **`BigdataDocumentViewer`** exposes **`BigdataDocumentWidget`**, and (for multiple widgets) **`BigdataWidgetManager`**. The legacy helper **`init(config)`** is an alias for `new BigdataDocumentWidget(config)` and returns the mounted instance. **UMD `BigdataDocumentViewer` members**
Property / method Type Description
`BigdataDocumentViewer.init` `function` **Deprecated.** `init(config)` is a legacy alias for `new BigdataDocumentWidget(config)`.
`BigdataDocumentViewer.BigdataDocumentWidget` `class` Constructor: pass the same `config` as above.
`BigdataDocumentViewer.BigdataWidgetManager` `class` Optional. Shared `manager` and base `theme` across widgets.
Store the instance from `new BigdataDocumentViewer.BigdataDocumentWidget(...)` if you need to call `destroy()` or use **`addListener`** / **`emit`**. (The legacy `BigdataDocumentViewer.init(...)` helper is deprecated.) ## Proxy contract (summary) **What your BFF should serve (per path)**
Aspect Expected behavior
Public doc **GET** `{proxyUrl}/v1/documents/{documentId}`. The response is either a **limited** view or includes a **signed** `url` for the full annotated content - the widget may follow with `/proxy-document` to load **RPJSON**. Align with [Fetch document](/api-reference/search/fetch-document).
Private doc **GET** `{proxyUrl}/contents/v1/documents/{documentId}/annotated`. The JSON includes a **signed** `url`; the viewer loads RPJSON via `/proxy-document`. See [Get annotated document](/api-reference/documents/get-annotated-document).
Proxy pass-through **GET** `{proxyUrl}/proxy-document?url={encodeURIComponent(signedUrl)}` - your server fetches `signedUrl`, returns the body to the client (RPJSON or bytes). **Required** when the browser must not call the signed host directly.
Original file **GET** `{proxyUrl}/contents/v1/documents/{documentId}/original` for PDF / download flows, as used by the file URL helper. Match [Get original document](/api-reference/documents/get-original-document) style responses.
**Headers** - if you set **`getHeaders`**, the widget merges the returned map onto requests the HTTP client makes to your **`proxyUrl`** (for example `Authorization: Bearer ...`). **Buffering and JSON** - if the viewer stays on “loading” while your proxy already finished, check that your framework is not transforming or stripping the JSON body, and that **CORS** and **`proxy-document`** are implemented for **GET**. ## Configuration reference ### Top-level options **`config` options (top level)**
Option Type Description
`container` `string` or `HTMLElement` **Required.** Where the widget mounts: a CSS selector that `document.querySelector` resolves, or a DOM element.
`proxyUrl` `string` **Required (for loading documents).** The base of your BFF. Without it, the inner queries for RPJSON will not run.
`instanceId` `string` Optional. Stable id for this instance when several widgets share a page; if omitted, the implementation uses `"document"`.
`manager` `BigdataWidgetManager` Optional. Shared manager across widgets for coordinated events and a common base theme.
`getHeaders` `function` Optional. `() => Promise>` - merged onto requests to your origin (e.g. bearer for your BFF).
`openDocument` `object` Optional. Initial `documentId`, `isPrivate`, and optional `matchesInformation`; see ** `openDocument` (shape) **.
`theme` `object` Optional. Visual settings; see ** `theme` **.
### `openDocument` (shape) **`openDocument` fields**
Field Type Description
`documentId` `string` **Required** when you open a document. The public document id or private **content** id your backend uses.
`isPrivate` `boolean` `true` = private **Content** flow ( `/contents/v1/.../annotated`); `false` = public `GET /v1/documents/{id}` path.
`matchesInformation` `object` **Optional** highlighting: `{ cqsChunks: [...] }` or `{ cnums: number[] }` from your search or app, when you need chunk alignment.
**Later navigation** - call **`instance.emit("opendocument", { documentId, isPrivate, matchesInformation? })`** to load another document (same instance). ### `theme` The widget runs inside an **open Shadow DOM**, so host-page CSS does not change its internal layout or colors. **Size the mount container** in your page - the widget **fills that container**. You can also set **`theme.width`** and **`theme.height`** if you need explicit dimensions on the widget instead of relying on the container alone. Theme values are merged with defaults. **`preset`** chooses a base palette before overrides: **`theme.preset` values**
`preset` Behavior
`light` Light background and text colors.
`dark` Dark background and text colors.
`auto` Follows the user’s system preference ( `prefers-color-scheme`).
**`theme` field reference**
Field Type Description
`primaryColor` `string` Accents (controls, highlights).
`backgroundColor` `string` Widget background.
`surfaceColor` `string` Surfaces and panels.
`textColor` `string` Primary text.
`textSecondaryColor` `string` Muted text.
`borderColor` `string` Borders and dividers.
`fontFamily` `string` Font stack; can default like the **chat** widget to inherit the host.
`fontSize` `string` Base size; defaults in the same family as the chat widget (e.g. `16px`).
`borderRadius` `string` Corner radius (e.g. `8px`).
`width` `string` Optional fixed width when not using a fully sized container.
`height` `string` Optional fixed height when not using a fully sized container.
`preset` `"light" | "dark" | "auto"` Base palette; see the table above.
## Multiple widgets and shared theming Create one **`BigdataWidgetManager`** with an optional shared `theme`, pass it as `manager` in each **`BigdataDocumentWidget` config**, and give each widget a distinct **`instanceId`**. That way instances stay isolated but can share base styling-**including** when you add the **chat** widget and **document viewer widget** on the same page. ## Instance lifecycle and events * **`destroy()`** - Unmounts React, clears the shadow root, and unregisters the instance. Call this when removing the widget from the page. * **`addListener(event, handler)`** / **`removeListener(event, handler)`** - Subscribe to widget events. Event payloads include `instanceName`, `timestamp`, and `data`: **`data` payload (summary) per event**
Event `data` shape (summary)
`ready` `{ widgetName: "BIGDATA_DOCUMENT_WIDGET" }` (fires after mount).
`opendocument` Same as **openDocument**: `documentId`, `isPrivate`, optional `matchesInformation`.
(For parity with the **chat** bus, the same type names exist across widgets. The document viewer widget is typically driven with **`opendocument`** to change what is on screen, see **`openDocument` (shape)** in [Top-level options](#top-level-options).) ## Troubleshooting > **Blank area or widget does not appear** > > Confirm the script URL loads (network tab, no 404). Ensure `container` > matches an existing element. Check the browser console for validation errors > (for example invalid `proxyUrl`). The inner fetch also needs a valid > **`openDocument` / `emit("opendocument", ...)`** with a **`documentId`**, > and a configured **`proxyUrl`**, or nothing will request RPJSON. > **CORS errors** > > Your `proxyUrl` origin must allow the site origin that embeds the widget > (`Access-Control-Allow-Origin` and related headers for `GET` and, if you use > `getHeaders`, the necessary preflight). > **401 or 403 from proxy** > > The proxy is rejecting the request before Bigdata. Verify server-side auth > and that forwarded credentials and **`getHeaders`** match what your BFF > expects. > **Document id works in the app but not the embed** > > For **Content**-originated files, set **`isPrivate: true`** and the > **`content_id`** your proxy expects. For **public** catalog content, set > **`isPrivate: false`** and the id you would pass to [Fetch > document](/api-reference/search/fetch-document). Mixing public vs private > flags is a frequent cause of 404s at the proxy. > **RPJSON never appears; loading never finishes** > > A missing or incorrect **`/proxy-document`** is common: the first response > returns a **signed** `url` that the browser must not call cross-origin. > Ensure **`GET .../proxy-document?url=`** downloads the **server-side** and > returns the JSON. If your BFF is slow but correct, also confirm you are not > stripping or buffering the response body. Check upstream **API key** > pass-through for **`/v1/documents/...`** and **`/contents/.../annotated`**. For personalized help, use [Support](/support). # Widgets Source: https://docs.bigdata.com/widgets/introduction Embed Bigdata research and document experiences in your own sites and applications. **Widgets** are drop-in browser bundles you host on a page alongside your own backend. They keep API credentials on **your** server while visitors interact with Bigdata capabilities in the UI. ## What’s in this section > **[Chat Widget](/widgets/chatbox-widget)** > > Add the research chat UI with a script tag, a proxy for your API key, and optional theming. > **[Document Viewer Widget](/widgets/document-viewer-widget)** > > Embed document viewing and navigation aligned with your Content API integration. > **[Connecting widgets](/widgets/connecting-widgets)** > > Use a shared BigdataWidgetManager to connect a chat and document viewer widget. This is how you can reference playgrounds as done in other parts of the documentation (we will probably want to update icons): > **[Search Service Playground](https://platform.bigdata.com/search/search-documents)** > > Search across your private content and other sources. In the playground, open the source selector and choose **My Files** to limit results to your uploaded documents. > **[Research Agent Playground](https://platform.bigdata.com/research-agent)** > > Run research over your private content and real-time data. In the playground, use the source selector and filter by **My Files** to ground answers in your documents.