October 20, 2026
Most investment teams consume datasets built around someone else's taxonomy. Generic sentiment can describe whether an article sounds positive or negative, but it cannot answer a strategy-specific question such as: *Is this company gaining market share from competitors?*
This developer workflow moves the point of differentiation upstream. Researchers define the theme, retrieve the relevant evidence across a full equity universe, distill the judgment of a large reasoning model into a smaller model, and test whether the resulting labels contain information beyond sentiment. After retrieval, the proprietary text and model workflow remain inside the firm's GPU environment.
## Key benefits
* **Create differentiated data:** Define the classes and evidence rules around an investment thesis instead of adapting a generic vendor taxonomy.
* **Retrieve at portfolio scale:** [Bigdata.com](https://bigdata.com) searches news and transcripts across thousands of companies.
* **Keep inference private:** Run teacher labeling, fine-tuning, corpus labeling, and signal construction on infrastructure controlled by the research team.
* **Reduce recurring compute:** Use the large reasoning model on a sample, then use the distilled student for full-corpus and subsequent labeling passes.
* **Evaluate economic value:** Compare the classified signal with a no-label, sentiment-only benchmark under the same portfolio construction.
* **Preserve auditability:** Materialize every stage as files so retrieval, labels, signals, and backtest inputs can be inspected or replaced independently.
## Architecture
The architecture is powered by [Bigdata.com](https://bigdata.com), which delivers comprehensive news and transcript search across thousands of companies. Alongside a price-history data source used in backtesting, these inputs drive the workflow. All processing steps—including sentence extraction, NVIDIA Nemotron 3 Embed 1B ranking, teacher inference, LoRA supervised fine-tuning, student inference, and signal construction—take place securely within a private GPU environment. By using files to hand off data between stages, each workflow component can be rerun or updated independently, without needing to rely on a single, monolithic service.
## How the workflow works
### 1. Define the research question twice
A robust pipeline needs two related but distinct specifications:
1. **Retrieval seeds** describe how the theme appears in text. For market share, examples cover both gains and losses: improving win rates, customers switching from competitors, losing shelf space, or growing slower than the market.
2. **A labeling taxonomy** defines the model's decision boundary. The market-share taxonomy maps evidence to `gainer`, `loser`, or `irrelevant` and explicitly rejects common near-misses such as absolute growth without a competitive comparison, industry-wide growth, or management plans without a realized outcome.
Both live in one file, `config/data_extraction.yaml`. Abridged from the market-share run:
```yaml theme={null}
theme:
id: market_share
classes:
gainer: 1 # long
loser: -1 # short
irrelevant: null # no position
body: |
GAINER: the COMPANY is gaining ground on competitors — won share,
accounts, or shelf space; win rates improved; grew faster than its market.
LOSER: the COMPANY is losing ground to competitors.
IRRELEVANT: anything else, including the near-misses — absolute growth
with no competitive reference (GROWING IS NOT GAINING SHARE), industry-
wide statements, and plans or guidance with no realized outcome.
...
universe_csv: input/universe_R1K_PiT.csv
search:
start_date: '2016-01-01'
end_date: '2026-10-01'
combined_category:
mode: INCLUDE
values: [news_premium, transcripts]
seeds:
statements:
positive:
- Company gained share in its core market at the expense of competitors
- Win rates against competitors improved meaningfully this period
- Management cited evidence of competitor customers switching to their platform
# ... 30 statements in total
negative:
- Company lost ground to competitors in its primary end market
- The firm is losing shelf space and channel presence to rivals
- Customer acquisition is falling behind the overall market growth rate
# ... 30 statements in total
```
The `classes` block is the only link between the taxonomy and the portfolio: each actionable class carries its trading sign, and `null` means no position. Everything below it is plain English the teacher model reads as its instructions.
This separation matters. Retrieval errors determine what evidence enters the corpus. Labeling errors determine what the dataset means. A downstream backtest cannot recover evidence that was never retrieved.
### 2. Retrieve and rank evidence across the universe
[Bigdata.com](https://bigdata.com) searches a licensed archive of premium news and earnings-call transcripts across thousands of companies, without a query for every name in every period. [Smart Batching](/use-cases/search-service/smart-batching) plans queries around content volume and co-mentions rather than a full company-by-period grid.
This implementation queries news and transcripts only, but the same search API covers the rest of the Bigdata.com content: regulatory filings, corporate communications such as press releases, expert-call interviews, podcasts, and investment research, all entity-resolved to the same company identifiers. Switching content types is a change to `combined_category` in `config/data_extraction.yaml`, so a theme whose evidence lives in filings or expert calls rather than in news can reuse the rest of the pipeline unchanged.
The local pipeline then:
1. Splits retrieved passages into sentences.
2. Embeds the sentences and seed statements with NVIDIA Nemotron 3 Embed 1B.
3. Computes similarity to the theme seeds.
4. Ranks the corpus so coverage becomes an explicit research parameter.
After sentence extraction, the market-share search yielded 4,226,092 passage-company records from January 2016 to October 2026. Ranking by similarity to the seeds and keeping the top 20% left 845,220 records. That 20% cut is the corpus used for labeling and the backtest. Keeping only the highest-ranked material increases precision but reduces the number of companies with an active signal. Retaining more material increases breadth while admitting weaker thematic matches.
### 3. Distill a large model's judgment into a small model
Calling a large hosted model on every passage creates a recurring cost and sends proprietary text outside the research environment. The pipeline instead uses a teacher-student pattern:
1. Draw a stratified sample across the similarity distribution.
2. Label that sample locally with **NVIDIA Nemotron 3 Super 120B**.
3. Split the teacher labels into training and held-out evaluation sets.
4. Fine-tune **NVIDIA Nemotron 3 Nano 4B** with LoRA supervised fine-tuning.
5. Select a checkpoint using accuracy, macro-F1, rare-class recall, and class-count drift.
6. Use the selected student to label the ranked corpus.
The implementation evaluates two text granularities. In **chunk mode**, the model reads the complete passage. In **sentence mode**, it reads the most relevant sentences first and stops when it finds actionable evidence. Chunk mode preserves context. Sentence mode attaches each signal to a more precise evidence span.
In the completed market-share run, the selected 4B checkpoints reproduced the teacher's held-out labels with **94.05% accuracy and 0.831 macro-F1** at chunk level, and **95.90% accuracy and 0.866 macro-F1** at sentence level. Macro-F1 is the figure to read: `irrelevant` is the majority class, so accuracy alone flatters any classifier on this distribution. In chunk mode, weighted labeling throughput rose from **7.4 rows per second** for the teacher on the sample to **286.3 rows per second** for the student on the full corpus, roughly **38.8x faster**. Sentence mode is not directly comparable: one row can require several model calls.
### 4. Convert labels into a daily signal
The theme configuration assigns a sign to each actionable class: `gainer = +1`, `loser = -1`, and `irrelevant` contributes no position. Labels are aggregated to a daily `(date, entity, signal)` table using a point-in-time cutoff.
The reference implementation forms a 50/50 long/short portfolio with a 5% single-name cap and up to 160 names per leg. It also builds a **sentiment-only benchmark** from the same retrieved records, using relevance and sentiment but no custom classification labels. This is the correct baseline for determining whether the distilled taxonomy adds information.
### 5. Validate the signal, not just the classifier
High held-out accuracy does not prove investment value. The workflow therefore tests two different questions:
* **Predictive content:** Pooled Spearman information coefficient measures whether the signal ranks subsequent returns.
* **Portfolio behavior:** Gross exposure, net exposure, active positions, turnover, return, volatility, and Information Ratio reveal whether a strong-looking result is caused by a thin or one-sided book.
For the market-share study, one-day Information Coefficient was strongest at the tightest 1% retrieval cut and declined as coverage widened. However, the tightest books were too sparse to maintain balanced exposure. The selected 20% operating point trades some per-passage strength for a broader, more investable cross-section. Raw labels also fire on a single day, which leaves the book under-invested. A one-day half-life keeps a classification active briefly after it appears and is the holding period used in the table below.
## Results from the reference market-share run
The comparison uses the same January 2016 to October 2026 price history for every row. The one-day half-life is applied to both classified books. The unsmoothed books are the same labels with no decay. The sentiment row is the unsmoothed benchmark built from Bigdata.com sentiment on the same retrieved records.
| Signal | Information Ratio | Annualized return | Volatility | Mean gross exposure |
| - | - | - | - | - |
| Sentence labels, 1-day half-life | **1.18** | **8.63%** | 7.33% | 97.0% |
| Chunk labels, 1-day half-life | **0.91** | **6.58%** | 7.24% | 91.7% |
| Sentence labels, unsmoothed | 0.97 | 6.75% | 6.92% | 58.7% |
| Chunk labels, unsmoothed | 0.95 | 5.88% | 6.19% | 47.3% |
| Sentiment-only, unsmoothed | 0.41 | 3.78% | 9.31% | 99.4% |
Solid lines are the one-day half-life. Dashed lines are the same labels unsmoothed. The gray line is the sentiment-only benchmark.
A white paper with the full methodology, evaluation, and results is available on request at [research@ravenpack.com](mailto:research@ravenpack.com).
## What is included
* A Jupyter notebook covering retrieval, labeling, distillation, evaluation, corpus inference, and backtesting
* Theme and model configuration files
* Resumable search, teacher, training, evaluation, and corpus-labeling stages
* NVIDIA Nemotron 3 Embed 1B semantic ranking
* NVIDIA Nemotron 3 Super 120B teacher serving
* NVIDIA Nemotron 3 Nano 4B LoRA fine-tuning and local inference
* Chunk-level and sentence-level labeling modes
* File-based interfaces for bringing your own corpus or prices
* A long/short backtest and sentiment-only benchmark
* Information-coefficient and holding-period checks across coverage levels
## Tested environment
* Python 3.12 or later
* Docker for model serving and training containers
* 2x NVIDIA H200 NVL (PCIe), serving the 120B NVFP4 teacher across both devices and later the fine-tuned 4B student. This is the configuration the reference run used, not an established minimum.
* Sufficient local storage for model weights, retrieved passages, labels, and price caches
* A Bigdata.com API key for retrieval
* A local price cache in the documented format (daily returns per company) for the backtest
The smaller student makes repeated inference materially cheaper, but the demonstrated pipeline still requires infrastructure appropriate for the teacher and fine-tuning stages.
## Software used
**NVIDIA technology**
* NVIDIA Nemotron 3 Embed 1B
* NVIDIA Nemotron 3 Super 120B A12B NVFP4
* NVIDIA Nemotron 3 Nano 4B BF16
* CUDA-enabled local model serving and distributed training
**Open-source and data components**
* [Bigdata.com](https://bigdata.com) Search API and [bigdata-smart-batching](https://pypi.org/project/bigdata-smart-batching/) library
* PyTorch, Transformers, PEFT, and TRL
* pandas, NumPy, and PyArrow
* Jupyter
* A price-history dataset
## Get started
Open the [dataset-generation pipeline notebook](https://github.com/Bigdata-com/bigdata-dataset-generation/blob/main/notebooks/dataset-generation-pipeline.ipynb). Two places need editing.
In `config/data_extraction.yaml`:
* the classes and their trading signs
* the labeling taxonomy
* the investable universe
* positive and negative retrieval seeds
* the search period
In the notebook itself:
* `TOP_FRACTION`, the share of the ranked corpus the student labels (0.2 in the reference run)
The same pipeline can then be applied to market share, margin pressure, balance-sheet risk, or another investment question whose evidence can be defined clearly in text.
## Resources
* [Bigdata dataset-generation repository](https://github.com/Bigdata-com/bigdata-dataset-generation)
* [NVIDIA AI Model Distillation for Financial Data](https://build.nvidia.com/nvidia/ai-model-distillation-for-financial-data/examplecard)
* [NVIDIA Quantitative Signal Discovery Agent](https://build.nvidia.com/nvidia/quantitative-signal-discovery-agent)
* [Bigdata.com](https://bigdata.com)
This content is for informational purposes only and does not constitute investment advice.
***
Francisco Gomez
Manager of Applied Intelligence (VP)
Francesco Ricigliano
Data Scientist