Baking Behavior into Weights: Fine-Tuning a Language Model for Financial Search

August 11, 2026

​

Summary

Prompt engineering is an effective way to discover how an AI system should behave. It becomes less attractive when the same multi-thousand-token rulebook has to be re-sent on every request. That was the situation behind Bigdata.com Smart Mode, which converts natural-language financial questions into structured search plans. A prompted frontier model could do the job, but with recurring latency, cost, and control trade-offs. So we distilled the planner’s stable behavior into Qwen3-1.7B, using parameter-efficient fine-tuning on tens of thousands of deliberately constructed examples, and cut the fixed prefix more than 70% by moving the rules out of the prompt and into the weights. The hard part was not choosing an open-source training library, or testing different models. It was designing the system around the model: simplifying the action space, covering the right decisions in the data, matching training and production formats, concentrating loss on the model’s output, and evaluating structured behavior rather than generic language-model quality. The resulting specialist matched or slightly improved the hosted planner on this task while materially reducing latency and cost.

​

The prompt had become part of the product

The primary user of a search API is increasingly not a human expert. It is an AI agent, orchestrated by a frontier model, acting for an analyst, portfolio manager, or application that does not directly touch the raw API. That changes the interface problem. A human who understands a financial search system can deliberately select sources, reporting periods, document types, entities, and other constraints. A general-purpose model often defaults to the simplest available call unless the domain rules are made explicit. Consider this query:

What did Micron’s latest 10-K say about Samsung?

This is not a semantic search for Samsung, Micron, and 10-K. The planner must understand that Micron is the reporting entity whose filing should be searched; Samsung is a mentioned entity expected inside it; “latest” is relative to the request time; the evidence must come from a specific document class; and the free-text query should preserve that narrow intent. Similar problems arise with fiscal calendars, source-specific requests, compound questions, ambiguous entity names, and cases where the correct outcome is no evidence at all. We initially handled this with a frontier model and extensive prompt engineering. That was the right starting point: prompts let us discover the rules quickly and refine the expected behavior without training anything. But the prompt kept growing. It accumulated instructions for entity roles, temporal interpretation, content selection, decomposition, structured output, and edge cases. The model was effectively receiving the same operating manual on every request. Beyond cost and latency, that also tied a latency-critical, high-volume path to an external dependency. At that point, the relevant question was no longer “Can prompting solve this?” It was “Which parts of this behavior should still be in the prompt?” The answer:

Dynamic context belongs in the prompt. Stable, repeated behavior is a candidate for the weights.

​

Before fine-tuning the model, simplify its world

Fine-tuning cannot compensate indefinitely for a poor interface. A single universal search action with a large schema forces the model to solve several problems at once: infer the required content, identify the relevant parameters, ignore unrelated fields, formulate the query appropriately, and produce valid structure. We first exposed a compact set of purpose-specific search capabilities, or search tools, with clean responsibility boundaries. Different content families require different filters and query-construction strategies, and collapsing them into one overloaded action is a false economy. For example, we naturally separated a search tool for earnings calls and corporate events from a tool for analyst and broker research. Separating them reduced the decision space, shortened capability descriptions, lowered schema ambiguity, simplified training examples, and made failures easier to diagnose. The principle is broader than search:

Do not give a language model an unnecessarily complicated interface and then compensate with a longer prompt. Reduce the action space first.

Cleaner tools do not automatically make the prompt shorter — more schemas can even add tokens. What they change is the structure of the problem: each capability carries one responsibility and a focused description, so every planning decision becomes more separable, easier to demonstrate in data, and easier to score. That separability is the precondition for fine-tuning, which is where the prompt really gets smaller.

​

Fine-tune behavior; retrieve knowledge

We did not fine-tune the model to memorize financial facts. Facts change, and they must remain current and attributable through retrieval. We specialized the decisions that are stable:

• Interpreting the information need
• Distinguishing reporting entities from entities merely mentioned
• Resolving financial and relative periods
• Selecting the appropriate capability
• Decomposing compound requests
• Producing valid structured plans
• Avoiding unsupported or unnecessary actions

Retrieval supplies the evidence. Fine-tuning teaches the model how to ask for it. For this query-planning task, a compact specialist was a better fit than repeatedly instructing a general-purpose model. The behavior is bounded, repeated, measurable, and latency-sensitive, and our evaluation showed that the smaller model could take on that role without sacrificing quality. Frontier models remain the right tool for open-ended reasoning and synthesis; our specialist sits between the frontier model (or a human) and the retrieval system. They are meant to coexist.

​

The real engineering work was the training distribution

For a model specialized on producing structured outputs with many dimensions, data quality is less about raw volume than about decision coverage. A large collection of similar queries can still produce a brittle model. Strict privacy and confidentiality policies mean that we do not use or inspect production query logs for training. Instead, we construct representative requests from domain expertise, client-informed use cases, trusted structured data, and controlled synthetic generation. This gives us deliberate coverage of realistic financial research patterns without relying on sensitive user data. We designed the dataset as an explicit coverage matrix rather than a corpus. Its dimensions span content intent (which tool, document type, source…), entity roles, temporal forms, entity multiplicity, explicit and implicit constraints, complete versus misleading hints, single-part versus compound questions, free-text versus primarily structural retrieval, and negative cases where no search action should be produced. The cross-product yields a few dozen distinct cells, each a category the planner must handle, and we built tens of thousands of examples across that space. The payoff of an explicit matrix is a closed loop: because every example carries its cell, evaluation produces per-dimension roll-ups. When the model underperforms, we see which cell is failing and we can generate targeted data for exactly that cell in the next iteration, instead of guessing.

​

Behavioral distillation, with control

A larger teacher model, Claude Sonnet 4.6, played two roles: generating realistic linguistic variation for each cell of the matrix, and producing the reference structured outputs the student learned to reproduce. Qwen3-1.7B was trained on those completed examples through supervised fine-tuning. More precisely, this was sequence-level behavioral distillation. We did not transfer hidden states or logits; we used a capable model to demonstrate the behavior and trained the smaller model to reproduce it. Synthetic data did not mean unconstrained model-generated data. The process combined expert-defined intents, trusted entities and periods, teacher-generated language variation, deterministic validation of structured labels, and human review of difficult slices. Fluent examples are easy to generate. A balanced distribution of correct decisions is much harder.

​

Match the inference context

Our first fine-tuning pass treated the task as a bare mapping: a natural-language query in, a structured payload out. At inference time, however, the model was always invoked with a concise system prompt defining its role, describing the available capabilities, and setting a small number of operating constraints. The target behavior had not changed, but the conditioning context had. That mismatch was not necessarily obvious from training loss alone, yet it became visible during inference-time evaluation. We therefore rebuilt the training examples with the same simplified system context, message structure, and output contract used at inference. For a compact specialist, the prompt is not merely deployment scaffolding. It is part of the task the model learns. Train–serve consistency therefore means matching the complete interaction format, not only the query and its expected payload. We also used the dataset to teach conservative planning: the model should not invent constraints or split a focused request into redundant calls. In structured search, precision depends as much on what the planner leaves unspecified as on what it adds.

​

Training the decision, not the boilerplate

We fine-tuned Qwen3-1.7B using LoRA/QLoRA (rank-32 adapters on the attention and MLP projections) with Unsloth on AWS SageMaker. Parameter-efficient fine-tuning was sufficient because the goal was behavioral adaptation, not relearning the model from scratch. LoRA keeps the pretrained model largely fixed and learns a small set of adaptation weights. QLoRA lowers training-memory requirements further by loading the base model in quantized form (4-bit) while training the adapters at higher precision. The job therefore fits comfortably on a single 24GB GPU (an ml.g5.xlarge / NVIDIA A10G) rather than requiring a distributed cluster. We initially evaluated a larger model variant, nevertheless, after the fine-tuning, the 1.7B model retained the task-specific quality while improving latency. The production lesson was clear:

For a constrained task, use the smallest model that reliably clears the evaluation gates.

We used standard assistant-only supervised fine-tuning: the model received the complete system context and request format used at inference, while the loss was computed only on the teacher-generated structured payload. Because decoder-only training commonly renders the full conversation as a single token sequence, we explicitly masked the system and user tokens so that the objective remained focused on the output the model was expected to produce.

​

Reproducible by construction

Training data is versioned (DVC) alongside the code; every run logs its exact data snapshot, hyperparameters, and metrics to MLflow and registers the resulting adapter in a model registry tagged with its evaluation results. Promotion between environments is metric-gated: a candidate must beat the current champion on the held-out evaluation set before it can advance. The full lifecycle, build → train → validate → quantize → deploy, runs as CI. Any deployed version traces back to its code revision, data snapshot, and eval results, which is what turns a one-off fine-tune into a capability the product can retrain on demand.

​

Evaluate the payload and the evidence separately

Training loss tells us whether the model is becoming better at reproducing its labels. It does not tell us whether the resulting search plan is operationally correct, or whether executing that plan retrieves evidence that can support an answer. We therefore evaluate the system at two distinct points of responsibility: the plan produced by the model (a schema of the search tools to be used) and the evidence returned by search.

​

Payload evaluation: did the model build the right search plan?

Before executing search, we compare the model’s structured output directly with a reference payload. This isolates the behavior learned during fine-tuning from the retrieval system that runs afterwards. These checks deliberately separate partial correctness from full correctness, which helps to distinguish failure modes. The free-text check, for instance, covers the decision to include query text, not its exact wording; placement- and structure-sensitive details are captured by the complete-plan check. Direct payload comparison is intentionally strict. A wrong filter can sometimes retrieve a chunk that happens to satisfy the request. That may look successful downstream, but it remains a planning error. We also review the checks across query types so a strong aggregate result does not hide a concentrated weakness.

​

Retrieval evaluation: did the plan return usable evidence?

We then execute the payload and evaluate the returned chunks. This answers a different question: whether the search system ultimately surfaced evidence that satisfied the information need. Constraint alignment and zero-result behavior are evaluated deterministically. Passage answerability uses the same narrow 1–4 rubric we developed for FinSearch-Eval (coming soon): This evaluation works at the chunk level because external search providers generally do not expose the filters, query transformations, or internal plans they applied. The evidence they return is the only output that can be compared consistently. That makes it an outcome evaluation, whereas the payload evaluation above is a direct and stricter test of the fine-tuned planner itself. The two layers are complementary rather than interchangeable. A plan can be structurally wrong and still get lucky in retrieval; a plan can also be correct while corpus coverage, ranking, or passage selection fails. Evaluating both lets us distinguish a model-generation error from a search-system error. A valid request for mentions of Tesla in a Coca-Cola earnings call, for example, should still produce the correct targeted payload. If the transcript contains no such mention, zero returned chunks are the correct retrieval outcome. The full retrieval evaluation methodology will be published as FinSearch-Eval.

​

Results: three views of the same change

No single metric captures the whole result. We looked at the model from three angles: what fine-tuning changed relative to the same base model, how the specialist compared with the prompted frontier model planner, and what the change meant end to end for retrieval quality, latency, and cost.

​

What fine-tuning changed in the base model

We first compared the fine-tuned model with the original Qwen3-1.7B using the same compact inference context on the held-out payload evaluation set. The out of the box model could sometimes produce a valid payload, but it rarely made the full set of planning decisions correctly. After fine-tuning, it reached 96.7% on the strict complete-plan check, which requires the selected capabilities, call count, arguments, and payload structure to match the reference. The compact prompt still supplied the runtime context; the weights now carried most of the stable planning behavior that previously had to be expressed through a much longer instruction set.

​

How the specialist compared with the prompted production planner

We then compared selected argument families against Claude Haiku 4.5 running the full production prompt. This is a separate field-level view of payload accuracy. The company check in the previous table asks whether the expected companies were extracted; here, reporting and mentioned companies are scored as separate fields, making their placement in the payload visible. The specialist matched or exceeded the prompted planner across all four fields, with improvements ranging from 2.1 to 6.0 percentage points. These are exact-match checks on the generated payload fields, not chunk-level FinSearch-Eval scores.

​

What changed end to end

Finally, we executed the generated plans and compared retrieval without model-based planning, the prompted hosted planner, and the fine-tuned specialist. These results should be read narrowly. They do not show that a 1.7B model is generally more capable than a frontier model. They show that a compact model, trained on a well-defined decision distribution, can be better suited to a constrained production task than a general-purpose model repeatedly instructed through a long prompt.

​

Serving is part of the model design

Fine-tuning improved planning quality and allowed us to move to a compact model, but model size alone does not determine production latency. The serving stack has to match the shape of the workload. After training, we merged the LoRA adapter into Qwen3-1.7B and served the results with vLLM on a SageMaker endpoint. vLLM exposed the same OpenAI-compatible tool-calling contract used by the rest of Smart Mode, so adopting the specialist required no change to the application interface. Planner requests have a distinctive inference profile: a largely identical prefix containing the system context and capability definitions, followed by request-specific hints and the natural-language question, with a comparatively short structured payload as output. That creates two clear sources of avoidable work: recomputing the shared prefix for every request, and repeatedly moving model weights through memory during generation. We addressed them separately. Prefix caching reduced repeated prefill work. vLLM reused the key-value cache for the identical system context and capability definitions instead of recomputing that prefix for each warm request. When the common prefix is tokenized identically and its cache blocks remain resident, the engine reuses the corresponding attention state rather than repeating the full prefill. The shared tokens are not literally free, the cache consumes GPU memory, entries can be evicted, and any change to the tokenized prefix limits reuse — but under steady traffic it removes a substantial amount of work before generation begins. AWQ W4A16 quantization reduced weight memory and bandwidth. We produced a post-training AWQ variant from the merged model, storing weights at 4-bit precision while keeping activations at 16-bit. This is distinct from QLoRA: QLoRA reduced memory while training the adapters, whereas AWQ compressed the final inference artifact. AWQ uses calibration activations to decide which weight channels to preserve, so we calibrated on a few hundred representative examples drawn from the same system-context, hint, and structured-payload distribution the planner serves, rather than on generic text. Quantization was therefore a post-training optimization of the same specialist, not a separate modeling project, and, on the same GPU, it buys throughput headroom that converts directly into cost. The broader lesson is that model size, prompt structure, cache reuse, and quantization are coupled. Fine-tuning made the behavior compact; inference engineering ensured that the specialization also produced practical latency and cost gains.

​

Conclusions

Prompt engineering was the discovery phase of this project. It made the domain rules explicit, exposed failure modes, and gave us a reference planner. Once those rules stabilized, the prompt was no longer merely runtime context. Together with the output contract and evaluation set, it had become a specification for a specialist model. This pattern is strongest when the same decisions recur frequently, change slowly, operate within a constrained action space, can be represented with validated examples, and can be scored directly. Fine-tuning is a weaker fit when the behavior is open-ended, changes continuously, or cannot be evaluated independently. The model is only one part of the transition. In Smart Mode, purpose-specific capabilities reduced the decision space; coverage-driven data represented the combinations the planner had to learn; the complete inference context was included during training; payload-level and retrieval-level evaluations kept failures attributable; and serving optimizations converted specialization into latency and cost gains. The result was not a model with more financial knowledge—retrieval still supplies the evidence. It was a compact model that performed one stable production behavior more consistently and with less repeated instruction.

That is where fine-tuning earns its place: when a mature prompt-defined workflow can be turned into a capability the product can test, version, and operate directly.

---

​

About Smart Mode

Bigdata.com Smart Mode turns natural-language financial questions into precise search over a large live corpus of news, earnings transcripts, SEC filings, analyst research, expert network calls, the web, and more. The fine-tuned planner described here is one component of that system. Explore Bigdata.com or review the developer documentation to build search and retrieval into your own agents.

---