This weblog is collectively written by Md Rahman, Arkaprabho Ghosh, Navin Bilwar, and Desh Shukla.
Government abstract
Cisco IT not too long ago evaluated fine-tuning embedding fashions utilizing NVIDIA Nemotron RAG fine-tuning recipe as a part of an effort to enhance retrieval accuracy for domain-specific enterprise information. The target was to not redesign present retrieval-augmented era (RAG) programs, however to know whether or not focused embedding fine-tuning might materially enhance semantic search high quality with cheap effort and quick turnaround. By this experiment, Cisco was capable of validate firsthand that embedding fine-tuning, mixed with artificial information era, can ship measurable accuracy positive factors inside a short while body. The experiment additionally demonstrated sturdy time-to-value, enabling speedy iteration and clear efficiency alerts with out lengthy coaching cycles or intensive guide labeling. The lowered turnaround of just a few days to know the quick advantages was a key end result of this collaboration.
The embedding mannequin coaching and analysis workflow was executed on Cisco AI PODs working Cisco UCS 885A infrastructure powered by NVIDIA HGX platform.
Drawback assertion
Previous to conducting this experiment, Cisco had carried out comparable embedding fine-tuning experiments utilizing earlier era fashions and smaller scale infrastructure. These prior efforts required important guide tuning of hyperparameters equivalent to batch measurement and variety of epochs, and outcomes have been typically tough to stabilize. Iteration cycles have been lengthy, making it difficult to discover completely different configurations or scale experiments. Regardless of some localized enhancements, key phrase search remained needed for a lot of domain-specific retrieval situations. There was additionally no standardized, end-to-end workflow that engineering groups might execute shortly and consider constantly throughout runs. Usually, these efforts would take weeks to months of guide effort for unsure positive factors.
How the effective‑tuning went and time to worth
On this experiment, Cisco used the NVIDIA NeMo Retriever embedding finetuning recipe, leveraging artificial information era to supply coaching alerts from present corpora. The recipe runs via 5 distinct levels: artificial information era (SDG), information preparation with hard-negative mining, contrastive fine-tuning, BEIR analysis, and ONNX mannequin export. The workflow was capable of run end-to-end efficiently. All experiments ran on a single NVIDIA H200 143 GB GPU hosted inside Cisco AI Pods constructed on Cisco UCS 885A programs. Finetuning runs accomplished inside hours of coaching time, enabling speedy experimentation throughout a number of dataset sizes and configurations. Using artificial information era eradicated the necessity for guide labeling, considerably decreasing overhead. This strategy allowed Cisco to iterate shortly, observe efficiency traits early, and validate whether or not embedding fine-tuning was price additional funding. The general time-to-value was considerably shorter than earlier efforts, with significant insights gained after solely a small variety of runs.
The five-stage pipeline structure:
Timings primarily based on ~925 paperwork / ~9,200 QA pairs / ~7,800 coaching examples on a single NVIDIA H200 GPU working on Cisco AI Pods with Cisco UCS 885A infrastructure. Precise period scales with information quantity.
Accuracy positive factors noticed
Throughout a number of experiments, the outcomes confirmed constant, measurable enhancements. Superb-tuning the NVIDIA 1-billion-parameter NV-EmbedQA mannequin on artificial domain-specific information yielded positive factors throughout all retrieval metrics, with NDCG@1 positive factors of +7.1 to +7.3 absolute factors (+9.9% to +11.1% relative). Recall@10 improved by as much as +6.8 factors (+8.5%), and MAP@10 by as much as +6.5 factors (+9.7%). Utilizing an on-premise 120B-parameter LLM for artificial information era, your entire pipeline ran with zero exterior API prices and with the information staying utterly on prem ensured information privateness. These positive factors held whilst dataset measurement elevated and retrieval duties grew to become more difficult. Importantly, enhancements have been noticed on domain-specific queries that beforehand carried out poorly with base embedding fashions. Whereas these outcomes characterize an preliminary baseline reasonably than a completely optimized end result, they offered sturdy affirmation that embedding fine-tuning can materially enhance retrieval high quality for enterprise-specific information.
Abstract of experiments
Desk 1. Retrieval efficiency comparability between the bottom embedding mannequin and the contrastively fine-tuned mannequin throughout two dataset sizes (334 and 925 paperwork). Superb-tuning constantly improves rating high quality throughout all BEIR analysis metrics.
Key Observations:
- Superb-tuning constantly improved retrieval high quality throughout all metrics.
- NDCG@1 confirmed the most important enchancment in top-level relevance.
- Good points have been steady throughout the 2 dataset sizes (334 and 925 paperwork).
- Improved Recall@10 and Map@10 positive factors indicative of higher protection and rating than the bottom embedding mannequin.
What shocked us
Probably the most surprising discovering was how shortly the recipe delivered actionable outcomes. Inside just a few days of beginning the experiment, we had measurable accuracy enhancements — a stark distinction to earlier efforts that took weeks to months. The artificial information era strategy produced coaching alerts of ample high quality to drive significant positive factors with out a single manually labeled instance. We have been additionally shocked by how nicely the enhancements generalized throughout question varieties, together with the rare-token identifier queries that had traditionally been the weakest level for semantic search.
Subsequent steps with engagement
Constructing on these outcomes, Cisco will proceed working with NVIDIA to systematically push accuracy additional. The following section of labor will focus on:
- Utilizing a hard and fast analysis set throughout runs in order that metrics can be immediately comparable
- Tuning the educational fee (attempting default, half, and double) and growing epochs from 3 to five
- Scaling coaching information to ~100K QA pairs to seek out the saturation level for the area
- Utilizing a bigger or higher-quality LLM for artificial information era to enhance QA pair constancy
- Making use of 10% warmup with cosine decay for extra steady convergence
- Growing hard-negative mining from 5 to 10 negatives per question for a stronger contrastive sign
- Refining artificial information era prompts to higher emphasize uncommon and domain-specific phrases — bug IDs, product identifiers, firmware variations — the place base fashions wrestle most
- Exploring chunk-aware coaching: utilizing actual doc chunks from a manufacturing vector database because the retrieval corpus, producing questions in opposition to these chunks by way of the LLM, and mapping every query to its optimistic chunk and hard-negative chunks — coaching the mannequin on the identical information distribution it will encounter in manufacturing, the place solutions could also be buried in longer textual content and chunking methods will range
Long term, the engagement will broaden to incorporate re-ranker fine-tuning and broader retrieval optimization as a part of a full end-to-end RAG enchancment effort.
Worth of the fine-tuning embedding mannequin
This experiment helps that leveraging a fine-tuning embedding mannequin can speed up time to manufacturing by offering a validated, end-to-end fine-tuning workflow that delivers measurable enhancements in days reasonably than months. The concepts and findings from this work are actively shaping the recipe’s evolution, whereas Cisco positive factors early entry to a maturing pipeline that shortens the trail from experimentation to manufacturing. The work additionally demonstrates how Cisco AI Pods primarily based on Cisco UCS 885A programs and NVIDIA H200 GPUs can present an efficient enterprise infrastructure basis for speedy embedding mannequin adaptation.
Key fine-tuning embedding mannequin advantages for companies
- Shield proprietary information (on-premises execution)
- Scale back assist prices (sooner decision, fewer escalations)
- No cloud API dependency (zero exterior prices)
- Quick time-to-value (full end-to-end pipeline — all 5 levels together with SDG, mining, coaching, analysis, and export — completes in 2-5 hours on a single GPU)
Key fine-tuning embedding mannequin advantages for builders
- No guide annotation required (artificial information era)
- Modular, hackable structure (5 distinct levels: SDG → Information Prep → Superb-Tune → Consider → Export)
- Manufacturing-ready outputs (ONNX export)
- Constructed-in analysis (BEIR — Benchmarking Info Retrieval — framework)
- Laborious unfavorable mining included (computerized high quality enhance)
Get began
The fine-tuning recipe for Llama Nemotron Embed 1B mannequin is on the market now as a whole, production-ready pipeline. Whether or not you’re constructing enterprise search, RAG functions, or domain-specific retrieval programs, this recipe supplies a transparent path from uncooked paperwork to deployed, domain-adapted embeddings.
Able to fine-tune your individual embedding mannequin?
👉 Discover the Nemotron Embed Superb-Tuning Recipe on GitHub
