Where Does Abacus.AI’s Smaug Model Improve Performance the Most?
Abacus.AI applied the same fine-tuning recipe to three very different open-weight models. The gains landed in completely different places & that tells you more than any single benchmark score.
TL;DR:
- Smaug Flash posts the line's largest jumps: +14.3 pts in agentic coding, +13.7 on AutomationBench, +19 on NL2Repo-Bench - directly targeting DeepSeek V4 Flash's failure mode in long tool-use loops.
- Smaug Mini (28B) beats Claude Sonnet 5, Claude Opus 4.6, and GPT-5.6 Luna on IFBench, AutomationBench, and JobBench - despite a slight dip in LiveBench agentic coding.
- Smaug Agentic shows the smallest score deltas (+0.3 overall) but ran 113 tasks over 7+ hours with a median of 78 agent steps and zero infrastructure errors - behavioral stability is the story, not the numbers.
In all three cases, the gains concentrate where the base model's gaps were widest. Same recipe, different floors.
Significance of AI Agents' Performance
An AI agent could provide an output without completing the task. In real-world scenarios, there is a need for the agent to comprehend the user query, identify an appropriate tool, analyze output, and take subsequent actions. After some number of cycles, the model would veer away from the initial goal, thus compromising on the credibility of the result.
It presents a problem for teams developing reliable agents. While a chatbot provides a response to a single request, an agent has to retain its context and make decisions based on related actions. With the extension of workflow, there would be additional cost implications as well as chances of mistakes.
Abacus.AI experienced similar shortcomings in its own operations, with agentic tasks taking several hours to complete, maintaining a large amount of context from stage to stage, and frequently invoking tools from other documents, APIs, and databases. These operations could run out of steam or not be completed successfully at all. Abacus.AI describes its findings on its research page in the following way: “Making a state-of-the-art model work like a good agent is a training challenge, not a prompting challenge.” General language models, despite their abilities, were not always ready for extended usage. Prompt caching eventually became less effective, and context grew bigger.
The basic approach was to improve the models unaided instead of trying to compensate for their shortcomings by using prompting or weight adjustments that would fix the identified problems. This way of working has resulted in creating the family of Smaug models. It is essential not only to ask how effective these models are but also where the fine-tuning works the best. The performance of the models depends greatly on the model and task type.
What Is the Smaug Line?
Smaug is a family of three open-weight models fine-tuned by Abacus.AI for agentic workloads. Each model is built on a different open-weight base, chosen for a distinct point on the capability–efficiency curve. All three are publicly available on Hugging Face.
A common methodology involving human-curated, actual agentic trails and artificial data based on hard cases was used for all the three bases, which is why the gain patterns are comparable and the differences are meaningful.
| Model | Base Model | Optimized For | Size |
|---|---|---|---|
| Smaug Flash | DeepSeek V4 Flash 0731 | Always-on enterprise agents, speed + cost | 304B |
| Smaug Mini | Qwen3.8-27B | Multimodal, compact, one-off tasks | 28B |
| Smaug Agentic | Kimi K3 | Frontier-scale, long-horizon coding | 2.8T (MoE) |
Applying the same recipe to three very different bases is itself a research claim: that the methodology is general enough to produce consistent, measurable improvements regardless of architecture or scale. What follows examines each model's results to determine where that claim holds strongest — and where trade-offs emerge.
Smaug Flash: The Biggest Gains in the Line
Smaug Flash is where the Smaug methodology delivers its largest benchmark jumps. It is an agentic fine-tune of DeepSeek V4 Flash, often known as the workhorse of the line, built for enterprise self-improving agents that run continuously. Its base, DeepSeek V4 Flash, is fast and cost-effective but susceptible to spins and confusion in long-context tool use. The fine-tuning targeted that specific failure mode.
The results on LiveBench reflect this precision. The +14.3 point gain in agentic coding ranging from 46.8 to 61.1 is the largest single-category delta in the entire Smaug line. On AutomationBench, the score improved by 13.7 points (25.1 to 38.83), and on NL2Repo-Bench the gain was 19 points (54.2 to 73.3). These are not marginal improvements; they represent the model closing a genuine capability gap in exactly the workflows it was designed for.
Smaug Flash vs DeepSeek V4 Flash vs Claude Sonnet 5:
¹ = vendor-reported score, not run under Abacus.AI's harness.
LiveBench Category-by-Category Breakdown:
These modest reductions in Data Analysis (-0.2) and Reasoning (-0.4) could be seen as a trade-off since it is more important to hone these skills to be useful for production agents, for example, the ability to use tools correctly and process longer contexts, than to retain full performance in reasoning. In terms of the benchmarks evaluating these skills, Smaug Flash outperforms Claude Sonnet 5.
Smaug Mini: The Surprising Story
Smaug Mini is an advanced iteration of the Qwen3.8-27B model and the small-scale version of the Smaug series with 27 billion parameters. The model is intended for multimodal tasks such as image/video interpretation, instruction following, and tool use.
With its comparatively smaller size, deployment could potentially become easier under certain conditions of hardware and inference configuration. It is important to note that Smaug Mini has achieved higher performance than several advanced closed models in certain benchmarks where the latter usually excel.
Smaug Mini vs Base and Frontier Closed Models:
¹ = vendor-reported score, not run under Abacus.AI's harness.
LiveBench Category Profile for Smaug Mini:
Basically, Smaug Flash has the biggest LiveBench gain in agentic coding, but because Qwen3.8-27B, the base of Smaug Mini is already strong at coding-style tasks, it’s largest gains are in Mathematics (+3.5) and Reasoning (+2.8) which leaves no room for improvement. The fine-tuning found the gaps that were actually open. On NL2Repo-Bench, Mini also gained 13.5 points over its base (42.3 to 55.8), a gain worth noting for code-related workflows.
The twist is important: Smaug Mini's agentic coding score (-0.6) is actually slightly lower than its base, yet it leads Claude Sonnet 5, Claude Opus 4.6, and GPT-5.6 Luna on IFBench (instruction following), AutomationBench, and JobBench. JobBench in particular shows a 17.1 point gain over the base (33.4 to 50.5). Herein lies the distinction between what a benchmark category is able to measure and what the model is capable of achieving in reality — since different benchmarks measure different abilities, and a drop in one area does not mean a lack in others.
Smaug Agentic: When Behavior Matters More Than Numbers
Smaug Agentic is the largest one in this line-up. It's a fine-tuning of Kimi K3, which is the state-of-the-art Mixture of Experts model from Moonshot AI, with 2.8 trillion parameters. Its benchmark deltas over the base are the smallest of the three Smaug models. The research page states this directly: "the behavioral change is bigger than the deltas."
Smaug Agentic vs Kimi K3 Base:
LiveBench Category Profile for Smaug Agentic:
Two benchmarks where the base model wins should be stated clearly: Terminal-Bench 2.1 (86.5 vs 88.3) and MMMU-Pro (81.0 vs 81.6). These are not hidden in footnotes in the official data, and they belong in any honest analysis.
What the numbers do not capture is what Abacus.AI describes as the model's primary behavioral improvement: suppressing runaway reasoning while preserving normal deliberation. Across 113 agentic coding tasks and more than seven hours of continuous operation, Smaug Agentic ran a median of 78 agent steps per task with no infrastructure errors and no timeouts. At frontier scale, the fine-tuning did not produce headline score jumps, rather it produced a model that does not break down in the long loops where frontier-scale models are actually deployed.
The Training Methodology Behind the Gains
The same methodology was applied to all three Smaug models. It combines human-curated traces from real agentic work with synthetic data grounded in hard, challenging examples. Training data is filtered to remove instances that exploit the scoring criteria of a task rather than actually solving it. The goal is to reward genuine problem-solving, not format compliance.
What differs across the three models is the specific fine-tuning approach:
- Smaug Flash: The attention factor matrices are alone modified with the help of three LoRA adapters, two of which have been trained by supervised learning and one with preference training. The rest of the components stay unchanged and byte identical to that of the official DeepSeek V4 Flash release. This ensures that the model is fully compatible with the base model on any inference stack, with the ability to have full 1-million-token context window.
- Smaug Agentic: Supervised fine-tune of Kimi K3 with reasoning tokens masked from the loss during training. This preserves the model's native thinking process while teaching better tool-use habits. Every architectural parameter is unchanged from the base, making it a drop-in replacement for any inference stack that serves Kimi K3.
- Smaug Mini: The same general methodology applied to a 27B dense base, producing consistent patterns: gains in the categories where the base had the most headroom, and trade-offs in areas already well-covered.
"The core philosophy of this methodology is captured in one quote on the Abacus.AI's research page: Most fine-tuning silently overwrites the reasoning of a model for the sake of task compliance – you get the format, but you lose the mind.” Smaug training recipes try to get around this pitfall by using the training budget on data which improves the model, not data which leverages benchmark scores. This might be a factor explaining the significant increase in agentic coding (+14.3 points) seen in Smaug Flash – its fine-tuning addressed the weakness of the base model."
What the Benchmarks Are Actually Measuring
Understanding where Smaug improves most requires understanding what each benchmark is designed to measure.
LiveBench is an ICLR Spotlight paper from Abacus.AI. It aims to prevent data contamination by generating questions based on the most up-to-date sources such as math contests, LeetCode questions, arXiv abstracts, news articles, and Kaggle datasets. Questions are refreshed every month to make it harder for models to have memorized the answers before training. Scoring is done based on ground truth, not LLM as a judge scoring method, which can be erroneous on challenging prompts. There are seven categories in LiveBench: Reasoning, Coding, Agentic Coding, Instruction Following, Mathematics, Language, and Data Analysis.
The other benchmarks each measure narrower skills:
- AutomationBench — real-world tool-use automation (600 public tasks, strict pass)
- DeepSWE 1.1 — software engineering executed by coding agents
- NL2Repo-Bench — translating natural language to repository-level code
- IFBench — instruction following precision (300 tasks, prompt-level loose)
- JobBench — multi-step job-level task completion (65 tasks, official protocol)
- GPQA Diamond — graduate-level science reasoning
- SciCode — scientific coding
- AA-LCR — long-context reasoning
- MMMU-Pro — multimodal understanding
Because these benchmarks measure different, though related, skills, a high score on one does not guarantee a high score on another. That is why Smaug Mini leads all compared closed models on AutomationBench and JobBench while showing a -0.6 delta in LiveBench's Agentic Coding category. They are not the same measure. It is also why some competitor scores marked ¹ in the tables above which are the vendor-reported figures must be interpreted carefully, not run under Abacus.AI's own harness.
Where Gains Are Biggest: Comparison Across All Three Models
| Category / Benchmark | Smaug Flash | Smaug Mini | Smaug Agentic |
|---|---|---|---|
| LiveBench Agentic Coding (delta vs base) | +14.3 ★ | -0.6 | +2.4 |
| LiveBench Instruction Following | +3.8 | +1.2 | -0.4 |
| LiveBench Mathematics | +3.3 | +3.5 ★ | -0.5 |
| LiveBench Reasoning | -0.4 | +2.8 | -0.4 |
| AutomationBench (delta vs base) | +13.7 ★ | +4.5 | +0.2 |
| NL2Repo-Bench (delta vs base) | +19.0 ★ | +13.5 | — |
| Behavioral endurance at scale | Stable enterprise loops | Compact, one-off tasks | 78 steps/task, 7+ hours, zero timeouts ★ |
The pattern across all three is consistent with one explanation: the gains concentrate where the base model's gaps were widest and where the training data was designed to push. Flash's base had the largest failure mode in agentic coding and hence it was prone to spinning in long tool-use loops in return, the fine-tuning produced the largest gain there (+14.3). Mini's base was already strong at coding, so the gains spread across math, reasoning, and language instead. Agentic's base was already at frontier capability, leaving little room for benchmark score improvement; the fine-tuning focused on behavioral reliability rather than raw capability.
Implications for Practical Use
It is more helpful when evaluating the use of Smaug models to start with the workload that the model is designed to handle.
For applications that demand sustained, long-context tool use, document processing, or repeated API calls across hours-long workflows, Smaug Flash's +14.3 agentic coding and +13.7 AutomationBench gains over its base are the most directly relevant data. Where multimodality and instruction-following tasks are concerned, and where a model that fits on a single GPU at 28B model size is desired, the relevant comparisons come from Smaug Mini’s IFBench score (82.0), AutomationBench (41.8), and JobBench (50.5), all beating Claude Sonnet 5 and GPT-5.6 Luna. Where longevity in hundreds of steps per agent is required without infrastructural failure, the behavioral stability information from Smaug Agentic (median: 78 steps, 7+ hours, zero timeouts from 113 tasks) carries far more weight than any benchmark delta.
Other useful metrics besides the benchmarks are: task completion ratio, latency at context length as in production, cost per thousand agent steps, and reproducibility.
Open weights provide access and deployment flexibility, but do not guarantee efficiency or ease of integration without testing against the actual workload. Hardware configuration, context length, and task difficulty all affect inference behavior in ways that benchmark scores cannot predict.
The benchmark results are a starting point for identifying which models are worth evaluating — not a substitute for testing on the actual task.
Conclusion
Each improvement seen in the Smaug series depends on what particular weakness this version’s fine-tuning aimed to solve. For example, Smaug Flash exhibits the biggest improvements in its own class, increasing scores by 14.3 points for LiveBench agentic coding, 13.7 points for AutomationBench, and 19 points for NL2Repo-Bench. This is because of the weaknesses that the original model had with the extended multi-step agent workflow. Smaug Mini, on the other hand, demonstrates improvements in mathematics (+3.5), reasoning (+2.8), and language (+2.7). It beats the mentioned frontier closed models in instruction following and job-level automation in spite of being a 28 billion parameter model. On the other hand, Smaug Mini shows decline in agentic coding score (by 0.6 points) in comparison with its original model. There is also a unique case of Smaug Agentic, which shows a modest improvement of +0.3 points but has high operational performance that is defined by 78 agent steps per task, seven plus hours of run time, and zero infrastructure errors.
The results stand out by virtue of their divergence from one model to another. The same training regimen is reflected differently based on the specific model that is used and the constraints of that particular model. The results also include information about those parts of the testing procedure where there is no improvement in performance. For instance, the Kimi K3 base model outperforms Smaug Agentic in Terminal-Bench 2.1 and MMMU-Pro while Smaug Mini shows some reduction in performance in LiveBench agentic coding. Including such limitations makes the results of this evaluation exercise more realistic.
Developers seeking a performance evaluation of various open-weight models for use in agent-based applications would find the results of Smaug family very useful. Instead of identifying the best model for any application, the performance results would help understand which models might be better under certain circumstances.