Why Budget AI Agent Swarms Only Outperform Frontier Models on Small Budgets
An analysis of Ramp Labs' security scanning data reveals that while cheaper AI models excel on small budgets, frontier models ultimately catch difficult bugs.
Recent data from Ramp Labs demonstrates how open-weight and alternative models perform when deployed as part of parallel coding agent swarms. In an evaluation involving seven real backend security vulnerabilities, a production scan using GPT-5.5 at high reasoning effort was compared against Kimi K2.6 and DeepSeek V4 Pro, which were operated at roughly a five-fold lower cost per token.
While a full-scale scan using top-tier models can easily exceed $20,000, deploying cheaper models allows developers to purchase multiple tries for the same expenditure. Because security scanning only requires a single successful trajectory to expose a flaw, evaluating models based purely on single-try success rates hides the financial advantage of volume.
The Math Behind Model Swarms
Using the pass@k estimator from OpenAI's 2021 Codex paper, researchers calculated the probability that at least one of multiple independent tries successfully hits a target. Based on 25 recorded attempts per model for each of the seven bugs, allocating a small budget to Kimi K2.6 yields a slight victory. Five Kimi attempts find an average of 4.06 bugs, compared to 3.64 bugs for a single GPT-5.5 try.
However, this advantage evaporates quickly as the budget scales. At two GPT-5.5 tries versus ten Kimi tries, the performance gap narrows, and by three tries, GPT-5.5 pulls ahead definitively. DeepSeek V4 Pro trails behind both, failing to match the frontier model's performance even at maximum allocated tries.
What It Means for Developers
For engineering teams building agent swarms, the choice between frontier and economical models depends heavily on task complexity and budget allocation. Developers can try top AI models cheaply through one API at https://apixoai.online. Economical models excel at covering straightforward surfaces and finding easily detectable issues, such as standard configuration flaws. However, subtle and complex vulnerabilities often require the deep reasoning capabilities of frontier models, which maintain a higher ceiling when handling difficult targets.
Furthermore, cost calculations must account for the actual session length rather than just base token pricing. Models that require prolonged thinking loops consume more tokens per finished task, altering the expected price ratio in production environments.
Source: Why Cheap Kimi Agents Only Beat GPT-5.5 on a Small Budget — Towards AI. Written by the Apixo team from that report.
One key for Claude, GPT, GLM, DeepSeek and more. Pay per token with crypto.
Get your API key
