R.I.P.
๐ป
Ghosted
When Does Restricting a Coding Agent to execute_code Help? A Regime $\times$ Agent-Design Ablation
July 12, 2026 ยท Grace Period ยท ๐ KDD 2026
Authors
Hong Yang, Qi Yu, Travis Desell
arXiv ID
2607.10569
Category
cs.SE: Software Engineering
Cross-listed
cs.AI,
cs.LG
Citations
0
Venue
KDD 2026
Abstract
Modern coding agents expose multiple tool surfaces -- IDE primitives, bash, and Model Context Protocol (MCP) code-execution -- and the field has shipped three contradictory claims about which one matters. We run the missing crossed comparison: an integrity-clean three-arm ablation (baseline / bash_only / code_only) on synthetic computation tasks and SWE-bench Mini modification tasks, holding model, harness, and prompts fixed, with two agents (Claude Code, OpenAI Codex CLI) so the comparison spans both regime and agent-design axes. Across the four resulting (regime, agent) cells, restricting the agent to a single execute_code MCP tool is cheaper than -- or statistically tied with -- its cheapest tool-rich rival in three cells (significantly on Artifact/Claude and SWE-bench/Codex; directionally on Artifact/Codex), with pass rates statistically tied within each cell. The lone exception is SWE-bench/Claude, where code_only is directionally costlier (+14.4%, not significant); a conditional-cost analysis localizes that gap to failure-cost on doomed-run trajectories, not a per-edit tax on successful runs. Two implications: the cheapest tool surface is jointly determined by task regime and agent design rather than by either axis alone, and the headline cost signal lives in cache-adjusted cost -- not pass rate, which is invariant across surfaces at the model sizes we evaluate. The benchmark harness, task suite, and analysis code are available at https://github.com/hyang0129/onlycodes.
Community Contributions
Found the code? Know the venue? Think something is wrong? Let us know!
๐ Similar Papers
In the same crypt โ Software Engineering
R.I.P.
๐ป
Ghosted
Microservices: yesterday, today, and tomorrow
๐
๐
The Cartographer
A Survey of Machine Learning for Big Code and Naturalness
R.I.P.
๐ป
Ghosted
An Overview on Smart Contracts: Challenges, Advances and Platforms
R.I.P.
๐ป
Ghosted
Slither: A Static Analysis Framework For Smart Contracts
R.I.P.
๐ป
Ghosted