Chronograph

Graph View

2026-10-29

11 min read

Martin Rosén-Lidholm
Martin Rosén-Lidholm

Daily D4 Digest — 2026-10-29

TL;DR

  • Coding agents collapse from 63% first-stage success to 29% when features evolve over multiple stages — inherited defects go unnoticed 95% of the time, making autonomous multi-step feature work a downstream cost timebomb
  • A new paper formally models how agent harnesses silently destroy a third of tool outputs through naive JSON truncation, causing 43% more tool calls and 2× computation — a hidden cost multiplier most teams never measure
  • Concordat introduces finite-state protocol enforcement for LLM agents that eliminated all 42 unauthorized actions across airline tasks with zero loss in task success — typestate for agents is here
  • “Counting the Bandages” argues that AI time-savings claims are fundamentally misleading: review time alone caps GPT-5’s real speedup at 3.6×, and over-trust trades clock minutes for escaped defects at a worse exchange rate
  • MCP’s natural-language tool metadata creates a prompt-injection surface with 63–100% attack success rates that three evaluated defenses cannot close below 55%

Call to Action

  • Evaluate your agent harness truncation policy — check if your framework naively slices JSON tool outputs; the context pipeline paper found 2 of 7 major frameworks do this, silently destroying evidence fields
  • Pilot Concordat-style protocol enforcement on any agent handling consequential actions (refunds, account changes); the open-source library eliminated unauthorized actions with no task-success penalty
  • Adopt the “ledger view” for AI productivity metrics — stop measuring stopwatch time and start tracking escaped defects alongside time saved, per the break-even defect cost framework

D1 — Agentic Engineering

Agents fail at feature evolution, not feature creation. FeatureArc evaluated 10 coding agents on 73 multi-stage feature tasks (2–10 stages each) and found that pooled success drops from 62.7% at the first stage to just 28.5% when the complete feature is tested on the agent’s final code. The mechanism is devastating: inherited defects appear in 72.9% of failed later stages, and agents leave earlier errors unnoticed ~95% of the time. Their self-tests almost never revisit earlier stages. This is the most concrete evidence yet that agentic engineering needs spec-first checkpoints between stages — autonomous multi-step feature work without human gates is not yet viable. (Also D4: each inherited defect is a downstream cost waiting to detonate.)

Evolving Agent Teams escape local optima through supervisor-driven restructuring. A multi-agent framework from Berkeley introduces a central supervisor that diagnoses when a team is stuck and restructures agent roles, backed by a persistent file system of learning documents and evolution history. The results on GPU kernel optimization are striking: a CUDA kernel generated from scratch achieves 1.2× mean speedup over production FlashInfer kernels, winning on 36 of 47 workloads. The pattern of a meta-agent that restructures the team rather than the code is a compelling orchestration primitive. Open-sourced at github.com/caoshiyi/EAT.

Stratum proposes persistent repository memory as infrastructure, not context. The paper argues that the real bottleneck for coding agents is a memory-architecture problem, not a context-length problem: mid-size repos exceed production context windows, KV cache storage costs 10⁴–10⁵× more per byte than indexed text, and 47% of co-edited file pairs are structurally disconnected. Their tiered memory system enables a fresh agent inheriting a prior session’s notes (written by a different model) to identify root-cause files in 21/24 trials vs. 6/24 baseline, at 4.5× lower median token cost. This directly addresses the FeatureArc problem: agents need memory across sessions. (Also D4: the token cost reduction is significant.)

InfraDev reveals that coding agents can barely handle production AI infrastructure. A benchmark of 36 tasks from real Megatron-LM, vLLM, and verl pull requests found the best frontier model resolves only 38.9% of tasks. Critically, 32% of submissions that pass all official tests are rejected by hidden probes — the agents’ own tests rarely check memory, time, or network traffic. With original PR descriptions as prompts (no re-engineered task framing), no model resolves a single task. This is a sobering reality check for anyone planning to use agents for infrastructure-level work. (Also D4: the gap between “passes tests” and “actually correct” is pure downstream cost.)

Anthropic admits nobody holds a mental model of Claude Code’s codebase. The Claude Code team now has Claude writing ~90% of its own code, with pull requests running 50+ review rounds overnight and reviewers spending most effort on pre-existing (not new) code. The team openly acknowledges lacking control over the system’s full architecture. This is the “human on the loop” transition in the wild — and the candid admission of lost control is either a warning or a preview, depending on your risk appetite.

kw-bench shows that harness quality, not model size, determines agent success on investigative knowledge work. On a five-year VAT reconstruction task across three generations of SAP, two model-agnostic harnesses scored 0.528–0.542 with DeepSeek v4.1 Flash while two lab harnesses (Claude Code, Codex) scored 0.00 on the same model. The failure mode was “inertia” — inability to keep many weak signals alive across a long session. On Claude Opus 5.5, the same tasks collapsed to minutes with near-perfect scores. The implication: for complex investigative work, the harness is the product, and model upgrades can make previously impossible tasks trivial overnight.

D2 — AI in the Product

No significant D2-specific updates today. The kw-bench finding (D1) has product implications for anyone building investigative AI products in audit/tax/compliance: a weeks-long engagement now costs minutes and a few dollars on frontier models, but only with the right harness architecture. This suggests the product moat is in the orchestration layer, not the model.

D3 — Build for Agents

MCP’s tool-description surface is fundamentally attackable. The Log-To-Leak paper demonstrates a prompt injection attack through MCP’s natural-language tool metadata that achieves 62.64–100% attack success rates across 18 model-server pairs, with legitimate task completion barely affected (≤6.63pp change in 16 of 18 pairs). The attack induces a post-hoc exfiltration call after legitimate tool use. Three evaluated defenses leave residual ASR above 55% on GitHub MCP with GPT-4o, and an instruction detector flags 85.53% of benign descriptions as suspicious. The conclusion — “successful task completion can coexist with unauthorized post-hoc disclosure” — means you cannot rely on functional correctness as a proxy for security. MCP needs host-controlled authorization for secondary tool calls.

Concordat enforces agent protocols through typed sessions, not prompt engineering. This open-source Python library treats an agent as a finite-state machine where the model is offered only tools legal in the current protocol state, illegal calls are rejected before side effects occur, and human-reserved decisions can only be requested. On tau-bench airline tasks with adversarial customers pressuring agents to skip confirmation, the standard agent made unauthorized changes in 13/50 tasks (GPT-4.1) and 29/128 task-runs (Claude Haiku 5.5); the Concordat agent made zero, with no significant drop in task success. Notably, tau-bench’s own reward function didn’t detect most violations. This is the strongest evidence yet that “bounded autonomy” — constraining agent behavior through formal protocol rather than prompting — works at no performance cost.

D4 — Cost of Ownership

Agent harnesses silently multiply computation costs through information attrition. This directly undermines the 2× output / ½× downstream cost equation: a third of all tool results (3,827 of 11,189) were truncated before reaching the model, despite the context window being only 5.75% utilized. The truncation produced broken JSON that systematically deleted specific evidence fields (one field reached the model in only 4.7% of calls). The measurable downstream cost: 43% more tool calls, 2× computation, and diagnostic yield dropping from 0.61 to 0.31. Two of seven surveyed agent frameworks cut tool output the same unsafe way. This is a hidden cost multiplier that most teams aren’t even aware of — it inflates inference costs while degrading quality, the exact opposite of the D4 promise.

Claimed AI time savings evaporate when you count escaped defects. The “Counting the Bandages” paper introduces a ledger view of AI productivity that accounts for escaped defects alongside clock time. Key findings: a calibrated worker’s escaped-defect rate doesn’t depend on model quality (until it’s good enough not to check at all); checking time grows logarithmically with error rate; and “over-trust buys minutes on the clock at a larger cost in total minutes.” Re-analyzing METR’s developer study from public data, the same 246 issues support a slowdown of 19–24%. In GDPval’s review-every-output workflow, review time alone caps GPT-5’s speedup at 3.6×. The break-even defect cost framework is the right mental model for any CTO evaluating agentic ROI: if you’re not measuring escaped defects, your productivity gains are borrowed from your support tiers and incident budgets.

FeatureArc quantifies the compounding cost of autonomous multi-stage work. When agents leave inherited defects unnoticed 95% of the time across feature evolution stages, each undetected defect compounds into downstream maintenance, debugging, and incident cost. The drop from 63% to 29% success across stages is not a productivity gap — it’s a tech debt factory. Without inter-stage verification gates, the speed is definitively borrowed.

Software Civil Engineering Lens

Today’s batch is exceptionally SCE-relevant — multiple papers independently converge on the same thesis from different angles.

Concordat is the most direct implementation of SCE principles we’ve seen. It operationalizes the Decider pattern for agent behavior: a finite-state protocol that is checked when defined, enforced at runtime, and supports behavioral Liskov substitution. This is spec-driven agent development — the protocol is the blueprint, the runtime is the inspector, and the agent operates with bounded autonomy within protocol constraints. The zero-violation result with no task-success penalty is the strongest empirical case yet for “human on the loop” via formal specification rather than manual review.

MARQ directly tests the “codes and norms” pillar. By infusing INCOSE requirement quality criteria as a criterion-specific knowledge stack, MARQ raises Claude Opus 5’s F1 from 0.55 to 0.80 on expert-labeled NASA requirements. This is codified engineering standards being machine-enforced — exactly the “codes/norms” pillar of SCE. The finding that gains are model-dependent and criterion-specific argues for modular, standards-aware tooling rather than general-purpose AI.

RefactorProof stress-tests the “simulation” pillar and finds it brittle. LLM-generated formal proofs survive only 4.3–26.8% of behavior-preserving refactorings unchanged. While simple one-identifier edits repair most failures, the brittleness means that formal verification via LLM-generated proofs cannot yet serve as the “terraform plan for domain logic” that SCE envisions — the proofs are coupled to implementation details rather than specifications. This is a concrete gap in the professionalization thesis: we need proofs that are spec-coupled, not code-coupled.

The “Counting the Bandages” framework provides the missing economic model for SCE adoption. If over-trust in AI trades visible speed for invisible defects, then the break-even defect cost is the number that determines when formal specification pays for itself. The logarithmic relationship between checking time and error rate means that improving specification quality (reducing the model’s error rate) yields diminishing returns on review time but linear returns on defect prevention. This is the economic argument for investing in blueprints.

FeatureArc’s inherited-defect finding is the most damning evidence for why software needs professionalization. The 95% undetected inheritance rate across stages is what happens when there are no inspection checkpoints, no structural simulation, and no specification to verify against. In civil engineering, this would be pouring floors on uncured foundations. The parallel is no longer metaphorical — it’s operational.

Sources