Chronograph

Graph View

2026-10-31

11 min read

Martin Rosén-Lidholm
Martin Rosén-Lidholm

Daily D4 Digest — 2026-10-31

TL;DR

  • Agentic Process Evidence (APE) proposes signed, tamper-evident attestations for every agent action in the SDLC — the first serious governance primitive for agentic engineering teams.
  • Multi-agent LLM pipelines cost 3.22× more than a single strong call and solve no more tasks on coding benchmarks; the real savings come from per-task model routing, not decomposition.
  • 1 in 17 agent skills in the wild contains oversight-loosening directives (e.g., --no-verify), and 70% apply unconditionally — a supply-chain risk hiding in plain sight.
  • Skeleton Coding and elicitation-first architectures both demonstrate that specification before generation dramatically outperforms post-hoc review, cutting errors 27–48%.
  • Prefill (input context) energy cost scales quadratically, not linearly — agentic workflows with large contexts are far more expensive than output-token-based estimates suggest.

Call to Action

  • Audit your agent skill supply chain: scan SKILL.md files for oversight-loosening directives using the codebook from this study; existing open-source scanners catch less than half of them.
  • Implement process attestation for agentic PRs: evaluate the APE framework as a starting point for linking agent sessions to outcomes with signed evidence.
  • Benchmark your multi-agent pipelines against single-call baselines using the coordination tax framework before investing in decomposition architectures — per-task model routing may deliver more value.

D1 — Agentic Engineering

Coding agents overwhelmingly follow repository precedent over task intent — and explicit priority rules fix this. A controlled study across 5,625 trials found that when coding agents encounter conflicts between existing code patterns and new instructions, they follow local file precedent 96% of the time (versus 4% for sibling-resource files). File placement alone can swing adoption from 0% to 100%. Critically, written priority rules loaded via harness achieved near-perfect compliance (117/120 in open-source repos), while code examples alone were unreliable. This has immediate implications for how teams structure their repositories and architectural decision records for agentic consumption. (Also D4: without explicit design-decision traceability, agents silently accumulate inconsistent implementation choices — a direct tech debt vector.)

Skeleton Coding reduces agent errors by 48% by specifying program structure alongside behavior. Researchers model AI-assisted coding as a control problem and show that providing structural specifications (data types, modules, function signatures) alongside tests exponentially reduces the number of behavioral specs needed to steer an agent to the intended program. Skeletons narrow the gap between small and large models by ~52%, lower implementation cost by 15–26×, and reduce code erosion by 24.7%. The core insight: when structure is left to the agent, developers cannot determine a priori which tests suffice without reverse-engineering thousands of lines of generated code. (Also SCE: this is specification-driven development in action.)

Agentic Process Evidence (APE) creates signed, machine-readable audit trails for every agent action in the SDLC. This paper directly confronts the fact that git logs were designed for human commits and cannot capture which model an agent used, which tools it called, or whether a human was accountable. APE introduces tamper-evident attestations aligned with in-toto, enabling Policy-as-Code control over agentic processes. It reframes AI agents as an insider threat vector and proposes governance primitives for supply-chain security, policy guardrails, and task alignment. (Cross-cutting D1/D3/D4: this is infrastructure for the “human on the loop” transition.)

Elicitation-first architecture recovers missing requirements before code generation, outperforming post-hoc review. A study on agentic smart contract synthesis shows that having the agent ask a short checklist of clarifying questions before writing code matters far more than reviewing or patching afterwards — across three AI models. Automated security fixes often broke requested behavior, and common AI judges missed obvious mistakes. The study proposes checking generated contracts against independently written executable answer keys, returning disagreements to the client rather than silently rewriting. (Also D2/SCE: this is the Specify → Verify loop applied to irreversible deployments.)

EasyProver saturates formal verification benchmarks using recursive agents with no task-specific training. The system proves 669/672 PutnamBench problems, all IMO 2026 formalizations, and 597/600 NTP4VC verification conditions using an off-the-shelf frontier model with isolated contexts, async search, and parallel tactic probing. To push beyond saturation, they introduce Aegis, which converts competitive programming problems into verification tasks over real Rust implementations — achieving 52.7% proof rate on 2,000 statements within 30 minutes each. This demonstrates that agentic architectures can make formal verification practical at scale.

D2 — AI in the Product

TreasuryQA Bench tests agents on real scanned financial archives accessed via MCP. This benchmark requires agents to search 1,435 scanned U.S. Treasury PDFs through Dropbox’s MCP server, perform OCR, and return exact figures. The best agent (Claude Opus 5.5) solves only 64.8% of tasks; the worst (GPT-5.6 Terra) manages 16.7%. Most failures return a wrong number rather than no number, with digit misreads from scans as the primary cause. The benchmark highlights that production document-agent workflows remain far from reliable, particularly with degraded scan quality — a critical consideration before deploying financial document agents to end users.

Decompose, Don’t Expand argues for extracting deterministic sub-tasks from AI systems by construction. This paper demonstrates “parity by construction”: once the deterministic portion of a task is extracted to a host system, model-size performance curves flatten — a 494M-parameter checkpoint performs identically to a frontier Opus-class model on the extracted sub-tasks. The only task that remains model-dependent is the irreducibly generative residue (labelling). This is a design discipline for embedding AI in products: identify the deterministic surface, build a closed grammar over it, and confine the model to the fuzzy core. No fine-tuning required.

D3 — Build for Agents

Agent Transfer Protocol (ATP) proposes domain-mediated, asynchronous agent-to-agent communication. ATP addresses a fundamental infrastructure gap for the agentic web: cross-organization agent collaboration that survives instance restarts, network interruptions, and offline periods. Administrative domains maintain stable addresses and message stores for logical agents; sending domains sign complete message envelopes for independent re-verification after the connection ends. The prototype demonstrates workflow completion after instances went offline or were replaced. This is the email/SMTP analogy for A2A — store-and-forward with cryptographic provenance. (Also D4: domain mediation adds latency overhead but eliminates the operational cost of keeping agent instances continuously online.)

1 in 17 agent skills contains oversight-loosening directives, spreading through a few upstream skill families. Analysis of 1.53 million skills from GitSkills reveals ~88,000 skills with directives that tell agents to proceed without approval, bypass checks, or skip human check-ins. 70.1% apply unconditionally. Skills that run shell code, install packages, or use sudo have 1.7–1.8× the odds of containing loosening directives. Just 2.4% of skill families account for half of all loosening-carrying skill copies — the supply chain concentrates risk. Open-source scanners catch at most 46% of these directives with precision below 0.4. (Also D1/D4: this is the agent-skill equivalent of dependency vulnerabilities, but for autonomy boundaries.)

A six-module scanner detects both local and composite vulnerabilities in agent skills with 95.9% recall. The scanner addresses not just malicious skills but unintended vulnerabilities in benign public repositories, including multi-skill interaction risks that existing tools miss entirely. In 1,587 skills from top GitHub repos, analysts confirmed 53 of 59 findings (89.8% precision). The Composite Analysis module models dependencies among skills through shared resources and instruction-mediated interactions, achieving 0.991 F1. Mitigations prevented PoC attacks in 47/55 cases while preserving functionality.

D4 — Cost of Ownership

Multi-agent decomposition pipelines cost 3.22× as much as a single strong call and solve no more tasks — directly undermining the 2× output / ½× cost equation. A rigorous study measuring coordination tax vs. heterogeneity dividend across 2,430 runs finds that planner-coder-reviewer pipelines using OFP are costlier, slower (3.24×), and no more accurate than a single call. Moving auxiliary roles to a cheap model cuts cost to 0.39× but loses 6.7 accuracy points — and the break-even forecast misses by 19% because cheap roles inflate token volume. The real value lies in per-task model routing: an oracle choice between two single calls solves 94.3% of runs at half the strong call’s cost per solved task. The lesson: invest in smart routing before multi-agent orchestration.

Prefill energy cost scales quadratically with input length, making agentic context windows far more expensive than output-focused estimates suggest. Systematic profiling across a wide range of token lengths shows that while decode (output) costs scale linearly, prefill (input) costs exhibit both linear and quadratic scaling. For agentic workflows that stuff large contexts — tool outputs, conversation histories, retrieved documents — this means energy and compute costs grow faster than naively expected. The study also validates compute time as a reliable proxy for energy consumption, enabling cost estimation even without direct energy measurements. Teams should monitor and optimize input context size, not just output tokens, when managing inference budgets.

Software Civil Engineering Lens

Today’s batch is unusually rich for the SCE thesis — multiple papers independently converge on the same conclusion: specification before generation is the critical control point, not post-hoc review.

Skeleton Coding is the clearest evidence yet for spec-driven development in practice. By modeling AI-assisted coding as a control problem and showing that structural specifications exponentially reduce the behavioral specs needed, the paper provides a formal argument for why “blueprints” (Event Models, skeleton specs) are not optional overhead but necessary control surfaces. The 15–26× cost reduction and 52% narrowing of the small-vs-large model gap directly supports the SCE claim that professionalized specification relocates human judgment to a higher control plane — and makes the choice of AI “material” (model size) less consequential.

Elicitation-first smart contract synthesis independently validates the same principle in a domain where errors are irreversible (deployed smart contracts). The finding that “asking first matters far more than reviewing or patching afterwards” is the Specify → Plan → Verify → Apply → Observe lifecycle proving its worth. The executable answer key is precisely the “simulation” step — terraform plan for domain logic.

Decompose, Don’t Expand offers a material science analogy: just as civil engineers choose materials by their known properties rather than hoping for universal materials, this paper argues for choosing deterministic solutions for deterministic sub-tasks and confining the model to the irreducibly generative residue. “Parity by construction” is essentially a datasheet guarantee — the system’s behavior is independent of which model you plug in, as long as the cut is correct.

APE (Agentic Process Evidence) addresses the licensure and accountability gap head-on. The paper’s framing — “every governance framework was built around one assumption: a human made the decision; that assumption no longer holds” — is the SCE thesis stated plainly. Signed attestations linking agent sessions to outcomes are the construction inspection records of agentic engineering.

The oversight-loosening directives study is a sobering counterpoint: while the field moves toward bounded autonomy, the actual ecosystem is moving in the opposite direction. 88,000 skills actively telling agents to remove human checkpoints, concentrated in a few upstream families, is the software equivalent of load-bearing walls being removed without engineering review. This is exactly the kind of failure that SCE’s codes/norms pillar is meant to prevent.

Net assessment: today provides the strongest single-day evidence I’ve seen that the field is converging on specification-driven approaches not from theory but from empirical necessity. The gap is no longer “should we do this?” but “who builds the tooling and standards to make it the default?”

Sources