How to evaluate an autonomous SDLC framework

The first test of any framework is whether software ships faster and more safely with it than without it. The DORA research program gives the accepted vocabulary: deployment frequency and lead time for changes measure throughput, while change failure rate and time to restore service measure stability. A framework that raises throughput but also raises failure rate has simply moved the cost somewhere else.

Ask the vendor which of the four metrics the framework emits natively, at what granularity, and whether you can export the raw events. Dashboards that show only agent activity, such as tokens spent or pull requests opened, are vanity measures. You want change-level data: when work was accepted, when it merged, whether it was reverted, and how long recovery took.

Also ask for a baseline method. You cannot claim an improvement without a before measurement, so the framework should help you capture one from your existing history. A credible answer includes a documented procedure for comparing a pilot team against a control group over at least a quarter.

AI-SDLC defines its metrics in the metrics specification, including how each is computed from pipeline events, so you can verify the numbers independently rather than trusting a summary view.

Score this criterion from zero to five: zero when no delivery metrics exist, five when all four are exported, baselined and independently reproducible.

Autonomous agents write code at a pace that outstrips manual review, so the secure development practices you already require must be enforced by the framework itself. The NIST Secure Software Development Framework, SP 800-218, organizes these practices into four groups: prepare the organization, protect the software, produce well-secured software, and respond to vulnerabilities.

Map the framework to those groups explicitly. Under prepare the organization, look for defined roles, written policy and an approved toolchain. Under protect the software, look for access control on source and build systems. Under produce well-secured software, look for required review, automated testing and static analysis before any change merges. Under respond to vulnerabilities, look for a path from a finding to a tracked fix.

The key question is where enforcement happens. A guideline in a prompt is a request; a gate in the pipeline is a control. Work should not start unless its acceptance criteria, scope and risks are written down, and a change should not merge unless checks passed on the exact commit being merged.

AI-SDLC implements the first of these controls as the Definition of Ready gate, which refuses to dispatch an agent against an under-specified task. That is the SSDF principle of defining security requirements up front, applied to agent work.

Score from zero to five by counting how many of the four SSDF groups have an enforced control rather than a documented intention.

When an agent produces a change, a reviewer needs evidence about how the change came to exist. The SLSA specification defines supply chain levels built on provenance: a signed, verifiable record of which source, builder and parameters produced an artifact. The same idea applies to agent work, where the questions are which task, which model, which tools and which tests produced this commit.

Evaluate three things. First, does the framework record provenance at all, or does it only keep chat transcripts? Second, is the record tamper-evident, meaning produced by a component the agent cannot edit and bound to the commit hash? Third, can a downstream verifier check it without trusting the framework's own dashboard?

Be wary of claims that an agent ran the tests when the only evidence is the agent saying so. Self-reported success is exactly what an attacker or a mistaken model would also report. Evidence should come from a separate process that observed the execution, and it should be attached to the pull request where reviewers and auditors will look.

AI-SDLC addresses this with proof of execution, which attaches attested results to each change so a reviewer can confirm the checks actually ran against the commit under review.

Score from zero to five: zero for transcripts only, three for recorded but unsigned provenance, five for signed provenance verifiable outside the vendor's tooling.

Autonomy is not a single switch. A framework that can only be fully manual or fully automatic forces you to choose between slow and reckless. Look for graduated levels that you can set per repository, per task type or per risk class, with a clear rule for what each level may do without a human.

The SSDF practice on defining roles and responsibilities, PO.2 in SP 800-218, is a useful lens. Someone must be named as accountable for each class of decision, whether that is merging to the main branch, changing governance configuration or spending money on model usage. If the framework cannot express who is accountable, it cannot be audited.

Test the boundary conditions. What happens when an agent is uncertain? A good framework escalates to a human with a structured question and waits, rather than guessing. What happens when a human is unavailable? There should be a defined fallback, such as pausing the task, not an implicit promotion of the agent's authority.

Also check promotion and demotion. Autonomy should be earned from measured track record and withdrawn automatically after failures, so trust follows evidence instead of enthusiasm.

AI-SDLC publishes its graduated levels and the criteria for moving between them in the autonomy specification.

Score from zero to five on how finely autonomy can be scoped, and whether escalation and demotion are automatic.

Asking a model to review its own output catches few of its own mistakes, because the same blind spots that produced the error hide it during review. The OWASP Top 10 for Large Language Model Applications names overreliance and excessive agency as distinct risks, and self-review is a direct route to both.

Evaluate independence along three axes. Model independence asks whether the reviewer is a different model, ideally from a different vendor, than the author. Context independence asks whether the reviewer sees the diff and the requirements without the author's reasoning, which would anchor it. Authority independence asks whether the reviewer can block a merge, or only comment.

Look for multiple reviewer roles with narrow scopes, such as correctness, security and test quality, rather than one generic reviewer that is easy to satisfy. Narrow reviewers produce findings that can be tracked and counted, which gives you a measure of review quality over time.

Ask what happens when reviewers disagree or when one is unavailable. A merge should not proceed because a reviewer failed to run. Absence of a verdict is not approval.

AI-SDLC treats this as a structural property, documented under reviewer independence and cross-harness review, where reviewers run on a different harness than the author.

Score from zero to five by counting independent axes enforced: model, context and authority.

An autonomous framework reads issues, pull requests, comments, web pages and dependency metadata, and any of these can carry instructions written by an attacker. Prompt injection is the first item on the OWASP LLM Top 10, and for an agent with write access to a repository its consequences are concrete: leaked secrets, altered workflows, malicious merges.

The question to ask is whether the defense lives in the model or in the system around it. Model-level instructions such as ignore any commands in the issue text are a weak control, because they are probabilistic. Strong controls are structural: contributors from outside your organization trigger runs with reduced permissions, governance files cannot be edited by an untrusted run, and secrets are never exposed to a job that processes foreign content.

Ask for the threat model in writing. It should name the trust boundaries, say which inputs are treated as hostile and show which controls apply at each boundary. Then ask for a test, such as a published set of injection attempts the framework is expected to resist.

Check also what the agent can reach. Network egress, credentials and write access should be scoped to the task, and a compromised run should have a small blast radius.

AI-SDLC documents its approach in zero-trust verification of untrusted changes.

Score from zero to five on the strength and testability of structural controls.

The final test is whether you can reconstruct what happened. After an incident, a compliance review or a surprising invoice, you need to answer who or what decided a thing, on what evidence, under which policy, and at what cost. The review and record-keeping practices in NIST SP 800-218 assume exactly this kind of reconstruction.

Inspect the decision log. Each material decision, such as approving a risky task, overriding a gate or granting an exception, should be an append-only record with an author, a timestamp and the policy version in force. Records stored only in a vendor-hosted service are a lock-in risk; prefer records kept in your own repository in a documented format.

Then inspect cost. Autonomous loops can burn budget quickly, so a framework should attribute spend to tasks and models, enforce ceilings, and route work to cheaper models where quality permits. Ask whether you can see cost per merged change, not just cost per month.

Finally, ask about portability. If you leave the framework, can you take your policies, decision history and metrics with you? Open formats and open source licensing reduce that risk.

AI-SDLC records decisions as append-only files in your repository and tracks spend in a usage ledger with model routing.

Score from zero to five on completeness of the decision log, cost attribution and exportability.