AI-SDLC Metrics and Observability Specification

Document type: Normative Status: Draft Spec version: v1alpha1

Table of Contents

1. Introduction
2. Metric Categories
3. OpenTelemetry Integration
4. Provenance Tracking
5. Audit Trail Requirements

1. Introduction

Beyond DORA's four keys, AI-augmented development requires purpose-built measurements. This document defines the AI-SDLC metrics framework, OpenTelemetry integration conventions, provenance tracking requirements, and audit trail specifications.

Metrics serve two purposes in the AI-SDLC Framework:

1. Governance — Metrics drive promotion and demotion decisions in the autonomy system
2. Observability — Metrics provide visibility into the health and performance of AI-augmented development workflows

2. Metric Categories

The framework defines five metric categories. Implementations MUST support collecting and reporting metrics from at least the first three categories (Task Effectiveness, Human-in-Loop, Code Quality).

2.1 Task Effectiveness

Metrics measuring how effectively agents complete assigned tasks.

Dimensions:

• agent — Agent name (AgentRole reference)
• complexity_tier — low, medium, high, critical
• task_type — implementation, review, testing, deployment

2.2 Human-in-Loop Indicators

Metrics measuring the frequency and nature of human involvement.

Dimensions:

• agent — Agent name
• autonomy_level — 0, 1, 2, 3
• reason — Reason for intervention/escalation

2.3 Code Quality

Metrics measuring the quality of AI-generated code.

Dimensions:

• author_type — ai-agent, human
• agent — Agent name (when author_type is ai-agent)
• language — Programming language
• repository — Repository identifier

2.4 Economic Efficiency

Metrics measuring the cost-effectiveness of AI agent usage.

Dimensions:

• model — Model identifier
• agent — Agent name
• team — Team/namespace
• stage — Pipeline stage name
• complexity — Complexity tier (low, medium, high, critical)
• repository — Repository identifier
• outcome — Execution outcome (success, failure, aborted)

2.5 Autonomy Trajectory

Metrics tracking the progression of agent autonomy over time.

Dimensions:

• agent — Agent name
• from_level — Previous level (for promotions)
• to_level — New level (for promotions)

3. OpenTelemetry Integration

The framework SHOULD define semantic conventions for AI-SDLC observability, extending OpenTelemetry's GenAI semantic conventions. All metric, trace, and log attribute names MUST use the ai_sdlc.* namespace prefix.

3.1 Traces

Implementations SHOULD generate spans for the following operations:

Spans SHOULD be linked into traces following the pipeline execution flow:

pipeline.stage (implement)
  └─→ agent.task (code-agent)
       ├─→ gate.evaluation (test-coverage)
       └─→ gate.evaluation (security-scan)
  └─→ handoff (code-agent → reviewer-agent)
pipeline.stage (review)
  └─→ agent.task (reviewer-agent)
       └─→ gate.evaluation (human-review)

3.2 Metrics

Implementations MUST expose metrics using the names and types defined in Section 2. Metrics SHOULD be exportable via OpenTelemetry Protocol (OTLP).

Instrument types:

• Gauge — For values that can go up and down (e.g., autonomy level, rates)
• Counter — For values that only increase (e.g., escalation count)
• Histogram — For distributions (e.g., task completion time, cost per task)

3.3 Logs

Implementations MUST produce structured logs for the following events:

Logs MUST be structured (JSON) and SHOULD be exportable via OTLP.

4. Provenance Tracking

Every AI-generated artifact MUST record provenance metadata. Provenance enables attribution, auditability, and regulatory compliance.

4.1 Required Fields

CostReceipt Fields

CostBreakdown Fields

ExecutionCostDetail Fields

4.2 Storage

Provenance metadata SHOULD be stored:

• In commit metadata (git trailers or notes)
• In PR descriptions or comments
• In a dedicated provenance store accessible via API

Implementations MUST ensure provenance records are immutable after creation.

4.3 Attribution

When QualityGate rules include requireAttribution: true, implementations MUST verify that provenance metadata is present and complete before admitting the artifact.

5. Audit Trail Requirements

Every action in the system MUST produce an immutable, tamper-evident audit log entry.

5.1 Required Audit Fields

5.2 Immutability

Audit log entries MUST NOT be modifiable after creation. Implementations SHOULD use append-only storage with integrity verification (e.g., hash chains, write-once storage).

5.3 Retention

Implementations SHOULD support configurable retention policies for audit logs. The default retention period SHOULD be at least 12 months to support regulatory compliance.

5.4 Access

Audit logs MUST be accessible to authorized users via API. Implementations SHOULD support filtering by actor, action, resource, and time range.