Docs Drift Guard – Demo

A 3-layer automation that keeps your API, its documentation, and its consumers in sync — and alerts your team on Slack when they aren't.

This repository demonstrates three complementary guards for enterprise APIs:

Layer	What it catches	How
Code-vs-Docs Drift	Developer changes the API but forgets to update the docs	Compares live OpenAPI to a stored baseline
Runtime Call Guard	Clients call the API with wrong field names or hit nonexistent endpoints	FastAPI middleware validates every request against the OpenAPI spec
Contract Tests	Accumulated mismatches from clients built against stale docs	pytest suite replays "old client" calls and reports what breaks

All three feed into Slack notifications so the team knows what to fix.

---

Quick Start

# 1. Clone & setup
git clone <repo-url> && cd docs-drift-guard-demo
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

# 2. Start the API server (in a separate terminal)
uvicorn app.main:app --reload

---

Demo Part 1: Code-vs-Docs Drift Detection

Detect when a developer changes the API code but doesn't update the docs.

# Save the current API as the baseline
python -m drift_guard.guard baseline

# Introduce a code change (adds 'role' field + GET /v1/users endpoint)
bash scripts/demo_drift.sh

# Detect the drift
python -m drift_guard.guard check
#   → "Added endpoint: GET /v1/users"
#   → "Schema change: User gained field role"
#   → Regenerates docs/api_reference.md
#   → Posts to Slack (if SLACK_WEBHOOK_URL is set)
#   → Exits with code 2

---

Demo Part 2: Runtime API Call Guard

Catch clients that are calling the API using field names from the stale manual docs (e.g. username instead of name).

# Simulate a client built from the stale docs
bash scripts/demo_bad_client.sh
#   → Sends POST with "username" (wrong) instead of "name" (correct)
#   → Sends GET to /v1/teams (doesn't exist)
#   → Each mismatch is logged by the middleware

# View the audit log
curl -s http://localhost:8000/v1/audit | python -m json.tool

# Get a Slack summary
python -m drift_guard.guard audit
#   → Summarises all runtime mismatches
#   → Posts to Slack
#   → Exits with code 3

The middleware runs on every request but never blocks — it logs issues and adds an X-API-Mismatch: true response header so callers know something was off.

---

Demo Part 3: Contract Tests (Clean Up the Mess)

Run a test suite that replays how a client built from the stale manual docs would call the API, and get a clear report of what's broken.

# Run the contract tests
pytest tests/test_contract.py -v

# Or run the full cleanup script (tests + audit report)
bash scripts/demo_cleanup.sh

The tests cover:

• Correct calls (should pass)
• Calls using username instead of name (should fail with 422)
• Calls to nonexistent endpoints (should get 404)
• Validation that the middleware flags all issues in the audit log

---

Full Demo Flow (for a Loom recording)

# Setup
source .venv/bin/activate
uvicorn app.main:app --reload     # separate terminal

# Part 1: Show the stale manual docs
cat docs/api_manual.md            # note: says "username", API uses "name"

# Part 2: Baseline + drift detection
python -m drift_guard.guard baseline
bash scripts/demo_drift.sh
python -m drift_guard.guard check  # drift detected!

# Part 3: Bad client calls + audit
bash scripts/demo_bad_client.sh    # middleware catches mismatches
python -m drift_guard.guard audit  # Slack summary

# Part 4: Contract tests
pytest tests/test_contract.py -v   # shows exactly what's broken

---

CLI Reference

python -m drift_guard.guard <command> [--url URL]

Command	What it does
baseline	Fetch OpenAPI, save snapshot, generate docs/api_reference.md
check	Compare to baseline, report drift, regenerate docs, notify Slack
audit	Fetch runtime mismatch log, summarise, notify Slack

Exit Codes

Code	Meaning
0	Success / no drift / no mismatches
1	Runtime error (server down, missing baseline, etc.)
2	Drift detected
3	Runtime mismatches found

---

Automation: CI Gate (GitHub Action)

A GitHub Action runs on every push and PR to main. It automatically:

1. Starts the API server
2. Runs drift_guard check — fails if the OpenAPI spec drifted from the baseline
3. Runs pytest tests/test_contract.py — fails if any contract tests break
4. Runs drift_guard audit — summarises runtime mismatches
5. Uploads artifacts/drift_summary.md and artifacts/audit_report.md as build artifacts
6. Posts to Slack if SLACK_WEBHOOK_URL is set as a repository secret

To enable Slack in CI: Add SLACK_WEBHOOK_URL as a repository secret in Settings > Secrets and variables > Actions.

If drift is detected, the PR will show a failing check with a clear message telling the developer to run python -m drift_guard.guard baseline and commit the updated files.

---

Automation: Pre-Commit Hook

Catch drift before code even gets pushed. Two ways to install:

Option A: Using the pre-commit framework (recommended)

pip install pre-commit
pre-commit install

This uses .pre-commit-config.yaml to run the drift check only when files under app/ are staged. It also runs ruff for linting.

Option B: Manual git hook

cp scripts/pre-commit-drift-check.sh .git/hooks/pre-commit
chmod +x .git/hooks/pre-commit

How it works:

• Only triggers when you stage changes to files in app/
• Starts a temporary server, compares the live OpenAPI to the baseline
• If drift is found, the commit is blocked with instructions on how to fix it
• To skip once (not recommended): git commit --no-verify

---

Slack Notifications

Set the SLACK_WEBHOOK_URL environment variable:

export SLACK_WEBHOOK_URL="https://hooks.slack.com/services/T.../B.../xxx"

If the variable is not set, the tool prints the JSON payload that would have been sent so you can verify the format.

---

Project Structure

docs-drift-guard-demo/
  app/
    main.py                 # FastAPI application + audit endpoint
    middleware.py            # API Call Guard middleware
    audit_log.py            # In-memory audit log
  docs/
    api_reference.md        # Auto-generated (by drift_guard)
    api_manual.md           # Hand-written – intentionally stale
  drift_guard/
    guard.py                # CLI entrypoint (baseline, check, audit)
    openapi_to_md.py        # OpenAPI JSON → Markdown converter
    drift_detect.py         # Compares two OpenAPI snapshots
    slack_notify.py         # Slack Incoming Webhook integration
  tests/
    test_contract.py        # Contract test suite
  artifacts/
    openapi_baseline.json   # Stored OpenAPI snapshot
    drift_summary.md        # Generated change summary
    audit_report.md         # Generated audit report
  scripts/
    bootstrap.sh            # One-command setup
    demo_drift.sh           # Introduces code drift
    demo_bad_client.sh      # Simulates stale-doc client calls
    demo_cleanup.sh         # Runs contract tests + audit
    pre-commit-drift-check.sh  # Git pre-commit hook
  .github/workflows/
    drift-guard.yml         # GitHub Action CI gate
  .pre-commit-config.yaml   # pre-commit framework config
  requirements.txt
  README.md

---

Tech Stack

• Python 3.11+
• FastAPI + Uvicorn
• Requests (Slack webhook)
• Rich (CLI output)
• Pytest (contract tests)
• Ruff (linting)

---

License

MIT – use freely for demos, talks, and internal tooling.