Skip to content

fix: bound telemetry growth during upload outages#426

Open
DivyamTalwar wants to merge 1 commit into
morphik-org:mainfrom
DivyamTalwar:fix/telemetry-retention-on-upload-failure
Open

fix: bound telemetry growth during upload outages#426
DivyamTalwar wants to merge 1 commit into
morphik-org:mainfrom
DivyamTalwar:fix/telemetry-retention-on-upload-failure

Conversation

@DivyamTalwar

Copy link
Copy Markdown

Summary

TELEMETRY_MAX_LOCAL_BYTES was only enforced after a successful telemetry proxy upload. If the proxy was down or rejected the payload, local telemetry files could grow beyond the configured budget.

This PR preserves the existing successful-upload cleanup behavior, but adds a scoped failed-upload cleanup path for logs/telemetry/. Successful uploads still truncate uploaded telemetry files before enforcing the global logs/ budget, while failed uploads enforce the configured budget only inside logs/telemetry/ so outage cleanup does not delete unrelated application logs.

Changes

  • Run best-effort telemetry budget enforcement after failed uploads.
  • Scope failed-upload cleanup to logs/telemetry/; keep successful-upload cleanup across logs/.
  • Preserve the existing successful-upload order: post payload, truncate uploaded files, then enforce budget.
  • Keep cleanup failures from escaping the uploader cycle and stopping future retries.
  • Document the uploader-cycle retention contract for self-hosted deployments: successful uploads enforce
    the configured budget across logs/, failed uploads enforce it only inside logs/telemetry/, and
    no-event cycles do not prune.
  • Add focused unit coverage for upload failures, no telemetry files, empty bundles, success ordering, cleanup failures, and disabled-budget behavior.

Why this matters

Self-hosted deployments should have predictable telemetry disk usage even when the telemetry proxy is unavailable. This makes the existing max_local_bytes configuration effective during upload failures without turning failed telemetry uploads into a global log cleanup pass.

Tests

Commands run:

  • .venv/bin/python -m pytest core/tests/unit/test_log_uploader_retention.py -q — passed, 6 tests.
  • uv run ty check core/services/log_uploader.py core/tests/unit/test_log_uploader_retention.py — passed.
  • .venv/bin/python -m py_compile core/services/log_uploader.py core/tests/unit/test_log_uploader_retention.py — passed.
  • .venv/bin/markdown-it docs/telemetry.md >/tmp/telemetry-pr11.html — passed.
  • git diff --check — passed.

Additional verification:

  • .venv/bin/python -m pytest core/tests/unit -q — failed during collection because the existing unit suite imports settings with database.provider = "postgres" and no POSTGRES_URI.
  • POSTGRES_URI=postgresql://user:pass@localhost:5432/test .venv/bin/python -m pytest core/tests/unit -q — started but was interrupted after it stalled for over one minute in ColPali model artifact fetching before test execution made meaningful progress.
  • uv run ruff check core/services/log_uploader.py core/tests/unit/test_log_uploader_retention.py — not run successfully because ruff is not installed in this environment.
  • uv run ruff format --check core/services/log_uploader.py core/tests/unit/test_log_uploader_retention.py — not run successfully because ruff is not installed in this environment.

Risk

Risk level: low/medium.

The code change is small and reuses the existing oldest-first deletion policy. The behavior change is intentional: deployments that configure a local telemetry budget can now prune oldest local telemetry files during upload outages instead of only after successful uploads. Rollback is straightforward by removing the failed-upload cleanup path.

Issue

No existing issue. This was discovered during repository analysis.

Notes for maintainers

This PR intentionally does not change retry behavior, telemetry payload format, or the existing successful-upload global cleanup behavior.

Enforce the configured byte budget for logs/telemetry when upload attempts fail, while preserving the existing successful-upload cleanup order across logs/. Keep the cleanup pass best-effort so filesystem pruning errors do not stop future uploader retries.

Constraint: Failed-upload cleanup must not become a surprise global log cleanup pass for self-hosted operators.

Rejected: Run the existing global logs/ budget sweep on every uploader cycle | that would delete unrelated logs during proxy outages and broaden the PR beyond telemetry retention.

Confidence: high

Scope-risk: narrow

Directive: Keep failure-path cleanup scoped to telemetry files unless a future PR deliberately changes the documented log-retention policy.

Tested: .venv/bin/python -m pytest core/tests/unit/test_log_uploader_retention.py -q; uv run ty check core/services/log_uploader.py core/tests/unit/test_log_uploader_retention.py; .venv/bin/python -m py_compile core/services/log_uploader.py core/tests/unit/test_log_uploader_retention.py; .venv/bin/markdown-it docs/telemetry.md >/tmp/telemetry-pr11.html; git diff --check

Not-tested: Full core/tests/unit run; it requires dummy POSTGRES_URI and stalled in ColPali model artifact fetching for over 1 minute. Ruff check/format unavailable because ruff is not installed in this environment.
@DivyamTalwar
DivyamTalwar marked this pull request as ready for review July 3, 2026 20:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant