fix: bound telemetry growth during upload outages#426
Open
DivyamTalwar wants to merge 1 commit into
Open
Conversation
Enforce the configured byte budget for logs/telemetry when upload attempts fail, while preserving the existing successful-upload cleanup order across logs/. Keep the cleanup pass best-effort so filesystem pruning errors do not stop future uploader retries. Constraint: Failed-upload cleanup must not become a surprise global log cleanup pass for self-hosted operators. Rejected: Run the existing global logs/ budget sweep on every uploader cycle | that would delete unrelated logs during proxy outages and broaden the PR beyond telemetry retention. Confidence: high Scope-risk: narrow Directive: Keep failure-path cleanup scoped to telemetry files unless a future PR deliberately changes the documented log-retention policy. Tested: .venv/bin/python -m pytest core/tests/unit/test_log_uploader_retention.py -q; uv run ty check core/services/log_uploader.py core/tests/unit/test_log_uploader_retention.py; .venv/bin/python -m py_compile core/services/log_uploader.py core/tests/unit/test_log_uploader_retention.py; .venv/bin/markdown-it docs/telemetry.md >/tmp/telemetry-pr11.html; git diff --check Not-tested: Full core/tests/unit run; it requires dummy POSTGRES_URI and stalled in ColPali model artifact fetching for over 1 minute. Ruff check/format unavailable because ruff is not installed in this environment.
DivyamTalwar
marked this pull request as ready for review
July 3, 2026 20:57
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
TELEMETRY_MAX_LOCAL_BYTESwas only enforced after a successful telemetry proxy upload. If the proxy was down or rejected the payload, local telemetry files could grow beyond the configured budget.This PR preserves the existing successful-upload cleanup behavior, but adds a scoped failed-upload cleanup path for
logs/telemetry/. Successful uploads still truncate uploaded telemetry files before enforcing the globallogs/budget, while failed uploads enforce the configured budget only insidelogs/telemetry/so outage cleanup does not delete unrelated application logs.Changes
logs/telemetry/; keep successful-upload cleanup acrosslogs/.the configured budget across
logs/, failed uploads enforce it only insidelogs/telemetry/, andno-event cycles do not prune.
Why this matters
Self-hosted deployments should have predictable telemetry disk usage even when the telemetry proxy is unavailable. This makes the existing
max_local_bytesconfiguration effective during upload failures without turning failed telemetry uploads into a global log cleanup pass.Tests
Commands run:
.venv/bin/python -m pytest core/tests/unit/test_log_uploader_retention.py -q— passed, 6 tests.uv run ty check core/services/log_uploader.py core/tests/unit/test_log_uploader_retention.py— passed..venv/bin/python -m py_compile core/services/log_uploader.py core/tests/unit/test_log_uploader_retention.py— passed..venv/bin/markdown-it docs/telemetry.md >/tmp/telemetry-pr11.html— passed.git diff --check— passed.Additional verification:
.venv/bin/python -m pytest core/tests/unit -q— failed during collection because the existing unit suite imports settings withdatabase.provider = "postgres"and noPOSTGRES_URI.POSTGRES_URI=postgresql://user:pass@localhost:5432/test .venv/bin/python -m pytest core/tests/unit -q— started but was interrupted after it stalled for over one minute in ColPali model artifact fetching before test execution made meaningful progress.uv run ruff check core/services/log_uploader.py core/tests/unit/test_log_uploader_retention.py— not run successfully becauseruffis not installed in this environment.uv run ruff format --check core/services/log_uploader.py core/tests/unit/test_log_uploader_retention.py— not run successfully becauseruffis not installed in this environment.Risk
Risk level: low/medium.
The code change is small and reuses the existing oldest-first deletion policy. The behavior change is intentional: deployments that configure a local telemetry budget can now prune oldest local telemetry files during upload outages instead of only after successful uploads. Rollback is straightforward by removing the failed-upload cleanup path.
Issue
No existing issue. This was discovered during repository analysis.
Notes for maintainers
This PR intentionally does not change retry behavior, telemetry payload format, or the existing successful-upload global cleanup behavior.