diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS new file mode 100644 index 0000000..3dc347e --- /dev/null +++ b/.github/CODEOWNERS @@ -0,0 +1,3 @@ +# Global owners +* @cherry-aggarwal @Brijesh-Thakkar @coderTanisha22 + diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md new file mode 100644 index 0000000..57ee575 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/bug_report.md @@ -0,0 +1,89 @@ +--- + +name: Bug Report +about: Report an issue or unexpected behavior in the project. +title: "[Bug]: " +labels: bug +assignees: "" +--- + +# Bug Report + +## Summary + +Provide a brief description of the bug. + +--- + +## Module + +Specify the module where the issue occurs. + +Example: + +* Telemetry +* OSDoctor +* FocusOS +* Blackbox +* Research Engine +* Dashboard + +--- + +## Steps to Reproduce + +Describe the steps required to reproduce the issue. + +1. +2. +3. + +--- + +## Expected Behavior + +Describe what should happen. + +--- + +## Actual Behavior + +Describe what actually happens. + +--- + +## Logs / Error Messages + +Paste any relevant logs, stack traces, or terminal output. + +```text +``` + +--- + +## Screenshots + +If applicable, attach screenshots or screen recordings. + +--- + +## Environment + +Please provide the following information: + +* Operating System: +* Programming Language / Version: +* Browser (if applicable): +* Other relevant details: + +--- + +## Possible Solution (Optional) + +If you have an idea of what might be causing the issue or how it could be fixed, describe it here. + +--- + +## Additional Notes + +Add any other information that may help reproduce or resolve the issue. diff --git a/.github/ISSUE_TEMPLATE/config.yml b/.github/ISSUE_TEMPLATE/config.yml new file mode 100644 index 0000000..2e1c553 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/config.yml @@ -0,0 +1,10 @@ +blank_issues_enabled: false + +contact_links: +- name: πŸ’¬ Questions & Discussions + about: Ask questions or discuss ideas before creating a new issue. + url: https://github.com/devlup-labs/CogniOS/discussions + +- name: πŸ“– Project Documentation + about: Read the project documentation before opening an issue. + url: https://github.com/devlup-labs/CogniOS#readme diff --git a/.github/ISSUE_TEMPLATE/development_task.md b/.github/ISSUE_TEMPLATE/development_task.md new file mode 100644 index 0000000..5b1f58f --- /dev/null +++ b/.github/ISSUE_TEMPLATE/development_task.md @@ -0,0 +1,81 @@ +--- + +name: Development Task +about: Create a new feature implementation task. +title: "[Feature]: " +labels: enhancement +assignees: "" +--- + +# Development Task + +## Module + +Specify the module this feature belongs to. + +Example: + +* Telemetry +* OSDoctor +* FocusOS +* Blackbox +* Research Engine +* Dashboard + +--- + +## Objective + +Clearly describe the feature that needs to be implemented. + +--- + +## Problem Statement + +What problem does this feature solve? + +--- + +## Proposed Solution + +Describe how you plan to implement this feature. + +--- + +## Tasks + +Break the implementation into smaller tasks. + +* [ ] +* [ ] +* [ ] +* [ ] + +--- + +## Acceptance Criteria + +The feature will be considered complete when: + +* [ ] +* [ ] +* [ ] +* [ ] + +--- + +## Related Documentation + +Mention any documentation that should be referred to before implementation. + +Examples: + +* Module README +* API Documentation +* Architecture Documentation + +--- + +## Additional Notes + +Add any diagrams, references, or implementation notes that may help contributors. diff --git a/.github/ISSUE_TEMPLATE/documentation.md b/.github/ISSUE_TEMPLATE/documentation.md new file mode 100644 index 0000000..48456e2 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/documentation.md @@ -0,0 +1,88 @@ +--- + +name: Documentation +about: Suggest improvements or updates to the project documentation. +title: "[Docs]: " +labels: documentation +assignees: "" +--- + +# Documentation Update + +## Summary + +Provide a brief description of the documentation change. + +--- + +## Type of Documentation + +Select all that apply by replacing the space inside the brackets with an **`x`**. + +Example: + +```text +- [x] Module Documentation +- [ ] API Documentation +``` + +* [ ] README +* [ ] Module Documentation +* [ ] API Documentation +* [ ] Architecture Documentation +* [ ] Code Comments +* [ ] Setup Guide +* [ ] Other + +--- + +## Files Affected + +List the documentation files that need to be updated. + +Example: + +```text +docs/modules/backend.md +README.md +backend/api/README.md +``` + +--- + +## Description + +Describe what needs to be added, updated, or corrected. + +--- + +## Reason for the Update + +Explain why this documentation change is needed. + +Examples: + +* New feature added +* Existing documentation is outdated +* Missing documentation +* Incorrect information +* Improve clarity for contributors + +--- + +## Related Issue(s) + +Mention any related GitHub issue(s). + +Example: + +```text +Closes #12 +Related to #8 +``` + +--- + +## Additional Notes + +Include any references, screenshots, diagrams, or links that may help reviewers. diff --git a/.github/ISSUE_TEMPLATE/feature_request.md b/.github/ISSUE_TEMPLATE/feature_request.md new file mode 100644 index 0000000..68abd34 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/feature_request.md @@ -0,0 +1,70 @@ +--- + +name: Feature Request +about: Propose a new feature or enhancement for the project. +title: "[Feature]: " +labels: enhancement +assignees: "" +--- + +# Feature Request + +## Summary + +Provide a brief description of the requested feature. + +--- + +## Module + +Specify the module this request is related to. + +Example: + +* Telemetry +* OSDoctor +* FocusOS +* Blackbox +* Research Engine +* Dashboard + +--- + +## Problem Statement + +Describe the current limitation or pain point. + +--- + +## Proposed Solution + +Describe your proposed feature and how it should work. + +--- + +## Alternatives Considered + +Describe any alternative approaches you have considered. + +--- + +## Expected Outcome + +Describe what success looks like after this feature is implemented. + +--- + +## Acceptance Criteria + +The feature will be considered complete when: + +* [ ] +* [ ] +* [ ] + +--- + +## Additional Context + +Add mockups, references, links, or any extra details that may help. + diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md new file mode 100644 index 0000000..6515727 --- /dev/null +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -0,0 +1,96 @@ +# Pull Request + +## Related Issue + +Closes # + +--- + +## Summary + +Provide a brief description of the changes made in this Pull Request. + +Example: + +* Implemented process telemetry collection +* Updated module documentation + +--- + +## Type of Change + +**Instructions:** Mark the applicable option(s) by replacing the space inside the brackets with an **`x`**. + +Example: + +```text +- [x] New feature +- [ ] Bug fix +``` + +Select all that apply: + +* [ ] New feature +* [ ] Bug fix +* [ ] Documentation update +* [ ] Refactoring +* [ ] Performance improvement +* [ ] Test addition +* [ ] Other (please specify) + +--- + +## Changes Made + +Describe the main changes introduced by this PR. + +* +* +* + +--- + +## Testing + +Describe how you tested your changes. + +* [ ] Project builds successfully +* [ ] Existing functionality verified +* [ ] New functionality tested +* [ ] Tests added/updated (if applicable) + +Additional testing details: + +--- + +## Documentation + +* [ ] Documentation updated +* [ ] No documentation changes required + +If updated, specify the affected documentation: + +--- + +## Checklist + +Before requesting a review, confirm the following: + +* [ ] My code follows the project's coding standards. +* [ ] I have reviewed my own code. +* [ ] I have updated relevant documentation. +* [ ] My changes do not introduce unnecessary files or dependencies. +* [ ] The project builds successfully. +* [ ] This Pull Request addresses a single feature or bug. + +--- + +## Screenshots / Logs (if applicable) + +Attach screenshots, terminal output, or logs that help demonstrate your changes. + +--- + +## Additional Notes + +Include any information reviewers should know before reviewing this Pull Request. diff --git a/.gitignore b/.gitignore index 210860c..9f59571 100644 --- a/.gitignore +++ b/.gitignore @@ -7,12 +7,20 @@ __pycache__/ .venv/ venv/ +# Environment & Config (IMPORTANT) +.env +config.ini + # Local data and generated artifacts -data/cognios.db +cognios_telemetry.db +*.db data/models/*.pt data/traces/* # Keep data directories present in git if marker files are added. -!data/models/.gitkeep !data/traces/.gitkeep + +# OS / IDE .DS_Store +.vscode/ +.idea/ diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..5d095d6 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,214 @@ +# Contributing to CogniOS + +First off, thank you for taking the time to contribute to **CogniOS**! πŸŽ‰ + +Whether you're fixing a bug, implementing a new feature, improving documentation, or optimizing existing code, your contributions are appreciated. + +Please read this guide before making your first contribution. + +--- + +# Development Workflow + +All development follows the GitHub Issue β†’ Branch β†’ Pull Request workflow. + +``` +Issue + ↓ +Assign Yourself + ↓ +Create a Branch + ↓ +Implement Changes + ↓ +Commit & Push + ↓ +Open Pull Request + ↓ +Code Review + ↓ +Merge +``` + +Direct pushes to the `main` branch are **not allowed**. + +--- + +# Before You Start + +Before working on a feature: + +1. Check the existing GitHub Issues. +2. Assign the issue to yourself. +3. If the required issue does not exist, create one describing the proposed feature or bug. +4. Wait for approval (if required) before beginning implementation. + +This helps prevent duplicate work and keeps development organized. + +--- + +# Branch Naming Convention + +Create a new branch from the latest `main` branch. + +Use the following naming convention: + +``` +feature/ +fix/ +docs/ +refactor/ +test/ +``` + +Examples: + +``` +feature/process-monitor +feature/anomaly-detector +fix/cpu-parser +docs/backend-api +refactor/cache-layer +``` + +--- + +# Coding Standards + +Please ensure your code follows these guidelines: + +* Write clean and readable code. +* Use meaningful variable and function names. +* Follow the coding style used within the module. +* Keep functions small and focused. +* Avoid unnecessary complexity. +* Remove unused code before submitting a Pull Request. +* Add comments only where they improve understanding. + +--- + +# Documentation + +Every contribution should include documentation updates whenever applicable. + +Documentation may include: + +* Function descriptions +* API endpoint updates +* Architecture changes +* README updates +* Module documentation + +If your implementation changes the behavior of a module, its documentation should also be updated. + +--- + +# Commit Message Guidelines + +Write clear and descriptive commit messages. + +Recommended format: + +``` +feat: implement process telemetry collector +fix: resolve memory parsing issue +docs: update backend documentation +refactor: simplify scheduler logic +test: add telemetry unit tests +``` + +Avoid vague commit messages such as: + +``` +update +changes +fixed stuff +final +temp +``` + +--- + +# Pull Request Guidelines + +Before opening a Pull Request: + +* Ensure your branch is up to date. +* Resolve merge conflicts. +* Verify the project builds successfully. +* Update relevant documentation. +* Ensure code follows project conventions. + +Each Pull Request should focus on **one feature or one bug fix**. + +Large unrelated changes should be split into multiple Pull Requests. + +--- + +# Code Review + +Every Pull Request will be reviewed before merging. + +Reviewers may request: + +* Code improvements +* Refactoring +* Additional documentation +* Bug fixes +* Performance improvements + +Please address review comments before requesting another review. + +--- + +# Testing + +Before submitting a Pull Request, verify that: + +* The project builds successfully. +* Existing functionality remains unaffected. +* New functionality works as intended. +* Any applicable tests pass successfully. + +Do not submit code that does not compile. + +--- + +# Reporting Bugs + +When reporting a bug, include: + +* Description of the issue +* Steps to reproduce +* Expected behavior +* Actual behavior +* Relevant logs or screenshots (if available) + +--- + +# Feature Requests + +When proposing a new feature, include: + +* Problem being solved +* Proposed solution +* Module(s) affected +* Expected outcome + +Discuss major architectural changes before beginning implementation. + +--- + +# Code of Conduct + +Please be respectful and collaborative. + +Constructive discussions and code reviews help improve the project for everyone. + +--- + +# Questions + +If you have questions regarding implementation, architecture, or project structure, please open a GitHub Discussion or contact the maintainers before starting work. + +Happy coding! πŸš€ diff --git a/README.md b/README.md index ef75438..8a6efbd 100644 --- a/README.md +++ b/README.md @@ -36,3 +36,306 @@ graph TD BB --> DASH RE --> DASH ``` +# CogniOS + +### Intelligent OS Observability & Adaptive Workload Optimization Platform + +> **CogniOS** is an AI-assisted Linux system observability and workload optimization platform that combines operating systems, machine learning, and systems engineering to provide intelligent performance monitoring, anomaly detection, workload-aware optimization, crash analysis, and scheduling researchβ€”all without modifying the Linux kernel. + +--- + +## Overview + +Modern operating systems expose a large amount of telemetry, but existing monitoring tools often present only raw statistics, leaving users to determine the root cause of performance issues themselves. + +CogniOS bridges this gap by collecting real-time system telemetry, analyzing workload behavior, detecting anomalies, recommending optimizations, and enabling post-crash analysis through an integrated and modular architecture. + +The project is designed as both: + +* A practical system observability platform for Linux +* A research framework for experimenting with AI-assisted operating system scheduling + +--- + +## Key Features + +* πŸ“Š Real-time Linux system telemetry collection +* 🩺 Intelligent anomaly detection for performance degradation +* 🎯 AI-based workload classification and optimization +* πŸ“¦ Rolling telemetry recording for crash replay +* πŸ”¬ Scheduling algorithm benchmarking and research +* πŸ“ˆ Interactive Streamlit dashboard +* πŸ—„οΈ Centralized SQLite telemetry database +* 🧩 Modular architecture for independent development and testing + +--- + +# Architecture + +``` + Linux System + β”‚ + β–Ό + Telemetry Collectors + (/proc, psutil, process APIs) + β”‚ + β–Ό + SQLite Database + β”‚ + β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” + β”‚ β”‚ β”‚ + β–Ό β–Ό β–Ό + OS Doctor FocusOS BlackBox + β”‚ β”‚ β”‚ + β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ + β–Ό + Research Engine + β”‚ + β–Ό + Streamlit Dashboard +``` + +--- + +# Core Modules + +## 🩺 OS Doctor + +OS Doctor continuously monitors the system for abnormal behavior using machine learning–based anomaly detection techniques. + +### Objective + +* Detect abnormal CPU, memory and I/O behavior +* Explain possible reasons for system slowdowns +* Identify unusual workload patterns +* Provide interpretable diagnostics from live telemetry + +--- + +## 🎯 FocusOS + +FocusOS is responsible for intelligent workload classification and adaptive system optimization. + +### Objective + +* Classify the current workload using machine learning +* Optimize process priorities +* Modify Linux scheduling parameters +* Improve overall system responsiveness + +Future versions may also support dynamic CPU affinity optimization and workload-aware scheduling. + +--- + +## πŸ“¦ BlackBox + +BlackBox acts as the system's flight recorder. + +It continuously stores recent telemetry in a rolling buffer, allowing developers to replay system activity after crashes or freezes. + +### Objective + +* Record recent system telemetry +* Preserve crash history +* Replay workload traces +* Assist in post-mortem debugging + +--- + +## πŸ”¬ Research Engine + +The Research Engine provides an experimentation platform for scheduling algorithms. + +### Objective + +* Compare classical scheduling algorithms +* Evaluate AI-based schedulers +* Replay collected workloads +* Benchmark scheduling performance +* Support reinforcement learning experiments + +Supported scheduling algorithms include: + +* FCFS +* Shortest Job First +* Round Robin +* Priority Scheduling + +Future versions may include reinforcement learning schedulers. + +--- + +## πŸ“Š Dashboard + +The Streamlit dashboard serves as the unified visualization layer of CogniOS. + +### Objective + +* Visualize system telemetry +* Display anomaly alerts +* Monitor workload classifications +* Compare scheduling results +* Provide a centralized monitoring interface + +--- + +## πŸ“‘ Collectors + +Collectors gather telemetry directly from Linux using lightweight system interfaces. + +### Objective + +* CPU monitoring +* Memory monitoring +* Disk monitoring +* Process monitoring +* Network statistics +* I/O statistics + +These collectors act as the data source for every other module. + +--- + +## πŸ—„οΈ Data Layer + +The data layer stores all telemetry generated by the collectors. + +### Responsibilities + +* Store real-time telemetry +* Maintain workload traces +* Persist crash recordings +* Provide data for ML training +* Support benchmarking experiments + +SQLite is used as the centralized storage backend. + +--- + +# Repository Structure + +``` +CogniOS/ +β”‚ +β”œβ”€β”€ blackbox/ # Rolling telemetry recorder and crash replay engine +β”œβ”€β”€ collectors/ # System telemetry collectors using /proc and psutil +β”œβ”€β”€ dashboard/ # Streamlit-based monitoring dashboard +β”œβ”€β”€ data/ +β”‚ β”œβ”€β”€ datasets/ # Labeled workload datasets for ML training +β”‚ β”œβ”€β”€ telemetry/ # Raw telemetry snapshots +β”‚ └── sqlite/ # SQLite database files +β”œβ”€β”€ focusos/ # AI workload classifier and adaptive optimization engine +β”œβ”€β”€ os_doctor/ # Real-time anomaly detection module +β”œβ”€β”€ research_engine/ # Scheduling simulator and benchmarking framework +β”œβ”€β”€ utils/ # Shared helper utilities used across modules +β”‚ +β”œβ”€β”€ cognios_as_daemon.py # Runs CogniOS as a background monitoring daemon +β”œβ”€β”€ config.py # Global project configuration +β”œβ”€β”€ db.py # SQLite database interface and helper functions +β”œβ”€β”€ main.py # Main application entry point +β”œβ”€β”€ overhead.py # Measures runtime overhead introduced by monitoring +β”œβ”€β”€ requirements.txt # Python dependencies +└── README.md +``` + +--- + +# Technology Stack + +| Category | Technology | +| ---------------------- | ------------------------------- | +| Language | Python | +| System Monitoring | psutil, Linux `/proc` | +| Database | SQLite | +| Machine Learning | PyTorch | +| Data Processing | NumPy, Pandas | +| Anomaly Detection | Isolation Forest (scikit-learn) | +| Reinforcement Learning | Stable-Baselines3 | +| Dashboard | Streamlit | + +--- + +# Getting Started + +## Clone the Repository + +```bash +git clone https://github.com//CogniOS.git +cd CogniOS +``` + +## Install Dependencies + +```bash +pip install -r requirements.txt +``` + +## Launch CogniOS + +```bash +python main.py +``` + +To run CogniOS as a background service: + +```bash +python cognios_as_daemon.py +``` + +--- + +# Development Philosophy + +Each major subsystem is designed as an independent module with its own documentation, enabling contributors to work on individual components without affecting the rest of the project. + +Every module contains its own dedicated `README.md` describing: + +* Module architecture +* Directory structure +* Components +* APIs +* Data flow +* Usage +* Future work + +--- + +# Roadmap + +* Real-time Linux telemetry collection +* Intelligent anomaly detection +* AI-assisted workload optimization +* Crash replay and forensic analysis +* Scheduling benchmark suite +* Reinforcement learning scheduler +* GPU telemetry support +* Windows and macOS support + +--- + +# Contributing + +We welcome contributions from developers interested in: + +* Operating Systems +* Machine Learning +* Systems Programming +* Linux Internals +* Performance Engineering +* Data Engineering + +Before contributing, please read the documentation for the module you wish to work on. + +--- + +# License + +This project is released under the MIT License. + +--- + +# Acknowledgements + +CogniOS is developed as an educational and research-focused project exploring the intersection of **Operating Systems**, **Artificial Intelligence**, and **Systems Engineering**. + +The project aims to provide a modular platform for learning, experimentation, and innovation in intelligent system observability and workload optimization. diff --git a/cognios_as_daemon.py b/cognios_as_daemon.py index cbf2700..4e68d68 100644 --- a/cognios_as_daemon.py +++ b/cognios_as_daemon.py @@ -61,7 +61,8 @@ def run_daemon(): metrics['net_packets_received'], metrics['net_errs'], metrics['net_drops'], - json.dumps(metrics['process_data']) + json.dumps(metrics['process_data']), + json.dumps(metrics['num_threads']) ) logging.info(f"Successfully saved metrics for timestamp: {metrics['timestamp']}") diff --git a/cognios_telemetry.db b/cognios_telemetry.db deleted file mode 100644 index b3ec599..0000000 Binary files a/cognios_telemetry.db and /dev/null differ diff --git a/collectors/layer1_system.py b/collectors/layer1_system.py index 8086307..a4592e3 100644 --- a/collectors/layer1_system.py +++ b/collectors/layer1_system.py @@ -1,3 +1,5 @@ +import enum +import enum import psutil import time from datetime import datetime,timezone @@ -31,16 +33,19 @@ def collect_layer1_metrics(): pass time.sleep(0.1) process_data = [] + num_threads=[] for pid, p in procs.items(): try: cpu = round(p.cpu_percent(), 2) mem = round(p.info.get("memory_percent") or 0.0, 2) if cpu > 0.5 or mem > 0.5: process_data.append((p.info.get("name"), cpu, mem)) + num_threads.append(p.num_threads()) except (psutil.NoSuchProcess, psutil.AccessDenied): pass except Exception: process_data = [] + num_threads= [] # CPU Metrics cpu_usage_percent = psutil.cpu_percent(interval=None) cpu_times = psutil.cpu_times() @@ -213,7 +218,8 @@ def collect_layer1_metrics(): "avg_temp":temp_avg, "max_temp":temp_max, "battery_percent":battery_percent, - "process_data":process_data + "process_data":process_data, + "num_threads":num_threads } if __name__ == "__main__": @@ -269,7 +275,8 @@ def collect_layer1_metrics(): metrics['net_packets_received'], metrics['net_errs'], metrics['net_drops'], - json.dumps(metrics['process_data']) + json.dumps(metrics['process_data']), + json.dumps(metrics["num_threads"]) ) print(f"Successfully saved Layer 1 metrics to database table 'layer1_sys' in {DB_PATH}!") except Exception as e: diff --git a/config.py b/config.py index c4c8d21..2674350 100644 --- a/config.py +++ b/config.py @@ -1 +1,2 @@ -DB_PATH = "cognios_telemetry.db" \ No newline at end of file +DB_PATH = "cognios_telemetry.db" +SLIDING_WIND_N = 120 diff --git a/db.py b/db.py index 1f3fbf3..79a5bf2 100644 --- a/db.py +++ b/db.py @@ -104,20 +104,27 @@ def create_connection(db_path): avg_temp REAL, max_temp REAL, battery_percent REAL, - process_data TEXT + process_data TEXT, + num_threads INTEGER ) ''') + # Ensure num_threads column exists for older database instances + cursor.execute("PRAGMA table_info(layer1_sys)") + columns = [row[1] for row in cursor.fetchall()] + if 'num_threads' not in columns: + cursor.execute("ALTER TABLE layer1_sys ADD COLUMN num_threads INTEGER") + conn.commit() return conn # Function to write the collected metrics into the database -def write_layer1(conn, timestamp, cpu_usage_percent, cpu_freq, cpu_user_time, cpu_system_time, cpu_idle_time, cpu_iowait_time, cpu_busy_time, cpu_ctx_switches, memory_percent, memory_used, memory_available, memory_cached, memory_buffers, swap_percent, swap_sin, swap_sout, disk_usage_percent, disk_read_mb_s, disk_write_mb_s, disk_read_time, disk_write_time, load_avg_1, load_avg_5, load_avg_15, total_processes, running_processes, sleeping_processes, zombie_processes, avg_temp, max_temp, battery_percent, net_rate_mb_s, net_bytes_sent, net_bytes_recv, net_packets_sent, net_packets_recv, net_errs, net_drops, process_data): +def write_layer1(conn, timestamp, cpu_usage_percent, cpu_freq, cpu_user_time, cpu_system_time, cpu_idle_time, cpu_iowait_time, cpu_busy_time, cpu_ctx_switches, memory_percent, memory_used, memory_available, memory_cached, memory_buffers, swap_percent, swap_sin, swap_sout, disk_usage_percent, disk_read_mb_s, disk_write_mb_s, disk_read_time, disk_write_time, load_avg_1, load_avg_5, load_avg_15, total_processes, running_processes, sleeping_processes, zombie_processes, avg_temp, max_temp, battery_percent, net_rate_mb_s, net_bytes_sent, net_bytes_recv, net_packets_sent, net_packets_recv, net_errs, net_drops, process_data,num_threads): cursor = conn.cursor() cursor.execute(''' - INSERT INTO layer1_sys (timestamp, cpu_usage_percent, cpu_freq, cpu_user_time, cpu_system_time, cpu_idle_time, cpu_iowait_time, cpu_busy_time, cpu_ctx_switches, memory_percent, memory_used, memory_available, memory_cached, memory_buffers, swap_percent, swap_sin, swap_sout, disk_usage_percent, disk_read_mb_s, disk_write_mb_s, disk_read_time, disk_write_time, net_rate_mb_s, net_bytes_sent, net_bytes_recv, net_packets_sent, net_packets_recv, net_errs, net_drops, load_avg_1, load_avg_5, load_avg_15, total_processes, running_processes, sleeping_processes, zombie_processes, avg_temp, max_temp, battery_percent, process_data) - VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + INSERT INTO layer1_sys (timestamp, cpu_usage_percent, cpu_freq, cpu_user_time, cpu_system_time, cpu_idle_time, cpu_iowait_time, cpu_busy_time, cpu_ctx_switches, memory_percent, memory_used, memory_available, memory_cached, memory_buffers, swap_percent, swap_sin, swap_sout, disk_usage_percent, disk_read_mb_s, disk_write_mb_s, disk_read_time, disk_write_time, net_rate_mb_s, net_bytes_sent, net_bytes_recv, net_packets_sent, net_packets_recv, net_errs, net_drops, load_avg_1, load_avg_5, load_avg_15, total_processes, running_processes, sleeping_processes, zombie_processes, avg_temp, max_temp, battery_percent, process_data, num_threads) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?,?, ?, ?, ?, ?, ?, ?, ?) ''', (timestamp, cpu_usage_percent, cpu_freq, cpu_user_time, @@ -156,5 +163,6 @@ def write_layer1(conn, timestamp, cpu_usage_percent, cpu_freq, cpu_user_time, cp avg_temp, max_temp, battery_percent, - process_data)) + process_data, + num_threads)) conn.commit() diff --git a/focusos/collector.py b/focusos/collector.py new file mode 100644 index 0000000..c64af60 --- /dev/null +++ b/focusos/collector.py @@ -0,0 +1,115 @@ +import sys +import os +sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) + +import collectors.layer1_system as layer1_system +import collectors.layer2_process as layer2_process +from utils import helpers +import db +import sqlite3 +import subprocess +import json + +DB_PATH = "cognios_telemetry.db" +conn = db.create_connection(DB_PATH) + +#collects relevant telemetry data from layer 1 table and returns it as a dictionary +def collect_snapshot(conn): + conn.row_factory = sqlite3.Row + cursor = conn.cursor() + + cursor.execute("SELECT timestamp, cpu_usage_percent, memory_percent, net_rate_mb_s, total_processes, process_data, num_threads FROM layer1_sys ORDER BY timestamp DESC LIMIT 1") + row = cursor.fetchone() + result_dict = {} + if row is not None: + result_dict = dict(row) + + # 1. Safely decode the JSON string from SQLite into a Python list + if result_dict.get('process_data'): + try: + processes = json.loads(result_dict['process_data']) + + #Sorting the list and retaining top 5 entries + result_dict['process_data'] = sorted(processes, key=lambda x: x['cpu'], reverse=True)[:5] + except (json.JSONDecodeError, TypeError): + result_dict['process_data'] = [] + else: + result_dict['process_data'] = [] + + if result_dict.get('num_threads'): + try: + result_dict['num_threads'] = json.loads(result_dict['num_threads']) + except (json.JSONDecodeError, TypeError): + result_dict['num_threads'] = [] + else: + result_dict['num_threads'] = [] + + return result_dict + + +#returns the active window name in lowercase, example: 'firefox' +def get_foreground_app(): + try: + window_id = subprocess.check_output(["xdotool", "getactivewindow"]).decode().strip() + temp_window = subprocess.check_output(["xdotool", "getwindowname", window_id]) + window = temp_window.decode("utf-8").strip().lower() + if " - " in window: + window = window.split(" - ")[-1] + return window + except subprocess.CalledProcessError: + window = "unknown" + return window + + +#returns 2 list[dict], one for top cpu and one for top memory +def get_top_processes(n = 5): + top_cpu, top_mem, _ = layer2_process.collect_process_telemetry() + + top_cpu_filtered = [] + top_mem_filtered = [] + + cpu_keys = ['pid', 'name', 'cpu_percent', 'thread_count', 'status'] + mem_keys = ['pid', 'name', 'memory_percent', 'thread_count', 'status'] + + top_cpu_filtered = [{key: d[key] for key in cpu_keys if key in d} for d in top_cpu] + top_mem_filtered = [{key: d[key] for key in mem_keys if key in d} for d in top_mem] + + return top_cpu_filtered, top_mem_filtered + + +#layer 1 must also be run as a daemon to get all outputs +if __name__ == "__main__": + + import asyncio + async def collector_test(): + print("Testing collector as sandbox\nPress Ctrl + C to finish test run\n") + count = 0 + try: + while True: + start_time = asyncio.get_event_loop().time() + count+=1 + + snapshot = collect_snapshot(conn) + print(snapshot) + foreground_app = get_foreground_app() + print(foreground_app) + + """if count%5 == 0: + top_cpu, top_mem = get_top_processes(n = 5) + print(top_cpu) + print(top_mem)""" + + print("_"*30) + execution_time = asyncio.get_event_loop().time() - start_time + sleep_time = max(0, 1.0 - execution_time) + await asyncio.sleep(sleep_time) + except asyncio.CancelledError: + print("Task cancelled") + finally: + conn.close() + print("Finished execution") + + try: + asyncio.run(collector_test()) + except KeyboardInterrupt: + print("Finished collector test run") \ No newline at end of file diff --git a/focusos/feature_engineer.py b/focusos/feature_engineer.py new file mode 100644 index 0000000..88c5365 --- /dev/null +++ b/focusos/feature_engineer.py @@ -0,0 +1,71 @@ +import pandas as pd +import sys +import os +sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) +from config import DB_PATH + +# Import the stateless database window function +from focusos.sliding_window import get_window_from_db + +# sliding window fetching all the 120 rows via the sliding_window module +def extract_features(df: pd.DataFrame): + try: + if df is None or df.empty: + return None + + # reversing the index so that 0 is oldest and -1 is newest + df = df.iloc[::-1].reset_index(drop=True) + + # cpu + cpu_mean = df["cpu_usage_percent"].mean() + cpu_max = df["cpu_usage_percent"].max() + cpu_variance = df["cpu_usage_percent"].var() + + # mem feature engineering + ram_mean = df["memory_percent"].mean() + ram_growth_rate = (df["memory_percent"].iloc[-1] - df["memory_percent"].iloc[0]) if len(df) > 0 else 0 + + # network + net_mean = (df["net_bytes_sent"] + df["net_bytes_recv"]).mean() + disk_io_mean = (df["disk_write_mb_s"] + df["disk_read_mb_s"]).mean() + + # processes + process_count_mean = df["total_processes"].mean() + # 'thread_count' is not in layer1_sys + thread_count_mean = 0 + # We use fillna('') so .str doesn't crash on missing process names + process_col = df["process_data"].fillna("").str.lower() + + vscode_active = int(process_col.str.contains("code|code-insiders|vsls-agent|antigravity|sublime", regex=True).any()) + browser_active = int(process_col.str.contains("chrome|firefox|brave|msedge", regex=True).any()) + compiler_active = int(process_col.str.contains("gcc|g\\+\\+|clang|rustc|javac|make", regex=True).any()) + + features = { + "cpu_mean": cpu_mean, + "cpu_max": cpu_max, + "cpu_variance": cpu_variance, + "ram_mean": ram_mean, + "ram_growth_rate": ram_growth_rate, + "network_mean": net_mean, + "disk_io_mean": disk_io_mean, + "process_count_mean": process_count_mean, + "thread_count_mean": thread_count_mean, + "vscode_active": vscode_active, + "browser_active": browser_active, + "compiler_active": compiler_active, + } + features_df = pd.DataFrame([features]) + print(features_df) + return features_df + + except Exception as e: + print(f"Window Extraction Error: {e}") + return None + +if __name__ == "__main__": + print("Fetching window from database...") + df = get_window_from_db() + if df is not None: + extract_features(df) + else: + print("Not enough data or database empty.") \ No newline at end of file diff --git a/focusos/readme.md b/focusos/readme.md new file mode 100644 index 0000000..48ecaad --- /dev/null +++ b/focusos/readme.md @@ -0,0 +1,547 @@ +# FocusOS β€” Module Documentation + +**Part of CogniOS | Adaptive Workload Understanding and System Optimization Engine** + +--- + +## Table of Contents + +1. [Module Overview](#1-module-overview) +2. [Workflow](#2-workflow) +3. [File Structure](#3-file-structure) +4. [Function Definitions](#4-function-definitions) +5. [Useful Telemetry Data](#5-useful-telemetry-data) +6. [Additional Information](#6-additional-information) +7. [Module Diagram](#7-module-diagram) +8. [Limitations](#8-limitations) + +--- + +## 1. Module Overview + +### What is FocusOS? + +FocusOS is the **adaptive optimization layer** of CogniOS. Its core responsibility is to understand *what kind of work the user is currently doing* β€” not just which app is open β€” and dynamically tune the operating system's resource allocation to make that work feel faster and smoother. + +Traditional OS schedulers are workload-agnostic. They don't know if the user is in a deep coding session, compiling a project, gaming, or on a video call. FocusOS bridges this gap by continuously inferring workload context from system telemetry and applying intelligent, context-aware optimizations. + +> Instead of: *"What app is open?"* +> FocusOS asks: *"What kind of workload is happening right now?"* + +### Workload Categories Detected + +| Workload | Description | +|----------|-------------| +| `Coding` | IDE active, moderate CPU, low network, Python/Node processes | +| `Compiling` | Compiler active, high CPU, high disk I/O | +| `Gaming` | Game process dominant, high GPU/CPU, low network variance | +| `Video Call` | Zoom/Meet active, high network, moderate CPU | +| `Browsing` | Browser dominant, variable network, moderate CPU | +| `Idle` | Low CPU/RAM/disk, no dominant foreground process | + +--- + +## 2. Workflow + +### Training Phase + +**Step 1 β€” Collect telemetry every second** + +- **Metric Sourcing:** Relevant metrics are collected from Layer 1 and Layer 2. +- **In-Memory Buffer Storage:** A buffer may be implemented to eliminate the overhead of writing directly to the database tables every second. +- **Feature Classification:** + - System features + - Process features β€” a dedicated function is defined to fetch process metrics. To accurately compute the process creation rate, `asyncio` is used. This function accepts the local database path as its primary input argument. + - Context features + +**Step 2 β€” Generation of time-averaged statistics and final feature vector** + +- **Sliding window execution:** A dedicated function is defined to calculate rolling statistical metrics. +- **Execution Interval:** This function is triggered strictly every 30 seconds utilizing `asyncio`. +- **Dataset Creation:** A 2D array encompassing all compiled feature vectors is generated to assemble the training dataset for the clustering model. +- **Feature Scaling:** Prior to passing the dataset to KMeans, the data is normalized using `StandardScaler` from `sklearn.preprocessing`. This ensures that all metrics carry a mathematically similar weightage during distance calculations. +- **Unsupervised Clustering:** The output of this training phase is a tracking array (`trained_array`) that assigns a definitive cluster integer to every input feature vector row. +- **Pseudo-Labeling:** To perform pseudo-labeling, the centroid of each individual cluster is evaluated. This is achieved by calculating the mathematical mean of every feature vector that has been mapped to that same cluster number within the `trained_array`. +- **Supervised Mapping Database:** A "trained database" is then constructed to store the original structural feature vectors, along with a column storing their corresponding mapped workload labels. +- **XGBoost Dataset Formulation:** For training the supervised XGBoost model, `x_train` is designated as the initial 2D telemetry array, while `y_train` is fed the calculated `trained_array` labels from KMeans. + +### Testing with Actual Data + +- On live telemetry data, the module evaluates the system state using `model.predict_proba()` to compute a probability distribution across the different workload categories. +- **Label Dictionary Mapping:** The mapping from cluster number to workload category is done via a predefined dictionary mapping. +- **Threshold:** If the prediction confidence score for any individual workload category exceeds **80%**, the optimization layer triggers to actively prioritize that specific process. + +--- + +## 3. File Structure + +``` +focusos/ +β”‚ +β”œβ”€β”€ telemetry/ +β”‚ └── collector.py # Reads psutil + /proc, emits 1-sec snapshots +β”‚ +β”œβ”€β”€ windowing/ +β”‚ └── sliding_window.py # Maintains a 120-sample rolling buffer +β”‚ +β”œβ”€β”€ features/ +β”‚ └── feature_engineer.py # Converts raw window β†’ statistical feature vector +β”‚ +β”œβ”€β”€ models/ +β”‚ β”œβ”€β”€ cluster_trainer.py # KMeans training for unsupervised discovery +β”‚ β”œβ”€β”€ label_mapper.py # Maps cluster IDs β†’ workload names (human step) +β”‚ └── classifier.py # XGBoost train + inference +β”‚ +β”œβ”€β”€ optimization/ +β”‚ └── optimizer.py # Applies nice(), cpu_affinity(), ionice() via psutil +β”‚ +β”œβ”€β”€ explanation/ +β”‚ └── llm_explainer.py # Builds LLM prompt from prediction + feature importance +β”‚ +β”œβ”€β”€ dashboard/ +β”‚ └── dashboard_server.py # Serves data to React frontend +β”‚ +β”œβ”€β”€ utils/ +β”‚ └── process_utils.py # Helper: get foreground app, detect compiler, etc. +β”‚ +β”œβ”€β”€ models_saved/ +β”‚ β”œβ”€β”€ kmeans_model.pkl # Saved KMeans model +β”‚ └── xgboost_model.pkl # Saved XGBoost classifier +β”‚ +└── main.py # Entry point: starts daemon loop +``` + +--- + +## 4. Function Definitions + +### `collector.py` + +```python +def collect_snapshot(conn) -> dict +``` + +**Purpose:** Collects one complete telemetry snapshot from `psutil` and `/proc` every second. Stores the snapshot into the shared SQLite database and returns it as a dict for the sliding window buffer. This is the heartbeat of FocusOS β€” everything downstream depends on this data. + +```python +timestamp < (strftime('%s','now') - 1) +``` + +--- + +```python +def get_foreground_app() -> str +``` + +**Purpose:** Detects the currently focused window title using `xdotool`. Returns the lowercased application name string. Falls back to `'unknown'` if no window manager is available or the call fails. + +```python +subprocess.check_output(['xdotool', 'getactivewindow']) +``` + +--- + +```python +def get_top_processes(n: int = 5) -> list[dict] +``` + +**Purpose:** Returns the top N processes sorted by CPU usage, and separately by memory usage. Each entry contains `pid`, `name`, `cpu_percent`, `memory_percent`, `num_threads`, `status`. Called every 5 seconds (Layer 2 telemetry). Only top-5 are stored to keep the database lean. + +```python +psutil.process_iter(['pid','name','cpu_percent','memory_percent','num_threads','status']) +``` + +--- + +### `sliding_window.py` + +```python +class SlidingWindowBuffer(maxlen: int = 120) +``` + +**Purpose:** Maintains a fixed-size FIFO deque of telemetry snapshots representing the last 2 minutes of system state (120 samples at 1/sec). When the buffer is full, the oldest entry is automatically dropped on each new addition. This is the temporal memory FocusOS reasons over. + +```python +self.buffer = deque(maxlen=120) +``` + +--- + +```python +def add(self, snapshot: dict) -> None +``` + +**Purpose:** Appends a new telemetry snapshot to the buffer. If the buffer is already at `maxlen`, the oldest snapshot is evicted automatically by the deque. Called once per second by the daemon loop. + +```python +self.buffer.append(snapshot) +``` + +--- + +```python +def is_ready(self) -> bool +``` + +**Purpose:** Returns `True` only when the buffer has accumulated exactly `maxlen` snapshots. FocusOS waits for a full 2-minute window before running its first inference. Prevents classification on incomplete data at startup. + +```python +return len(self.buffer) == self.buffer.maxlen +``` + +--- + +```python +def get_window(self) -> list[dict] +``` + +**Purpose:** Returns a plain list copy of the current buffer contents, ordered oldest to newest. This copy is passed to `feature_engineer.py` for statistical feature extraction every 30 seconds. + +```python +return list(self.buffer) +``` + +--- + +### `feature_engineer.py` + +```python +def extract_features(window: list[dict]) -> dict +``` + +**Purpose:** Converts the 120-snapshot sliding window into a flat statistical feature vector. Computes mean, max, and variance for CPU, RAM, network, and process signals. Also sets binary context flags (`vscode_active`, `compiler_active`). The output dict is the direct input to the XGBoost classifier. + +--- + +```python +def categorize_app(app_name: str) -> int +``` + +**Purpose:** Maps a raw foreground application name (window title string) to an integer category code used as a numerical feature in the ML model. Categories: `IDE=0`, `Browser=1`, `Game=2`, `Call=3`, `Terminal=4`, `Unknown=5`. Called inside `extract_features()`. + +```python +if any(k in app_name for k in ['code', 'pycharm', 'vim']): + return 0 # IDE +``` + +--- + +```python +def get_compiler_flag(window: list[dict]) -> int +``` + +**Purpose:** Scans the top process names across all 120 snapshots and returns `1` if any known compiler or build tool (`gcc`, `g++`, `make`, `clang`, `rustc`, `javac`, `cargo`) appeared in the top-5 CPU processes during the window. Returns `0` otherwise. + +```python +compiler_tools = ['gcc', 'g++', 'make', 'clang', 'rustc', 'javac', 'cargo'] +``` + +--- + +### `cluster_trainer.py` + +```python +def train_kmeans(X: np.ndarray, n_clusters: int = 5) -> KMeans +``` + +**Purpose:** Trains a KMeans clustering model on collected feature vectors to discover natural workload behavior groups. Called once during the initial bootstrapping phase before any labels exist. Saves the fitted model and scaler to `models_saved/`. This is the unsupervised first step of the semi-supervised pipeline. + +```python +model = KMeans(n_clusters=5, random_state=42, n_init=10) +model.fit(X_scaled) +``` + +--- + +```python +def assign_pseudo_labels(cluster_ids: np.ndarray, label_map: dict) -> list[str] +``` + +**Purpose:** Converts raw integer cluster IDs from KMeans output into human-readable workload label strings using a manually defined mapping (e.g. `{0: 'coding', 1: 'video_call', ...}`). This human labelling step is the bridge from unsupervised clustering to supervised classification. + +```python +return [label_map.get(cid, 'unknown') for cid in cluster_ids] +``` + +--- + +```python +def save_pseudo_labeled_dataset(X: np.ndarray, labels: list[str], path: str) -> None +``` + +**Purpose:** Persists the pseudo-labeled feature matrix to disk as a CSV file. Each row is one 2-minute window's feature vector, with its assigned workload label as the final column. This dataset is later loaded by `classifier.py` to train XGBoost. + +```python +df = pd.DataFrame(X, columns=FEATURE_NAMES) +df['label'] = labels +df.to_csv(path, index=False) +``` + +--- + +### `classifier.py` + +```python +def train_classifier(X_train: np.ndarray, y_train: np.ndarray) -> XGBClassifier +``` + +**Purpose:** Trains an XGBoost multi-class classifier on the pseudo-labeled workload dataset. Uses 100 estimators, max depth 6, and learning rate 0.1. Saves the fitted model to `models_saved/xgboost_model.pkl`. Called once after pseudo-labels are generated, and can be retrained periodically. + +```python +model = XGBClassifier(n_estimators=100, max_depth=6, learning_rate=0.1) +model.fit(X_train, y_train) +``` + +--- + +```python +def predict_workload(features: dict) -> tuple[str, float, dict] +``` + +**Purpose:** Runs a single real-time inference pass on a feature vector extracted from the current sliding window. Returns the predicted workload label, the confidence score for that label, and a dict of all class probabilities. Called every 30 seconds by the daemon loop. + +```python +probs = model.predict_proba(X)[0] +label = WORKLOAD_LABELS[np.argmax(probs)] +confidence = probs[np.argmax(probs)] +``` + +--- + +```python +def get_feature_importances() -> dict +``` + +**Purpose:** Extracts and returns the feature importance scores from the trained XGBoost model as a `{feature_name: score}` dict. The top 3 features by importance are passed to the LLM explanation layer to generate a human-readable reason for the current workload prediction. + +```python +return dict(zip(FEATURE_NAMES, model.feature_importances_)) +``` + +--- + +### `optimizer.py` + +```python +def apply_optimization(workload: str, confidence: float) -> None +``` + +**Purpose:** The action layer of FocusOS. Reads the detected workload and β€” only if confidence exceeds 80% β€” adjusts OS-level process scheduling by calling `nice()`, `cpu_affinity()`, and `ionice()` via psutil. Prioritizes relevant processes and deprioritizes background tasks. Logs all actions taken. + +--- + +```python +def prioritize_process(proc_name: str, nice_val: int) -> bool +``` + +**Purpose:** Finds a running process by name substring match and sets its CPU scheduling priority using `psutil.Process.nice()`. Returns `True` if the process was found and updated, `False` if the process was not running or access was denied. Called internally by `apply_optimization()`. + +```python +p = psutil.Process(pid) +p.nice(nice_val) +``` + +--- + +```python +def log_optimization_result(workload: str, confidence: float, actions: list[str]) -> None +``` + +**Purpose:** Writes the optimization event β€” workload label, confidence score, timestamp, and list of actions applied β€” to the `focusos_events` table in the shared CogniOS SQLite database. This log is consumed by the dashboard and by BlackBox for pre-crash workload reconstruction. + +```python +cur.execute(""" + INSERT INTO focusos_events (timestamp, workload, confidence, actions_json) + VALUES (?, ?, ?, ?) +""", (time.time(), workload, confidence, json.dumps(actions))) +``` + +--- + +### `llm_explainer.py` + +```python +def generate_explanation(prediction: str, confidence: float, top_features: dict) -> str +``` + +**Purpose:** Builds a structured JSON prompt from the workload prediction and top XGBoost feature importances, then sends it to the configured LLM (Ollama local or API). Returns a 2-sentence plain-English explanation of why the current workload was detected. Result is displayed on the dashboard. + +```python +prompt = { + "prediction": prediction, + "confidence": f"{confidence:.0%}", + "important_features": [...top 3 features...] +} +return call_llm(system_prompt, json.dumps(prompt)) +``` + +--- + +## 5. Useful Telemetry Data + +FocusOS reads from the shared CogniOS Telemetry Collector. Below are the signals most relevant to FocusOS and the reasoning behind each. + +### Layer 1: System Telemetry (Every 1 Second) + +| Signal | Why FocusOS Needs It | +|--------|---------------------| +| `cpu_percent` | Primary indicator: coding ~30-50%, compiling ~80-100% | +| `cpu_per_core` | Distribution matters: compiler uses all cores, game pins 2-4 | +| `cpu_user` / `cpu_system` | System-heavy = kernel work; user-heavy = application | +| `cpu_iowait` | High iowait = compiling or heavy disk-bound workload | +| `mem_percent` | Gaming and video editing push RAM high; coding is moderate | +| `mem_cached` | High cache = file-heavy workload (compiling, video) | +| `disk_read_mbps` | Compiling reads many source files | +| `disk_write_mbps` | Build artifacts being written = strong compiling signal | +| `net_bytes_sent` / `net_bytes_recv` | High recv = streaming/video call; near-zero = offline coding | +| `process_count` | Many new processes = build system spawning (make, cargo) | +| `context_switches` | High = many competing threads (compiling, game engine) | +| `load_1m` / `load_5m` | Sustained high load = compile or gaming; spike = anomaly | + +### Layer 2: Top Process Telemetry (Every 5 Seconds) + +| Signal | Why FocusOS Needs It | +|--------|---------------------| +| Top-5 process `name` | gcc/g++/make = compiling; zoom = video call; code = coding | +| Top process `cpu_percent` | One process at 90% = game; distributed = compile | +| Top process `memory_percent` | Memory-hungry process reveals workload type | +| `num_threads` | Many threads = parallel compile jobs or game engine | +| `read_bytes` / `write_bytes` | Compiler writes output; game engine streams assets | +| `status` | Zombie accumulation signals build system issues | + +### Layer 3: Process Metadata (Store Once per New Top-5 Entry) + +| Signal | Why FocusOS Needs It | +|--------|---------------------| +| `exe` (executable path) | `/usr/bin/gcc` is unambiguous compiling; `/opt/zoom/zoom` = video call | +| `cmdline` | `python train.py` = ML work; `make -j8` = compiling | +| `nice` (priority) | Know baseline before FocusOS modifies it (needed for rollback) | +| `create_time` | Long-lived compiler process = active sustained build | + +### Layer 4: Deep Diagnostics (Anomaly Only β€” from OS Doctor) + +| Signal | Relevance | +|--------|-----------| +| `cpu_affinity` | Avoid double-setting if process is already pinned | +| `io_priority` | Know existing I/O class before overriding | +| `open_files` | Compiler opens many `.c`/`.h` files β€” confirming signal | +| `net_connections` | Zoom/Teams keeps multiple open sockets β€” video call signal | + +### Final Feature Vector Passed to XGBoost + +```python +[ + mean_cpu, # Statistical mean of CPU over 2 min window + max_cpu, # Peak CPU during window + cpu_variance, # Low variance = sustained load; high = bursty + mean_ram, # Mean memory usage % + ram_growth_rate, # RAM trending up β†’ memory-intensive workload + mean_net_upload, # Video call = high upload + mean_net_download, # Streaming = high download + net_variance, # Low = steady call; high = burst downloads + mean_process_count, # Build systems spawn many processes + mean_thread_count, # Parallel compiler = many threads + top_proc_cpu_share, # 1 process at 90% = game; distributed = compile + foreground_app_category, # IDE=0, Browser=1, Game=2, Call=3, Terminal=4 + compiler_active, # 1 if gcc/make/cargo seen in top processes + vscode_active, # 1 if VSCode is foreground app +] +``` + +--- + +## 6. Additional Information + +### 6.1 Why Semi-Supervised (KMeans β†’ XGBoost)? + +| Approach | Problem | +|----------|---------| +| Pure supervised | Needs labeled data β€” hard to collect at scale | +| CNN on heatmap | Needs GPU, hard to explain, overkill for tabular data | +| Pure KMeans | Cannot generalize to new unseen patterns at inference time | +| **KMeans β†’ Pseudo-labels β†’ XGBoost** | No GPU, fast inference, explainable, works on small data | + +### 6.2 Why XGBoost Over Other Classifiers? + +| Property | Why It Matters for FocusOS | +|----------|---------------------------| +| Tabular data performance | Features are statistical β€” XGBoost excels here | +| Small dataset | Works well with hundreds to thousands of samples | +| Fast inference | Less than 1ms prediction, critical for 30-second real-time cycle | +| No GPU required | Runs on any Linux machine in the CogniOS target environment | + +### 6.3 Integration with Other CogniOS Modules + +| Module | Integration Point | +|--------|------------------| +| **Telemetry Collector** | FocusOS reads from the shared SQLite telemetry database (Layers 1-3) | +| **OS Doctor** | OS Doctor flags CPU/Memory > 90% anomalies so FocusOS avoids misclassifying a runaway process as "compiling" | +| **BlackBox** | FocusOS writes its workload timeline to the events table; BlackBox uses this to reconstruct pre-crash workload context | +| **Dashboard** | FocusOS pushes workload, confidence, probabilities, actions, and LLM explanation via WebSocket to the React frontend | + +### 7. Module Diagram +```mermaid +graph TD + subgraph Phase1["Phase 1: Telemetry Collection"] + A1["psutil (CPU, RAM, Disk, Net)"] --> B1["Telemetry Collector Daemon"] + A2["/proc filesystem (Process/thread stats)"] --> B1 + A3["Window Manager APIs (xprop / xdotool / compositor)"] --> B1 + B1 -->|Sample every 1 sec| C1["Raw Metrics Buffer"] + end + + subgraph Phase2["Phase 2: Sliding Window Generator"] + C1 --> D1["120-Second FIFO Queue"] + D1 --> E1["Matrix Representation (120 x Feature_Count)"] + end + + subgraph Phase3["Phase 3: Feature Engineering"] + E1 --> F1["Compute Statistical Metrics"] + F1 --> G1["CPU Stats (Mean, Max, Variance)"] + F1 --> G2["RAM Stats (Mean, Growth Rate)"] + F1 --> G3["Network Stats (Mean Up/Down, Variance)"] + F1 --> G4["Process Stats (Count, Threads, Top Share)"] + F1 --> G5["Context Stats (One-Hot Encoded Categories)"] + G1 & G2 & G3 & G4 & G5 --> H1["Final Feature Vector"] + end + + subgraph Phase4["Phase 4: Unsupervised Clustering"] + H1 -->|Batch Historical Data| I1["KMeans Clustering (k=5)"] + I1 --> J1["Discover 5 Resource Profiles"] + J1 --> K1["Analyze Profile Rules (e.g. High Net + Zoom)"] + K1 --> L1["Apply Human Pseudo-Labels (Coding, Compiling, etc.)"] + end + + subgraph Phase5["Phase 5: Supervised Classification"] + L1 --> M1["Labeled Tabular Dataset"] + M1 --> N1["Train XGBoost Classifier"] + H1 -->|Live Vector Every 30 sec| O1["XGBoost Real-Time Inference"] + N1 --> O1 + O1 --> P1{"Confidence > 80%?"} + end + + subgraph Phase6["Phase 6: Optimization"] + P1 -->|Yes| Q1["Identify Target Process"] + P1 -->|No| Q2["Maintain Current State"] + Q1 --> R1["Apply nice() / renice priority"] + Q1 --> R2["Configure sched_setaffinity() (CPU core separation)"] + end + + subgraph Phase7["Phase 7: Explanation & UI"] + O1 --> S1["Extract XGBoost Feature Importances"] + S1 --> T1["Build Context JSON Payload"] + T1 --> U1["Query Local LLM Explanation Layer"] + U1 --> V1["Generate Natural Language Explanation"] + V1 --> W1["Streamlit Dashboard Display"] + R1 --> W1 + R2 --> W1 + end +``` + +### 8. Limitations + +| Limitation | Description | +|------------|------------| +| **Optimisation Delay** | FocusOS relies on a 120 sample buffer to represent the last 2 minutes performance as a single feature vector. The module may not be able to apply optimisations for the the first 120 seconds after the dameon starts. | +| **Rigidity with Labels** | The pseudo labelling structure maps the system state from cluster numbers to definite workload categories. In case of complex or mixed workflows, system metrics may not fit one particular category, which may get misclassified or classified as β€œunknown”. | +| **Threshold for Optimisation** | If the confidence for a particular workload category is more than 80%, the optimisation layer is deployed. In case of a mixed workload, such as 55% compiling and 35% coding, the threshold will not be met, and the optimisation layer will be skipped. | +| **Dependence on OS Doctor for Anomaly Detection** | FocusOS is not able to distinguish between heavy workload and a broken application that is monopolising a core. If OS Doctor is unable to detect the anomaly, FocusOS may incorrectly prioritise this rogue application with system hardware. | \ No newline at end of file diff --git a/focusos/sliding_window.py b/focusos/sliding_window.py new file mode 100644 index 0000000..9c530e0 --- /dev/null +++ b/focusos/sliding_window.py @@ -0,0 +1,39 @@ +import os +import sys +import sqlite3 +import pandas as pd +from collections import deque +sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) +from config import DB_PATH +from config import SLIDING_WIND_N +# class SlidingWindowBuffer: +# def __init__(self, maxlen: int = SLIDING_WIND_N): +# # it initialises the deque with a maxlen=SLIDING_WIND_N +# self.buffer = deque(maxlen=maxlen) + +# def add(self, snapshot: dict) -> None: +# # this checks the deque if full then drops the oldest adds newest +# self.buffer.append(snapshot) + +# def is_ready(self) -> bool: +# # keeps check if maxlen is achieved or not +# return len(self.buffer) == self.buffer.maxlen + +# def get_window(self) -> list[dict]: +# # Returns a copy of the current buffer as a standard Python list. +# return list(self.buffer) +def get_window_from_db(db_path=DB_PATH, limit=SLIDING_WIND_N): + try: + with sqlite3.connect(db_path) as conn: + # it will fetch last nth rows n=limit from the db + query = f"select * from layer1_sys order by timestamp desc limit {limit}" + df = pd.read_sql(query, conn) + if df.empty: + return None + + # Reverses the dataframe so the oldest data is first, newest is last + df = df[::-1].reset_index(drop=True) + return df + except Exception as e: + print(f"Window Extraction Error: {e}") + return None diff --git a/main.py b/main.py index 6e2b21e..fcb68db 100644 --- a/main.py +++ b/main.py @@ -1 +1,29 @@ """Single startup script for CogniOS.""" +import subprocess +import sys + +print("Starting CogniOS System...") + +# Start the Telemetry Daemon in the background +# Output is routed to telemetry_daemon.log +telemetry_log = open('telemetry_daemon.log', 'w') +telemetry_process = subprocess.Popen( + [sys.executable, "cognios_as_daemon.py"], + stdout=telemetry_log, + stderr=telemetry_log +) +print(f"Telemetry Daemon started! (PID: {telemetry_process.pid})") +print("To view the live logs anytime, run: tail -f telemetry_daemon.log") +print("To stop the daemon, run: pkill -f cognios_as_daemon.py") +# Start the FocusOS Daemon in the background +# Output is routed to focusos.log +# focusos_log = open('focusos.log', 'w') +# focusos_process = subprocess.Popen( +# [sys.executable, "focusos_daemon.py"], +# stdout=focusos_log, +# stderr=focusos_log +#) +#print(f"FocusOS Daemon started! (PID: {focusos_process.pid})") + +#print("\nAll CogniOS modules are now running silently in the background!") +print("Your terminal is free to use.") diff --git a/telemetry_daemon.log b/telemetry_daemon.log new file mode 100644 index 0000000..48e0b65 --- /dev/null +++ b/telemetry_daemon.log @@ -0,0 +1,370 @@ +2026-07-10 21:15:51,985 - INFO - Starting CogniOS Daemon. Saving metrics to 'cognios_telemetry.db' every 1 second. +2026-07-10 21:15:52,210 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:45:51.986269+00:00 +2026-07-10 21:15:53,467 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:45:53.211139+00:00 +2026-07-10 21:15:54,734 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:45:54.467925+00:00 +2026-07-10 21:15:55,985 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:45:55.734385+00:00 +2026-07-10 21:15:57,256 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:45:56.986033+00:00 +2026-07-10 21:15:58,525 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:45:58.256736+00:00 +2026-07-10 21:15:59,798 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:45:59.525899+00:00 +2026-07-10 21:16:01,067 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:46:00.798561+00:00 +2026-07-10 21:16:02,327 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:46:02.067647+00:00 +2026-07-10 21:16:03,633 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:46:03.327908+00:00 +2026-07-10 21:16:04,887 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:46:04.633296+00:00 +2026-07-10 21:16:06,170 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:46:05.888062+00:00 +2026-07-10 21:16:07,424 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:46:07.170636+00:00 +2026-07-10 21:16:08,672 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:46:08.424896+00:00 +2026-07-10 21:16:09,919 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:46:09.673142+00:00 +2026-07-10 21:16:11,203 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:46:10.920119+00:00 +2026-07-10 21:16:12,449 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:46:12.203263+00:00 +2026-07-10 21:16:13,700 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:46:13.449281+00:00 +2026-07-10 21:16:14,951 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:46:14.700697+00:00 +2026-07-10 21:16:16,219 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:46:15.952074+00:00 +2026-07-10 21:16:17,481 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:46:17.220070+00:00 +2026-07-10 21:16:18,755 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:46:18.481916+00:00 +2026-07-10 21:16:19,994 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:46:19.756086+00:00 +2026-07-10 21:16:21,249 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:46:20.994280+00:00 +2026-07-10 21:16:22,526 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:46:22.249952+00:00 +2026-07-10 21:16:23,796 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:46:23.526867+00:00 +2026-07-10 21:16:25,060 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:46:24.796327+00:00 +2026-07-10 21:16:26,333 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:46:26.060977+00:00 +2026-07-10 21:16:27,604 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:46:27.333477+00:00 +2026-07-10 21:16:28,875 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:46:28.604269+00:00 +2026-07-10 21:16:30,149 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:46:29.875674+00:00 +2026-07-10 21:16:31,462 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:46:31.150138+00:00 +2026-07-10 21:16:32,715 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:46:32.462596+00:00 +2026-07-10 21:16:33,977 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:46:33.715607+00:00 +2026-07-10 21:16:35,286 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:46:34.978477+00:00 +2026-07-10 21:16:36,593 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:46:36.286974+00:00 +2026-07-10 21:16:37,860 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:46:37.594369+00:00 +2026-07-10 21:16:39,155 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:46:38.860712+00:00 +2026-07-10 21:16:40,392 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:46:40.155286+00:00 +2026-07-10 21:16:41,654 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:46:41.393004+00:00 +2026-07-10 21:16:42,950 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:46:42.655286+00:00 +2026-07-10 21:16:44,208 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:46:43.950780+00:00 +2026-07-10 21:16:45,477 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:46:45.208762+00:00 +2026-07-10 21:16:46,785 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:46:46.478427+00:00 +2026-07-10 21:16:48,091 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:46:47.785686+00:00 +2026-07-10 21:16:49,339 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:46:49.091465+00:00 +2026-07-10 21:16:50,653 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:46:50.339691+00:00 +2026-07-10 21:16:51,959 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:46:51.654782+00:00 +2026-07-10 21:16:53,271 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:46:52.960554+00:00 +2026-07-10 21:16:54,537 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:46:54.271228+00:00 +2026-07-10 21:16:55,802 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:46:55.537797+00:00 +2026-07-10 21:16:57,074 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:46:56.802350+00:00 +2026-07-10 21:16:58,341 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:46:58.075097+00:00 +2026-07-10 21:16:59,650 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:46:59.341298+00:00 +2026-07-10 21:17:00,958 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:47:00.651536+00:00 +2026-07-10 21:17:02,267 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:47:01.959319+00:00 +2026-07-10 21:17:03,574 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:47:03.268527+00:00 +2026-07-10 21:17:04,841 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:47:04.574318+00:00 +2026-07-10 21:17:06,144 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:47:05.841685+00:00 +2026-07-10 21:17:07,412 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:47:07.145169+00:00 +2026-07-10 21:17:08,660 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:47:08.413100+00:00 +2026-07-10 21:17:09,962 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:47:09.660784+00:00 +2026-07-10 21:17:11,258 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:47:10.962473+00:00 +2026-07-10 21:17:12,532 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:47:12.259092+00:00 +2026-07-10 21:17:13,785 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:47:13.532453+00:00 +2026-07-10 21:17:15,059 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:47:14.785753+00:00 +2026-07-10 21:17:16,322 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:47:16.059966+00:00 +2026-07-10 21:17:17,631 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:47:17.322574+00:00 +2026-07-10 21:17:18,932 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:47:18.631761+00:00 +2026-07-10 21:17:20,202 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:47:19.932636+00:00 +2026-07-10 21:17:21,466 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:47:21.203530+00:00 +2026-07-10 21:17:22,735 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:47:22.466803+00:00 +2026-07-10 21:17:24,010 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:47:23.735311+00:00 +2026-07-10 21:17:25,267 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:47:25.010733+00:00 +2026-07-10 21:17:26,517 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:47:26.267220+00:00 +2026-07-10 21:17:27,825 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:47:27.518098+00:00 +2026-07-10 21:17:29,095 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:47:28.825986+00:00 +2026-07-10 21:17:30,391 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:47:30.095657+00:00 +2026-07-10 21:17:31,659 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:47:31.392042+00:00 +2026-07-10 21:17:32,924 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:47:32.660005+00:00 +2026-07-10 21:17:34,179 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:47:33.925200+00:00 +2026-07-10 21:17:35,427 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:47:35.179413+00:00 +2026-07-10 21:17:36,710 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:47:36.427350+00:00 +2026-07-10 21:17:37,931 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:47:37.710884+00:00 +2026-07-10 21:17:39,196 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:47:38.931275+00:00 +2026-07-10 21:17:40,465 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:47:40.197079+00:00 +2026-07-10 21:17:41,740 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:47:41.465682+00:00 +2026-07-10 21:17:43,045 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:47:42.740303+00:00 +2026-07-10 21:17:44,296 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:47:44.046034+00:00 +2026-07-10 21:17:45,607 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:47:45.297387+00:00 +2026-07-10 21:17:46,917 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:47:46.608180+00:00 +2026-07-10 21:17:48,229 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:47:47.917840+00:00 +2026-07-10 21:17:49,531 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:47:49.229723+00:00 +2026-07-10 21:17:50,804 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:47:50.531999+00:00 +2026-07-10 21:17:52,103 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:47:51.804345+00:00 +2026-07-10 21:17:53,403 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:47:53.104410+00:00 +2026-07-10 21:17:54,715 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:47:54.404518+00:00 +2026-07-10 21:17:56,022 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:47:55.716498+00:00 +2026-07-10 21:17:57,322 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:47:57.023133+00:00 +2026-07-10 21:17:58,597 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:47:58.322932+00:00 +2026-07-10 21:17:59,865 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:47:59.598118+00:00 +2026-07-10 21:18:01,177 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:48:00.865400+00:00 +2026-07-10 21:18:02,441 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:48:02.178089+00:00 +2026-07-10 21:18:03,737 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:48:03.442425+00:00 +2026-07-10 21:18:04,994 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:48:04.737197+00:00 +2026-07-10 21:18:06,295 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:48:05.994742+00:00 +2026-07-10 21:18:07,561 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:48:07.295384+00:00 +2026-07-10 21:18:08,811 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:48:08.561628+00:00 +2026-07-10 21:18:10,102 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:48:09.811455+00:00 +2026-07-10 21:18:11,357 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:48:11.103038+00:00 +2026-07-10 21:18:12,590 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:48:12.358119+00:00 +2026-07-10 21:18:13,898 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:48:13.590512+00:00 +2026-07-10 21:18:15,166 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:48:14.898296+00:00 +2026-07-10 21:18:16,415 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:48:16.167003+00:00 +2026-07-10 21:18:17,682 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:48:17.415451+00:00 +2026-07-10 21:18:18,952 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:48:18.683095+00:00 +2026-07-10 21:18:20,264 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:48:19.953197+00:00 +2026-07-10 21:18:21,570 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:48:21.264476+00:00 +2026-07-10 21:18:22,881 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:48:22.570629+00:00 +2026-07-10 21:18:24,191 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:48:23.882287+00:00 +2026-07-10 21:18:25,500 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:48:25.192454+00:00 +2026-07-10 21:18:26,805 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:48:26.501694+00:00 +2026-07-10 21:18:28,114 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:48:27.806154+00:00 +2026-07-10 21:18:29,421 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:48:29.115090+00:00 +2026-07-10 21:18:30,734 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:48:30.422197+00:00 +2026-07-10 21:18:32,035 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:48:31.734932+00:00 +2026-07-10 21:18:33,301 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:48:33.036123+00:00 +2026-07-10 21:18:34,568 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:48:34.302136+00:00 +2026-07-10 21:18:35,832 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:48:35.568727+00:00 +2026-07-10 21:18:37,139 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:48:36.832601+00:00 +2026-07-10 21:18:38,408 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:48:38.139349+00:00 +2026-07-10 21:18:39,682 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:48:39.408822+00:00 +2026-07-10 21:18:40,944 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:48:40.683030+00:00 +2026-07-10 21:18:42,216 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:48:41.944609+00:00 +2026-07-10 21:18:43,512 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:48:43.216326+00:00 +2026-07-10 21:18:44,779 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:48:44.512336+00:00 +2026-07-10 21:18:46,049 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:48:45.779992+00:00 +2026-07-10 21:18:47,321 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:48:47.050253+00:00 +2026-07-10 21:18:48,591 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:48:48.321415+00:00 +2026-07-10 21:18:49,852 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:48:49.591899+00:00 +2026-07-10 21:18:51,125 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:48:50.852452+00:00 +2026-07-10 21:18:52,425 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:48:52.126679+00:00 +2026-07-10 21:18:53,677 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:48:53.425244+00:00 +2026-07-10 21:18:54,942 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:48:54.677746+00:00 +2026-07-10 21:18:56,249 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:48:55.942624+00:00 +2026-07-10 21:18:57,516 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:48:57.250170+00:00 +2026-07-10 21:18:58,790 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:48:58.516776+00:00 +2026-07-10 21:19:00,094 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:48:59.791169+00:00 +2026-07-10 21:19:01,402 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:49:01.094780+00:00 +2026-07-10 21:19:02,712 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:49:02.402978+00:00 +2026-07-10 21:19:03,942 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:49:03.712828+00:00 +2026-07-10 21:19:05,158 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:49:04.943322+00:00 +2026-07-10 21:19:06,408 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:49:06.158316+00:00 +2026-07-10 21:19:07,663 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:49:07.408570+00:00 +2026-07-10 21:19:08,918 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:49:08.664176+00:00 +2026-07-10 21:19:10,188 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:49:09.918742+00:00 +2026-07-10 21:19:11,453 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:49:11.189068+00:00 +2026-07-10 21:19:12,701 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:49:12.454093+00:00 +2026-07-10 21:19:13,950 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:49:13.702015+00:00 +2026-07-10 21:19:15,210 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:49:14.951198+00:00 +2026-07-10 21:19:16,478 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:49:16.210405+00:00 +2026-07-10 21:19:17,787 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:49:17.479080+00:00 +2026-07-10 21:19:19,049 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:49:18.787910+00:00 +2026-07-10 21:19:20,313 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:49:20.049473+00:00 +2026-07-10 21:19:21,564 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:49:21.314155+00:00 +2026-07-10 21:19:22,818 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:49:22.564682+00:00 +2026-07-10 21:19:24,069 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:49:23.819656+00:00 +2026-07-10 21:19:25,323 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:49:25.069500+00:00 +2026-07-10 21:19:26,625 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:49:26.323568+00:00 +2026-07-10 21:19:27,934 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:49:27.626252+00:00 +2026-07-10 21:19:29,181 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:49:28.934510+00:00 +2026-07-10 21:19:30,476 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:49:30.181654+00:00 +2026-07-10 21:19:31,734 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:49:31.476356+00:00 +2026-07-10 21:19:33,028 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:49:32.735016+00:00 +2026-07-10 21:19:34,337 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:49:34.029656+00:00 +2026-07-10 21:19:35,644 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:49:35.338139+00:00 +2026-07-10 21:19:36,891 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:49:36.645202+00:00 +2026-07-10 21:19:38,194 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:49:37.891416+00:00 +2026-07-10 21:19:39,502 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:49:39.194506+00:00 +2026-07-10 21:19:40,809 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:49:40.503094+00:00 +2026-07-10 21:19:42,099 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:49:41.810673+00:00 +2026-07-10 21:19:43,388 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:49:43.100167+00:00 +2026-07-10 21:19:44,673 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:49:44.389289+00:00 +2026-07-10 21:19:45,924 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:49:45.673502+00:00 +2026-07-10 21:19:47,174 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:49:46.924779+00:00 +2026-07-10 21:19:48,394 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:49:48.175223+00:00 +2026-07-10 21:19:49,659 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:49:49.394599+00:00 +2026-07-10 21:19:50,918 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:49:50.660114+00:00 +2026-07-10 21:19:52,206 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:49:51.918456+00:00 +2026-07-10 21:19:53,420 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:49:53.206329+00:00 +2026-07-10 21:19:54,704 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:49:54.420986+00:00 +2026-07-10 21:19:56,008 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:49:55.704543+00:00 +2026-07-10 21:19:57,319 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:49:57.009097+00:00 +2026-07-10 21:19:58,630 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:49:58.320823+00:00 +2026-07-10 21:19:59,939 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:49:59.630783+00:00 +2026-07-10 21:20:01,247 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:50:00.939940+00:00 +2026-07-10 21:20:02,549 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:50:02.248100+00:00 +2026-07-10 21:20:03,858 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:50:03.549492+00:00 +2026-07-10 21:20:05,162 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:50:04.858904+00:00 +2026-07-10 21:20:06,472 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:50:06.162993+00:00 +2026-07-10 21:20:07,734 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:50:07.472796+00:00 +2026-07-10 21:20:09,041 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:50:08.734634+00:00 +2026-07-10 21:20:10,335 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:50:10.042584+00:00 +2026-07-10 21:20:11,590 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:50:11.335358+00:00 +2026-07-10 21:20:12,837 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:50:12.590266+00:00 +2026-07-10 21:20:14,128 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:50:13.837341+00:00 +2026-07-10 21:20:15,418 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:50:15.128694+00:00 +2026-07-10 21:20:16,705 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:50:16.418484+00:00 +2026-07-10 21:20:17,980 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:50:17.706560+00:00 +2026-07-10 21:20:19,241 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:50:18.980473+00:00 +2026-07-10 21:20:20,509 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:50:20.241256+00:00 +2026-07-10 21:20:21,781 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:50:21.509253+00:00 +2026-07-10 21:20:23,092 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:50:22.781770+00:00 +2026-07-10 21:20:24,381 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:50:24.092437+00:00 +2026-07-10 21:20:25,670 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:50:25.381957+00:00 +2026-07-10 21:20:26,961 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:50:26.671109+00:00 +2026-07-10 21:20:28,261 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:50:27.961931+00:00 +2026-07-10 21:20:29,569 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:50:29.262215+00:00 +2026-07-10 21:20:30,878 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:50:30.570114+00:00 +2026-07-10 21:20:32,170 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:50:31.879061+00:00 +2026-07-10 21:20:33,460 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:50:33.170757+00:00 +2026-07-10 21:20:34,749 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:50:34.461132+00:00 +2026-07-10 21:20:36,009 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:50:35.749363+00:00 +2026-07-10 21:20:37,271 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:50:37.009324+00:00 +2026-07-10 21:20:38,526 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:50:38.271865+00:00 +2026-07-10 21:20:39,775 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:50:39.526770+00:00 +2026-07-10 21:20:41,065 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:50:40.775465+00:00 +2026-07-10 21:20:42,282 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:50:42.066031+00:00 +2026-07-10 21:20:43,539 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:50:43.283039+00:00 +2026-07-10 21:20:44,784 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:50:44.539977+00:00 +2026-07-10 21:20:46,032 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:50:45.784468+00:00 +2026-07-10 21:20:47,322 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:50:47.032941+00:00 +2026-07-10 21:20:48,613 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:50:48.323600+00:00 +2026-07-10 21:20:49,904 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:50:49.613893+00:00 +2026-07-10 21:20:51,207 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:50:50.904791+00:00 +2026-07-10 21:20:52,496 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:50:52.207749+00:00 +2026-07-10 21:20:53,745 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:50:53.496693+00:00 +2026-07-10 21:20:55,041 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:50:54.745918+00:00 +2026-07-10 21:20:56,304 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:50:56.042746+00:00 +2026-07-10 21:20:57,562 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:50:57.304736+00:00 +2026-07-10 21:20:58,799 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:50:58.562622+00:00 +2026-07-10 21:21:00,104 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:50:59.799992+00:00 +2026-07-10 21:21:01,414 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:51:01.105607+00:00 +2026-07-10 21:21:02,722 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:51:02.415093+00:00 +2026-07-10 21:21:04,029 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:51:03.722849+00:00 +2026-07-10 21:21:05,330 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:51:05.029819+00:00 +2026-07-10 21:21:06,621 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:51:06.331223+00:00 +2026-07-10 21:21:07,918 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:51:07.622168+00:00 +2026-07-10 21:21:09,213 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:51:08.919158+00:00 +2026-07-10 21:21:10,504 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:51:10.214448+00:00 +2026-07-10 21:21:11,814 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:51:11.504963+00:00 +2026-07-10 21:21:13,119 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:51:12.814918+00:00 +2026-07-10 21:21:14,410 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:51:14.119700+00:00 +2026-07-10 21:21:15,659 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:51:15.411589+00:00 +2026-07-10 21:21:16,908 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:51:16.659220+00:00 +2026-07-10 21:21:18,200 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:51:17.909941+00:00 +2026-07-10 21:21:19,501 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:51:19.201087+00:00 +2026-07-10 21:21:20,805 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:51:20.502189+00:00 +2026-07-10 21:21:22,058 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:51:21.805377+00:00 +2026-07-10 21:21:23,316 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:51:23.058679+00:00 +2026-07-10 21:21:24,555 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:51:24.316219+00:00 +2026-07-10 21:21:25,812 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:51:25.555508+00:00 +2026-07-10 21:21:27,044 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:51:26.812344+00:00 +2026-07-10 21:21:28,332 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:51:28.045106+00:00 +2026-07-10 21:21:29,552 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:51:29.332305+00:00 +2026-07-10 21:21:30,804 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:51:30.552681+00:00 +2026-07-10 21:21:32,061 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:51:31.804278+00:00 +2026-07-10 21:21:33,350 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:51:33.062248+00:00 +2026-07-10 21:21:34,615 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:51:34.350444+00:00 +2026-07-10 21:21:35,883 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:51:35.616029+00:00 +2026-07-10 21:21:37,121 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:51:36.883307+00:00 +2026-07-10 21:21:38,360 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:51:38.121673+00:00 +2026-07-10 21:21:39,612 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:51:39.360344+00:00 +2026-07-10 21:21:40,881 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:51:40.612780+00:00 +2026-07-10 21:21:42,153 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:51:41.881988+00:00 +2026-07-10 21:21:43,462 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:51:43.153764+00:00 +2026-07-10 21:21:44,764 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:51:44.463189+00:00 +2026-07-10 21:21:46,060 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:51:45.764802+00:00 +2026-07-10 21:21:47,365 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:51:47.061413+00:00 +2026-07-10 21:21:48,635 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:51:48.365918+00:00 +2026-07-10 21:21:49,895 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:51:49.636279+00:00 +2026-07-10 21:21:51,201 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:51:50.895851+00:00 +2026-07-10 21:21:52,511 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:51:52.202100+00:00 +2026-07-10 21:21:53,823 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:51:53.511830+00:00 +2026-07-10 21:21:55,115 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:51:54.824144+00:00 +2026-07-10 21:21:56,428 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:51:56.116491+00:00 +2026-07-10 21:21:57,737 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:51:57.428283+00:00 +2026-07-10 21:21:59,049 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:51:58.738523+00:00 +2026-07-10 21:22:00,344 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:52:00.050516+00:00 +2026-07-10 21:22:01,616 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:52:01.344568+00:00 +2026-07-10 21:22:02,923 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:52:02.617193+00:00 +2026-07-10 21:22:04,218 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:52:03.924117+00:00 +2026-07-10 21:22:05,529 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:52:05.218617+00:00 +2026-07-10 21:22:06,836 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:52:06.530265+00:00 +2026-07-10 21:22:08,102 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:52:07.837332+00:00 +2026-07-10 21:22:09,356 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:52:09.102913+00:00 +2026-07-10 21:22:10,664 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:52:10.357425+00:00 +2026-07-10 21:22:11,971 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:52:11.665547+00:00 +2026-07-10 21:22:13,261 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:52:12.972471+00:00 +2026-07-10 21:22:14,573 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:52:14.262327+00:00 +2026-07-10 21:22:15,847 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:52:15.573667+00:00 +2026-07-10 21:22:17,156 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:52:16.847238+00:00 +2026-07-10 21:22:18,449 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:52:18.156656+00:00 +2026-07-10 21:22:19,757 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:52:19.449953+00:00 +2026-07-10 21:22:21,067 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:52:20.758271+00:00 +2026-07-10 21:22:22,334 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:52:22.068227+00:00 +2026-07-10 21:22:23,647 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:52:23.334449+00:00 +2026-07-10 21:22:24,956 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:52:24.647346+00:00 +2026-07-10 21:22:26,222 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:52:25.956470+00:00 +2026-07-10 21:22:27,496 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:52:27.222397+00:00 +2026-07-10 21:22:28,764 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:52:28.497052+00:00 +2026-07-10 21:22:30,034 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:52:29.765279+00:00 +2026-07-10 21:22:31,300 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:52:31.034330+00:00 +2026-07-10 21:22:32,574 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:52:32.300875+00:00 +2026-07-10 21:22:33,846 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:52:33.574412+00:00 +2026-07-10 21:22:35,118 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:52:34.846538+00:00 +2026-07-10 21:22:36,366 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:52:36.118361+00:00 +2026-07-10 21:22:37,671 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:52:37.366406+00:00 +2026-07-10 21:22:38,945 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:52:38.671222+00:00 +2026-07-10 21:22:40,219 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:52:39.945479+00:00 +2026-07-10 21:22:41,484 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:52:41.219621+00:00 +2026-07-10 21:22:42,762 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:52:42.484429+00:00 +2026-07-10 21:22:43,999 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:52:43.762246+00:00 +2026-07-10 21:22:45,268 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:52:44.999910+00:00 +2026-07-10 21:22:46,580 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:52:46.268874+00:00 +2026-07-10 21:22:47,807 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:52:47.580239+00:00 +2026-07-10 21:22:49,039 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:52:48.808150+00:00 +2026-07-10 21:22:50,349 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:52:50.040285+00:00 +2026-07-10 21:22:51,650 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:52:51.349589+00:00 +2026-07-10 21:22:52,900 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:52:52.650682+00:00 +2026-07-10 21:22:54,146 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:52:53.900228+00:00 +2026-07-10 21:22:55,431 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:52:55.146572+00:00 +2026-07-10 21:22:56,679 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:52:56.431526+00:00 +2026-07-10 21:22:57,925 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:52:57.679735+00:00 +2026-07-10 21:22:59,173 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:52:58.925538+00:00 +2026-07-10 21:23:00,435 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:53:00.173984+00:00 +2026-07-10 21:23:01,700 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:53:01.435976+00:00 +2026-07-10 21:23:02,952 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:53:02.701057+00:00 +2026-07-10 21:23:04,207 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:53:03.952925+00:00 +2026-07-10 21:23:05,464 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:53:05.207654+00:00 +2026-07-10 21:23:06,713 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:53:06.464225+00:00 +2026-07-10 21:23:07,971 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:53:07.713589+00:00 +2026-07-10 21:23:09,229 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:53:08.971439+00:00 +2026-07-10 21:23:10,486 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:53:10.230102+00:00 +2026-07-10 21:23:11,772 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:53:11.486730+00:00 +2026-07-10 21:23:13,041 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:53:12.773479+00:00 +2026-07-10 21:23:14,343 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:53:14.041505+00:00 +2026-07-10 21:23:15,578 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:53:15.343470+00:00 +2026-07-10 21:23:16,865 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:53:16.579098+00:00 +2026-07-10 21:23:18,078 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:53:17.865385+00:00 +2026-07-10 21:23:19,332 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:53:19.078244+00:00 +2026-07-10 21:23:20,586 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:53:20.333112+00:00 +2026-07-10 21:23:21,848 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:53:21.586820+00:00 +2026-07-10 21:23:23,102 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:53:22.848857+00:00 +2026-07-10 21:23:24,397 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:53:24.102680+00:00 +2026-07-10 21:23:25,669 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:53:25.397977+00:00 +2026-07-10 21:23:26,937 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:53:26.669477+00:00 +2026-07-10 21:23:28,204 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:53:27.937485+00:00 +2026-07-10 21:23:29,455 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:53:29.205573+00:00 +2026-07-10 21:23:30,738 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:53:30.455346+00:00 +2026-07-10 21:23:31,999 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:53:31.738393+00:00 +2026-07-10 21:23:33,271 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:53:32.999935+00:00 +2026-07-10 21:23:34,520 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:53:34.271514+00:00 +2026-07-10 21:23:35,810 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:53:35.520529+00:00 +2026-07-10 21:23:37,077 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:53:36.810580+00:00 +2026-07-10 21:23:38,332 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:53:38.077881+00:00 +2026-07-10 21:23:39,591 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:53:39.333240+00:00 +2026-07-10 21:23:40,898 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:53:40.592169+00:00 +2026-07-10 21:23:42,164 - INFO - Successfully saved metrics for timestamp: 2026-07-10T15:53:41.899541+00:00