The Experience Continuity Engine is a predictive server degradation detection system designed to identify subtle signs of server distress before they trigger traditional downtime alerts. Rather than replacing existing load balancers or routing infrastructure, this system acts as an intelligent decision layer. It analyzes multivariate server telemetry in real-time and feeds early-warning routing signals to upstream infrastructure, enabling proactive traffic rerouting before the end-user experiences a failure.
Traditional load balancers and infrastructure health checks are inherently reactive. They typically rely on binary up/down signals—such as ping timeouts, consecutive HTTP 5xx responses, or hard threshold breaches (e.g., CPU > 95%). By the time a server fails a conventional health check, it is already actively dropping requests, degrading user experience, or fully offline.
This engine solves that by observing the combinatorial patterns of multiple metrics simultaneously (CPU utilization, memory usage, HTTP 500 error rates, and network latency). A server whose CPU is at 80% might be completely healthy, but a server at 80% CPU with steadily increasing network latency and a slight uptick in 500 errors is likely entering a cascading failure state. This predictive approach catches degradation cross-sectionally before individual metrics breach hard failure thresholds.
The system is designed as a stateless, localized machine learning pipeline. It is intentionally lightweight for low-latency inference.
[ simulator.py ] -> Generates synthetic telemtry data with modeled failure ramps
|
v
[ historical_logs.csv ] -> Training dataset
|
v
[ train.py ] -> Trains the anomaly detection model & determines threshold
|
v
[ model.joblib ] -> Serialized model artifacts
|
v
[ app.py ] -> FastAPI inference server exposing prediction endpoints
|
v
[ demo_client.py ] -> Simulates live production traffic querying the API
simulator.py: Generates synthetic multivariate time-series data modeling both baseline healthy traffic and specific cascading degradation scenarios.train.py: Ingests the historical logs, trains an unsupervised machine learning model, calculates a statistically sound anomaly threshold based on precision requirements, and exports the model artifacts.app.py: A FastAPI application that loads the trained model into memory and exposes a low-latency REST API for real-time telemetry scoring.demo_client.py: A test harness that streams simulated live server metrics to the API to demonstrate the system's real-time predictive behavior.
The core detection engine uses an IsolationForest algorithm. This unsupervised anomaly detection approach was chosen over a supervised classifier because, for this MVP, no labeled real-world failure data exists. The model is trained on synthetic data representing "normal" operating variance. By using an unsupervised method, the model learns the multidimensional boundaries of nominal server behavior and flags data points that fall statistically outside those boundaries. Given the lack of robustly labeled failure data, unsupervised anomaly detection is the most appropriate and honest approach.
The model was evaluated against a held-out test set containing both healthy baseline periods and simulated degradation events.
- True Negatives (TN): 3514
- False Positives (FP): 86
- False Negatives (FN): 756
- True Positives (TP): 1044
- Decision Threshold: -0.500710
| Metric | Score |
|---|---|
| Precision | 92.4% |
| Recall | 58.0% |
| F1 Score | 0.713 |
| Accuracy | 84.4% |
| Detection Point | ~42% into failure ramp |
Prioritizing Precision over Recall: The detection threshold was hand-tuned to strongly favor precision (92.4%) over recall (58.0%). In a production environment, false positives (rerouting healthy traffic unnecessarily) cause network instability, cache thrashing, and alert fatigue. False negatives simply mean the system defaults back to the traditional load balancer's reactive health check. The engine successfully detects failure signatures approximately 42% of the way through a degradation event, providing a reliable early-warning window without flapping on healthy traffic.
(Note: Accuracy alone (84.4%) is an inherently misleading metric here due to the roughly 2:1 class imbalance between healthy and failing states in the test set. Precision and the specific detection point are the true indicators of utility).
The inference server exposes two endpoints for integration.
A standard liveness probe for the API itself.
Request:
curl -s http://127.0.0.1:8000/healthResponse:
{
"status": "ok",
"model_loaded": true
}The primary inference endpoint. It accepts current server telemetry and returns a normalized health score and a recommended routing action.
Request:
curl -s -X POST -H "Content-Type: application/json" -d '{
"cpu_utilization": 82.5,
"memory_usage": 78.1,
"http_500_rate": 0.02,
"network_latency_ms": 145.0
}' http://127.0.0.1:8000/predictResponse:
{
"status": "degrading",
"health_score": 0.0,
"action": "reroute"
}status: Either"healthy"or"degrading".health_score: A normalized float between0.0and1.0, where1.0is perfectly healthy and0.0is critical distress.action: Evaluates the raw anomaly score against the pre-calculated threshold. Returns"none"for healthy states and"reroute"when degradation is detected.
To run the pipeline locally, execute the following commands in order:
# 1. Install required dependencies
pip install pandas scikit-learn fastapi uvicorn requests joblib
# 2. Generate the synthetic historical dataset
python simulator.py
# 3. Train the model and generate model.joblib
python train.py
# 4. Start the inference API (runs on localhost:8000)
uvicorn app:app --reload
# 5. In a separate terminal window, run the live simulation
python demo_client.pyThis project was built under a strict 5-hour hackathon time constraint, which drove several deliberate architectural choices:
- Stateless Inference: The model relies purely on point-in-time telemetry rather than rolling windows or time-series features (like moving averages or derivatives). This was chosen to keep API latency absolute minimum and to avoid managing state across distributed inference nodes, at the cost of some contextual awareness.
- Synthetic Data: Because securing and sanitizing real production failure logs was out of scope for a 5-hour build, the system relies on a custom simulator modeling expected baseline variance and linear degradation ramps.
- Static Percentile Threshold: The boundary between
MAINTAINandREROUTEis currently set using a static, hand-tuned percentile calculation derived during training. A production system would ideally utilize a dynamic threshold that adjusts to diurnal traffic patterns.
If developed beyond the MVP stage, the following enhancements would be prioritized to make the system production-ready:
- Time-Series Feature Engineering: Incorporate short-term rolling averages and rate-of-change metrics (e.g., latency velocity) into the feature set. This would provide the model with trajectory context, likely moving the detection point even earlier in the failure ramp.
- Real Telemetry Calibration: Re-train and calibrate the IsolationForest model against real, sanitized production Prometheus/Datadog logs to capture actual operational noise and non-linear failure modes.
- Decision Hysteresis: Implement a confirmation window (e.g., requiring 3 consecutive
REROUTEsignals over 15 seconds) before triggering a routing change to prevent "flapping" during momentary, harmless traffic bursts. - Live Visualization Dashboard: Build a lightweight frontend to visualize the real-time telemetry streams against the model's decision threshold, making the system's predictive capabilities observable to infrastructure operators.
Note: My specific role in this hackathon project was designing and building the ML/Predictive Degradation Engine module (data simulation, model training, threshold tuning, and API inference layer).