Skip to content

Multilog Analysis (Logx)#52

Open
Jeffrey-Moon wants to merge 6 commits into
mainfrom
multilog_analysis
Open

Multilog Analysis (Logx)#52
Jeffrey-Moon wants to merge 6 commits into
mainfrom
multilog_analysis

Conversation

@Jeffrey-Moon

@Jeffrey-Moon Jeffrey-Moon commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Added tests to test the features but they were AI generated as a result of me being lazy... If you can think of edge cases, let me know.

Zero-Copy Parser Optimization

  • Replaced Polars group_by loops with an $O(N)$ contiguous boundary detector.
  • Slices telemetry values into zero-copy NumPy array views of master columns, dropping parsing time to 60ms and saving 280MB of RAM per log for a test log with size $736.9\text{MB}$.
  • Parses log start timestamps from the header.

Run Catalog & Scanner

  • Indexes directory log headers in under 1ms per file.
  • Implemented date filtering and lazy evaluation in check_any with automatic garbage collection to prevent OOM leaks.
  • Added unified path resolution for directories, wildcards, and list inputs.

Overlay Graphing & Downsampling

  • Added plot_comparison and compare_summary to overlay relative time coordinates
  • Implemented max_points parameter using zero-copy step-slicing.
  • Fixed step calculation using ceiling division (n_points + max_points - 1) // max_points to enforce a strict upper bound.

Hybrid Semantic Search

  • Added optional SentenceTransformer vector search
  • Caches embeddings in-memory with automatic invalidation.
  • Lazy-loads imports to execute on keyword search if PyTorch/models are missing offline

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why does this function need to live in this file?

@Jeffrey-Moon Jeffrey-Moon Jul 18, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it was originally in csv.py but chukka said to just move it to init unless I misunderstood him. I think once we have more auxiliary functions we can create a new file

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants