bitvanes-core

Zero-trust ETL engine for AI/RAG workloads. Written in Rust, compiled to both wasm32-unknown-unknown (for the browser) and native targets (for the CLI).

What it does

A four-stage pipeline that transforms raw documents into Apache Arrow columnar chunks ready for vector database ingestion:

1. Parse - Markdown, HTML, plain text, JSON, or PDF into structural spans with heading ancestry and section classification. PDF requires the cli-pdf feature natively; the browser extracts PDF text via PDF.js.
2. Scrub - PII redaction (email, SSN, phone, credit card, API keys) via regex + Luhn validation, with an offset-delta map for projecting chunk positions back to the original document.
3. Chunk - BPE-aware splitting at structural boundaries using any of six OpenAI tokenizers (cl100k_base, o200k_base, r50k_base, p50k_base, p50k_edit, o200k_harmony), with optional overlap that is preserved even across oversized spans.
4. Assemble - Arrow RecordBatch with 9 columns (chunk_index, text, token_count, source_path, heading_path, section_kind, char offsets, embedding placeholder), exported via zero-copy FFI pointers.

Zero-trust guarantees

• All vocab files are embedded at compile time via include_str!.
• No network calls during parsing, scrubbing, or tokenization.
• In the browser, data is processed in a Web Worker sandbox.

Workspace layout

core/
  Cargo.toml              workspace manifest (2 crates)
  crates/
    core/                  bitvanes-core (pure library, no wasm-bindgen)
      src/
        schema.rs          PipelineConfig, ChunkSpec, EmbeddingConfig
        error.rs           BitVanesError
        parse/             markdown.rs, html.rs, text.rs
        scrub.rs           PII redaction + OffsetMap
        tokenize.rs        BPE wrapper (tiktoken-rs)
        chunk.rs           structural-boundary chunker
        arrow_io/          batch.rs, ffi.rs, ipc.rs, csv.rs
        pipeline.rs        full pipeline orchestration (+ run_pipeline_batch)
        embed.rs           Embedder trait (API foundation)
        parse/pdf.rs       native PDF extractor (cli-pdf feature)
    wasm/                  bitvanes-wasm (thin #[wasm_bindgen] wrapper)
      src/lib.rs           process(), release_batch(), array_ptr(), schema_ptr()

Build

# Native (for testing and the CLI)
cargo build --workspace
cargo test --workspace --all-features

# WebAssembly (for the browser)
wasm-pack build crates/wasm --target web --out-dir pkg

Usage from JavaScript

import init, { process, array_ptr, schema_ptr, release_batch, version } from './bitvanes_wasm.js';
await init();

const config = {
  format: "markdown",
  scrub: { patterns: ["email", "ssn"], custom: [] },
  chunk: { max_tokens: 512, overlap_tokens: 0, tokenizer: "cl100k_base" },
  source_label: "doc.md",
};

const slotId = process(config, new Uint8Array(fileBytes));
// Read via arrow-js-ffi using array_ptr(slotId) and schema_ptr(slotId)
// Then:
release_batch(slotId);

Cargo features

Feature	Default	Description
ipc	no	Arrow IPC stream output (StreamWriter) for CLI piping
csv	no	Arrow CSV output for data export
embeddings	no	On-device embedding generation via ONNX Runtime (ort)
parallel	no	Rayon-based parallel batch processing (native only)
cli-pdf	no	Native PDF text extraction via pdf-extract (not in wasm)

Zero-telemetry is unconditional. BPE vocab is embedded at compile time by tiktoken-rs (include_str!, no network code, no feature to disable it), so every build is fully offline. There is intentionally no embed-vocab feature.

Supported formats

Format	Feature	Native only?	Notes
Markdown	—	no	pulldown-cmark, GFM
HTML	—	no	scraper / html5ever
Plain text	—	no	Paragraph-based splitting
JSON	—	no	Structural (object → heading_path)
PDF	cli-pdf	yes	pdf-extract (browser uses PDF.js)
DOCX	office	yes	ZIP + quick-xml, headings/tables/lists
PPTX	office	yes	Slide-by-slide text extraction
XLSX	office	yes	calamine, memory-guarded for large sheets
EPUB	office	yes	OPF spine + HtmlParser per chapter
RTF	office	yes	rtf-parser, heading heuristics

Performance

Lever	Feature	Effect
Parallel regex sweep	parallel	Rayon across PII patterns (~4–8× on 8 patterns)
Parallel embedding	embeddings	Batch ONNX inference across chunks
Memory-mapped I/O	mmap	Zero-copy file reads for >1 MB files
Streaming IPC output	ipc	IpcStream<W> writes batches directly to disk/stdout
Wasm SIMD	(always on wasm)	target-feature=+simd128 via .cargo/config.toml

# Build the CLI with all performance features:
cargo build --release --features "office,mmap,parallel,ipc,csv,cli-pdf"

Output schema

The Arrow RecordBatch emitted by run_pipeline has 11 columns:

#	Column	Type	Nullable	Description
0	chunk_index	UInt32	no	0-based ordinal
1	chunk_id	Utf8	no	Deterministic blake3 hash of chunk content
2	text	Utf8	no	Scrubbed chunk text (ready for vector embedding)
3	token_count	UInt16	no	BPE token count of text
4	source_path	Utf8	no	Source filename/label for lineage
5	heading_path	List<Utf8>	yes	Heading ancestry (H1→H6)
6	section_kind	Dictionary<Int8, Utf8>	no	paragraph, code, heading, table_cell, etc.
7	char_offset_start	UInt32	no	Start offset into scrubbed text
8	char_offset_end	UInt32	no	End offset into scrubbed text
9	pii_metadata	List<Struct>	yes	PII findings overlapping this chunk
10	embedding	FixedSizeList<Float32, 1536>	yes	Dense vector (populated downstream)

pii_metadata struct fields

Each finding in the pii_metadata list is a struct with:

Field	Type	Description
entity	Utf8	Entity slug: email, ssn, credit_card, routing_number, ...
confidence	Float32	Weighted-additive score in [0, 1]
offset_start	Int32	Byte offset into the original (pre-scrub) text
offset_end	Int32	End byte offset (exclusive)
anchors	List<Utf8>	Contextual keywords that fired in the ±N-word window

PII scrubbing engine

Two-tier architecture with weighted-additive confidence scoring:

Tier 1 (always available, no model download):

Pattern	Base	Validator	Notes
email	0.85	—	RFC-5322-ish regex
ssn	0.60	—	US Social Security XXX-XX-XXXX
phone	0.65	—	E.164 (+1…)
credit_card	0.70	Luhn mod-10 gate	Floors at 0.90 on pass; drops on fail
routing_number	0.55	ABA checksum gate	9-digit US bank routing
aws_key	0.95	—	AKIA… prefix
github_pat	0.95	—	ghp_… / gho_… prefix
jwt	0.90	—	Three base64url segments

Each candidate's confidence is boosted by contextual anchor keywords (e.g. "social security", "credit card") found in a configurable ±N-word sliding window (default: 7 words, +0.10 per hit, capped at 0.99).

Tier 2 (stub): the PiiDetector trait + ModelDetector placeholder (gated behind the pii-model feature) provides the plug-in point for a future ONNX NER model. The pipeline contract is unchanged: one finding list, one offset map, one pii_metadata column.

Embeddings (native, embeddings feature)

When the embeddings feature is enabled, the pipeline can generate dense vector embeddings for each chunk using ONNX Runtime:

use bitvanes_core::{Embedder, OrtEmbedder, run_pipeline_with_embeddings};
use std::path::Path;

let embedder = OrtEmbedder::new(
    Path::new("models/model_quantized.onnx"),
    Path::new("models/tokenizer.json"),
    384,   // dimension (MiniLM-L6-v2)
    256,   // max sequence length
)?;

let batch = run_pipeline_with_embeddings(bytes, &config, &embedder)?;
// batch.column("embedding") now has real Float32 vectors.

The model file (e.g. MiniLM-L6-v2 quantized, ~22 MB) is fetched separately and cached. In the browser, use @xenova/transformers or onnxruntime-web for client-side embeddings — the Rust wasm module delegates this to JS.

License

Dual-licensed under MIT OR Apache-2.0.