AkagiCast

Real-time voice-driven VRM avatar — a lightweight VTuber application that captures microphone audio, extracts speech features, and drives a 3D VRM model's mouth, eyes, and body with natural idle animations.

Features

• Real-time lip sync — Microphone audio is analyzed via FFT and mapped to Japanese vowel visemes (A/I/U/E/O), driving VRM BlendShape morph targets.
• Natural idle animation — Breathing, head micro-movements, body sway, arm droop, and procedurally generated eye blinks.
• VRM 0.x / 1.0 support — Parses GLB binary extensions to discover BlendShape groups (including Japanese aliases like あ/い/う/え/お and まばたき).
• Transparent window mode — Runs with a transparent background by default for streaming/overlay use.
• Multi-device audio — Supports F32, I16, and U16 sample formats via cpal.
• Frame-rate independent smoothing — Exponential smoothing ensures natural mouth transitions regardless of frame rate.

Architecture

┌─────────────────────────────────────────────────────────────┐
│                        akagicast (bin)                       │
│  App entry, scene setup, audio polling, DSP scheduling       │
├──────────────┬──────────────────┬────────────────────────────┤
│ akagicast-   │ akagicast-dsp    │ akagicast-avatar           │
│ audio        │                  │                            │
│              │ RMS energy       │ VRM BlendShape parsing     │
│ cpal capture │ Hann-window FFT  │ Viseme morph application   │
│ Ring buffer  │ Spectral centroid│ Idle body animation        │
│ Crossbeam TX │ Band energies    │ Procedural blink system    │
│              │ Viseme mapping   │ (Bevy Plugin)              │
│              │ Exp. smoothing   │                            │
├──────────────┴──────────────────┴────────────────────────────┤
│                     akagicast-core                           │
│  Shared types: AudioChunk, VoiceVisemes, DspConfig, Errors  │
└─────────────────────────────────────────────────────────────┘

Prerequisites

• Rust (edition 2024, stable toolchain)
• Platform audio backend:
  • macOS: CoreAudio (built-in)
  • Linux: ALSA (libasound2-dev on Debian/Ubuntu, alsa-lib-devel on Fedora)
  • Windows: WASAPI (built-in)
• A VRM model placed in assets/ (tested with VRM 0.x; see assets/ for included sample models)

Quick Start

# Clone the repository
git clone <repo-url>
cd AkagiCast

# Place a .vrm model in assets/ (sample models are already included)
# Build and run
cargo run --bin akagicast --release

CLI Flags

Flag	Description
-d, --debug	Enable opaque window with debug overlay (shows "Press ESC to quit")

Controls

Key	Action
Esc	Quit the application

Adding Your Own VRM Model

1. Drop your .vrm file into the assets/ directory.
2. The application auto-detects VRM files in this priority order: suzuha.vrm, alicia.vrm, avatar.vrm, catbot.vrm, cool_loops.vrm, then any other *.vrm.
3. To use a specific model, rename it to one of the priority names or modify the candidate list in main.rs.

Building

# Debug build
cargo build

# Release build (recommended for production — LTO enabled)
cargo build --release

# Run tests
cargo test

The release profile uses thin LTO for optimized binary size and performance.

Project Structure

AkagiCast/
├── Cargo.toml              # Workspace manifest
├── assets/                 # VRM model files (.vrm)
├── bin/
│   └── akagicast/          # Main binary — application entry point
│       └── src/main.rs
└── crates/
    ├── akagicast-core/     # Shared types and error definitions
    ├── akagicast-audio/    # Audio capture (cpal) + lock-free ring buffer
    ├── akagicast-dsp/      # Digital signal processing — FFT, viseme extraction, smoothing
    └── akagicast-avatar/   # VRM loading, BlendShape driving, idle animation (Bevy Plugin)

Crate Responsibilities

• akagicast-core (lib.rs) — Defines AudioChunk, VoiceVisemes, DspConfig, and VtuberError. Has zero Bevy or audio dependencies; usable in any context.
• akagicast-audio (lib.rs) — Wraps cpal to capture mono-mixed microphone audio at 25ms intervals, sends chunks through a bounded ring buffer + crossbeam_channel. Handles F32/I16/U16 sample formats.
• akagicast-dsp (lib.rs) — Applies a Hann window, runs rustfft to compute magnitude spectra, extracts spectral centroid and band energies, maps features to A/I/U/E/O viseme weights, and applies exponential smoothing.
• akagicast-avatar (lib.rs) — Bevy Plugin that parses GLB JSON to discover morph target indices for viseme presets and blink, then drives MorphWeights each frame. Also provides procedural idle body animation (breathing, head sway, natural arm pose, blinking).
• akagicast (binary) (main.rs) — Wires everything together: sets up the Bevy app with camera/lighting, starts audio capture, polls audio on each frame, runs DSP, and feeds smoothed visemes into the avatar plugin.

How Lip Sync Works

1. Capture: cpal captures microphone audio at the device's native sample rate, downmixes to mono, and buffers it in a ring buffer.
2. Chunking: A dedicated thread pulls 25ms chunks from the ring buffer and sends them via crossbeam_channel.
3. Feature extraction (per frame in Bevy's Update schedule):
   • Compute RMS energy → noise gate → normalized mouth openness.
   • Apply Hann window → FFT (1024-point) → magnitude spectrum.
   • Compute spectral centroid (brightness) and low/mid/high band energy ratios.
   • Map centroid + band energies to A/I/U/E/O vowel weights via Gaussian scoring.
4. Smoothing: Exponential interpolation (lerp_speed = 15.0) prevents jitter; frame-rate independent via 1 - exp(-speed * dt).
5. Application: Smoothed visemes are written to the avatar's CurrentVisemes resource, which the avatar systems read to set BlendShape weights on the VRM face mesh.

License

MIT OR Apache-2.0