Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
119 changes: 119 additions & 0 deletions docs/superpowers/plans/2026-07-27-rtx4090-community-benchmark.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,119 @@
# RTX 4090 Community Benchmark Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Publish a clearly scoped, source-linked summary of the Fun-ASR Nano RTX 4090 community validation on the English and Chinese FunASR benchmark pages.

**Architecture:** Extend the product-site Jinja benchmark source template with one language-aware section and verify both generated routes. Reuse existing benchmark-card CSS patterns, keep all claims traceable to `sgl-project/sglang-omni#1170`, and clearly distinguish community validation from released upstream support.

**Tech Stack:** Jinja2 templates, pytest/BeautifulSoup generated-output tests, product-site validator, local HTTP server, Playwright screenshots.

## Global Constraints

- Do not claim that the SGLang-Omni integration has merged or shipped.
- Report only measurements published in `sgl-project/sglang-omni#1170`.
- Keep existing navigation and CSS unchanged.
- Preserve English and Chinese content parity.

---

### Task 1: Define the generated-output contract

**Files:**
- Modify: `web-pages/product-site/tests/test_output.py`

**Interfaces:**
- Consumes: generated `/benchmarks.html` and `/en/benchmarks.html`
- Produces: a regression test for the RTX 4090 section, evidence links, metrics, and release caveat

- [ ] **Step 1: Write the failing generated-output test**

Add `test_rtx4090_community_benchmark_is_bilingual_and_qualified`. For both language routes, require `[data-community-benchmark="rtx4090"]`, `105,067`, `16.11 GiB`, `0.0175`, `0.0164`, the four source URLs, and language-specific text stating the integration is not released.

- [ ] **Step 2: Run the focused test and verify it fails**

Run:

```bash
python -m pytest web-pages/product-site/tests/test_output.py -k rtx4090 -q
```

Expected: fail because the section is absent.

### Task 2: Add the bilingual community benchmark

**Files:**
- Modify: `web-pages/product-site/templates/benchmarks.html`

**Interfaces:**
- Consumes: `language`, existing `benchmark-record`, `benchmark-fields`, and `text-link` template styles
- Produces: `[data-community-benchmark="rtx4090"]` in both generated benchmark routes

- [ ] **Step 1: Add the section and evidence card**

Insert a new unframed section after the public-record list. Use Jinja language branches for headings and explanatory copy, while keeping model names, metrics, revisions, and URLs identical.

- [ ] **Step 2: Include complete evidence and boundaries**

Render hardware/software versions, SeedTTS EN/ZH c=1 and c=16 throughput/latency/quality, the 30-minute soak, memory, 30-second request limit, terminal-event-only behavior, dirty-worktree provenance, and pending upstream integration.

- [ ] **Step 3: Run the focused test**

```bash
python -m pytest web-pages/product-site/tests/test_output.py -k rtx4090 -q
```

Expected: pass.

### Task 3: Validate and publish the isolated branch

**Files:**
- Test: `web-pages/product-site/templates/benchmarks.html`
- Test: `web-pages/product-site/tests/test_output.py`

**Interfaces:**
- Consumes: product-site source and tests from Tasks 1-2
- Produces: a pushed `codex/rtx4090-benchmark-20260727` branch and reviewable pull request

- [ ] **Step 1: Run the product-site suite**

```bash
python -m pytest web-pages/product-site/tests -q
```

Expected: all tests pass.

- [ ] **Step 2: Build and validate generated output**

Run two builds, compare them recursively, and run `validate.py` on the output. Request the four linked GitHub/Gist sources and require HTTP status below 400.

- [ ] **Step 3: Inspect rendered pages**

Serve the generated output with:

```bash
python3 -m http.server 8765
```

Capture desktop `1440x1000` and mobile `390x844` screenshots of both pages. Confirm the new evidence card does not overflow and text does not overlap.

- [ ] **Step 4: Run repository checks**

Run:

```bash
git diff --check origin/main...HEAD
git status --short
```

Expected: no whitespace errors and only the intended files changed.

- [ ] **Step 5: Commit and publish**

```bash
git add web-pages/product-site/templates/benchmarks.html web-pages/product-site/tests/test_output.py
git commit -m "docs: add RTX 4090 Fun-ASR benchmark"
git push -u origin codex/rtx4090-benchmark-20260727
```

Open a ready pull request against `main` with the validation evidence and explicit community/unmerged caveat.
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
# RTX 4090 Community Benchmark Design

## Goal

Add a trustworthy, production-oriented summary of the completed Fun-ASR Nano single-RTX-4090 community validation to the FunASR benchmark pages without presenting unmerged SGLang-Omni integration work as an official supported runtime.

## Scope

- Update the product-site benchmark source template on `main`; do not edit generated `gh-pages` output.
- Render one self-contained community-validation section in both English and Chinese.
- Link the upstream benchmark issue, integration roadmap, draft PR, and raw JSON artifacts.
- Add a focused generated-output regression test.
- Leave navigation, styling, deployment registry, and all other pages unchanged.

## Content

The new section will identify the environment as a single RTX 4090 24 GB and report only measurements published in `sgl-project/sglang-omni#1170`:

- SeedTTS English and Chinese throughput, latency, and WER/CER at concurrency 1 and 16.
- The 30-minute mixed-concurrency soak result of 105,067 successful requests and zero unexpected errors.
- Startup and post-graph-capture memory.
- Operational constraints: 30-second request limit, no low-latency delta claim, dirty feature worktree provenance, and pending upstream integration.

The section will call this a community validation and will not claim support in a released SGLang-Omni version.

## Presentation

Use the product site's existing `benchmark-record` and `benchmark-fields` components so the data remains readable without horizontal scrolling on mobile. No new CSS or visual asset is needed because this is an evidence card inside the existing benchmark experience, not a new page or hero.

## Verification

- Run the product-site pytest suite, including a focused assertion for both generated language routes.
- Run the deterministic product-site build and validator.
- Check the source URLs return successful HTTP responses.
- Run `git diff --check`.
- Serve the generated site and inspect desktop and mobile screenshots for overflow or overlap.

## Rollback

The work is isolated on `codex/rtx4090-benchmark-20260727` from recorded `main` commit `1e0200916488bc749565aaf67818d056efbc57e8`. The generated-page experiment and original pages are retained in the patrol evidence directory and an unpushed `gh-pages` backup branch.
33 changes: 33 additions & 0 deletions web-pages/product-site/templates/benchmarks.html
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,39 @@ <h2>{{ '带完整限定条件阅读每一个数字' if language == 'zh' else 'Re
</div>
</section>

<section class="section benchmark-results" data-community-benchmark="rtx4090" aria-label="{{ 'RTX 4090 社区验证' if language == 'zh' else 'RTX 4090 community validation' }}">
<div class="section-heading compact-heading">
<p class="eyebrow">{{ '消费级 GPU 社区验证' if language == 'zh' else 'Consumer GPU community validation' }}</p>
<h2>{{ '单张 RTX 4090 上的 Fun-ASR Nano' if language == 'zh' else 'Fun-ASR Nano on one RTX 4090' }}</h2>
<p>{{ 'SGLang-Omni 社区完成了 SeedTTS 英文、中文全量测试和 30 分钟稳定性测试。以下结果对应被测 feature branch,不代表 SGLang-Omni 已发布该集成。' if language == 'zh' else 'The SGLang-Omni community completed full SeedTTS English and Chinese sweeps plus a 30-minute stability run. These results describe the tested feature branch, not a released SGLang-Omni integration.' }}</p>
</div>
<div class="benchmark-list">
<article class="benchmark-record">
<div class="benchmark-record-heading">
<div><p>SGLang-Omni 0.1.0</p><h3>Fun-ASR Nano</h3></div>
<strong class="benchmark-result">c=16: 114.26 EN / 127.25 ZH samples/s</strong>
</div>
<dl class="benchmark-fields">
<div><dt>{{ '硬件' if language == 'zh' else 'Hardware' }}</dt><dd>RTX 4090 24,564 MiB; Ryzen 9 7950X; 125 GiB RAM</dd></div>
<div><dt>{{ '软件' if language == 'zh' else 'Software' }}</dt><dd>CUDA 13.0; PyTorch 2.11.0; Transformers 5.6.0; Fun-ASR <code>854d88f</code></dd></div>
<div><dt>SeedTTS EN</dt><dd>1,088/1,088; c=1: 20.16 samples/s, 0.050 s; c=16: 114.26 samples/s, 0.139 s; WER 0.0175</dd></div>
<div><dt>SeedTTS ZH</dt><dd>2,020/2,020; c=1: 14.36 samples/s, 0.071 s; c=16: 127.25 samples/s, 0.125 s; CER 0.0164</dd></div>
<div><dt>{{ '稳定性' if language == 'zh' else 'Stability' }}</dt><dd>{{ '30 分钟混合 c=1/4/8/16;105,067 次成功请求;0 个意外错误' if language == 'zh' else '30-minute mixed c=1/4/8/16 soak; 105,067 successful requests; 0 unexpected errors' }}</dd></div>
<div><dt>{{ '显存' if language == 'zh' else 'Memory' }}</dt><dd>{{ '静态分配后 15.98 GiB;CUDA Graph 捕获后 16.11 GiB;退出后显存释放' if language == 'zh' else '15.98 GiB after static allocation; 16.11 GiB after graph capture; GPU memory released on exit' }}</dd></div>
<div><dt>{{ '测量口径' if language == 'zh' else 'Timing scope' }}</dt><dd>{{ '预热后三次测量;完整 SeedTTS sweep;无跳过样本' if language == 'zh' else 'Three measured repeats after warmup; complete SeedTTS sweeps; zero skipped clips' }}</dd></div>
<div><dt>{{ '请求边界' if language == 'zh' else 'Request boundary' }}</dt><dd>{{ '每次请求最长 30 秒;只观察到最终转写事件' if language == 'zh' else 'Maximum 30 seconds per request; only terminal transcript events were observed' }}</dd></div>
</dl>
<p class="benchmark-qualification"><strong>{{ '限定:' if language == 'zh' else 'Qualification: ' }}</strong>{{ '结果来自有未提交改动的 feature worktree,上游集成仍在推进;不能据此声称已发布支持或具备低延迟增量输出。' if language == 'zh' else 'Results came from a dirty feature worktree while upstream integration remained in progress; they do not establish released support or low-latency streaming deltas.' }}</p>
<ul class="evidence-list">
<li><a href="https://github.com/sgl-project/sglang-omni/issues/1170">{{ '完整评测与环境' if language == 'zh' else 'Full benchmark and environment' }}</a></li>
<li><a href="https://github.com/sgl-project/sglang-omni/issues/1120">{{ '消费级 GPU 路线图' if language == 'zh' else 'Consumer GPU roadmap' }}</a></li>
<li><a href="https://github.com/sgl-project/sglang-omni/pull/1171">{{ '集成草稿 PR' if language == 'zh' else 'Draft integration PR' }}</a></li>
<li><a href="https://gist.github.com/wirybeaver/ffa8a07f89066654a271a45b21592d25">{{ '原始 JSON 数据' if language == 'zh' else 'Raw JSON artifacts' }}</a></li>
</ul>
</article>
</div>
</section>

<section class="section realtime-method">
<div class="section-heading compact-heading">
<p class="eyebrow">{{ '实时服务' if language == 'zh' else 'Realtime services' }}</p>
Expand Down
27 changes: 27 additions & 0 deletions web-pages/product-site/tests/test_output.py
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,33 @@ def test_benchmark_page_warns_against_cross_profile_claims(built_site):
assert 'RTFx' in warning.get_text()


@pytest.mark.parametrize(
('relative', 'release_caveat'),
(
('benchmarks.html', '不代表 SGLang-Omni 已发布该集成'),
('en/benchmarks.html', 'not a released SGLang-Omni integration'),
),
)
def test_rtx4090_community_benchmark_is_bilingual_and_qualified(
built_site, relative, release_caveat
):
soup = read_soup(built_site / relative)
section = soup.select_one('[data-community-benchmark="rtx4090"]')

assert section
text = section.get_text(' ', strip=True)
for marker in ('105,067', '16.11 GiB', '0.0175', '0.0164', release_caveat):
assert marker in text

links = {link.get('href') for link in section.select('a[href]')}
assert {
'https://github.com/sgl-project/sglang-omni/issues/1170',
'https://github.com/sgl-project/sglang-omni/issues/1120',
'https://github.com/sgl-project/sglang-omni/pull/1171',
'https://gist.github.com/wirybeaver/ffa8a07f89066654a271a45b21592d25',
} <= links


def test_manifest_contains_all_detail_and_benchmark_routes(built_site):
manifest = __import__('json').loads(
(built_site / 'deployment-manifest.json').read_text(encoding='utf-8')
Expand Down
Loading