diff --git a/docs/superpowers/plans/2026-07-27-rtx4090-community-benchmark.md b/docs/superpowers/plans/2026-07-27-rtx4090-community-benchmark.md new file mode 100644 index 000000000..058f97f3e --- /dev/null +++ b/docs/superpowers/plans/2026-07-27-rtx4090-community-benchmark.md @@ -0,0 +1,119 @@ +# RTX 4090 Community Benchmark Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Publish a clearly scoped, source-linked summary of the Fun-ASR Nano RTX 4090 community validation on the English and Chinese FunASR benchmark pages. + +**Architecture:** Extend the product-site Jinja benchmark source template with one language-aware section and verify both generated routes. Reuse existing benchmark-card CSS patterns, keep all claims traceable to `sgl-project/sglang-omni#1170`, and clearly distinguish community validation from released upstream support. + +**Tech Stack:** Jinja2 templates, pytest/BeautifulSoup generated-output tests, product-site validator, local HTTP server, Playwright screenshots. + +## Global Constraints + +- Do not claim that the SGLang-Omni integration has merged or shipped. +- Report only measurements published in `sgl-project/sglang-omni#1170`. +- Keep existing navigation and CSS unchanged. +- Preserve English and Chinese content parity. + +--- + +### Task 1: Define the generated-output contract + +**Files:** +- Modify: `web-pages/product-site/tests/test_output.py` + +**Interfaces:** +- Consumes: generated `/benchmarks.html` and `/en/benchmarks.html` +- Produces: a regression test for the RTX 4090 section, evidence links, metrics, and release caveat + +- [ ] **Step 1: Write the failing generated-output test** + +Add `test_rtx4090_community_benchmark_is_bilingual_and_qualified`. For both language routes, require `[data-community-benchmark="rtx4090"]`, `105,067`, `16.11 GiB`, `0.0175`, `0.0164`, the four source URLs, and language-specific text stating the integration is not released. + +- [ ] **Step 2: Run the focused test and verify it fails** + +Run: + +```bash +python -m pytest web-pages/product-site/tests/test_output.py -k rtx4090 -q +``` + +Expected: fail because the section is absent. + +### Task 2: Add the bilingual community benchmark + +**Files:** +- Modify: `web-pages/product-site/templates/benchmarks.html` + +**Interfaces:** +- Consumes: `language`, existing `benchmark-record`, `benchmark-fields`, and `text-link` template styles +- Produces: `[data-community-benchmark="rtx4090"]` in both generated benchmark routes + +- [ ] **Step 1: Add the section and evidence card** + +Insert a new unframed section after the public-record list. Use Jinja language branches for headings and explanatory copy, while keeping model names, metrics, revisions, and URLs identical. + +- [ ] **Step 2: Include complete evidence and boundaries** + +Render hardware/software versions, SeedTTS EN/ZH c=1 and c=16 throughput/latency/quality, the 30-minute soak, memory, 30-second request limit, terminal-event-only behavior, dirty-worktree provenance, and pending upstream integration. + +- [ ] **Step 3: Run the focused test** + +```bash +python -m pytest web-pages/product-site/tests/test_output.py -k rtx4090 -q +``` + +Expected: pass. + +### Task 3: Validate and publish the isolated branch + +**Files:** +- Test: `web-pages/product-site/templates/benchmarks.html` +- Test: `web-pages/product-site/tests/test_output.py` + +**Interfaces:** +- Consumes: product-site source and tests from Tasks 1-2 +- Produces: a pushed `codex/rtx4090-benchmark-20260727` branch and reviewable pull request + +- [ ] **Step 1: Run the product-site suite** + +```bash +python -m pytest web-pages/product-site/tests -q +``` + +Expected: all tests pass. + +- [ ] **Step 2: Build and validate generated output** + +Run two builds, compare them recursively, and run `validate.py` on the output. Request the four linked GitHub/Gist sources and require HTTP status below 400. + +- [ ] **Step 3: Inspect rendered pages** + +Serve the generated output with: + +```bash +python3 -m http.server 8765 +``` + +Capture desktop `1440x1000` and mobile `390x844` screenshots of both pages. Confirm the new evidence card does not overflow and text does not overlap. + +- [ ] **Step 4: Run repository checks** + +Run: + +```bash +git diff --check origin/main...HEAD +git status --short +``` + +Expected: no whitespace errors and only the intended files changed. + +- [ ] **Step 5: Commit and publish** + +```bash +git add web-pages/product-site/templates/benchmarks.html web-pages/product-site/tests/test_output.py +git commit -m "docs: add RTX 4090 Fun-ASR benchmark" +git push -u origin codex/rtx4090-benchmark-20260727 +``` + +Open a ready pull request against `main` with the validation evidence and explicit community/unmerged caveat. diff --git a/docs/superpowers/specs/2026-07-27-rtx4090-community-benchmark-design.md b/docs/superpowers/specs/2026-07-27-rtx4090-community-benchmark-design.md new file mode 100644 index 000000000..50d03a4aa --- /dev/null +++ b/docs/superpowers/specs/2026-07-27-rtx4090-community-benchmark-design.md @@ -0,0 +1,40 @@ +# RTX 4090 Community Benchmark Design + +## Goal + +Add a trustworthy, production-oriented summary of the completed Fun-ASR Nano single-RTX-4090 community validation to the FunASR benchmark pages without presenting unmerged SGLang-Omni integration work as an official supported runtime. + +## Scope + +- Update the product-site benchmark source template on `main`; do not edit generated `gh-pages` output. +- Render one self-contained community-validation section in both English and Chinese. +- Link the upstream benchmark issue, integration roadmap, draft PR, and raw JSON artifacts. +- Add a focused generated-output regression test. +- Leave navigation, styling, deployment registry, and all other pages unchanged. + +## Content + +The new section will identify the environment as a single RTX 4090 24 GB and report only measurements published in `sgl-project/sglang-omni#1170`: + +- SeedTTS English and Chinese throughput, latency, and WER/CER at concurrency 1 and 16. +- The 30-minute mixed-concurrency soak result of 105,067 successful requests and zero unexpected errors. +- Startup and post-graph-capture memory. +- Operational constraints: 30-second request limit, no low-latency delta claim, dirty feature worktree provenance, and pending upstream integration. + +The section will call this a community validation and will not claim support in a released SGLang-Omni version. + +## Presentation + +Use the product site's existing `benchmark-record` and `benchmark-fields` components so the data remains readable without horizontal scrolling on mobile. No new CSS or visual asset is needed because this is an evidence card inside the existing benchmark experience, not a new page or hero. + +## Verification + +- Run the product-site pytest suite, including a focused assertion for both generated language routes. +- Run the deterministic product-site build and validator. +- Check the source URLs return successful HTTP responses. +- Run `git diff --check`. +- Serve the generated site and inspect desktop and mobile screenshots for overflow or overlap. + +## Rollback + +The work is isolated on `codex/rtx4090-benchmark-20260727` from recorded `main` commit `1e0200916488bc749565aaf67818d056efbc57e8`. The generated-page experiment and original pages are retained in the patrol evidence directory and an unpushed `gh-pages` backup branch. diff --git a/web-pages/product-site/templates/benchmarks.html b/web-pages/product-site/templates/benchmarks.html index fc5e78ef5..85365d9b8 100644 --- a/web-pages/product-site/templates/benchmarks.html +++ b/web-pages/product-site/templates/benchmarks.html @@ -52,6 +52,39 @@

{{ '带完整限定条件阅读每一个数字' if language == 'zh' else 'Re +
+
+

{{ '消费级 GPU 社区验证' if language == 'zh' else 'Consumer GPU community validation' }}

+

{{ '单张 RTX 4090 上的 Fun-ASR Nano' if language == 'zh' else 'Fun-ASR Nano on one RTX 4090' }}

+

{{ 'SGLang-Omni 社区完成了 SeedTTS 英文、中文全量测试和 30 分钟稳定性测试。以下结果对应被测 feature branch，不代表 SGLang-Omni 已发布该集成。' if language == 'zh' else 'The SGLang-Omni community completed full SeedTTS English and Chinese sweeps plus a 30-minute stability run. These results describe the tested feature branch, not a released SGLang-Omni integration.' }}

+
+
+
+
+

SGLang-Omni 0.1.0

Fun-ASR Nano

+ c=16: 114.26 EN / 127.25 ZH samples/s +
+
+
{{ '硬件' if language == 'zh' else 'Hardware' }}
RTX 4090 24,564 MiB; Ryzen 9 7950X; 125 GiB RAM
+
{{ '软件' if language == 'zh' else 'Software' }}
CUDA 13.0; PyTorch 2.11.0; Transformers 5.6.0; Fun-ASR 854d88f
+
SeedTTS EN
1,088/1,088; c=1: 20.16 samples/s, 0.050 s; c=16: 114.26 samples/s, 0.139 s; WER 0.0175
+
SeedTTS ZH
2,020/2,020; c=1: 14.36 samples/s, 0.071 s; c=16: 127.25 samples/s, 0.125 s; CER 0.0164
+
{{ '稳定性' if language == 'zh' else 'Stability' }}
{{ '30 分钟混合 c=1/4/8/16；105,067 次成功请求；0 个意外错误' if language == 'zh' else '30-minute mixed c=1/4/8/16 soak; 105,067 successful requests; 0 unexpected errors' }}
+
{{ '显存' if language == 'zh' else 'Memory' }}
{{ '静态分配后 15.98 GiB；CUDA Graph 捕获后 16.11 GiB；退出后显存释放' if language == 'zh' else '15.98 GiB after static allocation; 16.11 GiB after graph capture; GPU memory released on exit' }}
+
{{ '测量口径' if language == 'zh' else 'Timing scope' }}
{{ '预热后三次测量；完整 SeedTTS sweep；无跳过样本' if language == 'zh' else 'Three measured repeats after warmup; complete SeedTTS sweeps; zero skipped clips' }}
+
{{ '请求边界' if language == 'zh' else 'Request boundary' }}
{{ '每次请求最长 30 秒；只观察到最终转写事件' if language == 'zh' else 'Maximum 30 seconds per request; only terminal transcript events were observed' }}
+
+

{{ '限定：' if language == 'zh' else 'Qualification: ' }}{{ '结果来自有未提交改动的 feature worktree，上游集成仍在推进；不能据此声称已发布支持或具备低延迟增量输出。' if language == 'zh' else 'Results came from a dirty feature worktree while upstream integration remained in progress; they do not establish released support or low-latency streaming deltas.' }}

+ +
+
+
+

{{ '实时服务' if language == 'zh' else 'Realtime services' }}

diff --git a/web-pages/product-site/tests/test_output.py b/web-pages/product-site/tests/test_output.py index 404ccfc26..7367b7716 100644 --- a/web-pages/product-site/tests/test_output.py +++ b/web-pages/product-site/tests/test_output.py @@ -84,6 +84,33 @@ def test_benchmark_page_warns_against_cross_profile_claims(built_site): assert 'RTFx' in warning.get_text() +@pytest.mark.parametrize( + ('relative', 'release_caveat'), + ( + ('benchmarks.html', '不代表 SGLang-Omni 已发布该集成'), + ('en/benchmarks.html', 'not a released SGLang-Omni integration'), + ), +) +def test_rtx4090_community_benchmark_is_bilingual_and_qualified( + built_site, relative, release_caveat +): + soup = read_soup(built_site / relative) + section = soup.select_one('[data-community-benchmark="rtx4090"]') + + assert section + text = section.get_text(' ', strip=True) + for marker in ('105,067', '16.11 GiB', '0.0175', '0.0164', release_caveat): + assert marker in text + + links = {link.get('href') for link in section.select('a[href]')} + assert { + 'https://github.com/sgl-project/sglang-omni/issues/1170', + 'https://github.com/sgl-project/sglang-omni/issues/1120', + 'https://github.com/sgl-project/sglang-omni/pull/1171', + 'https://gist.github.com/wirybeaver/ffa8a07f89066654a271a45b21592d25', + } <= links + + def test_manifest_contains_all_detail_and_benchmark_routes(built_site): manifest = __import__('json').loads( (built_site / 'deployment-manifest.json').read_text(encoding='utf-8')