diff --git a/CHANGELOG.en.md b/CHANGELOG.en.md index e8ab488..29c6c89 100644 --- a/CHANGELOG.en.md +++ b/CHANGELOG.en.md @@ -46,6 +46,12 @@ All notable PocketRoot changes are recorded here. Semantic Versioning begins wit review; `apk-tools` and `pax-utils` have no remaining candidate-material engineering items, six origins still need package-specific material, and legal and redistribution gates remain closed. +- Strict external download-cache input for the RootFS source-review + materializer. A cache replaces network transport only: inputs remain + size-bounded and symlink/overlap-rejected, with pinned SHA-512 and canonical + extracted-aports-tree verification; v2 receipts distinguish network from + cache acquisition instead of fabricating a selected URL, and ambiguous + legacy v1 bundles must be regenerated through cache mode. - arm64 Simulator and unsigned-device final-link gates for the full Experimental graph. - Repository iOS 18 native smoke covering 17 preparation, boot, guest, 8 MiB sustained binary-output, stdout/stderr overflow, command, cancellation, recovery, shutdown, and 256 MiB Simulator lifecycle peak-memory checks. - A repository-external, unapproved RootFS double-build candidate path for the diff --git a/CHANGELOG.md b/CHANGELOG.md index dc3b34f..3a10824 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -41,6 +41,10 @@ PocketRoot 的重要变化记录在这里。首个公开版本发布后遵循 Se 75/75 个候选 payload 已完成 checksum-bound 工程复核，`apk-tools`、 `pax-utils` 的候选材料工程项关闭，另外 6 个 origin 仍需补逐包材料，法律和 再分发门禁保持关闭。 +- RootFS source-review materializer 新增严格仓库外下载缓存输入；缓存只替代网络 + 传输，仍逐项限制大小、拒绝 symlink/重叠路径、核对固定 SHA-512，并重新验证 + 解包后的 canonical aports tree；v2 receipt 会明确区分网络与缓存获取，不伪造 + selected URL，来源含糊的旧 v1 bundle 需通过缓存模式重新生成。 - 加入完整 Experimental graph 的 arm64 Simulator 与 unsigned device final-link gate。 - 加入 repository-owned iOS 18 native smoke App 和 runner，覆盖 17 项 prepare、boot、guest、8 MiB 持续二进制输出、stdout/stderr 超限、command、取消、recovery、shutdown 与 256 MiB Simulator 生命周期峰值内存门禁。 - native smoke 新增仓库外、未授权 RootFS 双构建候选入口：校验候选 provenance、 diff --git a/Compliance/RootFS/v0.3.3/README.md b/Compliance/RootFS/v0.3.3/README.md index ec966da..5f84494 100644 --- a/Compliance/RootFS/v0.3.3/README.md +++ b/Compliance/RootFS/v0.3.3/README.md @@ -104,6 +104,35 @@ ruby Scripts/prepare-rootfs-source-bundle.rb \ --verify /absolute/new/path/outside-the-repository/rootfs-v0.3.3-source-review ``` +网络不可用时，可把另一份 source-review 目录或同布局的只读目录作为下载缓存： + +When the network is unavailable, another source-review directory or a +read-only directory with the same cache layout can supply the downloads: + +```bash +ruby Scripts/prepare-rootfs-source-bundle.rb \ + --download-cache /absolute/existing/rootfs-v0.3.3-source-review \ + --output /absolute/new/path/outside-the-repository/rootfs-v0.3.3-source-review +``` + +缓存布局为 `downloads/aports/.tar.gz` 与 +`distfiles//`。缓存只替代网络传输；工具仍拒绝符号链接和 +仓库内/重叠路径，限制每个输入大小，核对固定 SHA-512，并重新校验解包后的规范化 +aports tree 后才原子提升输出。 + +The cache layout is `downloads/aports/.tar.gz` and +`distfiles//`. It replaces network transport only: +the tool still rejects symlinks and repository-local or overlapping paths, +bounds every input, verifies pinned SHA-512 digests, and revalidates each +canonical extracted aports tree before atomically promoting the output. +The receipt marks cached acquisition explicitly and records cache-relative +paths instead of claiming that an upstream URL was contacted; pinned upstream +origins remain in `SOURCE-ACQUISITION.json`. +Receipt schema v2 requires that explicit acquisition mode. A legacy v1 +source-review directory can be passed to `--download-cache` to regenerate a +verifiable v2 bundle, but it is no longer accepted directly by `--verify` +because its transport provenance is ambiguous. + `--verify` 同时核对所有普通文件摘要、目录集合和符号链接目标。该输出不会被 App、 Git 或 CI artifact 自动打包或上传。 diff --git a/Docs/RootFS.md b/Docs/RootFS.md index ac874e2..fd883e8 100644 --- a/Docs/RootFS.md +++ b/Docs/RootFS.md @@ -98,6 +98,23 @@ ruby Scripts/prepare-rootfs-source-bundle.rb \ --verify /absolute/new/path/outside-the-repository/rootfs-v0.3.3-source-review ``` +已有 source-review 或同布局的仓库外只读缓存可用于离线重建，不降低任何固定校验： + +```bash +ruby Scripts/prepare-rootfs-source-bundle.rb \ + --download-cache /absolute/existing/rootfs-v0.3.3-source-review \ + --output /absolute/new/path/outside-the-repository/rootfs-v0.3.3-source-review +``` + +缓存必须提供 `downloads/aports/.tar.gz` 和 +`distfiles//`。脚本拒绝 symlink、特殊/缺失文件、仓库内 +缓存及输入/输出重叠，逐项限制大小并核对 SHA-512；aports 仍会重新解包并校验规范化 +tree identity。缓存不构成来源批准，也不会改变 receipt 的固定上游来源。 +receipt 会明确标记缓存获取并记录缓存内相对路径，不会伪称访问了某个上游 URL； +固定上游来源仍保留在 `SOURCE-ACQUISITION.json`。 +receipt schema v2 强制包含该模式。旧 v1 source-review 的传输来源存在歧义，不再 +直接通过 `--verify`；仍可把它传给 `--download-cache`，重新生成可验证的 v2 bundle。 + 规范化 aports 目录身份覆盖条目类型、路径、普通文件权限位和内容摘要；物化时会保留 这些权限位，`--verify` 会再次校验。 diff --git a/Docs/en/RootFS.md b/Docs/en/RootFS.md index 99a88ef..33d6c24 100644 --- a/Docs/en/RootFS.md +++ b/Docs/en/RootFS.md @@ -73,6 +73,28 @@ ruby Scripts/prepare-rootfs-source-bundle.rb \ --verify /absolute/new/path/outside-the-repository/rootfs-v0.3.3-source-review ``` +An existing source-review directory, or a read-only external directory with +the same layout, can rebuild the bundle offline without weakening any pin: + +```bash +ruby Scripts/prepare-rootfs-source-bundle.rb \ + --download-cache /absolute/existing/rootfs-v0.3.3-source-review \ + --output /absolute/new/path/outside-the-repository/rootfs-v0.3.3-source-review +``` + +The cache must provide `downloads/aports/.tar.gz` and +`distfiles//`. The script rejects symlinks, +special or missing files, repository-local caches, and input/output overlap; +bounds each input; verifies SHA-512; and re-extracts every aports snapshot to +verify its canonical tree identity. A cache is not source approval and does +not change the receipt's pinned upstream origins. +The receipt marks cached acquisition explicitly and records cache-relative +paths rather than claiming that an upstream URL was contacted; the pinned +upstream origins remain in `SOURCE-ACQUISITION.json`. +Receipt schema v2 requires that explicit mode. A legacy v1 source-review +directory has ambiguous transport provenance and is no longer accepted by +`--verify`; pass it to `--download-cache` to regenerate a verifiable v2 bundle. + After that source-review directory verifies, extract the 21 pinned license/attribution candidates into another directory outside the repository: diff --git a/Scripts/prepare-rootfs-source-bundle.rb b/Scripts/prepare-rootfs-source-bundle.rb index 920f3ac..a00122d 100644 --- a/Scripts/prepare-rootfs-source-bundle.rb +++ b/Scripts/prepare-rootfs-source-bundle.rb @@ -38,6 +38,12 @@ def parse_options commands.on("--output DIR", "New absolute directory outside the repository") do |path| options[:output] = path end + commands.on( + "--download-cache DIR", + "Optional external cache with downloads/aports and distfiles payloads" + ) do |path| + options[:download_cache] = path + end commands.on("--validate-only", "Validate manifest without downloading") do options[:validate_only] = true end @@ -57,6 +63,10 @@ def parse_options raise OptionParser::InvalidOption, "select exactly one of --validate-only, --output, or --verify" end + if options[:download_cache] && !options[:output] + raise OptionParser::InvalidOption, + "--download-cache requires --output" + end options end @@ -101,7 +111,20 @@ def require_bundle_path(root, relative, label, type:) path end -def validate_output(path) +def resolve_external_directory(path, label) + directory = Pathname(path) + raise SourceBundleError, "#{label} must be absolute" unless directory.absolute? + raise SourceBundleError, "#{label} must not be a symlink" if directory.symlink? + raise SourceBundleError, "#{label} is not a directory" unless directory.directory? + + resolved = directory.realpath + if within_path?(resolved, repository_root) + raise SourceBundleError, "#{label} must be outside the repository" + end + resolved +end + +def validate_output(path, inputs = []) output = Pathname(path) raise SourceBundleError, "--output must be absolute" unless output.absolute? raise SourceBundleError, "--output already exists: #{output}" if output.exist? || output.symlink? @@ -113,10 +136,97 @@ def validate_output(path) if within_path?(resolved_output, repository_root) raise SourceBundleError, "--output must be outside the repository" end + inputs.each do |input| + if within_path?(resolved_output, input) || within_path?(input, resolved_output) + raise SourceBundleError, + "--output must not overlap an input directory" + end + end resolved_output end +def copy_cached_download( + cache, + relative, + destination, + expected_sha512, + maximum_bytes +) + source = require_bundle_path( + cache, + relative, + "download cache payload #{relative}", + type: :file + ) + if source.size > maximum_bytes + raise SourceBundleError, + "download cache payload #{relative} exceeds #{maximum_bytes} bytes" + end + + destination.dirname.mkpath + destination.delete if destination.exist? + digest = Digest::SHA512.new + source_lstat = source.lstat + flags = File::RDONLY + flags |= File::NOFOLLOW if File.const_defined?(:NOFOLLOW) + File.open(source, flags) do |input| + input_stat = input.stat + unless input_stat.file? && + input_stat.dev == source_lstat.dev && + input_stat.ino == source_lstat.ino + raise SourceBundleError, + "download cache payload changed while opening: #{relative}" + end + File.open(destination, "wb") do |output| + while (chunk = input.read(64 * 1_024)) + if output.pos + chunk.bytesize > maximum_bytes + raise SourceBundleError, + "download cache payload #{relative} exceeds #{maximum_bytes} bytes" + end + digest.update(chunk) + output.write(chunk) + end + end + end + unless digest.hexdigest == expected_sha512 + raise SourceBundleError, + "download cache payload #{relative} SHA-512 mismatch" + end +rescue StandardError + destination.delete if destination.exist? + raise +end + +def acquire( + urls, + destination, + expected_sha512, + maximum_bytes, + cache: nil, + cache_relative: nil +) + if cache + copy_cached_download( + cache, + cache_relative, + destination, + expected_sha512, + maximum_bytes + ) + return { + "acquisitionMode" => "download-cache", + "cachePath" => cache_relative + } + end + + { + "acquisitionMode" => "network", + "selectedURL" => + download(urls, destination, expected_sha512, maximum_bytes) + } +end + def download(urls, destination, expected_sha512, maximum_bytes) errors = [] urls.each do |url| @@ -474,8 +584,10 @@ def expected_bundle_path_types(sources, identities) end def verify_receipt(receipt, manifest, sources, root, identities) - unless receipt["schemaVersion"] == 1 && + acquisition_mode = receipt["acquisitionMode"] + unless receipt["schemaVersion"] == 2 && receipt["archive"] == manifest["archive"] && + %w[network download-cache].include?(acquisition_mode) && receipt["redistributionApproved"] == false raise SourceBundleError, "materialization receipt has invalid release metadata" end @@ -488,10 +600,20 @@ def verify_receipt(receipt, manifest, sources, root, identities) origin = source.fetch("sourceOrigin") snapshot = source.fetch("aportsSnapshot") recorded = receipt_sources[index] + snapshot_acquisition_valid = + recorded.is_a?(Hash) && + if acquisition_mode == "download-cache" + !recorded.key?("aportsSnapshotURL") && + recorded["aportsCachePath"] == + "downloads/aports/#{origin}.tar.gz" + else + !recorded.key?("aportsCachePath") && + recorded["aportsSnapshotURL"] == snapshot["url"] + end unless recorded.is_a?(Hash) && recorded["sourceOrigin"] == origin && recorded["aportsCommit"] == source["aportsCommit"] && - recorded["aportsSnapshotURL"] == snapshot["url"] && + snapshot_acquisition_valid && recorded["aportsCanonicalTreeFormat"] == manifest["aportsCanonicalTreeFormat"] && recorded["aportsCanonicalTreeSha256"] == @@ -522,10 +644,23 @@ def verify_receipt(receipt, manifest, sources, root, identities) end expected_distfiles.each_with_index do |distfile, offset| distfile_record = recorded_distfiles[offset] + expected_cache_path = + "distfiles/#{origin}/#{distfile.fetch("filename")}" + distfile_acquisition_valid = + distfile_record.is_a?(Hash) && + if acquisition_mode == "download-cache" + !distfile_record.key?("selectedURL") && + distfile_record["cachePath"] == expected_cache_path + else + !distfile_record.key?("cachePath") && + distfile["retrievalURLs"].include?( + distfile_record["selectedURL"] + ) + end unless distfile_record.is_a?(Hash) && distfile_record["filename"] == distfile["filename"] && distfile_record["sha512"] == distfile["sha512"] && - distfile["retrievalURLs"].include?(distfile_record["selectedURL"]) + distfile_acquisition_valid raise SourceBundleError, "materialization receipt has invalid distfile record for " \ "#{origin}/#{distfile["filename"]}" @@ -755,7 +890,7 @@ def notice_text(manifest, source_count, distfile_count) MARKDOWN end -def materialize(manifest, sources, output) +def materialize(manifest, sources, output, download_cache: nil) staging = output.parent.join(".#{output.basename}.tmp-#{Process.pid}-#{SecureRandom.hex(6)}") raise SourceBundleError, "staging path already exists: #{staging}" if staging.exist? @@ -766,11 +901,13 @@ def materialize(manifest, sources, output) origin = entry.fetch("sourceOrigin") snapshot = entry.fetch("aportsSnapshot") snapshot_archive = staging.join("downloads", "aports", "#{origin}.tar.gz") - snapshot_url = download( + snapshot_acquisition = acquire( [snapshot.fetch("url")], snapshot_archive, snapshot.fetch("sha512"), - MAX_SNAPSHOT_DOWNLOAD_BYTES + MAX_SNAPSHOT_DOWNLOAD_BYTES, + cache: download_cache, + cache_relative: "downloads/aports/#{origin}.tar.gz" ) snapshot_symlinks = extract_and_verify_snapshot( snapshot_archive, @@ -795,22 +932,27 @@ def materialize(manifest, sources, output) receipt_distfiles = entry.fetch("distfiles").map do |distfile| destination = staging.join("distfiles", origin, distfile.fetch("filename")) - selected_url = download( + distfile_acquisition = acquire( distfile.fetch("retrievalURLs"), destination, distfile.fetch("sha512"), - MAX_DISTFILE_DOWNLOAD_BYTES + MAX_DISTFILE_DOWNLOAD_BYTES, + cache: download_cache, + cache_relative: + "distfiles/#{origin}/#{distfile.fetch("filename")}" ) { "filename" => distfile.fetch("filename"), - "selectedURL" => selected_url, "sha512" => distfile.fetch("sha512") - } + }.merge( + distfile_acquisition["acquisitionMode"] == "download-cache" ? + {"cachePath" => distfile_acquisition.fetch("cachePath")} : + {"selectedURL" => distfile_acquisition.fetch("selectedURL")} + ) end - receipt_sources << { + receipt_source = { "sourceOrigin" => origin, "aportsCommit" => entry.fetch("aportsCommit"), - "aportsSnapshotURL" => snapshot_url, "aportsCanonicalTreeFormat" => manifest.fetch("aportsCanonicalTreeFormat"), "aportsCanonicalTreeSha256" => snapshot.fetch("canonicalTreeSha256"), @@ -822,6 +964,14 @@ def materialize(manifest, sources, output) end, "distfiles" => receipt_distfiles } + if snapshot_acquisition["acquisitionMode"] == "download-cache" + receipt_source["aportsCachePath"] = + snapshot_acquisition.fetch("cachePath") + else + receipt_source["aportsSnapshotURL"] = + snapshot_acquisition.fetch("selectedURL") + end + receipt_sources << receipt_source end staging.join("SOURCE-ACQUISITION.json").binwrite( @@ -831,8 +981,10 @@ def materialize(manifest, sources, output) notice_text(manifest, sources.length, sources.sum { |entry| entry.fetch("distfiles").length }) ) receipt = { - "schemaVersion" => 1, + "schemaVersion" => 2, "archive" => manifest.fetch("archive"), + "acquisitionMode" => + download_cache ? "download-cache" : "network", "redistributionApproved" => false, "sources" => receipt_sources } @@ -879,8 +1031,23 @@ def materialize(manifest, sources, output) if options.fetch(:validate_only) puts "RootFS source acquisition manifest is valid (#{sources.length} source origins)." else - output = validate_output(options.fetch(:output)) - materialize(manifest, sources, output) + download_cache = + if options[:download_cache] + resolve_external_directory( + options.fetch(:download_cache), + "--download-cache" + ) + end + output = validate_output( + options.fetch(:output), + [download_cache].compact + ) + materialize( + manifest, + sources, + output, + download_cache: download_cache + ) puts "Materialized verified RootFS source-review bundle at #{output}." end rescue SourceBundleError, OptionParser::ParseError, SystemCallError, diff --git a/Tests/Scripts/RootFSSourceBundleTests.rb b/Tests/Scripts/RootFSSourceBundleTests.rb index 485d799..8264798 100644 --- a/Tests/Scripts/RootFSSourceBundleTests.rb +++ b/Tests/Scripts/RootFSSourceBundleTests.rb @@ -68,6 +68,165 @@ def test_validates_and_materializes_external_bundle assert_includes verify_stdout, "Verified materialized RootFS source-review bundle" end + def test_materializes_from_checksum_verified_download_cache + cache = @temporary_directory.join("download-cache") + cache.join("downloads/aports").mkpath + cache.join("distfiles/demo").mkpath + FileUtils.copy_file( + @snapshot, + cache.join("downloads/aports/demo.tar.gz") + ) + FileUtils.copy_file( + @distfile, + cache.join("distfiles/demo/demo-source.txt") + ) + @snapshot.delete + @distfile.delete + output = @temporary_directory.join("cached-output") + + stdout, stderr, status = run_script( + "--download-cache", + cache.to_s, + "--output", + output.to_s + ) + + assert status.success?, "#{stdout}\n#{stderr}" + assert_equal "pkgname=demo\n", + output.join("aports/demo/APKBUILD").binread + assert_equal "demo source\n", + output.join("distfiles/demo/demo-source.txt").binread + receipt = JSON.parse( + output.join("MATERIALIZATION-RECEIPT.json").read + ) + assert_equal "download-cache", receipt.fetch("acquisitionMode") + source_receipt = receipt.fetch("sources").first + assert_equal "downloads/aports/demo.tar.gz", + source_receipt.fetch("aportsCachePath") + refute source_receipt.key?("aportsSnapshotURL") + distfile_receipt = source_receipt.fetch("distfiles").first + assert_equal "distfiles/demo/demo-source.txt", + distfile_receipt.fetch("cachePath") + refute distfile_receipt.key?("selectedURL") + _verify_stdout, verify_stderr, verify_status = + run_script("--verify", output.to_s) + assert verify_status.success?, verify_stderr + + distfile_receipt["selectedURL"] = + JSON.parse(@manifest_path.read) + .fetch("sources").first + .fetch("distfiles").first + .fetch("retrievalURLs").first + output.join("MATERIALIZATION-RECEIPT.json").write( + "#{JSON.pretty_generate(receipt)}\n" + ) + refresh_self_authored_integrity(output) + _tampered_stdout, tampered_stderr, tampered_status = + run_script("--verify", output.to_s) + refute tampered_status.success? + assert_includes tampered_stderr, + "materialization receipt has invalid distfile record" + end + + def test_rejects_download_cache_digest_mismatch_without_output + cache = @temporary_directory.join("download-cache-mismatch") + cache.join("downloads/aports").mkpath + cache.join("distfiles/demo").mkpath + cache.join("downloads/aports/demo.tar.gz").binwrite("wrong\n") + FileUtils.copy_file( + @distfile, + cache.join("distfiles/demo/demo-source.txt") + ) + output = @temporary_directory.join("cache-mismatch-output") + + _stdout, stderr, status = run_script( + "--download-cache", + cache.to_s, + "--output", + output.to_s + ) + + refute status.success? + assert_includes stderr, "SHA-512 mismatch" + refute output.exist? + end + + def test_verify_rejects_legacy_receipt_without_acquisition_mode + output = @temporary_directory.join("legacy-receipt-output") + _stdout, stderr, status = run_script("--output", output.to_s) + assert status.success?, stderr + + receipt_path = output.join("MATERIALIZATION-RECEIPT.json") + receipt = JSON.parse(receipt_path.read) + receipt["schemaVersion"] = 1 + receipt.delete("acquisitionMode") + receipt_path.write("#{JSON.pretty_generate(receipt)}\n") + refresh_self_authored_integrity(output) + + _verify_stdout, verify_stderr, verify_status = + run_script("--verify", output.to_s) + refute verify_status.success? + assert_includes verify_stderr, + "materialization receipt has invalid release metadata" + end + + def test_rejects_symlink_in_download_cache + cache = @temporary_directory.join("download-cache-symlink") + cache.join("downloads/aports").mkpath + cache.join("distfiles/demo").mkpath + cache.join("downloads/aports/demo.tar.gz").make_symlink(@snapshot) + FileUtils.copy_file( + @distfile, + cache.join("distfiles/demo/demo-source.txt") + ) + output = @temporary_directory.join("cache-symlink-output") + + _stdout, stderr, status = run_script( + "--download-cache", + cache.to_s, + "--output", + output.to_s + ) + + refute status.success? + assert_includes stderr, "must not contain a symlink" + refute output.exist? + end + + def test_rejects_repository_local_download_cache + output = @temporary_directory.join("repository-cache-output") + + _stdout, stderr, status = run_script( + "--download-cache", + REPOSITORY_ROOT.to_s, + "--output", + output.to_s + ) + + refute status.success? + assert_includes stderr, + "--download-cache must be outside the repository" + refute output.exist? + end + + def test_rejects_output_nested_in_download_cache + cache = @temporary_directory.join("overlapping-cache") + cache.mkpath + output = cache.join("nested-output") + + _stdout, stderr, status = run_script( + "--download-cache", + cache.to_s, + "--output", + output.to_s + ) + + refute status.success? + assert_includes stderr, + "--output must not overlap an input directory" + refute output.exist? + end + def test_materialization_preserves_and_verifies_executable_mode write_snapshot( @snapshot,