feat: metrics integration for commit and scan

[evindj]

This change implements part 2 of the core metrics. it covers RestCatalog spec, integration with commit and scan workflows. It leaves the logging metrics reporter out of scope, it will be added in a subsequent diff.
some other differences is that some of the http calls for the RestCatalogMetrics reporter is still synchronous a subsequent change will cover that.

[Copilot]

Pull request overview

Adds core metrics reporting to Iceberg C++ commit and scan workflows, including wiring metrics reporters through catalogs/tables and emitting REST-spec-compatible reports.

Changes:

• Emit CommitReport from Transaction::Commit() when a commit produces a new snapshot, and propagate table reporters into update operations.
• Collect and emit ScanReport (with scan planning counters/timers) from DataTableScan::PlanFiles(), including new manifest/file skip counters.
• Introduce RestMetricsReporter and catalog-level wiring (REST/SQL/InMemory) plus new end-to-end/unit tests for scan planning and REST reporter behavior.

Reviewed changes

Copilot reviewed 28 out of 29 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
src/iceberg/update/snapshot_update.h	Adds per-snapshot-update reporter plumbing (ReportWith) and stores a reporter pointer.
src/iceberg/transaction.cc	Adds commit timing/attempt metrics and emits CommitReport after successful snapshot-producing commits; propagates reporter to FastAppend in transactions.
src/iceberg/test/scan_planning_metrics_test.cc	New end-to-end tests validating ScanReport emission and scan planning counters.
src/iceberg/test/rest_metrics_reporter_test.cc	New tests asserting REST reporter suppresses HTTP failures.
src/iceberg/test/fast_append_test.cc	Adds integration tests verifying CommitReport emission behavior for snapshot vs property-only commits.
src/iceberg/test/CMakeLists.txt	Registers the new test sources in the build.
src/iceberg/table.h	Extends Table::Make/StagedTable::Make to accept an optional reporter; adds reporter accessors and combining API.
src/iceberg/table.cc	Stores reporter on tables; wires reporter + table name into scans and propagates reporter to FastAppend.
src/iceberg/table_scan.h	Extends scan context and builder API to include table name + metrics reporter.
src/iceberg/table_scan.cc	Collects scan planning metrics, attaches them to manifest planning, and emits ScanReport.
src/iceberg/manifest/manifest_reader.h	Adds SkipCounter() hook to count per-entry skips during manifest reading.
src/iceberg/manifest/manifest_reader.cc	Implements skip counting for partition/metrics/partition-set filtering.
src/iceberg/manifest/manifest_reader_internal.h	Stores skip_counter_ in the reader implementation.
src/iceberg/manifest/manifest_group.h	Adds ability to attach scan metrics to manifest planning.
src/iceberg/manifest/manifest_group.cc	Increments scan counters for manifests/files and delete-file result accounting; wires reader skip counter.
src/iceberg/delete_file_index.h	Adds optional scan metrics hook (non-owning) for delete-manifest accounting.
src/iceberg/delete_file_index.cc	Counts scanned/skipped delete manifests and skipped delete files while building the delete index.
src/iceberg/catalog/sql/sql_catalog.h	Stores a catalog-level reporter for tables loaded/created by SQL catalog.
src/iceberg/catalog/sql/sql_catalog.cc	Loads configured reporter and passes it into created/loaded tables.
src/iceberg/catalog/rest/rest_metrics_reporter.h	Introduces REST metrics reporter interface for POSTing reports to the REST endpoint.
src/iceberg/catalog/rest/rest_metrics_reporter.cc	Implements JSON serialization + POST and suppresses failures (fire-and-forget).
src/iceberg/catalog/rest/rest_catalog.h	Switches REST catalog HTTP client ownership to shared_ptr; adds per-table reporter builder.
src/iceberg/catalog/rest/rest_catalog.cc	Loads configured reporter and optionally combines it with a per-table REST reporter when enabled/supported.
src/iceberg/catalog/rest/meson.build	Adds new REST metrics reporter source to Meson build.
src/iceberg/catalog/rest/CMakeLists.txt	Adds new REST metrics reporter source to CMake build.
src/iceberg/catalog/rest/catalog_properties.h	Adds rest-metrics-reporting-enabled property.
src/iceberg/catalog/memory/in_memory_catalog.h	Stores a catalog-level reporter for in-memory catalog tables.
src/iceberg/catalog/memory/in_memory_catalog.cc	Loads configured reporter and passes it into created/loaded tables.
.gitignore	Ignores additional build output directories.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 46 out of 47 changed files in this pull request and generated 4 comments.

[Copilot]

Pull request overview

Copilot reviewed 46 out of 47 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (2)

src/iceberg/expression/sanitize_expression.cc:107

• The timestamp/time regexes use an unescaped '.' in the fractional-seconds groups (e.g. (.�d{1,9})), which matches any character rather than a literal dot. That can misclassify non-temporal strings as time/timestamp patterns and bucket them incorrectly.

  static const std::regex kTime(R"(\d{2}:\d{2}(:\d{2}(.\d{1,9})?)?)");
  static const std::regex kTimestamp(
      R"(\d{4}-\d{2}-\d{2}T\d{2}:\d{2}(:\d{2}(.\d{1,9})?)?)");
  static const std::regex kTimestampTz(
      R"(\d{4}-\d{2}-\d{2}T\d{2}:\d{2}(:\d{2}(.\d{1,9})?)?([-+]\d{2}:\d{2}|Z))");

src/iceberg/table_scan.cc:612

• When context_.metrics_reporter is set, ScanReport.table_name is taken from context_.table_name without validation. If a caller sets MetricsReporter() but forgets TableName(), reports will be emitted with an empty required field, which can violate the REST metrics spec / make reports unusable.

  if (context_.metrics_reporter) {
    ICEBERG_ASSIGN_OR_RAISE(auto projected_schema, ResolveProjectedSchema());
    const auto& schema_ptr = projected_schema.get();

[Copilot]

Pull request overview

Copilot reviewed 46 out of 47 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

src/iceberg/table_scan.cc:603

• ScanMetrics is always attached to ManifestGroup, even when no metrics_reporter is configured. This causes ManifestGroup/DeleteFileIndex to execute per-file/per-delete metrics bookkeeping (including looping delete_files to sum sizes) on every scan, which is a significant overhead when metrics are disabled.

  manifest_group->CaseSensitive(context_.case_sensitive)
      .Select(ScanColumns())
      .FilterData(filter())
      .IgnoreDeleted()
      .ColumnsToKeepStats(context_.columns_to_keep_stats)
      .PlanWith(context_.plan_executor)
      .ScanMetrics(scan_metrics);
  if (context_.ignore_residuals) {

[evindj]

possible merge conflicts stuff I will have a look.

[wgtmac]

I haven't finished reading this PR yet. Codex published my pending comments by accident. Will let you know when I'm done.

[Copilot]

Pull request overview

Copilot reviewed 58 out of 58 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 58 out of 58 changed files in this pull request and generated no new comments.

Suppressed comments (3)

src/iceberg/expression/sanitize_expression.cc:110

• The regexes meant to detect fractional seconds use an unescaped '.' (e.g. (.�\d{1,9})), which matches any character rather than a literal dot. This will incorrectly classify strings like 12:34:56x123 as a time/timestamp and may lead to inconsistent sanitization buckets.

  static const std::regex kTime(R"(\d{2}:\d{2}(:\d{2}(.\d{1,9})?)?)");
  static const std::regex kTimestamp(
      R"(\d{4}-\d{2}-\d{2}T\d{2}:\d{2}(:\d{2}(.\d{1,9})?)?)");
  static const std::regex kTimestampTz(
      R"(\d{4}-\d{2}-\d{2}T\d{2}:\d{2}(:\d{2}(.\d{1,9})?)?([-+]\d{2}:\d{2}|Z))");

src/iceberg/expression/sanitize_expression.cc:114

• Same issue as above for the nanosecond timestamp regexes: (.�\d{7,9}) treats '.' as "any character". This can cause invalid strings to be treated as timestamps (and bucketed as such) instead of falling back to hashing.

  static const std::regex kTimestampNs(
      R"(\d{4}-\d{2}-\d{2}T\d{2}:\d{2}(:\d{2}(.\d{7,9})?)?)");
  static const std::regex kTimestampTzNs(
      R"(\d{4}-\d{2}-\d{2}T\d{2}:\d{2}(:\d{2}(.\d{7,9})?)?([-+]\d{2}:\d{2}|Z))");

src/iceberg/update/snapshot_update.cc:208

• CommitMetrics::total_duration/attempts are documented as measuring the wall-clock duration and retry count of the commit attempt, but SnapshotUpdate::Commit() starts the timer and increments attempts even when this update is part of a Transaction (where PendingUpdate::Commit() only calls txn->Apply(*this) and does not perform the catalog UpdateTable call). This means commit duration/attempts reported for transactional updates will not reflect the actual commit RPC + retries, only apply/re-apply time.

[wgtmac]

Thanks for the work on this PR! I did a detailed code review and pushed the changes directly to the branch to avoid a long review-and-revision loop.

Summary of the changes:

• Aligned table full-name handling with Java’s catalog-owned design and preserved REST’s catalog-specific naming behavior.
• Removed unnecessary public APIs, including reporter-combination and legacy table-scan builder APIs.
• Simplified commit reporting so each snapshot update owns its metrics and reports once, only after a successful standalone or transaction commit.
• Reused retry-runner attempt accounting and removed redundant commit-attempt state.
• Aligned scan metrics accounting and naming across table scans, manifest planning, delete-file indexing, and deletion vectors.
• Improved Java parity for expression sanitization, including temporal values, hashes, transforms, unary predicates, and overflow handling.
• Reworked catalog reporter initialization and executor handling. REST reporting now supports both asynchronous execution with a user-provided executor and synchronous fallback without one.
• Propagated catalog reporter construction errors through Result instead of silently ignoring them.
• Strengthened and simplified the related tests, fixtures, names, and comments.

[Copilot]

Pull request overview

Copilot reviewed 58 out of 58 changed files in this pull request and generated no new comments.

Suppressed comments (1)

src/iceberg/expression/sanitize_expression.cc:109

• In the temporal string detection regexes, the fractional-second dot is unescaped (.), so it matches any character. This can incorrectly classify strings like 12:34:56X123 as time/timestamp and return "(time)" instead of hashing the value. Escape the dot in all affected patterns (use \. inside the raw strings).

  static const std::regex kDate(R"(\d{4}-\d{2}-\d{2})");
  static const std::regex kTime(R"(\d{2}:\d{2}(:\d{2}(.\d{1,9})?)?)");
  static const std::regex kTimestamp(
      R"(\d{4}-\d{2}-\d{2}T\d{2}:\d{2}(:\d{2}(.\d{1,9})?)?)");
  static const std::regex kTimestampTz(

[Copilot]

Pull request overview

Copilot reviewed 58 out of 58 changed files in this pull request and generated no new comments.

Suppressed comments (1)

src/iceberg/expression/sanitize_expression.cc:114

• The temporal regexes treat the fractional-seconds separator as . (match-any) rather than a literal dot. This makes non-temporal strings like 12:34:56x123 match kTime and be sanitized as (time) incorrectly. Escape the dot in all patterns that intend to match fractional seconds.

Result<std::string> SanitizeString(std::string_view value, int64_t now, int32_t today) {
  static const std::regex kDate(R"(\d{4}-\d{2}-\d{2})");
  static const std::regex kTime(R"(\d{2}:\d{2}(:\d{2}(.\d{1,9})?)?)");
  static const std::regex kTimestamp(
      R"(\d{4}-\d{2}-\d{2}T\d{2}:\d{2}(:\d{2}(.\d{1,9})?)?)");
  static const std::regex kTimestampTz(
      R"(\d{4}-\d{2}-\d{2}T\d{2}:\d{2}(:\d{2}(.\d{1,9})?)?([-+]\d{2}:\d{2}|Z))");