feat(inspect): implement SnapshotsTable scanning by WZhuo · Pull Request #801 · apache/iceberg-cpp

WZhuo · 2026-07-02T08:35:38Z

Summary

Implement Scan() for SnapshotsTable, adding the core scanning infrastructure to MetadataTable and materializing snapshot rows into Arrow arrays via ArrowRowBuilder.

Changes

`metadata_table.h` / `.cc`

Add SnapshotSelection struct with snapshot_id, as_of_timestamp, ref_name optional fields for time-travel queries
Add Scan() virtual method returning Result<ArrowArray> with a convenience zero-arg overload
Add supports_time_travel() concrete method driven by kind(), matching Java's TIME_TRAVEL_TABLE_TYPES static set (currently both kSnapshots and kHistory return false)
Default Scan() returns NotSupported

`snapshots_table.h` / `.cc`

Declare and implement SnapshotsTable::Scan() — iterates source_table()->snapshots() and writes 6 columns (committed_at, snapshot_id, parent_id, operation, manifest_list, summary) via ArrowRowBuilder
committed_at converted to microseconds since epoch, matching Java's snap.timestampMillis() \* 1000

Tests

metadata_table_test_base.h — shared MetadataTableTestBase with MockFileIO + MockCatalog fixture, FinishAndImport(), MakeTestSnapshots(), MakeTableWithSnapshots() helpers
metadata_table_test.cc — 2 framework tests (FactoryRejectsNullSourceTable, SupportsTimeTravel)
snapshots_table_test.cc — 11 tests (Construct, SchemaMatchesIcebergSchema, 9 scan data tests matching Java TestDataTaskParser fixture)
history_table_test.cc — 1 schema test

- Add Scan() virtual method and Scan() convenience overload to MetadataTable - Add SnapshotSelection struct for time-travel snapshot resolution - Add supports_time_travel() concrete method driven by kind() - Implement SnapshotsTable::Scan() to materialize snapshot rows via ArrowRowBuilder

Copilot

Pull request overview

Implements initial metadata-table scanning support by adding a Scan() API to MetadataTable (with snapshot-selection parameters for future time-travel) and providing a concrete SnapshotsTable::Scan() implementation that materializes snapshot rows into Arrow arrays. The PR also restructures/extends the metadata-table test suite to validate schemas and snapshot scanning behavior.

Changes:

Added SnapshotSelection and a virtual MetadataTable::Scan() API (with a convenience overload) plus supports_time_travel().
Implemented SnapshotsTable::Scan() to emit snapshot rows (6 columns) via ArrowRowBuilder.
Added/expanded tests and wired new test sources into the metadata-table test target.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
src/iceberg/inspect/metadata_table.h	Adds `SnapshotSelection`, `Scan()` API, and `supports_time_travel()` declaration/docs.
src/iceberg/inspect/metadata_table.cc	Implements default `Scan()` + `supports_time_travel()` and wires factory for kinds.
src/iceberg/inspect/snapshots_table.h	Declares `SnapshotsTable::Scan()` override.
src/iceberg/inspect/snapshots_table.cc	Implements snapshot scanning into Arrow via `ArrowRowBuilder`.
src/iceberg/test/metadata_table_test.cc	Simplifies base setup and adds `SupportsTimeTravel` test.
src/iceberg/test/metadata_table_test_base.h	New shared fixture/helpers for metadata table tests.
src/iceberg/test/snapshots_table_test.cc	New tests validating snapshots table construction/schema/scan output.
src/iceberg/test/history_table_test.cc	New schema test for history table.
src/iceberg/test/CMakeLists.txt	Adds new test sources to the metadata-table test target.

Copilot

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (1)

src/iceberg/inspect/metadata_table.cc:38

supports_time_travel() is currently hard-coded to return false. That matches today’s two Kind values, but it’s easy to forget to update once additional metadata table kinds are added, and it doesn’t reflect the docstring/PR description that this is kind-driven. Consider switching on kind() and making the non-exhaustive case unreachable to keep future additions honest.

bool MetadataTable::supports_time_travel() const noexcept { return false; }

Copilot

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 3 comments.

Copilot

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (1)

src/iceberg/inspect/metadata_table.cc:38

supports_time_travel() is documented (and described in the PR) as being driven by the metadata table kind, but the implementation always returns false regardless of kind. This makes the API misleading and forces future kinds to remember to update callers rather than the method itself.

bool MetadataTable::supports_time_travel() const noexcept { return false; }

Copilot

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.

wgtmac · 2026-07-29T15:15:04Z

+  /// \brief The snapshot ID to read.
+  std::optional<int64_t> snapshot_id;
+  /// \brief Read the snapshot that was current at this timestamp.
+  std::optional<TimePointMs> as_of_timestamp;


Should it be std::variant<std::monostate, int64_t, TimePointMs>? Otherwise we need to define the priority if both snapshot_id and as_of_timestamp are set.

wgtmac · 2026-07-29T15:15:29Z

+  /// \brief Read the snapshot that was current at this timestamp.
+  std::optional<TimePointMs> as_of_timestamp;
+  /// \brief Read the snapshot referenced by this named ref (branch or tag).
+  std::optional<std::string> ref_name;


Is std::string enough? An empty one is unset.

wgtmac · 2026-07-29T15:16:26Z

+  /// The default implementation returns NotSupported. Subclasses override this
+  /// to materialize their data.
+  virtual Result<ArrowArray> Scan(
+      const std::optional<SnapshotSelection>& snapshot_selection);


It doesn't seem necessary to add a std::optional on top of SnapshotSelection if all its values are optional.

wgtmac · 2026-07-29T15:22:53Z

+  ///
+  /// The default implementation returns NotSupported. Subclasses override this
+  /// to materialize their data.
+  virtual Result<ArrowArray> Scan(


Should we return ArrowArrayStream instead? My concern is that a single ArrowArray may lead to OOM if the result is too large. ArrowArrayStream is an iterator of ArrowArrays which is more scalable.

wgtmac · 2026-07-29T15:31:15Z

+    ICEBERG_RETURN_UNEXPECTED(AppendString(builder.column(4), snapshot->manifest_list));
+
+    // column 5: summary (map<string,string>)
+    ICEBERG_RETURN_UNEXPECTED(AppendStringMap(builder.column(5), snapshot->summary));


Snapshot::summary is not equivalent to Java’s Snapshot.summary() here. In C++, the map still contains the operation entry, while Java stores the operation separately and SnapshotsTable.snapshotToRow emits the remaining summary map. As a result, every normal row duplicates operation in both columns (the copied Java fixture has 10 summary entries, not 11). Please omit SnapshotSummaryFields::kOperation when materializing this map, while preserving null versus empty-map semantics.

wgtmac · 2026-07-29T15:31:27Z

 }

+Result<ArrowArray> SnapshotsTable::Scan(
+    const std::optional<SnapshotSelection>& /*snapshot_selection*/) {


Silently ignoring an invalid selection diverges from Java. SnapshotScan.useSnapshot, useRef, and asOfTime validate or resolve the requested snapshot before SnapshotsTable returns all known snapshots. With this implementation, Scan({.snapshot_id = 999}) succeeds and returns unrelated rows. Please validate the selector (or reject a non-null selection as unsupported) instead of treating invalid input as an unqualified scan.

wgtmac · 2026-07-29T15:32:01Z

+  ///
+  /// The snapshots table always returns every known snapshot, so the
+  /// snapshot_selection parameter is ignored.
+  Result<ArrowArray> Scan(


This override hides the zero-argument MetadataTable::Scan() overload under C++ name lookup. A caller using the public SnapshotsTable::Make() result cannot write snapshots_table->Scan(); only callers typed as MetadataTable compile, which is why the current tests miss it. Please add using MetadataTable::Scan; (or an explicit zero-argument overload).

WZhuo marked this pull request as ready for review July 2, 2026 08:46

WZhuo force-pushed the inspect branch 3 times, most recently from 28c4513 to a53f8ce Compare July 6, 2026 03:37

WZhuo changed the title ~~feat(inspect): expand MetadataTable framework for all 16 metadata table types~~ feat(inspect): implement SnapshotsTable scanning Jul 6, 2026

WZhuo force-pushed the inspect branch 3 times, most recently from 9bc23e1 to 04b657f Compare July 7, 2026 02:04

WZhuo force-pushed the inspect branch from 04b657f to efc76b8 Compare July 14, 2026 06:23

WZhuo force-pushed the inspect branch from efc76b8 to 3b11911 Compare July 17, 2026 02:19

manuzhang added this to the 0.4.0 milestone Jul 21, 2026

manuzhang requested a review from Copilot July 21, 2026 03:39

Copilot started reviewing on behalf of manuzhang July 21, 2026 03:42 View session

Copilot AI reviewed Jul 21, 2026

View reviewed changes

Comment thread src/iceberg/inspect/metadata_table.h Outdated

Comment thread src/iceberg/inspect/metadata_table.cc Outdated

Comment thread src/iceberg/test/metadata_table_test_base.h Outdated

Comment thread src/iceberg/inspect/metadata_table.h Outdated

fix(inspect): address metadata table review feedback

6c112cc

Copilot AI review requested due to automatic review settings July 22, 2026 05:32

Copilot started reviewing on behalf of WZhuo July 22, 2026 05:33 View session

Copilot AI reviewed Jul 22, 2026

View reviewed changes

Comment thread src/iceberg/test/snapshots_table_test.cc

Comment thread src/iceberg/test/snapshots_table_test.cc

test(inspect): address snapshots table review feedback

791474e

Copilot AI review requested due to automatic review settings July 23, 2026 01:44

Copilot started reviewing on behalf of WZhuo July 23, 2026 01:45 View session

Copilot AI reviewed Jul 23, 2026

View reviewed changes

Comment thread src/iceberg/test/metadata_table_test_base.h Outdated

Potential fix for pull request finding

a5c20c2

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings July 24, 2026 01:38

Copilot started reviewing on behalf of WZhuo July 24, 2026 01:38 View session

Fix formatting issues in metadata_table_test_base.h

ab62d36

Copilot AI reviewed Jul 24, 2026

View reviewed changes

Comment thread src/iceberg/test/metadata_table_test_base.h Outdated

Comment thread src/iceberg/test/metadata_table_test.cc Outdated

Comment thread src/iceberg/test/snapshots_table_test.cc Outdated

Copilot AI review requested due to automatic review settings July 24, 2026 01:43

Copilot started reviewing on behalf of WZhuo July 24, 2026 01:44 View session

Copilot AI reviewed Jul 24, 2026

View reviewed changes

Comment thread src/iceberg/inspect/snapshots_table.cc

Comment thread src/iceberg/test/metadata_table_test_base.h Outdated

fix(inspect): address remaining review feedback

6680bf9

Copilot AI review requested due to automatic review settings July 27, 2026 02:05

Copilot started reviewing on behalf of WZhuo July 27, 2026 02:05 View session

Copilot AI reviewed Jul 27, 2026

View reviewed changes

Comment thread src/iceberg/inspect/snapshots_table.cc

wgtmac reviewed Jul 29, 2026

View reviewed changes

Uh oh!

Conversation

WZhuo commented Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

metadata_table.h / .cc

snapshots_table.h / .cc

Tests

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

wgtmac Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

wgtmac Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

wgtmac Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

wgtmac Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

wgtmac Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

wgtmac Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

wgtmac Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

WZhuo commented Jul 2, 2026 •

edited

Loading

`metadata_table.h` / `.cc`

`snapshots_table.h` / `.cc`