Skip to content

*: re-vendor kvproto at b41e863 (2026-06-22) and record the vendored revision#550

Open
eduralph wants to merge 1 commit into
tikv:masterfrom
getwyrd:kvproto-refresh
Open

*: re-vendor kvproto at b41e863 (2026-06-22) and record the vendored revision#550
eduralph wants to merge 1 commit into
tikv:masterfrom
getwyrd:kvproto-refresh

Conversation

@eduralph

@eduralph eduralph commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Why: the vendored protos are an unrecorded late-2023/early-2024 vintage (errorpb.proto/pdpb.proto untouched since #324, one hand-patched addition from #519). Nothing records which kvproto revision they came from, and the staleness now blocks real features.

What: all vendored protos + the include tree re-vendored at pingcap/kvproto b41e863 — the revision client-go pins today — plus a new proto/VERSION recording it. Regenerated with the existing tonic-build pipeline (21 files); one mechanical source fix (pdpb.RequestHeader gained caller_component/caller_id). Newly available surface includes KvFlush/KvBufferBatchGet (pipelined DML), HealthFeedback, and pdpb.BatchScanRegions.

Two new protocol semantics arrive with the refresh and are handled rather than silently mis-parsed:

  1. errorpb.UndeterminedResult (an unknown raft apply outcome). Following client-go's shape — its transport never retries this ("should not retry … processed by the caller", region_request.go) and each action decides — the plan layer here retries by default (replaying an idempotent request resolves the uncertainty; the error escapes unchanged on backoff exhaustion), while plans for which a replay is unsafe are terminal on first sight: raw CAS (a replay would compare against its own effect and report succeed = false for a write that happened), the primary commit (client-go returns ErrResultUndetermined there, commit.go), and async-commit/1PC prewrites (the commit point — stricter than client-go, in the safe direction). Commit paths classify the surfaced error as UndeterminedError: at a commit point, "failed" must never be claimed when the transaction may already be durable.

  2. Shared locks (SharedLock/SharedPessimisticLock; real holders live in shared_lock_infos — "DO NOT read from the wrapper"). This client doesn't implement shared-lock resolution yet, and partial handling is worse than none (resolving the wrapper checks transaction 0; filtering on wrapper fields silently drops members). Resolution refuses them with an explicit error, and the API-v2 keyspace codecs are wrapper-aware (keyed on type — the empty logical key is valid — recursing into members), so scan results can't panic the truncation path. Full shared-lock support is deliberately follow-up work.

Verification: make check green; 67 lib tests (4 new covering the undetermined and shared-lock semantics); txn/raw/failpoint integration suites green against a local api-v2 cluster; and behavior compared against client-go by driving both clients through identical transactional and raw scenarios — no observable change.

Summary by CodeRabbit

  • New Features
    • Added snapshot-backup preparation streaming APIs and a new log-backup flush operation.
    • Extended keyspace, meta storage, deadlock, and resource-group management capabilities.
    • Introduced GC barrier/state controls, shard-aware coprocessor options, region bucket streaming, and versioned coprocessor requests.
    • Expanded encryption support for cloud KMS and file encryption metadata.
  • Bug Fixes
    • Improved handling of undetermined outcomes to avoid unsafe retries during commit.
    • Prevented shared locks from being incorrectly resolved or filtered.

@ti-chi-bot ti-chi-bot Bot added the dco-signoff: yes Indicates the PR's author has signed the dco. label Jul 15, 2026
@ti-chi-bot

ti-chi-bot Bot commented Jul 15, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign lonng for approval. For more information see the Code Review Process.
Please ensure that each of them provides their approval before proceeding.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot Bot added contribution This PR is from a community contributor. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Jul 15, 2026
@coderabbitai

coderabbitai Bot commented Jul 15, 2026

Copy link
Copy Markdown

Review Change Stack

Warning

Review limit reached

@eduralph, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 40 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: dc4e4750-9588-4a2a-9988-ad0d3839d55d

📥 Commits

Reviewing files that changed from the base of the PR and between ac3b830 and b01ca22.

⛔ Files ignored due to path filters (21)
  • src/generated/backup.rs is excluded by !**/generated/**
  • src/generated/cdcpb.rs is excluded by !**/generated/**
  • src/generated/coprocessor.rs is excluded by !**/generated/**
  • src/generated/deadlock.rs is excluded by !**/generated/**
  • src/generated/encryptionpb.rs is excluded by !**/generated/**
  • src/generated/errorpb.rs is excluded by !**/generated/**
  • src/generated/import_sstpb.rs is excluded by !**/generated/**
  • src/generated/keyspacepb.rs is excluded by !**/generated/**
  • src/generated/kvrpcpb.rs is excluded by !**/generated/**
  • src/generated/logbackup.rs is excluded by !**/generated/**
  • src/generated/meta_storagepb.rs is excluded by !**/generated/**
  • src/generated/metapb.rs is excluded by !**/generated/**
  • src/generated/mpp.rs is excluded by !**/generated/**
  • src/generated/pdpb.rs is excluded by !**/generated/**
  • src/generated/raft_cmdpb.rs is excluded by !**/generated/**
  • src/generated/raft_serverpb.rs is excluded by !**/generated/**
  • src/generated/resource_manager.rs is excluded by !**/generated/**
  • src/generated/resource_usage_agent.rs is excluded by !**/generated/**
  • src/generated/schedulingpb.rs is excluded by !**/generated/**
  • src/generated/tikvpb.rs is excluded by !**/generated/**
  • src/generated/tsopb.rs is excluded by !**/generated/**
📒 Files selected for processing (45)
  • proto/VERSION
  • proto/autoid.proto
  • proto/brpb.proto
  • proto/cdcpb.proto
  • proto/configpb.proto
  • proto/coprocessor.proto
  • proto/deadlock.proto
  • proto/debugpb.proto
  • proto/diagnosticspb.proto
  • proto/disaggregated.proto
  • proto/disk_usage.proto
  • proto/encryptionpb.proto
  • proto/enginepb.proto
  • proto/errorpb.proto
  • proto/gcpb.proto
  • proto/import_kvpb.proto
  • proto/import_sstpb.proto
  • proto/include/eraftpb.proto
  • proto/include/gogoproto/gogo.proto
  • proto/include/rustproto.proto
  • proto/keyspacepb.proto
  • proto/kvrpcpb.proto
  • proto/logbackuppb.proto
  • proto/meta_storagepb.proto
  • proto/metapb.proto
  • proto/mpp.proto
  • proto/pdpb.proto
  • proto/raft_cmdpb.proto
  • proto/raft_serverpb.proto
  • proto/recoverdatapb.proto
  • proto/replication_modepb.proto
  • proto/resource_manager.proto
  • proto/resource_usage_agent.proto
  • proto/schedulingpb.proto
  • proto/tikvpb.proto
  • proto/tracepb.proto
  • proto/tsopb.proto
  • src/pd/timestamp.rs
  • src/raw/client.rs
  • src/request/keyspace.rs
  • src/request/plan.rs
  • src/request/plan_builder.rs
  • src/transaction/lock.rs
  • src/transaction/mod.rs
  • src/transaction/transaction.rs
📝 Walkthrough

Walkthrough

The update vendors a newer kvproto revision, expands protobuf service and message contracts, standardizes code-generation options, and changes Rust retry and lock handling for undetermined results and shared-lock wrappers.

Changes

Protocol and transaction safety changes

Layer / File(s) Summary
Proto revision and foundational schema updates
proto/VERSION, proto/autoid.proto, proto/brpb.proto, proto/cdcpb.proto, proto/configpb.proto, proto/coprocessor.proto, proto/deadlock.proto, proto/debugpb.proto, proto/diagnosticspb.proto, proto/disaggregated.proto, proto/disk_usage.proto, proto/encryptionpb.proto, proto/enginepb.proto
Updates the kvproto revision and code-generation options, and adds backup, encryption, coprocessor, deadlock, and error schema declarations.
Service and operational contract expansion
proto/import_sstpb.proto, proto/keyspacepb.proto, proto/kvrpcpb.proto, proto/logbackuppb.proto, proto/meta_storagepb.proto, proto/metapb.proto, proto/mpp.proto, proto/pdpb.proto, proto/raft_cmdpb.proto, proto/raft_serverpb.proto, proto/resource_manager.proto, proto/resource_usage_agent.proto, proto/schedulingpb.proto, proto/tikvpb.proto, proto/tracepb.proto, proto/tsopb.proto
Adds RPCs and fields for imports, keyspaces, metadata storage, GC state, resource usage, scheduling, TiKV batching, and related public messages.
Terminal handling for undetermined results
src/request/plan.rs, src/request/plan_builder.rs, src/raw/client.rs, src/transaction/transaction.rs, src/pd/timestamp.rs
Adds terminal retry behavior for undetermined region results and applies it to compare-and-swap and transaction prewrite/commit flows.
Shared-lock rejection and keyspace encoding
src/request/keyspace.rs, src/request/plan.rs, src/transaction/lock.rs, src/transaction/mod.rs
Detects shared-lock wrappers, preserves their nested lock fields during encoding, and rejects them before lock resolution or cleanup.

Estimated code review effort: 5 (Critical) | ~120 minutes

Possibly related issues

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly matches the main change: re-vendoring kvproto at the specified revision and recording that revision.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

…revision

The vendored protos were an unrecorded late-2023/early-2024 vintage (errorpb
and pdpb untouched since tikv#324, one hand-patched addition from tikv#519). Re-vendor
everything at pingcap/kvproto b41e86365ce0 — the revision client-go currently
pins — and add proto/VERSION so the vintage is never again guesswork.

Regenerated with the existing tonic-build pipeline (21 files). One mechanical
source fix: pdpb.RequestHeader gained caller_component/caller_id.

Two new protocol semantics arrive with the refresh and are handled rather than
silently mis-parsed:

1. errorpb.UndeterminedResult (an unknown raft apply outcome). The transport
   never retries it in client-go (region_request.go: 'should not retry ...
   processed by the caller'); each action decides. Here: the plan layer retries
   by default (replaying an idempotent request resolves the uncertainty, and
   the error escapes UNCHANGED on backoff exhaustion), while plans for which a
   replay is unsafe are terminal on first sight — raw CAS (a replay would
   compare against its own effect and report a false failure), the primary
   commit (client-go returns ErrResultUndetermined there, commit.go), and
   async-commit/1PC prewrites (the commit point; stricter than client-go, in
   the safe direction). Commit paths classify the surfaced error as
   UndeterminedError: at a commit point, 'failed' must never be claimed when
   the transaction may already be durable.

2. Shared locks (SharedLock / SharedPessimisticLock, whose real holders live
   in shared_lock_infos — 'DO NOT read from the wrapper LockInfo'). This
   client does not implement shared-lock resolution yet, and partial handling
   is worse than none (resolving the wrapper checks transaction 0; filtering
   on wrapper fields silently drops members). Resolution therefore REFUSES
   them with an explicit error, before any filter. The API-v2 keyspace codecs
   are wrapper-aware (identifying wrappers by TYPE — the empty logical key is
   valid — and recursing into members), so scan results can never panic the
   truncation path.

Tests: 67 lib tests (4 new: undetermined terminal/preserved/recognized,
shared-lock refusal); txn/raw/failpoint integration suites green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Eduard R. <eduard@ralphovi.net>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

contribution This PR is from a community contributor. dco-signoff: yes Indicates the PR's author has signed the dco. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant