[FSTORE-2059][FSTORE-2060] Document collect, aggregate, and snowflake REST serving#614
Open
jimdowling wants to merge 3 commits into
Open
[FSTORE-2059][FSTORE-2060] Document collect, aggregate, and snowflake REST serving#614jimdowling wants to merge 3 commits into
jimdowling wants to merge 3 commits into
Conversation
… REST serving https://hopsworks.atlassian.net/browse/FSTORE-2059 https://hopsworks.atlassian.net/browse/FSTORE-2060 Document the collect(N) and aggregate query operations and the REST serving of snowflake-schema feature views. concepts/fs/feature_view/online_api.md introduces the two operations and how the online API serves them (primary-key reads, ordered index scans, RonSQL pushdown). user_guides/fs/feature_view/query.md adds the "Collect recent rows" and "Aggregate rows by entity" sections: order_by/ascending semantics, the (entity..., order column) primary-key requirement for online feature groups, filter-before-limit semantics and the online filter shape restriction, label and mutual-exclusion rules, the N and N-times-width caps, TTL-implied lookback, aggregation functions and output naming, and the window/TTL bound. The nested-join section now states the REST gates: full-primary-key hops, one feature store, and uniform join types (all inner or all left; mixed subtrees stay on the SQL client). user_guides/fs/feature_view/feature-vectors.md adds "Retrieve collected rows" (folded array-of-structs feature, scan_vectors) and "Retrieve aggregate features", and updates the client-choice section with the REST serving paths and the snowflake gates. The filter note now distinguishes point-read filters (not applied) from collect and aggregate filters (applied before folding). All claims verified against the implementation: query.py and feature_view.py signatures, QueryController collect/aggregate validation, PreparedStatementBuilder join gates and filter rendering, ConstructorController output naming, and both clients' empty-history fold. Examples use return_type="pandas" (get_feature_vector has no dict return type). markdownlint, snakeoil, and the strict mkdocs build pass. Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01L5mBKFhoVJBS7J7kCSKQVN
…d gates https://hopsworks.atlassian.net/browse/FSTORE-2059 https://hopsworks.atlassian.net/browse/FSTORE-2060 Updates the collect/aggregate guide for behavior that landed in the later review rounds. The aggregate section now states the creation-time type checks (sum and avg numeric-only, min and max non-complex, greatest and least integer-only), the whole-second window with a required TIMESTAMP event-time feature, the window anchoring contract (read time online, each training row's own event time in point-in-time training data), the (entity..., event_time) primary-key layout online aggregations share with collect, and the point-in-time gates: windowed aggregations join the root directly with an inner or left join, and shapes that cannot be computed point-in-time correct fail at feature view creation instead of producing training data with future leakage. The collect section notes that complex-typed value features cannot be collected on online-enabled feature groups and that n must be an integer. The feature-vectors guide documents empty aggregation windows (count zero, other functions missing) for single and batch reads. markdownlint, snakeoil, and the strict mkdocs build pass. Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01L5mBKFhoVJBS7J7kCSKQVN
Contributor
Author
|
Second commit (bf050b7) documents behavior from the later review rounds: aggregate type checks at creation, whole-second windows with a required TIMESTAMP event time, the window anchoring contract (read time online, each training row's own event time in point-in-time training data), the (entity..., event_time) online layout shared with collect, the point-in-time gates (direct inner/left join; incompatible shapes fail at creation), complex-typed collect features rejected on online feature groups, and empty aggregation windows (count zero, other functions missing). |
https://hopsworks.atlassian.net/browse/FSTORE-2059 https://hopsworks.atlassian.net/browse/FSTORE-2060 The engines now CAST count, sum, and avg outputs to the declared training-dataset types, so the cross-engine output type contract is guaranteed and worth stating: count is bigint; sum widens integers to bigint, floats to double, and decimal(p,s) to decimal(p+10,s); avg is double or decimal(p+4,s+4); min, max, greatest, and least keep the source type. Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01L5mBKFhoVJBS7J7kCSKQVN
Contributor
Author
|
Added one sentence ( |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Documents the
collect(N)andaggregatequery operations (FSTORE-2059) and REST serving of snowflake-schema feature views (FSTORE-2060).JIRA: https://hopsworks.atlassian.net/browse/FSTORE-2059, https://hopsworks.atlassian.net/browse/FSTORE-2060
Implementation PRs: logicalclocks/hopsworks-ee#3148, logicalclocks/hopsworks-ee#3149, logicalclocks/hopsworks-api#1040, logicalclocks/hopsworks-api#1041.
Pages
concepts/fs/feature_view/online_api.md: what collect and aggregate produce online, and how the online API serves them (primary-key reads, ordered index scans, RonSQL pushdown).user_guides/fs/feature_view/query.md: "Collect recent rows" and "Aggregate rows by entity" sections covering order semantics, the(entity..., order column)primary-key requirement for online feature groups, filter-before-limit and the online filter shape restriction, label and mutual-exclusion rules, the N and N-times-width caps, TTL-implied lookback, aggregation functions with output naming, and the window/TTL bound. The nested-join section now states the REST gates: full-primary-key hops, one feature store, uniform join types (all inner or all left).user_guides/fs/feature_view/feature-vectors.md: "Retrieve collected rows" (folded array-of-structs feature,scan_vectors) and "Retrieve aggregate features" sections, plus the REST client's serving paths and snowflake gates in the client-choice section.Verification
Every claim was checked against the implementation:
query.py/feature_view.pysignatures,QueryControllercollect and aggregate validation,PreparedStatementBuilderjoin gates and filter rendering,ConstructorControlleroutput naming, and the empty-history fold on both online clients. Code examples usereturn_type="pandas"becauseget_feature_vectorhas no dict return type. markdownlint, snakeoil, and the strict mkdocs build pass locally.🤖 Generated with Claude Code