Skip to content

Global temporal and leave-last-out splitting for next-item evaluation#708

Open
hieuddo wants to merge 4 commits into
PreferredAI:masterfrom
hieuddo:temporal-split
Open

Global temporal and leave-last-out splitting for next-item evaluation#708
hieuddo wants to merge 4 commits into
PreferredAI:masterfrom
hieuddo:temporal-split

Conversation

@hieuddo

@hieuddo hieuddo commented Jul 8, 2026

Copy link
Copy Markdown
Member

Description

The current NextItemEvaluation only supports pre-split data via from_splits with three datasets: Diginetica, RetailRocket, and Cosmetics; already preprocessed and uploaded in bundle train/val/test. We should move to more transparent data loading/splitting. Recent studies have also shown some flaws/issues in data splitting that lead to data leakage:

  • [1] Meng et al. (2020). Exploring Data Splitting Strategies for the Evaluation of Recommendation Models. RecSys 2020.
  • [2] Ji et al. (2023). A Critical Study on Data Leakage in Recommender System Offline Evaluation. ACM TOIS 2023.
  • [3] Hidasi & Czapp (2023). Widespread Flaws in Offline Evaluation of Recommender Systems. RecSys 2023.

This PR adds:

  • NextItemEvaluation.from_timestamps: a global temporal split, the leakage-free protocol recommended by [2], similar to the existing TimestampSplit for rating data.
  • NextItemEvaluation.leave_last_out: the per-user leave-one-out split, commonly used in the sequential literature. We'll raise a warning that this splitting method leaks cross-user future information; we support this because it is still very common.
  • A sort-order warning in SequentialDataset.build: warns when rows within a session are not chronological.
  • Add three new amazon_review datasets: Beauty, Sports_and_Outdoors, and Toys_and_Games; which are widely used in Semantic ID papers. For now, we only implement load_feedback to use these three as normal rating data. Any load_text or load_item_metadata are for future implementation with some Semantic ID models (e.g., TIGER, RQ-VAE).

Related Issues

Checklist:

  • I have added tests.
  • I have updated the documentation accordingly.
  • I have updated datasets/README.md (if you are adding a new dataset).

@hieuddo

hieuddo commented Jul 8, 2026

Copy link
Copy Markdown
Member Author

Quick runs (10 epochs) with SASRec and BERT4Rec show better performance than their corresponding results in TIGER paper

Beauty:

Model Recall@10 NDCG@10
SASRec 0.0669 0.0356
BERT4Rec 0.0656 0.0360

Sports:

Model Recall@10 NDCG@10
SASRec 0.0351 0.0182
BERT4Rec 0.0330 0.0174

Toys:

Model Recall@10 NDCG@10
SASRec 0.0739 0.0376
BERT4Rec 0.0688 0.0378

Comment thread README.md Outdated
@hieuddo hieuddo requested a review from qtuantruong July 9, 2026 03:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants