Sync mechanism

[Sporiff]

Right, I know you're all probably sick of the back and forth on this, so I thought I'd open this up as a discussion here so that we can keep everything together.

Through my various experiments and trials, it's become apparent that the sync mechanism is the hard problem to solve here, and it largely comes down to two things:

1. What is the model around which we are focusing sync? Device-based or user-based?
2. What is the entry point to that sync? General sync endpoint controlled by properties such as entity_type and action or per-entity sync?

Subscriptions sync

My current thinking around subscriptions is, to summarize:

1. No central records for subscriptions. I tried this and it got far too messy, far too quickly. Instead of maintaining a record for a subscription feed, each user/feed subscription should be its own database entry. It means duplication, but it also means nobody is updating data that other people are accessing.
2. Subscriptions are very low traffic by comparison to episodes. A person will listen to several episodes in one day, while they might only subscribe to a few new feeds a week and aren't likely to rapidly un/subscribe. The expected traffic is an initial bulk sync, followed by periodic new entries and occasional unsubscribe actions.

For this kind of thing, a single sync endpoint that keeps track of actions is trivial enough, for example:

[
  {
    "event_id": "6217f6e5-9bd5-4fc7-bd8c-8d0dea32d082",
    "action": "create",
    "action_timestamp": "2026-07-05T14:00:45+00:00",
    "entity_type": "subscription",
    "data": {
      "feed_url": "https://example.com/rss1",
      "guid": "63bdca44-e177-4b0c-8307-42e653b0ce54",
      "subscribed_at": "2023-07-04T08:23:13+00:00",
      "unsubscribed_at": "2026-07-03T12:20:14+00:00",
    }
  }
]

We could make a required heading for client IDs (X-Client-ID) so that clients don't receive their own updates when they sync by default, then have a param they can pass if they want to see everything.

Episodes sync

Episodes have one real sticking point in my mind: authority over changes. We can look at this a few ways:

1. Timestamp authority: the client records a UTC timestamp of the action and sends it with each update. The action with the latest timestamp becomes the authoritative "view" of the entity.
   • The benefit here is that this feels closest to the "user-centric" view of things; whatever the user did last in real life becomes the authoritative state.
   • The big weakness here is mutability. If a user syncs their devices, but a device comes online later with a later timestamp, it could cause problems.
   • We also risk big problems if the client records the timestamp wrong, for whatever reason.
2. Last-seen authority: the server responds with the latest version of the action that it has received, based on its own authoritative timestamps.
   • The benefit here is that the server is authoritative, and the clients are all beholden to it.
   • The big drawback is that drift can easily occur. What if a device comes online later, but passes data that happened much earlier? What if the client's clock drifts?

From this, it strikes me that we need to have some sort of versioning capability (optimistic concurrency model), where the server responds with the latest version of an entity. To give a simple example:

// Server sync object
[
  {
    "event_id": "6217f6e5-9bd5-4fc7-bd8c-8d0dea32d082",
    "action": "create",
    "action_timestamp": "2026-07-05T14:00:45+00:00",
    "entity_type": "subscription",
	"entity_id": "31c81c29-4953-43b9-9a6e-002b854517d8"
    "data": {
      "feed_url": "https://example.com/rss1",
      "guid": "63bdca44-e177-4b0c-8307-42e653b0ce54",
      "subscribed_at": "2023-07-04T08:23:13+00:00",
	  "version": 1, // The server responds with the latest version it holds
    }
  }
]

When the client receives these data from the server, it must then refer to the latest version that it's aware of when it wants to perform an update. For example:

// Client sync request
[
  {
    "event_id": "6217f6e5-9bd5-4fc7-bd8c-8d0dea32d082",
    "action": "upsert",
    "action_timestamp": "2026-07-05T14:00:45+00:00",
    "entity_type": "subscription",
	"entity_id": "31c81c29-4953-43b9-9a6e-002b854517d8"
	"required_version": 1, // The expected version is listed here
    "data": {
      "feed_url": "https://example.com/rss1",
      "guid": "63bdca44-e177-4b0c-8307-42e653b0ce54",
      "subscribed_at": "2023-07-04T08:23:13+00:00",
      "unsubscribed_at": "2026-07-03T12:20:14+00:00",
    }
  }
]

If the server receives this, and the version has since incremented, this is a mismatch. The client then needs to essentially "rebase" on top of the latest change, and then send it back with an updated expectation.

// Client
[
  {
    "event_id": "6217f6e5-9bd5-4fc7-bd8c-8d0dea32d082",
    "action": "upsert",
    "action_timestamp": "2026-07-05T14:00:45+00:00",
    "entity_type": "subscription",
	"entity_id": "31c81c29-4953-43b9-9a6e-002b854517d8"
	"required_version": 1,
    "data": {
      "feed_url": "https://example.com/rss1",
      "guid": "63bdca44-e177-4b0c-8307-42e653b0ce54",
      "subscribed_at": "2023-07-04T08:23:13+00:00",
      "unsubscribed_at": "2026-07-03T12:20:14+00:00",
    }
  }
]

// Server
[
  {
	"status": "conflict",
	"currentVersion": 3,
	"actions": [
	  ...
	]
  }
]

// Client syncs the latest version, rebases its changes, and sends a new update
[
  {
    "event_id": "6217f6e5-9bd5-4fc7-bd8c-8d0dea32d082",
    "action": "upsert",
    "action_timestamp": "2026-07-05T14:00:45+00:00",
    "entity_type": "subscription",
	"required_version": 3, // The client updates the required version to inform the server it is aware of the changes
    "data": {
      "feed_url": "https://example.com/rss1",
      "guid": "63bdca44-e177-4b0c-8307-42e653b0ce54",
      "subscribed_at": "2023-07-04T08:23:13+00:00",
      "unsubscribed_at": "2026-07-03T12:20:14+00:00",
    }
  }
]

To summarize: I'm trying to think of ways that we can avoid having too many conflicts. As I said above, I think subscriptions are unlikely to give too many issues here, but users will likely update episodes a lot more.

I want to hear opinions, alternatives, feedback, technical challenges. Whatever you've got, basically. Working this out would basically unblock us, I feel.

[keunes]

Thanks @Sporiff for starting this discussion.

To begin with a key question:

What is the model around which we are focusing sync? Device-based or user-based?

For me this is clear: it is user-based. The user must be central, and is assumed to have multiple devices on which they listen and do other actions. And I think we concluded this already before - gpodder.net seems more device-focussed, in its UX at least.

[keunes]

Creating a separate sub-thread for another key point:

What is the entry point to that sync? General sync endpoint controlled by properties such as entity_type and action or per-entity sync?

I can't comment on this, really. No idea what the benefits and downsides are of either. (From the APIs that I've consumed the per-entity endpoint seems kind of standard, but that's hardly an argument.)