WPB-26489 backend adminless scheduled jobs model for deletion and reminder by battermann · Pull Request #5289 · wireapp/wire-server

battermann · 2026-06-25T07:39:56Z

https://wearezeta.atlassian.net/browse/WPB-26489

Checklist

Add a new entry in an appropriate subdirectory of changelog.d
Read and follow the PR guidelines

TODO

add test with ephemeral user once related PR is merged

Copilot

Pull request overview

This PR introduces an Arbiter-backed scheduled-jobs subsystem across services, migrating the meetings-cleanup worker to run via Arbiter and adding initial “adminless” reminder/deletion scheduled jobs, along with an internal /i/jobs UI/API mount in Brig.

Changes:

Add Wire.API.Jobs, Wire.JobSubsystem, and Wire.JobStore to model/schedule jobs and persist minimal app-level metadata in Postgres.
Replace the standalone meetings cleanup cron thread with an Arbiter worker pool (scheduledJobs.pollInterval controls polling cadence).
Expose Arbiter’s internal UI + REST API through Brig’s internal routes under /i/jobs and /i/jobs/api/v1.

Reviewed changes

Copilot reviewed 56 out of 57 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
services/galley/src/Galley/Env.hs	Add Galley env field for passing a connection string into the job subsystem.
services/galley/src/Galley/App.hs	Wire `JobSubsystem` + `JobStore` interpreters and derive a Postgres connection string for Arbiter usage.
services/galley/galley.cabal	Add `arbiter-core` dependency for Galley.
services/galley/default.nix	Add `arbiter-core` Nix dependency for Galley.
services/brig/src/Brig/Run.hs	Adjust internal API wiring to pass `env` into `IAPI.servantSitemap`.
services/brig/src/Brig/App.hs	Initialize embedded Arbiter server/UI application and store it in Brig env.
services/brig/src/Brig/API/Internal.hs	Mount internal `/i/jobs` Raw app into Brig internal routes.
services/brig/default.nix	Add Arbiter packages (`arbiter-core`, `arbiter-servant`, `arbiter-servant-ui`).
services/brig/brig.cabal	Add Arbiter packages to Brig build-depends.
services/background-worker/test/Test/Wire/Util.hs	Extend test env with `arbiterConnStr` placeholder.
services/background-worker/test/Test/Wire/BackendNotificationPusherSpec.hs	Extend test env with `arbiterConnStr` placeholder.
services/background-worker/src/Wire/MeetingsCleanupWorker.hs	Refactor to expose `runCleanupOldMeetings` instead of owning the cron thread; move metrics increment into cleanup routine.
services/background-worker/src/Wire/Effects.hs	Add `JobSubsystem`/`JobStore` effects + interpreters and extend migration handling.
services/background-worker/src/Wire/BackgroundWorker/ScheduledJobs.hs	New scheduled-jobs dispatcher/worker wiring for meetings cleanup + adminless jobs via Arbiter.
services/background-worker/src/Wire/BackgroundWorker/Options.hs	Add `scheduledJobs` config section (poll interval).
services/background-worker/src/Wire/BackgroundWorker/Env.hs	Add `arbiterConnStr` to runtime env, derived from Postgres settings.
services/background-worker/src/Wire/BackgroundWorker.hs	Start the unified scheduled-jobs worker instead of the old meetings-cleanup worker thread.
services/background-worker/src/Wire/AdminlessJobsWorker.hs	New adminless reminder/deletion job handlers + scheduled job cleanup in app-level store.
services/background-worker/default.nix	Add `arbiter-core` dependency for background-worker.
services/background-worker/background-worker.integration.yaml	Add integration config for `scheduledJobs.pollInterval`.
services/background-worker/background-worker.cabal	Expose new modules and add `arbiter-core` dependency.
postgres-schema.sql	Update schema snapshot to include scheduled jobs table and related DB objects.
nix/manual-overrides.nix	Add overrides/jailbreaks for Arbiter package set.
nix/haskell-pins.nix	Add Arbiter source pin and package mapping.
libs/wire-subsystems/wire-subsystems.cabal	Add Arbiter deps and expose new job subsystem/store modules; add `cron` dep.
libs/wire-subsystems/test/unit/Wire/ConversationSubsystem/InterpreterSpec.hs	Add a `JobSubsystem` mock interpreter for conversation subsystem unit tests.
libs/wire-subsystems/src/Wire/Options/Keys.hs	Fix ASN.1 imports to use `crypton-asn1-*` packages explicitly.
libs/wire-subsystems/src/Wire/JobSubsystem/Workers.hs	New Arbiter worker runner helpers (recurring + one-off) with explicit defaults.
libs/wire-subsystems/src/Wire/JobSubsystem/Interpreter.hs	New Polysemy interpreter to schedule jobs into Arbiter and start workers.
libs/wire-subsystems/src/Wire/JobSubsystem.hs	New Polysemy effect and configuration types for scheduling/starting jobs.
libs/wire-subsystems/src/Wire/JobStore/Postgres.hs	New Postgres-backed interpreter for persisting minimal scheduled job metadata.
libs/wire-subsystems/src/Wire/JobStore.hs	New Polysemy effect for scheduled job metadata persistence.
libs/wire-subsystems/src/Wire/ConversationSubsystem/Update.hs	Schedule adminless reminder/deletion jobs when last admin leaves; add reminder event emission.
libs/wire-subsystems/src/Wire/ConversationSubsystem/Interpreter.hs	Add internal actions for adminless deletion/reminder execution paths.
libs/wire-subsystems/src/Wire/ConversationSubsystem/Action.hs	Factor conversation deletion into reusable `removeConversation`.
libs/wire-subsystems/src/Wire/ConversationSubsystem.hs	Add new internal conversation actions for adminless job execution.
libs/wire-subsystems/postgres-migrations/20260624095944-extend-job-model.sql	Add `scheduled_jobs` table + indexes migration.
libs/wire-subsystems/default.nix	Add Arbiter and `cron` dependencies for wire-subsystems.
libs/wire-api/wire-api.cabal	Expose new Jobs API modules in wire-api.
libs/wire-api/test/unit/Test/Wire/API/Conversation.hs	Extend event tests for new `ConvAdminlessReminder` type classification.
libs/wire-api/src/Wire/API/Team/FeatureFlags.hs	Accept both legacy and current JSON keys for search visibility defaults.
libs/wire-api/src/Wire/API/Routes/Internal/Jobs.hs	New internal route type for mounting the embedded jobs app.
libs/wire-api/src/Wire/API/Routes/Internal/Brig.hs	Add Jobs app route into Brig internal API type.
libs/wire-api/src/Wire/API/Jobs.hs	New job payload types, queue names, and registry definition for Arbiter.
libs/wire-api/src/Wire/API/Event/Conversation.hs	Add `ConvAdminlessReminder` event type and corresponding event payload schema.
libs/types-common/src/Data/Id.hs	Introduce `ScheduledJobId` and `ScheduledJob` id tag.
libs/extended/src/Hasql/Pool/Extended.hs	Add shared `postgresqlConnectionString` helper and refactor pool init to use it.
integration/test/Test/AdminlessGroups.hs	Add integration coverage for adminless scheduling and Arbiter execution path.
integration/test/Notifications.hs	Add helper to detect adminless reminder notifications.
flake.nix	Add Arbiter git input.
flake.lock	Lock Arbiter input revision/hash.
docs/src/developer/reference/config-options.md	Document `scheduledJobs.pollInterval`.
charts/wire-server/values.yaml	Add Helm value for `background-worker.config.scheduledJobs.pollInterval`.
charts/wire-server/templates/background-worker/configmap.yaml	Render scheduledJobs config into background-worker configmap.
changelog.d/2-features/WPB-26489	Feature changelog entry for Arbiter-backed scheduled jobs + adminless jobs.
changelog.d/0-release-notes/WPB-26489	Release notes for Arbiter scheduling and `/i/jobs` endpoints.
cabal.project	Add `source-repository-package` entries for Arbiter packages.

akshaymankar

Haven't read everything, comments based on just trying to understand the arbiter stuff.

Copilot

Pull request overview

Copilot reviewed 50 out of 51 changed files in this pull request and generated 6 comments.

Copilot

Pull request overview

Copilot reviewed 52 out of 53 changed files in this pull request and generated 2 comments.

Copilot

Pull request overview

Copilot reviewed 53 out of 54 changed files in this pull request and generated 2 comments.

Co-authored-by: Akshay Mankar <axeman@mailbox.org>

- golden tests - make orig user id optional - raise retryable exception explicitly - change log level to debug

akshaymankar · 2026-07-16T13:18:02Z

+      jitter: equal
+      # Maximum time to wait for in-flight jobs during shutdown.
+      gracefulShutdownTimeout: 30s
+      # How often the Arbiter reaper runs and how long worker heartbeats remain valid.


Does this also need to be configured on the K8s deployment?

akshaymankar · 2026-07-16T13:24:02Z

+      workerHeartbeatInterval: 10s
+      # Base and cap for exponential retry backoff.
+      backoffBase: 2.0
+      backoffCap: 1048576s


This is 12 days, 3 hours, 16 mins and 16s. Isn't it too much? Also, why this specific number?

Suggested change

backoffCap: 1048576s

backoffCap: 300s

Ah its 2^20, but I think its too much to wait 12 days before a retry.

This is the cap so we are not going to wait until it exceeds unless we massively increase the max attempts. But we can set it to 1 day maybe?

akshaymankar · 2026-07-16T15:09:28Z

  maxAttempts: 3     # total attempts incl. first run

+# Scheduled jobs
+scheduledJobs:


Maybe worth specifying how this is different from backgroundJobs since this can (and proabably will) replace background jobs soon.

for the rabbit mq backed jobs

zebot added the ok-to-test Approved for running tests in CI, overrides not-ok-to-test if both labels exist label Jun 25, 2026

battermann changed the base branch from WPB-26487-backend-scheduled-jobs-http-control-plane-using-arbiter to develop June 29, 2026 10:13

battermann force-pushed the WPB-26489-backend-adminless-scheduled-jobs-model-and-teardown-support branch from 7737422 to 4e704c1 Compare June 29, 2026 12:28

battermann marked this pull request as ready for review June 29, 2026 12:30

battermann requested review from a team as code owners June 29, 2026 12:30

battermann requested a review from Copilot June 29, 2026 12:52

Copilot started reviewing on behalf of battermann June 29, 2026 12:52 View session

Copilot AI reviewed Jun 29, 2026

View reviewed changes

akshaymankar reviewed Jun 29, 2026

View reviewed changes

battermann force-pushed the WPB-26489-backend-adminless-scheduled-jobs-model-and-teardown-support branch 2 times, most recently from 25c5413 to c9ed418 Compare July 6, 2026 07:42

battermann changed the title ~~WPB-26489 backend adminless scheduled jobs model and teardown support~~ WPB-26489 backend adminless scheduled jobs model for deletion and reminder Jul 6, 2026

battermann force-pushed the WPB-26489-backend-adminless-scheduled-jobs-model-and-teardown-support branch 2 times, most recently from 5e49c76 to 9d7fbf5 Compare July 9, 2026 14:27

battermann changed the base branch from develop to WPB-25325-backend-background-jobs-hasql-resource-pool-refactoring July 9, 2026 14:28

battermann force-pushed the WPB-26489-backend-adminless-scheduled-jobs-model-and-teardown-support branch from f0df3e3 to 33c7c37 Compare July 9, 2026 16:03

battermann requested a review from Copilot July 9, 2026 16:03

Copilot started reviewing on behalf of battermann July 9, 2026 16:04 View session

Copilot AI reviewed Jul 9, 2026

View reviewed changes

battermann force-pushed the WPB-26489-backend-adminless-scheduled-jobs-model-and-teardown-support branch from 33c7c37 to 35ea9ff Compare July 10, 2026 08:51

Base automatically changed from WPB-25325-backend-background-jobs-hasql-resource-pool-refactoring to develop July 10, 2026 12:37

battermann force-pushed the WPB-26489-backend-adminless-scheduled-jobs-model-and-teardown-support branch from dbb2409 to d4d78b4 Compare July 10, 2026 13:14

battermann requested a review from Copilot July 10, 2026 13:23

Copilot started reviewing on behalf of battermann July 10, 2026 13:24 View session

Copilot AI reviewed Jul 10, 2026

View reviewed changes

Comment thread libs/wire-subsystems/src/Wire/JobSubsystem/Workers.hs Outdated

Comment thread libs/wire-subsystems/src/Wire/JobSubsystem/Interpreter.hs Outdated

battermann requested a review from Copilot July 10, 2026 14:23

Copilot started reviewing on behalf of battermann July 10, 2026 14:24 View session

Copilot AI reviewed Jul 10, 2026

View reviewed changes

Comment thread services/background-worker/src/Wire/BackgroundWorker/Options.hs

Comment thread libs/wire-subsystems/src/Wire/ConversationSubsystem/Update.hs Outdated

battermann and others added 16 commits July 16, 2026 11:33

Update integration/test/Test/AdminlessGroups.hs

b4e220f

Co-authored-by: Akshay Mankar <axeman@mailbox.org>

update changelog

efebbd5

clean up

37eba57

- golden tests - make orig user id optional - raise retryable exception explicitly - change log level to debug

thread request ID through the job

5d56db7

make code more DRY using type level string

6ddaeee

encode MeetingsJob to empty JSON object

c9d6b4e

rename param

b06478c

refactor ArbiterAdapter

34a3ba8

upgrade arbiter and fix usage

0f8bc89

extract env ctor, remove team from dedup key and group key

e0584f7

make arbiter config runtime configurable

423cfb5

add ephemeral user test

6691488

fixed unused dependencies

a0f2600

make linter happy

09cc8ac

increase postgres connections for local tests

2972277

fixed allowGuest helper

dae09bf

battermann force-pushed the WPB-26489-backend-adminless-scheduled-jobs-model-and-teardown-support branch from 4dfdc8f to 818aa31 Compare July 16, 2026 11:33

run only single worker pool on shared job queue

0471132

battermann force-pushed the WPB-26489-backend-adminless-scheduled-jobs-model-and-teardown-support branch from 818aa31 to 0471132 Compare July 16, 2026 12:10

cabal.project: Remove pinned arbiter, use nix

ff00e5d

akshaymankar reviewed Jul 16, 2026

View reviewed changes

battermann added 7 commits July 17, 2026 09:39

split queue into meetings and conversations

37ace44

move workers to background-worker and remove the handlers

dae34f2

linter

42eeffb

honor graceful shutdown settings in k8s deployment

fbbbb52

set backoffCap to 1 day

5f75861

explain background vs scheduled jobs settings

0ba4c01

renamed all arbiter backed scheduledjob* to job* and use backgroundjob*

20fab16

for the rabbit mq backed jobs

battermann requested a review from akshaymankar July 17, 2026 14:30

replaced remaining "scheduled ..." wording in docs, comments, logs...

42c8fea

Uh oh!

Conversation

battermann commented Jun 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

TODO

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

akshaymankar left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

akshaymankar Jul 16, 2026

Choose a reason for hiding this comment

Uh oh!

akshaymankar Jul 16, 2026

Choose a reason for hiding this comment

Uh oh!

akshaymankar Jul 16, 2026

Choose a reason for hiding this comment

Uh oh!

battermann Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

akshaymankar Jul 16, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

battermann commented Jun 25, 2026 •

edited

Loading