32. Waking pollers with batched notifications after the commit
Date: 2026-10-08
Status
Accepted
Context
A due job waited for the next poll of a node: up to one pollInterval (5 seconds by default) for
a new job, a job resumed by an event or a child job, a retry, an ended sleep, a poll check, a wait
deadline or the next occurrence of a recurring job. Reacting faster meant a shorter interval, i.e.
more claim queries on every node, also while idle.
PostgreSQL’s LISTEN/NOTIFY can wake the nodes, but a transaction that issues NOTIFY takes a
database-wide lock at commit that is held through the WAL flush (async.c, PreCommit_Notify;
unchanged in PostgreSQL 19). Measured with pgbench on PostgreSQL 18: notifying commits cap at about
5,000 per second whatever the number of clients (writes that don’t notify reach 30,000–36,000),
while one connection sending 100 notifications per second next to them costs nothing measurable.
Alternatives considered (details and measurements in the
spec): a trigger on bikesched_jobs (atomic and
complete, but every due-now write takes the commit lock, also when nobody listens, and switching it
off needs a database setting or runtime DDL); pg_notify in each writing statement or right after
each commit (the same lock per write); notifying due-later work as well (every suspension would
take the lock); timer and next-due query only, without LISTEN (due-now work keeps the interval’s
latency); one elected node keeping the timer; an in-memory queue of all known due times instead of
the next-due query (incomplete by construction); the job id or node id as payload; pg_cron.
Decision
bikesched notifies itself, batched after the commit, and only about work due now:
- The stores report the jobs their writes make due on their
Sqlhandle (dueNow(type),dueAt(type, time));DatabaseAccesspasses them to the node’sDueJobsafter the commit, so rolled-back writes report nothing. A guard test lists every store statement that inserts or updates jobs as notifying or silent. DueJobsNotifierwakes the node’s own poller right away and collects the types made due now; the first type of an empty batch schedules one flush afternotifyInterval(100 ms), which sendsselect pg_notify('bikesched_jobs', t) from unnest(?::text[]) tin an auto-commit statement of its own. A node sends at most one notifying commit per interval; an idle node sends none.- Writes on the caller’s connection notify inside the caller’s transaction instead, since bikesched does not see that commit: sent on commit, not on rollback.
- Due-later times written by a node (suspensions, retries with a delay, compensation, occurrences,
delayed jobs) only arm that node’s one wake-up timer, kept on the poller thread. Before every
interval and timer poll, a bounded next-due query (
scheduled_atwithin the next poll interval, on the existing poll index) arms it for due times the node did not write. JobNotificationListenerholds one connection per node, borrowed on its own thread, and wakes the poller for notifications of the types the node handles. It reconnects after failures and checks after each connect that its own notification arrives (warning only).- A wake-up that finds no free slot makes the next released slot poll.
listenForDueJobs(true)by default;pollIntervalgoes up to 10 seconds as the safety net.
Consequences
- Due-now work starts within about
notifyIntervalon other nodes, own work right away, delayed work of the node when due; delayed jobs scheduled by another node (or on a caller’s connection) start up to onepollIntervallate; writes by plain SQL start on the next interval poll. - Job writes never take the notify commit lock. A lost notification (node dies between commit and
flush, failed flush) costs up to one
pollInterval. - Every write path that makes jobs due now must report them: a forgotten path is silently up to one
interval slower. The guard test and an
AGENTS.mdrule cover the stores; SQL built at runtime is not covered. - Each listening node holds one pool connection; PgBouncer in transaction mode silently drops
LISTEN(the self-check warns), so such setups turn listening off and lower the interval. - Core requires the PostgreSQL driver at compile time (
requires org.postgresql.jdbc, an automatic module) forPGConnection.getNotifications. - The poller, the listener and the timer live in the new package
internal.job.claim, which usesinternal.jobbut not the other way round;internal.jobreaches the timer throughDueTimes. - Follow-ups (adaptive poll interval, next-due query inside the claim, claiming
limit + 1, jitter, claim reports) are described in the spec.