Metrics
A JobListener set with BikeschedBuilder.jobListener(…) observes the jobs a node runs, an
EventListener set with BikeschedBuilder.eventListener(…) the events published through it. The
module bikesched-micrometer implements both with Micrometer meters; the core stays free of
Micrometer.
Micrometer
Bikesched bikesched =
Bikesched.builder(dataSource)
.nodeName("node-1")
.jobListener(new MicrometerJobListener(registry, Tags.of("node", "node-1")))
.eventListener(new MicrometerEventListener(registry, Tags.of("node", "node-1")))
.build();| Meter | Type | Tags | Measures |
|---|---|---|---|
bikesched.job.claim.delay |
timer | job.name |
how long after they were due jobs were claimed; rising values mean the nodes fall behind |
bikesched.job.attempts |
timer | job.name, outcome |
attempts by recorded outcome and their duration |
bikesched.job.attempts.abandoned |
counter | job.name |
attempts abandoned by the hard timeout |
bikesched.jobs.active |
gauge | attempts being processed by the node | |
bikesched.jobs.abandoned |
gauge | abandoned attempts whose thread is still running | |
bikesched.events.published |
counter | event.name, source |
published events; source is node (bikesched.newEvent) or job (job.newEvent) |
bikesched.event.deliveries |
counter | event.name |
waiting steps that received an event when it was published |
outcome is the JobAttemptOutcome in lower case: succeeded, retrying, failed, cancelled,
waiting, postponed (the handler called job.postpone), compensating, compensated,
compensation_retrying, compensation_failed, failing (the job’s failure
handler runs next; its run then reports failed or
compensation_failed), failure_handler_retrying or not_recorded (the job was released while it
ran, or the database write failed; it runs again). Alert e.g. on failed and compensation_failed.
The event meters count an event once, when it is published; a step that starts waiting later and
receives a kept event (see Retention) is not a delivery.
Events published in the caller’s transaction (connection(…)) are not counted, see Own
listeners.
Every meter also carries the tags passed to the constructor. Use one listener per node, and give
nodes that share a registry a distinguishing tag such as the node name, otherwise their gauges
collide. The meters only cover what a node does itself; the number of due jobs across all nodes is
a query on bikesched_jobs (see Database).
Own listeners
JobListener has three methods, all doing nothing by default:
| Method | Called |
|---|---|
onJobClaimed(JobClaim) |
before a claimed job runs; JobClaim has the job id and name, the attempt, whether it is a compensation, and the claim delay |
onAttemptEnded(JobAttemptResult) |
once per claim, after its outcome was recorded; JobAttemptResult has the claim, the JobAttemptOutcome, the duration and isAbandoned() |
onAbandonedAttemptReturned(JobClaim) |
when the thread of an attempt abandoned by the hard timeout finally returns |
The methods are called synchronously on the job’s thread, so they must be fast and thread-safe. Exceptions they throw are logged and ignored. A job’s outcome is reported by the node that ran it after the database commit, so it is lost if that node dies in between.
EventListener has one method, doing nothing by default:
| Method | Called |
|---|---|
onEventPublished(EventPublication) |
once per published event, after it committed; EventPublication has the event id, its name, the EventSource (NODE or JOB) and the number of waiting steps that received it |
It is called synchronously on the publishing thread: before bikesched.newEvent(…).publish()
returns, and in a job when its publishing step first publishes the event (not when a later run of
the handler finds the step and returns the stored id). Events published in the caller’s
transaction (connection(…)) are not reported: bikesched does not see whether that
transaction commits or rolls back. The payload and the event key are not part of the
EventPublication, since they can be sensitive or high-cardinality. Exceptions are logged and
ignored; publish() still returns the event id.
A class can implement both JobListener and EventListener.