Pacemaker Service

Purpose & Overview

The Pacemaker service is a high-reliability monitoring daemon (/usr/bin/tlbmc_pacemaker) designed to supervise the execution of the primary bmcweb process, enforce strict resource boundaries, and perform rate-limited, self-healing service restarts (systemctl restart bmcweb) whenever anomalies or health violations are detected.

By continuously evaluating process vitality, socket availability, and memory consumption, Pacemaker ensures robust long-term availability of the BMC Redfish management stack and prevents resource exhaustion.

Systemd Unit (pacemaker.service)

The standalone monitoring daemon is managed as a critical systemd service configured in pacemaker.service.

  • Description: tlBMC Pacemaker for gBMCWeb
  • After: bmcweb.service (Ensures bmcweb initialization precedes supervision)
  • ExecStart: /usr/bin/tlbmc_pacemaker
  • Restart: always (Ensures automated self-recovery if the pacemaker daemon itself terminates)
  • WantedBy: multi-user.target (Activates automatically upon system boot)

Execution Architecture

The standalone executable entry point is established in pacemaker_main.cc, which instantiates the central Pacemaker health coordinator and initializes a gRPC service runtime.

Configuration Flags

Pacemaker behavior is tailored via configurable Abseil command-line flags:

  • --pacemaker_interval: Defines the polling frequency between routine health verifications (default: 5 minutes).
  • --port: Configures the local listening port (default: 2998).

Execution Loop (PerformChecks())

The core monitoring lifecycle is governed by periodic asynchronous execution of PerformChecks() dispatched via TaskScheduler. During each evaluation cycle, the daemon executes a rigorous sequence of diagnostic verifications against the target process (bmcweb):

  1. Service Activity Verification: Validates that bmcweb.service maintains an active state via systemctl is-active bmcweb.
  2. Port Listening Verification: Confirms active socket availability on HTTPS port 443 via ss -tulpn | grep 443.
  3. Process Identification: Extracts the authoritative Main PID from systemctl status bmcweb.
  4. Memory Footprint Verification: Evaluates Resident Set Size (VmRSS) extracted from /proc/<pid>/status. If consumption breaches kMemoryUsageThreshold (300 MB), corrective mitigation is initiated.
  5. CPU Usage Tracking: Measures real-time processor utilization by parsing single-iteration top snapshots (top -n1).
  6. Monotonic Active Timestamp Tracking: Queries ActiveEnterTimestampMonotonic via systemctl show. By comparing this monotonic active timestamp against historical records, Pacemaker detects unsupervised service restarts occurring outside of its direct control.

Self-Healing & Error Logging

When a diagnostic verification fails or an unauthorized transition is identified, Pacemaker invokes standardized self-healing recovery actions managed by RecordError and RestartService.

Error Classifications (ErrorType)

Diagnostic anomalies are categorized and pushed into an internal circular buffer:

  • kUnknown (0): Unsupervised restart detected via shifted monotonic timestamps.
  • kServiceInactive (1): Target systemd service dropped out of active status.
  • kPortNotListening (2): Socket verification failed on HTTPS port 443.
  • kMemoryUsageAboveThreshold (3): Memory consumption exceeded the 300 MB boundary.

Rate-Limited Recovery (RestartService)

Remediation is executed by issuing systemctl restart bmcweb. To prevent system thrashing or cascading failures during chronic instability:

  • Restart Loop Clamping: Consecutive restart attempts are tracked and strictly clamped at kMaxConsecutiveRestartAttempts (10).
  • If the retry ceiling is breached, further restart attempts are aborted and an internal error is returned.

Telemetry Reporting (GetMonitoringData())

Pacemaker aggregates structured health metadata into a MonitoredData payload, periodically streaming formatted JSON telemetry directly to system syslog.

Monitored Telemetry Schema

The JSON payload exposes comprehensive historical circular buffers and state flags:

  • MemoryUsageRecentToOldest: Historical sequence of up to 12 recent memory consumption measurements (in bytes).
  • ErrorsRecentToOldest: Historical log of recent diagnostic anomalies including string-formatted type and timestamp.
  • CpuUsage: Most recent processor utilization percentage.
  • MemoryUsage: Current memory allocation footprint.
  • LastActiveTimestamp: Most recent monotonic execution entry timestamp.
  • LastResetTime: Timestamp of the most recent automated service recovery.
  • RestartTriggered: Boolean indicating active self-healing recovery state.
  • Pid: Active target process identification number.
  • ConsecutiveRestartAttempts: Current consecutive restart counter.
  • PacemakerSchedulerStats: Operational metrics from the underlying TaskScheduler.

Code References