tlbmc/g3doc/components/pacemaker.md

# Pacemaker Service

<!--*
# Document freshness: For more information, see go/fresh-source.
freshness: { owner: 'tlbmc-dev' reviewed: '2026-08-17' }
*-->

## Purpose & Overview

The **Pacemaker** service is a high-reliability monitoring daemon
(`/usr/bin/tlbmc_pacemaker`) designed to supervise the execution of the primary
`bmcweb` process, enforce strict resource boundaries, and perform rate-limited,
self-healing service restarts (`systemctl restart bmcweb`) whenever anomalies or
health violations are detected.

By continuously evaluating process vitality, socket availability, and memory
consumption, Pacemaker ensures robust long-term availability of the BMC Redfish
management stack and prevents resource exhaustion.

--------------------------------------------------------------------------------

## Systemd Unit (`pacemaker.service`)

The standalone monitoring daemon is managed as a critical systemd service
configured in
[pacemaker.service](http://google3/third_party/milotic/external/cc/tlbmc/pacemaker/pacemaker.service).

*   **Description**: tlBMC Pacemaker for gBMCWeb
*   **After**: `bmcweb.service` (Ensures `bmcweb` initialization precedes
    supervision)
*   **ExecStart**: `/usr/bin/tlbmc_pacemaker`
*   **Restart**: `always` (Ensures automated self-recovery if the pacemaker
    daemon itself terminates)
*   **WantedBy**: `multi-user.target` (Activates automatically upon system boot)

--------------------------------------------------------------------------------

## Execution Architecture

The standalone executable entry point is established in
[pacemaker_main.cc](http://google3/third_party/milotic/external/cc/tlbmc/pacemaker/pacemaker_main.cc),
which instantiates the central `Pacemaker` health coordinator and initializes a
gRPC service runtime.

### Configuration Flags

Pacemaker behavior is tailored via configurable Abseil command-line flags:

*   **`--pacemaker_interval`**: Defines the polling frequency between routine
    health verifications (default: `5` minutes).
*   **`--port`**: Configures the local listening port (default: `2998`).

--------------------------------------------------------------------------------

## Execution Loop (`PerformChecks()`)

The core monitoring lifecycle is governed by periodic asynchronous execution of
`PerformChecks()` dispatched via `TaskScheduler`. During each evaluation cycle,
the daemon executes a rigorous sequence of diagnostic verifications against the
target process (`bmcweb`):

1.  **Service Activity Verification**: Validates that `bmcweb.service` maintains
    an active state via `systemctl is-active bmcweb`.
2.  **Port Listening Verification**: Confirms active socket availability on
    HTTPS port `443` via `ss -tulpn | grep 443`.
3.  **Process Identification**: Extracts the authoritative `Main PID` from
    `systemctl status bmcweb`.
4.  **Memory Footprint Verification**: Evaluates Resident Set Size (`VmRSS`)
    extracted from `/proc/<pid>/status`. If consumption breaches
    `kMemoryUsageThreshold` (`300` MB), corrective mitigation is initiated.
5.  **CPU Usage Tracking**: Measures real-time processor utilization by parsing
    single-iteration top snapshots (`top -n1`).
6.  **Monotonic Active Timestamp Tracking**: Queries
    `ActiveEnterTimestampMonotonic` via `systemctl show`. By comparing this
    monotonic active timestamp against historical records, Pacemaker detects
    unsupervised service restarts occurring outside of its direct control.

--------------------------------------------------------------------------------

## Self-Healing & Error Logging

When a diagnostic verification fails or an unauthorized transition is
identified, Pacemaker invokes standardized self-healing recovery actions managed
by `RecordError` and `RestartService`.

### Error Classifications (`ErrorType`)

Diagnostic anomalies are categorized and pushed into an internal circular
buffer:

*   **`kUnknown` (`0`)**: Unsupervised restart detected via shifted monotonic
    timestamps.
*   **`kServiceInactive` (`1`)**: Target systemd service dropped out of active
    status.
*   **`kPortNotListening` (`2`)**: Socket verification failed on HTTPS port 443.
*   **`kMemoryUsageAboveThreshold` (`3`)**: Memory consumption exceeded the 300
    MB boundary.

### Rate-Limited Recovery (`RestartService`)

Remediation is executed by issuing `systemctl restart bmcweb`. To prevent system
thrashing or cascading failures during chronic instability:

*   **Restart Loop Clamping**: Consecutive restart attempts are tracked and
    strictly clamped at `kMaxConsecutiveRestartAttempts` (`10`).
*   If the retry ceiling is breached, further restart attempts are aborted and
    an internal error is returned.

--------------------------------------------------------------------------------

## Telemetry Reporting (`GetMonitoringData()`)

Pacemaker aggregates structured health metadata into a `MonitoredData` payload,
periodically streaming formatted JSON telemetry directly to system syslog.

### Monitored Telemetry Schema

The JSON payload exposes comprehensive historical circular buffers and state
flags:

*   **`MemoryUsageRecentToOldest`**: Historical sequence of up to 12 recent
    memory consumption measurements (in bytes).
*   **`ErrorsRecentToOldest`**: Historical log of recent diagnostic anomalies
    including string-formatted `type` and timestamp.
*   **`CpuUsage`**: Most recent processor utilization percentage.
*   **`MemoryUsage`**: Current memory allocation footprint.
*   **`LastActiveTimestamp`**: Most recent monotonic execution entry timestamp.
*   **`LastResetTime`**: Timestamp of the most recent automated service
    recovery.
*   **`RestartTriggered`**: Boolean indicating active self-healing recovery
    state.
*   **`Pid`**: Active target process identification number.
*   **`ConsecutiveRestartAttempts`**: Current consecutive restart counter.
*   **`PacemakerSchedulerStats`**: Operational metrics from the underlying
    `TaskScheduler`.

--------------------------------------------------------------------------------

## Code References

*   **Main Header**:
    [pacemaker.h](http://google3/third_party/milotic/external/cc/tlbmc/pacemaker/pacemaker.h)
*   **Core Implementation**:
    [pacemaker.cc](http://google3/third_party/milotic/external/cc/tlbmc/pacemaker/pacemaker.cc)
*   **Executable Entry Point**:
    [pacemaker_main.cc](http://google3/third_party/milotic/external/cc/tlbmc/pacemaker/pacemaker_main.cc)
*   **Systemd Unit**:
    [pacemaker.service](http://google3/third_party/milotic/external/cc/tlbmc/pacemaker/pacemaker.service)