True Hitless - SPI image change (1)
SPI image Package installation change: https://gbmc-internal-review.git.corp.google.com/c/meta-google-gbmc/+/241086
True-hitless-image Change: https://gbmc-internal-review.git.corp.google.com/c/meta-google-gbmc/+/241855
1. Add emmc-unavilable receipe to create emmc-unavailable.target
2. Modified emmc-available receipe to make it trigger emmc-available-postprocess
3. Add emmc-postprocess scripts and services
When emmc is available: emmc-available.target ACTIVE
=> emmc-available-postprocessing.service
=> emmc-true-hitless-enable.sh: This script enables the true-hitless services by mounting the true-hitless image (as needed) and copying systemd service files from the eMMC to the system's configuration directory (as needed). If the eMMC was previously unavailable, it restarts the eMMC services.
When emmc is unavailable: emmc-unavailable.target ACTIVE
=> emmc-unavailable-postprocessing.service
=> emmc-true-hitless-disable.sh: This script disables the true-hitless services by removing the emmc-override.conf files that were created during the true-hitless installation. It then reloads the systemd daemon to apply the changes (as needed).
When emmc is available and emmc services are up and running, but one of the emmc services crashed.
emmc-services-fallback will be triggered =>
emmc-true-hitless-disable.sh =>
emmc-services-restart
[emmc-available-postprocessing.service will also be triggered during true-hitless POST_INSTALL to enable true-hitless overrides/services]
Change-Id: I840c1e8335ff22ff7d57923d9c78ee8da6a6e114
Fusion-Link: fusion2 N/A
Tested: Tested build SPI image with true-hitless-image with no issues
Google-Bug-Id: 445241073
Signed-off-by: Baicheng Zong <baichengz@google.com>
diff --git a/recipes-google/emmc/emmc-available/emmc-available.target b/recipes-google/emmc/emmc-available/emmc-available.target
index 4c4dc8c..46c2d36 100644
--- a/recipes-google/emmc/emmc-available/emmc-available.target
+++ b/recipes-google/emmc/emmc-available/emmc-available.target
@@ -1,2 +1,3 @@
[Unit]
Description=eMMC is unlocked and mounted
+Conflicts=emmc-unavailable.target
diff --git a/recipes-google/true-hitless/true-hitless.bb b/recipes-google/true-hitless/true-hitless.bb
new file mode 100644
index 0000000..58193f5
--- /dev/null
+++ b/recipes-google/true-hitless/true-hitless.bb
@@ -0,0 +1,60 @@
+SUMMARY = "gBMC True Hitless Update Support"
+DESCRIPTION = "Provides systemd targets and services for true hitless updates, including eMMC availability detection, postprocessing scripts, and initialization."
+LICENSE = "CLOSED"
+LIC_FILES_CHKSUM = ""
+
+SRC_URI = " \
+ file://emmc-unavailable.target \
+ file://emmc-available-postprocessing.service \
+ file://emmc-unavailable-postprocessing.service \
+ file://emmc-true-hitless-enable.sh \
+ file://emmc-true-hitless-disable.sh \
+ file://true-hitless-initialize.service \
+ file://true-hitless-initialize.sh \
+ file://block-override.conf \
+"
+
+S = "${WORKDIR}"
+
+# To find the files in ${BPN}/ subdirectory
+FILESEXTRAPATHS:prepend := "${THISDIR}/${BPN}:"
+
+inherit systemd
+
+DEPENDS += "systemd"
+
+SYSTEMD_PACKAGES = "${PN}"
+SYSTEMD_SERVICE:${PN} = " \
+ emmc-unavailable.target \
+ emmc-available-postprocessing.service \
+ emmc-unavailable-postprocessing.service \
+ true-hitless-initialize.service \
+"
+SYSTEMD_AUTO_ENABLE = "enable"
+
+FILES:${PN} += "${libdir}/true-hitless"
+
+do_install() {
+ # Replace @MACHINE@ placeholder with actual machine name in scripts
+ if [ -z "${MACHINE}" ]; then
+ bbfatal "MACHINE variable is not set"
+ fi
+ sed -i 's|@MACHINE@|${MACHINE}|g' ${S}/emmc-true-hitless-enable.sh
+
+ # Install systemd targets and services to /usr/lib/systemd/system (systemd_system_unitdir)
+ install -d ${D}${systemd_system_unitdir}
+ install -m 0644 ${S}/emmc-unavailable.target ${D}${systemd_system_unitdir}
+ install -m 0644 ${S}/emmc-available-postprocessing.service ${D}${systemd_system_unitdir}
+ install -m 0644 ${S}/emmc-unavailable-postprocessing.service ${D}${systemd_system_unitdir}
+ install -m 0644 ${S}/true-hitless-initialize.service ${D}${systemd_system_unitdir}
+
+ # Install scripts to /usr/bin (bindir)
+ install -d ${D}${bindir}
+ install -m 0755 ${S}/emmc-true-hitless-enable.sh ${D}${bindir}
+ install -m 0755 ${S}/emmc-true-hitless-disable.sh ${D}${bindir}
+ install -m 0755 ${S}/true-hitless-initialize.sh ${D}${bindir}
+
+ # Install block-override.conf template to /usr/lib/true-hitless
+ install -d ${D}${libdir}/true-hitless
+ install -m 0644 ${S}/block-override.conf ${D}${libdir}/true-hitless
+}
diff --git a/recipes-google/true-hitless/true-hitless/block-override.conf b/recipes-google/true-hitless/true-hitless/block-override.conf
new file mode 100644
index 0000000..5f9d550
--- /dev/null
+++ b/recipes-google/true-hitless/true-hitless/block-override.conf
@@ -0,0 +1,16 @@
+# True-hitless block-override configuration
+# Blocks True-hitless services from multi-user.target
+# Services will only start after one of the postprocessing services completes
+[Unit]
+After=emmc-available-postprocessing.service
+After=emmc-unavailable-postprocessing.service
+# Check for completion flags created by postprocessing scripts
+# Using subfolder /run/true-hitless/ for better organization
+# The -complete suffix indicates the postprocessing is complete
+ConditionPathExists=|/run/true-hitless/emmc-available-postprocessing-complete
+ConditionPathExists=|/run/true-hitless/emmc-unavailable-postprocessing-complete
+
+[Install]
+WantedBy=
+WantedBy=emmc-available.target
+WantedBy=emmc-unavailable.target
diff --git a/recipes-google/true-hitless/true-hitless/emmc-available-postprocessing.service b/recipes-google/true-hitless/true-hitless/emmc-available-postprocessing.service
new file mode 100644
index 0000000..fb6bacb
--- /dev/null
+++ b/recipes-google/true-hitless/true-hitless/emmc-available-postprocessing.service
@@ -0,0 +1,10 @@
+[Unit]
+Description=eMMC Available Postprocessing
+
+[Service]
+Type=oneshot
+RemainAfterExit=yes
+ExecStart=/usr/bin/emmc-true-hitless-enable.sh
+
+[Install]
+WantedBy=emmc-available.target
\ No newline at end of file
diff --git a/recipes-google/true-hitless/true-hitless/emmc-true-hitless-disable.sh b/recipes-google/true-hitless/true-hitless/emmc-true-hitless-disable.sh
new file mode 100644
index 0000000..4f90696
--- /dev/null
+++ b/recipes-google/true-hitless/true-hitless/emmc-true-hitless-disable.sh
@@ -0,0 +1,42 @@
+#!/bin/sh
+set -eu
+
+# Define all variables
+SYSTEMD_RUNTIME_DIR="/run/systemd/system"
+OVERRIDE_CONFS="emmc-override.conf block-override.conf"
+TRUE_HITLESS_FLAGS_DIR="/run/true-hitless"
+UNAVAILABLE_FLAG_FILE="${TRUE_HITLESS_FLAGS_DIR}/emmc-unavailable-postprocessing-complete"
+changes_made=0
+
+# Remove emmc-override.conf and block-override.conf files created by true-hitless installation
+# When eMMC is unavailable during reboot or when we need to fall back from true-hitless
+# services to the main SPI image (e.g., due to a crash), these override configurations
+# must be removed to restore normal service behavior/bins/libs using the main image.
+for service_dir in "$SYSTEMD_RUNTIME_DIR"/*.service.d; do
+ if [ -d "$service_dir" ]; then
+ for override_conf_name in $OVERRIDE_CONFS; do
+ override_conf="$service_dir/$override_conf_name"
+ if [ -f "$override_conf" ]; then
+ echo "Removing override conf: $override_conf" >&2
+ rm -f "$override_conf"
+ changes_made=1
+ fi
+ done
+
+ # If directory is now empty, remove it
+ if [ -z "$(ls -A "$service_dir")" ]; then
+ echo "Removing empty directory: $service_dir" >&2
+ rmdir "$service_dir"
+ fi
+ fi
+done
+
+if [ "$changes_made" -eq 1 ]; then
+ systemctl daemon-reload
+fi
+
+# Create flag file to indicate postprocessing is complete and active
+echo "Creating completion flag file..." >&2
+mkdir -p "${TRUE_HITLESS_FLAGS_DIR}"
+touch "${UNAVAILABLE_FLAG_FILE}"
+echo "eMMC unavailable postprocessing completed successfully" >&2
diff --git a/recipes-google/true-hitless/true-hitless/emmc-true-hitless-enable.sh b/recipes-google/true-hitless/true-hitless/emmc-true-hitless-enable.sh
new file mode 100644
index 0000000..300eee4
--- /dev/null
+++ b/recipes-google/true-hitless/true-hitless/emmc-true-hitless-enable.sh
@@ -0,0 +1,80 @@
+#!/bin/sh
+set -eu
+
+# Configuration and paths
+EMMC_MOUNT="/mnt/luks-mmcblk0_fs"
+TRUE_HITLESS_BASE="${EMMC_MOUNT}/true-hitless"
+TRUE_HITLESS_IMAGE="${TRUE_HITLESS_BASE}/true-hitless-image-@MACHINE@.squashfs-xz"
+TRUE_HITLESS_ROOTFS="${TRUE_HITLESS_BASE}/rootfs"
+TRUE_HITLESS_SYSTEMD_DIR="${TRUE_HITLESS_ROOTFS}/usr/lib/systemd/system"
+RUNTIME_SYSTEMD_DIR="/run/systemd/system"
+EMMC_SERVICE_FILES="emmc-services-restart.service emmc-services-fallback.service"
+TRUE_HITLESS_RESTART_SCRIPT="${TRUE_HITLESS_ROOTFS}/usr/bin/emmc-services-restart.sh"
+RUNTIME_RESTART_SCRIPT="/run/emmc-services-restart.sh"
+TRUE_HITLESS_FLAGS_DIR="/run/true-hitless"
+AVAILABLE_FLAG_FILE="${TRUE_HITLESS_FLAGS_DIR}/emmc-available-postprocessing-complete"
+changes_made=0
+
+# Check existence of true-hitless image for current machine
+if [ ! -f "${TRUE_HITLESS_IMAGE}" ]; then
+ echo "Error: True-hitless image not found: ${TRUE_HITLESS_IMAGE}" >&2
+ exit 1
+fi
+
+# Mount true-hitless image
+mkdir -p "${TRUE_HITLESS_ROOTFS}"
+if ! mountpoint -q "${TRUE_HITLESS_ROOTFS}"; then
+ mount -t squashfs "${TRUE_HITLESS_IMAGE}" "${TRUE_HITLESS_ROOTFS}"
+fi
+
+# 0. Version Compatibility Check
+# TODO: Implement version compatibility check between main SPI image and true-hitless image
+
+# Copy systemd files from true-hitless image
+echo "Copying systemd files from true-hitless image..." >&2
+
+# 1. Copy emmc-override.conf from each service .d directory to /run/systemd/system
+for service_d_dir in "${TRUE_HITLESS_SYSTEMD_DIR}"/*.service.d; do
+ if [ -d "${service_d_dir}" ]; then
+ service_d_name=$(basename "${service_d_dir}")
+ dst_service_d="${RUNTIME_SYSTEMD_DIR}/${service_d_name}"
+
+ # Create destination directory if it doesn't exist
+ mkdir -p "${dst_service_d}"
+
+ # Copy emmc-override.conf
+ if [ -f "${service_d_dir}/emmc-override.conf" ]; then
+ echo "Copying emmc-override.conf to ${service_d_name}..." >&2
+ cp "${service_d_dir}/emmc-override.conf" "${dst_service_d}/"
+ changes_made=1
+ fi
+ fi
+done
+
+# 2. Copy specific service files
+for service in ${EMMC_SERVICE_FILES}; do
+ if [ -f "${TRUE_HITLESS_SYSTEMD_DIR}/${service}" ]; then
+ echo "Copying ${service}..." >&2
+ cp "${TRUE_HITLESS_SYSTEMD_DIR}/${service}" "${RUNTIME_SYSTEMD_DIR}/"
+ changes_made=1
+ fi
+done
+
+# 3. Copy emmc-services-restart.sh script
+if [ -f "${TRUE_HITLESS_RESTART_SCRIPT}" ]; then
+ echo "Copying emmc-services-restart.sh..." >&2
+ cp "${TRUE_HITLESS_RESTART_SCRIPT}" "${RUNTIME_RESTART_SCRIPT}"
+ chmod +x "${RUNTIME_RESTART_SCRIPT}"
+ changes_made=1
+fi
+
+# Reload systemd daemon only if changes were made
+if [ "$changes_made" -eq 1 ]; then
+ systemctl daemon-reload
+fi
+
+# Create flag file to indicate postprocessing is complete and active
+echo "Creating completion flag file..." >&2
+mkdir -p "${TRUE_HITLESS_FLAGS_DIR}"
+touch "${AVAILABLE_FLAG_FILE}"
+echo "eMMC available postprocessing completed successfully" >&2
diff --git a/recipes-google/true-hitless/true-hitless/emmc-unavailable-postprocessing.service b/recipes-google/true-hitless/true-hitless/emmc-unavailable-postprocessing.service
new file mode 100644
index 0000000..eb068fd
--- /dev/null
+++ b/recipes-google/true-hitless/true-hitless/emmc-unavailable-postprocessing.service
@@ -0,0 +1,10 @@
+[Unit]
+Description=eMMC Unavailable Postprocessing
+
+[Service]
+Type=oneshot
+RemainAfterExit=yes
+ExecStart=/usr/bin/emmc-true-hitless-disable.sh
+
+[Install]
+WantedBy=emmc-unavailable.target
\ No newline at end of file
diff --git a/recipes-google/true-hitless/true-hitless/emmc-unavailable.target b/recipes-google/true-hitless/true-hitless/emmc-unavailable.target
new file mode 100644
index 0000000..c943335
--- /dev/null
+++ b/recipes-google/true-hitless/true-hitless/emmc-unavailable.target
@@ -0,0 +1,3 @@
+[Unit]
+Description=eMMC Unavailable Target
+Conflicts=emmc-available.target
\ No newline at end of file
diff --git a/recipes-google/true-hitless/true-hitless/true-hitless-initialize.service b/recipes-google/true-hitless/true-hitless/true-hitless-initialize.service
new file mode 100644
index 0000000..75a66a9
--- /dev/null
+++ b/recipes-google/true-hitless/true-hitless/true-hitless-initialize.service
@@ -0,0 +1,14 @@
+[Unit]
+Description=True Hitless Initialization
+DefaultDependencies=no
+After=local-fs.target
+ConditionPathIsDirectory=/var/google
+Before=sysinit.target
+
+[Service]
+Type=oneshot
+RemainAfterExit=yes
+ExecStart=/usr/bin/true-hitless-initialize.sh
+
+[Install]
+WantedBy=sysinit.target
\ No newline at end of file
diff --git a/recipes-google/true-hitless/true-hitless/true-hitless-initialize.sh b/recipes-google/true-hitless/true-hitless/true-hitless-initialize.sh
new file mode 100644
index 0000000..eae4081
--- /dev/null
+++ b/recipes-google/true-hitless/true-hitless/true-hitless-initialize.sh
@@ -0,0 +1,136 @@
+#!/bin/sh
+set -eu
+
+TRUE_HITLESS_SOURCE="/var/google/true-hitless"
+SERVICE_LIST_FILE="${TRUE_HITLESS_SOURCE}/true-hitless-service-list.txt"
+BLOCK_OVERRIDE_TEMPLATE="/usr/lib/true-hitless/block-override.conf"
+SYSTEMD_RUNTIME_DIR="/run/systemd/system"
+SYSTEMD_USR_LIB_DIR="/usr/lib/systemd/system"
+SYSTEMD_ETC_DIR="/etc/systemd/system"
+DISABLE_FROM_TARGETS="multi-user.target.wants"
+ENABLE_TO_TARGETS="emmc-available.target.wants emmc-unavailable.target.wants"
+changes_made=0
+
+################################################################################
+# 1. Verify true-hitless package installation and enablement
+################################################################################
+if [ ! -d "${TRUE_HITLESS_SOURCE}" ]; then
+ echo "True-hitless package is not installed at ${TRUE_HITLESS_SOURCE}" >&2
+ exit 0
+fi
+if [ ! -f "${SERVICE_LIST_FILE}" ]; then
+ echo "True-hitless is not enabled (true-hitless service list file not found at ${SERVICE_LIST_FILE})" >&2
+ exit 0
+fi
+echo "True-hitless is enabled" >&2
+echo "Starting true-hitless initialization..." >&2
+
+################################################################################
+# 2. Verify required files exist
+################################################################################
+if [ ! -f "${BLOCK_OVERRIDE_TEMPLATE}" ]; then
+ echo "Error: Block override template not found at ${BLOCK_OVERRIDE_TEMPLATE}" >&2
+ exit 1
+fi
+
+################################################################################
+# 3. Read and parse service list
+################################################################################
+# Service list file format: One service per line with .service suffix
+# Example:
+# phosphor-abc@.service
+# xyz.openbmc_project.abcd@.service
+# my-service.service
+# No comments, no empty lines, no extra whitespace
+################################################################################
+echo "Reading service list from ${SERVICE_LIST_FILE}..." >&2
+service_list=$(cat "${SERVICE_LIST_FILE}")
+
+if [ -z "$service_list" ]; then
+ echo "No services found in ${SERVICE_LIST_FILE}" >&2
+ exit 0
+fi
+echo "Services to process: $service_list" >&2
+
+################################################################################
+# 4. Process each service
+################################################################################
+# Create target directories if they don't exist
+for target in $ENABLE_TO_TARGETS; do
+ mkdir -p "${SYSTEMD_ETC_DIR}/$target"
+done
+
+for service in $service_list; do
+ echo "Processing service: ${service}" >&2
+
+ service_override_dir="${SYSTEMD_RUNTIME_DIR}/${service}.d"
+ mkdir -p "$service_override_dir"
+
+ block_override_file="${service_override_dir}/block-override.conf"
+ echo " Installing block-override.conf" >&2
+ cp "${BLOCK_OVERRIDE_TEMPLATE}" "${block_override_file}"
+
+ # Find the service unit file
+ service_unit_path="${SYSTEMD_USR_LIB_DIR}/${service}"
+ if [ ! -f "$service_unit_path" ]; then
+ service_unit_path="${SYSTEMD_ETC_DIR}/${service}"
+ fi
+
+ if [ ! -f "$service_unit_path" ]; then
+ echo " Warning: Service unit file not found: ${service}" >&2
+ continue
+ fi
+
+ case "$service" in
+ *@.service)
+ echo " Template service detected, finding instances..." >&2
+ instances=$(systemctl list-units --all --no-legend "${service%@.service}@*.service" 2>/dev/null | awk '{print $1}')
+
+ if [ -z "$instances" ]; then
+ echo " No instances found" >&2
+ continue
+ fi
+ ;;
+ *)
+ echo " Regular service detected" >&2
+ # Treat regular service as a single instance
+ instances="$service"
+ ;;
+ esac
+
+ # Process all services
+ for instance in $instances; do
+ echo " Processing instance: $instance" >&2
+
+ # Remove symbolic links from disable targets
+ for target in $DISABLE_FROM_TARGETS; do
+ for systemd_dir in "$SYSTEMD_ETC_DIR" "$SYSTEMD_USR_LIB_DIR"; do
+ target_link="${systemd_dir}/$target/$instance"
+ if [ -L "$target_link" ]; then
+ echo " Disabling from $target: $instance (${systemd_dir})" >&2
+ rm "$target_link"
+ fi
+ done
+ done
+
+ # Add symbolic links to enable targets
+ for target in $ENABLE_TO_TARGETS; do
+ target_link="${SYSTEMD_ETC_DIR}/$target/$instance"
+ if [ ! -L "$target_link" ]; then
+ echo " Enabling to $target -> $instance" >&2
+ ln -sf "$service_unit_path" "$target_link"
+ fi
+ done
+ done
+
+ changes_made=1
+done
+
+# Reload systemd if any changes were made
+if [ "$changes_made" -eq 1 ]; then
+ echo "Reloading systemd daemon..." >&2
+ systemctl daemon-reload
+ echo "True-hitless initialization completed successfully" >&2
+else
+ echo "No services processed" >&2
+fi
