TH scripts improvments for resolving SHM issue Key Changes 1. Logic Migration MSV Scripts: Removed SHM (Shared Memory) removal and IPMI restart operations from MSV scripts, including pre-rm and postinstall. TH Scripts: Centralized these operations within the TH post-processing scripts: emmc-available-postprocessing emmc-unavailable-postprocessing 2. TH Upgrade Scenario Logic Operations are abstracted into a transition model between TH1 (old version/list) and TH2 (new version/list): First-time Onboarding: TH1 (empty) → TH2 Removal: TH1 → TH2 (empty) Upgrade/Downgrade: TH1 → TH2 3. eMMC run-time availability status change logic available → unavailable: Similar logic as TH1 -> TH2 (empty), but keep TH package existing in the system for both eMMC and /var/google. unavailable → available: Similar logic as TH1 (empty) -> TH2, but without copying/pasting work for TH packages Prerequisite for Execution: When either TH1 or TH2 includes bmcweb daemons, the following sequence is triggered (if bmcweb is not included in both TH, the following logic will be skipped): Clear SHM. Restart all daemons (Union of TH1 and TH2). Restart IPMI. 4. Previous block-override.conf has been discarded and the new content has been updated through recipes-google/true-hitless/true-hitless/true-hitless-initialize.sh. New content: "[Unit] ConditionPathExists=|/run/true-hitless/emmc-available-postprocessing-complete ConditionPathExists=|/run/true-hitless/emmc-unavailable-postprocessing-complete" Diff: 1. removed After=emmc-available-postprocessing so that we can restart TH daemons inside of emmc-available-postprocessing. 2. added ConditionPathExists=|/run/true-hitless/emmc-unavailable-postprocessing-complete constrain to avoid double initialization when eMMC is unavailable since reboot. Platforms-Affected: EMR, GF Tested: Detailed tested steps is in cl/908884346. Will also be tested through TH nightly/weekly test suite Google-Bug-Id: 492591099 Fusion-Link: fusion2 https://fusion2.corp.google.com/3a1ed5b5-ddce-4667-8687-a939dff5aff6 , tested experimental package - https://msvbrowse.corp.google.com/versionz?hash=1-713fb0f5_7f3d8598_4fd97d78_52e0eb86_f812b17b_a8707ec0_3ee9b308_d2df21c6 Change-Id: I7dc047d8cd2dd996d70c47e87c54d8dd9b75b979
diff --git a/recipes-google/true-hitless/true-hitless/emmc-services-restart.sh b/recipes-google/true-hitless/true-hitless/emmc-services-restart.sh index caae6fc..16eceb7 100644 --- a/recipes-google/true-hitless/true-hitless/emmc-services-restart.sh +++ b/recipes-google/true-hitless/true-hitless/emmc-services-restart.sh
@@ -1,17 +1,195 @@ #!/bin/bash +# +# eMMC Services Restart Script +# +# This script handles restarting services in the True Hitless list. +# It also handles shared memory clearing for bmcweb and chain actions +# if bmcweb is restarted. +# It detects an upgrade by checking for an old service list file. +set -eu + +################################################################################ +# 1. Variables, Constants and Initialization +################################################################################ + +# 1.1 Configuration Paths SERVICE_LIST_FILE="/var/google/true-hitless/true-hitless-service-list.txt" +OLD_SERVICE_LIST_FILE="/run/true-hitless/old-true-hitless-service-list.txt" +SYSTEMD_RUNTIME_OVERRIDE_DIR="/run/systemd/system" +PHOSPHOR_IPMI_NET_SERVICE="phosphor-ipmi-net@gbmcbr.service" +TLBMC_STATIC_SHM_INITIALIZED_FILE="/run/tlbmc/static_shm_initialized" +TLBMC_STATIC_SHM_FILE="/dev/shm/TlbmcStaticSharedMemory" + +# 1.2 Ensure service list exists before proceeding [ -f "$SERVICE_LIST_FILE" ] || { echo "Error: Service list file not found: $SERVICE_LIST_FILE" >&2; exit 0; } -# List of services to restart -EMMC_SERVICES="$(cat $SERVICE_LIST_FILE)" +# 1.3 Service Lists Content +# CONFIGURED_SERVICES: Services listed in the current configuration file. +# In upgrade/onboarding, this represents the target state. +# In uninstall, this represents the services to be reverted to SPI. +CONFIGURED_SERVICES="$(cat $SERVICE_LIST_FILE)" +[ -z "$CONFIGURED_SERVICES" ] && { echo "Empty service list" >&2; exit 0; } -[ -z "$EMMC_SERVICES" ] && { echo "Empty service list" >&2; exit 0; } +# 1.4 Mode Detection and Conditional Variables +# PREVIOUS_SERVICES: Services listed in the previous configuration file (if upgrading). +PREVIOUS_SERVICES="" +MODE="NORMAL" -service_restart_list="${EMMC_SERVICES//@.service/@*.service}" +# REMOVED_SERVICES: Services present in the previous list but not in the configured list. +# These services need to have their overrides removed and be reverted to SPI. +# This is only applicable during an upgrade. +REMOVED_SERVICES="" -echo "restarting services $service_restart_list" >&2 +if [ -f "$OLD_SERVICE_LIST_FILE" ]; then + MODE="UPGRADE" + PREVIOUS_SERVICES="$(cat $OLD_SERVICE_LIST_FILE)" + echo "Mode: UPGRADE" >&2 +else + echo "Mode: NORMAL" >&2 +fi -systemctl restart $service_restart_list || exit 1 +# 1.5 Global State Flags +has_bmcweb=false +need_daemon_reload=0 + +################################################################################ +# 2. List Comparison and Override Cleanup (Upgrade only) +################################################################################ + +if [ "$MODE" = "UPGRADE" ]; then + echo "Old services: $(echo $PREVIOUS_SERVICES)" >&2 + echo "New services: $(echo $CONFIGURED_SERVICES)" >&2 + + # 2.1 Find services in previous list but not in configured list + while IFS= read -r service || [[ -n "$service" ]]; do + [ -z "$service" ] && continue + if ! grep -q -x -F "$service" "$SERVICE_LIST_FILE"; then + REMOVED_SERVICES="$REMOVED_SERVICES $service" + fi + done < "$OLD_SERVICE_LIST_FILE" + + # 2.2 Clean up overrides for removed services + for service in $REMOVED_SERVICES; do + [ -z "$service" ] && continue + echo "Service ${service} was removed from true-hitless list. Removing block-override.conf and emmc-override.conf..." >&2 + + # Expand template services (bashism supported in this environment) + expanded_service="${service//@.service/@*.service}" + + for svc_dir in "${SYSTEMD_RUNTIME_OVERRIDE_DIR}"/${expanded_service}.d; do + if [ -d "${svc_dir}" ]; then + # Remove specific true-hitless overrides instead of the whole directory + if [ -f "${svc_dir}/block-override.conf" ]; then + rm -f "${svc_dir}/block-override.conf" + need_daemon_reload=1 + fi + + if [ -f "${svc_dir}/emmc-override.conf" ]; then + rm -f "${svc_dir}/emmc-override.conf" + need_daemon_reload=1 + fi + + # Remove directory only if it is now empty + if [ -d "${svc_dir}" ] && [ -z "$(ls -A "${svc_dir}")" ]; then + rmdir "${svc_dir}" + fi + fi + done + done + + # 2.3 Reload systemd daemon to pick up removed overrides before restarting services + if [ "$need_daemon_reload" -eq 1 ]; then + echo "Reloading systemd daemon..." >&2 + systemctl daemon-reload + fi +fi + +################################################################################ +# 3. Determine bmcweb Involvement +################################################################################ + +search_list="$CONFIGURED_SERVICES" +if [ "$MODE" = "UPGRADE" ]; then + search_list="$REMOVED_SERVICES $CONFIGURED_SERVICES" +fi + +if [[ "$search_list" == *"bmcweb.service"* ]]; then + has_bmcweb=true +fi + +################################################################################ +# 4. Clear Shared Memory (if needed) +################################################################################ + +if [ "$has_bmcweb" = true ]; then + echo "bmcweb involved, removing shared memory files: $TLBMC_STATIC_SHM_INITIALIZED_FILE and $TLBMC_STATIC_SHM_FILE" >&2 + rm -f "$TLBMC_STATIC_SHM_INITIALIZED_FILE" + rm -f "$TLBMC_STATIC_SHM_FILE" +fi + +################################################################################ +# 5. Restart Services +################################################################################ + +if [ "$MODE" = "UPGRADE" ]; then + # 5.1 Collect and restart removed services in batch to revert to SPI (Upgrade mode) + if [ -n "$REMOVED_SERVICES" ]; then + removed_batch="" + for service in $REMOVED_SERVICES; do + [ -z "$service" ] && continue + removed_batch="$removed_batch ${service//@.service/@*.service}" + done + if [ -n "$removed_batch" ]; then + echo "Restarting removed services (batch):$removed_batch" >&2 + systemctl restart $removed_batch || exit 1 + fi + fi + + # 5.2 Collect and restart configured services in batch (Upgrade mode) + configured_batch="" + for service in $CONFIGURED_SERVICES; do + [ -z "$service" ] && continue + configured_batch="$configured_batch ${service//@.service/@*.service}" + done + if [ -n "$configured_batch" ]; then + echo "Restarting configured services (batch):$configured_batch" >&2 + systemctl restart $configured_batch || exit 1 + fi +else + # Normal mode: Restart all configured services in one batch + configured_batch="" + for service in $CONFIGURED_SERVICES; do + [ -z "$service" ] && continue + configured_batch="$configured_batch ${service//@.service/@*.service}" + done + if [ -n "$configured_batch" ]; then + echo "Restarting services (batch):$configured_batch" >&2 + systemctl restart $configured_batch || exit 1 + fi +fi + + +################################################################################ +# 6. Perform Chain Actions for bmcweb Restart (if needed) +################################################################################ + +if [ "$has_bmcweb" = true ]; then + echo "Sleeping 60 seconds to allow sensors to restart after bmcweb restart..." >&2 + sleep 60 + + # Check if service exists before restarting + if systemctl status "${PHOSPHOR_IPMI_NET_SERVICE}" > /dev/null 2>&1 || [ $? -ne 4 ]; then + echo "Restarting ${PHOSPHOR_IPMI_NET_SERVICE} for true-hitless..." >&2 + systemctl restart "${PHOSPHOR_IPMI_NET_SERVICE}" + else + echo "${PHOSPHOR_IPMI_NET_SERVICE} does not exist. Skipping restart." >&2 + fi +fi + +################################################################################ +# 7. Cleanup and Exit +################################################################################ echo "eMMC services restart: Completed successfully" >&2 +exit 0
diff --git a/recipes-google/true-hitless/true-hitless/emmc-true-hitless-disable.sh b/recipes-google/true-hitless/true-hitless/emmc-true-hitless-disable.sh index 5bde297..7761408 100644 --- a/recipes-google/true-hitless/true-hitless/emmc-true-hitless-disable.sh +++ b/recipes-google/true-hitless/true-hitless/emmc-true-hitless-disable.sh
@@ -18,10 +18,22 @@ RUNTIME_UNAVAILABLE_COMPLETION_FLAG="${RUNTIME_STATUS_DIR}/emmc-unavailable-postprocessing-complete" changes_made=0 -# Remove emmc-override.conf and block-override.conf files created by true-hitless installation -# When eMMC is unavailable during reboot or when we need to fall back from true-hitless -# services to the main SPI image (e.g., due to a crash), these override configurations -# must be removed to restore normal service behavior/bins/libs using the main image. +# 1. Stop status update service to release image file +echo "Stopping true-hitless-status-update.service..." >&2 +SYSTEMD_IGNORE_CHROOT=1 systemctl stop true-hitless-status-update.service || true + +# 2. Stop all True Hitless services to ensure files are not busy during unmount +echo "Stopping all true-hitless services..." >&2 +if [ -f "$PERSISTENT_SERVICE_LIST_FILE" ]; then + for service in $(cat "$PERSISTENT_SERVICE_LIST_FILE"); do + [ -z "$service" ] && continue + expanded_service="${service//@.service/@*.service}" + echo "Stopping service: $expanded_service" >&2 + systemctl stop "$expanded_service" || true + done +fi + +# 3. Remove emmc-override.conf and block-override.conf files created by true-hitless installation for service_dir in "$SYSTEMD_RUNTIME_OVERRIDE_DIR"/*.service.d; do if [ -d "$service_dir" ]; then for override_conf_name in $SYSTEMD_OVERRIDE_CONF_FILES; do @@ -41,27 +53,7 @@ fi done -echo "Changing target back to multi-user.target if necessary" >&2 -for service in $(cat $PERSISTENT_SERVICE_LIST_FILE); do - service_path=/etc/systemd/system/emmc-available.target.wants/${service//@.service/@*.service} - ls $service_path >/dev/null || continue - echo "recover service $service_path" >&2 - cp -P $service_path /etc/systemd/system/multi-user.target.wants/ - rm -rf $service_path - changes_made=1 -done - -if [ "$changes_made" -eq 1 ]; then - echo "Reload and restart true hitless services..." >&2 - systemctl daemon-reload - systemctl restart emmc-services-restart.service -fi - -# Create flag file to indicate postprocessing is complete and active -echo "Creating completion flag file..." >&2 -mkdir -p "${RUNTIME_STATUS_DIR}" -touch "${RUNTIME_UNAVAILABLE_COMPLETION_FLAG}" - +# 4. Unmount eMMC image rootfs if mountpoint -q "${EMMC_TRUE_HITLESS_BASE_DIR}/rootfs"; then echo "unmount eMMC image rootfs" >&2 umount "$EMMC_TRUE_HITLESS_IMAGE_FILE" @@ -69,4 +61,16 @@ echo "eMMC image rootfs not mounted, skipping unmount." >&2 fi +# 5. Create flag file to indicate postprocessing is complete and active +echo "Creating completion flag file..." >&2 +mkdir -p "${RUNTIME_STATUS_DIR}" +touch "${RUNTIME_UNAVAILABLE_COMPLETION_FLAG}" + +# 6. Reload and restart true hitless services (they will start from SPI now) +if [ "$changes_made" -eq 1 ]; then + echo "Reload and restart true hitless services..." >&2 + systemctl daemon-reload + systemctl restart emmc-services-restart.service +fi + echo "eMMC unavailable postprocessing completed successfully" >&2
diff --git a/recipes-google/true-hitless/true-hitless/emmc-true-hitless-enable.sh b/recipes-google/true-hitless/true-hitless/emmc-true-hitless-enable.sh index 80ecffc..1568987 100644 --- a/recipes-google/true-hitless/true-hitless/emmc-true-hitless-enable.sh +++ b/recipes-google/true-hitless/true-hitless/emmc-true-hitless-enable.sh
@@ -103,8 +103,19 @@ systemctl daemon-reload fi -# Create flag file to indicate postprocessing is complete and active +# Create flag file to indicate postprocessing is complete and active. +# This is placed BEFORE the restart of services to avoid a deadlock during +# first-time onboarding. On first onboarding, services are active (running on SPI) +# and targeted for restart. However, they are blocked by the ConditionPathExists +# directive in block-override.conf which requires this flag file to exist. +# Placing this here satisfies the condition, allowing services to restart successfully. +# On boot, services are blocked by the flag file until this script runs, ensuring +# they start only after all overrides are placed. echo "Creating completion flag file..." >&2 mkdir -p "${RUNTIME_STATUS_DIR}" touch "${RUNTIME_AVAILABLE_COMPLETION_FLAG}" + +# Restart true-hitless services to apply overrides. +echo "Restarting true-hitless services..." >&2 +systemctl restart emmc-services-restart.service echo "eMMC available postprocessing completed successfully" >&2
diff --git a/recipes-google/true-hitless/true-hitless/true-hitless-initialize.sh b/recipes-google/true-hitless/true-hitless/true-hitless-initialize.sh index c7da346..97664bb 100644 --- a/recipes-google/true-hitless/true-hitless/true-hitless-initialize.sh +++ b/recipes-google/true-hitless/true-hitless/true-hitless-initialize.sh
@@ -5,10 +5,20 @@ # Persistent configuration paths PERSISTENT_TRUE_HITLESS_CONFIG_DIR="/var/google/true-hitless" PERSISTENT_SERVICE_LIST_FILE="${PERSISTENT_TRUE_HITLESS_CONFIG_DIR}/true-hitless-service-list.txt" -# Default template paths -DEFAULT_BLOCK_OVERRIDE_TEMPLATE="/usr/lib/true-hitless/block-override.conf" # Systemd configuration directories SYSTEMD_RUNTIME_OVERRIDE_DIR="/run/systemd/system" +# Content for block-override.conf +BLOCK_OVERRIDE_CONTENT="# True-hitless block-override configuration +# Blocks True-hitless services from multi-user.target +# Services will only start after one of the postprocessing services completes +[Unit] +# Check for completion flags created by postprocessing scripts +# Using subfolder /run/true-hitless/ for better organization +# The -complete suffix indicates the postprocessing is complete +# Removed After=emmc-available-postprocessing.service to avoid deadlock during install +ConditionPathExists=|/run/true-hitless/emmc-available-postprocessing-complete +ConditionPathExists=|/run/true-hitless/emmc-unavailable-postprocessing-complete" +# Systemd folder paths SYSTEMD_USR_LIB_DIR="/usr/lib/systemd/system" SYSTEMD_ETC_DIR="/etc/systemd/system" # Systemd target configuration @@ -30,16 +40,9 @@ echo "True-hitless is enabled" >&2 echo "Starting true-hitless initialization..." >&2 -################################################################################ -# 2. Verify required files exist -################################################################################ -if [ ! -f "${DEFAULT_BLOCK_OVERRIDE_TEMPLATE}" ]; then - echo "Error: Block override template not found at ${DEFAULT_BLOCK_OVERRIDE_TEMPLATE}" >&2 - exit 1 -fi ################################################################################ -# 3. Read and parse service list +# 2. Read and parse service list ################################################################################ # Service list file format: One service per line with .service suffix # Example: @@ -58,7 +61,7 @@ echo "Services to process: $service_list" >&2 ################################################################################ -# 4. Process each service +# 3. Process each service ################################################################################ # Create target directories if they don't exist for target in $SYSTEMD_ENABLE_TO_TARGETS; do @@ -72,8 +75,8 @@ mkdir -p "$service_override_dir" block_override_file="${service_override_dir}/block-override.conf" - echo " Installing block-override.conf" >&2 - cp "${DEFAULT_BLOCK_OVERRIDE_TEMPLATE}" "${block_override_file}" + echo " Generating block-override.conf" >&2 + echo "$BLOCK_OVERRIDE_CONTENT" > "$block_override_file" # Find the service unit file service_unit_path="${SYSTEMD_USR_LIB_DIR}/${service}" @@ -106,26 +109,6 @@ # Process all services for instance in $instances; do echo " Processing instance: $instance" >&2 - - # Remove symbolic links from disable targets - for target in $SYSTEMD_DISABLE_FROM_TARGETS; do - for systemd_dir in "$SYSTEMD_ETC_DIR" "$SYSTEMD_USR_LIB_DIR"; do - target_link="${systemd_dir}/$target/$instance" - if [ -L "$target_link" ]; then - echo " Disabling from $target: $instance (${systemd_dir})" >&2 - rm "$target_link" - fi - done - done - - # Add symbolic links to enable targets - for target in $SYSTEMD_ENABLE_TO_TARGETS; do - target_link="${SYSTEMD_ETC_DIR}/$target/$instance" - if [ ! -L "$target_link" ]; then - echo " Enabling to $target -> $instance" >&2 - ln -sf "$service_unit_path" "$target_link" - fi - done done changes_made=1
diff --git a/recipes-google/true-hitless/true-hitless/true-hitless-status-update.sh b/recipes-google/true-hitless/true-hitless/true-hitless-status-update.sh index 2a08d36..c12a0fe 100644 --- a/recipes-google/true-hitless/true-hitless/true-hitless-status-update.sh +++ b/recipes-google/true-hitless/true-hitless/true-hitless-status-update.sh
@@ -475,7 +475,7 @@ local services_json="[]" if [ -f "$PERSISTENT_SERVICE_LIST_FILE" ]; then - while IFS= read -r service_name; do + while IFS= read -r service_name || [[ -n "$service_name" ]]; do # Check if this is a template service (ends with @.service) if [[ "$service_name" == *@.service ]]; then # Template service - find all instances