Dell BIOS 복구 - Dell BIOS boggu

Instructions

This article details how to recover from a "No POST" issue caused by a corrupted BIOS or interrupted BIOS update.
 


Table of Contents

  • Description
  • Issue Identification
  • Solution
  • Troubleshooting

Description


This article details how to recover from a "No POST" issue caused by a corrupted BIOS or interrupted BIOS update.

It is not recommended to power cycle or otherwise interrupt a system BIOS update while it is in progress. This procedure is likely to cause a "No power, No POST" issue, where the system does not power on when the power button is pressed.

Dell EMC PowerEdge Servers, beginning with systems which support Intel Xeon Scalable Processors (formerly Skylake) and second Generation Intel Xeon Scalable Processors (formerly Cascade Lake), have a built-in BIOS recovery process.

This BIOS recovery process may be able recover systems where a BIOS update was interrupted or the BIOS was found corrupt and the system was hanging as described above. Follow the steps in this article to begin the BIOS recovery process.


Issue Identification

To verify that a BIOS update interruption occurred, check the server's LifeCycle Controller (LCC) logs for a similar sequence of events. You can find out how to access the LCC logs here.

SYS1003 System CPU Resetting.
SYS1000 System is turning on.
RAC0182 The iDRAC firmware was rebooted with the following reason: ac.
SUP0516 Updating firmware for System BIOS (BIOS.Setup.1-1) to version 2.2.9.
SYS1003 System CPU Resetting.
RED030 Reboot is complete.
SYS1000 System is turning on.
SYS1001 System is turning off.
SYS1003 System CPU Resetting.
RAC0702 Requested system power cycle.
USR107 The operation SetupJobQueue of the DCIM_JobService was performed by root.
RED029 A reboot is pending.
JCP027 The (installation or configuration) job RID_633911391716 is successfully created on iDRAC.
USR107 The operation CreateRebootJob of the DCIM_SoftwareInstallationService was performed by root.
RED002 Package successfully downloaded.
RED111 Successfully downloaded the update package details 28.7065 MB in 16.33 secs at 1.7579 MBps (14.0632 Mbps) [FOLDER05611398M_1_BIOS_5XWV7_WN64_2.2.9.EXE].
RED110 Downloading the FOLDER05611398M_1_BIOS_5XWV7_WN64_2.2.9.EXE update package.

Figure 1: Log excerpt showing interrupted BIOS update.

In the above log excerpt, starting from the bottom and going up, the BIOS update is initiated (Event ID JCP027). The system then experienced a power AC loss and subsequent power restoration (Event ID RAC0182) during the update.

If the update had completed without interruption the logs would show the following:

  • Event ID SUP0518: "Successfully updated the System BIOS (BIOS.Setup.1-1) firmware to version X.Y.Z" (with X.Y.Z being the BIOS version)
  • Event ID SYS1003: "System CPU Resetting".
 

Solution

Once the issue has been confirmed as an interrupted BIOS update issue, use the following steps to perform the BIOS recovery.

Steps to perform BIOS recovery:

  1. Check the LCC job queue, and delete all jobs to stop any interfering the BIOS recovery.
     
    • This can be performed from the iDRAC web GUI from Maintenance -> Job Queue. Select all jobs and click the "Delete" button.
       
    • Dell BIOS 복구 - Dell BIOS boggu

      Figure 2: iDRAC9 Job Queue with job IDs displayed
    • Alternatively, the job queue can be deleted with local or remote racadm command: racadm jobqueue delete -i JID_CLEARALL_FORCE
      Dell BIOS 복구 - Dell BIOS boggu

      Figure 3: SSH session showing "jobqueue delete -i JID_CLEARALL_FORCE" command
    • The Job Queue page should return no jobs.
      Dell BIOS 복구 - Dell BIOS boggu

      Figure 4: iDRAC9 Job Queue cleared
  2. SSH to iDRAC and run the command: racadm recover BIOS.Setup.1-1
    Dell BIOS 복구 - Dell BIOS boggu

    Figure 5:
    SSH session showing racadm recover BIOS.Setup.1-1 command
    Dell BIOS 복구 - Dell BIOS boggu
    Figure 5 A: SSH session showing racadm recover BIOS.Setup.1-1 command when console shows racadm>>
  3. In the LCC log, a message ID of UEFI0298 with description of "The BIOS Recovery process is started" should be present.
    Dell BIOS 복구 - Dell BIOS boggu

    Figure 6: iDRAC9 LCC log with Message IDs displayed
  4. After a few minutes, the system will power on from a recovery ROM image. The boot screen displays the "[Recovery BIOS]" text.

    Dell BIOS 복구 - Dell BIOS boggu

    Figure 7: Boot screen showing commencement of BIOS recovery

  5. The system then performs a BIOS update to flash the primary ROM. This is completed with the full BIOS image that is stored in iDRAC from the last successful BIOS update performed from iDRAC/LC.

    Dell BIOS 복구 - Dell BIOS boggu

    Figure 8: Boot screen showing progress bar of BIOS recovery

  6. Once the BIOS flash completes, the system should restart and boot from the primary ROM image. The LC log should show "UEFI0299: The BIOS Recovery process is completed".
    Dell BIOS 복구 - Dell BIOS boggu

    Figure 9: iDRAC9 LCC logs showing confirmation of BIOS recover completion
  7. Once BIOS is recovered, it will come up as a new system board detected as seen below in figure 10 and the standard easy recovery process applies.
    Dell BIOS 복구 - Dell BIOS boggu
    Figure 10: POST picture showing standard easy recovery options
  8. Press Y to start restore process.
  9. The server will restart and then it will show the "The system content was successfully restored." (figure 11)
    Dell BIOS 복구 - Dell BIOS boggu
    Figure 11: Picture showing system configuration restore
  10. Press Y again and the server will restart multiple times as in Figure 12
    Dell BIOS 복구 - Dell BIOS boggu
    Figure 12: Example of server restarting during configuration restore
** Cautionary note**

After doing the BIOS recovery if the Lifecycle controller goes into recovery mode then run the following commands to re-enable it:

  1. racadm set LifecycleController.LCAttributes.LifecycleControllerState 0
    • This command will disable the lifecycle controller
  2. racadm set LifecycleController.LCAttributes.LifecycleControllerState 1
    • This command will enable the lifecycle controller
 

 


Troubleshooting

In the event either the BIOS recovery does not work or an auto recovery occurs, but does not resolve, a BIOS image can be specified to perform recovery. It is recommended to provide a BIOS image of the same version as that which was on the system before the latest BIOS update attempt.

The process is similar to the steps above, with the addition of uploading the BIOS Dell Update Package (DUP) to the iDRAC web GUI.

  1. Clear the LCC job queue through the methods shown in the solutions section.
  2. Upload BIOS DUP to iDRAC web GUI. From initial iDRAC web page, select Maintenance -> System Update. Click "Choose File" button, and browse to BIOS DUP file.
    Dell BIOS 복구 - Dell BIOS boggu

    Figure 10: iDRAC9 System Update screen showing Manual Update option
     
  3. Select "Upload". Activate the check box for the BIOS update and select "Install Next Reboot"
    Dell BIOS 복구 - Dell BIOS boggu

    Figure 11: iDRAC9 Manual Update screen showing Update Detail check box and install Next Reboot command

  4. Perform steps 2 through 6 as in the first section above (SSH to iDRAC, run "racadm recover BIOS.Setup.1-1").