ISM 5.3.1.0 on ESX version 8.0

Sorry the server model is R740xD 

Server Firmware version: 7.00.00.153

Hello, may I ask you if you are using dell ISO for ESXi.
 https://dell.to/3Aibgmt
Customized Image of VMware ESXi Availability and Download Instructions | Dell US

This article provides information about how to download the VMware ESXi Customized Image ISO.
 
if yes, then you need to raise the case and we will help to compare server config between working/non working servers.
You can also ask to check if the service is running 

@DELL-Young E​ yes we are using Dell ISO and the service is running

On the problematic server:
[root@vlan25-ip102:/var/log] esxcli dell ism config -g
The iDRAC Service Module is running with Limited Functionality Mode hence some features are unavailable. Possible reasons are: 1) OS-to-BMC Passthrough setting in iDRAC is disabled 2) USBNIC interface on the host OS does not have a configured IP address.

vSwitchiDRACvusb is not present

[root@vlan25-ip102:~] esxcfg-vswitch -l
Switch Name      Num Ports   Used Ports  Configured Ports  MTU     Uplinks
vSwitch0         9216        1           128               1500  PortGroup Name                            VLAN ID  Used Ports  Uplinks
  VM Network                                0        0Switch Name      Num Ports   Used Ports  Configured Ports  MTU     Uplinks
vSwitchBMC       9216        2           128               1500  PortGroup Name                            VLAN ID  Used Ports  Uplinks
  BMC_Network                               0        1DVS Name         Num Ports   Used Ports  Configured Ports  MTU     Uplinks
vds-hc1          9216        8           512               9000    vmnic2,vmnic0  DVPort ID                               In Use      Client
  66                                      0
  67                                      0
  68                                      1           vmnic0
  69                                      1           vmnic2
  52                                      1           vmk0DVS Name         Num Ports   Used Ports  Configured Ports  MTU     Uplinks
vds-hc2          9216        9           512               9000    vmnic3,vmnic1  DVPort ID                               In Use      Client
  66                                      0
  67                                      0
  68                                      1           vmnic1
  69                                      1           vmnic3
  52                                      1           vmk2
  60                                      1           vmk3[root@vlan25-ip102:~] esxcfg-vswitch -l | grep -i "iDRAC Network"
[root@vlan25-ip102:~]

Hello, the error is self-explanatory, at the end of the day, the tech support will need to check TSR by the official ticket you raise, which I encouraged earlier.
"The iDRAC Service Module is running with Limited Functionality Mode hence some features are unavailable. Possible reasons are: 1) OS-to-BMC Passthrough setting in iDRAC is disabled 2) USBNIC interface on the host OS does not have a configured IP address."
This can all be checked from TSR.
Respectfully,

@DELL-Young E​ can you please share steps to raise a case

Hello,

If you are in US/Canada you can call 1-800-945-3355 to contact tech support and set up a case.

If outside US you can find your regional support on the Contact US page

https://dell.to/4feyDft

It looks like you have a dell dot com email address. Internal employees are encouraged to use your internal contacts to reach support instead of posting on the Dell Community Forum. Dell employees must be SMAC certified to post on public forums.