Unsolved
1 Rookie
•
1 Message
0
5
October 3rd, 2025 23:25
PowerEdge R760xa - unable to get PXB communication between two GPUs (stuck at NODE)
I'm trying to understand why I can't get true PCIe P2P communication between the two NVIDIA L40S GPUs installed in PCIe Slots 36 and 38 that are connected to the same socket (CPU 1) in my R760xa:
$ sudo nvidia-smi topo -m
GPU0 GPU1 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X NODE 0,2,4,6,8,10 0 N/A
GPU1 NODE X 0,2,4,6,8,10 0 N/A
NODE means both GPUs can communicate via the CPU they are both attached to (CPU 1). However, the goal is to get PXB (instead of NODE) which means the GPUs can communicate directly with each other over the PCIe bus, bypassing the CPU entirely. I need the lowest latency that direct P2P PXB provides, but I'm stuck at NODE.
From what I understand, maybe incorrectly, the following two settings could play a role (in addition to other things): Memory Mapped I/O above 4 GB (alt name Above 4G Decoding) and Resizable BAR Support.
For some reason I'm not able to locate either of these two in the BIOS.. are they actually supported in 16th Gen? Other thoughts what might be blocking the PXB comms?