The ASUS RS500A-E11-RS12U is an AMD EPYC 7003 (Milan) 1U single-socket server (barebone) that supports up to 16x DIMM, 12x NVMe, 3x PCIe 4.0 slots, 1x OCP 3.0, 2x M.2 and ASUS ASMB10-iKVM and in use for the # VIKINGS hosting platform. It contains the ASUS KRPA-U16 mainboard.

PCIe Gen4 x16,slots, LP (x16 link)                           
          ▲                                                  
          │PCIe Gen4 x16 slot, LP (x8 link)                  
          │            ▲                                     
          └┐           │  PCIe Gen4 x16 slot, FHHL (x16 link)
           │           │                       ▲             
┌────┬────┬┼───────────┼───────────────────────┼─────┐       
│    │    │├─────────┐ ├─────────┐   ┌─────────┴───┐ │       
│    │    │└─────────┘ └─────────┘   └─────────────┘ │       
│ PSU│ PSU│     ┌──┐  ┌───┐            ┌───────────┐ │       
│    │    │ ┌─┐ ├──┘  ├───┘┌──┐┌──┐┌──┐└─────┬─────┘ │       
│    │ ┌──┼─┴─┘ ├──┐  │    │  ││  ││  │      │       │       
│    │ │  │     ├──┘  │    ├──┘├──┘├──┘      │       │       
└────┴─┼──┴─────┼─────▼────┼───┼───┼─────────┼───────┘       
       ▼        ▼    VGA   │   ▼   │         ▼               
 QCode LED  2x USB3.2 Gen1 │ LAN1  ▼   OCP3.0 Gen4 x16       
                           │ 1Gbe LAN2                        
                  BMC ◄────┘      1Gbe                                                         

The mainboard does not support PCIe bifurcation. PCIe cards that internally use multiple controllers requiring bifurcation to achieve full aggregate bandwidth, e.g. Intel E810-XXVDA4 may and e.g. Intel E810-CQDA2 will operate with reduced total throughput. It is recommended to avoid such cards on this mainboard (or ensure they are installed in a sufficiently wide PCIe slot in the case of the Intel E810-XXVDA4 example).

The card has two internal controllers, each requiring x8 lanes. Without bifurcation, both controllers share the x16 slot as a single x16 link, but only one controller is actively used at full speed. The total aggregate bandwidth is limited to approx. 125 Gbps on a PCIe 4.0 x16 slot effective throughput, meaning both ports cannot simultaneously run at full 100 Gbps — only one port at a time can reach line rate.

Contrary to the E810-CQDA2 the Intel E810-XXVDA4 has one internal controller, which receives adequate PCIe bandwidth without requiring bifurcation. Since PCIe Gen4 offers sufficient bandwidth even on a single x8 link, and there are x8 and x16 slots available, there should be no problems regarding the aggregated maximum speed of 4x 25GbE.

When using 4-port network cards in one of the low-profile slots, the server chassis obscures one of the ports by about 0.5 mm. Inserting a GBIC (or SFP transceiver) is possible with some force, but subsequent removal is very difficult or impossible. We have found this issue with all the barebones we have handled so far. An issue like this with such well-designed hardware from a reputable vendor is unprecedented, but there are obvious workarounds:

• Potential Fixes
  • When purchasing network cards with 4 slots, use the OCP3.0 or full-height port if possible
  • For stuck GBICs, gently wiggle while pulling or use a flathead screwdriver for leverage without forcing, to prevent port damage
  • If you're feeling brave, you can use a Dremel tool to manipulate the areas. But seriously, only do this if you know what you're doing!

ASUS RS500A-E11-RS12U Tuning