KGPE-D16

From Vikings Wiki
Revision as of 00:10, 29 July 2020 by Thum (talk | contribs) (Fan Control)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search
File:KGPE-D16 mainboard.png
ASUS KGPE-D16 on the development stand at Vikings.

General Information

The ASUS KGPE-D16 is a AMD Family 10h/15h, dual-CPU server and workstation motherboard released end of 2009S. It is well supported and stable under coreboot and Libreboot, with all CPUs, RAM, and peripherals functioning normally.

A basic system diagram is available in the official manual, Appendix A.1 and has been confirmed to match the hardware shipping from ASUS. Not indicated are the PCIe lane widths for the gigabit network controller, which are both x1. All legacy PCI devices share the same bus, and partially due to this design the SP5100 has issues with bridging high-bandwidth PCI peripherals. As such, an external PCI-PCIe bridge is recommended should you need to interface a high bandwidth legacy PCI device to this system; ASMedia controllers have been verified to function correctly.

Northbridge functions are distributed between the CPU internal northbridge and the SR5690 northbridge, which is effectively a HyperTransport to ALink/PCIe translator and switch. There is a separate SP5100 southbridge device, adjacent to the northbridge and residing under the smaller heatsink of the two. This device provides all traditional southbridge services including the LPC bridge and SATA controllers. All southbridge-destined messages, including CPU-originated power state control messages over HyperTransport, pass through the CPU northbridge and are routed to the southbridge via the SR5690 northbridge device.

Incidentally, this design places the IOMMU, which is part of the SR5690, in the correct location to properly shield the main CPU from all unauthorized traffic. If the southbridge connected directly to a HyperTransport link there would be no way to prevent unauthorized DMA from legacy PCI devices connected to the southbridge, or even from the southbridge's embedded microprocessor.

Installation Notes

  • MUST be operated with one 8PIN EPS12V cable for operations with one CPU, and two 8PIN EPS12V cables for two CPUs;
  • coreboot/Libreboot must be flashed externally when migrating from the proprietary BIOS. After booting the new contents of the ROM at least once, flashrom can safely reprogram the ROM under Linux
  • When migrating from the proprietary BIOS, after flashing coreboot the CMOS memory MUST be cleared. Failing to clear the CMOS will typically result in odd hangs during the boot process
  • Enabling the serial console or EHCI debug console will drastically increase the time needed to boot
  • Having a serial console log level above 2 will drastically increase the time required for booting
  • The proprietary BMC module must be removed for coreboot to function - or flashed with the OpenBMC port
  • All CPUs are split in to two NUMA nodes as they are two 2/4/6/8 core CPU's in one package, memory is divided based on NUMA nodes (1 6282SE 16 core CPU, 2 Nodes, 32GB RAM, 16GB per node) and not properly aligning NUMA RAM will result in drastically decreased performance
  • Turbo 2 and power saving seems to require a tickless system to function (nohz=on in the kernel cmdline), otherwise the extra cores are always woken up and will never enter CC6

Fan Control

For general info regarding fancontrol please see fancontrol.

There are two ways to control fans on a KGPE-D16:

  • Using openBMC with an ASUS BMC module
  • Fancontrol/pwmconfig to control your fans from via your GNU/Linux operating system

Specific notes regarding fancontrol on KGPE-D16

openBMC Port on KGPE-D16

Install the OpenBMC port beta to the ASMB4-iKVM or ASMB5-iKVM modules that come with the main KGPE-D16 retail SKU, this provides fan control and a variety of other cool remote management features.

Limited thermal management with coreboot and coreboot-based BIOS replacements on KGPE-D16

TL;DR: If you have two HSFs installed they will always have the same fan speed. We recommend using the thermal sensors of the CPU in the back because it will always be the warmest.

The thermal management hardware of the KGPE-D16 is somewhat unusual and limited. It supports both 4-pin and 3-pin fans, however even though it contains a PWM controller with 8 hardware channels ASUS has only wired up two PWM channels to the fan connectors. To make matters worse, PWM channel 1 is routed to all 4-pin fans while PWM channel 2 is routed to all 3-pin fans.

Features

Hardware Features - at a glance
Format SSI-EEB
Socket G34
Max Processors 2
Max RAM 192 GB 128GB normal or 192GB via special configuration - See below
PCI-e slots 4 5 physical, 4 concurrent
PCI slots 1 Via PCI Bridge that also connects onboard AST graphics chip
Other Expansion Slots 1 PIKE ASUS Proprietary I/O Expansion Slot, Insert PIKE RAID card for second half of the motherboard SATA/SAS ports - half of the "PIKE" connector is simply a reversed PCI-e x4 slot.
EEPROM Type DIP 8 SPI Socket
Factory EEPROM Size 2MB
Max EEPROM Size ???
TPM YES Compatibility with TPM modules that fit ASUS TPM header. (Connected over LPC.) Note these modules are proprietary. Owner controlled CRTM
Crossfire XDMA ??? Has ACS and dual PCI-e 2.0 x16 slots, so it should work (reported working on vendor bios, need tester for coreboot)


Binary Situation
Blob Free Operations YES
Native GFX Init Partial Text Mode Only Now features proper EDID parsing.
BMC Open Source OpenBMC - open source remote management - Available for KGPE-D16 and KCMA-D8 boards via installation to the ASUS ASMB4-iKVM or ASMB5-iKVM module.


Virtualization
HVM YES
SLAT (RVI) YES
IOMMU YES v1.26 with Interrupt Remapping
IOMMU for Graphics YES Near-Native 3D gaming graphics performance with proper software configuration
PCI-e ACS YES
SR-IOV ??? Coreboot doesn't support SR-IOV
PCI-e ARI ??? Required for more than 8 SR-IOV VF per device, AMD's docs say the chipset supports it however there are no firmware implementations that feature it.

OpenBMC - Open Source Remote Management

Raptor Engineering worked on porting OpenBMC to the KGPE-D16 and KCMA-D8 under a crowdfunded contract. The ASUS ASMB4-iKVM or ASMB5-iKVM modules are required to use it.

More info here: https://www.raptorengineering.com/coreboot/kgpe-d16-bmc-port-status.php

Processor Summary

Family 10h (Opteron 6100) processors do not currently support the isochronous mode required to enable the IOMMU, Family 15h (Opteron 6200 & 6300) processors work well with the IOMMU enabled.

In addition to the 1 or 2 main CPUs, there are no less than three known secondary processors present on the mainboard. All are disabled when running under coreboot.

  • There is a very poorly documented microprocessor inside the SR5690; purpose and type unknown. It is believed this processor requires a firmware upload from the main platform firmware or via JTAG in order to start execution.
  • A single 8051 processor core is present inside the SB700 southbridge. It normally handles errata related to power states and may also be responsible for the blinking power LED in S3 suspend under the proprietary BIOS. It is believed accesses made by this processor are responsible for the flashrom write failure when the board is booted from the proprietary BIOS. This processor also requires a firmware upload from the main platform firmware or via JTAG in order to start execution.
  • The BMC has an integrated ARM core. This is disabled by pin strap when the BMC firmware module is not installed.

Some processors may be present on or activated by add-on modules:

  • The optional PIKE add-on cards use ARM cores to handle the SAS protocol, though this firmware is directly loaded from a Flash chip on the module and does not involve any non-local components (e.g. the main CPU never touches the firmware on these modules outside of a manual reflash operation). Raptor Engineering is currently unaware of any SAS controllers that operate without a secondary processor or use libre firmware; the protocol is simply too complex to handle via a mask ROM, and as there are only one or two suppliers of SAS controllers there is very little incentive to release the source code to the firmware. Writing a libre firmware to replace the existing firmware may technically be possible, however it is extremely unlikely this will ever happen due to the man-decades required.
  • Installing an ASUS iKVM firmware module will activate the ARM core in the BMC, which has full system access to all peripherals and possibly memory. It is not recommended to use this module as the firmware is both highly privileged and proprietary, and is known to contain at least one critical security bug.

Non-free Software & Security Alert

Vikings recommends the Opteron 6200 CPU series for their IOMMU support and stable operation without microcode updates (non-free software) tested in in more than ~500 CPU years in-house under heavy load without incident.

Vikings does NOT recommend using Opteron 6100 series because of missing IOMMU support (security!) or the Opteron 6300 series because it requires microcode updates for stability and IOMMU to work, a gain-root-via-NMI-exploit fix and other issues that can be found in AMD's Fam15h errata.

CPUs recommended

Processor sold by AMD Part Number Cores Requires microcode updates for secure operation (ref) Notes
Opteron 6386SE OS6386YETGGHK 16 Yes
Opteron 6328 OS6328WKT8GHK or OS6328WKT8GHKWOF 8 Yes
Opteron 6287SE (2nd fastest) 16 No
Opteron 6284SE ecx-Off-US-917385 16 No
Opteron 6282SE OS6282YETGGGU 16 No

https://en.wikipedia.org/wiki/List_of_AMD_Opteron_microprocessors

RAM

RAM HCL

The following RAM models and configurations have been tested by either Raptor Engineering or a third party and are know to work as of the stated GIT revision.

Manufacturer Model Max working RAM / CPU Size Speed Type ECC Populated Slots CPU Mainboard Type Firmware
Micron 36KSF2G72PZ-1G4E1 (N/A) 16GB DDR3-1333 Registered Yes A2 / C2 Opteron 6378 Coreboot 2268e0d or later
Micron MT36KSF1G72PZ-1G6M1FF 32GB 8GB DDR3-1600 Registered Yes All orange slots Opteron 6262HE 1.03G Internal development version of coreboot
Micron / HP MT36JSF2G72PZ-1G6E1LG (HP: 672612-081) 32GB 16GB DDR3-1600 Registered Yes A2 / C2 / E2 / G2 Opteron 6276 1.03G Libreboot 20160907
Hynix/Hyundai HMT151R7BFR4C-H9 16GB 4GB DDR3-1333 Registered Yes A2 / C2 / E2 / G2

A2 / B2 / C2 / D2

Opteron 6276 1.03G Libreboot 20160907
Kingston 9965525-055.A00LF 8GB DDR3-1600 Unbuffered Yes A2 / C2 / E2 / F2 Opteron 6328 Coreboot 9fba481
Kingston KVR16R11D4/16 (9965516-483.A00LF) 64GB 16GB DDR3-1600 Registered Yes All orange slots (128GB) Opteron 6278/6262HE Libreboot 20160907
Kingston KVR16R11D4K4/64I (9965516-477.A00LF) 64GB 16GB DDR3-1600 Registered Yes All orange slots (128GB) Opteron 6278/6262HE/6284SE Libreboot 20160907
crucial ("crucial by Micron") CT16G3ERSLD4160B (MT36KSF2G72PZ-1G6P1NE) 64GB 16GB DDR3-1600 Registered Yes All orange slots (128GB) Opteron 6278/6282SE/6284SE/6287SE 1.03G, 1.04 Libreboot 20160907
Micron MT36KSF2G72PZ-1G6E1FE 64GB 16GB DDR3-1600 Registered Yes All orange slots Opteron 6378 1.04 Internal development version of coreboot (2017)
Micron MT36KSF2G72PZ-1G6N1KG 64GB 16GB DDR3-1600 Registered Yes All orange slots Opteron 6378 1.04 Internal development version of coreboot (2017)
crucial ("crucial by Micron") CT16G3ERSLD4160B (MT36KSF2G72PZ-1G6P1NE) 192GB 16GB DDR3-1600 Registered Yes Leave H1, H2, G1, G2 empty (see page 2-16 in the ASUS manual), LVDDR3_SEL1 can be set to "Force 1.35V" Opteron 6278/6282SE/6284SE/6287SE 1.03G, 1.04 coreboot d6735b0

RAM issues

192 GB RAM Limitation

The KGPE-D16 doesn't work with more than 192 GB RAM and would need further work by coreboot developers, which is unlikely going to be funded at this point (June 2019).

CPU0 can be fully populated, however CPU1 has marginal routing. In order to use 192 GB of RAM it's necessary to either leave the two DIMM slots next to the CPUs unpopulated or the 4 closest on CPU1 unpopulated. In each case RAM training may work well, but system can be unstable. Stress-testing recommended. Behavior may be contingent on the boards PCB revision.

Todo: It's been reported by Ian K. that 256 GB work. Further tests required.

Miscellaneous Known Issues

EHCI debug console

The EHCI debug console causes severe USB problems under both Libreboot and coreboot. This typically manifests as very slow boot / slow typing on USB keyboards. This issue appears to extend to the KCMA-D8 and KFSN4-DRE boards as well.

MMIO Resources Limit

The coreboot 32bit MMIO space limits the use of large amounts of PCI-e devices, such as more than a few network interfaces or graphics cards with the limit coming up sooner for older multi-port NIC's that have a switched design (ex: 82576), vs the newer style native multi-port pci-e setup (i350)

This is the reason for the "Not enough MMIO resources for SR-IOV" error when you attempt to enable SR-IOV on a system with both a quad port NIC and the onboard interfaces.


PIKE 2008 Libreboot Issues

Libreboot 20160818, 20160902 and 20160907 all have a bug: in SeaBIOS, PCI options ROMs are loaded when available, by default. Technically speaking this isn't a problem, because an option ROM can be free or non-free. Loading the option ROM from the PIKE2008 module on the KGPE-D16 causes the system to hang at boot. It's possible to use this in the payload (if you use Linux as payload, or the Petitboot bootloader), or to boot (with SeaGRUB and/or SeaBIOS) from regular SATA and then use it in GNU/Linux. Linux is capable of using the PIKE2008 module without loading the option ROM.

Libreboot-unstable (or Libreboot git master) now disables loading PCI option ROMs, but previous releases with SeaGRUB (20160818-20160907) do not. You can work around this by running the following command: $ ./cbfstool yourrom.rom add-int -i 0 -n etc/pci-optionrom-exec You can find cbfstool in the Libreboot util archive with the libreboot release that you are using.

Miscellaneous Notes

The 4 total PCI-e slots may be limiting, but as the board has PCI-e ACS you may be able to use an external ACS supporting PCI-e expansion system - you would still have IOMMU security and performance as ACS support means that the devices beyond the external switch will be placed in separate IOMMU groups and thus you will maintain security and not have to use the unsafe attachment override for attaching devices to virtual machines.

NOTE: MMIO space limit dependent.

MCM/NUMA notes - Read if you play video games

NOTE: All G34 CPU's are dual-MCM thus with two NUMA nodes, if you play video games or need a single task with many threads the socket C32 single MCM/NUMA node KCMA-D8 with a 4386 might have improved performance although it is also possible to play games with a dual node CPU without stuttering.

The correct way to do this is to create a VM with properly pinned CPU's including iothread/emulator with all of the RAM on one node which is the same one that your interrupts for assigned devices such as graphics usb etc are being processed on.

Turbo Examples: If you have a 16 core CPU to obtain Turbo 2 you would select 2 modules and thus 4 cores from each MCM/NUMA node - then you allocate all of the hugepages/VM RAM on node 0 where the interrupts are assigned - this will provide the best gaming performance with a 16 core CPU. If you have dual 8 core 6328 CPU's the best VM gaming performance is gained by using both node zeros from both CPU's and hugepages RAM on the first node (zero) of the first CPU - this obtains 8 cores at 3.8ghz. You would also need to isolate the CPU's not in use by using the isolcpus kernel command line option and moving away interrupts if they somehow migrate to an isol'ed cpu.

Please contact Taiidan for advice on VM gaming for this board, how to obtain Crossfire xDMA in a VM, etc - with a capable graphics card you should be able to almost max out games circa 2017 at 1080p with one of the faster socket G34 CPU's.

Free Software Foundations "Respects Your Privacy" (RYF) certification

The Vikings D16 (a relabelled KGPE-D16) board is being sold with coreboot/Libreboot pre-installed. It is the first workstation/server mainboard that has been "RYF - Respects Your Freedom" certified by the Free Software Foundation on March 6th, 2017.

External links