legacy_hardware:kgpe-d16
Differences
This shows you the differences between two versions of the page.
| Both sides previous revisionPrevious revisionNext revision | Previous revision | ||
| legacy_hardware:kgpe-d16 [2026/04/11 23:13] – removed - external edit (Unknown date) 127.0.0.1 | legacy_hardware:kgpe-d16 [2026/04/12 08:48] (current) – ↷ Links adapted because of a move operation 44.221.105.234 | ||
|---|---|---|---|
| Line 1: | Line 1: | ||
| + | ======= ASUS KGPE-D16 ======= | ||
| + | |||
| + | |||
| + | {{ : | ||
| + | |||
| + | ======= General Information ======= | ||
| + | |||
| + | The ASUS KGPE-D16, //often referred to as " | ||
| + | |||
| + | The ASUS KGPE-D16 is a AMD Family 10h/15h, dual-CPU server and workstation motherboard released end of 2012. With a dual CPU setup the performance is still impressive today for a target that doesn' | ||
| + | |||
| + | < | ||
| + | |||
| + | Northbridge functions are distributed between the CPU internal northbridge and the SR5690 northbridge, | ||
| + | |||
| + | Incidentally, | ||
| + | |||
| + | ====== Installation Notes ====== | ||
| + | |||
| + | * Must be operated with 1x EPS12V cable for operations with one CPU, and 2x EPS12V cables for two CPUs | ||
| + | * coreboot must be flashed externally when migrating from the proprietary BIOS, for example with a [[https:// | ||
| + | * When migrating from the proprietary BIOS, after flashing coreboot the CMOS memory must be cleared via the appropriate jumper on the mainboard. Failing to clear the CMOS will typically result in odd hangs during the boot process | ||
| + | * Enabling the serial console or EHCI debug console will drastically increase the time needed to boot | ||
| + | * Having a serial console log level above 2 will drastically increase the time required for booting | ||
| + | * All CPUs are split in to two NUMA nodes as they are two 2/4/6/8 core CPU's in one package, memory is divided based on NUMA nodes (1 6282SE 16 core CPU, 2 Nodes, 32GB RAM, 16GB per node) and not properly aligning NUMA RAM will result in drastically decreased performance | ||
| + | * Turbo 2 and power saving seems to require a tickless system to function (**nohz=on** in the kernel cmdline), otherwise the extra cores are always woken up and will never enter CC6 | ||
| + | |||
| + | ===== Fan Control ===== | ||
| + | |||
| + | For general info regarding fancontrol please see [[: | ||
| + | |||
| + | There are two ways to control fans on a KGPE-D16: | ||
| + | |||
| + | * Using openBMC with an ASUS BMC module | ||
| + | * Fancontrol/ | ||
| + | |||
| + | ==== Fan Control Brain Dump From Around 2016 ==== | ||
| + | |||
| + | pwmconfig made fanctrontol config is not reboot safe. better use | ||
| + | / | ||
| + | |||
| + | # update-rc.d fancontrol defaults | ||
| + | |||
| + | standard: | ||
| + | |||
| + | # | ||
| + | INTERVAL=10 | ||
| + | DEVPATH=hwmon0=devices/ | ||
| + | DEVNAME=hwmon0=w83795g | ||
| + | FCTEMPS=hwmon0/ | ||
| + | FCFANS= hwmon0/ | ||
| + | MINTEMP=hwmon0/ | ||
| + | MAXTEMP=hwmon0/ | ||
| + | MINSTART=hwmon0/ | ||
| + | MINSTOP=hwmon0/ | ||
| + | |||
| + | Strategy is not to rely on / | ||
| + | `sensors-detect` for module loading, but instead to do it manually. | ||
| + | |||
| + | 1) Remove lm_sensors.serivce from autostart. | ||
| + | 2) Place needed modules in / | ||
| + | 3) Reboot. | ||
| + | |||
| + | improved: | ||
| + | INTERVAL=10 | ||
| + | # | ||
| + | # | ||
| + | FCTEMPS=/ | ||
| + | # | ||
| + | FCFANS=/ | ||
| + | #FCFANS= | ||
| + | hwmon4/ | ||
| + | # | ||
| + | MINTEMP=/ | ||
| + | MAXTEMP=/ | ||
| + | # | ||
| + | MINSTART=/ | ||
| + | # | ||
| + | MINSTOP=/ | ||
| + | # | ||
| + | |||
| + | ==== Specific notes regarding fancontrol on KGPE-D16 ==== | ||
| + | |||
| + | ===== openBMC Port on KGPE-D16 ===== | ||
| + | |||
| + | Install the OpenBMC port beta to the ASMB4-iKVM or ASMB5-iKVM modules that come with the main KGPE-D16 retail SKU, this provides fan control and a variety of other cool remote management features. | ||
| + | |||
| + | ==== Limited thermal management with coreboot and coreboot-based BIOS replacements on KGPE-D16 ==== | ||
| + | |||
| + | The thermal management hardware of the KGPE-D16 is somewhat unusual and limited. It supports both 4-pin and 3-pin fans, however even though it contains a PWM controller with 8 hardware channels, ASUS has only wired up two PWM channels to the fan connectors. To make matters worse, PWM channel 1 is routed to all 4-pin fans while PWM channel 2 is routed to all 3-pin fans. | ||
| + | |||
| + | **TL;DR:** If you have two CPU HSFs installed, both fans will run at the same speed. We recommend using the thermal sensors of the warmest CPU in the system depending on your setup. | ||
| + | |||
| + | ===== Features ===== | ||
| + | |||
| + | Hardware Features - at a glance | ||
| + | |||
| + | ^Format | ||
| + | ^Socket | ||
| + | ^Max Processors | ||
| + | ^Max RAM |256 GB | | ||
| + | ^PCI-e slots |4 | | ||
| + | ^PCI slots |1 | | ||
| + | ^Other Expansion Slots|1 PIKE | | ||
| + | ^EEPROM Type |DIP 8 SPI Socket| | ||
| + | ^Factory EEPROM Size |2MB | | ||
| + | ^Max EEPROM Size |16MB tested | ||
| + | ^TPM |YES | | ||
| + | |||
| + | |||
| + | Binary Situation | ||
| + | ^Blob Free Operations |YES | | ||
| + | ^Native GFX Init | ||
| + | ^BMC |Open Source| | ||
| + | |||
| + | Virtualization | ||
| + | ^HVM | YES | ||
| + | ^SLAT (RVI) |YES| | ||
| + | ^IOMMU | ||
| + | ^IOMMU for Graphics|YES| | ||
| + | ^PCI-e ACS |YES| | ||
| + | ^SR-IOV | ||
| + | ^PCI-e ARI |???| | ||
| + | |||
| + | ==== OpenBMC - Open Source Remote Management ==== | ||
| + | |||
| + | Raptor Engineering worked on porting OpenBMC to the KGPE-D16 and KCMA-D8 under a crowdfunded contract. The ASUS ASMB4-iKVM or ASMB5-iKVM modules are required to use it. | ||
| + | |||
| + | More info here: https:// | ||
| + | |||
| + | ===== CPU Summary ===== | ||
| + | |||
| + | Family 10h (Opteron 6100) processors do not currently support the isosynchronous mode required to enable the IOMMU. | ||
| + | Family 15h (Opteron 6200 & 6300) processors work well with the IOMMU enabled. | ||
| + | |||
| + | Vikings **recommends** the **Opteron 6200** CPU series for their IOMMU support and stable operation without microcode updates. | ||
| + | |||
| + | Vikings **recommends** the **Opteron 6300** CPU series for their IOMMU support, stable operation and better performance than the Opteron 6200 CPU series. Note microcode updates are required to run these CPUs. A [[https:// | ||
| + | |||
| + | In addition to the 1 or 2 main CPUs, there are no less than three known secondary processors present on the mainboard. All are disabled when running under coreboot. | ||
| + | |||
| + | * There is a very poorly documented microprocessor inside the SR5690; purpose and type unknown. It is believed this processor requires a firmware upload from the main platform firmware or via JTAG in order to start execution. | ||
| + | * A single 8051 processor core is present inside the SB700 southbridge. It normally handles errata related to power states and may also be responsible for the blinking power LED in S3 suspend under the proprietary BIOS. It is believed accesses made by this processor are responsible for the flashrom write failure when the board is booted from the proprietary BIOS. This processor also requires a firmware upload from the main platform firmware or via JTAG in order to start execution. | ||
| + | * The BMC has an integrated ARM core. This is disabled by pin strap when the BMC firmware module is not installed. | ||
| + | |||
| + | Some processors may be present on or activated by add-on modules: | ||
| + | |||
| + | * The optional PIKE add-on cards use ARM cores to handle the SAS protocol, though this firmware is directly loaded from a Flash chip on the module and does not involve any non-local components (e.g. the main CPU never touches the firmware on these modules outside of a manual reflash operation). Raptor Engineering is currently unaware of any SAS controllers that operate without a secondary processor or use libre firmware; the protocol is simply too complex to handle via a mask ROM, and as there are only one or two suppliers of SAS controllers there is very little incentive to release the source code to the firmware. Writing a libre firmware to replace the existing firmware may technically be possible, however it is extremely unlikely this will ever happen due to the man-decades required. | ||
| + | * Installing an ASUS iKVM firmware module will activate the ARM core in the BMC, which has full system access to all peripherals and possibly memory. It is not recommended to use this module as the firmware is both highly privileged and proprietary, | ||
| + | |||
| + | ==== Recommended CPUs ==== | ||
| + | |||
| + | ^ CPU ^ Part Number ^ Cores ^ Requires microcode updates ^ | ||
| + | | Opteron 6386SE (fastest) | ||
| + | | Opteron 6328 | OS6328WKT8GHK or OS6328WKT8GHKWOF|8 | ||
| + | | Opteron 6287 SE | OS6287YITGGGU | ||
| + | | Opteron 6284 SE | OS6284YETGGGU | ||
| + | | Opteron 6282 SE | OS6282YETGGGU | ||
| + | | Opteron 6262 HE | OS6262VATGGGU | ||
| + | |||
| + | https:// | ||
| + | |||
| + | ===== RAM ===== | ||
| + | |||
| + | ==== RAM HCL ==== | ||
| + | |||
| + | The following RAM models and configurations have been tested by either Raptor Engineering or a third party and are know to work as of the stated GIT revision. | ||
| + | |||
| + | ^Manufacturer | ||
| + | |**SK Hynix** | ||
| + | |Samsung | ||
| + | |Micron | ||
| + | |Micron | ||
| + | |Micron / HP |MT36JSF2G72PZ-1G6E1LG (HP: 672612-081) | ||
| + | |Hynix/ | ||
| + | |Hynix/ | ||
| + | |Kingston | ||
| + | |Kingston | ||
| + | |Kingston | ||
| + | |crucial (" | ||
| + | |Micron | ||
| + | |Micron | ||
| + | |crucial (" | ||
| + | |||
| + | ==== RAM Limitations ==== | ||
| + | |||
| + | === 192 GB RAM limitation, up to 256 GB RAM with specific DIMMs === | ||
| + | |||
| + | With most available DIMMs the KGPE-D16 does not work with more than 192 GB RAM. There is a specific RAM module (namely HMT42GR7AFR4A-PB) that works stable up to 256 GB RAM (the maximum of the mainboard) (see RAM HCL). | ||
| + | |||
| + | |||
| + | If you are using RAM that is not HMT42GR7AFR4A-PB, | ||
| + | |||
| + | < | ||
| + | Financed in March 2024, model " | ||
| + | |||
| + | ===== Miscellaneous Known Issues and Workarounds ===== | ||
| + | |||
| + | ==== No Boot Menu with Petitboot ==== | ||
| + | |||
| + | If your KGPE-D16 server or workstation is equipped with a discrete graphics processor, the on-board VGA is disabled. Petitboot builds of Vikings do not have drivers for most discrete graphics processors, so there is no boot screen on which to select boot media, for example. Your display should start working as soon as Linux from an installed drive has been loaded. | ||
| + | |||
| + | A workaround for this is to enable the VGA jumper and connect a screen to the on-board VGA output because Petitboot has the required drivers for the on-board GPU (Aspeed AST2050) included. This is useful when installing a new operating system for example. The jumper settings can be reverted afterwards. | ||
| + | ==== EHCI Debug Console ==== | ||
| + | |||
| + | The EHCI debug console causes severe USB problems under both Libreboot and coreboot. This typically manifests as very slow boot / slow typing on USB keyboards. This issue appears to extend to the KCMA-D8 and [[Board: | ||
| + | |||
| + | ==== MMIO Resources Limit ==== | ||
| + | |||
| + | The coreboot 32bit MMIO space limits the use of large amounts of PCI-e devices, such as more than a few network interfaces or graphics cards with the limit coming up sooner for older multi-port NIC's that have a switched design (ex: 82576), vs the newer style native multi-port pci-e setup (i350) | ||
| + | |||
| + | This is the reason for the "Not enough MMIO resources for SR-IOV" | ||
| + | |||
| + | ==== PIKE 2008 Libreboot Issues ==== | ||
| + | |||
| + | Libreboot 20160818, 20160902 and 20160907 all have a bug: in SeaBIOS, PCI options ROMs are loaded when available, by default. Technically speaking this isn't a problem, because an option ROM can be free or non-free. Loading the option ROM from the PIKE2008 module on the KGPE-D16 causes the system to hang at boot. It's possible to use this in the payload (if you use Linux as payload, or the Petitboot bootloader), | ||
| + | |||
| + | Libreboot-unstable (or Libreboot git master) now disables loading PCI option ROMs, but previous releases with SeaGRUB (20160818-20160907) do not. You can work around this by running the following command: '' | ||
| + | |||
| + | ==== Miscellaneous Notes ==== | ||
| + | |||
| + | The 4 total PCI-e slots may be limiting, but as the board has PCI-e ACS you may be able to use an external ACS supporting PCI-e expansion system - you would still have IOMMU security and performance as ACS support means that the devices beyond the external switch will be placed in separate IOMMU groups and thus you will maintain security and not have to use the unsafe attachment override for attaching devices to virtual machines. | ||
| + | |||
| + | NOTE: MMIO space limit dependent. | ||
| + | |||
| + | ===== MCM/NUMA notes - Read if you play video games ===== | ||
| + | |||
| + | NOTE: All G34 CPU's are dual-MCM thus with two NUMA nodes, if you play video games or need a single task with many threads the socket C32 single MCM/NUMA node KCMA-D8 with a 4386 might have improved performance although it is also possible to play games with a dual node CPU without stuttering. | ||
| + | |||
| + | The correct way to do this is to create a VM with properly pinned CPU's including iothread/ | ||
| + | |||
| + | Turbo Examples: If you have a 16 core CPU to obtain Turbo 2 you would select 2 modules and thus 4 cores from each MCM/NUMA node - then you allocate all of the hugepages/ | ||
| + | |||
| + | |||
| + | ====== Operating Systems Findings ====== | ||
| + | |||
| + | ===== Qubes ===== | ||
| + | |||
| + | |||
| + | |||
| + | |||
| + | ====== Free Software Foundations " | ||
| + | |||
| + | < | ||
| + | |||
| + | ====== External links ====== | ||
| + | |||
| + | * [[https:// | ||
| + | * [[https:// | ||
| + | |||
| + | [[Category: | ||