hardware:kgpe-d16
Differences
This shows you the differences between two versions of the page.
| Next revision | Previous revision | ||
| hardware:kgpe-d16 [2023/04/16 17:56] – created - external edit 127.0.0.1 | hardware:kgpe-d16 [2024/03/25 09:33] (current) – [General Information] wording thum | ||
|---|---|---|---|
| Line 1: | Line 1: | ||
| - | ======= KGPE-D16 ======= | + | ======= |
| {{ : | {{ : | ||
| Line 5: | Line 5: | ||
| ======= General Information ======= | ======= General Information ======= | ||
| - | The ASUS KGPE-D16 | + | The ASUS KGPE-D16, //often referred to as " |
| - | A basic system diagram is available in the official [[https:// | + | The ASUS KGPE-D16 is a AMD Family 10h/15h, dual-CPU server and workstation motherboard released end of 2012. With a dual CPU setup the performance is still impressive today for a target that doesn' |
| + | |||
| + | <del>A basic system diagram is available in the official [[https:// | ||
| Northbridge functions are distributed between the CPU internal northbridge and the SR5690 northbridge, | Northbridge functions are distributed between the CPU internal northbridge and the SR5690 northbridge, | ||
| Line 16: | Line 18: | ||
| * Must be operated with 1x EPS12V cable for operations with one CPU, and 2x EPS12V cables for two CPUs | * Must be operated with 1x EPS12V cable for operations with one CPU, and 2x EPS12V cables for two CPUs | ||
| - | * coreboot must be flashed externally when migrating from the proprietary BIOS. Alternatively the ROM chips can be easibly | + | * coreboot must be flashed externally when migrating from the proprietary BIOS, for example with a [[https:// |
| - | * When migrating from the proprietary BIOS, after flashing coreboot the CMOS memory | + | * When migrating from the proprietary BIOS, after flashing coreboot the CMOS memory |
| * Enabling the serial console or EHCI debug console will drastically increase the time needed to boot | * Enabling the serial console or EHCI debug console will drastically increase the time needed to boot | ||
| * Having a serial console log level above 2 will drastically increase the time required for booting | * Having a serial console log level above 2 will drastically increase the time required for booting | ||
| Line 47: | Line 49: | ||
| Hardware Features - at a glance | Hardware Features - at a glance | ||
| - | ^scope=" | + | |
| - | |scope=" | + | ^Format |
| - | |scope=" | + | ^Socket |
| - | |scope=" | + | ^Max Processors |
| - | |scope=" | + | ^Max RAM |256 GB | |
| - | |scope=" | + | ^PCI-e slots |4 | |
| - | |scope=" | + | ^PCI slots |1 | |
| - | |scope=" | + | ^Other Expansion Slots|1 PIKE | |
| - | |scope=" | + | ^EEPROM Type |DIP 8 SPI Socket| |
| - | |scope=" | + | ^Factory EEPROM Size |2MB | |
| - | |scope=" | + | ^Max EEPROM Size |16MB tested |
| - | |scope=" | + | ^TPM |YES | |
| Binary Situation | Binary Situation | ||
| - | ^scope=" | + | ^Blob Free Operations |
| - | |scope=" | + | ^Native GFX Init |
| - | |scope=" | + | ^BMC |Open Source| |
| Virtualization | Virtualization | ||
| - | ^scope=" | + | ^HVM | YES |
| - | |scope=" | + | ^SLAT (RVI) |YES| |
| - | |scope=" | + | ^IOMMU |
| - | |scope=" | + | ^IOMMU for Graphics|YES| |
| - | |scope=" | + | ^PCI-e ACS |
| - | |scope=" | + | ^SR-IOV |
| - | |scope=" | + | ^PCI-e ARI |???| |
| ==== OpenBMC - Open Source Remote Management ==== | ==== OpenBMC - Open Source Remote Management ==== | ||
| Line 80: | Line 83: | ||
| More info here: https:// | More info here: https:// | ||
| - | ===== Processor | + | ===== CPU Summary ===== |
| + | |||
| + | Family 10h (Opteron 6100) processors do not currently support the isosynchronous mode required to enable the IOMMU. | ||
| + | Family 15h (Opteron 6200 & 6300) processors work well with the IOMMU enabled. | ||
| + | |||
| + | Vikings **recommends** the **Opteron 6200** CPU series for their IOMMU support and stable operation without microcode updates. | ||
| - | Family 10h (Opteron | + | Vikings **recommends** the **Opteron |
| In addition to the 1 or 2 main CPUs, there are no less than three known secondary processors present on the mainboard. All are disabled when running under coreboot. | In addition to the 1 or 2 main CPUs, there are no less than three known secondary processors present on the mainboard. All are disabled when running under coreboot. | ||
| Line 94: | Line 102: | ||
| * The optional PIKE add-on cards use ARM cores to handle the SAS protocol, though this firmware is directly loaded from a Flash chip on the module and does not involve any non-local components (e.g. the main CPU never touches the firmware on these modules outside of a manual reflash operation). Raptor Engineering is currently unaware of any SAS controllers that operate without a secondary processor or use libre firmware; the protocol is simply too complex to handle via a mask ROM, and as there are only one or two suppliers of SAS controllers there is very little incentive to release the source code to the firmware. Writing a libre firmware to replace the existing firmware may technically be possible, however it is extremely unlikely this will ever happen due to the man-decades required. | * The optional PIKE add-on cards use ARM cores to handle the SAS protocol, though this firmware is directly loaded from a Flash chip on the module and does not involve any non-local components (e.g. the main CPU never touches the firmware on these modules outside of a manual reflash operation). Raptor Engineering is currently unaware of any SAS controllers that operate without a secondary processor or use libre firmware; the protocol is simply too complex to handle via a mask ROM, and as there are only one or two suppliers of SAS controllers there is very little incentive to release the source code to the firmware. Writing a libre firmware to replace the existing firmware may technically be possible, however it is extremely unlikely this will ever happen due to the man-decades required. | ||
| * Installing an ASUS iKVM firmware module will activate the ARM core in the BMC, which has full system access to all peripherals and possibly memory. It is not recommended to use this module as the firmware is both highly privileged and proprietary, | * Installing an ASUS iKVM firmware module will activate the ARM core in the BMC, which has full system access to all peripherals and possibly memory. It is not recommended to use this module as the firmware is both highly privileged and proprietary, | ||
| - | |||
| - | ==== Non-free Software & Security Alert ==== | ||
| - | |||
| - | Vikings **recommends** the Opteron 6200 CPU series for their IOMMU support and stable operation without microcode updates (non-free software) tested in in more than ~500 CPU years in-house under heavy load without incident. | ||
| - | |||
| - | Vikings does **NOT** recommend using Opteron 6100 series because of missing IOMMU support (security!) or the Opteron 6300 series because it requires microcode updates for stability and IOMMU to work, a [[https:// | ||
| ==== Recommended CPUs ==== | ==== Recommended CPUs ==== | ||
| - | ^Processor [[http:// | + | ^ CPU ^ Part Number ^ Cores ^ Requires microcode updates ^ |
| - | |scope=" | + | | Opteron 6386SE |
| - | |scope=" | + | | Opteron 6328 | OS6328WKT8GHK or OS6328WKT8GHKWOF|8 |
| - | |scope=" | + | | Opteron |
| - | |scope=" | + | | Opteron |
| - | |scope=" | + | | Opteron |
| - | | | + | | Opteron 6262 HE | OS6262VATGGGU |
| https:// | https:// | ||
| Line 120: | Line 122: | ||
| ^Manufacturer | ^Manufacturer | ||
| - | |// | + | |**SK Hynix** |
| - | |Micron | + | |Micron |
| |Micron | |Micron | ||
| |Micron / HP |MT36JSF2G72PZ-1G6E1LG (HP: 672612-081) | |Micron / HP |MT36JSF2G72PZ-1G6E1LG (HP: 672612-081) | ||
| |Hynix/ | |Hynix/ | ||
| - | |Kingston | + | |Hynix/ |
| + | |Kingston | ||
| |Kingston | |Kingston | ||
| |Kingston | |Kingston | ||
| Line 133: | Line 136: | ||
| |crucial (" | |crucial (" | ||
| - | ==== Max. RAM Limitations ==== | + | ==== RAM Limitations ==== |
| - | === 192 GB RAM limitation, up to 256 GB RAM with Vikings builds | + | === 192 GB RAM limitation, up to 256 GB RAM with specific DIMMs === |
| - | It has been reported that the KGPE-D16 | + | With most available DIMMs the KGPE-D16 |
| - | CPU0 can be fully populated, however CPU1 has marginal routing. In order to use 192 GB of RAM it's necessary to either leave the two DIMM slots next to the CPUs unpopulated or the 4 closest on CPU1 unpopulated. In each case RAM training may work well, but system can be unstable. Stress-testing recommended. Behavior may be contingent on the boards PCB revision.< | ||
| - | **Vikings can build your KGPE-D16 based workstation or server with 256 GB of ECC RAM by using carefully selected and tested DIMMs. Let us know if you're interested in such a build.** | + | If you are using RAM that is not HMT42GR7AFR4A-PB, please note the following: CPU0 can be fully populated, however CPU1 has marginal routing. In order to use 192 GB of RAM it's necessary to either leave the two DIMM slots next to the CPUs unpopulated or the 4 closest on CPU1 unpopulated. In each case RAM training may work well, but system can be unstable. Stress-testing recommended. Behavior may be contingent on the boards PCB revision. |
| - | //Although these are standard DIMMs that you can normally get on the free market, we use them to finance our expensive DIMM tests that we performed for the KGPE-D16. We therefore ask for your understanding that we cannot publish this at the moment. We will do this as soon as the costs are recovered. With the current rate of sales we expect this to be the case in late 2024// | + | <del>//Although these are standard DIMMs that you can normally get on the free market, we use them to finance our expensive DIMM tests that we performed for the KGPE-D16. We therefore ask for your understanding that we cannot publish this at the moment. We will do this as soon as the costs are recovered. With the current rate of sales we expect this to be the case in late 2024//</ |
| + | Financed in March 2024. | ||
| ===== Miscellaneous Known Issues and Workarounds ===== | ===== Miscellaneous Known Issues and Workarounds ===== | ||
| Line 151: | Line 154: | ||
| If your KGPE-D16 server or workstation is equipped with a discrete graphics processor, the on-board VGA is disabled. Petitboot builds of Vikings do not have drivers for most discrete graphics processors, so there is no boot screen on which to select boot media, for example. Your display should start working as soon as Linux from an installed drive has been loaded. | If your KGPE-D16 server or workstation is equipped with a discrete graphics processor, the on-board VGA is disabled. Petitboot builds of Vikings do not have drivers for most discrete graphics processors, so there is no boot screen on which to select boot media, for example. Your display should start working as soon as Linux from an installed drive has been loaded. | ||
| - | A workaround for this is to enable the VGA jumper and connect a screen to the on-board VGA output because Petitboot has the required drivers for the on-board GPU (Aspeed AST2050) included. This is useful when installing a new operating system for example. The jumper settings can be reverted afterwards. | + | A workaround for this is to enable the VGA jumper and connect a screen to the on-board VGA output because Petitboot has the required drivers for the on-board GPU (Aspeed AST2050) included. This is useful when installing a new operating system for example. The jumper settings can be reverted afterwards. |
| ==== EHCI Debug Console ==== | ==== EHCI Debug Console ==== | ||
| Line 183: | Line 185: | ||
| Turbo Examples: If you have a 16 core CPU to obtain Turbo 2 you would select 2 modules and thus 4 cores from each MCM/NUMA node - then you allocate all of the hugepages/ | Turbo Examples: If you have a 16 core CPU to obtain Turbo 2 you would select 2 modules and thus 4 cores from each MCM/NUMA node - then you allocate all of the hugepages/ | ||
| - | ===== Free Software Foundations " | ||
| - | The [[https:// | + | ====== Operating Systems Findings ====== |
| - | ===== External links ===== | + | ===== Qubes ===== |
| - | * [[https://www.asus.com/us/Commercial-Servers-Workstations/KGPED16/|Official product page]] | + | |
| - | * [[https:// | + | |
| - | * [[https://vikings.net/downloads/KGPE-D16_manual.pdf|Official manual (mirror)]] | + | |
| + | ====== Free Software Foundations " | ||
| + | |||
| + | The [[https://shop.vikings.net/product/vikings-kgpe-d16-mainboard-fsf-ryf-certified/|Vikings D16]] (a relabeled KGPE-D16) board is being sold with coreboot pre-installed. It is the first workstation/server mainboard that has been "RYF - Respects Your Freedom" | ||
| + | |||
| + | ====== External links ====== | ||
| + | |||
| + | * [[https:// | ||
| + | * [[https://www.raptorengineering.com/coreboot/kgpe-d16-bmc-port-status.php|OpenBMC on the ASUS KGPE-D16]] | ||
| [[Category: | [[Category: | ||
| + | |||
hardware/kgpe-d16.1681667805.txt.gz · Last modified: 2023/04/16 17:56 by 127.0.0.1