- 1 AMD's Compute-GPU Model
- 1.1 GCN3 Based Simulation
- 1.2 HSAIL Based Simulation
- 2 ARM's NoMali GPU Model
AMD's Compute-GPU Model
GCN3 Based Simulation
An HPCA paper was published in 2018 that describes the GCN3 model.
ISCA 2018 tutorial
A tutorial was held on June 2nd, 2018, in conjuction with the 45th International Symposium on Computer Architecture (ISCA). Our presentation can be found here: The AMD gem5 APU Simulator: Modeling GPUs Using the Machine ISA. The GCN3 ISA is supported in AMD's public pre-release gem5 repo on the branch agutierr/master-gcn3-staging.
Cloning the repository
To clone the repo with GCN3 support use the following command:
git clone https://gem5.googlesource.com/amd/gem5 -b agutierr/master-gcn3-staging
Building the simulator with GPU and GCN3 support
Currently, the GPU model only works with X86 and the VIPER protocol, which you can read about in the slides from AMD's 2018 ISCA tutorial. To build gem5 with a GCN3-based GPU model included use the following command:
scons -sQ -jN ./build/GCN3_X86/gem5.opt
Simulation support for ROCm
In contrast to HSAIL execution, the GCN3 model does not rely on an emulated runtime (i.e., a simulator-specific implementation of the GPU runtime API). Instead, the model was designed with enough fidelity to run the userspace components of an off-the-shelf version of the Radeon Open Compute platform (ROCm). ROCm is an open platform from AMD that implements Heterogeneous Systems Architecture (HSA) principles. More information about the HSA standard can be found on the HSA Foundation's website.
The model currently only works
with system-call emulation (SE) mode, therefore all kernel level driver functionality is modeled entirely within the
SE mode layer of gem5. In particular, the emulated GPU driver supports the necessary
it receives from the userspace code. The source for the emulated GPU driver can be found in:
- The GPU compute driver:
- The HSA device driver:
The HSA driver code models the basic functionality for an HSA agent, which is any device that can be targeted by the HSA runtime and accepts Architected Query Language (AQL) packets. AQL packets are a standard format for all HSA agents, and are used primarily to initiate kernel launches on the GPU. The base
HSADriver class holds a pointer to the HSA packet processor for the device, and defines the interface for any HSA device. An HSA agent does not have to be a GPU, it could be a generic accelerator, CPU, NIC, etc.
GPUComputeDriver derives from
HSADriver and is a device-specific implementation of an
HSADriver. It provides the implementation for GPU-specific
ROCm tool chain and software stack
In order to build and run applications for ROCm and GCN3 you need several ROCm components. These are:
- Heterogeneous Compute Compiler (HCC)
- Radeon Open Compute runtime (ROCr)
- Radeon Open Compute thunk (ROCt)
- HIP (optional)
Only the roc-1.6.x branch of the necessary ROCm components are supported, so be sure to include
-b roc-1.6.x when cloning.
The recommended compiler to build these components is gcc 5.4.0.
Alternatively, there are deb and yum packages for ROCm archived here: ROCm archive.
When building gem5's GPU model you must make sure that the
src/dev/hsa/kfd_ioctl.h header matches the
kfd_ioctl.h header that comes with ROCt.
The emulated driver,
src/gpu-compute/gpu_compute_driver.[hh|cc] relies on this file to interpret the
ioctl() codes that the thunk
Before running any ROCm-based GPU applications in gem5, you need to make sure that the simulated environment is set properly in
LD_LIBRARY_PATH in particular must point to the ROCm installation on your local machine.
Example HC applications
No GPU applications have been included with the release of GCN3/ROCm support in gem5, however there are several public repositories with HC/AMP applications that are known to work with gem5's GCN3-based GPU model:
These applications may contain host-to-device copies, which are not necessary in the APU systems modeled in gem5. It is advised that you remove all copies in these applications and instead directly pass pointers to host-allocated data structures to the GPU kernels. This if fine for an APU because a "pointer is a pointer" and may be shared across the host and device.
HSAIL Based Simulation
The HSAIL-based GPU model is still the model that is included in the mainline gem5 repository, however it is no longer supported and will be deprecated once GCN3 support is fully merged into the mainline. It is recommended that users start with the GCN3 model in AMD's public pre-release repository.
A tutorial was held in conjunction with MICRO-48. We have made the slides available from our 2015 tutorial titled: The AMD gem5 APU Simulator: Modeling Heterogeneous Systems in gem5.
Emualted CL Runtime
- Download the emulated OpenCL runtime.
CLOC is used to compile OpenCL kernels for use with gem5's GPU compute model. The most recent revision of CLOC that is known to work with gem5 is:
ARM's NoMali GPU Model
The NoMali GPU model models the interface used by ARM Mali GPUs. The model does not render or compute anything, but can be used to fake a GPU. This enables Android and ChromeOS experiments without software rendering which would otherwise make simulation results extremely misleading. It was presented in the 2015 gem5 User Workshop.
Getting started instructions are currently available for Android 4.4 (KitKat).