gem5

ISCA 2025: Toward Full-System Heterogeneous Simulation: Merging gem5-SALAM with Mainline gem5

Wed, 30 Jul 2025 00:00:00 +0000

Towards Full-System Heterogeneous Simulation in gem5

As SoC architectures grow increasingly heterogeneous, they now integrate not only CPUs and GPUs but also tightly coupled programmable accelerators tailored for specific workloads. These accelerators are critical for emerging domains such as mobile inference, AR/VR, real-time vision, and edge analytics. Unlike traditional CPU-GPU systems, modern heterogeneous platforms demand fine-grained coordination among diverse compute engines, shared memory subsystems, and software-managed execution models. Capturing these interactions requires a cycle-level, full-system simulator.

While gem5 has long supported detailed CPU simulation and, more recently, full-system GPU modeling, support for programmable accelerators remained external via tools like gem5-SALAM—built on gem5 v21.1. Although SALAM added accelerator-specific capabilities such as cycle-level datapath modeling, memory-mapped scratchpads, and hardware synthesis integration, it was isolated from the mainline. As a result, it could not leverage recent ISA, memory system, or configuration infrastructure updates, nor benefit from upstream validation.

To close this gap, we integrated SALAM’s accelerator infrastructure into gem5 mainline (develop branch v25). This unification elevates accelerators to first-class components alongside CPUs and GPUs, enabling full-system heterogeneous simulation under a single software stack. The result is a unified framework for modeling heterogeneous SoCs with realistic OS support, shared resource contention, and software-controlled task orchestration.

Integration at a Glance

We integrated SALAM’s accelerator modeling infrastructure into gem5-develop through a series of architectural, interface, and validation updates.

We began by integrating key accelerator modeling components from SALAM into gem5. These include the LLVMInterface, which executes LLVM IR kernels using a cycle-accurate datapath; the CommInterface, which provides software-visible control and interrupt signaling; and a suite of configurable memory components such as scratchpads, DMA engines, and stream buffers. Together, these elements enable detailed and flexible modeling of a wide range of accelerator microarchitectures and memory hierarchies. To support realistic SoC integration, accelerators and local memories can be grouped into an AccCluster, reflecting the modular structure of accelerator subsystems commonly found in commercial SoCs. For rapid prototyping, we also integrated and automated SALAM’s hardware profile generator, which converts user-defined timing specifications into executable datapath models – eliminating the need for manual microarchitectural implementation. Finally, we refactored CACTI-SALAM for compatibility with gem5’s infrastructure, enabling timing and energy estimation for scratchpad memories using CACTI’s file-based configuration methodology. These changes bring cycle-level accelerator modeling, full-system memory interaction, and scalable design space exploration into gem5 mainline.

We then updated SALAM’s accelerator infrastructure to match gem5’s latest design conventions. This included refactoring classes to use modern SimObject patterns, replacing unsafe pointer casts in LLVM instruction handling with type-safe 32-bit variables, and switching to gem5’s standardized random number generator for latency modeling. We fixed off-by-one errors in address range definitions to follow gem5’s inclusive-exclusive semantics, aligned environment and ISA configuration with gem5’s current setup, and added dynamic LLVM detection using llvm-config to simplify SCons-based compilation for datapath simulation.

Finally, we validated the integrated framework by ensuring it passed gem5’s pre-commit checks and full regression test suite. Additionally, we adapted SALAM’s original system validation tests to run within the unified environment and cross-validated the outputs against the original SALAM baseline to confirm functional equivalence. We plan to upstream these accelerator tests to the gem5-resources repository to support broader validation of the integrated SALAM components within gem5.

What This Enables

Broader Heterogeneity Studies

With accelerators now fully integrated into gem5 mainline, researchers can simulate complete heterogeneous systems comprising CPUs, GPUs, and custom accelerators—co-existing under a single OS kernel and sharing interconnects and memory. This allows detailed studies of performance interference, resource arbitration, and synchronization mechanisms across diverse compute engines, grounded in full-system behavior rather than simplified models.

System-Level Exploration

The framework supports rich exploration of architectural tradeoffs at the system level. Users can evaluate different memory organizations—such as private scratchpads, shared LLCs, or DMA-managed SPMs—and compare strategies for offloading, synchronization, and kernel placement. Static vs. dynamic scheduling, locality-aware memory partitioning, and software-managed DMA schemes can all be studied in realistic OS-driven settings.

Domain-Specific Workload Support

This infrastructure also enables architectural research targeting emerging domains like real-time vision, mobile inference, AR/VR, and edge computing. These applications demand predictable latency, software-accelerator coordination, and careful memory management. The integrated framework allows researchers to model and study these workloads using real software stacks and bootable Linux images, with accelerator behavior simulated at cycle-level fidelity.

Exploratory Studies in Non-Traditional Regimes

Finally, the toolchain enables exploration of accelerator operation under emerging regimes such as transient overclocking and advanced cooling. In our workshop paper, we use this framework to study one such case of a non-traditional operating regime: multi-GHz frequency scaling in accelerators, enabled by advanced cooling techniques such as immersion and cryogenic systems. We present a preliminary analysis of performance and power upper bounds across this range. The results show how system bottlenecks shift with increasing frequency, highlighting the importance of evaluating accelerator behavior in the context of host latency and memory interactions. Full details of the experimental setup and findings are included in our ISCA ’25 workshop paper.

Users can apply this framework to the use cases discussed above using built-in accelerator models and benchmarks, or extend it further by modeling their own custom accelerators.

Modeling Your Own Accelerator

Creating a new accelerator model in the integrated gem5 framework is simple. You begin by writing the desired accelerator algorithm in C/C++ and compiling to LLVM IR. A YAML-based hardware profile specifies instruction timing, functional unit latencies, and memory ports. This profile is processed by the hardware-profile generator to produce a cycle-level timing model.

The user then places the accelerator inside an AccCluster, attaches scratchpads or DMAs as needed, and configures the system topology using gem5’s Python interface. A host-side program running in the simulated OS coordinates with the accelerator via memory-mapped control registers and interrupts. The complete system is simulated using run_system.sh, producing statistics, optional power reports, and host-side console output.

Getting Started

To get started, set the following environment variables to your gem5 and benchmark root directories:

export M5_PATH=/path/to/gem5
export ACC_BENCH_PATH=/path/to/benchmarks

Clone and build gem5:

git clone https://github.com/akanksha-sc/gem5
cd gem5
scons build/ARM/gem5.opt -j$(nproc)

To generate a custom hardware profile (optional):

$M5_PATH/tools/hw_generator/HWProfileGenerator.py -b <benchmark_name>

To run CACTI-SALAM (optional energy/area estimation):

cd $M5_PATH/tools/cacti-SALAM
./run_cacti_salam.py --bench-list $ACC_BENCH_PATH/benchmarks.list

Run a benchmark (custom or built-in like bfs):

$M5_PATH/tools/run_system.sh --bench <benchmark_name> --bench-path <benchmark_path>

This boots Linux, launches a user-space driver, and simulates the accelerator. Outputs include stats.txt (performance counters), system.terminal (host console output), SALAM_power.csv (power/area estimates, if CACTI-SALAM is used). Additional examples and documentation included in src/hwacc/docs.

Conclusion

This integration positions gem5 as a unified, full-system simulator for heterogeneous SoCs—combining CPUs, GPUs, and programmable accelerators under one framework with realistic timing, software, and architectural detail. It opens the door to studies ranging from co-scheduling and memory-system tuning to high-frequency accelerator and advanced-cooling analyses. Next steps include merging the support into gem5 mainline, expanding the benchmark suite with domain-specific workloads, and extending full-system accelerator support to additional ISAs. We hope this foundation accelerates heterogeneous-system research across the community.

Acknowledgments

This work is supported in part by the Semiconductor Research Corporation and by the DOE’s Office of Science, Office of Advanced Scientific Computing Research through EXPRESS: 2023 Exploratory Research for Extreme Scale Science.

References

A. Chaudhari and M. D. Sinclair. “Toward Full-System Heterogeneous Simulation: Merging gem5-SALAM with Mainline gem5.” 6th gem5 Users’ Workshop, June 2025.
S. Rogers, J. Slycord, M. Baharani and H. Tabkhi, “gem5-SALAM: A System Architecture for LLVM-based Accelerator Modeling,” 2020 53rd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), Athens, Greece, 2020, pp. 471-482, doi: 10.1109/MICRO50266.2020.00047.

Running Bao Hypervisor on gem5

Tue, 12 Nov 2024 00:00:00 +0000

¹ Ritsumeikan University

This article presents a methodology for executing the Bao Hypervisor [1] on gem5 via the VExpress_GEM5_Foundation platform, enabling executions of two unmodified Linux virtual machines simultaneously.

It was tested on “x86_64 Ubuntu 22.04”.

Obtain and Build Software

Install the tools required to run Bao Hypervisor on gem5.

sudo apt update
sudo apt install git tree scons gcc g++ python3-dev \ 
make automake device-tree-compiler \
bison flex libssl-dev

Create a directory to store build outputs:

mkdir -p resources/binaries resources/semihosting

Download and setup the toolchain

Download the latest bare-metal cross-compile toolchain “aarch64-none-elf” from the Arm Developer’s website. Tested on “arm-gnu-toolchain-13.3.rel1-x86_64-aarch64-none-elf.

Once the cross compiler for aarch64 has been downloaded, the file should be extracted.
Then, define the cross compiler prefix to the CROSS_COMPILE environment variable.

The following example demonstrates the installation of “arm-gnu-toolchain-13.3.rel1-x86_64-aarch64-none-elf.tar.xz”.
When setting the environmental variable, be careful not to forget a “-” at the end.

# Downloaded "arm-gnu-toolchain-13.3.rel1-x86_64-aarch64-none-elf.tar.xz" 
# to current directory, then following command should be executed.
tar xvf arm-gnu-toolchain-13.3.rel1-x86_64-aarch64-none-elf.tar.xz
export CROSS_COMPILE=$(pwd)/arm-gnu-toolchain-13.3.rel1-x86_64-aarch64-none-elf/bin/aarch64-none-elf-

Set up base environment and build automation

Obtain the git repository for building Bao Hypervisor and Linux. Then, build it using the make command.

git clone https://github.com/h1demasa/bao-demos.git
cd bao-demos
make -j $(nproc)

Once the build is completed, copy the generated binary to the directory that gem5 will use.

cp wrkdir/imgs/fvp-a/fip.bin ../resources/binaries/fip.bin
cp wrkdir/imgs/fvp-a/bl1.bin ../resources/binaries/bl1.bin
cp wrkdir/imgs/fvp-a/linux+linux/bao.bin ../resources/semihosting/bao.bin

Before proceeding, return to the parent directory and check that your resources directory has the following structure:

cd ../
tree resources/

resources
    binaries
        bl1.bin
        fip.bin
    semihosting
        bao.bin

gem5 (simulator)

To run Bao Hypervisor with gem5, build gem5.

As the Performance Monitors Control Register is not implemented in gem5, numerous warning logs are generated when Bao is running. These logs affect gem5’s execution performance. To prevent these warning logs, apply the following patch to gem5.

git clone https://github.com/gem5/gem5.git -b v24.0.0.1
cd gem5
git apply ../bao-demos/platforms/gem5/gem5.patch

Build gem5 with the above changes:

scons -j $(nproc) build/ARM/gem5.opt
cd -
make -C gem5/util/term

Run and Connect to the Simulation

Now you can run the simulation:

export M5_PATH=resources/ && \
	gem5/build/ARM/gem5.opt gem5/configs/example/arm/baremetal.py \
	   --workload ArmTrustedFirmware \
	   --num-cores 4 \
	   --mem-size 4GB \
	   --machine-type VExpress_GEM5_Foundation \
	   --semi-enable --semi-path resources/semihosting

And connect to it:

gem5/util/term/m5term 3456 # Trusted Firmware-A, U-boot and Bao Hypervisor
gem5/util/term/m5term 3457 # VM 1 (Linux)
gem5/util/term/m5term 3458 # VM 2 (Linux)

It is recommended that you observe the various stages of the bootloader and the text indicating the Bao Hypervisor boot process, which can be accessed via port 3456. Similarly, the Linux kernel boot sequence can be observed via port 3457 and port 3458. Once the boot process is completed, A login shell is started. You can access it via ports 3457 and 3458. The username and password for this login are “root”.

In my environment, it took an hour for the Linux login shell to start up.
Please wait while enjoying a cup of coffee.

Demo

This is a demo video of running Bao Hypervisor on gem5.

[1] José Martins, Adriano Tavares, Marco Solieri, Marko Bertogna, and Sandro Pinto. “Bao: A Lightweight Static Partitioning Hypervisor for Modern Multi-Core Embedded Systems”. In Workshop on Next Generation Real-Time Embedded Systems (NG-RES 2020). Schloss Dagstuhl-Leibniz-Zentrum für Informatik. 2020. https://drops.dagstuhl.de/opus/volltexte/2020/11779/

gem5 Version 23.1 Release: A Leap Forward in Computer Architecture Simulation

Thu, 21 Dec 2023 00:00:00 +0000

The gem5 computer architecture simulation tool has released its latest milestone - Version 23.1. This release marks a significant step in gem5’s development as it transitions to GitHub and introduces several groundbreaking features and improvements. In this blog post, we’ll dive into the key changes and enhancements that come with gem5 v23.1.

A Shift to GitHub Development

One of the most notable changes in gem5 v23.1 is the transition to GitHub for development. This shift to a popular platform for collaborative software development fosters greater transparency and community involvement. During this release, an impressive 362 pull requests were merged, comprising 416 commits from 51 unique contributors. This level of community engagement reflects gem5’s commitment to open-source collaboration.

Streamlined Configuration with Kconfig

The gem5 simulator’s build configuration has undergone a transformation with the introduction of kconfig. While most gem5 builds remain backward-compatible, specialized builds now require the use of scons <kconfig command>. This change allows users to tailor their gem5 installations to specific needs and is detailed in the kconfig documentation.

Standard Library Improvements

The gem5 simulator’s v23.1 release introduces several standard library improvements, including the introduction of WorkloadResource to resource specialization. The older Workload and CustomWorkload classes have been deprecated in favor of the new obtain_resource and WorkloadResource classes in resource.py. Users can easily update their code by following the provided migration instructions.

Moreover, gem5 introduces “Suites”, a new category of resource. These suites enhance the versatility and functionality of gem5 simulations, opening up new possibilities for users.

API Changes and Enhancements

Version 23.1 brings several changes to the API, making it more informative and flexible. Resource objects now have their own id and category, and the __str__() function provides detailed information. Users can utilize environment variables like GEM5_RESOURCE_JSON and GEM5_RESOURCE_JSON_APPEND to customize data sources using JSON files, offering more control over resource management.

Support for Latest Tools and Platforms

Version 23.1 of gem5 ensures compatibility with the latest tools and platforms, with added support for clang 15 and clang 16. However, it’s important to note that gem5 no longer supports building on Ubuntu 18.04, and GCC 7, GCC 9, and clang 6 are no longer supported.

Full-System GPU Model Enhancements

This release significantly enhances the full-system GPU model. Users can now leverage gem5 to simulate the latest ROCm 5.7.1, and various updates enable PyTorch and TensorFlow simulations. The inclusion of a new packer disk image script containing ROCm 5.4.2, PyTorch 2.0.1, and TensorFlow 2.11 opens up exciting possibilities for GPU simulation.

RISC-V RVV 1.0 Implementation

A major achievement in gem5 v23.1 is the implementation of RISC-V RVV 1.0. This feature, the result of collaborative efforts from numerous contributors, brings most of the instructions in the 1.0 specification to life. RVV 1.0 is compatible with various CPU models, supports both FS and SE modes, and allows users to specify vector unit widths. While there is room for future improvements, this implementation is a significant step forward for RISC-V support in gem5.

ArmISA Changes and Improvements

Version 23.1 of gem5 introduces architectural support for several Arm extensions, enhancing SVE instruction support, addressing FEAT_SEL2 related issues, and implementing an Arm Capstone Disassembler. These changes and improvements further solidify gem5’s compatibility with Arm-based systems.

Other Notable Changes

Several other notable changes and improvements have been made, including:

Coherence protocol enhancements
Far atomics in CHI
Improved support for RISC-V privilege modes
Bug fixes and optimizations

Known Bugs and Issues

While gem5 v23.1 brings significant improvements, there are some known bugs and issues that the development team is actively addressing. The gem5 community encourages users to report any issues they encounter to help improve the tool’s reliability.

In conclusion, gem5 Version 23.1 is a remarkable release that reflects the project’s continued commitment to excellence, collaboration, and innovation in the field of computer architecture simulation. As gem5 transitions to GitHub, it opens new doors for community involvement and further advancements. Researchers, developers, and enthusiasts alike can look forward to an enhanced simulation experience with gem5 v23.1.

To get started with gem5 v23.1 and explore its new features, visit the official gem5 website and consult the documentation. Join the gem5 community, provide feedback, and contribute to the evolution of this powerful simulation tool. Your engagement is essential in shaping the future of computer architecture research and development.

Benchmarking Linkers within gem5

Thu, 16 Feb 2023 00:00:00 +0000

tl;dr: Use the mold linker for the fastest linking times when building gem5

People familiar with gem5 are aware of its lengthy compilation time, especially during the linking stage. This can become frustrating when even a minor edit necessitates re-linking previously compiled files. This can add several minutes to the process. Keeping this in mind, we evaluated a range of currently supported linkers with gem5 to determine which one is the most efficient.

To conduct these tests, we closely examined each of the linkers that gem5 currently supports including the current default linker, “ld”. The four additional supported linkers evaluated are “lld”, “bfd”, “gold”, and “mold”.

Our method for comparing these linkers was as follows: we first built gem5 normally by executing scons build/ALL/gem5.opt. Once the gem5.opt binary was compiled, we deleted it. Thus we were left with the compiled object files but no linked binary. Then, we compared runs of rebuilding/linking gem5 with each of the linkers using /usr/bin/time scons build/ALL/gem5.opt -j12 --linker=[linker-option], where /usr/bin/time measured the duration. We ran these tests on a system using an AMD EPYC 7402P 24-Core processor with a 3.35 GHz frequency.

During these tests, we observed that using a linker other than the default “ld” in our experimental setup forced a recompilation of all of the m5 files. However, if we deleted gem5.opt binary after this and ran the compilation again, only gem5.opt was rebuilt/linked, resulting in two distinct times. To compare the times, we needed to take into account the time it took for the first run to build all the m5 files, as well as the time it took for the second run to re-link gem5.opt. These are labeled below as “all m5” versus “last few”. In addition to these two runs, we also compared each run on a networked file system (NFS) and a local SSD to see if the storage type of the files had any impact on the run times. Finally, we performed one last run on the local SSD using the 48 available cores on our system to assess if it made any difference. Below, we present the elapsed time for each of these runs.

We found that among the four linkers we tested, “bfd” was the slowest, and “mold” was the fastest. Additionally, the difference between using -j12 and -j48 appeared to be insignificant.

Based on our results, we suggest using “mold” as the linker when working with gem5. It’s worth noting that using a particular linker had a more significant impact on the time taken than the storage location.

	NFS + all m5	NFS + last few	Local SSD + all m5	Local SSD + last few	Local SSD + -j48 + last few
ld	—	3:29.19	—	3:08.31	3:00.15
bfd	4:15.82	3:32.13	3:39.70	3:02.15	3:02.35
lld	2:16.22	1:54.25	1:52.94	1:13.12	1:13.16
gold	2:30.98	1:43.59	1:59.41	1:19.86	1:19.48
mold	1:48.62	1:07.08	1:08.18	0:28.23	0:27.89

In addition to just comparing the build times, we executed 100000000 ticks for each linked compilation to ensure that using these linkers wouldn’t cause any issues when actually using gem5, such as increased execution time or functional problems.

We achieved this by performing an x86 linux boot with an O3 CPU and a Ruby cache. The command to do so is provided below.

Command:

/usr/bin/time build/ALL/gem5.opt -re tests/gem5/configs/x86_boot_exit_run.py --cpu o3 --num-cpus 2 --mem-system mesi_two_level --dram-class DualChannelDDR4_2400 --boot-type init --resource-directory tests/gem5/resources --tick-exit 100000000

We found none of the linkers had a significant impact on the runtime of any tests and all tests completed successfully. This indicates that using linkers should not have any detrimental effects on experiments conducted within gem5.

Based on our findings, we can confidently recommend using the mold linker to speed up linking times when building gem5. If you’re interested in using mold, you can follow the instructions here to compile it. Once it’s properly installed, you can use it by passing --linker=mold while building gem5.

Here’s an example command: scons build/ALL/gem5.opt -j12 --linker=mold.

“Moving to full system simulation of GPU applications”

Mon, 13 Feb 2023 00:00:00 +0000

For over a decade gem5 has supported two modes of simulation: full system (FS) mode where the simulator uses a disk image and a kernel to boot an instance of Linux and run applications on the disk image and System-emulation (SE) mode where the simulator runs applications on the host machine and intercepts system calls and provides emulation for them. Up until a few years ago, binaries run in SE mode were required to be statically linked to be run. Dynamically linked binaries can now be run assuming the dynamic libraries are available on the host machine. For increasingly more complicated and specialized applications, such as GPU applications, the dynamic libraries may not be available on the host system or may be a different version than what is needed for the simulated application. In these cases, using FS mode is preferred.

This issue of unavailable or different version libraries also occurs with the GPU model in gem5. The GPU model currently runs in SE mode and requires an older version of AMD’s ROCm™ stack. This is problematic for a few reasons: (1) the user may not have a GPU and therefore does not need the ROCm™ stack installed locally (2) the user may not be on a system compatible with the ROCm™ installer to install the libraries or (3) the user has ROCm™ installed but does not have the specific version required for gem5. These issues are currently solved by running building and running gem5 using a docker image. This is not necessary when using FS mode, making GPUFS easier to run along with regular simulation.

Over the past two years, work has been being done to implement a simulated GPU device with all of the necessary components to communicate with the upstream GPU driver. With this work in place, it is now possible to use FS mode to simulate GPU applications. With the 22.1 release of gem5, we are announcing GPU FS mode (GPUFS) as the preferred method to simulate GPU applications and will eventually replace SE mode entirely. Based on our most recent testing, nearly all applications which worked in SE mode will work in FS mode. The remainder of this blog post discusses the use cases of FS mode along with additional benefits and known issues.

Use cases

The use cases for GPUFS are the same as SE mode GPU simulation. That is, we simulate a single GPU application, collect stats, and exit the simulation. Although GPUFS in theory provides the ability to do more advanced simulations such as simulating concurrent GPU applications or simulating multiple GPU devices, these are not supported in the current model.

Benefits of Full System

The primary benefit of GPUFS is avoiding issues with dynamic libraries. Currently SE mode GPU simulations need to be run within a docker image. This itself has many user facing complexities, such as environments (e.g., universities) which do not allow docker to be run and potentially different build directories for GPU simulation and non-GPU simulation, and many developer complexities including testing and keeping up with download locations of older out-of-date libraries.

With a simulated GPU device users will be able to fast-forward through memory copies in GPU applications. A basic GPU application has three main GPU-related library calls: (1) copy data to the GPU, (2) launch a kernel on the GPU, and (3) copy data to the host. On a real system data can be copied using a GPU kernel which reads from host memory and writes to device memory or by using the help of a DMA engine. With GPUFS, system DMA engines are implemented to copy data to/from GPU memory. These engines can be simulated functionally within gem5 to speed up simulation. As a result, users can copy several GBs of data to GPU memory in minutes of simulation time by avoiding the detailed simulation of copy kernels.

GPUFS also more easily allows users and developers to update the ROCm™ stack to the latest version with each gem5 release. This allows users to be able to use features of the latest ROCm™ stack which can mean less time spent backporting applications to older ROCm™ versions. GPUFS has currently been tested on ROCm 4.2, 4.3, 5.0, and 5.4 but any version above 4.0 should work. This testing was done on the core ROCm package only, so 1st party libraries (rocBLAS, rocFFT, rocSPARSE, etc.) have not been thoroughly tested.

Full system mode uses the full ROCm stack, including the kernel driver, rather than using the emulated driver developed for SE mode. This means users can modify the Linux kernel driver to research areas that are difficult to do in SE mode. Examples include virtual memory research such as utilizing flexible page sizes and exploring page fault handling, implementing new packet types for the new SDMA and PM4 processors, or using virtualization features.

Using full system

Like FS mode in general, users need a disk image and a kernel to run GPUFS. A packer script is provided in the gem5-resources repository under src/gpu-fs/disk-image. Additionally, the kernel is available for download or can be transferred out of the disk image. This disk image and kernel consist of an operating system and kernel version compatible with the official ROCm™ release notes. A prebuilt GPUFS disk image and GPUFS kernel are available for download.

Scripts are provided for the user which can take a GPU application as an argument and copy it into the disk image upon simulation start to run a GPU application without needing to modify or mount the disk image. The traditional “rcS” script approach can also be used to run applications which exist on the binary already, applications which may need further input files, or applications the user wishes to build from source files in the disk image. Applications can be built using a docker image provided at gcr.io/gem5-test/gpu-fs:v22-1 or building a local docker image using util/dockerfiles/gpu-fs/ in the gem5 repository. Using this docker allows users to build GPU applications without needing to install ROCm™ on their host machine and without wasting simulation time building source files on the disk image. If desired, users may also install the required ROCm™ version locally, even without an AMD GPU, and build applications on their host machine.

More information on how to setup GPUFS is provided in the README.md file in the gem5-resources repository at src/gpu-fs/README.md.

Known issues

There are some known issues that are actively being addressed which will not be completed until a future release after gem5 22.1. These issues are below. If you are using GPUFS and run into an issue that is not listed here, we encourage you to report the issue to gem5-users, JIRA, or the gem5 slack channel. A useful bug report will include both terminal output and gem5 output preferably with the following debug flags: --debug-flags=AMDGPUDevice,SDMAEngine,PM4PacketProcessor,HSAPacketProcessor,GPUCommandProc.

Currently KVM and X86 are required to run full system. Atomic and Timing CPUs are not yet compatible with the disconnected Ruby network required for GPUFS and is a work in progress.
Some memory accesses generate incorrect addresses causing hard page faults leading to simulation panics. This is currently being investigated with high priority.
The printf function does not work within GPU kernels. As a workaround, a gem5-specific print function is being developed.

Recap

Full system GPU simulation (GPUFS) is now the preferred method to run GPU applications in gem5 22.1+. GPUFS is intended to be used for the same use cases are SE mode GPU simulation. It has the benefits of avoiding simulation within docker, improved simulation speed by functionally simulating memory copies, and an easier update path for gem5 developers.

As users move to GPUFS, we expect there will be some bug reports. Users are encouraged to submit reports to the gem5-users mailing list, JIRA, or gem5 slack channel.

gem5-22.1 Released!

Fri, 30 Dec 2022 00:00:00 +0000

The latter half of 2022 saw 500 commits submitted to gem5, from 48 unique contributors

Our top 10 contributors for the v22.1 release were:

Bobby R. Bruce (143 commits)
Giacomo Travaglini (63 commits)
Gabe Black (53 commits)
Matthew Poremba (49 commits)
Jason Lowe-Power (19 commits)
Yu-hsin Wang (18 commits)
Zhantong Qiu (11 commits)
Earl Ou (10 commits)
Tiago Mück (9 commits)
Quentin Forcioli (9 commits)

We wish to thank all of the gem5 community who made this gem5 release possible. We look forward to your continued support for the upcoming v23.0 release.

The latest version of gem5 can be obtained by pulling the latest version of the gem5 git repo stable branch:

git clone https://gem5.googlesource.com/public/gem5

# or, within the gem5 repo

git switch stable
git pull

Major New Features

Several new features were included in the v22.1 release. Here we outline some key highlights and how to use them. For a more comprehensive list of contributions, please consult the Release Notes.

Multiple ISA Compilations

Prior to v22.1 users of gem5 could only compile a single ISA target into a gem5 binary. E.g., scons build/X86/gem5.opt would create a gem5 binary with the X86 ISA but no other. In gem5 v22.1 users can compile any combination of ISAs into a gem5 binary. The build_opts directory includes ALL which can be used to compile a binary containing all ISA targets with scons build/ALL/gem5.opt.

With multi-ISA binaries the ISA used in simulation specified via which core is used. For example, using an X86TimingSimpleCPU will use the X86 ISA. When using the standard library API, you can carry out simulations as before by specifying the ISA to setting up the system processor.

Below is a simple ARM program in SE mode:

from gem5.isas import ISA
from gem5.utils.requires import requires
from gem5.resources.resource import Resource
from gem5.components.memory import SingleChannelDDR3_1600
from gem5.components.processors.cpu_types import CPUTypes
from gem5.components.boards.simple_board import SimpleBoard
from gem5.components.cachehierarchies.classic.no_cache import NoCache
from gem5.components.processors.simple_processor import SimpleProcessor
from gem5.simulate.simulator import Simulator

# This ensures that the ARM ISA is compiled into the binary.
requires(isa_required=ISA.ARM)

cache_hierarchy = NoCache()
memory = SingleChannelDDR3_1600(size="32MB")

# Here we must specify the ISA we are using via the `isa` parameter.
processor = SimpleProcessor(cpu_type=CPUTypes.TIMING, isa=ISA.ARM, num_cores=1)

board = SimpleBoard(
    clk_freq="3GHz",
    processor=processor,
    memory=memory,
    cache_hierarchy=cache_hierarchy,
)

board.set_se_binary_workload(Resource("arm-hello64-static"))

simulator = Simulator(board=board)
simulator.run()

This can be run with build/ALL/gem5.opt and ARM ISA will automatically be used as it was specified via the isa parameter when setting SimpleProcessor.

API to set tick-based Exit Events

It is common when running gem5 to want the simulation to Exit at a particular simulation tick. The typical manner in which this is achieved is by moving the MAX_TICK value of the simulation but this is limiting as it only allows for a single tick to be specified as en exit event. Until v22.1, the API for specifying exits at others ticks was not well exposed and difficult to use. As such the following functions have been added to the m5 Python module:

setMaxTick(tick) : Used to to specify the maximum simulation tick.
getMaxTick() : Used to obtain the maximum simulation tick value.
getTicksUntilMax(): Used to get the number of ticks remaining until the maximum tick is reached.
scheduleTickExitFromCurrent(tick) : Used to schedule an exit exit event a specified number of ticks in the future.
scheduleTickExitAbsolute(tick) : Used to schedule an exit event as a specified tick.

The setMaxTick function provides a cleaner interface for setting the maximum tick the simulation is to run to. When reached a MAX_TICK exit event is returned by the simulator. The scheduleTickExit functions allow for the scheduling of any number of tick exit events. When reached a SCHEDULED_TICK exit event is returned by the simulator. The scheduleTickExitFromCurrent function schedules the exit event N ticks in the future, with N being provided by the user. The scheduleTickExitAbsolute function allows for the scheduling at a specific simulation tick (e.g., at Tick 1,000).

Below is a simple simulation showing the scheduling of SCHEDULED_TICK exit events at 100, 1000, 10000, and 100000 ticks in the future.

from gem5.resources.resource import Resource
from gem5.isas import ISA
from gem5.components.memory import SingleChannelDDR3_1600
from gem5.components.boards.simple_board import SimpleBoard
from gem5.components.cachehierarchies.classic.no_cache import NoCache
from gem5.components.processors.simple_processor import SimpleProcessor
from gem5.components.processors.cpu_types import CPUTypes
from gem5.simulate.simulator import Simulator
from gem5.simulate.exit_event import ExitEvent

import m5

board = SimpleBoard(
    clk_freq="3GHz",
    processor=SimpleProcessor(
        cpu_type=CPUTypes.TIMING,
        isa=ISA.X86,
        num_cores=1,
    ),
    memory=SingleChannelDDR3_1600(),
    cache_hierarchy=NoCache(),
)
board.set_se_binary_workload(Resource("x86-hello64-static"))


def scheduled_tick_generator():
    while True:
        print(f"Exiting at: {m5.curTick()}")
        yield False


simulator = Simulator(
    board=board,
    on_exit_event={ExitEvent.SCHEDULED_TICK: scheduled_tick_generator()},
)

m5.scheduleTickExitFromCurrent(100)
m5.scheduleTickExitFromCurrent(1000)
m5.scheduleTickExitFromCurrent(10000)
m5.scheduleTickExitFromCurrent(100000)

simulator.run()

In this script the Simulator’s on_exit_event parameter is utilized to handle the exit event by printing the tick number at exit, then continuing the simulation. The output of this simulation will be:

Exiting at: 100
Exiting at: 1000
Exiting at: 10000
Exiting at: 100000

The RISCVMatchedBoard

For some time there has been a desire to distribute SimObjects, stdlib components, and systems with “known-good” properties. That is, configurations capable of running simulations with reasonable fidelity to their real-world counter parts. While still early days, the RISCVMatchedBoard is our first step into this endeavor.

The RISCVMatchedBoard is based on SiFive’s HiFive Unmatched board: a RISC-V, 64 bit Linux development platform with a SiFive Freedom U740 multi-core processor. Research at UC Davis sat down with the HiFive Unmatched board and carefully benchmarked its properties, then translated this to a gem5 design. The design has been incorporated into the gem5 Standard Library and can be found at “src/python/gem5/prebuilt/riscvmatched/riscvmatched_board.py”.

Below is an example of using the RISCVMatchedBoard to run a Full-System simulation.

from gem5.prebuilt.riscvmatched.riscvmatched_board import RISCVMatchedBoard
from gem5.utils.requires import requires
from gem5.isas import ISA
from gem5.simulate.simulator import Simulator
from gem5.resources.workload import Workload

requires(isa_required=ISA.RISCV)

board = RISCVMatchedBoard(
    clk_freq="1.2GHz",
    l2_size="2MB",
    is_fs=True,
)

workload = Workload("riscv-ubuntu-20.04-boot")
board.set_workload(workload)

simulator = Simulator(board=board)
simulator.run()

Work is still on-going to fine tune the parameters of this board to have it more-so emulate the real-world board’s behavior and details will be published in the future to highlight our overall success. We will continue to develop “known-good” systems for gem5 and incorporate them as part of future gem5 releases.

SimPoints

An API for SimPoints has been added. SimPointing is a technique to substantially improve simulation time. It works by only sampling representative parts of a simulation then extrapolating statistical data accordingly. By doing this remaining parts of a simulated program can be skipped.

In combination with gem5 Workloads (see the section below for more information on Workloads) we can distribute binaries with SimPoint information and gem5 Checkpoints. These can then be executed via the SimPoint API, thus producing faster simulation runs.

Examples of using SimPoints with gem5 can be found in “configs/example/gem5_library/checkpoints/simpoints-se-checkpoint.py” and “configs/example/gem5_library/checkpoints/simpoints-se-restore.py”.

We are going to continue to expand and fine-tune this API. Of note, we are going to incorporate LoopPoints into the gem5 framework, which will allow for SimPoints to work in multi-core simulations (a current limitation of the SimPoints framework).

Workloads

As an expansion of the gem5-resources infrastructure, the concept of a Workload has been introduced. The gem5-resources infrastructure has, for several major releases, provides resources. As an example:

from gem5.resources.resource import Resource

image = Resource("x86-npb")
kernel = Resource("x86-linux-kernel-5.4.49")
binary = Resource("x86-print-this")

...

board.set_se_simpoint_workload(
    binary=binary,
    arguments=["hello", 1500]
)

Here we three resources are requested. “x86-npb” is an X86 disk image containing the NAS Parallel Benchmark suite built atop the Ubuntu operating system, “x86-linux-kernel-5.4.49” is the v5.4.49 Linux Kernel, and “x86-print-this” is a binary which accepts two arguments: a string to be printed, and the number of times to print it”. The board is then set to run the “x86-print-this” binary with the arguments “hello” and “1500”.

While powerful, obtaining resources in this manner has some limitations. First of all, running a simulation may require multiple resources to be maintained. Some resources will almost always require other resources to run. For example, our “x86-npb” disk image resource is useless without a kernel.

The other issue beyond obvious couplings of resources is resources may require particular parameters to be passed to be useful. For example, the “x86-npb” contains a suite of benchmark applications but specific command line parameters must be passed to specify what benchmark with what input is to be run. In efforts to simplify usage of gem5, we want users to simply specify what they want their simulated system to run. For example, “x86-npb-FS-input-A”.

The solution to these problems are Workloads. Workloads allow for the bundling of resources and any input parameters. User’s need only specify the workload they wish to run and the gem5 Standard Library, interfacing, with the gem5-resources infrastructure, will setup the simulation to run correctly.

from gem5.prebuilt.demo.x86_demo_board import X86DemoBoard
from gem5.resources.workload import Workload
from gem5.simulate.simulator import Simulator

board = X86DemoBoard()

board.set_workload(Workload("x86-ubuntu-18.04-boot"))

simulator = Simulator(board=board)
simulator.run()

“kernel” : “x86-linux-kernel-5.4.49”, “disk_image”:”x86-ubuntu-18.04-img”

Below we show using the “x86-ubuntu-18.04-boot” workload. This workload will use the “x86-linux-kernel-5.4.49” resource for the simulation kernel and the “x86-ubuntu-18.04-img” resource for the disk image. Upon boot completion the simulation will exit.

Another example would be the “x86-print-this-15000-with-simpoints” Workload. This specifies an SE workload running the “x86-print-this” binary resource, passing the parameters “print this” and “1500” with the “x86-print-this-1500-simpoints” simpoint resource.

At the time of writing the following Workloads are available to use:

“x86-ubuntu-18.04-boot” : Runs an X86 Ubuntu 18.04 boot.
“riscv-ubuntu-20.04-boot” : Runs an RISC-V Ubuntu 20.04 boot.
“arm64-ubuntu-20.04-boot” : Runs an ARM-64 Ubuntu 20.04 boot.
“x86-print-this-15000-with-simpoints” : Runs the “print-this” binary, print 15000 “print this” string to the terminal, with SimPoints.
“x86-print-this-15000-with-simpoints-and-checkpoint” : Runs the “print-this” binary, print 15000 “print this” string to the terminal, with SimPoints and checkpoints.

More Workloads will be added overtime to provide the gem5 community with a rich variety of workloads they may plug into their simulations.

pre-commit checks for developers

In order to help gem5 developers adhere to the gem5 style guide, we have added the pre-commit framework to the gem5 repository. The pre-commit framework manages git hooks for the repository. We use this to run a suite of checks on code users wish to submit to code-review via a hook run when the git commit command is executed.

In particular, we now utilize the black Python code formatter on all Python code submitted to the gem5 repository.

In v22.1, when compiling gem5, the user will be asked if they wish to install the pre-commit hook if not already installed. With consent from the user, the hook will be installed. If a user wishes to install the hook manually they may do so by running the following in the gem5 repository:

pip install -r requirements.txt
./util/pre-commit-install.sh

Once installed, any staged code will be processed when git commit is executed. Users may also run the pre-commit on any file or directory they wish by using the following command:

pre-commit run --files path/to/file/or/directory

We strongly advise users to install the pre-commit hooks. Our code-review pre-submit CI tests have been updated with the pre-commit tests. If they fail, users will be unable to incorporate their patches to the code-base until the code is refactored so the pre-commit tests pass.

GPU Full-System mode

In v21.2 we introduced GPU support with Syscall-Emulation (SE) mode and in v22.0 we laid the early framework for Full-System (FS) Support support. In v22.1 we’re happy to announce substantially improved support for GPU simulation in FS mode.

Until v22.1, SE mode was preferred due to its relative stability. However, GPU SE simulation needed the user to have a very specific environment on their host system, primarily due to the GPU simulator requiring a very specific ROCm software stack for dynamic linking to workloads. A Docker container was created to provide this environment this, naturally, required all simulations to be carried out within the container.

The full incorporation of FS mode for GPU simulation removes all host requirements and all SE-mode simulations may be run as an FS mode simulation.

The GPU FS mode has also has improved simulated speed by functionally simulating memory copies, and provides an easier update path for gem5 developers.

For the meantime, we strongly advise users to run GPU FS mode on an X86 host using KVM mode to skip boot and other irrelevant simulation tasks. GPU simulation is very resource intensive so care should be taken as to not simulate unnecessary code.

Community Affairs

In the summer of 2022 we held a gem5 Bootcamp at UC Davis. The first of its kind, the Bootcamp was setup to give early-stage computer architecture researchers an opportunity to learn gem5 for an intensive 5 day course. With over 80 applicants we selected 50 due to venue constraints and held the Bootcamp in July, hosting all attendees in UC Davis accommodation.

The course was carefully designed to take the researchers, most of who were 1st year PhD students, from a very basic understanding of gem5 through to complex tasks such as adding ISA instructions, developing models, learning about various differences in gem5 CPU models, accelerating simulations, and the gem5 GPU model. Leveraging expertise and UC Davis, we were also able to give attendees tailored advice on how to use gem5 for their particular research agendas in special 1-on-1 sessions. In a survey taken after the event, it was found that 95% of attendees “strongly agreed” that the Bootcamp was valuable to them in getting starting with gem5 and 100% of attendees were “more likely or “much more likely” to use gem5 in their research.

We have made efforts to archive the events via our YouTube channel and have encapsulated teaching materials in the Bootcamp website. We encourage users to utilize these resources for learning gem5.

Future events

In the February 2023 we’ll be holding a gem5 Tutorial at HPCA. This tutorial will give attendees a 3-hour crash course in using gem5. This will include emphasis on new gem5 features, such as that incorporated into the v22.1 release. Based on feedback from previous tutorials, we will also include a short session on using the GPU FS model in gem5.

In July (10th to the 14th), UC Davis will be hosting the 2nd gem5 Bootcamp. With similar goals to the Bootcamp held in 2023, this event will give an intensive 5 day course for early-career computer architecture researchers, particularly those in the first two years of a PhD programme. Please join the gem5 announce mailing list or keep an eye on or gem5 events page for upcoming information on registering interest for this event.

gem5-22.0 Released!

Sat, 18 Jun 2022 00:00:00 +0000

First of all, thank you to all of the contributors who made this another great gem5 release! We’ll be talking about this release and many other cool things that have happened in the gem5 community over the past few years at the gem5 workshop with ISCA 2022. You can find a livestream and recording on our youtube channel.

gem5 version 22.0 has been slightly delayed, but we a have a very strong release! This release has 660 changes from 48 unique contributors. While there are not too many big ticket features, the community has done a lot to improve the stablity and add bugfixes to gem5 over this release. That said, we have a few cool new features like full system GPU support, a huge number of Arm improvements, and an improved HBM model.

See below for more details!

New features

Arm now models DVM messages for TLBIs and DSBs accurately. This is implemented in the CHI protocol.
EL2/EL3 support on by default in ArmSystem
HBM controller which supports pseudo channels
Improved Ruby’s SimpleNetwork routing
Added x86 bare metal workload and better real mode support
Added round-robin arbitration when using multiple prefetchers
KVM Emulation added for ARM GIGv3
Many improvements to the CHI protocol

Many RISC-V instructions added

The following RISCV instructions have been added to gem5’s RISC-V ISA:

Zba instructions: add.uw, sh1add, sh1add.uw, sh2add, sh2add.uw, sh3add, sh3add.uw, slli.uw
Zbb instructions: andn, orn, xnor, clz, clzw, ctz, ctzw, cpop, cpopw, max, maxu, min, minu, sext.b, sext.h, zext.h, rol, rolw, ror, rori, roriw, rorw, orc.b, rev8
Zbc instructions: clmul, clmulh, clmulr
Zbs instructions: bclr, bclri, bext, bexti, binv, binvi, bset, bseti
Zfh instructions: flh, fsh, fmadd.h, fmsub.h, fnmsub.h, fnmadd.h, fadd.h, fsub.h, fmul.h, fdiv.h, fsqrt.h, fsgnj.h, fsgnjn.h, fsgnjx.h, fmin.h, fmax.h, fcvt.s.h, fcvt.h.s, fcvt.d.h, fcvt.h.d, fcvt.w.h, fcvt.h.w, fcvt.wu.h, fcvt.h.wu

Improvements to the stdlib automatic resource downloader

The gem5 standard library’s downloader has been re-engineered to more efficiently obtain the resources.json file. It is now cached instead of retrieved on each resource retrieval.

The resources.json directory has been moved to a more permament URL at http://resources.gem5.org/resources.json.

Tests have also been added to ensure the resources module continues to function correctly.

gem5 in SystemC support revamped

The gem5 in SystemC has been revamped to accomodate new research needs. These changes include stability improvements and bugs fixes. The gem5 testing suite has also been expanded to include gem5 in SystemC tests.

Improved GPU support.

Users may now simulate an AMD GPU device in full system mode using the ROCm 4.2 compute stack. Until v21.2, gem5 only supported GPU simulation in Syscall-Emulation mode with ROCm 4.0. See src/gpu-fs/README.md in gem5-resources and example scripts in configs/example/gpufs/ for example scripts which run GPU full system simulations.

A GPU Ruby random tester has been added to help validate the correctness of the CPU and GPU Ruby coherence protocols as part of every kokoro check-in. This helps validate the correctness of the protocols before new changes are checked in. Currently the tester focuses on the protocols used with the GPU, but the ideas are extensible to other protocols. The work is based on “Autonomous Data-Race-Free GPU Testing”, IISWC 2019, Tuan Ta, Xianwei Zhang, Anthony Gutierrez, and Bradford M. Beckmann.

An Arm board has been added to the gem5 Standard Library

Via this change, an ARM Board, ArmBoard, has been added to the gem5 standard library. This allows for an ARM system to be run using the gem5 stdlib components.

An example gem5 configuration script using this board can be found in configs/example/gem5_library/arm-ubuntu-boot-exit.py.

`createAddrRanges` now supports NUMA configurations

When the system is configured for NUMA, it has multiple memory ranges, and each memory range is mapped to a corresponding NUMA node. For this, the change enables createAddrRanges to map address ranges to only a given HNFs.

Jira ticker here: https://gem5.atlassian.net/browse/GEM5-1187.

API (user-facing) changes

CPU model types are no longer simply the model name, but they are specialized for each ISA

For instance, the O3CPU is now the X86O3CPU and ArmO3CPU, etc. This requires a number of changes if you have your own CPU models. See https://gem5-review.googlesource.com/c/public/gem5/+/52490 for details.

Additionally, this requires changes in any configuration script which inherits from the old CPU types.

In many cases, if there is only a single ISA compiled the old name will still work. However, this is not 100% true.

Finally, CPU_MODELS is no longer a parameter in build_opts/. Now, if you want to compile a CPU model for a particular ISA you will have to add a new file for the CPU model in the arch/ directory.

Many changes in the CPU and ISA APIs

If you have any specialized CPU models or any ISAs which are not in the mainline, expect many changes when rebasing on this release.

No longer use read/setIntReg (e.g., see https://gem5-review.googlesource.com/c/public/gem5/+/49766)
InvalidRegClass has changed (e.g., see https://gem5-review.googlesource.com/c/public/gem5/+/49745)
All of the register classes have changed (e.g., see https://gem5-review.googlesource.com/c/public/gem5/+/49764/)
initiateSpecialMemCmd renamed to initiateMemMgmtCmd to generalize to other command beyond HTM (e.g., DVM/TLBI)
OperandDesc class added (e.g., see https://gem5-review.googlesource.com/c/public/gem5/+/49731)
Many cases of TheISA have been removed

Bug Fixes

Fixed RISC-V call/ret instruction decoding. The fix adds IsReturn and IsCall flags for RISC-V jump instructions by defining a new JumpConstructor` in “standard.isa”. Jira Ticket here: https://gem5.atlassian.net/browse/GEM5-1139.
Fixed x86 Read-Modify-Write behavior in multiple timing cores with classic caches. Jira Ticket here: https://gem5.atlassian.net/browse/GEM5-1105.
The circular buffer for the O3 LSQ has been fixed. This issue affected running the O3 CPU with large workloaders. Jira Ticket here: https://gem5.atlassian.net/browse/GEM5-1203.
Removed “memory-leak”-like error in RISC-V lr/sc implementation. Jira issue here: https://gem5.atlassian.net/browse/GEM5-1170.
Resolved issues with Ruby’s memtest. In gem5 v21.2, If the size of the address range was smaller than the maximum number of outstandnig requests allowed downstream, the tester would get stuck trying to find a unique address. This has been resolved.

Variable in env in the SConscript files now requires you to use env['CONF'] to access them. Anywhere that env['<VARIABLE>'] appeared should noe be env['CONF']['<VARIABLE>']
Internal build files are now in a per-target gem5.build directory
All build variable are per-target and there are no longer any shared variables.

Other changes

New bootloader is required for Arm VExpress_GEM5_Foundation platform. See https://gem5.atlassian.net/browse/GEM5-1222 for details.
The MemCtrl interface has been updated to use more inheritance to make extending it to other memory types (e.g., HBM pseudo channels) easier.

The Case for Using Guix to Solve the gem5 Packaging Problem

Mon, 23 May 2022 00:00:00 +0000

¹ School of Electrical and Computer Engineering, Cornell University, Ithaca, NY
² The University of Tennessee Health Science Center, Memphis, TN
³ ElenQ Technology

This post will first describe the gem5 packaging problem before making the case for using Guix, a mature functional cross-platform package manager, for building, distributing, installing, and managing the gem5 ecosystem.

The gem5 Packaging Problem

The gem5 simulator has become the defacto standard for cycle-level simulation, and gem5 now supports evaluating a diverse set of workloads. Unfortunately, both the gem5 simulator and gem5 workloads still lack a compelling software packaging solution to simplify building, distributing, installing, and managing the gem5 ecosystem. In this section, we describe the gem5 simulator and workload packaging problems before sketching an ideal software packaging solution.

The gem5 Simulator Packaging Problem – The gem5 simulator is a complex piece of software with numerous build- and run-time dependencies including a modern C++ compiler, SCons, Boost, and Python. To mitigate dependency issues, the gem5 installation instructions strongly recommend using specific versions of Ubuntu. The gem5 simulator has numerous compile-time options to experiment with different ISAs, coherence protocols, and/or accelerators, and this in turn makes providing a single precompiled binary difficult. Even so, the gem5 community does point new researchers to a small number of precompiled Docker images. Given these challenges, the gem5 community has chosen not to support any kind of packaging for the gem5 simulator. Almost all researchers individually manage dependencies and recompile the gem5 simulator from source.

The gem5 Workload Packaging Problem – Building workloads to run on the gem5 simulator using syscall emulation can be just as challenging as building the gem5 simulator itself. These workloads must be cross-compiled meaning researchers must build a complete cross-compilation toolchain for each target architecture. Researchers might also need to build an emulator (e.g., QEMU) to test these workloads before moving to cycle-level simulation. Researchers must ensure the workloads only use static libraries and do not call any unsupported syscalls. Given these challenges, the gem5 community directly included precompiled binaries as part of the gem5 source distribution for many years. More recently, the community has migrated to pre-compiled binaries as part of the gem5 resources project.

An ideal software packaging solution would be: reproducible – easily duplicate precisely specified development environments; transparent – understand entire development environment including exact build configuration and version of every dependency; composable – easily integrate the gem5 simulator and workload into standard development environments without needing cumbersome, heavyweight containers; fast – leverage precompiled packages; distribution agnostic – enable researchers to use the Linux distribution of their choice; unified – same packaging solution can be used for both the gem5 simulator and workloads; portable – easily build gem5 workloads for native execution and/or target multiple ISAs for cycle-level simulation without manually managing multiple cross-compilation toolchains; and flexible – easily switch between development environments, modify existing packages, add new packages, produce reproducible workflows, and/or generate containers.

Using Guix for gem5

Guix is a mature functional cross-platform package manager with hundreds of committers and over 20K packages. In this section, we briefly describe our on-going efforts to use Guix for packaging both the gem5 simulator and gem5 workloads.

Using Guix to Package gem5 Simulators – We have developed a proof-of-concept Guix package for the gem5 simulator that handles all build- and run-time dependencies and installation¹. The package builds gem5 for six ISAs, ensures builds are reproducible by eliminating non-deterministic use of |__DATE__| and |__TIME__|, patches the build environment to work with SCons, and performs a well-structured install of the gem5 simulator binaries and example configurations. The package is: reproducible through the use of isolated and deterministic environments for experimenting with the gem5 simulator; transparent, since the complete recursive dependency graph is precisely specified; composable, since the gem5 simulator is installed just like any other tool without the need for a container; and distribution agnostic, since the package can be installed on Ubuntu, RHEL, SUSE, or even Guix System which is an entire distribution based exclusively on Guix. Derived packages could enable easily providing packages for different compile-time configurations. When merged upstream into the main Guix package repository, this package will be part of the Guix build farm enabling binary package substitution for fast installation.

Using Guix to Package gem5 Workloads – Guix already includes packages for QEMU and cross-compilation toolchains for commercial ISAs. We are contributing to the RISC-V port of Guix including packaging the RISC-V cross-compilation toolchain. We identified a simple Smith-Waterman sequence alignment Guix package as an interesting gem5 workload and developed a derived Guix package that patches the standard build process to produce a statically linked binary². In addition to being reproducible, transparent, composable, and distribution agnostic, using Guix is also portable, since we can easily cross-compile the package for ARM or RISC-V, and flexible, since it requires only eight lines of Guile code to create a new derived package supporting static compilation.

Case Study

The attached appendix describes step-by-step commands for a case study that: installs QEMU, gem5, and cross-compilers for x86, ARM, and RISC-V in an isolated environment; cross-compiles and runs Smith-Waterman for all three ISAs; and runs this benchmark on the in-order and out-of-order timing models. To reproduce this case study, a researcher first must download and install Guix³.

Add new channel

By default, Guix includes its own main package repository, but users can also create their own ``channels’’ that include third-party packages. We need to add such a channel to get access to the gem5 simulator package and the derived Smith-Waterman package.

 % cd $HOME/.config/guix
 % cat > channels.scm \
<<'END'
(use-modules (guix ci))
(list
 (channel
  (name 'gn-bioinformatics)
  (url (string-append "https://git.genenetwork.org/"
         "guix-bioinformatics/guix-bioinformatics.git"))
  (branch "master"))
 (channel-with-substitutes-available
  %default-guix-channel "https://ci.guix.gnu.org"))
END

Update Guix and install Smith-Waterman

We use guix pull to download all of the package descriptions from the main package repository along with any third-party packages. We then install the default Smith-Waterman package and run it natively. Here we use the default “profile”, but we could also install this package in a dedicated Guix “profile”, similar to Python’s virtual environment.

% mkdir -p $HOME/tmp/misc/test-guix
% cd $HOME/tmp/misc/test-guix
% guix pull
% guix install smithwaterman
% smithwaterman -p TGATTGTACCAAA TGATCATGTACCA

Install QEMU and gem5

We now install both the QEMU and gem5 packages for all architectures in the same profile.

% guix install qemu
% guix install gem5

Build and run Smith-Waterman for x86_64 ISA

We use guix build --target=x86_64-linux-gnu to cross-compile most Guix packages for x86_64. Here we cross-compile the derived package for Smith-Waterman which produces a statically linked executable that we then run on both QEMU and gem5.

% cd $HOME/tmp/misc/test-guix
% DIR=$(guix build \
         --target=x86_64-linux-gnu smithwaterman-static)
% ln -sf $DIR/bin/smithwaterman sw-x86_64
% qemu-x86_64 ./sw-x86_64 -p TGATTGTACCAAA TGATCATGTACCA
% gem5-x86.opt \
    $GUIX_PROFILE/share/gem5/configs/example/se.py \
    --cmd=./sw-x86_64 \
    --options="-p TGATTGTACCAAA TGATCATGTACCA"

Build and run Smith-Waterman for ARM ISA

We use guix build --target=aarch64-linux-gnu to cross-compile most Guix packages for ARM. Here we cross-compile the derived package for Smith-Waterman which produces a statically linked executable that we then run on both QEMU and gem5.

% cd $HOME/tmp/misc/test-guix
% DIR=$(guix build \
         --target=aarch64-linux-gnu smithwaterman-static)
% ln -sf $DIR/bin/smithwaterman sw-aarch64
% qemu-aarch64 ./sw-aarch64 -p TGATTGTACCAAA TGATCATGTACCA
% gem5-arm.opt \
    $GUIX_PROFILE/share/gem5/configs/example/se.py \
    --cmd=./sw-aarch64 \
    --options="-p TGATTGTACCAAA TGATCATGTACCA"

Build and run Smith-Waterman for RISC-V ISA

We use guix build --target=riscv64-linux-gnu to cross-compile most Guix packages for RISC-V. Here we cross-compile the derived package for Smith-Waterman which produces a statically linked executable that we then run on both QEMU and gem5.

% cd $HOME/tmp/misc/test-guix
% DIR=$(guix build \
         --target=riscv64-linux-gnu smithwaterman-static)
% ln -sf $DIR/bin/smithwaterman sw-riscv64
% qemu-riscv64 ./sw-riscv64 -p TGATTGTACCAAA TGATCATGTACCA
% gem5-riscv.opt \
    $GUIX_PROFILE/share/gem5/configs/example/se.py \
    --cmd=./sw-riscv64 \
    --options="-p TGATTGTACCAAA TGATCATGTACCA"

Run experiment on RISC-V ISA

Once we have used Guix to install the gem5 simulator and the gem5 workload packages, we can easily perform a computer architecture research experiment. Here we compare the performance of running Smith-Waterman on an in-order vs.~out-of-order RISC-V processor model.

% cd $HOME/tmp/misc/test-guix

% gem5-riscv.opt \
    --outdir=m5out-minor-sw \
    $GUIX_PROFILE/share/gem5/configs/example/se.py \
    --cmd=./sw-riscv64 \
    --options="-p TGATTGTACCAAA TGATCATGTACCA" \
    --cpu-type=MinorCPU --ruby

% gem5-riscv.opt \
    --outdir=m5out-o3-sw \
    $GUIX_PROFILE/share/gem5/configs/example/se.py \
    --cmd=./sw-riscv64 \
    --options="-p TGATTGTACCAAA TGATCATGTACCA" \
    --cpu-type=O3CPU --ruby

% grep system.cpu.numCycles m5out-minor-sw/stats.txt
% grep system.cpu.numCycles m5out-o3-sw/stats.txt

Acknowledgements

This work was supported by NSF PPoSS Award #2118709 and NLNet awards for GNUMes-RISCV and Guix-Riscv64.

Support for LupIO devices in gem5

Mon, 07 Feb 2022 00:00:00 +0000

The case for a comprehensive and open-source collection of I/O devices

As all the gem5 users probably know, researchers in computer architecture often need to build a full hardware system in order to, for example, experiment a novel micro-architectural approach. In such a scenario, creating I/O devices for the system is typically not the main goal but only a necessary step, and researchers will therefore look for the easiest hardware designs to implement. Unfortunately, they may still face multiple difficulties in the process.

First, even if many hardware designs are widely available in existing computing systems and therefore well-supported by software stacks, their specifications may still be difficult to implement: e.g., a 16550-compatible UART, an IDE disk storage, etc. Second, many hardware specifications are proprietary, which forces researchers to abide by their licensing terms. Last, even if they are able to find some open-source specifications that are easy to implement, it will only apply to a few devices within the designed full system.

Researchers in systems software face very similar challenges, but from the side of device drivers. When building novel operating systems, the development of a few device drivers is also a necessary step that researchers should optimize. However, developing drivers for typical devices can often be difficult (e.g., a 16550-compatible UART, an IDE disk storage, etc.).

Our LupIO’s collection of devices aims to bridge that gap. By providing a comprehensive and open-source collection of I/O devices, that are powerful enough to build complex multicore systems and yet straightforward to implement, LupIO can help researchers in computer architecture and systems software perform exploratory research more easily. Additionally, LupIO can be used as a teaching asset, as it is accessible to students at the undergraduate and graduate level.

Overview of LupIO

LupIO is a comprehensive and open-source collection of education-friendly I/O devices. This collection defines the interfaces of the most common devices found in modern RISC-based computers, and makes it possible to build complete systems using only LupIO devices, even complex symmetric multiprocessor (SMP) systems.

LupIO includes core devices (such as an interrupt controller or a timer) as well as general I/O devices (such as a block device, a real-time clock, or a terminal). LupIO devices are intended to be processor-agnostic, so they should be usable with any processor architecture supporting memory-mapped devices (e.g., RISC-V, ARM, MIPS, etc.).

Each device interface is designed to be simple and clear, with an optimal balance between features and complexity. The register maps exposed by the devices are neatly organized by type (e.g., data, control, and status) and arranged consistently across devices, in order to ease their programmability. Developing implementations of LupIO devices, as well as corresponding device drivers, is meant to be straightforward.

The full specifications of LupIO devices are available at https://gitlab.com/luplab/lupio/lupio-specs.

Implementation in gem5

Following a proof-of-concept implementation in QEMU (https://gitlab.com/luplab/lupio/qemu), Jason Lowe-Power and I decided to join forces and have the LupIO collection be ported to gem5, where it would reach more of the comparch/systems research community.

Last summer, we hired two talented undergraduate students from UC Davis, Laura Hinman and Melissa Jost, to work on this implementation.

They successfully implemented all eight devices of the LupIO collection (https://gem5.googlesource.com/public/gem5/+/refs/tags/v21.2.0.0/src/dev/lupio/) and created an example board, called LupV, based around a RISC-V processor (https://gem5.googlesource.com/public/gem5/+/refs/tags/v21.2.0.0/src/python/gem5/components/boards/experimental/lupv_board.py). This experimental board can boot Linux and even has SMP support! The source for the bootloader/kernel and disk image resources used in this example board can be found here.

Conclusion

While only a RISC-V based board was created, LupIO should technically be processor-agnostic. We are currently working to prove that by building other boards based on other processors and only embedding LupIO devices (e.g., currently trying with an ARM32).

You can also use any of the LupIO devices individually in your hardware system. The purely peripheral I/O devices, such as the real-time clock, the terminal, etc., should work in any MMIO-capable systems.

gem5-21.2 Released!

Tue, 28 Dec 2021 00:00:00 +0000

We are proud to announce version 21.2 of the the gem5 project. In this release we incorporated 790 commits from 33 unique authors, new and regular, from both academia and industry. We are, as always, thankful to all the time our community puts into maintaining and improving gem5.

21.2 Highlights

Enhanced Standard Library

Having existed since v21.1 under the now deprecated name “the components library”, the v21.2 release of gem5 moves the gem5 standard library out of alpha. The purpose of the gem5 standard library is to provide gem5 users a standard set of commonly used components and utilities to aid them in their research. Our overarching goal with the standard library is to remove “boilerplate” code from gem5 configuration files; making the 95% of activities that rarely change from simulation-to-simulation available in an “off-the-shelf” manner to users. As an example, a users wishing to experiment with the effects of cache sizes can used the gem5 standard library to setup a processor, memory system, and test on sensible benchmarks, thus freeing them to focus completely on the impact of cache size changes.

The gem5 standard library is a provided as Python package which contains the following:

Components: A set of Python classes which wrap gem5’s models. Some of the components are pre-configured to match real hardware (e.g., SingleChannelDDR3_1600) and others are parameterized. Components can be combined together into boards which can be simulated.
Resources: A set of utilities to obtain and incorporate resources (disk images, applications, kernels, etc.) into gem5 simulations. Using this module allows you to automatically download and use many of gem5’s prebuilt resources (e.g., kernels and disk images) from gem5-resources.
Simulate: Used to interface with gem5’s simulation/run capabilities. Note: This package is in beta. Expect API changes to this package in future releases. Feedback is appreciated.
Prebuilt: These are fully functioning prebuilt systems (boards) to use directly in gem5 simulations with minimal setup. This release includes an X86 demo board and an example of how it may be used.

Note: Usage of the gem5 standard library is optional. It does not change any established gem5 API, or how gem5 configuration scripts may be created. gem5 configuration scripts that functioned in v21.1 should continue to function in v21.2. We do, however, hope the gem5 library can aid users in creating simulations, as is the case with all libraries.

Users can find example configurations scripts that incorporate the gem5 standard library in the gem5 repository’s configs/example/gem5_library directory.

As an example of how simple the gem5 standard library can make running a gem5 simulation, consider the following script:

from gem5.prebuilt.demo.x86_demo_board import X86DemoBoard
from gem5.resources.resource import Resources
from gem5.simulate.simulator import Simulators

# Here we setup the board. The prebuilt X86DemoBoard allows for Full-System X86
# simulation.
board = X86DemoBoard()

# We then set the workload. Here we use the 5.4.49 Linux kernel with an X86
# Ubuntu OS. If these cannot be found locally they will be automatically
# downloaded.
board.set_kernel_disk_workload(
    kernel=Resource("x86-linux-kernel-5.4.49"),
    disk_image=Resource("x86-ubuntu-18.04-img"),
)

# We then setup the Simulator and run the simulation.
simulator = Simulator(board=board)
simulator.run()

This script can be executed with

scons build/X86/gem5.opt
./build/X86/gem5.opt <script>

The script will automatically obtain the correct linux kernel and a disk image containing Ubuntu 18.04 from gem5-resources (if not already present on the host system). It will then run a full-system X86 simulation to a complete boot of the operating system, then exit. Prior to the introduction of the gem5 standard library, a user would have to put in considerable effort to build such a simulation (100s of lines of python).

While we hope we have designed the standard library in an intuitive manner, users may reference the source under src/python/gem5.

In the coming month we will be updating the gem5 website with new tutorials and documentation on using the gem5 standard library.

Future work on the gem5 standard library

Over the next few gem5 releases we will be expanding the standard library to include more components and features. An big goal of ours is to provide prebuilt components and systems that are proven to be representative of real-world counterparts.

The Simulate module will be expanded, improved, and moved out of beta state as its role in the gem5 standard library becomes more clear.

If you wish to report a bug in the gem5 standard library or have a feature request, please submit it to gem5’s Jira site. Questions regarding usage of the standard library can be made to the gem5 user’s mailing list.

LupIO: Friendly IO Devices for gem5.

LupIO devices were developed by Prof. Joel Porquet-Lupine as a set of open-source I/O devices to be used for teaching. They were designed to model a complete set of I/O devices that are neither too complex to teach in a classroom setting, or too simple to translate to understanding real-world devices. A goal of two undergraduate students at UC Davis, Melissa Jost and Laura Hinman, was to work on incorporating LupIO devices into gem5. As such the gem5 v21.2 release includes a LupIO real-time clock, a random number generator, a terminal device, a block device, a system controller, a timer device, a programmable interrupt controller, and an inter-processor interrupt controller.

A more detailed outline of LupIO can be found in Prof. Porquet-Lupine’s paper “LupIO: a collection fo education-friendly I/O devices” and information on the wider LupLab research group can be found on their website.

Users wishing to try out LupIO devices can find an example script and README file in the configs/example/lupv directory.

Note: These LupIO devices have been built and tested for RISC-V. However, there is no reason these couldn’t be modified to work with other ISA targets if required or desired. We welcome further development by the gem5 community.

Arm improvements

In continued and welcome collaboration with Arm Holdings, improvements to gem5 Arm implementations have been made. They are:

Improved configurability for Arm architectural extensions: We have improved how architectural extensions are enabled/disabled for an Arm system. Rather than working with independent boolean values, we now use a unified ArmRelease object which models the architectural features supported by a FS/SE Arm simulation.
Arm TLB can store partial entries: It is now possible to configure an ArmTLB as a walk cache which stores intermediate PAs obtained during a translation table walk.
Implemented a multilevel LB hierarchy: Users can now compose/model a customizable multilevel TLB hierarchy in gem5. The default Arm MMU now has an Instruction LA TLB, a Data L1 TLB, and Unified (Instruction + Data) L2 TLB.
Provided an Arm example script for the gem5-SST integration.

GPU Improvements

Continued efforts by, primarily, AMD, Inc. and the University of Wisconsin have improved gem5’s GPU support. In this release:

Vega support: gfx900 (Vega) discrete GPUs are now both supported and tested with gem5-resources applications.
Additional GPU applications: The Pannotia graph analytics benchmark suite has been added to gem5-resources, including Makefiles, READMEs, and sample commands on how to run each application in gem5.
Regression Testing: Several GPU applications are now tested as part of the nightly and weekly regressions, which improves test coverage and avoids introducing inadvertent bugs.
Minor updates to the architectural model: Small changes and fixes have been made to the HSA queue size (to allow larger GPU applications with many kernels to run), and the TLB (to create GCN4- and Vega-specific TLBs). We have also added new instructions that were previously unimplemented in GCN4 and Vega, and fixed corner cases for some instructions that were leading to incorrect behavior.

gem5-SST bridge revived

In recent versions of gem5, we sadly lost the ability to integrate with the Structural Simulation Toolkit (SST). In collaboration with the SST community, we have revived support for connecting gem5 cores to the SST memory system. In v21.2 release, this has been tested for RISC-V and Arm. More information on setting up and running gem5 with SST can be found in ext/sst/README.md.

New/Changed APIS

[API CHANGE]: All SimObject declarations in SConscript files now require a sim_objects parameter that lists all SimObject classes declared in that file which need C++ wrappers (that is, SimObject classes which have a type attribute defined).
[NEW CHANGE]: There is now an optional enums parameter for SimObject classes which must list all the Enum types defined in that SimObject file. Technically, this should only include Enum types which generate C++ wrappers though, as of v21.2, all Enums do so.

Other v21.2 improvements

The master/slave terminology has been removed. This has been an on-going effort for several gem5 releases. The gem5 codebase is now free of its usage.
Arm v8.2-A FEAT_UAO has been implemented.
The “at” variants of the file system call have been implemented in SE mode.
The SConscripts have been refactored for improved modularity.
New “tester” CPUs have been introduced which mimic GUPS.

gem5

ISCA 2025: Toward Full-System Heterogeneous Simulation: Merging gem5-SALAM with Mainline gem5

Towards Full-System Heterogeneous Simulation in gem5

Integration at a Glance

What This Enables

Broader Heterogeneity Studies

System-Level Exploration

Domain-Specific Workload Support

Exploratory Studies in Non-Traditional Regimes

Modeling Your Own Accelerator

Getting Started

Conclusion

Acknowledgments

References

Running Bao Hypervisor on gem5

Obtain and Build Software

Download and setup the toolchain

Set up base environment and build automation

gem5 (simulator)

Run and Connect to the Simulation

Demo

gem5 Version 23.1 Release: A Leap Forward in Computer Architecture Simulation

A Shift to GitHub Development

Streamlined Configuration with Kconfig

Standard Library Improvements

API Changes and Enhancements

Support for Latest Tools and Platforms

Full-System GPU Model Enhancements

RISC-V RVV 1.0 Implementation

ArmISA Changes and Improvements

Other Notable Changes

Known Bugs and Issues

Benchmarking Linkers within gem5

“Moving to full system simulation of GPU applications”

Use cases

Benefits of Full System

Using full system

Known issues

Recap

gem5-22.1 Released!

Major New Features

Multiple ISA Compilations

API to set tick-based Exit Events

The RISCVMatchedBoard

SimPoints

Workloads

pre-commit checks for developers

GPU Full-System mode

Community Affairs

Future events

gem5-22.0 Released!

New features

Many RISC-V instructions added

Improvements to the stdlib automatic resource downloader

gem5 in SystemC support revamped

Improved GPU support.

An Arm board has been added to the gem5 Standard Library

createAddrRanges now supports NUMA configurations

API (user-facing) changes

CPU model types are no longer simply the model name, but they are specialized for each ISA

Many changes in the CPU and ISA APIs

Bug Fixes

Build-related changes

Other changes

The Case for Using Guix to Solve the gem5 Packaging Problem

The gem5 Packaging Problem

Using Guix for gem5

Case Study

Add new channel

Update Guix and install Smith-Waterman

Install QEMU and gem5

Build and run Smith-Waterman for x86_64 ISA

Build and run Smith-Waterman for ARM ISA

Build and run Smith-Waterman for RISC-V ISA

Run experiment on RISC-V ISA

Acknowledgements

Support for LupIO devices in gem5

The case for a comprehensive and open-source collection of I/O devices

Overview of LupIO

Implementation in gem5

`createAddrRanges` now supports NUMA configurations