Working Status of Benchmarks


This page provides information on the working status of some benchmark suites with gem5-20. The resources needed to run these benchmarks with gem5 can be found in gem5-resources repository or gem5 resources web page.

Boot Tests

The Linux Kernel boot tests rely on 5 LTS kernel releases (4.4.186, 4.9.186, 4.14.134, 4.19.83, and 5.4), four CPU models (kvmCPU, AtomicSimpleCPU, TimingSimpleCPU, O3CPU), three memory systems (classic, MI_Example, MESI_Two_Level) and two boot types (init, systemd).

Following is the description of the possible status of these runs:

timeout: experiment did not finish in a reasonable amount of time (12 hours: this time was chosen as we found similar successful cases did not exceed this limit on the same host machine).

not-supported: cases which are not yet supported in gem5.

success: cases where Linux booted successfully.

sim-crash: cases where gem5 crashed.

kernel-panic: cases where kernel went into panic during simulation.

Boot Tests Status with Classic Memory and init Boot Boot Tests Status with Classic Memory and systemd Boot Boot Tests Status with MI_Example Memory and init Boot Boot Tests Status with MI_Example Memory and systemd Boot [Boot Tests Status with MESI_Two_Level Memory and init Boot Boot Tests Status with MESI_Two_Level Memory and systemd Boot

Summary: kvmCPU works in all cases. AtomicSimpleCPU works fine with Classic memory system, but it is not supported with any Ruby protocols. TimingSimpleCPU always work for a single CPU core in case of Classic memory system, but fails to complete the booting process in the alloted time for more than one CPU cores. TimingSimpleCPU mostly works for any number of CPU cores for Ruby memory protocols except one case with MI_example and seven cases with MESI_Two_Level protocols, where gem5 crashes. However, all of these crashes show an error that the cache controller’s packet queue has grown beyond 100 packets. It is expected that recompiling gem5 after changing the default packet queue size to a bigger value will lead these simulations to work fine. For O3CPU, the Classic memory system is considered to not support more than one CPU core and with a single CPU core it works successfully in less than half of the cases and either shows kernel panic or simulation times out in the rest of the cases. For Ruby memory protocols as well, O3CPU simulations do not work in all the cases (except one), but either times out or show kernel panic or crash. The O3CPU (with Ruby) cases where simulations crash, the errors point to possible deadlocks or segmentation faults.

NPB Tests

These NPB tests use KVM CPU (1,8,16,32, and 64 cores) and TimingSimple CPU (1 and 8 cores) with MESI_Two_Level memory system.

NPB Status with KVM CPU

It is a known problem that if the number of simulated CPU cores increase, KVM simulations get stuck sometimes. A work around is to use lower number of event queues than the CPU cores. Although our scripts do that for more than 1 CPU core, the cases shown as timeout in the plot above suffer from this problem of getting stuck.

NPB Status with TimingSimple CPU and MESI_Two_Level Memory System

There are three cases with TimingSimple CPU which did not finish in the alloted time. There is no reason apparent in the generated results files (simout, simerr, system.pc.com_1.device). Without further analysis, it is hard to tell if the simulation is stuck or is proceeding normally and just need more time to finish.

Summary: Most of the tested KVM and TimingSimple CPU simulations of NPB work successfully. The rest of the cases could not result into success in the allocated simulation time (except one kernel panic case).

PARSEC Tests

The PARSEC Experiments have been run with the following configurations:

The result of experiments with gem5-20 is shown below:

PARSEC Status with KVM CPU PARSEC Status with TimingSimple CPU

The cases of unsuccessfull termination of simulation are shown below:

SPEC 2006 Tests

The following plot represent the status of SPEC2006 workloads for different CPUs and data sizes with respect to gem5-20, linux kernel version 4.19.83 and gcc version 7.5.0.

SPEC-2006 status for gem5-20

SPEC 2017 Tests

The following plot represent the status of SPEC2017 workloads with respect to gem5-20, linux kernel version 4.19.83 and gcc version 7.5.0.

SPEC-2017 status for gem5-20

GAPBS Tests

The results of the GAPBS experiments with gem5-20 are shown below. For this experiment the input graphs for the workloads are synthetically generated.

GAPBS status for gem5-20 kvm GAPBS status for gem5-20 atomic GAPBS status for gem5-20 simple GAPBS status for gem5-20 o3