Experimental Evaluation Methodology for the Era of No Steady Performance [pdf]

Experimental Evaluation Methodology for the Era of No Steady Performance JAROMÍR ANTOCH, Charles University, Czechia WALTER BINDER, Università della Svizzera italiana, Switzerland LUBOMÍR BULEJ, Charles University, Czechia FRANÇOIS FARQUET, Oracle Labs, Switzerland VOJTĚCH HORKÝ, Charles University, Czechia ALEKSANDAR PROKOPEC, Oracle Labs, Switzerland ANDREA ROSÀ, Università della Svizzera italiana, Switzerland PETR TŮMA, Charles University, Czechia Recent studies of virtual machine warm up have pointed out that even small deterministic microbenchmarks executed in tightly controlled circumstances often do not reach a steady state of peak performance. This impacts performance evaluation methodologies that focus on performance after warm up, because the lack of a steady state may violate common assumptions made when computing metrics such as the average performance or the confidence interval for that average. Our work examines the reported lack of steady state in the context of comparatively larger virtual machine workloads. We document and analyze similar lack of steady state and argue that it should be considered an inherent property of these workloads rather than a fault. We introduce an updated performance evaluation methodology for workloads whose execution exhibits segments of steady state performance separated by sudden performance changes. Using the Renaissance benchmark suite for the Java Virtual Machine, we show that the methodology can produce confidence intervals that miss the true performance over 20 % less often than the existing methodologies. CCS Concepts: • General and reference → Performance; Measurement; Experimentation. Additional Key Words and Phrases: Java, Benchmark, Methodology ACM Reference Format: Jaromír Antoch, Walter Binder, Lubomír Bulej, François Farquet, Vojtěch Horký, Aleksandar Prokopec, Andrea Rosà, and Petr Tůma. 2026. Experimental Evaluation Methodology for the Era of No Steady Performance. Proc. ACM Program. Lang. 10, OOPSLA1, Article 128 (April 2026), 33 pages. https://doi.org/10.1145/3798236 1 Introduction Performance evaluation of modern virtual machines often relies on executing benchmarks, or similar experimental workloads, whose performance characterizes the system under test. In an idealized performance evaluation scenario, that performance would be repeatable – the benchmark would exhibit the same performance when run repeatedly on the same platform in the same Authors’ Contact Information: Jaromír Antoch, [email protected], Charles University, Prague, Czechia; Walter Binder, [email protected], Università della Svizzera italiana, Lugano, Switzerland; Lubomír Bulej, [email protected], Charles University, Prague, Czechia; François Farquet, [email protected], Oracle Labs, Zurich, Switzerland; Vojtěch Horký, [email protected], Charles University, Prague, Czechia; Aleksandar Prokopec, aleksandar.prokopec@ oracle.com, Oracle Labs, Zurich, Switzerland; Andrea Rosà, [email protected], Università della Svizzera italiana, Lugano, Switzerland; Petr Tůma, [email protected], Charles University, Prague, Czechia. This work is licensed under a Creative Commons Attribution 4.0 International License. © 2026 Copyright held by the owner/author(s). ACM 2475-1421/2026/4-ART128 https://doi.org/10.1145/3798236 Proc. ACM Program. Lang., Vol. 10, No. OOPSLA1, Article 128. Publication date: April 2026. 128:2 J. Antoch, W. Binder, L. Bulej, F. Farquet, V. Horký, A. Prokopec, A. Rosà, P. Tůma conditions. Prior work [3, 11, 14, 19, 23, 24, 32] shows that this idealized perspective does not hold in practice. Multiple effects complicate matters and therefore must be understood and accounted for to correctly execute and interpret the benchmark experiments. One effect known to complicate measurements is warm up, which causes the initial benchmark performance to differ from the performance sustained over continuous execution. In managed environments, an obvious technical factor contributing to warm up is just-in-time compilation, which trades the initial expense of loading, profiling and compiling the application code for the ensuing benefit of executing optimized machine code. Another factor specific to managed environments is garbage collection. It typically requires multiple collection cycles to gradually tenure long lived objects and thus collect young objects more efficiently. Still other factors may include demand loading (of data or code) or priming of various caches at the hardware, system or application level. To facilitate performance evaluation in the presence of warm up, benchmark workloads progress past warm up while repeating the same computation multiple times. The performance of each repetition is reported individually so that the measurements from the cold repetitions can be discarded in an experiment that is to investigate sustained performance. Another effect that complicates measurements is run variability. Some aspects of the benchmark workload execution, such as the exact relative timing of concurrent operations, do not lend them- selves to sufficient control to achieve entirely equivalent benchmark runs. Besides leading directly to different performance of individual workload repetitions, execution differences can also influence feedback-directed just-in-time compilation and therefore expand their impact to future workload repetitions that use the compiled code. Other resource allocation decisions that happen only once per run may have similar effect, impacting each run in a potentially different manner that mere workload repetition in one run can not characterize. An accepted solution to run variability is an experiment design with multiple runs, each with fresh initialization, compilation and resource allocation decisions, and a corresponding statistical evaluation. A procedure described by Georges et al. [11] advocates averaging over measurements collected within each run and then computing a confidence interval over these averages. Kalibera et al. [23] present a more complex procedure with confidence intervals that take into account both variability of measurements within each run and variability between runs. The procedure is further extended by Kalibera et al. [22, 24, 26] with better variance estimates and an arbitrary depth of experiment structure. A similar procedure is used by Bakshy et al. [2] to compute standard error in experiments structured into batches collected across multiple hosts. Among the complications not yet addressed is the observation made by Barrett et al. [3] in their detailed study of virtual machine warm-up behavior, which shows that even small deterministic benchmark workloads may not exhibit steady performance after warm up. The authors note that with such workloads, the widely used steady-state detection method from Georges et al. [11] incorrectly identifies steady state in most workloads without one. Furthermore, the methodology of Georges et al. [11] assumes a workload always reaches a steady state, even if it may take a very long time, which is an assumption the work of Barrett et al. [3] potentially contradicts. The more recent methodology by Kalibera et al. [24] accepts that a workload may not reach a steady state, but then recommends using the same single sample from each run, which is not efficient and may introduce bias. In this paper, we argue that lack of steady performance may simply be a fact of life with workloads on modern virtual machines. We develop a new methodology that accounts for this fact – rather than evaluating performance under the common steady-state assumption, we model the workload Proc. ACM Program. Lang., Vol. 10, No. OOPSLA1, Article 128. Publication date: April 2026. Experimental Evaluation Methodology for the Era of No Steady Performance 128:3 as a random process that transitions between segments of steady-state performance, and provide a method for summarizing such performance using confidence intervals. Finally, we demonstrate the benefits of the model in a realistic performance evaluation scenario. In specific terms, our contributions are: - We extend the experiment of Barrett et al. [3] to include significantly larger code base and significantly longer run times, using the Renaissance benchmark suite [35] for the Java Virtual Machine. Larger code and longer run times provide additional information on the lack of steady performance beyond the scope of the original experiment. - We use a technical analysis of a selected performance change to illustrate why we believe that achieving steady performance after warm up may not be a useful ambition, and propose an updated evaluation methodology to reflect this situation. - We use simulation based on realistic statistical distributions of performance to quantify the difference in accuracy between the existing and updated evaluation methodologies, and to expose the impact of warm up detection mechanisms on the optimum experiment dimensions. For analytical computation, we show that the updated methodology produces confidence intervals that miss the true performance up to 20 % (normal quantiles) or 23 % (studentized quantiles) less often than the existing methodologies. For resampling, we show an improvement of up to 24 %, with the confidence intervals on average only 5.6 % wider than the ground truth data. We proceed by investigating the long term behavior in the context of the Renaissance benchmark suite in Section 2, and delve into the technical reasons behind the observed behavior in Section 3. We present our updated statistical model in Section 4 and evaluate the accuracy of the confidence intervals based on the model in Section 5. Section 6 delves into the practical implications of warm up detection on the experiment dimensions. After summarizing the methodological recommendations in Section 7, we close the paper with a look at the threats to validity in Section 8, related work in Section 9, and conclusion in Section 10. Our