SchedMark: Evaluating Scheduler Performance


In the last decade we have witnessed many new 
developments in process scheduling---developments 
driven by innovations in processor technology and by 
ever more versatile workloads. Many innovative 
schedulers are emerging in the scientific literature as 
well as  enhancements and patches to production 
operating systems. However, one difficulty persists in 
the evaluation and comparison of scheduling schemes, 
even between those aiming to address the same problem: 
the lack of an agreed-upon benchmark for scheduling. 
Many studies come up with their own measures, 
workloads, and metrics, thus making their comparison an 
apples-and-oranges situation. The problem will probably 
grow worse with the emergence of commodity multicore 
and multithreaded chips, which are likely to result in 
even more scheduling work. The lack of a reliable, 
reproducible, and portable way to measure scheduling 
will grow even more critical.

 

To address this need, we are developing SchedMark, a 
benchmark suite to evaluate the scheduler's impact on 
applications and workloads of various representative 
classes. SchedMark contrasts with previous approaches 
that measured either specific workloads (e.g., 
multimedia), or focused on system metrics (e.g., 
context-switch overhead). SchedMark will include 
synthetic desktop and server applications that 
represent a range of parameters and classes, such as 
continuous media, interactive, parallel, and sequential 
computation applications. The suite will measure not 
only global metrics, such as throughput, but also 
metrics that are of specific relevance to each 
application, such as dropped frames for continuous 
media, and response time for interactive applications. 
The set is designed to be portable, self-calibrating, 
and self-scaling, so that its results will remain 
comparable across a large set of architectures and 
operating systems. Workloads will be both static and 
dynamic, containing various combinations of classes of 
applications to capture the effect of scheduling 
decisions on co-interference and cache performance.