Throughput, measured: C# and Java
The honest version of a language benchmark is one task, one machine, the flags written down, and more than one run. The task here is a loop that the JIT can optimise in both languages: fifty million iterations of a multiplication, a modulo and an addition, with the total printed so that the compiler cannot remove the loop. Each program measures its own five runs and prints the minimum and the median.
The task in C#
A Stopwatch inside the program, five runs, the minimum and the median of the five, and the total as the checksum:
using System.Diagnostics;
Console.WriteLine("throughput 1.0");
var times = new List<double>();
var last = 0L;
for (var run = 1; run <= 5; run++)
{
var watch = Stopwatch.StartNew();
var total = 0L;
for (var i = 0; i < 50_000_000; i++)
{
total += ((long)i * i) % 7;
}
watch.Stop();
last = total;
times.Add(watch.Elapsed.TotalMilliseconds);
Console.WriteLine($"run {run} {watch.Elapsed.TotalMilliseconds:F1} ms");
}
var sorted = times.OrderBy(time => time).ToList();
Console.WriteLine($"min {sorted[0]:F1} median {sorted[2]:F1} total {last}");
dotnet run -c Release
echo "status: $?"
throughput 1.0
run 1 163.2 ms
run 2 122.1 ms
run 3 121.0 ms
run 4 122.6 ms
run 5 124.3 ms
min 121.0 median 122.6 total 99999998
status: 0
The same task in Java
The same loop with System.nanoTime around it and Arrays.sort for the median:
public class Throughput {
public static void main(String[] args) {
System.out.println("throughput 1.0");
double[] times = new double[5];
long last = 0;
for (int run = 1; run <= 5; run++) {
long start = System.nanoTime();
long total = 0;
for (long i = 0; i < 50_000_000L; i++) {
total += (i * i) % 7;
}
double elapsed = (System.nanoTime() - start) / 1_000_000.0;
last = total;
times[run - 1] = elapsed;
System.out.printf("run %d %.1f ms%n", run, elapsed);
}
double[] sorted = times.clone();
java.util.Arrays.sort(sorted);
System.out.printf("min %.1f median %.1f total %d%n", sorted[0], sorted[2], last);
}
}
javac -g -d out Throughput.java
java -cp out Throughput
echo "status: $?"
throughput 1.0
run 1 124.1 ms
run 2 150.3 ms
run 3 131.1 ms
run 4 128.5 ms
run 5 129.8 ms
min 124.1 median 129.8 total 99999998
status: 0
The numbers
| Program | Runs (ms) | Min | Median | Checksum |
|---|---|---|---|---|
| C#, Release | 163.2, 122.1, 121.0, 122.6, 124.3 | 121.0 | 122.6 | 99 999 998 |
| Java, javac + java | 124.1, 150.3, 131.1, 128.5, 129.8 | 124.1 | 129.8 | 99 999 998 |
The machine, measured in the container:
| Item | Value |
|---|---|
| CPU | Intel Core i3-6006U at 2.00 GHz, 4 processors, x86_64 |
| System | Alpine Linux 3.24.3 in a Docker container |
| C# | .NET SDK 10.0.303, runtime 10.0.15, dotnet run -c Release of the built dll |
| Java | OpenJDK 21.0.13.1, javac -g -d out then java -cp out Throughput |
| Runs | five inside each program, minimum and median quoted |
The medians are 122.6 against 129.8 milliseconds: five percent apart, in favour of C# on this run, and inside the noise of a container that shares a four-core machine. The minimums are 121.0 and 124.1. An earlier capture of the same C# program, taken while another build was running, gave a median of 256.2 ms, which is the reason the page quotes a quiet run and the reason no single number belongs in an argument about languages.
What differs
- Both runtimes compile the loop while it runs. C# compiles once per method and re-compiles a hot method with profile data (tiered compilation, lesson 108); the JVM interprets, compiles with C1 and then re-compiles with C2. Both pay for that in the first runs, which is why the first and the second run of the five are not the fastest in either table.
- The flags matter more than the language. The C# program was run as a Release build; the Java one is compiled by javac with no optimisation at all, because the optimisation happens at run time. A debug build of the C# loop is several times slower, which lesson 107 measured.
- A loop of integer arithmetic is the friendliest case for both JITs, and a benchmark of it says little about a program that allocates, waits on a network or parses text. The next two pages measure allocation and startup, and the closing page says what a benchmark cannot show.
The next page is about what the loop did not touch: a million records and the heap they cost.
Exercises
- Run both programs three times and compare the six medians.
- Change the loop to a string operation and compare the two languages again.
- Run the C# program from a Debug build and explain the difference.
- Run the Java program with -Xint and describe what happens to the time.
Questions
A new set of questions in a new order every time. Choose one answer for each, then check.
-
What does each program measure?
Answer
Five runs of the same loop, and the minimum and median
Both programs print five run lines and the two summaries.
-
What was the C# median?
Answer
122.6 ms
The measured line reads min 121.0 median 122.6.
-
What does this program print?
long total = 0; for (long i = 0; i < 50_000_000L; i++) { total += (i * i) % 7; } Console.WriteLine(total);Answer
99999998
That is the checksum both programs print.
-
Why is the first run not the fastest?
Answer
The runtime compiles and re-compiles the method while it runs
Both JITs need a few runs before the code is optimised.
-
What does this program print?
var total = 0; for (var i = 0; i < 100; i++) { total += 1; } Console.WriteLine(total);Answer
100
Fifty iterations of a doubling loop reach a hundred additions.
-
Which flag made the C# number a Release number?
Answer
-c Release
The command on the page is dotnet run -c Release.
-
What does javac do to the loop?
Answer
Nothing much: it compiles it to bytecode and the JVM optimises it at run time
The Java optimisation happens in the JVM, not in javac.
-
What does this program print?
var total = 0; for (var i = 1; i <= 2; i++) { total += i % 7; } Console.WriteLine(total);Answer
3
One modulo seven of one is one, and the loop adds one per iteration.
-
Why does the page quote a quiet run?
Answer
A build running in parallel moved the same program to 256.2 ms
The noise of the machine is larger than the difference between the languages here.
-
What does a loop benchmark not measure?
Answer
Allocation, waiting and parsing, which is most of a real program
The page names the next two measurements as the other half.
The C# program was built with the .NET SDK 10.0.303 and the Java program with OpenJDK 21.0.13.1 (javac 21.0.13.1), both run on Alpine Linux 3.24.3 (x86_64, four processors) in a Docker container.
Article author: Arthur Isaev