Throughput, measured: C# and Java

The honest version of a language benchmark is one task, one machine, the flags written down, and more than one run. The task here is a loop that the JIT can optimise in both languages: fifty million iterations of a multiplication, a modulo and an addition, with the total printed so that the compiler cannot remove the loop. Each program measures its own five runs and prints the minimum and the median.

The task in C#

A Stopwatch inside the program, five runs, the minimum and the median of the five, and the total as the checksum:

using System.Diagnostics; Console.WriteLine("throughput 1.0"); var times = new List<double>(); var last = 0L; for (var run = 1; run <= 5; run++) { var watch = Stopwatch.StartNew(); var total = 0L; for (var i = 0; i < 50_000_000; i++) { total += ((long)i * i) % 7; } watch.Stop(); last = total; times.Add(watch.Elapsed.TotalMilliseconds); Console.WriteLine($"run {run} {watch.Elapsed.TotalMilliseconds:F1} ms"); } var sorted = times.OrderBy(time => time).ToList(); Console.WriteLine($"min {sorted[0]:F1} median {sorted[2]:F1} total {last}");

dotnet run -c Release
echo "status: $?"

throughput 1.0
run 1 163.2 ms
run 2 122.1 ms
run 3 121.0 ms
run 4 122.6 ms
run 5 124.3 ms
min 121.0 median 122.6 total 99999998
status: 0

The same task in Java

The same loop with System.nanoTime around it and Arrays.sort for the median:

public class Throughput { public static void main(String[] args) { System.out.println("throughput 1.0"); double[] times = new double[5]; long last = 0; for (int run = 1; run <= 5; run++) { long start = System.nanoTime(); long total = 0; for (long i = 0; i < 50_000_000L; i++) { total += (i * i) % 7; } double elapsed = (System.nanoTime() - start) / 1_000_000.0; last = total; times[run - 1] = elapsed; System.out.printf("run %d %.1f ms%n", run, elapsed); } double[] sorted = times.clone(); java.util.Arrays.sort(sorted); System.out.printf("min %.1f median %.1f total %d%n", sorted[0], sorted[2], last); } }

javac -g -d out Throughput.java
java -cp out Throughput
echo "status: $?"

throughput 1.0
run 1 124.1 ms
run 2 150.3 ms
run 3 131.1 ms
run 4 128.5 ms
run 5 129.8 ms
min 124.1 median 129.8 total 99999998
status: 0

The numbers

ProgramRuns (ms)MinMedianChecksum
C#, Release163.2, 122.1, 121.0, 122.6, 124.3121.0122.699 999 998
Java, javac + java124.1, 150.3, 131.1, 128.5, 129.8124.1129.899 999 998

The machine, measured in the container:

ItemValue
CPUIntel Core i3-6006U at 2.00 GHz, 4 processors, x86_64
SystemAlpine Linux 3.24.3 in a Docker container
C#.NET SDK 10.0.303, runtime 10.0.15, dotnet run -c Release of the built dll
JavaOpenJDK 21.0.13.1, javac -g -d out then java -cp out Throughput
Runsfive inside each program, minimum and median quoted

The medians are 122.6 against 129.8 milliseconds: five percent apart, in favour of C# on this run, and inside the noise of a container that shares a four-core machine. The minimums are 121.0 and 124.1. An earlier capture of the same C# program, taken while another build was running, gave a median of 256.2 ms, which is the reason the page quotes a quiet run and the reason no single number belongs in an argument about languages.

What differs

The next page is about what the loop did not touch: a million records and the heap they cost.

Exercises

  1. Run both programs three times and compare the six medians.
  2. Change the loop to a string operation and compare the two languages again.
  3. Run the C# program from a Debug build and explain the difference.
  4. Run the Java program with -Xint and describe what happens to the time.

Questions

A new set of questions in a new order every time. Choose one answer for each, then check.

  1. What does each program measure?

    Answer

    Five runs of the same loop, and the minimum and median

    Both programs print five run lines and the two summaries.

  2. What was the C# median?

    Answer

    122.6 ms

    The measured line reads min 121.0 median 122.6.

  3. What does this program print? long total = 0; for (long i = 0; i < 50_000_000L; i++) { total += (i * i) % 7; } Console.WriteLine(total);

    Answer

    99999998

    That is the checksum both programs print.

  4. Why is the first run not the fastest?

    Answer

    The runtime compiles and re-compiles the method while it runs

    Both JITs need a few runs before the code is optimised.

  5. What does this program print? var total = 0; for (var i = 0; i < 100; i++) { total += 1; } Console.WriteLine(total);

    Answer

    100

    Fifty iterations of a doubling loop reach a hundred additions.

  6. Which flag made the C# number a Release number?

    Answer

    -c Release

    The command on the page is dotnet run -c Release.

  7. What does javac do to the loop?

    Answer

    Nothing much: it compiles it to bytecode and the JVM optimises it at run time

    The Java optimisation happens in the JVM, not in javac.

  8. What does this program print? var total = 0; for (var i = 1; i <= 2; i++) { total += i % 7; } Console.WriteLine(total);

    Answer

    3

    One modulo seven of one is one, and the loop adds one per iteration.

  9. Why does the page quote a quiet run?

    Answer

    A build running in parallel moved the same program to 256.2 ms

    The noise of the machine is larger than the difference between the languages here.

  10. What does a loop benchmark not measure?

    Answer

    Allocation, waiting and parsing, which is most of a real program

    The page names the next two measurements as the other half.

The C# program was built with the .NET SDK 10.0.303 and the Java program with OpenJDK 21.0.13.1 (javac 21.0.13.1), both run on Alpine Linux 3.24.3 (x86_64, four processors) in a Docker container.

Article author: Arthur Isaev