java tink library does not scale with java threads

[uploeger]

It seems that java tink library does not scale with java threads. Find attached a small java test program.

Find below test results with one and three threads, running on a unix (redhat) box with 24 vCPUs.

In both cases around 1.000.000 encryption operations are executed every second. The amount of encryption operations is not increasing when using more threads.

java AeadThread encrypt "this is a test" 10000000 1

origtext: this is a test

Thread: 1 for encryption loop: Thu May 25 13:55:22 CEST 2023

Thread: 1 after encryption loop: Thu May 25 13:55:32 CEST 2023

java AeadThread encrypt "this is a test" 10000000 3

origtext: this is a test

Thread: 2 for encryption loop: Thu May 25 13:53:35 CEST 2023

Thread: 3 for encryption loop: Thu May 25 13:53:35 CEST 2023

Thread: 1 for encryption loop: Thu May 25 13:53:35 CEST 2023

Thread: 1 after encryption loop: Thu May 25 13:54:02 CEST 2023

Thread: 2 after encryption loop: Thu May 25 13:54:02 CEST 2023

Thread: 3 after encryption loop: Thu May 25 13:54:02 CEST 2023

AeadThread.zip

[uploeger]

test environment:
java version: openjdk version "1.8.0_362"
java tink library version: 1.9.0

[tholenst]

Thanks for the report. I think the issue here is that you call "GetPrimitive" in each thread separately.

In Tink, GetPrimitive can be slow and performs optimizations. It also may use locks. You should share the Aead objects between threads instead.

Let me know if this doesn't solve the issue.

[tholenst]

I will close this so I have a better overview of open work items and since I assume that this is resolved. If this is still an issue, please reopen.

[uploeger]

I tested with shared aead object, but the result is the same.
Since I could not re-open a closed issue, I opened a new one.

AeadThread.zip

[tholenst]

Thanks for insisting, I can reproduce this.

Doing a thread dump with 5 threads I get that 2 of them are blocked as follows:

"4" tink-crypto/tink#18 prio=5 os_prio=0 cpu=34757.19ms elapsed=41.17s tid=0x00007f043c323b10 nid=0x5feb0 waiting for monitor entry [0x00007f03f31fc000]
java.lang.Thread.State: BLOCKED (on object monitor)
at sun.security.provider.SecureRandom.engineNextBytes(java.base@17.0.6/SecureRandom.java:222)

This is goes through at com.google.crypto.tink.subtle.Random.randBytes(Random.java:43)

See

tink-java/src/main/java/com/google/crypto/tink/subtle/Random.java

Line 41 in 374451d

public static byte[] randBytes(int size) {

This is confusing since this uses a ThreadLocal.

[tholenst]

Looking at it some more, a full stacktrace looks like this:

java.lang.Thread.State: BLOCKED (on object monitor)
at sun.security.provider.SecureRandom.engineNextBytes(java.base@17.0.6/SecureRandom.java:222)
- waiting to lock <0x000000060f001770> (a sun.security.provider.SecureRandom)
at sun.security.provider.NativePRNG$RandomIO.implNextBytes(java.base@17.0.6/NativePRNG.java:537)
at sun.security.provider.NativePRNG.engineNextBytes(java.base@17.0.6/NativePRNG.java:221)
at java.security.SecureRandom.nextBytes(java.base@17.0.6/SecureRandom.java:758)
at com.google.crypto.tink.subtle.Random.randBytes(Random.java:43)
at com.google.crypto.tink.subtle.AesGcmJce.encrypt(AesGcmJce.java:51)
at AeadThread.run(AeadThread.java:46)

In Random.randBytes we use a ThreadLocal SecureRandom, attempting to make sure that we don't block threads on each other. However, in NativePRNG:221 the object calls "INSTANCE.implNextBytes(bytes)" which goes to the static member INSTANCE -- hence destroying our efforts to avoid a global mutex.

Quick googling suggests that this is https://bugs.openjdk.org/browse/JDK-8098581 -- but this bug was fixed in 2016 and I'm using openjdk 17.0.6 :/

I think we need to do some investigating how to pick the correct randomness provider :/ See https://docs.oracle.com/en/java/javase/17/security/oracle-providers.html for some starting point.

[tholenst]

I will try to describe my current understand here. There might be bugs, I'm mainly writing this down in the hope that this helps me understand.

Let's say we call "java.security.SecureRandom.nextBytes()" (https://github.com/openjdk/jdk/blob/master/src/java.base/share/classes/java/security/SecureRandom.java). I am pretty sure that "threadSafe" is true, so we call "secureRandomSpi.engineNextBytes()" (separate instance for each thread).

On my system, secureRandomSpi is an object of type NativePRNG (https://github.com/openjdk/jdk/blob/master/src/java.base/unix/classes/sun/security/provider/NativePRNG.java). There, the call to engineNextBytes will be forwarded to a global instance of the private class RandomIO in this class -- it uses the method implNextBytes.

This class in turn asks an object of type "sun.security.provider.SecureRandom" (https://github.com/openjdk/jdk/blob/master/src/java.base/share/classes/sun/security/provider/SecureRandom.java) for "engineNextBytes". This is a synchronized method, and since we have a global lock from the "RandomIO" object above, we now lock on a global lock.

I feel I'm missing something.

[tholenst]

I think I understand somewhat better now.

In the end, the job of the RandomIO object above is to read from /dev/urandom or /dev/random. It produces a mix of a SHA1 PRNG stream and the results of reading from /dev/(u)random.

The above explains how we get the SHA1 PRNG stream (because we first block on this one). However, it is more interesting how to read from /dev/urandom in multiple threads, and it seems at least somewhat natural that Java read from multiple threads with a global instance.