ARTICLE AD BOX
From time to time, our production JFrog Artifactory instance becomes partially unavailable. During these incidents, the login page does not load and all user authentication requests fail, although some API endpoints continue to respond. The system does not crash or restart, CPU and memory usage remain within normal ranges, readiness and liveness probes continue to report “OK,” and the /router/api/v1/system/health endpoint reports all microservices as “HEALTHY.”
We checked the console.log , which repeatedly logs the following error:
2025-12-12T18:39:59.509L [35m[tomct][0m [WARNING] [ ] [io.grpc.netty.NettyServerHandler] [io.grpc.netty.NettyServerHandler onHeadersRead] - Exception in onHeadersRead()
java.util.concurrent.RejectedExecutionException: Task io.grpc.internal.SerializingExecutor@6840f075 rejected from java.util.concurrent.ThreadPoolExecutor@f2b035b[Running, pool size = 50, active threads = 50, queued tasks = 0, completed tasks = 611037]
We have created thread dumps at the time of the incident for the jf-access service. One of the 50 blocked threads shows the following stack trace:
"jf-access-grpc-293" #254480 [1695216] daemon prio=5 os_prio=0 cpu=11.66ms elapsed=2323.22s tid=0x00007f2a743b36c0 nid=1695216 waiting on condition [0x00007f2a3c4fd000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
- parking to wait for <0x00000000869680a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park([email protected]/LockSupport.java:371)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block([email protected]/AbstractQueuedSynchronizer.java:519)
at java.util.concurrent.ForkJoinPool.unmanagedBlock([email protected]/ForkJoinPool.java:3780)
at java.util.concurrent.ForkJoinPool.managedBlock([email protected]/ForkJoinPool.java:3725)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await([email protected]/AbstractQueuedSynchronizer.java:1712)
at java.util.concurrent.ArrayBlockingQueue.take([email protected]/ArrayBlockingQueue.java:420)
at io.grpc.stub.ClientCalls$BlockingResponseStream.waitForNext(ClientCalls.java:690)
at io.grpc.stub.ClientCalls$BlockingResponseStream.hasNext(ClientCalls.java:709)
at java.util.Iterator.forEachRemaining([email protected]/Iterator.java:132)
at org.jfrog.access.server.service.jfconfig.client.config.ConfigClient.lambda$getConfigByPatternStream$3(ConfigClient.java:82)
at org.jfrog.access.server.service.jfconfig.client.config.ConfigClient$$Lambda/0x00007f2ab541a238.accept(Unknown Source)
at java.util.Optional.ifPresent([email protected]/Optional.java:178)
at org.jfrog.access.server.service.jfconfig.client.config.ConfigClient.getConfigByPatternStream(ConfigClient.java:82)
at org.jfrog.access.server.service.jfconfig.JFConfigService.getConfigByPattern(JFConfigService.java:108)
at org.jfrog.access.server.service.auth.ldap.LdapGroupSettingStorage.list(LdapGroupSettingStorage.java:38)
The remaining 49 threads all follow the same execution pattern:
"jf-access-grpc-217" #182116 [1228419] daemon prio=5 os_prio=0 cpu=7111.59ms elapsed=182097.62s tid=0x00007f2a744217e0 nid=1228419 waiting on condition [0x00007f2a38bfd000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
- parking to wait for <0x0000000086bdb448> (a com.google.common.util.concurrent.SettableFuture)
at java.util.concurrent.locks.LockSupport.park([email protected]/LockSupport.java:221)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:561)
at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:111)
at com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:247)
at com.google.common.cache.LocalCache$LoadingValueReference.waitForValue(LocalCache.java:3628)
at com.google.common.cache.LocalCache$Segment.waitForLoadingValue(LocalCache.java:2211)
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2198)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2081)
at com.google.common.cache.LocalCache.get(LocalCache.java:4019)
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4042)
at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:5024)
at com.google.common.cache.ForwardingLoadingCache.get(ForwardingLoadingCache.java:48)
at org.jfrog.access.server.service.auth.ldap.LdapGroupServiceImpl.syncPopulatedUserGroups(LdapGroupServiceImpl.java:202)
We suspect that the jf-access service may be experiencing a deadlock or thread-pool starvation involving the jfconfig service.
Any help or support around this topic would be greatly appreciated!
