Deadlock when calling net.sf.ehcache.Cache.isKeyInCache from multiple threads

Type: Bug
Status: Resolved
Priority: 2 Major
Resolution: Not a Bug
Component/s: ehcache
Labels:

Assignee: ljacomet
Reporter: mbraeger
Created: February 21, 2014
Votes: 0
Watchers: 5
Updated:
March 13, 2014
Resolved:
February 24, 2014

Description

The deadlock occurred with ehcache-ee-2.7.5.jar

Found one Java-level deadlock:
=============================
"pool-11-thread-22":
  waiting for ownable synchronizer 0x00000007020bea48, (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
  which is held by "pool-13-thread-1"
"pool-13-thread-1":
  waiting for ownable synchronizer 0x0000000702fee538, (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
  which is held by "pool-13-thread-2"
"pool-13-thread-2":
  waiting for ownable synchronizer 0x00000007020bea48, (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
  which is held by "pool-13-thread-1"

Java stack information for the threads listed above:
===================================================
"pool-11-thread-22":
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000007020bea48> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:964)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1282)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:731)
        at net.sf.ehcache.store.chm.SelectableConcurrentHashMap$Segment.containsKey(SelectableConcurrentHashMap.java:741)
        at net.sf.ehcache.store.chm.SelectableConcurrentHashMap.containsKey(SelectableConcurrentHashMap.java:329)
        at net.sf.ehcache.store.MemoryStore.containsKey(MemoryStore.java:519)
        at net.sf.ehcache.Cache.isKeyInCache(Cache.java:2902)
        at cern.c2mon.server.cache.common.BasicCache.hasKey(BasicCache.java:67)
        at cern.c2mon.server.cache.common.TagLocationServiceImpl.getCache(TagLocationServiceImpl.java:63)
        at cern.c2mon.server.cache.common.TagLocationServiceImpl.getCopy(TagLocationServiceImpl.java:76)
        at cern.c2mon.server.supervision.impl.SupervisionTagNotifier.callCacheNotification(SupervisionTagNotifier.java:281)
        - locked <0x00000007930a3238> (a java.util.HashMap)
        at cern.c2mon.server.supervision.impl.SupervisionTagNotifier.callCacheNotification(SupervisionTagNotifier.java:314)
        - locked <0x00000007930a3238> (a java.util.HashMap)
        at cern.c2mon.server.supervision.impl.SupervisionTagNotifier.notifyEquipmentTags(SupervisionTagNotifier.java:261)
        at cern.c2mon.server.supervision.impl.SupervisionTagNotifier.notifySupervisionEvent(SupervisionTagNotifier.java:230)
        at cern.c2mon.server.supervision.impl.SupervisionNotifierImpl$SupervisionNotifyTask.run(SupervisionNotifierImpl.java:282)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
"pool-13-thread-1":
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x0000000702fee538> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:964)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1282)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:731)
        at net.sf.ehcache.store.chm.SelectableConcurrentHashMap$Segment.containsKey(SelectableConcurrentHashMap.java:741)
        at net.sf.ehcache.store.chm.SelectableConcurrentHashMap.containsKey(SelectableConcurrentHashMap.java:329)
        at net.sf.ehcache.store.MemoryStore.containsKey(MemoryStore.java:519)
        at net.sf.ehcache.Cache.isKeyInCache(Cache.java:2902)
        at cern.c2mon.server.cache.common.BasicCache.hasKey(BasicCache.java:67)
        at cern.c2mon.server.cache.common.TagLocationServiceImpl.getCache(TagLocationServiceImpl.java:63)
        at cern.c2mon.server.cache.common.TagLocationServiceImpl.get(TagLocationServiceImpl.java:81)
        at cern.c2mon.server.rule.evaluation.RuleEvaluatorImpl.evaluateRule(RuleEvaluatorImpl.java:151)
        at cern.c2mon.server.rule.evaluation.RuleEvaluatorImpl.evaluateRules(RuleEvaluatorImpl.java:113)
        at cern.c2mon.server.rule.evaluation.RuleEvaluatorImpl.notifyElementUpdated(RuleEvaluatorImpl.java:89)
        at cern.c2mon.server.rule.evaluation.RuleEvaluatorImpl.notifyElementUpdated(RuleEvaluatorImpl.java:39)
        at cern.c2mon.server.cache.listener.MultiThreadedCacheListener$NotifyTask.callCorrectMethod(MultiThreadedCacheListener.java:277)
        at cern.c2mon.server.cache.listener.MultiThreadedCacheListener$NotifyTask.run(MultiThreadedCacheListener.java:255)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
"pool-13-thread-2":
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000007020bea48> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:964)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1282)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:731)
        at net.sf.ehcache.store.chm.SelectableConcurrentHashMap$Segment.containsKey(SelectableConcurrentHashMap.java:741)
        at net.sf.ehcache.store.chm.SelectableConcurrentHashMap.containsKey(SelectableConcurrentHashMap.java:329)
        at net.sf.ehcache.store.MemoryStore.containsKey(MemoryStore.java:519)
        at net.sf.ehcache.Cache.isKeyInCache(Cache.java:2902)
        at cern.c2mon.server.cache.common.BasicCache.hasKey(BasicCache.java:67)
        at cern.c2mon.server.cache.common.TagLocationServiceImpl.getCache(TagLocationServiceImpl.java:63)
        at cern.c2mon.server.cache.common.TagLocationServiceImpl.get(TagLocationServiceImpl.java:81)
        at cern.c2mon.server.rule.evaluation.RuleEvaluatorImpl.evaluateRule(RuleEvaluatorImpl.java:151)
        at cern.c2mon.server.rule.evaluation.RuleEvaluatorImpl.evaluateRules(RuleEvaluatorImpl.java:113)
        at cern.c2mon.server.rule.evaluation.RuleEvaluatorImpl.notifyElementUpdated(RuleEvaluatorImpl.java:89)
        at cern.c2mon.server.rule.evaluation.RuleEvaluatorImpl.notifyElementUpdated(RuleEvaluatorImpl.java:39)
        at cern.c2mon.server.cache.listener.MultiThreadedCacheListener$NotifyTask.callCorrectMethod(MultiThreadedCacheListener.java:277)
        at cern.c2mon.server.cache.listener.MultiThreadedCacheListener$NotifyTask.run(MultiThreadedCacheListener.java:255)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)

Found 1 deadlock.

Comments

Matthias Braeger 2014-02-22

Dear Louis, You can close this bug report since we found the cause for the deadlock in our code. My apologies.

Kind regards, Matthias

Louis Jacomet Jacomet 2014-02-24

Resolving as requested

Dinakar Manchikalapudi 2014-03-12

What was the issue with your code Matthias. I have a lot of threads backed up with the exact same issue. Thanks for your time.

Matthias Braeger 2014-03-13

The Ehcache {{containsKey()}} method hangs when a second thread is holding a write lock on the same key through {{acquireWriteLock()}}. The second thread has first to release the write lock before {{containsKey()}} gets unlocked. Check also that you have not by mistake turned on the bulk loading-mode.

Dinakar Manchikalapudi 2014-03-13

my ehcache config is pretty simple …

for example:

<ehcache xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="ehcache.xsd"
        updateCheck="false" monitoring="autodetect"
        dynamicConfig="true"
        maxBytesLocalHeap="870M">
    <!-- NOTE: If you need close to accurate values for total number of effective caches and therefore their sizes, please assign a size for each of your caches in addition to the already configured global setting -->
    <defaultCache eternal="false" overflowToDisk="false" timeToLiveSeconds="1" timeToIdleSeconds="1" statistics="true" memoryStoreEvictionPolicy="LFU" />
	<cache name="Impl:CRONumberService:getCRONumber" eternal="false" timeToLiveSeconds="86400" timeToIdleSeconds="86400" statistics="true" maxBytesLocalHeap="250M" memoryStoreEvictionPolicy="LFU" />
	
</ehcache>

is this turned on by default? if I need to set this property .. where do I need to set it ?

Matthias Braeger 2014-03-13

Bulk loading is turned off by default. So you should rather check your write locks on ehcache.

Dinakar Manchikalapudi 2014-03-13

Is there a way to configure reads without waiting for read locks? If so, can you give an example?