• Bug
  • Status: Closed
  • 2 Major
  • Resolution: Cannot Reproduce
  • ehcache-core
  • drb
  • Reporter: galinar
  • December 20, 2010
  • 0
  • Watchers: 1
  • July 27, 2012
  • January 04, 2011

Description

Getting frequent Blocked threads and extremely high load load on our app servers during cache get operation. Below is the thread dump.Is there any helpful solution for this problem?

TP-Processor655” daemon prio=10 tid=0x0000002c33fc4800 nid=0x4d9 waiting for monitor entry [0x00000000493c3000] java.lang.Thread.State: BLOCKED (on object monitor) at net.sf.ehcache.Cache.isExpired(Cache.java:1886) - waiting to lock <0x0000002b4d3d9cc0> (a net.sf.ehcache.Element) at net.sf.ehcache.Cache.searchInMemoryStore(Cache.java:1325) at net.sf.ehcache.Cache.get(Cache.java:967) at net.sf.ehcache.Cache.get(Cache.java:944) at org.hibernate.cache.EhCache.get(EhCache.java:110) at org.hibernate.cache.ReadOnlyCache.get(ReadOnlyCache.java:33) at org.hibernate.event.def.DefaultLoadEventListener.loadFromSecondLevelCache(DefaultLoadEventListener.java:438) at org.hibernate.event.def.DefaultLoadEventListener.doLoad(DefaultLoadEventListener.java:314) at org.hibernate.event.def.DefaultLoadEventListener.load(DefaultLoadEventListener.java:113) at org.hibernate.event.def.DefaultLoadEventListener.proxyOrLoad(DefaultLoadEventListener.java:151) at org.hibernate.event.def.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:79) at org.hibernate.impl.SessionImpl.internalLoad(SessionImpl.java:655) at org.hibernate.type.EntityType.resolveIdentifier(EntityType.java:261) at org.hibernate.type.ManyToOneType.assemble(ManyToOneType.java:160) at org.hibernate.type.TypeFactory.assemble(TypeFactory.java:355) at org.hibernate.cache.entry.CacheEntry.assemble(CacheEntry.java:74) at org.hibernate.cache.entry.CacheEntry.assemble(CacheEntry.java:60) at org.hibernate.event.def.DefaultLoadEventListener.assembleCacheEntry(DefaultLoadEventListener.java:495) at org.hibernate.event.def.DefaultLoadEventListener.loadFromSecondLevelCache(DefaultLoadEventListener.java:460) at org.hibernate.event.def.DefaultLoadEventListener.doLoad(DefaultLoadEventListener.java:314) at org.hibernate.event.def.DefaultLoadEventListener.load(DefaultLoadEventListener.java:113) at org.hibernate.event.def.DefaultLoadEventListener.proxyOrLoad(DefaultLoadEventListener.java:151) at org.hibernate.event.def.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:79) at org.hibernate.impl.SessionImpl.internalLoad(SessionImpl.java:655) at org.hibernate.type.EntityType.resolveIdentifier(EntityType.java:261) at org.hibernate.type.ManyToOneType.assemble(ManyToOneType.java:160) at org.hibernate.type.TypeFactory.assemble(TypeFactory.java:355) at org.hibernate.cache.entry.CacheEntry.assemble(CacheEntry.java:74) at org.hibernate.cache.entry.CacheEntry.assemble(CacheEntry.java:60) at org.hibernate.event.def.DefaultLoadEventListener.assembleCacheEntry(DefaultLoadEventListener.java:495) at org.hibernate.event.def.DefaultLoadEventListener.loadFromSecondLevelCache(DefaultLoadEventListener.java:460) at org.hibernate.event.def.DefaultLoadEventListener.doLoad(DefaultLoadEventListener.java:314) at org.hibernate.event.def.DefaultLoadEventListener.load(DefaultLoadEventListener.java:113) at org.hibernate.event.def.DefaultLoadEventListener.proxyOrLoad(DefaultLoadEventListener.java:151) at org.hibernate.event.def.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:79) at org.hibernate.impl.SessionImpl.internalLoad(SessionImpl.java:655) at org.hibernate.type.EntityType.resolveIdentifier(EntityType.java:261) at org.hibernate.type.ManyToOneType.assemble(ManyToOneType.java:160) at org.hibernate.type.TypeFactory.assemble(TypeFactory.java:355) at org.hibernate.cache.StandardQueryCache.get(StandardQueryCache.java:94) at com.quinstreet.leadcapture.portal.common.caching.XPortalQueryCache.get(XPortalQueryCache.java:121) at org.hibernate.loader.Loader.list(Loader.java:1549) at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:395) at org.hibernate.hql.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:271) at org.hibernate.impl.SessionImpl.list(SessionImpl.java:844) at org.hibernate.impl.QueryImpl.list(QueryImpl.java:74) at com.quinstreet.leadcapture.portal.common.dao.impl.PortalVendorDAO.getPortalVendorDealersForPortal(PortalVendorDAO.java:601) at com.quinstreet.leadcapture.portal.common.dao.impl.PortalVendorDAO.getAllEduPortalVendorDealersForPortalAndServices(PortalVendorDAO.java:1129) at sun.reflect.GeneratedMethodAccessor172.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:292) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:155) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:122) at com.quinstreet.leadcapture.portal.common.service.register.RegisterSessIdentifierInterceptor.invoke(RegisterSessIdentifierInterceptor.java:68) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:144) at org.springframework.orm.hibernate3.HibernateInterceptor.invoke(HibernateInterceptor.java:97) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:144) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:174) at $Proxy54.getAllEduPortalVendorDealersForPortalAndServices(Unknown Source) at com.quinstreet.leadcapture.portal.common.delegate.PortalVendorDelegate.getAllEduPortalVendorDealersForPortalAndServices(PortalVendorDelegate.java:117) at com.quinstreet.leadcapture.portal.common.validation.CriteriaValidator.getMatchClientInfoList(CriteriaValidator.java:1629) at com.quinstreet.leadcapture.portal.common.validation.CriteriaValidator.validatePortalVendorsHier(CriteriaValidator.java:4018) at com.quinstreet.leadcapture.portal.common.validation.CriteriaValidator.validate(CriteriaValidator.java:639) at com.quinstreet.leadcapture.portal.common.validation.CriteriaValidator.validate(CriteriaValidator.java:596) at com.quinstreet.chainserver.clientmatch.ClientMatchHelper.getMatchResults(ClientMatchHelper.java:238) at com.quinstreet.chainserver.chainengine.ChainEngineImpl.isQualified(ChainEngineImpl.java:1387) at com.quinstreet.chainserver.chainengine.ChainEngineImpl.getNextSite(ChainEngineImpl.java:430) at com.quinstreet.chainserver.messagehandler.MessageHandler.handles(MessageHandler.java:252) at com.quinstreet.chainserver.integration.adapter.servlet.ChainAdapterServlet.doPost(ChainAdapterServlet.java:86) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at javax.servlet.http.HttpServlet.service(HttpServlet.java:810) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173) at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(Unknown Source) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationF

Comments

Chris Dennis 2010-12-20

Diagnosing the high load is difficult without more information, however the contention seen here on determining whether a particular Element has expired was removed in r1619. This means it should have been released in ehcache-core 2.0.0. It would be quite helpful if you tried to reproduce the high load situation using the latest ehcache-core release (2.3.0 or Fremantle (2.4.0) Beta).

Chris Dennis 2010-12-20

Can you please point me to the classes or code snippets that had been changed? I see that synchronization on element is still there in version 2.0. </div> {:.comment-heading} ### **Chris Dennis** 2010-12-20

Looking at the tagged ehcache-core-2.0.0 release in svn I don’t see the synchronization that you were getting in 1.7.2:

public final boolean isExpired(Element element) throws IllegalStateException, NullPointerException {
    checkStatus();
    return element.isExpired(configuration);
}

You can see the check-in that removes the synchronization here: http://svn.terracotta.org/fisheye/changelog/Ehcache/?cs=1619

{:.comment-heading} ### **Galina Rogozinsky** 2010-12-20

Hi Chris,

Is version 2.0 completely backward compatible with 1.7.2?

Thanks, Galina.

{:.comment-heading} ### **Steve Harris** 2010-12-20

Yep, 2.3 is backward compatible as well. Just drop in the jar and the newer versions of slf4j

{:.comment-heading} ### **Galina Rogozinsky** 2010-12-21

Hi Chris,

Where can I find list of all enhancements and bug fixes made between release 1.7.2 and 2.3.0?

Thanks for your help, Galina.

{:.comment-heading} ### **Steve Harris** 2010-12-21

http://ehcache.org/changes-report.html

or

https://jira.terracotta.org/jira/browse/EHC#selectedTab=com.atlassian.jira.plugin.system.project%3Achangelog-panel

{:.comment-heading} ### **Fiona OShea** 2011-01-04

Per Chris comments. This should not be an issue.

{:.comment-heading} ### **Galina Rogozinsky** 2011-01-04

Do we need slf4j-jdk14-1.5.8.jar and slf4j-api-1.5.8.jar or just one of them?

{:.comment-heading} ### **Gary Keim** 2011-01-04

If you’re using Maven, you only need to declare a dependency on one of the concrete bindings (slf4j-jdk14-1.5.8.jar), otherwise you need to add both libraries.

http://www.slf4j.org/faq.html#where_is_binding