JAR Locking / Resource Clean Up

I've upgraded the "help" demo app and checked again with Grails 1.2.1. Unfortunately I am still seeing the same issue.

The list of locked JAR files is as follows...

• ehcache-core-1.7.1.jar
• grails-core-1.2.1.jar
• groovy-all-1.6.7.jar

I've also attached the log produced by Tomcat 6.0.24 as it forces the app to undeploy.

Tomcat 6.0.24 log with Grails 1.2.1 app.

As far as I understand it there were specific changes made to 6.0.24 that improved Tomcat's ability to reload deployed applications. These changes aren't present in 6.0.20 so nothing can be done other than the user upgrading to the latest Tomcat version.

We can however looked at why these SEVERE errors are appearing

Most of the ThreadLocal warnings are Spring-related; see this thread: http://forum.springsource.org/showthread.php?t=84202

Tomcat 6.0.26 has some new resource clean up and permgen leak detection logic which will hopefully help fix some of the issues. Note that many of the issues are not just in Grails but in libraries like Spring and Apache commons

I tried again on Tomcat 6.0.26 with Grails 1.2.1. The number of SEVERE items has significantly dropped, but there are still a few. Including the list below.

I'm not certain about the first error message in the list since it is related to the MSSQL Driver, but the other six appear to be referencing classes from the org.codehaus.groovy and org.codehaus.groovy.grails packages. Are these issues with Groovy / Grails or are these as you mentioned before errors with the underlying Spring and Apache Commons libraries?

Thanks

SEVERE: A web application registered the JBDC driver [com.microsoft.sqlserver.jdbc.SQLServerDriver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
Mar 24, 2010 4:03:11 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
SEVERE: A web application created a ThreadLocal with key of type [org.codehaus.groovy.runtime.GroovyCategorySupport.MyThreadLocal] (value [org.codehaus.groovy.runtime.GroovyCategorySupport$MyThreadLocal@4a17d8]) and a value of type [java.lang.ref.SoftReference] (value [java.lang.ref.SoftReference@63725d]) but failed to remove it when the web application was stopped. To prevent a memory leak, the ThreadLocal has been forcibly removed.
Mar 24, 2010 4:03:11 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
SEVERE: A web application created a ThreadLocal with key of type [org.codehaus.groovy.runtime.GroovyCategorySupport.MyThreadLocal] (value [org.codehaus.groovy.runtime.GroovyCategorySupport$MyThreadLocal@4a17d8]) and a value of type [java.lang.ref.SoftReference] (value [java.lang.ref.SoftReference@e94c62]) but failed to remove it when the web application was stopped. To prevent a memory leak, the ThreadLocal has been forcibly removed.
Mar 24, 2010 4:03:11 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
SEVERE: A web application created a ThreadLocal with key of type [null] (value [org.codehaus.groovy.grails.commons.spring.ReloadAwareAutowireCapableBeanFactory$2@95c2b4]) and a value of type [java.lang.Boolean] (value [false]) but failed to remove it when the web application was stopped. To prevent a memory leak, the ThreadLocal has been forcibly removed.
Mar 24, 2010 4:03:11 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
SEVERE: A web application created a ThreadLocal with key of type [org.codehaus.groovy.runtime.GroovyCategorySupport.MyThreadLocal] (value [org.codehaus.groovy.runtime.GroovyCategorySupport$MyThreadLocal@4a17d8]) and a value of type [java.lang.ref.SoftReference] (value [java.lang.ref.SoftReference@9212f4]) but failed to remove it when the web application was stopped. To prevent a memory leak, the ThreadLocal has been forcibly removed.
Mar 24, 2010 4:03:11 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
SEVERE: A web application created a ThreadLocal with key of type [null] (value [org.codehaus.groovy.grails.commons.spring.ReloadAwareAutowireCapableBeanFactory$2@95c2b4]) and a value of type [java.lang.Boolean] (value [false]) but failed to remove it when the web application was stopped. To prevent a memory leak, the ThreadLocal has been forcibly removed.
Mar 24, 2010 4:03:11 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
SEVERE: A web application created a ThreadLocal with key of type [org.codehaus.groovy.runtime.GroovyCategorySupport.MyThreadLocal] (value [org.codehaus.groovy.runtime.GroovyCategorySupport$MyThreadLocal@4a17d8]) and a value of type [java.lang.ref.SoftReference] (value [java.lang.ref.SoftReference@1a755e]) but failed to remove it when the web application was stopped. To prevent a memory leak, the ThreadLocal has been forcibly removed.

yes. The GroovyCategory ones are problems in Groovy (not Grails) that we'll need a new release of Groovy for. I'm not sure whether the issue with org.codehaus.groovy.grails.commons.spring.ReloadAwareAutowireCapableBeanFactory is a problem with Grails or Spring at this stage. Will need to investigate further...

Related Groovy issue GROOVY-4130

I'm having this problem and couldn't find a way to solve it. I'm trying to deploy the application on Tomcat 6.0.26 and these are the errors:

SEVERE: A web application registered the JBDC driver [org.firebirdsql.jdbc.FBDriver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
19/04/2010 10:40:17 org.apache.catalina.loader.WebappClassLoader clearReferencesStopTimerThread
SEVERE: A web application appears to have started a TimerThread named [Timer-2] via the java.util.Timer API but has failed to stop it. To prevent a memory leak, the timer (and hence the associated thread) has been forcibly cancelled.
19/04/2010 10:40:17 org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
SEVERE: A web application created a ThreadLocal with key of type [org.codehaus.groovy.runtime.GroovyCategorySupport.MyThreadLocal] (value [org.codehaus.groovy.runtime.GroovyCategorySupport$MyThreadLocal@a0416a]) and a value of type [java.lang.ref.SoftReference] (value [java.lang.ref.SoftReference@126d2fe]) but failed to remove it when the web application was stopped. To prevent a memory leak, the ThreadLocal has been forcibly removed.

Grails version is 1.1.2

Is there any workaround to make it work?

At this point it looks like you have to restart tomcat after enough redeploys. Not really a workaround, I know (Tomcat 6.0.26, Grails 1.2.2 here)

I just commited fix for GROOVY-4130 both to Groovy 1.7.x & 1.8.x
It solves issue with ThreadLocals but the issue has nothing to do with locked jars
Graeme, please let me know if you have other problems between Groovy runtime & Grails

Thanks Alex!

The fix Alex commited entered groovy 1.7.3 and 1.8-beta-1. Since Grails 1.3.5 ships with groovy 1.7.5, does it contain this fix?

It is only partially fixed. Alex' fix took care of the category support stuff, but my Grails 1.3.5 app still gives messages on undeploy because of the following ThreadLocal objects:

org.codehaus.groovy.grails.orm.hibernate.validation.HibernateDomainClassValidator
org.codehaus.groovy.grails.web.sitemesh.GrailsContentBufferingResponse
org.springframework.security.core.context.SecurityContextImpl

@Raphael Yes the fix has been applied in Groovy

@Marcus Is it possible to post a reproducible example, there are clearly cases where this still happens but a vanilla Grails application doesn't exhibit any problems as far as I can tell

@Graeme
I am using tomcat 6.0.29, grails 1.3.7, and java 1.5 version. But when I undeploy or stop the tomcat server then i am getting below error in the catalina.log file of tomcat...

SEVERE: The web application [/application] created a ThreadLocal with key of type [null] (value [org.codehaus.groovy.grails.commons.spring.ReloadAwareAutowireCapableBeanFactory$2@1115340]) and a value of type [java.lang.Boolean] (value [false]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.

How to resolve this issue? Do you think there is a memory leak in ReloadAwareAutowireCapableBeanFactory class?
Or am i doing something wrong?

I am using apache-tomcat-6.0.32 , grails 1.3.7, Java jdk1.6.0_24 on a CentoOS 5.5.
My application just stops responding at some point and I can see the following logs:
Any workaround?

SEVERE: The web application [/nu] created a ThreadLocal with key of type [org.codehaus.groovy.grails.orm.hibernate.support.HibernatePersistenceContextInterceptor$2] (value [org.codehaus.groovy.grails.orm.hibernate.support.HibernatePersistenceContextInterceptor$2@7649a07e]) and a value of type [java.lang.Integer] (value [0]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.
Jun 3, 2011 4:23:13 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
SEVERE: The web application [/nu] created a ThreadLocal with key of type [org.codehaus.groovy.grails.orm.hibernate.validation.HibernateDomainClassValidator$1] (value [org.codehaus.groovy.grails.orm.hibernate.validation.HibernateDomainClassValidator$1@fb0a82c]) and a value of type [java.util.ArrayList] (value [[]]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.

That message is Tomcat stopping the application and automatically clearing out thread locals, but is not related to your application becoming unresponsive.

@Chaitanya Tomcat is auto-preventing a memory leak in this case. There are a few thread locals here and there that are not cleared and I'll look and getting them cleared for 1.4 M2

The errors seen here should be resolved by this commit:

https://github.com/grails/grails-core/commit/0f2c0631a8ed47e1d0d34bc240a894eed4b86261

If further errors are seen in 1.4 M2 please re-open

Hi,
I'm sorry but I'm still seeing this leak with tomcat 7 even with minimum application build by grails 2.
First I tried to undeploy and see the following error log by checking tomcat's new leak detection

SEVERE: The web application [/hellog2-0.1] appears to have started a thread named [Timer-1] but has failed to stop it. This is very likely to create a memory leak.
16-Dec-2011 9:29:26 AM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/hellog2-0.1] created a ThreadLocal with key of type [org.codehaus.groovy.grails.orm.hibernate.metaclass.AbstractSavePersistentMethod$1] (value [org.codehaus.groovy.grails
.orm.hibernate.metaclass.AbstractSavePersistentMethod$1@4552a650]) and a value of type [java.util.HashSet] (value [[]]) but failed to remove it when the web application was stopped. Threads are going
to be renewed over time to try and avoid a probable memory leak.
16-Dec-2011 9:29:57 AM org.apache.catalina.util.LifecycleBase stop
INFO: The stop() method was called on component [StandardEngine[Catalina].StandardHost[localhost].StandardContext[/hellog2-0.1]] after stop() had already been called. The second call will be ignored.

To be sure, I've monitored the PermGen graph from visual-gc, then found the PermGen stay same after undeployment done, and when I re-deploy the same application then the PermGen just increased.

So, whenever I'm repeating undeploy/deploy the same app, the PermGen keeps increasing.

The exact versions I've tested:

Java 1.6.0_17 (build 1.6.0_17-b04)
Grails 2.0.0
apache-tomcat-7.0.22

This isn't related to the threadlocal problems, it's about the "jar locking" problem:
http://docs.oracle.com/javase/7/docs/technotes/guides/net/ClassLoader.html

Since Java 1.7.0 there is a close() method in URLClassLoader.

Here is a simple (untested) patch for the issue with the AbstractSavePersistentMethod noted in the next-to-previous comment. The problem is that an (anonymous) subclass of ThreadLocal is being created, and that therefore keeps a reference to the classloader. By replacing the initialValue() method with a "manual" setting of the initial value we avoid that subclass and hence the issue. As I said, though, this is untested.

Btw., the shutdown handler here is mostly useless: it only removes the thread-local entry for the thread that happens to run the shutdown handler, but does nothing for all others. I think it could just as well be removed.

I have actually tested the patch now in tomcat 7.0.27 and it works. In the course of testing I also noticed that tomcat was complaining about another place where thread-locals were causing a leak:

org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/site-0.1] created a ThreadLocal with key of type [org.codehaus.groovy.grails.orm.hibernate.support.HibernatePersistenceContextInterceptor$2] (value [org.codehaus.groovy.grails.orm.hibernate.support.HibernatePersistenceContextInterceptor$2@57f16388]) and a value of type [java.lang.Boolean] (value [false]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/site-0.1] created a ThreadLocal with key of type [org.codehaus.groovy.grails.orm.hibernate.support.HibernatePersistenceContextInterceptor$3] (value [org.codehaus.groovy.grails.orm.hibernate.support.HibernatePersistenceContextInterceptor$3@28bda2d3]) and a value of type [java.lang.Integer] (value [0]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

I have attached a second patch for this one (same problem, same solution - and previous comment about shutdown hook basically being useless applies here too).

I've applied to the patches, if problems persist please re-open

I've tested with master latest and I got:

SEVERE: The web application [/leakapp] appears to have started a thread named [Timer-1] but has failed to stop it. This is very likely to create a memory leak.

I created a virtual testing environment and sample app: https://github.com/xymor/grails-5823-test

I noticed that too, but only when using H2 as the database, not when using PostgreSQL (i.e. only in dev/testing, not production). Don't know who the actual culprit is, though.