Details
-
Type:
Bug
-
Status:
Closed
-
Priority:
Critical
-
Resolution: Duplicate
-
Affects Version/s: 1.1-beta1
-
Fix Version/s: 1.2-M2
-
Component/s: None
-
Labels:None
-
Environment:Debian Linux, Sun JDK 6
Description
When running in the development environment, Grails throws a lot of these exceptions after a few minutes. The app starts to misbehave after that, controller calls start to fail indicating server error etc. This makes development mode almost impossible to work with.
It seems to have to do with scanning for changes in source files, and the problem does not occur in testing or production environment.
Error automatically restarting container: IO error scanning directory '/home/marcus/myproj/myproj/src/groovy' ...
: IO error scanning directory '/home/marcus/myproj/myproj/src/groovy'
at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1065)
at org.apache.tools.ant.DirectoryScanner.checkIncludePatterns(DirectoryScanner.java:856)
at org.apache.tools.ant.DirectoryScanner.scan(DirectoryScanner.java:817)
at org.apache.tools.ant.types.AbstractFileSet.getDirectoryScanner(AbstractFileSet.java:435)
at org.apache.tools.ant.taskdefs.MatchingTask.getDirectoryScanner(MatchingTask.java:192)
at org.codehaus.groovy.ant.Groovyc.execute(Groovyc.java:535)
at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:105)
at org.apache.tools.ant.Task.perform(Task.java:348)
at Package_groovy$_run_closure8.doCall(Package_groovy:274)
at RunApp_groovy$_run_closure3.doCall(RunApp_groovy:126)
at RunApp_groovy$_run_closure3.doCall(RunApp_groovy)
at RunApp_groovy$_run_closure1.doCall(RunApp_groovy:61)
at RunApp_groovy$_run_closure1.doCall(RunApp_groovy)
at gant.Gant.dispatch(Gant.groovy:193)
at gant.Gant.this$2$dispatch(Gant.groovy)
at gant.Gant.invokeMethod(Gant.groovy)
at gant.Gant.processTargets(Gant.groovy:377)
at gant.Gant.processTargets(Gant.groovy:314)
There are also sometimes other I/O exceptions about "too many open files".
This is indeed a "too many open files" issue. Grails was hitting a resource limit of 1024 open file descriptors per process by default on Debian (and other Linuxes too I suppose). It appears to hold multiple file descriptors to source files, I have observed as many as 20 descriptors for the same controller source file. (lsof output can be sent privately on request.) Every time I exercise the application a bit, like hitting a few controllers from two simultaneous browser windows, the number of open files increases from around 300 to as many as 1000. There are only about 300 files in this project (excluding Grails jar files kept open by the VM).
Grails should really use the inotify mechanism to monitor changes, which I believe does not require holding any file descriptors. Or at least not open the same file many times.
The workaround was to increase the resource limit to 4096 files with ulimit.