Grails

Error automatically restarting container

Details

  • Type: Bug Bug
  • Status: Closed Closed
  • Priority: Critical Critical
  • Resolution: Duplicate
  • Affects Version/s: 1.1-beta1
  • Fix Version/s: 1.2-M2
  • Component/s: None
  • Labels:
    None
  • Environment:
    Debian Linux, Sun JDK 6

Description

When running in the development environment, Grails throws a lot of these exceptions after a few minutes. The app starts to misbehave after that, controller calls start to fail indicating server error etc. This makes development mode almost impossible to work with.

It seems to have to do with scanning for changes in source files, and the problem does not occur in testing or production environment.

Error automatically restarting container: IO error scanning directory '/home/marcus/myproj/myproj/src/groovy' ...
: IO error scanning directory '/home/marcus/myproj/myproj/src/groovy'
        at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1065)
        at org.apache.tools.ant.DirectoryScanner.checkIncludePatterns(DirectoryScanner.java:856)
        at org.apache.tools.ant.DirectoryScanner.scan(DirectoryScanner.java:817)
        at org.apache.tools.ant.types.AbstractFileSet.getDirectoryScanner(AbstractFileSet.java:435)
        at org.apache.tools.ant.taskdefs.MatchingTask.getDirectoryScanner(MatchingTask.java:192)
        at org.codehaus.groovy.ant.Groovyc.execute(Groovyc.java:535)
        at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:105)
        at org.apache.tools.ant.Task.perform(Task.java:348)
        at Package_groovy$_run_closure8.doCall(Package_groovy:274)
        at RunApp_groovy$_run_closure3.doCall(RunApp_groovy:126)
        at RunApp_groovy$_run_closure3.doCall(RunApp_groovy)
        at RunApp_groovy$_run_closure1.doCall(RunApp_groovy:61)
        at RunApp_groovy$_run_closure1.doCall(RunApp_groovy)
        at gant.Gant.dispatch(Gant.groovy:193)
        at gant.Gant.this$2$dispatch(Gant.groovy)
        at gant.Gant.invokeMethod(Gant.groovy)
        at gant.Gant.processTargets(Gant.groovy:377)
        at gant.Gant.processTargets(Gant.groovy:314)

There are also sometimes other I/O exceptions about "too many open files".

Activity

Hide
Marcus Better added a comment -

This is indeed a "too many open files" issue. Grails was hitting a resource limit of 1024 open file descriptors per process by default on Debian (and other Linuxes too I suppose). It appears to hold multiple file descriptors to source files, I have observed as many as 20 descriptors for the same controller source file. (lsof output can be sent privately on request.) Every time I exercise the application a bit, like hitting a few controllers from two simultaneous browser windows, the number of open files increases from around 300 to as many as 1000. There are only about 300 files in this project (excluding Grails jar files kept open by the VM).

Grails should really use the inotify mechanism to monitor changes, which I believe does not require holding any file descriptors. Or at least not open the same file many times.

The workaround was to increase the resource limit to 4096 files with ulimit.

Show
Marcus Better added a comment - This is indeed a "too many open files" issue. Grails was hitting a resource limit of 1024 open file descriptors per process by default on Debian (and other Linuxes too I suppose). It appears to hold multiple file descriptors to source files, I have observed as many as 20 descriptors for the same controller source file. (lsof output can be sent privately on request.) Every time I exercise the application a bit, like hitting a few controllers from two simultaneous browser windows, the number of open files increases from around 300 to as many as 1000. There are only about 300 files in this project (excluding Grails jar files kept open by the VM). Grails should really use the inotify mechanism to monitor changes, which I believe does not require holding any file descriptors. Or at least not open the same file many times. The workaround was to increase the resource limit to 4096 files with ulimit.
Hide
Ingo Wiarda added a comment -

I have encountered this issue on Ubuntu-8.10 amd 64bit with Grails 1.0.4.

Show
Ingo Wiarda added a comment - I have encountered this issue on Ubuntu-8.10 amd 64bit with Grails 1.0.4.
Hide
carlos orrego added a comment -

I have encounter the same issue on Ubuntu-8.10 32 bits and grails 1.0.4.

Is there a work around?

should i go back to version 1.0.3 ??

Show
carlos orrego added a comment - I have encounter the same issue on Ubuntu-8.10 32 bits and grails 1.0.4. Is there a work around? should i go back to version 1.0.3 ??
Hide
Marcus Better added a comment -

The workaround, as stated in an earlier comment, is to increase the "nofile" resource limit. Check the man page for limits.conf for details.

Show
Marcus Better added a comment - The workaround, as stated in an earlier comment, is to increase the "nofile" resource limit. Check the man page for limits.conf for details.
Hide
Peter Ledbrook added a comment -

I've not come across this recently. Could you see if it happens with 1.1-beta3 when we release that?

Show
Peter Ledbrook added a comment - I've not come across this recently. Could you see if it happens with 1.1-beta3 when we release that?
Hide
Matt Raible added a comment -

How do I increase the "nofile" resource limit on OS X? The following doesn't seem to work:

sudo ulimit -n 12000

The following doesn't seem to help and I still get "256" when I run "ulimit -n".

sudo sysctl -w kern.maxfilesperproc=20480 kern.maxfiles=22528

Show
Matt Raible added a comment - How do I increase the "nofile" resource limit on OS X? The following doesn't seem to work: sudo ulimit -n 12000 The following doesn't seem to help and I still get "256" when I run "ulimit -n". sudo sysctl -w kern.maxfilesperproc=20480 kern.maxfiles=22528
Hide
Serge added a comment -

Updating to 1.1-RC2 fixes it for me, however the recommended fs.file-max increase didn't help with version 1.0.4.
It was 317,000 and I increased it to 500,000 and still had the same error.

Anyway, below are the details of the hardware/software I had a problem with:

Computer:Intel Core2Duo E5200 Dual Core, 2 x 2.5Ghz, 2MB Cache, 800Mhz FSB, 4GB DDR RAM
Operating System: Linux - CentOS 5 (CentOS release 5.2 (Final))
O/S Bit Type: 64 Bit
Grails: 1.0.4
Groovy Version: 1.6.0
JVM: 1.6.0_12
fs.file-max = 317000 (increasing to 500,000 doesn't help)

: IO error scanning directory '/project/folder/src/groovy'
at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1065)
at org.apache.tools.ant.DirectoryScanner.checkIncludePatterns(DirectoryScanner.java:856)
at org.apache.tools.ant.DirectoryScanner.scan(DirectoryScanner.java:817)
at org.apache.tools.ant.types.AbstractFileSet.getDirectoryScanner(AbstractFileSet.java:435)
at org.apache.tools.ant.taskdefs.MatchingTask.getDirectoryScanner(MatchingTask.java:192)
at org.codehaus.groovy.ant.Groovyc.execute(Groovyc.java:535)
at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:105)
at org.apache.tools.ant.Task.perform(Task.java:348)
at Package_groovy$_run_closure10.doCall(Package_groovy:349)
at RunApp_groovy$_run_closure3.doCall(RunApp_groovy:130)
at RunApp_groovy$_run_closure3.doCall(RunApp_groovy)
at RunApp_groovy$_run_closure1.doCall(RunApp_groovy:58)
at RunApp_groovy$_run_closure1.doCall(RunApp_groovy)
at gant.Gant.dispatch(Gant.groovy:271)
at gant.Gant.this$2$dispatch(Gant.groovy)
at gant.Gant.invokeMethod(Gant.groovy)
at gant.Gant.processTargets(Gant.groovy:436)
at gant.Gant.processArgs(Gant.groovy:37

Show
Serge added a comment - Updating to 1.1-RC2 fixes it for me, however the recommended fs.file-max increase didn't help with version 1.0.4. It was 317,000 and I increased it to 500,000 and still had the same error. Anyway, below are the details of the hardware/software I had a problem with: Computer:Intel Core2Duo E5200 Dual Core, 2 x 2.5Ghz, 2MB Cache, 800Mhz FSB, 4GB DDR RAM Operating System: Linux - CentOS 5 (CentOS release 5.2 (Final)) O/S Bit Type: 64 Bit Grails: 1.0.4 Groovy Version: 1.6.0 JVM: 1.6.0_12 fs.file-max = 317000 (increasing to 500,000 doesn't help) : IO error scanning directory '/project/folder/src/groovy' at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1065) at org.apache.tools.ant.DirectoryScanner.checkIncludePatterns(DirectoryScanner.java:856) at org.apache.tools.ant.DirectoryScanner.scan(DirectoryScanner.java:817) at org.apache.tools.ant.types.AbstractFileSet.getDirectoryScanner(AbstractFileSet.java:435) at org.apache.tools.ant.taskdefs.MatchingTask.getDirectoryScanner(MatchingTask.java:192) at org.codehaus.groovy.ant.Groovyc.execute(Groovyc.java:535) at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:105) at org.apache.tools.ant.Task.perform(Task.java:348) at Package_groovy$_run_closure10.doCall(Package_groovy:349) at RunApp_groovy$_run_closure3.doCall(RunApp_groovy:130) at RunApp_groovy$_run_closure3.doCall(RunApp_groovy) at RunApp_groovy$_run_closure1.doCall(RunApp_groovy:58) at RunApp_groovy$_run_closure1.doCall(RunApp_groovy) at gant.Gant.dispatch(Gant.groovy:271) at gant.Gant.this$2$dispatch(Gant.groovy) at gant.Gant.invokeMethod(Gant.groovy) at gant.Gant.processTargets(Gant.groovy:436) at gant.Gant.processArgs(Gant.groovy:37
Hide
Graeme Rocher added a comment -

This should be already fixed, since we resolved the last remaining file locking issue related to keeping GSPs open recently

Show
Graeme Rocher added a comment - This should be already fixed, since we resolved the last remaining file locking issue related to keeping GSPs open recently

People

Vote (1)
Watch (4)

Dates

  • Created:
    Updated:
    Resolved: