[jetty-users] Help trying to investigate a file descriptor leak

[m***@ow2.org]

Hello everybody.

I need a guess if the issue below could come from Jetty.
We run a webapp (XWiki) at https://www.ow2.org

The application is reached through Apache as a reverse proxy.
Jetty is 9.4.6.v20170531 running on Debian (9.3)
JVM is 8u151-b12-1~deb9u1

The issue I observe from the xwiki perspective is detailed at:

https://jira.xwiki.org/browse/XWIKI-15009

As far as I understand, the specific content of the webapp at
https://www.ow2.org/resources/ is statically served by Jetty and opened
by XWiki through it. (I don't know how).

The issue we observe is as time goes on, many files - not all of them -
in the folder 'resources' above like
https://www.ow2.org/resources/js/xwiki/meta.js are opened more than 2
times by the java process and that number keeps increasing. We observe
that by running:

lsof -p 21948 | egrep '(REG|DIR)' | awk '{print $9'} | sort | uniq -c
|sort -n

XWiki support team says they haven't got any report of such a behavior
recently.

By the way we're also using http://www.tuckey.org/urlrewrite/ which is
installed as a JAR in the webapp and called via web.xml. The
configuration of the rewrite filter is as stated in
http://www.xwiki.org/xwiki/bin/view/Documentation/AdminGuide/ShortURLs/#HUrlRewriteFilter

Unfortunately I can't disable the rewrite filter for testing purpose
right now as it would break URL schemes compatibility with URLs already
spread over the internet.

NB : The leak concerns *only* files in '/resources' which are expected
to be server statically (by Jetty in the current case).

Regards,

[Jan Bartel]

We haven't had any reports of file descriptor leaks that I'm aware of.

How are you serving static content with Jetty - are you using a
ResourceHandler, or are you using the DefaultServlet, or something else?

If you are using the DefaultServlet, check how it is configured - do you
have caching enabled? Look for the settings of these init-params:

- resourceCache
- maxCacheSize
- maxCachedFileSize
- maxCachedFiles
- useFileMappedBuffer

You can turn debug on to see more info on the handling of static files -
add -Dorg.eclipse.jetty.servlet.DefaultServlet.LEVEL=DEBUG and
-Dorg.eclipse.jetty.server.ResourceService.LEVEL=DEBUG to the command line
if you're using jetty's default logging, or otherwise configure whatever
logging system you're using to output highest level for those classes.

You can also specify jetty.dump.start=true at the very end of the jetty
command line. That will spit out most of the configuration of your jetty
server to the log just after it starts - might reveal something useful.

regards
Jan

Post by m***@ow2.org
Hello everybody.
I need a guess if the issue below could come from Jetty.
We run a webapp (XWiki) at https://www.ow2.org
The application is reached through Apache as a reverse proxy.
Jetty is 9.4.6.v20170531 running on Debian (9.3)
JVM is 8u151-b12-1~deb9u1
https://jira.xwiki.org/browse/XWIKI-15009
As far as I understand, the specific content of the webapp at
https://www.ow2.org/resources/ is statically served by Jetty and opened
by XWiki through it. (I don't know how).
The issue we observe is as time goes on, many files - not all of them -
in the folder 'resources' above like
https://www.ow2.org/resources/js/xwiki/meta.js are opened more than 2
times by the java process and that number keeps increasing. We observe
lsof -p 21948 | egrep '(REG|DIR)' | awk '{print $9'} | sort | uniq -c
|sort -n
XWiki support team says they haven't got any report of such a behavior
recently.
By the way we're also using http://www.tuckey.org/urlrewrite/ which is
installed as a JAR in the webapp and called via web.xml. The
configuration of the rewrite filter is as stated in
http://www.xwiki.org/xwiki/bin/view/Documentation/AdminGuide/ShortURLs/#
HUrlRewriteFilter
Unfortunately I can't disable the rewrite filter for testing purpose
right now as it would break URL schemes compatibility with URLs already
spread over the internet.
NB : The leak concerns *only* files in '/resources' which are expected
to be server statically (by Jetty in the current case).
Regards,
_______________________________________________
jetty-users mailing list
To change your delivery options, retrieve your password, or unsubscribe
from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users

[m***@ow2.org]

Hi Jan,

Le Tue, 20 Feb 2018 16:25:04 +1100,

Post by Jan Bartel
We haven't had any reports of file descriptor leaks that I'm aware of.

Thank you very much for spending time to answer.

Post by Jan Bartel
How are you serving static content with Jetty - are you using a
ResourceHandler, or are you using the DefaultServlet, or something else?

I have no kind of idea. I'm the system administrator, so I can only
provide informations about how I've installed jetty and deployed the
application.

I'm using the shipped init script, that reads from /etc/default/jetty.
In that file I have:

JETTY_HOME=/srv/apprun/jetty/current
JETTY_BASE=/srv/appdata/jetty/base-xxx
TMPDIR=/srv/appdata/jetty/temp-xxx
JAVA_OPTIONS='-Xmx4096m'

The webapp is deployed in base-xxx/webapps/root

Inside WEB-INF/ there, there is a web.xml which doesn't contain any
reference to the classes ResourceHandler,DefaultServlet you mention.

(I am providing web.xml as attachement)

Inside JETTY_BASE I have a classic start.ini file with the following
content:

$ cat start.ini | sed -e '/^#/ d' -e '/^$/ d'
--module=http-forwarded
--module=console-capture
--module=http
jetty.http.port=8100
--module=deploy
jetty.deploy.extractWars=false


About JETTY_HOME, jetty runtime is unpacked here, untouched.

Post by Jan Bartel
If you are using the DefaultServlet, check how it is configured - do
- resourceCache
- maxCacheSize
- maxCachedFileSize
- maxCachedFiles
- useFileMappedBuffer
You can turn debug on to see more info on the handling of static
files - add -Dorg.eclipse.jetty.servlet.DefaultServlet.LEVEL=DEBUG
and -Dorg.eclipse.jetty.server.ResourceService.LEVEL=DEBUG to the
command line if you're using jetty's default logging, or otherwise
configure whatever logging system you're using to output highest
level for those classes.

I am supposed to add the above -Dxx calls in /etc/default/jetty ? in
JAVA_OPTIONS variable ?

Post by Jan Bartel
You can also specify jetty.dump.start=true at the very end of the
jetty command line. That will spit out most of the configuration of
your jetty server to the log just after it starts - might reveal
something useful.

I'm adding this to start.ini I suppose. Thanks.

Post by Jan Bartel
regards

Regards,

Post by Jan Bartel
Jan

Post by m***@ow2.org
Hello everybody.
I need a guess if the issue below could come from Jetty.
We run a webapp (XWiki) at https://www.ow2.org
The application is reached through Apache as a reverse proxy.
Jetty is 9.4.6.v20170531 running on Debian (9.3)
JVM is 8u151-b12-1~deb9u1
https://jira.xwiki.org/browse/XWIKI-15009
As far as I understand, the specific content of the webapp at
https://www.ow2.org/resources/ is statically served by Jetty and
opened by XWiki through it. (I don't know how).
The issue we observe is as time goes on, many files - not all of
them - in the folder 'resources' above like
https://www.ow2.org/resources/js/xwiki/meta.js are opened more than
2 times by the java process and that number keeps increasing. We
lsof -p 21948 | egrep '(REG|DIR)' | awk '{print $9'} | sort | uniq
-c |sort -n
XWiki support team says they haven't got any report of such a
behavior recently.
By the way we're also using http://www.tuckey.org/urlrewrite/ which
is installed as a JAR in the webapp and called via web.xml. The
configuration of the rewrite filter is as stated in
http://www.xwiki.org/xwiki/bin/view/Documentation/AdminGuide/ShortURLs/#
HUrlRewriteFilter
Unfortunately I can't disable the rewrite filter for testing purpose
right now as it would break URL schemes compatibility with URLs
already spread over the internet.
NB : The leak concerns *only* files in '/resources' which are
expected to be server statically (by Jetty in the current case).
Regards,
_______________________________________________
jetty-users mailing list
To change your delivery options, retrieve your password, or
unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users

[Brian Reichert]

Post by m***@ow2.org
Hi Jan,
Le Tue, 20 Feb 2018 16:25:04 +1100,

Post by Jan Bartel
We haven't had any reports of file descriptor leaks that I'm aware of.

Thank you very much for spending time to answer.

Post by Jan Bartel
How are you serving static content with Jetty - are you using a
ResourceHandler, or are you using the DefaultServlet, or something else?

I have no kind of idea. I'm the system administrator, so I can only
provide informations about how I've installed jetty and deployed the
application.

Not assert what you're seeing is the same as what I was seeing, but I
observed some evidence of jetty retaining file descriptors when the
hot-deploy feature was utilized:

http://dev.eclipse.org/mhonarc/lists/jetty-dev/msg02777.html

In that, I was fixated on disk space consumption, but the disk space
was consumed as jetty was retaining references to deleted files.

That resulted in me opening this bug:

https://github.com/eclipse/jetty.project/issues/867

And the resulting assertion is that it's not a bug, but a known
behavior of Java that you can tune away.

Anyway, this may not be at all related to what you're seeing, but
wanted to provide a heads-up.

[Martin]

Le Mon, 26 Feb 2018 10:10:45 -0500,

Post by Brian Reichert

Post by m***@ow2.org
Hi Jan,
Le Tue, 20 Feb 2018 16:25:04 +1100,

Post by Jan Bartel
We haven't had any reports of file descriptor leaks that I'm aware of.

Thank you very much for spending time to answer.

Post by Jan Bartel
How are you serving static content with Jetty - are you using a
ResourceHandler, or are you using the DefaultServlet, or something else?

I have no kind of idea. I'm the system administrator, so I can only
provide informations about how I've installed jetty and deployed the
application.

Not assert what you're seeing is the same as what I was seeing, but I
observed some evidence of jetty retaining file descriptors when the
http://dev.eclipse.org/mhonarc/lists/jetty-dev/msg02777.html
In that, I was fixated on disk space consumption, but the disk space
was consumed as jetty was retaining references to deleted files.
https://github.com/eclipse/jetty.project/issues/867
And the resulting assertion is that it's not a bug, but a known
behavior of Java that you can tune away.
Anyway, this may not be at all related to what you're seeing, but
wanted to provide a heads-up.

Thanks Brian and Jan,

XWiki team fixed the issue in:
https://jira.xwiki.org/browse/XWIKI-15009


About Jetty, I still to not exactly understand that says:

https://jira.xwiki.org/browse/XWIKI-15009?focusedCommentId=98061&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-98061

But all in all it seems fixed.

Regards,