[jetty-users] Jetty server crashes

Discussion:

kapil gupta

2018-01-10 04:37:06 UTC

We are using jetty server and when it is on load then the server is crashed
without showing much of information.
We have also put it on JMX. The CPU usage and memory looks fine, but still
Jetty server crashes. Please let me know what parameters we should look for
and how can we isolate issue which is causing jetty server crash.

Greg Wilkins

2018-01-10 08:10:07 UTC

Permalink

Kapil,

I'm sorry but you've not given us enough information to say anything. What
do you mean by "crashed"? If it is the JVM that stops, then that is not a
Jetty problem. Is there a stack trace? Is there any logging? Is the
process still running? Can any requests be served? Is the connector still
listening? Can you run with debug? Why do you think it is jetty and not
some other component?

regards

Post by kapil gupta
We are using jetty server and when it is on load then the server is
crashed without showing much of information.
We have also put it on JMX. The CPU usage and memory looks fine, but still
Jetty server crashes. Please let me know what parameters we should look for
and how can we isolate issue which is causing jetty server crash.
_______________________________________________
jetty-users mailing list
To change your delivery options, retrieve your password, or unsubscribe
from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users

--
Greg Wilkins <***@webtide.com> CTO http://webtide.com

kapil gupta

2018-01-10 11:50:15 UTC

Permalink

Greg,

Actually below things are observed:

1. No process of jetty server running
2. MySQL still runs
3. CPU usage shows data fine
4. RAM is also fine

Not sure how can we isolate if issue could be because of JVM, can you
please guide on setting some parameters or moniotring them.

Thanks

Post by Greg Wilkins
Kapil,
I'm sorry but you've not given us enough information to say anything.
What do you mean by "crashed"? If it is the JVM that stops, then that is
not a Jetty problem. Is there a stack trace? Is there any logging? Is the
process still running? Can any requests be served? Is the connector still
listening? Can you run with debug? Why do you think it is jetty and not
some other component?
regards

Post by kapil gupta
We are using jetty server and when it is on load then the server is
crashed without showing much of information.
We have also put it on JMX. The CPU usage and memory looks fine, but
still Jetty server crashes. Please let me know what parameters we should
look for and how can we isolate issue which is causing jetty server crash.
_______________________________________________
jetty-users mailing list
To change your delivery options, retrieve your password, or unsubscribe
from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users

--
_______________________________________________
jetty-users mailing list
To change your delivery options, retrieve your password, or unsubscribe
from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users

Greg Wilkins

2018-01-10 15:07:09 UTC

Permalink

Kapil,

java is a very safe language and there should be no way for a java program
to "crash" and end up with no process running... unless the program calls
System.exit(), which Jetty does not. Thus it is possible that you have
hit a JVM problem, as sometimes the JVM will segv due to a bug and crash.

I suggest making sure your operating system is up to date with latest
patches and checking your JVM is a recent stable release.

If it still happens, you need to capture how it failed. Perhaps the start
script can log the exit code? Are there any crash files in the directory
where it ran?

regards

Post by kapil gupta
We are using jetty server and when it is on load then the server is
crashed without showing much of information.
We have also put it on JMX. The CPU usage and memory looks fine, but
still Jetty server crashes. Please let me know what parameters we should
look for and how can we isolate issue which is causing jetty server crash.
_______________________________________________
jetty-users mailing list
To change your delivery options, retrieve your password, or unsubscribe
from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users

_______________________________________________
jetty-users mailing list
To change your delivery options, retrieve your password, or unsubscribe
from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users

--
Greg Wilkins <***@webtide.com> CTO http://webtide.com

Brian Reichert

2018-01-10 15:15:55 UTC

Permalink

See if you're using any of these directives with your JVM:

-XX:+ExitOnOutOfMemoryError
-XX:+CrashOnOutOfMemoryError

That's one way the JVM could exit.

The JVM used to run jetty will, by default, generate output to
STDOUT and STDERR. Are you collecting that output into a log?
Unless the JVM was terminated with a KILL signal, it likely would
have recorded that it was shutting down for a specific reason.

It's possible your server has an Out-of-Memory killer, to help
manage a system running very low on resources, e.g.:

https://www.memset.com/docs/additional-information/oom-killer/

I have no direct experience with that feature of the OS, though.

--
Brian Reichert <***@numachi.com>
BSD admin/developer at large

Joakim Erdfelt

2018-01-10 15:16:29 UTC

Permalink

If the Java JVM did crash, I would expect a hs_err_pid<pid>.log file to
exist telling you why the java JVM crashed.

You can find that file in the working directory from where you started the
JVM.

Or you can configure the location on the JVM.
http://www.oracle.com/technetwork/java/javase/felog-138657.html#gbwcy

Post by Greg Wilkins
Kapil,
java is a very safe language and there should be no way for a java program
to "crash" and end up with no process running... unless the program calls
System.exit(), which Jetty does not. Thus it is possible that you have
hit a JVM problem, as sometimes the JVM will segv due to a bug and crash.
I suggest making sure your operating system is up to date with latest
patches and checking your JVM is a recent stable release.
If it still happens, you need to capture how it failed. Perhaps the
start script can log the exit code? Are there any crash files in the
directory where it ran?
regards

Post by kapil gupta
We are using jetty server and when it is on load then the server is
crashed without showing much of information.
We have also put it on JMX. The CPU usage and memory looks fine, but
still Jetty server crashes. Please let me know what parameters we should
look for and how can we isolate issue which is causing jetty server crash.
_______________________________________________
jetty-users mailing list
To change your delivery options, retrieve your password, or unsubscribe
from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users

Silvio Bierman

2018-01-11 09:30:47 UTC

Permalink

Kapil,

If you are running on Linux check the system log for kernel panics due
to lack of memory. Linux will ruthlessly kill the process with highest
memory load in such a situation. That has happened to me plenty of times
and also left me wondering where my JVM process had gone.

And to add to what Greg said: the System.exit thing can also happen in a
library. I used a library once that would do a System.exit in some error
situations killing my server now and then. That was a real puzzler.

Cheers,

Silvio

Post by kapil gupta
Greg,
1. No process of jetty server running
2. MySQL still runs
3. CPU usage shows data fine
4. RAM is also fine
Not sure how can we isolate if issue could be because of JVM, can you
please guide on setting some parameters or moniotring them.
Thanks
Kapil,
I'm sorry but you've not given us enough information to say
anything.Â What do you mean by "crashed"?Â If it is the JVM that
stops, then that is not a Jetty problem.Â Is there a stack trace?
Is there any logging? Is the process still running? Can any
requests be served? Is the connector still listening?Â Can you run
with debug?Â Why do you think it is jetty and not some other
component?
regards
We are using jetty server and when it is on load then the
server is crashed without showing much of information.
We have also put it on JMX. The CPU usage and memory looks
fine, but still Jetty server crashes. Please let me know what
parameters we should look for and how can we isolate issue
which is causing jetty server crash.
_______________________________________________
jetty-users mailing list
To change your delivery options, retrieve your password, or
unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users
<https://dev.eclipse.org/mailman/listinfo/jetty-users>
--
http://webtide.com
_______________________________________________
jetty-users mailing list
To change your delivery options, retrieve your password, or
unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users
<https://dev.eclipse.org/mailman/listinfo/jetty-users>
_______________________________________________
jetty-users mailing list
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users

kapil gupta

2018-01-12 04:38:13 UTC

Permalink

Thanks everyone for your replies.

Some more points into it:

1. @Silvio - As you mentioned, yes sometimes OOM killer invokes jetty
process to stop, but many of times it doesnt.
2. @Greg - We are using cloud, and running on CentOS 7. We have stopped
updates due to some reasons.
3. @Joakim - Yes, checked the thread dump and some times OOM killer
invokes jetty crash, sometimes I see below properties in dump file:
1.
2. Current thread (0x00007f18e01b9800): VMThread [stack:
0x00007f18d33f4000,0x00007f18d34f5000] [id=11584]
3.
4. Stack: [0x00007f18d33f4000,0x00007f18d34f5000],
sp=0x00007f18d34f3140, free space=1020k
5. Native frames: (J=compiled Java code, j=interpreted, Vv=VM code,
C=native code)
6. V [libjvm.so+0x9a320a] VMError::report_and_die()+0x2ea
7. V [libjvm.so+0x498d3b] report_vm_out_of_memory(char const*, int,
unsigned long, char const*)+0x9b
8. V [libjvm.so+0x82191e] os::Linux::commit_memory_impl(char*,
unsigned long, bool)+0xfe
9. V [libjvm.so+0x821e69] os::pd_commit_memory(char*, unsigned
long, unsigned long, bool)+0x29
10. V [libjvm.so+0x81bb6a] os::commit_memory(char*, unsigned long,
unsigned long, bool)+0x2a
11. V [libjvm.so+0x88d623] PSVirtualSpace::expand_by(unsigned
long)+0x53
12. V [libjvm.so+0x88e9f8] PSYoungGen::resize_generation(unsigned
long, unsigned long)+0xf8
13. V [libjvm.so+0x88db62] PSYoungGen::resize(unsigned long,
unsigned long)+0x22
14. V [libjvm.so+0x88af1d] PSScavenge::invoke_no_policy()+0xf3d
15. V [libjvm.so+0x88b761] PSScavenge::invoke()+0x41
16. V [libjvm.so+0x843f40]
ParallelScavengeHeap::failed_mem_allocate(unsigned long)+0x70
17. V [libjvm.so+0x9a4a97]
VM_ParallelGCFailedAllocation::doit()+0x97
18. V [libjvm.so+0x9abf35] VM_Operation::evaluate()+0x55
19. V [libjvm.so+0x9aa2fa]
VMThread::evaluate_operation(VM_Operation*)+0xba
20. V [libjvm.so+0x9aa67e] VMThread::loop()+0x1ce
21. V [libjvm.so+0x9aaaf0] VMThread::run()+0x70
22. V [libjvm.so+0x8238c8] java_start(Thread*)+0x108

I do not have much of knowledge in to these configuration, need you guys
help.
Is this because of my server goes OutOfMemory, if yes then why it doesnt
show the memory occupied more in VisualVM when we check heap size

Post by Greg Wilkins
Kapil,
If you are running on Linux check the system log for kernel panics due to
lack of memory. Linux will ruthlessly kill the process with highest memory
load in such a situation. That has happened to me plenty of times and also
left me wondering where my JVM process had gone.
And to add to what Greg said: the System.exit thing can also happen in a
library. I used a library once that would do a System.exit in some error
situations killing my server now and then. That was a real puzzler.
Cheers,
Silvio
Greg,
1. No process of jetty server running
2. MySQL still runs
3. CPU usage shows data fine
4. RAM is also fine
Not sure how can we isolate if issue could be because of JVM, can you
please guide on setting some parameters or moniotring them.
Thanks

Post by kapil gupta
We are using jetty server and when it is on load then the server is
crashed without showing much of information.
We have also put it on JMX. The CPU usage and memory looks fine, but
still Jetty server crashes. Please let me know what parameters we should
look for and how can we isolate issue which is causing jetty server crash.
_______________________________________________
jetty-users mailing list
To change your delivery options, retrieve your password, or unsubscribe
from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users

_______________________________________________
To change your delivery options, retrieve your password, or unsubscribe from this list, visithttps://dev.eclipse.org/mailman/listinfo/jetty-users
_______________________________________________
jetty-users mailing list
To change your delivery options, retrieve your password, or unsubscribe
from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users

Greg Wilkins

2018-01-12 10:04:33 UTC

Permalink

Kapil,

if your server is running out of memory, then it is most likely the
application rather than jetty itself.
I would advise:
+ increasing the memory allocated to your production JVMs
+ running a test system with realistic load and profiling the applications
memory usage both over time (for leaks) or just for memory hungry
operations.

regards

Post by kapil gupta
Thanks everyone for your replies.
process to stop, but many of times it doesnt.
stopped updates due to some reasons.
1.
0x00007f18d33f4000,0x00007f18d34f5000] [id=11584]
3.
4. Stack: [0x00007f18d33f4000,0x00007f18d34f5000],
sp=0x00007f18d34f3140, free space=1020k
5. Native frames: (J=compiled Java code, j=interpreted, Vv=VM code,
C=native code)
6. V [libjvm.so+0x9a320a] VMError::report_and_die()+0x2ea
7. V [libjvm.so+0x498d3b] report_vm_out_of_memory(char const*,
int, unsigned long, char const*)+0x9b
8. V [libjvm.so+0x82191e] os::Linux::commit_memory_impl(char*,
unsigned long, bool)+0xfe
9. V [libjvm.so+0x821e69] os::pd_commit_memory(char*, unsigned
long, unsigned long, bool)+0x29
10. V [libjvm.so+0x81bb6a] os::commit_memory(char*, unsigned
long, unsigned long, bool)+0x2a
11. V [libjvm.so+0x88d623] PSVirtualSpace::expand_by(unsigned
long)+0x53
12. V [libjvm.so+0x88e9f8] PSYoungGen::resize_generation(unsigned
long, unsigned long)+0xf8
13. V [libjvm.so+0x88db62] PSYoungGen::resize(unsigned long,
unsigned long)+0x22
14. V [libjvm.so+0x88af1d] PSScavenge::invoke_no_policy()+0xf3d
15. V [libjvm.so+0x88b761] PSScavenge::invoke()+0x41
16. V [libjvm.so+0x843f40] ParallelScavengeHeap::failed_mem_allocate(unsigned
long)+0x70
doit()+0x97
18. V [libjvm.so+0x9abf35] VM_Operation::evaluate()+0x55
19. V [libjvm.so+0x9aa2fa] VMThread::evaluate_operation(
VM_Operation*)+0xba
20. V [libjvm.so+0x9aa67e] VMThread::loop()+0x1ce
21. V [libjvm.so+0x9aaaf0] VMThread::run()+0x70
22. V [libjvm.so+0x8238c8] java_start(Thread*)+0x108
I do not have much of knowledge in to these configuration, need you guys
help.
Is this because of my server goes OutOfMemory, if yes then why it doesnt
show the memory occupied more in VisualVM when we check heap size
On Thu, Jan 11, 2018 at 3:00 PM, Silvio Bierman <

Post by kapil gupta
We are using jetty server and when it is on load then the server is
crashed without showing much of information.
We have also put it on JMX. The CPU usage and memory looks fine, but
still Jetty server crashes. Please let me know what parameters we should
look for and how can we isolate issue which is causing jetty server crash.
_______________________________________________
jetty-users mailing list
To change your delivery options, retrieve your password, or unsubscribe
from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users

--
Greg Wilkins <***@webtide.com> CTO http://webtide.com

kapil gupta

2018-01-12 10:15:47 UTC

Permalink

Sure Greg,

I will check out for the points suggested. But one thing, why my VisualVM
doesn show heap size hike, it hardly consumes 50-60% of allocated memory.

Post by Greg Wilkins
Kapil,
if your server is running out of memory, then it is most likely the
application rather than jetty itself.
+ increasing the memory allocated to your production JVMs
+ running a test system with realistic load and profiling the
applications memory usage both over time (for leaks) or just for memory
hungry operations.
regards

Post by kapil gupta
Thanks everyone for your replies.
process to stop, but many of times it doesnt.
stopped updates due to some reasons.
1.
0x00007f18d33f4000,0x00007f18d34f5000] [id=11584]
3.
4. Stack: [0x00007f18d33f4000,0x00007f18d34f5000],
sp=0x00007f18d34f3140, free space=1020k
5. Native frames: (J=compiled Java code, j=interpreted, Vv=VM
code, C=native code)
6. V [libjvm.so+0x9a320a] VMError::report_and_die()+0x2ea
7. V [libjvm.so+0x498d3b] report_vm_out_of_memory(char const*,
int, unsigned long, char const*)+0x9b
8. V [libjvm.so+0x82191e] os::Linux::commit_memory_impl(char*,
unsigned long, bool)+0xfe
9. V [libjvm.so+0x821e69] os::pd_commit_memory(char*, unsigned
long, unsigned long, bool)+0x29
10. V [libjvm.so+0x81bb6a] os::commit_memory(char*, unsigned
long, unsigned long, bool)+0x2a
11. V [libjvm.so+0x88d623] PSVirtualSpace::expand_by(unsigned
long)+0x53
12. V [libjvm.so+0x88e9f8] PSYoungGen::resize_generation(unsigned
long, unsigned long)+0xf8
13. V [libjvm.so+0x88db62] PSYoungGen::resize(unsigned long,
unsigned long)+0x22
14. V [libjvm.so+0x88af1d] PSScavenge::invoke_no_policy()+0xf3d
15. V [libjvm.so+0x88b761] PSScavenge::invoke()+0x41
16. V [libjvm.so+0x843f40] ParallelScavengeHeap::failed_mem_allocate(unsigned
long)+0x70
17. V [libjvm.so+0x9a4a97] VM_ParallelGCFailedAllocation
::doit()+0x97
18. V [libjvm.so+0x9abf35] VM_Operation::evaluate()+0x55
19. V [libjvm.so+0x9aa2fa] VMThread::evaluate_operation(
VM_Operation*)+0xba
20. V [libjvm.so+0x9aa67e] VMThread::loop()+0x1ce
21. V [libjvm.so+0x9aaaf0] VMThread::run()+0x70
22. V [libjvm.so+0x8238c8] java_start(Thread*)+0x108
I do not have much of knowledge in to these configuration, need you guys
help.
Is this because of my server goes OutOfMemory, if yes then why it doesnt
show the memory occupied more in VisualVM when we check heap size
On Thu, Jan 11, 2018 at 3:00 PM, Silvio Bierman <

Post by Greg Wilkins
Kapil,
If you are running on Linux check the system log for kernel panics due
to lack of memory. Linux will ruthlessly kill the process with highest
memory load in such a situation. That has happened to me plenty of times
and also left me wondering where my JVM process had gone.
And to add to what Greg said: the System.exit thing can also happen in a
library. I used a library once that would do a System.exit in some error
situations killing my server now and then. That was a real puzzler.
Cheers,
Silvio
Greg,
1. No process of jetty server running
2. MySQL still runs
3. CPU usage shows data fine
4. RAM is also fine
Not sure how can we isolate if issue could be because of JVM, can you
please guide on setting some parameters or moniotring them.
Thanks

Post by kapil gupta
We are using jetty server and when it is on load then the server is
crashed without showing much of information.
We have also put it on JMX. The CPU usage and memory looks fine, but
still Jetty server crashes. Please let me know what parameters we should
look for and how can we isolate issue which is causing jetty server crash.
_______________________________________________
jetty-users mailing list
To change your delivery options, retrieve your password, or
unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users

kapil gupta

2018-01-18 07:12:03 UTC

Permalink

Also one more thing we are facing issues when the request is not going to
jetty server, there is no entry in access-log and application keeps loading.

Any pointers would be helpful

Post by kapil gupta
Sure Greg,
I will check out for the points suggested. But one thing, why my VisualVM
doesn show heap size hike, it hardly consumes 50-60% of allocated memory.

Post by kapil gupta
Thanks everyone for your replies.
jetty process to stop, but many of times it doesnt.
stopped updates due to some reasons.
1.
0x00007f18d33f4000,0x00007f18d34f5000] [id=11584]
3.
4. Stack: [0x00007f18d33f4000,0x00007f18d34f5000],
sp=0x00007f18d34f3140, free space=1020k
5. Native frames: (J=compiled Java code, j=interpreted, Vv=VM
code, C=native code)
6. V [libjvm.so+0x9a320a] VMError::report_and_die()+0x2ea
7. V [libjvm.so+0x498d3b] report_vm_out_of_memory(char const*,
int, unsigned long, char const*)+0x9b
8. V [libjvm.so+0x82191e] os::Linux::commit_memory_impl(char*,
unsigned long, bool)+0xfe
9. V [libjvm.so+0x821e69] os::pd_commit_memory(char*, unsigned
long, unsigned long, bool)+0x29
10. V [libjvm.so+0x81bb6a] os::commit_memory(char*, unsigned
long, unsigned long, bool)+0x2a
11. V [libjvm.so+0x88d623] PSVirtualSpace::expand_by(unsigned
long)+0x53
12. V [libjvm.so+0x88e9f8] PSYoungGen::resize_generation(unsigned
long, unsigned long)+0xf8
13. V [libjvm.so+0x88db62] PSYoungGen::resize(unsigned long,
unsigned long)+0x22
14. V [libjvm.so+0x88af1d] PSScavenge::invoke_no_policy()+0xf3d
15. V [libjvm.so+0x88b761] PSScavenge::invoke()+0x41
16. V [libjvm.so+0x843f40] ParallelScavengeHeap::failed_mem_allocate(unsigned
long)+0x70
17. V [libjvm.so+0x9a4a97] VM_ParallelGCFailedAllocation
::doit()+0x97
18. V [libjvm.so+0x9abf35] VM_Operation::evaluate()+0x55
19. V [libjvm.so+0x9aa2fa] VMThread::evaluate_operation(
VM_Operation*)+0xba
20. V [libjvm.so+0x9aa67e] VMThread::loop()+0x1ce
21. V [libjvm.so+0x9aaaf0] VMThread::run()+0x70
22. V [libjvm.so+0x8238c8] java_start(Thread*)+0x108
I do not have much of knowledge in to these configuration, need you guys
help.
Is this because of my server goes OutOfMemory, if yes then why it doesnt
show the memory occupied more in VisualVM when we check heap size
On Thu, Jan 11, 2018 at 3:00 PM, Silvio Bierman <

Post by Greg Wilkins
Kapil,
If you are running on Linux check the system log for kernel panics due
to lack of memory. Linux will ruthlessly kill the process with highest
memory load in such a situation. That has happened to me plenty of times
and also left me wondering where my JVM process had gone.
And to add to what Greg said: the System.exit thing can also happen in
a library. I used a library once that would do a System.exit in some error
situations killing my server now and then. That was a real puzzler.
Cheers,
Silvio
Greg,
1. No process of jetty server running
2. MySQL still runs
3. CPU usage shows data fine
4. RAM is also fine
Not sure how can we isolate if issue could be because of JVM, can you
please guide on setting some parameters or moniotring them.
Thanks

Post by kapil gupta
We are using jetty server and when it is on load then the server is
crashed without showing much of information.
We have also put it on JMX. The CPU usage and memory looks fine, but
still Jetty server crashes. Please let me know what parameters we should
look for and how can we isolate issue which is causing jetty server crash.
_______________________________________________
jetty-users mailing list
To change your delivery options, retrieve your password, or
unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users

--
_______________________________________________
jetty-users mailing list
To change your delivery options, retrieve your password, or
unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users