[jetty-users] Jetty EatWhatYouKill - Deadlock or something!

Discussion:

Sarath Prabath Redlapalli Jaya

2018-01-05 17:50:53 UTC

Hi,

We've been running a java server with Jetty(Dropwizard) and the load of
system per node would be ~ 10k QPS as of now.

We ran Load Test, Connection Test and Peak Test(No Stress test yet.), but
couldn't reproduce what's happening on production when we chose to go live

Below is the error that spits out continuosly

[2018-01-05 16:05:00,874] [] [WARN] MSG - o.e.j.u.t.QueuedThreadPool
***@dw{STARTED,4000<=4000<=4000,i=0,q=10000}
rejected CEP:***@62241285{/127.0.0.1:49625<->/127.
0.0.1:7000,OPEN,fill=FI,flush=-,to=0/30000}{io=1/0,kio=1,
kro=1}->***@23c692fe[p=HttpParser{s=START,0 of
0},g=***@544fccad{s=START}]=>***@2
c630bfe{r=0,c=false,a=IDLE,uri=null}:runFillable:BLOCKING
[2018-01-05 16:05:00,874] [] [WARN] MSG - o.e.j.u.t.s.EatWhatYouKill
java.util.concurrent.RejectedExecutionException:
CEP:***@62241285{/127.0.0.1:49625<->/127.0.0.1:7000
,OPEN,fill=FI,flush=-,to=0/30000}{io=1/0,kio=1,
kro=1}->***@23c692fe[p=HttpParser{s=START,0 of
0},g=***@544fccad{s=START}]=>***@2
c630bfe{r=0,c=false,a=IDLE,uri=null}:runFillable:BLOCKING
at org.eclipse.jetty.util.thread.QueuedThreadPool.execute(Queue
dThreadPool.java:385)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProd
uce(EatWhatYouKill.java:249)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produc
e(EatWhatYouKill.java:140)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(Ea
tWhatYouKill.java:131)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$Reserve
dThread.run(ReservedThreadExecutor.java:243)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(Queued
ThreadPool.java:679)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedT
hreadPool.java:597)
at java.lang.Thread.run(Thread.java:745)

Tried to find an answer, but couldn't. Can anyone help. At this point the
server stops accepting new connections and doesn't serve new http requests.

But Jetty QueuedThreadPool Metrics show that thread Utilization is 100%

Thanks

Simone Bordet

2018-01-05 18:47:41 UTC

Permalink

Hi,

On Fri, Jan 5, 2018 at 6:50 PM, Sarath Prabath Redlapalli Jaya

Post by Sarath Prabath Redlapalli Jaya
Hi,
We've been running a java server with Jetty(Dropwizard) and the load of
system per node would be ~ 10k QPS as of now.
We ran Load Test, Connection Test and Peak Test(No Stress test yet.), but
couldn't reproduce what's happening on production when we chose to go live
Below is the error that spits out continuosly
[2018-01-05 16:05:00,874] [] [WARN] MSG - o.e.j.u.t.QueuedThreadPool
rejected
of
[2018-01-05 16:05:00,874] [] [WARN] MSG - o.e.j.u.t.s.EatWhatYouKill
of
at
org.eclipse.jetty.util.thread.QueuedThreadPool.execute(QueuedThreadPool.java:385)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:249)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:243)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:679)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:597)
at java.lang.Thread.run(Thread.java:745)
Tried to find an answer, but couldn't. Can anyone help. At this point the
server stops accepting new connections and doesn't serve new http requests.
But Jetty QueuedThreadPool Metrics show that thread Utilization is 100%

What Jetty version ?
Have you tried 9.4.8 ?

Are you using a bounded thread pool ?

If it replicates with 9.4.8, can you take a server dump as explained here:
https://www.eclipse.org/jetty/documentation/9.4.x/jetty-dump-tool.html

--
Simone Bordet
----
http://cometd.org
http://webtide.com
Developer advice, training, services and support
from the Jetty & CometD experts.

Sarath Prabath Redlapalli Jaya

2018-01-06 05:18:48 UTC

Permalink

Hi,

I'm using Dropwizard 1.2.0 -> Jetty 9.4.7.v20170914.

Dropwizard has a maxThreads setting for jetty. Following is the config i
used

maxThreads: 4000
minThreads: 4000

Connectors:
acceptorThreads: 500
selectorThreads: 100. Although i believe these counts are high, i was just
tweaking them.

Also, i'll try to take the dump

Thanks

Post by Simone Bordet
Hi,
On Fri, Jan 5, 2018 at 6:50 PM, Sarath Prabath Redlapalli Jaya

live

Post by Sarath Prabath Redlapalli Jaya
Below is the error that spits out continuosly
[2018-01-05 16:05:00,874] [] [WARN] MSG - o.e.j.u.t.QueuedThreadPool

4000<=4000<=4000,i=0,q=10000}

Post by Sarath Prabath Redlapalli Jaya
rejected

23c692fe[p=HttpParser{s=START,0

Post by Sarath Prabath Redlapalli Jaya
of

2c630bfe{r=0,c=false,a=IDLE,uri=null}:runFillable:BLOCKING

Post by Sarath Prabath Redlapalli Jaya
[2018-01-05 16:05:00,874] [] [WARN] MSG - o.e.j.u.t.s.EatWhatYouKill

23c692fe[p=HttpParser{s=START,0

Post by Sarath Prabath Redlapalli Jaya
of

2c630bfe{r=0,c=false,a=IDLE,uri=null}:runFillable:BLOCKING

Post by Sarath Prabath Redlapalli Jaya
at
org.eclipse.jetty.util.thread.QueuedThreadPool.execute(

QueuedThreadPool.java:385)

Post by Sarath Prabath Redlapalli Jaya
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.

doProduce(EatWhatYouKill.java:249)

Post by Sarath Prabath Redlapalli Jaya
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.

produce(EatWhatYouKill.java:140)

Post by Sarath Prabath Redlapalli Jaya
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(

EatWhatYouKill.java:131)

Post by Sarath Prabath Redlapalli Jaya
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(

ReservedThreadExecutor.java:243)

Post by Sarath Prabath Redlapalli Jaya
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(

QueuedThreadPool.java:679)

Post by Sarath Prabath Redlapalli Jaya
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(

QueuedThreadPool.java:597)

Post by Sarath Prabath Redlapalli Jaya
at java.lang.Thread.run(Thread.java:745)
Tried to find an answer, but couldn't. Can anyone help. At this point the
server stops accepting new connections and doesn't serve new http

requests.

Post by Sarath Prabath Redlapalli Jaya
But Jetty QueuedThreadPool Metrics show that thread Utilization is 100%

What Jetty version ?
Have you tried 9.4.8 ?
Are you using a bounded thread pool ?
https://www.eclipse.org/jetty/documentation/9.4.x/jetty-dump-tool.html
--
Simone Bordet
----
http://cometd.org
http://webtide.com
Developer advice, training, services and support
from the Jetty & CometD experts.
_______________________________________________
jetty-users mailing list
To change your delivery options, retrieve your password, or unsubscribe
from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users

Simone Bordet

2018-01-06 16:12:38 UTC

Permalink

Hi,

On Sat, Jan 6, 2018 at 6:18 AM, Sarath Prabath Redlapalli Jaya

Post by Sarath Prabath Redlapalli Jaya
Hi,
I'm using Dropwizard 1.2.0 -> Jetty 9.4.7.v20170914.
Dropwizard has a maxThreads setting for jetty. Following is the config i
used
maxThreads: 4000
minThreads: 4000
acceptorThreads: 500

Wow, no.
You don't need 500 acceptor threads. Typically 1 is more than enough.

Post by Sarath Prabath Redlapalli Jaya
selectorThreads: 100. Although i believe these counts are high, i was just
tweaking them.

I am almost sure that you or Dropwizard set a limit to the number of
tasks of the thread pool queue.
I would double check that, and possibly leave it unbounded.

Post by Sarath Prabath Redlapalli Jaya
Also, i'll try to take the dump

Yes, that is typically key to understand what's going on.

If you can, set QueuedThreadPool.setDetailedDump=true before dumping.

--
Simone Bordet
----
http://cometd.org
http://webtide.com
Developer advice, training, services and support
from the Jetty & CometD experts.

Sarath Prabath Redlapalli Jaya

2018-01-08 22:41:32 UTC

Permalink

Hi Simone,

I took the thread dump, and after initial observation, it was some rogue
heavy blocking I/O call that's keeping the thread busy.

Jetty's perfect :). Thanks for the help

Thanks

Post by Simone Bordet
Hi,
On Sat, Jan 6, 2018 at 6:18 AM, Sarath Prabath Redlapalli Jaya

Wow, no.
You don't need 500 acceptor threads. Typically 1 is more than enough.

Post by Sarath Prabath Redlapalli Jaya
selectorThreads: 100. Although i believe these counts are high, i was

just

Post by Sarath Prabath Redlapalli Jaya
tweaking them.

I am almost sure that you or Dropwizard set a limit to the number of
tasks of the thread pool queue.
I would double check that, and possibly leave it unbounded.

Post by Sarath Prabath Redlapalli Jaya
Also, i'll try to take the dump

Yes, that is typically key to understand what's going on.
If you can, set QueuedThreadPool.setDetailedDump=true before dumping.
--
Simone Bordet
----
http://cometd.org
http://webtide.com
Developer advice, training, services and support
from the Jetty & CometD experts.
_______________________________________________
jetty-users mailing list
To change your delivery options, retrieve your password, or unsubscribe
from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users