Thanks Joakim for a detailed response.
3.5-3.8 GB out 5 GB is written to disk and then it gets stuck. I will
verify your other suggestions. But the problem is the issue does not come
always. It is intermittent.
Post by Joakim ErdfeltThe relevant stacktrace ...
"dw-150 - POST /management/upgrade/uploadbundle" #150 prio=5 os_prio=0
tid=0x00007ffff0f7e800 nid=0x3a46 in Object.wait() [0x00007ffee0728000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at org.eclipse.jetty.server.HttpInput.blockForContent(HttpInput.java:565)
at
org.eclipse.jetty.server.HttpInput$1.blockForContent(HttpInput.java:1084)
at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:306)
- locked <0x0000000677a5d220> (a java.util.ArrayDeque)
at
org.apache.commons.fileupload.MultipartStream$ItemInputStream.makeAvailable(MultipartStream.java:999)
at
org.apache.commons.fileupload.MultipartStream$ItemInputStream.read(MultipartStream.java:903)
at java.io.InputStream.read(InputStream.java:101)
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1488)
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1465)
at
com.vnera.restapilayer.ManagementResource.flushFileToDisk(ManagementResource.java:2515)
at
com.vnera.restapilayer.ManagementResource.storeFileToDisk(ManagementResource.java:2441)
at
com.vnera.restapilayer.ManagementResource.uploadBundle(ManagementResource.java:2366)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
at
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144)
at
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161)
at
org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205)
at
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99)
at
org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389)
at
org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347)
at
org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102)
at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
at
org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)
at
org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305)
at
org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154)
at
org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473)
at
org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427)
at
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388)
at
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341)
at
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228)
at
io.dropwizard.jetty.NonblockingServletHolder.handle(NonblockingServletHolder.java:49)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
at
io.dropwizard.servlets.ThreadNameFilter.doFilter(ThreadNameFilter.java:34)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
at
io.dropwizard.jersey.filter.AllowedMethodsFilter.handle(AllowedMethodsFilter.java:45)
at
io.dropwizard.jersey.filter.AllowedMethodsFilter.doFilter(AllowedMethodsFilter.java:39)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
at
org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:308)
at
org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:262)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
at
org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)
at
org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
at
org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
at
org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
at
org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
at
org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
at
org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at
com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:241)
at io.dropwizard.jetty.RoutingHandler.handle(RoutingHandler.java:52)
at
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:455)
at io.dropwizard.jetty.BiDiGzipHandler.handle(BiDiGzipHandler.java:69)
at
org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:56)
at
org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:169)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.Server.handle(Server.java:530)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:347)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:256)
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:247)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:382)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)
Looks like you are using dropwizard + jersey + com.vnera.restapilayer +
commons-fileupload to handle your file upload.
Your setup isn't using Jetty for multipart/form-data upload in the way
you think.
First, don't use commons-fileupload, that's for Servlet 2.4 (and older)
environments.
Support for multipart/form-data parsing is built into the Servlet spec as
HttpServletRequest.getParts() and getPart(name) since Servlet 3.0
https://docs.oracle.com/javaee/7/api/javax/servlet/http/HttpServletRequest.html#getPart-java.lang.String-
<https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.oracle.com_javaee_7_api_javax_servlet_http_HttpServletRequest.html-23getPart-2Djava.lang.String-2D&d=DwMFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=P3_1pTtMQK06fFymYIWbyyzVU6nc0CcwfuZhLhexammvaiCaU0ieHeI7BWvfbbjE&m=v4n059clZKEoxfRB9GdzFxy5fdVagerkzRukYEqXuGY&s=4cpaDgQbjItJZ6eUBOhoEvyZ5xcl-UEJyicVNSwKaQs&e=>
If your code, or ANY of its Filters or 3rd party libraries uses any one
of the HttpServletRequest.getParameter*() methods before your attempt at
using commons-fileupload then your code will not work and fail.
Why? That's because the Servlet spec says we have to read the Request
URI query and Request form data (Input Stream) to generate the the
parameter list and Parts list (if request is a multipart/form)
Right now, based on your stacktrace, Jetty is only involved in giving you
a HttpServletRequest InputStream which is blocked waiting for more content
from the client.
This could be a bug in commons-fileupload (a common scenario)
A bug in your use of the servlet spec (based on the number of layers you
have going on, this is a high probability, try putting a breakpoint on
Jetty's org.eclipse.jetty.server.Request.getParameters() method to see if
the above example is going on)
Or a bug in your http client (a common scenario where the client says
`Content-Length: <big-number>` but never sends the complete <big-number> of
bytes, hence the blocking for content)
You can test Jetty's behavior in isolation easily enough.
$ mkdir jetty-downloads
$ cd jetty-downloads
$ curl -O
http://central.maven.org/maven2/org/eclipse/jetty/jetty-distribution/9.4.12.v20180830/jetty-distribution-9.4.12.v20180830.tar.gz
<https://urldefense.proofpoint.com/v2/url?u=http-3A__central.maven.org_maven2_org_eclipse_jetty_jetty-2Ddistribution_9.4.12.v20180830_jetty-2Ddistribution-2D9.4.12.v20180830.tar.gz&d=DwMFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=P3_1pTtMQK06fFymYIWbyyzVU6nc0CcwfuZhLhexammvaiCaU0ieHeI7BWvfbbjE&m=v4n059clZKEoxfRB9GdzFxy5fdVagerkzRukYEqXuGY&s=_xubfGKK_hd_3C7SVWVrgcXmXZXXNcRILUISoABgvng&e=>
$ tar -zxf tar -zxvf jetty-distribution-9.4.12.v20180830.tar.gz
$ cd jetty-distribution-9.4.12.v20180830/demo-base/
$ java -jar ../start.jar
now open a web browser to http://localhost:8080/test/dump/info
<https://urldefense.proofpoint.com/v2/url?u=http-3A__localhost-3A8080_test_dump_info&d=DwMFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=P3_1pTtMQK06fFymYIWbyyzVU6nc0CcwfuZhLhexammvaiCaU0ieHeI7BWvfbbjE&m=v4n059clZKEoxfRB9GdzFxy5fdVagerkzRukYEqXuGY&s=djwkat47OBQtJYqNahL2TBoMQ0DKwuigizKTa_M-ktI&e=>
scroll down to "Form to generate UPLOAD content"
use "File 1" or "File 2" on that form to select your 5gb data file.
then "submit"
On chrome you'll see an "Uploading" percentage at the bottom of the window.
when it's done uploading, you'll get a "Dump Servlet" response.
scroll down to "Parts:" (about 1/3rd down the page)
you'll see the results of HttpSerletRequest.getParts() here.
Action: Part{n=Action,fn=null,ct=null,s=6,tmp=true,file=null}
Part{n=file2,fn=,ct=application/octet-stream,s=0,tmp=true,file=null}
TextField: Part{n=TextField,fn=null,ct=null,s=7,tmp=true,file=null}
file1: Part{n=file1,fn=bigger.iso,ct=application/octet-stream,s=
*6806166040*
,tmp=true,file=/private/var/folders/gm/676mvm6j6131h37d0msqk1k40000gn/T/jetty-0.0.0.0-8080-test.war-_test-any-432009796042963862.dir/upload/MultiPart7642561650112679943}
That tells me that 6806166040 bytes were sent as part of "file1".
If I look at the original (pre-uploaded) file, I have ...
$ ls -la bigger.iso
-rw-r--r-- 1 joakim staff 6806166040 Oct 13 07:09 bigger.iso
I can see that the upload completed, the entire 6GB file was sent.
Post by Debraj MannaPOST /management/upgrade/uploadbundle is the api in context
Post by Debraj MannaI am using a jetty-server behind nginx reverse proxy. On trying to
upload a big file about 5 GB via Multi-part I am seeing sometimes the
upload is getting stuck. Please find attached the full thread dump when the
jetty server was stuck.
I have also placed the thread dump in the below gist
<https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.github.com_debraj-2Dmanna_20b62dd3cbc9e7badb05fdb01e513fcd&d=DwMFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=P3_1pTtMQK06fFymYIWbyyzVU6nc0CcwfuZhLhexammvaiCaU0ieHeI7BWvfbbjE&m=v4n059clZKEoxfRB9GdzFxy5fdVagerkzRukYEqXuGY&s=hv6TjDnnkkwY-eeWVqCmgwMyVrNGLjbU6jABfarQVp4&e=>
https://gist.github.com/debraj-manna/20b62dd3cbc9e7badb05fdb01e513fcd
<https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.github.com_debraj-2Dmanna_20b62dd3cbc9e7badb05fdb01e513fcd&d=DwMFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=P3_1pTtMQK06fFymYIWbyyzVU6nc0CcwfuZhLhexammvaiCaU0ieHeI7BWvfbbjE&m=v4n059clZKEoxfRB9GdzFxy5fdVagerkzRukYEqXuGY&s=hv6TjDnnkkwY-eeWVqCmgwMyVrNGLjbU6jABfarQVp4&e=>
Can someone let me know if this is some known issue in Jetty?
Environment
Ubuntu 14.04
Java 8
Jetty Server - 9.4.10, 9.4.8.v20171121
_______________________________________________
jetty-users mailing list
To change your delivery options, retrieve your password, or unsubscribe
from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users
<https://urldefense.proofpoint.com/v2/url?u=https-3A__dev.eclipse.org_mailman_listinfo_jetty-2Dusers&d=DwMFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=P3_1pTtMQK06fFymYIWbyyzVU6nc0CcwfuZhLhexammvaiCaU0ieHeI7BWvfbbjE&m=v4n059clZKEoxfRB9GdzFxy5fdVagerkzRukYEqXuGY&s=Hpubhy2jdJWf-a4vF37_8cj4qCx0buzb-3pjfuFprfQ&e=>
_______________________________________________
jetty-users mailing list
To change your delivery options, retrieve your password, or unsubscribe
from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users
<https://urldefense.proofpoint.com/v2/url?u=https-3A__dev.eclipse.org_mailman_listinfo_jetty-2Dusers&d=DwMFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=P3_1pTtMQK06fFymYIWbyyzVU6nc0CcwfuZhLhexammvaiCaU0ieHeI7BWvfbbjE&m=v4n059clZKEoxfRB9GdzFxy5fdVagerkzRukYEqXuGY&s=Hpubhy2jdJWf-a4vF37_8cj4qCx0buzb-3pjfuFprfQ&e=>
https://urldefense.proofpoint.com/v2/url?u=https-3A__dev.eclipse.org_mailman_listinfo_jetty-2Dusers&d=DwICAg&c=q3cDpHe1hF8lXU5EFjNM_A&r=P3_1pTtMQK06fFymYIWbyyzVU6nc0CcwfuZhLhexammvaiCaU0ieHeI7BWvfbbjE&m=v4n059clZKEoxfRB9GdzFxy5fdVagerkzRukYEqXuGY&s=Hpubhy2jdJWf-a4vF37_8cj4qCx0buzb-3pjfuFprfQ&e=
