[jetty-users] "early EOF" warning when https call from old application in 9.2.9.v2015022er

[lamphus]

Hello,

Am running an older application with dependencies on jetty 8.1.16v20140903
(jars: jetty-client, jetty-continuation, jetty-http, jetty-io,
jetty-security, jetty-server, jetty-servlet, jetty-util, jetty-xml). It
works with 8.1.21.v20160908 jars too. It runs inside a jetty 9.2.9.v20150224
without issues using only http.

When using only https however the application with the jetty 8 deps tries to
httprequest another application in the same container (jetty9) and the issue
reported is :

[java] 11031 [HttpClient-31] WARN
org.eclipse.jetty.client.HttpExchange -
EXCEPTION ***@34ff2b7c=GET//domain:443/application2/rest#WAITING
(1ms)->EXCEPTED(1ms)sent=4ms
[java] org.eclipse.jetty.io.EofException: early EOF
[java] at
org.eclipse.jetty.client.AbstractHttpConnection$Handler.early
EOF(AbstractHttpConnection.java:352)
[java] at
org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:3
23)
[java] at
org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.j
ava:235)
[java] at
org.eclipse.jetty.client.AsyncHttpConnection.handle(AsyncHttp
Connection.java:135)
[java] at
org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.j
ava:196)
[java] at
org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectC
hannelEndPoint.java:696)
[java] at
org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectCh
annelEndPoint.java:53)
[java] at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedT
hreadPool.java:608)
[java] at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedTh
readPool.java:543)
[java] at java.lang.Thread.run(Thread.java:745)

and it yields a functional issue.

I've seen something close to it here:
https://bugs.eclipse.org/bugs/show_bug.cgi?id=376273 but that issue should
have been vanquished in 8.10+ .

The older application does not start with 9.2.9.v20150224 jars.

Am not in a position to change the older applications dependencies and am
obviously a newbie with jetty, is there reason to believe the application
needs to be updated with newer dependencies to work well in jetty9 or that
this error would not occur if all applications were living inside a jetty 8
(using https)?


Thanks for any hints.



--
View this message in context: http://jetty.4.x6.nabble.com/early-EOF-warning-when-https-call-from-old-application-in-9-2-9-v2015022er-tp4966218.html
Sent from the Jetty User mailing list archive at Nabble.com.

[Greg Wilkins]

lamphus,

you have a situation that is rather: a) special; b) complex; c) dated

It it going to be hard to help via open source channels, but we will
attempt.

You have an application using a jetty-8.1.16 client to talk to a
jetty-9.2.9 server and logs an EarlyEoF exception on the server and a
"functional issue" in the application.

EarlyEOF is just an indication that the server was parsing a request and
the EOF arrived. Typically this is because the client gave up - either it
timed out or some other problem happened.

Does this happen on all requests, or just some requests.

Do you know the time the request was issued vs the time the "functional
issue" occurs and the earlyEOF logged? Is there a delay.

Can you run the client stand alone talking to the server rather than
talking to itself?

regards

[David Persson]

Hi, yep, this was understood, and I knew I ran the risk of 0 replies, so
I'm glad for yours : )

Thanks for the info regarding EOF.

This seems to happen for a specific request. The application instantiates a
http://www.eclipse.org/jetty/documentation/current/http-client.html and
issues a get request with an url argument that goes "
https://domain.topdomain/path" but what is sent according to the jetty log
appears to be "http://domain.topdomain:443" (I think- log says
"GET//domain.topdomain:443/geoserver/rest" and the way our jetty9 is
configured one can't reach it like this. It has to be
http://domain.topdomain/path (for a 302) or https://domain.topdomain/path
or https://domain.topdomain:443 . Am currently trying to instantiate an
SSLContextFactory for the httpclient constructor but am so far not seeing
an effect. It's obviously less "I know what I'm doing" and more "poke this
and see if that part moves" but I am picking up a few things..

Post by Greg Wilkins
lamphus,
you have a situation that is rather: a) special; b) complex; c) dated
It it going to be hard to help via open source channels, but we will
attempt.
You have an application using a jetty-8.1.16 client to talk to a
jetty-9.2.9 server and logs an EarlyEoF exception on the server and a
"functional issue" in the application.
EarlyEOF is just an indication that the server was parsing a request and
the EOF arrived. Typically this is because the client gave up - either it
timed out or some other problem happened.
Does this happen on all requests, or just some requests.
Do you know the time the request was issued vs the time the "functional
issue" occurs and the earlyEOF logged? Is there a delay.
Can you run the client stand alone talking to the server rather than
talking to itself?
regards
--
_______________________________________________
jetty-users mailing list
To change your delivery options, retrieve your password, or unsubscribe
from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users

[Simone Bordet]

Hi,

Hi, yep, this was understood, and I knew I ran the risk of 0 replies, so I'm
glad for yours : )
Thanks for the info regarding EOF.
This seems to happen for a specific request. The application instantiates a
http://www.eclipse.org/jetty/documentation/current/http-client.html and
issues a get request with an url argument that goes
"https://domain.topdomain/path" but what is sent according to the jetty log
appears to be "http://domain.topdomain:443" (I think- log says
"GET//domain.topdomain:443/geoserver/rest" and the way our jetty9 is
configured one can't reach it like this.

Mmm, that would be a gross bug that would have been reported ages ago.
Perhaps it's a logging bug, rather than an actual network protocol bug.

It has to be
http://domain.topdomain/path (for a 302) or https://domain.topdomain/path or
https://domain.topdomain:443 . Am currently trying to instantiate an
SSLContextFactory for the httpclient constructor but am so far not seeing an
effect. It's obviously less "I know what I'm doing" and more "poke this and
see if that part moves" but I am picking up a few things..

You should look at the server logs and try to understand why the
server rejects that request,
as the client has sent it (it is waiting for the response, but gets a
connection closed instead).

Especially, take a close look at what JDK you run on client and
server, what ciphers they have in common, what version of the TLS
protocol you have configured, etc.
My guess is that you have some TLS incompatibility between client and server.

See also: http://docs.oracle.com/javase/7/docs/technotes/guides/security/jsse/ReadDebug.html

[David Persson]

Hi, thanks for your reply.

The EOF call that yields a functional issue for the end user appears to be
made from the 'server side' of this application to its neighbour
application in the same Jetty9(and as such the same jre, 8), Geoserver. I
don't see it when looking at traffic between an end user and the
application. The application uses the jars mentioned in the original email,
among others (8.1.16v20140903) jetty-client in
https://github.com/boundlessgeo/GeoExplorer/blob/master/app/httpclient.js
which in turn is used by
https://github.com/boundlessgeo/GeoExplorer/blob/master/app/auth.js :

The troubled part:

exports.getStatus = function(request) {
var url = getAuthUrl(request);
var status = 401;
var headers = new Headers(request.headers);
var token = headers.get("Cookie");
var exchange = clientRequest({
url: url,
method: "GET",
async: false,
headers: headers
});
exchange.wait();
return exchange.status; //Status is 0, exchange.content is null for
https url
};


where url == https://domain/geoserver/rest but the logs report it as
GET//domain:443/geoserver/rest and if that is
http://domain:443/geoserver/rest then our jetty9 can't handle that request,
it gets aborted whether you try it from the application server or a client
machine. It may be that we should configure jetty9 to understand and
redirect http://domain:443 but I don't think just that would solve the
issue, it is probably a bit of a tls configuration issue as you say since
the official doc
http://www.eclipse.org/jetty/documentation/current/http-client.html says "

When you create a HttpClient instance using the parameterless constructor,
you will only be able to perform plain HTTP requests and you will not be
able to perform HTTPS requests.

In order to perform HTTPS requests, you should create first a
SslContextFactory, configure it, and pass it to the HttpClient constructor.
When created with a SslContextFactory, the HttpClient will be able to
perform both HTTP and HTTPS requests to any domain."

so I tried changing httpclient.js to import the util.ssl package and then
create the httpclient like so:

var sslcontxtfctry = new SslContextFactory(true);
var client = new HttpClient(sslcontxtfctry);

but this makes no perceivable difference. The GET is still to :443 and it
still EOFs.


It is entirely possible that this issue is outside of the scope of this
forum, if so I won't pursue it further.

If there's any further assistance to be had I shall of course be happy to
receive it : )

Regards

Post by David Persson
Hi,

Post by David Persson
Hi, yep, this was understood, and I knew I ran the risk of 0 replies, so

I'm

Post by David Persson
glad for yours : )
Thanks for the info regarding EOF.
This seems to happen for a specific request. The application

instantiates a

Post by David Persson
http://www.eclipse.org/jetty/documentation/current/http-client.html and
issues a get request with an url argument that goes
"https://domain.topdomain/path" but what is sent according to the jetty

log

Post by David Persson
appears to be "http://domain.topdomain:443" (I think- log says
"GET//domain.topdomain:443/geoserver/rest" and the way our jetty9 is
configured one can't reach it like this.

Mmm, that would be a gross bug that would have been reported ages ago.
Perhaps it's a logging bug, rather than an actual network protocol bug.

Post by David Persson
It has to be
http://domain.topdomain/path (for a 302) or

https://domain.topdomain/path or

Post by David Persson
https://domain.topdomain:443 . Am currently trying to instantiate an
SSLContextFactory for the httpclient constructor but am so far not

seeing an

Post by David Persson
effect. It's obviously less "I know what I'm doing" and more "poke this

and

Post by David Persson
see if that part moves" but I am picking up a few things..

You should look at the server logs and try to understand why the
server rejects that request,
as the client has sent it (it is waiting for the response, but gets a
connection closed instead).
Especially, take a close look at what JDK you run on client and
server, what ciphers they have in common, what version of the TLS
protocol you have configured, etc.
My guess is that you have some TLS incompatibility between client and server.
See also: http://docs.oracle.com/javase/7/docs/technotes/guides/
security/jsse/ReadDebug.html
--
Simone Bordet
----
http://cometd.org
http://webtide.com
Developer advice, training, services and support
from the Jetty & CometD experts.
_______________________________________________
jetty-users mailing list
To change your delivery options, retrieve your password, or unsubscribe
from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users

[Simone Bordet]

Hi,

Post by David Persson
Hi, thanks for your reply.
The EOF call that yields a functional issue for the end user appears to be
made from the 'server side' of this application to its neighbour application
in the same Jetty9(and as such the same jre, 8), Geoserver.

Okay. So have you enabled javax.net.debug=all ?
DEBUG level logs on the server ?
Why the server closes the connection ?

[David Persson]

It isn't clear to me where to enable javax.net.debug=all.

The server jetty9 does not have slf4jlog or javautillog defined, the level
is at debug when I open prunmgr//ES//nameofservice, the stack trace from
this server log is in the first email.

I don't even know whether "GET//domain:443/geoserver/rest" means an http
prefix or an https prefix but I suppose http since our Jetty9 says
connection reset to those.

Will probably need to hire external help here but thanks for your time.

Post by lamphus
Hi,

Post by David Persson
Hi, thanks for your reply.
The EOF call that yields a functional issue for the end user appears to

be

Post by David Persson
made from the 'server side' of this application to its neighbour

application

Post by David Persson
in the same Jetty9(and as such the same jre, 8), Geoserver.

Okay. So have you enabled javax.net.debug=all ?
DEBUG level logs on the server ?
Why the server closes the connection ?
--
Simone Bordet
----
http://cometd.org
http://webtide.com
Developer advice, training, services and support
from the Jetty & CometD experts.
_______________________________________________
jetty-users mailing list
To change your delivery options, retrieve your password, or unsubscribe
from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users

[Brian Reichert]

Post by David Persson
It isn't clear to me where to enable javax.net.debug=all.

Several ways, if you're just experiementing:

https://www.eclipse.org/jetty/documentation/9.4.x/start-jar.html

- You can add 'javax.net.debug=all' to your start.ini file.
- I think you can add '-Djavax.net.debug=all' on the command line.

Post by David Persson
I don't even know whether "GET//domain:443/geoserver/rest" means an http
prefix or an https prefix but I suppose http since our Jetty9 says
connection reset to those.

Well, port 443 implies HTTPS, but I haven't followed this whole
thread, so I don't know the context.

[Simone Bordet]

Hi,

Post by David Persson
It isn't clear to me where to enable javax.net.debug=all.

On the server that closes the connection, and on the client that
receives the connection closed.

Post by David Persson
The server jetty9 does not have slf4jlog or javautillog defined, the level
is at debug when I open prunmgr//ES//nameofservice, the stack trace from
this server log is in the first email.
I don't even know whether "GET//domain:443/geoserver/rest" means an http
prefix or an https prefix but I suppose http since our Jetty9 says
connection reset to those.

As I said, this is very unlikely.
The issue is probably something different.

Post by David Persson
Will probably need to hire external help here but thanks for your time.

You can hire us :)
https://webtide.com/

[David Persson]

Thanks both. (And good to know Simone )

Jetty9 server: Bits of my start.ini (installed as windows service) are
being ignored, like send server version, and now javax.net.debug=all. In
prunmgr however adding -Djavax.net.debug=all made the logging very active
indeed.

For every :443/../rest call this is repeated:


qtp999661724-87, fatal error: 10: General SSLEngine problem
javax.net.ssl.SSLHandshakeException: SSLv2Hello is disabled
qtp999661724-87, SEND TLSv1.2 ALERT: fatal, description =
unexpected_message
qtp999661724-87, WRITE: TLSv1.2 Alert, length = 2
qtp999661724-87, fatal: engine already closed. Rethrowing
javax.net.ssl.SSLHandshakeException: SSLv2Hello is disabled
HttpClient-332, called closeInbound()
HttpClient-332, fatal error: 80: Inbound closed before receiving peer's
close_notify: possible truncation attack?
javax.net.ssl.SSLException: Inbound closed before receiving peer's
close_notify: possible truncation attack?
HttpClient-332, SEND TLSv1.2 ALERT: fatal, description = internal_error
HttpClient-332, WRITE: TLSv1.2 Alert, length = 2
[Raw write]: length = 7
0000: 15 03 03 00 02 02 50 ......P
HttpClient-332, called closeInbound()
HttpClient-332, closeInboundInternal()
HttpClient-332, called closeInbound()
HttpClient-332, closeInboundInternal()
2017-02-01 17:30:30.515:WARN:oejc.HttpExchange:HttpClient-332: EXCEPTION
***@6787ac61
=GET//domain:443/geoserver/rest#WAITING(0ms)->EXCEPTED(0ms)sent=0ms
org.eclipse.jetty.io.EofException: early EOF

Right now TLS 1.0, 1.1 and 1.2 are accepted, no SSL version.

Regards,
David

Post by Simone Bordet
Hi,

Post by David Persson
It isn't clear to me where to enable javax.net.debug=all.

On the server that closes the connection, and on the client that
receives the connection closed.

Post by David Persson
The server jetty9 does not have slf4jlog or javautillog defined, the

level

Post by David Persson
is at debug when I open prunmgr//ES//nameofservice, the stack trace from
this server log is in the first email.
I don't even know whether "GET//domain:443/geoserver/rest" means an http
prefix or an https prefix but I suppose http since our Jetty9 says
connection reset to those.

As I said, this is very unlikely.
The issue is probably something different.

Post by David Persson
Will probably need to hire external help here but thanks for your time.

You can hire us :)
https://webtide.com/
--
Simone Bordet
----
http://cometd.org
http://webtide.com
Developer advice, training, services and support
from the Jetty & CometD experts.
_______________________________________________
jetty-users mailing list
To change your delivery options, retrieve your password, or unsubscribe
from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users