Discussion:
[jetty-users] Virtual host proxy problems
John English
2017-12-26 17:06:03 UTC
Permalink
Hi folks,

I have a Jetty (9.4.6) SSL server which uses virtual hosts. The server
listens on port 8443, and I use iptables to redirect incoming
connections on 443 to the server at 8443. Port 443 is the only one open
to the outside world.

There are two webapps: one is my primary webapp, which responds to
127.0.0.1, 192.168.1.200 (the first network adapter's internal IP) and
my primary domain name (say, www.foo.com). The other is a very simple
proxy servlet which responds to requests directed to 127.0.0.2,
192.168.1.201 (the second network adapter's IP), and my secondary domain
name (say, www.bar.com).

The proxy servlet is a simple derivation of ProxyServlet, which
redirects all incoming requests to a secondary server listening on port
8888 by overriding rewriteTarget() like this:

protected String rewriteTarget (HttpServletRequest request) {
String url = request.getRequestURL().toString();
String fwd = url.replaceFirst("https://(.*?)(:\\d+)/",
"http://127.0.0.1:8888/");
return fwd;
}

Thus, any incoming HTTPS request get converted to an equivalent request
to http://127.0.0.1:8888/.

What actually happens is as follows:

1) Running a browser on the server machine: Requests to
https://127.0.0.1:8443, https://192.168.1.200:8443 both go to the
primary webapp as expected. Requests to https://127.0.0.2:8443,
https://192.168.1.201:8443 both go to the secondary server on port 8888
as expected.

2) From another machine on the local network: Requests to
https://192.168.1.200:8443 go to the primary webapp as expected.
Requests to https://192.168.1.201:8443 go to the secondary server on
port 8888 as expected.

3) Requests to https://www.foo.com go to the primary webapp as expected.
Requests to https://www.bar.com result in an empty 502 response ("Bad
Gateway").

Can anyone suggest what might be going wrong here in case (3)?

Thanks,
--
John English
Simone Bordet
2017-12-26 19:13:03 UTC
Permalink
Hi,
Post by John English
3) Requests to https://www.foo.com go to the primary webapp as expected.
Requests to https://www.bar.com result in an empty 502 response ("Bad
Gateway").
Can anyone suggest what might be going wrong here in case (3)?
I would verify that you don't have a bad /etc/hosts on your local
machine, resolving bar.com to some address you don't expect.
Also, you did not report the Jetty configuration where you have setup
your virtual hosts.

Enabling DEBUG logging for the "org.eclipse.jetty" category is very
verbose but sometimes understandable enough also by users like you to
figure out what's wrong.
You may also file an issue and attach the logs there, or in this email thread.
--
Simone Bordet
----
http://cometd.org
http://webtide.com
Developer advice, training, services and support
from the Jetty & CometD experts.
John English
2017-12-26 20:04:36 UTC
Permalink
Post by Simone Bordet
Hi,
Post by John English
3) Requests to https://www.foo.com go to the primary webapp as expected.
Requests to https://www.bar.com result in an empty 502 response ("Bad
Gateway").
Can anyone suggest what might be going wrong here in case (3)?
I would verify that you don't have a bad /etc/hosts on your local
machine, resolving bar.com to some address you don't expect.
Nope.
Post by Simone Bordet
Also, you did not report the Jetty configuration where you have setup
your virtual hosts.
In my "webapps" directory, I have directories for the primary webapp,
default webapp and proxy webapp, with corresponding primary.xml,
default.xml and proxy.xml. In primary.xml I have this:

<Configure class="org.eclipse.jetty.webapp.WebAppContext">
<Set name="contextPath">/</Set>
<Set name="war"><Property name="jetty.webapps"
default="."/>/primary</Set>
<Set name="virtualHosts">
<Array type="java.lang.String">
<Item>www.foo.com</Item>
<Item>192.168.1.200</Item>
<Item>127.0.0.1</Item>
</Array>
</Set>
</Configure>

In proxy.xml I have this:

<Configure class="org.eclipse.jetty.webapp.WebAppContext">
<Set name="contextPath">/</Set>
<Set name="war"><Property name="jetty.webapps" default="."/>/proxy</Set>
<Set name="virtualHosts">
<Array type="java.lang.String">
<Item>www.bar.com</Item>
<Item>192.168.1.201</Item>
<Item>127.0.0.2</Item>
</Array>
</Set>
</Configure>

My default.xml is a catch-all for all other domain names or IP
addresses, which just responds with a 404 to all requests.

<Configure class="org.eclipse.jetty.webapp.WebAppContext">
<Set name="contextPath">/</Set>
<Set name="war"><Property name="jetty.webapps"
default="."/>/default</Set>
</Configure>
Post by Simone Bordet
Enabling DEBUG logging for the "org.eclipse.jetty" category is very
verbose but sometimes understandable enough also by users like you to
figure out what's wrong.
OK, I'll give it a whirl. Thanks.
--
John English
John English
2017-12-26 20:15:02 UTC
Permalink
Post by Simone Bordet
Hi,
Post by John English
3) Requests to https://www.foo.com go to the primary webapp as expected.
Requests to https://www.bar.com result in an empty 502 response ("Bad
Gateway").
Can anyone suggest what might be going wrong here in case (3)?
I would verify that you don't have a bad /etc/hosts on your local
machine, resolving bar.com to some address you don't expect.
I forgot to mention: I also see the request to bar.com in the request
log, where it's logged as e.g.

185.18.139.12 - - [26/Dec/2017:16:32:32 +0000] "GET / HTTP/1.1" 502 0

so DNS resolution is definitely not the issue. The fact that it's a "bad
gateway" response definitely suggests that the request is getting to the
proxy webapp.

Is there perhaps some way to run the proxy in a debug mode without
having to run the entire system in debug mode? This is a live system
with quite a few users, so I'm likely to get swamped with lots of
irrelevant information.
--
John English
Simone Bordet
2017-12-26 21:48:44 UTC
Permalink
Hi,
Is there perhaps some way to run the proxy in a debug mode without having to
run the entire system in debug mode? This is a live system with quite a few
users, so I'm likely to get swamped with lots of irrelevant information.
For ProxyServlet the logging category is derived from the servlet name
(in web.xml) and the subclass package and class name.
The logic is here:
https://github.com/eclipse/jetty.project/blob/jetty-9.4.x/jetty-proxy/src/main/java/org/eclipse/jetty/proxy/AbstractProxyServlet.java#L206

Enabling DEBUG for that category should tell you a bit more.

A blind shot: could it be that your virtual host is configured as
"www.bar.com", but you make a request to "bar.com" ?
--
Simone Bordet
----
http://cometd.org
http://webtide.com
Developer advice, training, services and support
from the Jetty & CometD experts.
John English
2017-12-27 08:34:34 UTC
Permalink
Post by Simone Bordet
Hi,
Is there perhaps some way to run the proxy in a debug mode without having to
run the entire system in debug mode? This is a live system with quite a few
users, so I'm likely to get swamped with lots of irrelevant information.
For ProxyServlet the logging category is derived from the servlet name
(in web.xml) and the subclass package and class name.
https://github.com/eclipse/jetty.project/blob/jetty-9.4.x/jetty-proxy/src/main/java/org/eclipse/jetty/proxy/AbstractProxyServlet.java#L206
Enabling DEBUG for that category should tell you a bit more.
Thanks. I'll see what I can do with that.
Post by Simone Bordet
A blind shot: could it be that your virtual host is configured as
"www.bar.com", but you make a request to "bar.com" ?
Nope.
--
John English
John English
2017-12-27 09:53:05 UTC
Permalink
Post by Simone Bordet
For ProxyServlet the logging category is derived from the servlet name
(in web.xml) and the subclass package and class name.
https://github.com/eclipse/jetty.project/blob/jetty-9.4.x/jetty-proxy/src/main/java/org/eclipse/jetty/proxy/AbstractProxyServlet.java#L206
Enabling DEBUG for that category should tell you a bit more.
Hmm, I seem to be doing something wrong. The servlet-name in web.xml is
"Proxy", and the servlet-class is "servlets.Proxy". I created a
resources/jetty-logging.properties which looks like this (copied from
the example in the documentation):

# Configure Jetty for StdErrLog Logging
org.eclipse.jetty.util.log.class=org.eclipse.jetty.util.log.StrErrLog
# Overall Logging Level is INFO
org.eclipse.jetty.LEVEL=INFO
# Detail Logging for proxy
Proxy.LEVEL=DEBUG

I then get a 500 response due to a NPE at line 65 of
ProxyServlet.service(): "if (_log.isDebugEnabled())..."

I then copied the code you linked to into my Proxy.init() and it
reported the servlet name as "servlets.Proxy.Proxy", but changing the
last line to "servlets.Proxy.Proxy.LEVEL=DEBUG" makes no difference; I
still get the same NPE. I also tried "servlets.Proxy.LEVEL=DEBUG"; still
no luck.

What am I doing wrong here?
--
John English
John English
2017-12-27 10:09:02 UTC
Permalink
Post by John English
What am I doing wrong here?
Oops, please ignore that -- I added a debug config parameter to my
servlet, added an override of init() to read it, and forgot to call
super.init() from my init()...
--
John English
Greg Wilkins
2017-12-27 11:07:49 UTC
Permalink
John,

what would also be really good to discover is what is the Host header of
the request that goes wrong: both incoming to the proxy and outgoing to the
second server.

What does the log on the 8888 server say? Does it see the request and what
response does it think it sent?

regards
Post by John English
Post by John English
What am I doing wrong here?
Oops, please ignore that -- I added a debug config parameter to my
servlet, added an override of init() to read it, and forgot to call
super.init() from my init()...
--
John English
_______________________________________________
jetty-users mailing list
To change your delivery options, retrieve your password, or unsubscribe
from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users
--
Greg Wilkins <***@webtide.com> CTO http://webtide.com
Continue reading on narkive:
Loading...