[jetty-users] Jetty Cannot Find File In Web Application (Duo Outage)

[David C Fuhs]

Good morning all:

We are running Shibboleth Identity Provider (IdP) V3.3.3 on Jetty 9.3.9 on a Red Hat Linux platform. A multifactor authentication flow added to the IdP redirects any user found in the MFA group in LDAP to duo.vm. This has worked without error from when we installed it until a failure last week.

The nature of the failure: A user in the MFA group clicked to login to a federated application. The IdP displayed its login page as expected. The user entered his uid + password and was successfully authenticated. The search of the MFA group succeeded, but only a partial copy of duo.vm was displayed: the external part of the page was displayed but the inner section with the green buttons (e.g., "send me a push") never loaded.

A JavaScript file that is part of the IdP web application calls to Duo to retrieve the content for the inner section of the page. That call never happened.

The IdP warn and process logs did not show anything interesting, but the log levels are set pretty low. The Jetty access log did show that on the server experiencing the outage, the IdP got an HTTP 404 not found error on this call:

GET /idp/js/Duo-Web-v2.min.js

Access logs on the other IdP servers at the same time all showed HTTP 200 codes on the same GET.

That file exists and is readable. Nothing in the Jetty or Duo configurations has been touched in a long time.

Restarting Jetty on the problematic IdP resolved the problem, but we don't know why the problem occurred in the first place, or how to prevent it in the future. The problematic IdP had been successfully authenticating MFA users for months with no reported problems.

We tested right after the failure against our development IdP instances and a hot spare production instance, but all of those systems behaved as expected.

We have made no Jetty configuration changes since we installed version 9.3.9.

The only recent changes to the production IdP server where the problem occurred was routine RHEL OS patching a week earlier (with no reported problems in the intervening week) plus ongoing additions to the CAS registry (conf/cas-protocol.xml), also with no reported side-effects. We have not added configurations for any new service providers recently, and have made no changes to any Duo pages or configuration items in months.

What would cause Jetty to not find that JavaScript file?

How can we prevent a recurrence?

Any insights greatly appreciated.

David Fuhs
Enterprise Applications
Information Security
California State University, Chico
***@csuchico.edu
530-898-4852

[Joakim Erdfelt]

Sounds like you have deployed your webapps in a way that the work directory
is in System temp (aka "/tmp" on redhat).
But now you have something that is cleaning out system temp periodically.
(which your system didn't do before)

If you are using a ${jetty.base} directory, add a directory under it (not
your ${jetty.home}, jetty-home, or jetty-distribution directory) called
"work" and restart.
Or ... you can configure a different location for java's java.io.tmpdir.

If you want to know more, look at how webapp temp directories function ->
https://stackoverflow.com/questions/19232182/jetty-starts-in-c-temp/19232771#19232771

Post by David C Fuhs
We are running Shibboleth Identity Provider (IdP) V3.3.3 on Jetty 9.3.9 on
a Red Hat Linux platform. A multifactor authentication flow added to the
IdP redirects any user found in the MFA group in LDAP to duo.vm. This has
worked without error from when we installed it until a failure last week.
The nature of the failure: A user in the MFA group clicked to login to a
federated application. The IdP displayed its login page as expected. The
user entered his uid + password and was successfully authenticated. The
search of the MFA group succeeded, but only a partial copy of duo.vm was
displayed: the external part of the page was displayed but the inner
section with the green buttons (e.g., “send me a push”) never loaded.
A JavaScript file that is part of the IdP web application calls to Duo to
retrieve the content for the inner section of the page. That call never
happened.
The IdP warn and process logs did not show anything interesting, but the
log levels are set pretty low. The Jetty access log did show that on the
server experiencing the outage, the IdP got an HTTP 404 not found error on
GET /idp/js/Duo-Web-v2.min.js
Access logs on the other IdP servers at the same time all showed HTTP 200
codes on the same GET.
That file exists and is readable. Nothing in the Jetty or Duo
configurations has been touched in a long time.
Restarting Jetty on the problematic IdP resolved the problem, but we don’t
know why the problem occurred in the first place, or how to prevent it in
the future. The problematic IdP had been successfully authenticating MFA
users for months with no reported problems.
We tested right after the failure against our development IdP instances
and a hot spare production instance, but all of those systems behaved as
expected.
We have made no Jetty configuration changes since we installed version 9.3.9.
The only recent changes to the production IdP server where the problem
occurred was routine RHEL OS patching a week earlier (with no reported
problems in the intervening week) plus ongoing additions to the CAS
registry (conf/cas-protocol.xml), also with no reported side-effects. We
have not added configurations for any new service providers recently, and
have made no changes to any Duo pages or configuration items in months.
What would cause Jetty to not find that JavaScript file?
How can we prevent a recurrence?
Any insights greatly appreciated.
David Fuhs
Enterprise Applications
Information Security
California State University, Chico
530-898-4852
_______________________________________________
jetty-users mailing list
To change your delivery options, retrieve your password, or unsubscribe
from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users