Receiving 403 with Tomcat 9, works with Tomcat 8

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Receiving 403 with Tomcat 9, works with Tomcat 8

Jörg Schaible
Hi,

we have a strange symptom after an upgrade from Tomcat 8 to Tomcat 9, because
we get a 403 for a call that works flawlessly with the previous version.

Let's describe the scenario: We have a customer with a Wordpress application
hosted on an Apache server. Some pages perform XMLHttpRequests to load and
embed HTML snippets from other sources. One such source is our (load-balanced)
web application running on Tomcat. These requests are using GET or POST,
depending on the situation. However, after the switch from Tomcat 8 to Tomcat
9, the GET request is replied by Tomcat with 403. And the only trace is an
entry in the access_log. However, if we use the request URL directly in the
browser, the call succeeds.

We are using a vanilla installation of Tomcat. The load-balancer will map the
HTTPS calls on port 443 to HTTP on port 8080. The only modification to the
configuration is in catalina.properties, where we skip the jar scanning:

- tomcat.util.scan.StandardJarScanFilter.jarsToSkip=*

And we have some additional attributes at the connector in the server.xml:

  <Connector
    port="8080" protocol="HTTP/1.1"
    connectionTimeout="20000"
    redirectPort="8443"
    maxThreads="1000"  
    acceptCount="400"
    allowHostHeaderMismatch="true" />

Originally we suspected the "allowHostHeaderMismatch" attribute, because it
changed its default from true in Tomcat 8 to false in Tomcat 9, but it had no
effect on the communication

If we look at the network analysis in the browser, we have following request
parameters (example):

====== %< ====
GET https://tomcat.test-server.local/app/service?param=1

The HTTP request header contains:
- Host: tomcat.test-server.local
- Origin: https://www.test-server.local
- Referrer: https://www.test-server.local/
- DNT: 1

The HTTP response header contains:
- Access-Control-Allow-Credentials: true
- Access-Control-Allow-Origin: https://www.test-server.local
- Cache-Control: no-cache
- Content-Type: text/xml;charset=UTF-8
- Server: Apache-Coyote/1.1
- Transfer-Encoding: chunked
====== %< ====

We found the switched default for "allowHostHeaderMismatch" by chance. Are
there other parameters in the Tomcat configuration that are new or have changed
their default, which may influence this communication?

What's the best way to analyze this on the Tomcat side? Are there any special
logger settings to get more info about this 403?

Regards,
Jörg



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Receiving 403 with Tomcat 9, works with Tomcat 8

Mark Thomas-2
Exact Tomcat 8 version?
Exact Tomcat 9 version?

How is CORS configured in your application?

Mark


On 06/02/2019 10:36, Jörg Schaible wrote:

> Hi,
>
> we have a strange symptom after an upgrade from Tomcat 8 to Tomcat 9, because
> we get a 403 for a call that works flawlessly with the previous version.
>
> Let's describe the scenario: We have a customer with a Wordpress application
> hosted on an Apache server. Some pages perform XMLHttpRequests to load and
> embed HTML snippets from other sources. One such source is our (load-balanced)
> web application running on Tomcat. These requests are using GET or POST,
> depending on the situation. However, after the switch from Tomcat 8 to Tomcat
> 9, the GET request is replied by Tomcat with 403. And the only trace is an
> entry in the access_log. However, if we use the request URL directly in the
> browser, the call succeeds.
>
> We are using a vanilla installation of Tomcat. The load-balancer will map the
> HTTPS calls on port 443 to HTTP on port 8080. The only modification to the
> configuration is in catalina.properties, where we skip the jar scanning:
>
> - tomcat.util.scan.StandardJarScanFilter.jarsToSkip=*
>
> And we have some additional attributes at the connector in the server.xml:
>
>   <Connector
>     port="8080" protocol="HTTP/1.1"
>     connectionTimeout="20000"
>     redirectPort="8443"
>     maxThreads="1000"  
>     acceptCount="400"
>     allowHostHeaderMismatch="true" />
>
> Originally we suspected the "allowHostHeaderMismatch" attribute, because it
> changed its default from true in Tomcat 8 to false in Tomcat 9, but it had no
> effect on the communication
>
> If we look at the network analysis in the browser, we have following request
> parameters (example):
>
> ====== %< ====
> GET https://tomcat.test-server.local/app/service?param=1
>
> The HTTP request header contains:
> - Host: tomcat.test-server.local
> - Origin: https://www.test-server.local
> - Referrer: https://www.test-server.local/
> - DNT: 1
>
> The HTTP response header contains:
> - Access-Control-Allow-Credentials: true
> - Access-Control-Allow-Origin: https://www.test-server.local
> - Cache-Control: no-cache
> - Content-Type: text/xml;charset=UTF-8
> - Server: Apache-Coyote/1.1
> - Transfer-Encoding: chunked
> ====== %< ====
>
> We found the switched default for "allowHostHeaderMismatch" by chance. Are
> there other parameters in the Tomcat configuration that are new or have changed
> their default, which may influence this communication?
>
> What's the best way to analyze this on the Tomcat side? Are there any special
> logger settings to get more info about this 403?
>
> Regards,
> Jörg
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Receiving 403 with Tomcat 9, works with Tomcat 8

Jörg Schaible
Hi Mark,

Am Mittwoch, 6. Februar 2019, 11:45:46 CET schrieb Mark Thomas:
> Exact Tomcat 8 version?
> Exact Tomcat 9 version?
>
> How is CORS configured in your application?

the VersionLoggerListener entries from the catalina.log files:

this is the machine with Tomcat 8:
============== %< ==================
- Server version:        Apache Tomcat/8.0.41
- Server built:          Jan 18 2017 22:19:39 UTC
- Server number:         8.0.41.0
- OS Name:               Windows Server 2012 R2
- OS Version:            6.3
- Architecture:          amd64
- Java Home:             D:\Programme\Java
- JVM Version:           1.8.0_121-b13
- JVM Vendor:            Oracle Corporation
- CATALINA_BASE:         D:\Programme\Tomcat
- CATALINA_HOME:         D:\Programme\Tomcat
- Command line argument: -Dcatalina.home=D:\Programme\Tomcat
- Command line argument: -Dcatalina.base=D:\Programme\Tomcat
- Command line argument: -Djava.endorsed.dirs=D:\Programme\Tomcat\endorsed
- Command line argument: -Djava.io.tmpdir=D:\Programme\Tomcat\temp
- Command line argument: -
Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
- Command line argument: -Djava.util.logging.config.file=D:
\Programme\Tomcat\conf\logging.properties
- Command line argument: exit
- Command line argument: -Xms5120m
- Command line argument: -Xmx30720m
============== %< ==================

this is the machine with Tomcat 9:
============== %< ==================
- Server Version:        Apache Tomcat/9.0.14
- Server built:          Dec 6 2018 21:13:53 UTC
- Server version number: 9.0.14.0
- OS Name:               Windows Server 2012 R2
- OS Version:            6.3
- Architektur:          amd64
- Java Home:             D:\Programme\OpenJDK11
- JVM Version:           11.0.2+9
- JVM Hersteller:            Oracle Corporation
- CATALINA_BASE:         D:\Programme\Tomcat
- CATALINA_HOME:         D:\Programme\Tomcat
- Command line argument: -Dcatalina.home=D:\Programme\Tomcat
- Command line argument: -Dcatalina.base=D:\Programme\Tomcat
- Command line argument: -Djava.io.tmpdir=D:\Programme\Tomcat\temp
- Command line argument: -
Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
- Command line argument: -Djava.util.logging.config.file=D:
\Programme\Tomcat\conf\logging.properties
- Command line argument: --add-opens=java.base/java.lang=ALL-UNNAMED
- Command line argument: --add-opens=java.base/java.io=ALL-UNNAMED
- Command line argument: --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
- Command line argument: exit
- Command line argument: abort
- Command line argument: -Xms5120m
- Command line argument: -Xmx30720m
============== %< ==================

The CORS-Settings from the web.xml:

============== %< ==================
  <filter>
    <filter-name>CorsFilter</filter-name>
    <filter-class>org.apache.catalina.filters.CorsFilter</filter-class>
    <init-param>
      <param-name>cors.exposedHeaders</param-name>
      <param-value>Set-Cookie</param-value>
    </init-param>
  </filter>
  <filter-mapping>
    <filter-name>CorsFilter</filter-name>
    <url-pattern>/*</url-pattern>
  </filter-mapping>
============== %< ==================

Regards,
Jörg



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Receiving 403 with Tomcat 9, works with Tomcat 8

Mark Thomas-2
On 06/02/2019 12:48, Jörg Schaible wrote:

> Hi Mark,
>
> Am Mittwoch, 6. Februar 2019, 11:45:46 CET schrieb Mark Thomas:
>> Exact Tomcat 8 version?
>> Exact Tomcat 9 version?
>>
>> How is CORS configured in your application?
>
> the VersionLoggerListener entries from the catalina.log files:
>
> this is the machine with Tomcat 8:
> ============== %< ==================
> - Server version:        Apache Tomcat/8.0.41
> - Server built:          Jan 18 2017 22:19:39 UTC

<snip/>

> - Server Version:        Apache Tomcat/9.0.14
> - Server built:          Dec 6 2018 21:13:53 UTC

You have almost 2 years of bug fixes between those versions.

Looks like you've hit the fixes for these bugs:
https://bz.apache.org/bugzilla/show_bug.cgi?id=62676
https://bz.apache.org/bugzilla/show_bug.cgi?id=62761
https://bz.apache.org/bugzilla/show_bug.cgi?id=62343 (CVE-2018-8014)


> The CORS-Settings from the web.xml:
>
> ============== %< ==================
>   <filter>
>     <filter-name>CorsFilter</filter-name>
>     <filter-class>org.apache.catalina.filters.CorsFilter</filter-class>
>     <init-param>
>       <param-name>cors.exposedHeaders</param-name>
>       <param-value>Set-Cookie</param-value>
>     </init-param>
>   </filter>
>   <filter-mapping>
>     <filter-name>CorsFilter</filter-name>
>     <url-pattern>/*</url-pattern>
>   </filter-mapping>
> ============== %< ==================

You need to set cors.allowed.origin to an appropriate value. See:
http://tomcat.apache.org/tomcat-9.0-doc/config/filter.html#CORS_Filter

Mark

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Receiving 403 with Tomcat 9, works with Tomcat 8

Jörg Schaible
Hi Mark,

Am Mittwoch, 6. Februar 2019, 15:32:26 CET schrieb Mark Thomas:

[snip]

> You need to set cors.allowed.origin to an appropriate value. See:
> http://tomcat.apache.org/tomcat-9.0-doc/config/filter.html#CORS_Filter

thanks for your pointers, but unfortunately even setting the value to '*' has
no effect, we still get the 403 for this request. Is there anything else we
can to to debug this? Some logger settings?

Regards,
Jörg



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Receiving 403 with Tomcat 9, works with Tomcat 8

Jörg Schaible
Hi Mark

Am Freitag, 8. Februar 2019, 09:30:32 CET schrieb Jörg Schaible:

> Hi Mark,
>
> Am Mittwoch, 6. Februar 2019, 15:32:26 CET schrieb Mark Thomas:
>
> [snip]
>
> > You need to set cors.allowed.origin to an appropriate value. See:
> > http://tomcat.apache.org/tomcat-9.0-doc/config/filter.html#CORS_Filter
>
> thanks for your pointers, but unfortunately even setting the value to '*'
> has no effect, we still get the 403 for this request. Is there anything
> else we can to to debug this? Some logger settings?

Just as final remark. After correcting the parameter name, the filter works as
expected. Sometimes the problem is in front of the keyboard... ;-)

Thanks and cheers,
Jörg





---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]