Tomcat 8.5 appends null characters

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Tomcat 8.5 appends null characters

Tuukka Ilomäki
Dear collective wisdom,
as the EOL of Tomcat 7 is looming, we are migrating our legacy app from Tomcat 7.0 to Tomcat 8.5. We deploy exactly the same war in both versions. For some reason Tomcat 8.5 adds null characters to the end of JavaScript files. For instance, jQuery file served by Tomcat 8.5 ends with “})( window ); �����” where five last characters are bytes 00. Tomcat 7.0 serves the same file correctly. In case it matters, the issue occurs in both windows 10 and windows server 2016 environments.

I assume this has something to do with character encoding. Original js-file is cp1252. I have perused Tomcat 8 migration guide and Tomcat encoding FAQ and experimented with the various options but so far with no luck. Any advice is greatly appreciated.

Tuukka

Reply | Threaded
Open this post in threaded view
|

Re: Tomcat 8.5 appends null characters

Martin Grigorov
Hi,

On Tue, May 19, 2020 at 3:32 PM Tuukka Ilomäki <[hidden email]>
wrote:

> Dear collective wisdom,
> as the EOL of Tomcat 7 is looming, we are migrating our legacy app from
> Tomcat 7.0 to Tomcat 8.5. We deploy exactly the same war in both versions.
> For some reason Tomcat 8.5 adds null characters to the end of JavaScript
> files. For instance, jQuery file served by Tomcat 8.5 ends with “})( window
> ); �����” where five last characters are bytes 00. Tomcat 7.0 serves the
> same file correctly. In case it matters, the issue occurs in both windows
> 10 and windows server 2016 environments.
>
> I assume this has something to do with character encoding. Original
> js-file is cp1252. I have perused Tomcat 8 migration guide and Tomcat
> encoding FAQ and experimented with the various options but so far with no
> luck. Any advice is greatly appreciated.
>

I'd advise you to convert your files to UTF-8.
Or tell Tomcat to write responses in your preferred encoding.


>
> Tuukka
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Tomcat 8.5 appends null characters

Christopher Schultz-2
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Martin,

On 5/19/20 09:50, Martin Grigorov wrote:

> Hi,
>
> On Tue, May 19, 2020 at 3:32 PM Tuukka Ilomäki
> <[hidden email]> wrote:
>
>> Dear collective wisdom, as the EOL of Tomcat 7 is looming, we are
>> migrating our legacy app from Tomcat 7.0 to Tomcat 8.5. We deploy
>> exactly the same war in both versions. For some reason Tomcat 8.5
>> adds null characters to the end of JavaScript files. For
>> instance, jQuery file served by Tomcat 8.5 ends with “})( window
>> ); �����” where five last characters are bytes 00. Tomcat 7.0
>> serves the same file correctly. In case it matters, the issue
>> occurs in both windows 10 and windows server 2016 environments.
>>
>> I assume this has something to do with character encoding.
>> Original js-file is cp1252. I have perused Tomcat 8 migration
>> guide and Tomcat encoding FAQ and experimented with the various
>> options but so far with no luck. Any advice is greatly
>> appreciated.
>>
>
> I'd advise you to convert your files to UTF-8. Or tell Tomcat to
> write responses in your preferred encoding.

Tailing nulls might also be a misinterpreted chunked response. (I'm
grasping at straws, here).

Tuukka, is your client communicating directly with Tomcat, or through
some kind of proxy or load balancer?

I've been using the same product with Tomcat since the 4.1 series, and
I upgraded through every single major version from there to 9.0. I've
never seen anything like that.

- -chris
-----BEGIN PGP SIGNATURE-----
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/

iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl7ESsgACgkQHPApP6U8
pFgreA//VI1lMAk/P9hnlx1WCU9Uz2Ccb/UYDZe3HVTO4FrjqUkPi3eOC5Zf5nu9
r0AfFRZVpdiih9z4DGPngarxsymqxi8VfIds2HJcGtbYtJ4Kxt1WBUBCB//EmzSr
YU70a6LL3IzbWecV641497JvfASFTCc7LUTbO6D48RFezoPPD+g3tAhPLsUNtOr9
hBD8nINL+g4HTUb5uA0+N60Sjp81BXLlYbnX5FGzAwxicZtgLkGir/au0yqWEOTC
kZ+ejPUKpYw4ingWrlv4iY6+3EPLsi2jiTu+nLkiHKmcVvcxVkWt3m+KpvVWef14
78WFHPMBeBhgt4vpX37+YSkeVUVqr+NyX2Fyp/6mW8cyXFgYIi7C4YAyLpF7zeNz
3xRll64nvGnKghD/jbiHUr1prqwECXUFNKfPBK2XBL7k1h1I7eRzUtkUIbmwWQ6T
LwmwY3LXQ1+TLoISppQ5Cb27z3SfPVqi/2dFXbdqiLF2VOpCNKJl1qSDtlVEN5Az
qMKInNfegUoHAvznFclJX/3pFEhdA1l+LWbpd1zVbOMfm9ZCAi0dUtW75NTuP/Ur
43NGSXbxiblrq0xhxY+qVbEs9lnz9vhK9wRApSdBfN7/errej1IAXhKk9lFgJx4K
O5I3nIfT2BtHIKj5eDDTlYJfTlvRkkzAV4MPGPmBjVHdvAWmfsw=
=JiQr
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Tomcat 8.5 appends null characters

Tuukka Ilomäki
In reply to this post by Tuukka Ilomäki
On 5/19/20 09:50, Martin Grigorov wrote:

>> Hi,
>>
>> On Tue, May 19, 2020 at 3:32 PM Tuukka Ilomäki
>> <[hidden email]> wrote:
>>
>>> Dear collective wisdom, as the EOL of Tomcat 7 is looming, we are
>>> migrating our legacy app from Tomcat 7.0 to Tomcat 8.5. We deploy
>>> exactly the same war in both versions. For some reason Tomcat 8.5
>>> adds null characters to the end of JavaScript files. For
>>> instance, jQuery file served by Tomcat 8.5 ends with “})( window
>>> ); �����” where five last characters are bytes 00. Tomcat 7.0
>>> serves the same file correctly. In case it matters, the issue
>>> occurs in both windows 10 and windows server 2016 environments.
>>>
>>> I assume this has something to do with character encoding.
>>> Original js-file is cp1252. I have perused Tomcat 8 migration
>>> guide and Tomcat encoding FAQ and experimented with the various
>>> options but so far with no luck. Any advice is greatly
>>> appreciated.
>>>
>>
>> I'd advise you to convert your files to UTF-8. Or tell Tomcat to
>> write responses in your preferred encoding.


I would prefer keeping CP1252 for two reasons. First, for a test period we can keep the same codebase for production running in Tomcat 7 and test running Tomcat 8.5. Secondly, initial experiments with UTF-8 caused failures in backend RPG call to AS400 (trailing nulls again there as well).


“tell Tomcat to write responses in your preferred encoding” is exactly what I am trying to do but it seems that I have not found all the places where I need to tell it.

>Tailing nulls might also be a misinterpreted chunked response. (I'm
>grasping at straws, here).
>
>Tuukka, is your client communicating directly with Tomcat, or through
>some kind of proxy or load balancer?
>
>I've been using the same product with Tomcat since the 4.1 series, and
>I upgraded through every single major version from there to 9.0. I've
>never seen anything like that.

In production environment we have httpd in front of Tomcat but now in testing phase we communicate directly to Tomcat.

My wild guess is that response size is calculated using UTF-8 but response is cp1252. This is suggested by the fact that if I add comment line “// ääää” that has four characters that are double-byte characters in UTF-8, then the file served by Tomcat has four more null bytes appended.

Having first written all of the above and after some serious digging deep into the bowels of legacy code it turned out that a filter was setting the encoding. (Following the golden rules of legacy code, no one remembers the rational for that piece of code.) For unknown reasons it had worked for almost a decade in Tomcat 7 but Tomcat 8.5 did not like it. The reason for the different behavior remains unknown, but I have found where to start tinkering.

Thanks to all for the suggestions – much appreciated.

Tuukka
Reply | Threaded
Open this post in threaded view
|

Re: Tomcat 8.5 appends null characters

Christopher Schultz-2
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Tuukka,

On 5/20/20 14:25, Tuukka Ilomäki wrote:

> On 5/19/20 09:50, Martin Grigorov wrote:
>>> Hi,
>>>
>>> On Tue, May 19, 2020 at 3:32 PM Tuukka Ilomäki
>>> <[hidden email]> wrote:
>>>
>>>> Dear collective wisdom, as the EOL of Tomcat 7 is looming, we
>>>> are migrating our legacy app from Tomcat 7.0 to Tomcat 8.5.
>>>> We deploy exactly the same war in both versions. For some
>>>> reason Tomcat 8.5 adds null characters to the end of
>>>> JavaScript files. For instance, jQuery file served by Tomcat
>>>> 8.5 ends with “})( window ); �����” where five last
>>>> characters are bytes 00. Tomcat 7.0 serves the same file
>>>> correctly. In case it matters, the issue occurs in both
>>>> windows 10 and windows server 2016 environments.
>>>>
>>>> I assume this has something to do with character encoding.
>>>> Original js-file is cp1252. I have perused Tomcat 8
>>>> migration guide and Tomcat encoding FAQ and experimented with
>>>> the various options but so far with no luck. Any advice is
>>>> greatly appreciated.
>>>>
>>>
>>> I'd advise you to convert your files to UTF-8. Or tell Tomcat
>>> to write responses in your preferred encoding.
>
>
> I would prefer keeping CP1252 for two reasons. First, for a test
> period we can keep the same codebase for production running in
> Tomcat 7 and test running Tomcat 8.5. Secondly, initial experiments
> with UTF-8 caused failures in backend RPG call to AS400 (trailing
> nulls again there as well).
>
>
> “tell Tomcat to write responses in your preferred encoding” is
> exactly what I am trying to do but it seems that I have not found
> all the places where I need to tell it.
>
>> Tailing nulls might also be a misinterpreted chunked response.
>> (I'm grasping at straws, here).
>>
>> Tuukka, is your client communicating directly with Tomcat, or
>> through some kind of proxy or load balancer?
>>
>> I've been using the same product with Tomcat since the 4.1
>> series, and I upgraded through every single major version from
>> there to 9.0. I've never seen anything like that.
>
> In production environment we have httpd in front of Tomcat but now
> in testing phase we communicate directly to Tomcat.

Okay, that's good.

> My wild guess is that response size is calculated using UTF-8 but
> response is cp1252. This is suggested by the fact that if I add
> comment line “// ääää” that has four characters that are
> double-byte characters in UTF-8, then the file served by Tomcat has
> four more null bytes appended.
Is this a static file? For static files, Tomcat will return the size
in bytes of the on-disk resource. The bytes are read from the disk and
thrown onto the network with zero interpretation.

Perhaps the on-disk file is in CP1252 but the HTTP response
Content-Type header says UTF-8 (which is incorrect)?

> Having first written all of the above and after some serious
> digging deep into the bowels of legacy code it turned out that a
> filter was setting the encoding.
This is where I thought we might end up (assuming that
chunked-encoding wasn't the problem).

> (Following the golden rules of legacy code, no one remembers the
> rational for that piece of code.) For unknown reasons it had worked
> for almost a decade in Tomcat 7 but Tomcat 8.5 did not like it. The
> reason for the different behavior remains unknown, but I have found
> where to start tinkering.

- -chris
-----BEGIN PGP SIGNATURE-----
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/

iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl7Gw3AACgkQHPApP6U8
pFhBkxAAiHX4h7FC02PX6TwYOhUbJvs2Nvu9XufIV/ktXVg1nFWhXG+7R6YIj7rX
IUW5TkQezTAIjQUidSKigNm6PFDOyQ/PsS6dnhk/n/E6jaJDpLZMfUlhYJQQWEaY
/BinS0V5SsotHyv7FCjFDClLAOaetpGAUeb0RatlLIIjjtHVT/75N27rgPQjL4yc
BYKNE7+RycLM0z6q0FoUPFYaS6fAsPoeq1YzvlbnQKdgMmz4Ju9cwH39Per9XKWf
Xu+GhviZ3bYO7qdPFZs7VmQKbaVUciFUW4nwK+lTEL9ZwUen/ojMxua4MWSapnVF
pJsbBEEH4Po2YmySUfyw7pcD8qJ/g9UwUMD/s1hR7ClYlO/QViAOvMh/32kjXWYV
+wrgEB96XvbcuK6G4uZGKgXgrUaO8VJm3u5IS9csJK9MAdy0FGIqxI7ev199/waW
38Y3OO1kJiyySHYEaFOQktv0HkE5aggCTPPx9EQNumW2OLEqEe9VtFNFdD5/0eWa
mP0u+x+Yo2U7k2w+zn4dQP+ADtv24suU+ErTd8N0ti7LsyXWedPpHnoBsZuSrCuG
zR8H79JA0afg1TK/8WlFrqZj3hS7nkjzmK1x+DFc21KnBVP7x68E2Gmw14/A0gs+
PLARiQFRIljXdJxls+s9lBj8b/dKfNVTvrFPePZpid654ibcKOU=
=K4e7
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]