File size truncated at 1.4GB during download from Tomcat WebApp

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

File size truncated at 1.4GB during download from Tomcat WebApp

Mauro Tridici
Dear Users,

I’m struggling with the problem mentioned in this mail subject.
When I try to download a 5GB sized file using two different tomcat web applications on two different virtual machines, I noticed that the browser download window shows a wrong file size (please take a look to the attached picture).
Download starts regularly but it never ends.

I tried to download different files using different web browsers, but nothing changed.
It seems that this problem is not related to the web applications (as I said, I’m testing two applications) and it is not related to a specific tomcat version.

I’m a “tomcat newbie”, could you please help me to understand the cause of this issue?
You can find below some additional information about virtual machines, os and tomcat.

1) virtual machine “serverA"

OS: CentOS Linux release 7.8.2003

TOMCAT: 
Server version: Apache Tomcat/7.0.76
Server built:   Mar 17 2020 23:48:55 UTC
Server number:  7.0.76.0
OS Name:        Linux
OS Version:     3.10.0-1127.8.2.el7.x86_64
Architecture:   amd64
JVM Version:    1.8.0_252-b09
JVM Vendor:     Oracle Corporation

2) virtual machine “serverB”

OS: CentOS Linux release 7.8.2003

TOMCAT:
Server version: Apache Tomcat/8.5.56
Server built:   Jun 3 2020 20:18:30 UTC
Server number:  8.5.56.0
OS Name:        Linux
OS Version:     3.10.0-1127.8.2.el7.x86_64
Architecture:   amd64
JVM Version:    1.8.0_252-b09
JVM Vendor:     Oracle Corporation


Thank you in advance,
Mauro



Reply | Threaded
Open this post in threaded view
|

Re: File size truncated at 1.4GB during download from Tomcat WebApp

Christopher Schultz-2
Mauro,

On 10/3/20 08:47, Mauro Tridici wrote:

> Dear Users,
>
> I’m struggling with the problem mentioned in this mail subject.
> When I try to download a 5GB sized file using two different tomcat web
> applications on two different virtual machines, I noticed that the
> browser download window shows a wrong file size (please take a look to
> the attached picture).
> Download starts regularly but it never ends.
>
> I tried to download different files using different web browsers, but
> nothing changed.
> It seems that this problem is not related to the web applications (as I
> said, I’m testing two applications) and it is not related to a specific
> tomcat version.
>
> I’m a “tomcat newbie”, could you please help me to understand the cause
> of this issue?
> You can find below some additional information about virtual machines,
> os and tomcat.
>
> *1) virtual machine “serverA"*
>
> OS: CentOS Linux release 7.8.2003
>
> TOMCAT: 
> Server version: Apache Tomcat/7.0.76
> Server built:   Mar 17 2020 23:48:55 UTC
> Server number:  7.0.76.0
> OS Name:        Linux
> OS Version:     3.10.0-1127.8.2.el7.x86_64
> Architecture:   amd64
> JVM Version:    1.8.0_252-b09
> JVM Vendor:     Oracle Corporation
>
> *2) virtual machine “serverB”*
>
> OS: CentOS Linux release 7.8.2003
>
> TOMCAT:
> Server version: Apache Tomcat/8.5.56
> Server built:   Jun 3 2020 20:18:30 UTC
> Server number:  8.5.56.0
> OS Name:        Linux
> OS Version:     3.10.0-1127.8.2.el7.x86_64
> Architecture:   amd64
> JVM Version:    1.8.0_252-b09
> JVM Vendor:     Oracle Corporation
>
>
> Thank you in advance,
> Mauro

Can you observe the HTTP response headers sent by the server along with
the file?

Also, how are you sending the file(s) from the application to the
client? Are you using sendfile? Are you using the DefaultServlet? Or do
you have your own code which reads the file and writes it to the client?

Does the download always stop at the same point (byte count), or it is
different each time?

My guess is:

1. The code is in your application (or at least, not using Tomcat's code
at all)

2. The code uses "int" variable to track the file size / set
Content-Length response header

3. The file size is 5.4GiB

4. 5.4 GiB will overflow a 32-bit int by 3.4GiB resulting in a value of
+1.4GiB final value

5. This value is being send to the client in the Content-Length header

6. The client is terminating the download after 1.4GiB (respecting the
Content-Length header)

-chris

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: File size truncated at 1.4GB during download from Tomcat WebApp

Mauro Tridici


On 3 Oct 2020, at 15:14, Christopher Schultz <[hidden email]> wrote:

Mauro,

On 10/3/20 08:47, Mauro Tridici wrote:
Dear Users,

I’m struggling with the problem mentioned in this mail subject.
When I try to download a 5GB sized file using two different tomcat web
applications on two different virtual machines, I noticed that the
browser download window shows a wrong file size (please take a look to
the attached picture).
Download starts regularly but it never ends.

I tried to download different files using different web browsers, but
nothing changed.
It seems that this problem is not related to the web applications (as I
said, I’m testing two applications) and it is not related to a specific
tomcat version.

I’m a “tomcat newbie”, could you please help me to understand the cause
of this issue?
You can find below some additional information about virtual machines,
os and tomcat.

*1) virtual machine “serverA"*

OS: CentOS Linux release 7.8.2003

TOMCAT: 
Server version: Apache Tomcat/7.0.76
Server built:   Mar 17 2020 23:48:55 UTC
Server number:  7.0.76.0
OS Name:        Linux
OS Version:     3.10.0-1127.8.2.el7.x86_64
Architecture:   amd64
JVM Version:    1.8.0_252-b09
JVM Vendor:     Oracle Corporation

*2) virtual machine “serverB”*

OS: CentOS Linux release 7.8.2003

TOMCAT:
Server version: Apache Tomcat/8.5.56
Server built:   Jun 3 2020 20:18:30 UTC
Server number:  8.5.56.0
OS Name:        Linux
OS Version:     3.10.0-1127.8.2.el7.x86_64
Architecture:   amd64
JVM Version:    1.8.0_252-b09
JVM Vendor:     Oracle Corporation


Thank you in advance,
Mauro

Can you observe the HTTP response headers sent by the server along with
the file?

Also, how are you sending the file(s) from the application to the
client? Are you using sendfile? Are you using the DefaultServlet? Or do
you have your own code which reads the file and writes it to the client?

Does the download always stop at the same point (byte count), or it is
different each time?

My guess is:

1. The code is in your application (or at least, not using Tomcat's code
at all)

2. The code uses "int" variable to track the file size / set
Content-Length response header

3. The file size is 5.4GiB

4. 5.4 GiB will overflow a 32-bit int by 3.4GiB resulting in a value of
+1.4GiB final value

5. This value is being send to the client in the Content-Length header

6. The client is terminating the download after 1.4GiB (respecting the
Content-Length header)

-chris

---------------------------------------------------------------------
To unsubscribe, [hidden email]
For additional commands, [hidden email]


HI Christopher,

thank you very much for your answer.

You can find below the HTTP response headers detected during the download fo file named disk-0.vmdk

172.16.1.8 - - [03/Oct/2020:15:25:53 +0200] "GET /irods-cloud-backend/virtualCollection/root HTTP/1.1" 401 951
172.16.1.8 - - [03/Oct/2020:15:25:53 +0200] "GET /irods-cloud-backend/collection/root?offset=0&path=%2Fidas%2Fhome%2Fsysm02 HTTP/1.1" 401 951
172.16.1.8 - - [03/Oct/2020:15:25:54 +0200] "GET /irods-cloud-backend/initialConf HTTP/1.1" 200 138
172.16.1.8 - - [03/Oct/2020:15:26:00 +0200] "OPTIONS /irods-cloud-backend/login/ HTTP/1.1" 200 -
172.16.1.8 - - [03/Oct/2020:15:26:00 +0200] "POST /irods-cloud-backend/login/ HTTP/1.1" 200 102
172.16.1.8 - - [03/Oct/2020:15:26:01 +0200] "GET /irods-cloud-backend/collection/root?offset=0&path=%2Fidas%2Fhome%2Fsysm02 HTTP/1.1" 200 8117
172.16.1.8 - - [03/Oct/2020:15:26:01 +0200] "GET /irods-cloud-backend/virtualCollection/root HTTP/1.1" 200 480
172.16.1.8 - - [03/Oct/2020:15:26:01 +0200] "GET /irods-cloud-backend/fileCreatorTemplate HTTP/1.1" 200 787
172.16.1.8 - - [03/Oct/2020:15:26:02 +0200] "GET /irods-cloud-backend/virtualCollection HTTP/1.1" 200 2304
172.16.1.8 - - [03/Oct/2020:15:40:29 +0200] "GET /irods-cloud-backend/download?path=/idas/home/sysm02/disk-0.vmdk HTTP/1.1" 200 1532646912

Unfortunately, I can't give you more details about how the sfiles are sent to the client because the 2 different web applications have not been created by me (this is one of web applications I’m using https://github.com/irods-contrib/irods-cloud-browser)
I’m sorry, but I’m not a developer.

The file download is always completed and it stops at the same point (1.4GB)


In your opinion, if the problem is related to the 32-bit overflow, is there something that I can do in order to solve this issue.
Since I can’t modify any change to these two applications, I would like to know if I can do something on the other sides.

Many thanks again for your patience.
Mauro








Reply | Threaded
Open this post in threaded view
|

Re: File size truncated at 1.4GB during download from Tomcat WebApp

calder
On Sat, Oct 3, 2020, 09:01 Mauro Tridici <[hidden email]> wrote:

>
>
> On 3 Oct 2020, at 15:14, Christopher Schultz <[hidden email]>
> wrote:
>
> Mauro,
>
> On 10/3/20 08:47, Mauro Tridici wrote
>
> I’m struggling with the problem mentioned in this mail subject.
> When I try to download a 5GB sized file using two different
>
>
 < snip >

In your opinion, if the problem is related to the 32-bit overflow, is there
> something that I can do in order to solve this issue.
> Since I can’t modify any change to these two applications, I would like to
> know if I can do something on the other sides.
>


This issue was reported to the IRODs team, they confirmed it, and pushed a
fix.  You should check your version.

The issue, which is now closed.

https://github.com/irods-contrib/irods-rest/issues/66
Reply | Threaded
Open this post in threaded view
|

Re: File size truncated at 1.4GB during download from Tomcat WebApp

calder
On Sat, Oct 3, 2020, 09:58 calder <[hidden email]> wrote:

> On Sat, Oct 3, 2020, 09:01 Mauro Tridici <[hidden email]> wrote:
>
>>
>>
>> On 3 Oct 2020, at 15:14, Christopher Schultz <
>> [hidden email]> wrote:
>>
>> Mauro,
>>
>> On 10/3/20 08:47, Mauro Tridici wrote
>>
>> I’m struggling with the problem mentioned in this mail subject.
>> When I try to download a 5GB sized file using two different
>>
>>
>  < snip >
>
> In your opinion, if the problem is related to the 32-bit overflow, is
>> there something that I can do in order to solve this issue.
>> Since I can’t modify any change to these two applications, I would like
>> to know if I can do something on the other sides.
>>
>
>
> This issue was reported to the IRODs team, they confirmed it, and pushed a
> fix.  You should check your version.
>
> The issue, which is now closed.
>
> https://github.com/irods-contrib/irods-rest/issues/66
>

I also see your involvement in this issue,  which was marked as a Dup.

https://github.com/irods-contrib/metalnx-web/issues/130

... and updated here
https://github.com/irods-contrib/metalnx-web/issues/143
Reply | Threaded
Open this post in threaded view
|

Re: File size truncated at 1.4GB during download from Tomcat WebApp

Mauro Tridici


> On 3 Oct 2020, at 17:03, calder <[hidden email]> wrote:
>
> On Sat, Oct 3, 2020, 09:58 calder <[hidden email]> wrote:
>
>> On Sat, Oct 3, 2020, 09:01 Mauro Tridici <[hidden email]> wrote:
>>
>>>
>>>
>>> On 3 Oct 2020, at 15:14, Christopher Schultz <
>>> [hidden email]> wrote:
>>>
>>> Mauro,
>>>
>>> On 10/3/20 08:47, Mauro Tridici wrote
>>>
>>> I’m struggling with the problem mentioned in this mail subject.
>>> When I try to download a 5GB sized file using two different
>>>
>>>
>> < snip >
>>
>> In your opinion, if the problem is related to the 32-bit overflow, is
>>> there something that I can do in order to solve this issue.
>>> Since I can’t modify any change to these two applications, I would like
>>> to know if I can do something on the other sides.
>>>
>>
>>
>> This issue was reported to the IRODs team, they confirmed it, and pushed a
>> fix.  You should check your version.
>>
>> The issue, which is now closed.
>>
>> https://github.com/irods-contrib/irods-rest/issues/66
>>
>
> I also see your involvement in this issue,  which was marked as a Dup.
>
> https://github.com/irods-contrib/metalnx-web/issues/130
>
> ... and updated here
> https://github.com/irods-contrib/metalnx-web/issues/143


Hi Calder,

thank you for your help and suggestions.
Unfortunately, I’m still experiencing the same problem despite I’m using the last available version.

Since I used also a different web application (not MetalNX) and I noticed the same problem, I was thinking that there is something to be changed in tomcat configuration.
I will try to start again from scratch.

Mauro

Reply | Threaded
Open this post in threaded view
|

Re: File size truncated at 1.4GB during download from Tomcat WebApp

calder
On Sat, Oct 3, 2020, 11:43 Mauro Tridici <[hidden email]> wrote:

>
> > On 3 Oct 2020, at 17:03, calder <[hidden email]> wrote:
> >
> > On Sat, Oct 3, 2020, 09:58 calder <[hidden email]> wrote:
> >
> >> On Sat, Oct 3, 2020, 09:01 Mauro Tridici <[hidden email]> wrote:
> >>
> >>>
> >>>
> >>> On 3 Oct 2020, at 15:14, Christopher Schultz <
> >>> [hidden email]> wrote:
> >>>
> >>> Mauro,
> >>>
> >>> On 10/3/20 08:47, Mauro Tridici wrote
> >>>
> >>> I’m struggling with the problem mentioned in this mail subject.
> >>> When I try to download a 5GB sized file using two different
> >>>
> >>>
> >> < snip >
> >>
> >> In your opinion, if the problem is related to the 32-bit overflow, is
> >>> there something that I can do in order to solve this issue.
> >>> Since I can’t modify any change to these two applications, I would like
> >>> to know if I can do something on the other sides.
> >>>
> >>
> >>
> >> This issue was reported to the IRODs team, they confirmed it, and
> pushed a
> >> fix.  You should check your version.
> >>
> >> The issue, which is now closed.
> >>
> >> https://github.com/irods-contrib/irods-rest/issues/66
> >>
> >
> > I also see your involvement in this issue,  which was marked as a Dup.
> >
> > https://github.com/irods-contrib/metalnx-web/issues/130
> >
> > ... and updated here
> > https://github.com/irods-contrib/metalnx-web/issues/143
>
>
> Hi Calder,
>
> thank you for your help and suggestions.
> Unfortunately, I’m still experiencing the same problem despite I’m using
> the last available version.
>
> Since I used also a different web application (not MetalNX) and I noticed
> the same problem, I was thinking that there is something to be changed in
> tomcat configuration.
> I will try to start again from scentsratch.
>

Just to be sure ... did you check again to be sure you have the patch
installed?   I ask because two people who had the problem reported back
that the patch fixed the issue for them.

Reading the comments, they did find where an "int" was being used for
"contentLength"  in the code.

>
Reply | Threaded
Open this post in threaded view
|

Re: File size truncated at 1.4GB during download from Tomcat WebApp

Mauro Tridici


> On 3 Oct 2020, at 19:32, calder <[hidden email]> wrote:
>
> On Sat, Oct 3, 2020, 11:43 Mauro Tridici <[hidden email]> wrote:
>
>>
>>> On 3 Oct 2020, at 17:03, calder <[hidden email]> wrote:
>>>
>>> On Sat, Oct 3, 2020, 09:58 calder <[hidden email]> wrote:
>>>
>>>> On Sat, Oct 3, 2020, 09:01 Mauro Tridici <[hidden email]> wrote:
>>>>
>>>>>
>>>>>
>>>>> On 3 Oct 2020, at 15:14, Christopher Schultz <
>>>>> [hidden email]> wrote:
>>>>>
>>>>> Mauro,
>>>>>
>>>>> On 10/3/20 08:47, Mauro Tridici wrote
>>>>>
>>>>> I’m struggling with the problem mentioned in this mail subject.
>>>>> When I try to download a 5GB sized file using two different
>>>>>
>>>>>
>>>> < snip >
>>>>
>>>> In your opinion, if the problem is related to the 32-bit overflow, is
>>>>> there something that I can do in order to solve this issue.
>>>>> Since I can’t modify any change to these two applications, I would like
>>>>> to know if I can do something on the other sides.
>>>>>
>>>>
>>>>
>>>> This issue was reported to the IRODs team, they confirmed it, and
>> pushed a
>>>> fix.  You should check your version.
>>>>
>>>> The issue, which is now closed.
>>>>
>>>> https://github.com/irods-contrib/irods-rest/issues/66
>>>>
>>>
>>> I also see your involvement in this issue,  which was marked as a Dup.
>>>
>>> https://github.com/irods-contrib/metalnx-web/issues/130
>>>
>>> ... and updated here
>>> https://github.com/irods-contrib/metalnx-web/issues/143
>>
>>
>> Hi Calder,
>>
>> thank you for your help and suggestions.
>> Unfortunately, I’m still experiencing the same problem despite I’m using
>> the last available version.
>>
>> Since I used also a different web application (not MetalNX) and I noticed
>> the same problem, I was thinking that there is something to be changed in
>> tomcat configuration.
>> I will try to start again from scentsratch.
>>
>
> Just to be sure ... did you check again to be sure you have the patch
> installed?   I ask because two people who had the problem reported back
> that the patch fixed the issue for them.
>
> Reading the comments, they did find where an "int" was being used for
> "contentLength"  in the code.
>
>>



Yes, Calder, I checked it.
I also changed installation mode: I deployed MetalNX using docker-compose (as indicate in the updated official page)
So, I think that also during installation latest version of MetalNX is used.

Now, my question is: why using two different web applications I obtained the same behaviour?
Is there something to be changed in tomcat configuration? Another user, on this forum, mentioned a “32bit problem”.
My virtual machine has a 64bit OS. I need to change it? Sorry for this stupid question...




Reply | Threaded
Open this post in threaded view
|

Re: File size truncated at 1.4GB during download from Tomcat WebApp

calder
On Sun, Oct 4, 2020, 16:12 Mauro Tridici <[hidden email]> wrote:

>
>
> > On 3 Oct 2020, at 19:32, calder <[hidden email]> wrote:
> >
> > On Sat, Oct 3, 2020, 11:43 Mauro Tridici <[hidden email]> wrote:
> >
> >>
> >>> On 3 Oct 2020, at 17:03, calder <[hidden email]> wrote:
> >>>
> >>> On Sat, Oct 3, 2020, 09:58 calder <[hidden email]> wrote:
> >>>
> >>>> On Sat, Oct 3, 2020, 09:01 Mauro Tridici <[hidden email]>
> wrote:
> >>>>
> >>>>>
> >>>>>
> >>>>> On 3 Oct 2020, at 15:14, Christopher Schultz <
> >>>>> [hidden email]> wrote:
> >>>>>
> >>>>> Mauro,
> >>>>>
> >>>>> On 10/3/20 08:47, Mauro Tridici wrote
> >>>>>
> >>>>> I’m struggling with the problem mentioned in this mail subject.
> >>>>> When I try to download a 5GB sized file using two different
> >>>>>
> >>>>>
> >>>> < snip >
> >>>>
> >>>> In your opinion, if the problem is related to the 32-bit overflow, is
> >>>>> there something that I can do in order to solve this issue.
> >>>>> Since I can’t modify any change to these two applications, I would
> like
> >>>>> to know if I can do something on the other sides.
> >>>>>
> >>>>
> >>>>
> >>>> This issue was reported to the IRODs team, they confirmed it, and
> >> pushed a
> >>>> fix.  You should check your version.
> >>>>
> >>>> The issue, which is now closed.
> >>>>
> >>>> https://github.com/irods-contrib/irods-rest/issues/66
> >>>>
> >>>
> >>> I also see your involvement in this issue,  which was marked as a Dup.
> >>>
> >>> https://github.com/irods-contrib/metalnx-web/issues/130
> >>>
> >>> ... and updated here
> >>> https://github.com/irods-contrib/metalnx-web/issues/143
> >>
> >>
> >> Hi Calder,
> >>
> >> thank you for your help and suggestions.
> >> Unfortunately, I’m still experiencing the same problem despite I’m using
> >> the last available version.
> >>
> >> Since I used also a different web application (not MetalNX) and I
> noticed
> >> the same problem, I was thinking that there is something to be changed
> in
> >> tomcat configuration.
> >> I will try to start again from scentsratch.
> >>
> >
> > Just to be sure ... did you check again to be sure you have the patch
> > installed?   I ask because two people who had the problem reported back
> > that the patch fixed the issue for them.
> >
> > Reading the comments, they did find where an "int" was being used for
> > "contentLength"  in the code.
> >
> >>
>
>
>
> Yes, Calder, I checked it.
> I also changed installation mode: I deployed MetalNX using docker-compose
> (as indicate in the updated official page)
> So, I think that also during installation latest version of MetalNX is
> used.
>
> Now, my question is: why using two different web applications I obtained
> the same behaviour?
> Is there something to be changed in tomcat configuration? Another user, on
> this forum, mentioned a “32bit problem”.
> My virtual machine has a 64bit OS. I need to change it? Sorry for this
> stupid question...
>


The "bit-ness" of the operating system doesn't influence the size (range)
of Java integers ... also, an "int" is the same size in a 32bit JVM and
64bit JVM.

You mention you have two different Java applications running on Tomcat ...
do they both use IRODS?

Or does one application use IRODS and the other application use some other
library?