[Bug 64762] New: CoyoteInputStream asynchronous read

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

[Bug 64762] New: CoyoteInputStream asynchronous read

Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=64762

            Bug ID: 64762
           Summary: CoyoteInputStream asynchronous read
           Product: Tomcat 9
           Version: 9.0.31
          Hardware: PC
                OS: Linux
            Status: NEW
          Severity: regression
          Priority: P2
         Component: Connectors
          Assignee: [hidden email]
          Reporter: [hidden email]
  Target Milestone: -----

Hi everyone

Since the upgrade from Tomcat 8 to tomcat 9 The
CoyoteInputStream.getInputStream() does not work properly anymore.

We run Debian 10 buster and Tomcat has the following ServerInfo:
Server version: Apache Tomcat/9.0.31 (Debian)
Server built:   Jul 15 2020 11:43:33 UTC
Server number:  9.0.31.0
OS Name:        Linux
OS Version:     4.19.0-10-amd64
Architecture:   amd64
JVM Version:    1.8.0_261-b12
JVM Vendor:     Oracle Corporation

When reading/uploading  larger files (i.e. a HTTPS form-post) it looks like not
everything is copied as it should, we get the following Stack
Trace:
org.apache.commons.fileupload.MultipartStream$MalformedStreamException: Stream
ended unexpectedly
        at
org.apache.commons.fileupload.MultipartStream$ItemInputStream.makeAvailable(MultipartStream.java:1033)
        at
org.apache.commons.fileupload.MultipartStream$ItemInputStream.read(MultipartStream.java:931)
        at java.io.InputStream.read(InputStream.java:101)
        at org.apache.commons.fileupload.util.Streams.copy(Streams.java:98)
        at org.apache.commons.fileupload.util.Streams.copy(Streams.java:68)
        at  …
        at org.apache.commons.chain.impl.ChainBase.execute(ChainBase.java:191)
        at
org.apache.commons.chain.generic.LookupCommand.execute(LookupCommand.java:305)
        at org.apache.commons.chain.impl.ChainBase.execute(ChainBase.java:191)
        at …
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:660)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
        at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
        at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at
org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
        at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at
org.apache.catalina.filters.CorsFilter.handleNonCORS(CorsFilter.java:352)
        at org.apache.catalina.filters.CorsFilter.doFilter(CorsFilter.java:171)

        at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at
org.apache.catalina.filters.ExpiresFilter.doFilter(ExpiresFilter.java:1227)

        at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
        at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
        at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541)
        at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
        at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
        at
org.apache.catalina.valves.SemaphoreValve.invoke(SemaphoreValve.java:160)
        at
com.privasphere.net.SemaphoreValve5340.invoke(SemaphoreValve5340.java:43)
        at
org.apache.catalina.valves.CrawlerSessionManagerValve.invoke(CrawlerSessionManagerValve.java:235)
        at
com.privasphere.net.CrawlerSessionValve5340.invoke(CrawlerSessionValve5340.java:50)
        at
org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:688)
        at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
        at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
        at
org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:367)
        at
org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
        at
org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)
        at
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1639)
        at
org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:748)


Our (surprising) workaround for now is as follows:. We look at how many bytes
are expected by request.getContentLength() and if what is read does not match
the expected size sleep some milli seconds. We do this in a while loop which is
iterated a maximal number of times or until everything is read. Normally this
solves the problem. So for example for an upload of 230 K, the inputstream
claims done after ~130K, after a few ms wait, another 100K appear and all works
fine.

Could you please have a look into that bug?

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 64762] CoyoteInputStream asynchronous read (wait after premature end and the rest comes)

Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=64762

Ralf Hauser <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
            Summary|CoyoteInputStream           |CoyoteInputStream
                   |asynchronous read           |asynchronous read (wait
                   |                            |after premature end and the
                   |                            |rest comes)
                 CC|                            |[hidden email]

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 64762] CoyoteInputStream asynchronous read (wait after premature end and the rest comes)

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=64762

--- Comment #1 from Ralf Hauser <[hidden email]> ---
The stream is read/copied with

org.apache.commons.io.IOUtils.copy()

In combination with Axis2, the soap client side sees

"Unexpected EOF in prolog"  

(there we haven't found the symptom fighting workaround yet, but we rather want
to *really solve it*)

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 64762] CoyoteInputStream asynchronous read (wait after premature end and the rest comes)

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=64762

--- Comment #2 from Christopher Schultz <[hidden email]> ---
Looks like you are doing multipart/form-data processing, as handled by
commons-fileupload. Is that using the Tomcat-provided Servlet-3.0 file-upload,
or are you using a different mechanism?

I would guess it's Tomcat, but Tomcat doesn't use commons-chain, so I'm
wondering what's happening, here.

You say this is an "asynchronous read" but you are using an InputStream, which
is a blocking stream. What do you mean by "asynchronous read"?

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 64762] CoyoteInputStream getInputStream() read (wait after premature end and the rest comes)

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=64762

Ralf Hauser <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
            Summary|CoyoteInputStream           |CoyoteInputStream
                   |asynchronous read (wait     |getInputStream() read (wait
                   |after premature end and the |after premature end and the
                   |rest comes)                 |rest comes)

--- Comment #3 from Ralf Hauser <[hidden email]> ---
No, we are not using "Tomcat-provided Servlet-3.0 file-upload" but
commons-fileupload-1.4.jar with launched by some legacy struts

"asynchronous" may be confusing - it is blocking, but after one thinks it is
done, more appears in the inputstream

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 64762] CoyoteInputStream getInputStream() read (wait after premature end and the rest comes)

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=64762

--- Comment #4 from Christopher Schultz <[hidden email]> ---
(In reply to Ralf Hauser from comment #3)
> No, we are not using "Tomcat-provided Servlet-3.0 file-upload" but
> commons-fileupload-1.4.jar with launched by some legacy struts

Okay, so Struts is handling the file-upload. I suspected as much because of the
use of commons-chain.

Is your code being used *at all* in this interaction, or is Struts handling the
file upload completely by itself.

> "asynchronous" may be confusing - it is blocking, but after one thinks it is
> done, more appears in the inputstream

It would be strange for an InputStream to return -1 from a read and then later
allow more reading. Is that really what you are observing?

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 64762] CoyoteInputStream getInputStream() read (wait after premature end and the rest comes)

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=64762

--- Comment #5 from mgrigorov <[hidden email]> ---
Can you reproduce this without Struts ? I.e. with plain Servlet

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 64762] CoyoteInputStream getInputStream() read (wait after premature end and the rest comes)

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=64762

--- Comment #6 from Mark Thomas <[hidden email]> ---
Whether it requires Struts or not, a *minimal* WAR (with source) that allows us
to reproduce this would be very helpful.

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 64762] CoyoteInputStream getInputStream() read (wait after premature end and the rest comes)

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=64762

--- Comment #7 from Ralf Hauser <[hidden email]> ---
(In reply to mgrigorov from comment #5)
> Can you reproduce this without Struts ? I.e. with plain Servlet

Yes, it happens also with the Axis2 Servlet where there is no struts
in-between.

As the CoyoteInputStream.markSupported() returns false from the parent
java.io.InputStream.java this only happens upon the first read of the stream

We'll provide more insights and possible some code once we progress further -
many thanks for the quick responses

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 64762] CoyoteInputStream getInputStream() read (wait after premature end and the rest comes)

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=64762

Mark Thomas <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |NEEDINFO

--- Comment #8 from Mark Thomas <[hidden email]> ---
Also, make sure you test against the latest 9.0.x release. There are a couple
of fixes that might relate to this since 9.0.31.

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]