Quantcast

[Bug 60594] New: RFC 7230/3986 url requirement that prevents unencoded curly braces should be optional, since it breaks existing sites

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
21 messages Options
12
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[Bug 60594] New: RFC 7230/3986 url requirement that prevents unencoded curly braces should be optional, since it breaks existing sites

Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=60594

            Bug ID: 60594
           Summary: RFC 7230/3986 url requirement that prevents unencoded
                    curly braces should be optional, since it breaks
                    existing sites
           Product: Tomcat 7
           Version: 7.0.73
          Hardware: All
                OS: All
            Status: NEW
          Severity: regression
          Priority: P2
         Component: Connectors
          Assignee: [hidden email]
          Reporter: [hidden email]
  Target Milestone: ---

Using the protocol="HTTP/1.1" connector (Coyote)

After upgrading a site to Tomcat 7.0.73 from 7.0.72 or from anything earlier, a
url with an unencoded { or } (ie. http://my.com?filter={"search":"isvalid"} ),
now returns a 400 error code and logs the following error message:

"INFO: Error parsing HTTP request header
 Note: further occurrences of HTTP header parsing errors will be logged at
DEBUG level.
java.lang.IllegalArgumentException: Invalid character found in the request
target. The valid characters are defined in RFC 7230 and RFC 3986"

Resolution:
Since this is a breaking change (aka regression failure), there should be an
option to override and turn this off (still reporting the first occurrence as
shown above), so that any existing site which experiences this can choose to
ignore this failure and continue as before, so they can deal with changing
their application at a later date, if they deem the security risk is
appropriate.

Defaulting the option to true (to enable the check) is perfectly fine, as long
as there is an option in a server and/or application config file to disable it,
and proper documentation on it.

Either this, or you clearly state in the release notes of 7.0.73, exactly what
will break, and recommend that users do not perform the Tomcat update, if they
are not ready to change their applications to comply, but I think this would
open up an even bigger can of worms.

Instead of just saying:
"Add additional checks for valid characters to the HTTP request line parsing so
invalid request lines are rejected sooner. (markt)" - this tells us nothing
about the impending doom we may face.

But, I would recommend just giving us the option to decide for ourselves.

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[Bug 60594] RFC 7230/3986 url requirement that prevents unencoded curly braces should be optional, since it breaks existing sites

Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=60594

Mark Thomas <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Severity|regression                  |enhancement

--- Comment #1 from Mark Thomas <[hidden email]> ---
Given that using an unencoded '{' or '}' in a URL is contrary to the RFCs and
that the fix that tightened the validation rules was in response to a security
vulnerability (CVE-2016-6816) I think it is unlikely that an option will be
introduced to make this validation optional.

It is quite likely that some sites could safely tolerate some characters.
However, it is also likely that the 'safe' set of invalid characters will vary
from site to site. That would therefore require a more complex configuration
option than simply allowing or disallowing a fixed set of characters.

Those interested in proposing a patch should look at lines 74-78 of
org.apache.tomcat.util.http.parser.HttpParser although I'll repeat I think it
is unlikely such a patch would be accepted.

All that code is static which means configuration via system properties -
something I'd prefer to see less of rather than more of in Tomcat.

For completeness, '|' seems to be another character that is fairly widely used
in unecoded form when it should be encoded.

Finally, changes related conformance to the relevant RFCs and Java EE
specifications are not treated as regressions. Therefore, I have moved this to
an enhancement request.

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[Bug 60594] RFC 7230/3986 url requirement that prevents unencoded curly braces should be optional, since it breaks existing sites

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=60594

Remy Maucherat <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[hidden email]

--- Comment #2 from Remy Maucherat <[hidden email]> ---
*** Bug 60616 has been marked as a duplicate of this bug. ***

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[Bug 60594] RFC 7230/3986 url requirement that prevents unencoded curly braces should be optional, since it breaks existing sites

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=60594

--- Comment #3 from Coty Sutherland <[hidden email]> ---
Created attachment 34684
  --> https://bz.apache.org/bugzilla/attachment.cgi?id=34684&action=edit
patch proposal

In response to the numerous complaints on the users list I decided to give this
a shot. I added a system property which contains a blacklist that's used for
validation of request targets rather than the long if statement that was there.
If a users needs to allow unencoded | characters then they can just remove it
from the blacklist defined in the tomcat.util.http.parser.HttpParser.blacklist
property.

If this looks good to everyone I can push it to whichever versions of tomcat we
want to allow an option for.

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[Bug 60594] RFC 7230/3986 url requirement that prevents unencoded curly braces should be optional, since it breaks existing sites

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=60594

--- Comment #4 from Mark Thomas <[hidden email]> ---
Allowing some of those (e.g. space) is extremely dangerous and should not be
allowed under any circumstances.

I generally dislike configuration via system property. That said, making this
per Connector will be significantly more invasive.

Any proposed patch needs to include documentation. That documentation needs to
include a very large, very clear warning the deviating from the default is a
security risk.

If this feature is implemented, I'd prefer to see the option to allow illegal
characters limited to a much smaller sub-set.

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[Bug 60594] RFC 7230/3986 url requirement that prevents unencoded curly braces should be optional, since it breaks existing sites

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=60594

--- Comment #5 from Coty Sutherland <[hidden email]> ---
(In reply to Mark Thomas from comment #4)
> I generally dislike configuration via system property. That said, making
> this per Connector will be significantly more invasive.

I agree on both points. The system property seemed to be the least invasive way
to achieve the desired result.

> Any proposed patch needs to include documentation. That documentation needs
> to include a very large, very clear warning the deviating from the default
> is a security risk.

Also agreed. Where would that documentation go?

> If this feature is implemented, I'd prefer to see the option to allow
> illegal characters limited to a much smaller sub-set.

Other than space, which characters should absolutely be excluded in all cases?
I can create a secondary list containing those and programmatically add them if
a user tries to remove them from the blacklist.

Also, my initial patch used a whitelist instead of a blacklist so that the
system property was either commented out by default, or contained a few
characters that were the exception to the rule. I inversed it to a blacklist to
remove some logic and make it perform better; do you think that a whitelist
would work better here? I can provide that patch also.

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[Bug 60594] RFC 7230/3986 url requirement that prevents unencoded curly braces should be optional, since it breaks existing sites

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=60594

--- Comment #6 from eolivelli <[hidden email]> ---
Hi, for my use cases I would like to have just a whitelist and let Tomcat
handle all the RFC blacklisted chars automatically. In my case I had to
whitelist curly braces and pipe.

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[Bug 60594] RFC 7230/3986 url requirement that prevents unencoded curly braces should be optional, since it breaks existing sites

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=60594

--- Comment #7 from Coty Sutherland <[hidden email]> ---
Created attachment 34687
  --> https://bz.apache.org/bugzilla/attachment.cgi?id=34687&action=edit
whitelist patch proposal

For reference, and so I don't accidentally delete it :)

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[Bug 60594] RFC 7230/3986 url requirement that prevents unencoded curly braces should be optional, since it breaks existing sites

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=60594

--- Comment #8 from Mark Thomas <[hidden email]> ---
I think I prefer the whitelist option but I'd like to see it limited to - at
this point - '{', '}' and '|'. Other characters can be considered on a case by
case basis.

Documentation should go in the system properties section of the config docs
although I'm still mulling over what a Connector config option might look like.

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[Bug 60594] RFC 7230/3986 url requirement that prevents unencoded curly braces should be optional, since it breaks existing sites

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=60594

--- Comment #9 from Coty Sutherland <[hidden email]> ---
Created attachment 34694
  --> https://bz.apache.org/bugzilla/attachment.cgi?id=34694&action=edit
whitelist proposal limiting characters with docs

OK, here's an updated whitelist patch restricting the characters that are
accepted to '{', '}', and '|'. I also included documentation for the property.

Let me know if that works better for you :)

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[Bug 60594] RFC 7230/3986 url requirement that prevents unencoded curly braces should be optional, since it breaks existing sites

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=60594

--- Comment #10 from Mark Thomas <[hidden email]> ---
Thanks for the updated patch. I like the overall design. Some detail comments:
- I think a different name is required. We might want to override other
restrictions in the future. Maybe requestTargetAllow
- The docs need to state which characters are valid in the allowed list
- What to do if some other invalid character is placed on the allowed list. Log
a warning?
- I'm still undecided on whether this should be per connector configuration

We also need to decide which versions to add this to. I currently thinking:
- 7.0.x - yes
- 8.0.x - yes
- 8.5.x - maybe
- 9.0.x - no

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[Bug 60594] RFC 7230/3986 url requirement that prevents unencoded curly braces should be optional, since it breaks existing sites

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=60594

--- Comment #11 from eolivelli <[hidden email]> ---
Please fix it in Tomcat 8.5.X too

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[Bug 60594] RFC 7230/3986 url requirement that prevents unencoded curly braces should be optional, since it breaks existing sites

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=60594

Coty Sutherland <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
  Attachment #34684|0                           |1
        is obsolete|                            |
  Attachment #34687|0                           |1
        is obsolete|                            |
  Attachment #34694|0                           |1
        is obsolete|                            |

--- Comment #12 from Coty Sutherland <[hidden email]> ---
Created attachment 34698
  --> https://bz.apache.org/bugzilla/attachment.cgi?id=34698&action=edit
Updated patch proposal including a warning message for characters that aren't
allowed

(In reply to Mark Thomas from comment #10)
> Thanks for the updated patch. I like the overall design. Some detail
> comments:

No problem.

> - I think a different name is required. We might want to override other
> restrictions in the future. Maybe requestTargetAllow

That makes sense.

> - The docs need to state which characters are valid in the allowed list

Agreed.

> - What to do if some other invalid character is placed on the allowed list.
> Log a warning?

I thought about that but since there isn't any logging there at the moment I
let it go. I think it's a good idea to log a warning though, so I'll add that.

> - I'm still undecided on whether this should be per connector configuration

That would be nice, but I haven't dug into the code enough to be able to
quickly provide a patch for it.

> We also need to decide which versions to add this to. I currently thinking:
> - 7.0.x - yes
> - 8.0.x - yes

+1

> - 8.5.x - maybe

I'd vote yes on adding the option to 8.5.x because the stable version is
already out and the behavior has changed. We obviously don't want to continue
allowing broken clients to work, but I don't think we can change this behavior
in a stable version, as evidenced by the users list complaints :)

> - 9.0.x - no

+1

I also noticed that the property being parsed was including the quotes, so I
changed the commented out example accordingly.

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[Bug 60594] RFC 7230/3986 url requirement that prevents unencoded curly braces should be optional, since it breaks existing sites

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=60594

--- Comment #13 from eolivelli <[hidden email]> ---
Coty, the patch looks good to me, can you please add the following chars to the
list of allowed characters ?

'\"' (double quote)
'#' (sharp)
'<' (left angle bracket)
'>' (right angle bracket)
'\\' (backslash)
'^' (accent)
'`' (accent)

I think in some case I would need the "space" too, but Mark remarked that is
would be very dangerous

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[Bug 60594] RFC 7230/3986 url requirement that prevents unencoded curly braces should be optional, since it breaks existing sites

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=60594

--- Comment #14 from Mark Thomas <[hidden email]> ---
You need to make a case for each of those to added to the potentially allowed
list. Without any such justification, I am -1 on expanding it beyond the
current three allowed characters.

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[Bug 60594] RFC 7230/3986 url requirement that prevents unencoded curly braces should be optional, since it breaks existing sites

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=60594

--- Comment #15 from eolivelli <[hidden email]> ---
OK Mark at this moment I'm running a patch in production to make all the
characters allowed.

I have evidence only on troubles for curly braches and pipe characters so the
patch looks good for me.

I will wait for the release of this patch in an official 8.5.x Tomcat version
and deploy it to production.

In case I need further characters a will create a new issue


Thank you

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[Bug 60594] RFC 7230/3986 url requirement that prevents unencoded curly braces should be optional, since it breaks existing sites

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=60594

--- Comment #16 from Remy Maucherat <[hidden email]> ---
-1 as well for any additional characters. People who are that desperate to run
into trouble can patch Tomcat easily.

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[Bug 60594] RFC 7230/3986 url requirement that prevents unencoded curly braces should be optional, since it breaks existing sites

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=60594

--- Comment #17 from Coty Sutherland <[hidden email]> ---
OK, cool. So unless someone else objects to the patch as-is, I'll commit it to
7.0.x - 8.5.x shortly.

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[Bug 60594] RFC 7230/3986 url requirement that prevents unencoded curly braces should be optional, since it breaks existing sites

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=60594

Coty Sutherland <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Resolution|---                         |FIXED
             Status|NEW                         |RESOLVED

--- Comment #18 from Coty Sutherland <[hidden email]> ---
Fixed in:

- 8.5.x for 8.5.12 onwards
- 8.0.x for 8.0.42 onwards
- 7.0.x for 7.0.76 onwards

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[Bug 60594] RFC 7230/3986 url requirement that prevents unencoded curly braces should be optional, since it breaks existing sites

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=60594

--- Comment #19 from Lulseged Zerfu <[hidden email]> ---
Hi

 We have found that we have problems with some characters that are not allowed
in request URI and would like to know if any filter or valve can be applied to
encode until clients get updated instead of responding with 400 Bad Request.

 We have millions of clients (both android and ios) that needs to follow the
RFC but it will take time but until then there must be some work around that
can be used.

BR
Lulseged

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

12
Loading...