Standards, specs for URL parameter ordering

[Christopher Schultz-2]

All,

I'm sorry for using users@ as my own personal Google but I'm sure
someone knows this off the top of their head and can save me a lot of
reading.

I'm wondering about which specs mention how to handle URL parameters
(and POST parameters, for that matter) in terms of ordering. For
example, if I have a URL like:

https://example.com/context/resource?a=1&b=2&c=3&a=6

(Note that I have "a" in there twice)

If I call request.getParameterNames(), is there a predictable order in
which those parameters will be handed back? I'd love to hear that not
only are they returned in "URL order" (that is, the left-most parameter
is the first returned in that enumeration) in Tomcat, but either the
servlet spec, the CGI spec, or some other spec dictates that order
explicitly.

Similarly, if I use request.getParameterMap and than iterate through the
keys or Map.Entry objects in it, does that behave predictably as well?

I have a situation where both URL parameters and POST parameters being
in URL-order (or document-order for POST parameters) would be highly
convenient, but I'd like to know if I can actually *rely* on that
behavior, or if I have to make arrangements to provide the ordering in a
separate parameter.

I'd like to know which specs mention these things if they are indeed
specified in any of them.

Thanks in advance,
-chris

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

[Mark Thomas-2]

On 16/02/2021 14:58, Christopher Schultz wrote:

> All,
>
> I'm sorry for using users@ as my own personal Google but I'm sure
> someone knows this off the top of their head and can save me a lot of
> reading.
>
> I'm wondering about which specs mention how to handle URL parameters
> (and POST parameters, for that matter) in terms of ordering. For
> example, if I have a URL like:
>
> https://example.com/context/resource?a=1&b=2&c=3&a=6
>
> (Note that I have "a" in there twice)
>
> If I call request.getParameterNames(), is there a predictable order in
> which those parameters will be handed back? I'd love to hear that not
> only are they returned in "URL order" (that is, the left-most parameter
> is the first returned in that enumeration) in Tomcat, but either the
> servlet spec, the CGI spec, or some other spec dictates that order
> explicitly.

Yes, they will be in that order. (See ApplicationHttpRequest.parameters,
ParameterMap.delgatedMap and LinkedHashMap

The order isn't explicitly defined in any specification I am aware of.
However, the Servlet spec does state (3.1) that query string parameters
should be presented before parameters parsed form the request body.
There are several ways to implement that but it is likely that an
implementation will simply maintain insertion order for keys and values
(where there are multiple values for the same key) in a single
collection. At least, Tomcat does this.

> Similarly, if I use request.getParameterMap and than iterate through the
> keys or Map.Entry objects in it, does that behave predictably as well?

Same answer as above.

> I have a situation where both URL parameters and POST parameters being
> in URL-order (or document-order for POST parameters) would be highly
> convenient, but I'd like to know if I can actually *rely* on that
> behavior, or if I have to make arrangements to provide the ordering in a
> separate parameter.

You can rely on this if you are using Tomcat. I don't see the underlying
design changing anytime soon. For other containers, YMMV.

> I'd like to know which specs mention these things if they are indeed
> specified in any of them.

The only thing I am aware of is the query string before request body
ordering in section 3.1. of the Servlet spec.

Mark

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

[Christopher Schultz-2]

Mark,

On 2/17/21 04:08, Mark Thomas wrote:

> On 16/02/2021 14:58, Christopher Schultz wrote:
>> All,
>>
>> I'm sorry for using users@ as my own personal Google but I'm sure
>> someone knows this off the top of their head and can save me a lot of
>> reading.
>>
>> I'm wondering about which specs mention how to handle URL parameters
>> (and POST parameters, for that matter) in terms of ordering. For
>> example, if I have a URL like:
>>
>> https://example.com/context/resource?a=1&b=2&c=3&a=6
>>
>> (Note that I have "a" in there twice)
>>
>> If I call request.getParameterNames(), is there a predictable order in
>> which those parameters will be handed back? I'd love to hear that not
>> only are they returned in "URL order" (that is, the left-most parameter
>> is the first returned in that enumeration) in Tomcat, but either the
>> servlet spec, the CGI spec, or some other spec dictates that order
>> explicitly.
>
> Yes, they will be in that order. (See ApplicationHttpRequest.parameters,
> ParameterMap.delgatedMap and LinkedHashMap
>
> The order isn't explicitly defined in any specification I am aware of.
> However, the Servlet spec does state (3.1) that query string parameters
> should be presented before parameters parsed form the request body.
> There are several ways to implement that but it is likely that an
> implementation will simply maintain insertion order for keys and values
> (where there are multiple values for the same key) in a single
> collection. At least, Tomcat does this.
>
>> Similarly, if I use request.getParameterMap and than iterate through the
>> keys or Map.Entry objects in it, does that behave predictably as well?
>
> Same answer as above.
>
>> I have a situation where both URL parameters and POST parameters being
>> in URL-order (or document-order for POST parameters) would be highly
>> convenient, but I'd like to know if I can actually *rely* on that
>> behavior, or if I have to make arrangements to provide the ordering in a
>> separate parameter.
>
> You can rely on this if you are using Tomcat. I don't see the underlying
> design changing anytime soon. For other containers, YMMV.
>
>> I'd like to know which specs mention these things if they are indeed
>> specified in any of them.
>
> The only thing I am aware of is the query string before request body
> ordering in section 3.1. of the Servlet spec.

Thanks for all that. Experiments with Tomcat showed that the parameter
order was what I was expecting. I was wondering if that was simply an
implementation detail or if it was spec-mandated. It seems it's an
implementation detail, though it may be a popular one.

I wonder how difficult it would be to lobby the Servlet EG to codify
that behavior in the specification. I'm sure it would be advantageous to
all application developers if they could predict the order of
parameter-processing on the server side given the structure of their
URLs and HTTP POST form data.

My past attempts to get Servlet EG to do anything have been fruitless,
even for simple things[1]. Any idea if this kind of request would ever
actually be considered?

-chris

[1] https://github.com/eclipse-ee4j/servlet-api/issues/130

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

[Mark Thomas-2]

On 17/02/2021 14:03, Christopher Schultz wrote:

> Mark,
>
> On 2/17/21 04:08, Mark Thomas wrote:
>> On 16/02/2021 14:58, Christopher Schultz wrote:
>>> All,
>>>
>>> I'm sorry for using users@ as my own personal Google but I'm sure
>>> someone knows this off the top of their head and can save me a lot of
>>> reading.
>>>
>>> I'm wondering about which specs mention how to handle URL parameters
>>> (and POST parameters, for that matter) in terms of ordering. For
>>> example, if I have a URL like:
>>>
>>> https://example.com/context/resource?a=1&b=2&c=3&a=6
>>>
>>> (Note that I have "a" in there twice)
>>>
>>> If I call request.getParameterNames(), is there a predictable order in
>>> which those parameters will be handed back? I'd love to hear that not
>>> only are they returned in "URL order" (that is, the left-most parameter
>>> is the first returned in that enumeration) in Tomcat, but either the
>>> servlet spec, the CGI spec, or some other spec dictates that order
>>> explicitly.
>>
>> Yes, they will be in that order. (See ApplicationHttpRequest.parameters,
>> ParameterMap.delgatedMap and LinkedHashMap
>>
>> The order isn't explicitly defined in any specification I am aware of.
>> However, the Servlet spec does state (3.1) that query string parameters
>> should be presented before parameters parsed form the request body.
>> There are several ways to implement that but it is likely that an
>> implementation will simply maintain insertion order for keys and values
>> (where there are multiple values for the same key) in a single
>> collection. At least, Tomcat does this.
>>
>>> Similarly, if I use request.getParameterMap and than iterate through the
>>> keys or Map.Entry objects in it, does that behave predictably as well?
>>
>> Same answer as above.
>>
>>> I have a situation where both URL parameters and POST parameters being
>>> in URL-order (or document-order for POST parameters) would be highly
>>> convenient, but I'd like to know if I can actually *rely* on that
>>> behavior, or if I have to make arrangements to provide the ordering in a
>>> separate parameter.
>>
>> You can rely on this if you are using Tomcat. I don't see the underlying
>> design changing anytime soon. For other containers, YMMV.
>>
>>> I'd like to know which specs mention these things if they are indeed
>>> specified in any of them.
>>
>> The only thing I am aware of is the query string before request body
>> ordering in section 3.1. of the Servlet spec.
>
> Thanks for all that. Experiments with Tomcat showed that the parameter
> order was what I was expecting. I was wondering if that was simply an
> implementation detail or if it was spec-mandated. It seems it's an
> implementation detail, though it may be a popular one.
>
> I wonder how difficult it would be to lobby the Servlet EG to codify
> that behavior in the specification. I'm sure it would be advantageous to
> all application developers if they could predict the order of
> parameter-processing on the server side given the structure of their
> URLs and HTTP POST form data.
>
> My past attempts to get Servlet EG to do anything have been fruitless,
> even for simple things[1]. Any idea if this kind of request would ever
> actually be considered?

I don't see why not. Open an issue and we'll see.

> -chris
>
> [1] https://github.com/eclipse-ee4j/servlet-api/issues/130

Things should start improving now we have reached the point in Jakarta
EE where we can evolve the specs and add new functionality. I hopeful
for a LOT of additional clarity and a fair number of small new features
in servlet.next.

Mark

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

[Konstantin Kolinko]

ср, 17 февр. 2021 г. в 12:09, Mark Thomas <[hidden email]>:

>
> On 16/02/2021 14:58, Christopher Schultz wrote:
> > All,
> >
> > I'm sorry for using users@ as my own personal Google but I'm sure
> > someone knows this off the top of their head and can save me a lot of
> > reading.
> >
> > I'm wondering about which specs mention how to handle URL parameters
> > (and POST parameters, for that matter) in terms of ordering. For
> > example, if I have a URL like:
> >
> > https://example.com/context/resource?a=1&b=2&c=3&a=6
> >
> > (Note that I have "a" in there twice)
> >
> > If I call request.getParameterNames(), is there a predictable order in
> > which those parameters will be handed back? I'd love to hear that not
> > only are they returned in "URL order" (that is, the left-most parameter
> > is the first returned in that enumeration) in Tomcat, but either the
> > servlet spec, the CGI spec, or some other spec dictates that order
> > explicitly.
>
> Yes, they will be in that order. (See ApplicationHttpRequest.parameters,
> ParameterMap.delgatedMap and LinkedHashMap
>
> The order isn't explicitly defined in any specification I am aware of.
> However, the Servlet spec does state (3.1) that query string parameters
> should be presented before parameters parsed form the request body.

1. When there are multiple values, the order of values is preserved.

Java Servlet 4.0 spec has an example in its text that shows how the
order of values is preserved (chapter 3.1 "HTTP Protocol Parameters"):

<quote>Data from the query string and the post body are aggregated
into the request parameter set. Query string data is presented before
post body data. For example, if a request is made with a query string
of a=hello and a post body of a=goodbye&a=world, the resulting
parameter set would be ordered a=(hello, goodbye, world)</quote>

2. Original specification for url-encoded parameters is not HTTP, but
HTML specification. The place where I first saw it many years ago was
here:

https://tools.ietf.org/html/rfc1866#section-8.2.1
HTML 2.0 spec

> The fields are listed in the order they appear in the document

Personally, I would not rely on the order of names provided by
getParameterNames(), as it would be surprising for a client if
processing of a request depends on the order of parameter names. In
the same way as whether reordering of input fields on a web form could
break its processing.

Best regards,
Konstantin Kolinko

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

[Christopher Schultz-2]

All,

On 2/17/21 09:22, Mark Thomas wrote:

> On 17/02/2021 14:03, Christopher Schultz wrote:
>> Mark,
>>
>> On 2/17/21 04:08, Mark Thomas wrote:
>>> On 16/02/2021 14:58, Christopher Schultz wrote:
>>>> All,
>>>>
>>>> I'm sorry for using users@ as my own personal Google but I'm sure
>>>> someone knows this off the top of their head and can save me a lot of
>>>> reading.
>>>>
>>>> I'm wondering about which specs mention how to handle URL parameters
>>>> (and POST parameters, for that matter) in terms of ordering. For
>>>> example, if I have a URL like:
>>>>
>>>> https://example.com/context/resource?a=1&b=2&c=3&a=6
>>>>
>>>> (Note that I have "a" in there twice)
>>>>
>>>> If I call request.getParameterNames(), is there a predictable order in
>>>> which those parameters will be handed back? I'd love to hear that not
>>>> only are they returned in "URL order" (that is, the left-most parameter
>>>> is the first returned in that enumeration) in Tomcat, but either the
>>>> servlet spec, the CGI spec, or some other spec dictates that order
>>>> explicitly.
>>>
>>> Yes, they will be in that order. (See ApplicationHttpRequest.parameters,
>>> ParameterMap.delgatedMap and LinkedHashMap
>>>
>>> The order isn't explicitly defined in any specification I am aware of.
>>> However, the Servlet spec does state (3.1) that query string parameters
>>> should be presented before parameters parsed form the request body.
>>> There are several ways to implement that but it is likely that an
>>> implementation will simply maintain insertion order for keys and values
>>> (where there are multiple values for the same key) in a single
>>> collection. At least, Tomcat does this.
>>>
>>>> Similarly, if I use request.getParameterMap and than iterate through the
>>>> keys or Map.Entry objects in it, does that behave predictably as well?
>>>
>>> Same answer as above.
>>>
>>>> I have a situation where both URL parameters and POST parameters being
>>>> in URL-order (or document-order for POST parameters) would be highly
>>>> convenient, but I'd like to know if I can actually *rely* on that
>>>> behavior, or if I have to make arrangements to provide the ordering in a
>>>> separate parameter.
>>>
>>> You can rely on this if you are using Tomcat. I don't see the underlying
>>> design changing anytime soon. For other containers, YMMV.
>>>
>>>> I'd like to know which specs mention these things if they are indeed
>>>> specified in any of them.
>>>
>>> The only thing I am aware of is the query string before request body
>>> ordering in section 3.1. of the Servlet spec.
>>
>> Thanks for all that. Experiments with Tomcat showed that the parameter
>> order was what I was expecting. I was wondering if that was simply an
>> implementation detail or if it was spec-mandated. It seems it's an
>> implementation detail, though it may be a popular one.
>>
>> I wonder how difficult it would be to lobby the Servlet EG to codify
>> that behavior in the specification. I'm sure it would be advantageous to
>> all application developers if they could predict the order of
>> parameter-processing on the server side given the structure of their
>> URLs and HTTP POST form data.
>>
>> My past attempts to get Servlet EG to do anything have been fruitless,
>> even for simple things[1]. Any idea if this kind of request would ever
>> actually be considered?
>
> I don't see why not. Open an issue and we'll see.
>
>> -chris
>>
>> [1] https://github.com/eclipse-ee4j/servlet-api/issues/130
>
> Things should start improving now we have reached the point in Jakarta
> EE where we can evolve the specs and add new functionality. I hopeful
> for a LOT of additional clarity and a fair number of small new features
> in servlet.next.

Filed: https://github.com/eclipse-ee4j/servlet-api/issues/393

Any support from the Tomcat community would be greatly appreciated.

-chris

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

[Christopher Schultz-2]

Konstantin,

On 2/17/21 10:35, Konstantin Kolinko wrote:

> ср, 17 февр. 2021 г. в 12:09, Mark Thomas <[hidden email]>:
>>
>> On 16/02/2021 14:58, Christopher Schultz wrote:
>>> All,
>>>
>>> I'm sorry for using users@ as my own personal Google but I'm sure
>>> someone knows this off the top of their head and can save me a lot of
>>> reading.
>>>
>>> I'm wondering about which specs mention how to handle URL parameters
>>> (and POST parameters, for that matter) in terms of ordering. For
>>> example, if I have a URL like:
>>>
>>> https://example.com/context/resource?a=1&b=2&c=3&a=6
>>>
>>> (Note that I have "a" in there twice)
>>>
>>> If I call request.getParameterNames(), is there a predictable order in
>>> which those parameters will be handed back? I'd love to hear that not
>>> only are they returned in "URL order" (that is, the left-most parameter
>>> is the first returned in that enumeration) in Tomcat, but either the
>>> servlet spec, the CGI spec, or some other spec dictates that order
>>> explicitly.
>>
>> Yes, they will be in that order. (See ApplicationHttpRequest.parameters,
>> ParameterMap.delgatedMap and LinkedHashMap
>>
>> The order isn't explicitly defined in any specification I am aware of.
>> However, the Servlet spec does state (3.1) that query string parameters
>> should be presented before parameters parsed form the request body.
>
> 1. When there are multiple values, the order of values is preserved.
>
> Java Servlet 4.0 spec has an example in its text that shows how the
> order of values is preserved (chapter 3.1 "HTTP Protocol Parameters"):
>
> <quote>Data from the query string and the post body are aggregated
> into the request parameter set. Query string data is presented before
> post body data. For example, if a request is made with a query string
> of a=hello and a post body of a=goodbye&a=world, the resulting
> parameter set would be ordered a=(hello, goodbye, world)</quote>

Yes, that's the section Mark was referencing. While the example may be
instructive, it is not explicit. I'm just requesting explicit language
in the spec for that assumption.

> 2. Original specification for url-encoded parameters is not HTTP, but
> HTML specification. The place where I first saw it many years ago was
> here:
>
> https://tools.ietf.org/html/rfc1866#section-8.2.1
> HTML 2.0 spec
>
>> The fields are listed in the order they appear in the document

+1

> Personally, I would not rely on the order of names provided by
> getParameterNames(), as it would be surprising for a client if
> processing of a request depends on the order of parameter names. In
> the same way as whether reordering of input fields on a web form could
> break its processing.

I disagree: both HTML and HTTP make it clear that form-data ordering is
important and should be preserved. Given the importance placed on that
ordering, I think that same importance and reliable ordering should be
extended explicitly into the Servlet Specification explicitly.

In my use-case, I have a series of questions being presented on the
screen. When I receive the answers to those questions, I'd like to be
able to rely on the form-data ordering to know which questions were
presented first and which ones later when they are all on the same form.

Without being able to rely on form-data ordering, the only way to "know"
which question was asked first, second, etc. would be to either put a
hidden form element on the page describing the question-order (back to
the server) or to have individual hidden elements for each question
specifying the order. Something like this:

   <input type="hidden" name="question_3762_order" value="1" />
   <input type="hidden" name="question_213_order" value="2" />
   ...

If the ordering can be reliable, less data needs to be exchanged.

I have presented a justification for this in the issue filed for the
Servlet API[1] and I'm happy to hear any critique you may have of that
justification.

For emphasis:
> Personally, I would not rely on the order of names provided by
> getParameterNames()
It is precisely because of the ambiguity of the specification that I am
requesting clarification and, further, explicit definitive required
behavior of the Servlet Specification such that application developers
*can* rely on that ordering.

-chris

[1] https://github.com/eclipse-ee4j/servlet-api/issues/393

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

[Konstantin Kolinko]

ср, 17 февр. 2021 г. в 20:05, Christopher Schultz
<[hidden email]>:

>
> Konstantin,
>
> [...]
>
> Without being able to rely on form-data ordering, the only way to "know"
> which question was asked first, second, etc. would be to either put a
> hidden form element on the page describing the question-order (back to
> the server) or to have individual hidden elements for each question
> specifying the order. Something like this:
>
>    <input type="hidden" name="question_3762_order" value="1" />
>    <input type="hidden" name="question_213_order" value="2" />
>    ...

An alternative solution can be to include both question id and order
in the field name. E.g.

<input name="answer_1_3762" placeholder="Answer for Q3762">
<input name="answer_2_213" placeholder="Answer for Q213">

Well, your use case is a valid one.

BTW, for the same grounds for forms submitted with mime type of
"multipart/form-data"
the HttpServletRequest.getParts() method could return an ordered
collection (a List).
As of now, the API is specified to return a Collection<Part> [1].

[1] https://javaee.github.io/javaee-spec/javadocs/javax/servlet/http/HttpServletRequest.html#getParts--

Best regards,
Konstantin Kolinko

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]