Working tc native build

classic Classic list List threaded Threaded
18 messages Options
Reply | Threaded
Open this post in threaded view
|

Working tc native build

markt
With some additional information from Mladen regrading the build tools
to use, I now have a working build environment for the Tomcat Native
connector binaries. This is documented at:
http://wiki.apache.org/tomcat/BuildTcNativeWin

What I don't have, yet, is the ability to reproduce the entire tcnative
release (mainly because I haven't tried yet).

Clearly we need a 1.1.31 release ASAP. The key question at this point is
for Mladen: Are you in a position to roll a 1.1.31 release and if so in
what timescale?

Until we have an answer to that question I intend to start on trying to
create a full 1.1.31 release locally.

It would be good if others were able to recreate the tcnative build as
well to validate the steps on the wiki.

Mark

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Working tc native build

markt
On 01/07/2014 16:04, Mark Thomas wrote:
> With some additional information from Mladen regrading the build tools
> to use, I now have a working build environment for the Tomcat Native
> connector binaries. This is documented at:
> http://wiki.apache.org/tomcat/BuildTcNativeWin

(excluding a working I64 build that I am still looking at)

> What I don't have, yet, is the ability to reproduce the entire tcnative
> release (mainly because I haven't tried yet).
>
> Clearly we need a 1.1.31 release ASAP. The key question at this point is
> for Mladen: Are you in a position to roll a 1.1.31 release and if so in
> what timescale?
>
> Until we have an answer to that question I intend to start on trying to
> create a full 1.1.31 release locally.
>
> It would be good if others were able to recreate the tcnative build as
> well to validate the steps on the wiki.
>
> Mark
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Working tc native build

mturk
In reply to this post by markt
On 07/01/2014 05:04 PM, Mark Thomas wrote:

> With some additional information from Mladen regrading the build tools
> to use, I now have a working build environment for the Tomcat Native
> connector binaries. This is documented at:
> http://wiki.apache.org/tomcat/BuildTcNativeWin
>
> What I don't have, yet, is the ability to reproduce the entire tcnative
> release (mainly because I haven't tried yet).
>
> Clearly we need a 1.1.31 release ASAP. The key question at this point is
> for Mladen: Are you in a position to roll a 1.1.31 release and if so in
> what timescale?
>

Yes I am. I can do that by tomorrow.


Regards
--
^TM

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Working tc native build

markt
On 07/07/2014 07:52, Mladen Turk wrote:

> On 07/01/2014 05:04 PM, Mark Thomas wrote:
>> With some additional information from Mladen regrading the build tools
>> to use, I now have a working build environment for the Tomcat Native
>> connector binaries. This is documented at:
>> http://wiki.apache.org/tomcat/BuildTcNativeWin
>>
>> What I don't have, yet, is the ability to reproduce the entire tcnative
>> release (mainly because I haven't tried yet).
>>
>> Clearly we need a 1.1.31 release ASAP. The key question at this point is
>> for Mladen: Are you in a position to roll a 1.1.31 release and if so in
>> what timescale?
>>
>
> Yes I am. I can do that by tomorrow.

I'm guessing you are catching up on e-mail. I rolled a 1.1.31 release in
the middle of last week and just announced the result of the release vote.

It is probably worth you casting you eye over the release instructions
[1] and the release itself to see what I missed / got wrong. Konstantin
spotted a few things and there are probably more. If there is anything
drastically wrong, we can always roll 1.1.32.

Cheers,

Mark

[1] https://wiki.apache.org/tomcat/BuildTcNativeWin

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Working tc native build

mturk
On 07/07/2014 10:30 AM, Mark Thomas wrote:

> On 07/07/2014 07:52, Mladen Turk wrote:
>> On 07/01/2014 05:04 PM, Mark Thomas wrote:
>>> With some additional information from Mladen regrading the build tools
>>> to use, I now have a working build environment for the Tomcat Native
>>> connector binaries. This is documented at:
>>> http://wiki.apache.org/tomcat/BuildTcNativeWin
>>>
>>> What I don't have, yet, is the ability to reproduce the entire tcnative
>>> release (mainly because I haven't tried yet).
>>>
>>> Clearly we need a 1.1.31 release ASAP. The key question at this point is
>>> for Mladen: Are you in a position to roll a 1.1.31 release and if so in
>>> what timescale?
>>>
>>
>> Yes I am. I can do that by tomorrow.
>
> I'm guessing you are catching up on e-mail. I rolled a 1.1.31 release in
> the middle of last week and just announced the result of the release vote.

Yeah, sorry. I got lost in all those mail.

>
> It is probably worth you casting you eye over the release instructions
> [1] and the release itself to see what I missed / got wrong. Konstantin
> spotted a few things and there are probably more. If there is anything
> drastically wrong, we can always roll 1.1.32.
>

Sure.


Cheers
--
^TM

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Working tc native build

Christopher Schultz-2
In reply to this post by markt
Mark,

On 7/1/14, 11:44 AM, Mark Thomas wrote:
> On 01/07/2014 16:04, Mark Thomas wrote:
>> With some additional information from Mladen regrading the build tools
>> to use, I now have a working build environment for the Tomcat Native
>> connector binaries. This is documented at:
>> http://wiki.apache.org/tomcat/BuildTcNativeWin
>
> (excluding a working I64 build that I am still looking at)

I just glanced through the current contents of BuildTcNativeWin on the
wiki, and I can still see this comment:

>> IA64 OpenSSL build failed. Commenting out destest from ms\nt.mak
worked around the issue.

The problem is that the tests require a compatible supporting
architecture. That is, if you cross-compile to IA64 then try to run the
tests, the tests will fail because you are running on x86_84 and IA64 is
not available. The same thing will happen if you try to build/test on
ia32: you can produce x86_64 binaries, but you will be unable to test them.

Therefore, it would be safest to specify in the instructions that either
x86_64 or IA64 should be used for the build -- that way, at least one of
the 64-bit architectures can be tested along with the 32-bit one. It's
also worth mentioning that a 32-bit environment would need to
comment-out *all* the 64-bit tests, and not just the IA64 ones.

Thanks for continuing to work on this, Mark. I'm glad you've been able
to remove Cygwin as a requirement... looking at your initial build
instructions seemed overly complicated -- or at least had many more
requirements than strictly necessary. I'm not sure how much of Mladen's
magic environment is strictly necessary, either: I think a lot of it
could be put into scripts that ship with the tcnative source distribution.

I am still going to continue to work on getting all this stuff to build
using Microsoft Visual Studio Express: it should be possible to do, and
I *have* been able to get a 32-bit build working. But these days, I
think not having a 64-bit build available is not a practical solution,
of course.

-chris


signature.asc (943 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Working tc native build

markt
On 07/07/2014 14:48, Christopher Schultz wrote:

> Mark,
>
> On 7/1/14, 11:44 AM, Mark Thomas wrote:
>> On 01/07/2014 16:04, Mark Thomas wrote:
>>> With some additional information from Mladen regrading the
>>> build tools to use, I now have a working build environment for
>>> the Tomcat Native connector binaries. This is documented at:
>>> http://wiki.apache.org/tomcat/BuildTcNativeWin
>>
>> (excluding a working I64 build that I am still looking at)
>
> I just glanced through the current contents of BuildTcNativeWin on
> the wiki, and I can still see this comment:
>
>>> IA64 OpenSSL build failed. Commenting out destest from
>>> ms\nt.mak
> worked around the issue.
>
> The problem is that the tests require a compatible supporting
> architecture.

I don't believe that is the problem. I shouldn't need a supporting
architecture to build the tests (that part that is failing), only to
run them.

> That is, if you cross-compile to IA64 then try to run the tests,
> the tests will fail because you are running on x86_84 and IA64 is
> not available. The same thing will happen if you try to build/test
> on ia32: you can produce x86_64 binaries, but you will be unable to
> test them.

Agreed.

> Therefore, it would be safest to specify in the instructions that
> either x86_64 or IA64 should be used for the build -- that way, at
> least one of the 64-bit architectures can be tested along with the
> 32-bit one. It's also worth mentioning that a 32-bit environment
> would need to comment-out *all* the 64-bit tests, and not just the
> IA64 ones.

I don't believe any of the tests are specifically 32-bit or 64-bit.

> Thanks for continuing to work on this, Mark. I'm glad you've been
> able to remove Cygwin as a requirement... looking at your initial
> build instructions seemed overly complicated -- or at least had
> many more requirements than strictly necessary. I'm not sure how
> much of Mladen's magic environment is strictly necessary, either: I
> think a lot of it could be put into scripts that ship with the
> tcnative source distribution.

Keep in mind that these are build instructions for a completely clean
OS install so there are a number of tools lists that I would normally
expect to be present on a developers machine.

In terms of how much of this is necessary, experience suggests that
most of it is required to avoid having to install a Visual C Runtime
distribution. If you are prepared to do that (or more specifically
prepared to make your users do that) then yes, this could get a lot
simpler. Personally, I'm in favour of making the release manager jump
through a few hoops rather than making our users jump through hoops.

> I am still going to continue to work on getting all this stuff to
> build using Microsoft Visual Studio Express: it should be possible
> to do, and I *have* been able to get a 32-bit build working. But
> these days, I think not having a 64-bit build available is not a
> practical solution, of course.

I'm not sure it is possible to build with Visual Studio Express
without introducing a dependency on a Visual C Runtime distribution
but if you can find a way around that issue that would be great.

Mark


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Working tc native build

Christopher Schultz-2
Mark,

On 7/7/14, 11:31 AM, Mark Thomas wrote:

> On 07/07/2014 14:48, Christopher Schultz wrote:
>> Mark,
>>
>> On 7/1/14, 11:44 AM, Mark Thomas wrote:
>>> On 01/07/2014 16:04, Mark Thomas wrote:
>>>> With some additional information from Mladen regrading the
>>>> build tools to use, I now have a working build environment for
>>>> the Tomcat Native connector binaries. This is documented at:
>>>> http://wiki.apache.org/tomcat/BuildTcNativeWin
>>>
>>> (excluding a working I64 build that I am still looking at)
>>
>> I just glanced through the current contents of BuildTcNativeWin on
>> the wiki, and I can still see this comment:
>>
>>>> IA64 OpenSSL build failed. Commenting out destest from
>>>> ms\nt.mak
>> worked around the issue.
>>
>> The problem is that the tests require a compatible supporting
>> architecture.
>
> I don't believe that is the problem. I shouldn't need a supporting
> architecture to build the tests (that part that is failing), only to
> run them.
Aah, I didn't realize that building the tests was failing, not running
them. I assumed without checking that "destest" was something like
"destination test" and therefore would require e.g. IA64 to be runnable.
Sorry for the noise.

>> That is, if you cross-compile to IA64 then try to run the tests,
>> the tests will fail because you are running on x86_84 and IA64 is
>> not available. The same thing will happen if you try to build/test
>> on ia32: you can produce x86_64 binaries, but you will be unable to
>> test them.
>
> Agreed.
>
>> Therefore, it would be safest to specify in the instructions that
>> either x86_64 or IA64 should be used for the build -- that way, at
>> least one of the 64-bit architectures can be tested along with the
>> 32-bit one. It's also worth mentioning that a 32-bit environment
>> would need to comment-out *all* the 64-bit tests, and not just the
>> IA64 ones.
>
> I don't believe any of the tests are specifically 32-bit or 64-bit.
The OpenSSL docs (INSTALL.W64) specifically say that to test the builds,
you must test on the target platform:

"
Naturally test-suite itself has to be executed on the target platform.
"

>> Thanks for continuing to work on this, Mark. I'm glad you've been
>> able to remove Cygwin as a requirement... looking at your initial
>> build instructions seemed overly complicated -- or at least had
>> many more requirements than strictly necessary. I'm not sure how
>> much of Mladen's magic environment is strictly necessary, either: I
>> think a lot of it could be put into scripts that ship with the
>> tcnative source distribution.
>
> Keep in mind that these are build instructions for a completely clean
> OS install so there are a number of tools lists that I would normally
> expect to be present on a developers machine.
Right. Both of us are trying to come up with instructions that anyone --
even someone without an existing dev environment -- can follow.

> In terms of how much of this is necessary, experience suggests that
> most of it is required to avoid having to install a Visual C Runtime
> distribution. If you are prepared to do that (or more specifically
> prepared to make your users do that) then yes, this could get a lot
> simpler. Personally, I'm in favour of making the release manager jump
> through a few hoops rather than making our users jump through hoops.

Agreed, though it would be nice if the procedure could be performed by
end users as well.

>> I am still going to continue to work on getting all this stuff to
>> build using Microsoft Visual Studio Express: it should be possible
>> to do, and I *have* been able to get a 32-bit build working. But
>> these days, I think not having a 64-bit build available is not a
>> practical solution, of course.
>
> I'm not sure it is possible to build with Visual Studio Express
> without introducing a dependency on a Visual C Runtime distribution
> but if you can find a way around that issue that would be great.

Evidently, the packages that I have installed do not include
depends.exe. A quick Google search yields this tool:
http://www.dependencywalker.com/

I'll rebuild the 32-bit version (which I can do reliably) and see what
the dependencies are.

Thanks,
-chris


signature.asc (943 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Working tc native build

Christopher Schultz-2
Mark,

On 7/8/14, 9:58 AM, Christopher Schultz wrote:

> Mark,
>
> On 7/7/14, 11:31 AM, Mark Thomas wrote:
>> On 07/07/2014 14:48, Christopher Schultz wrote:
>>> Mark,
>>>
>>> On 7/1/14, 11:44 AM, Mark Thomas wrote:
>>>> On 01/07/2014 16:04, Mark Thomas wrote:
>>>>> With some additional information from Mladen regrading the
>>>>> build tools to use, I now have a working build environment for
>>>>> the Tomcat Native connector binaries. This is documented at:
>>>>> http://wiki.apache.org/tomcat/BuildTcNativeWin
>>>>
>>>> (excluding a working I64 build that I am still looking at)
>>>
>>> I just glanced through the current contents of BuildTcNativeWin on
>>> the wiki, and I can still see this comment:
>>>
>>>>> IA64 OpenSSL build failed. Commenting out destest from
>>>>> ms\nt.mak
>>> worked around the issue.
>>>
>>> The problem is that the tests require a compatible supporting
>>> architecture.
>>
>> I don't believe that is the problem. I shouldn't need a supporting
>> architecture to build the tests (that part that is failing), only to
>> run them.
>
> Aah, I didn't realize that building the tests was failing, not running
> them. I assumed without checking that "destest" was something like
> "destination test" and therefore would require e.g. IA64 to be runnable.
> Sorry for the noise.
>
>>> That is, if you cross-compile to IA64 then try to run the tests,
>>> the tests will fail because you are running on x86_84 and IA64 is
>>> not available. The same thing will happen if you try to build/test
>>> on ia32: you can produce x86_64 binaries, but you will be unable to
>>> test them.
>>
>> Agreed.
>>
>>> Therefore, it would be safest to specify in the instructions that
>>> either x86_64 or IA64 should be used for the build -- that way, at
>>> least one of the 64-bit architectures can be tested along with the
>>> 32-bit one. It's also worth mentioning that a 32-bit environment
>>> would need to comment-out *all* the 64-bit tests, and not just the
>>> IA64 ones.
>>
>> I don't believe any of the tests are specifically 32-bit or 64-bit.
>
> The OpenSSL docs (INSTALL.W64) specifically say that to test the builds,
> you must test on the target platform:
>
> "
> Naturally test-suite itself has to be executed on the target platform.
> "
>
>>> Thanks for continuing to work on this, Mark. I'm glad you've been
>>> able to remove Cygwin as a requirement... looking at your initial
>>> build instructions seemed overly complicated -- or at least had
>>> many more requirements than strictly necessary. I'm not sure how
>>> much of Mladen's magic environment is strictly necessary, either: I
>>> think a lot of it could be put into scripts that ship with the
>>> tcnative source distribution.
>>
>> Keep in mind that these are build instructions for a completely clean
>> OS install so there are a number of tools lists that I would normally
>> expect to be present on a developers machine.
>
> Right. Both of us are trying to come up with instructions that anyone --
> even someone without an existing dev environment -- can follow.
>
>> In terms of how much of this is necessary, experience suggests that
>> most of it is required to avoid having to install a Visual C Runtime
>> distribution. If you are prepared to do that (or more specifically
>> prepared to make your users do that) then yes, this could get a lot
>> simpler. Personally, I'm in favour of making the release manager jump
>> through a few hoops rather than making our users jump through hoops.
>
> Agreed, though it would be nice if the procedure could be performed by
> end users as well.
>
>>> I am still going to continue to work on getting all this stuff to
>>> build using Microsoft Visual Studio Express: it should be possible
>>> to do, and I *have* been able to get a 32-bit build working. But
>>> these days, I think not having a 64-bit build available is not a
>>> practical solution, of course.
>>
>> I'm not sure it is possible to build with Visual Studio Express
>> without introducing a dependency on a Visual C Runtime distribution
>> but if you can find a way around that issue that would be great.
>
> Evidently, the packages that I have installed do not include
> depends.exe. A quick Google search yields this tool:
> http://www.dependencywalker.com/
>
> I'll rebuild the 32-bit version (which I can do reliably) and see what
> the dependencies are.
Evidently, the VS Express 2013 package can't build due to some odd bugs
described here:
http://markmail.org/thread/b7ze2fyrqyr6fb7i

I had to install the "Platform SDK" and was able to produce a 32-bit
binary again. I'll have to rollback my VM and see if I can build with
the SDK (which includes VS2010, evidently) only, to simplify things.

Anyway, here's what the above tool says tcnative-1.dll requires in terms
of direct dependencies:

- USER32.dll
- PSAPI.dll
- SHLWAPI.dll
- KERNEL32.dll
- ADVAPI32.dll
- WS2_32.dll
- MSWSOCK.dll
- MSVCR100.dll

Is that last one the one you were concerned about?

If so, what's the procedure for statically-linking that library into
tcnative ... or, better yet, why is that library not necessary when
using MSVS 2006 or whatever?

I checked, and the openssl.exe binary that was built also requires that
same library (specifically, MSVCR100.dll).

Thanks,
-chris


signature.asc (943 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Working tc native build

markt
On 08/07/2014 16:39, Christopher Schultz wrote:

> Anyway, here's what the above tool says tcnative-1.dll requires in
> terms of direct dependencies:
>
> - USER32.dll - PSAPI.dll - SHLWAPI.dll - KERNEL32.dll -
> ADVAPI32.dll - WS2_32.dll - MSWSOCK.dll - MSVCR100.dll
>
> Is that last one the one you were concerned about?

Yes.

> If so, what's the procedure for statically-linking that library
> into tcnative ... or, better yet, why is that library not necessary
> when using MSVS 2006 or whatever?

Using VS6 or Mladen's toolkit, it builds against msvcrt.dll which is
part of the base OS.

For reasons I haven't dug into, later versions of Visual Studio build
upon a newer version of that library and despite quite a lot of
searching I haven't found a way to make later versions of Visual
Studio build against the older dll.

> I checked, and the openssl.exe binary that was built also requires
> that same library (specifically, MSVCR100.dll).

Yeah, I'd expect that.

Mark


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Working tc native build

Christopher Schultz-2
Mark,

On 7/8/14, 12:14 PM, Mark Thomas wrote:

> On 08/07/2014 16:39, Christopher Schultz wrote:
>
>> Anyway, here's what the above tool says tcnative-1.dll requires in
>> terms of direct dependencies:
>>
>> - USER32.dll - PSAPI.dll - SHLWAPI.dll - KERNEL32.dll -
>> ADVAPI32.dll - WS2_32.dll - MSWSOCK.dll - MSVCR100.dll
>>
>> Is that last one the one you were concerned about?
>
> Yes.
>
>> If so, what's the procedure for statically-linking that library
>> into tcnative ... or, better yet, why is that library not necessary
>> when using MSVS 2006 or whatever?
>
> Using VS6 or Mladen's toolkit, it builds against msvcrt.dll which is
> part of the base OS.
>
> For reasons I haven't dug into, later versions of Visual Studio build
> upon a newer version of that library and despite quite a lot of
> searching I haven't found a way to make later versions of Visual
> Studio build against the older dll.
Here is an exhaustive explanation of what in the world is going on:
http://kobyk.wordpress.com/2007/07/20/dynamically-linking-with-msvcrtdll-using-visual-c-2005/

I found this via SO:
http://stackoverflow.com/questions/10166412/how-to-link-against-msvcrt-dll-instead-of-msvcr100-dll-in-vc-10-0

The upshot is that one should use the Windows Driver (Development) Kit
(WDK/DDK) instead of Visual Studio in order to get a more modern
compiler (than the 2005 version) whilst enjoying a build against MSVCRT.dll.

The SO answer mentions that the Windows 8 WDK no longer links against
the system MSVCRT.dll, so it looks like the Windows 7 WDK is the latest
version that will produce acceptable results.

I'll investigate whether the WDK (which includes a C/C++ compiler and
tool chain) alone can build OpenSSL, APR, and tcnative. If so, your
existing instructions might be able to reduce another prerequisite (the
Windows SDK itself).

Thanks,
-chris


signature.asc (943 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Working tc native build

Rainer Jung-3
In reply to this post by markt
On 08.07.2014 18:14, Mark Thomas wrote:

> On 08/07/2014 16:39, Christopher Schultz wrote:
>
>> Anyway, here's what the above tool says tcnative-1.dll requires in
>> terms of direct dependencies:
>>
>> - USER32.dll - PSAPI.dll - SHLWAPI.dll - KERNEL32.dll -
>> ADVAPI32.dll - WS2_32.dll - MSWSOCK.dll - MSVCR100.dll
>>
>> Is that last one the one you were concerned about?
>
> Yes.
>
>> If so, what's the procedure for statically-linking that library
>> into tcnative ... or, better yet, why is that library not necessary
>> when using MSVS 2006 or whatever?
>
> Using VS6 or Mladen's toolkit, it builds against msvcrt.dll which is
> part of the base OS.
>
> For reasons I haven't dug into, later versions of Visual Studio build
> upon a newer version of that library and despite quite a lot of
> searching I haven't found a way to make later versions of Visual
> Studio build against the older dll.

The dependency on the modern (versioned) msvcrXXX.dll only gets
problematic when you need to mix binaries and libs build with different
MSVC versions in the same process.

For instance building modules for the Apache web server and the server
itself with different MSVC versions can get you in trouble, because the
msvcrXXX.dll version depends 1:1 on the MSVC version and different
versions of the lib are not expected to interact nicely in the same process.

In the tcnative case, this would only happen, if either the jvm itself
or another native agent or library loaded into the jvm would be linked
against a different msvcrXXX.dll. Concerning agents we can't be safe
because we can't control what users load. Concerning the jvm I did a
quick check with 1.7.0_51 64 bit on Windows 7 and depends.exe show the
dependency to msvcr100.dll in bin/server/jvm.dll. The same for Java 8.

So to me it looks one can only either use the old way of building
against the old msvcrt.dll without version - which seems to be no longer
really supported and might vanish - or sync on the msvc version that is
used to build the jvm and hope they keep it stable per jvm major version.

For end users the dependency on the dll is not a big problem, because
Microsoft provides it for redistribution or download. Of course we can't
bundle it due to license incompatibility.

Regards,

Rainer

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Working tc native build

Christopher Schultz-2
Rainer,

On 7/10/14, 8:36 PM, Rainer Jung wrote:

> On 08.07.2014 18:14, Mark Thomas wrote:
>> On 08/07/2014 16:39, Christopher Schultz wrote:
>>
>>> Anyway, here's what the above tool says tcnative-1.dll requires in
>>> terms of direct dependencies:
>>>
>>> - USER32.dll - PSAPI.dll - SHLWAPI.dll - KERNEL32.dll -
>>> ADVAPI32.dll - WS2_32.dll - MSWSOCK.dll - MSVCR100.dll
>>>
>>> Is that last one the one you were concerned about?
>>
>> Yes.
>>
>>> If so, what's the procedure for statically-linking that library
>>> into tcnative ... or, better yet, why is that library not necessary
>>> when using MSVS 2006 or whatever?
>>
>> Using VS6 or Mladen's toolkit, it builds against msvcrt.dll which is
>> part of the base OS.
>>
>> For reasons I haven't dug into, later versions of Visual Studio build
>> upon a newer version of that library and despite quite a lot of
>> searching I haven't found a way to make later versions of Visual
>> Studio build against the older dll.
>
> The dependency on the modern (versioned) msvcrXXX.dll only gets
> problematic when you need to mix binaries and libs build with different
> MSVC versions in the same process.
>
> For instance building modules for the Apache web server and the server
> itself with different MSVC versions can get you in trouble, because the
> msvcrXXX.dll version depends 1:1 on the MSVC version and different
> versions of the lib are not expected to interact nicely in the same
> process.
>
> In the tcnative case, this would only happen, if either the jvm itself
> or another native agent or library loaded into the jvm would be linked
> against a different msvcrXXX.dll. Concerning agents we can't be safe
> because we can't control what users load. Concerning the jvm I did a
> quick check with 1.7.0_51 64 bit on Windows 7 and depends.exe show the
> dependency to msvcr100.dll in bin/server/jvm.dll. The same for Java 8.
Something doesn't quite add up, here: we have been producing builds
against the "system" MSVCRT.dll library for .. ever, and the JVM has
probably been built against MSVCR100.dll for a while, but there have
been no reports of tcnative burning to the ground on Windows. Doesn't
that mean that "libs built with different MSVC versions in the same
process" aren't a problem.. at least .. not always? Maybe the deal is
that we use only simple calls from the library and therefore it doesn't
matter which one gets called at runtime.

> So to me it looks one can only either use the old way of building
> against the old msvcrt.dll without version - which seems to be no longer
> really supported and might vanish - or sync on the msvc version that is
> used to build the jvm and hope they keep it stable per jvm major version.

It can still be done using the Windows Driver Development Kit, but most
people don't have the DDK sitting around for "normal" Windows development.

> For end users the dependency on the dll is not a big problem, because
> Microsoft provides it for redistribution or download. Of course we can't
> bundle it due to license incompatibility.

Any chance that MSVCR100.dll and friends are provided by recent OSs? On
my Windows 8 VM I can see these files in /windows/system32:

07/25/2012  11:06 PM            77,824 msvcirt.dll
07/11/2012  10:01 PM           613,840 msvcp110_clr0400.dll
07/25/2012  11:06 PM           572,416 msvcp60.dll
08/30/2012  08:52 PM            17,888 msvcr100_clr0400.dll
07/11/2012  10:01 PM           856,016 msvcr110_clr0400.dll
07/26/2012  01:26 AM           654,848 msvcrt.dll

In my particular case, would we need to bundle anything with tcnative?

Thanks,
-chris


signature.asc (943 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Working tc native build

markt
In reply to this post by Rainer Jung-3
On 11/07/2014 01:36, Rainer Jung wrote:

> In the tcnative case, this would only happen, if either the jvm itself
> or another native agent or library loaded into the jvm would be linked
> against a different msvcrXXX.dll. Concerning agents we can't be safe
> because we can't control what users load. Concerning the jvm I did a
> quick check with 1.7.0_51 64 bit on Windows 7 and depends.exe show the
> dependency to msvcr100.dll in bin/server/jvm.dll. The same for Java 8.

As long as all versions of Java depend on the same msvcrXXX.dll then
we'd only need to ship one version of tc-native. As soon as there are
multiple dependencies we'd need to build and ship multiple versions.

> So to me it looks one can only either use the old way of building
> against the old msvcrt.dll without version - which seems to be no longer
> really supported and might vanish - or sync on the msvc version that is
> used to build the jvm and hope they keep it stable per jvm major version.

Right now the prudent thing to do appears to ensure that multiple
committers have a built environment (preferably on a VM) for the current
build process.

> For end users the dependency on the dll is not a big problem, because
> Microsoft provides it for redistribution or download. Of course we can't
> bundle it due to license incompatibility.

Indeed. Most systems will probably have it installed already. If we
opted to depend on the version that shipped with Java we'd know it was
present. That said, it still makes me uncomfortable. My preference is to
stick to the current dependencies for as long as we can.

Mark


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: Working tc native build

Konstantin Preißer-3
In reply to this post by Christopher Schultz-2
Hi Christopher,

> -----Original Message-----
> From: Christopher Schultz [mailto:[hidden email]]
> Sent: Friday, July 11, 2014 7:50 PM
> To: Tomcat Developers List
> Subject: Re: Working tc native build

> > For end users the dependency on the dll is not a big problem, because
> > Microsoft provides it for redistribution or download. Of course we can't
> > bundle it due to license incompatibility.
>
> Any chance that MSVCR100.dll and friends are provided by recent OSs? On
> my Windows 8 VM I can see these files in /windows/system32:
>
> 07/25/2012  11:06 PM            77,824 msvcirt.dll
> 07/11/2012  10:01 PM           613,840 msvcp110_clr0400.dll
> 07/25/2012  11:06 PM           572,416 msvcp60.dll
> 08/30/2012  08:52 PM            17,888 msvcr100_clr0400.dll
> 07/11/2012  10:01 PM           856,016 msvcr110_clr0400.dll
> 07/26/2012  01:26 AM           654,848 msvcrt.dll
>
> In my particular case, would we need to bundle anything with tcnative?

I think the above dlls with "_clr0400.dll" are DLLs used by the .Net Framework only (at leat their descriptions read "Microsoft® .NET Framework"), so I don't think they can/will be used by C++ applications.

AFAIK, even recent Windows OSes don't contain current Visual C++ Runtime libraries by default. Users will need to install the corresponding redistributable package - e.g. if you compile with VS 2010, the user will need to install the "Microsoft Visual C++ 2010 Redistributable Package" [1] (x86 or x64) which e.g. will install "msvcr100.dll" to the windows\system32 directory.
If you compile with VS 2013, the user will need the "Visual C++ Redistributable Packages for Visual Studio 2013" [2] (x86 or x64) which e.g. installs "msvcr120.dll", and so on.

However, I don't know why MS doesn't include those runtimes with current OSes just like it is done with .Net Framework. (Maybe to reduce maintenance when system only needs a specific runtime, but not all, as there have been security updates for some C++ Redistributables in the past).


[1] http://www.microsoft.com/en-us/download/details.aspx?id=14632
[2] http://www.microsoft.com/en-us/download/details.aspx?id=40784



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: Working tc native build

Konstantin Preißer-3
Hi,

> -----Original Message-----
> From: Konstantin Preißer [mailto:[hidden email]]
> Sent: Friday, July 11, 2014 8:18 PM
> To: 'Tomcat Developers List'
> Subject: RE: Working tc native build
>
> Hi Christopher,
>
> > -----Original Message-----
> > From: Christopher Schultz [mailto:[hidden email]]
> > Sent: Friday, July 11, 2014 7:50 PM
> > To: Tomcat Developers List
> > Subject: Re: Working tc native build
>
> > > For end users the dependency on the dll is not a big problem, because
> > > Microsoft provides it for redistribution or download. Of course we can't
> > > bundle it due to license incompatibility.
> >
> > Any chance that MSVCR100.dll and friends are provided by recent OSs? On
> > my Windows 8 VM I can see these files in /windows/system32:
> >
> > 07/25/2012  11:06 PM            77,824 msvcirt.dll
> > 07/11/2012  10:01 PM           613,840 msvcp110_clr0400.dll
> > 07/25/2012  11:06 PM           572,416 msvcp60.dll
> > 08/30/2012  08:52 PM            17,888 msvcr100_clr0400.dll
> > 07/11/2012  10:01 PM           856,016 msvcr110_clr0400.dll
> > 07/26/2012  01:26 AM           654,848 msvcrt.dll
> >
> > In my particular case, would we need to bundle anything with tcnative?
>
> I think the above dlls with "_clr0400.dll" are DLLs used by the .Net Framework
> only (at leat their descriptions read "Microsoft® .NET Framework"), so I don't
> think they can/will be used by C++ applications.
>
> AFAIK, even recent Windows OSes don't contain current Visual C++ Runtime
> libraries by default. Users will need to install the corresponding
> redistributable package - e.g. if you compile with VS 2010, the user will need
> to install the "Microsoft Visual C++ 2010 Redistributable Package" [1] (x86 or
> x64) which e.g. will install "msvcr100.dll" to the windows\system32 directory.
> If you compile with VS 2013, the user will need the "Visual C++
> Redistributable Packages for Visual Studio 2013" [2] (x86 or x64) which e.g.
> installs "msvcr120.dll", and so on.
>
> However, I don't know why MS doesn't include those runtimes with current
> OSes just like it is done with .Net Framework. (Maybe to reduce
> maintenance when system only needs a specific runtime, but not all, as there
> have been security updates for some C++ Redistributables in the past).
>
>
> [1] http://www.microsoft.com/en-us/download/details.aspx?id=14632
> [2] http://www.microsoft.com/en-us/download/details.aspx?id=40784

FJY, When I just tried to compile a simple C++ DLL (exporting simple functions to solve a Sudoku) using Visual Studio 2013, the resulting DLL was 11 KB, and when using Dependeny Walker [1] view the DLL dependencies, it showed "msvcr120.dll" and "kernel32.dll".

However, when I change the project configuration (projectname.vcxproj) from
  <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'" Label="Configuration">
    (...)
    <UseOfMfc>false</UseOfMfc>
  </PropertyGroup>

to

  <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'" Label="Configuration">
    (...)
    <UseOfMfc>Static</UseOfMfc>
  </PropertyGroup>

and compile the DLL, it has 83 KB instead of 11 KB, and Dependency Walker only shows "kernel32.dll", but no VC runtime. I then tested this with a VC++ project that will create an .EXE, and with the first option, it will fail to run on a fresh Windows 8 system complaining about missing msvcr120.dll; whereas with the second option, it will run without any problems.

So maybe with this option the necessary functions from the C runtime have been included in the generated DLL, avoiding a run-time depencency to the VC++ runtime DLLs. However, I have only little knowledge about C++ so I cannot really comment on how this setting (UseOfMfc) works.


Regards,
Konstantin Preißer


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Working tc native build

Christopher Schultz-2
Konstantin,

On 7/11/14, 3:02 PM, Konstantin Preißer wrote:

> Hi,
>
>> -----Original Message-----
>> From: Konstantin Preißer [mailto:[hidden email]]
>> Sent: Friday, July 11, 2014 8:18 PM
>> To: 'Tomcat Developers List'
>> Subject: RE: Working tc native build
>>
>> Hi Christopher,
>>
>>> -----Original Message-----
>>> From: Christopher Schultz [mailto:[hidden email]]
>>> Sent: Friday, July 11, 2014 7:50 PM
>>> To: Tomcat Developers List
>>> Subject: Re: Working tc native build
>>
>>>> For end users the dependency on the dll is not a big problem, because
>>>> Microsoft provides it for redistribution or download. Of course we can't
>>>> bundle it due to license incompatibility.
>>>
>>> Any chance that MSVCR100.dll and friends are provided by recent OSs? On
>>> my Windows 8 VM I can see these files in /windows/system32:
>>>
>>> 07/25/2012  11:06 PM            77,824 msvcirt.dll
>>> 07/11/2012  10:01 PM           613,840 msvcp110_clr0400.dll
>>> 07/25/2012  11:06 PM           572,416 msvcp60.dll
>>> 08/30/2012  08:52 PM            17,888 msvcr100_clr0400.dll
>>> 07/11/2012  10:01 PM           856,016 msvcr110_clr0400.dll
>>> 07/26/2012  01:26 AM           654,848 msvcrt.dll
>>>
>>> In my particular case, would we need to bundle anything with tcnative?
>>
>> I think the above dlls with "_clr0400.dll" are DLLs used by the .Net Framework
>> only (at leat their descriptions read "Microsoft® .NET Framework"), so I don't
>> think they can/will be used by C++ applications.
>>
>> AFAIK, even recent Windows OSes don't contain current Visual C++ Runtime
>> libraries by default. Users will need to install the corresponding
>> redistributable package - e.g. if you compile with VS 2010, the user will need
>> to install the "Microsoft Visual C++ 2010 Redistributable Package" [1] (x86 or
>> x64) which e.g. will install "msvcr100.dll" to the windows\system32 directory.
>> If you compile with VS 2013, the user will need the "Visual C++
>> Redistributable Packages for Visual Studio 2013" [2] (x86 or x64) which e.g.
>> installs "msvcr120.dll", and so on.
>>
>> However, I don't know why MS doesn't include those runtimes with current
>> OSes just like it is done with .Net Framework. (Maybe to reduce
>> maintenance when system only needs a specific runtime, but not all, as there
>> have been security updates for some C++ Redistributables in the past).
>>
>>
>> [1] http://www.microsoft.com/en-us/download/details.aspx?id=14632
>> [2] http://www.microsoft.com/en-us/download/details.aspx?id=40784
>
> FJY, When I just tried to compile a simple C++ DLL (exporting simple functions to solve a Sudoku) using Visual Studio 2013, the resulting DLL was 11 KB, and when using Dependeny Walker [1] view the DLL dependencies, it showed "msvcr120.dll" and "kernel32.dll".
>
> However, when I change the project configuration (projectname.vcxproj) from
>   <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'" Label="Configuration">
>     (...)
>     <UseOfMfc>false</UseOfMfc>
>   </PropertyGroup>
>
> to
>
>   <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'" Label="Configuration">
>     (...)
>     <UseOfMfc>Static</UseOfMfc>
>   </PropertyGroup>
>
> and compile the DLL, it has 83 KB instead of 11 KB, and Dependency
> Walker only shows "kernel32.dll", but no VC runtime. I then tested this
> with a VC++ project that will create an .EXE, and with the first option,
> it will fail to run on a fresh Windows 8 system complaining about
> missing msvcr120.dll; whereas with the second option, it will run
> without any problems.
>
> So maybe with this option the necessary functions from the C runtime
> have been included in the generated DLL, avoiding a run-time
> depencency to the VC++ runtime DLLs. However, I have only little
> knowledge about C++ so I cannot really comment on how this setting
> (UseOfMfc) works.
Right: you have created a statically-linked DLL (which sounds funny to
me) which does not have a dependency on the MSVCRT stuff: its all bundled.

I wonder how much more bloated tcnative-1.dll would be if we just
statically-linked the DLL with MSVCRT... it might make this pain go away.

-chris


signature.asc (943 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Working tc native build

Rainer Jung-3
In reply to this post by Christopher Schultz-2
On 11.07.2014 19:50, Christopher Schultz wrote:

> Rainer,
>
> On 7/10/14, 8:36 PM, Rainer Jung wrote:
>> On 08.07.2014 18:14, Mark Thomas wrote:
>>> On 08/07/2014 16:39, Christopher Schultz wrote:
>>>
>>>> Anyway, here's what the above tool says tcnative-1.dll requires in
>>>> terms of direct dependencies:
>>>>
>>>> - USER32.dll - PSAPI.dll - SHLWAPI.dll - KERNEL32.dll -
>>>> ADVAPI32.dll - WS2_32.dll - MSWSOCK.dll - MSVCR100.dll
>>>>
>>>> Is that last one the one you were concerned about?
>>>
>>> Yes.
>>>
>>>> If so, what's the procedure for statically-linking that library
>>>> into tcnative ... or, better yet, why is that library not necessary
>>>> when using MSVS 2006 or whatever?
>>>
>>> Using VS6 or Mladen's toolkit, it builds against msvcrt.dll which is
>>> part of the base OS.
>>>
>>> For reasons I haven't dug into, later versions of Visual Studio build
>>> upon a newer version of that library and despite quite a lot of
>>> searching I haven't found a way to make later versions of Visual
>>> Studio build against the older dll.
>>
>> The dependency on the modern (versioned) msvcrXXX.dll only gets
>> problematic when you need to mix binaries and libs build with different
>> MSVC versions in the same process.
>>
>> For instance building modules for the Apache web server and the server
>> itself with different MSVC versions can get you in trouble, because the
>> msvcrXXX.dll version depends 1:1 on the MSVC version and different
>> versions of the lib are not expected to interact nicely in the same
>> process.
>>
>> In the tcnative case, this would only happen, if either the jvm itself
>> or another native agent or library loaded into the jvm would be linked
>> against a different msvcrXXX.dll. Concerning agents we can't be safe
>> because we can't control what users load. Concerning the jvm I did a
>> quick check with 1.7.0_51 64 bit on Windows 7 and depends.exe show the
>> dependency to msvcr100.dll in bin/server/jvm.dll. The same for Java 8.
>
> Something doesn't quite add up, here: we have been producing builds
> against the "system" MSVCRT.dll library for .. ever, and the JVM has
> probably been built against MSVCR100.dll for a while, but there have
> been no reports of tcnative burning to the ground on Windows. Doesn't
> that mean that "libs built with different MSVC versions in the same
> process" aren't a problem.. at least .. not always? Maybe the deal is
> that we use only simple calls from the library and therefore it doesn't
> matter which one gets called at runtime.

Folklore says mixing msvcrt.dll and a single numbered one is fine,
mixing multiple numbered ones in the same binary not.

>> So to me it looks one can only either use the old way of building
>> against the old msvcrt.dll without version - which seems to be no longer
>> really supported and might vanish - or sync on the msvc version that is
>> used to build the jvm and hope they keep it stable per jvm major version.
>
> It can still be done using the Windows Driver Development Kit, but most
> people don't have the DDK sitting around for "normal" Windows development.

I can't comment, I never tried that way for any build.

>> For end users the dependency on the dll is not a big problem, because
>> Microsoft provides it for redistribution or download. Of course we can't
>> bundle it due to license incompatibility.
>
> Any chance that MSVCR100.dll and friends are provided by recent OSs? On
> my Windows 8 VM I can see these files in /windows/system32:
>
> 07/25/2012  11:06 PM            77,824 msvcirt.dll
> 07/11/2012  10:01 PM           613,840 msvcp110_clr0400.dll
> 07/25/2012  11:06 PM           572,416 msvcp60.dll
> 08/30/2012  08:52 PM            17,888 msvcr100_clr0400.dll
> 07/11/2012  10:01 PM           856,016 msvcr110_clr0400.dll
> 07/26/2012  01:26 AM           654,848 msvcrt.dll

Like Konstantin said, typically it is *not* provided.

> In my particular case, would we need to bundle anything with tcnative?

I'd say if we ever switch to building using normal MSVC, it should
simply be documented, that tcnative has a dependency on msvcr100.dll
(with the number 100 adopted to the version of MSVC which was used for
the build) and that you can get that from MS (the redistributable
download). I'd expect, that as long as we stick to the same version,
that's used to build the JVM, tcnative would find the dll loaded already
by the JVM (but haven't tested that assumption). If that is true, we
could document that as well.

Currently since we only depend on msvcrt.dll everything is fine, except
that building that way is a bit more complicated and MS might stop that
possibility in the future.

Regards,

Rainer

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]