Building tcnative on win32

classic Classic list List threaded Threaded
32 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Building tcnative on win32

Christopher Schultz-2
All,

I've spent some time trying to script a soup-to-nuts repeatable build of
tcnative on win32. I've made some progress but I'm hitting a few snags
that perhaps some folks could help with.

First, I want the script to fetch as much stuff as possible for the user
instead of relying on a great deal of previously-installed software. My
current requirements -- other than MSVC++ -- are for wget.exe and
7za.exe to be available. If anyone knows how to script an HTTP download
or a ZIP-decompress from a Windows command-line without either of those
tools, I'd greatly appreciate some insight.

(This is where I mention that, really, getting anything done on Windows
that doesn't involve Microsoft Excel and nothing else is absolutely
excruciating. Running Windows 8 in a VM means that even editing my batch
script is a challenge, as the meta keys from Mac -> Windows don't match
up and I feel like I'm learning to type all over again. Sign.)

We need to patch both openssl and libapr before building them, so we
need "patch". There's a GNU win32 ports project where I can get
patch.exe, but its got two problems:

1. It needs to be patched (ha!) to *not* require administrator access.
Why does patch.exe need administrator access to run? Your guess is as
good as mine.

2. It seems to be crashing when I try to actually patch something:
C:\Users\IEUser\AppData\Local\Temp\build-tcnative\build\openssl-1.0.1h>\Users\IEUser\Desktop\tools\patch.exe
-i
..\tomcat-native-1.1.30-win32-src\jni\native\srclib\openssl\openssl-msvcrt.patch
-p 0 --dry-run
patching file util/pl/VC-32.pl

Assertion failed: hunk, file ../patch-2.5.9-src/patch.c, line 354

This application has requested the Runtime to terminate it in an unusual
way.
Please contact the application's support team for more information.

If anyone knows of a good "patch" program for win32, I'd really
appreciate that, too.

I've gotten some help from the apr folks with building APR on win32 as
well. Their instructions on the web site are outright lies, but it seems
that instructions that can be found elsewhere (like in the source
distro) that are significantly less complicated (e.g. "run nmake") and
seem to work. I have to get apr building against OpenSSL but that is
apparently fairly trivial. Once I get get those beasts built, I can
follow Mladen's instructions for building tcnative which are fairly
straightforward.

Ultimately, my goal is to get a script that can be maintained that does
the following:
1. Obtains all sources necessary (openssl, apr, tcnative)
2. Patches all sources as necessary
3. Builds each component
4. Produces a statically-linked binary for distribution

Does anyone know why we bother with a statically-linked tcnative.dll
rather than just shipping a "regular" DLL and have users provide their
own apr.dll and openssl.dll? It seems like it might be easier if we
didn't have to build everything ourselves. Also, for things like
"upgrading OpenSSL", it would often be a drop-in replacement instead of
having to re-build everything from scratch.

Thanks,
-chris


signature.asc (943 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Building tcnative on win32

Konstantin Kolinko
2014-06-14 1:36 GMT+04:00 Christopher Schultz <[hidden email]>:

> All,
>
> I've spent some time trying to script a soup-to-nuts repeatable build of
> tcnative on win32. I've made some progress but I'm hitting a few snags
> that perhaps some folks could help with.
>
> First, I want the script to fetch as much stuff as possible for the user
> instead of relying on a great deal of previously-installed software. My
> current requirements -- other than MSVC++ -- are for wget.exe and
> 7za.exe to be available. If anyone knows how to script an HTTP download
> or a ZIP-decompress from a Windows command-line without either of those
> tools, I'd greatly appreciate some insight.

Maybe Java + Apache Ant?

Or maybe NAnt has some tasks for that
http://nant.sourceforge.net/

> (This is where I mention that, really, getting anything done on Windows
> that doesn't involve Microsoft Excel and nothing else is absolutely
> excruciating. Running Windows 8 in a VM means that even editing my batch
> script is a challenge, as the meta keys from Mac -> Windows don't match
> up and I feel like I'm learning to type all over again. Sign.)

For simple editing I use FAR Manager built-in editor.
http://www.farmanager.com/screenshots.php?l=en

> We need to patch both openssl and libapr before building them, so we
> need "patch". There's a GNU win32 ports project where I can get
> patch.exe, but its got two problems:
> (....)

This package? How did you install it? (Interactive installer, or zip).
http://gnuwin32.sourceforge.net/packages/patch.htm

[quote] from "Installation, Usage and Help" of the above page
On MS-Windows, the patchfile must be a text file, i.e. CR-LF must be
used as line endings. A file with LF may give the error: "Assertion
failed, hunk, file patch.c, line 343," unless the option '--binary' is
given.
[/quote]

Best regards,
Konstantin Kolinko

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: Building tcnative on win32

Konstantin Preißer-3
In reply to this post by Christopher Schultz-2
Hi Christopher,

> -----Original Message-----
> From: Christopher Schultz [mailto:[hidden email]]
> Sent: Friday, June 13, 2014 11:36 PM
> To: [hidden email]
> Subject: Building tcnative on win32
>
> All,
>
> I've spent some time trying to script a soup-to-nuts repeatable build of
> tcnative on win32. I've made some progress but I'm hitting a few snags
> that perhaps some folks could help with.
>
> First, I want the script to fetch as much stuff as possible for the user
> instead of relying on a great deal of previously-installed software. My
> current requirements -- other than MSVC++ -- are for wget.exe and
> 7za.exe to be available. If anyone knows how to script an HTTP download
> or a ZIP-decompress from a Windows command-line without either of those
> tools, I'd greatly appreciate some insight.

If you look for a more powerful scripting environment included in Windows than cmd.exe, you can take a look at Windows Powershell [1] which has been introduced with Vista.
I haven't worked much with Powershell yet so I can't comment on how useful it is, but I know that is based on .Net so you should be able to do anything that you could do in C#/.Net.

For example, to download a file you should be able to use System.Web.WebClient like it is described at [2].
For extracting a ZIP file, you could use System.IO.Compression.ZipFile.ExtractToDirectory() as described at [3].


Regards,
Konstantin Preißer


[1] http://en.wikipedia.org/wiki/Windows_PowerShell
[2] http://answers.oreilly.com/topic/2006-how-to-download-a-file-from-the-internet-with-windows-powershell/
[3] http://serverfault.com/a/461812


> (This is where I mention that, really, getting anything done on Windows
> that doesn't involve Microsoft Excel and nothing else is absolutely
> excruciating. Running Windows 8 in a VM means that even editing my batch
> script is a challenge, as the meta keys from Mac -> Windows don't match
> up and I feel like I'm learning to type all over again. Sign.)
>
> We need to patch both openssl and libapr before building them, so we
> need "patch". There's a GNU win32 ports project where I can get
> patch.exe, but its got two problems:
>
> 1. It needs to be patched (ha!) to *not* require administrator access.
> Why does patch.exe need administrator access to run? Your guess is as
> good as mine.
>
> 2. It seems to be crashing when I try to actually patch something:
> C:\Users\IEUser\AppData\Local\Temp\build-tcnative\build\openssl-
> 1.0.1h>\Users\IEUser\Desktop\tools\patch.exe
> -i
> ..\tomcat-native-1.1.30-win32-src\jni\native\srclib\openssl\openssl-
> msvcrt.patch
> -p 0 --dry-run
> patching file util/pl/VC-32.pl
>
> Assertion failed: hunk, file ../patch-2.5.9-src/patch.c, line 354
>
> This application has requested the Runtime to terminate it in an unusual
> way.
> Please contact the application's support team for more information.
>
> If anyone knows of a good "patch" program for win32, I'd really
> appreciate that, too.
>
> I've gotten some help from the apr folks with building APR on win32 as
> well. Their instructions on the web site are outright lies, but it seems
> that instructions that can be found elsewhere (like in the source
> distro) that are significantly less complicated (e.g. "run nmake") and
> seem to work. I have to get apr building against OpenSSL but that is
> apparently fairly trivial. Once I get get those beasts built, I can
> follow Mladen's instructions for building tcnative which are fairly
> straightforward.
>
> Ultimately, my goal is to get a script that can be maintained that does
> the following:
> 1. Obtains all sources necessary (openssl, apr, tcnative)
> 2. Patches all sources as necessary
> 3. Builds each component
> 4. Produces a statically-linked binary for distribution
>
> Does anyone know why we bother with a statically-linked tcnative.dll
> rather than just shipping a "regular" DLL and have users provide their
> own apr.dll and openssl.dll? It seems like it might be easier if we
> didn't have to build everything ourselves. Also, for things like
> "upgrading OpenSSL", it would often be a drop-in replacement instead of
> having to re-build everything from scratch.
>
> Thanks,
> -chris


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Building tcnative on win32

Christopher Schultz-2
In reply to this post by Konstantin Kolinko
Konstantin,

On 6/13/14, 5:56 PM, Konstantin Kolinko wrote:

> 2014-06-14 1:36 GMT+04:00 Christopher Schultz <[hidden email]>:
>> All,
>>
>> I've spent some time trying to script a soup-to-nuts repeatable build of
>> tcnative on win32. I've made some progress but I'm hitting a few snags
>> that perhaps some folks could help with.
>>
>> First, I want the script to fetch as much stuff as possible for the user
>> instead of relying on a great deal of previously-installed software. My
>> current requirements -- other than MSVC++ -- are for wget.exe and
>> 7za.exe to be available. If anyone knows how to script an HTTP download
>> or a ZIP-decompress from a Windows command-line without either of those
>> tools, I'd greatly appreciate some insight.
>
> Maybe Java + Apache Ant?
>
> Or maybe NAnt has some tasks for that
> http://nant.sourceforge.net/
I considered Nant, but I wanted to limit the dependencies required to
build tcnative.

>> We need to patch both openssl and libapr before building them, so we
>> need "patch". There's a GNU win32 ports project where I can get
>> patch.exe, but its got two problems:
>> (....)
>
> This package? How did you install it? (Interactive installer, or zip).
> http://gnuwin32.sourceforge.net/packages/patch.htm
>
> [quote] from "Installation, Usage and Help" of the above page
> On MS-Windows, the patchfile must be a text file, i.e. CR-LF must be
> used as line endings. A file with LF may give the error: "Assertion
> failed, hunk, file patch.c, line 343," unless the option '--binary' is
> given.
> [/quote]
Thanks for that. Adding --binary fixes the problem for the openssl
patch. The APR patch seems to not to apply no matter what. I think it
may be because the APR patch is for 1.4.x and the current version of
libapr is 1.5.1. I manually applied the patch and have a new patch
available. I'm not sure how to detect the version of libapr being used
to apply the right patch, but we can provide both patches and let users
apply the correct one for their environment.

Thanks,
-chris


signature.asc (943 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Building tcnative on win32 [progress!]

Christopher Schultz-2
In reply to this post by Christopher Schultz-2
All,

On 6/13/14, 5:36 PM, Christopher Schultz wrote:
> I've spent some time trying to script a soup-to-nuts repeatable build of
> tcnative on win32. I've made some progress but I'm hitting a few snags
> that perhaps some folks could help with.

I've got openssl and libapr patched and built. Now, I'm working on
Mladen's instructions for building tcnative itself. Quoting from his BZ
comment:

> It's very simple. The biggest problem is compiling apr and openssl.
> OpenSSL needs to be patched to allow /MD with static lib (something I
> tried to convince openssl developers for years without luck). We have
> the patch in the source dist. Then you have to put apr and openssl
> include and lib files to some directory
>
> C:\foo\bar\include
> C:\foo\bar\lib
>
> Then execute:
> nmake -f NMAKEMakefile WITH_APR=C:\foo\bar WITH_OPENSSL=C:\foo\bar APR_DECLARE_STATIC=1 [ENABLE_OCSP=1]
>
> When building APR I also modify apr.hw and set  APR_HAVE_IPV6 to 1.
I have patched APR appropriately and executed this command:

C:\work\tomcat-native-1.1.30-src> nmake -f NMAKEMakefile
"WITH_APR=%MYTEMP%\build\apr"
"WITH_OPENSSL=%MYTEMP%\build\openssl-%OPENSSL_VERSION%" APR_DECLARE_STATIC=1

(MYTEMP\build\apr contains the apr build and
MYTEMP\build\openssl-OPENSSL_VERSION contains the openssl build)

nmake responds with the following:

Microsoft (R) Program Maintenance Utility Version 10.00.30319.01
Copyright (C) Microsoft Corporation.  All rights reserved.

NMAKEmakefile.inc(123) : fatal error U1050: Must specify CPU environment
variable (X86, X64, I64) x86
Stop.

Okay, so I have to set the CPU environment variable. I didn't know that,
but the build script helpfully tells me I have to, and gives me a list
of recognized values.

I also noticed that I hadn't installed Java (duh) and so the next build
failed because jni.h could not be found. After installing Java and
setting JAVA_HOME appropriately, nmake now reports:

C:\work\tomcat-native-1.1.30-src> SET CPU=X86
C:\work\tomcat-native-1.1.30-src> SET JAVA_HOME="C:\Program
Files\Java\jdk1.7.0_60"
C:\work\tomcat-native-1.1.30-src> nmake -f NMAKEMakefile
"WITH_APR=%MYTEMP%\build\apr"
"WITH_OPENSSL=%MYTEMP%\build\openssl-%OPENSSL_VERSION%" APR_DECLARE_STATIC=1

Microsoft (R) Program Maintenance Utility Version 10.00.30319.01
Copyright (C) Microsoft Corporation.  All rights reserved.

NMAKEmakefile.inc(330) : fatal error U1023: syntax error in expression
Stop.

This is NMAKEmakefile.inc, starting on line 330:
!IF DEFINED(JAVA_HOME) && "$(JAVA_HOME)" != ""
JAVA_INCLUDES=-I "$(JAVA_HOME)\include" -I "$(JAVA_HOME)\include\win32"
!ENDIF

I'm not nmake expert, but I don't see any problems with the above
make-script. I checked to see that things like DEFINED(), &&, and != are
used in other places, and everything seems in order.

Any suggestions?

Thanks,
-chris


signature.asc (943 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Building tcnative on win32 [progress!]

Christopher Schultz-2
All,

On 6/18/14, 1:50 PM, Christopher Schultz wrote:

> All,
>
> On 6/13/14, 5:36 PM, Christopher Schultz wrote:
>> I've spent some time trying to script a soup-to-nuts repeatable build of
>> tcnative on win32. I've made some progress but I'm hitting a few snags
>> that perhaps some folks could help with.
>
> I've got openssl and libapr patched and built. Now, I'm working on
> Mladen's instructions for building tcnative itself. Quoting from his BZ
> comment:
>
>> It's very simple. The biggest problem is compiling apr and openssl.
>> OpenSSL needs to be patched to allow /MD with static lib (something I
>> tried to convince openssl developers for years without luck). We have
>> the patch in the source dist. Then you have to put apr and openssl
>> include and lib files to some directory
>>
>> C:\foo\bar\include
>> C:\foo\bar\lib
>>
>> Then execute:
>> nmake -f NMAKEMakefile WITH_APR=C:\foo\bar WITH_OPENSSL=C:\foo\bar APR_DECLARE_STATIC=1 [ENABLE_OCSP=1]
>>
>> When building APR I also modify apr.hw and set  APR_HAVE_IPV6 to 1.
>
> I have patched APR appropriately and executed this command:
>
> C:\work\tomcat-native-1.1.30-src> nmake -f NMAKEMakefile
> "WITH_APR=%MYTEMP%\build\apr"
> "WITH_OPENSSL=%MYTEMP%\build\openssl-%OPENSSL_VERSION%" APR_DECLARE_STATIC=1
>
> (MYTEMP\build\apr contains the apr build and
> MYTEMP\build\openssl-OPENSSL_VERSION contains the openssl build)
>
> nmake responds with the following:
>
> Microsoft (R) Program Maintenance Utility Version 10.00.30319.01
> Copyright (C) Microsoft Corporation.  All rights reserved.
>
> NMAKEmakefile.inc(123) : fatal error U1050: Must specify CPU environment
> variable (X86, X64, I64) x86
> Stop.
>
> Okay, so I have to set the CPU environment variable. I didn't know that,
> but the build script helpfully tells me I have to, and gives me a list
> of recognized values.
>
> I also noticed that I hadn't installed Java (duh) and so the next build
> failed because jni.h could not be found. After installing Java and
> setting JAVA_HOME appropriately, nmake now reports:
>
> C:\work\tomcat-native-1.1.30-src> SET CPU=X86
> C:\work\tomcat-native-1.1.30-src> SET JAVA_HOME="C:\Program
> Files\Java\jdk1.7.0_60"
> C:\work\tomcat-native-1.1.30-src> nmake -f NMAKEMakefile
> "WITH_APR=%MYTEMP%\build\apr"
> "WITH_OPENSSL=%MYTEMP%\build\openssl-%OPENSSL_VERSION%" APR_DECLARE_STATIC=1
>
> Microsoft (R) Program Maintenance Utility Version 10.00.30319.01
> Copyright (C) Microsoft Corporation.  All rights reserved.
>
> NMAKEmakefile.inc(330) : fatal error U1023: syntax error in expression
> Stop.
>
> This is NMAKEmakefile.inc, starting on line 330:
> !IF DEFINED(JAVA_HOME) && "$(JAVA_HOME)" != ""
> JAVA_INCLUDES=-I "$(JAVA_HOME)\include" -I "$(JAVA_HOME)\include\win32"
> !ENDIF
>
> I'm not nmake expert, but I don't see any problems with the above
> make-script. I checked to see that things like DEFINED(), &&, and != are
> used in other places, and everything seems in order.
>
> Any suggestions?
Aw, geez. Evidently, among other things, Windows is bad at quotes, too.

There is a difference between:

C:\work\tomcat-native-1.1.30-src> SET JAVA_HOME="C:\Program
Files\Java\jdk1.7.0_60"

AND

C:\work\tomcat-native-1.1.30-src> SET JAVA_HOME=C:\Program
Files\Java\jdk1.7.0_60

(note the lack of quotes around the path in the second example)

So, tcnative is building, now.

I'm getting a lot of crypto-related build (not link) errors, so I'll
have to follow-up on those.

-chris


signature.asc (943 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Building tcnative on win32 [progress!]

Christopher Schultz-2
All,

On 6/18/14, 1:58 PM, Christopher Schultz wrote:
> I'm getting a lot of crypto-related build (not link) errors, so I'll
> have to follow-up on those.

Okay, so it seems that OpenSSL's include file is ... broken.

C:\Users\IEUser\Desktop>type
%MYTEMP%\build\openssl-1.0.1h\include\openssl\openslv.h
../../crypto/opensslv.h

That's not a valid C header file. It comes right out of the OpenSSL
source distribution, and its used by tcnative directly. All kinds of
compiler errors are thrown when that gets included from
include/ssl_private.h

The file OPENSSL_HOME/include/openssl/../../crypto/opensslv.h does
contain a legitimate C header file.

I checked on my Mac and a Linux box and I can confirm that
/usr/include/openssl/opensslv.h contains what the
../../crypto/opensslv.h file referenced above contains.

It seems that the OpenSSL source package contains symlinks. I haven't
found a command-line tar program on win32 that can expand an archive
containing symlinks.

This is getting frustrating. :(

-chris


signature.asc (943 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

RE: Building tcnative on win32

Konstantin Preißer-3
In reply to this post by Christopher Schultz-2
Hi Christopher,

> -----Original Message-----
> From: Christopher Schultz [mailto:[hidden email]]
> Sent: Wednesday, June 18, 2014 7:05 PM

> >> First, I want the script to fetch as much stuff as possible for the user
> >> instead of relying on a great deal of previously-installed software. My
> >> current requirements -- other than MSVC++ -- are for wget.exe and
> >> 7za.exe to be available. If anyone knows how to script an HTTP download
> >> or a ZIP-decompress from a Windows command-line without either of
> those
> >> tools, I'd greatly appreciate some insight.
> >
> > Maybe Java + Apache Ant?
> >
> > Or maybe NAnt has some tasks for that
> > http://nant.sourceforge.net/
>
> I considered Nant, but I wanted to limit the dependencies required to
> build tcnative.

Would Windows Powershell in this case work for you (as mentioned in my previous email)?
For example, to download a ZIP file and extract it, you could use a script like this that downloads Tomcat 8.0.8 x64 ZIP file and extract it to a "work" subfolder:

[[[
try
{
    # Add "System.IO.Compression.FileSystem" assembly so we can extract ZIP archives.
    # Note: This requires .Net 4.5 or higher (included since Windows 8 / Server 2012)
    Add-Type -AssemblyName System.IO.Compression.FileSystem

    # Work folder where we download and extract files.
    $workFolder = Join-Path $pwd "work"

    # Remove the work folder if it exists, then (re-)create it.
    If (Test-Path $workFolder) {
        Remove-Item $workFolder -Recurse
    }
    (New-Item $workFolder -ItemType directory) | Out-Null

    $downloadUrl = "http://apache.openmirror.de/tomcat/tomcat-8/v8.0.8/bin/apache-tomcat-8.0.8-windows-x64.zip"
    $downloadPath = Join-Path $workFolder "MyTomcatDownload.zip"

    Write-Host "Downloading $downloadUrl to $downloadPath ..."
    Write-Host ""

    # Note: "Invoke-WebRequest http://xxx -OutFile xxx" will read the whole content to memory, then writing it to disk.
    (New-Object System.Net.WebClient).DownloadFile($downloadUrl, $downloadPath)

    # Extract the ZIP archive to the current directory.
    Write-Host "Extracting ZIP archive..."
    Write-Host ""
    [System.IO.Compression.ZipFile]::ExtractToDirectory($downloadPath, $workFolder)

    Write-Host "Finished."
}
catch
{
    Write-Host ("Exception: " + $_.Exception.GetType().FullName) -ForegroundColor Red
    Write-Host ("Message: " + $_.Exception.Message) -ForegroundColor Red
}
]]]


To run the script if it is saved as "MyScript1.ps" in the current directory, you can use the following command (setting ExecutionPolicy is required to allow to run scripts in the current session):

Powershell.exe -ExecutionPolicy RemoteSigned .\MyScript.ps1


Hope this helps.

Regards,
Konstantin Preißer


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: Building tcnative on win32 [progress!]

Konstantin Preißer-3
In reply to this post by Christopher Schultz-2
Hi Christopher,

> -----Original Message-----
> From: Christopher Schultz [mailto:[hidden email]]
> Sent: Wednesday, June 18, 2014 9:53 PM
 
> It seems that the OpenSSL source package contains symlinks. I haven't
> found a command-line tar program on win32 that can expand an archive
> containing symlinks.
>
> This is getting frustrating. :(

WinRAR can create Symlinks on Windows when extracting .tar.gz files (Note that by default, you need administrative rights to create a Symlink on a NTFS drive).

Although Winrar.exe isn't a command-line tool, you can start it from command-line to extract a file. Unfortunately, when I try to run this command from an elevated cmd.exe to extract the OpenSSL source code package, WinRAR fails to create the Symlinks ("the system cannot find the specified path"):
Start /wait "" "%ProgramFiles%\WinRAR\winrar.exe" x openssl-1.0.1h.tar.gz

However, when opening an elevated Winrar.exe (GUI Version) and extract the .tar.gz file there, WinRAR successfully creates the symlinks though. Maybe there is some bug in Winrar so it cannot create symlinks under some conditions.


Regards,
Konstantin Preißer




---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: Building tcnative on win32 [progress!]

Konstantin Preißer-3
> -----Original Message-----
> From: Konstantin Preißer [mailto:[hidden email]]
> Sent: Wednesday, June 18, 2014 11:42 PM

> Although Winrar.exe isn't a command-line tool, you can start it from
> command-line to extract a file. Unfortunately, when I try to run this
> command from an elevated cmd.exe to extract the OpenSSL source code
> package, WinRAR fails to create the Symlinks ("the system cannot find the
> specified path"):
> Start /wait "" "%ProgramFiles%\WinRAR\winrar.exe" x openssl-1.0.1h.tar.gz

OK, I think the problem is when running the above command, WinRAR uses a relative path to create a symlink which fails. It works if you specify an absolute path as target:

Start /wait "" "%ProgramFiles%\WinRAR\winrar.exe" x openssl-1.0.1h.tar.gz C:\MyFolder\OpenSSL-Output\

With this command it will extract it to C:\MyFolder\OpenSSL-Output\ and create the symlinks successfully.


Unfortunately, WinRAR is proprietary software (not free). I have tried 7-Zip, but instead of actually creating the symlinks, it just seems to copy the file to the path where the symlink should be created.


Regards,
Konstantin Preißer


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Building tcnative on win32 [progress!]

Christopher Schultz-2
Konstantin,

On 6/18/14, 5:54 PM, Konstantin Preißer wrote:

>> -----Original Message-----
>> From: Konstantin Preißer [mailto:[hidden email]]
>> Sent: Wednesday, June 18, 2014 11:42 PM
>
>> Although Winrar.exe isn't a command-line tool, you can start it from
>> command-line to extract a file. Unfortunately, when I try to run this
>> command from an elevated cmd.exe to extract the OpenSSL source code
>> package, WinRAR fails to create the Symlinks ("the system cannot find the
>> specified path"):
>> Start /wait "" "%ProgramFiles%\WinRAR\winrar.exe" x openssl-1.0.1h.tar.gz
>
> OK, I think the problem is when running the above command, WinRAR uses a relative path to create a symlink which fails. It works if you specify an absolute path as target:
>
> Start /wait "" "%ProgramFiles%\WinRAR\winrar.exe" x openssl-1.0.1h.tar.gz C:\MyFolder\OpenSSL-Output\
>
> With this command it will extract it to C:\MyFolder\OpenSSL-Output\ and create the symlinks successfully.
>
>
> Unfortunately, WinRAR is proprietary software (not free). I have
> tried 7-Zip, but instead of actually creating the symlinks, it just
> seems to copy the file to the path where the symlink should be
> created.
I was using 7za.exe and it was creating files whose contents contained
the path-target of the symlink instead of what you describe above.

I'd be perfectly happy if it would just duplicate the files instead of
creating symlinks, but that does not seem to be the actual behavior.

-chris


signature.asc (943 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

RE: Building tcnative on win32 [progress!]

Konstantin Preißer-3
Hi Christopher,

> -----Original Message-----
> From: Christopher Schultz [mailto:[hidden email]]
> Sent: Thursday, June 19, 2014 1:31 AM
> To: Tomcat Developers List
> Subject: Re: Building tcnative on win32 [progress!]
>
> Konstantin,
>
> On 6/18/14, 5:54 PM, Konstantin Preißer wrote:
> >> -----Original Message-----
> >> From: Konstantin Preißer [mailto:[hidden email]]
> >> Sent: Wednesday, June 18, 2014 11:42 PM
> >
> >> Although Winrar.exe isn't a command-line tool, you can start it from
> >> command-line to extract a file. Unfortunately, when I try to run this
> >> command from an elevated cmd.exe to extract the OpenSSL source code
> >> package, WinRAR fails to create the Symlinks ("the system cannot find the
> >> specified path"):
> >> Start /wait "" "%ProgramFiles%\WinRAR\winrar.exe" x openssl-
> 1.0.1h.tar.gz
> >
> > OK, I think the problem is when running the above command, WinRAR uses
> a relative path to create a symlink which fails. It works if you specify an
> absolute path as target:
> >
> > Start /wait "" "%ProgramFiles%\WinRAR\winrar.exe" x openssl-
> 1.0.1h.tar.gz C:\MyFolder\OpenSSL-Output\
> >
> > With this command it will extract it to C:\MyFolder\OpenSSL-Output\ and
> create the symlinks successfully.
> >
> >
> > Unfortunately, WinRAR is proprietary software (not free). I have
> > tried 7-Zip, but instead of actually creating the symlinks, it just
> > seems to copy the file to the path where the symlink should be
> > created.
>
> I was using 7za.exe and it was creating files whose contents contained
> the path-target of the symlink instead of what you describe above.
>
> I'd be perfectly happy if it would just duplicate the files instead of
> creating symlinks, but that does not seem to be the actual behavior.

Sorry, you are right: After I looked at the files, I saw that 7z.exe created files which contained only the path for the symlinks, instead of duplicating the file.  :(

Unfortunately, I also have not found another tool which could duplicate the symlink file or actually create the NTFS symlink other than WinRAR.


Regards,
Konstantin Preißer


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Building tcnative on win32 [progress!]

Christopher Schultz-2
Konstantin,

On 6/18/14, 7:40 PM, Konstantin Preißer wrote:

> Hi Christopher,
>
>> -----Original Message-----
>> From: Christopher Schultz [mailto:[hidden email]]
>> Sent: Thursday, June 19, 2014 1:31 AM
>> To: Tomcat Developers List
>> Subject: Re: Building tcnative on win32 [progress!]
>>
>> Konstantin,
>>
>> On 6/18/14, 5:54 PM, Konstantin Preißer wrote:
>>>> -----Original Message-----
>>>> From: Konstantin Preißer [mailto:[hidden email]]
>>>> Sent: Wednesday, June 18, 2014 11:42 PM
>>>
>>>> Although Winrar.exe isn't a command-line tool, you can start it from
>>>> command-line to extract a file. Unfortunately, when I try to run this
>>>> command from an elevated cmd.exe to extract the OpenSSL source code
>>>> package, WinRAR fails to create the Symlinks ("the system cannot find the
>>>> specified path"):
>>>> Start /wait "" "%ProgramFiles%\WinRAR\winrar.exe" x openssl-
>> 1.0.1h.tar.gz
>>>
>>> OK, I think the problem is when running the above command, WinRAR uses
>> a relative path to create a symlink which fails. It works if you specify an
>> absolute path as target:
>>>
>>> Start /wait "" "%ProgramFiles%\WinRAR\winrar.exe" x openssl-
>> 1.0.1h.tar.gz C:\MyFolder\OpenSSL-Output\
>>>
>>> With this command it will extract it to C:\MyFolder\OpenSSL-Output\ and
>> create the symlinks successfully.
>>>
>>>
>>> Unfortunately, WinRAR is proprietary software (not free). I have
>>> tried 7-Zip, but instead of actually creating the symlinks, it just
>>> seems to copy the file to the path where the symlink should be
>>> created.
>>
>> I was using 7za.exe and it was creating files whose contents contained
>> the path-target of the symlink instead of what you describe above.
>>
>> I'd be perfectly happy if it would just duplicate the files instead of
>> creating symlinks, but that does not seem to be the actual behavior.
>
> Sorry, you are right: After I looked at the files, I saw that 7z.exe created files which contained only the path for the symlinks, instead of duplicating the file.  :(
>
> Unfortunately, I also have not found another tool which could duplicate the symlink file or actually create the NTFS symlink other than WinRAR.
Seems like the intertubes agree. :(

After more digging, it appears that OpenSSL, after making things a pain
in the ass (broken "symlinks" on win32 tarball-expansions), the build
process creates an inc32/openssl directory containing all of the C
header files you might need.

So, instead of sensibly using the OPENSSL_HOME where I expanded the
source and build the library, I can simply copy inc32/openssl/*.h to
somewhere else that then has include/openssl/*.h where tcnative's build
can find it.

tcnative is now compling, but it's having trouble locating the APR
libraries. Time to take a break for some more World Cup action. It's
much more fun than fighting builds on Windows.

-chris


signature.asc (943 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Building tcnative on win32 [alpha]

Christopher Schultz-2
In reply to this post by Christopher Schultz-2
All,

I have a first-stab at a Windows batch script that can do almost
everything. I've attached it to this post.

*NOTE THAT YOU DO NOT HAVE TO BUY/OWN ANY MICROSOFT SOFTWARE TO BUILD
OpenSSL, APR, and tcnative*

I did all my work on a Windows 8.1 VirtualBox VM that I downloaded from
https://www.modern.ie/en-us/virtualization-tools. So, if you want to
configure a build environment and don't want to jack-up your own
computer to get this all done, you can build OpenSSL, APR, and tcnative
all without buying any software.

When you are done, you can copy the binaries to a real machine and
scuttle the VM. Sounds like a good deal to me.

There are a few prerequisites:

1. If you don't already have Microsoft Visual Studio C++ available,
   download and install Microsoft Visual Studio Express 2013, which
   you can find here:
   http://www.microsoft.com/en-us/download/details.aspx?id=40787

   Note that, in desperation, at some point, I also installed the
   Microsoft Windows SDK. I'm not sure if this is necessary, or if
   either of these packages will be sufficient. In the future, I'll
   untangle the web and post simpler instructions:
   http://www.microsoft.com/en-us/download/details.aspx?id=8279

2. Download and install Java JDK. You might be able to get away with
   the JRE but I got the whole JDK because I didn't want to mess-around
   with missing components (like header files).

3. Download GnuWin32 wget from here:
   http://gnuwin32.sourceforge.net/packages/wget.htm

4. Download 7zip command-line version (7za.exe) from here:
   http://www.7-zip.org/download.html

5. Put wget.exe and 7za.exe into a directory called "tools" in some
   work directory. build-tcnative.bat will use a working directory
   under %TEMP% for downloads and builds. You can also change the
   value of the %TOOLS% environment variable in the script to point
   wherever you want, but I haven't gone back to replace all
   references to tools\foo.exe to %TOOLS%\foo.exe so it might not work
   yet. Also put build-tcnative.bat into this work directory
   (not "tools", but one level up).

6. Dump this text into a file called patch.exe.manifest in your work
   directory:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<assembly xmlns="urn:schemas-microsoft-com:asm.v1"
manifestVersion="1.0">
<assemblyIdentity
version="2.5.0.0"
processorArchitecture="X86"
name="Patch"
type="win32"
/>
<description>A tool for applying diff-generated patches.</description>
<!-- Identify the application security requirements. -->
<trustInfo xmlns="urn:schemas-microsoft-com:asm.v3">
<security>
<requestedPrivileges>
<requestedExecutionLevel
level="asInvoker"
uiAccess="false"/>
</requestedPrivileges>
</security>
</trustInfo>
</assembly>


7. Launch cmd.exe and run VC++'s vcvars32.bat script or, better yet,
   launch the cmd.exe config that gets installed when you install
   VC++.

8. Change to the directory containing your "tools" directory from step
   5, then run build-tcnative.bat.

On my VM, the build gets as far as linking tcnative before it fails. It
fails because I haven't copied the artifacts for APR to the right
places, yet.

Hopefully, this will be enough to get some other folks started who are
waiting for an update to tcnative-1.1.30 that includes OpenSSL 1.0.1h.

Enjoy,
-chris

build-tcnative.bat (9K) Download Attachment
signature.asc (943 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

RE: Building tcnative on win32 [alpha]

Konstantin Preißer-3
Hi Christopher,

good work for creating a script to build tcnative. Below are some comments/review of it.

> -----Original Message-----
> From: Christopher Schultz [mailto:[hidden email]]
> Sent: Thursday, June 19, 2014 3:21 PM
>
> All,
>
> I have a first-stab at a Windows batch script that can do almost
> everything. I've attached it to this post.
>
> *NOTE THAT YOU DO NOT HAVE TO BUY/OWN ANY MICROSOFT SOFTWARE
> TO BUILD
> OpenSSL, APR, and tcnative*
>
> I did all my work on a Windows 8.1 VirtualBox VM that I downloaded from
> https://www.modern.ie/en-us/virtualization-tools. So, if you want to
> configure a build environment and don't want to jack-up your own
> computer to get this all done, you can build OpenSSL, APR, and tcnative
> all without buying any software.
>
> When you are done, you can copy the binaries to a real machine and
> scuttle the VM. Sounds like a good deal to me.
>
> There are a few prerequisites:
>
> 1. If you don't already have Microsoft Visual Studio C++ available,
>    download and install Microsoft Visual Studio Express 2013, which
>    you can find here:
>    http://www.microsoft.com/en-us/download/details.aspx?id=40787
>
>    Note that, in desperation, at some point, I also installed the
>    Microsoft Windows SDK. I'm not sure if this is necessary, or if
>    either of these packages will be sufficient. In the future, I'll
>    untangle the web and post simpler instructions:
>    http://www.microsoft.com/en-us/download/details.aspx?id=8279

If you installed Visual Studio, then you shouldn't need a Windows SDK as the necessary tools are already included in VS. (It would be more convenient if one doesn't need to install VS as it installs a lot of other components, but I don't think there is some other MS build environment - Windows SDK does not seem to include the necessary tools.)


>
> 2. Download and install Java JDK. You might be able to get away with
>    the JRE but I got the whole JDK because I didn't want to mess-around
>    with missing components (like header files).
>
> 3. Download GnuWin32 wget from here:
>    http://gnuwin32.sourceforge.net/packages/wget.htm

I downloaded the "Binaries Zip" file version of wget.exe, but it seems to require some additional DLLs that are not included. I then downloaded the "Setup" version of it, and copied wget.exe + all DLLs that the Setup installed to the tools directory.


>
> 4. Download 7zip command-line version (7za.exe) from here:
>    http://www.7-zip.org/download.html
>
> 5. Put wget.exe and 7za.exe into a directory called "tools" in some
>    work directory. build-tcnative.bat will use a working directory
>    under %TEMP% for downloads and builds. You can also change the
>    value of the %TOOLS% environment variable in the script to point
>    wherever you want, but I haven't gone back to replace all
>    references to tools\foo.exe to %TOOLS%\foo.exe so it might not work
>    yet. Also put build-tcnative.bat into this work directory
>    (not "tools", but one level up).
>
> 6. Dump this text into a file called patch.exe.manifest in your work
>    directory:
>
> <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
> <assembly xmlns="urn:schemas-microsoft-com:asm.v1"
> manifestVersion="1.0">
> <assemblyIdentity
> version="2.5.0.0"
> processorArchitecture="X86"
> name="Patch"
> type="win32"
> />
> <description>A tool for applying diff-generated patches.</description>
> <!-- Identify the application security requirements. -->
> <trustInfo xmlns="urn:schemas-microsoft-com:asm.v3">
> <security>
> <requestedPrivileges>
> <requestedExecutionLevel
> level="asInvoker"
> uiAccess="false"/>
> </requestedPrivileges>
> </security>
> </trustInfo>
> </assembly>

Ah, I now understand why you wrote in an earlier mail that this patch tool needs administrator rights - it does not include a proper manifest. Starting with Windows Vista, Windows uses such manifest to determine if an EXE should be run with Administrator rights (showing an UAC dialog) or user rights. For EXEs that don't include such a manifest, it cannot know whether it should run as administrator, so it checks the filename for patterns like "setup", "install", "patch", "update" etc.

So, you should be fine by renaming "patch.txt" to something like "pt.exe" - it will make the UAC symbol on the file go away, and it will run without administrator rights.


> 7. Launch cmd.exe and run VC++'s vcvars32.bat script or, better yet,
>    launch the cmd.exe config that gets installed when you install
>    VC++.
>
> 8. Change to the directory containing your "tools" directory from step
>    5, then run build-tcnative.bat.

I think there are some problems with this BAT file.

1) I think you unintentionally used double-quotes on several places where normally they shouldn't be, e.g. for setting environment variables:
SET MYTEMP="%TEMP%\build-tcnative"

Note, that in CMD.EXE, the SET command includes everything after the =, so the double quotes are actually included in the environment variable. This could cause problems if concatenating it with some command; however, cmd.exe seems to accept commands like '"C:\Windows\system32"\calc.exe'; but it would be better to correctly set the environment variables.


2) When I run tcnative.dll from a CMD.exe that has run VC++'s vcvars32.bat, then I get a problem when downloading patch.exe. This seems because the URL that is included in the bat file (http://downloads.sourceforge.net/project/gnuwin32/patch/2.5.9-7/patch-2.5.9-7-bin.zip) leads to a HTML page which generates a download link, instead of actually downloading the EXE file.

It seems to work if I use a URL like http://vorboss.dl.sourceforge.net/project/gnuwin32/patch/2.5.9-7/patch-2.5.9-7-bin.zip, but I need to insert a 'md "%myTemp%\downloads' command to create the directory; otherwise wget seems unable to download it.


3) There also seems to be something wrong with this command:
mt.exe -manifest "%MYTEMPpatch.exe.manifest "-outputresource:%TOOLS%\patch.exe;#1"
I think it should be something like:
mt.exe -manifest "patch.exe.manifest "-outputresource:%TOOLS%\patch.exe;#1"


4) The script seems to require perl.exe (you did not mention this in your mail ;-)
I have not yet tried to run it with perl installed, as I get an error that patch.exe cannot find "apr-1.5.patch".


Regarding the dependency to wget.exe, I think that can be avoided by making a Powershell Script (as said earlier) as it allows to directly download a file from command line. I think this also will make the script easier to write as you can write more in the style of an programming/scripting language instead of Windows/DOS commands.

I can look about creating a Powershell version of the script if you have finished it. What do you think?


>
> On my VM, the build gets as far as linking tcnative before it fails. It
> fails because I haven't copied the artifacts for APR to the right
> places, yet.
>
> Hopefully, this will be enough to get some other folks started who are
> waiting for an update to tcnative-1.1.30 that includes OpenSSL 1.0.1h.
>
> Enjoy,
> -chris

One other thing: Do you know which version of VC++ was used to date to build Tomcat Native? If I open it with some EXE inspection tools, I can see that it has a dependency to msvcrt.dll, which seems like an older version of VC++ (maybe VC++ 6.0 which is a really old version - but I'm not sure).
I don't know much about building native (C/C++) apps; however if I create a C++ DLL with Visual Studio 2013, then the DLL has a dependency to msvcr120.dll, which maybe is not included by default in Windows (at least not in earlier versions of Windows like Windows 7/2008R2).

I don't know if it is possible to change the dependency to the older msvcrt.dll or include the DLL in the build; but if not, then maybe users will need to install a "Visual C++ Redistributable Packages for Visual Studio 2013" package [1], like PHP does this on their Windows binaries site [2].


[1] http://www.microsoft.com/en-us/download/details.aspx?id=40784
[2] http://windows.php.net/download/


Regards,
Konstantin Preißer


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Building tcnative on win32 [alpha]

Christopher Schultz-2
In reply to this post by Christopher Schultz-2
All,

Attached is an updated script that can build *a* tcnative binary, but I
haven't tested anything, yet. I suspect the whole build is not correct
because tcnative-1.dll is far too small (in bytes) to be correct:

File             My Build    1.1.30 Official Build
openssl.exe       1692160    1630208
tcnative-1.dll     176640    1141760
tcnative-1-src.pdb 176128     176128

So, tcnative does not seem to be building statically (wild guess). If
anyone can give me some clues as to why that's happening, it would be
very helpful to me.

This script also only builds the 32-bit binaries. Once I get these
working, switching to build the x86-64 and IA64 binaries should be trivial.

Enjoy,
-chris

On 6/19/14, 9:21 AM, Christopher Schultz wrote:

> All,
>
> I have a first-stab at a Windows batch script that can do almost
> everything. I've attached it to this post.
>
> *NOTE THAT YOU DO NOT HAVE TO BUY/OWN ANY MICROSOFT SOFTWARE TO BUILD
> OpenSSL, APR, and tcnative*
>
> I did all my work on a Windows 8.1 VirtualBox VM that I downloaded from
> https://www.modern.ie/en-us/virtualization-tools. So, if you want to
> configure a build environment and don't want to jack-up your own
> computer to get this all done, you can build OpenSSL, APR, and tcnative
> all without buying any software.
>
> When you are done, you can copy the binaries to a real machine and
> scuttle the VM. Sounds like a good deal to me.
>
> There are a few prerequisites:
>
> 1. If you don't already have Microsoft Visual Studio C++ available,
>    download and install Microsoft Visual Studio Express 2013, which
>    you can find here:
>    http://www.microsoft.com/en-us/download/details.aspx?id=40787
>
>    Note that, in desperation, at some point, I also installed the
>    Microsoft Windows SDK. I'm not sure if this is necessary, or if
>    either of these packages will be sufficient. In the future, I'll
>    untangle the web and post simpler instructions:
>    http://www.microsoft.com/en-us/download/details.aspx?id=8279
>
> 2. Download and install Java JDK. You might be able to get away with
>    the JRE but I got the whole JDK because I didn't want to mess-around
>    with missing components (like header files).
>
> 3. Download GnuWin32 wget from here:
>    http://gnuwin32.sourceforge.net/packages/wget.htm
>
> 4. Download 7zip command-line version (7za.exe) from here:
>    http://www.7-zip.org/download.html
>
> 5. Put wget.exe and 7za.exe into a directory called "tools" in some
>    work directory. build-tcnative.bat will use a working directory
>    under %TEMP% for downloads and builds. You can also change the
>    value of the %TOOLS% environment variable in the script to point
>    wherever you want, but I haven't gone back to replace all
>    references to tools\foo.exe to %TOOLS%\foo.exe so it might not work
>    yet. Also put build-tcnative.bat into this work directory
>    (not "tools", but one level up).
>
> 6. Dump this text into a file called patch.exe.manifest in your work
>    directory:
>
> <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
> <assembly xmlns="urn:schemas-microsoft-com:asm.v1"
> manifestVersion="1.0">
> <assemblyIdentity
> version="2.5.0.0"
> processorArchitecture="X86"
> name="Patch"
> type="win32"
> />
> <description>A tool for applying diff-generated patches.</description>
> <!-- Identify the application security requirements. -->
> <trustInfo xmlns="urn:schemas-microsoft-com:asm.v3">
> <security>
> <requestedPrivileges>
> <requestedExecutionLevel
> level="asInvoker"
> uiAccess="false"/>
> </requestedPrivileges>
> </security>
> </trustInfo>
> </assembly>
>
>
> 7. Launch cmd.exe and run VC++'s vcvars32.bat script or, better yet,
>    launch the cmd.exe config that gets installed when you install
>    VC++.
>
> 8. Change to the directory containing your "tools" directory from step
>    5, then run build-tcnative.bat.
>
> On my VM, the build gets as far as linking tcnative before it fails. It
> fails because I haven't copied the artifacts for APR to the right
> places, yet.
>
> Hopefully, this will be enough to get some other folks started who are
> waiting for an update to tcnative-1.1.30 that includes OpenSSL 1.0.1h.
>
> Enjoy,
> -chris
>

build-tcnative.bat (12K) Download Attachment
signature.asc (943 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Building tcnative on win32 [alpha]

Christopher Schultz-2
In reply to this post by Konstantin Preißer-3
Konstantin,

On 6/19/14, 12:20 PM, Konstantin Preißer wrote:
> good work for creating a script to build tcnative. Below are some
> comments/review of it.

Thanks for your review. See below for more.

FYI I *have* been reading your prior comments. I didn't want to get into
Power Shell because of my lack of experience with it: working with
plain-old batch files is something I am more familiar with, even though
it is astoundingly miserable when compared to UNIX-style shell scripting.

>> -----Original Message-----
>> From: Christopher Schultz [mailto:[hidden email]]
>> Sent: Thursday, June 19, 2014 3:21 PM
>>
>> All,
>>
>> I have a first-stab at a Windows batch script that can do almost
>> everything. I've attached it to this post.
>>
>> *NOTE THAT YOU DO NOT HAVE TO BUY/OWN ANY MICROSOFT SOFTWARE
>> TO BUILD
>> OpenSSL, APR, and tcnative*
>>
>> I did all my work on a Windows 8.1 VirtualBox VM that I downloaded from
>> https://www.modern.ie/en-us/virtualization-tools. So, if you want to
>> configure a build environment and don't want to jack-up your own
>> computer to get this all done, you can build OpenSSL, APR, and tcnative
>> all without buying any software.
>>
>> When you are done, you can copy the binaries to a real machine and
>> scuttle the VM. Sounds like a good deal to me.
>>
>> There are a few prerequisites:
>>
>> 1. If you don't already have Microsoft Visual Studio C++ available,
>>    download and install Microsoft Visual Studio Express 2013, which
>>    you can find here:
>>    http://www.microsoft.com/en-us/download/details.aspx?id=40787
>>
>>    Note that, in desperation, at some point, I also installed the
>>    Microsoft Windows SDK. I'm not sure if this is necessary, or if
>>    either of these packages will be sufficient. In the future, I'll
>>    untangle the web and post simpler instructions:
>>    http://www.microsoft.com/en-us/download/details.aspx?id=8279
>
> If you installed Visual Studio, then you shouldn't need a Windows SDK
> as the necessary tools are already included in VS. (It would be more
> convenient if one doesn't need to install VS as it installs a lot of
> other components, but I don't think there is some other MS build
> environment - Windows SDK does not seem to include the necessary
> tools.)
Good to know. When I've got something that is mostly working, I'll
re-build the entire VM and document the procedure better.

>> 2. Download and install Java JDK. You might be able to get away with
>>    the JRE but I got the whole JDK because I didn't want to mess-around
>>    with missing components (like header files).
>>
>> 3. Download GnuWin32 wget from here:
>>    http://gnuwin32.sourceforge.net/packages/wget.htm
>
> I downloaded the "Binaries Zip" file version of wget.exe, but it
> seems
> to require some additional DLLs that are not included. I then downloaded
> the "Setup" version of it, and copied wget.exe + all DLLs that the Setup
> installed to the tools directory.
I think I may have used a slightly different wget.exe. I'll track that
one down, too. The script gives you two possibilities for where you
might want to get wget.exe. I think I used the "first" of the two listed.

>> 4. Download 7zip command-line version (7za.exe) from here:
>>    http://www.7-zip.org/download.html
>>
>> 5. Put wget.exe and 7za.exe into a directory called "tools" in some
>>    work directory. build-tcnative.bat will use a working directory
>>    under %TEMP% for downloads and builds. You can also change the
>>    value of the %TOOLS% environment variable in the script to point
>>    wherever you want, but I haven't gone back to replace all
>>    references to tools\foo.exe to %TOOLS%\foo.exe so it might not work
>>    yet. Also put build-tcnative.bat into this work directory
>>    (not "tools", but one level up).
>>
>> 6. Dump this text into a file called patch.exe.manifest in your work
>>    directory:
>>
>> <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
>> <assembly xmlns="urn:schemas-microsoft-com:asm.v1"
>> manifestVersion="1.0">
>> <assemblyIdentity
>> version="2.5.0.0"
>> processorArchitecture="X86"
>> name="Patch"
>> type="win32"
>> />
>> <description>A tool for applying diff-generated patches.</description>
>> <!-- Identify the application security requirements. -->
>> <trustInfo xmlns="urn:schemas-microsoft-com:asm.v3">
>> <security>
>> <requestedPrivileges>
>> <requestedExecutionLevel
>> level="asInvoker"
>> uiAccess="false"/>
>> </requestedPrivileges>
>> </security>
>> </trustInfo>
>> </assembly>
>
> Ah, I now understand why you wrote in an earlier mail that this
> patch
> tool needs administrator rights - it does not include a proper manifest.
> Starting with Windows Vista, Windows uses such manifest to determine if
> an EXE should be run with Administrator rights (showing an UAC dialog)
> or user rights. For EXEs that don't include such a manifest, it cannot
> know whether it should run as administrator, so it checks the filename
> for patterns like "setup", "install", "patch", "update" etc.
Oh, so it I just renamed the binary to something else, it probably
wouldn't complain? That's great security right there.

> So, you should be fine by renaming "patch.txt" to something like
> "pt.exe" - it will make the UAC symbol on the file go away, and it
> will run without administrator rights.

Nice. I'm happy patching it for now, but it does represent another
prerequisite for someone building the whole thing: getting the manifest
file onto the disk. If batch files had "here" documents, I'd just put
the manifest right into the script, but I can't... and escaping all that
stuff for ECHO line-of-manifest >> manifest.xml just isn't in the cards
today ;)

>> 7. Launch cmd.exe and run VC++'s vcvars32.bat script or, better yet,
>>    launch the cmd.exe config that gets installed when you install
>>    VC++.
>>
>> 8. Change to the directory containing your "tools" directory from step
>>    5, then run build-tcnative.bat.
>
> I think there are some problems with this BAT file.
>
> 1) I think you unintentionally used double-quotes on several places
> where normally they shouldn't be, e.g. for setting environment variables:
>
> SET MYTEMP="%TEMP%\build-tcnative"
While I agree, it does not seem to break anything. I made that mistake
with the PATH environment variable and then *everything* stopped
working. No fun figuring out that SET FOO="blah" and SET FOO=blah are
two different things. Thanks, DOS. :p

> Note, that in CMD.EXE, the SET command includes everything after the
> =, so the double quotes are actually included in the environment
> variable. This could cause problems if concatenating it with some
> command; however, cmd.exe seems to accept commands like
> '"C:\Windows\system32"\calc.exe'; but it would be better to correctly
> set the environment variables.

I'm happy to turn over my script to someone to do some cleanup before,
say, including it in the distribution.

> 2) When I run tcnative.dll from a CMD.exe that has run VC++'s
> vcvars32.bat, then I get a problem when downloading patch.exe. This
> seems because the URL that is included in the bat file
> (http://downloads.sourceforge.net/project/gnuwin32/patch/2.5.9-7/patch-2.5.9-7-bin.zip)
> leads to a HTML page which generates a download link, instead of
> actually downloading the EXE file.

Oops. Perhaps I pasted that link incorrectly at some point. I'll fix
that before final publication.

> It seems to work if I use a URL like
> http://vorboss.dl.sourceforge.net/project/gnuwin32/patch/2.5.9-7/patch-2.5.9-7-bin.zip,
> but I need to insert a 'md "%myTemp%\downloads' command to create the
> directory; otherwise wget seems unable to download it.

Okay, I'll check that out.

> 3) There also seems to be something wrong with this command:
> mt.exe -manifest "%MYTEMPpatch.exe.manifest "-outputresource:%TOOLS%\patch.exe;#1"
> I think it should be something like:
> mt.exe -manifest "patch.exe.manifest "-outputresource:%TOOLS%\patch.exe;#1"

Got it.

> 4) The script seems to require perl.exe (you did not mention this in your mail ;-)

Yeah, I totally forgot to mention that. I figured I'd include it with my
next update (which I also forgot).

> I have not yet tried to run it with perl installed, as I get an error
> that patch.exe cannot find "apr-1.5.patch".

Hah.. yes. The APR patch includes in the tcnative source distro doesn't
seem to work with APR 1.5.1. I think it's because it was made against
APR 1.4.8 and the two versions are different enough that patch gives up.
Here's the patch for APR 1.5.1:

---
\Users\IEUser\AppData\Local\Temp\build-tcnative\build\apr-1.5.1\include\apr.hw
     Fri Jun 07 08:57:56 2013
+++ apr.hw      Wed Jun 18 07:49:57 2014
@@ -282,7 +282,7 @@
 #define APR_HAVE_IN_ADDR        1
 #define APR_HAVE_INET_ADDR      1
 #define APR_HAVE_INET_NETWORK   0
-#define APR_HAVE_IPV6           0
+#define APR_HAVE_IPV6           1
 #define APR_HAVE_MEMMOVE        1
 #define APR_HAVE_SETRLIMIT      0
 #define APR_HAVE_SIGACTION      0
@@ -346,7 +346,7 @@
 /* If we have a TCP implementation that can be "corked", what flag
  * do we use?
  */
-#define APR_TCP_NOPUSH_FLAG       @apr_tcp_nopush_flag@
+#define APR_TCP_NOPUSH_FLAG       0

 /* Is the TCP_NODELAY socket option inherited from listening sockets?
  */


> Regarding the dependency to wget.exe, I think that can be avoided by
> making a Powershell Script (as said earlier) as it allows to directly
> download a file from command line. I think this also will make the
> script easier to write as you can write more in the style of an
> programming/scripting language instead of Windows/DOS commands.
>
> I can look about creating a Powershell version of the script if you
> have finished it. What do you think?

Anything that makes this procedure replicable by someone else is a Good
Thing.

After Mladen's encouragement that "it just takes 10 minutes" it's been
pretty frustrating trying to create these artifacts.

>> On my VM, the build gets as far as linking tcnative before it fails. It
>> fails because I haven't copied the artifacts for APR to the right
>> places, yet.
>>
>> Hopefully, this will be enough to get some other folks started who are
>> waiting for an update to tcnative-1.1.30 that includes OpenSSL 1.0.1h.
>>
>> Enjoy,
>> -chris
>
> One other thing: Do you know which version of VC++ was used to date
> to build Tomcat Native?
Mladen mentioned it at one time, but I can't remember where. He also
mentioned how to modify one of the components to build it statically.
I'm searching for that reference right now but I'm having a hard time
finding it.

Basically, I'm working off of these resources:
1. https://issues.apache.org/bugzilla/show_bug.cgi?id=56363#c10
2. INSTALL.W32 script in OpenSSL distribution
3. Thread on apr-dev@ : http://markmail.org/thread/ecu3gwgr77cb6mtm
4. Random directory listings after things have built. ;)
   That's how I found out that OpenSSL puts their headers in inc32/
after the build. That's also how I found that there is an out32/ and
out32dll/ directory where the former has the statically-linked version
of openssl.exe in it.

> If I open it with some EXE inspection tools, I can see that it has a
> dependency to msvcrt.dll, which seems like an older version of VC++
> (maybe VC++ 6.0 which is a really old version - but I'm not sure).
> I don't know much about building native (C/C++) apps; however if I
> create a C++ DLL with Visual Studio 2013, then the DLL has a
> dependency to msvcr120.dll, which maybe is not included by default in
> Windows (at least not in earlier versions of Windows like Windows
> 7/2008R2).

Whatever Mladen tends to build it with is probably correct.

How does one inspect a DLL in this way to see dependencies?

Thanks,
-chris


signature.asc (943 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

RE: Building tcnative on win32 [alpha]

Konstantin Preißer-3
Hi Christopher,

thank you for your reply.

> -----Original Message-----
> From: Christopher Schultz [mailto:[hidden email]]
> Sent: Thursday, June 19, 2014 6:57 PM
>
> Konstantin,
>
> On 6/19/14, 12:20 PM, Konstantin Preißer wrote:
> > good work for creating a script to build tcnative. Below are some
> > comments/review of it.
>
> Thanks for your review. See below for more.
>
> FYI I *have* been reading your prior comments. I didn't want to get into
> Power Shell because of my lack of experience with it: working with
> plain-old batch files is something I am more familiar with, even though
> it is astoundingly miserable when compared to UNIX-style shell scripting.

OK, no problem. I admit it is not easy to learn using Powershell as it has completely new syntax/commands. (It has some aliases to make transition easier, e.g. you can write echo instead of Write-Host, wget/curl instead of Invoke-WebRequest, cd instead of Set-Location etc.)


> > Ah, I now understand why you wrote in an earlier mail that this
> > patch
> > tool needs administrator rights - it does not include a proper manifest.
> > Starting with Windows Vista, Windows uses such manifest to determine if
> > an EXE should be run with Administrator rights (showing an UAC dialog)
> > or user rights. For EXEs that don't include such a manifest, it cannot
> > know whether it should run as administrator, so it checks the filename
> > for patterns like "setup", "install", "patch", "update" etc.
>
> Oh, so it I just renamed the binary to something else, it probably
> wouldn't complain? That's great security right there.

Yes - "Not complain" here means it does not run with administrator priviliges (If you explicitely run it as administrator, then you still get the UAC dialog). It is just some guessing of Windows that automatically request you to run a tool as administrator (UAC dialog) if it doesn't have a correct manifest, so this shouldn't be security-related.


> > 4) The script seems to require perl.exe (you did not mention this in your
> mail ;-)
>
> Yeah, I totally forgot to mention that. I figured I'd include it with my
> next update (which I also forgot).
>
> > I have not yet tried to run it with perl installed, as I get an error
> > that patch.exe cannot find "apr-1.5.patch".
>
> Hah.. yes. The APR patch includes in the tcnative source distro doesn't
> seem to work with APR 1.5.1. I think it's because it was made against
> APR 1.4.8 and the two versions are different enough that patch gives up.
> Here's the patch for APR 1.5.1:
>
> ---
> \Users\IEUser\AppData\Local\Temp\build-tcnative\build\apr-
> 1.5.1\include\apr.hw
>      Fri Jun 07 08:57:56 2013
> +++ apr.hw      Wed Jun 18 07:49:57 2014
> @@ -282,7 +282,7 @@
>  #define APR_HAVE_IN_ADDR        1
>  #define APR_HAVE_INET_ADDR      1
>  #define APR_HAVE_INET_NETWORK   0
> -#define APR_HAVE_IPV6           0
> +#define APR_HAVE_IPV6           1
>  #define APR_HAVE_MEMMOVE        1
>  #define APR_HAVE_SETRLIMIT      0
>  #define APR_HAVE_SIGACTION      0
> @@ -346,7 +346,7 @@
>  /* If we have a TCP implementation that can be "corked", what flag
>   * do we use?
>   */
> -#define APR_TCP_NOPUSH_FLAG       @apr_tcp_nopush_flag@
> +#define APR_TCP_NOPUSH_FLAG       0
>
>  /* Is the TCP_NODELAY socket option inherited from listening sockets?
>   */
>
>
> > Regarding the dependency to wget.exe, I think that can be avoided by
> > making a Powershell Script (as said earlier) as it allows to directly
> > download a file from command line. I think this also will make the
> > script easier to write as you can write more in the style of an
> > programming/scripting language instead of Windows/DOS commands.
> >
> > I can look about creating a Powershell version of the script if you
> > have finished it. What do you think?
>
> Anything that makes this procedure replicable by someone else is a Good
> Thing.
>
> After Mladen's encouragement that "it just takes 10 minutes" it's been
> pretty frustrating trying to create these artifacts.

OK.


> > One other thing: Do you know which version of VC++ was used to date
> > to build Tomcat Native?
>
> Mladen mentioned it at one time, but I can't remember where. He also
> mentioned how to modify one of the components to build it statically.
> I'm searching for that reference right now but I'm having a hard time
> finding it.
>
> Basically, I'm working off of these resources:
> 1. https://issues.apache.org/bugzilla/show_bug.cgi?id=56363#c10
> 2. INSTALL.W32 script in OpenSSL distribution
> 3. Thread on apr-dev@ : http://markmail.org/thread/ecu3gwgr77cb6mtm
> 4. Random directory listings after things have built. ;)
>    That's how I found out that OpenSSL puts their headers in inc32/
> after the build. That's also how I found that there is an out32/ and
> out32dll/ directory where the former has the statically-linked version
> of openssl.exe in it.
>
> > If I open it with some EXE inspection tools, I can see that it has a
> > dependency to msvcrt.dll, which seems like an older version of VC++
> > (maybe VC++ 6.0 which is a really old version - but I'm not sure).
> > I don't know much about building native (C/C++) apps; however if I
> > create a C++ DLL with Visual Studio 2013, then the DLL has a
> > dependency to msvcr120.dll, which maybe is not included by default in
> > Windows (at least not in earlier versions of Windows like Windows
> > 7/2008R2).
>
> Whatever Mladen tends to build it with is probably correct.
>
> How does one inspect a DLL in this way to see dependencies?

I used the tool "Dependency Walker" from [1]. When I open tcnative.dll from current Tomcat 8.0.8 Windows x64 build, it shows following entries:

User32.dll
Psapi.dll
Shlwapi.dll
Kernel32.dll
Advapi32.dll
Ws2_32.dll
Mswsock.dll
Msvcrt.dll

So it seems to use Msvcrt.dll as C runtime library. However, I have not yet tried your updated script to see what dependencies that build will have.


Regards,
Konstantin Preißer


[1] http://www.dependencywalker.com/


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Building tcnative on win32 [alpha]

Christopher Schultz-2
In reply to this post by Christopher Schultz-2
All,

It seems that a bit of cleanup, changing an option for OpenSSL build,
and a *complete* re-download and re-build of everything has resulted in
what appears to be a 1.5MiB tcnative-1.dll file, which without any
testing at all seems like I probably have done things right.

The attached script will build a statically-linked 32-bit tcnative-1.dll
binary and a statically-linked 32-bit openssl.exe binary. I'll work on
getting the other architectures built, soon.

Does anyone have a 32-bit Windows system on which the Tomcat APR (with
SSL) tests could be run? I'd like to send a DLL and have someone test it
more competently than I perhaps could.

There is another dependency I didn't mention before: ActiveState Perl.
You only need this if you are going to use OpenSSL with tcnative but,
honestly, who is going to bother building the whole thing without using
OpenSSL?

Anyhow, here are updated instructions for building OpenSSL, APR, and
tcnative on win32:

*NOTE THAT YOU DO NOT HAVE TO BUY/OWN ANY MICROSOFT SOFTWARE TO BUILD
OpenSSL, APR, and tcnative*

I did all my work on a Windows 8.1 VirtualBox VM that I downloaded from
https://www.modern.ie/en-us/virtualization-tools. So, if you want to
configure a build environment and don't want to jack-up your own
computer to get this all done, you can build OpenSSL, APR, and tcnative
all without buying any software.

When you are done, you can copy the binaries to a real machine and
scuttle the VM. Sounds like a good deal to me.

There are a few prerequisites:

1. If you don't already have Microsoft Visual Studio C++ available,
   download and install Microsoft Visual Studio Express 2013, which
   you can find here:
   http://www.microsoft.com/en-us/download/details.aspx?id=40787

2. Download and install Java JDK. You might be able to get away with
   the JRE but I got the whole JDK because I didn't want to mess-around
   with missing components (like header files).

3. If you are going to use OpenSSL (which you are), you'll need
   ActiveState Perl, which you can find here:
   http://www.activestate.com/activeperl/downloads

3. Download GnuWin32 wget from here:
   http://gnuwin32.sourceforge.net/packages/wget.htm
   I've heard that the above wget.exe might have a problem, so you can
   use http://users.ugent.be/~bpuype/cgi-bin/fetch.pl?dl=wget/wget.exe
   instead. I believe the latter tool is the same but statically-linked.

4. Download 7zip command-line version (7za.exe) from here:
   http://www.7-zip.org/download.html

5. Put wget.exe and 7za.exe into a directory called "tools" in some
   work directory. build-tcnative.bat will use a working directory
   under %TEMP% for downloads and builds. You can also change the
   value of the %TOOLS% environment variable in the script to point
   wherever you want.

6. Launch cmd.exe and run VC++'s vcvars32.bat script or, better yet,
   launch the cmd.exe config that gets installed when you install
   VC++.

7. Create a "work" directory and CD to it. Put build-tcnative.bat
   in this directory and run it.
   When you are done, there will be a "build" directory created
   that contains all the binaries.

A few notes:

1. A few of the APR tests fail. I'm checking with the APR folks to see
if they are spurious or not. One the tests requires memcached to be
available and that one of course fails.

2. Only the 32-bit binary without OCSP is built. More architectures, later.

Hopefully, this will be enough to get some other folks started who are
waiting for an update to tcnative-1.1.30 that includes OpenSSL 1.0.1h.

Enjoy,
-chris

build-tcnative.bat (14K) Download Attachment
signature.asc (943 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Building tcnative on win32 [alpha]

Christopher Schultz-2
In reply to this post by Konstantin Preißer-3
Konstantin,

On 6/19/14, 12:20 PM, Konstantin Preißer wrote:
> One other thing: Do you know which version of VC++ was used to date
> to build Tomcat Native?

I tried removing the Windows SDK and I was unable to link the compiled
objects. I think there are some required libraries in the SDK.

-chris


signature.asc (943 bytes) Download Attachment
12