completely automated (for real) Let's Encrypt on embedded Tomcat

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

completely automated (for real) Let's Encrypt on embedded Tomcat

Garret Wilson
Hi, everyone. I'm back already. (I had intended to leave the list to
focus my efforts elsewhere, but … here I am again.)

I just realized there is a big SSL problem for small applications, and I
want to fix it. First a little review of where we are.

Servlet containers are becoming less important and less desirable in
today's world, because we don't want to deploy and maintain some sort of
high-level container infrastructure (in the Java EE container sense, not
the Docker sense) just to deploy an application in it. Modern
distributed micrososervice applications have a bunch of
service/worker/agent application that are identical and redundant. You
spin up as many as you need; if some go down, you (or an orchestrator)
spins up others.

For this reason libraries like Spring Boot allow you to deploy your Java
application as a standalone JAR with embedded Tomcat. The JAR represents
the completely independent application. You just throw it on a node and
it runs and provides a web server or whatever. So we we should be able
to throw a Spring Boot JAR on something like AWS Elastic Beanstalk and
it just runs. I found out it is far from that simple, and SSL is one of
the major problems.

There seem to be two ways to get SSL support. On something like AWS
Elastic Beanstalk, you deploy a load balancer in front of your EC
instances. Elastic Beanstalk will (using the AWS Route 53 DNS) configure
SSL to the load balancer, spin up EC instances as needed (each running
your standalone JAR), and connect the load balancer to the EC instances,
all in a (sort of) automated fashion. But note that the SSL endpoint is
the load balancer, and the load balancer costs money! Even if you're
just running just a single standalone JAR instance requiring a single EC
instance, that load balancer sits there and drains cash. Significant
cash if you just want to run a little program with SSL support.

What's the other option to deploy a standalone JAR? Configure an AWS EC
instance (or a VM with another provider), configure certbot, configure
Tomcat, save some files locally on the machine, etc. All this manual
work. I just want to run the standalone JAR! In short, if I have a
standalone program I want to run, I either have to configure and
maintain a VM like I did in the year 2000, or get into the nightmare of
Kubernetes-like orchestration with the endless configurations and/or the
high costs.

I propose to create a module that integrates with embedded Tomcat that:

 1. You indicate what domain you're hosting for (as part of the
    application configuration or as an environment variable when
    deployed, for example).
 2. When your application starts running, it automatically connects to
    Let's Encrypt using RFC 8555 (or whatever is needed) and requests a
    certificate, based upon the IP address it's running on.
 3. The module exposes the correct HTTP paths and/or connects to a
    configured DNS as needed for validation.
 4. The module receives the certificates and caches them in memory or in
    a temporary file as needed and provides them to Tomcat; Tomcat now
    is serving using SSL/TLS.
 5. If the application dies, who cares? You start up another one. It
    automatically does the same thing (on another machine or wherever it
    is running) and the application is running SSL/TLS. It's that
    simple. You don't need to run certbot. You don't need to manually
    copy files on the system. You don't even need to know where the
    application is going to run. You just need an executable JAR with
    this new module, and you run it. Done.
 6. (Many variations exists where multiple JARs are running but one is
    the "leader" for Let's Encrypt, and they communicate and share the
    cashed certificate until the node dies. Or there are variations
    using Docker. The first step is the radical one, and then all sorts
    of possibilities open up.)

 From glancing over the Let's Encrypt docs and having had hands-on
experience embedding Tomcat, that seems completely doable to me. And I'm
ready to start.

But first, what work has been done in this area already? I'm aware of
Chris' slides from 2018, but those techniques require some combination
of certbot, keytool, non-embedded Tomcat, symlinks,OS scripts, manually
file system manipulation, etc. I think at ApacheCon 2019 Chris mentioned
some more work has been done on this, but I don't recall where it was.

Please point me to the latest work and ideas for Tomcat+Let's Encrypt so
that I don't spend two months doing something that is already been done,
or before I find out it is impossible.

As it stands I want fully automated SSL/TLS configuration just by
running a standalone JAR, and I don't see that existing anywhere. I'm
not prepared to pay AWS for a load balancer just to run a little app,
and I got tired of manual Linux setup and scripts and general sysadmin
work around 20 years ago. It's the cloud. It should act like the cloud.

Garret

Reply | Threaded
Open this post in threaded view
|

Re: completely automated (for real) Let's Encrypt on embedded Tomcat

Martynas Jusevičius
https://github.com/AtomGraph/letsencrypt-tomcat

On Sun, Oct 4, 2020 at 8:04 PM Garret Wilson <[hidden email]> wrote:

>
> Hi, everyone. I'm back already. (I had intended to leave the list to
> focus my efforts elsewhere, but … here I am again.)
>
> I just realized there is a big SSL problem for small applications, and I
> want to fix it. First a little review of where we are.
>
> Servlet containers are becoming less important and less desirable in
> today's world, because we don't want to deploy and maintain some sort of
> high-level container infrastructure (in the Java EE container sense, not
> the Docker sense) just to deploy an application in it. Modern
> distributed micrososervice applications have a bunch of
> service/worker/agent application that are identical and redundant. You
> spin up as many as you need; if some go down, you (or an orchestrator)
> spins up others.
>
> For this reason libraries like Spring Boot allow you to deploy your Java
> application as a standalone JAR with embedded Tomcat. The JAR represents
> the completely independent application. You just throw it on a node and
> it runs and provides a web server or whatever. So we we should be able
> to throw a Spring Boot JAR on something like AWS Elastic Beanstalk and
> it just runs. I found out it is far from that simple, and SSL is one of
> the major problems.
>
> There seem to be two ways to get SSL support. On something like AWS
> Elastic Beanstalk, you deploy a load balancer in front of your EC
> instances. Elastic Beanstalk will (using the AWS Route 53 DNS) configure
> SSL to the load balancer, spin up EC instances as needed (each running
> your standalone JAR), and connect the load balancer to the EC instances,
> all in a (sort of) automated fashion. But note that the SSL endpoint is
> the load balancer, and the load balancer costs money! Even if you're
> just running just a single standalone JAR instance requiring a single EC
> instance, that load balancer sits there and drains cash. Significant
> cash if you just want to run a little program with SSL support.
>
> What's the other option to deploy a standalone JAR? Configure an AWS EC
> instance (or a VM with another provider), configure certbot, configure
> Tomcat, save some files locally on the machine, etc. All this manual
> work. I just want to run the standalone JAR! In short, if I have a
> standalone program I want to run, I either have to configure and
> maintain a VM like I did in the year 2000, or get into the nightmare of
> Kubernetes-like orchestration with the endless configurations and/or the
> high costs.
>
> I propose to create a module that integrates with embedded Tomcat that:
>
>  1. You indicate what domain you're hosting for (as part of the
>     application configuration or as an environment variable when
>     deployed, for example).
>  2. When your application starts running, it automatically connects to
>     Let's Encrypt using RFC 8555 (or whatever is needed) and requests a
>     certificate, based upon the IP address it's running on.
>  3. The module exposes the correct HTTP paths and/or connects to a
>     configured DNS as needed for validation.
>  4. The module receives the certificates and caches them in memory or in
>     a temporary file as needed and provides them to Tomcat; Tomcat now
>     is serving using SSL/TLS.
>  5. If the application dies, who cares? You start up another one. It
>     automatically does the same thing (on another machine or wherever it
>     is running) and the application is running SSL/TLS. It's that
>     simple. You don't need to run certbot. You don't need to manually
>     copy files on the system. You don't even need to know where the
>     application is going to run. You just need an executable JAR with
>     this new module, and you run it. Done.
>  6. (Many variations exists where multiple JARs are running but one is
>     the "leader" for Let's Encrypt, and they communicate and share the
>     cashed certificate until the node dies. Or there are variations
>     using Docker. The first step is the radical one, and then all sorts
>     of possibilities open up.)
>
>  From glancing over the Let's Encrypt docs and having had hands-on
> experience embedding Tomcat, that seems completely doable to me. And I'm
> ready to start.
>
> But first, what work has been done in this area already? I'm aware of
> Chris' slides from 2018, but those techniques require some combination
> of certbot, keytool, non-embedded Tomcat, symlinks,OS scripts, manually
> file system manipulation, etc. I think at ApacheCon 2019 Chris mentioned
> some more work has been done on this, but I don't recall where it was.
>
> Please point me to the latest work and ideas for Tomcat+Let's Encrypt so
> that I don't spend two months doing something that is already been done,
> or before I find out it is impossible.
>
> As it stands I want fully automated SSL/TLS configuration just by
> running a standalone JAR, and I don't see that existing anywhere. I'm
> not prepared to pay AWS for a load balancer just to run a little app,
> and I got tired of manual Linux setup and scripts and general sysadmin
> work around 20 years ago. It's the cloud. It should act like the cloud.
>
> Garret
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: completely automated (for real) Let's Encrypt on embedded Tomcat

Christopher Schultz-2
In reply to this post by Garret Wilson
Garret,

On 10/4/20 14:04, Garret Wilson wrote:

> Hi, everyone. I'm back already. (I had intended to leave the list to
> focus my efforts elsewhere, but … here I am again.)
>
> I just realized there is a big SSL problem for small applications, and I
> want to fix it. First a little review of where we are.
>
> Servlet containers are becoming less important and less desirable in
> today's world, because we don't want to deploy and maintain some sort of
> high-level container infrastructure (in the Java EE container sense, not
> the Docker sense) just to deploy an application in it. Modern
> distributed micrososervice applications have a bunch of
> service/worker/agent application that are identical and redundant. You
> spin up as many as you need; if some go down, you (or an orchestrator)
> spins up others.

Microservices won't work the way you want with Let's Encrypt. You have
two options:

1. Hit Let's Encrypt every time you launch a new instance of the
microservice to deploy a new certificate

2. Handle the certificate provisioning elsewhere (e.g. ELB)

#1 just won't work. LE won't re-issue a certificate more frequently than
every 6 weeks or something like that. So that really leaves you with #2.

What you really want is for the orchestrator to provide the certificate
and key to the nodes as they come on-line. This represents a bit of a
security issue, but not any moreso than any node trying to connect to
the orchestrator in the first place IMHO.

So instead of trying to get LE to work with Tomcat (which does work, but
requires some care and feeding), maybe we should try to get Tomcat to
load its crypto material from other places.

One way to do that would be with e.g. Amazon's key storage service. I'm
not familiar with that. I know it can store various types of keys; not
sure about certificates and if we can pull a private key out of it.

Honestly, your best bet would probably be to use ELB and just pay for
it. You only pay for data-transfer, so a dormant ELB costs you virtually
nothing.

Instead of spinning-up an EC2 instance for your service, maybe you
should be looking at Lambda instead. You can probably get your costs
down more that way than trying to eliminate the ELB.

BTW, if your instance isn't running at all (e.g. the orchestrator has
decided that zero nodes are required), what do clients contact to make
that initial connection for the orchestrator to spin-up that first instance?

> For this reason libraries like Spring Boot allow you to deploy your Java
> application as a standalone JAR with embedded Tomcat. The JAR represents
> the completely independent application. You just throw it on a node and
> it runs and provides a web server or whatever. So we we should be able
> to throw a Spring Boot JAR on something like AWS Elastic Beanstalk and
> it just runs. I found out it is far from that simple, and SSL is one of
> the major problems.
>
> There seem to be two ways to get SSL support. On something like AWS
> Elastic Beanstalk, you deploy a load balancer in front of your EC
> instances. Elastic Beanstalk will (using the AWS Route 53 DNS) configure
> SSL to the load balancer, spin up EC instances as needed (each running
> your standalone JAR), and connect the load balancer to the EC instances,
> all in a (sort of) automated fashion. But note that the SSL endpoint is
> the load balancer, and the load balancer costs money! Even if you're
> just running just a single standalone JAR instance requiring a single EC
> instance, that load balancer sits there and drains cash. Significant
> cash if you just want to run a little program with SSL support.
>
> What's the other option to deploy a standalone JAR? Configure an AWS EC
> instance (or a VM with another provider), configure certbot, configure
> Tomcat, save some files locally on the machine, etc. All this manual
> work. I just want to run the standalone JAR! In short, if I have a
> standalone program I want to run, I either have to configure and
> maintain a VM like I did in the year 2000, or get into the nightmare of
> Kubernetes-like orchestration with the endless configurations and/or the
> high costs.
>
> I propose to create a module that integrates with embedded Tomcat that:
>
> 1. You indicate what domain you're hosting for (as part of the
>    application configuration or as an environment variable when
>    deployed, for example).
> 2. When your application starts running, it automatically connects to
>    Let's Encrypt using RFC 8555 (or whatever is needed) and requests a
>    certificate, based upon the IP address it's running on.
> 3. The module exposes the correct HTTP paths and/or connects to a
>    configured DNS as needed for validation.
> 4. The module receives the certificates and caches them in memory or in
>    a temporary file as needed and provides them to Tomcat; Tomcat now
>    is serving using SSL/TLS.
> 5. If the application dies, who cares? You start up another one. It
>    automatically does the same thing (on another machine or wherever it
>    is running) and the application is running SSL/TLS. It's that
>    simple. You don't need to run certbot. You don't need to manually
>    copy files on the system. You don't even need to know where the
>    application is going to run. You just need an executable JAR with
>    this new module, and you run it. Done.
> 6. (Many variations exists where multiple JARs are running but one is
>    the "leader" for Let's Encrypt, and they communicate and share the
>    cashed certificate until the node dies. Or there are variations
>    using Docker. The first step is the radical one, and then all sorts
>    of possibilities open up.)
> > From glancing over the Let's Encrypt docs and having had hands-on
> experience embedding Tomcat, that seems completely doable to me. And I'm
> ready to start.
>
> But first, what work has been done in this area already? I'm aware of
> Chris' slides from 2018, but those techniques require some combination
> of certbot, keytool, non-embedded Tomcat, symlinks,OS scripts, manually
> file system manipulation, etc. I think at ApacheCon 2019 Chris mentioned
> some more work has been done on this, but I don't recall where it was.
>
> Please point me to the latest work and ideas for Tomcat+Let's Encrypt so
> that I don't spend two months doing something that is already been done,
> or before I find out it is impossible.

This exists. Romain Manni-Bucau from the TomEE project has one, and
there are probably others.

https://github.com/rmannibucau/letsencrypt-manager

YMMV if you want to re-launch a Tomcat docker image every few hours.

> As it stands I want fully automated SSL/TLS configuration just by
> running a standalone JAR, and I don't see that existing anywhere. I'm
> not prepared to pay AWS for a load balancer just to run a little app,
> and I got tired of manual Linux setup and scripts and general sysadmin
> work around 20 years ago. It's the cloud. It should act like the cloud.

-chris


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: completely automated (for real) Let's Encrypt on embedded Tomcat

Garret Wilson
Thank you so much for replying, Chris. Responses below.

On 10/5/2020 8:53 AM, Christopher Schultz wrote:

> Microservices won't work the way you want with Let's Encrypt. You have
> two options:
>
> 1. Hit Let's Encrypt every time you launch a new instance of the
> microservice to deploy a new certificate
>
> 2. Handle the certificate provisioning elsewhere (e.g. ELB)
>
> #1 just won't work. LE won't re-issue a certificate more frequently than
> every 6 weeks or something like that. So that really leaves you with #2.

It's good to know about the six-week limit, but you discarded #1 too
quickly. Can't the microservice simply store the credentials in S3 or
one of a hundred other data stores? (Note that I care less about
"microservices" as such at the moment. I just want a turnkey deployment
of a single application for now. But the idea is the same.)

> What you really want is for the orchestrator to provide the certificate
> and key to the nodes as they come on-line.

Look these "orchestrators" are a configuration nightmare at the moment.
They end up being worse than my just configuring a CentOS machine from
scratch. Plus I have to pay for all those extra services. Read
https://leebriggs.co.uk/blog/2019/04/13/the-fargate-illusion.html and
try not to shudder.

> So instead of trying to get LE to work with Tomcat (which does work, but
> requires some care and feeding), maybe we should try to get Tomcat to
> load its crypto material from other places.

There is already a Java library (https://github.com/shred/acme4j) for
talking to Let's Encrypt. It sounds like it does everything I need. I'll
need to investigate more, but here are my initial doubts:

 1. Does acme4j allow me to verify my certificate behind another port?
    (e.g. ElasticBeanstalk deploys a JAR behind NGINX port 5000 by
    default. I'm still reading RFC 8555 to find out if the ACME server
    has to connect back on a certain port for verification.)
 2. Once I have the Let's Encrypt certificate, can I convert to PKCS12
    for Tomcat completely in the application without shelling out to
    openssl or keytool? I'm hoping Bouncy Castle and/or acme4j-util will
    allow me to do that.
 3. Once I have the PKCS12, how do I feed it to the embedded Tomcat?

Chris, where can I get more information on the latter questions about
getting this certificate to Tomcat once I have it?

> One way to do that would be with e.g. Amazon's key storage service. I'm
> not familiar with that. I know it can store various types of keys; not
> sure about certificates and if we can pull a private key out of it.

I think your idea of storing this stuff elsewhere is a good first
stepping stone to get to where I want to go.

Just to get the ball rolling, I could manually run some Let's Encrypt
client, and then store the certificate in S3. Then the first component I
would write would be steps #2 and #3 above. I am already familiar with
the AWS Java library, so I could quickly figure out how to pull the
certificates off S3. But I need your help in finding out how to convert
them to PKCS12 (or whatever; this is new territory for me) on the fly
and feed them to the embedded Tomcat.

> Honestly, your best bet would probably be to use ELB and just pay for
> it. You only pay for data-transfer, so a dormant ELB costs you virtually
> nothing.

Last month I deployed a test application on Elastic Beanstalk on a
domain nobody knows about just to see how it worked. The ELB cost me $16
in a month with basically nobody using it! That adds up quickly. I have
several little apps I want to toss up. See also my question
https://serverfault.com/q/1036276 .


> Instead of spinning-up an EC2 instance for your service, maybe you
> should be looking at Lambda instead. You can probably get your costs
> down more that way than trying to eliminate the ELB.

I want to drop either a self-contained JAR file or a Docker image
somewhere, and have it immediately start running with SSL support,
without my configuring a VM or running scripts. When I have a new
version, I want to drop a new JAR or Docker image and have it
automatically replace the other one. I don't want to maintain a VM. I
have a target price of let's say $5/month for everything (although $3
would be better). Does AWS Lambda give me that? If so, please point me
to the guides.

Garret
Reply | Threaded
Open this post in threaded view
|

Re: completely automated (for real) Let's Encrypt on embedded Tomcat

Christopher Schultz-2
Garret,

On 10/5/20 12:21, Garret Wilson wrote:

> Thank you so much for replying, Chris. Responses below.
>
> On 10/5/2020 8:53 AM, Christopher Schultz wrote:
>> Microservices won't work the way you want with Let's Encrypt. You have
>> two options:
>>
>> 1. Hit Let's Encrypt every time you launch a new instance of the
>> microservice to deploy a new certificate
>>
>> 2. Handle the certificate provisioning elsewhere (e.g. ELB)
>>
>> #1 just won't work. LE won't re-issue a certificate more frequently than
>> every 6 weeks or something like that. So that really leaves you with #2.
>
> It's good to know about the six-week limit, but you discarded #1 too
> quickly. Can't the microservice simply store the credentials in S3 or
> one of a hundred other data stores? (Note that I care less about
> "microservices" as such at the moment. I just want a turnkey deployment
> of a single application for now. But the idea is the same.)

Sure, it can contain S3 credentials and you can pick-up your key and
certificate (or, better yet, the whole keystore) there, but at that
point you have "moved" the problem outside of Tomcat, right?

You can have a "certificate renewal service" that writes to S3.

I suppose you could put that directly into Tomcat, but Tomcat is not
likely to ship with an Amazon-specific feature built-into it.

I could imagine a LifecycleListener which has been written with this
kind of thing in mind. (I'm not sure that a LifecycleListener would be
able to intervene early enough in the process of a non-embedded Tomcat
startup to do this, btw; just an idea.)

Another idea would be to use embedded Tomcat (or, at your suggestion,
Spring Boot) and fetch the keystore from some "standard" location of
your choosing. Again, that would be (appropriately, IMO) outside Tomcat
code.

>> What you really want is for the orchestrator to provide the certificate
>> and key to the nodes as they come on-line.
>
> Look these "orchestrators" are a configuration nightmare at the moment.
> They end up being worse than my just configuring a CentOS machine from
> scratch. Plus I have to pay for all those extra services. Read
> https://leebriggs.co.uk/blog/2019/04/13/the-fargate-illusion.html and
> try not to shudder.
>
>> So instead of trying to get LE to work with Tomcat (which does work, but
>> requires some care and feeding), maybe we should try to get Tomcat to
>> load its crypto material from other places.
>
> There is already a Java library (https://github.com/shred/acme4j) for
> talking to Let's Encrypt. It sounds like it does everything I need. I'll
> need to investigate more, but here are my initial doubts:
>
> 1. Does acme4j allow me to verify my certificate behind another port?
>    (e.g. ElasticBeanstalk deploys a JAR behind NGINX port 5000 by
>    default. I'm still reading RFC 8555 to find out if the ACME server
>    has to connect back on a certain port for verification.)
> 2. Once I have the Let's Encrypt certificate, can I convert to PKCS12
>    for Tomcat completely in the application without shelling out to
>    openssl or keytool? I'm hoping Bouncy Castle and/or acme4j-util will
>    allow me to do that.

This can be done, but it's non-trivial. For example, Tomcat contains
code to package PEM-encoded DER files (good old OpenSSL-style =====BEGIN
CERTIFICATE===== things) into an in-memory keystore to configure JSSE.
It seems like it would be straightforward, but it turns out not to be in
all cases. YMMV.

> 3. Once I have the PKCS12, how do I feed it to the embedded Tomcat?

If it's a file on the disk, it's easy: just use the path.

> Chris, where can I get more information on the latter questions about
> getting this certificate to Tomcat once I have it?

This mailing list is a good place to start (and likely finish).

>> One way to do that would be with e.g. Amazon's key storage service. I'm
>> not familiar with that. I know it can store various types of keys; not
>> sure about certificates and if we can pull a private key out of it.
>
> I think your idea of storing this stuff elsewhere is a good first
> stepping stone to get to where I want to go.
>
> Just to get the ball rolling, I could manually run some Let's Encrypt
> client, and then store the certificate in S3. Then the first component I
> would write would be steps #2 and #3 above. I am already familiar with
> the AWS Java library, so I could quickly figure out how to pull the
> certificates off S3. But I need your help in finding out how to convert
> them to PKCS12 (or whatever; this is new territory for me) on the fly
> and feed them to the embedded Tomcat.

Go back to my presentation on Let's Encrypt and you'll see how to use
openssl to convert to a keystore if that's what you want. Or, better
yet, skip that step entirely and use the PEM-encoded DER files that
Let's Encrypt already provides to you. You just have to work-out
file-permissions issues.

>> Honestly, your best bet would probably be to use ELB and just pay for
>> it. You only pay for data-transfer, so a dormant ELB costs you virtually
>> nothing.
>
> Last month I deployed a test application on Elastic Beanstalk on a
> domain nobody knows about just to see how it worked. The ELB cost me $16
> in a month with basically nobody using it! That adds up quickly. I have
> several little apps I want to toss up. See also my question
> https://serverfault.com/q/1036276 .

Oh, I forgot that you have to pay for lb-hours as well as transfer.
Sorry about that. Yeah, they don't keep services up and running for free. :)

What you have to remember is that AWS is often MORE EXPENSIVE than
running your own stuff. It's just WAY more flexible and "responsive". If
I want to get a new server installed by my infra provider into a DC, ?I
usually have to call them, tell them what I want, get some quotes for
specific hardware, ask them to order it, install it, but an OS on it,
etc. and maybe that takes 3 total weeks. With AWS, I can do it in 5 minutes.

Auto-scale with a traditional co-located deployment strategy? Not possible.

>> Instead of spinning-up an EC2 instance for your service, maybe you
>> should be looking at Lambda instead. You can probably get your costs
>> down more that way than trying to eliminate the ELB.
>
> I want to drop either a self-contained JAR file or a Docker image
> somewhere, and have it immediately start running with SSL support,
> without my configuring a VM or running scripts. When I have a new
> version, I want to drop a new JAR or Docker image and have it
> automatically replace the other one. I don't want to maintain a VM.

I hear that. I still don't understand how you are supposed to get OS
updates working for an application deployed into Elastic Beanstalk. You
immediately get a bunch of EC2 instances. Do I have to watch them for OS
updates? Do I have to apply them? OR does AWS just wave their hands and
say "they'll be fine" and auto-update and auto-re-build them whenever
they want. I have regulatory requirements that say that I need to
document my patch cycles. What do I say for this stuff? "Amazon says I
don't need to worry my pretty little head?" Really, I'm speaking from a
position of profound ignorance on this point. I really have no idea how
this stuff is supposed to work and I haven't put any time into
discovering the real answers.

For my AWS deployments, I treat them all just like I do with regular DC
deployments where I instrument and maintain each and ever service. I'd
love to be free of that burden.

> I have a target price of let's say $5/month for everything (although
> $3 would be better). Does AWS Lambda give me that? If so, please
> point me to the guides.

I think you always need the lb, and if the lb prices are too high for
you, then you need to look for other solutions.

-chris

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: completely automated (for real) Let's Encrypt on embedded Tomcat

Garret Wilson
On 10/5/2020 2:42 PM, Christopher Schultz wrote:
> …
> Sure, it can contain S3 credentials and you can pick-up your key and
> certificate (or, better yet, the whole keystore) there, but at that
> point you have "moved" the problem outside of Tomcat, right?

No, not at all. The major problems are:

 1. Generating the certificate automatically.
 2. Feeding the certificate to Tomcat automatically.

If the "extra" thing I have to do is specify an environment variable
with the name of the S3 bucket to use as a certificate state, then that
is a teeny, tiny problem. That is not really even a problem.

Can you imagine if in my spring boot application I could run it using
"java -jar my-app.jar --cert-work-bucket my-bucket" and it would just go
get a certificate automatically?

> You can have a "certificate renewal service" that writes to S3.

I want it built into my application as a module. You just include the
module, specify the domain name (and maybe a couple of other details
needed by RFC 8555, such as a contact email), and boom, it all happens
automatically.

Why does it have to be more difficult than that? It shouldn't be.

> I suppose you could put that directly into Tomcat, but Tomcat is not
> likely to ship with an Amazon-specific feature built-into it.


Read my original email. I never intended to put this directly into
standalone Tomcat (although if you want to put it into Tomcat you're
welcome to). I want to use this with an application running on embedded
Tomcat. Spring Boot is a prime example. I'll just extend the Spring Boot
embedded Tomcat module and extend/modify that as needed. If that
requires modifying Tomcat code, fine.

> Another idea would be to use embedded Tomcat (or, at your suggestion,
> Spring Boot) and fetch the keystore from some "standard" location of
> your choosing. Again, that would be (appropriately, IMO) outside Tomcat
> code.

That was always the intention. In my introduction to my original email I
explained that the modern approach is moving away from something like
standalone Tomcat to self-contained executables that run their own
servers, whether embedded Tomcat or whatever. (I was late to the party,
and even two years ago I wasn't getting it.)

> 1. Does acme4j allow me to verify my certificate behind another port?
>     (e.g. ElasticBeanstalk deploys a JAR behind NGINX port 5000 by
>     default. I'm still reading RFC 8555 to find out if the ACME server
>     has to connect back on a certain port for verification.)
> 2. Once I have the Let's Encrypt certificate, can I convert to PKCS12
>     for Tomcat completely in the application without shelling out to
>     openssl or keytool? I'm hoping Bouncy Castle and/or acme4j-util will
>     allow me to do that.
> This can be done, but it's non-trivial. For example, Tomcat contains
> code to package PEM-encoded DER files (good old OpenSSL-style =====BEGIN
> CERTIFICATE===== things) into an in-memory keystore to configure JSSE.
> It seems like it would be straightforward, but it turns out not to be in
> all cases. YMMV.

Non-trivial as it may be, it /only needs to be done once/. If I have a
converter, then I can use it a thousand times. A million times. And
suddenly the deployment becomes a piece of cake.

In reality, today's style of handling SSL is what matches your
description of "It seems like it would be straightforward, but it turns
out not to be …". So why do we keep doing all this difficult, manual,
tedious, not-trivial stuff to deploy certificates the hard way, when we
could put our efforts into a single no-trivial task of making a
converter so that Tomcat can use the Let's Encrypt certificates
directly? We do it once. It's hard, but then everything else is easy. I
don't get why we want to spend another decade doing it the hard way when
we can spend one year on a different hard task and then do SSL the easy
way for the other nine.

(The frustration isn't directed at you. It's just in general in software
development I see the industry—why does the most basic of things have to
be so difficult?)

> 3. Once I have the PKCS12, how do I feed it to the embedded Tomcat?
> If it's a file on the disk, it's easy: just use the path.

Can I pass it in memory? If not, why not? How is memory less accessible
than a file?

>> Chris, where can I get more information on the latter questions about
>> getting this certificate to Tomcat once I have it?
> This mailing list is a good place to start (and likely finish).

Of course I really super-appreciate the help on this mailing list, and
I'll be asking lots more questions. But I also don't want to ask things
you've already answered elsewhere. I thought sure in one of your
ApacheCon 2019 presentations you mentioned you had made more progress
than the ApacheCon 2018 slides, such as auto-reloading or something. If
there is further documentation let me know.

> …
> Go back to my presentation on Let's Encrypt and you'll see how to use
> openssl to convert to a keystore if that's what you want.

I want to make any conversions completely in the application without
depending on some utility installed in the OS. So no making a shell call
to openssl. If OpenSSL can do something, why can't Java?

> Or, better
> yet, skip that step entirely and use the PEM-encoded DER files that
> Let's Encrypt already provides to you.

I have no idea what a PEM-encoded DER file is, but I'll certainly learn.
And if it means that Tomcat can directly use something provided by Let's
Encrypt with no conversion required, than that's awesome!

So what's the problem? One piece of code talks to Let's Encrypt and
stores this DER file thing on S3. Another piece of code pulls the DER
file from S3 and starts up an embedded Tomcat with it. (All in same app,
as needed.) Seems straightforward to me.

> You just have to work-out
> file-permissions issues.

I still don't get why files have to be involved. I pulled a DER file
from S3. I have it in memory. Embedded Tomcat is running in memory. They
are in the same JVM. Why do I have to put something in a file?

> …
>
> What you have to remember is that AWS is often MORE EXPENSIVE than
> running your own stuff.

It shouldn't be, in general. Part of the whole cloud value proposition
is that you only pay for what you use.

The problem here is that this big SSL antipattern hurdle is making me
need to use things I don't need for a tiny app. I'm wanting to remove
that hurdle.

> …
> I hear that. I still don't understand how you are supposed to get OS
> updates working for an application deployed into Elastic Beanstalk.

See
https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/environment-platform-update-managed.html 
.

But I don't want to over-focus on Elastic Beanstalk.

The issue here is that I want a standalone app with embedded Tomcat that
automatically 100% configures SSL. For now Elastic Beanstalk is just one
vehicle to run that app. In the future we may have platforms (if we
don't already) that completely hide the OS, and the JAR is just running
as a service.

We have the same sort of thing with static sites on AWS. Remember that
static site generator, Guise Mummy, I showed off at ApacheCon 2019? I
finished it, and all my sites are now using it. Guise Mummy generates
and deploys a static site to S3, but then configures CloudFront to cache
and serve it with edge access around the world, with HTTPS. (See
https://www.globalmentor.com/ as an example.) What web server is serving
those files? What OS is hosting the web server? I have no idea. It's
irrelevant. How do I update the OS? I don't need to. For all practical
purposes there is no OS. My site consists of static files, and AWS is
hosting them.

Eventually the same should go for my JAR file. I give AWS my JAR file.
AWS starts up my JAR file. It has HTTPS automatically. I shouldn't have
to do a single thing.

We are almost there, except for this huge SSL problem where I have to do
basically what I did in 2000 to get SSL turned on. (Maybe it's a little
easier. Not much.)

> Really, I'm speaking from a
> position of profound ignorance on this point. I really have no idea how
> this stuff is supposed to work and I haven't put any time into
> discovering the real answers.


Chris, I'm learning all this stuff too. I had to get over a bunch of
gotchas to get this S3->CloudFront with automatic SSL stuff working for
a static site. And you can see from my Stack Overflow question
https://stackoverflow.com/q/63650514 that I naively approached Elastic
Beanstalk from the same view, thinking it would transparently handle
everything for me. It doesn't. But that's where we want to go eventually.

So I have a lot to learn. One of the first things I'm learning is that
SSL is a problem Everywhere. All the gazillions of cute demos don't
include SSL. If you do set up SSL, you have to jump through all these
hoops, or pay lots extra to have a load balancer. Really, there isn't an
easy SSL solution for anything in the cloud that I can find that isn't a
static site. If you know of something, then please go answer my Server
Fault question: https://serverfault.com/q/1036276 .

If nobody else will do this, I will do this. It needs to be done,
because I don't enjoy playing sysadmin just to set up SSL when I'd
rather be writing applications and deploying them.

Garret

Reply | Threaded
Open this post in threaded view
|

Re: completely automated (for real) Let's Encrypt on embedded Tomcat

James H. H. Lampert
I'm coming into this conversation late, so what I say could be
completely irrelevant, but when I recently set up an independent (i.e.,
not behind httpd) Tomcat server on one of our AWS EC2 instances, and
could not get certbot to function at all, to save my life, I ended up
using something called "LEGO." It *does* require one to shut the Tomcat
server down during the renewal process (because it has to take over the
port briefly), but it also *does* play nicely with a Tomcat server
that's doing its own SSL.

--
James H. H. Lampert

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: completely automated (for real) Let's Encrypt on embedded Tomcat

Christopher Schultz-2
In reply to this post by Garret Wilson
Garret,

On 10/5/20 19:45, Garret Wilson wrote:

> On 10/5/2020 2:42 PM, Christopher Schultz wrote:
>> …
>> Sure, it can contain S3 credentials and you can pick-up your key and
>> certificate (or, better yet, the whole keystore) there, but at that
>> point you have "moved" the problem outside of Tomcat, right?
>
> No, not at all. The major problems are:
>
> 1. Generating the certificate automatically.
> 2. Feeding the certificate to Tomcat automatically.
>
> If the "extra" thing I have to do is specify an environment variable
> with the name of the S3 bucket to use as a certificate state, then that
> is a teeny, tiny problem. That is not really even a problem.
>
> Can you imagine if in my spring boot application I could run it using
> "java -jar my-app.jar --cert-work-bucket my-bucket" and it would just go
> get a certificate automatically?

What if your Docker container would just run certbot on launch?

>> You can have a "certificate renewal service" that writes to S3.
>
> I want it built into my application as a module. You just include the
> module, specify the domain name (and maybe a couple of other details
> needed by RFC 8555, such as a contact email), and boom, it all happens
> automatically.
>
> Why does it have to be more difficult than that? It shouldn't be.

Fair enough.

>> I suppose you could put that directly into Tomcat, but Tomcat is not
>> likely to ship with an Amazon-specific feature built-into it.
>
>
> Read my original email. I never intended to put this directly into
> standalone Tomcat (although if you want to put it into Tomcat you're
> welcome to). I want to use this with an application running on embedded
> Tomcat. Spring Boot is a prime example. I'll just extend the Spring Boot
> embedded Tomcat module and extend/modify that as needed. If that
> requires modifying Tomcat code, fine.
>
>> Another idea would be to use embedded Tomcat (or, at your suggestion,
>> Spring Boot) and fetch the keystore from some "standard" location of
>> your choosing. Again, that would be (appropriately, IMO) outside Tomcat
>> code.
>
> That was always the intention. In my introduction to my original email I
> explained that the modern approach is moving away from something like
> standalone Tomcat to self-contained executables that run their own
> servers, whether embedded Tomcat or whatever. (I was late to the party,
> and even two years ago I wasn't getting it.)

I'm not sure I'd say "modern approach". It's certainly popular these
days, for sure.

>> 1. Does acme4j allow me to verify my certificate behind another port?
>>     (e.g. ElasticBeanstalk deploys a JAR behind NGINX port 5000 by
>>     default. I'm still reading RFC 8555 to find out if the ACME server
>>     has to connect back on a certain port for verification.)
>> 2. Once I have the Let's Encrypt certificate, can I convert to PKCS12
>>     for Tomcat completely in the application without shelling out to
>>     openssl or keytool? I'm hoping Bouncy Castle and/or acme4j-util will
>>     allow me to do that.
>> This can be done, but it's non-trivial. For example, Tomcat contains
>> code to package PEM-encoded DER files (good old OpenSSL-style =====BEGIN
>> CERTIFICATE===== things) into an in-memory keystore to configure JSSE.
>> It seems like it would be straightforward, but it turns out not to be in
>> all cases. YMMV.
>
> Non-trivial as it may be, it /only needs to be done once/. If I have a
> converter, then I can use it a thousand times. A million times. And
> suddenly the deployment becomes a piece of cake.
>
> In reality, today's style of handling SSL is what matches your
> description of "It seems like it would be straightforward, but it turns
> out not to be …". So why do we keep doing all this difficult, manual,
> tedious, not-trivial stuff to deploy certificates the hard way, when we
> could put our efforts into a single no-trivial task of making a
> converter so that Tomcat can use the Let's Encrypt certificates
> directly? We do it once. It's hard, but then everything else is easy. I
> don't get why we want to spend another decade doing it the hard way when
> we can spend one year on a different hard task and then do SSL the easy
> way for the other nine.
>
> (The frustration isn't directed at you. It's just in general in software
> development I see the industry—why does the most basic of things have to
> be so difficult?)
>
>> 3. Once I have the PKCS12, how do I feed it to the embedded Tomcat?
>> If it's a file on the disk, it's easy: just use the path.
>
> Can I pass it in memory? If not, why not? How is memory less accessible
> than a file?

Yes, you can pass it in-memory, but you have to arrange to load the key
material from somewhere into memory. And sometimes the key material can
be in a surprising series of formats.

>>> Chris, where can I get more information on the latter questions about
>>> getting this certificate to Tomcat once I have it?
>> This mailing list is a good place to start (and likely finish).
>
> Of course I really super-appreciate the help on this mailing list, and
> I'll be asking lots more questions. But I also don't want to ask things
> you've already answered elsewhere. I thought sure in one of your
> ApacheCon 2019 presentations you mentioned you had made more progress
> than the ApacheCon 2018 slides, such as auto-reloading or something. If
> there is further documentation let me know.

Not auto-reloading, but reloading the SSLHostConfig does indeed work,
now. If you had an on-disk keystore, the configuration doesn't change
but it needs to be reloaded. You can do that via JMX by calling the
reload() (or similar?) method on that MX bean.

If you build the keystore in-memory, I'm not exactly sure what you'd
need to do in order to get Tomcat to bounce the SSLSocketFactory in that
way.

>> Go back to my presentation on Let's Encrypt and you'll see how to use
>> openssl to convert to a keystore if that's what you want.
>
> I want to make any conversions completely in the application without
> depending on some utility installed in the OS. So no making a shell call
> to openssl. If OpenSSL can do something, why can't Java?

It can.

>> Or, better
>> yet, skip that step entirely and use the PEM-encoded DER files that
>> Let's Encrypt already provides to you.
>
> I have no idea what a PEM-encoded DER file is, but I'll certainly learn.

This:

-----BEGIN CERTIFICATE-----
stuff
-----END CERTIFICATE-----

It's an easier format IMHO to deal with than, say, a PKCS12 keystore.

Wonderfully, and somewhat surprisingly (to me, who finds Java's crypto
APIs needlessly complicated), Java can load these trivially:

java.security.cert.CertificateFactory.getInstance("X.509").generateCertificates([inputstream]);

This returns a Collection<Certificate> which you can simply iterate
through and do whatever.

For loading keys, it's not as straightforward. :(

> And if it means that Tomcat can directly use something provided by Let's
> Encrypt with no conversion required, than that's awesome!
>
> So what's the problem? One piece of code talks to Let's Encrypt and
> stores this DER file thing on S3. Another piece of code pulls the DER
> file from S3 and starts up an embedded Tomcat with it. (All in same app,
> as needed.) Seems straightforward to me.

You also need to periodically re-execute the ACME request in case the
application has been running long enough for the certificate to "age".
So you'd better do it on startup and also every X days or whatever.

>> You just have to work-out
>> file-permissions issues.
>
> I still don't get why files have to be involved. I pulled a DER file
> from S3.

Sorry. I think of that as a "file".

> I have it in memory. Embedded Tomcat is running in memory. They
> are in the same JVM. Why do I have to put something in a file?

You'll have to see what is required to tell Tomcat to reload an
SSLHostConfig that was programmatically-created. That's over my head at
this point.

>> What you have to remember is that AWS is often MORE EXPENSIVE than
>> running your own stuff.
>
> It shouldn't be, in general. Part of the whole cloud value proposition
> is that you only pay for what you use.

Yes, but if you need 100% utilization then you ave to pay for it. Like
you are seeing with e.g. ELB charges. The ELB has to be up even if your
application isn't running any active nodes.

> The problem here is that this big SSL antipattern hurdle is making me
> need to use things I don't need for a tiny app. I'm wanting to remove
> that hurdle.

If you'd bite the ELB bullet, you could just get free auto-renewing
certificates from Amazon. *shrug*

>> I hear that. I still don't understand how you are supposed to get OS
>> updates working for an application deployed into Elastic Beanstalk.
>
> See
> https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/environment-platform-update-managed.html
> .
>
> But I don't want to over-focus on Elastic Beanstalk.
>
> The issue here is that I want a standalone app with embedded Tomcat that
> automatically 100% configures SSL. For now Elastic Beanstalk is just one
> vehicle to run that app. In the future we may have platforms (if we
> don't already) that completely hide the OS, and the JAR is just running
> as a service.
>
> We have the same sort of thing with static sites on AWS. Remember that
> static site generator, Guise Mummy, I showed off at ApacheCon 2019? I
> finished it, and all my sites are now using it. Guise Mummy generates
> and deploys a static site to S3, but then configures CloudFront to cache
> and serve it with edge access around the world, with HTTPS. (See
> https://www.globalmentor.com/ as an example.) What web server is serving
> those files? What OS is hosting the web server? I have no idea. It's
> irrelevant. How do I update the OS? I don't need to. For all practical
> purposes there is no OS. My site consists of static files, and AWS is
> hosting them.

So your web server is S3.

> Eventually the same should go for my JAR file. I give AWS my JAR file.
> AWS starts up my JAR file. It has HTTPS automatically. I shouldn't have
> to do a single thing.
>
> We are almost there, except for this huge SSL problem where I have to do
> basically what I did in 2000 to get SSL turned on. (Maybe it's a little
> easier. Not much.)
>
>> Really, I'm speaking from a position of profound ignorance on this
>> point. I really have no idea how this stuff is supposed to work and
>> I haven't put any time into discovering the real answers.
>
>
> Chris, I'm learning all this stuff too. I had to get over a bunch of
> gotchas to get this S3->CloudFront with automatic SSL stuff working for
> a static site. And you can see from my Stack Overflow question
> https://stackoverflow.com/q/63650514 that I naively approached Elastic
> Beanstalk from the same view, thinking it would transparently handle
> everything for me. It doesn't. But that's where we want to go eventually.

ELB ~= CloudFront, right? If CloudFront handles your SSL for you, why
not let ELB do it for you in this context?

It doesn't have to be as hard as you are making it sound. And your
application shouldn't have to deal with this. You ave decided that your
application should handle it, and so you are making more work for
yourself (and your application). If you separate those concerns, you
will have a simpler system. Well, it's a more complicated system
overall, but each part is simpler.

> So I have a lot to learn. One of the first things I'm learning is that
> SSL is a problem Everywhere. All the gazillions of cute demos don't
> include SSL. If you do set up SSL, you have to jump through all these
> hoops, or pay lots extra to have a load balancer. Really, there isn't an
> easy SSL solution for anything in the cloud that I can find that isn't a
> static site. If you know of something, then please go answer my Server
> Fault question: https://serverfault.com/q/1036276 .
>
> If nobody else will do this, I will do this. It needs to be done,
> because I don't enjoy playing sysadmin just to set up SSL when I'd
> rather be writing applications and deploying them.

If you are building a Docker container and it won't kill you to have
anything running on it besides "java -jar myapp.jar" then I would
suggest that you set things up using the process I detailed in that 2019
ACNA presentation you mentioned. It would require that you:

1. Fetch files from S3 on Docker deployment (this seeds the node with
any existing keys + certs)

2. Run certbot on Docker deployment (which may decide not to update if
the files are fresh enough)

3. Configure cron to call the script included with the ACNA 2019
presentation once per week

4. That cron script needs to be updated to push any new files generated
by certbot -> S3

5. Profit

Or you could build a bunch of Java code because you want your
application itself to handle this.

-chris

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: completely automated (for real) Let's Encrypt on embedded Tomcat

Christopher Schultz-2
In reply to this post by James H. H. Lampert
James,

On 10/5/20 19:59, James H. H. Lampert wrote:
> I'm coming into this conversation late, so what I say could be
> completely irrelevant, but when I recently set up an independent (i.e.,
> not behind httpd) Tomcat server on one of our AWS EC2 instances, and
> could not get certbot to function at all, to save my life, I ended up
> using something called "LEGO."

Thanks for mentioning LEGO. Any time I've been mentioning certbot, you
can replace that with $your-favorite-acme-client.

> It *does* require one to shut the Tomcat server down during the
> renewal process (because it has to take over the port briefly), but
> it also *does* play nicely with a Tomcat server that's doing its own
> SSL.

You *should* be able to do this without stopping Tomcat, but it might
end up complicating other things. If you have a reverse proxy server,
this is trivial to avoid. If you are binding Tomcat directly to port 80,
this is not so easy.

Another option is to use DNS-based authentication where your web server
isn't involved.

-chris

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: completely automated (for real) Let's Encrypt on embedded Tomcat

James H. H. Lampert
On 10/6/20 2:48 PM, Christopher Schultz wrote:
> Thanks for mentioning LEGO. Any time I've been mentioning certbot, you
> can replace that with $your-favorite-acme-client.

You're welcome.

LEGO definitely cut my Gordian Knot on that particular project, wherein
Certbot absolutely, positively, refused to work. The other AWS
installations either have httpd handling the https front-ending, or they
have a load-balancer handling it, with Amazon itself providing the certs.

So I'm always likely to suggest it as a second choice (and one that
appears to play much more nicely with Tomcat) if certbot won't do it in
a given situation.

--
JHHL

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: completely automated (for real) Let's Encrypt on embedded Tomcat

Garret Wilson
In reply to this post by Christopher Schultz-2
As always thanks for the discussion, Chris. More replies and a new idea
below:

On 10/6/2020 2:45 PM, Christopher Schultz wrote:
> …
> What if your Docker container would just run certbot on launch?

But then I'm back to being a sysadmin, because the Docker container is
like a little OS and I have to set up the OS, update it, install certbot
if needed (based on the OS version!), ensure it's the certbot that has
the bugfixes I want and behaves as I expect, install Tomcat on the OS,
etc. I have to set up certbot with systemd or or whatever to run
periodically. All that stuff goes into my Dockerfile. So I'm really
doing the same thing as I would on a VM, except that Docker makes it
easier to reproduce. But it's conceptually not much different than being
sysadmin on a VM, then freezing the VM and duplicating that.

>
> If you build the keystore in-memory, I'm not exactly sure what you'd
> need to do in order to get Tomcat to bounce the SSLSocketFactory in that
> way.

OK, I'll look into it. But it boggles my mind that an SSLSocketFactory
would have a harder time using actual bytes of a certificate than it
would loading it from disk. Because once it loads it from disk, it has
the bytes in memory, no? So the only problem would be the API—whether it
was built to unnecessarily require disk storage, or if it allows a
"hook" to provide the bytes at the point in the logic between
loading-from-disk and using-the-cert.

>
>> I have no idea what a PEM-encoded DER file is, but I'll certainly learn.
> This:
>
> -----BEGIN CERTIFICATE-----
> stuff
> -----END CERTIFICATE-----
>
Ooh, that stuff. Yeah, I've definitely used that. I just never knew what
it was. I never really got into the deep murky depths of certificates,
but I guess now I'll have, seeing as that no one else in the last 20
years has really made this easy.

>> I still don't get why files have to be involved. I pulled a DER file
>> from S3.
> Sorry. I think of that as a "file".

Ah, there's an important distinction to be made here. As developers we
often say "file" when we talk about a bunch of bytes that has some
coherent format — a JPEG file, an Ogg Vorbis file, a JSON file. But (and
the specs only really started making this clear around the time XML
became a spec, as least as far as I know) from another view it's only a
file if it's stored in a file system. (I'll explain why below.) So
really we can talk about an "XML document", for example, that may never
be stored in a file — we may construct an XML DOM tree completely in
memory and pass it to some other method.

>
> So your web server is S3.

Ah, no!

S3 is a database. It could be PostgreSQL or Cassandra. It's a data
store. In particular it's a key-value store. It stores things. It has a
feature that allows you to set a configuration so that some web server
will automatically serve the BLOBs from the S3 data store, along with
some metadata. But is the web server "part of" S3? I'll bet not. Surely
there is some pool of workers that pull data from the S3 data store and
serves the data. I could probably write my own web server to do the same
thing. But AWS makes it transparent; it's part of the infrastucture. I
just provide the data and declaratively tell it what I want served. Then
it is served. I don't have to configure a server. There is some server
in the cloud.

But there's actually another step! Since I'm serving the site via
CloudFront, the CloudFront layer (servers around the world in different
countries!) actually connects to the web site serving the S3 data and
/copies it to CloudFront/! So my data is actually being served to the
end user by some CloudFront server, in some country closest to the user.
Is this Tomcat? Is it NGINX? Is it Apache? I don't know. Do I care what
it is? No. In fact, I'd rather not know, because I want to configure the
distribution and serving /independent from any server implementation/.

So my web server is something on some CloudFront deployment in some
country. For all I know different CloudFront local deployments use
different server types. I don't know and I don't care.

>
> ELB ~= CloudFront, right?

Sort of. It is equivalent in that 1) it is the direct endpoint
connection with the user, and 2) I can configure it to use the AWS
managed certificates.


> If CloudFront handles your SSL for you, why
> not let ELB do it for you in this context?


That's what I'm doing now! And that's what I would prefer doing. And
that's what I would need to do for a large application!

But the problem (going back to the motivation for all this) is that the
ELB costs a lot of money. I don't absolutely need an ELB for a small app
that I'm testing, or an app that will have 10 users, or an app that I
don't care that crashes and immediately gets restarted. Why should I pay
$16 or more per month for an ELB if I don't need it? When I have five
little apps I want to deploy?

Think about the thousands of little apps people want to deploy around
the world. Is their choice really between paying $5/month on Digital
Ocean and completely managing a VM, or setting up some intricate set of
replicate servers with load balancers and the like? Why can't they just
drop a JAR somewhere and let it run with SSL, pay $3/month for it, and
not have to become CentOs sysadmin? That is what I'm getting at.

> It doesn't have to be as hard as you are making it sound.

OK, I guess I just haven't found the guide for the simple drop-in-a-JAR
and get an application on a custom domain with SSL. Could you send me
that URL and I'll read it.

>
> If you are building a Docker container and it won't kill you to have
> anything running on it besides "java -jar myapp.jar" then I would
> suggest that you set things up using the process I detailed in that 2019
> ACNA presentation you mentioned.

Could you provide me a link to that? Because I think I only have your
2018 Tomcat Let's Encrypt slides. The 2019 one is the one I was looking for.

> It would require that you:
>
> 1. Fetch files from S3 on Docker deployment (this seeds the node with
> any existing keys + certs)
>
> 2. Run certbot on Docker deployment (which may decide not to update if
> the files are fresh enough)
>
> 3. Configure cron to call the script included with the ACNA 2019
> presentation once per week
>
> 4. That cron script needs to be updated to push any new files generated
> by certbot -> S3

Ah, if this is the "simple" you were talking about, then … that's not
simple in my book. It requires you know all sorts of intricacies about
certbot. I've ran into bugs with different versions, and different
versions of CentOS come distributed with different certbot versions, for
instance. So then am I going to have to build certbot from scratch? Then
comes the nightmare of making sure I have the right libraries that match
what certbot needs, and these things are always changing.

Oh, and cron? Sure, I'll bet you know that inside and out because you've
been working on it for years. I have no doubt you know it 10 times
better than I do. But it's another bunch of intricacies and gotchas
someone has to learn.

But shouldn't we be moving on from cron anyway?
https://trstringer.com/systemd-timer-vs-cronjob/

But why should I learn about any of this!!! I just want to drop a JAR
and have it run on the web! All the stuff you're talking about is what
the infrastructure should handle. It is completely orthogonal to the
application functionality.

If I keep talking about this it will seem like I'm ranting and arguing,
and I don't want to come across that way because I really want to
produce something useful, and I'll very much need your help with the
questions and problems I run into. So I was just getting excited
explaining my thoughts. ;)

But anyway, let me tell you the idea I had this morning. In a way, you
hinted at it in your reply. Why do I need to use S3 as a store if my
application is running on AWS, and AWS already has the AWS Certificate
Manager which already manages an SSL certificate with renewal! In
essence the AWS Certificate Manager is the "data store/state" like S3,
and I don't even need to call Let's Encrypt.

What I need to do is to write code in my application to merely ask AWS
Certificate Manager for the certificate when it starts up, and then pass
that to Tomcat.

I have already written code for Guise that sets up a certificate in AWS
Certificate Manager and configures the DNS in Route 53 so that the
certificate gets updated. It's all a single command `guise deploy`. It's
all written.

Now I just need to write the code to 1) get the certificate from AWS,
and 2) give it to Tomcat. And #2 will be the hardest, probably, so I'll
be asking lots of questions.

In the future I can go back and add the automatic Let's Encrypt
certificate request part, but for the first step I'll look into using
the one configured in AWS Certificate Manager. I'll go do more research …

Garret

Reply | Threaded
Open this post in threaded view
|

Re: completely automated (for real) Let's Encrypt on embedded Tomcat

Christopher Schultz-2
Garret,

On 10/7/20 13:12, Garret Wilson wrote:

> As always thanks for the discussion, Chris. More replies and a new idea
> below:
>
> On 10/6/2020 2:45 PM, Christopher Schultz wrote:
>> …
>> What if your Docker container would just run certbot on launch?
>
> But then I'm back to being a sysadmin, because the Docker container is
> like a little OS and I have to set up the OS, update it, install certbot
> if needed (based on the OS version!), ensure it's the certbot that has
> the bugfixes I want and behaves as I expect, install Tomcat on the OS,
> etc. I have to set up certbot with systemd or or whatever to run
> periodically. All that stuff goes into my Dockerfile. So I'm really
> doing the same thing as I would on a VM, except that Docker makes it
> easier to reproduce. But it's conceptually not much different than being
> sysadmin on a VM, then freezing the VM and duplicating that.

Building a Dockerfile makes you a sysadmin. If you were deploying a WAR
file into Elastic Beanstalk I might have more sympathy for your situation.

>> If you build the keystore in-memory, I'm not exactly sure what you'd
>> need to do in order to get Tomcat to bounce the SSLSocketFactory in that
>> way.
>
> OK, I'll look into it. But it boggles my mind that an SSLSocketFactory
> would have a harder time using actual bytes of a certificate than it
> would loading it from disk.

I didn't say that. I said that Tomcat's reload() method re-reads the
configuration. If you use Tomcat embedded, then there is no
configuration, and therefore reload() has nothing to reload. You will
have to see what needs to be done, there. Perhaps you can re-set the
SSLSocketFactory and then call reload() which will re-build the
SSLContext, etc. and all will be well. I'm just not making any promises.

> Because once it loads it from disk, it has
> the bytes in memory, no? So the only problem would be the API—whether it
> was built to unnecessarily require disk storage, or if it allows a
> "hook" to provide the bytes at the point in the logic between
> loading-from-disk and using-the-cert.

Right.

>>> I have no idea what a PEM-encoded DER file is, but I'll certainly learn.
>> This:
>>
>> -----BEGIN CERTIFICATE-----
>> stuff
>> -----END CERTIFICATE-----
>>
> Ooh, that stuff. Yeah, I've definitely used that. I just never knew what
> it was. I never really got into the deep murky depths of certificates,
> but I guess now I'll have, seeing as that no one else in the last 20
> years has really made this easy.
>
>>> I still don't get why files have to be involved. I pulled a DER file
>>> from S3.
>> Sorry. I think of that as a "file".
>
> Ah, there's an important distinction to be made here. As developers we
> often say "file" when we talk about a bunch of bytes that has some
> coherent format — a JPEG file, an Ogg Vorbis file, a JSON file. But (and
> the specs only really started making this clear around the time XML
> became a spec, as least as far as I know) from another view it's only a
> file if it's stored in a file system. (I'll explain why below.) So
> really we can talk about an "XML document", for example, that may never
> be stored in a file — we may construct an XML DOM tree completely in
> memory and pass it to some other method.
>
>>
>> So your web server is S3.
>
> Ah, no!
>
> S3 is a database. It could be PostgreSQL or Cassandra. It's a data
> store. In particular it's a key-value store. It stores things. It has a
> feature that allows you to set a configuration so that some web server
> will automatically serve the BLOBs from the S3 data store, along with
> some metadata. But is the web server "part of" S3? I'll bet not.

S3 has a web interface, so it's definitely a web server. You can say
it's a key-value store (which it is), but because you fetch objects from
it using HTTP GET, it's also a web server.

> Surely there is some pool of workers that pull data from the S3 data
> store and serves the data. I could probably write my own web server
> to do the same thing. But AWS makes it transparent; it's part of the
> infrastucture. I just provide the data and declaratively tell it what
> I want served. Then it is served. I don't have to configure a server.
> There is some server in the cloud.
>
> But there's actually another step! Since I'm serving the site via
> CloudFront, the CloudFront layer (servers around the world in different
> countries!) actually connects to the web site serving the S3 data and
> /copies it to CloudFront/! So my data is actually being served to the
> end user by some CloudFront server, in some country closest to the user.
> Is this Tomcat? Is it NGINX? Is it Apache? I don't know. Do I care what
> it is? No. In fact, I'd rather not know, because I want to configure the
> distribution and serving /independent from any server implementation/.

CloudFront is a caching web server, just like if you were to use httpd +
mod_cache. S3 is the origin server and CF is the caching reverse proxy.

> So my web server is something on some CloudFront deployment in some
> country. For all I know different CloudFront local deployments use
> different server types. I don't know and I don't care.
>
>> ELB ~= CloudFront, right?
>
> Sort of. It is equivalent in that 1) it is the direct endpoint
> connection with the user, and 2) I can configure it to use the AWS
> managed certificates.
>
>> If CloudFront handles your SSL for you, why not let ELB do it for
>> you in this context?
>
> That's what I'm doing now! And that's what I would prefer doing. And
> that's what I would need to do for a large application!
>
> But the problem (going back to the motivation for all this) is that the
> ELB costs a lot of money. I don't absolutely need an ELB for a small app
> that I'm testing, or an app that will have 10 users, or an app that I
> don't care that crashes and immediately gets restarted. Why should I pay
> $16 or more per month for an ELB if I don't need it? When I have five
> little apps I want to deploy?
>
> Think about the thousands of little apps people want to deploy around
> the world. Is their choice really between paying $5/month on Digital
> Ocean and completely managing a VM, or setting up some intricate set of
> replicate servers with load balancers and the like? Why can't they just
> drop a JAR somewhere and let it run with SSL, pay $3/month for it, and
> not have to become CentOs sysadmin? That is what I'm getting at.

Probably because there is a minimum cost to these things, and companies
like Digital Ocean and Amazon are in business to make money.

>> It doesn't have to be as hard as you are making it sound.
>
> OK, I guess I just haven't found the guide for the simple drop-in-a-JAR
> and get an application on a custom domain with SSL. Could you send me
> that URL and I'll read it.

TLS simply requires some basic infrastructure that isn't free. Even when
it's free.

>> If you are building a Docker container and it won't kill you to have
>> anything running on it besides "java -jar myapp.jar" then I would
>> suggest that you set things up using the process I detailed in that 2019
>> ACNA presentation you mentioned.
>
> Could you provide me a link to that? Because I think I only have your
> 2018 Tomcat Let's Encrypt slides. The 2019 one is the one I was looking
> for.

http://tomcat.apache.org/presentations.html#latest-lets-encrypt


>> It would require that you:
>>
>> 1. Fetch files from S3 on Docker deployment (this seeds the node with
>> any existing keys + certs)
>>
>> 2. Run certbot on Docker deployment (which may decide not to update if
>> the files are fresh enough)
>>
>> 3. Configure cron to call the script included with the ACNA 2019
>> presentation once per week
>>
>> 4. That cron script needs to be updated to push any new files generated
>> by certbot -> S3
>
> Ah, if this is the "simple" you were talking about, then … that's not
> simple in my book. It requires you know all sorts of intricacies about
> certbot. I've ran into bugs with different versions, and different
> versions of CentOS come distributed with different certbot versions, for
> instance. So then am I going to have to build certbot from scratch? Then
> comes the nightmare of making sure I have the right libraries that match
> what certbot needs, and these things are always changing.

You could set up a docker file for this and use docker-compose to
laminate the tomcat docker container with the certbot docker container.

I've never used Docker for anything. Isn't this what Docker is all about?

> Oh, and cron? Sure, I'll bet you know that inside and out because you've
> been working on it for years. I have no doubt you know it 10 times
> better than I do. But it's another bunch of intricacies and gotchas
> someone has to learn.
>
> But shouldn't we be moving on from cron anyway?
> https://trstringer.com/systemd-timer-vs-cronjob/

If you like new, shiny things then go ahead. cron has worked for 50 years.

> But why should I learn about any of this!!! I just want to drop a JAR
> and have it run on the web! All the stuff you're talking about is what
> the infrastructure should handle. It is completely orthogonal to the
> application functionality.

So you don't want to learn cron but you'll happily go learn about Quartz
and put that into your application instead?

It's all the same. It just depends upon which language you want to use
to set up all this stuff.

> If I keep talking about this it will seem like I'm ranting and arguing,
> and I don't want to come across that way because I really want to
> produce something useful, and I'll very much need your help with the
> questions and problems I run into. So I was just getting excited
> explaining my thoughts. ;)
>
> But anyway, let me tell you the idea I had this morning. In a way, you
> hinted at it in your reply. Why do I need to use S3 as a store if my
> application is running on AWS, and AWS already has the AWS Certificate
> Manager which already manages an SSL certificate with renewal! In
> essence the AWS Certificate Manager is the "data store/state" like S3,
> and I don't even need to call Let's Encrypt.

Exactly. And it will auto-deploy to ELB, but you don't want to use ELB.
So...

> What I need to do is to write code in my application to merely ask AWS
> Certificate Manager for the certificate when it starts up, and then pass
> that to Tomcat.

That should work. Take care not to allow it to expire while your
application is running.

> I have already written code for Guise that sets up a certificate in AWS
> Certificate Manager and configures the DNS in Route 53 so that the
> certificate gets updated. It's all a single command `guise deploy`. It's
> all written.
>
> Now I just need to write the code to 1) get the certificate from AWS,
> and 2) give it to Tomcat. And #2 will be the hardest, probably, so I'll
> be asking lots of questions.

#2 won't be hard. #3 is "getting Tomcat to reload when a new key+cert is
available" and that's the "hard" part.

> In the future I can go back and add the automatic Let's Encrypt
> certificate request part, but for the first step I'll look into using
> the one configured in AWS Certificate Manager. I'll go do more research …

-chris

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: completely automated (for real) Let's Encrypt on embedded Tomcat

Garret Wilson
In reply to this post by Garret Wilson
On 10/7/2020 10:12 AM, Garret Wilson wrote:
> …
> But anyway, let me tell you the idea I had this morning. In a way, you
> hinted at it in your reply. Why do I need to use S3 as a store if my
> application is running on AWS, and AWS already has the AWS Certificate
> Manager which already manages an SSL certificate with renewal! In
> essence the AWS Certificate Manager is the "data store/state" like S3,
> and I don't even need to call Let's Encrypt.

Darn, it turns out AWS doesn't allow me to directly use AWS Certificate
Manager certificates directly in my application.

    You cannot install your ACM certificate or your private ACM Private
    CA certificate directly on your AWS based website or application.
    (https://docs.aws.amazon.com/acm/latest/userguide/acm-services.html)

I guess it's back to Let's Encrypt using the original approach then.

I've already purchased a book on the intricacies of SSL certificates and
I have more on my shopping list. I'll start working on this and get back
to you with questions.

Cheers,

Garret