100M Let’s Encrypt certificates – numbers you don’t know

If you want to run a server with a green padlock in your browser, you need an SSL certificate signed by one of companies, which were included in your operating system or browser. 

Let’s Encrypt has quickly become the biggest issuer of these SSL certificates since its launch in October 2016. It is now the largest certificate provider for internet facing servers. It is not clear how many servers are secured by its certificates, but at least 80 per cent of certificates valid at any time was issued by Let’s Encrypt (this estimate is based on a Frost&Sullivan report on SSL/TLS certificates from 2016 and actual data from Let's Encrypt .

It has reached 100,000,000 of issued certificates earlier this week and I think it’s time to look under the bonnet and talk about some other interesting facts behind the Let’s Encrypt technology.

Certificate validity

This is one of the biggest controversies around Let’s Encrypt. One of the goals of Let’s Encrypt has always been automation. It is a not-for-profit organisation funded by a number of large companies to introduce encryption (HTTPS) to the whole of the internet. They believe that automation is the key to achieve this goal and to encourage it, they decided their certificates will be only valid for 90 days.

Do you know that your Let’s encrypt certificate is in fact valid only 89 days and 23 hours?

As servers are not always fully synchronised with the real time, Let’s Encrypt back dates the beginning of validity by one hour. So you definitely have to renew any of your certificates in 90 days less 1 hour.

Limit on the number of certificates

When there is something free, people tend to over-use it. While there may be 80 per cent of the certificates from Let’s Encrypt, the actual number of servers protected may be much lower.

Do you know that in one week you can request 20 new certificates followed by 1,000 renewals, but if you do it the other way round the requests for new certificates will be rejected?

That is a reason why you can only get 20 new certificates per week for any given registered domain (second level domain – SLD).  However, it would not be fair if you can’t renew certificates to your existing servers so you can do an unlimited number of certificate renewals. The devil is in the fact that these renewals eat into your weekly allowance so you should be careful and plan your operations well if you’re a big user of the Let’s Encrypt certificates.

Do you know that if you regularly request a certificate every 8 hours and 25 minutes, some of your requests will start being rejected in less than 90 days (after your first renewal)?

Once you reach three months worth of your certificate allowance, i.e., around 240 certificates, just regular renewals may easily consume all your allowance for new certificates. You need to start planning certificate operations in batches so you can keep increasing your certificate inventory as your company grows.

Let’s Encrypt and ISP providers

Each user of Let’s Encrypt has to create its account. It doesn’t really matter much, if you are responsible for your own servers. If you start requesting certificates for your customers, you have to keep an eye on some more rate restrictions.

Do you know that you can create only 10 new accounts in 3 hours from one IPv4 address, but you are allowed 500 registrations from IPv6/48 range?

If you can manage all your customers from one account, then it is the recommended approach. If you want to separate users – just in case some of “account keys” get compromised, it pays off to use a management server with IPv6 as that can create 500 new accounts in 3 hours. 

Let’s encrypt and Online Certificate Status (OCSP)

When you think about certificate issuance, it’s a nice key management application with a low transaction rate. The biggest daily peak of Let’s Encrypt was 1.2 million certificates, which translates into 14 certificates per second.

Do you know that you are only allowed to request only 2,000 OCSP requests per second from each of your servers?

This is not true about online certificate status validation (the OCSP protocol), where browsers may request a real-time response each time they connect to a web server. This means a potentially huge transaction rate and Let’s Encrypt tries to protect itself with a particular rate limit of 2,000 requests per second per server. 

We need something a bit more scalable for growing internet and one of the current solutions is something called OCSP stapling, which is currently being rolled out. OCSP stapling is a functionality of web servers, which needs to be supported by browsers as well. Web-servers send cached OCSP responses to browsers, so they don’t need to query Let’s Encrypt directly. 

A complete list of limits and restrictions

While issuing certificates sounds like a pretty straightforward task, its implementation is a large system with many moving parts. It is necessary to manage its performance via various restrictions and limits. Let’s Encrypt goal to automate certificate management requires a reduction of certificate complexity as well as restricting the number of transactions.

100M certificates, how many server downtimes?

I believe that just simple automation of certificate issuance is not enough. Things break and they are more likely to break, if you look after servers with different operating systems or network configurations.

Many people will say that all you need is to “hook your renewal client to cron” and it will never fail. I’m not so sure you will sleep happily if your business depended on it. We should remember that an expired certificate takes the whole server down. No one can use it – it’s a perfect denial of service.

If you think it can’t happen to you, it did happen to DEFCON, one of the biggest hacker conventions, and it did happen to many other big and small companies.

There are several monitoring systems, you can use. Some free online services (alphabetical order):

• CertificateMonitor.org;
• KeyChest.net; or
• LetsMonitor.org.

There are standalone applications (GitHub projects):

• Certinel; or
• Checkssl

And there is also a number of commercial plug-ins or extensions to IT management systems.

Dan Cvrcek, co-founder, Enigma Bridge
Image Credit: Yuri Samoilov / Flickr