Solution for the Let's Encrypt 404 Not Found Problem

Let's Encrypt is a great service, which provides SSL certificates for free, which is very important for safety on the internet. One of the (maybe even only) downside of this service is that the certificate durations are only 3 months. A few days ago, the certificate for berkersonmez.com is expired. Since I did not configure auto renewal, I tried to manually renew the certificate by using Certbot's command:

sudo certbot certonly --webroot -w /home/zzz/berkersonmez.com -d berkersonmez.com -d www.berkersonmez.com

Unexpectedly, I got the following error:

Cert is due for renewal, auto-renewing...
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for berkersonmez.com
http-01 challenge for www.berkersonmez.com
Using the webroot path /etc/zzz/berkersonmez.com for all unmatched domains.
Waiting for verification...
Cleaning up challenges
Failed authorization procedure. berkersonmez.com (http-01): urn:acme:error:unaut                                             horized :: The client lacks sufficient authorization :: Invalid response from ht                                             tp://berkersonmez.com/.well-known/acme-challenge/xbA1zJB3s--lk8acMKs5D1_tTNrcvNC                                             w8X-Nr5JCp8I: "<html>
<head><title>404 Not Found</title></head>
<body bgcolor="white">
<center><h1>404 Not Found</h1></center>
<hr><center>", www.berkersonmez.com (http-01): urn:acme:error:unauthorized :: Th                                             e client lacks sufficient authorization :: Invalid response from http://www.berk                                             ersonmez.com/.well-known/acme-challenge/JEok4-X6vix6uNA_M1i84uJUz1GJkq1Y6b7aLoDw                                             uM0: "<html>
<head><title>404 Not Found</title></head>
<body bgcolor="white">
<center><h1>404 Not Found</h1></center>
<hr><center>"

Basically, Certbot puts a challenge file which Let's Encrypt server tries to access over HTTP. In my case, the challenge file could not be accessed and 404 File not found is returned from my server.

I tried to google for the solution, and found out that there are many posts about this problem on Let's Encrypt's community pages such as this and this and this etc. Generally, in these posts people suggested creating a manual test file in the directory the Let's Encrypt's challenge files are created (".well-known/acme-challenge") and trying to access the file from the browser. I did go ahead and checked that the file I created can be accessed from my browser and no 404 errors occurred. I also looked at my Nginx configuration thinking maybe a security configuration is disallowing Let's Encrypt's servers from accessing the url. After a lot of frustration, I ignored the certificate problem for a few days, which resulted in some problems.

After a while, I found this thread where the issue is resolved by removing the IPv6 AAAA record from the DNS records of the server. I checked my DNS manager and saw that there were AAAA records for my domain. It is mentioned in the said thread that if there is AAAA record for the domain, Let's Encrypt tries to access your server over IPv6. I tried deleting the AAAA records, and retried renewing the certificate. When I got the same error I thought the trick did not work, but it would probably resolve the issue if I had waited a little. Also, since removing AAAA records may cause some other problems, I wanted to find a better solution.

I checked Nginx access logs and saw that Let's Encrypt server indeed tries to access the challenge file over IPv6:

2600:3000:1511:200::1d - - [17/Aug/2017:20:33:26 +0300] "GET /.well-known/acme-challenge/JEok4-X6vix6uNA_M1i84uJUz1GJkq1Y6b7aLoDwuM0 HTTP/1.1" 404 152 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)

Trying to curl using IPv6 actually made me pretty sure the issue was about this:

$ curl -IkL6 https://berkersonmez.com/.well-known/test.txt
curl: (35) gnutls_handshake() failed: An unexpected TLS packet was received.

This time I aimed to allow access over IPv6. While searching, I found this page where the Nginx settings for IPv6 access is presented. Therefore, I added neccessary lines as following:

server {
    listen 80;
    listen [::]:80 ipv6only=on;
    ...
}
server {
    listen 443;
    listen [::]:443 ipv6only=on;
    ...
}

After restarting Nginx, I was able to renew the certificate using Certbot's command.

Too Long Didn't Read

If you encounter 404 Not Found errors during Let's Encrypt certificate renewal, suspect that the issue may be about IPv6. Try applying these steps:

Make sure Certbot's webroot configuration is correct.
Add a test.txt file inside "<webroot>/.well-known/acme-challenge" and try to access it from your browser.
Try to access the test file via Curl with following commands. See if attempt with "-IkL6" option gives error while the other one works.
- curl -IkL4 <URL>
- curl -IkL6 <URL>
Try configuring your server to allow IPv6 access through your domain name. If you are using Nginx, you need to add listen directives like this to your configuration file:
- listen [::]:80 ipv6only=on;
If none works, try removing AAAA records from your DNS manager.

EDIT: Three months later and I again encountered a problem during renewal. Just keep in mind that, if you are doing a http to https redirection, be sure to serve challenge files directly through http. Add an exception to the redirection like this:

server {
    listen 80;
    listen [::]:80 ipv6only=on;
    server_name berkersonmez.com www.berkersonmez.com;

    location /.well-known {
        root /home/zzz/berkersonmez.com;
        allow all;
    }
    location / {
        return 301 https://berkersonmez.com$request_uri;
    }
}