This is really weird. Looking at the NGINX error logs in more detail I can see that another one of my sites (example2.com) was giving the same errors.
2021/06/25 02:23:07 [error] 27666#27666: recv() failed (110: Connection timed out) while requesting certificate status, responder: r3.o.lencr.org, peer: 184.28.220.10:80, certificate: "/etc/letsencrypt/live/example.com/fullchain.pem"
2021/06/25 02:23:07 [error] 27666#27666: OCSP responder prematurely closed connection while requesting certificate status, responder: r3.o.lencr.org, peer: 184.28.220.10:80, certificate: "/etc/letsencrypt/live/example.com/fullchain.pem"
2021/06/25 02:23:14 [error] 27666#27666: recv() failed (110: Connection timed out) while requesting certificate status, responder: r3.o.lencr.org, peer: 184.28.220.74:80, certificate: "/etc/letsencrypt/live/example.com/fullchain.pem"
2021/06/25 02:23:14 [error] 27666#27666: OCSP responder prematurely closed connection while requesting certificate status, responder: r3.o.lencr.org, peer: 184.28.220.74:80, certificate: "/etc/letsencrypt/live/example.com/fullchain.pem"
2021/06/25 02:23:14 [error] 27666#27666: connect() to [2600:1406:1400::b81c:cb20]:80 failed (101: Network is unreachable) while requesting certificate status, responder: r3.o.lencr.org, peer: [2600:1406:1400::b81c:cb20]:80, certificate: "/etc/letsencrypt/live/example.com/fullchain.pem"
2021/06/25 02:23:14 [error] 27666#27666: connect() to [2600:1406:1400::b81c:cb13]:80 failed (101: Network is unreachable) while requesting certificate status, responder: r3.o.lencr.org, peer: [2600:1406:1400::b81c:cb13]:80, certificate: "/etc/letsencrypt/live/example.com/fullchain.pem"
2021/06/25 02:31:56 [error] 27666#27666: recv() failed (110: Connection timed out) while requesting certificate status, responder: r3.o.lencr.org, peer: 184.28.220.10:80, certificate: "/etc/letsencrypt/live/example.com/fullchain.pem"
2021/06/25 02:31:56 [error] 27666#27666: OCSP responder prematurely closed connection while requesting certificate status, responder: r3.o.lencr.org, peer: 184.28.220.10:80, certificate: "/etc/letsencrypt/live/example.com/fullchain.pem"
2021/06/25 02:32:03 [error] 27666#27666: recv() failed (110: Connection timed out) while requesting certificate status, responder: r3.o.lencr.org, peer: 184.28.220.74:80, certificate: "/etc/letsencrypt/live/example.com/fullchain.pem"
2021/06/25 02:32:03 [error] 27666#27666: OCSP responder prematurely closed connection while requesting certificate status, responder: r3.o.lencr.org, peer: 184.28.220.74:80, certificate: "/etc/letsencrypt/live/example.com/fullchain.pem"
2021/06/25 02:32:03 [error] 27666#27666: connect() to [2600:1406:1400::b81c:cb20]:80 failed (101: Network is unreachable) while requesting certificate status, responder: r3.o.lencr.org, peer: [2600:1406:1400::b81c:cb20]:80, certificate: "/etc/letsencrypt/live/example.com/fullchain.pem"
2021/06/25 02:32:03 [error] 27666#27666: connect() to [2600:1406:1400::b81c:cb13]:80 failed (101: Network is unreachable) while requesting certificate status, responder: r3.o.lencr.org, peer: [2600:1406:1400::b81c:cb13]:80, certificate: "/etc/letsencrypt/live/example.com/fullchain.pem"
2021/06/25 03:18:33 [error] 27666#27666: recv() failed (110: Connection timed out) while requesting certificate status, responder: r3.o.lencr.org, peer: 184.28.220.10:80, certificate: "/etc/letsencrypt/live/example2.com/fullchain.pem"
2021/06/25 03:18:33 [error] 27666#27666: OCSP responder prematurely closed connection while requesting certificate status, responder: r3.o.lencr.org, peer: 184.28.220.10:80, certificate: "/etc/letsencrypt/live/example2.com/fullchain.pem"
2021/06/25 03:18:41 [error] 27666#27666: recv() failed (110: Connection timed out) while requesting certificate status, responder: r3.o.lencr.org, peer: 184.28.220.74:80, certificate: "/etc/letsencrypt/live/example2.com/fullchain.pem"
2021/06/25 03:18:41 [error] 27666#27666: OCSP responder prematurely closed connection while requesting certificate status, responder: r3.o.lencr.org, peer: 184.28.220.74:80, certificate: "/etc/letsencrypt/live/example2.com/fullchain.pem"
2021/06/25 03:18:41 [error] 27666#27666: connect() to [2600:1406:1400::b81c:cb20]:80 failed (101: Network is unreachable) while requesting certificate status, responder: r3.o.lencr.org, peer: [2600:1406:1400::b81c:cb20]:80, certificate: "/etc/letsencrypt/live/example2.com/fullchain.pem"
2021/06/25 03:18:41 [error] 27666#27666: connect() to [2600:1406:1400::b81c:cb13]:80 failed (101: Network is unreachable) while requesting certificate status, responder: r3.o.lencr.org, peer: [2600:1406:1400::b81c:cb13]:80, certificate: "/etc/letsencrypt/live/example2.com/fullchain.pem"
The logs are filled with these errors for these 2 sites multiple times every day. There are only 15 days of NGINX error logs available so I don't know how long this has been going on but at least 15 days.
Here is the weird part. Since I manually renewed the cert for example.com yesterday, these NGINX errors have completely stopped (for both sites even though I didn't renew the cert for example2.com).
I also have another site example3.com with no entries at all in the NGINX error log.
example.com and example2.com both have an additional site set up on a subdomain, while example3.com does not.
These dev sites also don't have any entries in the NGINX error log.
It could be a coincidence, but the only 2 sites with entries in the error log are the 2 domains that have an additional site on the same domain.
The other thing I noticed is that all of these log entries are trying to connect to IPv6 addresses but my VPS does not have IPv6 networking.