rudR
ZITADEL•5mo ago•
8 replies
rud

Staggering log volume when database is unavailable

We had service on a database the other day, and Zitadel is very enthusiastic about reporting errors, either DNS lookup failure or connection errors.

Here I reproduced the behaviour with a 4.1.3 install running in Docker Compose, and I stopped the 
db
 instance, so DNS resolution would fail. These logs are from the Zitadel container:

time=2025-09-11T09:33:17.066Z level=ERROR msg="producer: Error fetching queue settings" err="failed to connect to `user=zitadel_user database=zitadel`: hostname resolving error: lookup db on 127.0.0.11:53: no such host"

time=2025-09-11T09:33:17.539Z level=ERROR msg="producer: Error fetching jobs" err="failed to connect to `user=zitadel_user database=zitadel`: hostname resolving error: lookup db on 127.0.0.11:53: no such host" queue=execution

time=2025-09-11T09:33:17.539Z level=ERROR msg="producer: Error fetching jobs" err="failed to connect to `user=zitadel_user database=zitadel`: hostname resolving error: lookup db on 127.0.0.11:53: no such host"


We're seeing about 1mio log-lines per minute, which I think is risky behaviour for stability. Would it at all be possible to add some throttling to the connection retry mechanism? It seems like it might be a bit too eager, at least to me.
Was this page helpful?