Notifier errors since upgraded to 3.0.4

Hello!

I am looking for help to understand and debug an issue I have with my Zitadel service.
I recently upgraded my self-hosted Zitadel instance from 2.63.4 to 3.0.4 and since then I am getting recurring errors related to some Notifier resource.

Every 30 minutes I see the two following errors logs:

level=ERROR msg="Notifier: Error from notification wait" err="unexpected EOF"
level=ERROR msg="Notifier: Error running listener (will attempt reconnect after backoff)" attempt=180 err="unexpected EOF" sleep_duration=8m30.106954557s

level=ERROR msg="Notifier: Error from notification wait" err="unexpected EOF"
level=ERROR msg="Notifier: Error running listener (will attempt reconnect after backoff)" attempt=180 err="unexpected EOF" sleep_duration=8m30.106954557s

(small note: it feels like the retry backoff delay is not working properly since the error happens every 30 minutes)

I tried multiple small configuration adjustments but none of them resolved it. Unfortunately, I also could not find anything on the internet that would point me in the right direction when it comes to fixing it.

Thank you in advance

craigzourOP•5/23/25, 12:25 PM

(Bump)

Rajat Singh•5/23/25, 12:29 PM

hey @craigzour thanks for the bump, looking into it

Rajat•5/26/25, 9:12 AM

hey @craigzour The error indicates that the Notifier component is experiencing an abrupt termination of its connection to the notification service or message broker. This could be due to multiple reasons, such as network interruptions, misconfigurations.
I will take it to my team and ket them have a look

Rajat•5/26/25, 9:15 AM

also, what exactly you tried when you say "I tried multiple small configuration adjustments but none of them resolved it"

craigzourOP•5/26/25, 12:38 PM

Hello @Rajat .

Thank you for spending some time looking into this. I tried to make some small Zitadel configuration adjustments such as:

Tweaking the database connection options (MaxOpenConns and MaxConnLifetime)
Settings up the Notifications options with either LegacyEnabled set to true or false (because I thought it was related to the Notifier word included in the error message)

craigzourOP•5/26/25, 12:51 PM

Also just to make sure we are on the same page, we have not touched anything else outside of that Zitadel upgrade.

Ccraigzour Hello @Rajat . Thank you for spending some time looking into this. I tried to m...

Rajat•5/28/25, 9:16 AM

thanks for the update @craigzour I will send this to my team internally and will get back to you.

adlerhurst•6/3/25, 8:50 PM

Hi there
30 minutes is the default max connection lifetime to the database. Can you increase or decrease that value so that we figure out if thats the reason.

adlerhurst•6/3/25, 8:53 PM

Ah sorry i skipped that message, changing the conn max lifetime didn’t change the interval of the log?

craigzourOP•6/4/25, 12:50 PM

Hello!
This is correct. Even though we changed to the Notifier errors cadence stayed the same (every 30 mins or so)

craigzourOP•6/4/25, 12:54 PM

Here is our current config file in case it helps debugging (we reverted that maxconnlifetime change since we still had the issue)

adlerhurst•6/13/25, 5:54 PM

Thanks for the info we try to reproduce

craigzourOP•6/13/25, 6:19 PM

Thanks you for the update

adlerhurst•6/18/25, 7:55 AM

hi there
I just created this issue for tracking: https://github.com/zitadel/zitadel/issues/10092

GitHub

Error running listener · Issue #10092 · zitadel/zitadel

From this discord thread: https://discord.com/channels/927474939156643850/1374467931672543262 They see the following error logs every 30 minutes: level=ERROR msg="Notifier: Error from notifica...

craigzourOP•6/18/25, 12:30 PM

Hello! Perfect! Thank you

craigzourOP•8/25/25, 2:55 PM

Hello!

Wanted to share some new information about this as I spent some time investigating on it recently.

I tried to find what code was throwing that recurring error and found out it was not in Zitadel directly but in a package named Riverqueue which has been integrated in Zitadel 6 months ago. It was not part of the Zitadel version we were using before (2.63.4) the migration to 3.0.4.

After investigating and testing various things I discovered that our AWS RDS Proxy has an Idle timeout set to 30 minutes which is exactly the frequency at which we get those errors. I tried to increase that value to 8 hours (maximum allowed) and noticed that we were only getting one error every 8 hours.

In the AWS RDS Proxy logs I also found out that we were getting that log just before getting the Notifier error

So all that to say that I think the issue is between Riverqueue and our AWS RDS Proxy. I believe Riverqueue needs an open connection with the database that never ends but our Proxy terminates it because it does not see any activity.

craigzourOP•8/25/25, 3:01 PM

I don't know much about Riverqueue but I am guessing that removing the AWS RDS Proxy would probably solve my problem.

Since you folks probably know more about it, please let me know if you have a solution in terms of configuration that would allow me to keep that Proxy

Maybe there is a way for Riverqueue to keep that connection alive or go around that AWS RDS pinning thing.

craigzourOP•8/25/25, 7:43 PM

FYI: I did remove the AWS RDS Proxy and have not seen that issue come back.

adlerhurst•9/8/25, 7:34 AM

oh wow these are great news. Thanks for digging so deep into the problem

adlerhurst•9/8/25, 7:47 AM

FYI, this PR will fix the issue because Zitadel does not change the application_name anymore: https://github.com/zitadel/zitadel/pull/10560

GitHub

fix(projections): overhaul the event projection system by adlerhurs...

This PR overhauls our event projection system to make it more robust and prevent skipped events under high load. The core change replaces our custom, transaction-based locking with standard Postgre...