4.x Upgrade Issue
Hello!
Self hosting. I recently upgraded from 3.3.2 to 4.3.0. The migrations worked. Life was good. EXCEPT... I have what I assume is a decryption issue. OIDC fails with the error Message=Errors.Internal Parent=(ID=CRYPT-Jaik2R Message=error decrypting value Parent=(invalid character '\u0097' looking for beginning of value))" oidc_error.description=Errors.Internal oidc_error.type=server_error status_code=500. The UI flakes. I have reverted to 3.4.2 which works fine. But any 4.x series seems to fail.
Running in Cloud Run, where it's insanely easy to direct traffic amongst versions. So appreciative for being able to just revert to the older version.
Any ideas where to look? More debugging I can do?
4 Replies
Hi @euphline, I'm an AI assistant for ZITADEL. While a team member reviews this question, I'll see if I can find relevant sources for your question.
Answer
I wasn't able to find a direct answer, here are some relevant sources:
1. [Troubleshoot ZITADEL] (1) :
Includes general troubleshooting steps for self-hosted ZITADEL, which may help in diagnosing upgrade and runtime issues.
2. [Do you offer support for self-hosting?] (2) :
Describes support options for self-hosted deployments, including access to technical account management for deeper debugging.
ā
To learn more, see the sources I considered:
[1] Troubleshoot ZITADEL
[2] Do you offer support for self-hosting?
ā
Feel free to tag @Ask Inkeep with additional questions.
So ... I did a deep dive into the zitadel code with the help of AI. It pointed me to the web_keys1 table. I went through lots of machinations to try to clean that thing out and get them to recreate to no avail. But finally, after enabling the web keys feature flag, I was able to add a new OIDC web key (I used ED25519) and delete the old ones. It worked.
Glad I could be helpful. Feel free to create a new thread with any new questions you may have.
So all in all - the upgrade had a bit of a painful hurdle, but ... I'm over it and life is good. Now on to figuring out the migration to the new login page... š