4.x Upgrade Issue
Hello!
Self hosting. I recently upgraded from 3.3.2 to 4.3.0. The migrations worked. Life was good. EXCEPT... I have what I assume is a decryption issue. OIDC fails with the error Message=Errors.Internal Parent=(ID=CRYPT-Jaik2R Message=error decrypting value Parent=(invalid character '\u0097' looking for beginning of value))" oidc_error.description=Errors.Internal oidc_error.type=server_error status_code=500. The UI flakes. I have reverted to 3.4.2 which works fine. But any 4.x series seems to fail.
Running in Cloud Run, where it's insanely easy to direct traffic amongst versions. So appreciative for being able to just revert to the older version.
Any ideas where to look? More debugging I can do?
10 Replies
Hi @euphline, I'm an AI assistant for ZITADEL. While a team member reviews this question, I'll see if I can find relevant sources for your question.
Answer
I wasn't able to find a direct answer, here are some relevant sources:
1. [Troubleshoot ZITADEL] (1) :
Includes general troubleshooting steps for self-hosted ZITADEL, which may help in diagnosing upgrade and runtime issues.
2. [Do you offer support for self-hosting?] (2) :
Describes support options for self-hosted deployments, including access to technical account management for deeper debugging.
ā
To learn more, see the sources I considered:
[1] Troubleshoot ZITADEL
[2] Do you offer support for self-hosting?
ā
Feel free to tag @Ask Inkeep with additional questions.
So ... I did a deep dive into the zitadel code with the help of AI. It pointed me to the web_keys1 table. I went through lots of machinations to try to clean that thing out and get them to recreate to no avail. But finally, after enabling the web keys feature flag, I was able to add a new OIDC web key (I used ED25519) and delete the old ones. It worked.
Glad I could be helpful. Feel free to create a new thread with any new questions you may have.
So all in all - the upgrade had a bit of a painful hurdle, but ... I'm over it and life is good. Now on to figuring out the migration to the new login page... š
@euphline did you follow any migration guide to perform the upgrade? I have a self-hosted instance running 2.xx.x and im worried ill run into things like this and want to be careful
Not per-se. But I can walk you through what I did. (Although it's possible I skipped steps now and then and had to re-run things - which may have resulted in some of my challenges).
What kind of environment are you running in?
( @Shardool )
Ah i see, Running on kubernetes with chart v8.11.2 and zitadel v2.67.2!
Great. So I was in Cloud Run, which is slightly different, but same concept.
Step 0: Get ready - Get docker instances of the versions you're going to upgrade through.
Step 1: Backup your database. š (I didn't need it, but it way lowered my blood pressure).
Step 2: Upgrade one major version at a time. So go to v3.x first. Install a job (not a service) to do a one-time run of the new version of Zitadel with the setup parameter. This is SAFE. You'll be able to downgrade afterwords. Really. You can even do it while the existing code is running!!
Step 3: Backup your database. š
Step 4: Go ahead and swap your 2.67.2 service for the 3.x service.
Step 5: Do the same to go to 4.x
In my case, I had a decryption issue with web keys that required some creativity, but I was able to solve it by going back to the previous version and running there until I figured it out.
Zitadel is incredibly well behaved in upgrades in my experience. If anything, so well behaved it's kinda scary. Because who would think you could just back up to the other version in parallel even after database changes have happened? I mean really. Yeah - it works!
Oh, also, you can re-run that "setup" job as many times as you want.
(Not that it does any good. But it'll make you feel like you're doing something while you're busy wandering in the desert).
thank you haha! yea im using helm-chart so the setup job does run automatically when i upgrade the chart version
š¤ hoping for no issues
š¤ on this side too. I'm curious to hear how it goes!