Very slow migration (2.61.0 --> 3.3.0)
Hi team!
First of all, thank you for this amazing piece of software! Really great dev experience so far.
We are self-hosting Zitadel on Kubernetes. As we continue developing our application that will eventually integrate with Zitadel, we decide to update our dependencies, including Zitadel itself. We upgraded from 2.61.0 to 3.3.0. The database contains ~84,000 users.
Our Kubernetes deployment uses hooks to ensure that
zitadel-setup runs ahead of new deployment. However, we've been having an issue with the job timing out. It currently has a timeout of, 9000s but still doesn't fully complete the migration successfully. It seemingly gets stuck at the user table migration.
Instead, we wiped the database and re-imported, which is only an option in the dev environment. I would like to improve this process in the future. Is a very slow migration expected? What are best practices to ensure a swift update?
Thanks!!5 Replies
Hm can you share how big the tables are? Esp. the events2 table.
We have seen some downtime when migrating from 2x to 3x but I only saw like 10-60mins cases so far.
Thank you btw. for the nice words!
I do not have that database server any more but will try to reproduce - thinking about it, I wonder if the
zitadel-setup job in Kubernetes could be limited in terms of compute, resulting in a very slow process. I note that the default deployment didn't have cpu/memory limits set for the job, so not sure how it works out in practice.Hm yeah it could be that the zitadel job is too restricted in CPU or Memory or that the DB is too restricted.
The DB should be okay, it's running on a pretty beefy RDS instance on AWS - I will play with the sizing to see if it makes a difference.
Did you check the settings on the zitadel job as well?