Zitadel Fails DB Migration
Hello,
I use Zitadel on Docker on a Debian 12 VPS. As database I have a Docker backed CRDB container. I have automatic updates through Watchtower on both.
A month ago or so, after 2.69, I had the problem that some others had of the startup DB “verify migration” failing due to timeout. I kept 2.67.1 installed until yesterday, when CRDB started giving me problems. Now everytime I start Zitadel (also tried starting it with the DB mirroring to Postgres config) I get an error at startup, again, during DB verification and, then, another error that states Zitadel cannot connect to CRDB.
Tried downgrading CRDB, tried upgrading Zitadel to latest. Nothing.
Both are running on latest now.
Here are the logs I see:
https://pastebin.com/n5Ef4k78
Thank you
Pastebin
Attaching to zitadel_crdb_1, zitadel_db_1, zitadel_zitadel_1crdb_1 ...
Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time.
12 Replies
This is my Docker compose file:
https://pastebin.com/kYNbZuMd
Generally I use no external config, only ENV vars. The one you see is the mirroring one. Same errors both with normal startup or mirroring.
Pastebin
version: '3.8'services: zitadel: restart: 'always' networks:...
Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time.
Anyone? I have all my services with Login through Zitadel, I need to fix this
hi @Elmond thanks for your question, will look into this tomorrow 🙂
Hello @Rajat, thank you very much, any news?
hey @Elmond aplogies, I am looking into this
Hi @Elmond it could be a timeout issue, might have to look more and check with someone from within the team.
It looks like the main issue is the
49_init_permitted_orgs_function database migration timing out during Zitadel's startup. This timeout seems to destabilize CockroachDB, causing the subsequent connection reset/refused errors.Okay, I’ll wait for an update, thanks @Rajat
Hi @Elmond I have raised this internally, I will have an answer soon with soemthing
Thank you very much. Is there, by any chance, a prioritized support where I can pay a fee to get the issue looked at faster?
And if the prices are not astronomical for a student ahah
hi @Elmond can you try to chnage the DB config with the env flag ZITADEL_DATABASE_COCKROACH_OPTIONS this variable is present in defaults.yaml
With statement_timeout parameter.
you can do soemthing like
Will do tomorrow morning and update you. Thanks
time="2025-04-03T10:04:38Z" level=fatal msg="unable to initialize the database" caller="/home/runner/work/zitadel/zitadel/cmd/initialise/init.go:67" error="failed to connect to
user=root database=zitadel: 172.25.0.2:26257 (crdb): server error: ERROR: option "statement_timeout=600000" is invalid, must have prefix '-c' or '--' (SQLSTATE 08P01)"
It is not accepting it
environment:
- 'ZITADEL_EXTERNALPORT=443'
- 'ZITADEL_EXTERNALSECURE=true'
- 'ZITADEL_TLS_ENABLED=false'
- 'ZITADEL_DATABASE_COCKROACH_HOST=crdb'
- 'ZITADEL_EXTERNALDOMAIN=account.indst.eu'
- 'ZITADEL_LOGSTORE_ACCESS_STDOUT_ENABLED=true'
- 'ZITADEL_FIRSTINSTANCE_ORG_HUMAN_PASSWORDCHANGEREQUIRED=false'
- 'ZITADEL_DATABASE_COCKROACH_OPTIONS=statement_timeout=600000'
This is what I have as Environment
@Rajatcan you pls try
'ZITADEL_DATABASE_COCKROACH_OPTIONS=-c statement_timeout=600000'
If this doesnt work, I will check with my teamlevel=info msg="verify migration" caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:43" name=49_init_permitted_orgs_function
zitadel_1 | time="2025-04-03T16:58:05Z" level=error msg="failed to rollback transaction" caller="/home/runner/work/zitadel/zitadel/internal/database/database.go:58" error="sql: transaction has already been committed or rolled back"
zitadel_1 | time="2025-04-03T16:58:05Z" level=fatal msg="migration failed" caller="/home/runner/work/zitadel/zitadel/cmd/setup/setup.go:304" error="timeout: context deadline exceeded" name=49_init_permitted_orgs_function
zitadel_1 | time="2025-04-03T16:58:07Z" level=info msg="initialization started" caller="/home/runner/work/zitadel/zitadel/cmd/initialise/init.go:74"
zitadel_1 | time="2025-04-03T16:58:08Z" level=fatal msg="unable to initialize the database" caller="/home/runner/work/zitadel/zitadel/cmd/initialise/init.go:67" error="failed to connect to
user=root database=zitadel: 172.26.0.2:26257 (crdb): failed to receive message: read tcp 172.26.0.4:60052->172.26.0.2:26257: read: connection reset by peer"
zitadel_zitadel_1 exited with code 1
zitadel_1 | time="2025-04-03T16:58:10Z" level=info msg="initialization started" caller="/home/runner/work/zitadel/zitadel/cmd/initialise/init.go:74"
zitadel_1 | time="2025-04-03T16:58:10Z" level=fatal msg="unable to initialize the database" caller="/home/runner/work/zitadel/zitadel/cmd/initialise/init.go:67" error="failed to connect to user=root database=zitadel: 172.26.0.2:26257 (crdb): dial error: dial tcp 172.26.0.2:26257: connect: connection refused"
zitadel_zitadel_1 exited with code 1
crdb_1 | *
crdb_1 | * WARNING: ALL SECURITY CONTROLS HAVE BEEN DISABLED!
crdb_1 | *
crdb_1 | * This mode is intended for non-production testing only.
crdb_1 | *
Same original error
Even if I set it to 6000000
@Rajat
Any news?