Pallab
Pallab•8mo ago

Need help upgrading form v2.55.8 to v2.67.1

I have an instance of Zitadel running on EKS that uses PostgreSQL running on RDS. Current Zitadel Version: v2.55.8 Upgrade Version: v2.67.1 Postgres Version: 15.7 The instance was deployed using the official helm chart. The zitadel-setup job gets stuck at the following migration step 
caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:130" migration step=29_init_fields_for_project_grant
caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:130" migration step=29_init_fields_for_project_grant
This is the corresponding log from the database. 
2025-01-29 13:00:46 UTC:10.230.9.194(42190):postgres@zitadel:[29296]:STATEMENT: INSERT INTO eventstore.events2 (
instance_id
, "owner"
, aggregate_type
, aggregate_id
, revision
, creator
, event_type
, payload
, "sequence"
, created_at
, "position"
, in_tx_order
) VALUES
($1, $2, $3, $4, $5, $6, $7, $8, $9, statement_timestamp(), EXTRACT(EPOCH FROM clock_timestamp()), $10)
RETURNING created_at, "position";
2025-01-29 13:00:46 UTC:10.230.27.208(56588):postgres@zitadel:[28383]:ERROR: duplicate key value violates unique constraint "events2_pkey"
2025-01-29 13:00:46 UTC:10.230.27.208(56588):postgres@zitadel:[28383]:DETAIL: Key (instance_id, aggregate_type, aggregate_id, sequence)=(250999327709734704, project, 253045174010121988, 3515980) already exists.
2025-01-29 13:00:46 UTC:10.230.9.194(42190):postgres@zitadel:[29296]:STATEMENT: INSERT INTO eventstore.events2 (
instance_id
, "owner"
, aggregate_type
, aggregate_id
, revision
, creator
, event_type
, payload
, "sequence"
, created_at
, "position"
, in_tx_order
) VALUES
($1, $2, $3, $4, $5, $6, $7, $8, $9, statement_timestamp(), EXTRACT(EPOCH FROM clock_timestamp()), $10)
RETURNING created_at, "position";
2025-01-29 13:00:46 UTC:10.230.27.208(56588):postgres@zitadel:[28383]:ERROR: duplicate key value violates unique constraint "events2_pkey"
2025-01-29 13:00:46 UTC:10.230.27.208(56588):postgres@zitadel:[28383]:DETAIL: Key (instance_id, aggregate_type, aggregate_id, sequence)=(250999327709734704, project, 253045174010121988, 3515980) already exists.
I've tried running a zitadel setup cleanup from the debug pod as documented here: https://github.com/zitadel/zitadel-charts/blob/main/README.md However, the issue still persists and I'm unsure on how to resolve this. I tried searching the forum but haven't been able to figure out the right approach. Any help is appreciated. Thanks.
GitHub
zitadel-charts/README.md at main · zitadel/zitadel-charts
This repository contains Helm charts for running ZITADEL in Kubernetes - zitadel/zitadel-charts
11 Replies
fabienne
fabienne•8mo ago
@muhlemmer @stebenz are you able to help here?
Pallab
PallabOP•8mo ago
Going through previous posts on the forum, I came across this pull request that seems to fix the issue. However, this was added in a migration step that comes much later. https://github.com/zitadel/zitadel/pull/9089 However, I may be entirely wrong about how I am looking at the issue.
GitHub
fix: set correct owner on project grants by adlerhurst · Pull Reque...
Which Problems Are Solved In versions previous to v2.66 it was possible to set a different resource owner on project grants. This was introduced with the new resource based API. The resource owner ...
Unknown User
Unknown User•8mo ago
Message Not Public
Sign In & Join Server To View
Pallab
PallabOP•8mo ago
@stebenz Unfortunately, the setup still gets stuck on the same migration step with v2.67.5 
time="2025-01-31T08:42:47Z" level=warning msg="migration already started, will check again in 5 seconds" caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:130" migration step=29_init_fields_for_project_grant
time="2025-01-31T08:42:52Z" level=warning msg="migration already started, will check again in 5 seconds" caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:130" migration step=29_init_fields_for_project_grant
time="2025-01-31T08:42:57Z" level=warning msg="migration already started, will check again in 5 seconds" caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:130" migration step=29_init_fields_for_project_grant
time="2025-01-31T08:42:47Z" level=warning msg="migration already started, will check again in 5 seconds" caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:130" migration step=29_init_fields_for_project_grant
time="2025-01-31T08:42:52Z" level=warning msg="migration already started, will check again in 5 seconds" caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:130" migration step=29_init_fields_for_project_grant
time="2025-01-31T08:42:57Z" level=warning msg="migration already started, will check again in 5 seconds" caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:130" migration step=29_init_fields_for_project_grant
Is there something I should look at in order to move past this step? Performed cleanup and tried again. Still gets stuck at the same step 
zitadel@zitadel-debug-jj8qn:/app$ zitadel setup cleanup --config /zitadel-secret-config-yaml/db_config.yaml --config /config/zitadel-config-yaml
INFO[0000] cleanup started                               caller="/home/runner/work/zitadel/zitadel/cmd/setup/cleanup.go:33"
INFO[0000] cleanup migration                             caller="/home/runner/work/zitadel/zitadel/cmd/setup/cleanup.go:52" name=29_init_fields_for_project_grant
zitadel@zitadel-debug-jj8qn:/app$ zitadel setup cleanup --config /zitadel-secret-config-yaml/db_config.yaml --config /config/zitadel-config-yaml
INFO[0000] cleanup started                               caller="/home/runner/work/zitadel/zitadel/cmd/setup/cleanup.go:33"
INFO[0000] cleanup migration                             caller="/home/runner/work/zitadel/zitadel/cmd/setup/cleanup.go:52" name=29_init_fields_for_project_grant
Unknown User
Unknown User•7mo ago
Message Not Public
Sign In & Join Server To View
FFO
FFO•7mo ago
Unknown User
Unknown User•7mo ago
Message Not Public
Sign In & Join Server To View
FFO
FFO•7mo ago
Hm can you try something like zitadel setup --cleanup Oh just realised you did already Hm this is weird, I will look into this a little later, maybe I spot something... can you share your config? What happens if you upgrade to 2.55.9?
Pallab
PallabOP•7mo ago
The upgrade to v2.55.9 works. But upgrading to v2.56.0 and above gets stuck at this migration step 
29_init_fields_for_project_grant
29_init_fields_for_project_grant
Tried increasing the timeout as suggested here: https://github.com/zitadel/zitadel-charts/issues/281 But that did not work either.
FFO
FFO•7mo ago
Hm let me ask around, usually its the timeout, to what number did you increase it?
Pallab
PallabOP•7mo ago
@FFO We took it all the way up to 3600 seconds. @FFO Were you able to find something that can help with the issue?

Did you find this page helpful?