KvH•3mo ago

Migrating Zitadel version stuck on step47_fill_membereship_fields

Environment: I'm self hosting Zitadel with a Postgres database. Version: Currently running 2.66.3. Trying to update to 3.2.3. Stack: I'm running Zitadel in a Kubernetes cluster in Azure and the Postgres database also in Azure running v15.12. Since it concerns membership fields, maybe it has something to do with the custom user roles? I'm setting the ZITADEL_INTERNALAUTHZ_ROLEPERMISSIONMAPPINGS arg to add some custom user roles on top of the existing roles. What you expected to happen: The migration runs successfully. What went wrong: The migration seems to get stuck on the migration step=47_fill_membership_fields. The migration has currently been running for 40 minutes, repeating the same warning message. See the attached .txt It has been a while since I've had the time to update the Zitadel instance. My previous attempt for migrating from v2.66.3 to 2.77.3 was unsuccessful for the same reason. Since I had some spare time now I wanted to try the update again, this time to the newest currently available v3.2.3. However the same problem has occurred since last time, so I assume the problem is introduced somewhere between v2.66.3 and v2.77.3. Anyone any idea as to why this is happening?

Zitadel_migration_st...

11 Replies

KvHOP•3mo ago

From what I can find from the code of step47 is that it checks for every instance that is created (which in my case is only the ZITADEL instance) and checks for all IAM roles/memberships in that instance. These memberships are then set again in the migration in possibly a new way/new table, which I'm not sure what specifically changed. These memberships for almost all of the users that have one on the instance level, have at least one or two of my custom roles:

242729515267129720    instance    242729515267129720    instance.member.changed    1639    1    2024-05-14 11:21:02.629842+00    {"roles": ["IAM_DC_ADMIN", "IAM_DC_OPERATIONAL"], "userId": "261108991784910980"}    247793137223729364    242729515267129720    1715685662.630443    0

242729515267129720    instance    242729515267129720    instance.member.changed    1639    1    2024-05-14 11:21:02.629842+00    {"roles": ["IAM_DC_ADMIN", "IAM_DC_OPERATIONAL"], "userId": "261108991784910980"}    247793137223729364    242729515267129720    1715685662.630443    0

Could it be that these custom roles are the issue when running this migration step? For reference, my custom roles look like this:

[
  // List of other pre-existing Zitadel roles.
  ...
  {
    "Role": "PROJECT_GRANT_OWNER_VIEWER",
    "Permissions": [
      "policy.read",
      "project.read",
      "project.grant.read",
      "project.grant.member.read",
      "user.read",
      "user.global.read",
      "user.grant.read",
      "user.membership.read"
    ]
  },
  {
    "Role": "IAM_DC_ADMIN",
    "Permissions": [
      "iam.read",
      "iam.write",
      "iam.member.read",
      "iam.member.write",
      "user.read",
      "user.write",
      "user.grant.read",
      "user.grant.write",
      "user.grant.delete",
      "org.member.write",
      "org.member.delete"
    ]
  },
  {
    "Role": "IAM_DC_FINANCE",
    "Permissions": [
      "iam.read",
      "org.read",
      "org.create",
      "org.write",
      "org.member.read",
      "org.member.write",
      "user.read",
      "user.write",
      "user.grant.read",
      "user.grant.write",
      "project.grant.read",
      "project.grant.write"
    ]
  },
  {
    "Role": "IAM_DC_OPERATIONAL",
    "Permissions": [
      "iam.read",
      "org.read",
      "org.write",
      "org.member.read",
      "org.member.write",
      "user.read",
      "user.write",
      "user.grant.read",
      "user.grant.write",
      "project.grant.read"
    ]
  }
]

[
  // List of other pre-existing Zitadel roles.
  ...
  {
    "Role": "PROJECT_GRANT_OWNER_VIEWER",
    "Permissions": [
      "policy.read",
      "project.read",
      "project.grant.read",
      "project.grant.member.read",
      "user.read",
      "user.global.read",
      "user.grant.read",
      "user.membership.read"
    ]
  },
  {
    "Role": "IAM_DC_ADMIN",
    "Permissions": [
      "iam.read",
      "iam.write",
      "iam.member.read",
      "iam.member.write",
      "user.read",
      "user.write",
      "user.grant.read",
      "user.grant.write",
      "user.grant.delete",
      "org.member.write",
      "org.member.delete"
    ]
  },
  {
    "Role": "IAM_DC_FINANCE",
    "Permissions": [
      "iam.read",
      "org.read",
      "org.create",
      "org.write",
      "org.member.read",
      "org.member.write",
      "user.read",
      "user.write",
      "user.grant.read",
      "user.grant.write",
      "project.grant.read",
      "project.grant.write"
    ]
  },
  {
    "Role": "IAM_DC_OPERATIONAL",
    "Permissions": [
      "iam.read",
      "org.read",
      "org.write",
      "org.member.read",
      "org.member.write",
      "user.read",
      "user.write",
      "user.grant.read",
      "user.grant.write",
      "project.grant.read"
    ]
  }
]

Rajat•3mo ago

hey @KvH thanks for your question, since its a DB issue, the least naive solution first - increase resources and wait longer if you can?. Meanwhile I will check with my engineers internally what we can do here.

KvHOP•3mo ago

Hey @Rajat , Thanks for the response. I'll check if I can let it run overnight or maybe over the weekend. Let me know if you find out something else.

KvHOP•3mo ago

Hello @Rajat , I've had the opportunity to try and run the migration overnight, but to no avail. Attached I have the logfile of the Zitadel pod and a screenshot of some of the metrics of the Postgres server of the last day. I've started the update just past 5PM (which is propably 3PM on the Azure server) and have let it run until this morning 8AM (6AM server time).

zitadel_try-update_1...

Rajat•3mo ago

hi @KvH I will raise this internally

KvHOP•3mo ago

Hello @Rajat , Have you had any time to look at this issue?

Rajat•3mo ago

hey @KvH I will tag @adlerhurst here as its a postgres issue 🙂

KvHOP•3mo ago

Hey @adlerhurst , Any chance you can help me with this issue? Hello @adlerhurst , We would like to keep up to date with the Zitadel releases, but this bug makes that impossible for us without having to rebuild our Zitadel Database from scratch and we can't really do that. Do you have some time to help with this issue?

adlerhurst•3mo ago

Hi there Im currently a bit short on time but i give my best to have a look at it

KvHOP•2mo ago

Hey @adlerhurst. I was wondering if you have had any time to check into this issue?

adlerhurst•4w ago

Hi @KvH Thanks for your patience. It looks like a previous run of zitadel setup got killed before it finished. Can you please run the zitadel setup cleanup with the same configuration flag as you use for setup and send the output?

Migrating Zitadel version stuck on step47_fill_membereship_fields

Did you find this page helpful?