wlinna•4d ago

Multiple errors when importing data from v2.71.6 to v4.1.0

I have tried following this guide https://zitadel.com/docs/guides/migrate/sources/zitadel multiple times, I have recreated my new Zitadel instance multiple times. However, the result is this (from a test org):

{
    "errors": [
        {
            "type": "domain_isverified",
            "id": "281717763746037762_example.com",
            "message": "ID=V3-DKcYh Message=Errors.Org.Domain.AlreadyExists Parent=(ERROR: duplicate key value violates unique constraint \"unique_constraints_pkey\" (SQLSTATE 23505))"
        },
        {
            "type": "login_policy",
            "id": "281717763746037762",
            "message": "ID=Org-FEd32 Message=Errors.IDPConfig.NotExisting"
        },
        {
            "type": "user_link",
            "id": "282588122661781507_282583967918456835",
            "message": "ID=COMMAND-39nf2 Message=Errors.IDPConfig.NotExisting"
        },
        {
            "type": "trigger_action",
            "id": "0_0",
            "message": "ID=COMMAND-Dfhj5 Message=Errors.Flow.FlowTypeMissing"
        }
    ],
    "success": {
        "orgs": [
            {
                "orgId": "281717763746037762",
                "projectIds": ["281717802887282690"],
                "oidcAppIds": ["281717948026978306"],
                "humanUserIds": ["282588122661781507", "307682321706778626"],
                "actionIds": ["282443599746957315"],
                "orgMembers": ["307682321706778626"],
                "domains": ["example.com"]
            }
        ]
    }
}

{
    "errors": [
        {
            "type": "domain_isverified",
            "id": "281717763746037762_example.com",
            "message": "ID=V3-DKcYh Message=Errors.Org.Domain.AlreadyExists Parent=(ERROR: duplicate key value violates unique constraint \"unique_constraints_pkey\" (SQLSTATE 23505))"
        },
        {
            "type": "login_policy",
            "id": "281717763746037762",
            "message": "ID=Org-FEd32 Message=Errors.IDPConfig.NotExisting"
        },
        {
            "type": "user_link",
            "id": "282588122661781507_282583967918456835",
            "message": "ID=COMMAND-39nf2 Message=Errors.IDPConfig.NotExisting"
        },
        {
            "type": "trigger_action",
            "id": "0_0",
            "message": "ID=COMMAND-Dfhj5 Message=Errors.Flow.FlowTypeMissing"
        }
    ],
    "success": {
        "orgs": [
            {
                "orgId": "281717763746037762",
                "projectIds": ["281717802887282690"],
                "oidcAppIds": ["281717948026978306"],
                "humanUserIds": ["282588122661781507", "307682321706778626"],
                "actionIds": ["282443599746957315"],
                "orgMembers": ["307682321706778626"],
                "domains": ["example.com"]
            }
        ]
    }
}

IDP settings did not get transferred. In addition, the users are not activated and Zitadel is telling me to resend the activation email. Is there a practical way to import data from the old Zitadel instance? My old Zitadel v2.71.6 uses Cockroach while the new one uses Postgres.

9 Replies

Ask Inkeep•4d ago

Hi @wlinna, I'm an AI assistant for ZITADEL. While a team member reviews this question, I'll see if I can find relevant sources for your question. Answer For migrating from CockroachDB to PostgreSQL, ZITADEL provides the mirror command specifically for database-to-database migrations (1) . Since you're moving from v2.71.6 with CockroachDB to a newer version with PostgreSQL, this is the recommended approach (2) . The mirror command allows you to copy data from one database to another and is designed for use cases like migrating from CockroachDB to PostgreSQL (1) . Here's the process: 1. Prepare the destination database:

zitadel init --config /path/to/your/new/config.yaml
zitadel setup --for-mirror --config /path/to/your/new/config.yaml

zitadel init --config /path/to/your/new/config.yaml
zitadel setup --for-mirror --config /path/to/your/new/config.yaml

2. Mirror the data:

zitadel mirror --system --config /path/to/your/mirror/config.yaml

zitadel mirror --system --config /path/to/your/mirror/config.yaml

3. Initialize and verify:

zitadel setup --for-mirror --config /path/to/your/new/config.yaml
zitadel mirror verify --system --config /path/to/your/mirror/config.yaml

zitadel setup --for-mirror --config /path/to/your/new/config.yaml
zitadel mirror verify --system --config /path/to/your/mirror/config.yaml

The errors you're seeing with IDP configurations and user activation issues suggest that the API-based import method you're using may not be transferring all data types completely. The mirror command is specifically designed to create an exact copy of the database, which should preserve all configurations including IDP settings . Note that ZITADEL v3 requires PostgreSQL and will not start with CockroachDB configured (2) . The mirror functionality addresses this requirement by enabling complete data migration from CockroachDB to PostgreSQL (1) . If you continue experiencing issues, you might want to click the "Ask for help" button to have a team member review your specific migration scenario. To learn more, see the sources I considered: [1] Mirror data from cockroach to postgres [2] Technical Advisory 10015 Feel free to tag @Ask Inkeep with additional questions.

wlinnaOP•3d ago

I just tested the same guide between v2.71.6 and v2.71.6 (yes, the same version), and I still have the same problems

Rajat•3d ago

hey @wlinna have you tried those mirror commands that inkeep suggested?

wlinnaOP•3d ago

Hello, I haven't had the opportunity yet, because mirroring is not as simple in my case because the databases are not exposed to the public Internet I'm still thinking what would be the best way to do that

Rajat•3d ago

the reason i say this is because theres no one-click migration from your old Cockroach to Postgres that brings everything (users, IdPs, actions, policies) perfectly. Imo, you need to export from old zitadel -> transfer files via tunneling->import to the new zitadel -> recreate IDPs -> activate users(new activation unless credentials were migrated) so soemtying like Export from source: curl --request POST ... > export.json Transfer file securely between environments(scp??) Import to target: curl --request POST ... --data '{"data_orgsv1": "$(cat export.json)"}'

wlinnaOP•3d ago

I doubt that would work. What you suggest would lead to users losing their IDP config, and lots of suspicious/scary emails to users (I have already received multiple "Initialize User" emails just by trying this export/import thing) Users would have to reconfigure their IDPs, but why would we make them suffer through that? The mirror strategy seems better (that's my hunch at least)

Rajat•3d ago

yes, using mirror would be the best possible way BUT how about this First extract IDP configuration details (including their internal IDs) from your old instance using Zitadel's admin API Before migration, create identical IDP configurations in your new instance with matching IDs Only then run the migration

wlinnaOP•3d ago

Thanks for the suggestion. I was thinking of a variation of this too. Iis it possible to even create IDP configurations with specific IDs though the UI? What I was thinking is to create identical IDPs first, and then replace all the old IDP IDs in export.json with the new ones, and then import it.

Rajat•3d ago

No, you cannot create IDP configurations with specific/custom IDs through the UI.

Multiple errors when importing data from v2.71.6 to v4.1.0

Did you find this page helpful?