Login V2 not working with virtual instance?
Hi, I'm trying to test out the virtual instance functionality, but I'm running into a problem when trying to sign in to the non-default instance.
I'm running the standard docker compose from the docs (https://raw.githubusercontent.com/zitadel/zitadel/main/docs/docs/self-hosting/deploy/docker-compose.yaml), except for the fact that I added a system api user via the
ZITADEL_SYSTEMAPIUSERS env var.
The default instance works fine. I can go to localhost:8080 and it'll redirect me to :3000 for the login v2 ui, which allows me to login just fine.
I then decided to create a virtual instance via /system/v1/instances/_create to test the virtual instance system. I gave it the custom domain test.localhost. The instance creation succeeds, but when I now go to test.localhost:8080 it redirects me to localhost:3000, which shows a blank http500 page. Even if I try to correct the url to test.localhost:3000, it still errors out.
I looked in the docker logs and I see the following:
If I had to guess what's happening here I'd say the zitadel instance is looking up the auth request for the wrong virtual instance and is failing because it won't exist for the default instance. That said, I'm not knowledgeable enough about how zitadel works to know how to properly debug or fix this.
If anyone knows how to solve this issue or how to help me debug, help would be much appreciated.
Thanks!9 Replies
Hi @Noah, I'm an AI assistant for ZITADEL. While a team member reviews this question, I'll see if I can find relevant sources for your question.
Answer
I wasn't able to find a direct answer, here are some relevant sources:
1. External ZITADEL Access : This explains how ZITADEL uses the Host header to determine which virtual instance to use, and covers configuration for ExternalDomain, ExternalPort, and ExternalSecure settings.
2. Issue with authorization request details : A similar issue where someone experienced authorization request problems in a KinD cluster setup, with the error suggesting protocol or domain configuration issues.
If you'd like me to click the "Ask for help" button to tag a team member for assistance with this virtual instance configuration issue, let me know!
Feel free to tag @Ask Inkeep with additional questions.
Got it. Tagging @Rajat for additional help.
For what it's worth: this also happens on a kubernetes deployment where the login ui v2 lives at
/ui/v2/loginhello @Noah thanks for reaching out! I will review this with my team and get back to you shortly
hi, do you have an update?
hello @Noah apologies for the delay, we are updating our docs to better describe this process, it's still under review but you can see it here:
- https://github.com/zitadel/zitadel/pull/10698
GitHub
docs(login): add Virtual Instances Documentation for Login App by p...
This PR adds comprehensive documentation explaining how the ZITADEL login application implements virtual instances support for multi-tenant deployments.
Which Problems Are Solved
VIRTUAL_INSTANCE...
Hi @fcoppede, I’m experiencing the exact same issue that Noah described.
I’ve spent the past few days trying to resolve it both locally (on Rancher Desktop) and on Google GKE, but without success. The main (default) instance works fine, but I can’t log into the ZITADEL Console for any newly created Virtual Instance.
I’m currently using ZITADEL v4.2.2 and the official Helm Chart v9.5.0.
Local setup: Rancher Desktop with the built-in Traefik as the ingress controller (for both zitadel and zitadel-login).
Cloud setup: GKE with the Google ingress controller.
In both environments, the behavior is the same.
Virtual instance setup:
I followed this guide for enabling virtual instance support in zitadel-login:
https://github.com/zitadel/zitadel/pull/10698
I added the following environment variables to zitadel-login:
- name: AUDIENCE
value: https://auth.example.com
- name: SYSTEM_USER_ID
value: system-user-1
- name: SYSTEM_USER_PRIVATE_KEY
value: LS0tLS1CRUdJTiB….
Initially, I ran into an error when using my private key as mentioned in the guide:
TypeError: "pkcs8" must be PKCS#8 formatted string
To fix this, I converted the key to PKCS#8:
openssl pkcs8 -topk8 -inform PEM -outform PEM -nocrypt \
-in system-user-1.pem \
-out system-user-1_pkcs8.pem
Then I base64-encoded it:
base64 < system-user-1_pkcs8.pem | tr -d '\n' > system-user-1_pkcs8.pem.b64
I used this value for SYSTEM_USER_PRIVATE_KEY in values.yaml, and it worked as expected.
Permissions issue:
For my system user system-user-1, I initially set:
- MemberType: System
Roles:
- "SYSTEM_OWNER"
After logging in at https://auth.example.com/ui/v2/login
, I got the following browser error:
{"error":{"name":"ConnectError","rawMessage":"No matching permissions found (AUTH-AWfge)","code":7,...}}
The zitadel-login logs showed:
Error [ConnectError]: [permission_denied] No matching permissions found (AUTH-5mWD2)
The only way I could fix this (for the main instance only) was to also assign the IAM_LOGIN_CLIENT role:
- MemberType: System
Roles:
- "SYSTEM_OWNER"
- "IAM_LOGIN_CLIENT"
This made login and the console work again (still for first instance only)
This suggests the system user and its key pair are valid, but it’s unclear why IAM_LOGIN_CLIENT is required — it’s not mentioned in the docs. Could you clarify this?
GitHub
docs(login): add Virtual Instances Documentation for Login App by p...
This PR adds comprehensive documentation explaining how the ZITADEL login application implements virtual instances support for multi-tenant deployments.
Which Problems Are Solved
VIRTUAL_INSTANCE...
Token behavior:
I noticed that ZITADEL_SERVICE_USER_TOKEN_FILE only works if I comment out the environment variables that enable virtual instance support:
- AUDIENCE
- SYSTEM_USER_PRIVATE_KEY
- SYSTEM_USER_ID
As soon as these are set, it looks like ZITADEL_SERVICE_USER_TOKEN (or the *_FILE variant) is completely ignored.
If this is expected behavior, it’s not documented anywhere.
New instance creation:
Here’s how I’m creating new instances:
Generate a JWT:
zitadel-tools key2jwt \
--audience=https://auth.example.com \
--key=system-user-1.pem \
--issuer=system-user-1
Call the System API:
POST https://auth.example.com/system/v1/instances/_create
The API call succeeds and returns the new instance ID.
However, after the first instance is created, trying to access the console at
https://vendor01.auth.example.com/ui/console/
shows a blank screen, and zitadel-login logs:
Error [ConnectError]: [not_found] Auth Request does not exist (QUERY-Thee9)
I can open https://vendor01.auth.example.com/ui/v2/login/loginname
, and the login form loads, but trying to log in with the user created during instance creation results in “user not found”.
In the database, I see that the newly created instance is missing a login-client user.
I could manually create one, but that would mean having an extra login client per instance — and since ZITADEL_SERVICE_USER_TOKEN is ignored when SYSTEM_USER_ID and SYSTEM_USER_PRIVATE_KEY are set, it feels like a dead end anyway.
Any guidance would be greatly appreciated.
Thanks in advance for your help!
Ok, I've found answer to my first question about using IAM_LOGIN_CLIENT
https://github.com/zitadel/zitadel-charts/issues/379
But after spending more time trying to switch from Google Ingress to Nginx ingress with all suggested config options (found in official docs)
this still doesn't work, no matter what I do I'm getting: Error [ConnectError]: [not_found] Auth Request does not exist (QUERY-Thee9)
It seems like I'm not the only one who struggles with getting it working,
and seemingly ZITADEL staff is not paying much attention to help here...
Thanks anyway (said to myself...).
GitHub
Multi instance setup with LoginV2 · Issue #379 · zitadel/zitadel-...
Preflight Checklist I could not find a solution in the existing issues, docs, nor discussions I have joined the ZITADEL chat Describe your problem I tested the updated Helm Chart with LoginV2 but i...