ppenguin•2w ago

First test-deployment using nomad: cannot login (using loginv2)

I made an initial deployment of Zitadel in a nomad cluster. After first start I saved the 2 .pat file's contents to vault, and render them back to ZITADEL_SERVICE_USER_TOKEN_FILE=/local/login-client.vault.pat for both the login and main container. When I try to login, I get in the main container:

2025-08-27T16:32:07.518788054+02:00 stderr F time="2025-08-27T14:32:07Z" level=info msg="server is listening on [::]:8080" caller="/home/runner/work/zitadel/zitadel/cmd/start/start.go:677"
\,2025-08-27T16:32:54.628370448+02:00 stderr F time="2025-08-27T14:32:54Z" level=warning msg="token verifier repo: decrypt access token" caller="/home/runner/work/zitadel/zitadel/internal/authz/repository/eventsourcing/eventstore/token_verifier.go:282" error="ID=APP-8EF0zZ Message=invalid token"
2025-08-27T16:32:54.628370448+02:00 stderr F time="2025-08-27T14:32:54Z" level=warning msg="token verifier repo: verify JWT access token" caller="/home/runner/work/zitadel/zitadel/internal/authz/repository/eventsourcing/eventstore/token_verifier.go:286" error="parsing of request failed: token contains an invalid number of segments"
2025-08-27T16:32:54.628441579+02:00 stderr F time="2025-08-27T14:32:54Z" level=warning msg="authz: verify access token" caller="/home/runner/work/zitadel/zitadel/internal/api/authz/context.go:114" error="ID=APP-Reb32 Message=invalid token" org_domain= org_id=

2025-08-27T16:32:07.518788054+02:00 stderr F time="2025-08-27T14:32:07Z" level=info msg="server is listening on [::]:8080" caller="/home/runner/work/zitadel/zitadel/cmd/start/start.go:677"
\,2025-08-27T16:32:54.628370448+02:00 stderr F time="2025-08-27T14:32:54Z" level=warning msg="token verifier repo: decrypt access token" caller="/home/runner/work/zitadel/zitadel/internal/authz/repository/eventsourcing/eventstore/token_verifier.go:282" error="ID=APP-8EF0zZ Message=invalid token"
2025-08-27T16:32:54.628370448+02:00 stderr F time="2025-08-27T14:32:54Z" level=warning msg="token verifier repo: verify JWT access token" caller="/home/runner/work/zitadel/zitadel/internal/authz/repository/eventsourcing/eventstore/token_verifier.go:286" error="parsing of request failed: token contains an invalid number of segments"
2025-08-27T16:32:54.628441579+02:00 stderr F time="2025-08-27T14:32:54Z" level=warning msg="authz: verify access token" caller="/home/runner/work/zitadel/zitadel/internal/api/authz/context.go:114" error="ID=APP-Reb32 Message=invalid token" org_domain= org_id=

and in the login container:

Error [ConnectError]: [unauthenticated] Errors.Token.Invalid (AUTH-7fs1e)

Error [ConnectError]: [unauthenticated] Errors.Token.Invalid (AUTH-7fs1e)

Any ideas what to check next?

7 Replies

Ask Inkeep•2w ago

Hi @ppenguin, I'm an AI assistant for ZITADEL. While a team member reviews this question, I'll see if I can find relevant sources for your question. Answer I wasn't able to find a direct answer to your specific Nomad deployment issue with PAT files and token validation errors, here are some relevant sources: 1. Configure personal access token authentication for service users : This covers how PATs work in ZITADEL and mentions that PATs are ready-to-use tokens that can be added as Authorization headers. 2. Authenticate service users and client applications : This explains the different authentication methods for service users, including PATs, and notes that PATs have limitations around centralized validation and potential leakage risks. The token parsing errors you're seeing ("token contains an invalid number of segments") suggest the PAT content may have been corrupted during the vault storage/retrieval process. You might want to verify the PAT content integrity and check if there are any encoding issues when transferring between vault and the containers. Would you like me to provide general guidance on troubleshooting token authentication issues, or would you prefer to click the "Ask for help" button to have a team member review your specific Nomad configuration? Feel free to tag @Ask Inkeep with additional questions. Got it. Tagging @Rajat for additional help.

ppenguinOP•2w ago

I verified that the .pat contents was the same as the one generated with first instance. So it's no "corruption" issue. If I do a clean-slate deploy and with the LOGINV2 URLs all with IP:port of the login service (which is the exposed address, i.e. the LAN address of the host running the containers), I get a login prompt but when I enter the admin user (from ZITADEL_FIRSTINSTANCE_ORG_HUMAN_USERNAME) and verified I by checking the database, I get Could not get the context of the user. and the corresponding password is not accepted. Edit: the IP was a red herring, I got to the same point now with the local (lan) host name (fyi !=localhost). But still not able to login with the first ORG_HUMAN user.

Rajat•2w ago

hey @ppenguin welcome to the server and thanks for your question, looking into it

ppenguinOP•2w ago

Just for clarity: the last status (with login impossible) does not have the Token.Invalid issue anymore; the only thing I see in zitadel-login logs is

2025-08-28T13:19:08.315907586+02:00 stderr F No session lifetime provided, using default of 24 hours.
2025-08-28T13:19:08.687948492+02:00 stderr F Could not read session cookie
2025-08-28T13:19:44.454558419+02:00 stderr F Ignored error: no session cookie found

2025-08-28T13:19:08.315907586+02:00 stderr F No session lifetime provided, using default of 24 hours.
2025-08-28T13:19:08.687948492+02:00 stderr F Could not read session cookie
2025-08-28T13:19:44.454558419+02:00 stderr F Ignored error: no session cookie found

@Rajat interesting: running zitadel:v3.4.0 container with zitadel-login:v4.0.3 with the same config settings works! (Or at least I can login, haven't tested advanced management yet). Curios what (regression?) might be the reason?

Rajat•2w ago

I can check with my team, but keep using v4 as we'd soon update our cloud offering to v4 too, I will check with my team as to what could be the reason(maybe a bug that got fixed in later releases)

ppenguinOP•2w ago

Thanks! Just to make clear: the working version is v3.4.0, the non-working is 4.x. So my goal is to make 4.x work. (I'm only using zitadel-login v4 for both cases, since there is no loginv2 for 3.x) Additional (related) issue: Even though I set the env vars to use loginv2 it was porbably just disregarded in the main v3.4.0 container? So after I got reverse proxy working and started using it, I noticed that I had no MFA login options per default. It worked by sending an invite to register MFA to a user, but now trying to get the same user to login from another device resulted in "no passkey available". Then I tried ticking the box at "Use new login interface" as admin user, but that seems to now have locked me out completely. (Could've thought of that before🤦‍♂️ ). I think I'll have to untick that box directly in the database? Which table? EDIT: I found how to revert (but still want loginv2 on the long term): UPDATE projections.apps7_oidc_configs SET login_version=1 WHERE login_version=2;

Rajat•4d ago

hey @ppenguin I read the thread, what I understood from this is that right now, still, the v4.x still doesnt work for you BUT v3.4.0 works AND earlier token/PAT errors are gone, yes?

First test-deployment using nomad: cannot login (using loginv2)

Did you find this page helpful?