Mattia•3mo ago

JWT auth fails when adding custom domain

Hi everyone 👋 I deployed a ZITADEL v3.3.0 instance to my K8s cluster using the Helm Chart and now want to configure a custom domain. From what I found, this must be done via the API (no UI anymore?) — is that correct? I tried the AddCustomDomain endpoint, using the ResourceID from /ui/console/instance?id=organizations as the instanceID, but got:

{"code": 5, "message": "Not Found"}

{"code": 5, "message": "Not Found"}

I also tested the old API with no success. For auth, I followed the System API guide, created a system user, and generated a JWT with:

zitadel-tools key2jwt \
  --audience=https://zitadel.domain.com \
  --key=system-user-1.pem \
  --issuer=system-user-1

zitadel-tools key2jwt \
  --audience=https://zitadel.domain.com \
  --key=system-user-1.pem \
  --issuer=system-user-1

Since I couldn’t mount the key in Helm, I base64-encoded the public key and added it under keyData, then ran helm upgrade. The docs mention this JWT approach is for "test" purposes, so I’m unsure if it works for production. When I tried listing instances:

curl --request POST \
  --url https://zitadel.domain.com/system/v1/instances/_search \
  --header 'Authorization: Bearer ey...' \
  --header 'Content-Type: application/json'

curl --request POST \
  --url https://zitadel.domain.com/system/v1/instances/_search \
  --header 'Authorization: Bearer ey...' \
  --header 'Content-Type: application/json'

...I received:

{
  "code": 16,
  "message": "Errors.Token.Invalid (AUTH-7fs1e)"
}

{
  "code": 16,
  "message": "Errors.Token.Invalid (AUTH-7fs1e)"
}

Am I missing something with the token, audience, or signing? Appreciate any help! 🙌

12 Replies

MattiaOP•3mo ago

Helm values:

zitadel:
  zitadel:
    configmapConfig:
      ExternalSecure: true
      ExternalDomain: zitadel.domain.com
      TLS:
        Enabled: false
      SystemAPIUsers:
        - system-user-1:
            KeyData: ABC...
            Memberships:
              - MemberType: System
                Roles:
                  - "SYSTEM_OWNER"
                  - "IAM_OWNER"
                  - "ORG_OWNER"

zitadel:
  zitadel:
    configmapConfig:
      ExternalSecure: true
      ExternalDomain: zitadel.domain.com
      TLS:
        Enabled: false
      SystemAPIUsers:
        - system-user-1:
            KeyData: ABC...
            Memberships:
              - MemberType: System
                Roles:
                  - "SYSTEM_OWNER"
                  - "IAM_OWNER"
                  - "ORG_OWNER"

Raccine•3mo ago

Hi there @Mattia! Thanks for reaching out - Let me loop in an engineer who can help you troubleshoot this issue more closely! @Matías ☺️

Matías•3mo ago

Hi @Mattia, thanks for reaching out! Let's try to troubleshoot this together: 1. Check that your System-API user is loaded: Watch the pod logs after helm upgrade – you should see a line similar to: system API key for user system-user-1 initialised succesfully. If this line is missing, the key was not picked up and every request will fail. Check that KeyData is correctly indented and line-wrapped. Put the base64-encoded key in one line, indent exactly two spaces under the user name. 2. Verify you have a valid self-signed JWT: The audience must match exactly what ZITADEL resolves to now (scheme + host + optional port). With an ingress on https://zitadel.domain.com:

zitadel-tools key2jwt \
    --key system-user-1.pem \
    --issuer system-user-1 \
    --audience https://zitadel.domain.com

zitadel-tools key2jwt \
    --key system-user-1.pem \
    --issuer system-user-1 \
    --audience https://zitadel.domain.com

Required claims are: iss, sub, aud, iat, exp. 60 min TTL is typical. The error you received is returned when any of those claims are wrong or the key does not match what ZITADEL has in memory. - make sure aud is exactly https://zitadel.domain.com (keep the s in https) - verify that the public key, not the private key, is in KeyData - confirm server time and pod time are correct (JWT iat/exp window) 3. Adding the custom domain: You are right, this needs to be done through the API. There is currently no UI option for self-hosted instances. I would recommend using the deprecated v1 System API. The resource based (v2) API is still in the beta phase. 4. {"code":5,"message":"Not Found"} error when calling AddCustomDomain: To decide which instance should handle an incoming request it looks first at the HTTP/1 Host header. If the header is unknown, the routing layer bails out immediately with gRPC status NOT_FOUND (code 5). Pleaes make sure that you use the existing working domain (zitadel.domain.com) as the Host header and include the correct instanceId in the path parameter. I look forward to hearing back from you!

MattiaOP•2mo ago

Hi @Matías , thank you for your help! It seems like my key isn’t being loaded — I don’t see the log line system API key for user ... in any of the three running pods, nor in the zitadel-setup or zitadel-init jobs. Here’s the relevant part of my Helm values file:

zitadel:
  image:
    tag: "v3.3.0"
  zitadel:
    configmapConfig:
      SystemAPIUsers:
        - system-user-1:
            # Base64-encoded public key
            KeyData: <base-64-encoded-public-cert>

zitadel:
  image:
    tag: "v3.3.0"
  zitadel:
    configmapConfig:
      SystemAPIUsers:
        - system-user-1:
            # Base64-encoded public key
            KeyData: <base-64-encoded-public-cert>

I initially deployed the instance a few weeks ago and updated it recently with:

helm upgrade --install zitadel . -n zitadel --create-namespace

helm upgrade --install zitadel . -n zitadel --create-namespace

The resulting config map looks correct (snippet below):

data:
  zitadel-config-yaml: |-
    ExternalDomain: zitadel.domain.com
    ExternalSecure: true
    Machine:
      Identification:
        Hostname:
          Enabled: true
        Webhook:
          Enabled: false
    SystemAPIUsers:
    - system-user-1:
        KeyData: <base-64-encoded-public-cert>
    TLS:
      Enabled: false

data:
  zitadel-config-yaml: |-
    ExternalDomain: zitadel.domain.com
    ExternalSecure: true
    Machine:
      Identification:
        Hostname:
          Enabled: true
        Webhook:
          Enabled: false
    SystemAPIUsers:
    - system-user-1:
        KeyData: <base-64-encoded-public-cert>
    TLS:
      Enabled: false

To be safe, I also scaled the deployment down to 0 and back up to 3 to ensure all pods use the updated config map — but still no sign of the key being picked up in logs. Is there anything else I might be missing to get the system API key loaded?

Matías•2mo ago

Hey @Mattia , just a heads up, I'm taking a look at this with my team and I will get back to you shortly with a reply and/or next steps to troubleshoot.

Perrotti•2mo ago

@Matías same error here... no log entry for key being loaded, and I even tried setting the env var on the deployment itself, but no luck also, when I run the system api call, I get an error saying that issuer should be my https address, not system-user-1

Matías•2mo ago

Hey @Perrotti, could you share the exact error message? And helm values as well. I've bumped the issue internally to try and get a faster reply.

Perrotti•2mo ago

sure! well, I'm doing things like this:

zitadel:
  zitadel:
    masterkey: "123123123123"
    masterkeySecretName: ""
    configmapConfig:
      ExternalSecure: true
      ExternalDomain: iam.domain.com
      TLS:
        Enabled: false
      Database:
        Postgres:
          Host: rds.fqdn.sa-east-1.rds.amazonaws.com
          Port: 5432
          Database: zitadel
          MaxOpenConns: 20
          MaxIdleConns: 10
          MaxConnLifetime: 30m
          MaxConnIdleTime: 5m
          User:
            Username: zitadel_user
            SSL:
              Mode: disable
          Admin:
            Username: admin
            SSL:
              Mode: disable
      SystemAPIUsers:
      - https://iam.domain.com:
          # Base64-encoded public key
          KeyData: base64encodedkey

zitadel:
  zitadel:
    masterkey: "123123123123"
    masterkeySecretName: ""
    configmapConfig:
      ExternalSecure: true
      ExternalDomain: iam.domain.com
      TLS:
        Enabled: false
      Database:
        Postgres:
          Host: rds.fqdn.sa-east-1.rds.amazonaws.com
          Port: 5432
          Database: zitadel
          MaxOpenConns: 20
          MaxIdleConns: 10
          MaxConnLifetime: 30m
          MaxConnIdleTime: 5m
          User:
            Username: zitadel_user
            SSL:
              Mode: disable
          Admin:
            Username: admin
            SSL:
              Mode: disable
      SystemAPIUsers:
      - https://iam.domain.com:
          # Base64-encoded public key
          KeyData: base64encodedkey

I set the user name to my fqdn as per the last bugs regarding this. then, I created a JWT with zitadel-tools key2jwt --audience=https://iam.domain.com --key=system-user-1.pem --issuer=https://iam.domain.com, but I get { "code": 16, "message": "Errors.Token.Invalid (AUTH-7fs1e)", "details": [ { "@type": "type.googleapis.com/zitadel.v1.ErrorDetail", "id": "AUTH-7fs1e", "message": "Errors.Token.Invalid" } ] } and, on zitadel's logs time="2025-07-16T19:44:40Z" level=warning msg="token verifier repo: verify JWT access token" caller="/home/runner/work/zitadel/zitadel/internal/authz/repository/eventsourcing/eventstore/token_verifier.go:287" error="invalid signature (invalid signature: no possible keys matches)" I've tried to indent the SystemAPIUsers several times in multiple ways, setting an env var and the content as a json string, but I'm always getting this invalid signature now and I don't see any entries on the log for loaded key or something like that is there any private key that's already on a standard zitadel install that I can use to sign this? also, I'm running v3.3.2 @Mattia not sure if you've got this fixed, but please that a look at https://discord.com/channels/927474939156643850/1277030942606884904. And @Matías I don't think there's this system API key log message at all... at least not on the vanilla code 🙂 make sure to define the port on the jwt sub, aud and iss fields to be the same as it's on the top of your pod's log (the https://zitadel.domain.com:port-number/ui/console line and the one below that). and you'll always get the usual JWT check errors, the System User JWT Check runs in parallel and it doesn't output anything. no errors, no party hats, nothing.

Matías•2mo ago

And @Matías I don't think there's this system API key log message at all... at least not on the vanilla code

Yes, you got that right, sorry about the confusion there! I got things mixed up 🙇‍♂️ I just received a note by one of our backend Engineers. I just saw that other thread that you tagged, thanks for sharing your experience and the steps you followed there, that's immensely helpful! :gigilove: @Mattia could you please take a look at the above and let me know if that helps? 🙏

MattiaOP•2mo ago

@Matías @Perrotti you were absolutely right. I had to include port 8080 in the aud when generating the token:

zitadel-tools key2jwt \
  --key system-user-1.pem \
  --issuer system-user-1 \
  --audience https://zitadel.domain.com:8080

zitadel-tools key2jwt \
  --key system-user-1.pem \
  --issuer system-user-1 \
  --audience https://zitadel.domain.com:8080

After generating the token with the correct audience, I was able to successfully list the instances:

curl --request POST \
  --url https://zitadel.domain.com/system/v1/instances/_search \
  --header 'Authorization: Bearer ey123' \
  --header 'Content-Type: application/json'

curl --request POST \
  --url https://zitadel.domain.com/system/v1/instances/_search \
  --header 'Authorization: Bearer ey123' \
  --header 'Content-Type: application/json'

Then I used the instance ID in the path (as described here https://zitadel.com/docs/apis/resources/system/system-service-add-domain) to add the custom domain. Thanks again for the support 🙏 It would be awesome if this detail about the port in the audience were mentioned in the docs! 🙂 (Sorry for the late response, I was busy the last few weeks.)

Matías•2mo ago

@Mattia thanks for letting me know! I'm glad to hear this is working for you now. I've added a task in my backlog to update the documentation to reflect this 🙇‍♂️

Gigi the Giraffe (Zitadel)•2mo ago

🎉 Looks like you just helped out another community member! Thanks for being so helpful <@463821425220911104>! You're now one step closer to leveling up—keep up the amazing peer support! 🚀

JWT auth fails when adding custom domain

Did you find this page helpful?