Zitadel in Kubernetes and access from within the cluster

Achoo! Kraken · 2025-10-28T13:22:16.213Z

Hi! I'm trying to set up Zitadel running in Kubernetes and having trouble with domain-based instance identification. I can access the admin UI and make API calls using the **external domain**; all good. Now, I also want to make calls to Zitadel **from within** the cluster (using the `...svc.cluster.local` Service endpoint) and... I'm stuck. Trying to access the internal endpoint from within a temporary pod: ```# curl http://zitadel.zitadel.svc.cluster.local:8080/.well-known/openid-configuration unable to set instance using origin &{zitadel.zitadel.svc.cluster.local:8080 https} (ExternalDomain is mydomain.com): ID=QUERY-1kIjX Message=Instance not found. Make sure you got the domain right.``` ...which is fair enough, but how do I fix that? ( also, it says 'https' there, but I'm using http as it's internal comms ) - I read all the docs, helm chart specs, config specs, github tickets and tutorials I could find - `/ui/console/instance?id=domain` doesn't offer adding new domains - adding extra domains to the Organisation doesn't change anything ( I tried all variants, with and without the port; they go straight into *Verified domains* as I have verification disabled ) - Helm `zitadel.selfSignedCert.additionalDnsName` seems to be for the cert only This is self-hosted K8S, latest Zitadel version deployed using the official Helm chart from charts.zitadel.com.

A

Ask Inkeep•10/28/25, 1:22 PM

Hi @Achoo! Kraken, I'm an AI assistant for ZITADEL. While a team member reviews this question, I'll see if I can find relevant sources for your question.

A

Ask Inkeep•10/28/25, 1:22 PM

Answer

You need to add

zitadel.zitadel.svc.cluster.local

zitadel.zitadel.svc.cluster.local

zitadel.zitadel.svc.cluster.local

zitadel.zitadel.svc.cluster.local as an additional instance domain using the System API (1) (2) . The Helm chart's

ExternalDomain

ExternalDomain

ExternalDomain

ExternalDomain setting only defines the primary domain (3) .

Organization domains won't work for this—they're intended for routing users by login methods to their correct organization, not for instance-level access (1) .

To add the internal domain, use the System API's

AddDomain

AddDomain

AddDomain

AddDomain endpoint after ZITADEL is running (2) . The Helm chart cannot declare multiple instance domains during deployment (3) .

Your Kubernetes Ingress should route both your external domain (

mydomain.com

mydomain.com

) and the internal service domain (

zitadel.zitadel.svc.cluster.local

zitadel.zitadel.svc.cluster.local

zitadel.zitadel.svc.cluster.local

zitadel.zitadel.svc.cluster.local) to the ZITADEL service . ZITADEL uses the

Host

Host

Host

Host header to identify the correct instance .

Note that if you change the

ExternalDomain

ExternalDomain

ExternalDomain

ExternalDomain configuration later, you need to rerun ZITADEL's setup phase for the changes to take effect (1) .

A

Ask Inkeep•10/28/25, 1:22 PM

To learn more, see the sources I considered:

[1] External ZITADEL Access
[2] Trouble accessing the service using different hostnames.
[3] Set multiple instance domains via helm chart

A

Ask Inkeep•10/28/25, 1:22 PM

Feel free to tag @Ask Inkeep with additional questions.

B

bawsky•10/28/25, 2:10 PM

Since Zitadel can run multiple (virtual) instances, you should specify what is the specify you want if it doesn't match the domain you're using. You can do that using the

Host

Host

Host

Host request header.

E.g.:

curl http://zitadel.zitadel.svc.cluster.local:8080/.well-known/openid-configuration \
  -H 'Host: zitadel.my-cool-domain.com'

curl http://zitadel.zitadel.svc.cluster.local:8080/.well-known/openid-configuration \
  -H 'Host: zitadel.my-cool-domain.com'

curl http://zitadel.zitadel.svc.cluster.local:8080/.well-known/openid-configuration \
  -H 'Host: zitadel.my-cool-domain.com'

curl http://zitadel.zitadel.svc.cluster.local:8080/.well-known/openid-configuration \
  -H 'Host: zitadel.my-cool-domain.com'

(assuming your Zitadel server run on
zitadel.my-cool-domain.com
zitadel.my-cool-domain.com
zitadel.my-cool-domain.com
zitadel.my-cool-domain.com)

A

Achoo! KrakenOP•10/28/25, 2:10 PM

@Ask Inkeep Ok, I'm trying to use the InstanceApi to list instances as the first step. I'm calling: zitadelClient.betaInstances.listInstances( ... ) with the "Automatically Initialized IAM Admin" user (that's an "Iam Owner' permission level) and getting:

body: '{"code":"permission_denied","message":"No matching permissions found

From what I see it should be authorised to manage instances. What other permission do I need? Or is the error a red herring?

A

Ask Inkeep•10/28/25, 2:10 PM

Thanks for the follow-up, @Achoo! Kraken. I'll get back to you soon.

A

Ask Inkeep•10/28/25, 2:10 PM

Answer

I wasn't able to find a direct answer, here are some relevant sources:

[Instance resource API extended permission checks][(5) ]: This GitHub issue discusses permission checks on instance endpoints and mentions that instance admins (e.g., IAM_OWNER) should be permitted to call instance service endpoints with their own instance ID, but when calling with any other instance ID, a permission denied error is returned (1) .
[[permission_denied] No matching permissions found (AUTH-5mWD2) ][(6) ]: This Discord thread mentions that users need
```
IAM_OWNER
```
IAM_OWNER
```
IAM_OWNER
```
IAM_OWNER
or
```
IAM_LOGIN_CLIENT
```
IAM_LOGIN_CLIENT
```
IAM_LOGIN_CLIENT
```
IAM_LOGIN_CLIENT
roles, and the solution involved adding these roles in the default admin account's membership section questions-help-bugs[permission_denied] No matching permissions found (AUTH-5mWD2) .

A

Ask Inkeep•10/28/25, 2:10 PM

To learn more, see the sources I considered:

[1] Instance resource API extended permission checks
questions-help-bugs[permission_denied] No matching permissions found (AUTH-5mWD2)

A

Ask Inkeep•10/28/25, 2:10 PM

Feel free to tag @Ask Inkeep with additional questions.

B

bawsky•10/28/25, 2:15 PM

I've stumbled upon this myself but can't remember where I found the

Host:

Host:

Host:

Host: header tip... Hope it's documented somewhere but tbh I think I found it on a GitHub issue

Btw, see here to learn more about virtual instances.

(I never had a use for that myself.. we run 3 separate Zitadel deploys in our infrastructure, but they're all isolated a have only one "virtual instance" (i.e. the default instance) each. I think it's less confusing that way)

ZITADEL Docs

Instance Structure

Bbawsky I've stumbled upon this myself but can't remember where I found the `Host:` head...

A

Achoo! KrakenOP•10/28/25, 2:17 PM

That's very good to know, thank you, but also... I'd like to use Zitadel as the auth provider to other cluster services, and they are configurable with the auth + .well-known discoverability endpoint only

Can't use the Host: trick with them...

B

bawsky•10/28/25, 2:31 PM

Ah right... Tbh I don't think you can do anything if you can't edit the request headers on those other services.

I suppose one thing you can try is to add a new custom domain to your instance... I've never tried that, but it may work if you simply add

zitadel.zitadel.svc.cluster.local

zitadel.zitadel.svc.cluster.local

zitadel.zitadel.svc.cluster.local

zitadel.zitadel.svc.cluster.local
v1 api: https://zitadel.com/docs/apis/resources/system/system-service-add-domain
v2 api: https://zitadel.com/docs/apis/resources/instance_service_v2/zitadel-instance-v-2-instance-service-add-custom-domain

(not sure how to do that from the UI)

A

Achoo! KrakenOP•10/28/25, 2:32 PM

Yes, I'm fiddling with the API and it seems there is a permission missing for certain ops:
// client.betaInstances.getInstance -> allowed
// client.betaInstances.addCustomDomain -> DENIED
// client.betaInstances.listInstances -> DENIED

A

Achoo! KrakenOP•10/28/25, 2:34 PM

I'll try the V1 API next, maybe it's the 'beta' part...

AAsk Inkeep To learn more, see the sources I considered: [[1] Instance resource API exten...

A

Ask InkeepAPP•10/28/25, 3:13 PM

Got it. Tagging @Rajat for additional help.

A

Achoo! KrakenOP•10/28/25, 3:17 PM

https://github.com/zitadel/zitadel/issues/9452 (closed May 21)
says that "List Instances" and "Add Custom Domain" were migrated to V2 API, and explicitly marks V1 equivalents as deprecated; therefore I assume the V2 endpoints should work.

I don't want to develop my solution using a deprecated, soon-to-be-removed API, and would appreciate help with figuring out how to authorise my serviceaccount to use those API calls.

GitHub

Add instance requests to resource API · Issue #9452 · zitadel/zit...

As a developer I want to be able to perform all requests relevant to instance, using the new resource API. Acceptance Criteria All needed requests are implemented All requests can be called by an a...

B

bawsky•10/28/25, 4:03 PM

Looking at the documentation for that v2 endpoint, it seems you need

system.domain.write

system.domain.write

system.domain.write

system.domain.write permission, which by default (according to here) is only included in the

SYSTEM_OWNER

SYSTEM_OWNER

SYSTEM_OWNER

SYSTEM_OWNER role

So I suppose you just have to make sure your user has that role

A

Achoo! KrakenOP•10/28/25, 4:05 PM

OK, I found the problem.

https://zitadel.com/docs/guides/manage/console/managers says
IAM Managers: This is the highest level. Users with IAM Manager roles are able to manage the whole Instance.

When set up via Helm chart, this is NOT TRUE that it can manage the whole instance.

https://zitadel.com/docs/apis/resources/instance_service_v2/zitadel-instance-v-2-instance-service-add-custom-domain
lists Required permissions:
system.domain.write

and it appears that this permission is ONLY granted to Role: "SYSTEM_OWNER"

Therefore Role: "IAM_OWNER" IS NOT the highest level.

ZITADEL Docs

To configure managers in ZITADEL go to the resource where you like to add it (e.g Instance, Organization, Project, GrantedProject).

ZITADEL Docs

Add Custom Domain

Bbawsky Looking at the documentation for that v2 endpoint, it seems you need `system.dom...

A

Achoo! KrakenOP•10/28/25, 4:07 PM

Haha, splendid

I'm glad we both came to the same conclusion.

Now I need to figure out how to grant the SYSTEM_OWNER role to my user, ideally with the Helm chart. Otherwise I'll have to hack IAM_OWNER permissions in the chart values, but that's ewww..

A

Achoo! KrakenOP•10/28/25, 4:11 PM

There's the SystemAPIUsers section, looks promising...

# - superuser2:
#     # If no memberships are specified, the user has a membership of type System with the role "SYSTEM_OWNER"
#     KeyData: <base64 encoded key>     # or you can directly embed it as base64 encoded value

# - superuser2:
#     # If no memberships are specified, the user has a membership of type System with the role "SYSTEM_OWNER"
#     KeyData: <base64 encoded key>     # or you can directly embed it as base64 encoded value

# - superuser2:
#     # If no memberships are specified, the user has a membership of type System with the role "SYSTEM_OWNER"
#     KeyData: <base64 encoded key>     # or you can directly embed it as base64 encoded value

# - superuser2:
#     # If no memberships are specified, the user has a membership of type System with the role "SYSTEM_OWNER"
#     KeyData: <base64 encoded key>     # or you can directly embed it as base64 encoded value

B

bawsky•10/28/25, 4:12 PM

Hm, granting those "system" permissions seems tricky

I'd assume the

/admin/v1/members/:userId

/admin/v1/members/:userId

/admin/v1/members/:userId

/admin/v1/members/:userId endpoint would work for that but seems like it doesn't...

AAchoo! Kraken There's the SystemAPIUsers section, looks promising... ``` # - superuser2: # ...

B

bawsky•10/28/25, 4:13 PM

Oh nice! That indeed looks promising!
Please let me know if that works!

B

bawsky•10/28/25, 4:15 PM

Just found this documentation for this exact subject:
https://zitadel.com/docs/guides/integrate/zitadel-apis/access-zitadel-system-api

B

bawsky•10/28/25, 4:23 PM

It seems that a few particular manual steps are required to access that API... Unfortunately though I don't think the helm chart has builtin support for this, so you'll need to build your own solution (generating the RSA key pair/pointing the Zitadel configuration to it/generate a signed JWT using the RSA secret)

F

FriendlyForeignAgent•10/28/25, 6:06 PM

I had the same issue last week!
Somewhere in the docs its says that the auth used for this user and how its registered is incompatible to other account types.

Currently (in development with tilt) we have a seed script runner that configures zitadel and its k8s environment. One part of that is to generate local RSA certs, generate a k8s secret with a format that zitadel expects (secretConfigRef in helm i think), add the pub key to that secret:

const secretYaml = `
apiVersion: v1
kind: Secret
metadata:
  name: zitadel-system-api-config
  namespace: default
type: Opaque
stringData:
  config.yaml: |
    SystemAPIUsers:
      - ${userId}:
          KeyData: ${publicKeyBase64}
          Memberships:
            - MemberType: System
              Roles:
                - "SYSTEM_OWNER"
                - "IAM_OWNER"
                - "ORG_OWNER"
`;

const secretYaml = `
apiVersion: v1
kind: Secret
metadata:
  name: zitadel-system-api-config
  namespace: default
type: Opaque
stringData:
  config.yaml: |
    SystemAPIUsers:
      - ${userId}:
          KeyData: ${publicKeyBase64}
          Memberships:
            - MemberType: System
              Roles:
                - "SYSTEM_OWNER"
                - "IAM_OWNER"
                - "ORG_OWNER"
`;

const secretYaml = `
apiVersion: v1
kind: Secret
metadata:
  name: zitadel-system-api-config
  namespace: default
type: Opaque
stringData:
  config.yaml: |
    SystemAPIUsers:
      - ${userId}:
          KeyData: ${publicKeyBase64}
          Memberships:
            - MemberType: System
              Roles:
                - "SYSTEM_OWNER"
                - "IAM_OWNER"
                - "ORG_OWNER"
`;

const secretYaml = `
apiVersion: v1
kind: Secret
metadata:
  name: zitadel-system-api-config
  namespace: default
type: Opaque
stringData:
  config.yaml: |
    SystemAPIUsers:
      - ${userId}:
          KeyData: ${publicKeyBase64}
          Memberships:
            - MemberType: System
              Roles:
                - "SYSTEM_OWNER"
                - "IAM_OWNER"
                - "ORG_OWNER"
`;

We have a k8s job that merges some of these zitadel secrets in cluster, but this will work standalone too.

Then we make a client (with zitadel-node package, though principle is the same)

See below.

And ignore the crappy code, my goal was to make it functional

F

FriendlyForeignAgent•10/28/25, 6:07 PM

/**
 * ZitadelSuperadmin - System API client with superadmin privileges
 * Uses self-signed JWTs for authentication (NOT OAuth)
 * Requires SystemAPIUsers configuration in ZITADEL runtime config
 */
export class ZitadelSuperadmin {
  private privateKey: CryptoKey | null = null;
  private userId: string;
  private baseUrl: string;
  private cachedToken: string | null = null;
  private tokenExpiry: number = 0;

  constructor(userId: string, baseUrl: string) {
    this.userId = userId;
    this.baseUrl = baseUrl.replace(/\/$/, "");
  }

  async loadPrivateKey(privateKeyPem: string): Promise<void> {
    this.privateKey = await convertPKCS1ToPKCS8(privateKeyPem);
  }

  /**
   * Generate a self-signed JWT for System API authentication
   */
  private async generateSystemToken(): Promise<string> {
    if (!this.privateKey) {
      throw new Error("Private key not loaded");
    }

    const now = Math.floor(Date.now() / 1000);
    const exp = now + 3600; // 1 hour

    const payload = {
      iss: this.userId,
      sub: this.userId,
      aud: this.baseUrl,
      iat: now,
      exp: exp,
    };

    
    const jwt = await new SignJWT(payload)
      .setProtectedHeader({ alg: "RS256" })
      .sign(this.privateKey);

    this.cachedToken = jwt;
    this.tokenExpiry = exp;

    return jwt;
  }

  // Cont below

/**
 * ZitadelSuperadmin - System API client with superadmin privileges
 * Uses self-signed JWTs for authentication (NOT OAuth)
 * Requires SystemAPIUsers configuration in ZITADEL runtime config
 */
export class ZitadelSuperadmin {
  private privateKey: CryptoKey | null = null;
  private userId: string;
  private baseUrl: string;
  private cachedToken: string | null = null;
  private tokenExpiry: number = 0;

  constructor(userId: string, baseUrl: string) {
    this.userId = userId;
    this.baseUrl = baseUrl.replace(/\/$/, "");
  }

  async loadPrivateKey(privateKeyPem: string): Promise<void> {
    this.privateKey = await convertPKCS1ToPKCS8(privateKeyPem);
  }

  /**
   * Generate a self-signed JWT for System API authentication
   */
  private async generateSystemToken(): Promise<string> {
    if (!this.privateKey) {
      throw new Error("Private key not loaded");
    }

    const now = Math.floor(Date.now() / 1000);
    const exp = now + 3600; // 1 hour

    const payload = {
      iss: this.userId,
      sub: this.userId,
      aud: this.baseUrl,
      iat: now,
      exp: exp,
    };

    
    const jwt = await new SignJWT(payload)
      .setProtectedHeader({ alg: "RS256" })
      .sign(this.privateKey);

    this.cachedToken = jwt;
    this.tokenExpiry = exp;

    return jwt;
  }

  // Cont below

/**
 * ZitadelSuperadmin - System API client with superadmin privileges
 * Uses self-signed JWTs for authentication (NOT OAuth)
 * Requires SystemAPIUsers configuration in ZITADEL runtime config
 */
export class ZitadelSuperadmin {
  private privateKey: CryptoKey | null = null;
  private userId: string;
  private baseUrl: string;
  private cachedToken: string | null = null;
  private tokenExpiry: number = 0;

  constructor(userId: string, baseUrl: string) {
    this.userId = userId;
    this.baseUrl = baseUrl.replace(/\/$/, "");
  }

  async loadPrivateKey(privateKeyPem: string): Promise<void> {
    this.privateKey = await convertPKCS1ToPKCS8(privateKeyPem);
  }

  /**
   * Generate a self-signed JWT for System API authentication
   */
  private async generateSystemToken(): Promise<string> {
    if (!this.privateKey) {
      throw new Error("Private key not loaded");
    }

    const now = Math.floor(Date.now() / 1000);
    const exp = now + 3600; // 1 hour

    const payload = {
      iss: this.userId,
      sub: this.userId,
      aud: this.baseUrl,
      iat: now,
      exp: exp,
    };

    
    const jwt = await new SignJWT(payload)
      .setProtectedHeader({ alg: "RS256" })
      .sign(this.privateKey);

    this.cachedToken = jwt;
    this.tokenExpiry = exp;

    return jwt;
  }

  // Cont below

/**
 * ZitadelSuperadmin - System API client with superadmin privileges
 * Uses self-signed JWTs for authentication (NOT OAuth)
 * Requires SystemAPIUsers configuration in ZITADEL runtime config
 */
export class ZitadelSuperadmin {
  private privateKey: CryptoKey | null = null;
  private userId: string;
  private baseUrl: string;
  private cachedToken: string | null = null;
  private tokenExpiry: number = 0;

  constructor(userId: string, baseUrl: string) {
    this.userId = userId;
    this.baseUrl = baseUrl.replace(/\/$/, "");
  }

  async loadPrivateKey(privateKeyPem: string): Promise<void> {
    this.privateKey = await convertPKCS1ToPKCS8(privateKeyPem);
  }

  /**
   * Generate a self-signed JWT for System API authentication
   */
  private async generateSystemToken(): Promise<string> {
    if (!this.privateKey) {
      throw new Error("Private key not loaded");
    }

    const now = Math.floor(Date.now() / 1000);
    const exp = now + 3600; // 1 hour

    const payload = {
      iss: this.userId,
      sub: this.userId,
      aud: this.baseUrl,
      iat: now,
      exp: exp,
    };

    
    const jwt = await new SignJWT(payload)
      .setProtectedHeader({ alg: "RS256" })
      .sign(this.privateKey);

    this.cachedToken = jwt;
    this.tokenExpiry = exp;

    return jwt;
  }

  // Cont below

F

FriendlyForeignAgent•10/28/25, 6:07 PM

private async getToken(): Promise<string> {
    const now = Math.floor(Date.now() / 1000);

    if (!this.cachedToken || now >= this.tokenExpiry - 300) {
      return await this.generateSystemToken();
    }

    return this.cachedToken;
  }

  private async request<T>(
    endpoint: string,
    options: {
      method?: string;
      body?: any;
    } = {}
  ): Promise<T> {
    const token = await this.getToken();
    const url = `${this.baseUrl}${endpoint}`;

    const response = await ofetch<T>(url, {
      method: options.method || "GET",
      headers: {
        Authorization: `Bearer ${token}`,
        "Content-Type": "application/json",
      },
      body: options.body,
    });

    return response;
  }

  async listInstances(queries?: any) {
    return this.request(
      "/zitadel.instance.v2beta.InstanceService/ListInstances",
      {
        method: "POST",
        body: {
          queries: queries || [],
        },
      }
    );
  }
// Other privilaged methods
}

private async getToken(): Promise<string> {
    const now = Math.floor(Date.now() / 1000);

    if (!this.cachedToken || now >= this.tokenExpiry - 300) {
      return await this.generateSystemToken();
    }

    return this.cachedToken;
  }

  private async request<T>(
    endpoint: string,
    options: {
      method?: string;
      body?: any;
    } = {}
  ): Promise<T> {
    const token = await this.getToken();
    const url = `${this.baseUrl}${endpoint}`;

    const response = await ofetch<T>(url, {
      method: options.method || "GET",
      headers: {
        Authorization: `Bearer ${token}`,
        "Content-Type": "application/json",
      },
      body: options.body,
    });

    return response;
  }

  async listInstances(queries?: any) {
    return this.request(
      "/zitadel.instance.v2beta.InstanceService/ListInstances",
      {
        method: "POST",
        body: {
          queries: queries || [],
        },
      }
    );
  }
// Other privilaged methods
}

private async getToken(): Promise<string> {
    const now = Math.floor(Date.now() / 1000);

    if (!this.cachedToken || now >= this.tokenExpiry - 300) {
      return await this.generateSystemToken();
    }

    return this.cachedToken;
  }

  private async request<T>(
    endpoint: string,
    options: {
      method?: string;
      body?: any;
    } = {}
  ): Promise<T> {
    const token = await this.getToken();
    const url = `${this.baseUrl}${endpoint}`;

    const response = await ofetch<T>(url, {
      method: options.method || "GET",
      headers: {
        Authorization: `Bearer ${token}`,
        "Content-Type": "application/json",
      },
      body: options.body,
    });

    return response;
  }

  async listInstances(queries?: any) {
    return this.request(
      "/zitadel.instance.v2beta.InstanceService/ListInstances",
      {
        method: "POST",
        body: {
          queries: queries || [],
        },
      }
    );
  }
// Other privilaged methods
}

private async getToken(): Promise<string> {
    const now = Math.floor(Date.now() / 1000);

    if (!this.cachedToken || now >= this.tokenExpiry - 300) {
      return await this.generateSystemToken();
    }

    return this.cachedToken;
  }

  private async request<T>(
    endpoint: string,
    options: {
      method?: string;
      body?: any;
    } = {}
  ): Promise<T> {
    const token = await this.getToken();
    const url = `${this.baseUrl}${endpoint}`;

    const response = await ofetch<T>(url, {
      method: options.method || "GET",
      headers: {
        Authorization: `Bearer ${token}`,
        "Content-Type": "application/json",
      },
      body: options.body,
    });

    return response;
  }

  async listInstances(queries?: any) {
    return this.request(
      "/zitadel.instance.v2beta.InstanceService/ListInstances",
      {
        method: "POST",
        body: {
          queries: queries || [],
        },
      }
    );
  }
// Other privilaged methods
}

F

FriendlyForeignAgent•10/28/25, 6:09 PM

Hope that helps

A

Achoo! KrakenOP•10/29/25, 4:38 PM

Thank you @bawsky ! And also thank you so much @FriendlyForeignAgent , it's good to know I wasn't wildly off the track. What library does the 'convertPKCS1ToPKCS8' function come from?

Anyway, I'm still on it, and concluded I'm stuck. Seems to happen a lot with Zitadel

This time with System API access.

Here's the complete record of what I'm doing, in shell commands, but my Typescrtipt implementation gives the same error.

Generate keys:

openssl genrsa -traditional -out system-owner.pem 2048
openssl rsa -in system-owner.pem -outform PEM -pubout -out system-owner.pub
openssl pkcs8 -topk8 -inform PEM -in system-owner.pem -out system-owner-pkcs8.pem -nocrypt

openssl genrsa -traditional -out system-owner.pem 2048
openssl rsa -in system-owner.pem -outform PEM -pubout -out system-owner.pub
openssl pkcs8 -topk8 -inform PEM -in system-owner.pem -out system-owner-pkcs8.pem -nocrypt

openssl genrsa -traditional -out system-owner.pem 2048
openssl rsa -in system-owner.pem -outform PEM -pubout -out system-owner.pub
openssl pkcs8 -topk8 -inform PEM -in system-owner.pem -out system-owner-pkcs8.pem -nocrypt

openssl genrsa -traditional -out system-owner.pem 2048
openssl rsa -in system-owner.pem -outform PEM -pubout -out system-owner.pub
openssl pkcs8 -topk8 -inform PEM -in system-owner.pem -out system-owner-pkcs8.pem -nocrypt

Install Zitadel via Helm with the following config:

  zitadel-config-yaml: |-
    ..
    SystemAPIUsers:
    - system-owner:
        # system-owner.pub: -----BEGIN PUBLIC KEY-----MIIB..QAB-----END PUBLIC KEY-----
        KeyData: LS0t..S0tLQ==
        Memberships:
        - MemberType: System
          Roles:
          - SYSTEM_OWNER
          - IAM_OWNER
          - ORG_OWNER

  zitadel-config-yaml: |-
    ..
    SystemAPIUsers:
    - system-owner:
        # system-owner.pub: -----BEGIN PUBLIC KEY-----MIIB..QAB-----END PUBLIC KEY-----
        KeyData: LS0t..S0tLQ==
        Memberships:
        - MemberType: System
          Roles:
          - SYSTEM_OWNER
          - IAM_OWNER
          - ORG_OWNER

  zitadel-config-yaml: |-
    ..
    SystemAPIUsers:
    - system-owner:
        # system-owner.pub: -----BEGIN PUBLIC KEY-----MIIB..QAB-----END PUBLIC KEY-----
        KeyData: LS0t..S0tLQ==
        Memberships:
        - MemberType: System
          Roles:
          - SYSTEM_OWNER
          - IAM_OWNER
          - ORG_OWNER

  zitadel-config-yaml: |-
    ..
    SystemAPIUsers:
    - system-owner:
        # system-owner.pub: -----BEGIN PUBLIC KEY-----MIIB..QAB-----END PUBLIC KEY-----
        KeyData: LS0t..S0tLQ==
        Memberships:
        - MemberType: System
          Roles:
          - SYSTEM_OWNER
          - IAM_OWNER
          - ORG_OWNER

I also tried the below version, with the same result:

    SystemAPIUsers:
    - system-owner:
        KeyData: LS0t..LQ==

    SystemAPIUsers:
    - system-owner:
        KeyData: LS0t..LQ==

    SystemAPIUsers:
    - system-owner:
        KeyData: LS0t..LQ==

    SystemAPIUsers:
    - system-owner:
        KeyData: LS0t..LQ==

( I find this YAML syntax very confusing so I used KYAML as the config source as such:

    SystemAPIUsers: [
        {
            "system-owner": {
                 "KeyData": "LS0t..LQ=="
            }
        }
    ]

    SystemAPIUsers: [
        {
            "system-owner": {
                 "KeyData": "LS0t..LQ=="
            }
        }
    ]

    SystemAPIUsers: [
        {
            "system-owner": {
                 "KeyData": "LS0t..LQ=="
            }
        }
    ]

    SystemAPIUsers: [
        {
            "system-owner": {
                 "KeyData": "LS0t..LQ=="
            }
        }
    ]

)

Generate the auth token:

go install github.com/zitadel/zitadel-tools@latest
export SYSTEM_OWNER_JWT=$(~/go/bin/zitadel-tools key2jwt --audience=https://zitadel.kube.local:999 --key=system-owner-pkcs8.pem --issuer=system-owner)

go install github.com/zitadel/zitadel-tools@latest
export SYSTEM_OWNER_JWT=$(~/go/bin/zitadel-tools key2jwt --audience=https://zitadel.kube.local:999 --key=system-owner-pkcs8.pem --issuer=system-owner)

go install github.com/zitadel/zitadel-tools@latest
export SYSTEM_OWNER_JWT=$(~/go/bin/zitadel-tools key2jwt --audience=https://zitadel.kube.local:999 --key=system-owner-pkcs8.pem --issuer=system-owner)

go install github.com/zitadel/zitadel-tools@latest
export SYSTEM_OWNER_JWT=$(~/go/bin/zitadel-tools key2jwt --audience=https://zitadel.kube.local:999 --key=system-owner-pkcs8.pem --issuer=system-owner)

A

Achoo! KrakenOP•10/29/25, 4:38 PM

And make the call:

curl --request POST -k \
  --url https://zitadel.kube.local:999/zitadel.instance.v2beta.InstanceService/ListInstances \
  --header "Authorization: Bearer ${SYSTEM_OWNER_JWT}" \
  --header 'Connect-Protocol-Version: 1' \
  --header 'Content-Type: application/json' \
  --data '{}'

curl --request POST -k \
  --url https://zitadel.kube.local:999/zitadel.instance.v2beta.InstanceService/ListInstances \
  --header "Authorization: Bearer ${SYSTEM_OWNER_JWT}" \
  --header 'Connect-Protocol-Version: 1' \
  --header 'Content-Type: application/json' \
  --data '{}'

curl --request POST -k \
  --url https://zitadel.kube.local:999/zitadel.instance.v2beta.InstanceService/ListInstances \
  --header "Authorization: Bearer ${SYSTEM_OWNER_JWT}" \
  --header 'Connect-Protocol-Version: 1' \
  --header 'Content-Type: application/json' \
  --data '{}'

curl --request POST -k \
  --url https://zitadel.kube.local:999/zitadel.instance.v2beta.InstanceService/ListInstances \
  --header "Authorization: Bearer ${SYSTEM_OWNER_JWT}" \
  --header 'Connect-Protocol-Version: 1' \
  --header 'Content-Type: application/json' \
  --data '{}'

Result:

{"code":"unauthenticated","message":"Errors.Token.Invalid (AUTH-7fs1e)","details":[{"type":"zitadel.v1.ErrorDetail","value":"CgpBVVRILTdmczFlEhRFcnJvcnMuVG9rZW4uSW52YWxpZA","debug":{"id":"AUTH-7fs1e","message":"Errors.Token.Invalid"}}]}

{"code":"unauthenticated","message":"Errors.Token.Invalid (AUTH-7fs1e)","details":[{"type":"zitadel.v1.ErrorDetail","value":"CgpBVVRILTdmczFlEhRFcnJvcnMuVG9rZW4uSW52YWxpZA","debug":{"id":"AUTH-7fs1e","message":"Errors.Token.Invalid"}}]}

{"code":"unauthenticated","message":"Errors.Token.Invalid (AUTH-7fs1e)","details":[{"type":"zitadel.v1.ErrorDetail","value":"CgpBVVRILTdmczFlEhRFcnJvcnMuVG9rZW4uSW52YWxpZA","debug":{"id":"AUTH-7fs1e","message":"Errors.Token.Invalid"}}]}

{"code":"unauthenticated","message":"Errors.Token.Invalid (AUTH-7fs1e)","details":[{"type":"zitadel.v1.ErrorDetail","value":"CgpBVVRILTdmczFlEhRFcnJvcnMuVG9rZW4uSW52YWxpZA","debug":{"id":"AUTH-7fs1e","message":"Errors.Token.Invalid"}}]}

From what I saw, error AUTH-7fs1e may be related to an incorrect permission set?

If anyone has any clue what I'm doing wrong I'd truly appreciate that.
@Rajat do you happen to have a solid, end-to-end working example? I'm not sure what is going on, but it doesn't feel right that it takes a moderately skilled person 3 days (and counting) to deploy Zitadel on Kubernetes

A

Achoo! KrakenOP•10/29/25, 4:41 PM

Also, what is it with the docs advertising endpoints as 'v2', but those just return "error 5: Not found", and the actual responses come from "v2beta", as in

zitadel.instance.**v2beta**.InstanceService

zitadel.instance.**v2beta**.InstanceService

zitadel.instance.**v2beta**.InstanceService

zitadel.instance.**v2beta**.InstanceService.

F

FriendlyForeignAgent•10/29/25, 5:04 PM

So high level overview:

Generate RSA key pairs locally. This is how I did it.```javascript// Generate private key with traditional format (PKCS#1)// DONE WITH BUN SYNTAX await $`openssl genrsa -traditional -out ${privateKeyPath} 2048`.quiet(); // Export public key await $`openssl rsa -in ${privateKeyPath} -outform PEM -pubout -out ${publicKeyPath}`.quiet() // Read public key and encode as base64 const publicKey = fs.readFileSync(publicKeyPath, "utf-8"); const publicKeyBase64 = Buffer.from(publicKey).toString("base64"); // THIS SHOVES THE PUB KEY INTO A K8S SECRET AND does kubectl apply await createOrPatchSystemApiSecret(publicKeyBase64, userId);```

Shove it into kubernetes config that zitadel knows how to access (make sure the secret aligns with values zitadel expects, set with configSecretName and configSecretKey

# Reference the name of a secret that contains ZITADEL configuration.
# This secret is created by the seed script and contains SystemAPIUsers config
configSecretName: zitadel-system-api-config
# The key under which the ZITADEL configuration is located in the secret.
configSecretKey: config.yaml

# Reference the name of a secret that contains ZITADEL configuration.
# This secret is created by the seed script and contains SystemAPIUsers config
configSecretName: zitadel-system-api-config
# The key under which the ZITADEL configuration is located in the secret.
configSecretKey: config.yaml

# Reference the name of a secret that contains ZITADEL configuration.
# This secret is created by the seed script and contains SystemAPIUsers config
configSecretName: zitadel-system-api-config
# The key under which the ZITADEL configuration is located in the secret.
configSecretKey: config.yaml

# Reference the name of a secret that contains ZITADEL configuration.
# This secret is created by the seed script and contains SystemAPIUsers config
configSecretName: zitadel-system-api-config
# The key under which the ZITADEL configuration is located in the secret.
configSecretKey: config.yaml

mine looks like that.

Ensure the format of yaml is good. the example I gave in above post is functional (works with above helm values).
Ensure that this is done pre zitadel setup. Also, you must allow zitadel init (or setup, i forget) job to run to actually make use of this. If you only run the normal startup process from existing install it wont pick up secrets and set system user.

Allow zitadel to bootstrap as usual and it will have a system user (wont show up in console anywhere, its a diff user type).

At that point you have the user in zitadel and a key pair on your machine.

Now you want to make requests after zit is up. use the class i sent before which will handle loading the key, constructing the jwt, singing it and providing it to the zit api calls.

F

FriendlyForeignAgent•10/29/25, 5:05 PM

If you do that, it will set everything up zitadel side. Just ensure those important things like allowing zitadel to start up properly (run the init job) and enusure its looking for the correct secret and key name.

A

Achoo! KrakenOP•10/29/25, 5:10 PM

I'll try the separate secret route and review the commands! I roll back to previous versions of dev VMs before each attempt so I guarantee that Zitadel gets installed with no previous config

Thank you. I shall report back

A

Achoo! KrakenOP•10/30/25, 2:02 AM

Nothing. @Rajat How do I check if the SystemAPIUsers is even read, and read properly by Zitadel? I went through all the relevant-looking tables in all schemas of the DB and don't see anything -- which may mean that Zitadel is not reading my config OR I'm looking in the wrong place.
Where in the DB is this stuff stored?

Is there an official example of how to configure SystemAPIUsers with the Helm chart?

I followed the docs, I followed what FriendlyForeignAgent posted above, including copying and pasting the secrets etc. and it just does. not. work for me.

I'm sorry, I'm a little desperate here. Please help.

A

Achoo! KrakenOP•10/30/25, 2:05 AM

By 'does not work' I mean that I'm only ever getting the same error AUTH-7fs1e -- I even tried to generate a random PEM and tried taht. Same result -- probably means that Zitadel is not seeing my config, but I'm out of ideas of what to do.

R

Rajat•10/30/25, 2:05 AM

hey @Achoo! Kraken I maybe able to help more when I am back from my vacation but I found this JWT auth fails when adding custom domain and this github issue that may give you some ideas. if they do any trick or help.

A

Achoo! KrakenOP•10/30/25, 2:14 AM

Thank you, there seems to be good info there, I'll work on this more tomorrow, and enjoy your holidays

A

Achoo! KrakenOP•10/30/25, 4:12 PM

Sorted. It was the port

Zitadel's internal Service uses plain HTTP and port 8080. Externally, it's on HTTPS and port 999 (that's where all administrative services live).

So, naturally, I set the JWT audience to use HTTPS and port :999 because, duh, that's where it lives, right?

Naaah:

the audience needs to use the INTERNAL port,
but the audience scheme ( HTTP or HTTPS ) has to match the EXTERNAL endpoint.

@Rajat a couple of suggestions:

mention that maybe in your docs, because it is not obvious, like not at all
in case of aud mismatch the error I was getting was: "issuer does not match: Expected: https://zitadel.kube.local:999, got: system-owner", which is a) just plain wrong -- the issuer was fine b) it reinforces the perception that the external endpoint should be used in the token
maybe some sort of message displayed during app startup would be good? For the longest time I had no clue where I was going wrong, or even if Zitadel was loading my key at all.
documentation bug https://zitadel.com/docs/apis/resources/instance_service_v2/zitadel-instance-v-2-instance-service-add-custom-domain -- the correct parameter name is
```
domain
```
```
domain
```
not
```
customDomain
```
customDomain
```
customDomain
```
customDomain

For completeness, the relevant config part is:

zitadel:
  configmapConfig:
    ExternalSecure: true
    TLS:
      Enabled: false

zitadel:
  configmapConfig:
    ExternalSecure: true
    TLS:
      Enabled: false

zitadel:
  configmapConfig:
    ExternalSecure: true
    TLS:
      Enabled: false

zitadel:
  configmapConfig:
    ExternalSecure: true
    TLS:
      Enabled: false

Anyway, off to cleaning up the unholy mess that I made while debugging this

and thank you so much to everyone!

@FriendlyForeignAgent @bawsky @Rajat

R

Rajat•11/3/25, 6:37 PM

hey @Achoo! Kraken thank you for your feedbacks and I am glad it workd out.
I will take those feedback to the team so that they could be fixed/updated.

R

Rajat•11/3/25, 11:02 PM

hey @Achoo! Kraken first of all thank you for your detailed feedback. This was super helpful. I have raised it internally. Someone from the team should look at them and I will get back to you on this thread once I have more updates. Besides that, thanks a lot for using zitadel. What kind of use case are you looking to use zitadel with?. Just curious to learn more about what you are building