Zitadel in Kubernetes and access from within the cluster
Hi! I'm trying to set up Zitadel running in Kubernetes and having trouble with domain-based instance identification. I can access the admin UI and make API calls using the external domain; all good.
Now, I also want to make calls to Zitadel from within the cluster (using the
...svc.cluster.local Service endpoint) and... I'm stuck.
Trying to access the internal endpoint from within a temporary pod:
...which is fair enough, but how do I fix that?
( also, it says 'https' there, but I'm using http as it's internal comms )
- I read all the docs, helm chart specs, config specs, github tickets and tutorials I could find
- /ui/console/instance?id=domain doesn't offer adding new domains
- adding extra domains to the Organisation doesn't change anything ( I tried all variants, with and without the port; they go straight into Verified domains as I have verification disabled )
- Helm zitadel.selfSignedCert.additionalDnsName seems to be for the cert only
This is self-hosted K8S, latest Zitadel version deployed using the official Helm chart from charts.zitadel.com.19 Replies
Hi @Achoo! Kraken, I'm an AI assistant for ZITADEL. While a team member reviews this question, I'll see if I can find relevant sources for your question.
Answer
You need to add
zitadel.zitadel.svc.cluster.local as an additional instance domain using the System API (1) (2) . The Helm chart's ExternalDomain setting only defines the primary domain (3) .
Organization domains won't work for this—they're intended for routing users by login methods to their correct organization, not for instance-level access (1) .
To add the internal domain, use the System API's AddDomain endpoint after ZITADEL is running (2) . The Helm chart cannot declare multiple instance domains during deployment (3) .
Your Kubernetes Ingress should route both your external domain (mydomain.com) and the internal service domain (zitadel.zitadel.svc.cluster.local) to the ZITADEL service . ZITADEL uses the Host header to identify the correct instance .
Note that if you change the ExternalDomain configuration later, you need to rerun ZITADEL's setup phase for the changes to take effect (1) .
To learn more, see the sources I considered:
[1] External ZITADEL Access
[2] Trouble accessing the service using different hostnames.
[3] Set multiple instance domains via helm chart
Feel free to tag @Ask Inkeep with additional questions.Since Zitadel can run multiple (virtual) instances, you should specify what is the specify you want if it doesn't match the domain you're using. You can do that using the
Host request header.
E.g.:
(assuming your Zitadel server run on zitadel.my-cool-domain.com)@Ask Inkeep Ok, I'm trying to use the InstanceApi to list instances as the first step. I'm calling: zitadelClient.betaInstances.listInstances( ... ) with the "Automatically Initialized IAM Admin" user (that's an "Iam Owner' permission level) and getting:
body: '{"code":"permission_denied","message":"No matching permissions found
From what I see it should be authorised to manage instances. What other permission do I need? Or is the error a red herring?
Thanks for the follow-up, @Achoo! Kraken. I'll get back to you soon.
Answer
I wasn't able to find a direct answer, here are some relevant sources:
1. [Instance resource API extended permission checks][(5) ]: This GitHub issue discusses permission checks on instance endpoints and mentions that instance admins (e.g., IAM_OWNER) should be permitted to call instance service endpoints with their own instance ID, but when calling with any other instance ID, a permission denied error is returned (1) .
2. [[permission_denied] No matching permissions found (AUTH-5mWD2) ][(6) ]: This Discord thread mentions that users need
IAM_OWNER or IAM_LOGIN_CLIENT roles, and the solution involved adding these roles in the default admin account's membership section (2) .
To learn more, see the sources I considered:
[1] Instance resource API extended permission checks
[2] [permission_denied] No matching permissions found (AUTH-5mWD2)
Feel free to tag @Ask Inkeep with additional questions.I've stumbled upon this myself but can't remember where I found the
Host: header tip... Hope it's documented somewhere but tbh I think I found it on a GitHub issue 🤷♂️
Btw, see here to learn more about virtual instances.
(I never had a use for that myself.. we run 3 separate Zitadel deploys in our infrastructure, but they're all isolated a have only one "virtual instance" (i.e. the default instance) each. I think it's less confusing that way)ZITADEL Docs
Instance Structure
That's very good to know, thank you, but also... I'd like to use Zitadel as the auth provider to other cluster services, and they are configurable with the auth + .well-known discoverability endpoint only 😕 Can't use the Host: trick with them...
Ah right... Tbh I don't think you can do anything if you can't edit the request headers on those other services.
I suppose one thing you can try is to add a new custom domain to your instance... I've never tried that, but it may work if you simply add
zitadel.zitadel.svc.cluster.local
v1 api: https://zitadel.com/docs/apis/resources/system/system-service-add-domain
v2 api: https://zitadel.com/docs/apis/resources/instance_service_v2/zitadel-instance-v-2-instance-service-add-custom-domain
(not sure how to do that from the UI)Yes, I'm fiddling with the API and it seems there is a permission missing for certain ops:
// client.betaInstances.getInstance -> allowed
// client.betaInstances.addCustomDomain -> DENIED
// client.betaInstances.listInstances -> DENIED
I'll try the V1 API next, maybe it's the 'beta' part... 🤔
Got it. Tagging @Rajat for additional help.
https://github.com/zitadel/zitadel/issues/9452 (closed May 21)
says that "List Instances" and "Add Custom Domain" were migrated to V2 API, and explicitly marks V1 equivalents as deprecated; therefore I assume the V2 endpoints should work.
I don't want to develop my solution using a deprecated, soon-to-be-removed API, and would appreciate help with figuring out how to authorise my serviceaccount to use those API calls.
GitHub
Add instance requests to resource API · Issue #9452 · zitadel/zit...
As a developer I want to be able to perform all requests relevant to instance, using the new resource API. Acceptance Criteria All needed requests are implemented All requests can be called by an a...
OK, I found the problem.
https://zitadel.com/docs/guides/manage/console/managers says
IAM Managers: This is the highest level. Users with IAM Manager roles are able to manage the whole Instance.
When set up via Helm chart, this is NOT TRUE that it can manage the whole instance.
https://zitadel.com/docs/apis/resources/instance_service_v2/zitadel-instance-v-2-instance-service-add-custom-domain
lists Required permissions:
system.domain.write
and it appears that this permission is ONLY granted to Role: "SYSTEM_OWNER"
Therefore Role: "IAM_OWNER" IS NOT the highest level.
ZITADEL Docs
To configure managers in ZITADEL go to the resource where you like to add it (e.g Instance, Organization, Project, GrantedProject).
ZITADEL Docs
Add Custom Domain
Haha, splendid 🙂 I'm glad we both came to the same conclusion.
Now I need to figure out how to grant the SYSTEM_OWNER role to my user, ideally with the Helm chart. Otherwise I'll have to hack IAM_OWNER permissions in the chart values, but that's ewww..
There's the SystemAPIUsers section, looks promising...
Hm, granting those "system" permissions seems tricky 🤔
I'd assume the
/admin/v1/members/:userId endpoint would work for that but seems like it doesn't...
Oh nice! That indeed looks promising!
Please let me know if that works!
Just found this documentation for this exact subject:
https://zitadel.com/docs/guides/integrate/zitadel-apis/access-zitadel-system-api
It seems that a few particular manual steps are required to access that API... Unfortunately though I don't think the helm chart has builtin support for this, so you'll need to build your own solution (generating the RSA key pair/pointing the Zitadel configuration to it/generate a signed JWT using the RSA secret)
I had the same issue last week!
Somewhere in the docs its says that the auth used for this user and how its registered is incompatible to other account types.
Currently (in development with tilt) we have a seed script runner that configures zitadel and its k8s environment. One part of that is to generate local RSA certs, generate a k8s secret with a format that zitadel expects (secretConfigRef in helm i think), add the pub key to that secret:
We have a k8s job that merges some of these zitadel secrets in cluster, but this will work standalone too.
Then we make a client (with zitadel-node package, though principle is the same)
See below.
And ignore the crappy code, my goal was to make it functional 🙂
Hope that helps
Thank you @bawsky ! And also thank you so much @FriendlyForeignAgent , it's good to know I wasn't wildly off the track. What library does the 'convertPKCS1ToPKCS8' function come from?
Anyway, I'm still on it, and concluded I'm stuck. Seems to happen a lot with Zitadel 😩
This time with System API access.
Here's the complete record of what I'm doing, in shell commands, but my Typescrtipt implementation gives the same error.
Generate keys:
Install Zitadel via Helm with the following config:
I also tried the below version, with the same result:
( I find this YAML syntax very confusing so I used KYAML as the config source as such:
)
Generate the auth token:
And make the call:
Result:
From what I saw, error AUTH-7fs1e may be related to an incorrect permission set?
If anyone has any clue what I'm doing wrong I'd truly appreciate that.
@Rajat do you happen to have a solid, end-to-end working example? I'm not sure what is going on, but it doesn't feel right that it takes a moderately skilled person 3 days (and counting) to deploy Zitadel on Kubernetes 😔
Also, what is it with the docs advertising endpoints as 'v2', but those just return "error 5: Not found", and the actual responses come from "v2beta", as in
zitadel.instance.**v2beta**.InstanceService.So high level overview:
1. Generate RSA key pairs locally. This is how I did it.
2. Shove it into kubernetes config that zitadel knows how to access (make sure the secret aligns with values zitadel expects, set with configSecretName and configSecretKey
mine looks like that.
3. Ensure the format of yaml is good. the example I gave in above post is functional (works with above helm values).
4. Ensure that this is done pre zitadel setup. Also, you must allow zitadel init (or setup, i forget) job to run to actually make use of this. If you only run the normal startup process from existing install it wont pick up secrets and set system user.
Allow zitadel to bootstrap as usual and it will have a system user (wont show up in console anywhere, its a diff user type).
At that point you have the user in zitadel and a key pair on your machine.
5. Now you want to make requests after zit is up. use the class i sent before which will handle loading the key, constructing the jwt, singing it and providing it to the zit api calls.
If you do that, it will set everything up zitadel side. Just ensure those important things like allowing zitadel to start up properly (run the init job) and enusure its looking for the correct secret and key name.
I'll try the separate secret route and review the commands! I roll back to previous versions of dev VMs before each attempt so I guarantee that Zitadel gets installed with no previous config 😉 Thank you. I shall report back 🙂