Fidget•3mo ago

Login v2 pod not sending CUSTOM_REQUEST_HEADERS

Use-case: Kubernetes self-hosted proof-of-concept Environment: Self-Hosting on EKS Version: ghcr.io/zitadel/zitadel:v4.0.0 Stack: Zitadel UI v2 What you expected to happen: Zitadel UI for basic management login What went wrong: Login v2 gets 400 error due to issue with Host header to zitadel API instance Attachments: On the zitadel-login pods I see the environment variable file is set: cat /.env-file/.env

/.env-file/.env
ZITADEL_SERVICE_USER_TOKEN_FILE="/login-client/pat"
ZITADEL_API_URL="http://zitadel:8080"
CUSTOM_REQUEST_HEADERS="Host:zitadel.example.com"

/.env-file/.env
ZITADEL_SERVICE_USER_TOKEN_FILE="/login-client/pat"
ZITADEL_API_URL="http://zitadel:8080"
CUSTOM_REQUEST_HEADERS="Host:zitadel.example.com"

Testing from the CLI with curl and wget I can send host headers and get a 200 response. However the pods logs show connection errors experienced when the CUSTOM_REQUEST_HEADERS aren't sent.

ZITADEL_SERVICE_USER_TOKEN_FILE=/login-client/pat is set and file exists, setting ZITADEL_SERVICE_USER_TOKEN to the files content
   ▲ Next.js 15.4.0-canary.86
   - Local:        http://localhost:3000
   - Network:      http://0.0.0.0:3000

Error [ConnectError]: [unavailable] connect ECONNREFUSED 172.20.216.193:8080
    at <unknown> (ConnectError: [unavailable] connect ECONNREFUSED 172.20.216.193:8080)
    at h.from (.next/server/chunks/981.js:9:112772)
    at aB (.next/server/chunks/981.js:9:74614)
    at ClientHttp2Session.h (.next/server/chunks/981.js:9:82106) {
  rawMessage: 'connect ECONNREFUSED 172.20.216.193:8080',
  code: 14,
  metadata: Headers {},
  details: [],
  [cause]: Error: connect ECONNREFUSED 172.20.216.193:8080
      at <unknown> (Error: connect ECONNREFUSED 172.20.216.193:8080) {
    errno: -111,
    code: 'ECONNREFUSED',
    syscall: 'connect',
    address: '172.20.216.193',
    port: 8080
  }
}

ZITADEL_SERVICE_USER_TOKEN_FILE=/login-client/pat is set and file exists, setting ZITADEL_SERVICE_USER_TOKEN to the files content
   ▲ Next.js 15.4.0-canary.86
   - Local:        http://localhost:3000
   - Network:      http://0.0.0.0:3000

Error [ConnectError]: [unavailable] connect ECONNREFUSED 172.20.216.193:8080
    at <unknown> (ConnectError: [unavailable] connect ECONNREFUSED 172.20.216.193:8080)
    at h.from (.next/server/chunks/981.js:9:112772)
    at aB (.next/server/chunks/981.js:9:74614)
    at ClientHttp2Session.h (.next/server/chunks/981.js:9:82106) {
  rawMessage: 'connect ECONNREFUSED 172.20.216.193:8080',
  code: 14,
  metadata: Headers {},
  details: [],
  [cause]: Error: connect ECONNREFUSED 172.20.216.193:8080
      at <unknown> (Error: connect ECONNREFUSED 172.20.216.193:8080) {
    errno: -111,
    code: 'ECONNREFUSED',
    syscall: 'connect',
    address: '172.20.216.193',
    port: 8080
  }
}

5 Replies

Ask Inkeep•3mo ago

Hi @Fidget, I'm an AI assistant for ZITADEL. While a team member reviews this question, I'll see if I can find relevant sources for your question. Answer I wasn't able to find a direct answer in the documentation for your specific Kubernetes configuration issue, here are some relevant sources: 1. Set up Zitadel with Docker Compose : The Docker Compose documentation shows the CUSTOM_REQUEST_HEADERS environment variable being used with the login container, and mentions that if you can't use the network_mode service:zitadel, you can pass the environment variable CUSTOM_REQUEST_HEADERS=Host:localhost instead (1) . 2. Discord message from @blackr4t on 2024-10-02 : A community discussion mentions that regardless of what header comes in, it will forward the domain as ZITADEL_API_URL without https://, and there's work in progress to make this configuration easier (2) . Your issue appears to be related to network connectivity between the login container and the ZITADEL API service in your Kubernetes environment. The CUSTOM_REQUEST_HEADERS configuration should help with host header forwarding, but the connection refused error suggests a networking issue rather than just header configuration. Would you like me to help you with a general approach to troubleshooting this Kubernetes networking issue, or would you prefer to click the "Ask for help" button to have a team member review your specific setup? To learn more, see the sources I considered: [1] Set up Zitadel with Docker Compose [2] Discord message from @blackr4t on 2024-10-02 Feel free to tag @Ask Inkeep with additional questions.

FidgetOP•3mo ago

I don't believe it's a connection refused due to being able to hit the same address:port combination with wget from the same pod container. On login pod replicate connection error without host header . /.env-file/.env && wget -q -O - "${ZITADEL_API_URL}"

wget: server returned error: HTTP/1.1 404 Not Found

wget: server returned error: HTTP/1.1 404 Not Found

On login pod show success with host header . /.env-file/.env && wget -q -O - --header "${CUSTOM_REQUEST_HEADERS}" "${ZITADEL_API_URL}"

<!doctype html>
<html lang="en">
  <head>
    <meta charset="utf-8"/>
    <title>ZITADEL • Console</title>
    <base href="/ui/console/"/>
    <meta name="viewport" content="width=device-width, initial-scale=1"/>
    <link id="appFavicon" rel="icon" type="image/x-icon" href="favicon.ico"/>
    <link rel="stylesheet" href="./assets/icons/line-awesome/css/line-awesome.min.css"/>
    <link rel="manifest" href="manifest.webmanifest"/>
    <meta name="theme-color" content="#e6768b"/>
...
    <meta name="robots" content="none"/>
  <link rel="stylesheet" href="styles.45af6733371ffe5f.css"></head>

  <body>
    <cnsl-root></cnsl-root>
    <noscript>Please enable JavaScript to continue using this application.</noscript>
  <script src="runtime.9d86acfd175fdf65.js" type="module"></script><script src="polyfills.4258d1ba3f1b5f57.js" type="module"></script><script src="scripts.cfdbc03626b7e4d0.js" defer></script><script src="main.1d2591fd5283265b.js" type="module"></script></body>
</html>

<!doctype html>
<html lang="en">
  <head>
    <meta charset="utf-8"/>
    <title>ZITADEL • Console</title>
    <base href="/ui/console/"/>
    <meta name="viewport" content="width=device-width, initial-scale=1"/>
    <link id="appFavicon" rel="icon" type="image/x-icon" href="favicon.ico"/>
    <link rel="stylesheet" href="./assets/icons/line-awesome/css/line-awesome.min.css"/>
    <link rel="manifest" href="manifest.webmanifest"/>
    <meta name="theme-color" content="#e6768b"/>
...
    <meta name="robots" content="none"/>
  <link rel="stylesheet" href="styles.45af6733371ffe5f.css"></head>

  <body>
    <cnsl-root></cnsl-root>
    <noscript>Please enable JavaScript to continue using this application.</noscript>
  <script src="runtime.9d86acfd175fdf65.js" type="module"></script><script src="polyfills.4258d1ba3f1b5f57.js" type="module"></script><script src="scripts.cfdbc03626b7e4d0.js" defer></script><script src="main.1d2591fd5283265b.js" type="module"></script></body>
</html>

fcoppede•2mo ago

hello @Fidget thanks for reaching out! can you try running the last version of Zitadel and let me know if that error persists? thanks!

FidgetOP•2mo ago

@Federico

configMapGenerator:
  - name: helm-zitadel
    literals:
      - externalSecure=true
      - externalDomain=${component}.${internal_domain}
      - externalPort=443
      - |-
        values.yaml=# https://github.com/zitadel/zitadel/blob/main/cmd/defaults.yaml
        zitadel:
          masterkeySecretName: zitadel-masterkey
          dbSslCaCrtSecret: "rootca-zitadel"
          dbSslAdminCrtSecret: "mtls-cnpg-zitadel-superuser"
          dbSslUserCrtSecret: "mtls-cnpg-zitadel-app"
          serverSslCrtSecret: "https-zitadel"
          #debug:
          #  enabled: true
          configmapConfig:
            Database:
              postgres:
        image:
          repository: "ghcr.io/zitadel/zitadel"
          tag: "v4.2.0"
        login:
          image:
            repository: "ghcr.io/zitadel/zitadel-login"
            tag: "v4.2.0"
        initJob:
          command: "zitadel"
        setupJob:
          machinekeyWriter:
            image:
              repository: "alpine/k8s"
              tag: "1.34.0"
        env:
          - name: ZITADEL_DEFAULTINSTANCE_FEATURES_LOGINV2_REQUIRED
            value: "false"
          - name: ZITADEL_DATABASE_POSTGRES_HOST
            valueFrom:
              secretKeyRef:
                name: cnpg-zitadel-app
                key: host
          - name: ZITADEL_DATABASE_POSTGRES_PORT
            valueFrom:
              secretKeyRef:
                name: cnpg-zitadel-app
                key: port
          - name: ZITADEL_DATABASE_POSTGRES_DATABASE
            valueFrom:
              secretKeyRef:
                name: cnpg-zitadel-app
                key: dbname
          - name: ZITADEL_DATABASE_POSTGRES_USER_USERNAME
            valueFrom:
              secretKeyRef:
                name: cnpg-zitadel-app
                key: username
          - name: ZITADEL_DATABASE_POSTGRES_USER_SSL_MODE
            value: verify-full
          - name: ZITADEL_TLS_ENABLED
            value: "false"

configMapGenerator:
  - name: helm-zitadel
    literals:
      - externalSecure=true
      - externalDomain=${component}.${internal_domain}
      - externalPort=443
      - |-
        values.yaml=# https://github.com/zitadel/zitadel/blob/main/cmd/defaults.yaml
        zitadel:
          masterkeySecretName: zitadel-masterkey
          dbSslCaCrtSecret: "rootca-zitadel"
          dbSslAdminCrtSecret: "mtls-cnpg-zitadel-superuser"
          dbSslUserCrtSecret: "mtls-cnpg-zitadel-app"
          serverSslCrtSecret: "https-zitadel"
          #debug:
          #  enabled: true
          configmapConfig:
            Database:
              postgres:
        image:
          repository: "ghcr.io/zitadel/zitadel"
          tag: "v4.2.0"
        login:
          image:
            repository: "ghcr.io/zitadel/zitadel-login"
            tag: "v4.2.0"
        initJob:
          command: "zitadel"
        setupJob:
          machinekeyWriter:
            image:
              repository: "alpine/k8s"
              tag: "1.34.0"
        env:
          - name: ZITADEL_DEFAULTINSTANCE_FEATURES_LOGINV2_REQUIRED
            value: "false"
          - name: ZITADEL_DATABASE_POSTGRES_HOST
            valueFrom:
              secretKeyRef:
                name: cnpg-zitadel-app
                key: host
          - name: ZITADEL_DATABASE_POSTGRES_PORT
            valueFrom:
              secretKeyRef:
                name: cnpg-zitadel-app
                key: port
          - name: ZITADEL_DATABASE_POSTGRES_DATABASE
            valueFrom:
              secretKeyRef:
                name: cnpg-zitadel-app
                key: dbname
          - name: ZITADEL_DATABASE_POSTGRES_USER_USERNAME
            valueFrom:
              secretKeyRef:
                name: cnpg-zitadel-app
                key: username
          - name: ZITADEL_DATABASE_POSTGRES_USER_SSL_MODE
            value: verify-full
          - name: ZITADEL_TLS_ENABLED
            value: "false"

This values.yaml in a configmap with Flux to deploy via helm release worked to get a email login ui, so now I can actually explore zitadel, thank you so very much!

Gigi the Giraffe (Zitadel)•2mo ago

🎉 Looks like you just helped out another community member! Thanks for being so helpful <@187445980088631296>! You're now one step closer to leveling up—keep up the amazing peer support! 🚀

Login v2 pod not sending CUSTOM_REQUEST_HEADERS

Did you find this page helpful?