TomasP•15mo ago

Kubernetes ingress error on SAML auth

We have a Zitadel instance running in a Kubernetes cluster, but when trying to use the MockSAML service, we get sort of upstream header error in the ingress service. We tracked it to a sort of a header issue. Did you encounter this sort of issue maybe? The error:

[error] 1904#1904: *29000294 upstream sent too big header while reading response header from upstream, client: <IP>, server: ~^(?<subdomain>[\w-]+)\.auth-dev\.server\.net$, request: "POST /ui/login/login/externalidp/saml/acs HTTP/2.0", upstream: "grpc://10.224.2.168:8080", host: "test-organization.server.net", referrer: "https://mocksaml.com/"

[error] 1904#1904: *29000294 upstream sent too big header while reading response header from upstream, client: <IP>, server: ~^(?<subdomain>[\w-]+)\.auth-dev\.server\.net$, request: "POST /ui/login/login/externalidp/saml/acs HTTP/2.0", upstream: "grpc://10.224.2.168:8080", host: "test-organization.server.net", referrer: "https://mocksaml.com/"

We tried various configurations for the ingress adapter, this is the annotation config we're at right now:

  ingress:
            enabled: true
            className: nginx
            annotations:
              nginx.ingress.kubernetes.io/ssl-redirect: 'true'
              nginx.ingress.kubernetes.io/backend-protocol: 'GRPC'
              nginx.ingress.kubernetes.io/client-body-buffer-size: 64k
              nginx.ingress.kubernetes.io/client-header-buffer-size: 100k
              nginx.ingress.kubernetes.io/http2-max-header-size: 96k
              nginx.ingress.kubernetes.io/large-client-header-buffers: 4 100k
              nginx.ingress.kubernetes.io/proxy-body-size: 150m
              nginx.ingress.kubernetes.io/proxy-buffer-size: 96k
              nginx.ingress.kubernetes.io/proxy-read-timeout: "1000"
              nginx.ingress.kubernetes.io/proxy-send-timeout: "1000"
              nginx.ingress.kubernetes.io/server-snippet: |
                client_header_buffer_size 100k;
                large_client_header_buffers 4 100k;
                grpc_buffer_size 16k;
...

  ingress:
            enabled: true
            className: nginx
            annotations:
              nginx.ingress.kubernetes.io/ssl-redirect: 'true'
              nginx.ingress.kubernetes.io/backend-protocol: 'GRPC'
              nginx.ingress.kubernetes.io/client-body-buffer-size: 64k
              nginx.ingress.kubernetes.io/client-header-buffer-size: 100k
              nginx.ingress.kubernetes.io/http2-max-header-size: 96k
              nginx.ingress.kubernetes.io/large-client-header-buffers: 4 100k
              nginx.ingress.kubernetes.io/proxy-body-size: 150m
              nginx.ingress.kubernetes.io/proxy-buffer-size: 96k
              nginx.ingress.kubernetes.io/proxy-read-timeout: "1000"
              nginx.ingress.kubernetes.io/proxy-send-timeout: "1000"
              nginx.ingress.kubernetes.io/server-snippet: |
                client_header_buffer_size 100k;
                large_client_header_buffers 4 100k;
                grpc_buffer_size 16k;
...

We use Zitadel v2.46.2. Thanks!

4 Replies

Unknown User•15mo ago

Message Not Public

FFO•15mo ago

actually I think one should do that 😄

Unknown User•15mo ago

Message Not Public

TomasPOP•15mo ago

Seems that the server-snippets annotation was not working in our setup, we had to add this to nginx controller config:

Kubernetes ingress error on SAML auth

Did you find this page helpful?