Shardool•5mo ago

Service user fetching /oauth/v2/token timing out

Zitadel version: v2.67.2 Running 3 instances behind nginx Hello folks, Have been running zitadel in production for a couple of months now and am starting to issues with service users fetching tokens from zitadel timing out. This happens pretty often between successful requests. We have about ~5 service users that will request for new tokens every 5 seconds or so with their key. We started off ok but have noticed degradation after running this setup for ~15 days. We should ideally cache the tokens but is this behavior expected? Nginx logs:

2025/02/03 16:57:13 [error] 26#26: *94 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 192.168.45.171, server: auth.***.***, request: "POST /oauth/v2/token HTTP/2.0", upstream: "grpc://192.168.17.41:8080", host: "auth.***.***"
192.168.45.171 - - [03/Feb/2025:16:57:13 +0000] "POST /oauth/v2/token HTTP/2.0" 504 160 "-" "Go-http-client/2.0" 855 60.002 [***-prod-zitadel-auth-8080] ] 192.168.17.41:8080 0 60.001 504 2e330162c99f856fec6ede6765bbec8d
2025/02/03 16:57:29 [error] 26#26: *113 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 192.168.45.171, server: auth.***.***, request: "POST /oauth/v2/token HTTP/2.0", upstream: "grpc://192.168.60.40:8080", host: "auth.***.***"
192.168.45.171 - - [03/Feb/2025:16:57:29 +0000] "POST /oauth/v2/token HTTP/2.0" 504 160 "-" "Go-http-client/2.0" 855 60.001 [***-prod-zitadel-auth-8080] ] 192.168.60.40:8080 0 60.000 504 01421677980b20d9f1920e0e37f9d581

2025/02/03 16:57:13 [error] 26#26: *94 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 192.168.45.171, server: auth.***.***, request: "POST /oauth/v2/token HTTP/2.0", upstream: "grpc://192.168.17.41:8080", host: "auth.***.***"
192.168.45.171 - - [03/Feb/2025:16:57:13 +0000] "POST /oauth/v2/token HTTP/2.0" 504 160 "-" "Go-http-client/2.0" 855 60.002 [***-prod-zitadel-auth-8080] ] 192.168.17.41:8080 0 60.001 504 2e330162c99f856fec6ede6765bbec8d
2025/02/03 16:57:29 [error] 26#26: *113 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 192.168.45.171, server: auth.***.***, request: "POST /oauth/v2/token HTTP/2.0", upstream: "grpc://192.168.60.40:8080", host: "auth.***.***"
192.168.45.171 - - [03/Feb/2025:16:57:29 +0000] "POST /oauth/v2/token HTTP/2.0" 504 160 "-" "Go-http-client/2.0" 855 60.001 [***-prod-zitadel-auth-8080] ] 192.168.60.40:8080 0 60.000 504 01421677980b20d9f1920e0e37f9d581

Attaching zitadel logs for reference. Seeing unable to filter events / duplicate key violates unique constraint pretty often.

message.txt

4 Replies

ShardoolOP•4mo ago

zitadel config:

ingress:
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
    nginx.ingress.kubernetes.io/backend-protocol: GRPC
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
  className: nginx
  enabled: true
  hosts:
  - host: auth.***.***
    paths:
    - path: /
      pathType: Prefix
  tls:
  - hosts:
    - auth.***.***
    secretName: zitadel-tls
zitadel:
  configSecretKey: config.yaml
  configSecretName: <secret>
  configmapConfig:
    Database:
      Postgres:
        Admin:
          ExistingDatabase: ***
          SSL:
            Mode: require
          Username: ***
        Database: zitadel
        MaxConnIdleTime: 5m
        MaxConnLifetime: 30m
        MaxIdleConns: 10
        MaxOpenConns: 20
        Port: 5432
        User:
          SSL:
            Mode: require
          Username: zitadel
    ExternalDomain: auth.***.***
    ExternalPort: 443
    ExternalSecure: true
    TLS:
      Enabled: false
  masterkeySecretName: zitadel-masterkey

ingress:
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
    nginx.ingress.kubernetes.io/backend-protocol: GRPC
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
  className: nginx
  enabled: true
  hosts:
  - host: auth.***.***
    paths:
    - path: /
      pathType: Prefix
  tls:
  - hosts:
    - auth.***.***
    secretName: zitadel-tls
zitadel:
  configSecretKey: config.yaml
  configSecretName: <secret>
  configmapConfig:
    Database:
      Postgres:
        Admin:
          ExistingDatabase: ***
          SSL:
            Mode: require
          Username: ***
        Database: zitadel
        MaxConnIdleTime: 5m
        MaxConnLifetime: 30m
        MaxIdleConns: 10
        MaxOpenConns: 20
        Port: 5432
        User:
          SSL:
            Mode: require
          Username: zitadel
    ExternalDomain: auth.***.***
    ExternalPort: 443
    ExternalSecure: true
    TLS:
      Enabled: false
  masterkeySecretName: zitadel-masterkey

config.yml only has the database configuration implemented token caching on service users and issue doesn't happen as much but would be nice to know the root cause

Raccine•4mo ago

Hey @Shardool! Thanks for reaching out - Let me actually loop in @adlerhurst who might be able to give you some more insight into why this is happening

adlerhurst•4mo ago

Hi there Which version of zitadel are you running?

ShardoolOP•3mo ago

running v2.67.2

Service user fetching /oauth/v2/token timing out

Did you find this page helpful?