ShardoolS
ZITADEL•12mo ago•
5 replies
Shardool

Service user fetching /oauth/v2/token timing out

Zitadel version: v2.67.2
Running 3 instances behind nginx

Hello folks,

Have been running zitadel in production for a couple of months now and am starting to issues with service users fetching tokens from zitadel timing out. This happens pretty often between successful requests.

We have about ~5 service users that will request for new tokens every 5 seconds or so with their key. We started off ok but have noticed degradation after running this setup for ~15 days. We should ideally cache the tokens but is this behavior expected?

Nginx logs:
2025/02/03 16:57:13 [error] 26#26: *94 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 192.168.45.171, server: auth.***.***, request: "POST /oauth/v2/token HTTP/2.0", upstream: "grpc://192.168.17.41:8080", host: "auth.***.***"
192.168.45.171 - - [03/Feb/2025:16:57:13 +0000] "POST /oauth/v2/token HTTP/2.0" 504 160 "-" "Go-http-client/2.0" 855 60.002 [***-prod-zitadel-auth-8080] ] 192.168.17.41:8080 0 60.001 504 2e330162c99f856fec6ede6765bbec8d
2025/02/03 16:57:29 [error] 26#26: *113 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 192.168.45.171, server: auth.***.***, request: "POST /oauth/v2/token HTTP/2.0", upstream: "grpc://192.168.60.40:8080", host: "auth.***.***"
192.168.45.171 - - [03/Feb/2025:16:57:29 +0000] "POST /oauth/v2/token HTTP/2.0" 504 160 "-" "Go-http-client/2.0" 855 60.001 [***-prod-zitadel-auth-8080] ] 192.168.60.40:8080 0 60.000 504 01421677980b20d9f1920e0e37f9d581


Attaching zitadel logs for reference. Seeing unable to filter events / duplicate key violates unique constraint pretty often.
message.txt35.95KB
Was this page helpful?