OIDC calls taking 2-3s each, but DB queries are all sub-millisecond

F

fabienne•10/7/24, 5:05 PM

@muhlemmer can you help here?

F

FFO•10/8/24, 2:44 PM

Hm the redirect in our cloud takes about 190ms

F

FFO•10/8/24, 2:46 PM

To give us something to work with here.

On QA we use 4CPU/16GBmem/100GBsdd as cloud sql postgres with 20 connections

F

FFO•10/8/24, 2:46 PM

Can you share us some insights of how you DB stats look like?

F

FFO•10/9/24, 1:43 PM

Hm can you configure tracing? That might give us a clue.

https://github.com/zitadel/zitadel/blob/main/cmd/defaults.yaml#L11

F

FFO•10/11/24, 6:34 PM

Let me quickly pull our config, maybe that helps you.

F

FFO•10/11/24, 6:36 PM

Database:
  postgres:
    Host: ${db_host}
    Port: 5432
    Database: ${db}
    MaxOpenConns: 15
    MaxIdleConns: 12
    MaxConnLifetime: 30m
    MaxConnIdleTime: 2m
    Options: ""
    User:
      SSL:
        Mode: require
    Admin:
      SSL:
        Mode: require

Database:
  postgres:
    Host: ${db_host}
    Port: 5432
    Database: ${db}
    MaxOpenConns: 15
    MaxIdleConns: 12
    MaxConnLifetime: 30m
    MaxConnIdleTime: 2m
    Options: ""
    User:
      SSL:
        Mode: require
    Admin:
      SSL:
        Mode: require

Database:
  postgres:
    Host: ${db_host}
    Port: 5432
    Database: ${db}
    MaxOpenConns: 15
    MaxIdleConns: 12
    MaxConnLifetime: 30m
    MaxConnIdleTime: 2m
    Options: ""
    User:
      SSL:
        Mode: require
    Admin:
      SSL:
        Mode: require

Database:
  postgres:
    Host: ${db_host}
    Port: 5432
    Database: ${db}
    MaxOpenConns: 15
    MaxIdleConns: 12
    MaxConnLifetime: 30m
    MaxConnIdleTime: 2m
    Options: ""
    User:
      SSL:
        Mode: require
    Admin:
      SSL:
        Mode: require

So ATM we use the default pool sizes.

Is your zitadel in the same region as the db?

F

FFO•10/15/24, 2:11 PM

48vCPU and 96GB of ram

48vCPU and 96GB of ram

48vCPU and 96GB of ram

48vCPU and 96GB of ram well that is a big machine... we see good latencies already from 4cpus/16GB mem

F

FFO•10/15/24, 3:13 PM

Uhm these authrequest lookups should be way faster

F

FFO•10/15/24, 3:13 PM

Do you have the query analyzer enabled on GCP?

ZZGRDaniel We're in a similar boat. Without performance at scale, we aren't going to be ab...

F

FFO•10/16/24, 4:55 PM

Let me assure you, it is top priority

We have 2 full time engineers just on this topic.

We also made good progress and have a test that pulls through 1200/rps on less then 6 cores

A

adlerhurst•10/17/24, 8:18 AM

Hey there if you have any loadtests you can share i would be happy to talk about it to further understand how you plan to use zitadel

A

adlerhurst•10/17/24, 1:08 PM

Great thanks for your help