boblack_zocker•4mo ago

Performance breaks by unknown reasons

Hello, our production self-hosted zitadel is doing misterious things in the kubernetes. Out of nowhere he start big read/write operations and by that it stops working for about 5minutes till the process is done and then it functions normally again. is it doing some kind of maintance in the background or why is it doing that?

8 Replies

Raccine•4mo ago

Hey @boblack_zocker! Thanks for reaching out - Let me loop in an engineer to look into this for you! ☺️

boblack_zockerOP•4mo ago

Thanks, is it possible to get an answer today? our production is having some outages because of this :). thanks

Rajat Singh•4mo ago

@boblack_zocker apologies for what you are facing we are still looking into this , we will get back to you.

adlerhurst•4mo ago

Do you see what statements are executed?

boblack_zockerOP•4mo ago

@adlerhurst what do you mean? i have some errorlogs but not more its running in its own pod in kubernetes it started with this logs: 2025-02-24 11:10:56.494 time="2025-02-24T10:10:56Z" level=info msg="process events failed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/v2/handler.go:413" error="context deadline exceeded" projection=projections.instance_domains2025-02-24 11:10:56.310
time="2025-02-24T10:10:56Z" level=info msg="process events failed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/v2/handler.go:413" error="context deadline exceeded" projection=projections.device_auth_requests2 2025-02-24 11:10:56.096
time="2025-02-24T10:10:55Z" level=info msg="query failed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/repository/sql/query.go:129" error="timeout: context already done: context deadline exceeded" 2025-02-24 11:10:56.074
time="2025-02-24T10:10:55Z" level=info msg="query failed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/repository/sql/query.go:129" error="context deadline exceeded" is this a problem with zitadel or the database of zitadel?

adlerhurst•4mo ago

I mean from the database some query insights. Can you share your zitadel config, im mostly interested in the projections part

boblack_zockerOP•4mo ago

we got the default.yaml zitadel config what exacly do you need to see on the query insights?

adlerhurst•4mo ago

it looks to me like the statements take too long. What you can try is to set the following variables: to disable the timeouts

Projections:
  TransactionDuration: 0
Auth:
  Spooler:
    TransactionDuration: 0 #ZITADEL_AUTH_SPOOLER_TRANSACTIONDURATION

Admin:
  Spooler:
    TransactionDuration: 10s

Projections:
  TransactionDuration: 0
Auth:
  Spooler:
    TransactionDuration: 0 #ZITADEL_AUTH_SPOOLER_TRANSACTIONDURATION

Admin:
  Spooler:
    TransactionDuration: 10s

Performance breaks by unknown reasons

Did you find this page helpful?