Dismal performance of Users API V2
We have been using the Management API gRPC endpoints for some time and recently decided to try the Users API v2, primarily because of the organization filter. However, the difference in performance is so significant that we find it unusable.
The
management.v1.ListUsers call takes roughly 1.5 seconds to return 9,800+ results (unpaginated response), whereas users.v2.ListUsers takes 54–56 seconds to complete the same operation. What gives?21 Replies
Uhm that sounds unhealthy. Can you share some insights on your infra?
Like DB type/size and connection settings for the DB connection?
Oh yeah, sorry, should have included it in the original post.
we are running CockroachDB 23.1.9 and ZITADEL 2.60.2. Cockroach has three instances with 4 vCPU and 24 GB RAM.
Connection options:
MaxOpenConns: 200
MaxIdleConns: 70
MaxConnLifetime: 30m
MaxConnIdleTime: 5m
Thanks for the additional info. Can you check the cockroach query view on the most expensive queries?
The connection settings are on the higher end of what we usually see.
Sure, I'll get back to you shortly, I'll have to run the test again
This kind of sounds like an index issue to me 😄
yeah, most likely, although we didn't change the filter parameters (except for the organization) so such sudden change is unexpected.
I've tried without an org filter as well, same result. My initial assumption was that maybe we forgot to run
zitadel setup or something like that.I quickly checked out internal logs on postgres and have not seen a lot of high latency cases
Unknown User•7mo ago
Message Not Public
Sign In & Join Server To View
Ok thanks for the added info
Did you also test a recent version like 2.65.x?
Unknown User•7mo ago
Message Not Public
Sign In & Join Server To View
Perfect can you share your test?
That certainly helps us
Unknown User•7mo ago
Message Not Public
Sign In & Join Server To View
fwiw, the slowest query that I could find. Doesn't look like a datastore problem, as @Luka mentioned
haha no worries
Let us investigate and come back
Unknown User•7mo ago
Message Not Public
Sign In & Join Server To View
We are still investigating this, but we think it could have something todo with either reading the instance config too many times or the authorization being checked excessively
In version 2.65.x we have a cache that might change the behavior.
You could try adding a instance cache in memory and see how that goes
https://zitadel.com/docs/self-hosting/manage/cache#examples
Unknown User•7mo ago
Message Not Public
Sign In & Join Server To View
Thank you so much, I appreciate it. Let me know if I can help somehow
Unknown User•7mo ago
Message Not Public
Sign In & Join Server To View
Thanks for the heads up, looks like I'll be stuck with management api for a while 🙂
Unknown User•5mo ago
Message Not Public
Sign In & Join Server To View