FWe are currently working a lot on some performance improvements. This is the issue we are currently working on: https://github.com/zitadel/zitadel/issues/8352
Fservice users 1000rps
Fthe write to the eventstore.events can't be disabled, this is basically the source of trhough for ZITADEL where all the data is stored.
FZITADEL is based on evensourcing so everything that happens results in events in that table
FI was working with Tim on the 1000/rps benchmark recently and we with 4CPU Zitadel and a 4CPU/16GB Postgres we get to around 600/rps but we identified a small bottleneck in the milestones which should bring us to 1000/rps
FHappy to share more insights as they come around 
FWe have not seen problems with a large events table in itself.
Only some queries made us some pain. Which had some index problems
Only some queries made us some pain. Which had some index problems
FSo cleanup is not yet on that table available, but I would not worry too much ATM
FCan you both share what calls you want to achieve 1000/rps?
FI think this one was resolved in 2.63 with a feature flag.
https://github.com/zitadel/zitadel/releases/tag/v2.63.0
https://github.com/zitadel/zitadel/releases/tag/v2.63.0
FFYou can try enable with this api here for an instance https://zitadel.com/docs/apis/resources/feature_service_v2/feature-service-set-instance-features
FdisableUserTokenEvent
FCan you share results? And what exactly you are doing?
FGot it, with client_credentials it could well be that your zitadel also bottlenecks with CPU.
How does your setup look like and what stats you see in app/db
How does your setup look like and what stats you see in app/db
FHm... since you use one service user it could be that old events clog things up.
FNo lets try something else first.
We could delete the old event that clog things up with
But please only do this when you enabled the flag I mentioned before
We could delete the old event that clog things up with
But please only do this when you enabled the flag I mentioned before
FAnd after you have done a backup or use PITR or are able to recreate the DB
FThis should take some pressure of the DB I would say since the old events do not linger around anymore
FWe usually only recommend leader/follower for HA purpose.
ATM we do not support read replicas but will release a cache interface soon (in a few days) for PG unlogged tables and after that for redis (valkey, ...)
ATM we do not support read replicas but will release a cache interface soon (in a few days) for PG unlogged tables and after that for redis (valkey, ...)
S@FFO this cache implementation going to be used by zitadel cloud too or just for self hosters?
FThis will be available in the oss code.
Sthe only endpoint I really have latency concerns with is the introspection one, since it's in the critical latency sensitive happy path for most of my calls - the other api calls, while it'd be nice if they're fast, I'm not going to notice 20ms shaved off a list all users in my org call as much
Sso quite looking forward to any improvements in this area, it's ~100ms latency penalty currently for me doing an introspection lookup on the opaque tokens
SYeah basically done the same with wrapping the introspection call with a simple in memory ttl cache with 60s expiry. I hope the official sdk implementation can just internalise this as an option
S100ms from aws (eks for us) out to the internet and into gcp (and likely a different region) and then a processing latency and back is probably expected behaviour tbh - not much else I can do on that
FDo you have a specific scenario in regard of scaling on which we can try to create a precise answer?
Our cloud service and some fairly big customers so far have not had problems as we constantly improve our business logic and the indexes for better scaling.
What we certainly at one point need to add is an archiving process (basically making a snapshopt of an aggregate and move the old events to a bulk storage)
Our cloud service and some fairly big customers so far have not had problems as we constantly improve our business logic and the indexes for better scaling.
What we certainly at one point need to add is an archiving process (basically making a snapshopt of an aggregate and move the old events to a bulk storage)
FWhat inputs are you looking for?
J@FFO Hey, I also have a question about the growth of the events table. What strategies do you have for dealing with this growth? Currently we have around ~10 different Orgs, 7k users and the DB grows quickly. This amount of Orgs / users is a small fraction of what we want to have when we complete the migration to Zitadel. Currently we are on v2.71.9 (self-hosted) and custom Hosted Login app. Does the upgrade to Zitadel v3.x.x or v4.x.x change anything in this matter? Does disableUserTokenEvent option still make any difference?
RVery interested how others are dealing with this issue and if the updrage to V4.x.x wil change this behavior or if the disableUserTokenEvent option wil make any difference for us.
FHey, we are actually working on a big change for this... starting with v5 we will start switching to a relational model BUT! We will keep the events in a seperate table for audit purposes but you will be able to define retention for that table more easily