Experiencing downtime

Error message

❯ curl https://login.blp-digital.ch/.well-known/openid-configuration                                                                                                                                                                                                                                                   6s
failed to connect to `host=internal-zitadel-cloud-gg8.gcp-europe-west6.cockroachlabs.cloud user=zitadel_prod database=prod`: server error (ERROR: internal error while retrieving user account memberships: relation "system.public.privileges" does not exist (SQLSTATE 28000))

❯ curl https://login.blp-digital.ch/.well-known/openid-configuration                                                                                                                                                                                                                                                   6s
failed to connect to `host=internal-zitadel-cloud-gg8.gcp-europe-west6.cockroachlabs.cloud user=zitadel_prod database=prod`: server error (ERROR: internal error while retrieving user account memberships: relation "system.public.privileges" does not exist (SQLSTATE 28000))

Happening with other instances, couldn't see anything on https://status.zitadel.com

ZITADEL Cloud Status

ZITADEL Cloud's status page provides information on the current status and incident history of ZITADEL Cloud.

5 Replies

DWalderOP•3y ago

Saw just now the status update

Unknown User•3y ago

Message Not Public

FFO•3y ago

Yeah our database had a severe outage. We will follow up with a post-mortem once we analysed the details with the vendor. Sorry for any impact we caused.

DWalderOP•3y ago

Related to this incident, we experienced another issue: - openID configuration can't be fetched due to Zitadel being down - frontend keeps retrying -> 429 from Google Cloud Armor - issue is fixed on Zitadel - takes a while for users to be able to login again since they keep being rejected with 429 We should probably implement a backoff on our frontend, just wanted to report the event

FFO•3y ago

Great observation, we definitely recommend implementing exponential backoff and retry. Once we collected all the data we need we will look into our rate limits as well. But I guess to some degree the helped us starting under load more easy 😎

Experiencing downtime

Did you find this page helpful?