Construcker•2d ago

Whole environment is wiped? Emergency

Use-case: A fullstack project in web (Saas) Environment: Self hosting Version: 3.2.2 Stack: Using it for access and user control for authentication, using docker and Azure What you expected to happen: I was kicked out of my login with a {"error":"invalid_request","error_description":"Errors.App.NotFound"} error What went wrong: All users and projects in the zitadel project is gone, unknown cause Attachments: {"error":"invalid_request","error_description":"Errors.App.NotFound"} Only the ZITADEL Admin user remains, but our projects, users, settings and everything is missing How can this be possible that it's all wiped? Are there any solution for a backup? Sorry for not a fully flavoured post, wipe of users has me checking logs and everything to figure out why it's been wiped 😅 Thank you for your time!

3 Replies

Ask Inkeep•2d ago

Hi @Construcker, I'm an AI assistant for ZITADEL. While a team member reviews this question, I'll see if I can find relevant sources for your question. Answer I understand this is an emergency situation. Let me address your backup and recovery options based on the available information. Backup and Recovery Options Database-Level Recovery Since ZITADEL is event sourced, the table eventstore.events contains all the records needed to reconstruct the system state (1) . The timestamp of your last record in the events table defines up to which point in time ZITADEL can restore its state . For database restoration, you can use standard PostgreSQL backup procedures, though there are some considerations. One user noted that when using pg_dump and restoring with psql zitadel < backup.sql, migration issues can occur because certain elements already exist (2) . Alternative Recovery Methods There are two potential approaches mentioned in the community: 1. Import/Export API: ZITADEL has an admin service for importing data: https://zitadel.com/docs/apis/resources/admin/admin-service-import-data 2. System User Approach: You can create a System User with appropriate roles (IAM_OWNER or ORG_OWNER) and use the Management API to restore data (3) . This involves creating a JWT and calling the user API to update users . Immediate Steps Since you still have the ZITADEL Admin user, you should be able to create new organizations and projects through the Console UI. However, this won't restore your lost data. Prevention for the Future For production setups, the documentation emphasizes proper backup strategies (1) . When designing your backup strategy, focus on the eventstore.events table as it contains the complete system state . I'd recommend c licking the "Ask for help" button to tag a team member - this situation requires immediate attention from the ZITADEL team to help investigate what caused the data loss and explore recovery options specific to your setup. To learn more, see the sources I considered: [1] ZITADEL Production Setup [2] Restoring backup DB doesn't work [3] How to regain access in a self hosted instance? Feel free to tag @Ask Inkeep with additional questions.

ConstruckerOP•2d ago

@Ask Inkeep How could docker affect the data wipe? One of my devs suggested it might have been running in a docker instance, so when the environment was restarted it wiped the data

Ask Inkeep•2d ago

Thanks for the follow-up, @Construcker. I'll get back to you soon. Answer Your developer is absolutely correct - Docker volume configuration is a critical factor that could cause data loss in ZITADEL deployments. Docker Volume Configuration Issues The official Docker Compose configuration includes persistent volumes for the database (1) :

volumes:
  data:

volumes:
  data:

However, if your setup doesn't properly configure persistent volumes, restarting the Docker environment would indeed wipe all data. The database volume must be properly configured to persist data across container restarts . Common Docker Data Persistence Problems A community discussion specifically addresses this concern about persisting configuration and data using Docker (2) . The key issue is that without proper volume configuration, Docker containers are ephemeral by nature - when they're removed and recreated, any data stored inside the container filesystem is lost. For PostgreSQL in the official example, the volume is configured as (1) :

volumes:
  - 'data:/var/lib/postgresql/data:rw'

volumes:
  - 'data:/var/lib/postgresql/data:rw'

What Likely Happened If your Docker setup was missing proper volume mounts or using temporary volumes, restarting the environment would: 1. Destroy the existing containers 2. Create new containers with fresh filesystems
3. Result in a clean ZITADEL instance with only the default admin user Prevention Ensure your Docker Compose configuration includes persistent volumes for: - PostgreSQL data storage - Any ZITADEL configuration files - Machine keys and PAT files The official documentation emphasizes using proper volume configuration to avoid exactly this type of data loss scenario. I'd recommend clicking the "Ask for help" button to have a team member help you verify your Docker volume configuration and explore any potential recovery options from your Azure environment backups. To learn more, see the sources I considered: [1] Set up Zitadel with Docker Compose [2] Persisting config/data using Docker Feel free to tag @Ask Inkeep with additional questions.

Whole environment is wiped? Emergency

Did you find this page helpful?