tfmm•10mo ago

All of a sudden zitadel console giving "error_description":"Errors.App.NotFound"

Not sure what changed, but when trying to access the console, I get this error: {"error":"invalid_request","error_description":"Errors.App.NotFound"} I did update the container image recently, but not sure if that was related or not. The console app is listed twice in my projections.app7 postgres table. I attempted removing one, but there was no change. I'm going to try to restore the deleted one and delete the other, but figured i'd as for some help here while I work on that.

24 Replies

tfmmOP•10mo ago

restoring and deleting the other console app in the apps7 table had no affect I am also now noticing there are 2 copies of the various *-API apps in the table.

FFO•10mo ago

Can you share a little how you deployed zitadel and what version you are using?

tfmmOP•10mo ago

deployed via docker compose. version is 2.65.0 i did do a restore from my previous cockroach-db based install as the migration was not working for me i'm assuming this is something to do with having restored my old instance to this one, but i'm not sure where to go to fix it. do i need to delete all of the -API and console app references? if so from what db tables to be sure everything is properly cleaned up? downgrading to 2.64.2 has no affect should there be something in one of the apps__configs tables for the console? so doing some DB Digging, it appears the console app id is the older one of the 2 that exist in my apps table, it is enabled as far as i can see, and it does exist. I'm unsure what they're used for, but the sequence numbers are different in the instances and apps tables

FFO•10mo ago

Ok, so you use CRDB, right? Can you share the stdout logs from Zitadel please.

tfmmOP•10mo ago

no, using postgres now. was using CRDB prior to october, but console worked after the re-spin and switch to postgres and yes, let me grab the logs

tfmmOP•10mo ago

message.txt

FFO•10mo ago

Hm how do you compose args look like? I got the feeling that some migration were not run

tfmmOP•10mo ago

that's my assumption too compose file is this:

services:
  zitadel:
    restart: 'always'
    image: 'ghcr.io/zitadel/zitadel:latest'
    command: 'start-from-init --masterkey "$ZITADEL_MASTERKEY" --tlsMode external'
    environment:
      - ZITADEL_DATABASE_POSTGRES_HOST=db
      - ZITADEL_DATABASE_POSTGRES_PORT=5432
      - ZITADEL_DATABASE_POSTGRES_DATABASE=$ZITADEL_DB_NAME
      - ZITADEL_DATABASE_POSTGRES_USER_USERNAME=$ZITADEL_DB_USERNAME
      - ZITADEL_DATABASE_POSTGRES_USER_PASSWORD=$ZITADEL_DB_PASSWORD
      - ZITADEL_DATABASE_POSTGRES_USER_SSL_MODE=disable
      - ZITADEL_DATABASE_POSTGRES_ADMIN_USERNAME=$ZITADEL_DB_ADMIN_USERNAME
      - ZITADEL_DATABASE_POSTGRES_ADMIN_PASSWORD=$ZITADEL_DB_ADMIN_PASSWORD
      - ZITADEL_DATABASE_POSTGRES_ADMIN_SSL_MODE=disable
      - ZITADEL_EXTERNALSECURE=true
      - ZITADEL_EXTERNALDOMAIN=sso.redacted.co
      - ZITADEL_EXTERNALPORT=443
    depends_on:
      db:
        condition: 'service_healthy'
    ports:
      - '60110:8080'

  db:
    restart: 'always'
    image: postgres:16-alpine
    environment:
      - POSTGRES_USER=$ZITADEL_DB_ADMIN_USERNAME
      - POSTGRES_PASSWORD=$ZITADEL_DB_ADMIN_PASSWORD
      - POSTGRES_DB=$ZITADEL_DB_NAME
      - PGUSER=postgres
    healthcheck:
      test: ["CMD-SHELL", "pg_isready", "-d", "zitadel", "-U", "postgres"]
      interval: '10s'
      timeout: '30s'
      retries: 5
      start_period: '20s'
    volumes:
      - /containers/data/zitadel/postgres:/var/lib/postgresql/data

services:
  zitadel:
    restart: 'always'
    image: 'ghcr.io/zitadel/zitadel:latest'
    command: 'start-from-init --masterkey "$ZITADEL_MASTERKEY" --tlsMode external'
    environment:
      - ZITADEL_DATABASE_POSTGRES_HOST=db
      - ZITADEL_DATABASE_POSTGRES_PORT=5432
      - ZITADEL_DATABASE_POSTGRES_DATABASE=$ZITADEL_DB_NAME
      - ZITADEL_DATABASE_POSTGRES_USER_USERNAME=$ZITADEL_DB_USERNAME
      - ZITADEL_DATABASE_POSTGRES_USER_PASSWORD=$ZITADEL_DB_PASSWORD
      - ZITADEL_DATABASE_POSTGRES_USER_SSL_MODE=disable
      - ZITADEL_DATABASE_POSTGRES_ADMIN_USERNAME=$ZITADEL_DB_ADMIN_USERNAME
      - ZITADEL_DATABASE_POSTGRES_ADMIN_PASSWORD=$ZITADEL_DB_ADMIN_PASSWORD
      - ZITADEL_DATABASE_POSTGRES_ADMIN_SSL_MODE=disable
      - ZITADEL_EXTERNALSECURE=true
      - ZITADEL_EXTERNALDOMAIN=sso.redacted.co
      - ZITADEL_EXTERNALPORT=443
    depends_on:
      db:
        condition: 'service_healthy'
    ports:
      - '60110:8080'

  db:
    restart: 'always'
    image: postgres:16-alpine
    environment:
      - POSTGRES_USER=$ZITADEL_DB_ADMIN_USERNAME
      - POSTGRES_PASSWORD=$ZITADEL_DB_ADMIN_PASSWORD
      - POSTGRES_DB=$ZITADEL_DB_NAME
      - PGUSER=postgres
    healthcheck:
      test: ["CMD-SHELL", "pg_isready", "-d", "zitadel", "-U", "postgres"]
      interval: '10s'
      timeout: '30s'
      retries: 5
      start_period: '20s'
    volumes:
      - /containers/data/zitadel/postgres:/var/lib/postgresql/data

FFO•10mo ago

Hm that looks right, its kind of weird to not see the migrations at statup of zitadel This is really weird

tfmmOP•10mo ago

so i just noticed in an old db backup, there is a config for the console in the apps_oidc_configs table, but they are not present if i do a select now on that table should i re-add that to the db?

FFO•10mo ago

you can give it a try 😄 Its broken already an other idea would be to try the migration cleanup

tfmmOP•10mo ago

how do i trigger that?

FFO•10mo ago

I think it was somehting like 'setup --masterkey "$ZITADEL_MASTERKEY" cleanup' So it should be an arg from the zitadel binary

tfmmOP•10mo ago

thanks, i have to run, but i will try that later and report back. I appreciate the help

FFO•10mo ago

Happy to assist!

tfmmOP•10mo ago

cleanup didn't appear to do the trick, going to try restoring the oidc table tomorrow.

time="2024-11-15T02:51:59Z" level=info msg="cleanup started" caller="/home/runner/work/zitadel/zitadel/cmd/setup/cleanup.go:33"
time="2024-11-15T02:51:59Z" level=info msg="there is no stuck migration please run `zitadel setup`" caller="/home/runner/work/zitadel/zitadel/cmd/setup/cleanup.go:48"

time="2024-11-15T02:51:59Z" level=info msg="cleanup started" caller="/home/runner/work/zitadel/zitadel/cmd/setup/cleanup.go:33"
time="2024-11-15T02:51:59Z" level=info msg="there is no stuck migration please run `zitadel setup`" caller="/home/runner/work/zitadel/zitadel/cmd/setup/cleanup.go:48"

restoring the missing oidc_configs entries also did not resolve. 😦 still the same errors in the zitadel container log:

time="2024-11-15T13:23:59Z" level=info msg="server is listening on [::]:8080" caller="/home/runner/work/zitadel/zitadel/cmd/start/start.go:570"
time=2024-11-15T13:24:09.633Z level=WARN msg="request error" oidc_error.parent="ID=QUERY-wu6Ee Message=Errors.App.NotFound Parent=(sql: no rows in result set)" oidc_error.description=Errors.App.NotFound oidc_error.type=invalid_request status_code=400
2024/11/15 13:24:09 ERROR: Failed to extract ServerMetadata from context
time=2024-11-15T13:24:10.822Z level=WARN msg="request error" oidc_error.parent="ID=QUERY-wu6Ee Message=Errors.App.NotFound Parent=(sql: no rows in result set)" oidc_error.description=Errors.App.NotFound oidc_error.type=invalid_request status_code=400
2024/11/15 13:24:10 ERROR: Failed to extract ServerMetadata from context
time=2024-11-15T13:24:16.055Z level=WARN msg="request error" oidc_error.parent="ID=QUERY-wu6Ee Message=Errors.App.NotFound Parent=(sql: no rows in result set)" oidc_error.description=Errors.App.NotFound oidc_error.type=invalid_request status_code=400

time="2024-11-15T13:23:59Z" level=info msg="server is listening on [::]:8080" caller="/home/runner/work/zitadel/zitadel/cmd/start/start.go:570"
time=2024-11-15T13:24:09.633Z level=WARN msg="request error" oidc_error.parent="ID=QUERY-wu6Ee Message=Errors.App.NotFound Parent=(sql: no rows in result set)" oidc_error.description=Errors.App.NotFound oidc_error.type=invalid_request status_code=400
2024/11/15 13:24:09 ERROR: Failed to extract ServerMetadata from context
time=2024-11-15T13:24:10.822Z level=WARN msg="request error" oidc_error.parent="ID=QUERY-wu6Ee Message=Errors.App.NotFound Parent=(sql: no rows in result set)" oidc_error.description=Errors.App.NotFound oidc_error.type=invalid_request status_code=400
2024/11/15 13:24:10 ERROR: Failed to extract ServerMetadata from context
time=2024-11-15T13:24:16.055Z level=WARN msg="request error" oidc_error.parent="ID=QUERY-wu6Ee Message=Errors.App.NotFound Parent=(sql: no rows in result set)" oidc_error.description=Errors.App.NotFound oidc_error.type=invalid_request status_code=400

FFO•10mo ago

Hmm can you try the following start-from-setup --init-projections=true (add the rest of your arguments)

tfmmOP•10mo ago

sure, give me a couple no change, same behavior and log entries just as another point of reference, this appears to have only broken the console itself, i can still use other apps just fine, which is the only thing making me hesitate blowing the whole thing up and re-configuring it. well that and i don't want to have to deal with resetting the few user passwords that are setup.

FFO•10mo ago

Hm this is really weird. Do you in the startup logs from zitadel see that migrations are being checked/run?

tfmmOP•10mo ago

yes, and i don't notice any failures, but let me dig through them just to be sure i didn't miss anything yeah, i don't see any failures is there a way to get a backup of the users and apps without being able to acquire and admin token? at this point i'm ok with having to reconfigure the rest, but would prefer to not have to nuke my users/apps config

FFO•10mo ago

Hm let us fix your install @adlerhurst or @livio might have a clue what fails

tfmmOP•10mo ago

ok, i'm definitely open to that, the less work the better haha. if you need or want any database content or anything, just let me know i do see several duplicate tables for example user_auth_methods4 and user_auth_methods5, which is firming up the idea of migrations not doing something properly in my opinion i just have no clue how to fix it Holy crap i found the issue. No idea what caused it, but i had 2 orgs in the projections.orgs table, one was set to disabled. I enabled it, and now i can get into console. the disabled org was the one created when I did a fresh install, and then did a restore from my old crdb-based backup. apparently you can't disable the default org? @FFO thanks again for the help sorting through this.

FFO•10mo ago

Oh nice catch. Yeah the default project can/should not be disabled since it holds console and the apis. I wonder though how that happend 😄

tfmmOP•10mo ago

I probably did it trying to clean up after the import, i 100% blame myself 😄

All of a sudden zitadel console giving "error_description":"Errors.App.NotFound"

Did you find this page helpful?