SystemUser Works for Debug Stock Image but not Non-Debug Image

There's a lot of issues using the SystemUser.

I'm trying to upgrade our self-hosted instance of Zitadel from 2.71.10 (self-compiled source code) to 2.71.17 (v2:latest as of writing). However, I'd like to use the stock image if possible.

I'm noticing that

ghcr.io/zitadel/zitadel:v2.71.17-debug

ghcr.io/zitadel/zitadel:v2.71.17-debug

ghcr.io/zitadel/zitadel:v2.71.17-debug

ghcr.io/zitadel/zitadel:v2.71.17-debug works fine as expected. However,

ghcr.io/zitadel/zitadel:v2.71.17

ghcr.io/zitadel/zitadel:v2.71.17

ghcr.io/zitadel/zitadel:v2.71.17

ghcr.io/zitadel/zitadel:v2.71.17 does not, giving the same errors other community members have reported relating to SystemUsers:

rpc error: code = Unauthenticated desc = Errors.Token.Invalid (AUTH-7fs1e)

rpc error: code = Unauthenticated desc = Errors.Token.Invalid (AUTH-7fs1e)

rpc error: code = Unauthenticated desc = Errors.Token.Invalid (AUTH-7fs1e)

rpc error: code = Unauthenticated desc = Errors.Token.Invalid (AUTH-7fs1e)

time="2025-10-17T18:45:31Z" level=info msg=activity caller="/home/runner/work/zitadel/zitadel/internal/activity/activity.go:105" domain="https://{{mydomain}}:443" grpcStatus=0 httpStatus=200 instance=322930458348224793 isSystemUser=false method=/zitadel.admin.v1.AdminService/Healthz org= path= requestMethod= trigger=resourceAPI user=

time="2025-10-17T18:45:31Z" level=info msg=activity caller="/home/runner/work/zitadel/zitadel/internal/activity/activity.go:105" domain="https://{{mydomain}}:443" grpcStatus=0 httpStatus=200 instance=322930458348224793 isSystemUser=false method=/zitadel.admin.v1.AdminService/Healthz org= path= requestMethod= trigger=resourceAPI user=

time="2025-10-17T18:45:31Z" level=info msg=activity caller="/home/runner/work/zitadel/zitadel/internal/activity/activity.go:105" domain="https://{{mydomain}}:443" grpcStatus=0 httpStatus=200 instance=322930458348224793 isSystemUser=false method=/zitadel.admin.v1.AdminService/Healthz org= path= requestMethod= trigger=resourceAPI user=

time="2025-10-17T18:45:31Z" level=info msg=activity caller="/home/runner/work/zitadel/zitadel/internal/activity/activity.go:105" domain="https://{{mydomain}}:443" grpcStatus=0 httpStatus=200 instance=322930458348224793 isSystemUser=false method=/zitadel.admin.v1.AdminService/Healthz org= path= requestMethod= trigger=resourceAPI user=

Notably:

isSystemUser=false

isSystemUser=false

isSystemUser=false

isSystemUser=false

Using the -debug flavour:

time="2025-10-17T20:03:12Z" level=info msg=activity caller="/home/runner/work/zitadel/zitadel/internal/activity/activity.go:105" domain="{{mydomain}}:443" grpcStatus=0 httpStatus=200 instance=322930458348224793 isSystemUser=true method=/zitadel.admin.v1.AdminService/ListOrgs org= path= requestMethod= trigger=resourceAPI user=superuser

time="2025-10-17T20:03:12Z" level=info msg=activity caller="/home/runner/work/zitadel/zitadel/internal/activity/activity.go:105" domain="{{mydomain}}:443" grpcStatus=0 httpStatus=200 instance=322930458348224793 isSystemUser=true method=/zitadel.admin.v1.AdminService/ListOrgs org= path= requestMethod= trigger=resourceAPI user=superuser

time="2025-10-17T20:03:12Z" level=info msg=activity caller="/home/runner/work/zitadel/zitadel/internal/activity/activity.go:105" domain="{{mydomain}}:443" grpcStatus=0 httpStatus=200 instance=322930458348224793 isSystemUser=true method=/zitadel.admin.v1.AdminService/ListOrgs org= path= requestMethod= trigger=resourceAPI user=superuser

time="2025-10-17T20:03:12Z" level=info msg=activity caller="/home/runner/work/zitadel/zitadel/internal/activity/activity.go:105" domain="{{mydomain}}:443" grpcStatus=0 httpStatus=200 instance=322930458348224793 isSystemUser=true method=/zitadel.admin.v1.AdminService/ListOrgs org= path= requestMethod= trigger=resourceAPI user=superuser

isSystemUser=true

isSystemUser=true

isSystemUser=true

isSystemUser=true

This makes me think there is an issue with the system user keys we're using, but I'm not clear how this would differ between the debug and non-debug flavour. Working with the non-debug flavour is difficult because it does not have a shell I can use to look around.

Ask Inkeep•10/17/25, 6:48 PM

Hi @Billy, I'm an AI assistant for ZITADEL. While a team member reviews this question, I'll see if I can find relevant sources for your question.

Ask Inkeep•10/17/25, 6:49 PM

Answer

I wasn't able to find a direct answer, here are some relevant sources:

Discord message from @sirchrispbacon on 2025-09-17 : This shows similar SystemUser authentication issues with the error "invalid signature (invalid signature: no possible keys matches) " and suggests ensuring the public key in ZITADEL matches the private key used to sign the JWT (1) .
questions-help-bugs-debug vs non -debug Zitadel images on ghcr : The -debug images include a shell and debugging tools while non-debug images are minimal and lack these tools, but this doesn't explain authentication differences questions-help-bugs-debug vs non -debug Zitadel images on ghcr .

Ask Inkeep•10/17/25, 6:49 PM

To learn more, see the sources I considered:

[1] Discord message from @sirchrispbacon on 2025-09-17
questions-help-bugs-debug vs non -debug Zitadel images on ghcr

Ask Inkeep•10/17/25, 6:49 PM

Feel free to tag @Ask Inkeep with additional questions.

AAsk Inkeep To learn more, see the sources I considered: [[1] Discord message from @sirch...

Ask InkeepAPP•10/17/25, 6:49 PM

Got it. Tagging @Rajat for additional help.

nullsenseOP•10/17/25, 8:09 PM

Following the steps here, I've made a manual request to my Zitadel instance using curl:

curl --request POST \
  --url "$AUDIENCE/system/v1/instances/_search" \
  --header "Authorization: Bearer $jwt_token" \
  --header "Content-Type: application/json"

curl --request POST \
  --url "$AUDIENCE/system/v1/instances/_search" \
  --header "Authorization: Bearer $jwt_token" \
  --header "Content-Type: application/json"

With the debug flavour, I get the correct happy response:

{
  "details":{
    "totalResult":"2"
  },
  "result":[
    {
      "id":"322930458348224793",
      "details":{
        "sequence":"29",
        "creationDate":"2025-06-03T19:05:52.744139Z",
        "changeDate":"2025-06-03T19:05:52.744139Z",
        "resourceOwner":"322930458348224793"
      },
      "state":"STATE_RUNNING",
      "name":"{{name}}",
      "version":"v2.71.17",
      "domains":[
        {
          "details":{
            "sequence":"33",
            "creationDate":"2025-06-03T19:05:52.744139Z",
            "changeDate":"2025-06-03T19:05:52.744139Z",
            "resourceOwner":"322930458348224793"
          },
          "domain":"{{name}}-pwx4tp.{{mydomain}}",
          "generated":true
        },
        {
          "details":{
            "sequence":"33",
            "creationDate":"2025-06-03T19:05:52.744139Z",
            "changeDate":"2025-06-03T19:05:52.744139Z",
            "resourceOwner":"322930458348224793"
          },
          "domain":"{{mydomain}}",
          "primary":true
        }
      ]
    }
  ]
}

{
  "details":{
    "totalResult":"2"
  },
  "result":[
    {
      "id":"322930458348224793",
      "details":{
        "sequence":"29",
        "creationDate":"2025-06-03T19:05:52.744139Z",
        "changeDate":"2025-06-03T19:05:52.744139Z",
        "resourceOwner":"322930458348224793"
      },
      "state":"STATE_RUNNING",
      "name":"{{name}}",
      "version":"v2.71.17",
      "domains":[
        {
          "details":{
            "sequence":"33",
            "creationDate":"2025-06-03T19:05:52.744139Z",
            "changeDate":"2025-06-03T19:05:52.744139Z",
            "resourceOwner":"322930458348224793"
          },
          "domain":"{{name}}-pwx4tp.{{mydomain}}",
          "generated":true
        },
        {
          "details":{
            "sequence":"33",
            "creationDate":"2025-06-03T19:05:52.744139Z",
            "changeDate":"2025-06-03T19:05:52.744139Z",
            "resourceOwner":"322930458348224793"
          },
          "domain":"{{mydomain}}",
          "primary":true
        }
      ]
    }
  ]
}

With the non-debug flavour, I get the following error:

{"code":16, "message":"Errors.Token.Invalid (AUTH-7fs1e)", "details":[{"@type":"type.googleapis.com/zitadel.v1.ErrorDetail", "id":"AUTH-7fs1e", "message":"Errors.Token.Invalid"}]}

{"code":16, "message":"Errors.Token.Invalid (AUTH-7fs1e)", "details":[{"@type":"type.googleapis.com/zitadel.v1.ErrorDetail", "id":"AUTH-7fs1e", "message":"Errors.Token.Invalid"}]}

nullsenseOP•10/17/25, 8:16 PM

I'm really not sure what I'm missing. As noted with the bot, the debug flavour should only include extra debugging tools (shell and busybox). I'm not sure why there is such a big behaviour difference.

nullsenseOP•10/21/25, 11:39 AM

@Matías @elina_shoko bump

Rajat•10/21/25, 1:57 PM

hey @nullsense thanks for the bump I will take a look at it and will get back to you

nullsenseOP•10/23/25, 12:57 PM

Any update @Rajat? Let me know if you need any new information

Rajat•10/23/25, 2:03 PM

hey @Billy I will look into it today, soo many new questions piled up in the last 2 days or so

Rajat•10/23/25, 2:48 PM

hey @Billy two things here.

Toue zitadel version is VERY OLD, if you look at our roadmap the latest version is v4.x.xv4.x.x so consider upgrading to solve most of the problem.

Add the block explicitly when you start the non-debug container.

# zitadel.yaml
SystemAPIUsers:
superUser:
 KeyData: "<your ed25519 public key>"

# zitadel.yaml
SystemAPIUsers:
superUser:
 KeyData: "<your ed25519 public key>"

nullsenseOP•10/23/25, 2:50 PM

Thanks @Rajat

Toue zitadel version is VERY OLD, if you look at our roadmap the latest version is v4.x.x so consider upgrading to solve most of the problem.

This work is part of that. However, I've seen this behaviour with the v4.x.x images as well (debug vs non-debug)

Add the block explicitly when you start the non-debug container.

Does it not support file path in the config? (That's what I currently use)

SystemAPIUsers:
  - superuser:
      Path: <keyName>.pub
      Memberships:
        - MemberType: System
          Roles:
            - "SYSTEM_OWNER"
            - "IAM_OWNER"
            - "ORG_OWNER"

SystemAPIUsers:
  - superuser:
      Path: <keyName>.pub
      Memberships:
        - MemberType: System
          Roles:
            - "SYSTEM_OWNER"
            - "IAM_OWNER"
            - "ORG_OWNER"

I'm not convinced why this would be a difference in the debug vs non-debug image flavours either...

Nnullsense Thanks @Rajat > Toue zitadel version is VERY OLD, if you look at our roadmap th...

Rajat•10/23/25, 8:08 PM

you are not wrong, I am trying to suggest various things that ight work, it could be possible that its an indent issue, here's a similar answer on github that helped the user stuck in the somewhat similar issue like yours because the main difference you're experiencing between debug and non-debug images likely relates to how system user authentication is handled in the minimal scratch-based container versus the debug container that includes additional tools.

nullsenseOP•10/23/25, 8:41 PM

@Rajat
Ok I tried that suggestion out and unfortunately the extra indentation behaves similarly as to if I remove the KeyData or Path line completely.