nullsense
nullsense•4d ago

SystemUser Works for Debug Stock Image but not Non-Debug Image

There's a lot of issues using the SystemUser. I'm trying to upgrade our self-hosted instance of Zitadel from 2.71.10 (self-compiled source code) to 2.71.17 (v2:latest as of writing). However, I'd like to use the stock image if possible. I'm noticing that ghcr.io/zitadel/zitadel:v2.71.17-debug works fine as expected. However, ghcr.io/zitadel/zitadel:v2.71.17 does not, giving the same errors other community members have reported relating to SystemUsers: 
rpc error: code = Unauthenticated desc = Errors.Token.Invalid (AUTH-7fs1e)
rpc error: code = Unauthenticated desc = Errors.Token.Invalid (AUTH-7fs1e)
time="2025-10-17T18:45:31Z" level=info msg=activity caller="/home/runner/work/zitadel/zitadel/internal/activity/activity.go:105" domain="https://{{mydomain}}:443" grpcStatus=0 httpStatus=200 instance=322930458348224793 isSystemUser=false method=/zitadel.admin.v1.AdminService/Healthz org= path= requestMethod= trigger=resourceAPI user=
time="2025-10-17T18:45:31Z" level=info msg=activity caller="/home/runner/work/zitadel/zitadel/internal/activity/activity.go:105" domain="https://{{mydomain}}:443" grpcStatus=0 httpStatus=200 instance=322930458348224793 isSystemUser=false method=/zitadel.admin.v1.AdminService/Healthz org= path= requestMethod= trigger=resourceAPI user=
Notably: isSystemUser=false Using the -debug flavour: 
time="2025-10-17T20:03:12Z" level=info msg=activity caller="/home/runner/work/zitadel/zitadel/internal/activity/activity.go:105" domain="{{mydomain}}:443" grpcStatus=0 httpStatus=200 instance=322930458348224793 isSystemUser=true method=/zitadel.admin.v1.AdminService/ListOrgs org= path= requestMethod= trigger=resourceAPI user=superuser
time="2025-10-17T20:03:12Z" level=info msg=activity caller="/home/runner/work/zitadel/zitadel/internal/activity/activity.go:105" domain="{{mydomain}}:443" grpcStatus=0 httpStatus=200 instance=322930458348224793 isSystemUser=true method=/zitadel.admin.v1.AdminService/ListOrgs org= path= requestMethod= trigger=resourceAPI user=superuser
isSystemUser=true This makes me think there is an issue with the system user keys we're using, but I'm not clear how this would differ between the debug and non-debug flavour. Working with the non-debug flavour is difficult because it does not have a shell I can use to look around.
2 Replies
Ask Inkeep
Ask Inkeep•4d ago
Hi @Billy, I'm an AI assistant for ZITADEL. While a team member reviews this question, I'll see if I can find relevant sources for your question. Answer I wasn't able to find a direct answer, here are some relevant sources: 1. Discord message from @sirchrispbacon on 2025-09-17 : This shows similar SystemUser authentication issues with the error "invalid signature (invalid signature: no possible keys matches) " and suggests ensuring the public key in ZITADEL matches the private key used to sign the JWT (1) . 2. -debug vs non -debug Zitadel images on ghcr : The -debug images include a shell and debugging tools while non-debug images are minimal and lack these tools, but this doesn't explain authentication differences (2) . ​ To learn more, see the sources I considered: [1] Discord message from @sirchrispbacon on 2025-09-17 [2] -debug vs non -debug Zitadel images on ghcr ​ Feel free to tag @Ask Inkeep with additional questions. Got it. Tagging @Rajat for additional help.
nullsense
nullsenseOP•3d ago
Following the steps here, I've made a manual request to my Zitadel instance using curl: 
curl --request POST \
  --url "$AUDIENCE/system/v1/instances/_search" \
  --header "Authorization: Bearer $jwt_token" \
  --header "Content-Type: application/json"
curl --request POST \
  --url "$AUDIENCE/system/v1/instances/_search" \
  --header "Authorization: Bearer $jwt_token" \
  --header "Content-Type: application/json"
With the debug flavour, I get the correct happy response: 
{
  "details":{
    "totalResult":"2"
  },
  "result":[
    {
      "id":"322930458348224793",
      "details":{
        "sequence":"29",
        "creationDate":"2025-06-03T19:05:52.744139Z",
        "changeDate":"2025-06-03T19:05:52.744139Z",
        "resourceOwner":"322930458348224793"
      },
      "state":"STATE_RUNNING",
      "name":"{{name}}",
      "version":"v2.71.17",
      "domains":[
        {
          "details":{
            "sequence":"33",
            "creationDate":"2025-06-03T19:05:52.744139Z",
            "changeDate":"2025-06-03T19:05:52.744139Z",
            "resourceOwner":"322930458348224793"
          },
          "domain":"{{name}}-pwx4tp.{{mydomain}}",
          "generated":true
        },
        {
          "details":{
            "sequence":"33",
            "creationDate":"2025-06-03T19:05:52.744139Z",
            "changeDate":"2025-06-03T19:05:52.744139Z",
            "resourceOwner":"322930458348224793"
          },
          "domain":"{{mydomain}}",
          "primary":true
        }
      ]
    }
  ]
}
{
  "details":{
    "totalResult":"2"
  },
  "result":[
    {
      "id":"322930458348224793",
      "details":{
        "sequence":"29",
        "creationDate":"2025-06-03T19:05:52.744139Z",
        "changeDate":"2025-06-03T19:05:52.744139Z",
        "resourceOwner":"322930458348224793"
      },
      "state":"STATE_RUNNING",
      "name":"{{name}}",
      "version":"v2.71.17",
      "domains":[
        {
          "details":{
            "sequence":"33",
            "creationDate":"2025-06-03T19:05:52.744139Z",
            "changeDate":"2025-06-03T19:05:52.744139Z",
            "resourceOwner":"322930458348224793"
          },
          "domain":"{{name}}-pwx4tp.{{mydomain}}",
          "generated":true
        },
        {
          "details":{
            "sequence":"33",
            "creationDate":"2025-06-03T19:05:52.744139Z",
            "changeDate":"2025-06-03T19:05:52.744139Z",
            "resourceOwner":"322930458348224793"
          },
          "domain":"{{mydomain}}",
          "primary":true
        }
      ]
    }
  ]
}
With the non-debug flavour, I get the following error: 
{"code":16, "message":"Errors.Token.Invalid (AUTH-7fs1e)", "details":[{"@type":"type.googleapis.com/zitadel.v1.ErrorDetail", "id":"AUTH-7fs1e", "message":"Errors.Token.Invalid"}]}
{"code":16, "message":"Errors.Token.Invalid (AUTH-7fs1e)", "details":[{"@type":"type.googleapis.com/zitadel.v1.ErrorDetail", "id":"AUTH-7fs1e", "message":"Errors.Token.Invalid"}]}
I'm really not sure what I'm missing. As noted with the bot, the debug flavour should only include extra debugging tools (shell and busybox). I'm not sure why there is such a big behaviour difference.

Did you find this page helpful?