Arnau•2w ago

Login V2: gRPC call to zitadel.user.v2.UserService.ListUsers returns 403, organization scoped

Use-case: Multi tenancy PaaS Environment: Self hosting Version: 4.5.0 and 4.6.2 in beta envs, 3.3.0 in production (using old zitadel/typescript repo) Hello everyone, Not sure if it's reproducible as well for some of you, but gRPC call to zitadel.user.v2.UserService.ListUsers endpoint returns a 403 permission_denied error to the Login V2 application when the URL contains a pre-defined organization query param resolved from scopes from the auth request, AND most strangely only for one same user email, even after deleteing + recreating the user.

 Error [ConnectError]: [permission_denied] HTTP 403
    at <unknown> (.next/server/chunks/5397.js:10:108067)
    at v (.next/server/chunks/5397.js:10:108544)
    at next (.next/server/chunks/5397.js:10:144756)
    at async Object.unary (.next/server/chunks/5397.js:10:144015)
    at async Object.i [as listUsers] (.next/server/chunks/5397.js:1:1763)
    at async aw (.next/server/chunks/8101.js:1:108130)
    at async s (.next/server/chunks/8101.js:1:613) {
  rawMessage: 'HTTP 403',
  code: 7,
  metadata: Headers {
    server: 'awselb/2.0',
    date: 'Wed, 29 Oct 2025 08:21:33 GMT',
    'content-type': 'application/grpc',
    'content-length': '0',
    'grpc-status': '7',
    'grpc-message': 'permission denied'
  },
  details: [],
  cause: undefined,
  digest: '2436281361'
}

 Error [ConnectError]: [permission_denied] HTTP 403
    at <unknown> (.next/server/chunks/5397.js:10:108067)
    at v (.next/server/chunks/5397.js:10:108544)
    at next (.next/server/chunks/5397.js:10:144756)
    at async Object.unary (.next/server/chunks/5397.js:10:144015)
    at async Object.i [as listUsers] (.next/server/chunks/5397.js:1:1763)
    at async aw (.next/server/chunks/8101.js:1:108130)
    at async s (.next/server/chunks/8101.js:1:613) {
  rawMessage: 'HTTP 403',
  code: 7,
  metadata: Headers {
    server: 'awselb/2.0',
    date: 'Wed, 29 Oct 2025 08:21:33 GMT',
    'content-type': 'application/grpc',
    'content-length': '0',
    'grpc-status': '7',
    'grpc-message': 'permission denied'
  },
  details: [],
  cause: undefined,
  digest: '2436281361'
}

The IAM_LOGIN_CLIENT manager role is granted to the Service Account used by Login V2. Also tried adding IAM_OWNER, sadly with the same result. Strangely, the REST endpoint /v2/users returns the expected search result. I opened this issue for it: https://github.com/zitadel/zitadel/issues/10995

GitHub

[Bug]: Login V2: gRPC call to zitadel.user.v2.UserService.ListUsers...

Preflight Checklist I could not find a solution in the documentation, the existing issues or discussions I have joined the ZITADEL chat Environment Self-hosted Version 3.3.0 (with custom zitadel/ty...

12 Replies

Ask Inkeep•2w ago

Hi @Arnau, I'm an AI assistant for ZITADEL. While a team member reviews this question, I'll see if I can find relevant sources for your question. Answer I wasn't able to find a direct answer, here are some relevant sources: 1. Org Owner cannot see all users (1) : A user reported that after upgrading ZITADEL, they could only see their own user in the Users tab despite being an Org Owner, though the API request on 'management/v1/users/_search' worked as expected and showed all users. 2. Discord message from @dhene on 2025-09-26 (2) : A user experienced permission_denied errors when trying to list organizations despite having the Org Owner role, suggesting potential permission-related issues with organization-scoped requests. To learn more, see the sources I considered: [1] Org Owner cannot see all users [2] Discord message from @dhene on 2025-09-26 Feel free to tag @Ask Inkeep with additional questions. Got it. Tagging @Rajat for additional help.

ArnauOP•2w ago

Happens the same with v3.3.0 on the server side, with the code from zitadel/typescript archived repository... a possible clue that the issue is on the API itself? Latest findings: seems the constant way to reproduce the 403 permission denied, is searching for an e-mail address that match the following criteria: - They are exactly 34 characters long (not 33, not 35) - The domain part of the address contains a hyphen character (-). Example:

test.34characters@thisis-atest.com

test.34characters@thisis-atest.com

`` We don't have any specific verified domain in the organizations.

Rajat•2w ago

hey @Arnau interesting investigation 🔬 did it ONLY exist for 34 characters long email address you think?.

ArnauOP•2w ago

As far as we can see, yes. Happens across different AWS envs and Zitadel instances, with a single email address. As it was a prio we managed to fix it in a try/catch fashion to fallback to REST /v2/users and it worked, but definitely there's something there that bothers me.

Rajat•2w ago

hey @Arnau could you please open an issue for the same?. It should be noticed by the engineering team.

ArnauOP•2w ago

Sure, is this one: https://github.com/zitadel/zitadel/issues/10995

Rajat•7d ago

ah you already opened it last week, I should've noticed. Thanks https://github.com/zitadel/zitadel/issues/10995#issuecomment-3488698581 I have tested it and it works fine for me

ArnauOP•7d ago

Thanks for the support Rajat. Could you test it with organizationId query as well? The JSON stringified query object from the Login V2 app is this one:

[
    {
        "$typeName": "zitadel.user.v2.SearchQuery",
        "query": {
            "case": "orQuery",
            "value": {
                "$typeName": "zitadel.user.v2.OrQuery",
                "queries": [
                    {
                        "$typeName": "zitadel.user.v2.SearchQuery",
                        "query": {
                            "case": "emailQuery",
                            "value": {
                                "$typeName": "zitadel.user.v2.EmailQuery",
                                "emailAddress": "test.34characters@thisis-atest.com",
                                "method": 1
                            }
                        }
                    }
                ]
            }
        }
    },
    {
        "$typeName": "zitadel.user.v2.SearchQuery",
        "query": {
            "case": "organizationIdQuery",
            "value": {
                "$typeName": "zitadel.user.v2.OrganizationIdQuery",
                "organizationId": "308125743508954293"
            }
        }
    }
]

[
    {
        "$typeName": "zitadel.user.v2.SearchQuery",
        "query": {
            "case": "orQuery",
            "value": {
                "$typeName": "zitadel.user.v2.OrQuery",
                "queries": [
                    {
                        "$typeName": "zitadel.user.v2.SearchQuery",
                        "query": {
                            "case": "emailQuery",
                            "value": {
                                "$typeName": "zitadel.user.v2.EmailQuery",
                                "emailAddress": "test.34characters@thisis-atest.com",
                                "method": 1
                            }
                        }
                    }
                ]
            }
        }
    },
    {
        "$typeName": "zitadel.user.v2.SearchQuery",
        "query": {
            "case": "organizationIdQuery",
            "value": {
                "$typeName": "zitadel.user.v2.OrganizationIdQuery",
                "organizationId": "308125743508954293"
            }
        }
    }
]

Rajat•7d ago

hey @Arnau sure, let me test it and will get back to you.

Rajat•7d ago

it worked

    testReq := &user.ListUsersRequest{
        Queries: []*user.SearchQuery{
            {
                Query: &user.SearchQuery_EmailQuery{
                    EmailQuery: &user.EmailQuery{
                        EmailAddress: "test.34characters@thisis-atest.com",
                    },
                },
            },
            {
                Query: &user.SearchQuery_OrganizationIdQuery{
                    OrganizationIdQuery: &user.OrganizationIdQuery{
                        OrganizationId: "299415687774972672",
                    },
                },
            },
        },
    }

    testReq := &user.ListUsersRequest{
        Queries: []*user.SearchQuery{
            {
                Query: &user.SearchQuery_EmailQuery{
                    EmailQuery: &user.EmailQuery{
                        EmailAddress: "test.34characters@thisis-atest.com",
                    },
                },
            },
            {
                Query: &user.SearchQuery_OrganizationIdQuery{
                    OrganizationIdQuery: &user.OrganizationIdQuery{
                        OrganizationId: "299415687774972672",
                    },
                },
            },
        },
    }

here's the output - 2025/11/05 10:38:29 :x: User not found in the organization ( this happened because I deleted the user after the testing yesterday 😅 ) - 2025/11/05 10:40:23 :white_check_mark: Found user: email:"test.34characters@thisis-atest.com" is_verified:true created the user again and it found it.

ArnauOP•7d ago

Thanks for checking @Rajat. I also tried with a new vanilla compose setup and I'm not able to reproduce it neither locally, with the same Org config and same e-mail/loginnames/IdP links... Seems it only happens consistently in our AWS envs, with the emails also being consistent. This Prometheus query also shows 0 metric results,

grpc_server_grpc_status_code_total{
    service="zitadel",
    grpc_method="/zitadel.user.v2.UserService/ListUsers",
    return_code="403"
}

grpc_server_grpc_status_code_total{
    service="zitadel",
    grpc_method="/zitadel.user.v2.UserService/ListUsers",
    return_code="403"
}

But I can see the K8S pod of the Login V2 app tracing the errors (and fallbacking to REST, due our custom try/catch). I'll review if there's something at infra/WAF level that could be blocking" those gRPC requests. TBH is the strangest issue we had so far.

Rajat•7d ago

oh wow, yes, seems like the case. Maybe its a platform specific issue as I was never able to reproduce it(also the fact that its ONLY happening for exactly 34 chars email, seems very specific)

Login V2: gRPC call to zitadel.user.v2.UserService.ListUsers returns 403, organization scoped

Did you find this page helpful?