Flaky error from /oauth/v2/token

Latest Self-hosted version that has been tested: v2.71.6
Behavior not present in (at least) v2.58.3

In some of my integration tests, I am finding that the

/oauth/v2/token

/oauth/v2/token

/oauth/v2/token

/oauth/v2/token endpoint is returning a flaky error.

time=2025-05-14T15:27:24.757Z level=ERROR msg="request error" oidc_error.parent="ID=OIDC-AhX2u Message=Errors.Internal Parent=(invalid signature (error fetching keys: ID=QUERY-Tha6f Message=Errors.AuthNKey.NotFound Parent=(sql: no rows in result set)))" oidc_error.description=Errors.Internal oidc_error.type=server_error status_code=500

time=2025-05-14T15:27:24.757Z level=ERROR msg="request error" oidc_error.parent="ID=OIDC-AhX2u Message=Errors.Internal Parent=(invalid signature (error fetching keys: ID=QUERY-Tha6f Message=Errors.AuthNKey.NotFound Parent=(sql: no rows in result set)))" oidc_error.description=Errors.Internal oidc_error.type=server_error status_code=500

time=2025-05-14T15:27:24.757Z level=ERROR msg="request error" oidc_error.parent="ID=OIDC-AhX2u Message=Errors.Internal Parent=(invalid signature (error fetching keys: ID=QUERY-Tha6f Message=Errors.AuthNKey.NotFound Parent=(sql: no rows in result set)))" oidc_error.description=Errors.Internal oidc_error.type=server_error status_code=500

time=2025-05-14T15:27:24.757Z level=ERROR msg="request error" oidc_error.parent="ID=OIDC-AhX2u Message=Errors.Internal Parent=(invalid signature (error fetching keys: ID=QUERY-Tha6f Message=Errors.AuthNKey.NotFound Parent=(sql: no rows in result set)))" oidc_error.description=Errors.Internal oidc_error.type=server_error status_code=500

I am unable to reproduce this locally and can only get it to fail in CI/CD.

I have checked the thread questions-help-bugsUnable to create proper Access Token with the same error code, but the more I'm debugging the more my issue seems different since it does indeed hit successfully.

This seems like a race condition with the database.

Comments below will be an example of my test flow. @Raccine, I'd appreciate if you'd be able to point me to the right person to get support for this.

nilsenseOP•5/14/25, 4:01 PM

Here is a sample of some of my test flow:

userResp, err := zitadelClient.AddMachineUser(tenant1Secret.OrgId, t.Name(), "My Machine Account", "First machine account used for API XY.", user.AccessTokenType_ACCESS_TOKEN_TYPE_BEARER)
        require.NoError(t, err)
        testUserId := userResp.GetUserId()

        user, err := zitadelClient.GetMachineUserByName(tenant1Secret.OrgId, t.Name())
        require.NoError(t, err)
        require.NotNil(t, user)
        require.Equal(t, user.UserId, testUserId)

        _, err = zitadelClient.AddPersonalAccessToken(tenant1Secret.OrgId, testUserId, &timestamppb.Timestamp{Seconds: time.Now().Add(time.Minute*10).UnixMilli() / 1000})
        require.NoError(t, err)
        //testPat := token.GetToken()

        key, err := zitadelClient.AddMachineKey(tenant1Secret.OrgId, testUserId, &timestamppb.Timestamp{Seconds: time.Now().Add(time.Minute*10).UnixMilli() / 1000}, true)
        require.NoError(t, err)
        var testUserSecret apiSecret
        err = json.Unmarshal(key.GetKeyDetails(), &testUserSecret)
        require.NoError(t, err)
        testUserSecret.ProjectId = tenant1Secret.UserSecret.ProjectId
        testUserSecret.TenantId = tenant1Secret.UserSecret.TenantId
        testUserSecret.Domain = tenant1Secret.UserSecret.Domain
        require.Equal(t, testUserSecret.secret.UserId, testUserId)

        // TODO: Use native zitadel client
        userTokenExpiry := time.Now().Add(time.Hour).Unix()
        _, err = getZitadelUserToken(testUserSecret, userTokenExpiry)
        require.NoError(t, err) **//FAILS HERE**

        // TODO: Remove
        zitadelClient.RemoveUser(tenant1Secret.OrgId, testUserId)

userResp, err := zitadelClient.AddMachineUser(tenant1Secret.OrgId, t.Name(), "My Machine Account", "First machine account used for API XY.", user.AccessTokenType_ACCESS_TOKEN_TYPE_BEARER)
        require.NoError(t, err)
        testUserId := userResp.GetUserId()

        user, err := zitadelClient.GetMachineUserByName(tenant1Secret.OrgId, t.Name())
        require.NoError(t, err)
        require.NotNil(t, user)
        require.Equal(t, user.UserId, testUserId)

        _, err = zitadelClient.AddPersonalAccessToken(tenant1Secret.OrgId, testUserId, &timestamppb.Timestamp{Seconds: time.Now().Add(time.Minute*10).UnixMilli() / 1000})
        require.NoError(t, err)
        //testPat := token.GetToken()

        key, err := zitadelClient.AddMachineKey(tenant1Secret.OrgId, testUserId, &timestamppb.Timestamp{Seconds: time.Now().Add(time.Minute*10).UnixMilli() / 1000}, true)
        require.NoError(t, err)
        var testUserSecret apiSecret
        err = json.Unmarshal(key.GetKeyDetails(), &testUserSecret)
        require.NoError(t, err)
        testUserSecret.ProjectId = tenant1Secret.UserSecret.ProjectId
        testUserSecret.TenantId = tenant1Secret.UserSecret.TenantId
        testUserSecret.Domain = tenant1Secret.UserSecret.Domain
        require.Equal(t, testUserSecret.secret.UserId, testUserId)

        // TODO: Use native zitadel client
        userTokenExpiry := time.Now().Add(time.Hour).Unix()
        _, err = getZitadelUserToken(testUserSecret, userTokenExpiry)
        require.NoError(t, err) **//FAILS HERE**

        // TODO: Remove
        zitadelClient.RemoveUser(tenant1Secret.OrgId, testUserId)

nilsenseOP•5/14/25, 4:02 PM

Would also appreciate if someone is able to point me to the go sdk that I can replace the rest call to /oauth/v2/token, (which is what I use for getZitadelUserToken(testUserSecret, userTokenExpiry)getZitadelUserToken(testUserSecret, userTokenExpiry)

nilsenseOP•5/15/25, 7:11 PM

Thank you. I've gone through the documentation and am asking becasue I can't find the API that matches that endpoint. **

nilsenseOP•5/20/25, 3:41 PM

I have opened a Github issue bug here

GitHub

[Bug]: Potential Race Condition - flaky error from /oauth/v2/token ...

Preflight Checklist I could not find a solution in the documentation, the existing issues or discussions I have joined the ZITADEL chat Environment Self-hosted Version 2.71.6 Database PostgreSQL Da...

nilsenseOP•5/21/25, 5:28 PM

@Rajat Anyway you could take a look at this too?

nilsenseOP•5/29/25, 5:46 PM

bump

Rajat•6/2/25, 9:36 AM

hey @nilsense we have taken in it consideration, I will check with my team when can we work on it or any other input from the engineers. Altho, I'd like to ask you too, would you be interested in doing this PR given if engineers can give more input/context on the issue?. please lmk

I will get back to you with more inputs of the engineers on the issue

nilsenseOP•6/2/25, 1:13 PM

Hey there @Rajat, thanks for getting back to me. I'm generally willing to contribute but it would depend on the context as well. I've already spent a lot of time trying to figure it out (unsuccessfully). Let me know what your people say and we can go from there! (Hoping to get this solved in v2 if possible)

Rajat•6/2/25, 1:18 PM

nice and thanks @nilsense let me discuss with my team and I will come back with something, what did you already tried already

you can tell me here and I'd try to unblock you with that

nilsenseOP•6/2/25, 1:19 PM

Just tracing the error more or less. I wasn't exactly sure if it was something I was doing wrong on my side or Zitadel's side. But confirming the issue, how it is caused, and a potential path to resolve it would help me get in further (if I have time).

Rajat•6/6/25, 9:07 AM

thanks @nilsense engineering team is aware of the ongoing issues but we are growing very fast, with better tests coverage, it WILL get better. I will ask someone within the team to get back on your issue with some feedback.
Hope this helps
thanks

nilsenseOP•6/6/25, 4:21 PM

Thanks @Rajat ! I understand the miss due to the test coverage, but I'm most afraid that some of these things will not get fixed before support for v2 expires. I appreciate advocating having someone on your team look at the issue. Thanks Rajat!

userResp, err := zitadelClient.AddMachineUser(tenant1Secret.OrgId, t.Name(), "My Machine Account", "First machine account used for API XY.", user.AccessTokenType_ACCESS_TOKEN_TYPE_BEARER) require.NoError(t, err) testUserId := userResp.GetUserId() user, err := zitadelClient.GetMachineUserByName(tenant1Secret.OrgId, t.Name()) require.NoError(t, err) require.NotNil(t, user) require.Equal(t, user.UserId, testUserId) _, err = zitadelClient.AddPersonalAccessToken(tenant1Secret.OrgId, testUserId, &timestamppb.Timestamp{Seconds: time.Now().Add(time.Minute*10).UnixMilli() / 1000}) require.NoError(t, err) //testPat := token.GetToken() key, err := zitadelClient.AddMachineKey(tenant1Secret.OrgId, testUserId, &timestamppb.Timestamp{Seconds: time.Now().Add(time.Minute*10).UnixMilli() / 1000}, true) require.NoError(t, err) var testUserSecret apiSecret err = json.Unmarshal(key.GetKeyDetails(), &testUserSecret) require.NoError(t, err) testUserSecret.ProjectId = tenant1Secret.UserSecret.ProjectId testUserSecret.TenantId = tenant1Secret.UserSecret.TenantId testUserSecret.Domain = tenant1Secret.UserSecret.Domain require.Equal(t, testUserSecret.secret.UserId, testUserId) // TODO: Use native zitadel client userTokenExpiry := time.Now().Add(time.Hour).Unix() _, err = getZitadelUserToken(testUserSecret, userTokenExpiry) require.NoError(t, err) **//FAILS HERE** // TODO: Remove zitadelClient.RemoveUser(tenant1Secret.OrgId, testUserId)