nilsense•4w ago

Flaky error from /oauth/v2/token

Latest Self-hosted version that has been tested: v2.71.6 Behavior not present in (at least) v2.58.3 In some of my integration tests, I am finding that the /oauth/v2/token endpoint is returning a flaky error.

time=2025-05-14T15:27:24.757Z level=ERROR msg="request error" oidc_error.parent="ID=OIDC-AhX2u Message=Errors.Internal Parent=(invalid signature (error fetching keys: ID=QUERY-Tha6f Message=Errors.AuthNKey.NotFound Parent=(sql: no rows in result set)))" oidc_error.description=Errors.Internal oidc_error.type=server_error status_code=500

time=2025-05-14T15:27:24.757Z level=ERROR msg="request error" oidc_error.parent="ID=OIDC-AhX2u Message=Errors.Internal Parent=(invalid signature (error fetching keys: ID=QUERY-Tha6f Message=Errors.AuthNKey.NotFound Parent=(sql: no rows in result set)))" oidc_error.description=Errors.Internal oidc_error.type=server_error status_code=500

I am unable to reproduce this locally and can only get it to fail in CI/CD. I have checked the thread here with the same error code, but the more I'm debugging the more my issue seems different since it does indeed hit successfully. This seems like a race condition with the database. Comments below will be an example of my test flow. @Raccine, I'd appreciate if you'd be able to point me to the right person to get support for this.

10 Replies

nilsenseOP•4w ago

Here is a sample of some of my test flow:

userResp, err := zitadelClient.AddMachineUser(tenant1Secret.OrgId, t.Name(), "My Machine Account", "First machine account used for API XY.", user.AccessTokenType_ACCESS_TOKEN_TYPE_BEARER)
        require.NoError(t, err)
        testUserId := userResp.GetUserId()

        user, err := zitadelClient.GetMachineUserByName(tenant1Secret.OrgId, t.Name())
        require.NoError(t, err)
        require.NotNil(t, user)
        require.Equal(t, user.UserId, testUserId)

        _, err = zitadelClient.AddPersonalAccessToken(tenant1Secret.OrgId, testUserId, &timestamppb.Timestamp{Seconds: time.Now().Add(time.Minute*10).UnixMilli() / 1000})
        require.NoError(t, err)
        //testPat := token.GetToken()

        key, err := zitadelClient.AddMachineKey(tenant1Secret.OrgId, testUserId, &timestamppb.Timestamp{Seconds: time.Now().Add(time.Minute*10).UnixMilli() / 1000}, true)
        require.NoError(t, err)
        var testUserSecret apiSecret
        err = json.Unmarshal(key.GetKeyDetails(), &testUserSecret)
        require.NoError(t, err)
        testUserSecret.ProjectId = tenant1Secret.UserSecret.ProjectId
        testUserSecret.TenantId = tenant1Secret.UserSecret.TenantId
        testUserSecret.Domain = tenant1Secret.UserSecret.Domain
        require.Equal(t, testUserSecret.secret.UserId, testUserId)

        // TODO: Use native zitadel client
        userTokenExpiry := time.Now().Add(time.Hour).Unix()
        _, err = getZitadelUserToken(testUserSecret, userTokenExpiry)
        require.NoError(t, err) **//FAILS HERE**

        // TODO: Remove
        zitadelClient.RemoveUser(tenant1Secret.OrgId, testUserId)

userResp, err := zitadelClient.AddMachineUser(tenant1Secret.OrgId, t.Name(), "My Machine Account", "First machine account used for API XY.", user.AccessTokenType_ACCESS_TOKEN_TYPE_BEARER)
        require.NoError(t, err)
        testUserId := userResp.GetUserId()

        user, err := zitadelClient.GetMachineUserByName(tenant1Secret.OrgId, t.Name())
        require.NoError(t, err)
        require.NotNil(t, user)
        require.Equal(t, user.UserId, testUserId)

        _, err = zitadelClient.AddPersonalAccessToken(tenant1Secret.OrgId, testUserId, &timestamppb.Timestamp{Seconds: time.Now().Add(time.Minute*10).UnixMilli() / 1000})
        require.NoError(t, err)
        //testPat := token.GetToken()

        key, err := zitadelClient.AddMachineKey(tenant1Secret.OrgId, testUserId, &timestamppb.Timestamp{Seconds: time.Now().Add(time.Minute*10).UnixMilli() / 1000}, true)
        require.NoError(t, err)
        var testUserSecret apiSecret
        err = json.Unmarshal(key.GetKeyDetails(), &testUserSecret)
        require.NoError(t, err)
        testUserSecret.ProjectId = tenant1Secret.UserSecret.ProjectId
        testUserSecret.TenantId = tenant1Secret.UserSecret.TenantId
        testUserSecret.Domain = tenant1Secret.UserSecret.Domain
        require.Equal(t, testUserSecret.secret.UserId, testUserId)

        // TODO: Use native zitadel client
        userTokenExpiry := time.Now().Add(time.Hour).Unix()
        _, err = getZitadelUserToken(testUserSecret, userTokenExpiry)
        require.NoError(t, err) **//FAILS HERE**

        // TODO: Remove
        zitadelClient.RemoveUser(tenant1Secret.OrgId, testUserId)

Would also appreciate if someone is able to point me to the go sdk that I can replace the rest call to /oauth/v2/token, (which is what I use for getZitadelUserToken(testUserSecret, userTokenExpiry)

Unknown User•4w ago

Message Not Public

nilsenseOP•4w ago

Thank you. I've gone through the documentation and am asking becasue I can't find the API that matches that endpoint. **

Unknown User•4w ago

Message Not Public

nilsenseOP•3w ago

I have opened a Github issue bug here

GitHub

[Bug]: Potential Race Condition - flaky error from /oauth/v2/token ...

Preflight Checklist I could not find a solution in the documentation, the existing issues or discussions I have joined the ZITADEL chat Environment Self-hosted Version 2.71.6 Database PostgreSQL Da...

nilsenseOP•2w ago

@Rajat Anyway you could take a look at this too? bump

Rajat•4d ago

hey @nilsense we have taken in it consideration, I will check with my team when can we work on it or any other input from the engineers. Altho, I'd like to ask you too, would you be interested in doing this PR given if engineers can give more input/context on the issue?. please lmk 🙂 I will get back to you with more inputs of the engineers on the issue

nilsenseOP•4d ago

Hey there @Rajat, thanks for getting back to me. I'm generally willing to contribute but it would depend on the context as well. I've already spent a lot of time trying to figure it out (unsuccessfully). Let me know what your people say and we can go from there! (Hoping to get this solved in v2 if possible)

Rajat•4d ago

nice and thanks @nilsense let me discuss with my team and I will come uo with something, what did you already tried already 😄 you can tell me here and I'd try to unblock you with that

nilsenseOP•4d ago

Just tracing the error more or less. I wasn't exactly sure if it was something I was doing wrong on my side or Zitadel's side. But confirming the issue, how it is caused, and a potential path to resolve it would help me get in further (if I have time).

Flaky error from /oauth/v2/token

Did you find this page helpful?