Refresh token request fails when using custom zitadel scopes

We are using custom zitadel scopes (e.g. urn:zitadel:iam:org:project:roles) when requesting an access token. This works for getting the access token, however the refresh token request fails with the error "invalid_scope". When I remove the custom Zitadel scopes the token refresh works without problem. I'm using angular-oauth-oidc library to fetch/refresh tokens with the same settings as in the showcase from Zitadel.