NodeJS Call Cancelled uncaught exception when making call to zitadel API
So I've noticed an issue where one of our nodejs services using the js sdk ends up restarting the container and killing the pod on kubernetes. See screenshot for the issue.
It's intermittent, it's happening infrequently, and primarily after prolonged inactivity in the pod (overnight for example).
Since it's an uncaught exception it's causing istio to issue a 503 back to the client caller, as the upstream target to the proxy is understandably and justifiably dead (issuing TCP RST packets before the pod dies).
Any thoughts?
5 Replies
This might be a red herring, and that the code 1, and details of call cancelled might be the outcome of the uncaught exception (and thus cancelling the call from the client side) but I can't be sure
getting an expected FIN ACK when the pod dies and restarts followed by the expected Client Hello of the pod/nodejs restarting and reinitailising a connection to the API a few seconds later
attached log for when the request comes in and dies (nothing in the TCP dump indicates anything unusual but it's all hidden inside the TLS stream anyway)
One thing stands out to me but I have no expertise in gRPC as a protocol really:
Code 8 being as far as I can tell, RESOURCE_EXHAUSTED - server is basically cutting the call server side for an unknown reason
I wonder if I'm hitting the 10 rps over a minute gRPC quota mhmm
https://github.com/smartive/zitadel-node/issues/443 thrown it into an issue on the off chance it's solved upstream and as a record for this question in case anyone else has it
GitHub
Intermittent failures causing node to panic · Issue #443 · smartive...
Bit of a strange one but figured I'd ask, I've got an issue where if a nodejs microservice is idle for some time (not known how long yet for a pattern) and then receives a request which req...