When I develop LLM agent based service (e.g. call openAI api async) using multiple workers to parallel processing large requests, I use uvicorn in APP code to start multiple workers. In server side, a ```remote conntection disconnected``` error happens. But from log, I can see openai request has response but in client side, it causes failure.
After checking log, code and googling, still cannot find what causes the error. But following fastAPI recommendation, gunicorn is recommended in production. So I try to start server using ```gunicorn -w n --worker-class uvicorn.workers.Uvicorn main:app```. The problem is solved. From the server log, it is found multiple workers really started working. When I use unicorn to start multiple workers, always confusing why it looks only one or few worker working, not really parallel, when monitoring CPU usage.
It looks uvicorn cannot handle multiple workers and eventts scheduler, particulally when calling the outside API, which has limit request rates and a little long response time. But what is the root reason is still not understanding.
-
Previous
Daily update signal forecasting for QQQ [Update 2024-11-26] -
Next
AGI, I don’t expect what I can see.
