aiohttp_ClientSession_issue
Investigate aiohttp.ClientSession randomly failure when batch calling APIs, e.g. Client error for XXX: Can not write request body for API_URL
What is the issue?
The issue happends when you call your API endpoint (e.g. FastAPI server) from server side using batch. Normally we use asynchronous + semaphore + aiohttp to control traffic sent to server. Sometimes, it works fine. Sometime server return nothing, and the client catch errors, e.g. Client error for XXX: Can not write request body for server_url, e.g. http://0.0.0.0:port/...
Googling cannot find the reason what causes the issue. After reading the aiohttp.ClientSession source code, I guess the issue maybe caused by Timeout setting when calling ClientSession. Almost all use default setting. But default Timeout, total = 300 seconds. It maybe enough for adhoc usage. But for production, e.g. your server need to call OpenAPI API to do reasoning and process, only 1 openai API call may spending a few seconds (e.g. 6s, or even more > 10s in multi-modality & long response, e.g 2K output token).
The issue looks randomly. You cannot repeat.
The project is to test whether the issue is caused by Timeout and how configure parameters affect failure rate
Test Experiments
Test parameters:
- Argument in asyncio.Semaphore
- Argument (total = xxx) in aiohttp.ClientTimeout
- Work number
- Start server: Uvicorn or Gunicorn
Practical setting
In general, increase Timeout(total=XX) can reduce failure. In aiohttp ClientSession instance initialize the default `timeout = sentinel`, which means `DEFAULT_TIMEOUT: Final[ClientTimeout] = ClientTimeout(total=5 * 60)`, i.e. default 300 seconds.
Estimate total setting in timeout
total >= seconds_of_API_call (average or max) * #_of_request_async_task / (#_of_worker #_of_Semaphore (parallel call))
e.g. in the demo
seconds_of_API_call = 1s
#_of_Semaphore = 2
#_of_worker = 2
#_of_request_async = 10
total >= (1*10) / (2 * 2) = 2.5Hope this analysis can help you solve the API call failure issue.
Code and detail analysis on the affect of parameter setting on API call success and failure rate in Github: aiohttp_ClientSession_issue
-
Previous
AGI, I don’t expect what I can see. -
Next
Resource sharing for AI fintech product development
