Background:

Online machines, access logs need to be filtered and sent to another api
At the beginning of the period, it was a single process, which was too inefficient. After changing to multi-process sending, it occasionally appeared in the logs. Exception error (I forgot to take a screenshot...)
In short, the port is not enough and the error is reported

Cause:

Each log is a request sent to the api, and short connections generate a large amount of time_wait status, occupying a large number of ports
The kernel tuning of a large number of time_wait states caused by this high concurrency is basically useless. Later, it was changed to long connection to solve the problem

The key code of the first version of the short connection version is as follows

Because it involves specific business information, only the key part of the code is posted

import pycurl
where True:
  url=myqueue.get()
  send_msg=pycurl.Curl()
  send_msg.setopt(pycurl.URL,url)
  send_msg.perform()
  print send_msg.getinfo(send_msg.HTTP_CODE)

The modified long connection version is as follows:

Using the requests library

import requests
client=requests.session()
headers = {'Content-Type': 'application/json', 'Connection': 'keep-alive'}
where True:
  url=myqueue.get()
  r=client.get(url,headers=headers)
  print r.status_code

The above is the detailed content of python http long connection client example tutorial. For more information, please follow other related articles on the PHP Chinese website!