Optimization practice of a PHP application_PHP tutorial

I recently reviewed an optimization practice I did before and found that some common optimization methods can still be reused. As the system runs for a long time, there will always be problems and bottlenecks of one kind or another. It’s not scary to have problems. We have something to do with "beating the tiger" - it’s nothing more than locating the problem ->analyzing the problem->proposing a solution ->Practice->Result feedback->Summary and then optimize.
Problem description: The system was developed using PHP5 + Zend framework. After the data scale and access volume increased (tens of millions), the background apache server load became too high. During peak access hours (such as the period from get off work to 10pm every day) (especially on Friday), the machine CPU load will soar to more than 170. The high CPU load causes the processing of requests to slow down accordingly, so this problem needs to be solved urgently.
Problem analysis: After several consecutive days of observation and analysis, when the CPU usage reaches 100%, the system CPU usage accounts for a large proportion, and the user CPU usage is not very high. In addition, the front-end haproxy and squid cache CPUs The load is very low, and the hit ratio of memcached and Squid can generally reach about 60%.
Analyzing the backend's access-log, we found that a large portion of requested User-Agents are search crawlers;
At the same time, xdebug was configured on apache, and a set of performance data was measured on the main pages during the idle period. By using kcachegrind to analyze the measured data (how to configure xdebug, you can search with soso) and found:
The performance data is not stable enough, and the test data will vary greatly between the same requests
The slow points are scattered
In most cases, memcached access is relatively slow (more than 100ms)
Solution Through the above preliminary analysis, a series of adjustments were gradually made to the existing procedures.
The first thing to consider is whether we can find a way to increase the Hit ratio of the front-end Squid cache, thereby reducing the number of requests that penetrate Squid and reach the back-end Apache.
Considering that a considerable number of requests originate from Crawler, previously Squid cache would only cache requests with a language cookie set, and requests from Crawler did not have cookie information. So I thought of defaulting all requests from Crawler to the language of zh_CN, and then modifying the configuration of haproxy to transfer all requests from common Crawler with User-Agent to squid cache.
Modify the php code and set the cache time of some pages to be longer
After the above two steps, the number of requests reaching apache has indeed been reduced, but this does little to help the problem of excessive CPU load, so I looked for another method.
Secondly, according to the results of using xdebug profiling, the interaction with memcached takes a long time, so I wonder if I can find a way to make memcached respond to requests faster, so that each request can be completed faster, thereby reducing concurrency.
Through code analysis, it was found that online memcached uses poll(), and the number of memcached connections remains around 1,000 during busy times, and memcached's CPU usage is around 30%. Obviously, the poll() method is very inefficient when handling so many concurrent connections. So I recompiled memcached to use epoll() to process requests. After replacing it with epoll, memcached's cpu usage dropped from about 30% to about 3%, which is 10 times!
In addition, the hit ratio of memcached is not particularly high, and the number of items swapped out is also relatively high, so I thought of partitioning the contents of the cache. I originally planned to do manual partitioning, but later I found that the latest memcache extension of PHP can support cache-based partitioning. The key is automatically partitioned, and new memcached instances can be added without modifying the program code (need to modify the configuration file:-)). So I upgraded the php memcache extension of each apache, and then added a new memcached to the configuration file. This completes the content partition of memcached. The effect after the modification is more significant, and the loading time of the page is much shorter than before the modification.
After these two steps of adjustment, the efficiency of memcached is higher than before, but the load of apache is still high. I have no choice but to think of other solutions!
Further in-depth analysis mentioned earlier that the CPU usage of the main system is very high. To find the reason, we can only go deep into the kernel:) From now on, our strace journey begins. To paraphrase a Nike advertising slogan: Just strace it!
Strace was performed on the httpd process during peak hours, using the following methods
strace -p PID -c gives summary
strace -p PID -o output.log Write to file and study slowly
strace -p PID -e trace=file Only look at syscalls related to filesystem operations
strace -p PID -elstat64,stat64,open,getcwd only traces these syscalls
…
From the above strace analysis, the following conclusions are drawn:
lstat64, stat64, open, etc. There are so many syscalls
The above syscalls do take up a lot of time! More than 60% of the time is robbed by them, orz
The vast majority of syscalls fail. It’s really a case of repeated failure
With the above data, we have found the direction of the problem, which is to find where these meaningless system calls come from.
After analysis, when PHP wants to load a certain class, it will search for the files corresponding to the class in a series of directories defined in include_path, and try each directory until it is found. Well, this method is obviously relatively inefficient. Is there a better way to accomplish this? The answer is yes, there is! And there’s more than one way!
When calling require_once(), write the absolute path as the parameter (Guys write Zend Framework didn’t understand this at first; it was updated later))
Use __autoload() to lazy load the class, which means that it will be loaded only when it is really needed, instead of requiring_once all the class files that may be used.
The problem has been found, but there is another problem to solve. Pay attention to using absolute paths in the code during development. The only thing that can be improved is to change it to lazy loading. However, a large number of require_once in Zend Framework use relative paths, which causes problems - the problem I am talking about here is the CPU load we are talking about in this article. The root cause of the excessive problem.
OK, now that the problem is found, let’s solve it. Write a script to automatically generate the Class -> File Path correspondence, and generate the correspondence files between all classes in the code and all classes in Zend Framework. Comment out all require_once in the code and in the Zend Framework library. Then conduct detailed testing before going live. The results were astonishing, the load dropped to within 3! ! Problem solved.
Summary:
Anyone who writes code knows that there will always be problems wherever there may be problems. There will be a cause for any problem (even if it is not found yet). Solving it from the root is the best way. It doesn’t matter what problem is solved. I hope everyone can learn this solution. Ideas and good use of tools. ok, that’s it for this case.