What software do you use to write crawlers in Python?

If you are learning programming, then "crawlers" are definitely things you cannot ignore. So, what tools do you need before learning python crawler?

Code editing operating environment: (Recommended learning: Python video tutorial)

Personal PyCharm Community Edition is recommended. Of course, IDLE that comes with Python will also work, as will Notepad, as long as you are used to it.

F12 Developer Tools

Look at the source code: quickly locate element analysis xpath: 1. Google browser is recommended here, you can right-click on the source code interface to view

Packet Capture Tool

Recommend httpfox, a plug-in for the Firefox browser. It is better than the F12 tool that comes with Google Firefox. It can conveniently check the website's packet receiving and sending packets. Information

is an absolute path. When encountering some dynamically generated icons (common ones include list page turning buttons, etc.), the erratic absolute path is likely to cause errors. Therefore, it is recommended here that during actual analysis, it is only used as a reference. Remember to remember the following figure. Remove the "x:" in the xpath box. It seems that this is the syntax of the early version of xpath. It is currently incompatible with some modules (such as scrapy). It is better to delete it to avoid errors.

Regular Expression Test Tool

For more Python related technical articles, please visit the Python Tutorial

The above is the detailed content of What software do you use to write crawlers in Python?. For more information, please follow other related articles on the PHP Chinese website!