Use superagent
to crawl the web page. When encountering a redirection, I cannot get the response body after the redirection. How to solve the problem and how to obtain the redirected webpage
I want to get the res of the 501 page, but it jumps to page 37018, causing me to get the empty res of the 501 page.
Tested web pages that could not be crawled and found two situations
The response code when entering the webpage is 200 at first, and after a while it refreshes to 304
The response code redirects from 301 to 200 upon entry, and refreshes to 304 after a while
Tested the web pages that could be crawled and found two situations
The response code redirects from 301 to 200 upon entry, and refreshes to 304 after a while
The response code when entering the webpage is 200 at first, and after a while it refreshes to 304
Ah, let me wipe it, then there will be no difference. I don’t know if this has anything to do with me not being able to crawl the content, orz
Add more
The problem is not redirection, but my regular expression matching problem
Don’t you even read the official documentation?
Following redirects