I am a newbie to web crawling, so my understanding of http requests is very shallow. When I inspect certain requests I send to a website, such as add to cart, the dev tools show that there are more values in the "Cookie" header of the request, which are set by "set -cookie" response header is set.
How does this work?
More specifically, one cookie in the "set-cookie" header of the response will contain the cart id, but when looking at all cookie values in the request header, there will also be a cart id in the "Cookie" header , but there are other cookies such as "checkout session lookup" and "checkout session token".
However, these cookies are not set in the "set-cookie" header of the response, so how do they get there?
Suppose I want to grab the complete "Cookie" value of all cookies included in the request header, not just those cookies in the response header. Is there a way to do this?
Thank you for your help!
![[Web front-end] Node.js quick start](https://img.php.cn/upload/course/000/000/067/662b5d34ba7c0227.png)
I think this may be related to the JavaScript on this page. For example, the following code sets a new cookie:
document.cookie="username=John Doe";Therefore, cookies can not only be sent through request headers.
To get the full cookie, I think you might need a headless browser.