Scrape Amazon product listings utilising Scrapy & residential proxies

To get started with Scrapy, you will first need to install it using the methods provided in its documentation. Check here for more information

Once you have an active subscription, you can find your credentials & proxy addresses in Dashboard > Residential > Proxy Setup

Navigate to settings.py in /amazon/amazon/ folder and modify the following lines to authenticate.

DECODO_USER = 'username' ## Decodo Username (Sub-user)
DECODO_PASSWORD = 'password' ## Password for your user
DECODO_ENDPOINT = 'gate.decodo.com' ## Endpoint you'd like to use
DECODO_PORT = '7000' ## Port of the endpoint you are using.

Navigate to the project folder and run the following command:

scrapy crawl amazon_search

Amazon search results will be saved in /amazon/data folder in a .csv format