Skip to main content
17,826 questions
Newest Active Bountied Unanswered

Question list filters

Filter by
Sorted by
Tagged with
Score of 2
1 answer
81 views

I've been learning how to use scrapy, and made some practice code, and it seems to be connecting to the target URL, but not returning any item Here's the code: import scrapy class BookspiderSpider(...
Score of 0
0 answers
171 views

I'm creating a project to add to a not-yet-existent portfolio and I have some things I'm confused about. The goal of the project is to create the most comprehensive collection of artwork (drawings,...
Score of 3
1 answer
81 views

What I want to do is to increase the line width of Scrapy logs. I'm logging to a file. When Scrapy logs (assuming it's pprint) the lines are getting wrapped at 80. Yes, that's the default for Python's ...
Score of 1
1 answer
134 views

My version of scrapy is 2.11.0 I am learning scrapy and the code they give as an example to try is this: from pathlib import Path import scrapy class QuotesSpider(scrapy.Spider): name = "...
Score of 2
2 answers
200 views

I'm making a tutorial on how to scrape with Scrapy. For that, I use Quarto/RStudio and the website https://quotes.toscrape.com/. For pedagogic purposes, I need to run a first crawl on the first page, ...
Score of 1
1 answer
232 views

I'm quite new to web scraping, and in particular in using Scrapy's spiders, pipelines... I'm getting some 202 status from some spider requests' response, hence the page content is not available yet ...
Score of 2
1 answer
115 views

After starting a spider, there is the problem with freezing on a stage when pipeline must enable. There is no errors, just scrapy-playwrigth script, but it stopes on beggining before even starts ...
Score of 0
1 answer
232 views

I'm far from a Python expert and this is my first Scrapy project. I installed Scrapy using Brew. I've been able to do some basics with Scrapy and making progress. I need to add Beautiful Soup to clean ...
Score of 0
0 answers
132 views

I’m writing a sitemap XML parser using lxml.etree.iterparse class Sitemap: """Class to parse Sitemap (type=urlset) and Sitemap Index (type=sitemapindex) files""" ...
Score of 0
1 answer
123 views

My scrapy logic is as follows: get all rows from child_page_table where parent_page_id is null for each row, if parent_page_id is (still) null, yield a Request with callback scrape_page [scrape_page] ...
Score of 2
1 answer
94 views

I am trying to make a web crawler with Scrapy which fetches some html pages and saves them via default Request callback i.e. parse() The thing is, I want the spider to stop crawling pending or ...
Score of -1
2 answers
179 views

Trying to build a scraper that extracts nutritional information from each product page on Sainsbury (for eg, scraping energy values out of https://www.sainsburys.co.uk/gol-ui/product/sainsburys-...
Score of 0
1 answer
89 views

I have pressed the command scrapy or scrapy crawl bookspider -o bookdata.csv and the error looks like this: Traceback (most recent call last): File "C:\Users\Tunansh Vatsa\AppData\Local\...
Score of -4
1 answer
93 views

I had a small webcrawler that was written using scrapy and since I didn't want to run it against real site during development I used a local mirror. Mirror was served with python -m http.server 8000 ...
Score of 2
1 answer
92 views

        I have a scrapy Crawlspider that parses reviews, using a scrapy-rotating-proxies.         But when I tried to connect to the site I got the 507 status code. In ...

15 30 50 per page
1
2 3 4 5
…
1189