Setup web crawler with Python and Scrapy on Raspberry Pi

A web crawler (also known as spider or spiderbot) is an internet bot that continually browses web pages, typically for web indexing purposes. Typically Search Engines use web crawling ito scan the web and be aware of contents, links and websites relations. These data are processed to understand what results better fit users queries. Crawlers consume resources on visited systems. For this reason, mechanisms exist for public sites not…