Scrapy Custom Development and Support
We offer support and development for Scrapy, a fast high-level screen scraping and web crawling framework, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing.
We often use Scrapy in conjunction with Mechanize and Selenium. Mechanize is a python based library for handling stateful web browsing and selenium can be used for web browsing automation.
These tools allow us to build custom crawlers to extract information from sources, store it in a a large index and then make that data available via a variety of means from direct database access to a web service.
