crawler:scrapyarchitecturecode
Differences
This shows you the differences between two versions of the page.
| crawler:scrapyarchitecturecode [2017/01/06 17:16] – [Write items to MongoDB] admin | crawler:scrapyarchitecturecode [2022/10/29 16:15] (current) – external edit 127.0.0.1 | ||
|---|---|---|---|
| Line 859: | Line 859: | ||
| return url_is_from_spider(request.url, | return url_is_from_spider(request.url, | ||
| </ | </ | ||
| + | ==== Integrate Scrapy with Other Systems ==== | ||
| + | Integrate via below systems: | ||
| + | * Database: MySQL, MongoDB | ||
| + | * Cache: Redis Cache, Cm Cache -> You can **start multiple spider instances that share a single redis queue**. Best suitable for **broad multi-domain crawls**. | ||
crawler/scrapyarchitecturecode.1483722991.txt.gz · Last modified: 2022/10/29 16:15 (external edit)