crawler:scrapyarchitecturecode
Differences
This shows you the differences between two versions of the page.
| crawler:scrapyarchitecturecode [2017/01/06 17:18] – [scope of allowed_domains] admin | crawler:scrapyarchitecturecode [2022/10/29 16:15] (current) – external edit 127.0.0.1 | ||
|---|---|---|---|
| Line 860: | Line 860: | ||
| </ | </ | ||
| ==== Integrate Scrapy with Other Systems ==== | ==== Integrate Scrapy with Other Systems ==== | ||
| + | Integrate via below systems: | ||
| + | * Database: MySQL, MongoDB | ||
| + | * Cache: Redis Cache, Cm Cache -> You can **start multiple spider instances that share a single redis queue**. Best suitable for **broad multi-domain crawls**. | ||
crawler/scrapyarchitecturecode.1483723108.txt.gz · Last modified: 2022/10/29 16:15 (external edit)