crawler:scrapyarchitecturecode
Differences
This shows you the differences between two versions of the page.
| crawler:scrapyarchitecturecode [2017/01/06 17:19] – [Integrate Scrapy with Other Systems] admin | crawler:scrapyarchitecturecode [2022/10/29 16:15] (current) – external edit 127.0.0.1 | ||
|---|---|---|---|
| Line 861: | Line 861: | ||
| ==== Integrate Scrapy with Other Systems ==== | ==== Integrate Scrapy with Other Systems ==== | ||
| Integrate via below systems: | Integrate via below systems: | ||
| - | * database: MySQL, MongoDB | + | * Database: MySQL, MongoDB |
| - | * Cache: Redis Cache, Cm Cache | + | * Cache: Redis Cache, Cm Cache -> You can **start multiple spider instances that share a single redis queue**. Best suitable for **broad multi-domain crawls**. |
crawler/scrapyarchitecturecode.1483723196.txt.gz · Last modified: 2022/10/29 16:15 (external edit)