my notes

User Tools

Site Tools

Trace:
crawler:scrapyexamples

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
crawler:scrapyexamples [2015/05/15 01:14] – [Scrapy OpenSources] admincrawler:scrapyexamples [2022/10/29 16:15] (current) – external edit 127.0.0.1
Line 188: Line 188:
 </code> </code>
 ===== Scrapy OpenSources ===== ===== Scrapy OpenSources =====
 +Sort follow filter **Most stars**
 +  * https://github.com/scrapinghub/portia
 +  * https://github.com/gnemoug/distribute_crawler -> distribute spiders  
   * https://github.com/darkrho/scrapy-redis -> distributed spiders with single redis for receiving items   * https://github.com/darkrho/scrapy-redis -> distributed spiders with single redis for receiving items
-  * https://github.com/gnemoug/distribute_crawler -> distribute spiders+  * https://github.com/geekan/scrapy-examples
   * https://github.com/holgerd77/django-dynamic-scraper -> Manage scrapy spiders via django admin   * https://github.com/holgerd77/django-dynamic-scraper -> Manage scrapy spiders via django admin
 +  * https://github.com/scrapinghub/scrapyjs
   * https://github.com/scrapy/scrapyd   * https://github.com/scrapy/scrapyd
   * https://github.com/scrapinghub/scrapylib -> collection of code   * https://github.com/scrapinghub/scrapylib -> collection of code
Line 198: Line 202:
   * https://github.com/kalessin/finance -> Crawl finance data and store to mysql server   * https://github.com/kalessin/finance -> Crawl finance data and store to mysql server
   * https://github.com/istresearch/scrapy-cluster   * https://github.com/istresearch/scrapy-cluster
 +  * https://github.com/scrapy/scrapely
   * https://github.com/octoberman/scrapy-indeed-spider   * https://github.com/octoberman/scrapy-indeed-spider
   * https://github.com/dcondrey/scrapy-spiders -> collection of spiders   * https://github.com/dcondrey/scrapy-spiders -> collection of spiders
Line 212: Line 217:
   * https://github.com/walbuc/Django-Scrapy -> good architecture with django:   * https://github.com/walbuc/Django-Scrapy -> good architecture with django:
     * sqlite 3 in dev, postgres or mongodb in prod     * sqlite 3 in dev, postgres or mongodb in prod
 +    * Using Celery: Distributed Task Queue
     * Using redis server for caching     * Using redis server for caching
     * Scrapy create webservice REST and Django using REST API to get data     * Scrapy create webservice REST and Django using REST API to get data
Line 217: Line 223:
   * https://github.com/amferraz/9gag-scraper   * https://github.com/amferraz/9gag-scraper
   * https://github.com/junwei-wang/GoogleLoginSpider   * https://github.com/junwei-wang/GoogleLoginSpider
-  * https://github.com/lodow/portia-proxy -> You can **annotate a web page** to identify the data you wish to extract+  * https://github.com/lodow/portia-proxy -> You can **annotate a web page** to identify the data you wish to  
 +  * https://github.com/voliveirajr/seleniumcrawler -> scrapy with selenium 
 +extract
   * https://github.com/pelick/VerticleSearchEngine   * https://github.com/pelick/VerticleSearchEngine
   * https://github.com/eliangcs/pystock-crawler   * https://github.com/eliangcs/pystock-crawler
crawler/scrapyexamples.1431652492.txt.gz · Last modified: (external edit)