http://mynotes.babies.vn/ 2026-05-17T09:24:48+00:00 http://mynotes.babies.vn/ http://mynotes.babies.vn/lib/exe/fetch.php?media=wiki:dokuwiki.svg text/html 2022-10-29T16:15:47+00:00 Anonymous (anonymous@undisclosed.example.com) http://mynotes.babies.vn/doku.php?id=crawler:scrapy&rev=1667060147&do=diff Scrapy refer: * <http://doc.scrapy.org/en/latest/> code examples: * <https://code.google.com/p/scrapy-tutorial/> git clone https://code.google.com/p/scrapy-tutorial/ * <https://code.google.com/p/scrapy-spider/> svn checkout http://scrapy-spider.googlecode.com/svn/trunk/ scrapy-spider-read-only * <https://code.google.com/p/kateglo-crawler/> svn checkout http://kateglo-crawler.googlecode.com/svn/trunk/ kateglo-crawler-read-only text/html 2022-10-29T16:15:47+00:00 Anonymous (anonymous@undisclosed.example.com) http://mynotes.babies.vn/doku.php?id=crawler:scrapyarchitecturecode&rev=1667060147&do=diff Scrapy Architecture Code Scrapy commands Overview about scrapy commands * Scrapy command format scrapy --help Scrapy 1.0.3 - project: templatedownload Usage: scrapy <command> [options] [args] Available commands: bench Run quick benchmark test check Check spider contracts commands crawl Run a spider edit Edit spider fetch Fetch a URL using the Scrapy downloader genspider Generate new spider using pre-defined templates list … text/html 2022-10-29T16:15:47+00:00 Anonymous (anonymous@undisclosed.example.com) http://mynotes.babies.vn/doku.php?id=crawler:scrapyexamples&rev=1667060147&do=diff Scrapy Examples Download entire site with scrapy from scrapy.selector import HtmlXPathSelector from scrapy.contrib.spiders import CrawlSpider, Rule from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor as sle class BabySpider(CrawlSpider): name = "baby" allowed_domains = ["babies.vn"] start_urls = [ "http://shop.babies.vn/index.php" ] rules = [ Rule(sle(allow=("/*.html")), callback='parse_template'), ] def parse_template(self, response)…