my notes

User Tools

Site Tools

Trace:
crawler:scrapy

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

crawler:scrapy [2016/08/17 00:53] – [linkextractors package] admincrawler:scrapy [2022/10/29 16:15] (current) – external edit 127.0.0.1
Line 149: Line 149:
 SET VS90COMNTOOLS=%VS100COMNTOOLS% SET VS90COMNTOOLS=%VS100COMNTOOLS%
 </code> => Fix error: Unable to find vcvarsall.bat </code> => Fix error: Unable to find vcvarsall.bat
 +  * upgrade setuptools:<code bat>
 +pip install -U setuptools
 +</code>
 === Install pyopenssl === === Install pyopenssl ===
 Step by steop install openssl: Step by steop install openssl:
Line 526: Line 529:
 </code> </code>
 === extract links with linkextractors === === extract links with linkextractors ===
-Extract files in html file:<code python>+Extract files in html file which links in **tags=('script','img')** and **attrs=('src')**:<code python>
 filesExtractor = sle(allow=("/*"), tags=('script','img'), attrs=('src'), deny_extensions = []) filesExtractor = sle(allow=("/*"), tags=('script','img'), attrs=('src'), deny_extensions = [])
 links = [l for l in self.filesExtractor.extract_links(response) if l not in self.seen] links = [l for l in self.filesExtractor.extract_links(response) if l not in self.seen]
crawler/scrapy.1471395189.txt.gz · Last modified: 2022/10/29 16:15 (external edit)