python爬取知乎

此项目的功能是爬取知乎用户信息以及人际拓扑关系，爬虫框架使用scrapy，数据存储使用mongo，下载这些数据感觉也没什么用，就当为大家学习scrapy提供一个例子吧。

## 使用方法

### 本地运行

爬虫程序依赖mongo和rabbitmq，因此这两个服务必须正常运行和配置。为了加快下载效率，图片下载是异步任务，因此在启动爬虫进程执行需要启动异步worker，启动方式是进入zhihu_spider/zhihu目录后执行下面命令:

# -*- coding=utf8 -*-
from scrapy import cmdline

cmdline.execute（“scrapy crawl zhihu“.split（））

 属性            大小     日期    时间   名称
----------- ---------  ---------- -----  ----
     目录           0  2016-03-20 14:42  zhihu_spider-master\
     文件          30  2016-03-20 14:42  zhihu_spider-master\.gitignore
     文件        3056  2016-03-20 14:42  zhihu_spider-master\README.md
     目录           0  2016-03-20 14:42  zhihu_spider-master\doc\
     文件      753450  2016-03-20 14:42  zhihu_spider-master\doc\image.png
     文件      532877  2016-03-20 14:42  zhihu_spider-master\doc\people.png
     文件      413151  2016-03-20 14:42  zhihu_spider-master\doc\relation.png
     文件      126198  2016-03-20 14:42  zhihu_spider-master\doc\主页.png
     文件      111628  2016-03-20 14:42  zhihu_spider-master\doc\代码.png
     文件        3386  2016-03-20 14:42  zhihu_spider-master\doc\流程图.graffle
     文件       97151  2016-03-20 14:42  zhihu_spider-master\doc\流程图.png
     文件         232  2016-03-20 14:42  zhihu_spider-master\docker-compose.yml
     文件          60  2016-03-20 14:42  zhihu_spider-master\requirements.txt
     目录           0  2016-03-20 14:42  zhihu_spider-master\zhihu\
     文件          96  2016-03-20 14:42  zhihu_spider-master\zhihu\main.py
     文件         254  2016-03-20 14:42  zhihu_spider-master\zhihu\scrapy.cfg
     目录           0  2016-03-20 14:42  zhihu_spider-master\zhihu\zhihu\
     文件           0  2016-03-20 14:42  zhihu_spider-master\zhihu\zhihu\__init__.py
     文件         686  2016-03-20 14:42  zhihu_spider-master\zhihu\zhihu\constants.py
     文件        1092  2016-03-20 14:42  zhihu_spider-master\zhihu\zhihu\items.py
     文件        2536  2016-03-20 14:42  zhihu_spider-master\zhihu\zhihu\pipelines.py
     文件        3654  2016-03-20 14:42  zhihu_spider-master\zhihu\zhihu\settings.py
     目录           0  2016-03-20 14:42  zhihu_spider-master\zhihu\zhihu\spiders\
     文件         161  2016-03-20 14:42  zhihu_spider-master\zhihu\zhihu\spiders\__init__.py
     文件        7657  2016-03-20 14:42  zhihu_spider-master\zhihu\zhihu\spiders\profile.py
     目录           0  2016-03-20 14:42  zhihu_spider-master\zhihu\zhihu\tools\
     文件          44  2016-03-20 14:42  zhihu_spider-master\zhihu\zhihu\tools\__init__.py
     文件         696  2016-03-20 14:42  zhihu_spider-master\zhihu\zhihu\tools\async.py