Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

如何控制抓取评论的数量? #329

Open
tomriddle1234 opened this issue Apr 29, 2024 · 3 comments
Open

如何控制抓取评论的数量? #329

tomriddle1234 opened this issue Apr 29, 2024 · 3 comments

Comments

@tomriddle1234
Copy link

请问https://weibo.com/ajax/statuses/buildComments 这个api的文档在哪里?新浪文档中找不到。
不许抓取那么多评论,如何控制排序和数量?

@Mar2812
Copy link

Mar2812 commented Apr 30, 2024

yield Request(url, callback=self.parse, meta={'source_url': url, 'mid':mid, 'c':0,'max_count':666})

在start_request中改为这个,c表示计数器,每保存一个评论就加一,能大概控制数量,但不够精准,仅供思路参考哈

其次排序,我看代码里面是先抓一级评论然后抓二级评论,默认按热度排序

@tomriddle1234
Copy link
Author

yield Request(url, callback=self.parse, meta={'source_url': url, 'mid':mid, 'c':0,'max_count':666})
这句在源码哪里?我搜全文都没有,找不到max_count和c

@Edward-WZ
Copy link

我咋根本就无法抓取评论区。。。只能爬微博正文

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants