如何从通过Javascript加载的页面上抓取数据

问题描述：

我想使用beautifulsoup在此页面上删除评论-https://www.x....s.com/video_id/the-suburl

I want to scrap the comments off this page using beautifulsoup - https://www.x....s.com/video_id/the-suburl

通过Java单击可加载评论.注释是分页的，每个页面也都在单击时加载注释.我希望获取所有评论，对于每个评论，我想获取海报个人资料网址，评论，否.喜欢，没有喜欢和张贴时间(如页面上所述).

The comments are loaded on click via Javascript. The comments are paginated and each page loads comments on click too. I wish to fetch all comments, for each comment, I want to get the poster profile url, the comment, no. of likes, no of dislikes, and time posted (as stated on the page).

评论可以是字典列表.

我该怎么办?

答

此脚本将打印在页面上找到的所有注释:

This script will print all comments found on the page:

import json
import requests
from bs4 import BeautifulSoup


url = 'https://www.x......com/video_id/gggjggjj/'
video_id = url.rsplit('/', maxsplit=2)[-2].replace('video', '')

u = 'https://www.x......com/threads/video/ggggjggl/{video_id}/0/0'.format(video_id=video_id)
comments = requests.post(u, data={'load_all':1}).json()

for id_ in comments['posts']['ids']:
    print(comments['posts']['posts'][id_]['date'])
    print(comments['posts']['posts'][id_]['name'])
    print(comments['posts']['posts'][id_]['url'])
    print(BeautifulSoup(comments['posts']['posts'][id_]['message'], 'html.parser').get_text())
    # ...etc.
    print('-'*80)

如何从通过Javascript加载的页面上抓取数据

相关推荐