使用RVest从meta和button标记中抓取信息

问题描述：

我正在尝试从葡萄酒销售商的页面上抓取用户的平均评分(满分5星)和评分数.我们的5星的平均星级似乎在 button 标签中，而评分数在 meta 标签中.
这是HTML:

I am trying to scrape the average user ratings (out of 5 stars) and number of ratings from a wine seller's page. The average stars our of 5 seems to be in a button tag while the number of rating is in a meta tag.
Here is the HTML:

<div class="bv_avgRating_component_container notranslate">
    <button
      type="button"
      class="bv_avgRating"
      aria-expanded="false"
      aria-label="average rating value is 4.5 of 5."
      id="avg-rating-button"
      role="link"
      itemprop="ratingValue"
    >
      4.5
    </button>
  </div>
  <div class="bv_numReviews_component_container">
    <meta itemprop="reviewCount" content="95" />
   &nbsp;
   <button
      type="button"
      class="bv_numReviews_text"
      aria-label="Read 95 Reviews"
      aria-expanded="false"
      id="num-reviews-button"
      role="link"
    >
      (95)
    </button>
  </div>

我尝试过的事情:

library(tidyverse)
library(rvest)

x <- "/wine/red-wine/cabernet-sauvignon/amici-cabernet-sauvignon-napa/p/20095750?s=918&igrules=true"
ratings <- read_html(paste0("https://www.totalwine.com", x)) %>% 
  html_nodes(xpath = '//meta[@itemprop="reviewCount"]') %>% 
  html_attr('content') #returns character(empty)

ratings <- read_html(paste0("https://www.totalwine.com", x)) %>% 
  html_nodes("meta") %>% 
  html_attr("content") #returns chr [1:33]

ratings <- read_html(paste0("https://www.totalwine.com", x)) %>% 
  html_nodes("div meta") %>% 
  html_attr("content") #returns chr [1:21]

ratings <- read_html(paste0("https://www.totalwine.com", x)) %>% 
  html_nodes("meta[itemprop=reviewCount]") %>% 
  html_attr("content") #returns character(empty)

最终，我要提取的两点是 4.5 和 content ="95" .

At the end of the day, the two points I am trying to extract are 4.5 and content="95".

答

打开开发工具"的网络"选项卡并重新加载页面，您将看到此页面从 https://www.totalwine加载数据.com/product/api/product/product-detail/v1/getProduct/20095750-1?shoppingMethod = INSTORE_PICKUP& state = US-CA& storeId = 918 (这是一个JSON文件):通过此获取所需的评分和评论数:

Open the Network tab of Dev Tool and reload the page, you'll see that this page loads data from https://www.totalwine.com/product/api/product/product-detail/v1/getProduct/20095750-1?shoppingMethod=INSTORE_PICKUP&state=US-CA&storeId=918 (which is a JSON file): Get the rating and review count you want by this:

data <- jsonlite::fromJSON("https://www.totalwine.com/product/api/product/product-detail/v1/getProduct/20095750-1?shoppingMethod=INSTORE_PICKUP&state=US-CA&storeId=918")
rating <- data$customerAverageRating
reviews_count <- data$customerReviewsCount

更新:如果您不熟悉网络抓取领域，您可能想知道为什么我根本不使用 rvest .事实是，此页面使用JS生成内容，并且 rvest 无法处理JS，它仅在加载JS之前读取HTML.

Update: If you're new to the web-scraping field, you're probably wondering why I didn't use rvest at all. The thing is, this page uses JS to generate the content and rvest cannot handle JS, it only reads the HTML before JS loaded.

使用RVest从meta和button标记中抓取信息

相关推荐