为什么我的 Scrapy 代码返回一个空数组?
我正在为 wunderground.com 构建一个网络爬虫,但我的代码返回[]"的值,表示英寸_雨和湿度.有人能明白为什么会这样吗?
I am building a web scraper for wunderground.com, but I my code returns the value of "[]" for inches_rain and humidity. Could anyone see why this is happening?
# -*- coding: utf-8 -*-
import scrapy
from scrapy.selector import Selector
import time
from wunderground_scraper.items import WundergroundScraperItem
class WundergroundComSpider(scrapy.Spider):
name = "wunderground"
allowed_domains = ["www.wunderground.com"]
start_urls = (
'http://www.wunderground.com/q/zmw:10001.5.99999',
)
def parse(self, response):
info_set = Selector(response).xpath('//div[@id="current"]')
list = []
for i in info_set:
item = WundergroundScraperItem()
item['description'] = i.xpath('div/div/div/div/span/text()').extract()
item['description'] = item['description'][0]
item['humidity'] = i.xpath('div/table/tbody/tr/td/span/span/text()').extract()
item['inches_rain'] = i.xpath('div/table/tbody/tr/td/span/span/text()').extract()
list.append(item)
return list
我也知道湿度和英寸雨项目设置为相同的 xpath,但这应该是正确的,因为一旦信息在数组中,我只需将它们设置为数组中的某些值.
I also know that the humidity and inches_rain items are set to the same xpath, but that should be correct because once the information is in an array I just set them to certain values from the array.
让我建议一个更可靠和可读的 XPath 来定位,举个例子,湿度"值,其中基础是湿度"列标签:
Let me suggest a more reliable and readable XPath to locate, for the sake of an example, "Humidity" value where the base is that "Humidity" column label:
"".join(i.xpath('.//td[dfn="Humidity"]/following-sibling::td//text()').extract()).strip()
现在输出 45%.
仅供参考,您的 XPath 至少有一个问题 - tbody
标记 - 从 XPath 表达式中删除它.
FYI, your XPath had at least one problem - the tbody
tag - remove it from the XPath expression.