通过scrapy中的回调函数传递元元素
当我通过回调函数传递元素时,就像在官方 scrapy 文档中找到的这个例子一样.
When I pass elements through callback function like in this example found on the official scrapy documentation.
我想知道传递给 parse_page2
的元素 item
在上述函数中修改后是否可以在 parge_page1
函数中检索修改.
I was wondering if the element item
passed to parse_page2
once modified inside the aforementioned function could be retrieved modified in the parge_page1
function.
我的意思是假设下面的例子.在 parse_page2
函数中,我们将 response.url
添加到other_url"字段中.
I mean assume the example below. In the parse_page2
function we add the response.url
into the 'other_url' field.
在parse_page2
执行完成后,是否有办法在parse_page1
中获取'other_url'?
Does it exist a way to get 'other_url' inside parse_page1
after the execution of parse_page2
has completed?
def parse_page1(self, response):
item = MyItem()
item['main_url'] = response.url
request = scrapy.Request("http://www.example.com/some_page.html",
callback=self.parse_page2)
request.meta['item'] = item
return request
def parse_page2(self, response):
item = response.meta['item']
item['other_url'] = response.url
return item
您可以简单地在 meta dict 中传递 response.url,并在您的 parse_page2 函数中创建项目,而不是在 parse_page1 函数中创建您的项目.
Instead of creating your item in the parse_page1 function, you can simply pass the response.url in the meta dict, and create the item in your parse_page2 function.
def parse_page1(self, response):
return Request(url="http://www.example.com/some_page.html",
meta={'main_url':reponse.url},
callback=self.parse_page2)
def parse_page2(self, response):
item = MyItem()
item['main_url'] = response.meta['main_url']
item['other_url'] = response.url
return item
或者,如果你真的想从 parse_page2 返回信息,你可以回调 parse_page1,并在你的函数中添加一个条件:
Or, if you really want to return the info from the parse_page2, you can callback parse_page1, and add a conditional in your function:
def parse_page1(self, response):
if "other_url" in response.meta:
do something
else:
do something else