Scrapy:如何从脚本中导出 Json

问题描述:

我用scrapy创建了一个网络爬虫,但我的电话号码有问题,因为它是一个脚本.脚本是:

I created a web crawler with scrapy, but I've a problem with phone number because it is into a script. The script is:

<script data-n-head="true" type="application/ld+json">{"@context":"http://schema.org","@type":"LocalBusiness","name":"Clínica Dental Reina Victoria 23","description":".TU CLÍNICA DENTAL DE REFERENCIA EN MADRID","logo":"https://estaticos.qdq.com/CMS/directory/logos/c/l/clinica-dental-reina-victoria.png","image":"https://estaticos.qdq.com/coverphotos/098/535/ed1c5ffcf38241f8b83a1808af51a615.jpg","url":"https://www.clinicadental-reinavictoria.es/","hasMap":"https://www.google.com/maps/search/?api=1&query=40.4469174,-3.7087934","telephone":"+34915340309","address":{"@type":"PostalAddress","streetAddress":"Av. Reina Victoria 23","addressLocality":"MADRID","addressRegion":"Madrid","postalCode":"28003"}}</script>

此脚本在不同的页面中更改,但只更改电话号码

This script change in diferents page, but only change the phone number

我用 Xpath 提取脚本

I extract the script with Xpath

data = response.xpath('/html/head/script[3]').extract()
        decoded = json.loads(data.telephone("utf-8"))
        ml_item['datos'] = decoded['telephone']

我认为我需要自定义管道来提取电话号码

I think that I need the custom pipelines for extract the phone number

在 pipelines.py 我添加了 jsonWriter 行

In pipelines.py i added the jsonWriter line

ITEM_PIPELINES = {'mercado.pipelines.MercadoPipeline': 500,
                    'mercado.pipelines.MercadoImagenesPipeline': 600,
                    'mercado.pipelines.JsonWriterPipeline': 800, }

但我需要在 pipelines.py 中添加一些代码来定义 JsonWriterPipeline.控制台返回此错误:

But I need added some code in pipelines.py for define JsonWriterPipeline. The console return this error:

raise NameError("Module '%s' doesn't define any object named '%s'" % (module, name))
NameError: Module 'mercado.pipelines' doesn't define any object named 'JsonWriterPipeline'

我将所有数字与名称、网站等其他信息一起保存在 CSV 文件中...

I save all numbers in a CSV file with other information like Name, Web, etc...

如果你已经抓取了 script 标签内的内容,那就很简单了

it's simple if you already crawled the contents inside the script tag

import re

script = '{"@context":"http://schema.org","@type":"LocalBusiness","name":"Clínica Dental Reina Victoria 23","description":".TU CLÍNICA DENTAL DE REFERENCIA EN MADRID","logo":"https://estaticos.qdq.com/CMS/directory/logos/c/l/clinica-dental-reina-victoria.png","image":"https://estaticos.qdq.com/coverphotos/098/535/ed1c5ffcf38241f8b83a1808af51a615.jpg","url":"https://www.clinicadental-reinavictoria.es/","hasMap":"https://www.google.com/maps/search/?api=1&query=40.4469174,-3.7087934","telephone":"+34915340309","address":{"@type":"PostalAddress","streetAddress":"Av. Reina Victoria 23","addressLocality":"MADRID","addressRegion":"Madrid","postalCode":"28003"}}'

phone_number = re.search(r'"telephone":"(.*?)","address"', script).group(1)

print(phone_number)