通过网络表单提交数据并提取结果

问题描述：

我的python级别是新手.我从来没有写过网络爬虫或爬虫.我已经编写了一个 python 代码来连接到一个 api 并提取我想要的数据.但是对于一些提取的数据，我想获得作者的性别.我找到了这个网站 http://bookblog.net/gender/genie.php 但缺点是没有可用的 api.我想知道如何编写python将数据提交到页面中的表单并提取返回数据.如果我能在这方面得到一些指导，那将是非常有帮助的.

My python level is Novice. I have never written a web scraper or crawler. I have written a python code to connect to an api and extract the data that I want. But for some the extracted data I want to get the gender of the author. I found this web site http://bookblog.net/gender/genie.php but downside is there isn't an api available. I was wondering how to write a python to submit data to the form in the page and extract the return data. It would be a great help if I could get some guidance on this.

这是dom的形式:

<form action="analysis.php" method="POST">
<textarea cols="75" rows="13" name="text"></textarea>
<div class="copyright">(NOTE: The genie works best on texts of more than 500 words.)</div>
<p>
<b>Genre:</b>
<input type="radio" value="fiction" name="genre">
fiction&nbsp;&nbsp;
<input type="radio" value="nonfiction" name="genre">
nonfiction&nbsp;&nbsp;
<input type="radio" value="blog" name="genre">
blog entry
</p>
<p>
</form>

结果页dom:

<p>
<b>The Gender Genie thinks the author of this passage is:</b>
male!
</p>

答

无需使用机械化，只需在 POST 请求中发送正确的表单数据即可.

No need to use mechanize, just send the correct form data in a POST request.

另外，使用正则表达式来解析 HTML 也是一个坏主意.最好使用像 lxml.html 这样的 HTML 解析器.

Also, using regular expression to parse HTML is a bad idea. You would be better off using a HTML parser like lxml.html.

import requests
import lxml.html as lh


def gender_genie(text, genre):
    url = 'http://bookblog.net/gender/analysis.php'
    caption = 'The Gender Genie thinks the author of this passage is:'

    form_data = {
        'text': text,
        'genre': genre,
        'submit': 'submit',
    }

    response = requests.post(url, data=form_data)

    tree = lh.document_fromstring(response.content)

    return tree.xpath("//b[text()=$caption]", caption=caption)[0].tail.strip()


if __name__ == '__main__':
    print gender_genie('I have a beard!', 'blog')

通过网络表单提交数据并提取结果

相关推荐