使用Lambda从S3读取数据
我在AWS的S3存储桶中存储了一系列json文件.
I have a range of json files stored in an S3 bucket on AWS.
我希望使用AWS lambda python服务来解析此json并将解析的结果发送到AWS RDS MySQL数据库.
I wish to use AWS lambda python service to parse this json and send the parsed results to an AWS RDS MySQL database.
我有一个稳定的python脚本,用于解析和写入数据库.我需要使用lambda脚本来遍历json文件(添加它们时).
I have a stable python script for doing the parsing and writing to the database. I need to lambda script to iterate through the json files (when they are added).
每个json文件都包含一个列表,简单地由results = [content]
Each json file contains a list, simple consisting of results = [content]
我想要的是伪代码:
- 连接到S3存储桶(
jsondata
) - 读取JSON文件(
results
)的内容 - 对此数据执行脚本(
results
)
- Connect to the S3 bucket (
jsondata
) - Read the contents of the JSON file (
results
) - Execute my script for this data (
results
)
我可以列出我拥有的存储桶:
I can list the buckets I have by:
import boto3
s3 = boto3.resource('s3')
for bucket in s3.buckets.all():
print(bucket.name)
给予:
jsondata
但是我无法访问该存储桶以读取其结果.
But I cannot access this bucket to read its results.
似乎没有read
或load
函数.
我希望有
for bucket in s3.buckets.all():
print(bucket.contents)
编辑
我误会了一些东西. Lambda必须自己下载文件,而不是在S3中读取文件.
I am misunderstanding something. Rather than reading the file in S3, lambda must download it itself.
来自在此似乎您必须给lambda一个下载路径,它可以从该路径访问文件本身
From here it seems that you must give lambda a download path, from which it can access the files itself
import libraries
s3_client = boto3.client('s3')
def function to be executed:
blah blah
def handler(event, context):
for record in event['Records']:
bucket = record['s3']['bucket']['name']
key = record['s3']['object']['key']
download_path = '/tmp/{}{}'.format(uuid.uuid4(), key)
s3_client.download_file(bucket, key, download_path)
You can use bucket.objects.all()
to get a list of the all objects in the bucket (you also have alternative methods like filter
, page_size
and limit
depending on your need)
These methods return an iterator with S3.ObjectSummary
objects in it, from there you can use the method object.get
to retrieve the file.