Python中的AWS Lambda和Open CV的无服务器问题
我正在开发一种微服务,以分析上传到S3 AWS Bucket的图像.我正在使用无服务器框架.我正在使用virtualenv通过PIP和serverless-python-requirements插件安装依赖项,以将这些依赖项部署到Lambda函数.
I am developing a microservice to analyze an image uploaded to an S3 AWS Bucket. I am using Serverless framework. I am using virtualenv to install the dependencies with PIP and serverless-python-requirements plugin to deploy these dependencies to the Lambda function.
但是,由于缺少.so文件,部署微服务时出现错误.我得到的错误是
However I am having an error when I deploy the microservice because of a missing .so file. The error I get is
无法导入模块'handlers.image':libgthread-2.0.so.0:无法打开共享对象文件:没有此类文件
我的serverless.yml文件是
My serverless.yml file is
service: sls-covid
provider:
name: aws
runtime: python3.8
profile: testuser
stage: ${opt:stage, 'staging'}
environment: ${file(environment.yml):${self:provider.stage}}
region: ${self:provider.environment.REGION}
iamRoleStatements:
# para poder leer y escribir en el bucket
- Effect: "Allow"
Action:
- "s3:*"
Resource: "*"
custom:
pythonRequirements:
dockerizePip: non-linux
package:
individually: true
exclude:
- node_modules/**
- env/**
- package.json
- package-lock.json
functions:
analyzeImage:
# para analizar la imagen cuando se almacena en S3
handler: handlers.image.analyze
events:
- s3:
bucket: ${self:service}-${self:provider.stage}-images
event: s3:ObjectCreated:*
rules:
- prefix: uploads/
plugins:
- serverless-python-requirements
相关代码为:
import json
import logging
import boto3
from pydicom import dcmread
import numpy as np
# here the code stops working
import cv2
from pydicom.filebase import DicomBytesIO
logger = logging.getLogger()
logger.setLevel(logging.INFO)
s3 = boto3.client('s3')
def analyze(event, context):
print("=== event: ", event)
# code to read uploaded image from bucket and create a ds object
# code below is never executed since the error appears when the cv2 library is loaded
img = cv2.resize(ds.pixel_array, (224, 224))
我的requirements.txt内容如下:
And my requirements.txt contents is as follows:
numpy==1.18.4
opencv-python==4.2.0.34
pandas==1.0.3
pydicom==1.4.2
python-dateutil==2.8.1
pytz==2018.7
six==1.14.0
我的问题是.如何使用插件将opencv依赖项正确上传到我的lambda函数?我应该以其他方式执行此操作吗?
My question is. How can I properly upload opencv dependencies to my lambda function using the plugin? Should I do this in another way?
谢谢!
Based on the doc, you need to include those manually using dockerExtraFiles
option.
=======================更新=======================
======================= UPDATE =======================
打包Python依赖关系并为AWS Lambda构建部署程序包可能非常棘手.深入研究后,需要进行以下操作才能使其正常运行.
Packaging Python dependencies and building deployment package for AWS Lambda can be quite tricky. After digging around, here are the things that you need to do to make it work.
FROM lambci/lambda:build-python3.8
RUN yum -y install libXext libSM libXrender
步骤2:修改 serverless.yml
service: sls-covid
provider:
name: aws
runtime: python3.8
profile: testuser
stage: ${opt:stage, 'staging'}
environment: ${file(environment.yml):${self:provider.stage}}
region: ${self:provider.environment.REGION}
iamRoleStatements:
# para poder leer y escribir en el bucket
- Effect: "Allow"
Action:
- "s3:*"
Resource: "*"
custom:
pythonRequirements:
dockerizePip: non-linux
########### BEGIN ###########
dockerFile: Dockerfile
dockerExtraFiles:
- /lib64/libgthread-2.0.so.0
- /lib64/libglib-2.0.so.0
- /lib64/libSM.so.6
- /lib64/libICE.so.6
- /lib64/libXrender.so.1
- /lib64/libXext.so.6
- /lib64/libX11.so.6
- /lib64/libuuid.so.1
- /lib64/libxcb.so.1
- /lib64/libXau.so.6
########### END ###########
package:
individually: true
exclude:
- node_modules/**
- env/**
- package.json
- package-lock.json
functions:
analyzeImage:
# para analizar la imagen cuando se almacena en S3
handler: handlers.image.analyze
events:
- s3:
bucket: ${self:service}-${self:provider.stage}-images
event: s3:ObjectCreated:*
rules:
- prefix: uploads/
plugins:
- serverless-python-requirements
第3步:Hello world测试
$ sls invoke -f analyzeImage --log
null
--------------------------------------------------------------------
START RequestId: 00ccd940-cf8a-46ed-8671-65e597f997a0 Version: $LATEST
=== event: {}
END RequestId: 00ccd940-cf8a-46ed-8671-65e597f997a0
REPORT RequestId: 00ccd940-cf8a-46ed-8671-65e597f997a0 Duration: 1.41 ms Billed Duration: 100 ms Memory Size: 1024 MB Max Memory Used: 121 MB Init Duration: 916.32 ms
希望有帮助.