GCloud上传httplib2.RedirectMissingLocation:已重定向,但响应缺少Location:标头
我正在尝试使用简单的python程序将小文件上传到gcloud
I am attempting to upload a small file to gcloud using a simple python program
client = storage.Client(project=GCLOUD_PROJECT)
bucket = client.get_bucket(GCLOUD_BUCKET)
blob = bucket.blob(GCLOUD_FILE_ON_CLOUD)
blob.upload_from_filename(GCLOUD_FILE_LOCAL)
直到最近它一直在工作,并且发生了一些变化.现在,每当我上传大于5MB的文件时,都会出现以下错误.小于或等于5MB的文件将通过.大小还不足以实现可恢复的上传,是吗?
It had been working until recently and something changed. Now, whenever I upload a file greater than 5MB I get the below error. Files less than or equal to 5MB goes through. The size isn't large enough to implement resumable upload, is it?
Traceback (most recent call last):
File "/Users/mmumshad/PycharmProjects/quiz-python-flask-angular/gcloud_upload.py", line 40, in <module>
blob.upload_from_filename(GCLOUD_FILE_LOCAL)
File "/Users/mmumshad/PycharmProjects/quiz-python-flask-angular/venv36/lib/python3.6/site-packages/gcloud/storage/blob.py", line 597, in upload_from_filename
encryption_key=encryption_key, client=client)
File "/Users/mmumshad/PycharmProjects/quiz-python-flask-angular/venv36/lib/python3.6/site-packages/gcloud/storage/blob.py", line 543, in upload_from_file
http_response = upload.stream_file(use_chunks=True)
File "/Users/mmumshad/PycharmProjects/quiz-python-flask-angular/venv36/lib/python3.6/site-packages/gcloud/streaming/transfer.py", line 1086, in stream_file
response = send_func(self.stream.tell())
File "/Users/mmumshad/PycharmProjects/quiz-python-flask-angular/venv36/lib/python3.6/site-packages/gcloud/streaming/transfer.py", line 1215, in _send_chunk
return self._send_media_request(request, end)
File "/Users/mmumshad/PycharmProjects/quiz-python-flask-angular/venv36/lib/python3.6/site-packages/gcloud/streaming/transfer.py", line 1125, in _send_media_request
self.bytes_http, request, retries=self.num_retries)
File "/Users/mmumshad/PycharmProjects/quiz-python-flask-angular/venv36/lib/python3.6/site-packages/gcloud/streaming/http_wrapper.py", line 423, in make_api_request
check_response_func=check_response_func)
File "/Users/mmumshad/PycharmProjects/quiz-python-flask-angular/venv36/lib/python3.6/site-packages/gcloud/streaming/http_wrapper.py", line 371, in _make_api_request_no_retry
redirections=redirections, connection_type=connection_type)
File "/Users/mmumshad/PycharmProjects/quiz-python-flask-angular/venv36/lib/python3.6/site-packages/oauth2client/transport.py", line 175, in new_request
redirections, connection_type)
File "/Users/mmumshad/PycharmProjects/quiz-python-flask-angular/venv36/lib/python3.6/site-packages/oauth2client/transport.py", line 282, in request
connection_type=connection_type)
File "/Users/mmumshad/PycharmProjects/quiz-python-flask-angular/venv36/lib/python3.6/site-packages/httplib2/__init__.py", line 1986, in request
cachekey,
File "/Users/mmumshad/PycharmProjects/quiz-python-flask-angular/venv36/lib/python3.6/site-packages/httplib2/__init__.py", line 1685, in _request
content,
httplib2.RedirectMissingLocation: Redirected but the response is missing a Location: header.
当我调试时,我看到以下内容.
When I debug I see the below.
{
'content-type': 'text/plain; charset=utf-8',
'range': 'bytes=0-1048575',
'content-length': '0',
'date': 'Sun, 19 Jan 2020 23:52:13 GMT',
'server': 'UploadServer',
'alt-svc': 'quic=":443"; ma=2592000; v="46,43",h3-Q050=":443"; ma=2592000,h3-Q049=":443"; ma=2592000,h3-Q048=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000',
'status': '308'}
我的点子列表以供参考.操作系统是MAC OSx.我也在Linux上进行了测试.同样的问题.
My pip list for reference. OS is MAC OSx. I tested on Linux as well. Same issue.
adal 1.2.2
bson 0.5.8
cachetools 3.1.1
certifi 2019.9.11
cffi 1.13.1
chardet 3.0.4
cryptography 2.8
dnspython 1.16.0
gcloud 0.18.3
gitdb2 2.0.6
GitPython 3.0.5
google-auth 1.6.3
googleapis-common-protos 1.51.0
httplib2 0.16.0
idna 2.7
itsdangerous 1.1.0
MarkupSafe 1.1.1
oauth2client 4.1.3
oauthlib 3.1.0
pip 19.0.3
protobuf 3.11.2
pyasn1 0.4.7
pyasn1-modules 0.2.7
pycparser 2.19
PyJWT 1.7.1
python-dateutil 2.8.0
requests 2.19.1
requests-oauthlib 1.2.0
rsa 4.0
setuptools 40.8.0
six 1.12.0
smmap2 2.0.5
urllib3 1.23
websocket-client 0.56.0
Werkzeug 0.16.0
这是上周的工作.最近有什么变化吗?
This was working last week. Has something changed recently?
解决方案
gcloud
软件包已弃用两次,并且与httplib2>=0.16
不兼容.正确的解决方案是使用google-cloud-*
软件包家族.
Solutions
gcloud
package is deprecated twice and is not compatible with httplib2>=0.16
. Proper solution is to use google-cloud-*
packages family.
google-api-python-client>=1.7.12
使用的是redirect_codes API,请升级,它可以正常工作.
google-api-python-client>=1.7.12
is using redirect_codes API, please upgrade, it just works.
httplib2 v0.17.0刚刚发布,具有修改被视为重定向的响应代码集的功能.如果可以修改创建Http
对象的代码,这是最好的选择:
httplib2 v0.17.0 is just released with ability to modify set of response codes treated as redirects. This is the best option if you can modify code that creates Http
object:
http = httplib2.Http()
http.redirect_codes = http.redirect_codes - {308}
不可能,请编辑您的requirements.txt以固定httplib2<0.16.0
Iff that's not possible, edit your requirements.txt to pin down httplib2<0.16.0
Google云存储服务器使用HTTP 308 用于特殊的可恢复上传功能,该功能有点类似于重试相同方法到相同位置" ,但不完全是这样.
google cloud storage server uses HTTP 308 for special resumable uploads feature which somewhat resembles "retry same method to same location", but not quite.
以上(可能)是PyPI软件包 google-resumable-media 的理由.与通用的HTTP客户端不同,该版本由gcloud相关软件包的最新版本使用,并以类似的方式处理200和308.
Above is (probably) rationale for PyPI package google-resumable-media which is used by more recent incarnations of gcloud related packages and handles 200 and 308 in similar manner, unlike generic HTTP client should.
历史背景:
- 2016 软件包gcloud 已被弃用,转而使用软件包
google-cloud
- 2017 google-cloud-python 从httplib2切换HTTP传输请求
- 2018 软件包google-cloud 被再次弃用,转而支持google-cloud- *包
- 2020 httplib2 v0.16获得了对每个 RFC7538 的308个重定向的支持.
- 2016 package gcloud was deprecated in favor of package
google-cloud
- 2017 google-cloud-python switches HTTP transport from httplib2 to requests
- 2018 package google-cloud was deprecated again in favor of google-cloud-* packages
- 2020 httplib2 v0.16 gained support for 308 redirects per RFC7538
很抱歉有坏消息.作为HTTP爱好者,我偏向308支持.如果您有更好的主意,如何更好地处理这种情况,请与我们联系.
Sorry for bad news. As HTTP enthusiast, I'm biased towards 308 support. Please reach if you have better idea how to handle this situation more gracefully.