将PyPI软件包添加到Composer环境时出现错误

问题描述:

我刚刚创建了一个新的Cloud Composer环境,尝试向该环境中添加一些PyPI软件包(例如 google-cloud-pubsub )时遇到错误.

I just created a new Cloud Composer environment and I'm getting an error when trying to add some PyPI packages to the environment (for example google-cloud-pubsub).

这是有关环境的数据:

This is the data about the environment:

尝试添加 google-cloud-pubsub 时遇到的错误只是这样:无法安装PyPI软件包.

The error I get when trying to add google-cloud-pubsub is just this: Failed to install PyPI packages.

有人知道吗?

谢谢:)

云构建:

+ python3 -m pip install -r requirements.txt
Collecting google-cloud-pubsub==1.2.0 (from -r requirements.txt (line 1))
  Downloading https://files.pythonhosted.org/packages/c5/9e/e95b75789ba589e022f856754da8c6a7e7d607861254c7fd8ccdf5d7bdac/google_cloud_pubsub-1.2.0-py2.py3-none-any.whl (121kB)
Requirement already satisfied: apache-airflow==1.10.2-composer in /usr/local/lib/airflow (from -r requirements.txt (line 2))
Collecting google-api-core[grpc]<2.0.0dev,>=1.14.0 (from google-cloud-pubsub==1.2.0->-r requirements.txt (line 1))
  Downloading https://files.pythonhosted.org/packages/63/7e/a523169b0cc9ce62d56e07571db927286a94b1a5f51ac220bd97db825c77/google_api_core-1.16.0-py2.py3-none-any.whl (70kB)
Collecting grpc-google-iam-v1<0.13dev,>=0.12.3 (from google-cloud-pubsub==1.2.0->-r requirements.txt (line 1))
  Downloading https://files.pythonhosted.org/packages/65/19/2060c8faa325fddc09aa67af98ffcb6813f39a0ad805679fa64815362b3a/grpc-google-iam-v1-0.12.3.tar.gz
Requirement already satisfied: alembic<0.9,>=0.8.3 in /opt/python3.6/lib/python3.6/site-packages (from apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: configparser<3.6.0,>=3.5.0 in /opt/python3.6/lib/python3.6/site-packages (from apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: croniter<0.4,>=0.3.17 in /opt/python3.6/lib/python3.6/site-packages (from apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: dill<0.3,>=0.2.2 in /opt/python3.6/lib/python3.6/site-packages (from apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: enum34~=1.1.6 in /opt/python3.6/lib/python3.6/site-packages (from apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: flask<0.13,>=0.12.4 in /opt/python3.6/lib/python3.6/site-packages (from apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: flask-appbuilder==1.12.1 in /opt/python3.6/lib/python3.6/site-packages (from apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: flask-admin==1.5.2 in /opt/python3.6/lib/python3.6/site-packages (from apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: flask-caching<1.4.0,>=1.3.3 in /opt/python3.6/lib/python3.6/site-packages (from apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: flask-login<0.5,>=0.3 in /opt/python3.6/lib/python3.6/site-packages (from apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: flask-swagger==0.2.13 in /opt/python3.6/lib/python3.6/site-packages (from apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: flask-wtf<0.15,>=0.14.2 in /opt/python3.6/lib/python3.6/site-packages (from apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: funcsigs==1.0.0 in /opt/python3.6/lib/python3.6/site-packages (from apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: future<0.17,>=0.16.0 in /opt/python3.6/lib/python3.6/site-packages (from apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: gitpython>=2.0.2 in /opt/python3.6/lib/python3.6/site-packages (from apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: gunicorn<20.0,>=19.4.0 in /opt/python3.6/lib/python3.6/site-packages (from apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: iso8601>=0.1.12 in /opt/python3.6/lib/python3.6/site-packages (from apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: jsonschema~=3.0 in /opt/python3.6/lib/python3.6/site-packages (from apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: json-merge-patch==0.2 in /opt/python3.6/lib/python3.6/site-packages (from apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: jinja2<=2.10.0,>=2.7.3 in /opt/python3.6/lib/python3.6/site-packages (from apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: markdown<3.0,>=2.5.2 in /opt/python3.6/lib/python3.6/site-packages (from apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: pandas<1.0.0,>=0.17.1 in /opt/python3.6/lib/python3.6/site-packages (from apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: pendulum==1.4.4 in /opt/python3.6/lib/python3.6/site-packages (from apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: psutil<6.0.0,>=4.2.0 in /opt/python3.6/lib/python3.6/site-packages (from apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: pygments<3.0,>=2.0.1 in /opt/python3.6/lib/python3.6/site-packages (from apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: python-daemon<2.2,>=2.1.1 in /opt/python3.6/lib/python3.6/site-packages (from apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: python-dateutil<3,>=2.3 in /opt/python3.6/lib/python3.6/site-packages (from apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: python-nvd3==0.15.0 in /opt/python3.6/lib/python3.6/site-packages (from apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: requests<3,>=2.20.0 in /opt/python3.6/lib/python3.6/site-packages (from apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: setproctitle<2,>=1.1.8 in /opt/python3.6/lib/python3.6/site-packages (from apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: sqlalchemy<1.3.0,>=1.1.15 in /opt/python3.6/lib/python3.6/site-packages (from apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: tabulate<=0.8.2,>=0.7.5 in /opt/python3.6/lib/python3.6/site-packages (from apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: tenacity==4.8.0 in /opt/python3.6/lib/python3.6/site-packages (from apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: thrift>=0.9.2 in /opt/python3.6/lib/python3.6/site-packages (from apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: tzlocal>=1.4 in /opt/python3.6/lib/python3.6/site-packages (from apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: unicodecsv>=0.14.1 in /opt/python3.6/lib/python3.6/site-packages (from apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: werkzeug<0.15.0,>=0.14.1 in /opt/python3.6/lib/python3.6/site-packages (from apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: zope.deprecation<5.0,>=4.0 in /opt/python3.6/lib/python3.6/site-packages (from apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Collecting googleapis-common-protos<2.0dev,>=1.6.0 (from google-api-core[grpc]<2.0.0dev,>=1.14.0->google-cloud-pubsub==1.2.0->-r requirements.txt (line 1))
  Downloading https://files.pythonhosted.org/packages/05/46/168fd780f594a4d61122f7f3dc0561686084319ad73b4febbf02ae8b32cf/googleapis-common-protos-1.51.0.tar.gz
Requirement already satisfied: six>=1.10.0 in /opt/python3.6/lib/python3.6/site-packages (from google-api-core[grpc]<2.0.0dev,>=1.14.0->google-cloud-pubsub==1.2.0->-r requirements.txt (line 1))
Requirement already satisfied: pytz in /opt/python3.6/lib/python3.6/site-packages (from google-api-core[grpc]<2.0.0dev,>=1.14.0->google-cloud-pubsub==1.2.0->-r requirements.txt (line 1))
Requirement already satisfied: google-auth<2.0dev,>=0.4.0 in /opt/python3.6/lib/python3.6/site-packages (from google-api-core[grpc]<2.0.0dev,>=1.14.0->google-cloud-pubsub==1.2.0->-r requirements.txt (line 1))
Requirement already satisfied: setuptools>=34.0.0 in /opt/python3.6/lib/python3.6/site-packages (from google-api-core[grpc]<2.0.0dev,>=1.14.0->google-cloud-pubsub==1.2.0->-r requirements.txt (line 1))
Requirement already satisfied: protobuf>=3.4.0 in /opt/python3.6/lib/python3.6/site-packages (from google-api-core[grpc]<2.0.0dev,>=1.14.0->google-cloud-pubsub==1.2.0->-r requirements.txt (line 1))
Requirement already satisfied: grpcio<2.0dev,>=1.8.2; extra == "grpc" in /opt/python3.6/lib/python3.6/site-packages (from google-api-core[grpc]<2.0.0dev,>=1.14.0->google-cloud-pubsub==1.2.0->-r requirements.txt (line 1))
Requirement already satisfied: python-editor>=0.3 in /opt/python3.6/lib/python3.6/site-packages (from alembic<0.9,>=0.8.3->apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: Mako in /opt/python3.6/lib/python3.6/site-packages (from alembic<0.9,>=0.8.3->apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: itsdangerous>=0.21 in /opt/python3.6/lib/python3.6/site-packages (from flask<0.13,>=0.12.4->apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: click>=2.0 in /opt/python3.6/lib/python3.6/site-packages (from flask<0.13,>=0.12.4->apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: colorama<1,>=0.3.9 in /opt/python3.6/lib/python3.6/site-packages (from flask-appbuilder==1.12.1->apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: Flask-OpenID<2,>=1.2.5 in /opt/python3.6/lib/python3.6/site-packages (from flask-appbuilder==1.12.1->apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: Flask-Babel<1,>=0.11.1 in /opt/python3.6/lib/python3.6/site-packages (from flask-appbuilder==1.12.1->apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: Flask-SQLAlchemy<3,>=2.3 in /opt/python3.6/lib/python3.6/site-packages (from flask-appbuilder==1.12.1->apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: wtforms in /opt/python3.6/lib/python3.6/site-packages (from flask-admin==1.5.2->apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: PyYAML>=3.0 in /opt/python3.6/lib/python3.6/site-packages (from flask-swagger==0.2.13->apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: ordereddict in /opt/python3.6/lib/python3.6/site-packages (from funcsigs==1.0.0->apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: gitdb2>=2.0.0 in /opt/python3.6/lib/python3.6/site-packages (from gitpython>=2.0.2->apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: attrs>=17.4.0 in /opt/python3.6/lib/python3.6/site-packages (from jsonschema~=3.0->apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: pyrsistent>=0.14.0 in /opt/python3.6/lib/python3.6/site-packages (from jsonschema~=3.0->apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: MarkupSafe>=0.23 in /opt/python3.6/lib/python3.6/site-packages (from jinja2<=2.10.0,>=2.7.3->apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: numpy>=1.12.0 in /opt/python3.6/lib/python3.6/site-packages (from pandas<1.0.0,>=0.17.1->apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: pytzdata>=2018.3.0.0 in /opt/python3.6/lib/python3.6/site-packages (from pendulum==1.4.4->apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: lockfile>=0.10 in /opt/python3.6/lib/python3.6/site-packages (from python-daemon<2.2,>=2.1.1->apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: docutils in /opt/python3.6/lib/python3.6/site-packages (from python-daemon<2.2,>=2.1.1->apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: python-slugify>=1.2.5 in /opt/python3.6/lib/python3.6/site-packages (from python-nvd3==0.15.0->apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /opt/python3.6/lib/python3.6/site-packages (from requests<3,>=2.20.0->apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: idna<2.9,>=2.5 in /opt/python3.6/lib/python3.6/site-packages (from requests<3,>=2.20.0->apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: certifi>=2017.4.17 in /opt/python3.6/lib/python3.6/site-packages (from requests<3,>=2.20.0->apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /opt/python3.6/lib/python3.6/site-packages (from requests<3,>=2.20.0->apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: monotonic>=0.6 in /opt/python3.6/lib/python3.6/site-packages (from tenacity==4.8.0->apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: rsa>=3.1.4 in /opt/python3.6/lib/python3.6/site-packages (from google-auth<2.0dev,>=0.4.0->google-api-core[grpc]<2.0.0dev,>=1.14.0->google-cloud-pubsub==1.2.0->-r requirements.txt (line 1))
Requirement already satisfied: cachetools>=2.0.0 in /opt/python3.6/lib/python3.6/site-packages (from google-auth<2.0dev,>=0.4.0->google-api-core[grpc]<2.0.0dev,>=1.14.0->google-cloud-pubsub==1.2.0->-r requirements.txt (line 1))
Requirement already satisfied: pyasn1-modules>=0.2.1 in /opt/python3.6/lib/python3.6/site-packages (from google-auth<2.0dev,>=0.4.0->google-api-core[grpc]<2.0.0dev,>=1.14.0->google-cloud-pubsub==1.2.0->-r requirements.txt (line 1))
Requirement already satisfied: python3-openid>=2.0 in /opt/python3.6/lib/python3.6/site-packages (from Flask-OpenID<2,>=1.2.5->flask-appbuilder==1.12.1->apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: Babel>=2.3 in /opt/python3.6/lib/python3.6/site-packages (from Flask-Babel<1,>=0.11.1->flask-appbuilder==1.12.1->apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: smmap2>=2.0.0 in /opt/python3.6/lib/python3.6/site-packages (from gitdb2>=2.0.0->gitpython>=2.0.2->apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: text-unidecode==1.2 in /opt/python3.6/lib/python3.6/site-packages (from python-slugify>=1.2.5->python-nvd3==0.15.0->apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: pyasn1>=0.1.3 in /opt/python3.6/lib/python3.6/site-packages (from rsa>=3.1.4->google-auth<2.0dev,>=0.4.0->google-api-core[grpc]<2.0.0dev,>=1.14.0->google-cloud-pubsub==1.2.0->-r requirements.txt (line 1))
Requirement already satisfied: defusedxml in /opt/python3.6/lib/python3.6/site-packages (from python3-openid>=2.0->Flask-OpenID<2,>=1.2.5->flask-appbuilder==1.12.1->apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Building wheels for collected packages: grpc-google-iam-v1, googleapis-common-protos
  Running setup.py bdist_wheel for grpc-google-iam-v1: started
  Running setup.py bdist_wheel for grpc-google-iam-v1: finished with status 'done'
  Stored in directory: /root/.cache/pip/wheels/de/3a/83/77a1e18e1a8757186df834b86ce6800120ac9c79cd8ca4091b
  Running setup.py bdist_wheel for googleapis-common-protos: started
  Running setup.py bdist_wheel for googleapis-common-protos: finished with status 'done'
  Stored in directory: /root/.cache/pip/wheels/2c/f9/7f/6eb87e636072bf467e25348bbeb96849333e6a080dca78f706
Successfully built grpc-google-iam-v1 googleapis-common-protos
Installing collected packages: googleapis-common-protos, google-api-core, grpc-google-iam-v1, google-cloud-pubsub
  Found existing installation: googleapis-common-protos 1.5.8
    Uninstalling googleapis-common-protos-1.5.8:
      Successfully uninstalled googleapis-common-protos-1.5.8
  Found existing installation: google-api-core 1.8.1
    Uninstalling google-api-core-1.8.1:
      Successfully uninstalled google-api-core-1.8.1
  Found existing installation: grpc-google-iam-v1 0.11.4
    Uninstalling grpc-google-iam-v1-0.11.4:
      Successfully uninstalled grpc-google-iam-v1-0.11.4
Successfully installed google-api-core-1.16.0 google-cloud-pubsub-1.2.0 googleapis-common-protos-1.51.0 grpc-google-iam-v1-0.12.3
You are using pip version 9.0.3, however version 20.0.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
+ python3 -m pip check
google-cloud-spanner 1.8.0 has requirement grpc-google-iam-v1<0.12dev,>=0.11.4, but you have grpc-google-iam-v1 0.12.3.
google-cloud-bigtable 0.32.0 has requirement grpc-google-iam-v1<0.12dev,>=0.11.4, but you have grpc-google-iam-v1 0.12.3.
You are using pip version 9.0.3, however version 20.0.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
The command '/bin/sh -c bash installer.sh $COMPOSER_PYTHON_VERSION' returned a non-zero code: 1
ERROR
ERROR: build step 0 "gcr.io/cloud-builders/docker" failed: step exited with non-zero status: 1

错误无法安装PyPI软件包"表示Cloud Build无法为您的Cloud Composer环境构建Docker映像.这通常是因为PyPI软件包安装失败,或者存在不兼容的依赖项.在您的情况下,是后者:

The error "Failed to install PyPI packages" means Cloud Build was unable to build Docker images for your Cloud Composer environment. This is usually because there was a PyPI package installation failure, or there were incompatible dependencies. In your case, it is the latter:

+ python3 -m pip check
google-cloud-spanner 1.8.0 has requirement grpc-google-iam-v1<0.12dev,>=0.11.4, but you have grpc-google-iam-v1 0.12.3.
google-cloud-bigtable 0.32.0 has requirement grpc-google-iam-v1<0.12dev,>=0.11.4, but you have grpc-google-iam-v1 0.12.3.

要检查不兼容的依赖项(并防止它们破坏您的环境),Composer在所有软件包安装结束时运行 pip check .您将需要解决这些冲突,环境更新才能成功.

To check for incompatible dependencies (and to prevent them from breaking your environment), Composer runs pip check at the end of all package installs. You will need to resolve these conflicts for the environment update to succeed.