{"id":618,"date":"2018-07-24T17:59:24","date_gmt":"2018-07-24T09:59:24","guid":{"rendered":"https:\/\/vinta.ws\/code\/?p=618"},"modified":"2026-03-17T01:15:55","modified_gmt":"2026-03-16T17:15:55","slug":"integrate-with-google-cloud-api-in-python","status":"publish","type":"post","link":"https:\/\/vinta.ws\/code\/integrate-with-google-cloud-api-in-python.html","title":{"rendered":"Integrate with Google Cloud API in Python"},"content":{"rendered":"

google-cloud<\/code>, Python idiomatic clients for Google Cloud Platform services. There is an older Python library also officially supported by Google, google-api-python-client<\/code>, which is in maintenance mode.<\/p>\n

ref:
\nhttps:\/\/github.com\/googleapis\/google-cloud-python<\/a>
\nhttps:\/\/github.com\/googleapis\/google-api-python-client<\/a><\/p>\n

Installation<\/h2>\n
$ pip install google-cloud\n\n# you could also only install specific components\n$ pip install google-cloud-storage<\/code><\/pre>\n
ref:
\nhttps:\/\/pypi.org\/search\/?q=google+cloud<\/a><\/p>\n
Google Cloud Storage<\/h2>\n
It is worth noting that, initializing storage.Client()<\/code> is a blocking call.<\/p>\n
ref:
\nhttps:\/\/googleapis.github.io\/google-cloud-python\/latest\/storage\/buckets.html<\/a>
\nhttps:\/\/cloud.google.com\/storage\/docs\/reference\/libraries<\/a><\/p>\n
Upload From String<\/h3>\n
from google.cloud.storage.bucket import Bucket\nfrom google.cloud.storage.blob import Blob\n\ndef upload_from_string(bucket_id, content, filename, content_type):\n    client = storage.Client()\n    bucket = Bucket(client, bucket_id)\n    blob = Blob(filename, bucket)\n    blob.upload_from_string(content, content_type)<\/code><\/pre>\n
Upload From An URL<\/h3>\n
from google.cloud import storage\nimport requests\n\ndef upload_from_url(bucket_id, filename, url):\n    client = storage.Client()\n    session = requests.Session()\n    with session.get(url, stream=True) as response:\n        bucket = client.get_bucket(bucket_id)\n        blob = bucket.blob(filename)\n        blob.upload_from_file(response.raw, content_type=response.headers.get('Content-Type'))<\/code><\/pre>\n
Update A File's Metadata<\/h3>\n
from google.cloud import storage\n\ndef update_metadata(bucket, filepath, new_metadata):\n    bucket = task.storage_client.get_bucket(bucket)\n    blob = bucket.get_blob(filepath)\n    blob.metadata = {**blob.metadata, **new_metadata} if blob.metadata else new_metadata\n    blob.patch()\n\nnew_metadata = {\n    'Link': '<https:\/\/api.example.com\/users\/57c16f5bb811055b66d8ef46>; rel=\"user\"',\n}<\/code><\/pre>\n
ref:
\nhttps:\/\/github.com\/GoogleCloudPlatform\/google-cloud-python\/issues\/1185<\/a><\/p>\n
Copy A File<\/h3>\n
from google.cloud import storage\n\ndef copy_file(source_bucket, source_name, destination_bucket, destination_name):\n    storage_client = storage.Client()\n    source_bucket = storage_client.get_bucket(source_bucket)\n    source_file = source_bucket.blob(source_name)\n    destination_bucket = storage_client.get_bucket(destination_bucket)\n    destination_file = source_bucket.copy_blob(source_file, destination_bucket, destination_name)\n    return destination_file\n\nfile_ext_mapping = {\n    'image\/jpeg': 'jpg',\n    'video\/mp4': 'mp4',\n}\nfile_ext = file_ext_mapping[original_message.media.mimetype]\nsource_name = f'messages\/{original_message.id}.{file_ext}'\ndestination_name = f'messages\/{new_message.id}.{file_ext}'\n\ncopy_file(\n    source_bucket='asia.uploads.example.com',\n    source_name=source_name,\n    destination_bucket='asia.uploads.example.com',\n    destination_name=destination_name,\n)<\/code><\/pre>\n
ref:
\nhttps:\/\/cloud.google.com\/storage\/docs\/json_api\/v1\/objects\/copy<\/a>
\nhttps:\/\/cloud.google.com\/storage\/docs\/renaming-copying-moving-objects#storage-copy-object-python<\/a><\/p>\n
Copy A Folder With Batch Operations<\/h3>\n
from google.cloud import storage\n\ndef copy_files(source_bucket_name, source_name_prefix, destination_bucket_name, fix_destination_name_func=None):\n    storage_client = storage.Client()\n    source_bucket = storage_client.get_bucket(source_bucket_name)\n    destination_bucket = storage_client.get_bucket(destination_bucket_name)\n    blobs = source_bucket.list_blobs(prefix=source_name_prefix)\n\n    # YOU CANNOT DO THIS\n    # blobs is a HTTP iterator\n    # blobs.num_results always return 0\n    # if not blobs.num_results:\n    #     raise ValueError(f'No objects matched: gs:\/\/{source_bucket.name}\/{source_name_prefix}')\n\n    with storage_client.batch():\n        for source_blob in blobs:\n            destination_name = fix_destination_name_func(source_blob.name) if callable(fix_destination_name_func) else source_blob.name\n            source_bucket.copy_blob(source_blob, destination_bucket, destination_name)\n    return True\n\nsource_bucket_name = 'asia.uploads.example.com'\ndestination_bucket_name = 'asia.contents.example.com'\nsource_name_prefix = 'media\/123'\n\ncopy_files(\n    source_bucket_name=source_bucket_name,\n    destination_bucket_name=destination_bucket_name,\n    source_name_prefix=source_name_prefix,\n    fix_destination_name_func=lambda source_name: source_name.replace(source_name_prefix, 'forum-posts'),\n)<\/code><\/pre>\n
equals to<\/p>\n
$ gsutil cp -r \"gs:\/\/asia.uploads.example.com\/media\/123\/*\" \"gs:\/\/asia.contents.example.com\/\"<\/code><\/pre>\n
ref:
\nhttps:\/\/cloud.google.com\/storage\/docs\/listing-objects<\/a><\/p>\n
batch()<\/code> does not guarantee the order of executions, so do not mix different types of calls in the same batch. For instance, the batch should not be a mixture of "copy a.txt" then delete a.txt<\/code>.<\/p>\n
ref:
\nhttps:\/\/googlecloudplatform.github.io\/google-cloud-python\/latest\/storage\/batch.html<\/a><\/p>\n
Upload A File Directly To A Bucket<\/h3>\n
We first need to generate a signed upload URL, and then we can upload the file to the URL.<\/p>\n
import base64\nimport datetime\nimport time\n\nfrom oauth2client.client import GoogleCredentials\nimport yarl\n\ncredentials = GoogleCredentials.get_application_default()\n\ndef signurl(method, url, content_type=None, expires_at=None, md5sum=None, meta=None):\n    method, is_resumable = method.upper(), False\n    if method in ['RESUMABLE']:\n        method, is_resumable = 'POST', True\n    path = yarl.URL(url).path\n\n    def signature():\n        def _signature_parts():\n            def _meta():\n                for key, value in (meta or {}).items():\n                    yield 'x-goog-meta-{key}:{value}'.format(key=key, value=value)\n                if is_resumable:\n                    yield 'x-goog-resumable:start'\n\n            yield method\n            yield md5sum or ''\n            # we need to use curl -H 'content-type:'<\/code> to upload if we sign an empty content-type\n            yield content_type or 'application\/octet-stream'\n            yield str(int(time.mktime(expires_at.timetuple()))) if expires_at else ''\n            yield from sorted(_meta())\n            yield path\n\n        _, signature = credentials.sign_blob('n'.join(_signature_parts()))\n        return base64.b64encode(signature).decode('utf-8')\n\n    def params():\n        yield 'GoogleAccessId', credentials.service_account_email\n        if expires_at:\n            yield 'Expires', int(time.mktime(expires_at.timetuple()))\n        yield 'Signature', signature()\n\n    return str(yarl.URL(url).with_query(**dict(params())))\n\nsignurl(\n    method='RESUMABLE',\n    url='https:\/\/storage.googleapis.com\/asia.uploads.example.com\/media\/your-filename.ext'\n    expires_at=datetime.datetime.utcnow() + datetime.timedelta(hours=24),\n)<\/code><\/pre>\n
$ curl -v -X 'POST' \n-H 'content-type: application\/octet-stream' \n-H 'x-goog-resumable:start' \n-d '' 'THE_SIGNED_UPLOAD_URL'\n\n$ curl -v -X PUT \n--upload-file whatever.mp4 \nTHE_URL_FROM_LOCATION_HEADER_OF_THE_ABOVE_RESPONSE<\/code><\/pre>\n
ref:
\nhttps:\/\/cloud.google.com\/storage\/docs\/access-control\/signed-urls#signing-resumable<\/a>
\nhttps:\/\/cloud.google.com\/storage\/docs\/uploading-objects<\/a>
\nhttps:\/\/cloud.google.com\/storage\/docs\/json_api\/v1\/how-tos\/upload<\/a>
\nhttps:\/\/cloud.google.com\/storage\/docs\/json_api\/v1\/how-tos\/resumable-upload<\/a>
\nhttps:\/\/cloud.google.com\/storage\/docs\/xml-api\/resumable-upload<\/a><\/p>\n
Enable CORS For A Google Cloud Storage Bucket<\/h3>\n
$ gsutil cors get gs:\/\/your_bucket_name\n\n$ cat cors.json\n[\n  {\n    \"origin\": [\"*\"],\n    \"responseHeader\": [\"Content-Type\", \"x-goog-resumable:start\"],\n    \"method\": [\"GET\", \"PUT\", \"\"]\n  }\n]\n$ gsutil cors set cors.json gs:\/\/your_bucket_name<\/code><\/pre>\n
ref:
\nhttps:\/\/cloud.google.com\/storage\/docs\/gsutil\/commands\/cors<\/a>
\nhttps:\/\/medium.com\/imersotechblog\/upload-files-to-google-cloud-storage-gcs-from-the-browser-159810bb11e3<\/a>
\nhttp:\/\/andrewvos.com\/uploading-files-directly-to-google-cloud-storage-from-client-side-javascript<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
google-cloud, Python idiomatic clients for Google Cloud Platform services. There is an older Python library also officially supported by Google, google-api-python-client, which is in maintenance mode.<\/p>\n","protected":false},"author":1,"featured_media":619,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[4,116],"tags":[114,2],"class_list":["post-618","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-about-python","category-about-web-development","tag-google-cloud-platform","tag-python"],"_links":{"self":[{"href":"https:\/\/vinta.ws\/code\/wp-json\/wp\/v2\/posts\/618","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/vinta.ws\/code\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/vinta.ws\/code\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/vinta.ws\/code\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/vinta.ws\/code\/wp-json\/wp\/v2\/comments?post=618"}],"version-history":[{"count":0,"href":"https:\/\/vinta.ws\/code\/wp-json\/wp\/v2\/posts\/618\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/vinta.ws\/code\/wp-json\/wp\/v2\/media\/619"}],"wp:attachment":[{"href":"https:\/\/vinta.ws\/code\/wp-json\/wp\/v2\/media?parent=618"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/vinta.ws\/code\/wp-json\/wp\/v2\/categories?post=618"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/vinta.ws\/code\/wp-json\/wp\/v2\/tags?post=618"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}