search
Search
Join our weekly DS/ML newsletter layers DS/ML Guides
menu
menu search toc more_vert
Robocat
Guest 0reps
Thanks for the thanks!
close
Comments
Log in or sign up
Cancel
Post
account_circle
Profile
exit_to_app
Sign out
help Ask a question
Share on Twitter
search
keyboard_voice
close
Searching Tips
Search for a recipe:
"Creating a table in MySQL"
Search for an API documentation: "@append"
Search for code: "!dataframe"
Apply a tag filter: "#python"
Useful Shortcuts
/ to open search panel
Esc to close search panel
to navigate between search results
d to clear all current filters
Enter to expand content preview
icon_star
Doc Search
icon_star
Code Search Beta
SORRY NOTHING FOUND!
mic
Start speaking...
Voice search is only supported in Safari and Chrome.
Navigate to
A
A
brightness_medium
share
arrow_backShare
Twitter
Facebook

Cloning a bucket in Google Cloud Storage using Python

Cloud Computing
chevron_right
Google Cloud Platform
chevron_right
Cloud Storage
chevron_right
Python client library
schedule Jul 2, 2022
Last updated
local_offer Cloud Computing
Tags

Although cloning a bucket in Google Cloud Storage is not supported directly, we can still emulate this behaviour by creating a new bucket with the same properties as the source bucket, and then populating the new bucket with the files in the source bucket.

Prerequisites

To follow along with this guide, please make sure to have:

  • created a service account and downloaded the private key (JSON file) for authentication (please check out my detailed guide)

  • installed the Python client library for Google Cloud Storage:

    pip install --upgrade google-cloud-storage

Creating a bucket with the same properties as source bucket

The first step is to create a bucket with the same properties as the source bucket in Google Cloud Storage (GCS):

from google.cloud import storage
path_to_private_key = './gcs-project-354207-099ef6796af6.json'
client = storage.Client.from_service_account_json(json_credentials_path=path_to_private_key)

bucket_src = client.get_bucket('example-bucket-skytowner')
print(f'Source bucket location: {bucket_src.location}')
print(f'Source bucket storage_class: {bucket_src.storage_class}')

new_bucket = storage.Bucket(client, 'example-bucket-skytowner-clone')
new_bucket.storage_class = bucket_src.storage_class
new_bucket = client.create_bucket(new_bucket, location=bucket_src.location)
Source bucket location: US
Source bucket storage_class: STANDARD

Note the following:

  • we have first authenticated ourselves using the private key of our service account.

  • we fetched the bucket meta-information from GCS using the get_bucket(~) method.

  • we can access the location and storage class using the location and storage_class properties of the fetched bucket.

  • we then create a Bucket object using the constructor storage.Bucket(~) - at this point, we have not yet created an actual bucket on GCS. We then set the same storage class as well as the location for this new bucket.

  • we finally send a request to GCS to create the bucket using the create_bucket(~) method.

Now once we head over to the GCS web console, we should see our cloned bucket:

Notice how the location and storage class are the same.

Populating the cloned bucket with files of the source bucket

Now that we have cloned the bucket, we need to insert all the files that are in the source bucket into this new cloned bucket. We can do so using the following script:

blobs_src = client.list_blobs('example-bucket-skytowner')
for blob_src in blobs_src:
print(f'Copied [{blob_src.name}] into new bucket')
blob_new = bucket_src.copy_blob(blob_src, new_bucket, new_name=blob_src.name)
Copied [cat.png] into new bucket

Here:

  • we first fetch the meta-information of all blobs (files) in the source bucket using list_blob(~)

  • we then iterate over this list of blobs, and then use the copy_blob(~) method to insert each blob into the cloned bucket.

mail
Join our newsletter for updates on new DS/ML comprehensive guides (spam-free)
robocat
Published by Isshin Inada
Edited by 0 others
Did you find this page useful?
Ask a question or leave a feedback...
0
thumb_down
0
chat_bubble_outline
0
settings
Enjoy our search
Hit / to insta-search docs and recipes!