search
Search
Login
Math ML Join our weekly DS/ML newsletter
menu
menu search toc more_vert
Robocat
Guest 0reps
Thanks for the thanks!
close
Comments
Log in or sign up
Cancel
Post
account_circle
Profile
exit_to_app
Sign out
help Ask a question
Share on Twitter
search
keyboard_voice
close
Searching Tips
Search for a recipe:
"Creating a table in MySQL"
Search for an API documentation: "@append"
Search for code: "!dataframe"
Apply a tag filter: "#python"
Useful Shortcuts
/ to open search panel
Esc to close search panel
to navigate between search results
d to clear all current filters
Enter to expand content preview
icon_star
Doc Search
icon_star
Code Search Beta
SORRY NOTHING FOUND!
mic
Start speaking...
Voice search is only supported in Safari and Chrome.
Navigate to
A
A
brightness_medium
share
arrow_backShare
Twitter
Facebook

Uploading files on Google Cloud Storage using Python

Cloud Computing
chevron_right
Google Cloud Platform
chevron_right
Cloud Storage
chevron_right
Python client library
schedule Jul 1, 2022
Last updated
local_offer Cloud Computing
Tags

Prerequisites

To follow along with this guide, please make sure to have:

  • created a GCP project

  • created a service account and downloaded the private key (JSON file) for authentication

  • installed the Python client library for Google Cloud Storage (GCS):

    pip install --upgrade google-cloud-storage

If you haven't, then please check out my detailed guide first!

Creating a bucket on Google Cloud Storage

All files on Google Cloud Storage (GCS) must reside in a bucket, which means that we must first create a bucket before we can upload files. We can either create buckets using the web GCS console (refer to my guidelink on how to do so), or we can use the Python client library:

from google.cloud import storage

# Authenticate ourselves using the service account private key
path_to_private_key = './gcs-project-354207-099ef6796af6.json'
client = storage.Client.from_service_account_json(json_credentials_path=path_to_private_key)

bucket = client.create_bucket('example-bucket-skytowner')
print(f'Bucket with name [{bucket.name}] created on GCS!')
Bucket with name [example-bucket-skytowner] created on GCS!

Here, we are using the create_bucket(~) method to create a bucket called example-bucket-skytowner on GCS.

WARNING

Bucket names must be globally unique and cannot be changed later.

Uploading a single file on Google Cloud Storage

To upload a text file called sample.txt that resides in the same directory as the Python script onto GCS, use the upload_from_filename(~) function:

from google.cloud import storage
# Authenticate ourselves using the service account private key
path_to_private_key = './gcs-project-354207-099ef6796af6.json'
client = storage.Client.from_service_account_json(json_credentials_path=path_to_private_key)

bucket = storage.Bucket(client, 'example-bucket-skytowner')
# Name of the file on the GCS once uploaded
blob = bucket.blob('uploaded_sample.txt')
# Path of the local file
blob.upload_from_filename('sample.txt')

Now, if we head over to the web console for GCS, we should see our uploaded file uploaded_sample.txt:

WARNING

If a file with the same name already exists in GCS, then that file will be overwritten.

Uploading multiple files on Google Cloud Storage

There is no way to upload files in a batch in GCS. Therefore, to upload multiple files on GCS, we must iteratively call upload_from_filename(~):

from google.cloud import storage

path_to_private_key = './gcs-project-354207-099ef6796af6.json'
client = storage.Client.from_service_account_json(json_credentials_path=path_to_private_key)
bucket = storage.Bucket(client, 'example-bucket-skytowner')

list_files_to_upload = ['sample1.txt','sample2.txt']

for str_file_name in list_files_to_upload:
# The name of file on GCS once uploaded
blob = bucket.blob(str_file_name)
# Path of the local file to upload
blob.upload_from_filename(str_file_name)

Here, we are uploading two files: sample1.txt and sample2.txt which both live in the same directory as this Python file.

After running this code, we should see two new files on the web GCS console:

Uploading a folder on Google Cloud Storage

Suppose we have a local folder called to_upload containing two files:

📁 to_upload
├─ cat.png
├─ sample.txt

Unfortunately, GCS does not provide a direct method for uploading directories, let alone for bulk uploads. Therefore, we must use the glob library to fetch the relative path of the files we wish to upload, and then iterate over them as we call upload_from_filename(~):

from google.cloud import storage
import glob

path_to_private_key = './gcs-project-354207-099ef6796af6.json'
client = storage.Client.from_service_account_json(json_credentials_path=path_to_private_key)
bucket = storage.Bucket(client, 'example-bucket-skytowner')

for str_path_file in glob.glob('to_upload/**'):
print(f'Uploading {str_path_file}')
# The name of file on GCS once uploaded
blob = bucket.blob(str_path_file)
# The content that will be uploaded
blob.upload_from_filename(str_path_file)
print(f'Path in GCS: {str_path_file}')
Uploading to_upload/cat.png
Path in GCS: to_upload/cat.png
Uploading to_upload/sample.txt
Path in GCS: to_upload/sample.txt

Note that even though we do not directly upload a folder, GCS will automatically create a folder called to_upload when uploading a file named to_upload/cat.png - pretty convenient!

Once running this code, we should see our folder on the GCS web console:

Inside this folder, we have our two files:

Uploading variable value on Google Cloud Storage

Instead of uploading a file, we can upload a variable value on GCS using the upload_from_string(~) method:

from google.cloud import storage

path_to_private_key = './gcs-project-354207-099ef6796af6.json'
client = storage.Client.from_service_account_json(json_credentials_path=path_to_private_key)
bucket = storage.Bucket(client, 'example-bucket-skytowner')

# The name of file on GCS once uploaded
blob = bucket.blob('uploaded_sample.txt')
# The content that will be uploaded
blob.upload_from_string('Hello world', content_type='text/plain')

Note the following:

  • this will upload a text file with the content 'Hello world'.

  • the name of this file on GCS will be uploaded_sample.txt.

After running this code, we should see our new text file on the GCS web console:

mail
Join our newsletter for updates on new DS/ML comprehensive guides (spam-free)
robocat
Published by Isshin Inada
Edited by 0 others
Did you find this page useful?
thumb_up
thumb_down
Ask a question or leave a feedback...