Uploading files on Google Cloud Storage using Python
Start your free 7-days trial now!
Prerequisites
To follow along with this guide, please make sure to have:
created a GCP project
created a service account and downloaded the private key (JSON file) for authentication
installed the Python client library for Google Cloud Storage (GCS):
pip install --upgrade google-cloud-storage
If you haven't, then please check out my detailed guide first!
Creating a bucket on Google Cloud Storage
All files on Google Cloud Storage (GCS) must reside in a bucket, which means that we must first create a bucket before we can upload files. We can either create buckets using the web GCS console (refer to my guidelink on how to do so), or we can use the Python client library:
from google.cloud import storage
# Authenticate ourselves using the service account private keypath_to_private_key = './gcs-project-354207-099ef6796af6.json'client = storage.Client.from_service_account_json(json_credentials_path=path_to_private_key)
bucket = client.create_bucket('example-bucket-skytowner')print(f'Bucket with name [{bucket.name}] created on GCS!')
Bucket with name [example-bucket-skytowner] created on GCS!
Here, we are using the create_bucket(~) method to create a bucket called example-bucket-skytowner on GCS.
Bucket names must be globally unique and cannot be changed later.
Uploading a single file on Google Cloud Storage
To upload a text file called sample.txt that resides in the same directory as the Python script onto GCS, use the upload_from_filename(~) function:
from google.cloud import storage# Authenticate ourselves using the service account private keypath_to_private_key = './gcs-project-354207-099ef6796af6.json'client = storage.Client.from_service_account_json(json_credentials_path=path_to_private_key)
bucket = storage.Bucket(client, 'example-bucket-skytowner')# Name of the file on the GCS once uploadedblob = bucket.blob('uploaded_sample.txt')# Path of the local fileblob.upload_from_filename('sample.txt')
Now, if we head over to the web console for GCS, we should see our uploaded file uploaded_sample.txt:
If a file with the same name already exists in GCS, then that file will be overwritten.
Uploading multiple files on Google Cloud Storage
There is no way to upload files in a batch in GCS. Therefore, to upload multiple files on GCS, we must iteratively call upload_from_filename(~):
from google.cloud import storage
path_to_private_key = './gcs-project-354207-099ef6796af6.json'client = storage.Client.from_service_account_json(json_credentials_path=path_to_private_key)bucket = storage.Bucket(client, 'example-bucket-skytowner')
list_files_to_upload = ['sample1.txt','sample2.txt']
for str_file_name in list_files_to_upload: # The name of file on GCS once uploaded blob = bucket.blob(str_file_name) # Path of the local file to upload blob.upload_from_filename(str_file_name)
Here, we are uploading two files: sample1.txt and sample2.txt which both live in the same directory as this Python file.
After running this code, we should see two new files on the web GCS console:
Uploading a folder on Google Cloud Storage
Suppose we have a local folder called to_upload containing two files:
📁 to_upload ├─ cat.png ├─ sample.txt
Unfortunately, GCS does not provide a direct method for uploading directories, let alone for bulk uploads. Therefore, we must use the glob library to fetch the relative path of the files we wish to upload, and then iterate over them as we call upload_from_filename(~):
from google.cloud import storageimport glob
path_to_private_key = './gcs-project-354207-099ef6796af6.json'client = storage.Client.from_service_account_json(json_credentials_path=path_to_private_key)bucket = storage.Bucket(client, 'example-bucket-skytowner')
for str_path_file in glob.glob('to_upload/**'): print(f'Uploading {str_path_file}') # The name of file on GCS once uploaded blob = bucket.blob(str_path_file) # The content that will be uploaded blob.upload_from_filename(str_path_file) print(f'Path in GCS: {str_path_file}')
Uploading to_upload/cat.pngPath in GCS: to_upload/cat.pngUploading to_upload/sample.txtPath in GCS: to_upload/sample.txt
Note that even though we do not directly upload a folder, GCS will automatically create a folder called to_upload when uploading a file named to_upload/cat.png - pretty convenient!
Once running this code, we should see our folder on the GCS web console:
Inside this folder, we have our two files:
Uploading variable value on Google Cloud Storage
Instead of uploading a file, we can upload a variable value on GCS using the upload_from_string(~) method:
from google.cloud import storage
path_to_private_key = './gcs-project-354207-099ef6796af6.json'client = storage.Client.from_service_account_json(json_credentials_path=path_to_private_key)bucket = storage.Bucket(client, 'example-bucket-skytowner')
# The name of file on GCS once uploadedblob = bucket.blob('uploaded_sample.txt')# The content that will be uploadedblob.upload_from_string('Hello world', content_type='text/plain')
Note the following:
this will upload a text file with the content
'Hello world'.the name of this file on GCS will be
uploaded_sample.txt.
After running this code, we should see our new text file on the GCS web console:
