Writing file to Google Cloud Storage from Google Colab
Start your free 7-days trial now!
Writing to Google Cloud Storage (GCS) from Google Colab is easy since all the required libraries are already pre-installed.
1. Authentication for interacting with Google Cloud Storage
The first step is to authenticate yourself:
from google.colab import authauth.authenticate_user()
This should open up a dialogue for authentication - make sure to use the same account that you use for the Google Cloud Platform (GCP) project.
2. Uploading file on Google Cloud Storage using gcloud terminal command
Now that we are authenticated, we can use the pre-installed gcloud command to select our project:
project_id = 'gcs-project-354207'!gcloud config set project {project_id}
Here, note the following:
we must prefix the terminal command using
!.{project_id}is a placeholder that will automatically get replaced with the value ofproject_id.
Finding the Project ID
To get the GCP project ID, head over to the GCP console and click on the header dropdown menu:
Now, we can find the project ID:
Sample text file to upload to Google Cloud Storage
Let's suppose we want to upload the following text file from Google Colab to GCS:
with open('./sample.txt', 'w') as f: f.write('hello world')
This creates a text file called sample.txt in the root directory of your Colab instance machine:
Now, to upload this text file to GCS, use the built-in gsutil command:
# You can find the full reference here: https://cloud.google.com/storage/docs/gsutil/commands/cpbucket_name = 'example-bucket-skytowner'!gsutil cp /sample.txt gs://{bucket_name}/
Copying file:///sample.txt [Content-Type=text/plain].../ [1 files][ 11.0 B/ 11.0 B] Operation completed over 1 objects/11.0 B.
Note that if you have not yet created a bucket, then you can do so by using the mb (make bucket) command:
# You can find the full reference here: https://cloud.google.com/storage/docs/gsutil/commands/mbbucket_name = 'example-bucket-skytowner'!gsutil mb gs://{bucket_name}
Creating gs://example-bucket-skytowner/...
Once running this code, we should see our uploaded sample.txt file:
