search
Search
Join our weekly DS/ML newsletter layers DS/ML Guides
menu
menu search toc more_vert
Robocat
Guest 0reps
Thanks for the thanks!
close
Comments
Log in or sign up
Cancel
Post
account_circle
Profile
exit_to_app
Sign out
help Ask a question
Share on Twitter
search
keyboard_voice
close
Searching Tips
Search for a recipe:
"Creating a table in MySQL"
Search for an API documentation: "@append"
Search for code: "!dataframe"
Apply a tag filter: "#python"
Useful Shortcuts
/ to open search panel
Esc to close search panel
to navigate between search results
d to clear all current filters
Enter to expand content preview
icon_star
Doc Search
icon_star
Code Search Beta
SORRY NOTHING FOUND!
mic
Start speaking...
Voice search is only supported in Safari and Chrome.
Navigate to
A
A
brightness_medium
share
arrow_backShare
Twitter
Facebook

Writing file to Google Cloud Storage from Google Colab

Cloud Computing
chevron_right
Google Cloud Platform
chevron_right
Cloud Storage
schedule Jul 1, 2022
Last updated
local_offer Cloud Computing
Tags

Writing to Google Cloud Storage (GCS) from Google Colab is easy since all the required libraries are already pre-installed.

1. Authentication for interacting with Google Cloud Storage

The first step is to authenticate yourself:

from google.colab import auth
auth.authenticate_user()

This should open up a dialogue for authentication - make sure to use the same account that you use for the Google Cloud Platform (GCP) project.

2. Uploading file on Google Cloud Storage using gcloud terminal command

Now that we are authenticated, we can use the pre-installed gcloud command to select our project:

project_id = 'gcs-project-354207'
!gcloud config set project {project_id}

Here, note the following:

  • we must prefix the terminal command using !.

  • {project_id} is a placeholder that will automatically get replaced with the value of project_id.

Finding the Project ID

To get the GCP project ID, head over to the GCP console and click on the header dropdown menu:

Now, we can find the project ID:

Sample text file to upload to Google Cloud Storage

Let's suppose we want to upload the following text file from Google Colab to GCS:

with open('./sample.txt', 'w') as f:
f.write('hello world')

This creates a text file called sample.txt in the root directory of your Colab instance machine:

Now, to upload this text file to GCS, use the built-in gsutil command:

# You can find the full reference here: https://cloud.google.com/storage/docs/gsutil/commands/cp
bucket_name = 'example-bucket-skytowner'
!gsutil cp /sample.txt gs://{bucket_name}/
Copying file:///sample.txt [Content-Type=text/plain]...
/ [1 files][ 11.0 B/ 11.0 B]
Operation completed over 1 objects/11.0 B.

Note that if you have not yet created a bucket, then you can do so by using the mb (make bucket) command:

# You can find the full reference here: https://cloud.google.com/storage/docs/gsutil/commands/mb
bucket_name = 'example-bucket-skytowner'
!gsutil mb gs://{bucket_name}
Creating gs://example-bucket-skytowner/...

Once running this code, we should see our uploaded sample.txt file:

mail
Join our newsletter for updates on new DS/ML comprehensive guides (spam-free)
robocat
Published by Isshin Inada
Edited by 0 others
Did you find this page useful?
Ask a question or leave a feedback...