Uploading a file on Databricks and reading the file in a notebook
In this guide, we will go through the steps of uploading a simple text file on Databricks, and then reading this file using Python in a Databricks notebook.
Uploading a file on Databricks
To upload a file on Databricks, click on Upload Data:
Here, even though the label is Upload Data, the file does not have to contain data (e.g. CSV file) - it can be any file like a JSON file.
Next, select the file that you wish to upload, and then click on Next:
Here, we'll be uploading a text file called
Next, copy the file path in Spark API Format:
Finally, click on Done - the text file has now been uploaded to
dbfs, which is Databricks' file system.
Copying file from DBFS to local file system on driver node
The problem with
dbfs is that the file in
dbfs cannot be directly accessed in Python code. Therefore, we must copy this file over to the standard file system of the driver node like so:
# The dbfs path that you copied in the previous stepdbfs_path = 'dbfs:/FileStoreemail@example.com/sample.txt'local_path = 'file:///tmp/sample.txt'# Copy (cp) the file from dbfs to local file system in driver nodedbutils.fs.cp(dbfs_path, local_path)True
After running this code, the
sample.txt will be written to the local file system on the driver node in the root
Reading the file using Python
Finally, we can read the content of the file using Python like so:
with open('/tmp/sample.txt') as f:content = f.read()print(content)Hello world
Here, we see that the content of the text file is