Use Boto3 to open an AWS S3 file directly

In this example I want to open a file directly from an S3 bucket without having to download the file from S3 to the local file system.  This is a way to stream the body of a file into a python variable, also known as a ‘Lazy Read’.

import boto3
 
s3client = boto3.client(
    's3',
    region_name='us-east-1'
)
 
# These define the bucket and object to read
bucketname = mybucket 
file_to_read = /dir1/filename 

#Create a file object using the bucket and object key. 
fileobj = s3client.get_object(
    Bucket=bucketname,
    Key=file_to_read
    ) 
# open the file object and read it into the variable filedata. 
filedata = fileobj['Body'].read()

# file data will be a binary stream.  We have to decode it 
contents = filedata.decode('utf-8')) 

# Once decoded, you can treat the file as plain text if appropriate 
print(contents)

And that is all there is to it. Be careful when reading in very large files. Also this example works will with text files. I use it alot when saving and reading in json data from an S3 bucket.

Good Luck.

Tagged , , , , , . Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *

Solve : *
24 × 1 =