Using Python Boto3 with Amazon AWS S3 Buckets

I’m here adding some additional Python Boto3 examples, this time working with S3 Buckets.

So to get started, lets create the S3 resource, client, and get a listing of our buckets.

import boto3

s3 = boto3.resource('s3')
s3client = boto3.client('s3')

response = s3client.list_buckets()
for bucket in response["Buckets"]:
    print(bucket['Name'])

Here we create the s3 client object and call ‘list_buckets()’. Response is a dictionary and has a key called ‘Buckets’ that holds a list of dicts with each bucket details.

To list out the objects within a bucket, we can add the following:

    theobjects = s3client.list_objects_v2(Bucket=bucket["Name"])
    for object in theobjects["Contents"]:
        print(object["Key"])

Note that if the Bucket has no items, then there will be no Contents to list and you will get an error thrown “KeyError: ‘Contents’.

Each object returned is a dictionary with Key Value pairs describing the object. Boto3 Docs are you friend here: https://boto3.readthedocs.io/en/latest/reference/services/s3.html#S3.Client.list_objects_v2

Now if the Bucket has over 1,000 items, the list_objects is limited to 1000 replies. To get around this, we need to use a Paginator.

import boto3

s3 = boto3.resource('s3')
s3client = boto3.client('s3')

response = s3client.list_buckets()
print(response)
for bucket in response["Buckets"]:
    # Create a paginator to pull 1000 objects at a time
    paginator = s3client.get_paginator('list_objects')
    pageresponse = paginator.paginate(Bucket=bucket["Name"])
    
    # PageResponse Holds 1000 objects at a time and will continue to repeat in chunks of 1000. 
    for pageobject in pageresponse:
        for file in pageobject["Contents"]:
            print(object["Key"])