I’m here adding some additional Python Boto3 examples, this time working with S3 Buckets.
So to get started, lets create the S3 resource, client, and get a listing of our buckets.
import boto3 s3 = boto3.resource('s3') s3client = boto3.client('s3') response = s3client.list_buckets() for bucket in response["Buckets"]: print(bucket['Name'])
Here we create the s3 client object and call ‘list_buckets()’. Response is a dictionary and has a key called ‘Buckets’ that holds a list of dicts with each bucket details.
To list out the objects within a bucket, we can add the following:
theobjects = s3client.list_objects_v2(Bucket=bucket["Name"]) for object in theobjects["Contents"]: print(object["Key"])
Note that if the Bucket has no items, then there will be no Contents to list and you will get an error thrown “KeyError: ‘Contents’.
Each object returned is a dictionary with Key Value pairs describing the object. Boto3 Docs are you friend here: https://boto3.readthedocs.io/en/latest/reference/services/s3.html#S3.Client.list_objects_v2
Now if the Bucket has over 1,000 items, the list_objects is limited to 1000 replies. To get around this, we need to use a Paginator.
import boto3 s3 = boto3.resource('s3') s3client = boto3.client('s3') response = s3client.list_buckets() print(response) for bucket in response["Buckets"]: # Create a paginator to pull 1000 objects at a time paginator = s3client.get_paginator('list_objects') pageresponse = paginator.paginate(Bucket=bucket["Name"]) # PageResponse Holds 1000 objects at a time and will continue to repeat in chunks of 1000. for pageobject in pageresponse: for file in pageobject["Contents"]: print(object["Key"])