01/24/2023 10:14 | Category: aws

Tags: pythonbotos3

finding an object in s3 using boto3 without using a list objects method

I've seen quite a few implementations of a "does this object exist?" method that use Boto3 as a pagination tool to sift through a list of objects in a bucket.

This is objectively (hehe) fine; however it should only be implemented for a list of validations rather than a single item.

The issue with this approach is in a few places:

  • O(n) time to resolve the match via pagination, given n is the # of objects in the bucket
  • Repetitive calls to S3 that incur charges
  • Slower roundabout returns due to repetitive API calls, they even have potential for throttling

A great solution to this is to use the head_object() method of the Boto3 library. This returns the metadata for a given object if it exists, a botocore.errorfactory.ClientError if it does not.


Example implementation

import boto3
from botocore.client import BaseClient
from botocore.errorfactory import ClientError

s3_client = boto3.client('s3')

def s3_obj_exists(s3_client: BaseClient, s3_key: str) -> bool:
        return True

    except ClientError:
        # obj not found

    return False