upload files from local directory to aws s3 bucket

In this article, I will focus on how python Boto3 library makes it easy for us to upload a single or all file(s) in amazon s3 bucket. Amazon Web Services (AWS) Simple Storage Service (S3) is a Storage-as-a-Service provided by Amazon.  It’s a general purpose object store, the objects are grouped under a name space called “buckets.” The buckets are unique across the entire AWS S3.

Copy and past below scripts to create the function.

def upload_local_file_to_aws_s3 (FILE_NAME_DIR, AWS_BUCKET_NAME, AWS_ACCESS_KEY, AWS_ACCESS_SECRET_KEY):
    from boto3.s3.transfer import S3Transfer
    #have all the variables populated which are required below
    client = boto3.client('s3', aws_access_key_id=AWS_ACCESS_KEY,aws_secret_access_key=AWS_ACCESS_SECRET_KEY)
    transfer = S3Transfer(client)
    Key = AWS_S3_FILE_KEY +'/' + os.path.basename(FILE_NAME_DIR)
    transfer.upload_file(FILE_NAME_DIR, AWS_BUCKET_NAME, Key)

Don’t forget to change your bucket and directory name, access and secret key before execute the function.

AWS_ACCESS_KEY         = "YOUR ACCESS KEY"
AWS_ACCESS_SECRET_KEY  = "YOU SECRET KEY"
AWS_BUCKET_NAME        = "YOUR BUCKET NAME"
FILE_NAME_DIR          = "YOU LOCAL DIRECTORY"
def upload_local_file_to_aws_s3 (FILE_NAME_DIR, AWS_BUCKET_NAME, AWS_ACCESS_KEY, AWS_ACCESS_SECRET_KEY)

Conclusion:
Python Boto3 library makes it bery easy for us to upload a single or all file(s) from our local directory to amazon s3 bucket. Click here if you want to know how to delete all files in aws s3 bucket.

Published by Jean Joseph

Jean Joseph is a Database, Big Data, Data Warehouse Platform, Data Pipeline, Database Architecture Solutions Provider as well as a Data Engineer enthusiast among other disciplines.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: