日々精進

新しく学んだことを書き留めていきます

S3にPythonのオブジェクトを保存したり読み込んだりするコード

意外と手こずったのでコードを貼っておく。

import io, pickle, boto3
BUCKET = "バケット名"

def upload_to_s3(file, content):
    s3 = boto3.resource('s3')
    s3.Bucket(BUCKET).put_object(Key=file, Body=content)

def upload_object_to_s3(file, obj):
    pickle_buffer = io.BytesIO()
    pickle.dump(obj, pickle_buffer)
    upload_to_s3(file, pickle_buffer.getvalue())
    
def download_pickle_from_s3(file) -> object:
    with io.BytesIO() as data:
        s3 = boto3.resource('s3')
        s3.Bucket(BUCKET).download_fileobj(file, data)
        data.seek(0)
        return pickle.load(data)

使い方は以下。

upload_object_to_s3("LightGBM_model.pickle", models)
saved_model = download_pickle_from_s3("LightGBM_model.pickle")

参考: stackoverflow.com

qiita.com

boto3.amazonaws.com

docs.python.org