abstractkitchen.com logo

How to Upload Files to DigitalOcean Spaces with Python

Posted by Dmitry

Updated on

python digitalocean digitalocean spaces files boto3

Sometimes I need to upload something to DigitalOcean Spaces. Here is my snippet on how to upload files to DigitalOcean Spaces with boto3 and python. First you'll need to install dependencies.

pip install boto3

Upload local file

From boto3 library we'll need upload_file method. For some content type magic we'll use mimetypes. More on this later. Create a file, let's say digitalocean.py.

import boto3
import mimetypes


def get_spaces_client(**kwargs):
    """
    :param kwargs:
    :return:
    """
    region_name = kwargs.get("region_name")
    endpoint_url = kwargs.get("endpoint_url")
    key_id = kwargs.get("key_id")
    secret_access_key = kwargs.get("secret_access_key")

    session = boto3.session.Session()

    return session.client(
        's3',
        region_name=region_name,
        endpoint_url=endpoint_url,
        aws_access_key_id=key_id,
        aws_secret_access_key=secret_access_key
    )


def upload_file_to_space(spaces_client, space_name, file_src, save_as, **kwargs):
    """
    :param spaces_client: Your DigitalOcean Spaces client from get_spaces_client()
    :param space_name: Unique name of your space. Can be found at your digitalocean panel
    :param file_src: File location on your disk
    :param save_as: Where to save your file in the space
    :param kwargs
    :return:
    """

    is_public = kwargs.get("is_public", False)
    content_type = kwargs.get("content_type")
    meta = kwargs.get("meta")

    if not content_type:
        file_type_guess = mimetypes.guess_type(file_src)

        if not file_type_guess[0]:
            raise Exception("We can't identify content type. Please specify directly via content_type arg.")

        content_type = file_type_guess[0]

    extra_args = {
        'ACL': "public-read" if is_public else "private",
        'ContentType': content_type
    }

    if isinstance(meta, dict):
        extra_args["Metadata"] = meta

    return spaces_client.upload_file(
        file_src,
        space_name,
        save_as,

        # boto3.s3.transfer.S3Transfer.ALLOWED_UPLOAD_ARGS
        ExtraArgs=extra_args
    )

Now, when you want to upload file from your local disk to digitalocean spaces simply import your python module.

import digitalocean


client = digitalocean.get_spaces_client(
    region_name="nyc3",
    endpoint_url="https://nyc3.digitaloceanspaces.com",
    key_id="my-spaces-key",
    secret_access_key="my-spaces-secret-access-key"
)

digitalocean.upload_file_to_space(client, "space-name", "local/my_photo.jpg", "place/here/my_photo.jpg")

How to Set DigitalOcean Spaces File Permissions to Public?

By default, file will be uploaded in private mode. Which means, you can't share this file url with others. If you want to change it, then use is_public=True.

In boto3 this is implemented via ACL property, but Spaces only support "public-read" and "private" values. That's why i reduced configuration to is_public argument.

digitalocean.upload_file_to_space(
    client,
    "space-name",
    "local/my_photo.jpg",
    "place/here/my_photo.jpg",
    is_public=True
)

Uploaded files content type. binary/octet-stream error

If you uploaded files with boto3 before, then you might've stumbled upon a mime-type issue. I mean, "binary/octet-stream" or "octet/binary" mime-types in S3 or DigitalOcean Spaces. I resolved this problem with mimetypes.guess_type, but if you want to specify content type directly, then use content_type argument.

digitalocean.upload_file_to_space(
    client,
    "space-name",
    "local/my_styles.css",
    "place/my_styles.css",
    content_type="text/css"
)

Set File Metadata in upload_file_to_space

digitalocean.upload_file_to_space(
    client,
    "space-name",
    "local/my_styles.css",
    "space/path/my_styles.css",
    meta={
      "x-amz-meta-my-key": "your-value"
    }
)

How to Upload Files To Bunny.net Storage

python bunny_net storage snippet

Bunny.net is a well-known CDN and storage among developers. Here is my python snippet on how to upload a file to the storage.

Upload Byte Array File with put_object

First, add this function to digitalocean.py module.

def upload_bytes_array_to_space(spaces_client, space_name, file_body, save_as, **kwargs):
    """
    :param spaces_client: Your DigitalOcean Spaces client from get_spaces_client()
    :param space_name: Unique name of your space. Can be found at your digitalocean panel
    :param file_body: Byte Array File
    :param save_as: Where to save your file in the space
    :param kwargs:
    :return:
    """

    is_public = kwargs.get("is_public", False)
    content_type = kwargs.get("content_type")
    meta = kwargs.get("meta")

    args = {
        "Bucket": space_name,
        "Body": file_body,
        "Key": save_as,
        "ACL": "public-read" if is_public else "private"
    }

    if content_type:
        args["ContentType"] = content_type

    if isinstance(meta, dict):
        args["Metadata"] = meta

    return spaces_client.put_object(**args)

Use it in your module as described previously. Open your file in binary mode and send it to space. I'm not doing here any mime type detection, so it's up to you to set it directly via content_type.

with open("scripts/test/test.jpg", "rb") as my_file:
    digitalocean.upload_bytes_array_to_space(
        client,
        "space-name",
        my_file,
        "space/path/test6.jpg",
        is_public=True
    )

How to Set File Metadata?

If you want to pass additional metadata to your file, simply add meta key to upload_bytes_array_to_space.

with open("scripts/test/test.jpg", "rb") as my_file:
    digitalocean.upload_bytes_array_to_space(
        client,
        "space-name",
        my_file,
        "space/path/test6.jpg",
        is_public=True,
        meta={
            "x-amz-meta-my-key": "your-value"
        }
    )

How to Create Spaces key_id and secret_access_key?

Visit official documentation from DigitalOcean.

A Short Guide To The Chinese Coordinate System

Have your ever searched google maps china offset? Most people who was in China yes. Here is my story behind this question.

The Definitive Guide To Sitemaps With Python

open source python

Sitemaps are important. Especially for big websites. It is always a good idea to develop your website with SEO in mind. Unfortunately, most developers ignore this part.

How to Import CSV Files to PostgreSQL with Pgfutter

csv postgresql

Sometimes I need to upload large CSV files to PostgreSQL. CSV file might have hundreds of columns, that's why i want a tool that can do some magic for me.

Flask Boilerplate and Your Guide to Flask in 2022. With SQLAlchemy.

boilerplate open source flask

Flask-Backbone is my take to create an initial structure and a set of rules, so most of my flask projects are easy to support.

How to Copy Text to Clipboard With Javascript

javascript DOM browser

Here is a short snippet on how to copy text to the clipboard with Javascript. Your visitors will thank you. Most likely not, but here we are.

How to Upload Files To Bunny.net Storage

python bunny_net storage snippet

Bunny.net is a well-known CDN and storage among developers. Here is my python snippet on how to upload a file to the storage.

How to Create Jinja2 Filters in Flask

python flask jinja2 snippet

In this post, I'll talk about filters. Jinja2 has a list of built-in filters, and Flask leverages them.

JavaScript Document Onload + TypeScript version

javascript DOM snippet basics typescript
🗃 JavaScript Basics

Universal code-snippet to know when HTML is ready for modern browsers.