S3 clients: AWS CLI, boto3 and rclone

The four settings

Every S3 client asks for the same four things. These are the values for Subnomic Storage:

Requests are signed with AWS Signature Version 4, which every current client uses by default. The access-key dialog in the console shows the endpoint too; the secret is shown once, so copy it before you close the dialog.

AWS CLI

Keep the key in a profile of its own, so it never mixes with your AWS credentials:

aws configure set aws_access_key_id <access key id> --profile subnomic
aws configure set aws_secret_access_key <secret> --profile subnomic
aws configure set region eu-west-1 --profile subnomic

Then pass the profile and the endpoint with each command:

aws --profile subnomic --endpoint-url https://objects.subnomic.com s3 ls
aws --profile subnomic --endpoint-url https://objects.subnomic.com s3 cp ./backup.tar.gz s3://acme-backups/2026/10/backup.tar.gz
aws --profile subnomic --endpoint-url https://objects.subnomic.com s3 sync ./dist s3://acme-assets/dist

In CI, environment variables do the same job. This is the shape of the step that publishes Subnomic’s own release binaries:

- name: Upload to Subnomic Storage
  env:
    AWS_ACCESS_KEY_ID: ${{ secrets.SUBNOMIC_OBJECTS_ACCESS_KEY }}
    AWS_SECRET_ACCESS_KEY: ${{ secrets.SUBNOMIC_OBJECTS_SECRET_KEY }}
    AWS_REGION: eu-west-1
  run: aws --endpoint-url https://objects.subnomic.com s3 sync ./dist/ s3://acme-releases/${{ github.ref_name }}/

Python (boto3)

Give the client the endpoint and the region, and ask for path-style addressing explicitly:

import os

import boto3
from botocore.config import Config

s3 = boto3.client(
    "s3",
    endpoint_url="https://objects.subnomic.com",
    region_name="eu-west-1",
    aws_access_key_id=os.environ["SUBNOMIC_ACCESS_KEY_ID"],
    aws_secret_access_key=os.environ["SUBNOMIC_SECRET_ACCESS_KEY"],
    config=Config(s3={"addressing_style": "path"}),
)

s3.upload_file("backup.tar.gz", "acme-backups", "2026/10/backup.tar.gz")

for obj in s3.list_objects_v2(Bucket="acme-backups", Prefix="2026/").get("Contents", []):
    print(obj["Key"], obj["Size"])

upload_file switches to a multipart upload on its own for a large file, and multipart uploads are supported.

A link that expires

A presigned URL lets somebody without a key fetch one object until it expires, and nothing needs to change on the bucket to allow it:

url = s3.generate_presigned_url(
    "get_object",
    Params={"Bucket": "acme-backups", "Key": "2026/10/backup.tar.gz"},
    ExpiresIn=3600,  # seconds; at most 7 days
)

Seven days is the ceiling, as it is on AWS. For a link you can revoke before it expires, use the share link in the console’s browser view instead — see Sharing a single file.

rclone

Add a remote to rclone.conf (rclone config file prints where it is). provider = Other is rclone’s setting for an S3-compatible service; path-style is already rclone’s default for it.

[subnomic]
type = s3
provider = Other
access_key_id = <access key id>
secret_access_key = <secret>
endpoint = https://objects.subnomic.com
region = eu-west-1

rclone lsd subnomic:
rclone copy ./photos subnomic:acme-media/photos --progress
rclone sync ./dist subnomic:acme-assets/dist

rclone sync deletes from the destination whatever is not in the source. Run it with --dry-run first when the destination already holds anything you care about.

Moving data in, and out

Because the API is S3, moving to or from Subnomic Storage needs no export tool: point one client at both ends. With rclone, add a second remote for the other provider and copy between them:

rclone copy otherprovider:old-bucket subnomic:acme-archive --progress

The same command, the other way round, is how you leave. Why we built it that way is in An object store you can leave.

What a key can reach

An access key belongs to the workspace, not to the person who created it, and it keeps working after they leave — revoke a leaver’s keys as part of offboarding. Keys can carry an expiry, and disabling one takes effect on the next request. Buckets, keys, sharing and limits are covered in Object storage.