S3 API Compatibility

This article explains how Orchesto delivers compatibility with, and extends, the AWS S3 API as well as known limitations.

Feature Matrix

Orchesto supports all of the core object storage functions in the Amazon S3 API (that is, create, read and delete operations needed to manage buckets and objects). The following list gives a basic outline:

LevelOperationsSupport
ServiceGET
BucketHEAD, PUT, DELETE, GET, location
Bucketpolicy, acl, versioning
ObjectHEAD, PUT¹, GET, DELETE²
Objectpolicy, acl, versioning, torrent

¹ Also supporting multi-part upload and copy.

² Delete either one object or multiple objects.

Note that Amazon S3 API features not outlined in the above table are not supported by Orchesto.

Compatibility

To simplify integration into existing architectures, Orchesto is designed to provide a high degree of transparency against targeted storage environments. In particular, it can preserve region, bucket and object names as well as security credentials and custom HTTP header fields.

Deploying Orchesto as a forward proxy against a storage provider with maximum transparency for a given client requires:

  1. configuration of Orchesto with the storage backend with all regions, security credentials and buckets as-is;

  2. configuration of the client to change its AWS S3 API endpoint URL to reference Orchesto instead

For example, let's assume you have buckets and objects in Amazon AWS S3, and you already use the AWS CLI tool to work with your storage.

$ aws s3 ls
2018-05-17 10:14:32 concretely
2018-05-17 10:14:31 cylindrocellular
2018-05-17 10:14:28 distinctiveness
2018-05-17 10:14:31 inexactitude
2018-05-17 10:14:31 olefinic
2018-05-17 10:14:27 poligraphical
2018-05-17 10:14:31 potful
2018-05-17 10:14:27 sportsmanlike
2018-05-17 10:14:32 unsymbolically

Then, by adding a corresponding Amazon storage backend, security credential, and buckets to Orchesto, the only configuration change required in clients is the endpoint URL.

$ aws s3 ls --endpoint-url http://127.0.0.1:9090
2018-05-23 15:17:22 concretely
2018-05-23 15:17:22 cylindrocellular
2018-05-23 15:17:22 distinctiveness
2018-05-23 15:17:22 inexactitude
2018-05-23 15:17:22 olefinic
2018-05-23 15:17:22 poligraphical
2018-05-23 15:17:22 potful
2018-05-23 15:17:22 sportsmanlike
2018-05-23 15:17:22 unsymbolically

When preparing Orchesto to enable access to already existing buckets, the corresponding virtual buckets that you create will have a distinct creation time.

To work as a multi-point gateway, Orchesto abstracts discrepancies in the object storage model across storage providers. This includes idiosyncracies in S3 API implementations and aligment with protocol translation.

As all storage resources are private, only authenticated requests can access them. Anonymous requests are relayed to remote storage providers, delegating access control to enable existing security policies to govern anonymous access.