Minio as S3-Compatible Storage

Difficulty: Advanced
Estimated Cloudlet Usage: 1-16, depending on number of nodes created

In this guide we will look at setting up cloud storage as an S3-compatible object store. You’ll find this useful if, for example, you are running an application that might have very high storage needs for static content like images, PDFs, and other objects. A great example is Omeka, a popular publishing platform for rich collections. Omeka Classic supports using S3 storage for all uploads and traditionally that would mean setting up an account at Amazon Web Services and using an S3 bucket. We have a guide on that here. However can we setup cloud storage at Reclaim Cloud and integrate it with an Omeka site hosted elsewhere? Absolutely and in this guide will look at the full process.

To start we are going to use an application in the Marketplace called Minio. Minio is an object storage server software that is compatible with Amazon S3 storage and best of all completely open source. Click here to start an install. When installing it you’ll see a prompt for how many nodes you’d like to run. Minio is a highly available redundant storage solution in which any particular file is stored on multiple nodes to avoid loss of data. For testing 1 node might be ok but for any production work we recommend at least 4 nodes. Give it an environment URL and name and click Install and within a few minutes you’ll be up and running.

After the install is complete you are provided with a web-based utility to manage your storage buckets and even upload individual files. Make sure to store your access key and secret as you’ll need those later. In the management interface I’ve created a bucket called “omeka” that I will use for uploads from my Omeka Classic install hosted on a Reclaim Hosting shared hosting account. You’ll also need to make sure the bucket has read access to the public. You can click the menu next to the bucket name and set a policy for read only public access.

Screen Shot 2020-06-03 at 2.49.50 PM

Screen Shot 2020-06-03 at 2.29.07 PM

We won’t cover the install process of Omeka Classic here as we have a guide at Installing Omeka Classic on Reclaim Hosting that you can use if running on Reclaim Hosting but this will work with any self-hosted Omeka Classic install.

We also need to install a separate plugin that extends the S3 functionality of Omeka to be used with services other than AWS. You can find a work in progress plugin at GitHub - timmmmyboy/omeka-amazon-s3-storage-adapter: Omeka storage adapter for Amazon S3 based on the official AWS SDK. and follow this guide to install Uploading Plugins to Omeka

After install we need to edit the configuration file located at application/config/config.ini.

In this file you’ll find a sample S3 configuration commented out. You’ll need to at least uncomment and fill in the adapter, access key, secret key, bucket, and endpoint using the information from you Minio install. Since we are using a custom plugin you’ll also change the storage adapter to Omeka_Storage_Adapter_AmazonS3. Here is how my configuration looks for comparison:

; Storage ;
;;;;;;;;;;;

; Sample S3 cloud storage configuration
;
; The accessKeyId, secretAccessKey, and bucket options are all required.
; If the expiration option is set, files will be uploaded with "private"
; access, and Omeka will generate URLs that are only valid for a limited
; time.  If the expiration option is missing or left commented out,
; uploaded files will always be publicly readable. Leave the forceSSL  
; option commented out unless site is served over HTTPS-only *and* the 
; the web browser is issuing mixed-content warnings.
;
storage.adapter = "Omeka_Storage_Adapter_AmazonS3"
storage.adapterOptions.accessKeyId = "q60pUTm8D7"
storage.adapterOptions.secretAccessKey = "OtfTaXAGf7"
storage.adapterOptions.bucket = "omeka"
storage.adapterOptions.region = "us-east-1"
storage.adapterOptions.endpoint = "https://objectstore.uk.reclaim.cloud"; Custom S3 endpoint (optional)
;storage.adapterOptions.expiration = 10 ; URL expiration time (in minutes)
;storage.adapterOptions.forceSSL = 1 ; Boolean value (optional)

Once that’s saved you can test out adding an item. If all goes well Omeka should function just like it normally would, however a key difference is that the image stored is using no local storage in the hosting account, but rather utilizing Minio as cloud storage so we can grow over time with no limits.

Minio is a great option for cloud storage giving you a lot of control and should integrate with other applications that are S3 compatible as well. For WordPress this may be useful How to Get MinIO and Other S3-Compatible Storage Providers to Work with WP Offload Media

1 Like

So I have been trying this out, and I am happy to say both connecting via Transmit and using s3cmd line tool for syncing files from my desktop works. It’s pretty cool, the commands for locally creating, removing, and syncing buckets work well, which means I now just have to figure out how to offload my blog’s WordPress images over to Minio and I can leave both Amazon’s S3 and Digital Ocean’s Spaces in the dust. Thanks for this guide/tutorial, it worked seamlessly for me and hopefully I’ll have a WordPress example to add to your Omeka example here soon…

Quick follow-up, that Delicious Brains post linked to above takes you through setting up the WP Offload Media plugin for MinIO, and I can gladly report it works a treat. I am currently downloading all my uploads backups from S3 and Spaces and syncing them with MinIO using s3cmd-so good.

I’ll write a longer post and link it here, but the " Configuring WP Offload Media" part of that post is where you start, and you will need to download and tweak the WP Amazon s3 and Cloudfront Tweaks plugin.

1 Like

Bit more here in this post about working with MinIO and using the WP Offload Media and the Tweaks plugins to add specific filters for MinIO: MinIO and Object Storage in the Reclaim Cloud | bavatuesdays

1 Like