It took me a while to get things working with Mezzanine and s3 and cloudfront. Here is the setup I settled on:

First install django-storages-redux, rather than django-storages. To do so, just make sure django-storages is completely uninstalled and install the redux package:

pip install django-storages-redux

You don't need to do anything else; then add storages in INSTALLED_APPS (just like you would for the regular module). You'll also need to install boto.

The basic settings

############ Bucket and creds
CLOUDFRONT_DOMAIN = "<your_cloudfront_domain>"   # if using cloudfront
AWS_STORAGE_BUCKET_NAME =  <your_bucket_name>'
AWS_ACCESS_KEY_ID = '<your_id>'
AWS_SECRET_ACCESS_KEY = '<your_sec_key>

########### Optional settings

# tells AWS to add properties to the files, such that when they
# get served from s3 they come with this header telling the browser to cache for
# life
AWS_HEADERS = { 
                     'Expires': 'Thu, 31 Dec 2099 20:00:00 GMT',
                     'Cache-Control': 'max-age=94608000',
                     }
# Used to make sure that only changed files are uploaded with collectstatic
AWS_PRELOAD_METADATA = True

#http://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-query-string-auth.html
# allows authenticating with creds in querystring for temp access to a resource
# Setting to False if not needed helps get rid of uwanted qstrings in compressed
# output
AWS_QUERYSTRING_AUTH = False

# if you don't need these
AWS_S3_SECURE_URLS = False
AWS_S3_ENCRYPTION = False
# from boto.s3.connection import ProtocolIndependentOrdinaryCallingFormat
# AWS_S3_CALLING_FORMAT = ProtocolIndependentOrdinaryCallingFormat()

########### required

#AWS_S3_CUSTOM_DOMAIN = '%s.s3.amazonaws.com' % AWS_STORAGE_BUCKET_NAME    # if not using cloudfront

# This was necessary or got IncompressibleFileError w Django compressor
AWS_S3_CUSTOM_DOMAIN = CLOUDFRONT_DOMAIN

# Static storage
STATICFILES_LOCATION = 'static'
STATICFILES_STORAGE = 'custom_storages.StaticStorage'

# If using django-compressor it needs to temp cache files somewhere
# make sure this matches your COMPRESS_ROOT too
STATIC_ROOT = '/home/djangoUser/temp_static/'
# STATIC_URL = "https://%s/%s/" % (AWS_S3_CUSTOM_DOMAIN, STATICFILES_LOCATION)   # if basic s3
STATIC_URL = "https://%s/%s/" % (CLOUDFRONT_DOMAIN, STATICFILES_LOCATION)   # if cloudfront
ADMIN_MEDIA_PREFIX = STATIC_URL + 'grappelli/'
# ADMIN_MEDIA_PREFIX = STATIC_URL + 'admin/'

# Media storage
MEDIAFILES_LOCATION = 'media'
# MEDIA_URL = "https://%s/%s/" % (AWS_S3_CUSTOM_DOMAIN, MEDIAFILES_LOCATION)
MEDIA_URL = "https://%s/%s/" % (CLOUDFRONT_DOMAIN, MEDIAFILES_LOCATION)
 MEDIA_ROOT = ''
DEFAULT_FILE_STORAGE = 'custom_storages.MediaStorage'

Most of these settings are self-explanatory. I tried to include in the commented versions which settings are for pure s3 and which are for cloudfront. Some settings like STATIC_ROOT are not needed if you are not using django-compressor. The reason you need it with compressor is because it needs a (at least temporary) local filesystem cache of static files to work out the compressed static files from. These files are taken from whatever dir COMPRESS_ROOT points to. Given that Django offers collectstatic to the STATIC_ROOT, you should make sure COMPRESS_ROOT=STATIC_ROOT, and this should all just work, with one other small tweak to the custom static storage class. I'll talk about that momentarily, but essentially it just tweaks the storage so that as well as pushing up to s3 on saves locally. See here. Anyway, you can ignore all that if you're not interested in django-compressor.

Ensuring static and media files are stored in different dirs

In the above settings, I am using custom storage classes for both static and media files. They are defined as

# custom_storages.py
from django.core.files.storage import get_storage_class
from django.conf import settings
from storages.backends.s3boto import S3BotoStorage
from filebrowser_safe.storage import S3BotoStorageMixin



class CachedS3BotoStorage(S3BotoStorage, S3BotoStorageMixin):


    def __init__(self, *args, **kwargs):
        super(CachedS3BotoStorage, self).__init__(*args, **kwargs)
        self.local_storage = get_storage_class(
            "compressor.storage.CompressorFileStorage")()

    def save(self, name, content):
        name = super(CachedS3BotoStorage, self).save(name, content)
        self.local_storage._save(name, content)
        return name


class MediaStorage(S3BotoStorage, S3BotoStorageMixin):
    location = settings.MEDIAFILES_LOCATION

 StaticStorage = lambda: CachedS3BotoStorage(location=settings.STATICFILES_LOCATION)

There are a few things going on here:

  1. We want to define two custom storage classes sub-classing S3BotoStorage because we want to set the location attribute to be different for static and media files. This ensure that static files go in the 'static' dir of the bucket and media files go in the 'media' dir.
  2. We use the S3BotoStorageMixin filebrowser_safe. Without this Mezzanine's filebrowser will just die whenever you try to upload a file or peruse your media library. You'll get various errors saying isdir is not defined and so forth. The mixin defines those methods for s3 storage, although admittedly rather badly as things go at snails pace it seems.
  3. For the static we define CachedS3BotoStorage as per the django-compressor docs. This defines a local_storage attribute, and makes it so that the save method now writes the static file to that local storage aswell as pushing up to s3. This all means that collectstatic will copy the files both to the local filesystem at the STATIC_ROOT dir, and also push up to s3. Now if the COMPRESSED_ROOT=STATIC_ROOT, django-compressor can use this cache of static files to create its compressed files (I'm not quite sure how this could work on something like Heroku given the ephemeral nature of the fs storage, anyone know if django-compressor will work there?)

Bucket policy, users IAM and cors

Instead of using the AWS ID and key for your Amazon main account, it makes sense to add a user with only the necessary privileges needed on this s3 bucket. This way if your server ever gets compromised, then the worst the attacker can do is mess with s3, not your whole Amazon account!

To create a user login to your AWS account and navigate to IAM. Select Users then "Create New Users", input the username(s), and ensure "create security credentials for each user" is ticked. You'll be presented with an "Access Key ID" and " Secret Access Key" for each user. You should download these and securely save them, as you will never see them again otherwise. These are now the creds you should input into the settings for Mezzanine above.

It's good practice now to disable the root access keys, and amazon will show you a warning somewhere about this if you haven't done it already.

If you're happy with that user having full access to s3 (all buckets), then you can attach a policy to that user, by clicking 'Attach Policy' and searching for "s3". Finally just add the result "AmazonS3FullAccess". I don't recommend this for a user that is to be associated with a given Django app, but I use it for a user whose creds I use with the 's3cmd` tool on my local machine, where I want full access to multiple buckets for various reasons.

If you want to refine things so the user only has access to a single bucket, then you'll need to amend the policy on the bucket itself. First grab the "User ARN" string from the summary page that is displayed when clicking on your user. Next navigate to services>s3 and click your bucket. Click the properties tab over on the right and expand the permissions section. Now you can add a policy that looks something like this:

{
    "Version": "2008-10-17",
    "Statement": [
        {
            "Sid": "PublicReadForGetBucketObjects",
            "Effect": "Allow",
            "Principal": {
                "AWS": "*"
            },
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::<your_bucket_name>/*"
        },
        {
            "Sid": "",
            "Effect": "Allow",
            "Principal": {
                "AWS": "<your_user_arn>"
            },
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::<your_bucket_name>/*",
                "arn:aws:s3:::<your_bucket_name>"
            ]
        }
    ]

}

This basically says allow anybody to read any object from this bucket. Allow your newly created user to do anything (read/modify/whatever) this bucket.

The final thing is to add a CORS policy to the bucket:

<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
    <CORSRule>
        <AllowedOrigin>*</AllowedOrigin>
        <AllowedMethod>GET</AllowedMethod>
        <MaxAgeSeconds>3000</MaxAgeSeconds>
        <AllowedHeader>Authorization</AllowedHeader>
    </CORSRule>
</CORSConfiguration>

settings custom storages: what some of the complex settings actually mean

Issues

You can check the related posts for some of the big stumbling blocks I came across along the way, but some smaller things of note:

  1. We need to set MEDIA_ROOT='' will s3 when using Mezzanine and filebrowser, otherwise we'll run into SuspiciousOperation errors or even worse, hard to debug 400 responses with no trace (Django 1.6). This is explained this post. Fine, but the Mezzanine thumbnail tag currently operates by caching thumbnails to local filesystem before pushing up to s3. With MEDIA_ROOT='', this will result in an upload dir full of thumbnails being written somewhere to your fs (no worries if on Heroku, as it only needs to be cached there temporarily anyway). Nevertheless, it's important to know where this directory is going to be written, so that you can set permissions correctly on it. For me Apache was running in daemon mode as user "djangoUser", and the thumnail dir was written under "/home/djangoUser" when MEDIA_ROOT=''. If you can't figure out where this dir is, and you are getting permissions errors, try adding print os.getcwd() into the source code of the mezzanine_tags.py thumbnail tag.

  2. As I mentioned, the S3BotoStorageMixin allows filebrowser to work with s3 by providing the required isdir method and other methods that filebrowser needs and that django-storages (redux or otherwise) don't provide. Nevertheless, it is agaonizingly slow now browsing your media library with filebrowser. I think there has to be a better mixin soon that refines these methods and allows some speedup. At the moment it's so slow that I just opt completely remove filebrowser-safe, and have Mezzanine fall back to the usual Django file upload widget (just pip uninstall filebrowser-safe is all you need to do, no changes to settings or code needed). It's not as nice as filebrowser, as all you can do is upload, so I need to manage my media via s3 directly. Also for images added in the rich text, I need to add the URL directly from s3 (I'll have to use a find and replace script in the future if my static hosting changes), but it works for me at the moment, and it's A LOT faster.

Current rating: 5

About Lee

I am a Theoretical Physics PhD graduate now working in the technology sector. I have strong mathematical skills and originally started in heavy duting scientific computing, but now I work mostly with Python and the Django framework. I am available for hire now, so check out my resume and get in touch.

Comments