A partial archive of meta.discourse.org as of Tuesday July 18, 2017.

Setting up file and image uploads to S3

zogstrip

So, you want to use S3 to handle image uploads? Here's the definitive guide:

S3 registration

Head over to http://aws.amazon.com/s3/ and click on .

During the create account process, make sure you provide payment information, otherwise you won't be able to use S3. There's no registration fee, you will only be charged for what you use, if you exceed the free usage tier.

User creation

Creating a user account

Sign in to AWS Management Console and click on to access the AWS Identity and Access Management (IAM) console which enables you to manage access to your AWS resources.

We need to create a user account, so click on the Users link on the left handside and then the button. Type in a descriptive user name and make sure the "Generate an access key for each User" checkbox is checked.

Here's the critical step: make sure you either download the credentials or you copy and paste somewhere safe both Access Key ID and Secret Access Key values. We will need them later.

Setting permissions

Once the user is created, we need to set him permission. Select the user you've just created in the upper panel, click on the Permissions tab in the lower panel and then click the button.

In the "Manage User Permissions" popup, select the radio button and click the select button to manually enter the permission.

Type in a descriptive name for the policy and use the following piece of code as a template for your policy document:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:*",
      "Resource": [
        "arn:aws:s3:::name-of-your-bucket",
        "arn:aws:s3:::name-of-your-bucket/*"
      ]
    }
  ]
}

First, some warnings about your bucket name:

Make sure you change both references to "name-of-your-bucket" with the name of the bucket you will use for your Discourse instance before applying the policy.

Discourse configuration

Now that you've properly set up S3, the final step is to configure your Discourse forum. Make sure you're logged in with an administrator account and go the Settings section in the admin panel.

Type in "S3" in the textbox on the right to display only the relevant settings:

You will need to:

  • Check the "enable_s3_uploads" checkbox to activate the feature
  • Paste in both "Access Key Id" and "Secret Access Key" in their respective text fields
  • Enter the name of the bucket you've authorized in the "s3 upload bucket"

The "region" setting is optional and defaults to "us-east-1". You should enter the location (eg. eu-west-1, sa-east-1, etc...) that is nearest to your users for better performances.

Enjoy

That's it. From now on, all your images will be uploaded to and served from S3.

Note how you did not have to create your S3 bucket? That's because Discourse will automagically create it for you if it doesn't already exists. :wink:

computerdruid

Note that the free usage tier only lasts for 12 months, so either don't forget about it and leave it active, or create an alert in AWS to have it notify you when it charges you money.

djensen47

Hitting the like button wasn't enough, I just have to saw this is awesome. Over at http://axisandallies.org/forums (which is down right now :disappointed:) people have uploaded tens of thousands of files and it's unmanageable with old style forums.

So thank you thank you thank you.

zogstrip

Just curious, what makes it unmanageable?

djensen47

On SMF there's a bunch of reasons:

  • All files are uploaded into a single directory
  • Files cannot be moved via the OS without breaking things
  • Filenames are generated
sam

To me this seems like a sensible constraint, only other 2 options are

  • Store file hashes and run recovery jobs that figure out where files really are based on a hash and a full scan of the filesystem.
  • Store attachements in the db, which is a world of pain.
djensen47

I agree, I probably wasn't specific enough. SMF doesn't really allow you to create your own directory structure. If you write a script to move a file and update the database, it still won't find it. There's a bunch of hardcoded crap in the code.

sam

Our design should be safe for moving the install, our tables really should only store relative locations.

haiku

I followed these instructions to the letter and I'm getting "Sorry, there was an error uploading that file. Please try again." every time.

Anyone got any ideas?

zogstrip

Have you got any errors in the logs?

haiku

There are some errors in production_errors.log, Amazon returns 403 "The request signature we calculated does not match the signature you provided. Check your key and signing method." It sounds like I put in the keys wrong, but I've checked that..

zogstrip

What's your S3 user policy?

haiku

This:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:*",
      "Resource": [
        "arn:aws:s3:::discourse-4mix",
        "arn:aws:s3:::discourse-4mix/*"
      ]
    }
  ]
}

Sidenote: how do you get syntax highlighting on code blocks (like in the op?)

zogstrip

Did you figure it out?

The highlighting engine is unfortunately not smart enought to detect that your code block is javascript. You can force it using GitHub's fenced code blocks.

joallard

I have the same "signature" problem.

Edit: I just changed the keys, and it went away. Maybe I had a trailing space? @haiku would you check yours?

l_g_k_h

Faced a problem below..

Installed latest discourse today. Setup S3 using the guide above but when creating new topic with image upload, it still loads from local /uploads folder.

So I checked Amazon S3 console, there was no bucket created.
Tried creating bucket manually, but same issue.
Checked keys such that there is no trailing space.
Checked logs, there was no errors relating to S3.


How can I debug this?

Discourse Version: 0.9.8.11
Git Version: a1b501c3fba126a3bc1705bea69a6397196b396e

UPDATED:
It is working now, guess it was some sort of delay/cache?

joegoggins

Does anyone have any suggestions about how to migrate file system stored images into S3?

If I need to write a script to make it happen that's cool (and I'd be happy to share it), just figured I'd ask here before diving into it.

philnelson

I would also like some kind of howto for this.

modius

Might want to add to the howto a recommendation to not use dots in the bucket name. This is allowed by AWS but prevents you from referencing the bucket under SSL.

I came a cropper over this one earlier:


codinghorror

Very good point, done in the code and here too. We've had 5-6 reports of bucket problems due to periods in the name, something we definitely want to avoid in the future.