Many Ruby on Rails apps use Amazon AWS S3 buckets for storing assets. When dealing with files uploaded by front-end web or mobile clients there are many factors you should consider to make the whole process secure and performant. In this blog post, I will describe common pitfalls and an optimal solution when it comes to handling client-side file uploads.
I will be using Fog gem (version 2.0.0 at the time of writing) in the examples. It is a dependency of Carrierwave and Paperclip two popular file uploader gems. If you use either of them, you should already have it included in your app. If not just add:
It is a lower level API than Carrierwave, Paperclip or Shrine gems. Getting familiar with it will help you understand how AWS S3 buckets work without all the high-level magic provided by these gems. I find it more pleasant to work with than an official Ruby AWS SDK.
AWS S3 setup
You will need Amazon AWS credentials to start working with file uploads. One common mistake is to use your primary user credentials instead of creating an IAM user with limited permissions.
If your primary credentials are compromised you could wake up with a huge bill because of Bitcoin mining bots. Losing credentials with S3 permissions only is much less severe.
Add an IAM user
I will not elaborate on how to add an IAM Amazon AWS user. There is plenty of other tutorials about it. Make sure you give your user correct permission policies. To follow this tutorial you should use
AmazonS3FullAccess policy for your IAM user.
AWS credentials on a client-side
You should never store your AWS credentials in a client-side app. One misconception is that you can store them in native mobile apps because they are compiled and therefore secure. Remember that there are tools which allow to decompile an app and retrieve plain text strings from its binary.
In the following examples, I will explain how to provide secure access to a private Amazon S3 buckets without exposing your credentials to a client side.
If you accidentally commit your AWS credentials to a GitHub repo make sure to remove them.
Setup a test bucket
You will need an S3 bucket to follow this tutorial. Let’s configure an API client and create an empty bucket to work with:
BTW it is a good practice to use
fetch for reading ENV variables. It is always better to fail fast when you forget to configure something in a given environment instead of having to track bugs caused by unexpected
Now that we have confirmed that our credentials are correct and our test bucket is ready let me describe incorrect ways to handle file uploads.
Insecure uploads to a public bucket
I will not elaborate on the client side in this tutorial. I will simulate client-side uploads with simple cURL HTTP requests. It can be done with all the front-end technologies.
Configure bucket for public uploads
ACL stands for Access Control List. It can easily be configured via Ruby SDK:
Now everyone can upload a file directly to the bucket without an authentication:
It issues a HTTP PUT request with a file binary data in its body.You should receive a 200 HTTP response. To double check that file is actually in a bucket run the following command:
Leaving your bucket open to the public is a serious security threat. Malicious users could freely read all the data and modify it if your bucket read and write access is publicly available. It is still better than having your AWS credentials compromised but I highly discourage you from ever using this approach in production apps.
If for whatever strange reason you must use publicly open buckets in your application make sure to enable access logs for them to have at least a minimal insight and control.
Secure uploads via a Rails server
Another way to upload files from a client side is to proxy them through your Rails server. In that case, an authentication mechanism is provided by our own app, probably in form of a token or a session cookie.
This time our bucket should be set as private:
Below examples assume that you have some kind of authentication in place but cURL requests to a Rails app do not include it for simplicity.
Now let’s see a sample Rails implementation:
To interact with this controller you can use the following cURL commands:
Objects uploaded this way will be private and readable only by authenticated clients. You can double check it:
This solution is secure. AWS credentials are not exposed to the client side and only authenticated users can interact with bucket resources. In real life application, you would probably want to scope the bucket access on per user or group basis depending on your business logic.
Proxied file transfers issue
Although this solution is secure there are some serious performance related problems with proxying both download an upload process through Ruby servers.
Imagine that multiple mobile clients with a slow internet connection start uploading or downloading large files from your servers. It would mean that your main application process is blocked for all the other requests. It would be like running database queries which take a dozen of seconds and could easily bring your whole application down. What’s worse you would have to pay for all the file transfer bandwidth twice.
Luckily AWS S3 offers a neat solution to this problem.
Secure and direct uploads
Now we know how to upload assets directly from clients in an insecure way and how to do it securely through our servers sacrificing performance.
Let’s combine the best of both.
Direct S3 file uploads with presigned URLs
We will use so-called pre-signed URLs in this example. It is a way to generate URLs, valid only for a short amount of time which you can use to access assets on private buckets.
Here’s how they look in action:
To upload a file you have to do the following:
It will return an URL which you can use to upload a file with a given filename:
What’s important is that this upload URL will only be valid for a specified
TIME_TO_ACCESS period. Client-side must be programmed to access the URL right after receiving it from the server.
Similarly, you can retrieve a download URL from a show action:
and use it to download the desired asset directly from a bucket:
[WARNING] remember to parse the JSON from raw cURL command result, otherwise, the URL will not validate correctly with AWS API.
Advantages of pre-signed URLs
This solution is optimal in terms of security and performance. Files are not transferred through our servers so bandwidth and even the slowest mobile clients are not a problem.
Security credentials never leave the server, our bucket is private and URLs we share are only issued to authenticated clients and valid for a short period of time.
It is essential to make the validity period as short as possible. Imagine that your app’s user logged into an account on someone else’s computer and accessed his S3 assets there. If your signed URLs will be valid longer they could be retrieved from the browser’s history even if your user logged out.
The same if he accessed your app via a compromised network. Middleman could obtain his access token and asset links but it would not be a serious security threat. Token gets invalidated after sign out and links no longer grant access to the bucket resources because they expired.
With the high-level APIs provided by popular S3 gems, it is very easy to overlook some detail and make a mistake which could compromise the security or kill the performance of your app. Getting to know how AWS S3 works on a lower level will allow you to make more thoughtful decisions when designing your app’s infrastructure.