From Paperclip to Active Storage: An incremental, zero-downtime approach

I recently switched Doorkeeper from using Paperclip to Active Storage for storing and processing uploaded files. The approach we took was a bit different from what I saw documented elsewhere, and this article explains why we took the approach we did, how we did, and things to watch out for when making the switch regardless of the approach you take.

Background

Doorkeeper is an event management platform for communities focused on the Japanese market. I started it as a side project, but since then it has grown into a mature business.

Using Paperclip since 2010

We’ve used Paperclip to handle user uploads since 2010. Paperclip was deprecated with the release of Active Storage, and though we explored moving to Active Storage at the time, as it didn’t seem to easily support CDNs, we decided to leave things as is.

As Paperclip isn’t compatible with Ruby 3.0, and Rails 6.1 brought an officially sanctioned way of using Active Storage with a CDN, I decided it was time to make the switch.

120k attachments to, 34 GB of files

At the time of the switch, we had 120k attachments stored with Paperclip, representing 34 GB of uploaded files. While this was enough that switch was a serious undertaking, I wasn’t concerned about things like minimizing the cost of file transfers, and so didn’t need a solution that would say leave the original uploads in place.

250 requests per minute, 60 uploads per day

As uploads are primarily performed by event organizers, and participants greatly outnumber organizers, we have very few uploads compared to our overall traffic. In a typical day we get around 60 uploads, while our Rails application is receiving about 250 requests per minute.

Because of this imbalance, having a CDN was quite important to us, both to ensure responsiveness to the visitors of an event page, and to avoid unnecessary load for our Rails application.

Goals of our switching strategy

In making the switch, I wanted a strategy that would have zero downtime, be easy to rollback, and could be done in an incremental fashion.

Zero downtime

While planned downtime can sometimes be the simplest approach, it also means you need to make a tradeoff between disrupting customers and disrupting your own sleep (assuming you and your customers live in the same time zone). Besides that, planned downtime offends my pride as a developer, and so if possible, I wanted to avoid it.

Easy to rollback

To minimize the impact of something going wrong, I wanted an approach that would allow us to quickly switch back from Active Storage to Paperclip.

Incremental

Attachments in Doorkeeper were spread across six models. I wanted to be able to incrementally migrate from models that appear in non-critical places and are few in number (e.g., for our listing of companies that host tech meetups in Tokyo) to those that appear all over the place and are numerous (e.g., user avatars). This way, if we didn’t anticipate something correctly, and there was an issue with the migration, we’d likely catch it in a low impact place.

Strategy for switching

With those goals in mind, I devised the following strategy for gracefully switching to Active Storage.

1. All new uploads go to both Paperclip and Active Storage

As a first step, we proactively stored all new uploads with both Paperclip and Active Storage. Because both Paperclip and Active Storage directly map the name of the attachment attribute in the model to what they store in the database, I created a module that would handle this:

By extending this module, and changing has_attached_file to has_paperclip_attachment_with_active_storage, uploads would be stored using both plugins, while only the Paperclip version would be used to actually serve the uploads (and so no other code needed to be changed).

2. Backfill past uploads from Paperclip to Active Storage

Rather than attempting to convert existing Paperclip attachments to Active Storage ones, I decided the simplest approach would be to just store an attachment in Paperclip based on the Active Storage one. This, when combine with the previous step, would mean we could have all attachments stored using both plugins, and thus seamlessly switch between which one we used to actually serve the attached file.

While there was a number of strategies I could of used to backfill the existing attachments, as it was a resource intensive task that could be parallelized, I decided creating an Active Job would be the best approach:

Running a command like PaperclipToActiveStorageJob.generate!(User, :avatar) spawns the individual jobs that actually backfill the Active Storage attachments.

3. Use Active Storage to serve uploads

Once all the past uploads were backfilled for a given model, I switched it over to use Active Storage instead of Paperclip to serve the uploads.

As part of this step, I also removed the code for storing the attachment to Paperclip. This created the risk that if I needed to roll back, I could theoretically lose an upload. A more conservative approach would have been to continue to store the attachment to both, and drop storing it to Paperclip only after I was sure I wouldn’t need to roll back.

Differences in image processing between Active Storage and Paperclip

Both Paperclip and Active Storage let you apply transformations to images, such as resizing them to fit within a specified set of dimensions. However, the way that they do this is different, and some of the subtle applications weren’t obvious to me until we deployed to production and encountered edge cases.

Active Storage only transforms a subset of images

Paperclip attempts to process any attachment by invoking imagemagick with the appropriate arguments. Active Storage is more conservative, and if you call the variant method on an attachment with a content type that is not in ActiveStorage.variable_content_types, it raises an exception. Doorkeeper had saved a number of attachments that weren’t considered variable by Active Storage, notably image/x-png and image/svg+xml, and thus couldn’t be transformed.

To resolve the error, I inspected the files one at a time, which as there was only one hundred or so was feasible to do. Almost all of them were quite old (e.g. images for events that had been held seven years ago). For the ones that were still being actively used (e.g. a logo for a community), I manually transcoded it to an acceptable format. Otherwise, for ones that were no longer used (e.g. an image for an event that was held seven years ago by a community that no longer holds events), I just deleted the attachment.

Additionally, I changed the validation for the attachment to restrict it to ones with a content type of Rails.application.config.active_storage.variable_content_types, rather than something more permissive like /\Aimage\/.*\Z/. I initially had tried to use Active Storage.variable_content_types directly, however, in production I found that it was initialized only after my models were loaded, resulting in my validations not working.

Active Storage treats PDFs differently than images

With Paperclip, styles you defined would be applied both to images and PDFs. Active Storage differentiates between variants (images) and previews (PDFs, videos). For one of Doorkeeper’s attachments we accepted both image and PDF uploads, and regardless of what the user uploads, we want to provide the same image preview. To do this, we defined a helper method that invoked either the preview or variant method, whichever was appropriate:

Regenerating transformations is resource intensive

In making the switch, the URLs to our attachments changed. As most of these attachments were in publicly accessible pages, it meant that as crawlers picked up the new image URLs, they would visit them, triggering the transformation of images. This peaked at us receiving approximately 400 transcoding requests per minute, which used a significant percentage of our computing resources.

Around this time, we were also noticing out of memory errors related to image processing. This is something we experienced very occasionally in the past with paperclip, but we saw it happening more frequently after the migration to Active Storage. It seems likely that this increase in errors was related to the migration, though it subsided within a couple of days, before we were able to make a concrete diagnosis.