A common technique for autoscaling Resque workers on Heroku uses the after_enqueue and after_perform hooks that run within each Resque worker to adjust the current worker scale up and down, respectively. The after_perform hook watches the pending job count. When no jobs remain, this hook will scale all workers down.

When scaling workers up, the Heroku API is additive; thus, additional workers can be scaled while others are actively processing jobs. As the Heroku API does not currently allow individually specified workers to scale down, however, it is necessary to wait until all workers are inactive before performing a scale down operation.

While the after_perform scale down approach mitigates against shutting an active worker down while pending jobs exist, a race condition can occur where the pending job count hits zero, and a scale down operation is invoked on the Heroku API prior to a subsequent job being enqueued from the app and reserved on a worker which will be shutdown in the current scale operation.

Recently, I have been working on the resque-heroku-scaler gem which autoscales Resque workers through a separate monitor process. The scaler monitor process polls for pending jobs against the specified Resque Redis backend. When scaling down, this scaler process locks each worker from attempting to reserve any future pending jobs before a scale down operation is performed.

To get started, include the scaler tasks in lib/tasks/scaler.rake

require 'resque/tasks'
require 'resque/plugins/heroku_scaler/tasks'

task "resque:setup" => :environment

In your Procfile, configure the scaler process using:

scaler: bundle exec rake resque:heroku_scaler

To run the scaler process, use the following command.

heroku scale scaler=1

Require the worker extensions within the app running your workers. For example, in lib/tasks/resque.rake. These extensions to the worker run loop poll for a shared flag within the Redis backend set by the scaler process which indicates the worker should enter an inactive scaling state. After a worker has entered this scaling state, it will cease attempting to check for incoming pending jobs. Once all workers have indicated they are inactive, the scaler process issues the scale down operation using the Heroku API.

require 'resque/tasks'

task "resque:setup" => :environment do
  require 'resque-heroku-scaler'
  ENV['QUEUE'] = '*'
end

In your development environment, the scaler process can run local worker processes using the rush library. To configure, use the following in your scaler rake file.

require 'resque/tasks'
require 'resque/plugins/heroku_scaler/tasks'

task "resque:setup" => :environment do
  if Rails.env.development?
    require 'resque-heroku-scaler'
    ENV["RUSH_PATH"] ||= File.expand_path('/path/to/app', __FILE__)
    Resque::Plugins::HerokuScaler.configure do |c|
      c.scale_manager = :local
    end
  end
end

Heroku provides an excellent platform to scale your Ruby or Node.js application through the cloud. Built on top of Amazon EC2, Heroku alleviates much of the server administration work typically associated with scaling applications out as user demand increases. Heroku is PaaS; Salesforce and Matz agree.

On the new Cedar stack, scaling your app’s web and background processes across N machines can be achieved using the scale command. While explicit scale commands work well for web processes as traffic increases, background workers tend to have a lot of free time if not managed automatically, running $36 a month per worker.

Several libraries currently autoscale background workers. For Resque based workers, Daniel Huckstep provides a cool approach to scale based on load. For Delayed Job, Heroku’s Pedro Belo has provided a DJ fork which supports autoscaling on Bamboo and Aspen. Additional DJ autoscale libraries include HireFire and Workless.

While I would prefer to use Github’s Resque for performance and manageability, to avoid an additional, potentially paid dependency on Redis at this stage, we are currently using Pedro’s Delayed Job autoscaling changes at RRemind. To support Cedar, I have made a couple of updates in a fork of the Delayed Job gem.

This implementation will autoscale a single worker up and down as background requests are generated by the application. Job retries are not currently supported. To use, create an initializer similar to below. In your local development environment, the rush library provides process management.

Delayed::Worker.auto_scale = true
Delayed::Worker.max_attempts = 1

if Rails.env.production?
  Delayed::Worker.auto_scale_manager = :cedar
else
  Delayed::Worker.auto_scale_manager = :local
end

To schedule a custom background job or delay send from a mailer, respectively:

Delayed::Job.enqueue YourAwesomeJob.new
UserMailer.delay.welcome_email(context)

In your Procfile, add:

worker: bundle exec rake jobs:work

Efficiently autoscaling your background workers eliminates spending for idle dynos. For a background load of 20 hours per month, you’d pay $1 autoscaling compared to $36 for an idle background worker running the entire month.

Over the past several years, PCI DSS has prompted quite a bit of change on the web in credit card processing. There is much to be read on the topic; though, in short, since 2004 a consortium of credit card companies have established the governing standard for how merchants must deal with credit cards online. Achieving compliance, however, can be quite costly.

Enter Braintree, a service based payment solution providing a merchant account, your processing gateway, and recurring billing on a single platform. By externalizing the credit card surface area within your application, Braintree’s unique Transparent Redirect technology allows you to preserve the sales experience within the context of your site while alleviating the hurdles associated with achieving PCI compliance.

In concert with the Braintree's Vault, Transparent Redirect enables developers to productively integrate recurring subscription billing. To remotely create customers within the vault using transparent redirect, call the create customer data method to embed the redirect data within the form as a hidden field.

class PaymentController < ApplicationController
  def signup
    @tr_data = Braintree::TransparentRedirect.create_customer_data(
      :redirect_url => "http://example.com/payment/confirm"
    )
  end
end

This data field will include the callback location to your confirmation action to be invoked after sensitive credit card data has been externalized to Braintree in the initial remote form post.

<%= form_tag Braintree::TransparentRedirect.url, :method => :post do %>
  <%= hidden_field_tag "tr_data", @tr_data %>
  <%= text_field_tag "customer[credit_card][number]" %>
  <%= text_field_tag "customer[credit_card][expiration_date]" %>
<% end %>

In your confirmation callback action, the customer record can be confirmed using the query string parameter below. The result will contain the identifier tokens which represent the remote customer and credit card, respectively.

result = Braintree::TransparentRedirect.confirm(request.query_string)
customer_id = result.customer.id
payment_token = result.customer.credit_cards[0].token

Lastly, to establish a recurring subscription associated with the remote credit card, use the payment token along with the selected plan.

result = Braintree::Subscription.create(
  :payment_method_token => payment_token,
  :plan_id => "super_awesome_plan"
)

Braintree's innovative approach to externalizing credit card information through transparent redirection takes the pain out of PCI compliance while preserving the user experience within the context of your site. We are looking forward to using their APIs more at RRemind. Braintree is paving the way in today’s demanding world of web based credit card processing. Use them.