Setting up MongoDB, MongoMapper, GridFS, Devise & Carrierwave on OSX@matsimitsu

For a little side project I wanted to experiment with MongoDB and GridFS on Rails 2.3.5, but it took me a while to get everything up and running.

It wasn’t really a problem with Mongrel, but as soon as Apache and Passenger came in the picture the problems began.

The gems

Development setup:

config.gem 'carrierwave'
config.gem 'devise'
config.gem 'mongo_mapper', :version => '0.7.0'
config.gem 'mongomapper_ext'
config.gem 'will_paginate'

If you want to use GridFS with MongoDB, make sure you use MongoMapper 0.7.0, because later versions cause:

mongo/GridFS not found

Test setup:

config.gem 'rspec', :lib => 'spec'
config.gem 'rspec-rails', :lib => 'spec/rails'

config.gem 'machinist_mongo',
  :version => '1.1.0',
  :lib => 'machinist/mongo_mapper'

config.gem 'faker'
config.gem 'steak'
config.gem 'capybara'

MongoMapper

To use MongoMapper add the following to your config/initializers:

# config/initializers/mongo_mapper.rb

# load YAML and connect
database_yaml = YAML::load(File.read(RAILS_ROOT + '/config/database.yml'))
if database_yaml[Rails.env] && database_yaml[Rails.env]['adapter'] == 'MongoDB'
  mongo_database = database_yaml[Rails.env]
  MongoMapper.connection = Mongo::Connection.new(mongo_database['host'], 27017, :pool_size => 5, :timeout => 5)
  MongoMapper.database =  mongo_database['database']
end

if defined?(PhusionPassenger)
   PhusionPassenger.on_event(:starting_worker_process) do |forked|
     MongoMapper.connection.connect_to_master if forked
   end
end

The PusionPassenger section can be removed if you are not planning to run this on Apache and Passenger.

If you do plan to run it, make sure this is in your config. It will make sure new spawned Apache children still have a MongoDB database connection and prevent you from pulling out all your hair.

You can now setup your config/database.yml config. Example:

# config/database.yml

development:
  adapter: mongodb
  database: mmyapp_development
  host: localhost

Devise

Setting up Devise is pretty easy, just use the following snippet:

# config/initializers/devise.rb

Devise.setup do |config|
  config.mailer_sender = "info@example.com"
  config.orm = :mongo_mapper
end

And use it in your (user) model:

class User < Base
  include MongoMapper::Document
  include MongoMapperExt::Slugizer

  devise :registerable, :database_authenticatable, :recoverable, :rememberable, :trackable, :validatable
end

Since we are on MongDB you can skip the migrations!

Carrierwave

Next up is image uploading, I used Carrierwave.

Again some configuration, this time make sure that the application_#{Rails.env} is consistent with your database.yml file.

# config/initializers/carrierwave.rb

CarrierWave.configure do |config|
  config.grid_fs_database = "application_#{Rails.env}"
  config.grid_fs_host = 'localhost'
  config.grid_fs_access_url = "/GridFS"
  config.storage = :grid_fs
end

With Carrierwave configured we can now implement it in our model.

An example is:

class Photo
  require 'carrierwave/orm/mongomapper'
  include MongoMapper::Document

  mount_uploader :image, ImageUploader
end

Note the mount_uploader. This will tell Carrierwave to look for a file called image_uploader in /app/uploaders/.

That file should contain some of the following:

# app/uploaders/image_uploader.rb
class ImageUploader < CarrierWave::Uploader::Base
  include CarrierWave::ImageScience

  storage :grid_fs

  def store_dir
    "files/#{model.id}"
  end

  def extension_white_list
    %w(jpg jpeg gif png)
  end

  version :small_thumb do
    process :resize_to_fill => [100,100]
  end
end

A couple of things are important here, first the storage :grid_fs. It will tell Carrierwave to store the files in GridFS.

Next is the def store_dir. This is used to generate the path to the images.

/GridFS/files/[ID]/[filename].[ext]

Note that the path is prefixed with /GridFS. This is the value we have set in the config of Carrierwave.

To serve out the files from Rails we are using a GridFS metal. This will increase the speed of image serving.

# app/metal/grid_fs.rb

# Rails metal to be used with Carrierwave (GridFS) and MongoMapper

require 'rubygems'
require 'mongo'
require 'mongo/GridFS'

# Allow the metal piece to run in isolation
require(File.dirname(__FILE__) + "/../../config/environment") unless defined?(Rails)

class GridData < Rails::Rack::Metal
  def self.call(env)
    if env["PATH_INFO"] =~ /^\/GridFS\/(.+)$/
      key = $1
      if ::GridFS::GridStore.exist?(MongoMapper.database, key)
        ::GridFS::GridStore.open(MongoMapper.database, key, 'r') do |file|
          [200, {'Content-Type' => file.content_type}, [file.read]]
        end
      else
        [404, {'Content-Type' => 'text/plain'}, ['File not found.']]
      end
    else
      [404, {'Content-Type' => 'text/plain'}, ['File not found.']]
    end
  end
end

An important thing to note is the regex part:

if env["PATH_INFO"] =~ /^\/GridFS\/(.+)$/

The GridFS in the regex is the same as in your Carrierwave config!

OSX and open files

At some point when I began stress testing MongoDB by requesting a lot of files from GridFS. MongoDB would become unresponsive and eventually crash the Rails application.

After a deep dive into the logs I found out that MongoDB has a lot of files open. More then the 256 that OSX wil allow by default.

To fix this use the following from serverfault, it will basically set the open files limit a lot higher for OSX.

To change any of these limits, add a line (you may need to create the file first) to /etc/launchd.conf, the arguments are the same as passed to the launchctl command. For example:

sudo echo "limit maxfiles 1024 unlimited" > /etc/launchd.conf

However launchd has already started your login shell, so the simplest way to make these changes take effect is to restart our machine.

That’s it!

Well that’s it! you should now have a working Rails 2.3.5 stack with MongoMapper, files served from GridFS, authentication from Devise and image uploading with Carrierwave.