Packs is a code organization technique introduced by Shopify and Gusto. It’s a “specification” for organizing (lots of) code in a Rails application. In short, you can imagine that you split your Majestic Monolith internally into multiple local “microservices” that are still part of the same Rails application.
Each “pack” is a separate directory and partially mimics the Rails application structure (app/models, app/controllers, etc.). The idea is to have a clear separation of concerns and to make it easier to work on a specific part of the application without having to deal with the whole monolith when working on a specific feature.
We at pludoni have been using that in two of our projects for about 6 months now, and we are quite happy with the results. Also, I recently gave a short intro presentation on “Packs” at the Ruby User Group Frankfurt which I want to recap here in this post.
Code structuring by feature
By default, in Rails applications, code is structured by type of code (models, controllers, views, jobs, mailers, and the infamous “services”). The mantra Convention over Configuration leads to a structure that is easy to understand and navigate. But as the application grows, it becomes harder to find the code you are looking for. Especially, if you start adding namespaces (modules) to bundle related code together, you end up with a structure that is hard to navigate and jump between related classes.
find . | grep billing
app/models/billing/invoice.rb
app/models/billing/payment.rb
app/controllers/billing/invoices_controller.rb
app/views/billing/invoices/index.html.erb
app/views/billing/invoices/...
app/jobs/billing/invoice_remind_unpaid_job.rb
#...
app/javascript/billing/index.js
#...
spec/models/billing/invoice_spec.rb
spec/requests/billing/invoice_flow_spec.rb
#...
Imagine, your app has a “Billing” concern, that includes models, controllers and jobs, and maybe a significant section of your config/routes.rb. All these files are spread across the application foler, and you have to jump between directories to work on a feature. If your app grows, you might have hundreds of files in app/models. Especially, when your app code grows more horizontally - that is, you add more and more relatively independent features, such as: Billing, Auth, statistics/analytics, onboarding, import/export, API, Admin-UI, etc - then might be a good time to start looking into Packs or similar feature-based code organization.
Why Packs? What’s wrong with Engines?
Engines (also named Rails plugins) are a way to share reusable code across multiple Rails applications. They are always a Gem and facilitate the sharing of code, views, controllers, etc. between multiple Rails applications. They are a great way to share code between multiple applications, but they have some downsides when it comes to code organization within a single application:
- Engines are isolated: They are isolated from the main application, which means that you can’t access the main application’s code from the engine (easily). Also, referencing the styling (layout) of the “Main-App” can be tricky. Also, accessing global objects, such as Current-User is more difficult than necessary.
- Increased complexity: By favoring reusability, Engines introduce complexity that we don’t need in a monolith, such as: distribution as a Gem, “Dummy App” management; needs to be familiar with the variants: “–mountable”, and
isolate_namespace. - Added boilerplate: Engines are Gems, so you need: Railtie, Gemfile, Gemfile.lock, gemspec, and maybe Initializers. On the monolith side, you need the Gem declaration or explicit mounting in the routes.
Thus, let me introduce you to Packs, which get rid of the downsides for single-use integrated mini-apps
Getting started
In a bog-standard Rails application, just add the packs-rails Gem to your Gemfile:
bundle add packs-rails
# create package.yml in root of app
bundle binstub packwerk
bin/packwerk init
Packs-Rails is also “Convention-Over-Configuration”, and assume your feature-Packs live under ./packs/. I suggest, you keep it that way. packs also ships with a bare-bones generator that can create a base pack skeleton for you:
bin/packs create packs/billing
# you can create more base folder
mkdir -p packs/billing/app/models/billing
mkdir -p packs/billing/app/controllers/billing
mkdir -p packs/billing/app/views/billing
mkdir -p packs/billing/app/jobs/billing
mkdir -p packs/billing/spec/models/billing
mkdir -p packs/billing/spec/requests/billing
# ...
After restarting your Rails app, autoloading should already work, that means, it doesn’t matter, if you put a file under app/models/billing/invoice.rb or packs/billing/app/models/billing/invoice.rb. As you notice, there is no automatic module namespacing, like you might be familiar with in Engines. I welcome this, as it makes it easier to understand and navigate.
That should be enough to get started! After you get more familiar with Packs, you can try to move more code into Packs. In the following sections, I document some ways, to integrate more type of code into a Pack.
Recipes
This sections contains some recipes, how to integrate parts, that are not always needed and depend on your stack.
AutoAnnotateModels
We use annotate-models to put a schema comment at the top of each model file. You need to tell annotate about those paths:
# lib/tasks/auto_annotate_models.rake
if Rails.env.development?
require 'annotate'
task :set_annotation_options do
# You can override any of these by setting an environment variable of the
# same name.
Annotate.set_defaults(
'model_dir' => (Dir['packs/*/app/models'] + ["app/models"]).join(','),
#...
Routes
Each Pack can have its own routes file. Starting from Rails 6.1, you can include routes that match the convention config/routes/ROUTE_NAME.rb with draw(:ROUTE_NAME) in your main routes file.
# config/routes.rb
Rails.application.routes.draw do
draw(:billing)
# ...
end
In your Pack, you can define routes as usual:
# packs/billing/config/routes/billing.rb
namespace :billing do
resources :invoices
end
Needs server restart.
Vite / TypeScript
We like to move some of related frontend code into the Pack as well. For example, we use Vite as a frontend build tool and TypeScript as a language. We can tell both to look into the packs:
// vite.config.ts
export default defineConfig({
resolve: {
alias: {
'@billing': path.resolve(__dirname, 'packs/billing/app/javascript'),
},
},
// ...
});
And Typescript:
# tsconfig.json
{
"compilerOptions": {
"baseUrl": ".",
"paths": {
"@billing/*": ["packs/billing/app/javascript/*"]
}
}
}
GraphQL frontend queries
In one app, we use GraphQL for a lot of Frontend queries. We use graphqlcodegen and graphql-request to compile the queries ahead as a simple Javascript function call.
overwrite: true
schema: "app/graphql/schema.json"
documents:
- "app/javascript/graphql/queries/**/*.graphql"
- "packs/*/app/graphql/queries/**/*.graphql"
generates:
app/javascript/graphql_queries/requests.ts:
config:
avoidOptionals: true
immutableTypes: true
constEnums: true
disableDescriptions: true
plugins:
- "typescript"
- typescript-operations
- typescript-graphql-request
Phlex Components
We dabble with Phlex for some of our Frontend (Non-JS) components. We can tell Phlex to look into the Packs as well:
# config/application.rb
module MyApp
class Application < Rails::Application
config.autoload_paths << "#{root}/app/views/components"
Dir['packs/*/app/views/components'].each do |path|
config.autoload_paths << path
end
#...
Rspec/Test-Unit
Packs-Rails has documentation, on how to integrate your test suite and make just running the tests in a Pack work. I suggest you read the official documentation.
For Rspec, just add this line to .rspec
--require packs/rails/rspec
Parallel Testing using test-boosters
We use test_boosters to parallelize our test suite. You need to explicitly tell test-boosters to also run your Pack tests, with Rspec:
export TEST_BOOSTERS_RSPEC_TEST_FILE_PATTERN="{packs,spec}/**/*_spec.rb"
rspec_booster --job $CI_NODE_INDEX/$CI_NODE_TOTAL
I18n-Tasks
We use i18n-tasks to manage our translation files. Using the write: configs + i18n-tasks normalize -p command, you can define the deterministic file for each translation key based on it’s prefix.
So, for each pack we define custom translations like this:
# Locale files to write new keys to, based on a list of key pattern => file rules. Matched from top to bottom:
# `i18n-tasks normalize -p` will force move the keys according to these rules
write:
- ['{billing, models.billing_jobboard, components.billing }.*', 'packs/billing/config/locales/%{locale}.yml']
- ['simple_form.*.billing_*.*', 'packs/billing/config/locales/%{locale}.yml']
- config/locales/%{locale}.yml
Generator
To make creating new packs easier, I strongly suggest, you create a generator based on your App’s needs. Here is a starting point:
rails generate generator pack
vim lib/generators/pack/pack_generator.rb
Here is our current generator:
# lib/generators/pack/pack_generator.rb
class PackGenerator < Rails::Generators::NamedBase
source_root File.expand_path("templates", __dir__)
def create_pack_subdir
pack_dir = "./packs/#{file_name}"
empty_directory pack_dir
%w[app/models app/controllers app/views app/jobs app/mailers app/jobs].each do |dir|
empty_directory "#{pack_dir}/#{dir}/#{file_name}"
end
empty_directory "#{pack_dir}/app/javascript"
empty_directory "#{pack_dir}/config/locales"
empty_directory "#{pack_dir}/config/routes"
route_file = <<~RUBY
# use your routes here:
# namespace :#{file_name} do
# resources :#{file_name}
# end
RUBY
create_file "#{pack_dir}/config/routes/#{file_name}.rb", route_file
route = " draw(:#{file_name})\n"
insert_into_file "config/routes.rb", route, after: "Rails.application.routes.draw do\n"
create_file "#{pack_dir}/config/environment.rb", <<~RUBY
# this file is only for vim-rails to pick it up as a Rails environment
RUBY
create_file "#{pack_dir}/package.yml", <<~YAML
enforce_dependencies: true
# Turn on privacy checks for this package
# enforcing privacy is often not useful for the root package, because it would require defining a public interface
# for something that should only be a thin wrapper in the first place.
# We recommend enabling this for any new packages you create to aid with encapsulation.
enforce_privacy: false
# By default the public path will be app/public/, however this may not suit all applications' architecture so
# this allows you to modify what your package's public path is.
# public_path: app/public/
dependencies:
- "."
YAML
readme = <<~README
# #{file_name}
TODO: Write description of this pack
README
create_file "#{pack_dir}/README.md", readme
empty_directory "#{pack_dir}/spec/models/#{file_name}"
empty_directory "#{pack_dir}/spec/controllers/#{file_name}"
empty_directory "#{pack_dir}/spec/controllers/#{file_name}"
puts "Restart the server to setup load paths"
end
TBD: migrations
We still put the migrations into db/migrate.
Conclusion
Packs-Rails is great, as you can start easily, and just make your new feature in the Pack structure, but still keep your main app intact. For our main apps, we strive to make every new feature into a new Pack, and we have begun to extract related classes into a pack. At the moment, we are at about 10 Packs in one of our apps. The best feature is, with each Pack you get the “New App Smell”! ✨🤩✨ every single time. It’s easier to load a feature into your head and not be overwhelmed with hundreds of files. Also, you can run all related tests in one command.
If you have additional recipes or questions, feel free to reach out to me on Mastodon!
Follow up resources
- Pennylane Engineering: Scaling our Ruby on RailsMonolith using Packwerk
- rubyatscale/packs-rails
- engineering.gusto: How To Guide to Ruby Packs Gustos Gem Ecosys
If you’ve got some packs in place, feel free to check out packwerk by Shopify, which is a kind of linting tool, that checks if your packs respect the defined dependencies and privacy rules. We don’t use it yet, but it might be helpful if you have a larger team and/or many packs.