August 03, 2008
Our Git deployment workflow
At weplay, we recently switched to pure git version control from git-svn.
Now that we've had a couple weeks for the dust to settle, I'd like to
share our workflow for managing deployments to our staging and production
clusters.
We started by outlining the goals of our system:
All code that's pushed to our staging and production environments must be in GitHub. Nothing goes straight from a local repository to our servers.
Any developer can deploy our most recent work to staging.
Any developer can deploy the code on staging into production. We (try to) avoid deploying anything to production that hasn't been pushed to staging first.
Any developer can see a diff between "What is deployed" and "What I'm about to deploy."
Any developer can branch from the production codebase for time sensitive tweaks and fixes. These need to be staged before they're deployed to production too.
That functionality is available in an automated, safe, easy to use form. (Hint: rake)
We ruled out having "production" and "staging" tags because updating tags across a tree of remotes doesn't seem to work smoothly. We also ruled out doing git merges from master into our staging branch. Our staging server/codebase jumps around from the latest changes to the production code (up to a week old) based on what we need to test, so we really just want to replace it.
We settled on using remote git branches for production and staging on origin and hard resetting them to other branches to simulate a copy. We treat these branches more like tags and never commit to them directly. This has worked great so far.
Here's what our rake tasks look like:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 |
class GitCommands def diff_staging `git fetch` puts `git diff origin/production origin/staging` end def tag_staging(branch_name) verify_working_directory_clean `git fetch` `git branch -f staging origin/staging` `git checkout staging` `git reset --hard origin/#{branch_name}` `git push -f origin staging` `git checkout master` `git branch -D staging` end def tag_production verify_working_directory_clean `git fetch` `git branch -f production origin/production` `git checkout production` `git reset --hard origin/staging` `git push -f origin production` `git checkout master` `git branch -D production` end def branch_production(branch_name) verify_working_directory_clean `git fetch` `git branch -f production origin/production` `git checkout production` `git branch #{branch_name}` `git checkout #{branch_name}` `git push origin #{branch_name}` end protected def verify_working_directory_clean return if `git status` =~ /working directory clean/ raise "Must have clean working directory" end end namespace :tag do desc <<-DESC Update the staging branch to prepare for a staging deploy. Defaults to master. Optionally specify a BRANCH=name DESC task :staging do branch_name = ENV['BRANCH'] || "master" GitCommands.new.tag_staging(branch_name) end desc "Update the remove production branch to prepare for a release" task :production => ['diff:staging'] do GitCommands.new.tag_production end end namespace :diff do desc "Show the differences between the staging branch and the production branch" task :staging do GitCommands.new.diff_staging end end namespace :branch do desc "Branch from production for tweaks or bug fixs. Specify BRANCH=name" task :production do branch_name = ENV['BRANCH'] raise "You must specify a branch name using BRANCH=name" unless branch_name GitCommands.new.branch_production end end namespace :deploy do desc "Tag and deploy staging" task :staging => "tag:staging" do `cap staging deploy:long` end end |
The last one (rake deploy:staging) simply wraps up the common task of tagging
our latest code to be pushed to staging and initiating a staging deploy.
Note: To use most of these commands, your local working directory must be clean. If we have outstanding changes in our tree when we need to run them, we use git-stash to temporarily move them out of the way.
Thanks to Scott Chacon for helping us work this out. Be sure to check out his GitCasts and Git Internals PDF.
August 06, 2008 at 1:05 AM
Thanks for sharing. I’ve put up your rake tasks as a Gist: http://gist.github.com/4169
August 06, 2008 at 7:02 PM
Nice.
tag\_producton -> tag\_production
August 06, 2008 at 8:48 PM
I’ve simplified tag_staging, tag_production, branch_production – check out gist 4294.
BUT I really really really think this is wrong. (not the ruby code, but the methodology).
You are loosing the best parts of Git by continually rewriting history. Any time you see a push -f or reset in the normal flow you are miss-using git.
Anyone can stage a branch that isn’t a descendant of origin/production then rake tag:production. All of a sudden all that history is gone!
August 10, 2008 at 5:48 PM
Henrik—Fixed thanks.
Jesse—That was my original reaction as well when we’re setting this up, and a few other people have shared similar comments. Functionally, we’re leveraging git’s branches as if they were tags that we can replace at will across our remotes. Because we never commit to the staging and production branches directly (which our rake tasks help enforce), we never lose history.
We’re interested in alternatives, and one of the reasons I posted this is to see if people had suggestions on how we could do better.
Looking at the docs for git-merge, I wonder if instead of a reset—hard we could do a merge using the “ours” strategy. That might let us workaround the problem with a vanilla merge, where we just want to replace instead and not deal with conflicts.
There’s also the possibility that it can be better done with tags, but from my brief experiments that seems to be problematic. We’d also have to force push tag updates, which I’m not sure is any better.
Thoughts?
August 11, 2008 at 5:19 PM
thanks for the post, i’ve been attempting to standardize a similar git workflow for my team, and its interesting to see another’s implementation. so please continue posting as this matures. quick thought: change your GitCommands instance methods to class methods, and you can eliminate the ‘new’ keyword in your calls.
August 15, 2008 at 9:42 AM
Bryan – this is a great post, and looks very useful. We’re just starting to move some of our rails apps to git, and this will definitely be helpful. One comment on the script: you’re not checking the return values of any of the git commands. If one of them fails, (like git co xxx), then the rest of the commands (like git reset—hard) will still run. I’m not sure what kind of problems this could cause.
August 15, 2008 at 11:50 PM
Good point, Tammer. I’ll see about checking for success response codes.
August 21, 2008 at 11:40 AM
The whole procedure looks like you’re using git as if it was subversion. Hard resets, forced pushes, etc, built into everything.