Our Git deployment workflow

At weplay, we recently switched to pure git version control from git-svn. Now that we've had a couple weeks for the dust to settle, I'd like to share our workflow for managing deployments to our staging and production clusters.

We started by outlining the goals of our system:

  • All code that's pushed to our staging and production environments must be in GitHub. Nothing goes straight from a local repository to our servers.

  • Any developer can deploy our most recent work to staging.

  • Any developer can deploy the code on staging into production. We (try to) avoid deploying anything to production that hasn't been pushed to staging first.

  • Any developer can see a diff between "What is deployed" and "What I'm about to deploy."

  • Any developer can branch from the production codebase for time sensitive tweaks and fixes. These need to be staged before they're deployed to production too.

  • That functionality is available in an automated, safe, easy to use form. (Hint: rake)

We ruled out having "production" and "staging" tags because updating tags across a tree of remotes doesn't seem to work smoothly. We also ruled out doing git merges from master into our staging branch. Our staging server/codebase jumps around from the latest changes to the production code (up to a week old) based on what we need to test, so we really just want to replace it.

We settled on using remote git branches for production and staging on origin and hard resetting them to other branches to simulate a copy. We treat these branches more like tags and never commit to them directly. This has worked great so far.

Here's what our rake tasks look like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
class GitCommands

  def diff_staging
    `git fetch`
    puts `git diff origin/production origin/staging`
  end
  
  def tag_staging(branch_name)
    verify_working_directory_clean
    
    `git fetch`
    `git branch -f staging origin/staging`
    `git checkout staging`
    `git reset --hard origin/#{branch_name}`
    `git push -f origin staging`
    `git checkout master`
    `git branch -D staging`
  end
  
  def tag_production
    verify_working_directory_clean
    
    `git fetch`
    `git branch -f production origin/production`
    `git checkout production`
    `git reset --hard origin/staging`
    `git push -f origin production`
    `git checkout master`
    `git branch -D production`
  end

  def branch_production(branch_name)
    verify_working_directory_clean
    
    `git fetch`
    `git branch -f production origin/production`
    `git checkout production`
    `git branch #{branch_name}`
    `git checkout #{branch_name}`
    `git push origin #{branch_name}`
  end
  
protected

  def verify_working_directory_clean
    return if `git status` =~ /working directory clean/
    raise "Must have clean working directory"
  end
end


namespace :tag do
  desc <<-DESC
    Update the staging branch to prepare for a staging deploy.
    Defaults to master. Optionally specify a BRANCH=name
  DESC
  
  task :staging do
    branch_name = ENV['BRANCH'] || "master"
    GitCommands.new.tag_staging(branch_name)
  end

  desc "Update the remove production branch to prepare for a release"
  task :production => ['diff:staging'] do
    GitCommands.new.tag_production
  end
end

namespace :diff do
  desc "Show the differences between the staging branch and the production branch"
  task :staging do
    GitCommands.new.diff_staging
  end
end

namespace :branch do
  desc "Branch from production for tweaks or bug fixs. Specify BRANCH=name"
  task :production do
    branch_name = ENV['BRANCH']
    raise "You must specify a branch name using BRANCH=name" unless branch_name
    GitCommands.new.branch_production
  end
end

namespace :deploy do
  desc "Tag and deploy staging"
  task :staging => "tag:staging" do
    `cap staging deploy:long`
  end
end

The last one (rake deploy:staging) simply wraps up the common task of tagging our latest code to be pushed to staging and initiating a staging deploy.

Note: To use most of these commands, your local working directory must be clean. If we have outstanding changes in our tree when we need to run them, we use git-stash to temporarily move them out of the way.

Thanks to Scott Chacon for helping us work this out. Be sure to check out his GitCasts and Git Internals PDF.

8 Responses to “Our Git deployment workflow”

  1. # Chris Says:

    Thanks for sharing. I’ve put up your rake tasks as a Gist: http://gist.github.com/4169

  2. # Henrik N Says:

    Nice.

    tag\_producton -> tag\_production

  3. # Jesse Andrews Says:

    I’ve simplified tag_staging, tag_production, branch_production – check out gist 4294.

    BUT I really really really think this is wrong. (not the ruby code, but the methodology).

    You are loosing the best parts of Git by continually rewriting history. Any time you see a push -f or reset in the normal flow you are miss-using git.

    Anyone can stage a branch that isn’t a descendant of origin/production then rake tag:production. All of a sudden all that history is gone!

  4. # Bryan Helmkamp Says:

    Henrik—Fixed thanks.

    Jesse—That was my original reaction as well when we’re setting this up, and a few other people have shared similar comments. Functionally, we’re leveraging git’s branches as if they were tags that we can replace at will across our remotes. Because we never commit to the staging and production branches directly (which our rake tasks help enforce), we never lose history.

    We’re interested in alternatives, and one of the reasons I posted this is to see if people had suggestions on how we could do better.

    Looking at the docs for git-merge, I wonder if instead of a reset—hard we could do a merge using the “ours” strategy. That might let us workaround the problem with a vanilla merge, where we just want to replace instead and not deal with conflicts.

    There’s also the possibility that it can be better done with tags, but from my brief experiments that seems to be problematic. We’d also have to force push tag updates, which I’m not sure is any better.

    Thoughts?

  5. # Micah Geisel Says:

    thanks for the post, i’ve been attempting to standardize a similar git workflow for my team, and its interesting to see another’s implementation. so please continue posting as this matures. quick thought: change your GitCommands instance methods to class methods, and you can eliminate the ‘new’ keyword in your calls.

  6. # Tammer Saleh Says:

    Bryan – this is a great post, and looks very useful. We’re just starting to move some of our rails apps to git, and this will definitely be helpful. One comment on the script: you’re not checking the return values of any of the git commands. If one of them fails, (like git co xxx), then the rest of the commands (like git reset—hard) will still run. I’m not sure what kind of problems this could cause.

  7. # Bryan Helmkamp Says:

    Good point, Tammer. I’ll see about checking for success response codes.

  8. # Chris Hoeppner Says:

    The whole procedure looks like you’re using git as if it was subversion. Hard resets, forced pushes, etc, built into everything.

Leave a Reply

You may use Textile markup in your comments.