Drupalcamp Pakistan: Automating Drupal Deployment

The goal of automating deployment is to make introducing new features easier. In this post we will learn how to set up a workflow that will move code and configuration of your drupal site from your local development station, to development, to staging and to production all by a using your version control and a push of a button.

Why Automatic deployment

• Its faster. Deploying automatically or semi automatically is a lot faster. You dont need to transfer the database settings manualy. You dont need to transfer the code manualy. You dont have to set the settings correctly on production or staging.
• Its less error prone. A process that is automated and executed every time is less likely to contain errors.
• Its well documented. An automated process almost always involves some kind of script being executed. This script is in version control. You can see how it evolves and why is became like it is.
• Reproducable. The process can be reproduced. You know exactly what has happend to the site.
• History. You can have a complete history of all the builds.
• Continous integration. Automating deployment brings us also closer to CI. Being able to introduce changes automatically and test them automatically reduces regression and makes you sleep better at night.

Our basic components

• A virtual private server with a lamp stack.
• Version control Git.
• Drush
• Drupal of course. With a correct repository layout.
• Dev Staging Production setup.
• Continous integration server
• Deployment scripts

How to set it all up?
TIP:
If you want to test it all out on a free cloud server. Learn how to create a free cloud server to play with and try it all out. Super fun and great for proof of concepts.

Lamp stack
In this post you can read how to configure a vps http://www.dominiquedecooman.com/blog/automate-installing-drupal-ubuntu-...

Drupal project
We will use this repository layout to structure our project:
- bash
-- updates - update scripts (see further)
-- installs - install scripts
-- scripts - random scripts needed in the project
- docroot - contains the drupal site
- documentation - contains documentation
- etc
-- drupal - contains settings files, robot.txt, ...
-- ssh - contains the config settings (see next)
-- aliasses - contains the alias files (see further)
-- vhost - contains a vhost template (see next)

Vhost file

<VirtualHost *:80>
  ServerName ddcdemo.local
  ServerAlias *.ddcdemo.local
  DocumentRoot /home/quickstart/websites/ddcdemo.local/docroot
  <Directory /home/quickstart/websites/ddcdemo.local/docroot>
    Options Indexes FollowSymLinks MultiViews
    AllowOverride All
    Order allow,deny
    allow from all
  </Directory>
</VirtualHost>

ssh config file

Host hera
HostName 92.243.15.236
User admin

Dev - staging - production
On your server create three folders where jenkins will able to build into. For example:

• /var/www/ddcdemo.dev
• /var/www/ddcdemo.staging
• /var/www/ddcdemo.production

Create vhosts on the server connecting to these folder so you sites are accessible. We use dummy urls but mostly it will be real ones. (We put and entry in our /etc/hosts locally to be able to see the sites on ddcdemo.dev, ddcdemo.staging, ddcdemo.prod)

Version control and Git flow
Read all about using git flow here: http://dominiquedecooman.com/blog/git-flow-minimizing-overhead
We will use the braching model as a basis of our workflow.

Drush
Drush is the drupal shell. If you are not yet using it definitly check it out. http://drupal.org/project/drush

We will use it to execute our commands to build our site. We will also use aliases. http://drupal.org/node/670460

Alias template: To be placed in the "~/.drush" folder

// environment dev
$aliases['ddcdemo.dev'] = array(
  'root' => '/var/www/ddcdemo.dev/docroot',
  'remote-host' => '92.243.15.236',
  'remote-user' => 'admin',
);
 
// environment test
$aliases['ddcdemo.test'] = array(
  'root' => '/var/www/ddcdemo.test/docroot',
  'remote-host' => '92.243.15.236',
  'remote-user' => 'admin',
);
 
// environment prod
$aliases['ddcdemo.prod'] = array(
  'root' => '/var/www/ddcdemo.prod/docroot',
  'remote-host' => '92.243.15.236',
  'remote-user' => 'admin',
);
 
// Local instance
$aliases['ddcdemo.local'] = array(
  'root' => '/home/quickstart/websites/ddcdemo.local/docroot',
  'uri' => 'ddc.local',
);

CI server
Install -> https://wiki.jenkins-ci.org/display/JENKINS/Installing+Jenkins+on+Ubuntu
Why?
We use jenkins to automate our workflow. Jenkins allows you to configure a jobs that execute commands to build the site and to introduce our new code and configuration. It will then execute tests and return feedback. It also provides a history of all builds.

Automate the workflow
Exporting database changes
To use continous integration with drupal and to work in team in general you cannot make database changes on the environments. This will not only be very error prone, you might forget something when deploying your feature, it is uncontrolable.

The features module allows us to export drupal configuration (code) into files. for example it will make a representation of a view and export that into a file. This file can be committed into the repository and deployed in a controled way. (http://drupal.org/project/features)

Features allows you to export most components. Here is an overview on how to structure them and some ideas on using install profiles in the deployment process: http://www.slideshare.net/nuvoleweb/code-driven-development-using-featur...

What you cant deploy with features should be done with hook_update(). You can code the database change in this hook and the drush updb command in the deploy script will execute all hook updates and you change will be deployed.

//Hook update example

Configuring jenkins jobs
First we will create a job per environment. Then we will create one or more testbot jobs. This way we will be able to deploy our new code to each environment. Jenkins will always connect to a repository and do a checkout of the code in a workspace. We will then sync these workspace directories with our folders

• Basic setup
• General settings
• Repository configuration
• Build triggers
• Post build actions
• Build section
• Restore the latest production database
• This site is building
• Copy files
• Update the code
• Copy build script
• Overrides
• Execute deploy script on test database
• Executed automated tests and store results
• Execute deploy script on the real database
• Unset the build message

//Testbot job - development job
#!/bin/sh
 
cd ../../docroot
 
#D7
 
#Update site
drush updb -y
 
#Enable ui modules
drush en field_ui update schema devel feeds_ui views_ui switcheroo dblog glue -y
 
#Feature reverts
drush fra -y
 
# Clear caches
drush cc all
 
#Reindex content Optional
#drush search-api-clear apachesolr_index
#drush sapi-aq
#drush queue-run-concurrent search_api_indexing_queue 8
 
# Run cron
drush cron
 
#Set dev settings
drush vset --yes --always-set preprocess_css 0
drush vset --yes --always-set preprocess_js 0
drush vset --yes --always-set cache 0
drush vset --yes --always-set block_cache 0
drush vset --yes --always-set page_compression 0
drush vset --yes --always-set error_level 2
 
#Run tests
drush en simpletest -y
drush test-run DDCDEMO

//Staging job
#!/bin/sh
 
cd ../../docroot
 
#D7
 
#Update site
drush updb -y
 
#Enable ui modules
drush dis field_ui update schema devel feeds_ui views_ui switcheroo dblog -y
 
#Feature reverts
drush fra -y
 
# Clear caches
drush cc all
 
#Reindex content Optional
#drush search-api-clear apachesolr_index
#drush sapi-aq
#drush queue-run-concurrent search_api_indexing_queue 8
 
# Run cron
drush cron
 
#Set dev settings
drush vset --yes --always-set preprocess_css 1
drush vset --yes --always-set preprocess_js 1
drush vset --yes --always-set cache 1
drush vset --yes --always-set block_cache 1
drush vset --yes --always-set page_compression 1
drush vset --yes --always-set error_level 0
 
#Run tests
drush en simpletest -y
drush test-run DDCDEMO

//Production job
#!/bin/sh
 
cd ../../docroot
 
#D7
 
#Update site
drush updb -y
 
#Enable ui modules
drush dis field_ui update schema devel feeds_ui views_ui switcheroo dblog -y
 
#Feature reverts
drush fra -y
 
# Clear caches
drush cc all
 
#Reindex content Optional
#drush search-api-clear apachesolr_index
#drush sapi-aq
#drush queue-run-concurrent search_api_indexing_queue 8
 
# Run cron
drush cron
 
#Set dev settings
drush vset --yes --always-set preprocess_css 1
drush vset --yes --always-set preprocess_js 1
drush vset --yes --always-set cache 1
drush vset --yes --always-set block_cache 1
drush vset --yes --always-set page_compression 1
drush vset --yes --always-set error_level 0

Writing deployment script

• Enable/Disable ui modules
• Set environment specific variables
• Error_level
• Set Cache
• Revert features
• Update the database
• Optional: reindex
• Run cron
• Flush cache
• Running tests

The is a dedicated blog post on what to use in the script http://www.dominiquedecooman.com/blog/drupal-7-tip-how-automate-and-cont... The post overlaps a bit with this one but it focusses more on what to automate than how to automate.

Differences per environment
Each environment as you can see in the script a differences. This way you can control what you are doing on each environment.

Reroll
To reroll something simply use git to rollback any changes and recommit a hotfix and deploy. Due to the automation you will be able to do it fast and easy.

You should be able to do this now
Develop on feature branch locally

git flow feature start news
drush dl views
drush en views -y
drush dl features
drush dl ftools
drush en features ftools -y
//Create a feature called ddcdemo_news add the article content type
git add *
git commit -m "initial setup"

Now we have to share the feature with our team mates and have it tested in our CI. We have to push the code so we can check it out in our testbot job to test our feature.

git flow feature publish news

You can now use the jenkins job to deploy the code of the feature branch. Type in the branch field "origin/feature/news" Fix tests if needed.

To stop developing on the feature

git flow feature finish news
git push origin develop

Now git flow has merged all our work into the develop branch. After we push it to the origin. Our CI system should now be able to detect the change and automatically execute our develop build. The code that was merged into develop will now be deployed on our development environment and tested.

Note: You can also run tests locally but simpletest is pretty slow so practically it is better to let the server handle it. There are some optimisation that can be done but they are beyond the scope of this post.

If any tests are failing we should fix them.

Release
Now we need to create a release to deploy on our staging environment so our client is able to test it.

git flow release start v0.1.2

When we are done adding to our release branch we can push it remote and use it as a branch to deploy on staging.

git flow release publish v0.1.2

Now in Jenkins we can fill in our branch name in our staging job and get that tested. If all goes well we can finish our release branch and it will be merged in both the master and the develop branch

git flow release finish v0.1.2
git push origin master

We can keep track of the version by storing this in a text file in the root of the project.

Now we should be ready to deploy the master branch on our production environment. Lets execute our production job. Note that no tests are executed. We did all the testing on our other environments. We are using the contributed simpletest module and the remotetestcase which allows us execute tests directly onto the database so we dont need to do complex setups. But a disadvantages is that it pollutes the database with test content.

Hotfix
Create hotfix branch - v0.1.2.1

git flow hotfix start v0.1.2.1

Finish the hotfix branch. This will merge the change into develop and the master. Push the master to the central repository. Push the production job and the fix is on production.

git flow hotfix finish v0.1.2.1
git push origin master

For more on gitflow read this: http://yakiloo.com/getting-started-git-flow/

Conclusion
What we have now is a powerful way of deploying code in a controlled way. When developing proper tests for each feature we will be able to minimize regression and we are able to release automatically in a controlled way.