A Drupal Enterprise Quality Experience

This post will summarize the experience of introducing drupal into an enterprise. How we achieved a level of quality in drupal development needed for an enterprise application.

It will cover:

• Setup : The drupal 6 site and the integration with the other apps in the enterprise.
• Level of quality
• Testing : How we implemented tests using simpletest and selenium.
• Staging : How we used features, hook_update (update_api) and deploy to stage our application in a dev/staging/prod release cycle and the problems we encountered.
• Automation : How we used ant, shell scripts and drush to:
  • Automate post-install for staging.
  • Automate releasing versions and testing.
  • Automate initialize the project.
• Development Management : How we managed to estimate and not to lose track of anything managing this complex project.
• Garding Performance : How we kept an eye on performance.
• Security and maintenance/
• Training Developers : How we combined developing and training developers at the same time. This resulting in drupal being adopting in the organisation.
• Conclusion

Setup

The extranet we are building is a window on all internal services. The goal is to aggregate all that data for these external services and let users/clients interact with it using drupal.
The elements:

• We have the web/php machine which harbours the drupal installation.
• A set of webservices to query an internal server, which contains all sorts of info about the organisation, clients, products, ... Goal is that clients can review there hosted products.
• We have a service that holds all ticketing data that is queried by a webservice to retrieve tickets. Goal is that clients can view tickets concerning them.
• There is a shell script that retrieves text and images from a an internal wiki to display in the drupal front end.
• All authentication to drupal and all the other services is done by an SSO (shibboleth) which is in his turn connected to an LDAP containing all info about clients. All is https because pages contain sensitive information about the clients hosted the services.
• Roles and permissions are assiged by custom code based on external rules retrieved from the LDAP.
• Some other internal services which are really old (1999 old) the organisation has that generated html where exposed using iframes. To construct the url we needed retrieve elements using webservices.

This is the version 1.0, integration with other services like the intranet, business logic modules, ... are still being planned.

As you can see drupal would receive a central place in the enterprise integrating tightly with all existing apps. Some newer apps were designed with a webservice model in mind which allow other apps to connect easly with it, which is great. On the other hand some legacy apps werent that friendly and required a less sexy solutions.

This showed immediatly an advantage drupal could have in the entreprise. Imagine for example that drupal was used to create the wiki? The services module could be used to let the main drupal communicate with the wiki. An easy and clean implementation. It would also mean your teams would only need to know drupal and not 10 million other apps. A lesson was learned and the clients intranet that was sheduled for review will now be ported to a drupal commons.

Level of qualtiy

All components in the chain are staged from local development -> integration -> preproduction -> production.

We worked in a scrum system with each week a demo.

Before a story could move to done and be deployed it had to respond to a "fini fini" which means it has to respond to a list a checks to assure it is 100% done.

• Are permssions configured?
• Is all transalated to French?
• Is all work documented, especially patches? (People change positions all the time in large organisations so good docs are essential)
• Is all work tested using simple test and selenium?
• Are all automation scripts up to date? (post-install, initialisation, test and release scripts)
  Are site conventions met? (for example expose new admin links to a custom admin menu)
• Is performance acceptable? (profile with xhprof)
• Is security acceptable? (external team)
• Are all configurations and translations automaticaly exportable to the other environments?
• Can you demo the story to the client on preproduction?

Clearly for most organisations this level of quality is simple unaffordable but for an enterprise app this standard quality.

Testing

Simple test
Simpletest is the standard way to test functionality on a drupal site. Here you can find a very good article that explains what simpletest does [http://www.comm-press.de/en/blog/simple-test-drupal].

We used the shib_auth module to wire drupals authentication system to the session provided by shibboleth. This means also that almost all standard simple tests for drupal fail because of this authentication. We decided to ignore tests provided by standard drupal module and we focused on our own tests.

We decided to use Simpletest to do all of the standard unit testing. Here is an execellent book that explains the principles of unit testing [http://www.amazon.com/Pragmatic-Unit-Testing-Java-JUnit/dp/0974514012]. Atomic tests testing our apis and webservices. The client had already a broad experience with unit testing. The team was used to really code test driven.

We immediatly had some trouble with simpletest. Because simpletest installs a drupal for each and every test it is very slow. Even with the optimisations proposed here [http://drupal.org/node/466972] it is still slow. To slow to continuously run tests. I found an article on drupal.org [http://drupal.org/node/758662] and apparantly in drupal 7 it is possible to do unit testing with simple test without sandboxing the whole drupal site on every test. It is possible to attack an instance and start running tests. This would obviously go a lot faster but you would have to take into account that you need to write tear down functions to clean up your database. Unfornatly this feature is not available on drupal 6 and we did not have the time to backport it. So we decided to run test only after completion of the functionality. Not very Test driven development. But we still had our code tested at the end.

We wrote tests for every functionality trying to keep the tests as short as possible and we would create a test file for each module we created. We also worked with ant, mainly because the client had a lot of experience with it and because they knew how to integrated the output of ant to a Hudson server. Simpletest comes with a file runTests.sh which can be launched by ant. We implemented that so we could launch all test, a test by group and a specific test class. Here is the script file. The script combines detecting, executing and cleaning. Ofcourse we used drush were possible.
Now we had only hudson to set up. On every commit the code base on hudson is updated and tests are launched.

Continous integration with drupal is possible. Note that in drupal 7 the core contains the test framework so you dont have to patch the core.

Selenium
We choose selenium to do our functional tests. Selenium is a great tool with requires some expertise to effectively use. Luckly the client had expertise using it. We used the PEAR phpUnit to code tests for selenium. Here is an example of a test:

It is pretty easy to code tests, you can even click your tests together using the firefox selenium plugin and export them for php.

With ant we could launch the selenium-server.jar which launches our tests.

But had some obstacles to cross.

• We needed to make selenium work with https. In firefox you ll need to export certificates, on ie you can set a variable.
• We needed to make it work on ie. Install phpunit on windows caused some pain but we managed to do it.
• We had to pass shibboleth each time.

All obstacles were managed.
Note that testing with selenium is very expensive. It is a pretty slow process to execute on multiple platforms/browsers. Ie needed sometimes a different approach to test stuff which took even more time. But you are practically sure your app will do what you want. To give an idea it took 20% of dev time per story to write and complete selenium testing .

Staging

We made all configuration portable using a method which basicly consists of features, deploy and hook update. see staging problem:http://dominiquedecooman.com/blog/drupal-staging-problem

New to this is that we also contributed a patch with our functions to update_api module http://drupal.org/project/update_api so we had a toolkit to use in hook update.
Here is a piece of code from our install file where our hook updates reside.

Here we also thouched a painpoint drupal has, namely some poor core apis. I think everyone is aware of this problem, Dries has set it as his first priority for drupal 8 [http://buytaert.net/8-steps-for-drupal-8 Step1 : Create better separation between framework and product]. So I guess the problem is known. Since we had to code every change we made using the UI we encountered a lot missing apis.

For example try to change a theme and configure its vars. Or use the submit function to delete a role, ... I m not saying it is not possible but the problem is that you realy have to dig in the core functions, understand all the actions and assemble the needed ones to do what needs to be done. If you have a lot of expertise and you know your way around the core code you ll manage but this means everyone who wants a stageable site, and allmost any enterprise client needs this, need to invest a lot of time in training people until they reach this level of expertise.

Better apis would make drupal easier to understand and thus the investment of training would decline and therefore drupal would be better adopted.
Better core apis will make drupal more usable and integratable from a developers point of view.

Automation

Automate post-install for staging
The organisation has a deployment tool which basicaly takes a tarball and deploys it on the server. After it is been deployed a post install script is run. Together with drush we managed to set up the necessairy to deploy drupal. (Note that to make drush render its output correctly we made some adaptations. )

Automate releasing versions and testing.
Before you could create a release we made it mandatory that all tests were pasted. After this happend an svn version got tagged and a tarball created.

Automate initialize the project.
Since we have a lot of tests and integrations with other apps like LDAP, internal config, wiki, ... We created an init file that would fill our drupal and external apps with data for a (users, content, ) test game. The result is an initialisation script that automates this task.

All the scripts are ran from terminal using "ant".

The use of bash scripts and drush made our live much easier. It takes an investment to create them but they are a huge timesaver in the long run.

For example when a new developer needs to set up his workstation he checks out a working copy, he runs the init script and he is ready to go. He codes. He runs his tests. He creates his release, svn is tagged. Deploys his tarball on the server and the post install script will be called executing the hook updates and deploying config stored with feature module. Imagine doing all this by hand, all the time?

Development management

To estimate completion time of a complex project like this you have to look beyond the features that will appear on the screen. This is not allways easy to explain. As a client/end user you dont see the tests, the scripts, the staging, the docs, the performance tweaks, the security updates, the platform installation, ... (hopefully) you only see an app that will work all the time. This will be thanks to all the work done behind the scenes.

To estimate each story we used roughly this key to estimate the time passed. Note that all work was done by pair programming. (hours are based on 40h)

• 35% development of the feature. (14h)
• 30% writing and running tests + refactoring code accordingly (12h)
• 10% deployment (exporting features, writing hook updates, deploying) (4h)
• 5% scripts (2h)
• 5% doc (2h)
• 5% demo (2h)
• 10% translation, perf, security, permissions, conventions. (depening on nature of functionality) (4h)

To manage this project we had to cooperate with a lot of people in the organisation. This is not allways time that shows up at the end of the week. For example to do a webservice you need to call some people in Paris to give you the right urls. This could quickly add up to some hours.

How did we not loose track of anything? We chose to work scrum. The scrum process protected us from being swamped in requests to introduce this and that while staying agile at the same time. It allowed us to estimate correctly and develop at maximum productivity creating functionality everyone agreed upon.

Garding Performance

We were only responsable for the performance on the application level. All other performance tweaks at server level where performed by the operation teams. For example integrating memcached in drupal is our task, installing the daemon on the server not.

To keep an eye on performance we used the profiller xhprof which can be used with devel. Installation is easy (http://techportal.ibuildings.com/2009/12/01/profiling-with-xhprof/)

Using this tools we monitored each page to see if we had any performance problems. For example we detected that a request to webservice was very slow. With xhprof it shows immediatly, you can compare the actual cpu time used versus the total load time. This showed us that our app was in fact waiting for the other server to respond. We alerted the other team, they found there was a problem with one of there sql servers and fixed it. Xhprof is defenitly a time saver.

Working with a profiller shows you how the app is structured, how the framework works, what is exectued, ... Its something you can learn a lot from. Xhprof togetter with common sense will make you think about performance all the time.

Security and maintenance

Security was a big deal in this project since it contains sensitve data about clients and the organisation.
An audit will be performed by an external entity.
We were only responsible on the drupal application level for security. On the server side we had https, sso, authentication proxy, ... but this was not our task.

We always checked as part of our "fini fini" that all form, url, ... input was sanitized so no sql or xss attacks could occur.

We always followed new security releases for modules and core. Modules that were patched were moved to a seperate folder. Patches were documented so upon core/module relase one could examine if there are patches and if it is still necessairy to apply them again.

I m not a security expert but double checking input and output and following updates is a good basic strategy to prevent evil from happening. Audits by security experts are a definitly necessairy if you work with critical data.

Training developers

This was probably the biggest challenge. How to train a team of already experienced php developers to use drupal and deliver an app that lives up to the quality standards. How to transfer knowledge effectively? How to make the team work on its own?

The method we used was probably not the fastest but I think in the long run the most effective. Instead of me developing stuff we immediatly started to let the team members develop.

Since we were trying to get a product done at the same time we did not take the approach: a node is blabla, a block is blabla, ... And I m glad we didnt because it would bore the hell out of everyone and after a month we wouldnt even had coded a module.

We posed the question what do we want to do? For example we want to retrieve data x from source y with a webservice and display it to user z with a role and permissions under conditions a,b,c. How do you do this in drupal? To answer this question you touch a lot of concepts, modules and apis.

So I architected the solution and then I explained every concept needed to complete the functionality. From blocks to modules, hooks, apis, ... Everytime we encounter a concept I would explain it in detail and how it fits as a piece of the puzzle. Ofcourse the real power of learning lies in repetition and exercise. So forexample the theming system was explained probably three times. The entry point being, we want to render some output using some html, how do we do that?

The first few weeks they were still a bit in the dark but as time progressed they saw how drupal was used: Creating functionality using existing code and integrating it with each other using configuration and custom code. Soon the awesome power of drupal became obvious to them, they learned to search drupal.org when in trouble, we posted issues, we even contributed a little module (http://drupal.org/project/jquery_jstree).

They could see that this vast community was creating and improving code. Almost anything we wanted to do was provided by a module and ways to interact with it. It was clear that with understanding of six concepts one can do almost anything with drupal. Upon these six points we would regularly mesure ourselves, we identified:

• Framework
• Core + APIs
• Contrib Modules + APIs
• Managing code, confifuration and content (features,deploy,hook_update,drush & use of scripts)
• Combining existing modules
• Best practices
• Data
• Internal data
• database: data types (node, taxonomy, ...)
• session
• External data : Webservices
• Interaction with users
• Users
• Authentication
• Access models
• Interaction with data
• Quality
• Performance
• Tests
• Security
• UI
• Information architecture by combining modules and using their apis to integrate. (views, cck, panels, block, taxonomy, contexts...)
• Jquery
• Templating
• Problem solving
• How to trace bugs
• Tools (devel, logs, debugger, xhprof, ...)
• Use Drupal.org, ask help from community

The reason why drupal has such a steep learning curve is because it you need to know a lot of little things that are individualy not that difficult to understand but the problem is it are a lot of those little things. Especialy when you want to create an enterprise level app.
On this level getting up and running requires a lot of knowledge about a lot of concepts. For instance: How do you stage config and content? How do you create a test? ...

When solved these questions you come to creating your functionality. You get introduced to hundreds of concepts, blocks, nodes, taxonomy, views, cck, fields, formatters, feature, rules, ...
To create a functionality you need to know which modules to use and how to configure them.
When you achieved your first functionality using the existing concepts and modules you want to extend the thing using hooks, apis, in your own modules.

To effectively use drupal you need to combine stuff. To be able to do that you need to know that stuff exists. This make the learning curve steep.

However if you get to this level you notice anything is possible with drupal.
In fact the developers were so enthousiastic that they started wondering why are we still building stuff from zero? Drupal gives us the head start we need. We were doing the same thing over and over again. Why are we maintaining ten differnet apps? Now we can trust the framework and focus on developing the fun stuff instead of recreating the framework over and over again. They even started proposing to start the new projects in drupal to their chiefs.

Altough drupal isnt the magic solutions. It is not click click click done. You still need to get work done. But what it offers not only a framework but an entire universe.

EDIT: I ve been working on this post for a while and when I saw Dries keynote and he spoke about the fact that the best ecosystem will win. I would certainly think so too. This experience made clear to me to it is that what a client wants. One ecosystem that everyone loves working with.

Conclusion

For me it was a quality experience, I learned a lot and I can only say drupal is ready to deliver enterprise quality.

EDIT : After watching Dries keynote this morning. Even though Drupal 6 does already a good job and Drupal 7 will do even better. If the community can achieve the points mentioned by Dries like staging config and content, serving data in multiple formats and improving the apis then Drupal 8 will truely be THE killer enterprise content app.

EDIT:Scripts have been removed