air traffic controlAhh the pilot project. It’s a term that software sales people hate to hear but almost every major software implementation project works its way through this phase. Even I have recommended using pilot projects as a safe way of determining the suitability of software for a particular purpose.

The notion of a pilot project isn’t new. The idea is to implement the entire software solution being proposed on a minor scale with real data to see if it can meet the needs of the organization. Often, just like in drug testing, there is a ‘control group’ and a ‘test group’. These groups are best organized as similar sized departments with a similar workload. The test group will use the new system, the control group will continue with normal procedures. At the end of the pilot, the performance of both groups will be evaluated and the relative efficiency of the new system can be measured.

It’s a great plan. After all, the only change between the two groups will be the introduction of the new software so the effects should be easy to test. For people who have just finished their MBA and are now working in the middle-management ranks, this provides perhaps the only opportunity to determine a Return on Investment (ROI). This is particularly difficult to come up with otherwise with certain systems such as project management software where the hard-cost benefits are difficult to determine and the soft-cost benefits such as bringing in the project early are tough to prove in advance.

The final result of the project is the identification of the right software to implement and even some tips on how to implement it. All of this at little to no cost, right?

So, if it’s such a great plan, why aren’t more pilot projects successful? Often a pilot project is undertaken only to find out that the results are never tabulated or that the results are delivered but they result in no decision or that the project is terminated prior to completion.

There are a number of hidden challenges in a pilot project which must be dealt with if you are to have any chance of success.

First of all there is a fundamental underestimation that I find in almost 100% of pilot projects which I encounter. Often the software itself for such a project is offered for little or no cost and the assumption is then made that the cost of the pilot project is therefore zero. When we’re talking about an enterprise system, nothing could be further from the truth.

In order to see the results of any enterprise software system, it must not only be installed, it must be implemented. As anyone who has done such an implementation can tell you, the costs of installation and purchase of the software are often dwarfed by the size of the rest of the implementation budget. There is legacy data to be configured and uploaded, the configuration of the system to be determined, reports and entry forms to be created or adjusted, training, more training, external systems to be linked, practices and procedures to be created then adopted and, (did I mention?) more training.

This is so whether you are looking to try the system out on one week of data or one out of fifty projects or one out of 20 departments.

Once a system has been implemented, expanding its use to the rest of the company is an incremental cost which is the least costly and least effort of the project overall.

An implementation of an enterprise system of almost any kind for the first department or first project or first core element of an organization may be scheduled to take weeks, yet, so many organizations that I speak to talk about a pilot project like it will take a week of effort and will cost the organization nothing.

Without an understanding of how much effort such a pilot will really take, management can quickly become frustrated and your sponsorship of the project will be in jeopardy.

Even if there is adequate management support, there are a few other pitfalls to overcome. First of all, it is ongoingly stunning to me that almost no pilot projects I encounter actually have metrics, for how the test will be measured. I see this over and over and over again. When pressed to how the pilot team will know that the test was a success (or even if it’s complete) the general feedback seems to be “We’ll know.” As though some feeling will be imparted to the team when the system is good enough.

Also, for the test to be fair, there should be no other influencing factors that would mask the effectiveness of the system in question. That’s great but in today’s business environment, everything is changing all the time. And actually isolating a team to avoid such influence would be an unnatural influence by itself.

It’s also worthwhile to mention that the implementation of any significant change in the systems of a group almost always results in a short-term drop in performance as the personnel adjust to a new way of doing business and as the analysis or data from the new system is integrated into the group’s decision making process. Very short term implementations such as a pilot program may show negative results where a longer term perspective might show a better example of the potential benefits.

So, is it hopeless, are you now reduced to choosing software from the description on the side of box? It’s not quite that bad. For some types of software, it is easier to implement a pilot than others. If we take project scheduling systems, for example, it is relatively easy to have one department work with one tool while the rest of the organization works with another. You can even have implementations done by staff who have been previously familiar with different tools to show how they can be put to their best advantage.

For other types of software, however, pilot projects may not be your most effective evaluation method. You might use other tools such as reference site visits or the use of an independent consultant to help shape the overall implementation plan. This cost and effort would be recoverable regardless of which system is ultimately selected and is often very revealing to the internal cultural issues that will need to be overcome to take advantage of the enterprise system in question.

One way or the other, the most important thing to remember: pilot projects aren’t free. Make sure you’re getting a good return on investment on the effort and money that the pilot itself will cost.