Article: The challenge of evaluating EPM software

It happens to me all the time.  The mail arrives and in it is an RFP.  They’re three dreaded letters around my office and I’ll tell you why.  The “Request for Proposal” is a construct of purchasing departments designed to compare vendors with similar products to sell.  The RFP is supposed to list in exacting detail exactly what kind of proposal the client requires and the respondents are expected to prepare their proposal and give the best possible price to the client.  The client evaluates all the proposals together and then knows who can give the best possible solution to them.

It works great when the client is looking to buy army boots or kitchen knives or storage containers.  It’s virtually useless when used to purchase an enterprise system and here’s why:

First of all, let’s take a look at how we assemble the RFP.  The beginning and end of the document are boilerplate.  They list the process of how to respond and the legal terms of what responding may mean and what you must or must not agree to.  In fact, most of this is not that useful, since if you are the winning bidder, you’ll need to go a whole new round of terms negotiations with the contracts people but either way, this is not our big concern.  No, it’s the bit in the middle we want to talk about.

In the middle we’re supposed to have a list of those requirements in ‘exacting detail’.  The client must assemble the various parties involved to see what they would like in our new enterprise system.  Perhaps a committee is convened or a series of focused workshops to work through all the implications of each requirement.  Perhaps there is a long period devoted to developing the business requirements that will resolve to system features.

 If only…

In fact, what’s more likely is the project manager assigned to this project will have a short meeting or perhaps two during which he or she will make a request that the participants represent their departments by listing all the features that are important to them.  Each person goes back to their group and sends an email saying “This is your only chance to have any impact on what we’ll need in the new enterprise system.  Speak up now or forever hold your peace.”  The departments respond in an uneven fashion.  Some department members give the request serious consideration and submit a long list of things.  Others give only a perfunctory response.

The project manager now assembles these items into a spreadsheet and adds a few columns: “Priority”, “Comment”, “Included”.  The end result looks like this:

Instructions: For each Priority enter: M: Must have I: Important to have but not essential N: Nice to have For each Included enter: Y: Yes, this is included in your product N: No, this is not included in your product E: This could be included by adding additional effort which has been identified, scoped and priced into your proposal
Feature	Priority (M/I/N)	Included (Y/N/E)	Comment

Gosh, it looks good so far, doesn’t it?  The spreadsheet fills out quickly.  For the more organized clients, on the internal copy (the one not sent to the RFP respondents) is a column called “Weighted”.  This is where the relevant score weighting for each item is entered.

 The project manager now merges line after line, categorizing them perhaps to make sure there are no big feature holes.  In the end, what is received is a remarkable list of features that looks just like every EPM system brochure on the market today.

Now, the vendors get this list and scratch their heads.  God knows, I’ve done it plenty.  They look at the list of features listed and try to make sense of why some features are listed 2, 3 even 4 times but each worded a little differently.  They try to reconcile some features which seem diametrically opposed to each other.  They ask questions for features which seem deliberately vague like “The system must automatically transfer data with our internal LIMS environment”.  Invariably the acronym LIMS is not explained and the details of what the expectation may be are not available.

But there is a profound problem with this process.  Have you spotted it yet?  The RFP is describing the solution not the problem.  It’s a massive disconnect and no one ever talks about it.  The purchaser or the project manager assembles a list of every feature all his or her users can come up with and presents that list to the vendors but what is not at all clear is whether the delivery of those features will make the lives of anyone at the client company any better.

So, what happens?  Vendors go through an exercise of preparing a hundred-page or sometimes a multi-hundred page response.  The client receives several of these.   The selection committee reviews them all.  They all say the same thing: “We can deliver every feature you’ve described.”  Sometimes vendors tick off every line item with a “yes”, even the ones that don’t make sense or are in complete conflict with other line items.

The hapless readers must wade through these documents trying to determine which vendor makes sense.  The weighting scheme tells them what features are more or less important but the scores after everything is done are very close.  The biggest impact on scores?  The neatness of the work.  I know, it sounds childish but ask around and see if it’s not true.  The vendor with the most colourful, neatest, easiest to read response always scores well.

Now a short list is assembled and a couple of vendors are invited to do demonstrations.  Typically the meetings are a couple of hours or less.  In the most frustrating cases, the client asks that the demonstration look exactly like what their system will look like after it’s purchased, configured, populated with data, customized and enhanced.  This obviously can’t be presented.  But, the meeting is important because it’s the first time the vendor gets to meet the end users and ask “What on earth is it you need to accomplish?  What are you suffering from?  What is the problem?”

In the end, purchasing decisions are often made one level above the selection committee anyway.  An executive makes that decision not based on a requirement analysis but on the recommendation of a trusted advisor, or a friend or colleague who once bought a similar system.  Will it work?  Sometimes and sometimes not but it’s a certainty the system purchased was not done based on what the company is struggling with.

“But that’s the best purchasing process we can come up with,” I’m sometimes told.

I truly doubt it. 

If you’re considering moving to a new EPM system in the New Year, I’ve got a couple of recommendations for your purchasing process.

First, start with the problem, not a description of what you think the solution should be.  List the business challenges that you are finding it difficult to overcome.  They can be articulated in terms of business decisions that you find difficult to make or business process inefficiencies which you wish to improve.  Saying “the system must help us decide to increase or decrease our future resource commitments with sub-contractors but delivering resource capacity planning such that we can see a projection for the next 90 days of resource requirements and resource capacity.” is incredibly more powerful than saying “The system must have resource capacity planning reports.”

A vendor who understands the business challenges will feel more latitude to craft a response which may be more creative and, even better, in ways that you’ve never imagined.

Next, put less attention on the vendor demonstrations and more on vendor references.  Probably the most valuable thing you can do during this process is arrange to visit the sites of other clients of the vendor.  A vendor salesperson demonstration will always look sharp but a real client will almost always be honest enough to explain the parts of the deployment that went great and those which were more challenging.

Finally, make sure that management gets on board with this process early and stays involved.  There’s little more debilitating than having a selection group work at something like this for months only to find out that none of their work will be considered when someone from senior management makes the final decision.

Happy buying!