Executives often ask me for the impossible and what they want is best exemplified by an often used phrase, “A sip from the firehose.” In most organizations today there is no shortage of data and in the project and portfolio management realm in particular we have a plethora of systems and processes that can generate data but no manager can possibly sift through thousands or even millions of records in order to determine what decisions they should make.

The result is that most significant decisions about projects including the decision to start the project in the first place is made on an ad-hoc basis.

The technique to avoid this is well known. Organizations should choose “Key Performance Indicators” (KPIs )which will surface information that is critical to making those business decisions. But, how does one choose those KPIs? How do you know that the KPIs you’re using are producing the result you require? Let’s take a look.

Choosing KPIs

First of all, where do we look for choosing the right Key Performance Indicator and how will we measure it? These metrics will be used in a variety of ways. We can use these metrics for distinguishing one project from another. We can use them in the project selection process, the stage gate process, the resource priority allocation process and the project review process. There is no limit to the number of things we could measure.

One place to look it through the main elements of PMI’s PMBOK (Project Management Body of Knowledge). In each section you might have possible measure that would lead to better management of the project.

In the Scope area, for example, you might have scope creep or scope change as a measure. In the Time area you might look at delays in the schedule. In the Cost area you might look at resource costs or Return on Investment. In the Communications area you might look at the timeliness of schedule updates or the status of project reporting. In the HR area you might look at Resource Variance or Resource Load/Overload or Resource balance. In the Risk area you might look at the volume of risk items and their severity or urgency and so on and so on.

Not every metric will make sense in every context. For example, in a private organization, projected Return on Investment might be a critical measure. In a public organization this might be completely inappropriate. There, citizen satisfaction or alignment with legislative requirements might be much more appropriate.

Some of the criteria for a good KPI would be this:

Ø The KPI reflects some element of corporate strategy

So, measuring ontime delivery might make sense in most organizations, but where corporate strategy is more focused on quality or safety, focusing on ontime delivery might result in management making decisions that are diametrically opposed to the corporate mission.

Ø The KPI is actionable

All too often we see organizations with spectacular dash boards containing lovely graphics and charts and, when we ask what action is supposed to happen if this indicator turns red or if the needle on that guage goes to the danger zone, no one knows the answer. The whole point of a measuring and reporting on a KPI is to ultimately take action.

Ø The KPI should result in obtainable objectives

Creating a metric which no one ever has a chance of getting out of the “red” and into the “green” makes no sense. It results in urgent emergency action day after day after day and ultimately is self-defeating. The objective that the KPI represents (overload less than 110% of availability for example) must be attainable.

Ø The KPI should be measureable

It’s tempting to make KPIs which are wholly subjective, where someone just plugs in a number or a color. If this is based on a feeling rather than some empirical measure, the KPI typically has little value and yet the decisions made on the basis of the KPI may be just as significant as those based on other empical measures.

Ø The KPI should not be duplicated

Just because we have a dozen different ways to display late schedules, showing all those KPIs doesn’t add value. In fact, it can be confusing and disruptive.

Ø And finally, the KPI and the actions associated with it should be understood by all decision makers. Having a common understanding among all the decision makers of what a particular measure and indicator means is critical to using KPIs effectively.

KPI Challenges

There are a number of easy mistakes that people doing this exercise for the first time face. Here are some of the most common:

Ø “Great news, I have hundreds of KPIs!”

This isn’t great news at all. Remember, what we’re after is a sip from the firehose. If you open the tap too far; if you let too much water through the hose, you’ll knock management from its feet and they won’t be able to absorb the information. Typically we look for a handful of KPIs. As Chris Iervolino, an expert in Business Performance Management (BPM) said, “Somewhere in the extensive negotiations of creating the KPIs, the ‘K’ got lost.” So, just because you can measure a thing, you don’t need to measure it.

Ø “Great news, we have only one KPI!”

Having too few KPIs is also not useful. There are always a couple of opposing forces and results that can be identified in the project management business and the whole point here is to make business decisions. Those opposing forces need to be identified and displayed in some way.

Ø KPIs are too subjective

There’s a strong incentive to invent KPIs for which there is no measure; no metric. The result is an indicator behind which is just a subjective decisions based on someone’s feeling or intuition. While many business decisions are made on a manager’s intuition, when we put subjectivity into a Key Performance Indicator, we take a subjective perspective and display it as an empirical or data-sourced result. The effect of this can be negative.

Ø The KPI has questionable completeness or quality

Imagine that we have a Key Performance Indicator that shows our resource capacity. The graphic shows the organization’s current status of all work and all resource availability and projects the expected over or under load of resources against our project and nonproject tasks in the future. But the indicator only has current project data for 20% of the projects. Another 60% of the project data is at least 30 days old and 20% of the data is missing altogether. Imagine what business decision might be made on the basis of this indicator alone and how potentially damaging that would be. When KPIs are displayed, we always recommend that there is some indicator of the indicator’s quality and the completeness of the underlying data.

Wrap up

The great news about applying the Key Performance Indicator paradigm to project and portfolio management data is that there’s typically lots of great data to choose from and the nature of the project management process results in a lot of that data being of very high formalized quality. That’s perfect for a productive KPI exercise.

If you’re getting started on choosing your KPIs, the best advice I can give is to start small. Start with 3 or 4 indicators and get the measurements of them, the display of them and the actions associated to them right. The benefits of doing this can be profound and don’t worry, you can always expand the number of indicators in the future.

Happy measuring!

Anyone who has worked with project management systems knows that the way you display data can dramatically affect the decions people make from it. This is why we often see GANTT charts with critical activities in red. I’m reminded of one of my very first sales of project scheduling software back in the 80’s. I don’t dare share the name of the organization but it was a large utility. We’d made this sale a few days earlier and now I got a call for “technical assistance”.

“I have a big problem. All my tasks are red,” the hapless client reported.

“Oh, that’s not a problem at all,” I replied. “Red tasks just mean that you are looking at the critical tasks. These tasks have been marked as red because they are on the critical path.”

“That’s another thing,” said my new client. “That word ‘critical’. That’s not going to work for us.”

There was a moment of silence. I was speechless. (For people who’ve met me you will know how unusual this is.)

“Perhaps the word ‘priority’ would be more appropriate,” I hesitantly replied. “Yes, that’s excellent!” said the client. “Now, what can you do about changing these colors? I need to get rid of these red tasks”

For those of you who have grown up with critical path methodology you might laugh. “This person needs training,” you might say. But I learned something very important from the interaction. My first reaction was to think that we needed to get in there right away and train people to distinguish between red tasks and blue tasks but I realized that the problem wasn’t his, it was mine. His perspective was that red tasks were a problem and that he’d have to take action to eliminate all the red from his report before he could give it to management. If you’re familiar with the critical path algorithm, this was going to be a problem because there are always tasks which are critical.

The challenge for the user wasn’t trivial. Even if I had trained him, he knew that his management would not appreciate the distinctions of critical vs. non-critical and would see red tasks the way a bull sees a red cape – they’d want to charge at them with as much force as they could muster.

When data moves beyond highly skilled users and into the hands of people who will only interact with it only occaisionally, choosing the display mode of the data is extremely important. In this day and age, the desire of management to get “real-time” dashboards and “live displays” of projects can lead to unhealthy project environments. Let’s consider the following very simple dashboard:

The situation here seems quite straightforward. Project 1 is running very late, Project 2 is slightly late and Project 3 is on time.

Showing this display instead of a complex barchart might be very appropriate to management. This display will draw attention to the schedule of Project 1. What should be done? Most managers would now query the project manager of Project 1 to ask why the project is late and what can be done to get it back on time.

That’s great so far. But next week when management sees the same report, is the same action appropriate? Probably not. Just having displayed the dashboard once has changed its context. If we display the identical dashboard with identical results next week, management will be likely to ask a very differerent question: “Have things improved since last week?” The display doesn’t show this.

So, we have the same data, the same author of the data, the same display, the same reader of the data but the reaction is quite different. This is because the context is very important. The manager of the project system has a couple of choices now. He or she can make a report that shows a year of icons for this display so the time factor can show management when the task becomes red and then goes back to the yellow caution and hopefully then to green. Or, they can try something like this:

Now both Project 1 and Project 2 are significantly behind schedule but we’ve added a new indicator to the graph. Now the trend of the scheduled delay is displayed. The eye naturally goes to the two red Projects: Project 1 and Project 2 and then to the right where we see that the trend for Project 1 seems to be improving and the trend for Project 2 seems to be getting worse. Also a concern is now Project 3 where the schedule is still on time but the trend is very much in the wrong direction. It looks like resources have been pulled from Projects 2 and 3 to work on Project 1 which is improving at the cost of the schedule of Project 3.

This is all still pretty good at the moment but you can see how the paradigm of such data expands exponentially. There is nothing quite so attractive to senior management than colored dashboard indicators and there’s a whole industry of people making indicators and formulas to drive them. In the simple example above, new indicators might be ordered up in a heartbeat. Was the improvement in Project 1 actually done at hte cost of Projects 2 and 3? What was the relative return on investment of working on Project 1 instead of 2 and 3. Perhaps this Red indicator caused us to move resources from our most important client to our least important internal project. How was the move of resources to respond to the red “X” in Project one aligned to our strategic goals? What did it cost?

Before you know it we’ll have a page full of symbols, curves, flashing lights and glowing buttons. There are a few other things missing from the whole display that are often overlooked. They can be summed up as timeliness and completeness.

Are we looking at all the data? Perhaps the project schedule is showing late but only half the tasks have been updated and when the data is all collected, the indicator will turn green. There’s no indicator in this simple display about how complete the data is or isn’t.

Is the data up to date? Perhaps Projects 2 and 3 were updated yesterday but Project 1 was only updated 90 days ago. Should they even be displayed on the same page? The data might no longer be relevant when compared from one project to another yet there is no indicator on the display that the data is all homogenous.

When we create display systems such as dashboards and summary reports we have to consider these things. I have a couple of basic rules about dashboards:

1. Less is moreJust because we can measure a thing doesn’t mean we should. Imagine a page with 500 colored indicators with 100 different shapes being used. That’s obviously visually stimulating but will it be useful? Almost certainly not. Yet, a page with one color on it (just red for example) isn’t useful either. That tells you to get into action but not where.

1. There must be actionEvery indicator should be able to have a related action to it. E.g. If the traffic light is red and the arrow beside it is red, then the VP must call the project manager immediately and review an action plan for getting back on track.

1. The indicator must have qualityWe have an expectation that project data that is reviewed by management has already been approved in some way. Yet management often asks for real-time dashboards that shows data long before it’s been reviewed or approved. Showing the quality of the data by either showing the level of approval, completeness or timeliness right on the dashboard or through some other process is key to being able to count on the decisions that will be made from the data.

1. The indicator is made for a particular audience.Making graphics dashboards with colored, animated flashy graphics is fun and, in this day and age of technology, not that hard. Designing such a display that makes an organization more effective is much tougher. So, every display we make is written with a particular audience in mind. Who will read this display and what is their context for the data.

The way you display data and what you display can make decisions and action possible that was impossible in the past. By the same token, a badly designed display can cause decision makers to make the incorrect decision inadvertently. So, think about what action such displays will cause as you’re designing them.