Pharmaceutical, biotechnology, and medical device sponsors and contract research organizations (CROs) are hiring sites to do one thing: produce high-quality data. In pursuit of that single and seemingly simple goal, research sites have to do hundreds of things, including training highly competent researchers, recruiting numerous appropriately qualified research subjects, and managing complex nuances of the research protocol.
It’s difficult enough to implement the processes and functions necessary to obtain high-quality data, but if you succeed in implementing the myriad policies and procedures needed, you will have only come half way to the goal. A procedure is only valuable if it is producing the desired results, and if you are not measuring the results, you will have no idea if you are winning. Further, each procedure you add may have negative implications for your other procedures— these are the inevitable “unintended consequences.” In such a research environment, key performance indicators (KPIs) are needed, and a tool known as a “balanced scorecard” is a must.
Why Measurements Matter
As clinical researchers, we know that in order to determine the safety and efficacy of an investigational product, medication, or device, we have to study it, and studying it means measuring it. We have to measure its safety against other medications or devices in its class, as well as its efficacy.
If we invent a new potential antibiotic in an existing class of safe and effective antibiotics, we have every reason to believe it will work. Yet we also accept that we positively cannot proceed without first studying this new potential antibiotic. We accept this as fundamental to the industry in which we work.
In our work life, we invent new processes to fix the ills of our businesses. These processes are designed well and should certainly work. However, all too often we implement processes and walk away. We often simply make the assumption that the processes will work. It is far too seldom that we take the time to then study the intervention, measure the results, and evaluate the effect on our other processes. It is equivalent in our clinical research world to considering matters of efficacy and side effect profiles. Too seldom do we pay attention to our own performance with the same rigor we apply in clinical research.
Interestingly, the simple act of measuring things improves the thing being measured, even in the absence of a meaningful intervention. This is equivalent to the placebo effect in our clinical research trials. This is known as the Hawthorne effect, and is sometimes explained as “what gets measured gets done,” going as far back as now famous experiments from the 1920s.1 According to one description of the experiments, “By the time everything had been returned to the way it was before the changes had begun, productivity at the factory was at its highest level.”2
A measurement is concrete, usually measures one thing, and tends to be quantitative. A metric describes qualities and requires a measurement of baseline characteristics. As an example, I can measure the amount of gas in my car and I can measure the distance I’ve traveled. The number of miles I’ve driven per gallon (MPG) of gas used is a metric, and is far more valuable than simply knowing the individual measurements that went into it. A key performance indicator (KPI) is the next improvement in measurement.
A KPI is a metric that is deemed to be a critical evaluation of the success of a process. There may be many measurements and metrics that are useful regarding that process, but there should only be one or two KPIs that provide a high-level, quick evaluation of the performance of that process. Is it doing well, or is it doing poorly?
You can think of a KPI as having your finger on your pulse. If during a normal part of your day your pulse is between 60 and 100, you’re probably doing just fine. However, if your pulse (a KPI in healthcare) is outside that range, then it’s time to dig deeper with more measurements and metrics to determine the cause. This is the same way you should measure the health of your organization.
A balanced scorecard is the collection of KPIs that look at the health of your organization from multiple different perspectives at the same time. Maintaining each of these KPIs within their desired range maximizes the health of your organization. (See Table 1* for a summary of terminology related to this article.)
It’s important to note also that metrics, KPIs, and the components of your balanced scorecard can be either leading or lagging indicators. If your metric is a lagging indicator, it’s providing you with information about what happened in the past without any remaining time to intervene and improve performance (leaving the opportunity to affect change in the future only). A leading indicator is a metric that is providing you with current data that are predictive of future outcomes.
Your leading indicators can provide you with an opportunity to intervene and impact performance before it is too late. Leading indicators give you the opportunity to affect change actively.
The balanced scorecard was a concept first discussed by Robert S. Kaplan and David P. Norton in Harvard Business Review in 1992.3 It’s important to note that as you work to improve one metric, you can many times worsen others. For example, if a business is concerned only about its financial performance, it can achieve its financial targets at the expense of employee satisfaction and engagement along with the quality it’s providing to its customers and many of its stakeholders. Similarly, it’s easy to imagine achieving our operational measures at the expense of our financial performance.
Kaplan and Norton argued that a balanced scorecard consists of four distinct perspectives (see Figure 1 for a simplified visualization). These are the customer perspective, financial perspective, internal business perspective, and future perspective (innovation and learning). The balanced scorecard allows us to see when we’re improving in one area at the expense of another, and it provides us with an ability to see where all of our important domains are maximized.
That Kaplan and Norton used four domains is irrelevant. Additionally, the four domains that they chose are less relevant than the domains that are pertinent specifically to any given industry (see Table 2* for some possibilities). For example, Southwest Airlines is renowned for putting its employees first, and certainly would list the employee perspective as one of its domains in a balanced scorecard.
Suffice it to say that each business needs to choose which perspectives are most important in its unique case. Four balanced perspectives may be a good recommendation, but three or five would work just as well. Each of these perspectives should be tied directly to a mission-critical strategy of your organization.
A Few Words of Caution
First, just because we can measure something doesn’t mean that we should. It’s very easy to allow ourselves to begin measuring just about everything in our surroundings. Always ask yourself why you are measuring something. If the thing you’re measuring is not directly linked to some mission-critical performance factor, consider stopping that measurement.
Measuring things takes time and effort from you and your team; be cautious not to focus that time and energy where there is little benefit. Determine in advance what your desired outcome is. What needs to be measured in order to monitor and improve performance on the desired outcome?
Sometimes, what really needs to be measured can be difficult. There’s a tendency to measure what’s easiest, and “the easy to measure drives out the hard, even when the latter is more important.”4 The thing you really should be measuring may not be the easiest. If you have the capabilities to measure the right thing, always do so.
Second, always be wary of unintended consequences, since “human beings adjust behavior based upon the metrics they’re held against.”5 If the only thing you speak with your team about is enrollment, your team members will likely pay more attention to enrollment numbers than they will to the quality of the data. You will also increase the likelihood of fraud at your site. Fraud is rare, but if all you focus on is enrollment, then the risk of fraud in your organization will be higher than an organization that focuses on quality. It’s an unintended consequence of a narrow focus on enrollment.
A Site’s Balanced Scorecard
Let’s design a balanced scorecard for a research site.
What are our mission-critical domains?
These domains need to take into account all stakeholders and the site’s strategy. After choosing your domains, you should be able to confidently state that a balanced strategy to win in all the domains chosen maximizes your success. If you can still come up with a scenario where you can fail at your strategy, then consider whether you have chosen a balanced collection of domains.
What are the most significant KPIs to measure within each domain?
To choose your KPIs, first consider what the functional areas touching this domain are in your operating model. Next, decide what result or outcome you wish to achieve in each area, and think about the activities or actions that drive that result. Finally, identify the measurements or metrics that let you know that the right activities are being performed, and that the right outcomes are resulting.
As stated at the beginning of this article, research sites are hired by sponsors and CROs to do one thing and one thing only—produce high-quality data. To produce data, sites need to recruit patients, and it is mission-critical that customers be served well. Therefore, the customer perspective should be part of the balanced scorecard, and should contain KPIs regarding enrollment numbers and data quality.
A sample KPI regarding enrollment numbers could be percent of time achieving sponsor goal enrollment, and a sample regarding the quality of the data would be the percent error rate. Now, the percent of time achieving sponsor goal is, by default, a lagging indicator, since you can only calculate that once goal enrollment was achieved. If the enrollment window closed prior to achieving goal enrollment, you no longer have any ability to intervene regarding that specific research trial.
Your percent error rate, however, can be tracked in real time. That makes it a leading indicator, allowing you to intervene and improve the quality of the data within an existing research trial prior to that research trial ending. For example, if you see that your percent error rate is unacceptably high in a given research trial, that enables you to drill deeper into that trial and try to determine the reasons why. Are the error rates higher because of trial complexity? Is there confusion regarding interpretation of the protocol? Do the staff site members require retraining?
Additional KPIs pertinent to your customers could include total enrollment % of goal across all studies, % sponsor repeat business, or customer satisfaction rate. What is important is that you choose where you want to focus. You can’t be all things to all people. How do you want to best serve your customers?
The financial domain is certainly necessary for a research site’s balanced scorecard. Research is financially challenging for sites; ignoring this domain could be perilous. Repeat the process above to choose the KPIs you believe will maximize the financial health of your organization.
On the other hand, a research site only paying attention to its financial perspective can produce a very short-term view of performance. That site may immediately improve its gross revenue and net income by taking on as many trials as possible, and by processing as many patients as possible through those trials. That site may maintain a lean staff size to further enhance its financial profit, but it’s reasonable to assume such a model would run the risks of increased error rates, lower subject satisfaction, and increased employee turnover due to the overly lean staff size. The long-term view of performance could therefore show that site’s model as being ultimately flawed.
The internal perspective can’t be ignored. It would include KPIs regarding employee engagement, payroll, or perhaps length of employment. If your business, for example, determines that your more senior research coordinators consistently produce higher quality data and higher levels of subject recruitment, then it would be reasonable to create a strategy for your business regarding methods to increase the number of senior research coordinators. Your KPIs regarding this strategy would fall into this internal perspective.
However, if you want to compete for research trials on cost, then perhaps you need to keep your payroll costs as low as possible, and should choose different KPIs to help guide you in achieving your strategy.
Now it’s Your Turn
You need to do the work from here. All sites and all strategies are different. Therefore, site leaders cannot all use the same balanced scorecard and the same KPIs to achieve their strategies. You need to use the processes discussed to determine your balancing point, and how you will research your own outcomes to know with certainty that you are balancing your success.
The balanced scorecard allows you to bind your short-term activities to a longer view on performance. Once you’ve initiated a balanced scorecard in your organization, this scorecard will alter the foundations of how you run your business. Your scorecard translates your mission, vision, and strategy into operational metrics.
Your scorecard will alter what you speak about in your meetings. It will increase alignment throughout your organization, so that your entire team is rowing in the same direction at the same time. Your scorecard will affect business planning so that financial budgeting is in alignment with strategic goals. It will create a mechanism for continuous improvement and your organization will have improved levels of learning.6 You will have achieved the ability to research your own activities as well as you research investigational products, medications, and devices.
As clinical researchers, we measure things for a living, and yet as leaders of sites, small CROs, and small sponsor companies, we are amazingly poor at measuring our own behaviors. However, measuring our own behaviors is the only path to continuous improvement.
A balanced scorecard provides us with the capability to maintain a long-range review that integrates the perspectives most critical to long-term success. Focusing strongly on one perspective—and one perspective only—can produce outsized results for that perspective, but it’s unacceptable to allow your other critical perspectives to suffer.
The ultimate win is achieved when you are able to maximize performance throughout your balanced scorecard. To do that, you need to push evenly in each of the domains you deem critical. As you improve one domain, others may slip a little in their optimal range. As you improve others, you may lose ground on your first; but remember the Hawthorne effect.
If we pay attention to all of our mission-critical perspectives, we will continue to make incremental improvements in each, and we will ensure that our end results are the healthiest they can possibly be.
- Mayo E. The Human Problems of an Industrial Civilisation (Macmillan, 1933); 2nd ed. (Harvard University, 1946).
- “The Hawthorne Effect.” The Economist November 3, 2008. www.economist.com/node/12510632
- Kaplan RS, Norton DP. “The Balanced Scorecard.” Harvard Business Review (1992).
- Caulkin S. “The Rule is Simple: Be Careful What You Measure.” The Guardian February 9, 2008. https://www.theguardian.com/business/2008/feb/10/businesscomment1
- Ariely D. “You Are What You Measure.” Harvard Business Review June 2010.
- Kaplan RS, Norton DP. “Using the Balanced Scorecard as a Strategic Management System.” Harvard Business Review July–August 2007.
Jeff Kingsley, DO, MBA, CPI, (firstname.lastname@example.org) is chief executive officer of IACT Health in Columbus, Ga., and vice chair of the Association Board of Trustees for ACRP.
*To see all figures and/or tables published originally in this article, please visit the full-issue PDF of the August 2016 Clinical Researcher.