Making Metrics Matter The Changing Paradigm of R&D Metrics As the pharmaceutical industry comes under greater scrutiny across the board, everything from discovery processes to R&D productivity to profit margins is subject to review. In response, top industry executives are reexamining the efficiency of their R&D operations. In an effort to self-scrutinize, many companies are taking a quantitative approach and reviewing and refining their metrics systems to realize efficiencies and boost R&D productivity, as well as evaluate and motivate their staff. By defining metrics and collecting data, companies can observe and improve everything from strategic portfolio decisions to resource allocation. Having a historical record, as well as a real-time view of operations, allows executives to make decisions regarding resources, such as staff and technology, and eliminate bottlenecks and overlaps. Finding efficiencies in R&D can lead to increased productivity, which translates into improvements for both the top and bottom line. Thought Leaders Karen Briegs. Director, Marketing and Product Development, 3C Company, Kenilworth, N.J.; 3C Company helps clients achieve strategic advantage by applying industry-specific technology to their business processes. For more information, visit 3cpharma.com. John A. Cloward. Associate Director, Pfizer Global Research and Development, New London, Conn.; Pfizer discovers, develops, manufactures, and markets leading prescriptions for humans and animals. For more information, visit pfizer.com. Krish Ghosh, Ph.D., MBA. Director, Project Planning and Information, Wyeth Research, Collegeville, Pa.; Wyeth Research is the research division of Wyeth Pharmaceuticals, a part of Wyeth, a research-based pharmaceutical and healthcare products company. For more information, visit wyeth.com. Linda Martin. Principal, KMR Group Inc., Chicago; KMR provides consulting services for the international pharmaceutical industry, including pharmaceutical companies, CROs, and biotech companies. For more information, visit kmrgroup.com. Tom Privette, Ph.D. Senior Director, Global Business Development, Wyeth Pharmaceuticals, Collegeville, Pa.; Wyeth Research is the research division of Wyeth Pharmaceuticals, a part of Wyeth, a research-based pharmaceutical and healthcare products company. For more information, visit wyeth.com. John D. Rhodes. U.S. and Global Managing Partner, Life Sciences, Deloitte & Touche LLP, Parsippany, N.J.; Deloitte is a professional services firms, providing audit, tax, financial advisory services, and consulting. For more information, visit deloitte.com/us/lifesciences. Bruce E. Seligmann, Ph.D. Founder, Chairman, and Chief Scientific Officer, High Throughput Genomics Inc., Tucson, Ariz.; HTG provides ArrayPlate qNPA technology and services for the life-sciences industry, addressing current unmet needs and enabling a new era of drug discovery and diagnostics. For more information, visit htgenomics.com. William K. Sietsema, Ph.D. VP, Clinical and Regulatory Strategic Planning, Kendle, Cincinnati; Kendle is a global CRO delivering clinical-development solutions to help the world’s biopharmaceutical companies maximize product life cycles and grow market share. For more information, visit kendle.com. Jeffry L. Vaught, Ph.D. Senior VP and President of Research and Development, Cephalon Inc., West Chester, Pa.; Cephalon is an international biopharmaceutical company. For more information, visit cephalon.com. Ronald S. Waife. President, Waife & Associates Inc., Needham, Mass.; Waife & Associates is a change management consultancy in biopharmaceutical clinical research. For more information, visit waife.com. Demetris N. Zambas. Associate Director, Clinical Operations, Schering-Plough Research Institute, Schering-Plough Corp., Kenilworth, N.J., Schering-Plough is a global science-based healthcare company with leading prescription, consumer, and animal health products. For more information, visit schering-plough.com. David S. Zuckerman. President, Customized Improvement Strategies LLC, St. Louis; Customized Improvement Strategies is a management consulting firm that focuses on process improvement, organizational development, leadership development, and change management. For more information, visit rx-business.com. DEFINING METRICS Metrics can mean different things to different groups. But industry experts agree that defining terms is an important first step in setting a company on the path to metrics success. Privette. At Wyeth, we started with the highest level of strategic metrics we had in place and set out to define those. We asked questions such as, how do we internally define a compound going to a development track, how exactly do we define the IND milestone, start of Phase II, start of Phase III, and so on. We had to carefully define the stage-gate decisions and the criteria for a decision to come up with the definition of the metric. And then once we defined about 15 key metrics, we put in place a system to collect the key metrics data. Although a company may already be collecting financial data, personnel data, clinical operations data, discovery data, and so on, the data are never conveniently centralized for access. Therefore, there is also a significant effort required to develop a system that captures the key metrics in a way that is efficient. It took us about six months to get a set of fairly good definitions and put a rudimentary system in place to capture these metrics and record them and report them back to senior management and eventually to the organization. This is an ongoing process, and we have been at it for about four years now. We fix and adjust things as we learn more. The metrics change as the business dictates, but our processes also improve as we move along and get more experience with the various components of the system. Waife. There are many different types of metrics, and they range from measurements of overall corporate or portfolio performance to how well an individual is doing his or her job. Right now there is a metric buzz surrounding every one of those areas. Zambas. Companies that use metrics well start with reality. If a company has a good means of tracking what gets done historically, then chances are it has developed a good algorithm of what its resource needs are for various trials, based on their complexity, their length, their phase, and other factors. Once a company has these data then it can look toward its next program, establish what studies are needed to accomplish the program, and, using the algorithm from the study level, roll that up into what the entire resource needs are and what the end timeline for the whole effort will be. This is where the balancing act starts. Martin. In our experience we have found that certain measures such as cycle times, attrition, success rates, throughput investments, productivity, and the value of what is being produced have the most significance. As far as a broad framework, these valuations are at the core of what should be measured in R&D. Sietsema. The term metrics is used in a lot of different ways by different people. I believe there are three categories of metrics. The first is metrics that can be applied specifically to clinical trials, measuring how well a clinical trial performs. The next is organization metrics, or development organization metrics, where one determines how well a company is performing as a drug-development organization. The third type is efficiency metrics, which looks at cost and time. Briegs. A primary question on most of our clients’ minds is why aren’t their metrics working? There are three types of metrics being used that span the entire company, not just clinical development: time-based metrics, which measure milestone to milestone, such as last patient visit to database lock; quantity-based metrics, which are centered around patient involvement, such as how many sites have been initiated and how many patients are at those sites; and quality-based metrics, which are focused on the data aspect, such as how many protocol amendments had to be made and how many data discrepancies per validated data point there were. Cloward. Decades of research have shown that when a metric is chosen, people respond to that metric. Some companies have metrics based on simple numbers, such as the number of compounds coming out of discovery, the number of first-in-human trials, or the number of Phase II starts, but they might not have clear guidance or policies to ensure that the numbers that are achieved are really contributing toward the bottom line. For example, if a company rewards people on the number of compounds, it will get large numbers of compounds, but that doesn’t necessarily translate into quality or real value. It is worth making the effort to choose the right metrics, define those metrics, and make sure they are well-understood throughout the company. While it may be easy to choose some obvious milestone, this is not necessarily a smart way to run the business. Zuckerman. Traditionally, people in this industry measure cycle times, timeliness, and achieving milestones because these are the most obvious and easiest to measure. Of course, pharmaceutical companies want to be faster and be on schedule, but the problem is these measures do not really get at the cause of what drives cycle times, and they risk suboptimizing processes. The data-management department may make changes that make processes faster, but these changes may create errors that get passed on to the statistics department. This is the biggest hurdle, and companies are starting to address this challenge. People are starting to talk about quality and efficiency and then moving on to the organizational, financial, and customer-service issues. Vaught. The traditional metrics approach, before things began changing in the last 10 years, was that researchers would find a compound that was interesting. They would do some safety pharmacology studies, recommend the compound for development, push it to development, and then it would become part of the pipeline. That decision process was driven predominately out of a research interest, which may or may not have been coupled to a therapeutic strategy or integrated across all divisions. Along the process there was an evaluation of the molecular target, the disease, the competitive analysis, and so on, but there was no communication between all the groups to ensure that going forward there would be a probability of success for the compound. Seligmann. Pharmaceutical companies begin to measure the metrics for a drug-discovery program at the level of gene expression. When putting that process together, the very first metric of a program is a validated target, and the next is efficacy. As researchers move along the drug-discovery line, metabolism is a metric that comes into play. Another metric that is important is the ease of synthesis in the scale up. Then there are program-specific metrics, such as metrics of efficiency and how long it takes to get to each step of drug development. ADDRESSING METRICS CHALLENGES Although many companies have a metrics system in place, experts say many of the traditional systems are flawed. Rhodes. One of the biggest problems with traditional metrics is the absence of metrics. The R&D area hasn’t had clear metrics or explicit criteria to establish metrics, for example, in the area of compound acceptability. One metric might be the patient visit, but traditionally there has been no clear way of communicating within the R&D organizations what the R&D metrics should be. One of the biggest shifts occurring is a better definition of those metrics. Because there is a unique employee base in the discovery and development area, there has been a lot of latitude and maybe an absence of metrics around what discovery should be and how short-term and long-term goals should be measured. Zambas. One of the critical issues with metrics systems is that every department picks a different key milestone that is important, for example, first site initiated or database lock. But these milestones are only valuable within the context of where they were developed. The more robust the tracking is, the more accurately the variability of the research business can be reflected. This area is constantly moving, and if a company is limiting itself to only tracking a couple of key points it will wind up without a complete picture. Briegs. I don’t think that the problems with metrics are the metrics per se. The problem right now is that companies are collecting metrics that nobody trusts because data come from systems that are either not mature, are old and outdated, or are custom developed and too rigid because they’ve been too tightly bound by work processes and can’t capture reality as it changes. Additionally, some of the systems cannot meet the more aggressive metrics and performance measures needed today. Ghosh. Although we have come a long way in collecting and analyzing appropriate data that can be used for developing metrics, scorecards, dashboards, different levels of reports –in terms of granularity, management presentations, analyst presentations, and so on — the information is still fragmented and is collected from multiple sources. We have recently started integrating all of the processes and information, which may relate to not only project planning — starting from early development and tracking the information all the way to launch — but also cycle times, success rates, costs, resources, and value. We are developing a Wyeth Research enterprise management tool where all the information will reside, so we can help senior management make better decisions using that information. Sietsema. Many of the existing metrics being used are good. Oftentimes where companies fail is in their ability to effectively use the metrics they have chosen. It takes effort to capture metrics so it is important to minimize the work required to capture the information. Most project teams will focus their efforts more on the execution of getting work done rather than tracking the metrics to allow future improvement of performance. The key to improving any person, process, organization, or performance is to understand what was done in the past so it can be done better in the future. Zuckerman. Most companies are measuring metrics at too low a level. It is easiest to look at the clinical-trial execution process because there are a lot of data that are easy to measure. There is nothing wrong with this, as long as there isn’t an expectation that those measures will help the company solve problems elsewhere. The process won’t be improved until companies start measuring at a higher level and earlier in the process. Waife. Most metrics as applied to clinical development have been, and still are, time-interval based. Traditional metrics are based on a milestone, on how long it takes an organization or a team or a department to move a project, a trial, or another large functional activity. The problem with milestone-based metrics, which everybody still pursues with great enthusiasm, is that there are so many variables to achieving that milestone that the metric is nearly useless. For example, the interval between last patient visit to database lock is a metric that every clinical-development department focuses on, and it is often used as the basis of defining the introduction of electronic data capture or to assess whether the clinical organization is doing a good job. To simply and grossly compare how many days or weeks or months it took to lock a database on one project, in one company, at one time, in a specific year, with one set of sites, in one therapeutic area and compare that with any other project is useless because of all of the variables. There can be as many as 20 different variables that can impact a metric. The biggest problem with traditional metrics is that the variables are not taken into account. Zambas. In general, milestone metrics and measures of effort are very closely related. To measure effort one has to look at some time points, from the beginning of an effort to the end of an effort. It is a matter of interpreting the data available. If there is a robust list of items, then the deltas between them are the time points at which various efforts can be applied. A company can easily set up algorithms for resource needs for those efforts if the list of milestones is robust enough. Seligmann. For too long, industry metrics have been based on one target measurement of efficacy and separate measurements of specificity. The trend in the industry is to look at the efficacy of a handful of targets that act in concert to induce a disease, act in concert to reverse the process of disease, or mediate the effect of a drug or mediate an adverse effect of a drug. From a strategic sense, the metrics going forward will evaluate whether all of the targets involved can be measured or at least if all of the targets involved at periodic intervals can be profiled. This then determines whether a program is on target. The process is facilitated and, in fact, can be carried out more precisely using multiplexed high-content assays, measuring the complete target profile. The result will be greater efficiency, greater productivity, and greater success. Vaught. The problem is that the bar for innovative therapeutics is extremely high now. It is going to take some time for us to learn how to use proteomics, genomics, and other technologies to better address human disease. The preclinical models currently available are validated based on drugs that work in people. But with new technologies, if we start with a different molecular target that could be curative, there is no animal model or data to show that it works and is effective in human beings. The types of targets that are being approached today are very different from what have been seen in medicine in the last 20 years, which makes developing metrics extremely challenging. The industry is adopting a variety of measures to increase efficiencies, decrease costs, and provide innovative therapeutics to the caregivers, as well as patients but the challenges of meeting this next level of therapeutic are exceedingly complex. Briegs. Arguments about metrics tend to focus on the quality of the data as opposed to focusing on what the data are indicating. No one trusts the data; everyone wants to keep their own spreadsheets, and that leads to a lot of insecurity with the numbers. Companies also are collecting too many metrics; some companies are collecting several hundred performance metrics measures, and most of them aren’t actionable. Companies need to choose metrics that can be acted upon in real time, such as those that indicate a downward trend in performance, as well as those that are predictive of problems later down the line. METRICS MODELS As companies refine existing metrics systems or begin to implement such systems for the first time, experts at the forefront of this evolution are developing models and best practices for metric success. Privette. There are various levels of metrics depending on their purpose. High-level strategic metrics or objectives are established by senior management based on the strategic goals of the organization. These high-level metrics can be rolled down to a divisional or functional level, and even further to a department or process level where they become very operational in their utility. At the operational or process level, metrics are usually focused on measuring and managing the processes, with metrics that are directly related to measures of efficiency and are actionable. The goal is to have performance objectives at all levels, which if achieved, will roll up and ensure success at the higher levels in the organization. Basically, this is an integrated scorecard system for alignment that transcends all levels in a logical and systemic fashion, with a mechanism for measuring and communicating progress. Sietsema. Kendle recently adopted the balanced scorecard system, which is a way of implementing cultural metrics performance as it applies to everyone’s job. It is applied on several levels — departmental, divisional, and corporate — not just at the clinical-trial level. This system gets to the core of why most metrics don’t work, which is because metrics aren’t built into the culture. When adopting the balanced scorecard, a company is embracing the process as part of its corporate culture. This is not an easy thing to do; it takes a lot of effort to get people on board, and management has to explain that it is a way of measuring how well people do something so they can do it better in the future. Waife. What we emphasize in our work is the transition from milestone metrics to units of work. What should be measured is how much effort it takes to produce a unit of work. Effort, not time, is the critical item to be measured. One of the reasons why this often isn’t done is because everybody wants to compare himself or herself with everybody else, which is a milestone method rather than a unit-of-work method. The unit-of-work measurement gets to the nuts and bolts of the operations of the organization and calculates how much effort, how many man hours, and what external dollars have to be spent on a task. All of this is calculated against some standard unit of measurement of work. To use the example of last patient visit to database lock, the new metric would be the level of effort required to clean a CRF page. This eliminates many of the variables. Rhodes. Good coordination of communications is needed in the R&D function to make metrics work. There needs to be joint accountability so that as metrics and criteria are defined in the discovery area this can be communicated to preclinical development, for example. In addition to more explicit metrics, the communication and discussion of those metrics, and what compounds would be acceptable for development, is critical for organizations much earlier in the process. A company should involve not only the discovery group, but preclinical and the commercial side to coordinate discovery pursuits that are good for health, as well as products that are commercially feasible. The biggest opportunity for companies in defining metrics is finding what targets to pursue versus screening and pursuing all possibilities. That is where the inefficiency has been in the R&D area — on the pursuit of broad spectrums. Briegs. We have worked with companies that are very much focused on their metrics, and the timeline depends on how complex they try to make this. We helped a company bring visibility to trial metrics with a six-to-eight week timeline to develop the metrics and the tool product. If companies know what metrics they want to use, this can be done quickly. But it can take a year and a half if they really struggle over very basic things, such as deciding what they are going to measure. I think it can be, and should be, done quickly. If organizations can get people to stay away from the little details and focus on the major points, then developing a metrics system can be done quickly. Zuckerman. By continuing to measure cycle times and quality at the clinical-trial level, there will be faster trials. But whether a company’s clinical-development plans, its portfolios, and its decision making are better is not impacted at all by these types of metrics. Managers have to start looking at the quality of protocols and measuring at a higher level. Briegs. Metrics can be used at a higher portfolio level to shift resources around. If a particular trial or process is starting to lag, a company can start to look at where there will be shortages or overages of staff and explore the possibility of starting, for example, an investigator-initiated trial or a second-line trial into that space. Companies can begin to use their resources much better this way. Metrics are not just about individual or group productivity; they can be used at a portfolio level to move resources around actively and fill any gaps leading to better overall productivity of the company. Martin. Historically, companies tended to look at metrics in isolation. We work with companies to put the measures together and understand the true impact on productivity. There has been a very focused approach on individual measures, but not a holistic approach, which I think needs to take place for better portfolio decisions, as well as for better management of the R&D organization as a whole. Portfolio management has become more sophisticated in the industry in terms of looking at the various risk assessments and evaluations. Some of the measures, such as success rate or the actual risk assessment, are key to making stage-gate decisions, as well as guiding the overall strategy of the organization. Companies also have to take a multifactorial approach to resource investment to determine what is going to impact that project, as well as to make better decisions around that portfolio. Ghosh. As Dr. Bob Ruffolo, president of Wyeth Research, has stated, “What drives our R&D productivity? It is the Wyeth R&D Productivity Model.” Our current definition of the R&D productivity model is 12 projects coming in to development every year, and two projects are expected to be out, that is, successfully launched with an average target cycle time of six and a half years. We also consider how valuable these two products are that come out in the end. We work to value the overall portfolio of the company, the pipeline projects as well as the in-line products that are in the market. We implemented the R&D productivity model in 2001, and we expect to reach steady state in the 2006-2007 timeframe, assuming that all of the assumptions for cycle times and success rates hold true. We have had higher than expected success in moving projects through the early phases of development. This has challenged our R&D budget in the recent years. This is definitely a good problem to have, but it creates a disruption in the steady flow of compounds in a steady state and, hence, affects the overall value of the portfolio. Cloward. I have worked in project management in several different industries and have seen companies make significant investments in processes and systems and cultural changes to improve the management of resources. Other industries have put in a lot of effort over the years to analyze metrics and to understand the full impact the information has throughout the business. I believe many large pharma companies are still struggling with these issues. GETTING EVERYONE ON BOARD A major hurdle in implementing a metrics system is overcoming the fear and misconceptions associated with the process. Many experts have identified that a barrier to the success of a metrics program is the overwhelming belief that metrics have repercussions and can lead to job losses. Briegs. The major question is how can an organization transition from the idea that metrics mean punishment to the idea that metrics mean improvement? Management has to communicate to employees that the reason for capturing metrics is to help, not to hurt. If managers notice that a person or group is having problems on a particular project, maybe they will realize that more resources are needed and they can provide extra staff so a project can get done on time. If an organization can transition people to understand that metrics are not a way to dock someone’s pay, but to make sure that senior management is aware of, and able to give attention to, teams that need assistance, then that is a brilliant and positive thing. But there is a whole mentality that has to be changed. Sietsema. People are afraid of being measured because, if expectations are not met, this can impact salaries, performance ratings, and job retention. Martin. Often companies put together metrics programs but they don’t have buy in from the various levels of the organization. Companies need to develop a process whereby members of the organization believe in and understand the metrics. Essentially these metrics need to be justified, agreed upon, and scrutinized at the top. This is key to the organization. It does take a fair amount of time and energy, and there are costs associated with capturing metrics. Managers need to understand that this commitment is part and parcel of running their business. Waife. It is not enough to say executives need to communicate the importance of metrics in a positive manner. From the beginning, they need to communicate very clearly and continuously that metrics are being used to improve everyone’s performance, to determine where the company is doing things well, and to understand the obstacles that are in the way of individual staff members. Managers need to let everyone know that a metrics system is not for finding people who are doing a bad job. Vaught. Scientists and discovery people hate metrics. Discovery is a very creative environment. Managers can’t go to discovery scientists and tell them that next week they have to come up with an innovative therapeutic for treating Alzheimer’s disease. Nonetheless, boundaries can be provided, and a manager can say this is where we are and here is where we need to be. We talk about the process with our discovery group. We ask them what they have to do and what resources they need and whether any technologies or processes can be changed to make their jobs more efficient and to shorten overall time frames. When they are part of the process, rather than the process being implemented without their input, this can go a long way toward buy in. The members of the organization need to understand what the corporation is trying to achieve. Development organizations are more accustomed to metrics because development can be timed. For example, it’s possible to know how long a toxicology study is going to take and how long it should take to develop an analytical procedure. Zambas. Issues with buy in are common in every company and every industry. The very first hurdle is getting people to go along with change. The second is the big-brother aspect. Companies don’t want people in the research environment to feel as though they are on an assembly line. Researchers should feel as though they are part of the creative process, and the more exposure they are given to the results of detailed tracking the more they will feel as though they are part of the process. Rhodes. Buy in comes reluctantly because there is skepticism about the outcome, which people assume is reduced budgets and loss of jobs. In fact, the use of metrics may result in more money being made available for the right opportunities and the job impact can only be measured as time unfolds. Sometimes, metrics may, in fact, result in outsourcing if the company can determine that it can focus more of its own resources on a particular area of interest. Martin. People in the organization need to see measures being used, and this can be done in a variety of ways. Ensuring transparency by posting measures throughout the organization is a good thing. Companies need to understand that metrics are not just measurements to be discussed in the board room; the information is something the entire organization needs to be consistently aware of. Briegs. If the data are hidden under the covers of a large system or buried deep into some report then nobody pays attention until somebody has a pet project and publishes a set of statistics. This is counterproductive. Choosing a few metrics or measures and making sure they are available on a regular basis is important, whether through a dashboard or updated every week and put on a Website. People have to see how metrics change from week to week or day to day. Martin. Senior management needs to place some visible value on the metrics in terms of how the information is going to be used, which can be tying goals and incentives to other parts of the organization and sanctioning various programs around measures. Too often, a company says it is going to look at measures and then the resources are not available to do anything with those measures. This can be counterproductive for an organization. Zambas. Metrics are a directive from our senior management. We initiated this years ago because management wanted a better means of planning and, if anything, the significance of that planning has grown. Rhodes. Setting the metrics strategy has to come from the head of R&D in coordination with the CEO and chief operations officer. The head of R&D and all key leaders who head up the different therapeutic areas have to be totally committed. Other functions of the company have to be committed as well as the commercial and manufacturing aspects so that there is a seamless supply chain. Organizational teams need to understand why the company is doing the measurement, where the organization is headed, how the data will impact the success of the company, and how people can be a part of that. Otherwise the process will not succeed. Vaught. Senior management is very involved in the metrics, and we review them on a monthly basis. We have an advantage because of the company’s size; we do not use a top-down approach. Rather, it is a bottom-up approach. Functional teams say they may need a certain amount of time to complete a project and management simply challenges by asking why. We ask if there is any way it can be done better and if we can provide them with information, materials, resources, or technologies that would allow a project to be completed faster. Clearly, our objectives for the company and our shareholders are to get things done as expeditiously and as cost-effectively as possible but there are certain things that can’t be done quickly. Ghosh. Originally, Wyeth took a top-down approach strategy to roll the metrics from the top (mega) level to the micro or the sub-micro levels. But the direction we are heading now is a bottom-up approach, where we will have a unified enterprise management system to measure and collect the data in one place. Everybody will get to see numbers coming from one source, rather than one metric having multiple numbers and functions depending on who provided the data, when was it viewed, what system generated it, and in which format it was generated. In the new process/system all metrics from the ground levels will be expected to roll up to the highest level. Cloward. It is a very difficult business challenge to assess whether people are making good recommendations and good decisions based on the data available when the result isn’t known until much later. This can’t be solved by simply coming up with a quick fix to evaluate whether people are making their numbers. Privette. The biggest challenges Wyeth experienced in implementing its metrics system were getting everyone aligned, getting buy in, and addressing any cynicism or resistance, which is endemic to any large organization. But senior management was very clear about the need to get behind this. There was urgency as far as the survival of the company and the need to change the way we were doing business. There were hundreds of people involved in developing the scorecards and the plans on how to reach these goals, so it wasn’t just imposed; the method was from senior management, but the teams were put together to decide how to get there. After a year, there was positive feedback from employees. When a company sets out to increase productivity by 300% to 500%, and it actually does it in a way that people see what they have accomplished, it makes them feel good. Very important attributes of Wyeth’s metrics system are its transparency and having the ability to communicate our accomplishments. Ghosh. There are great challenges in aligning resources to projects in development, whether they be dollars, which are relatively fungible and easier to reallocate, or aligning the FTEs to projects, so that projects can be easily accelerated, slowed down, terminated, or kept moving at the same speed. This, I believe, is one of the biggest challenges that the industry is facing today, and this will not improve significantly in the very near future. A huge advantage for any pharmaceutical company over its competitors is to develop strategic resource management capabilities that will mobilize and reallocate resources, mostly FTEs, efficiently across the entire portfolio, and can be used to develop, measure, and track a return on resources. Additionally, this will greatly help senior management in effective decision making through “options” and “impact” analyses. Sietsema. For a company’s development team, bringing metrics into the culture is a challenge. It is important for project leaders to play a role. It also helps to have someone whose role is to keep the metrics, and that can easily be a full-time job. Ghosh. In addition to commercial and technical risks there are operational risks that could be related to resource constraints, task executions, patient-enrollment issues, and new data requirements for a novel target that were not anticipated, to name a few. The operational risks are the most difficult part to quantify and to set targets against. Wyeth is somewhat an industry leader in looking into operational risk and how to quantify it and incorporate it into the company’s plans so that we can do better portfolio management. Vaught. Applying a rigid metric to clinical research presents the biggest challenge. The question becomes how rigid should that metric be? For example, if the metric pertains to meeting an objective by a certain date, then the risk is that something important will be missed. Martin. Some companies have implemented a systematic process not only for defining but developing measures. One challenge is to determine what the process is once people have metrics in hand and to drive some decisions from of those measures. Cloward. Integrating R&D metrics with the organization’s goals is critical, and this is why the process has to start with the senior executives. The corporate strategy needs to migrate to individual goals and then individual goals will support the metrics of the corporate strategy. Zambas. If a company works backward from what it would like to accomplish in a year, and at the same time looks forward as to what capacity it has, it can identify gaps. Every company has to determine whether accomplishing what it wants is significant enough to warrant increasing resources or outsourcing or whether priorities need to be readdressed. Seligmann. Companies have to understand the metrics of program progress and be able to make decisions on when to kill certain programs or shift resources to other programs. Ultimately the goal is to produce drugs as efficiently as possible with the highest chance of success. Briegs. Improving processes and speed and time to market, particularly in the development timeline, is critical. These are areas that haven’t necessarily been paid attention to before. Pharmaceutical companies are under such pressure to bring costs down and are searching for ways to improve their speed and processes while retaining high quality. Companies are doing this by trying to bring visibility to some of the areas that are causing overlaps, allowing people the ability to see the data, and bringing visibility to the metrics that demonstrate where performance improvement can truly better the organization. Vaught. Our executive committee outlines a project and its metrics and sets a time it wants a project completed by. Then the head of clinical operations and myself go back to the teams. We have a broad steering committee with input from the marketing and sales folks. We outline the objectives and ask each group to report back how decisions impact them. We identify the points where several projects are hitting simultaneously, and everyone has an opportunity to provide input on the project. If it is not doable, then we go back and either find a way of getting it all done at the same time or modify the corporation’s expectations. PharmaVOICE welcomes comments about this article. E-mail us at feedback@pharmavoice.com Karen Briegs 3C Company The problems surrounding metrics involve the quantity and accuracy of the metrics and the equitability to an individual’s role, not the metric itself. David Zuckerman Customized Improvement Strategies LLC When an organization knows its goals, is measuring at the right level, and the senior executives show that they are committed to metrics, then things will really start rolling. Dr. Bruce Seligmann High Throughput Genomics Inc. A technology such as a qNPA platform can be used to improve the measure of the metrics to advance compounds, as well as to generate efficiency, to increase productivity, and to decrease timelines. John Rhodes Deloitte & Touche LLP Companies have to set their strategy before implementing metrics. If there is no communication of a strategy, a company may find it is on a slower path to discovery, development, and commercialization. Ronald Waife Waife & Associates Inc. Metrics need steady executive commitment and positive follow through. Metrics too often are used as a stick instead of as a carrot. Dr. Tom Privette Wyeth Research One cannot attribute productivity success to a metrics system. What can be attributed to an increase in productivity is the use of the system to change the organization’s behavior and get it moving in the right direction. Metrics are a tool; the challenge is changing the culture and aligning the people to hit the targets. Linda Martin KMR Group Inc. Many companies have been using what we might call lagging metrics. They know what their historical performance has been, but there has not been enough emphasis on measures that can provide real-time information to help change an approach midstream. Dr. William Sietsema Kendle Tracking Metrics is an important process. Too often, companies don’t track correctly because they are so wrapped up in doing what needs to be done that they forget to track what they could be doing better. John Cloward Pfizer Global Research & Development Easily countable short-term metrics don’t necessarily contribute to long-term profitability or effective use of resources. Better metrics can lead to improved efficiency and effectiveness by making sure that the resources are applied where the greatest potential exists. Dr. Krish Ghosh Wyeth Research R&D productivity is highly linked to several variables: number of project starts, development cycle times, success rates, development costs, and value. In addition, the commercial risks lead to large uncertainties in predicting the market and coming up with the right revenue forecasts. Demetris Zambas Schering-Plough Research Institute Our metrics model includes historical reference, and the algorithms we have developed become more accurate with each additional year of information that goes into the system. The level of quality we have is a key benefit. PharmaVOICE Reader Survey: R&D Metrics Is your organization actively and adequately measuring R&D operational and performance metrics? Response Yes 41.9% Somewhat 30.2% No 20.9% Does not apply 2.3% Other 4.7% How important are metrics to your R&D organization? Response Very important 62.8% Somewhat important 30.2% Not important 2.3% Does not apply 2.3% Other 2.3% SOUND BITES FROM THE POLLING FIELD Metrics seem to have taken over the industry to the extent that they can be an obstacle to working effectively. Management is so worried about metrics that managers have lost sight of the big picture. Clinical research is not a production line, and companies would be better off hiring and supporting empowered professional employees. Not all metrics are of equal value. Meaningful metrics will ultimately allow for a reduction of overall drug-discovery cycle times, a reduction in attrition, and produce more regulatory approvals. To accomplish these goals, discovery front-end metrics are much more important than quantity metrics. In other words, quantity metrics, while easier to tally, may not be the answer to all of the industry’s woes. Some distinction has to be made as to the metrics involved at various levels of R&D. What’s helpful as a metric for preclinical may not work for clinical development. And even within clinical development, metrics may very well differ depending on the trial phase or the type of product or device. Source: PharmaVOICE, Titusville, N.J. For more information, visit pharmavoice.com. Note: The survey’s 43 respondents represent the following types of companies: 67% pharmaceutical, 14% biotechnology, 12% biopharmaceutical, 5% device/diagnostic, and 2% medical consulting. Positions represented by these respondents include VPs, directors, managers, engineers, and scientists. Metrics Versus Milestones Understanding the difference between metrics and milestones is an important distinction for a company to make when embarking on a metrics project. Milestones are critical points in time during the development program and are often targets that the team wishes to “hit.” There are endless milestones that could be measured, and the team should decide which are most important to its operation and can lead to the greatest insights regarding performance. Metrics, on the other hand, are performance data; these often measure the elapsed time between milestones or provide a quantitative measure of performance. Importantly, the team should be cautious not to track too many milestones and metrics. Many a project team has fallen into the trap of measuring so many milestones and metrics that it devotes enormous energy to this effort and is distracted from its main goals. One or two dozen milestones and metrics should be sufficient. “ Many organizations confuse the definition between a metric and a milestone,” says William K. Sietsema, VP, clinical and regulatory strategic planning, at Kendle. “A metric is something that can be used to provide information about performance, how long it took, or how much it cost. A milestone is a discrete point in time or a discrete event in a program, and the two are separate.” Milestones and metrics are important ways to keep the team on track, to highlight opportunities for improvement, and to educate and better select external suppliers. But the project team should be careful that the measurement of milestones and metrics do not become its reason for being. SELECTED MILESTONES IN CLINICAL DEVELOPMENT Study milestones Date on which protocol was signed Date of first approval by IRB/IEC Investigator meeting date Date on which first patient was enrolled in the trial (FPI) Date on which last patient finished the trial (LPO) Date on which last case report form arrives in house Database lock date Delivery date for unblinded study results Date on which final study report was signed Program milestones IND or CTA filing date End-of-Phase II meeting date Phase III go/no-go decision date NDA or CTD filing date NDA or CTD approval date Launch date SELECTED metrics in clinical development Study metrics Time to write protocol Time to obtain IRB/IEC approvals (average, min, max) Time to negotiate clinical investigator contracts (average, min, max) Time to enroll patients Number of patients enrolled (site-specific) Number of queries per patient Time to resolve queries (average, min, max) Time to retrieve case report forms (average, min, max, interval from LPO to arrival of last case report form in house) Time to lock database Time to write clinical study report Program metrics Time to prepare an IND or CTA Time to prepare NDA or CTD Time from completion of last study to completion of NDA or CTD Time from approval of NDA or CTD to launch Dr. Jeffry Vaught Cephalon Inc. Ultimately metrics come down to technology and people, as well as having a system that is understood from the lowest-level person to the highest-level person so everyone is on the same page.

Download PDF