Publications

Measuring Up: Benchmarking Public Services (1998)

With the advent of the "reinventing government" movement, government agencies are focused on increasing their accountability to the public they service. This involves developing and using performance measures to both document the state of public services and provide a guide for how they should be improved. While there is a legitimate role for comparative analysis in improving public services, the use of performance measures can be a two-edged sword. Favorable results identify high performers — jurisdictions that are renowned for cost efficiency, high-quality services, or innovative practices — resulting in positive attention and popular acclaim for top managers and officials, and hopefully front-line workers. On the other hand, poor results identify poor performers, and can add fuel to the movement for contracting out or even eliminating certain services in those jurisdictions. Thus, at the outset it should be recognized that in the public sector all benchmarking takes place in a political atmosphere which may affect the objectivity of any exercise in which the performance of government agencies is examined.

Benchmarking: what is it?


A benchmark is a standard used by an organization to measure its performance. Broadly defined benchmarks can serve as goals that an organization sets out to achieve as part of its overall mission. For example, goal-oriented benchmarks that reflect a jurisdiction’s social or economic well-being might include literacy rates, employment growth, infant mortality, or personal energy consumption. For each set of measurements, goals can be developed to serve as tools for focusing the efforts of government agencies and targeting public spending.

Benchmarking can also be focused on agency or workplace level outcomes. At this level, benchmarking involves developing performance measures for specific functions and comparing these outcomes across organizations, or between different branches or divisions within the same organization. From this type of study, an agency can determine where it stands in relation to others performing similar work and why — an exercise called "gap analysis." If the agency is not a high performer, then the superior methods of top organizations may be identified and imported; thus this process is often called "benchmarking for best practices."

Just about any comparative study can be viewed as a benchmarking exercise and just about anything that can be measured can be benchmarked. However, benchmarking differs from simple comparative studies in several respects. First, it is supposed to be an ongoing process, not a one-time study. Second, it goes beyond simple ranking to include factors that determine an organization’s rankings; that leads to goal-setting, and informed decisions on funding or work process improvements. Finally, it is not a quick and easy exercise, but a process that involves considerable time, support and funding.

Goal-oriented benchmarking


In this era of increased governmental accountability, there is movement towards "performance-based budgeting," whereby goals are set and funding decisions are based on how effectively specific programs meet the goals. While traditional government budgets focus on inputs (that is, how much money is to be spent on a program or by an agency), new "results-based" budgeting techniques aim to link funding decisions with outcomes, with increased funding going to programs with the best results in meeting the goals of the organization. In order for this to work, performance measurements must be in place, as well as standards to aspire to.

While performance measurement systems are becoming more commonplace, performance measurement, benchmarking and budgeting systems are not yet fully integrated in the public sector. However, according to a recent survey appearing in the Public Administration Review, performance-based budgeting initiatives are underway in 47 out of 50 states.

The Oregon benchmarking initiative has been the most publicized of the state-level goal-oriented benchmarking efforts. Starting in 1989, the Oregon Progress Board developed a comprehensive set of benchmarks encompassing a wide variety of economic, environmental, and quality-of-life measures, eventually amounting to 259 measurable goals. As described in Governing magazine, the goals ranged "from lowering rates of teenage pregnancy to increasing personal income; from boosting the state’s manufacturing base to cutting air and water pollution; from increasing Oregonians’ knowledge of basic geography to lowering their blood pressure."

Oregon’s benchmarking program, known as "Oregon Shines," aimed to (1) measure the state’s current status with regard to the 259 benchmarks; (2) track the state’s progress in improving those outcomes; and (3) guide funding decisions. In 1993, then-Governor Barbara Roberts offered financial incentives to agencies that adopted benchmarks, prompting agencies and local governments to add hundreds of their own. Smaller benchmarking groups sprung up in some locations.

Despite the positive publicity the program received from advocates of "reinventing government," Oregon Shines proved to be unwieldy. As the number of benchmarks proliferated, so did skepticism about the program. Oregon Shines failed to receive legislative renewal in 1995 but Roberts’ successor, Governor John Kitzhaber, convened a task force to examine the state’s benchmarking efforts. The result was "Oregon Shines II," under which the number of benchmarks was pared down to 92, grouped into broad policy areas corresponding to quality employment, safe communities, and a healthy environment. This scaled-down program was approved in the 1997 legis-lative session under a bill that established the Oregon Progress Board as an advisory body. Now the Progress Board is responsible for developing a "strategy for Oregon that addresses the ... needs and aspirations of the people of Oregon" and submitting biennial reports describing Oregon’s progress "based on the specific indicators the board adopts for measuring the attainment of strategic goals."

The relevance of Oregon’s benchmarking efforts to labor organizations was always dubious, and is even more so now. Whether the Progress Board will play a significant role in budgetary decision-making in the future remains to be seen.

Process-oriented benchmarking


While their application in government agencies is new, benchmarking and "best practices" are not new concepts. Private businesses have borrowed ideas from other businesses for a long time, and have been using benchmarking techniques for at least 15 years. The federal government’s National Performance Review (NPR) brought benchmarking to the forefront in the public sector, as the report recommended the adoption of a variety of best practices borrowed from state and local governments and industry. Now state and local governments are examining the use of benchmarking to evaluate and improve the delivery of a variety of services.

In response to a city-wide "competitiveness" initiative oriented toward contracting out, San Diego Local 127 and the city’s wastewater utility management used benchmarking techniques to keep their work in-house. With the assistance of an outside engineering consultant, they compared the processes, procedures, budget and staffing of the utility’s Operation and Management Division with those of other public and private operators. The analysis led to a revised six-year operational plan, including specific quantitative goals relating to budget and staff efficiencies. As long as these goals are met, wastewater operations will remain in-house. To insure against back-sliding by the city, the "Bid-to-Goal" program is in the form of a memorandum of understanding between the union and the city. A portion of the savings resulting from this program will be distributed to employees through gainsharing.

Since benchmarking is new to the public sector, and is typically viewed as an exercise undertaken by managers and consultants, public employee unions have little experience as partners. However, this might change as the trend towards performance measurement continues, since the interests of public employees are directly affected by decisions based on benchmarking studies. While using comparative data for goal-setting or identifying best practices can be beneficial, benchmarking studies can be also be used for other purposes that are detrimental to employees and the union. An unfavorable comparative study could be used to justify contracting-out initiatives, downsizing or demands for concessions. In fact, some studies may be designed with this in mind. Given the ways in which data can be manipulated to arrive at a pre-determined "conclusion," the union has a strong interest in seeing that benchmarking studies are carried out properly and that the results are used constructively.

In order to participate meaningfully in a study, or to evaluate the results of a management or consultant’s report, it is useful to know how an agency should carry out a benchmarking study.

Here are the basic steps:


Determine the purpose and scope of the project. The reasons for the study should be clearly spelled out and the basic parameters of the study need to be established, such as timelines, available funding, the size and composition of the benchmarking team, the role (if any) of consultants, the work processes to be studied, and so forth.

Identify what to benchmark. Once the purpose and scope of the study have been determined, the next step is identifying what should be measured. A variety of performance measures can be used. These include:

  • Inputs: the amount of resources used in delivering the service, such as money, personnel, or equipment.
  • Outcomes: measurable units of the services that are delivered in a given time frame, such as the number of inquiries responded to, number of repairs made and tons of solid waste collected.
  • Effectiveness: indicators that reflect timeliness, compliance, accuracy or satisfaction with services delivered. Examples include error rates, missed stops, response time and volume of complaints.
  • Efficiency: the cost per unit of output, such as cost per ton of solid waste collected or cost per inspection performed.
  • Productivity: the amount of time spent to produce a unit of output, such as the average amount of time to perform an inspection or the average time to process a form.
  • Service demand indicators: when comparing data between different jurisdictions, it is important to keep in mind factors that might help explain differences in performance. Examples include the total population served, average age of building inspected or vehicle serviced, population density and number of businesses requiring permits.

Identify benchmark partners. Generally, there are three categories of benchmarking studies, using three types of partners:

  • In internal benchmarking studies, different divisions or branches within a single organization are compared. This type of study is useful for establishing internal standards and for practicing benchmarking before branching out to other agencies. The benchmarking partners in this type of study would be several offices or branches within a single organization that perform similar work.
  • External benchmarking compares similar agencies in other jurisdictions. These external comparisons involve reviewing how the agency undertaking the study performs in relation to the other agencies. Partners should be jurisdictions similar in size and other characteristics in the same general geographical proximity. Larger jurisdictions will need to move further away to find comparable agencies. With services that are geographically based, such as solid waste collection or emergency medical services, it is also necessary to compare jurisdictions with a similar land area. There is some controversy over whether private agencies that provide the same service as a public agency should be included in a benchmarking study. While this can be dangerous, in sectors where our main competitors are private operations (e.g., wastewater), it may be hard to justify excluding them.
  • Best practices benchmarking involves identifying the best performing agencies, and using them as the basis of comparison. While there is no exhaustive list of "good agencies," anecdotal information usually exists that can be used to identify high performers for many services. Through professional conferences, literature, personal contacts, or even the internet, agencies with a reputation for good service can be found.

Collect internal data. Once the scope of the study, the performance measures and the comparable agencies have been determined, it is time to collect the data. Initial data-gathering involves the collection of baseline data from the agency conducting the study so that current performance levels are known.

Collect external data. To encourage their cooperation, providing data should not require major effort on the part of the surveyed agencies. Special care must be taken to ensure that the measures are calculated similarly across the participants. Also, as an incentive to participate, the results of the survey should be shared with all participants.

Perform "gap analysis." A "performance gap" is the difference between how an agency performs compared to others in the survey. An agency can compare itself to the "average" performer or to the highest performer, depending on its goals. In any case, the comparative data is used to determine the degree to which the agency outperforms or underperforms its counterparts, and to identify issues that warrant further examination to explain these results.

Establish action plans to close gaps. If a significant gap is found with respect to one or more performance measures, then the next steps are determining why, formulating a plan to close the gap, and establishing improvement goals. From the standpoint of a public employee union, the best means for accomplishing this is through a labor/management initiative that aims to borrow, adapt and adopt the "best practices" of the high-performing agencies. An action plan would link proposed changes with agency goals, identify tasks to be accomplished and establish timelines for the completion of these tasks.

Implement an improvement plan and monitor results. Once the "best practices" have been imported, the effect of changes in work processes should be documented to see if they lead to improved performance measures.

Recalibrate benchmarks. Many practitioners see benchmarking as an ongoing process, linked with other mechanisms for continuous improvements in the quality and efficiency of services. Thus, benchmarks should be periodically reviewed to keep up with trends in the field and maintain consistency with agency goals.

Problems and pitfalls

One major concern about benchmarking is that, like all public-sector performance and budgetary issues, it is an inherently political process. The services or partners that elected officials or administrators examine are likely to be politic-ally motivated. There might be pressure for a gap analysis to lead to a pre-determined conclusion, such as building a case for contracting out.

Second, agency managers might be tempted to manipulate data to make themselves look good. This applies not only to the agency conducting the study, but also to agencies participating in the study.

Third, measuring outcomes — especially variables like "customer satisfaction" — is extremely difficult. Given this difficulty, there is an ever-present danger of inappropriate comparisons being made between jurisdictions with different measurement methods.

This leads to the problem of establishing the cause and effect relationship between service delivery methods and performance outcomes. Special attention must be paid to "service demand indicators" (discussed earlier) to ensure that differences in outcomes are indeed the product of different work methods and not other factors, such as characteristics of the population served.

Conclusion


While the practice of benchmarking is not yet widespread in the public sector, it warrants the attention of public employee unions. The state-level, goal-oriented efforts, such as Oregon Shines, have been too far removed from day-to-day operations to be of much relevance to workers and their unions. However, to the extent that states are beginning to experiment with "performance-based budgeting," macro-level performance measurement issues might soon be of more significance.

Benchmarking seems more relevant to unions at the workplace or agency level. While unions should always be on the lookout for the misuse of comparative studies, process-oriented benchmarking can be beneficial. When an agency must analyze its operations, develop performance measures, compare itself to others, and identify best practices, there can be numerous opportunities to improve the service from within. Where the climate is right, involvement in such an effort could be useful in preventing contracting out and increasing employee involvement in decision-making. At any rate, no matter what the intent of management, awareness of benchmarking principles could prove beneficial to a union confronted with a comparative study. Where management’s intent is hostile, it is useful to be able to identify the flaws in the study. Where management’s intent is to improve the delivery of services by public employees, worker and union participation make the study more effective.

When considering whether to become involved in a benchmarking project, these considerations are important for the union:

  • The benchmarking effort should support a labor/management initiative to improve the delivery of public services from within. The union should be skeptical of efforts that have no clear purpose or that might be used to justify privatization or other harmful measures.
  • Benchmarking partners should be chosen with care, to ensure that appropriate comparisons are being made. The union’s input can be crucial in identifying sites where labor/management efforts are underway and/or other "best practices" are in effect. Sites where work is contracted out or that are too dissimilar to allow for meaningful comparison should be avoided.
  • The gap analysis should be undertaken with the understanding that if the results indicate that there is room for improvement, this will be addressed through a labor/management initiative that aims to identify and import "best practices" from leading organizations.

If these conditions are met, then the benchmarking effort can be beneficial for the employees delivering the services that are studied. For more information about benchmarking, contact the AFSCME Department of Research and Collective Bargaining Services at (202) 429-1215.

Print Version