Posts Tagged ‘transportation’

Cash for Clunkers: Measuring Performance

May 5, 2010

The GAO recently released a report reviewing the impact of Cash for Clunkers. Beyond good information about that program, this report impressively discusses general questions regarding stimulus implementation and performance measurement. Here’s the conclusion of the GAO’s report: (Note: NHTSA refers to the National Highway Traffic Safety Administration; the emphasis is ours.)

The implementation and results of the CARS program offer potential lessons learned for future vehicle retirement or similar incentive programs. First, the program produced economic and environmental benefits, achieving its broad objectives. However, the extent of the program’s effects is uncertain.

Second, before a program is underway, steps must be taken to determine what impacts are going to be measured and what data will be required to measure them. Moreover, steps must be taken to ensure that the data are reliable. NHTSA relied heavily on the consumer survey for data on the economic and environmental benefits of the CARS program. However, there is a potential risk to the reliability of estimates based on this survey data, because NHTSA did not follow some generally accepted survey design and implementation practices, largely because it had limited time to establish and administer the program.

Finally, given the number of stakeholders that are financially affected by the auto industry, it would be important to collect and consider information on how a future program would affect these stakeholders and take mitigating actions, as appropriate.

It’s hard not to get the sense that the fly-by-the-seat-of-your-pants nature of Cash for Clunkers made it, at best, difficult to ascertain the real benefits of the program. Given that some facets of the stimulus are designed specifically to spur data collection and use (as in the education and health care fields), we’re often surprised by the data shortcomings in other  stimulus spending as well as the lack of strong performance management thinking.

We appreciate that officials wanted to execute the program quickly, but if you can’t evaluate how beneficial a program is, how can you know whether or not it was wise to do it in the first place, much less do it quickly? How can you learn how to do it better next time?

We should also note that this second takeaway—“ before a program is underway, steps must be taken to determine what impacts are going to be measured and what data will be required to measure them”—sounds to us like some of the issues we’re hearing with respect to other stimulus projects. In several of the stimulus program measurement conversations we’ve had, jobs and spending seem to be the only real measures of success for certain programs.

Finally, anyone interested in designing and conducting valid surveys should definitely read Appendix II of this report, in which the GAO analyzes the NHTSA’s consumer survey of Cash for Clunkers, pointing out its weaknesses.

Advertisements

Green Means Go

April 30, 2010

We can’t resist pointing out a management tactic in New Hampshire that we spotted in the CSG report we discussed yesterday:

One key procedural change at the state agency level that may have had an impact on greasing the wheels involved a simple piece of paper to green-light stimulus funded projects. Stimulus director Bud Fitch, the state’s deputy attorney general, required every stimulus project to have a green cover sheet that signified it was a “Go.”

“What the green paper does is it says when you come in in the morning and your routine stack is there, if halfway down the pile is something with a green paper, pull that out, do it immediately, send it on to the next step, where normally it might not be gotten to for two or three days or sometimes even a week,” Fitch said.

Sounds cool. But are other states still using stacks of paper, instead of emails or databases?

“Shovel Ready or Not?”

April 29, 2010

“Because states had to choose projects quickly and in many cases chose them well before the Recovery Act was even written, many either didn’t consider the distressed areas criteria until late in the planning process or used their own criteria rather than the federal requirements.”

That powerful comment about the distribution of transportation funds, comes from a recent Council of State Governments report, “Shovel Ready or Not?” The report examines four fundamental facets of highway stimulus spending: the speed with which states spent the money; where those dollars went; how the money was accounted for; and the impact of the spending on jobs and other measures. We recommend the entire report, but nuggets like the one quoted above were particularly interesting.

For example, the report notes that some high-unemployment states received less funding per capita than other states with lower unemployment. That was a direct result of the fact that ARRA relied on existing federal programs and formulas for its allocations.

There’s also some fascinating commentary on the tension between spending speedily and spending well. States that were hard-pressed to get all the funds expended quickly, as we’ve noted in earlier posts, weren’t necessarily dawdlers, but in many cases were using the funds to tackle larger, longer-term projects than the states which spent more quickly. The CSG report highlights Florida’s efforts to build interchanges in Miami and Tampa as an example of those bigger projects.

The CSG also discusses whether a less than proportionate share of money, measured by population or economic impact went to urban areas. It is a tricky question that gets to the challenge of prioritizing the nation’s transportation needs. The short answer seems to be that less money did go to cities. The longer answer is that, in many cases, it might have been impractical or undesirable to focus more money on cities at the expense of other distressed areas.

Finally, CSG underscores the difficulty of assessing the impact of the stimulus spending, not only with respect to complex job counts, but the even greater challenge of figuring out the long-term transportation benefits gained for the nation. As it currently stands, neither the states nor the federal government are able to provide a full performance impact of ARRA spending. Will they ever? Or will the stimulus’s legacy be written, as this report suggests, “state by state, community by community, project by project and job by job?”

Cash for Clunkers: Digging Deeper

April 15, 2010

The debate over the economic benefits of last summer’s “Cash for Clunkers” program continues. Recently, the White House blog and Edmunds.com exchanged salvos in the argument.  But the economic benefits are only one part of the question. Examining performance metrics other than economic impact — such as the environmental benefits — sheds additional light on what exactly Cash for Clunkers accomplished.

Happily, fedgazette editor Ronald A. Wirtz  has a piece on the website of the Federal Reserve Bank of Minneapolis that digs  into data regarding some of the fuel efficiency savings that the country could gain from Cash for Clunkers.

Wirtz looked at the program in Minnesota, Montana, North Dakota, South Dakota, Wisconsin and the Upper Peninsula of Michigan. He found that gas efficiency gains were genuine, but the devil was in the details. Explained Wirtz:

“Average fuel efficiency between trade-ins and newly purchased vehicles rose about 50 percent, from roughly 15.5 miles per gallon to 24.

“But that covers up a lot of variation, part of which suggests a sop to owners of older, typically larger vehicles who used the opportunity to upgrade to something with only marginally better gas mileage.

“For example, about one quarter of the new vehicles purchased through the clunker program in the Upper Peninsula and the Dakotas had fuel efficiency gains of four miles per gallon or less. In many cases, older trucks and SUVs were simply traded in for newer but only marginally more fuel-efficient versions. Only 10 percent of all vehicles bought in the district under the program got 30 mpg or better.”  (see Chart 1).

The gains shouldn’t  be brushed aside, of course. As Wirtz is careful to note, even seemingly small fuel efficiency upgrades can save enormous amounts of fuel, which becomes more evident when using “Gallons per Thousand Miles” rather than “Miles per Gallon.”

Is Faster Better?

April 13, 2010

One of the most significant measures currently being applied to stimulus spending is the speed with which dollars are put to work. We get that. It doesn’t stimulate the economy very much today, to spend money next year. But there’s a delicate balance here. The Recovery Act also suggests that states should be looking toward the long-term benefits of the dollars spent. And that’s not easy to do, if the primary goal is to crank the dollars out as quickly a possible.

Former Governor Tim Kaine

Perhaps the best example of this tension is the October back and forth between Congressman James Oberstar, Chairman of the U.S. House Committee on Transportation and Infrastructure and former Governor of Virginia Tim Kaine. Oberstar complained that Virginia was the slowest of the states in getting its transportation funding out into the field. Kaine responded that his state was different from others, in that it was using its ARRA money to create new projects, not just to keep old ones going. What’s more, he pointed to the careful thoughtful process Virginia uses to make sure its infrastructure dollars are well spent.

We asked some transportation officials in Virginia about the exchange, and were struck by something that chief financial officer Reta Busher said regarding the challenges of transportation management, “The short term nature of what we are dealing with is troublesome in a business where a project can take anywhere from 18 months to 5 years.”

We think there’s an important point in having a range of different measures. If you concentrate too much on speed, then quality may be sacrificed, as we wrote in a column in Governing magazine. With a range of measures, no one element gets an inordinate or inappropriate amount of focus. It’s fine to measure speed, but that information also has to be weighted against how well the services are working — or what the dollars are accomplishing — in the long run.

How much does it cost?

March 18, 2010

We don’t want to get into the politics of the debate about road signs that advertise the use of stimulus dollars, but it does point out a significant shortcoming in government generally. That’s the difficulty in coming up with good reliable figures about what things cost.

Last month,  PolitiFact looked at the controversy over road signs, following up on comments by former Gov. Sarah Palin that one state spent one million dollars to post signs that advertised where stimulus money was being spent. PolitiFact did some research and concluded that the state was Ohio, but that the one million dollar figure was suspect.

It was based on the probably erroneous idea that each sign was about $3000.  But the Ohio Department of Transportation told PolitiFact that there was no way to come up with the cost of the signs.

Why? Because the costs of all construction signs in a construction zone were rolled together. PolitiFact pointed to inquiries in other states by the Associated Press, which  uncovered figures that were substantially lower than the $3000 estimate — Michigan estimated a sign costs $400 to $500. Illinois put the amount at $1000 ($300 for the sign and another $700 for labor). Colorado officials put the labor plus materials cost at between $750 and $1,200.

Of course this is small change, but wouldn’t it be useful for a state like Ohio to be able to clearly answer questions about what things cost — even if they’re small things? That would not only lead to useful cost comparisons, but help ensure that debates on spending were based on accurate information.

Keeping up with the GAO

March 8, 2010
Keeping up with the Government Accountability Office’s reports about the Recovery Act is a pretty demanding undertaking. But we’re going to try to do so on your behalf, and offer you summaries of the most important reports as they come out (as well as links to others).
The most comprehensive new report was the GAO’s major 174-page look at the Recovery Act “One Year Later”.  It gives a very clear picture of the pace at which dollars are being distributed, the vast array of oversight activities and a variety of agency specific topics (like the wage issues that have stalled the delivery of weatherization funds).
A few of its important points:

* Data quality. For individuals who have been following the fracas in the press over the last six months about data issues — like jobs reported for Congressional Districts that don’t exist, double counted jobs, incorrect dollar figures, etc. — the GAO report is surprisingly comforting. Lots of improvements were put in place for the second reporting period, which ended on December 31, 2009. New simplified job data have helped increase accuracy, along with ways to flag potential errors as data is input. (For example, the system will now stop people from proceeding if they input a Congressional District that doesn’t match their zip code.)

* Maintenance of Effort. One of the trickier issues pertaining to the Recovery Act are the “maintenance of effort” requirements, particularly in transportation.  A continuing budget crisis, coupled with legislative cuts and changes, has made it difficult for many states to certify to DOT that the stimulus money will actually be adding to previously planned spending. The Federal Highway Administration is expecting lots of states to send in revised certifications this week, according to the GAO. Whether or not states meet maintenance of effort requirements is likely to be an ongoing question.

* Capacity. As states and local governments grapple with all their responsibilities for reporting Recovery Act information, capacity questions continue to loom large. This problem will become increasingly acute as beleaguered budgets necessitate layoffs and temporary furloughs. For example: The GAO reports that capacity has been a barrier in housing agencies, causing some to bypass applying for competitive grants. This is particularly a problem for smaller agencies. Capacity issues also create obstacles for small agencies in particular in dealing with the distribution of weatherization dollars and the administrative requirements

Here are the two other GAO reports that came out last week.

Recovery Act: California’s Use of Funds and Efforts to Ensure Accountability
Recovery Act: Factors Affecting the Department of Energy’s Program Implementation