TECHNICAL FEATURE:

Fuel Economy And Bad Science

It's not really good form for one magazine to criticise another in public, but sometimes the temptation is too great to withstand. In this case the object of our concern is a recently-published article demonstrating that eight cars promoted as being environmentally friendly actually turn out to use more fuel and emit more CO2 than the official figures suggest.

The consequence of the increased fuel consumption is obvious. As for the CO2 emissions, each of these cars is rated in VED Band B, while the results of the tests suggest that they should all be in Band C, implying car tax payments of £120 per year rather than the £35 which owners would actually be charged.

The basis of the article is that the official fuel economy and CO2 emissions tests (which are actually the same thing) do not give a realistic idea of what will happen in real life. This is not the first time that this point has been made, though in fairness the magazine made a much better job of demonstrating it than the BBC's Watchdog programme did a few years ago.

In EU countries, the official test is as we described it in our fuel economy testing article published in 2003. It is conducted in a laboratory within very tight limits (if the procedure falls outside those limits the test is abandoned), and it's based on an analysis of the difference between the quality of the air going into the engine and that coming out of the exhaust pipe. Since a certain amount of fuel useage can be precisely related to a given change in air quality, this is actually a more accurate method than measuring how much petrol or diesel is fed into the engine, which has its own problems.

Watchdog conducted its own test by putting a gallon of fuel into the tank of three cars and driving them until they stopped. This was an insane idea for several reasons, not least that the full gallon could not possibly have been used (our 2003 article pointed out many other flaws). The test procedure used for the more recent magazine article had less of a Monty Python feel to it, though it did involve directly measuring quantities of fuel.

It also diverged from the official test in another, very significant way. The official test is conducted over two "virtual" routes, while the magazine test involved five real-world road routes. And already we've come to the point where the whole exercise can be dismissed as a waste of time.

There is no point in saying that a car's true fuel consumption is 54.5mpg rather than the claimed 61.4mpg when you're not putting it through the same test. It is completely ludicrous, and here's why.

Imagine that you want to measure the average height of a dozen people. You insist that they are all barefoot, and you measure them from the ground to the top of their skulls. You come up with an average height of five foot six.

I measure the same group, but I don't make any restriction on footwear. Some are still barefoot, but some are wearing trainers, some are in wellington boots and some are in high heels. I also measure them to the top of their hairlines, which is significant because two of them are bald and one has a Mohican. The average figure I get is five foot nine.

Which of us is right? We're both right, but our results can't be compared. We performed different tests. My result doesn't invalidate yours and yours doesn't invalidate mine. Just don't go trying to mix them up, as the magazine did by comparing its own results to that of the official test.

One thing you'll have noticed about the average height example is that your test is a lot more sensible than mine is. Measuring a barefoot person to the top of his or her skull is repeatable. If you did it every day for a year you'll still come up with a result of five foot six. My test could produce a different result every time: one of the barefoot people might wear shoes the following day, the girl in high heels (I'm assuming it's a girl) might bring in another pair, and the guy with the Mohican might get it shaved off. My test can not safely be described as repeatable.

Nor can the economy test performed by the magazine. Oh, they did their best with this, avoiding other traffic by driving the cars early in the morning and so on, but in the real world it isn't possible to avoid changes in conditions which would have a significant effect on the result (and by significant I mean to the extent of 0.1mpg, the smallest increment measured).

And even if it were, what relevance does this have? You would have a figure which applied to those five routes driven by those drivers in those conditions. It would not apply in any other conditions (or, if so, only coincidentally). Any of the following will have an effect on the car's economy: different route, different speed, different driver, different number of passengers, different amount of luggage, different weather, different traffic conditions, different amounts of roadworks, different number of windows open, different use of the lights, wipers, radio, air-conditioning, satellite navigation.

There are other elements too, but just consider those ones. Try to work out the number of options in there (how many billion separate routes can be driven in the UK, for example?) and then don't just add them all together - multiply them together to find just a fraction of the possible fuel consumption figures you could get for your car.

One of the conclusions of the article is this: "If the official consumption figures say you'll get 60mpg, you should expect to get 60mpg." No, you shouldn't. It is a wild, crazy assumption. It is madness. There is no possible test which could accurately predict the fuel economy you will get from your car in this way. Suggesting otherwise is a) misguided, b) attention-seeking or c) just plain daft. Take your pick.

The official economy test is flawed. Any car can be made to use more or less fuel and emit more or less CO2 than the figures suggest. Sometimes the laboratory results can be far removed from reality - in a car with a turbocharged engine, for example, the turbo will hardly spin at all during the whole process, leading to a falsely impressive figure which bears little resemblance to what happens when you put your foot down (which largely explains the increasing tendency of manufacturers to replace large-capacity, naturally-aspirated engines with small-capacity turbocharged ones, as explained in our feature, The Rise Of The Small Engine).

But although the official test does not provide much accurate information - though no less than any other conceivable test could do - it is at least repeatable. Put the same car through it twice and you're extremely likely to get the same result; and it does provide as good a comparison as you're going to get between two different vehicles. Improving on it would be very difficult, if not impossible, and if you want to try inventing one, employing bad science is not the way to go about it.