Thursday, April 15, 2010

Health care comparisions statistics: On the dangers of jumping to conclusions

I have been wanting to write an entry about this for a few years actually but just couldn't find the time. Now that I have had the time to spend digging into the data, I'm quite happy that I did because my initial gut feeling about what was reported was actually right; it is often necessary to dig deeper into research and really try to understand what the information that is presented to us really means.

Have a read:

New York Times, November 4, 2007
Beyond Those Health Care Numbers
“STATEMENT 1 The United States has lower life expectancy and higher infant mortality than Canada, which has national health insurance.
The differences between the neighbors are indeed significant. Life expectancy at birth is 2.6 years greater for Canadian men than for American men, and 2.3 years greater for
Canadian women than American women. Infant mortality in the United States is 6.8 per 1,000 live births, versus 5.3 in Canada.
These facts are often taken as evidence for the inadequacy of the American health system. But a recent study by June and Dave O’Neill, economists at Baruch College, from which
these numbers come, shows that the difference in health outcomes has more to do with broader social forces.
For example, Americans are more likely than Canadians to die by accident or by homicide. For men in their 20s, mortality rates are more than 50 percent higher in the United States than in Canada, but the O’Neills show that accidents and homicides account for most of that gap. Maybe these differences have lessons for traffic laws and gun control, but they teach us nothing about our system of health care.”

If you read these paragraphs without thinking about it too much, what do you take from it? Essentially that a very large part of the difference in mortality rates between the U.S. and Canada is due to accidents and homicides, isn't it? Well, if you do, then you would have been misled. Not that I believe that is was intentional but rather, the author has written the piece in such a way that readers can easily overlook what is bone-fide information, and what is undue extrapolation.

The O’Neill report merely points out two factors as evidence that using life expectancy is not the best tool to compare both systems:
Life expectancy shares similar problems as a measure of a country’s quality of health care. It is influenced by infant mortality and at older ages and does not delineate between causes of death susceptible to improved medical treatment ad those that are not (deaths from homicide, auto and other accidents).” (O'NEILL, 2007, p7)
That is the main point, and the accidents/homicides is merely an example of how the life expectancy statistics is the not a great tool to be used as a comparator of medical system outcomes.
Actually, once one reads Mr. Mankiw’s carefully, the conclusion is that “The bottom line is that many statistics on health outcomes say little about our system of health care”. But in the process, he highlights facts that lead the reader towards the wrong logical path.

What bothered me is that the O'Neill report got quoted in a few blogs as evidence that most of the gap between life expectancy in the U.S. and Canada is due to accidents and homicides (or, in a broader sense, all non-medical events causing death). But beyond the fact that the report doesn't say that at all, I thought it would be worth taking some time to dig into the data that the O'Neills are using to see if I can start from where they left off and see if the non-medical events can explain the large mortality gap.

(Note: Click on the charts to get larger/readable versions)
Table 4 (above) is the principal data piece in the O'Neill report which shows evidence that at least a portion of the gap has a non-medical cause.
Something that peeked my curiosity is the fact that the chart didn't show the weight of each age-group in the overall mortality numbers. Because, obviously, as the non-medical deaths happen at a younger age, they would significantly drag average death years down, but only if the number of non-medical death relative to the total number of deaths is high. The data actually shows us that it isn't.

The next two charts were constructed by me based on the same data the O'Neill used. The first chart contains data for Canada and the second, for the U.S.

The age groups are on the left hand side. The 'Age group used for average' is the age used for computation of the overall average age of death (see my notes for additional details).
Next column lists the total population in each age group. Note that this column is there for your reference only as I do not use it for any calculations. The other columns are self-explanatory. The 'Unadjusted age total' is simply the number of death in the age group times age group age used for average, used for the sake of final average computation. The 'adjusted Nb of death' and 'adjusted age' remove the non-medical events to compute an 'adjusted average death age'.

With a more detailed chart such as this one, it is quite easy to see that although accidents might explain a large portion of the gap for the 20-40 age group, the proportion of those death relative to the total number of deaths is minimal, and therefore has not as great an impact of average age of death as one could be led to believe based on Table 4.

But, by how much does it really impact the average age of death?
Including all causes of death
, I arrive at an average age of death of 72.29 years for the U.S. If we exclude all non-medical causes of death and re-calculate the average age of death, this number is now 74.17 years. The gain is therefore 1.88 years. This is quite significant. It means that if the U.S. could have figured a way to avoid all non-medical fatalities in 2004, the average age of death could have been 1.88 years higher.

Doing the same calculation for Canada, I arrive at 74.15 years pre-adjustment, and 75.46 years with the non-medical causes removed. The gain here is 1.31 years.

Therefore, the 1.86 years delta between the U.S. and Canada unadjusted average death years, falls down to 1.29 years once all non-medical causes have been removed from the data.
This essentially suggests that 31%, or one-third of the difference between the average age of death in Canada versus the U.S. can be explained by non-medical events. Quite significant but not quite sufficient to state that non-medical events explain why there's such a large gap in life expectancy between the two countries.

The O'Neill study also hints that higher obesity in the U.S. might explain the remainder of the difference. In their study, O'Neill used the 'diseases of heart' (table 4) from the data to show that they were a significant percentage of the gap in the older age groups.

However, when I remove the 'Diseases of heart' from my adjusted average age of death calculations (in addition to removing the non-medical causes), I obtain the following interesting data: the non adjusted U.S vs Canada still remains 1.86 years (obviously) but the adjusted age is now 1.65 years, a difference of just 11%.

To me, this highlights the danger of drawing conclusions based on multi-variable inputs; in this case, while obesity is undeniably a factor in the prevalence of heart conditions, it also will impact other diseases. But the health system can also have impacts on obesity through prevention and education. In fact, I'm sure we could find a thousand ways how to either use the medical mortality data to have stats say one thing or another. There are many variables interacting which can be hard to isolate and therefore, it makes any conclusion hazardous, while in the case of non-medical events, it is quite clear-cut: the deaths occurred outside of the system altogether.

Now, one last thing. Since the matter of infant mortality has also been brought up in the report, let's see what my model says about this: If we remove infant deaths and non-medical deaths? (by completely removing all 0-1 age group deaths in both Canada and the U.S.), the unadjusted death death delta (Canada minus U.S.) is 1.5979. The adjusted delta is 1.055, or, a reduction of the Canada/U.S. mortality delta of 34%
A large chunk for sure, but still not the cause. Again here, the reason is that the total number of death in the 0-1 age group is not large enough as a proportion of total death to have a truly game-changing impact on the results.

In conclusion, I would say that for the same reason that using 'Diseases of heart' as a tool to explain the differences in health outcome between the U.S. and Canada, using the life expectancy as the sole indicator and evidence of the 'superiority' of the Canadian system is very risky. However, the point of this blog entry was to dispel the myth that the higher rates of accidents and homicides in the U.S. could explain the large gap in mortality rates. The 2004 data referenced by the O'Neill report actually doesn't support that assertion. Nor does it support the assertion that 'infant mortality' nor 'Diseases of the heart' account for all of the gap. What the O'Neill report says is that the life expectancy statistics are an inadequate tool to compare systems. But although life expectancy is a weak measure of comparison of health care system, the large mortality gap between the U.S. and Canada that remains, even once adjusted to remove non-medical events, certainly highlights the need to identify the causes of this gap.

While the data cannot provide evidence that the health delivery system is the cause, the 2004 data also cannot exclude this explanation.

  • 94 was used for the 90+ average as it is roughly the calculated median age of death based on the 90+ population mortality figures. From mortfinal2004_worktableipt1.pdf, the U.S. figures were: 90-94 years...... 227,661. 95-99 years...... 87,446. 100 years & over. 18,526. I tried some other figures to make sure I wasn't introducing too much of a bias but varying this figure had little impact on the results.
  • 0.1 has been used for the '0-1 year' mortality population as opposed to the average which would be 0.5 as most infant mortality happens at birth. 0.1 seems good enough a proxy. The percentage of total death attributable to that age group is relatively small (1.2%) which indicates that changing the average age group figure will not have a significant impact on the results.
  • As the U.S. data was not readily consumable as the Canadian one's, I had to merge from multiple files to get the absolute mortality numbers and therefore, some absolute numbers are reconstituted from rates. They are highlighted in grey in the charts. The error margin this process introduces is not significant.
  • The areas highlighted in cyan are reconstituted by totaling their table sub-components as the consolidated number was not available. However, some tables were not available from the US data. To assess the significance of those missing data pieces, I totaled the sub-components for the consolidated number where data was available to the reconstituted one with some tables missing. The difference was less than 0.25% and I therefore concluded that the partial figures were within a reasonable error margin.
  • Discrepancies between my age figures and official life expectancy ones are due to the fact that my numbers are not actuarial life expectancy figures but rather, figures derived from death events and therefore, do not represent the life expectancy at birth of an individual born in 2004 but rather, the average lifespan of individuals in 2004.

List of sources:
  • CDC/NCHS, National Vital Statistics System: LCWK1. Deaths, percent of total deaths, and death rates for the 15 leading causes of death in 5-year age groups, by race and sex: United States, 2004
  • Worktable I. Deaths from each cause, by 5-year age groups, race, and sex: United States, 2004
  • Statistics Canada, Catalogue 84F0209XIE, "Mortality, Summary List of Causes", 2004. (Stats used start at page 64 in the PDF file)
  • HEALTH STATUS, HEALTH CARE AND INEQUALITY:CANADA VS. THE U.S., NBER Working Paper 13429, June E. O'Neill and Dave M. O'Neill, 2007