I’m not disagreeing with the usefulness of such a graph, and my comment isn’t related to the complexities and difficulties of creating such a metric.

I’m saying that whatever the metric is, it is combining costs and benefits (risks and rewards, etc.) into a single number, and that the natural way to express this is with a metric that centers on 0, with values above that being, on balance, helpful, and values below that, harmful. Expressing this as a percentage scale with a 50% cutoff point seems arbitrary and non-standard for a cost / benefit analysis.

My other point is that the word “effectiveness” is misleading. In the everyday use of this word, something that is a little bit effective is better that nothing at all – but that’s not true here, where values below 50% effectiveness are worse than nothing.

Again, I agree with your main point, but a confusing or misleading graph can be worse than none at all.

What would you use instead? I chose “effectiveness” here and a 50% cut off because it’s a simple concept that would convey the gist w/o the burden of a more accurate but harder to describe metric. This is a hypothetical graph after all. In reality I expect the task force looked at data from a variety of sources not necessarily directly compatible with each other. Risk of false positives times risks from unnecessary biopsies could be compared to risks from either missing or diagnosing cancer outside of a routine mammogram. Those risks could be portrayed in a compatible way, but I suspect that the actual evaluation methodology might have included other features not so easily comparable. (This might be why the task force didn’t create such a chart, it was hard, but IMHO would have been immensely useful even if approximate.)

Additionally, graphing results like this might have shed light on how effective the current “gold standard” method is. One thing that struck me in the task force’s recommendation is that mammography is a pretty lousy screening technique for women under 50. (A graph might have made this clearer.) I’m puzzled as to why that aspect wasn’t discussed more heavily and why no one advocated more research dollars into creating a better screening technique for younger women. That might have been a more accurate headline, “Task Force Finds Mammograms Less Effective Than Thought For Women Between 40 And 50″

It implies that for any population group the test is at least somewhat effective – after all, something 10% effective is better than nothing, right? In fact, for pretty much any medical procedure – including mammography – there are measurable risks that make a procedure actually harmful for certain population groups.

Your discussion of the amniocentesis test makes this clear, but it’s a mistake to refer to the balancing point as being “%50 effective” – that should be the “0%” point. Values above that are helpful, and values below that are harmful. Which is very different from calling something “40% effective”.