In Tuesday’s Wall Street Journal there was an article on heart treatments which have begun to be questioned because of problems that have arisen. It points out that this “reflect[s] a persistent phenomenon in medicine where doctors and patients embrace new technology only to find that it may not be good medicine once exposed to rigorous testing.”

This statement reminded me of the similar case with the investment world, where certain models are adopted without sufficient testing. Just think about the model AIG used to decide which credit default swap contracts to enter into; the model had been developed by a Yale professor and apparently had a known shortcoming, but this didn’t stop AIG from taking the risk side of way too many such instruments. Other models, too, that hadn’t been properly vetted, through rigorous testing, failed once the sub-prime mortgage crisis hit.

Our industry tends to love complex models, especially when they’ve been developed by someone whose name ends with “PhD.” I will credit Nassim Taleb for pointing out some of these problems (The Black Swan), though at times  I find his rants a bit excessive. But we can’t question the shortcomings of many of the risk models when put to the tests of recent years.

When I was in the military we trained under situations that were designed to match very closely what we might encounter in battle; unfortunately, risk models aren’t always put through such tests, perhaps because the future is so uncertain; thus Taleb’s suggestion not even to bother to try to use any ex ante measures. But his advice and analysis will no doubt have little impact on what we do.