TSG recently completed its most recent survey, which deals with performance attribution. Jed Schneider, CIPM, FRM discussed many of the results at a luncheon held in NYC, where we learned that, as with the prior three editions of this research, most folks prefer transaction-based attribution, though roughly half use holdings-based. The probable reasons as to why the contradiction are interesting, but not part of this post.
For several years, I have attempted to encourage a proper and unbiased evaluation of the two methods, to determine the true differences, and whether there is a point when one cannot justify using the holdings-based approach; a point where the use of transaction-based attribution is a “must.” But, no one chose to do the research, so I did. And I discussed some of my preliminary findings at last year’s PMAR (Performance Measurement, Attribution & Risk) conferences, and will discuss further findings at this year’s events.
Today, I merely want to discuss the three problems with the holdings-based approach. And, in “David Letterman” style, I will do it in reverse order of significance.
Number 3: we will usually have a residual. For many readers, this will seem odd to have been placed third, because surely the residual is the main problem with holdings-based, but I’d say it is not. A “residual” is a non-zero amount which reflects the inability to fully reconcile to the excess return. Recall that with relative attribution, our goal is to completely account for the excess return. However, with holdings-based models we often (actually, more like usually) can’t do this, but have a “residual,” meaning we don’t fully account for the excess return. In reality, it’s worse than that.
Number 2: getting the proportions wrong. Here I mean that the amount that is assigned to the different effects may, in realty, be incorrect. We may have too much or too little assigned to allocation, for example. Thus, it’s contribution to the excess return is over or understated. This goes beyond merely not reconciling to the excess return; rather, the numbers that are produced can be allocated in a manner which doesn’t properly align with reality; their proportions are incorrect.
Number 1: having the wrong signs. To me, this is the major problem. What do I mean? Well, with holdings based attribution, we may be showing a positive selection effect, but in reality, it’s negative! And so, instead of saying “great job!” we should be saying “you’ve got to do better!” And the problem is, you won’t know this. You’ll see a positive selection effect and conclude that those decisions were good ones that contributed to the excess return, when in reality the decisions hurt your performance!
This third problem (#1, actually) means that the results will be misleading, spurious, invalid. What’s the risk of this happening? I think you’ll be surprised.
An article will be forthcoming which will provide additional details on my research. Suffice it to say, the results are fairly startling, and should encourage most users of holdings-based models to seriously consider switching.
p.s., for more details on the attribution survey, contact Patrick Fowler.
p.p.s., our next survey deals with the GIPS(R) Standards (Global Investment Performance Standards). Please join in! It will begin this summer.