Lies and Social Desirability Bias

Regarding the low faith they have in survey responses and other evidence based upon anything other than what people actually do, Bryan Caplan laments many economists' rejection of "all-pervasive testimony on lame methodological grounds" Caplan argues that words ought to be trusted far more often, especially when there is no reason to be wary of social desirability bias. I am not sure I agree with Caplan that economists have an unsound attitude to words (I personally trust some surveys), but his blog post reminds me of an interesting approach to this problem, first developed by Judith Droitcourt Miller in her mid-1980's PhD dissertation at George Washington University.

 

In her doctoral thesis, A New Survey Technique for Studying Deviant Behavior, Miller outlines a very clever way of ascertaining the prevalence of lies to protect sensitive information by simple survey methods. Suppose one wants to find out how much one can trust survey responses on, say, how much people recycle. I believe there is widespread pressure on most people to say that they recycle, but it is also quite clear that recycling is a deeply unpleasant activity. As the Simpsons' Mr Burns sarcastically commented on the phenomenon, "Yes, I can't wait to start pawing through my garbage like some starving raccoon".

 

Miller's proposal is to take two groups believed to be statistically the same. Call them Groups A and B and send them  questionnaires containing a series of mainly rather harmless statements, as well as the statement of interest. Group A's form might ask respondents to indicate the number of statements with which they agree out of a list such as the following:

• I remember what I was doing when I learnt of the atrocities of 11th September 2001.
• I prefer the beach to holidaying in cities.
• I always recycle my refuse.
• My favourite colour is blue.
• My favourite film was released in the 1990's.

 

Group B gets the same statements, but the recycling one is separated from the others, requiring its own "yes" or "no" answer; the other statements are grouped together and respondents indicate the number of them with which they agree. One might in addition want to vary the order of the statements, to ease worries that a particular place among the statements will draw extra attention.

 

The effect of social desirability bias can be estimated from looking at the responses from the two groups. If recycling is privately unpleasant and in high regard socially, one should expect social desirability bias to manifest itself in widespread agreement with the single statement among Group B members. At the same time, the average number of statements with which respondents agree should be lower among members of Group A. Because in Group A's survey, disagreeing with the statement that one is an avid recycler does not stand out as much.

 

This is a very clever strategy for getting at lies in survey data. I wish it were original with me.