Archives for posts with tag: surveys

Last year I heard some atheists say that Christians are more charitable than secularists. My gut told me that this could not be accurate, so I investigated. This investigation resulted in two blog posts: The Myth of Christian Charity, part 1 and part 2. After these were published, Gregory Paul alerted me to a book published in 2006, Who Really Cares, in which Arthur C. Brooks makes extremely strong claims about religion and generosity. Late in 2010 there appeared another book, Robert Putnam and David Campbell’s American Grace, which makes similar claims. I decided that, since the findings in these high-profile books were supposedly based on the statistical analysis of large-scale survey data—that is, they looked like science—they should be rebutted (if they are false) in the scientific literature. I did a ton of research, verified that they are false, and wrote a paper, which is now under review by a scientific journal. While we wait to hear about the paper, here is a layperson’s summary.



Religious representatives have always claimed that religion is a good thing, and that its many benefits include an improvement in morality. Religious people, they say, are kinder than the unchurched. Repetitions of this claim have embedded the phrase “Christian charity” in our language.

In recent years, professional scholars have reported finding empirical support for this traditional claim. Most prominent among these are Arthur C. Brooks (Who Really Cares? Who Gives, Who Doesn’t, and Why It Matters, 2006), and Robert Putnam and David Campbell (American Grace: How Religion Divides and Unites Us, 2010). In this post I will call these three men “the traditionalists.”

“When it comes to charity,” says Brooks, “America is two nations—one charitable, the other uncharitable”; compared to the non-religious, “religious people are, inarguably, more charitable in every measurable way.” (Emphasis in the original.) Putnam and Campbell vigorously agree:

Some Americans are more generous than others. … In particular, religiously observant Americans are more generous with time and treasure than demographically secular Americans. … The pattern is so robust that evidence of it can be found in virtually every major national survey of American religious and social behavior. Any way you slice it, religious people are simply more generous.

This would be an astonishing result, a stunning vindication for advocates of religion everywhere, if it were valid. But it is not.

Methodological Challenges

The traditional hypothesis is that religiosity fosters generosity. To support this claim scientifically, we would have to (1) measure many people’s religiosity and generosity, (2) show that, on average, those people who have more of the former also have more of the latter, and (3) show that the former causes the latter. (The claim is that being religious makes people generous, rather than that being generous makes people religious, or that some third factor causes the first two.) The traditionalists fail to accomplish all three of these goals.

The business of sociology depends almost exclusively on surveys. Rather than observing people’s thoughts and feelings—which is impossible—the sociologist surveys them about their thoughts and feelings. Behavior, too, is generally inquired about rather than observed. But there is a problem: survey respondents tend to give answers that are flattering rather than true. This is called social desirability bias. In any community, behaviors considered good will be over-reported, and those considered bad will be under-reported. The problem is especially severe with behaviors to which strong norms are attached. Being generous and being religious are ideal exemplars of this category.

“Generosity can be measured most simply by measuring gifts of time and money,” write Putnam and Campbell. But surveys do not measure such gifts—they measure reports of such gifts. And these reports are anonymous, unverified, and subject to strong social pressures.

For a measure of religiosity, most surveys use frequency of church attendance. That one goes to services regularly is easy to say, hard to verify, and subject to strong community norms. In decades of surveys, 40 percent of Americans have reliably reported going to church pretty much every Sunday. It turns out that about half of them are liars. In the 90s, scientists found ways to count how many people were really attending. The number is much closer to 20 percent than to 40 percent. (See for example C. Kirk Hadaway and Penny Long Marler, “How Many Americans Attend Worship Each Week? An Alternative Approach to Measurement,” Journal for the Scientific Study of Religion 44(3):307-322 [2005].)

The traditionalists cite page after page of statistics showing a strong positive correlation between religiosity and generosity. But this tells us nothing of interest, because both attributes are likely to be over-reported, and in the case of generosity we don’t know by how much. Neither book mentions or addresses this enormous methodological problem.

In the measurement of generosity a more technical problem appears. Throughout both these books (and in the sociology-of-religion literature generally), the words generosity and charity are used interchangeably (as synonyms for altruism, benevolence, compassion, and so on). But charity has an additional sense. In the U.S. tax code, and in standard English, a charity is a nonprofit corporation; donations to such organizations are also called charity—a term easily confused with generosity.

Note, however, that generosity is not the same thing as donating to a nonprofit organization. These are different concepts. The first means, voluntarily helping others at some cost to oneself. The second means, giving money to an organization that qualifies as “not for profit” under the U.S. tax code.

Donations to one’s own church are tax-deductible. But that does not make them charitable, in the older sense of the word. They are membership dues for a social club. They do not benefit the wider community, as would, for example, donations to the Red Cross. They certainly should not be used as a proxy for the noble attribute we call generosity. Yet, this is exactly what Brooks and Putnam-and-Campbell do. In these books, the words charity and generosity are used to mean people’s (self-reported) charitable donations, including money given to their own church. Thus the measurement of generosity, which was already distorted by social desirability bias, is further distorted by a confusion of terminology.

Another technical issue relates to the measurement of religiosity. Is church attendance a good proxy by which to measure how religious people are? Perhaps not, if people report twice as much of it as they should. What else might we use? We could try frequency of prayer, or of Bible study, or how “certain” one is about the existence of God. And all these would be self-reports—but there is deeper problem here. How could we tell which of these things is more appropriate? In other words, what is religiosity?

Well, it is a matter of opinion. To verify this, notice that for any behavior (or attitude or quality) one party chooses as the epitome of religiosity, another party can say, “But that’s not really being religious,” and name some other behavior (or attitude or quality). There is no independent standard to which such claims be compared. If a man says, “I am highly religious,” nothing anyone else might say can prove him wrong. Even if they point out that he has previously described himself as an atheist, he can still say, for example, “I attend my wife’s church, and act as a deacon at the Sunday school”—or, “I have a very spiritual attitude toward life.” And no one can prove that these facts are less important to his religiosity than whether he professes to believe in God.

But if there is no evidence that can prove that a person is not religious, this means that we do not have a working definition of religiosity. And this means that the concept of religiosity is not useful in scientific research.

One finding is unimpeachable. People who go to church often give more money to churches than do people who go to church less often. But there is all the difference in the world between this finding and the claim that “Any way you slice it, religious people are simply more generous.”

Behavioral Observations

I mentioned that almost all sociological studies are based on data from surveys. There have been a few studies on religion and behavior where actual behavior was observed. (Brooks and Putnam-and-Campbell mention none of them.)

In the 1973 experiment of John M. Darley and C. Daniel Batson, the subjects (all students at Princeton Theological Seminary) “encountered a shabbily dressed person slumped by the side of the road.” Some were on their way to give a talk on the parable of the Good Samaritan; others had been assigned a topic unrelated to generosity. Those who (presumably) had generosity on their minds were not more likely than the others to stop and offer help to the slumped-over person. Also uncorrelated with their helping responses was their religiosity, as measured by a previous interview.

Some of the subjects were told, “Oh, you’re late. They were expecting you a few minutes ago. We’d better get moving.” This hurry condition had a significant effect on the subjects’ behavior. The authors conclude:

A person not in a hurry may stop and offer help to a person in distress. A person in a hurry is likely to keep going. Ironically, he is likely to keep going even if he is hurrying to speak on the parable of the Good Samaritan, thus inadvertently confirming the point of the parable. (Indeed, on several occasions, a seminary student going to give his talk on the parable of the Good Samaritan literally stepped over the victim as he hurried on his way!)

In a 1975 experiment by Ronald E. Smith, Gregory Wheeler, and Edward Diener, students in a large introductory psychology class were given an opportunity to cheat on a class test. On another, apparently unrelated occasion, they were asked to volunteer to help out some developmentally disabled children. Meanwhile, also seemingly unconnected with these events, a questionnaire was used to measure the strength of their religious affiliations. On the basis of this questionnaire, the subjects were divided into four groups: “Jesus people” (a term current in the 1970s, and not considered derogatory), religious, nonreligious, and atheists. No differences in either the rate of cheating or the rate of volunteering were observed between the four groups.

In another experiment (Lawrence V. Annis, Psychological Reports, 1976) subjects completed a questionnaire designed to measure “degree of commitment to traditional tenets of Western religion,” “location of religious values in the individual’s hierarchy of values,” and “frequency of religious behaviors like church attendance and private prayer.” Later, with no apparent connection to the questionnaire, each subject “happened” to see a woman carrying a ladder. The woman went into another room and closed the door; a few moments later there was an audible crash, designed to sound as if the woman had perhaps climbed the ladder and then fallen off. The subject then either opened the door or did not. None of Annis’s three measures of religious commitment bore any correlation with the likelihood of a subject’s opening the door.


Scientists who have taken the traditional hypothesis seriously and tested it experimentally have come up empty-handed. No evidence has been found for the proposition that religiosity fosters generosity. And that is not surprising, when we consider that no one even knows what religiosity is. People who describe themselves as religious tend also to describe themselves as generous. But this relation does not obtain in their actual behavior toward other human beings.

[The following is adapted from my unpublished book, No One Believes in God.]

Why do so many people believe?, we want to ask. But how big is this “so many”? Can we quantify it? How many are the believers – and how do we know?

Of course, there have been surveys. According to them, something like 85 percent of everyone in the world believes in some kind of god. But we cannot trust these polls. They are misleading. The methodology is flawed. Let me explain.

The goal of such surveys is to measure the popularity of various types of religious belief. But of course you can’t measure anyone’s belief directly, the way you can their temperature or their blood oxygen level. Are there indirect ways? Sure, you can just ask them about it! – but the thing is, how do you put the questions? What is the wording? You’d have to be pretty clever, I would think, to get decent results. A bald “Do you believe in god?”, for example, would elicit skewed, hard-to-interpret and probably worthless results.

What do the best pollsters use? Let’s take a look at some of the questions from a recent poll by the “highly respected” Pew Forum.

Q.30 Do you believe in God or a universal spirit?

Q.33 Do you believe in life after death?

Q.35 Do you think there is a heaven, where people who have led good lives are eternally rewarded?

Q.36 Do you think there is a hell, where people who have led bad lives and die without being sorry are eternally punished?

What – that’s what the Pew asked? I thought they were the real pros! They ask the same questions that the really stupid polls use. Jesus Christ.

Polls like this are misleading because to ask people “Do you believe in god?” is not an effective way to find out whether they believe in god. Here are just two of the many reasons that we know this to be true.

Most obvious and important is the strength of the social norms surrounding professions of “belief in god”. You answer the door, or the telephone, and a pollster is there. He starts asking you strange questions. In a vast number of households, if you are asked whether you “believe in god”, to say anything but yes would be to invite brutal reprisals from your family and neighbors. And in places where the risk is not so dire, there can still be severe pressure to say the right thing. This is a fact of life in any place where religion has serious influence, and that’s an awful lot of places.

The yes answer, on the other hand, is perfectly safe. There is no cost. Whether or not it is required, nothing follows from it. It’s not like saying, for example, “I believe in recycling”, which might be seen as committing you to a change in lifestyle. After you have affirmed your “belief in god”, no further action is required – so heck, why not say it? (Even if your dad is an atheist, he’ll just think you’re being funny. It’s not like he’s going to beat you or withhold meals or something, as a devout Christian or Muslim parent might do with a kid who said no.)

These observations have the following consequence. Of the people answering yes, some unmeasured but significant proportion have made this response not because they understand themselves to have a belief but because they fear the personal consequences of saying – not believing, but saying – the wrong thing. They feel constrained to say certain words, regardless of their actual belief. Even where the local norm is worded as a requirement that one believe, the practical result is rather that one is required to profess such belief. After all, no one can see into your mind. They cannot tell what you really think. Nor, most likely, do they care. They want you to say the right thing. That is the real demand. It’s about the words you say, especially to outsiders.

“Belief in god” is probably best seen as an idiom. It is not a state of mind but simply a stock phrase. Its meaning is not at all clear; however – and this is interesting – in practice, in the quotidian experience of the religious “believer”, the question of belief’s meaning, its nature and definition, does not arise. People are not asking themselves what it really means when they say they “believe in god”. They assume that they know. Whereas actually, they don’t know the first thing about it.

When we go to the trouble of surveying people on their “religious beliefs”, one of the things we should try to find out is what they personally mean when they use those words. But polls like this one – and they are all like this one – do nothing to help us understand this. They tell us nothing about belief. They assume that everyone already knows just what it is. But no one knows this.

Imagine a different poll. This one is measuring belief in Poseidon, so it asks people: “Do you believe in Poseidon?”

This is a very different question from the one we were just considering. Compared to Poseidon, the “god” in the other question sounds awfully vague. Asked whether I believe in “god”, I might find myself wondering nervously which god is being asked about. However, most respondents will recognize, probably before the question comes up, that there is a soothing, obvious, no-need-to-even-mention-it answer, which goes something like “You know which one: the one that most people talk about, and that most questions ask about. The one that most people, you know, believe in.”

And this assumption is written into the poll. People are going to assume that the “god” being asked about is the “god” that everyone always talks about. The fact that so many people pronounce this word so often is part of the de facto, unexamined definition of what the word means.

And now I really have to wonder whether this poll has measured anything at all. I suspect not. I suspect that every such survey is a travesty of research. Yet, the methodology goes entirely unquestioned. This is how religious belief is “tested”: you ask people absurd, pointless, misleading questions that can throw no light at all on what belief in god actually is – other than reminding us, between the lines, that it’s what you’re supposed to do. Heck, maybe that’s the real intention behind these stupid polls.

Many people believe in God. Many people believe in belief in God. What’s the difference? People who believe in God are sure that God exists, and they are glad, because they hold God to be the most wonderful of all things. People who moreover believe in belief in God are sure that belief in God exists (and who could doubt that?), and they think that this is a good state of affairs … . Given the way religious concepts and practices have been designed, the very behaviors that would be clear evidence of belief in God are also behaviors that would be clear evidence of (only) belief in belief in God. … This fact makes it hard to tell who – if anybody! – actually believes in God in addition to believing in belief in God. —Daniel Dennett

As far as I can tell, no one has ever obtained an actual measurement of the prevalence of religious belief, in this country or any other. The pollsters have no idea how many people believe in god, only how many are likely to say that they do, which is a completely different thing. They have neglected to notice that simply asking people this question, in these words, is worse than pointless. Their asking it in this way proves that they do not understand what the question means. Nor are they trying to understand. Nor are they helping us to do so.

In fact, there are many reasons to suppose that the prevalence of religious belief is greatly overestimated by almost everyone. Dennett’s “belief in belief” effect is one reason. Later in the book we will see more of them, the cumulative effect of which is to bring plausible estimates down very close to zero.