<This page is under construction>

© Michael J. Rosenfeld, 2009, 2012


Notes on terminology for evaluation of research.


1) What we mean and donít mean by bias.

What we donít mean: Statistical bias has nothing to do with personal bias or prejudice.


There are two kinds of errors relevant to statistical measurement. One is noise, the other is bias.


Given a set of observed measurements X,


X= Xo+ bias + random error


where Xo is the perfect theoretical measure, which we are never able to quite reach.


The expected value of X, or letís say the average of our X measurements, is

E(X)= Xo+ bias


bias is a constant which skews our measurements. We usually canít tell exactly what the bias is because the true theoretical values, Xo are not measurable. The only thing we have to examine is X, which comes with bias already in it.


Variance of X comes entirely from the random error.


Key to remember: bias skews the results, whereas random errors increase the variance but do not skew the results.


In a random sample, larger sample size can help reduce the influence of random noise (I will explain this later in the class). But larger sample size usually does nothing to minimize the effect of bias.


The world of research is full of biases and potential biases, only some of which we will actually discuss and exemplify in this class.


2) Sampling frame. This is the universe of individuals from whom we want to know something about (potential voters, all persons, adults). Think of this as the list of people we might want to randomly sample from. Of course, in real life, we usually donít have a full and complete list of everyone in the desired sampling frame. Another word for the sampling frame is the sampling universe, this is the population your data pertain to. You *must* always know what your sampling frame is. We make hypotheses about the wider universe, or the sampling frame, but we measure the sample. Keep the two separate in your mind.


3) Random sampling versus Convenience sampling

* Random sampling occurs when every subject in your sampling frame has an equal and random chance of being sampled. A random sample is generalizable to the whole sampling frame.


The purpose of statistical analysis is usually to use the data in our sample to test hypotheses about the whole sampling frame.


* Convenience sampling occurs when you find the subjects who are easiest or most convenient to find, and interview them. Convenience samples are not generalizable, meaning even if we know a lot about our sample, we may not be able to make inference about the wider sampling frame.


There are other kinds of sampling as well, which fall in between these extremes:

* Snowball Sampling

* Respondent Driven Sampling (a variety of Snowball Sampling)

* Stratified Random Sampling


And what if our data contain not a sample but the entire sampling frame? All 50 states, for instance? This is what Rice refers to as sampling fraction=1. Certainly, if the sampling fraction=1, we are not going to making probabilistic arguments about the sampling frame because we have measured the whole sampling frame. There may still be some use in fitting models to the data, but standard errors of coefficients may not be meaningful.



Kinds of bias or potential bias

<page under construction>





When you might see it, what effect it might have. examples

Sampling Bias

The people we find to put into our sample may not be representative of the whole sampling frame we are interested in.

* Traditionally, surveys are done with Random Digit Dialing. RDD reaches only people who have phones. When home phones were a luxury, this was a very biased sample of all homes. Now that a lot of young people have cell phones instead of land lines, new biases are introduced (because a lot of survey companies wonít call cell phones).

* Example. In 1936, the Literary Digest surveyed 10 million Americans recruited via telephone directories and automobile ownership lists. The surveyed population was planning to vote for Alf Landon over FDR by a landslide. In the actual election, FDR won in a landslide. Where did the discrepancy come from? In 1936, telephone and automobile owners were a very unusual, or biased subset of the desired survey frame of all voters.

* If you know exactly by how much a certain population is under- or over-represented in your sample, then it is possible (in theory) to correct the sampling bias by applying weights.

Selection Bias

Bias due to the way the data are selected. More importantly in the social sciences, selection bias occurs as a result of people making choices.

* People choose to marry or choose to send their children to parochial school, or choose to work (rather than staying home). Marriage, parochial school, and work are not randomly distributed in the population. If married people turn out to be different from unmarried people, it may be only because the kinds of people who marry are different from the kind of people who donít marry.

* Example: people who cohabit before marriage have a higher divorce rate. Is this because pre-marital cohabitation leads to divorce? Or perhaps the kind of people who donít believe in pre-marital cohabitation are also the kind of people who donít believe in (and therefore would be less likely to get a) divorce. See Andrew Cherlinís Marriage, Divorce, Remarriage.

* Selection bias crops up in some way in almost every social study.

Attrition Bias

Attrition removes respondents from your sample often for unseen reasons

Every longitudinal study is affected by attrition bias. Some individuals are ďlost to follow-up.Ē These individuals turn out to be different in some key, unmeasurable ways, from other individuals who are easier to track down. Homeless people or poor people have higher attrition. Subjects who are traveling salesmen have higher attrition. The lowest attrition population (the people who are easiest to find again if you have had them in your survey already) are middle aged suburban homeowners with good professional employment.

Mortality bias

A kind of attrition bias.

* If you are studying a high mortality population, such as old people or people with AIDS, the people who survive and remain in your sample may be fundamentally different from the people who have already died.

* In the Freedpeopleís study (described by Davidson and Lytle), the former slaves interviewed in their 70s and 80s had already lived much longer than most former slaves. This may mean that the surviving slaves were less likely to have experienced the worst of slavery.

Social Acceptability Bias

People say what they think you want to hear. People donít tell you things about themselves that may reflect poorly (or they imagine may reflect poorly) on them.

* Socially stigmatized behavior is almost always under-reported.

* Example: abortions are underreported in self-report compared to clinic and hospital report.

Response Bias

Some people donít want to answer questions or participate in surveys

* If the kind of people who donít want to answer a survey are different from the kind of people who are willing to answer, then response bias is a factor.

* Survey researchers try to limit response bias by doing everything they can to keep response rates high. Nonetheless, response rates have been falling over time in all surveys.

Recall Bias

What was this again? I forgot

* If you are asking people to recall something that happened in the past, you get a lot of noise (because people canít remember) but you also get bias. The passage of time makes people feel differently about events.

* Example: Alwin, Cohen, and Newcomb, in Political Attitudes Over the Life Span report that 46% of women in their sample reported voting for Nixon in 1960, when surveyed just after the 1960 election. When the same women were asked in 1984 who they had voted for in 1960, only 26% said they voted for Nixon.

Interviewer Bias

People give different answers depending on who the interviewer is (or who they think the interviewer is).

* See, for example, the Freedpeopleís study (Davidson and Lytle). There is a lot of experimental evidence that black and white interviewers elicit different kinds of answers.