Bad Methodology - Blog Posts

@johnhocksbur

This isn’t how statistical methodology works.

If you want to be able to generalize the results of your survey to the general population, you have to use some form of random sampling*, you can’t just ask random people on the internet. Twitter polls (etc.) have two main flaws:

1. Response is voluntary, which means that people who don’t care are less likely to answer, and (on questions where this is applicable) people with more middle-of-the-road or less-shocking answers are less likely to answer. 

2. They operate using “convenience sampling,” which is basically what it sounds like and tends to bias the results in favor of whatever opinion is held by the people in the group likely to notice the survey. A political survey on the Fox News website will tend to have more conservative responses than the general population; a sports survey on the Boston Globe website will tend to have more pro-Red Sox responses than the general population; a survey on a Twitter page will tend to have more whatever-the-twitter-users-followers think responses than the general population.

(I did a brief Google search to see if this has been surveyed reliably and didn’t find anything, although possibly I could find something in an academic database. If anyone can find a reliable survey, I would be interested in seeing what the results.)

*This is somewhat complicated by the fact that it is nearly impossible to do a perfectly random sample. Phone surveys in which callers are randomly chosen and the response rate is high are generally close enough in surveys of Americans, although they aren’t perfect.

This is so interesting to me. 65% of people would rather experience rape than be falsely accused of rape.