Across the sciences, researchers use a spectrum of tools or “instruments” to collect information and then make inferences about human preferences and behavior. These tools vary in the degree of control the researcher traditionally has had over the conditions of data collection. Surveys are an instance of such an instrument. Though widely used across social science, business, and even in computer science as user studies, surveys are known to have bugs. Although there are many tools for designing web surveys, few address known problems in survey design.
They have also traditionally varied in their media and the conditions under which they are administered. Some tools we consider are:
Observational studies
Allowing no control over how data are gathered, observational studies are analogous to data mining — if the information is not readily available, the researcher simply cannot get it.
Surveys
The next best approach is to run a survey. Surveys have similar intent as observational studies, in that they are not meant to have an impact on the subject(s) being studied. However, surveys are known to have flaws that bias results. These flaws are typically related to the language of individual survey questions and the structure and control flow of the survey instrument itself.
True Experiments
If a research is in the position of having a high degree of control over all variables of the experiment, they can randomly assign treatments and perform what is known as a “true experiment”. These experiments require little modeling, since the researcher can simply using hypothesis testing to distinguish between effect and noise.
Quasi-Experiments
Quasi-experiments are similar to true experiments, except they relax some of the requirements of true experiments and are typically concerned with understanding causality.
In the past, there has been little fluidity between these four approaches to data collection, since the media used to implement each was dramatically different. However, with the proliferation of data on the web and the ease of issuing questionnaires on such platforms as facebook, SurveyMonkey, and Mechanical Turk, the implementation of these studies of human preferences and behavior have come to share many core features.
Despite similarities between these tools, quality control techniques for experiments have been largely absent from the design and deployment of surveys. There has been an outpouring of web tools and services for designing and hosting web surveys, aimed at non-programmers. While there are some tools and services available for experiments, they tend to be domain-specific and targeted to niche populations of researchers. The robust statistical approaches used in experimental design should inform survey design, and the general, programmatic approaches to web survey design should be available for experimental design.
This is good, I just got confused on the observational part. I guess by observational you mean what in linguistics we call a corpus study – you just search and analyze information that’s already “out there”, which may involve gathering what’s out there in some systematic way first, and possibly annotating it. But there’s no control over the frequencies of the situations, which leads to sparsity in some areas. The upsides, though, are that things are more naturalistic, you lack task effects (there’s that phrase you wanted to look up), and you can get way more data. With sophisticated statistics (I think data mining is a synonym of that?), you can account for the uncontrolled distribution at least to a point. And when you control your distribution, you run the risk of introducing sampling bias. I don’t know if you want to go into observational studies very much since that’s not what SurveyMan is about, but some of these ideas might help. Also, I don’t know what a quasi-experiment is.
I think you’re right about all the synonyms for observational studies — I think unifying techniques and terminology should be a goal of data science, since many disciplines are doing the same thing, but in slightly different ways.
Regarding what SurveyMan has to offer, it really isn’t focused on observational studies/data mining. There was a picture I drew on the white board for Molly that illustrated how previously experiments and quasi-experiments were grouped together because they were conducted in highly controlled environments (i.e. a lab), whereas surveys and observational studies were conducted in the wild, where you have very little control over the environment. Surveys have traditionally aspired to be like observational studies, but inherently suffer from the “probe effect”. We’re shifting surveys over to the (quasi-)?experiments category, since we’re exercising a higher degree of control than we had before. What’s interesting to me is that this is quite clearly a product of being able to deploy surveys on the web. While this new technology has made surveys more robust, using the same platforms has actually degraded the integrity of experiments — where before they were conducted in a lab, now you have no idea what the conditions are under which the person is taking them. The assumption here is that you’ll drown out the noise caused by these uncontrolled environments by gathering significantly more data from a significantly broader population than before.
re : quasi-experiments. They’re used either when you cannot randomly assign a variable (e.g. it’s hard to reassign sex), or when you hold other variables constant on purpose in order to determine causality. This is something Emery’s quite keen on right now.