The University of Massachusetts Amherst
Categories
Experimental Resources

Web-based experiments

On the “P-Side” at least, people have recently been running web-based experiments, and getting excellent results. I’m starting this thread as a way of sharing resources, and also perhaps talking about issues that come up in this domain.

The two projects I’ve seen come to fruition are Wendell Kimper‘s experiment on variation in Finnish vowel harmony, and Claire Moore-Cantwell‘s wug-test experiment on Hebrew denominals. Wendell used LimeSurvey, which we have installed on the departmental server (ask me if you’d like to use it). LimeSurvey didn’t work for Claire (it couldn’t pass a variable from one part of the experiment to another), so she developed her own java-based software that fit her needs. Interim summary: we have LimeSurvey, and it is a simple solution for relatively straightforward experiments – if you have more detailed notes from first-hand experience, please add them below.

We have the good fortune of having Michael Becker in our midst this year, and he has recently been working on web-presentation software he and Jonathan Levine call Experigen. He has run several experiments using this software, and demo’d it for a group of us this summer, several of whom immediately decided to set up their own experiments with this elegant set of tools. It requires a little more expertise in html, etc. than LimeSurvey, but we should have or get that anyway, right? Again, those with more experience, please comment below.

2 replies on “Web-based experiments”

Another tool that is very useful, and for which there is an increasing community of users, is the Ibex Farm: http://spellout.net/ibexfarm/. It’s a flexible online experimental delivery system that can implements a variety of experimental goodies, including acceptability tasks, self-paced reading and latin square designs. It just hit its 200th experiment, and is used people in Masha Polinksy’s lab, Florian Jaeger’s lab, Elsi Kaiser’s lab, among many others. Here, a bunch of folks on the “S-Side” have used this with good results. Jesse Harris has set up a pipeline that integrates Ibex Farm with Amazon’s Mechanical Turk, which allows the user to manage recruitment and payment through MT, and experimental delivery through Ibex Farm, which is a very nice setup.

The ease and speed of web-based experimenting means it’s going to be pretty heavily used going forward. Given this, another useful topic of discussion here is quality control over internet experiments. We have some idea of how to make sure that people are reasonably focused and from the target population when we bring them into the lab, something that is a little more difficult with online experiments.

With respect to quality control, I had some success (or at least I think it was successful) using catch trials to weed out untrustworthy subjects in my experiment. I was interested in probabilistic data for a variable process, but I included a handful of stimuli where subjects would not be expected to exhibit variable behavior. I threw out the data of any subject who got a sufficient number of these wrong, taking that as indication that they were either not actually native speakers of Finnish or that they weren’t really attending to the task. I don’t remember off hand how much data that ended up being (my recollection is that it was not very much, but not totally trivial either). I suppose it would be beneficial to go through the data from subjects who were rejected on that basis, and see if they actually behaved any differently on the test trials.

Leave a Reply

Your email address will not be published. Required fields are marked *