Research design. The research design is the logic by which the results answer the research question, e.g. suppose one is studying whether regular use of pain–killers harms people, given the current US opioid crisis. A simple approach might be to interview people in hospitals to find out whether they are on pain–killers and measure their state of health. If the results were that those on painkillers were less healthy than those not on painkillers, does this mean that pain–killers reduce health? Unfortunately, the research design only shows a correlation, not what causes what. Indeed, as sicker people are more likely to need pain–killers it may be that sickness is causing pain–killer use, not the other way around. Such problems are common in research, so the research design aspect of the method states the logic by which the results will answer the research question. Stating the research design logic may only take a sentence or two but is very important. Before doing your research, imagine it gives ideal results and ask yourself “Does this really answer the research question?”
Experiments that seek to establish causes must be especially clear about research design. The basic logic of an experiment is to assign subjects to treatments to see what causes what. The treatment is the independent variable that the experimenter applies and the result is the dependent variable that is measured. This logic assumes that subject cases are applied to treatments in a random and matched way, so in a study of how people rate different browsers, they can’t choose the browser they rate as that is not random, nor is it matched, as more people would choose popular than unpopular browsers. This introduces their choice as a spurious cause of browser preference, and differences found could be attributed to subject differences not the treatment. This issue gives these types of experiment:
- Experimental design. Subjects are randomly assigned to all treatments.
- Quasi-experimental design. Subjects are randomly assigned to some treatments, e. g. if one cause is male-female gender, subjects cannot be assigned randomly.
- Repeated measure design. Subjects try first one treatment then another, e. g. try one browser then another. Since all subjects do all treatments, there is no need for random assignment and the subjects are the same for all trials. However there can be order effects, so the order that subjects try the software must be randomized to avoid it affecting the results.
Choosing a design. Discuss the research design with your advisor before proceeding, and once you select a design, read up on it in more detail. Some common designs are:
- One-shot case study or ex post facto design. A single group of subjects is measured after some intervention, e. g. studying the effect of new security measures. The problem here is that even if subjects liked the new measures, perhaps they equally liked the old ones?
- Two-group post-test only without random assignment. In this static (one-time) design, a group without the intervention is also measured, e. g. smokers are compared to non-smokers for health effects. The problem here is that people with weak health initially may be precisely those who choose smoke, e.g. soldiers about to enter battle may smoke as they figure they won’t last long anyway but smoking didn’t cause their death.
- One-group pretest-posttest. A single group of subjects is measured before and after some intervention, e. g. measure health before and after a diet. The problem here is that other factors may operate over that time, e.g. the World Cup may be on causing many subjects to lose sleep. Who is to say it was diet that caused any effects found?
- Two-group pretest-posttest. Subjects are randomly assigned to a treatment group and a control group. Both are measured before and after some intervention, e. g. to assess a new diet, the treatment group will try it while the control group doesn’t. This controls for everything but expectancy bias.
- Solomon four-group design. The most complex design as it also controls for experimental bias. Two groups don’t do the pretest, so it takes a lot of work to do.
Control group. The control group is a set of subjects who do everything the same but don’t receive the treatment, e. g. in medical research they get a placebo sugar pill not the new drug. This only controls for spurious causes if the subjects are randomly allocated, so don’t let subjects choose to be in the control group or not. A control group is important if subjects know they are being observed, e. g. in the Hawthorne effect, experimenters found that painting a room a lighter color improved work rate, but later found painting it dark had the same effect! What improved the work rate was not the paint but that they were being observed by researchers. A control group avoids such problems. Randomly allocate subjects to a control group to control for spurious effects.