Types of experiment and their variables

• Laboratory experiment — the IV is manipulated in a highly controlled environment. High internal validity and replicability, but risks demand characteristics and low ecological validity.
• Field experiment — the IV is manipulated in a natural setting. Higher mundane realism, but less control over extraneous variables, and there are ethical problems where participants have not consented.
• Natural experiment — the researcher does not manipulate the IV; it varies naturally and would have happened anyway (e.g. Rutter's Romanian orphans). No random allocation, so causal conclusions are risky.
• Quasi-experiment — the IV is a pre-existing characteristic of the participants (age, gender, having a diagnosis). Again, the IV is not manipulated and participants cannot be randomly allocated, so confounding variables are a real threat.

Variables and control. The independent variable is manipulated; the dependent variable is measured; both must be operationalised (defined so that they can be measured). An extraneous variable is any other variable that might affect the DV if not controlled (a nuisance variable); a confounding variable is one that varies systematically with the IV, so we cannot tell which caused the change in the DV.

Threats to validity: demand characteristics (participants guess the aim and change their behaviour — the 'please-U' or 'screw-U' effect) and investigator effects (the researcher's expectations or behaviour influence the results). Controls: single-blind and double-blind procedures, standardised instructions, and randomisation.

Hypotheses. A directional (one-tailed) hypothesis states the direction of the difference or relationship and is used when previous research points one way. A non-directional (two-tailed) hypothesis states only that there will be a difference or relationship, and is used when previous research is absent or contradictory. The null hypothesis states there is no difference/relationship beyond that expected by chance — and it is the null hypothesis that a statistical test either rejects or fails to reject.

Experimental design and sampling

Experimental design

• Independent groups — different participants in each condition. Problem: participant variables. Solution: random allocation. Also needs more participants.
• Repeated measures — the same participants in every condition, so participant variables are controlled. Problem: order effects (practice, fatigue, boredom) and greater risk of demand characteristics. Solution: counterbalancing (ABBA) — half do A then B, half do B then A.
• Matched pairs — different participants, but matched on variables relevant to the study (e.g. IQ, age). Controls participant variables without order effects, but matching is time-consuming and can never be perfect.

Sampling

• Random — every member of the target population has an equal chance of selection (names from a hat, random number generator). Free from researcher bias, but time-consuming, and the sample can still be unrepresentative by chance.
• Systematic — every nth member of the sampling frame. Objective, but again time-consuming.
• Stratified — the composition of the sample reflects the proportions of sub-groups (strata) in the population, with participants randomly selected from each stratum. The most representative method, but the strata cannot capture every difference.
• Opportunity — whoever is available. Quick and cheap, but unrepresentative and open to researcher bias.
• Volunteer (self-selected) — participants respond to an advert. Easy and gives committed participants, but produces a volunteer bias (a particular type of person volunteers).

Ethics (the BPS Code). Informed consent (or presumptive/prior general/retrospective consent where deception is necessary); deception must be justified by an ethics committee weighing the cost-benefit; protection from harm — participants must leave in the same state they arrived; the right to withdraw; confidentiality and privacy. The key remedies are a full debrief and the right to withdraw data.

Choose the design

Observations, self-report, correlations and case studies

• Observations — naturalistic vs controlled; covert vs overt; participant vs non-participant. Behaviour is recorded using behavioural categories, which must be operationalised, observable and mutually exclusive, and sampled by event sampling (count each occurrence) or time sampling (record what is happening at fixed intervals). Time sampling is efficient but may miss behaviours.
• Self-report — questionnaires (open and closed questions; Likert, rating and fixed-choice items) and interviews (structured, unstructured, semi-structured). Both are vulnerable to social desirability bias and acquiescence bias. Design principles: avoid leading questions, double-barrelled questions, jargon and emotive language; pilot the questionnaire first.
• Correlations — measure the strength and direction of an association between two co-variables, expressed as a correlation coefficient from -1 to +1. Correlations cannot demonstrate causation (there may be an untested third variable), and they can be misleading where the relationship is curvilinear.
• Case studies — an in-depth, idiographic investigation of one person, group or event, often using several methods and often longitudinal. Rich, detailed data (HM, Little Hans, Clive Wearing), but cannot be generalised and are prone to researcher subjectivity.
• Content analysis and thematic analysis — a method for turning qualitative material (interviews, media) into data. Coding counts instances of a category, producing quantitative data; thematic analysis identifies recurrent themes and keeps the data qualitative.
• Peer review — independent scrutiny of research by experts before publication, to allocate funding, validate quality and suggest amendments. Criticisms: reviewers may be anonymous and use that to criticise rivals, there is publication bias towards positive and 'headline' findings, and it may suppress opposition to the mainstream view.

Reliability and validity — and how to improve them

Reliability = consistency.

• Test-retest — administer the same test to the same person on two occasions; a correlation coefficient of +0.80 or more indicates good reliability.
• Inter-observer reliability — two or more observers score the same behaviour independently and their totals are correlated (again, aim for +0.80). It is improved by operationalising the behavioural categories more precisely and by training observers.
• Improving reliability: standardise procedures in experiments; in questionnaires, replace ambiguous or open questions with fixed-choice items; in interviews, use the same interviewer or train interviewers to avoid leading questions.

Validity = whether we are measuring what we intended to measure, and whether the finding is legitimate.

• Internal validity — did the IV really cause the change in the DV, or was it a confounding variable, demand characteristics or investigator effects?
• External validity — can the finding be generalised? Ecological validity (to other settings and real life), population validity (to other people) and temporal validity (to other eras — a real problem for Asch's 1950s conformity data).
• Face validity — does the measure look, on the face of it, as though it measures what it claims to? Concurrent validity — does it correlate closely (again, about +0.80) with an established measure?
• Improving validity: use a control group, standardise procedures, use single/double-blind designs, and — in qualitative research — use triangulation (comparing evidence from several sources).

Features of science: paradigm and paradigm shift (Kuhn) · theory construction and hypothesis testing · falsifiability (Popper: a theory that cannot be proved wrong is not scientific) · replicability · objectivity and the empirical method.

Which kind of validity?

Levels of measurement and descriptive statistics

Levels of measurement (you must know these to choose a test):

• Nominal — data in categories (how many people chose A, B or C). Each item appears in only one category. It is a count, not a measurement.
• Ordinal — data that can be ranked, but where the intervals between points are not equal — for example scores on a rating scale, because one person's '7' is not the same as another's. Ordinal data therefore lack precision.
• Interval — data measured with equal, standardised units: time in seconds, temperature, number of words recalled. This is the most precise level.

Measures of central tendency: the mean (uses every score, so it is most sensitive — but it is distorted by outliers); the median (unaffected by extreme scores, but ignores much of the data); the mode (the only option for nominal data).

Measures of dispersion: the range (highest minus lowest — simple, but affected by a single extreme value) and the standard deviation (the average distance of each score from the mean). A large SD means the scores are widely spread and the mean is not very representative; a small SD means the data are tightly clustered and all participants responded similarly.

Distributions: in a normal distribution the mean, median and mode are all at the mid-point and the curve is symmetrical. In a positive skew the long tail is to the right and the mode is to the left of the mean; in a negative skew the long tail is to the left and the mean is pulled below the mode.

Calculate the mean

Percentages from a table

Probability, significance and errors

An inferential test tells us how likely it is that our result occurred by chance. Psychology's usual level of significance is p ≤ 0.05 — a 5% (or less) probability that the result is due to chance. If the test is significant, we reject the null hypothesis.

• A more stringent level (p ≤ 0.01) is used where a Type I error would be especially serious — for example in drug trials, or where the study is a replication of a controversial finding.
• Type I error — a false positive: we reject the null hypothesis when it was in fact true (we claim a difference that isn't there). More likely with a lenient significance level such as 0.10.
• Type II error — a false negative: we accept the null hypothesis when it was actually false. More likely with a stringent level such as 0.01.

Using critical value tables. To find the critical value you need three things: whether the hypothesis is one-tailed or two-tailed; the N (or degrees of freedom); and the level of significance. Then compare:

Degrees of freedom for chi-square: df = (number of rows − 1) × (number of columns − 1).

Degrees of freedom

The sign test, step by step

The sign test is used when you are looking for a difference, the design is repeated measures (related), and the data are nominal (or can be reduced to a simple direction of change).

• Step 1. For each participant, record the sign of the change: a + if the score went up, a − if it went down, and 0 if there was no change.
• Step 2. Discard all the zeros (participants who showed no change). N is the number of participants left.
• Step 3. Count the pluses and the minuses. The calculated value of S is the number of the less frequent sign.
• Step 4. Look up the critical value of S for that N, the chosen significance level, and whether the test is one- or two-tailed.
• Step 5. The result is significant if the calculated value of S is equal to or less than the critical value.

Worked example. 15 participants are tested before and after therapy. Nine improve (+), four get worse (−), and two show no change (0). We discard the two zeros, so N = 13. The less frequent sign is the minus, so S = 4. The critical value of S for N = 13 at p ≤ 0.05, two-tailed, is 2. Because 4 is greater than 2, the result is not significant and we fail to reject the null hypothesis.

Calculate S

What is N?

Choosing the right statistical test

Three questions decide the test. (1) Am I testing a difference or a correlation? (2) If a difference, is the design related (repeated measures or matched pairs) or unrelated (independent groups)? (3) What is the level of measurement?

The parametric tests (the two t-tests and Pearson's r) are more powerful, but require interval data, a normal distribution and homogeneity of variance.

Choose the test (1)

Choose the test (2)

Significant or not?

Type I or Type II?

Pilot studies, data types and the scientific report

• Pilot studies — a small-scale trial run of an investigation, carried out before the real thing. The aim is to find out whether aspects of the design do not work — ambiguous questionnaire items, instructions that confuse participants, a task that is too easy — so that they can be modified and time and money saved. In observational research, a pilot allows observers to be trained and behavioural categories to be checked.
• Primary data — information collected first-hand by the researcher for the purpose of the investigation. It is authentic and fits the aim exactly, but it takes time and money to gather.
• Secondary data — information already collected by someone else (government statistics, journal articles, existing datasets). It is inexpensive and immediately available, but it may be outdated, incomplete or not quite fit the researcher's aim.
• Meta-analysis — a form of research using secondary data, in which the findings of many studies on the same question are combined and analysed together, often producing an effect size. It creates a much larger, more representative sample and greater confidence in conclusions — but it is vulnerable to publication bias, because researchers may omit studies with negative or non-significant findings.
• Presenting data: a bar chart shows the frequency of discrete categories, with gaps between the bars; a histogram shows continuous data, with no gaps and the area of the bar representing frequency; a scattergram shows the relationship between two co-variables in a correlation.
• Sections of a scientific report: abstract (a short summary of the whole study) → introduction (a literature review ending in the aims and hypotheses) → method (design, sample, apparatus, procedure, ethics — detailed enough for replication) → results (descriptive and inferential statistics) → discussion (what the findings mean, limitations, implications) → referencing.
• Implications for the economy — psychological research has real economic value. Attachment research supports flexible working and shared parental leave; research into eyewitness testimony saves police time and prevents costly miscarriages of justice; effective treatments for depression and OCD reduce absence from work, which the economy pays for.

Which test?

Tap a scenario, then the level of data it uses. Getting the level of measurement right is the first step to choosing the correct test.

Match the design to the test

Tap an item on the left, then its partner on the right.

The big ideas to know

You have covered the whole of AQA 4.2.3 Research methods — the highest-value topic on the course. Press Finish to see your score.