This mini-lesson covers AQA 4.11 — Big Data: data sets so large or so fast-moving that the traditional relational approach simply cannot cope, and what we do instead.
The three Vs — volume, velocity, variety
Why relational databases break down
Distributed processing and MapReduce
Functional programming for big data
The fact-based model and the graph schema
Work through each screen, answer the questions as you go (some are recall, some are calculations you must actually work out) and collect ⭐ stars. Press Start when you're ready.
4.11 · Definition
What makes data big
Big data is not just "a lot of data". AQA defines it by three characteristics — and any one of them can be the thing that breaks you:
Volume — too large to store or process on a single machine. Petabytes, not gigabytes.
Velocity — arriving so fast that it must be processed as it streams. A stock exchange feed, sensors on a jet engine, fraud checks on every card swipe.
Variety — structured (tables), semi-structured (JSON, XML) and unstructured (video, free text, images) all mixed together.
Why a relational database cannot cope:
it demands a fixed schema defined in advance — but unstructured data has no such shape;
it scales vertically (buy a bigger server), and there is a hard ceiling on how big one server can be;
ACID guarantees and joins across billions of rows become ruinously expensive.
The change of mindset: big data scales horizontally — add more cheap, ordinary machines. At that scale, machine failure is not an emergency, it is a routine daily event, so the software must simply expect nodes to die and carry on.
4.11 · Distribution
Distributed processing and MapReduce
A single machine reading a petabyte at 100 MB/s would take months. So the data is split across many machines and each one processes its own slice locally. Moving the computation to the data is far cheaper than moving the data to the computation.
MapReduce is the standard pattern:
Map — each node applies the same function to its own chunk, emitting (key, value) pairs. Every node works independently and in parallel.
Shuffle — pairs with the same key are gathered together on one node.
Reduce — the values for each key are combined into a single result.
Input: "the cat sat on the mat the end"
MAP → (the,1) (cat,1) (sat,1) (on,1)
(the,1) (mat,1) (the,1) (end,1) // 8 pairs
SHUFFLE→ the:[1,1,1] cat:[1] sat:[1]
on:[1] mat:[1] end:[1]
REDUCE → (the,3) (cat,1) (sat,1) (on,1)
(mat,1) (end,1) // 6 pairs
Why this scales: the map stage has no shared state, so ten thousand nodes can run it at once with no locking and no coordination. Double the machines and you roughly halve the time. That is impossible for code that mutates shared variables.
Calculate
Your turn — MapReduce
1MapReduce performs a word count on the input "the cat sat on the mat the end". How many (key, value) pairs does the reduce stage output?
pairs
Hint: Reduce outputs one pair per distinct word. Count the distinct words — 'the' appears three times but is only one key.
Calculate
Your turn — distributed speed-up
2A 1 TB data set must be read. One node reads at 100 MB/s. Taking 1 TB = 1 000 000 MB, and splitting the work evenly across 50 nodes, how many seconds does the read take?
seconds
Hint: First find the time for a single node: 1 000 000 ÷ 100. Then divide that between the 50 nodes.
4.11 · Functional
Why big data uses functional programming
Big data and functional programming fit together for one hard technical reason: parallelism is safe only without shared mutable state.
A pure function has no side effects: the same input always gives the same output, and nothing outside it is changed.
Immutable data can never be modified. To "change" it you create a new value.
Consequently a computation can be run twice, run on a different node, or run out of order, and the answer is identical. When a node dies mid-job — and at this scale one always does — the framework simply re-runs that task somewhere else. No locks, no race conditions, no corruption.
Contrast with imperative code: two threads incrementing a shared counter can interleave and lose an update. That bug is impossible in pure functional code, because there is no shared counter to corrupt. Immutability is not an aesthetic preference here — it is the precondition for the whole architecture.
4.11 · Models
The fact-based model and the graph schema
The fact-based model stores data as an ever-growing list of immutable facts. Each fact is:
atomic — it cannot be split further;
timestamped — we know exactly when it became true;
immutable — it is never updated or deleted, only superseded by a later fact;
uniquely identifiable, so duplicates can be spotted.
(user:42, city, "Leeds", 2019-03-01)
(user:42, city, "Bristol", 2024-08-14) // the old fact still exists
Nothing was overwritten. You can reconstruct the state of the world at any past moment, audit every change, and recover from a bad batch of code by simply recomputing from the raw facts.
A graph schema stores highly connected data as nodes (entities), edges (relationships) and properties on both. Relationships are first-class citizens — stored directly, not inferred by joining tables.
Where the graph wins: "find friends of friends who like this band" is a nightmare of self-joins in SQL and a trivial traversal in a graph database. Social networks, recommendation engines and fraud rings are all fundamentally graph-shaped problems.
Calculate
Your turn — graph edges
3In a social graph, 5 people are all mutually connected (every person is a friend of every other, and friendship is undirected). How many edges does the graph contain?
edges
Hint: Each of the 5 people connects to the other 4 — but that counts every friendship twice, once from each end.
Quick check
Why functional?
?Why is a pure function with no side effects so valuable in a distributed big-data system?
Quick check
The fact-based model
?In a fact-based model, a user moves from Leeds to Bristol. What happens to the original record?
Quick check
Relational limits
?Why does a traditional relational database struggle with a stream of unstructured video, free text and sensor readings?
Sort it
Which V is it?
Tap a scenario, then tap the characteristic it illustrates.
🗻 Volume
⚡ Velocity
🎨 Variety
Match it
Match the idea to its name
Tap an item on the left, then its partner on the right.
Idea
Name
Recap
The big ideas to know
Three Vs: volume (too big for one machine) · velocity (must process as it streams) · variety (structured, semi- and unstructured)
Relational limits: fixed schema in advance · vertical scaling hits a ceiling · joins and ACID cost too much at this scale
Distributed: move the computation to the data · scale horizontally · expect node failure as routine
MapReduce: map in parallel → shuffle by key → reduce to one value per key
Functional: pure functions and immutable data → no shared state → safe parallelism and free re-runs