TfL publishes something I didn’t know existed until recently: a “typical day” of crowding for every Underground station, broken into 15-minute slices, for every line that stops there. It isn’t live data, just a model of an ordinary day. It covers the whole network, though, and it comes with a second dataset that estimates how full each train is as it leaves each station.

Most crowding charts I’ve seen answer “where is it busy?” I wanted to know whether this data could show something that moves: whether the rush starts in one place and arrives in another, whether stations have different characters, and what happens inside a train between the platforms. So I pulled the data for 269 stations and looked before deciding what the story was.

Data & licence. Crowding data from the TfL Unified API (StopPoint/{id}/Crowding/{line}), restricted to the 11 Underground lines and fetched on 26 September 2026. Powered by TfL Open Data. Contains OS data © Crown copyright and database rights 2016 and Geomni UK Map data © and database rights 2019. Used under the TfL Open Data terms. The data is one modelled typical day: no weekday/weekend split, and nothing between 02:00 and 05:00. Distances are straight-line kilometres from Charing Cross. Notebook: eda_notebook.ipynb.

A typical day

First, the whole network added together. Every station, every line, 15 minutes at a time.

It’s the familiar two-humped camel. The busiest slice of the day is 08:15 (210,474 flow units across the network); the evening peak at 17:45 is about 5% lower (200,336). The evening hump is wider, though, and holds slightly more of the day: 28.1% of all flow falls between 16:00 and 19:00, against 26.3% between 07:00 and 10:00. Those six rush hours carry 54.4% of the flow in a dataset that covers 21 hours.

None of that is surprising. The interesting parts come from pulling this total apart.

Where it’s busy, and a ranking I got wrong first

Add up each station across all the lines that serve it and the ranking looks as you’d expect: Oxford Circus (316,602), King’s Cross St Pancras (284,292), Green Park, Victoria, Waterloo. It’s also very concentrated. The busiest 27 stations, 10% of the network, carry 53.5% of all the flow.

My first version of this ranking had Brixton in second place. Brixton is busy, but not busier than King’s Cross. It turned out that for most stations TfL returns several unlabeled flow series per time slice, in no particular order, and my first parser kept just one of them. Brixton, the end of the Victoria line, happens to have only one series, so it kept everything while the big interchanges lost most of theirs. The same bug made Moorgate look more than twelve times busier in the morning than in the evening. Summing all the series fixed both. The honest caveat is that I still don’t know what each series is (TfL doesn’t label them), so I treat the sum as “typical passenger flow” rather than a confirmed count of entries and exits.

The rush reaches the centre last

Here’s the network hour by hour. Press play.

It’s pretty, but by volume the centre dominates every frame, so the animation mostly shows the city breathing in and out. To see movement I needed timing rather than size. For each station I found its busiest 15-minute slice between 05:00 and 11:00, then coloured the map by that time.

Now the map shows a direction. The earliest stations are all a long way out: Dagenham Heathway, Kingsbury and Queensbury peak at 06:30, Barking and East Ham at 06:45. The centre is dark, peaking around 08:30. Plotted against distance, it’s a clear slope:

The correlation between morning peak time and distance is −0.55. It’s what you’d expect if people leave home at roughly the same time relative to when they need to arrive: the further out you live, the earlier you get on.

I checked one way this could be an artefact. A few central work stations (Temple, Goodge Street, St Paul’s and others) are so quiet in the morning that their “morning peak” is just the last slice of the window, 10:45, which would drag the centre later. Dropping those 7 stations makes the pattern stronger, not weaker: the correlation goes to −0.63 and the medians in the table don’t move. It isn’t a smooth gradient line by line, either. On the Northern line south of the river, Clapham North peaks at 07:30, earlier than Morden at 08:00. The wave shows up in the aggregate, not neatly at every stop.

The evening column is the one I didn’t expect. I assumed the wave would run in reverse after work, with the centre peaking first and the suburbs later as people got home. Instead the outer stations still peak earlier (17:00 against 17:45, correlation −0.52). I don’t have a clean explanation. Because TfL doesn’t say which flow series is entries and which is exits, I can’t check whether that’s people leaving the suburbs or arriving in them, so I’m leaving it as an open question rather than inventing a story.

Home stations and work stations

The timing hints at something simpler: some stations are busiest in the morning and others in the evening. Dividing each station’s evening peak by its morning peak splits the network almost cleanly.

At the bottom are the home stations. At Elm Park, 23 km out on the District line, the evening peak is a tenth of the morning peak. Pinner and Queensbury are close behind. At the top are the work stations: Goodge Street’s evening peak is 10.5 times its morning peak, with Mansion House, Temple and Chancery Lane not far off. Across all 203 stations the correlation between this ratio (on a log scale) and distance is −0.6.

Try it: look up any station’s day in the Lab →

The exceptions are the fun part. Canary Wharf, 7.5 km out, sits among the central stations at 5.1: a second business district doing exactly what the City does. And a handful of stations beyond 20 km are busier in the evening than the morning: Heathrow and Uxbridge, which are destinations in their own right rather than places people commute from.

That home/work split is also why I think the flow numbers lean towards people entering stations. If the sum counted entries and exits equally, a home station would be busy twice a day, not once. It’s still an inference, though.

Night-out stations and a station with no rush hour

Two more measures pick out different types of station: the share of a station’s day that falls after 20:00, and the share inside the two rush windows.

The median station takes 8.3% of its daily flow after 20:00. Covent Garden takes 32.3%, Leicester Square 30.5%, Piccadilly Circus 25.9%. That’s the West End, as you’d guess. At the other end, the median station does 56.3% of its business in the rush hours, while Heathrow Terminal 5 does 35.8% and Terminals 2 & 3 36.9%. Planes don’t keep office hours.

Four stations, four shapes. Elm Park spikes once in the morning. Goodge Street spikes once in the evening. Leicester Square builds slowly through the day and stays high late. Heathrow is roughly flat, a long plateau with no rush hour at all.

Inside the train

Station flow shows where people get on and off. The second dataset shows what happens in between: for each station and direction, a band from 0 to 6 for how full the train is when it leaves. Higher is fuller. That makes it possible to follow a single line and watch a train fill.

Here is the westbound Central line in the morning, from Epping through east London, the City and into the West End.

Read the 08:15 column from top to bottom. A train leaving Epping is at band 1. By Leytonstone it’s at 4, by Stratford 5, and leaving Bethnal Green and Liverpool Street it hits 6, the top of the scale. Then it empties: 5 leaving Bank, 4 at St Paul’s and Chancery Lane, and by Holborn it’s down to 2, where it stays all the way to Marble Arch. The crowd boards across east London and gets off in the City. On the West End stretch, from Tottenham Court Road to Marble Arch, the train never goes above band 2 all morning.

Across the whole network, 37 of 747 station-to-station stretches ever reach band 6. The ones that stay at band 4 or above the longest:

Does train fullness show the same inward wave as station flow? Partly. For each inbound stretch I took a weighted average of when in the morning it’s full. That time does get later nearer the centre (correlation −0.41): about 08:00 for stretches 8–15 km out, 08:09 at 3–8 km, 08:22 in the centre. But the outermost stretches, beyond 15 km, come in at 08:08, later than the band inside them. A 0–6 scale is coarse, and trains near the ends of lines are fairly empty whenever you measure them, so I wouldn’t read much into that last row. The wave is clearest in the station data.

What the data says

• The morning rush moves inward. Across the network, outer stations peak around 08:00 and central ones around 08:30. The pattern holds in aggregate even though individual lines are bumpier.
• Stations have characters. Home stations spike in the morning, work stations in the evening, and the ratio between them tracks distance from the centre, with Canary Wharf, Heathrow and Uxbridge as the exceptions you’d hope a good measure would catch.
• The West End and the airport break the pattern. Covent Garden does a third of its business after 20:00; Heathrow barely has a rush hour.
• You can watch a train fill up. The morning Central line goes from band 1 at Epping to the top band at Bethnal Green and back down to band 2 by Holborn.
• The evening doesn’t run the wave in reverse. Outer stations still peak first after work, and without labelled entries and exits I can’t say why.

The part I’ll remember is the bug. My first chart said Brixton was the second-busiest station on the Underground, and it looked plausible enough that I nearly kept it. The station-by-station timing held up once I summed the series properly, but the ranking only became believable after I stopped trusting the first number. The notebook is here if you want to follow your own line.

The shape of the Gita showed the pictures: a map of 700 verses, the regions the text falls into, the way its three voices each use different words. This post is the plumbing underneath those pictures. Before you can ask a text a statistical question you have to turn it into something a machine can hold, and I wanted that step to be honest: nothing appearing from nowhere, and the whole thing rebuildable from the source with one command.

What comes out is a graph of 5,314 nodes and 21,947 relationships, and every single one of them can be traced back to a verse file.

Method note. The source is my digital edition of the Gita: 701 verse records, each a markdown file with the chapter and verse number, the Sanskrit, a transliteration, a word-by-word gloss, and an English translation. Everything below is deterministic and idempotent (re-running rebuilds the same graph with no duplicates), and the pure parsing and graph-building logic is covered by 89 unit tests that touch neither the database nor the network. Code and the full ontology live in the gita-knowledge-graph folder on GitHub.

The verse is the unit

Everything hangs off the verse. Parsing one is deliberately boring, exact string and regular-expression work rather than anything clever: pull the chapter and verse number from the front matter, take the text under the ## Translation heading, and read the word-by-word table under ## Word Meanings. That gives the spine of the graph: one Text, its 18 chapters, and 701 verses, chained in reading order so you can walk the book verse by verse.

Two facts about each verse are worth a little care. The first is who speaks it. The Gita marks a change of speaker with a prefix, “The Blessed Lord said,” or “Arjuna said,”, but most verses carry no prefix at all because the speaker has not changed. So the parser reads the prefix when it is there and otherwise inherits the previous speaker, which is exactly how a human reads it. That resolves all 701 verses to one of four voices and, from the speaker, an addressee.

The second is the honorifics. The Gita almost never uses plain names; it calls Arjuna Partha and Dhananjaya, and Krishna Hrishikesha and Madhava. I tag those with a rule-based matcher seeded from a fixed list of 22 epithets, not a statistical name-finder. That is a deliberate choice: the list is small and known, so a rule is both more accurate and more honest than a model that might hallucinate a name.

Two vocabularies

A verse has two texts, and I did not want to throw either away. So the graph carries two parallel vocabularies.

The English layer comes from the translation. A standard NLP pipeline (spaCy) lemmatises it and keeps the nouns and verbs, which become 1,171 distinct terms joined to their verses by 6,156 weighted links. This is the statistical layer, the one that later feeds themes.

The Sanskrit layer comes from the word-by-word gloss. This one needed more care, because a word-by-word table is full of grammatical noise: pronouns, particles, and speaker markers that carry no meaning. So the parser drops those with a stoplist and normalises inflected surface forms back toward their root, so that karmani, karmana, and karma-phala all land on karma. That leaves 3,340 Sanskrit terms linked to their verses 7,995 times. The design rule I held to: English drives the embeddings later, because it is reproducible, but Sanskrit drives meaning, because it is the source language.

From words to ideas

Two vocabularies are still just words. The next layer turns them into ideas, and it does so deterministically, with no model guessing.

Themes are 13 broad topics (karma, dharma, bhakti, jnana, yoga, and so on), each defined as a set of English lemmas. A verse links to a theme when it uses those lemmas, and the strength of the link is just how often. Concepts are 22 philosophical categories grounded the other way, in the Sanskrit terms: a verse expresses atman when it contains Sanskrit words rooted in atman. The two layers meet on the nine names they share (karma, dharma, yoga, bhakti, jnana, moksha, atman, brahman, guna), bridged so a query can cross from the English index into the Sanskrit ontology.

Because every verse now carries themes, you can ask which ideas keep company. This is node similarity over the shared verses: two themes score highly when they tend to appear in the same verses.

The cast, the conches, and the glories

The nicest part to build was the narrative layer, because it is pulled straight out of the glosses rather than curated by hand. Scan the word meanings for known names and you recover the cast: 18 characters, from Arjuna and Krishna down to minor warriors, joined to the verses that name them by 310 links.

Two smaller entity layers come from the same glosses. Chapter 1 names six war-conches, and the graph knows which warrior blows each one:

And Chapter 10, where Krishna lists his divine glories, is captured as the 13 verses that explicitly declare “I am…” (in Sanskrit, asmi). I kept that layer strict on purpose: it records that Krishna declares a glory, but it does not try to pair each “I am” with its object, because the Sanskrit word order there is inconsistent and any automatic pairing would be guessing. I would rather record less and have all of it be right.

Meaning by number

The last layer is the one the map is built on. Every verse gets a 768-dimensional embedding of its English translation only, from a pinned all-mpnet-base-v2 model, and I compare every pair by cosine similarity. Rather than keep every faint resemblance, I calibrated a threshold: the notebook tries several cutoffs, checks how much of the text stays connected and how many cross-chapter theme links survive, and settles on 0.55, which keeps 2,260 verse-to-verse similarity links.

One deliberate piece of friction here: those similarity edges are the only part of the build that is not written automatically. The notebook stops and makes me eyeball a sample of the pairs and approve them before it loads them. It is the one place where the graph makes a judgement about meaning, so it is the one place I kept a human in the loop.

The whole thing, and why it is built this way

Stack all of that up and you get the graph. Here is its backbone, everything except the two dense word layers, which would otherwise drown the picture:

The one principle that kept the whole thing honest is that every node and edge has a provenance, and it is always one of four kinds. It is a seed (a hand-curated constant, like the chapter names or the theme definitions), something extracted directly from a verse (the speaker, the epithets, the Sanskrit terms, the conches), something derived deterministically from what was extracted (the themes and concepts), or something computed by the pinned model (the embeddings and the similarity). Nothing in the graph is a guess dressed as a fact, and if you disagree with a link you can always trace it back to the verse it came from.

That discipline is the actual point. The pictures in the shape of the Gita are only worth looking at because the thing underneath them is dull, checkable, and rebuilds from the text with one command. The analysis was the fun half; this is the half that makes it trustworthy.

The Bhagavad Gita is a conversation of about 700 verses, 18 chapters, one battlefield. I have been building a verse-by-verse digital edition of it as The Gita Project, and once every verse had its Sanskrit, a transliteration, a word-by-word gloss, and an English translation, a different question started nagging at me. Not what does it say, which people have argued about for two thousand years, but a smaller and more answerable one: what does the text look like when you map it?

So I loaded all of it into a graph database, gave every verse a numeric fingerprint, and let a few standard algorithms draw the picture. This post is what came back. None of it settles a single theological argument. But the data does say some concrete things, and one of them, about how the three speakers talk, surprised me.

Method note. The source is my own digital edition of the Gita: 701 verse records (the text is traditionally counted as 700; this edition also carries Arjuna’s opening question in Chapter 13, which some recensions omit, so it runs to 701), each with its English translation and its Sanskrit Word Meanings. I loaded them into a local Neo4j graph, then layered themes, Sanskrit-grounded concepts, the character cast, and semantic similarity on top. Every verse also gets a 768-dimensional embedding of its English translation only, from a pinned sentence-transformers/all-mpnet-base-v2 model, and verse-to-verse similarity edges are kept above a calibrated cosine threshold of 0.55 (2,260 pairs). The graph analytics use the Neo4j Graph Data Science library. The whole thing is deterministic and rebuildable; code and the full ontology live in the gita-knowledge-graph folder on GitHub. How the graph itself is built is a companion post, Building a knowledge graph of the Bhagavad Gita. Numbers in this post come straight from those notebooks.

A map of 700 verses

Every verse is a point in 768-dimensional space. That is impossible to look at, so I squashed it down to two dimensions with t-SNE, where verses that mean similar things end up near each other. Then I coloured each point by the community a graph algorithm (Louvain) finds in the similarity network, and labelled each region by the theme that is most distinctive to it.

The algorithm splits the text into 47 communities, and it is a genuinely tight partition: 77% of all similarity links fall inside a community rather than between communities, and the modularity score is 0.69. A second, independent algorithm (Leiden) recovers a closely matching split of 49 groups, so this is not an artefact of one method.

The communities also do not respect the chapter numbers, which is the part I keep returning to. Colour the exact same map by chapter instead, and the colours smear across the whole plane rather than forming 18 neat blocks:

The largest semantic communities each span ten to sixteen different chapters. The Gita returns to its core ideas again and again, in different chapters, in language similar enough that a machine groups them without ever being told what a chapter is. The chapter divisions organise the reading; the topics ignore them.

What each region is about

Naming a cluster is harder than finding it, because the obvious method fails. If you just ask which theme is heaviest in each community, almost every region comes back “karma”, because action is discussed nearly everywhere. So instead I measured lift: how much more a theme appears in a community than in the text overall. That surfaces what makes a region distinctive rather than what is simply common.

Pick the verse nearest each community’s centre and the regions come into focus. The biggest community, 90 verses, is a devotion cluster; its centre is 12.6, “But to those who worship Me, renouncing all actions in Me…”. A second region of 73 verses is pure karma-yoga; its centre is 3.30, “Renouncing all actions in Me, with the mind centered on the Self…”. A third, 44 verses, is about the senses and the mind, centred on 2.55, “When a man completely casts off, O Arjuna, all the desires of the mind…”.

One honest caveat before anyone reads too much into it: these are semantic clusters, grouped by how the English translations read, not a doctrinal map. They line up loosely with the tradition’s sense that the Gita moves through action, knowledge, and devotion, but I did not build them to prove that, and I would not lean on them to.

The centre of gravity

If verses are a network, some sit closer to the middle than others. PageRank, the algorithm that made Google, scores a verse highly when many other well-connected verses resemble it. Run it on the similarity network and the single most central verse in the whole Gita is 12.6, followed by 12.7, both from Chapter 12, the Bhakti Yoga chapter on devotion.

I want to be careful about what this means. PageRank rewards a verse for echoing many others, so the “centre” is really the text’s most representative language, the lines that restate its recurring promise most plainly. That does not make them the most important verses. It only means that if you had to pick the ones the rest of the book most sounds like, the machine points at Chapter 12’s devotional summations.

Three voices, three vocabularies

The Gita is a dialogue, and the graph knows who speaks each verse. Four voices carry it, but the floor is not shared evenly at all.

The interesting question is not who talks most, it is whether they talk differently. To test that I used keyness, the standard corpus-linguistics measure: for each speaker, a log-likelihood test flags the words they use far more often than the other voices do. It is the same idea behind “signature words”. The result splits the three main speakers cleanly, and it reads like the story itself.

Krishna speaks the language of the teacher: his signature words are action, self, sacrifice, knowledge, attachment, intellect. Arjuna, the warrior having a breakdown at the edge of a war he does not want to fight, over-uses kill, family, battle, destruction, and, tellingly, mouth and tooth, the vocabulary of his terrifying vision of the divine in Chapter 11. Sanjaya, the narrator relaying the scene to a blind king, deals in son, conch, archer, army, king, the furniture of the battlefield he is describing. Nobody labelled these voices by role; a frequency test pulled the teacher, the panicking student, and the war reporter apart on its own.

There is a quieter signal in the same data. Arjuna leans into the theme of dharma, duty, nearly three times as heavily as the text does on average (a lift of about 2.9), which is exactly the knot he is tied in. The machine found his crisis by counting words.

The honest limits

A few things this cannot do, stated plainly so the pictures above are not oversold. The embeddings are built from the English translation only, so the map reflects one translator’s choices as much as the Sanskrit; a different translation would move the dots. The keyness test runs over content words (nouns and verbs), so it captures what each voice talks about better than how they phrase it. The community detection is semantic, not doctrinal, and the second algorithm agrees with the first on only about 60% of the exact assignments (adjusted Rand index around 0.58), so treat the fine boundaries as soft. And a t-SNE map distorts distance to fit everything on a page; read it for neighbourhoods, not for precise gaps.

What survives all of that is small but real. The Gita’s topics run across its chapters rather than staying inside them, its most representative language is devotional, and its three voices use measurably different words that match the roles they play. None of it needed me to interpret a single verse.

The map and the build both leaned on the English translation, because that is what the embeddings are made from. But the graph also carries the other half of every verse: the word-by-word Sanskrit gloss, normalized into 3,340 Sanskrit terms. This post does the least glamorous thing you can do to a vocabulary, which is count it, and finds that the counting says more than I expected.

Method note. The Sanskrit terms come from the per-verse Word Meanings tables, with particles and pronouns dropped and inflected forms normalized toward their roots. The graph records that a term appears in a verse, but not how many times, so throughout this post frequency means document frequency: how many of the 701 verses a term shows up in. A “hapax” is therefore a term that appears in exactly one verse. Two honest caveats up front: the normalization is imperfect (you will see a stray inflected form or pronoun below), and document-frequency counting slightly inflates how rich the vocabulary looks. Code lives in the gita-knowledge-graph folder on GitHub; numbers come straight from the notebook.

The most repeated words are the ideas

Rank the Sanskrit terms by how many verses contain them, and the top of the list is not incidental vocabulary. It is the philosophy itself.

Karma appears in 92 of the 701 verses, more than one verse in eight. Under it sit jnana, atman, yoga, manas, brahman: exactly the concepts the tradition says the Gita is about. You could have guessed the list. What I did not expect was that a blind word-count would recover it with no idea of meaning at all.

I am keeping this honest, so: the raw list also has a couple of passengers. The pronoun mayi (“unto me”) slips past the stoplist, the name Arjuna is frequent for obvious reasons, and you can spot buddhih sitting separately from buddhi, an inflected form the normaliser missed. None of that changes the shape of the top of the list, but it is the kind of thing that would, if I swept it under the rug, quietly make the numbers a lie.

It behaves like a language

Word frequencies in natural language follow Zipf’s law: the n-th most common word appears about proportionally to 1/n, so a log-log plot of frequency against rank falls on roughly a straight line. Does a 700-verse Sanskrit text, counted by document frequency, do the same?

It does, with a fitted slope of about -0.69. That is shallower than the textbook -1, which is expected: document frequency flattens the very top (a word can only be counted once per verse, no matter how often it is chanted inside that verse), and the text is short. But the law is unmistakably there. The Gita’s Sanskrit is not a special code; it distributes its words the way language does.

Most words appear once

The flip side of a Zipf curve is a long, thin tail, and here it is dramatic. 56% of all Sanskrit terms, 1,865 of the 3,340, appear in exactly one verse. Three quarters appear in two verses or fewer. The vocabulary is a small hard core of repeated concepts sitting on top of a huge scatter of words used once and never again.

You can watch that happen by reading the text in order and counting how many new words each verse brings in:

The curve hardly flattens. By the halfway point of the book only about 1,970 of the 3,340 terms have appeared, so the second half is still introducing more than a thousand new ones. In quantitative-linguistics terms the vocabulary-growth (Heaps’) exponent is about 0.88, high for a text this size.

Here is where I have to be careful, because it is tempting to read that as “the Gita is extraordinarily rich” and stop. Part of the richness is real: it ranges over metaphysics, ethics, devotion, and a battlefield. But part of it is Sanskrit’s morphology, a single root throws off many inflected surface forms, and my normalization only pulls them toward their roots, not perfectly onto them (remember buddhih and buddhi). So the type count is inflated, and the honest reading is: the Gita has a genuinely broad vocabulary, made to look even broader by the language’s grammar and my imperfect tidying of it.

Which chapters are densest

Finally, not every chapter carries the same weight of vocabulary per verse.

The densest chapter is 16 (Daivasura Sampada Vibhaga, the divine and demoniac natures) at about 17 terms per verse, a chapter that works by piling up lists of qualities. Chapter 1, Arjuna’s despair on the field, is close behind at nearly 14, thick with the names of warriors and weapons rather than concepts. The leanest is Chapter 7 at about 9, where Krishna settles into steady, repetitive teaching and reuses the same core words. That is enumeration, not depth: chapter 16 lists where chapter 7 repeats.

None of this interprets a single verse; it only counts them. But the count lands on something real: the words the Gita says most often are its core ideas, the frequencies follow the same law as any language, and it keeps reaching for new words right to the last chapter.

Then the summer came. The Met Office now says a record-warm UK summer is increasingly likely, with temperatures staying well above average. So I did the obvious thing: grabbed the June and July readings, added them to the same 37-year daily series, and looked. I wanted to resist the tidy headline (“record heat drains the reservoirs!”) and let the readings talk first, because they usually say something more interesting than the slogan.

Data & licence. London reservoir levels via the London Datastore, sourced from the Environment Agency’s water situation reports. Public sector information licensed under the Open Government Licence v2.0. Data now runs 1 Jan 1989 to 31 Jul 2026. The “warmest summer” framing is context from the Met Office (link above); this analysis works only with reservoir levels, not temperature. Notebook: reservoir_analysis_2026-07.ipynb.

First, just add the days

No cleverness here. I extended the sawtooth by two months and shaded 2026 so the newest readings stand out against four decades.

Zoom into the right-hand edge and the shape is clear: after peaking in spring, both lines fall through June and July. The orange Lower Thames line keeps going, sliding to 72% by 31 July (the Lower Lee ends at 82%). That’s a real drop, but a real drop isn’t a finding. The daily view can’t tell me whether 72% is alarming or ordinary for late July. For that I need to compare 2026 against its own history.

How unusual is it? Draw the normal band

The honest test is the same one that flagged 2022 in the first post: plot 2026 on top of the normal band. The grey envelope is the full day-by-day min-to-max of every other year (1989-2025); the dashed line is the typical year. If 2026 sits inside the band, it’s within normal. If it rides the edge, it isn’t.

Here the two groups part ways, and this is the actual story.

The Lower Thames (bottom panel) starts the year mid-band, tracks the typical line into spring, then peels away and rides the bottom edge of the grey band all summer. Its 72% on 31 July isn’t just low: it matches the lowest level ever recorded for that date in the 37-year record. For late July, the Thames group has never been emptier. It’s on the floor of its own range.

The Lower Lee (top panel) tells a softer version. It fell too, but it’s sitting comfortably inside the band, around the 24th percentile for the date: low-ish, nowhere near a record. Same city, same summer, two different responses. The structural gap between these systems, which the first post flagged, is doing real work here.

So the headline the slogan wanted, “record heat empties the reservoirs,” is half right and half wrong, and the interesting half is the wrong one. One group is at a record low for the date; the other is just a bit dry. Which raises the obvious question: did the Thames get there because this summer drained it unusually fast?

Was the drawdown actually the fastest?

A hot, dry summer should show up as speed: how far the reservoirs fall between late spring and mid-summer. So for every year I measured the June-to-July drawdown (average level in the last week of May minus the last week of July) and ranked them.

And here the data complicates the neat narrative. The Lower Thames fell about 17 points this summer, steep enough to rank 5th-fastest of 38 summers, but not the record. 2022 fell faster (about 24 points), and so did several other years. The Lower Lee’s 13-point fall ranks 7th; its record belongs to 2018 (about 22 points). A summer the Met Office thinks may be the warmest on record did not produce the fastest drawdown on record.

So how is the Thames at a record low for the date if the fall wasn’t record-breaking? Because of where it started. The Lee refilled to a brimming 100% over winter; the Thames only reached about 93%. A slightly lower starting line, plus a steep but not unprecedented fall, was enough to land it on the floor. The record isn’t about how hard this summer pulled. It’s about the summer landing on a spring that never quite topped the Thames up.

What the data says

Reading the data rather than the headline:

• The Lower Thames is at a record low for the date: 72% on 31 July, matching the emptiest late-July in 37 years. That part of the scary story is true.
• But the drawdown wasn’t the fastest. 2022 and 2018 both drained faster. Record heat did not mean record-speed emptying.
• Starting height mattered as much as the heat. The Thames began summer a touch lower than the Lee and ended on the floor; the Lee started full and stayed inside its normal band.
• The real test is still ahead. July isn’t the seasonal trough; that comes in September or October. If the Thames keeps hugging the bottom edge, that autumn minimum is the number to watch, against the deep droughts of 1996-97 the first post dug up.

None of this downplays the summer. A record-low-for-the-date reservoir is worth watching. But the outcome I’d have guessed (“hottest summer, fastest drain”) isn’t the one the data supports. The reservoirs are spending a full winter’s savings fast, not running on empty, and the number that will really matter hasn’t happened yet. I’ll come back in October. The notebook is here if you want to watch the floor with me.

There’s a game I play on walks. Spot a blue plaque, cover the name with my thumb, and guess who lived there. I’m almost always wrong, and that’s the point: a blue plaque is a tiny act of civic memory, a decision that this person, in this building, is worth stopping a stranger in the street for. I’d walked past dozens before it occurred to me to ask the obvious question: who makes that list, and what does the whole list look like at once?

English Heritage runs the scheme, and their website will show you the plaques twelve at a time. I wanted all of them on one screen. So I scraped the lot, all 1,037 plaques, and went looking for the shape of London’s memory. What I found is that the shape itself is the story.

What the data is

Every plaque has a page: a name, an address, a category, the exact words on the ceramic, and a pin on a map. The listing is served by a quiet little search API, and each plaque’s coordinates are tucked into the map code on its own page. Two passes (one for the list, one to open all 1,037 detail pages) and it folds into a single tidy table.

Data & licence. Plaque data scraped in August 2026 from the English Heritage blue plaques site (listing API + individual plaque pages). The content belongs to English Heritage; this is a personal, non-commercial analysis of publicly visible information, with attribution. The full scraper and analysis notebook live in my CodePlayground repo.

Before trusting any of it, I checked how complete each field was. Names, boroughs, coordinates, inscriptions and categories all came back at basically 100%. Birth and death years covered 96% of entries: 991 of the plaques commemorate a specific person (the rest mark buildings and events). One thing I couldn’t get: the date each plaque went up. It isn’t published anywhere on the site, which quietly kills the question I most wanted to ask: how long after death does recognition arrive? Some questions the data just won’t answer, and it’s more honest to say so than to fudge it.

First, put them all on the map

No aggregation, no cleverness. Just drop all 1,036 plaques-with-coordinates onto London and colour them by category.

You don’t need statistics to see it. There’s a dense, glowing core in the centre and west, and then the rest of London, enormous, populous, historic London, thins out to scattered dots. My first thought was that I’d made a mistake. I hadn’t. London’s official memory really is packed into a small footprint. (Just how tightly packed, statistically, is the subject of the sequel to this post, where the same points become a geometry problem.)

Three boroughs hold two-thirds of it

I counted, and the concentration is even starker than the map suggests.

Just three boroughs, Westminster, Kensington & Chelsea, and Camden, hold 69% of every blue plaque in London. Westminster alone has 334, nearly a third of the whole scheme. Meanwhile Barking & Dagenham, Sutton and the City of London manage one apiece.

Zoom in past the borough line and the clustering gets almost comically specific. The single most-plaqued street in London is Cheyne Walk in Chelsea, with ten; the densest postcode district is NW3, Hampstead, with sixty-nine.

And each borough remembers a different kind of person. Westminster’s plaques lean towards politicians; Kensington & Chelsea and Camden lean towards writers and artists. Westminster remembers power; Chelsea remembers art. That’s not a coincidence: it’s the first hint of the feedback loop I’ll come back to.

Some of this is honest history: the West End and Bloomsbury were where the writers, scientists and statesmen of a certain era clustered, near the salons and institutions and each other. But some of it is self-fulfilling. Plaques mark the grand, surviving townhouses of central London, and grand surviving townhouses are exactly where well-documented, well-connected, plaque-worthy lives happened. The map isn’t just showing where remarkable people lived. It’s showing where the kind of person the scheme was built to remember lived.

Who gets remembered

Sort the plaques by what the person is remembered for, and London reveals itself as, above all, a city of writers.

Literature comes first, then politics and administration, then the fine arts and music. It’s a portrait of what a culture chooses to enshrine: the people who left documents (books, laws, paintings, scores), the kind of legacy that keeps a name legible a century later.

The gap isn’t flat

Here’s the number the scheme itself is quietly self-conscious about. There’s no gender field in the data, so I inferred it from first names (with an offline name-to-gender library) and honorifics like Sir and Dame. It’s imperfect: 122 names it couldn’t resolve, and it will misfire on some, so I only report the share among names it could resolve. Even so, the result is stark: fewer than one in five commemorated people are women.

But the single figure hides the interesting part. Split the women’s share by field and it swings wildly, from near-parity in one category to an absolute, unbroken zero in others.

Women appear most in philanthropy and reform (close to half) and on the stage: theatre, film, dance. They all but vanish from politics (3%) and hit a flat zero in engineering, industry and invention. The pattern isn’t really about the plaques; it’s about which doors were open to women in the first place, preserved in ceramic. The honorifics say the same thing more bluntly: 174 knighted Sirs to 25 Dames.

There is one hopeful thread. Stack the commemorated people by their birth decade and the pink band, however thin, grows as you move toward the present.

Among people born in the mid-1700s the female share was under 5%; among those born around 1900 it’s up past 30%. The scheme is trying to correct, and English Heritage has said as much publicly. You can watch that intention arrive, one decade at a time, but you’re watching it climb out of a very deep hole.

What the walls actually say

The inscriptions are their own small corpus. Throw all 1,037 into a pile and count the words, and the most common ones read like a poem about the dataset in miniature: poet, writer, painter, novelist, artist, and, sitting right among them, Sir.

There’s also a quiet grammar to how each plaque relates its person to its building. Nearly every one states a verb, and one verb runs away with it.

725 plaques, seven in ten, simply say “lived here.” Not born, not died, not worked. Lived. The blue plaque is a marker of domesticity made historic: this ordinary front door held an extraordinary ordinary life. And almost all of them say it on the same object: 88% of the plaques are the familiar ceramic roundel, with a stubborn handful in bronze, stone or slate.

How long did they live?

A small, humane detour. For the 991 people, birth and death years give a distribution of lifespans.

The median is 72, with a long tail into the nineties: Sir Robert Mayer made it to 106. Which makes a certain grim sense: the surest route onto a wall of long-term public memory is to do enough, for long enough, to be remembered. But the left tail is where the heartbreak lives: a cluster who died in their twenties and got a plaque anyway: John Keats at 26, the sculptor Gaudier-Brzeska at 24, the SOE agent Violette Szabo at 24. Fame usually rewards patience. Just occasionally it rewards a comet.

Memory isn’t merit

Put the threads together and they all point the same way. The geography (a tiny central core), the gender (fewer than one in five, and zero in whole professions), the categories (writers and statesmen), the honorifics (Sirs outnumbering Dames seven to one), these aren’t four separate findings. They’re four views of a single filter.

Blue plaques don’t really mark where remarkable people lived. They mark where the documented, connected, establishment class lived, and that class was central, male, literary and titled. The filter is self-reinforcing: grand surviving houses in Zone 1 are both where those lives happened and where a plaque can be hung today, so the memory keeps pooling in the same square mile.

None of this is a knock on the plaques. I still play the guessing game, and I still lose. But a wall of memory is also a mirror, and it’s worth occasionally asking who’s reflected in it and who isn’t: the outer boroughs, the women who never got the door opened, the engineers who built the city and got nothing.

A companion piece

That’s the human story. There’s also a purely mathematical one hiding in the same 1,036 dots: how clustered London’s memory is (provably), what territory each plaque owns, and the shortest possible walking tour of the whole city. I pulled that apart in a second post: The geometry of memory →. If you’d rather poke at the data yourself, the scraper and notebook are one afternoon of pandas and Plotly. Go find the plaque nobody put up.

In the first half of this I scraped every English Heritage blue plaque in London and asked who gets remembered. The headline was that London’s memory is astonishingly concentrated: you could see it just by dropping the dots on a map.

But “you can see it” is not the same as “it’s true.” This post is the data scientist’s follow-up: take the same 1,036 geolocated plaques and treat them as a spatial point pattern and an optimisation problem. Can I prove the clustering? What territory does each plaque own? Where are the hotspots, found by an algorithm, not my eyeballs? And, for fun, what’s the shortest walking tour of the whole city?

Method note. The 1,036 plaque locations were scraped in August 2026 from the English Heritage blue plaques site, as a personal, non-commercial analysis with attribution (the full licence note is in part one). Everything below works in the British National Grid (EPSG:27700), so distances and areas are in real metres, not degrees. Code lives in the geospatial notebook.

Is it clustered? Prove it.

Before any picture, a statistic. The eye is easily fooled, so the honest first move is a test. The Clark-Evans index compares the average distance from each plaque to its nearest neighbour against what you’d expect if the same number of plaques were scattered at random across the same area. Below 1 means clustered; above 1 means spread out; a big z-score means it’s not luck.

The index comes out at R = 0.54, with a z-score of about -28. In plain terms: plaques sit roughly half as far from their nearest neighbour as random scattering would predict, and the chance of that happening by luck is essentially nil. The median plaque has another plaque just 100 metres away. Here’s the distribution behind the number:

Your nearest plaque: a Voronoi map

Statistics proven, now the pretty part. A Voronoi diagram carves the plane into one cell per plaque, where every point in a cell is closer to that plaque than to any other, the plaque’s “catchment area.” Colour each cell by its size and the density story becomes visceral.

In the West End the cells are so small they blur into a mosaic, a plaque every hundred metres, each owning a scrap of pavement. Out toward Bromley or Croydon a lone plaque can own kilometres in every direction. The area of your nearest-plaque territory is, in effect, an inverse density map, and it screams the same thing the dots did, now with an area attached to it.

Try it: break the clustering yourself in the Lab’s Voronoi playground →

The heat of memory

The same information, smoothed into a continuous surface: a kernel-style density heatmap. No borough lines, no cells, just where memory glows hottest.

Hotspots, found by algorithm

I don’t want to decide where the clusters are: that’s cheating. So I handed the job to DBSCAN, a density-based clustering algorithm: any group of at least six plaques all within 350 metres of one another becomes a cluster; everything else is “scattered.” Told only the coordinates, it recovers the hotspots on its own.

It lands exactly where you’d expect: one giant blob over Westminster, another over Kensington & Chelsea, satellites in Bloomsbury and Hampstead, but the point is that it found them. Half of all London’s plaques fall into just the two largest clusters.

The memory network

A different lens from graph theory: what’s the shortest possible set of links that connects every plaque into one network, with no loops? That’s a minimum spanning tree, and drawn on the map it looks like the nervous system of London’s memory.

The whole tree is 371 km long, but look at how the “wire” is spent. It’s dense and short in the centre, where neighbours are 100 m apart, and it throws long lonely spans out to the isolated plaques on the fringe.

The Grand Tour: a travelling salesman in London

Finally, the classic. Pick one representative plaque per borough (the one nearest each borough’s centroid) and ask: what’s the shortest loop that visits all of them and returns home? That’s the Travelling Salesman Problem, and I solved it from scratch rather than calling a solver, because the method is half the fun.

The recipe is two moves. First, a nearest-neighbour heuristic: start somewhere, always walk to the closest unvisited borough. It’s greedy and it leaves ugly crossings. Then 2-opt: repeatedly find two edges that cross, snip them, and reconnect the other way, which always shortens the tour, until no swap helps.

Nearest-neighbour alone gives a 219 km tour with tell-tale crossings. A pass of 2-opt untangles it down to 180 km, an 18% saving for a couple of dozen lines of Python, and the visual proof is that the crossings are gone. That is the whole point of local search: a dumb first guess, plus a simple “is this knot removable?” rule, gets you most of the way to optimal.

What the geometry taught me

• The clustering isn’t a trick of the eye: Clark-Evans R = 0.54, z of about -28 makes it statistically undeniable.
• Voronoi territories turn density into area: tiny tiles downtown, whole boroughs at the edge.
• DBSCAN rediscovers the hotspots from coordinates alone, a nice reminder that the structure is in the data, not in my assumptions.
• A hand-rolled nearest-neighbour + 2-opt TSP shaves ~18% off the naive route, which is the whole value proposition of local-search optimisation in miniature.

If the first post was about who London remembers, this one was about the shape of that memory, and the shape, it turns out, is provably, beautifully lopsided. The notebook is here if you want to run the tour yourself.

I landed in London that November, a few months too late for it. You don’t have to have been here to have heard about it, though. The grass had gone the colour of straw, the hosepipe bans had come and gone, and “drought” was a headline word all summer. It was one of the first pieces of local weather-lore I picked up. So when I found a dataset with daily reservoir levels for London going back to 1989, the first thing I did was hunt for 2022, the drought everyone had told me about. I expected a cliff. The worst dip in the whole record.

It wasn’t. And that little wrongness, a memory I’d inherited rather than lived, is what turned a quick afternoon poke into this post.

What the data is

The London Datastore publishes the level of two big reservoir groups that supply the city:

• the Lower Lee group, and
• the Lower Thames group.

One row per day, one number per group, all the way from 1 January 1989 to 31 May 2026. Each number is just percent of capacity: how full the reservoirs were that day. That’s the entire dataset. No fancy features, no engineered columns. Just ~13,665 days of “how much water is in the tank.”

The part that delighted me, before I plotted anything, was how complete it is. 37 years, one reading every single day, and when I checked for gaps in the calendar there were none. 16 missing values in total, out of about 27,000. Public datasets tend to be messier than that. This one is a quiet feat of record-keeping.

Data & licence. London reservoir levels via the London Datastore, sourced from the Environment Agency’s water situation reports for England. Contains public sector information licensed under the Open Government Licence v2.0. The full analysis notebook lives in my CodePlayground repo.

Where these two groups sit

The dataset never says where these reservoirs are; “Lower Lee” and “Lower Thames” are just two column names. They’re real bodies of water, sitting in two very different corners of London. The Lower Lee group is the Lee Valley chain up in the north-east, above Tottenham and Walthamstow. The Lower Thames group is the cluster of big reservoirs out west, towards Staines and Heathrow. Two systems, one city, fed by separate rivers.

To draw them, I pulled the reservoir outlines straight from OpenStreetMap and coloured each polygon by its group. There are no coordinates in the levels CSV, so this is a separate open-data layer stitched on top.

Map data. Reservoir outlines and basemap © OpenStreetMap contributors, licensed under the Open Database Licence (ODbL). Fetched via the Overpass API in the analysis notebook.

First, just plot everything

Before slicing or summarising anything, it pays to be a little dumb and just plot every single day. No aggregation, no smoothing. Just drop all 13,665 points on a chart and see what the shape tells you.

There it is: a sawtooth. Up through the winter, down through the summer, over and over for close to four decades. Your eye catches the deep dips: the mid-1990s, a rough patch around 2005-06, and yes, 2022. Those are the droughts. But hold that thought about 2022, because the daily view is too noisy to rank them. For that we need to fold time.

The heartbeat: what an average year looks like

If every year follows near enough the same fill-and-drain rhythm, then I should be able to squash all 37 years onto a single 12-month clock and see the underlying pulse. Statisticians call this a climatology. I like to think of it as taking the reservoir’s resting heart rate.

So I averaged every January together, every February together, and so on. The shaded bands show the full range each month has ever spanned; the solid lines are the averages:

The Lower Thames peaks in late winter (think February, when it’s been raining for months and nobody’s watering a garden) and bottoms out around September to October, right before the autumn rains kick in and start the refill. The Lower Lee does the same dance but sits a few points lower the whole way round. That gap between the two groups isn’t random jitter; it’s there in almost every year. Two reservoir systems, same city, different personalities.

Which years ran driest?

Here’s where I went back for 2022. The fairest single number for “how bad was that year” is the annual minimum: the lowest the reservoirs fell at the worst moment of that year. Short bars are the scary years.

This is the bit that made me sit up. I’d arrived to the story of 2022 rather than the summer itself, so I had no yardstick of my own to check it against. The numbers were my only witness. For the Lower Thames, the driest years on record are:

2022, the drought I’d been assured was the worst, comes in fifth. The brutal squeeze was 1996-97, a slow two-year drawdown that pulled the Thames group down to 42%. For the Lower Lee, the deepest year was 1991 (48%), with 1992 close behind. The droughts that ran London’s reservoirs driest happened in the early-to-mid nineties, and most of us have forgotten them.

Memory is a headline. The data is a diary. When they disagree, I know which one I trust.

Putting 2022 under the microscope

None of this means 2022 wasn’t a real drought. It was. “Real drought” and “worst on record” are two different claims, though, and the honest way to tell them apart is to draw the year against its own history. Here’s 2022 for the Lower Thames, plotted on top of the normal band: the grey envelope is the full min-to-max range every other year has ever traced, day by day, and the dashed line is the typical year.

You can watch the story unfold across the year. 2022 starts out ordinary, then peels away from the typical line through that hot, dry summer and rides along the bottom edge of the band by autumn, low for the time of year, hugging the record floor without breaking it. That’s what a bad year looks like when it isn’t the record. The normal band is, to me, the single most honest chart in the whole notebook, because it answers the one question that matters: how unusual was this?

What I took away

• Water has a heartbeat. Fill in winter, draw down to an autumn low, then do it again the next year. Once you see the rhythm, it shows up everywhere.
• The Lower Thames runs fuller than the Lower Lee. Same pattern, year after year. It’s baked into how the two systems work.
• The drought you remember isn’t the worst one. 2022 made the news. 1996-97 set the record. The loud year sticks; the deep years fade.
• Complete data is a gift. The cleaning step was dull, so every surprise waited in the findings instead.

If you want to poke at it yourself (different reservoir, different year, your own definition of “drought”), the whole thing is a single Jupyter notebook with pandas and Plotly. Grab the CSV from the London Datastore, drop it in, and go looking for your own wrong memory. That’s the fun part.

Overview

The battery.py module contains several classes that model different aspects of a battery’s behavior:

• BasicSOHCalculator: Calculates the battery’s State of Health (SOH) based on the energy cycled and depth of discharge (DOD).
• TemperatureEfficiencyAdjuster: Adjusts the battery’s charging and discharging efficiencies based on temperature.
• Battery: Represents a battery with various properties and methods for charging and discharging.

Let’s dive into each of these classes and understand how they work.

BasicSOHCalculator

This class calculates the battery’s State of Health (SOH) based on the energy cycled and depth of discharge (DOD). The calculate_soh method takes in the initial SOH, the amount of energy cycled, and the DOD, and calculates the new SOH based on a degradation rate.

def calculate_soh(self, soh: float, energy_cycled_mwh: float, dod: float):
    """
    Calculates the State of Health (SOH) based on the given parameters.

    Args:
        soh (float): The initial State of Health.
        energy_cycled_mwh (float): The amount of energy cycled in megawatt-hours.
        dod (float): The depth of discharge as a fraction (0 to 1).

    Returns:
        float: The updated State of Health (SOH) after degradation.

    """
    base_degradation = 0.000005
    dod_factor = 2 if dod > 0.5 else 1
    degradation_rate = base_degradation * energy_cycled_mwh * dod_factor
    return soh * (1 - degradation_rate)

In the accompanying graph, observe the progression of State of Charge (SOC) and State of Health (SOH) values over time, illustrating the dynamic behavior of the battery system.

TemperatureEfficiencyAdjuster

This class adjusts the battery’s charging and discharging efficiencies based on the temperature. The adjust_efficiency method takes in the current temperature and the current efficiencies, and adjusts them based on a simple rule: for every degree Celsius the temperature is away from 25°C, the efficiencies decrease by 1%.

def adjust_efficiency(
    self,
    temperature_c: float,
    charge_efficiency: float,
    discharge_efficiency: float,
):
    """
    Adjusts the charge and discharge efficiency of the battery based on the temperature.

    Args:
        temperature_c (float): The temperature in degrees Celsius.
        charge_efficiency (float): The current charge efficiency of the battery.
        discharge_efficiency (float): The current discharge efficiency of the battery.

    Returns:
        Tuple[float, float]: A tuple containing the adjusted charge efficiency and discharge efficiency.
    """
    temp_effect = abs(temperature_c - 25) * 0.01
    new_charge_efficiency = max(0.5, min(charge_efficiency - temp_effect, 1.0))
    new_discharge_efficiency = max(
        0.5, min(discharge_efficiency - temp_effect, 1.0)
    )
    return new_charge_efficiency, new_discharge_efficiency

In the accompanying graph, observe the progression of State of Charge (SOC) and State of Health (SOH) values over time for each temperature, illustrating the dynamic behavior of the battery system.

Battery

This is the main class that represents a battery. It has various properties like capacity, efficiency, state of charge (SOC), state of health (SOH), and temperature. It also has methods for charging and discharging the battery, which adjust the efficiencies and update the SOH and cycle count.

The charge and discharge methods first adjust the efficiencies based on the temperature, then calculate the actual energy that can be charged or discharged, and finally update the SOC. They also call the update_soh_and_cycles method to update the SOH and cycle count.

def charge(self, energy_mwh: float):
    """
    Charges the battery with the specified amount of energy.

    Args:
        energy_mwh (float): The amount of energy to be charged in megawatt-hours.
    """
    self.adjust_efficiency_for_temperature()
    energy_mwh = min(energy_mwh, self.max_charge_rate_mw * self.duration_hours)
    actual_energy_mwh = energy_mwh * self.charge_efficiency
    self.soc = min(self.soc + actual_energy_mwh / self.capacity_mwh, 1.0)
    self.update_soh_and_cycles(energy_mwh)

The update_soh_and_cycles method updates the SOH using the BasicSOHCalculator and updates the cycle count based on the energy cycled.

def update_soh_and_cycles(self, energy_mwh: float):
    """
    Updates the state of health (SOH) and cycle count of the battery based on the energy cycled.

    Args:
        energy_mwh (float): The amount of energy cycled in megawatt-hours.
    """    
    self.energy_cycled_mwh += energy_mwh
    dod = 1.0 - self.soc
    self.soh = self.soh_calculator.calculate_soh(self.soh, energy_mwh, dod)
    self.check_and_update_cycles()

To explore the battery module in action, check out the Jupyter notebook here, where its functionality is demonstrated.

Wrapping Up

That’s a brief overview of the battery.py module in our OpenEnergy project. We’ve covered how the module models a battery’s behavior, including how it adjusts efficiency based on temperature, calculates the state of health, and handles charging and discharging operations. I hope this gives you a better understanding of how we’re simulating energy storage systems in our project. Stay tuned for more deep dives into other parts of the codebase!

As always, if you have any questions or suggestions, feel free to leave a comment below or open an issue on our GitHub repository. Happy coding!

Overview

This module combines the functionality of various pricing strategies including simulated price, average price, and forecasted price. It provides a unified interface for interacting with different pricing models.

Interface

The interface serves as a contract for all pricing models. It defines the methods that all pricing models should implement. This ensures that regardless of the pricing model used, the interaction remains consistent.

The interfaces.py file contains five interfaces: IPriceData, IPriceEnvelopeGenerator, IPriceNoiseAdder, and IPriceDataHelper. Each interface defines a set of methods that classes implementing the interface must provide.

IPriceData

The IPriceData interface is used for classes that retrieve price data. It defines a single method, get_prices, which takes a date as input and returns a tuple containing two lists of floats representing the buy and sell prices for that date.

class IPriceData(ABC):
    """Interface for retrieving price data."""

    @abstractmethod
    def get_prices(self, date: datetime.date) -> t.Tuple[t.List[float], t.List[float]]:
        """Get the prices for a specific date.

        Args:
            date (datetime.date): The date for which to retrieve the prices.

        Returns:
            Tuple[List[float], List[float]]: A tuple containing two lists of floats.
                The first list represents the buy prices, and the second list represents
                the sell prices.
        """
        pass

IPriceEnvelopeGenerator

The IPriceEnvelopeGenerator interface is used for classes that generate price envelopes. It defines a single method, generate, which takes a date as input and returns a list of price envelopes for that date.

class IPriceEnvelopeGenerator(ABC):
    """
    Interface for generating price envelopes.
    """

    @abstractmethod
    def generate(self, date: datetime.date) -> t.List[float]:
        """
        Generate price envelopes for the given date.

        Args:
            date (datetime.date): The date for which to generate price envelopes.

        Returns:
            List[float]: A list of price envelopes.
        """
        pass

IPriceNoiseAdder

The IPriceNoiseAdder interface is used for classes that add noise to a list of prices. It defines a single method, add, which takes a list of prices as input and returns a new list of prices with added noise.

class IPriceNoiseAdder(ABC):
    """
    Interface for adding noise to a list of prices.
    """

    @abstractmethod
    def add(self, prices: t.List[float]) -> t.List[float]:
        """
        Adds noise to the given list of prices.

        Args:
            prices (List[float]): The list of prices to add noise to.

        Returns:
            List[float]: The list of prices with added noise.
        """
        pass

IPriceDataHelper

The [IPriceDataHelper] interface is designed for classes that assist in handling price data, offering a suite of methods for date and price data manipulation. This interface facilitates obtaining the current date and time, calculating the date and time a week prior to a specified date, retrieving data for the preceding week, fetching data for the current date, and extracting prices for the current date.

class IPriceDataHelper(ABC):
    @abstractmethod
    def get_current_date(self, date: datetime.date) -> datetime.datetime:
        """Retrieve the current date and time based on a given date.

        Args:
            date (datetime.date): The date to convert to datetime.

        Returns:
            datetime.datetime: The current date and time.
        """
        pass

    @abstractmethod
    def get_prior_date(
        self, current_date: datetime.datetime, delta_days: int
    ) -> datetime.datetime:
        """Calculate the date prior to the current date by a specified number of days.

        Args:
            current_date (datetime.datetime): The current date from which to calculate the prior date.
            delta_days (int): The number of days before the current date to calculate.

        Returns:
            datetime.datetime: The calculated prior date.
        """
        pass

    @abstractmethod
    def get_prior_data(
        self,
        current_date: datetime.datetime,
        prior_date: datetime.datetime,
        data: pd.DataFrame,
    ) -> pd.DataFrame:
        """Retrieve data for a period between the prior date and the current date from a given dataset.

        Args:
            current_date (datetime.datetime): The end date of the period for which to retrieve data.
            prior_date (datetime.datetime): The start date of the period for which to retrieve data.
            data (pd.DataFrame): The dataset from which to retrieve the data.

        Returns:
            pd.DataFrame: The filtered dataset containing data between the prior and current dates.
        """
        pass

    @abstractmethod
    def get_current_date_data(
        self, current_date: datetime.datetime, data: pd.DataFrame
    ) -> pd.DataFrame:
        """Retrieve data for the current date from a given dataset.

        Args:
            current_date (datetime.datetime): The date for which to retrieve data.
            data (pd.DataFrame): The dataset from which to retrieve the data.

        Returns:
            pd.DataFrame: The filtered dataset containing data for the current date.
        """
        pass

    @abstractmethod
    def get_prices_current_date(
        self, current_date_data: pd.DataFrame, column_name: str
    ) -> t.List[float]:
        """Extract price data for the current date from a given dataset.

        Args:
            current_date_data (pd.DataFrame): The dataset containing data for the current date.
            column_name (str): The name of the column from which to extract the prices.

        Returns:
            List[float]: The list of prices for the current date.
        """
        pass

Simulated Price

The simulated price model uses statistical methods to generate a price that simulates market conditions. It’s useful for testing how systems would react to different market conditions.

The file we’ll look at is simulated_price.py. This file contains the logic for generating simulated price data. It’s a crucial part of the project as it allows us to create realistic, yet artificial, price data for testing and development purposes.

SimulatedPriceEnvelopeGenerator

The SimulatedPriceEnvelopeGenerator class is responsible for generating a simulated price envelope based on a sine wave. The envelope is defined by a number of intervals, a minimum and maximum price, and a peak start and end index.

The generate method is where the magic happens. It generates a list of price values for a given date. The prices are calculated based on the position of the interval in relation to the peak start and end. If the interval is within the peak, the price is calculated using a sine wave function. If it’s outside the peak, the price is calculated using a smaller amplitude sine wave. A random adjustment is also added to each price to simulate real-world price fluctuations.

class SimulatedPriceEnvelopeGenerator(IPriceEnvelopeGenerator):
    def generate(self, date: datetime.date) -> t.List[float]:
        """
        Generates a simulated price envelope for the given date.

        Args:
            date (datetime.date): The date for which to generate the price envelope.

        Returns:
            List[float]: A list of price values representing the price envelope.
        """
        # ... code omitted for brevity ...
        for i in range(self.num_intervals):
            # ... code omitted for brevity ...
            prices.append(price)
        return prices

SimulatedPriceNoiseAdder

The SimulatedPriceNoiseAdder class adds simulated noise to the list of prices generated by the SimulatedPriceEnvelopeGenerator. This is done to make the simulated prices more realistic. The noise is added by randomly adjusting each price within a specified noise level. There’s also a chance for a price spike to occur, which multiplies the price by a specified spike multiplier.

class SimulatedPriceNoiseAdder(IPriceNoiseAdder):
    def add(self, prices: t.List[float]) -> t.List[float]:
        """
        Adds simulated noise to a list of prices.

        Args:
            prices (List[float]): The list of prices to add noise to.

        Returns:
            List[float]: The list of prices with simulated noise added.
        """
        # ... code omitted for brevity ...
        for price in prices:
            # ... code omitted for brevity ...
            noisy_prices.append(new_price)
        return noisy_prices

SimulatedPriceModel

The SimulatedPriceModel class brings it all together. It uses an instance of SimulatedPriceEnvelopeGenerator to generate a list of prices for a given date, and then adds noise to these prices using an instance of SimulatedPriceNoiseAdder. The result is a tuple containing two lists of prices: one with the original prices and one with the noisy prices.

class SimulatedPriceModel(IPriceData):
    def get_prices(self, date: datetime.date) -> t.Tuple[t.List[float], t.List[float]]:
        """
        Generates simulated prices for the given date.

        Args:
            date (datetime.date): The date for which prices need to be generated.

        Returns:
            Tuple[List[float], List[float]]: A tuple containing two lists of prices.
            The first list represents the prices without noise and spikes,
            and the second list represents the prices with noise and spikes.

        """
        # ... code omitted for brevity ...
        prices_with_noise_and_spikes = self.noise_adder.add(prices)
        return prices, prices_with_noise_and_spikes

Below is the graph depicting the outcomes from the SimulatedPriceModel, showcasing the simulated prices:

To explore the more on SimulatedPriceModel , check out the Jupyter notebook here, where its functionality is demonstrated.

Average Price

The average price model calculates the average price based on historical data. It’s a simple yet effective model for predicting future prices when the market conditions are stable.

The average_price.py file contains the HistoricalAveragePriceModel class, which calculates historical average prices. This class is a key component of the project as it provides the historical context needed to understand current price data.

HistoricalAveragePriceModel

The HistoricalAveragePriceModel class is initialized with a data provider and an optional boolean indicating whether to interpolate missing values in the data. The data provider is used to retrieve price data, and the interpolation option determines whether missing values in the data are filled in using linear interpolation.

class HistoricalAveragePriceModel(IPriceData):
    def __init__(self, data_provider: IDataProvider, interpolate: bool = True, prior_days: int = DAYS_IN_WEEK):
        # ... code omitted for brevity ...
        if self.interpolate:
            self.data[self.PRICE_COLUMN].interpolate(method="linear", inplace=True)

get_prices

The get_prices method is the main method of the class. It takes a date as input and returns a tuple containing the average prices for the last week and the prices for the current date. The method uses helper methods to get the current date, the date a week prior, the data for the last week, the average prices for the last week, the data for the current date, and the prices for the current date.

def get_prices(self, date: datetime.date) -> t.Tuple[t.List[float], t.List[float]]:
    """
    Get the average prices for the last week and the prices for the current date.

    Args:
        date (datetime.date): The current date.

    Returns:
        Tuple[List[float], List[float]]: A tuple containing the average prices for the last week
        and the prices for the current date.
    """
    # ... code omitted for brevity ...
    return average_prices_last_week, prices_current_date

get_average_prices_last_week

The get_average_prices_last_week method calculates the average prices for the last week. It takes a DataFrame containing the price data for the last week as input and returns a list of average prices for each hour of the day. The method groups the data by hour and calculates the mean price for each group.

def get_average_prices_last_week(
    self, last_week_data: pd.DataFrame
) -> t.List[float]:
    """
    Calculate the average prices for the last week.

    Args:
        last_week_data (pd.DataFrame): The price data for the last week.

    Returns:
        List[float]: A list of average prices for each hour of the day.
    """
    # ... code omitted for brevity ...
    return (
        last_week_data.groupby(last_week_data.index.hour)[self.PRICE_COLUMN]
        .mean()
        .tolist()
    )

Below is the graph depicting the mean and standard deviation observed from the HistoricalAveragePriceModel, showcasing the simulated prices:

To explore the more on HistoricalAveragePriceModel , check out the Jupyter notebook here, where its functionality is demonstrated.

Forecasted Price

The forecasted price model uses advanced statistical methods or machine learning algorithms to predict future prices. It’s the most complex model and can adapt to changing market conditions.

The forecasted_price.py file contains the ForecastPriceModel class, which implements the IPriceData and IForecaster interfaces. This class provides methods for training, forecasting, and evaluating prices.

ForecastPriceModel

The ForecastPriceModel class is initialized with a data provider, a feature engineer, a machine learning model, a history length, a forecast length, and an optional boolean indicating whether to interpolate missing values in the data. The data provider is used to retrieve price data, the feature engineer is used to perform feature engineering on the data, and the machine learning model is used for forecasting.

class ForecastPriceModel(IPriceData, IForecaster):
    def __init__(
        self,
        data_provider: IDataProvider,
        feature_engineer: IFeatureEngineer,
        model: IModel,
        history_length=7 * 24,
        forecast_length=24,
        interpolate: bool = True,
        prior_days: int = DAYS_IN_WEEK,
    ):
        # ... code omitted for brevity ...

get_prices

The get_prices method is the main method of the class. It takes a date as input and returns a tuple containing the forecasted prices and the actual prices for the given date. The method uses helper methods to get the current date, the date a week prior, the data for the last week, the forecasted prices for the current date, the data for the current date, and the prices for the current date.

def get_prices(self, date: datetime.date) -> t.Tuple[t.List[float], t.List[float]]:
    """
    Get the forecasted prices and actual prices for a given date.

    Args:
        date (datetime.date): The date for which to get the prices.

    Returns:
        Tuple[List[float], List[float]]: A tuple containing the forecasted prices and actual prices.
    """
    # ... code omitted for brevity ...
    return forecasted_prices, prices_current_date

train, forecast, evaluate, save_model, load_model

The train, forecast, evaluate, save_model, and load_model methods are used to train the machine learning model, forecast prices, evaluate the forecasted prices, save the trained model to a file, and load a trained model from a file, respectively.

def train(self, df):
    """
    Train the forecast price model.

    Args:
        df (pandas.DataFrame): The training data.
    """
    # ... code omitted for brevity ...

def forecast(self, df):
    """
    Forecast prices using the trained model.

    Args:
        df (pandas.DataFrame): The data to forecast.

    Returns:
        pandas.DataFrame: The forecasted prices.
    """
    # ... code omitted for brevity ...

def evaluate(self, y_true, y_pred):
    """
    Evaluate the forecasted prices.

    Args:
        y_true (numpy.ndarray): The true prices.
        y_pred (numpy.ndarray): The forecasted prices.

    Returns:
        float: The evaluation metric.
    """
    # ... code omitted for brevity ...

def save_model(self, file_path):
    """
    Save the trained model to a file.

    Args:
        file_path (str): The path to the file.
    """
    # ... code omitted for brevity ...

@staticmethod
def load_model(file_path) -> IModel:
    """
    Load a trained model from a file.

    Args:
        file_path (str): The path to the file.

    Returns:
        IModel: The loaded model.
    """
    # ... code omitted for brevity ...

Below is the graph depicting the outcomes from the ForecastPriceModel, showcasing the simulated prices:

To explore the more on ForecastPriceModel , check out the Jupyter notebook here, where its functionality is demonstrated.

To use any of the pricing models, you need to create an instance of the model and call the appropriate methods as defined in the interface. The specific implementation details depend on the programming language and the design of your software.

Please note that this is a high-level overview. For detailed information, refer to the specific documentation for each pricing model and the source code.

Remember, the best way to learn is by doing. So, I encourage you to clone the repository, play around with the code, and see what you can create. Happy coding!