The data engine behind a UK cinema listings site

yourlocalcinema.com ↗ (opens in a new tab) lists showtimes for cinemas across the UK, and those times live on each cinema's own website. I built and maintain the system that collects them: 176 scrapers, and a control room where the owner runs them all from one screen, without a terminal and without a developer. The work started on Fiverr in 2023 and has continued directly since. Each scraper writes two files, every screening it finds and the subtitled screenings only, and one click merges them into the files the listings use. It is a background system feeding the site (more about me). This page is about how it's built: the decisions behind it, why they were made and what each one prevents.

How it's built

Five decisions carry most of the weight. Each one is written the same way: what was decided, why, and what it prevents.

One template, 176 scrapers

Decision: every scraper is one Python file holding one class, and every one declares the same few things: the cinema, its website and its two output files. Each has one main function that returns how many rows it found. The app finds the scrapers by scanning their folder, so adding a cinema means adding a file, with nothing to register. One scraper can also cover a whole chain: its list of venues comes from a shared spreadsheet, matched on the website address. Why: with one structure, one runner can launch any of the 176 without knowing anything about that cinema's website. Every scraper also writes the same two files: every screening it finds, and the subtitled screenings only, which the dashboard calls the Accessible CSV.

What it prevents: one-off code per cinema that only its author can run, and a list of scrapers to keep in step by hand.

The control room

• Dashboard, in the browser: the owner presses a run button.
• ylc_web, a small web server on the owner's own computer, starts the scrapers.
• Each scraper runs as its own process and writes two CSV files: all screenings, and subtitled only.
• Merge CSVs combines them, skipping stale files, into the two files the listings use.

Decision: the owner runs everything from a dashboard in the browser, served by ylc_web, a small web server that runs on the owner's own computer and is reachable only from it. The dashboard updates live as scrapers start and finish, with no page reload. The owner can run every cinema, one group (normal, proxy or browser), any ticked selection, or a single cinema from its row, and can stop one cinema, everything at once, or press Stop & Quit, which also shuts the app down. Why: a status has to come from the result, not a guess. A crash is detected from the scraper's exit code, and its full output opens in an error panel for that cinema. A clean run takes its counts from the scraper's last line, or from the CSV rows if that line is missing, and zero rows means “no screenings” or “no subtitled”. A name filter, a status filter and sortable columns find any cinema among the 176.

What it prevents: the owner needing a terminal, or a developer, to see what ran and what failed.

Running scrapers without breaking the app

Decision: each scraper runs as its own separate process, not as a thread inside the app. Why: a thread can't be force-stopped and a process can, and one scraper that crashes or hangs must not take the dashboard down with it. Stop kills the scraper and everything it started, including a browser. The first desktop app ran scrapers as threads; it served for two and a half years, until ylc_web replaced it in June 2026. The same thinking shaped how output is captured. Each scraper's output is written to a temporary file, not read through a pipe. The browser scrapers print so much that they filled the pipe's fixed buffer and froze: a run sat at “running”, and no browser ever appeared. Files have no such limit. Ordinary scrapers run a fixed number at a time. Browser scrapers are heavy, so only one runs at a time, in its own lane alongside the others, and Run all, Run selected and the group buttons share the same lanes.

What it prevents: a stuck scraper nobody can stop, a crash that takes the dashboard with it, and a run that hangs without a word. If you are hiring, this is the section I'd point you to.

Getting past blocks

Decision: not every cinema website is happy to be read by a script, so the scrapers fall into three groups, and the dashboard can run each on its own: normal, proxy and browser. Some cinema sites refuse repeated requests from one address, so those scrapers get a proxy. The app hands it in at launch, and the scraper itself doesn't change. A few sites only show their times to a real browser, so those scrapers drive a headless browser. Network calls retry with growing pauses. Why: handing the proxy in from outside keeps the shared structure intact. A scraper that needs a proxy follows the same template as one that doesn't, and the runner launches both the same way.

What it prevents: a brief network hiccup reported as a failure, and proxy settings copied into every scraper that needs one.

Keeping it alive

Decision: maintenance is a routine, not a rescue. Cinema websites change, and when one does, its scraper breaks; that is normal, and the system is built around it. Fixes ship as monthly batches, and the loop is short: the owner sends the error log, gets the fixed scrapers back, and the next launch pulls them in. On the owner's Mac, double-clicking one start file pulls the latest code, reinstalls dependencies only when they've changed, and opens the dashboard once the server is ready. If the update fails, because the computer is offline or git isn't installed, it says why and starts the version already there. Why: a failed update must never stop the app. The last step guards the listings themselves. Merge CSVs combines every cinema's files into the two combined files the listings use, all screenings and subtitled only, and it skips any cinema file older than two days.

What it prevents: a broken scraper turning into an emergency, an update that leaves the owner with nothing to run, and stale times slipping into the listings.

What it looks like in use

These screenshots come from a development run of the same dashboard the owner uses.

Run any set of cinemas

Run all starts every cinema, the group buttons start the normal, proxy or browser scrapers, and ticking rows builds any selection. Run selected starts just those, in the same lanes as everything else, and each running row has its own Stop button.

What went wrong, in full

When a scraper crashes, the app knows from its exit code. The full captured output opens in an error panel for that cinema, so the owner sees the scraper's own last words, not a summary of them.

Three kinds of problem

The Errors tab sorts every problem into three groups: the scraper crashed, it found no screenings, or it found no subtitled screenings. One shared function does the sorting, so this tab, the dashboard and the downloadable error file always agree.

Every run, remembered

The Logs tab keeps the run history across sessions: one row per cinema run, with its start and finish times, its duration and the rows it found. Like the dashboard, it has a name filter, a status filter and sortable columns.

One click to merge

Merge CSVs combines every cinema's files into the two combined files the listings use: all screenings, and subtitled only. Any cinema file older than two days is left out, so stale times never slip in. A confirmation shows how many rows each file holds.

What this project shows

• A long-running client relationship: since 2023, from the first desktop app to the control room in use today.
• A convention that scales: one structure that all 176 scrapers follow, so one runner launches every one of them.
• Engineering for a concrete failure: one process per scraper because a thread can't be stopped, and output in files because a full pipe froze runs.
• A tool a non-developer runs alone: the owner runs, stops, reads errors and merges without a terminal.
• Maintenance as a routine, not a rescue: monthly fix batches, picked up by the next launch, and an update step that can't stop the app from starting.

Want something like this?