Senior Crawler Developer · technical assessment
A crawler whose shape scales.
A queue-based web crawler built in Nest.js — you POST a URL, a worker fetches and extracts it asynchronously, and every slow, fragile part of scraping is governed centrally. Built for International Showtimes, designed for 25,000+ sites.
Scored 100% against traumpalast.de — a live German cinema chain, i.e. International Showtimes' own domain. Fetched by the lightweight http engine, no proxy, in 2.19s: redirect followed to the canonical URL, cross-origin CDN assets (static.filmtheaterbetriebe.de) resolved to absolute, German unicode intact, favicon fallback correct. Cheap-first — proven on their home turf.
01 The brief & constraints
What was mandated, and what was left open
The assessment was a two-part specification with the stack named explicitly. That framing decided several things before a line was written — most importantly the language.
| Part | Requirement | Mandated tools |
|---|---|---|
| Part 1 the crawler |
Crawl a given URL and extract title, meta description, favicon, script + stylesheet URLs, image URLs. Queue & process jobs. REST API to start / monitor / cancel. Unit tests. | Nest.js · Axios · Cheerio · BullMQ · Redis · Swagger |
| Part 2 anti-blocking |
Headless-browser fetching. Rotating user-agents. Rotating IPs via proxy/VPN, configured by env. Rate limiting. Unit tests. | Puppeteer · proxy/VPN · env config |
Endpoints (fixed)
POST /crawl · GET /status/:id · DELETE /cancel/:id — named exactly by the spec. Contracts kept intact.
Deadline
Document said 5 working days; recruiter framed it as 3. Treated 3 as the real budget with slack in reserve.
No target site
The crawler takes any URL. So Part 2 is about proving I know the anti-bot mechanisms — not defeating one specific WAF.
My scraping background is primarily Python. But the spec names Nest.js, Axios, Cheerio, BullMQ and Puppeteer by name — the exact tools the team uses. Submitting Python would read as dodging the test. So: build in the required stack, lean on real TypeScript fluency. The scraping concepts transfer; only the syntax was new. Stated plainly, never hidden.
02 Architecture at a glance
Queue-first, not fetch-in-the-request
The single most important decision. Crawling is slow and failure-prone, so it must never run inside the HTTP request. Everything else follows from that shape.
POST /crawl ─▶ CrawlController ─▶ CrawlService.enqueue() ─▶ BullMQ queue (Redis)
│
▼
GET /status/:id ◀─ CrawlService.getStatus() ◀─────────── CrawlProcessor (worker)
GET /crawls ◀─ CrawlService.list() │ 1. pick engine
DELETE /cancel/:id ─▶ CrawlService.cancel() │ 2. rotate UA + proxy
GET /health ◀─ QueueHealthIndicator │ 3. fetch page
│ 4. extract data
▼
PageFetcher (strategy)
├─ AxiosFetcher (http)
└─ PuppeteerFetcher (browser)
│
▼
ExtractionService (Cheerio)Why the queue earns its place
Throughput is governed centrally — rate limit, concurrency, retries with backoff, and job state all live in one place instead of being reinvented per request. That is exactly what matters going from one site to many.
Three properties it buys for free
Durability — a crashed worker doesn't lose the job. Backpressure — the limiter protects targets and the proxy pool. Horizontal scale — add worker processes, zero code change.
03 Component walkthrough
Every part, and the one job it owns
The codebase is organised by responsibility, not by type. Each unit below does one thing and is testable in isolation.
API layer crawl.controller.ts · dto/
The HTTP surface. Thin controllers; all request/response shapes are decorated DTOs, so Swagger and validation both derive from the same source. A global ValidationPipe (whitelist + transform) rejects bad input before it reaches logic.
Orchestration crawl.service.ts
Enqueue, status lookup, history listing, cancellation. The only place that talks to the queue. Maps a BullMQ job to the API's status DTO through one shared helper.
Worker crawl.processor.ts
The BullMQ consumer. Per job: pick the engine, rotate UA + proxy, fetch, extract, return the result. Honours a worker-level rate limit and cooperative cancellation checkpoints.
Fetcher strategy fetcher/
A PageFetcher interface with two implementations — AxiosFetcher (http) and PuppeteerFetcher (browser). The processor chooses per job; nothing downstream knows which ran.
Extraction extraction/extraction.service.ts
Pure and I/O-free: extract(html, baseUrl) in, structured data out. All URLs resolved to absolute against the post-redirect URL, de-duplicated; data: / javascript: dropped. Favicon has a real browser-like fallback chain.
Anti-blocking anti-blocking/
Small round-robin UserAgentRotator and ProxyRotator fed from env. Empty proxy list ⇒ direct fetch. Rate limiting is the BullMQ limiter, not ad-hoc sleeps.
Health health/queue-health.indicator.ts
Terminus probe. Returns 200 only when Redis is reachable AND ≥1 worker is registered — else 503. A queue with no worker silently accepts jobs and never runs them; the probe refuses to call that healthy.
Observability Bull Board · /admin/queues
The standard BullMQ dashboard, read-only, reading the same Redis. Drill into any job: input URL, extracted result, error and retry count.
Cancellation is honest about what's possible
A reviewer will poke this. The semantics are deliberate:
| Job state | DELETE /cancel/:id does | Result |
|---|---|---|
| waiting | Removed from the queue outright. | cancelled 200 |
| active | Sets a cancelRequested flag; worker aborts at its next checkpoint (before / after fetch). | 200 → settles failed "Cancelled by user" |
| completed | Nothing — cannot cancel finished work. | 409 Conflict |
No pretending a running fetch can be force-killed. Modelling the real state machine beats faking instant cancellation — and it's exactly the kind of nuance that matters at scale.
04 Project structure
One responsibility per folder
New concerns get their own directory rather than swelling a service. The layout is the documentation.
8 spec files · 43 offline unit tests. Every new unit ships with a spec; the suite touches no network, Redis or browser.
05 Design decisions
Each choice, and the trade-off it accepts
Every decision here is defensible from first principles — and each one gave something up. Naming the trade-off is how you show it was a decision, not a default.
Queue-first architecture
PageFetcher strategy interface
Extraction is pure / I/O-free
Results stored on the job (no DB)
Config over constants
Docker + system Chromium on Render
06 The dashboard question
Why Bull Board — and not a custom dashboard
A dashboard was never requested. When I considered adding visual crawl history, this is the comparison I ran — and the reasoning matters more than the result.
Bull Board chosen
- ~15 lines; reads the same Redis the worker uses.
- Real job drill-down: input, result, error, retries, state.
- Signals fluency with the BullMQ ecosystem — the tool a team actually reaches for.
- Zero UI to maintain; nothing to get subtly wrong.
- On-domain for a backend role.
Custom dashboard rejected
- Hours of React + streaming/polling + a persistence layer for history.
- New failure surface; a half-polished UI hurts more than none.
- Scores in a lane the rubric doesn't grade — this is a crawler role.
- For a senior hire, gilding the spec can read as over-building.
- Swagger already covers interactive testing.
Doing exactly the spec cleanly, with good judgement about what to leave out, is itself the senior signal. Bull Board captured the real value (queue visibility) at a fraction of the cost, on the team's own terms.
07 Anti-blocking depth
What's implemented — and how I'd escalate
The assessment target has no serious defenses, so the depth lives in the design and the documented escalation path, not in over-engineering the demo.
Implemented
UA rotation — round-robin pool, per-request. Proxy/VPN rotation — round-robin over an env-configured list, works with any provider or a local VPN gateway; credentials masked in output. Rate limiting — BullMQ worker limiter (N jobs / window) protects both target and proxy pool. Browser engine — Puppeteer for JS-only pages, opt-in per job.
How I'd handle the hard cases
Cloudflare / JS challenges — escalate that domain to the browser engine + residential proxies; the strategy interface already allows it. Akamai / fingerprinting — stealth-patched browser, consistent TLS/JA3 + header/UA coherence, human-like pacing. CAPTCHA — a solver behind the same PageFetcher seam, invoked only on detection.
Start every domain on the cheap HTTP engine. When a site starts returning blocks or JS-only shells, auto-escalate that domain to browser + residential proxies. Most of 25k sites never need a browser — you pay for it only where required.
08 Scaling
From one URL to 25,000+ sites
The assessment is a single-URL crawler. International Showtimes tracks 25k+ sites across 120+ markets. The queue-first shape is deliberately the one that gets there — by adding pieces, not rewriting.
Workers scale out
N worker processes against one queue, zero code change. The limiter stays global.
Per-domain politeness
Rate-limit and concurrency per host (a queue/group per domain) so one slow site can't starve others.
Adaptive engine
HTTP by default; auto-escalate to browser + residential proxies only where a domain demands it.
Frontier & dedup
A seen-URL set + scheduling frontier in Redis so 25k sites don't re-crawl or thunder.
Durable store
Postgres for results/history behind a repository interface; Redis stays the live queue + dedup cache. First thing I'd add for production.
Observability
Per-domain success / block / latency rates — Bull Board is already wired; metrics extend it so you see defenses change before data goes stale.
09 Shortcomings & next
What it doesn't do — stated plainly
Honesty over impression. These are known and bounded, documented up front — not discovered after the fact.
No durable persistence
Results & history live in Redis with 24h retention — not an audit log. Postgres behind a repository interface is the documented next step.
Single-page crawl only
No link-following / depth. The queue makes depth-N additive (enqueue discovered URLs) — deliberately left out of scope, not blocked.
No live CAPTCHA / WAF solving
The escalation path is documented and fits behind the fetcher seam; the target didn't need it, so it wasn't built to be built.
No per-domain rate limiting yet
The limiter is global. Per-host politeness is the key change for true multi-site scale.
Free-tier deploy caveats
Render free instances sleep (30–60s cold start) and are memory-tight for the browser engine. Fine for demo; a paid instance for real load.
No end-to-end / load suite
Unit coverage is strong and offline; I'd add an integration test against ephemeral Redis and a small k6 load profile next.