Scheduled Automation

Scheduled Hacker News Scrape, Summarize, and Email Digest

A scheduled scrape-to-store-to-deliver pipeline: Trigger.dev runs a weekday cron that scrapes the top Hacker News stories through a Browserbase proxy, summarizes each with an LLM, and emails a formatted digest.

What This Builds

This recipe builds a recurring extraction-and-delivery pipeline. Every weekday at 09:00, a Trigger.dev scheduled task scrapes the top three stories on Hacker News, fetches and summarizes each article with an LLM, and emails a clean digest. It is a complete scrape → structure → deliver loop that runs unattended in the cloud.

The interesting part is the durable, fan-out shape: one parent task discovers the article list, then triggers a child task per article. Each child runs with its own retries, so a single slow or paywalled page does not fail the whole run.

The Stack

  • Trigger.dev is the durable background-job runtime. It provides the cron schedule, the parent/child task model, automatic retries with backoff, and the deployment target.
  • Browserbase is a hosted headless-browser service. The task connects Puppeteer to Browserbase over a WebSocket endpoint so scraping runs through a managed, proxied browser rather than directly from the worker (which the Trigger.dev terms require for third-party sites).
  • OpenAI turns each article’s extracted text into a 2-3 sentence summary.
  • Resend plus React Email renders and delivers the final HTML digest.

Step-by-Step Outline

  1. Create a Trigger.dev project, plus accounts for Browserbase, OpenAI, and Resend; put the API keys in .env and mirror them into the Trigger.dev project environment variables.
  2. Define the parent scheduled task summarizeHackerNews with cron: { pattern: "0 9 * * 1-5" }. On each run it connects Puppeteer to Browserbase, loads news.ycombinator.com, and extracts the title and link of the top three .athing rows.
  3. Fan out with scrapeAndSummarizeArticle.batchTriggerAndWait(...), passing each article’s link as an idempotencyKey so re-runs do not duplicate work.
  4. In the child task, connect Puppeteer again, block images/styles/fonts via request interception, navigate to the article, and extract the main article text (capped, e.g. to ~1500 chars). Configure retry.maxAttempts: 3 so flaky pages retry.
  5. Summarize the extracted text with an OpenAI chat completion and return { title, link, summary }.
  6. Back in the parent, collect the successful child outputs, render them through a React Email template, and send the digest with Resend.
  7. Add the Puppeteer build extension to trigger.config.ts, run npx trigger.dev@latest dev to test locally, then deploy to run on the weekday schedule in production.

Why This Shape Works

Separating discovery (parent) from extraction (child) gives per-item retries and idempotency for free, which is exactly what unreliable web sources demand. Routing the browser through Browserbase keeps scraping compliant and resilient to bot defenses, while Trigger.dev’s cron and retry primitives remove the need to run and babysit your own scheduler.

Source

Adapted from the official Trigger.dev tutorial “How to scrape a website using Browserbase, Puppeteer, OpenAI and Trigger.dev” by James Ritchie (2024-10-23): https://trigger.dev/blog/scrape-hacker-news