Private Web Research Agent with Self-Hosted SearXNG

What This Builds

This recipe builds a research agent whose search tool is your own infrastructure. Instead of reaching for SerpAPI, Bing, or another metered, external search API, the agent queries a self-hosted SearXNG metasearch engine running in Docker. All search traffic stays local, there’s no per-query bill, and you choose which upstream engines are queried.

The agent searches via SearXNG, scrapes the best results into clean text, and reasons over them — the same loop as a commercial research agent, but private and free to run. This is the right shape when queries are sensitive (legal, internal, personal) or when you’re running local LLMs and don’t want your retrieval layer leaking tokens to a third party.

The Stack

• SearXNG — an open-source, privacy-respecting metasearch engine; run it in Docker, enable the JSON output format, and call its /search endpoint. It aggregates results from many engines without storing user data.
• LangChain — SearxSearchWrapper / the SearxNG tool plugs the local engine straight into an agent as a search tool; the agent loop decides when to search and what to query.
• Firecrawl (or Crawl4AI) — extracts clean markdown from the URLs SearXNG returns, so the model reads readable text. A self-hostable scraper keeps the whole pipeline private.
• Ollama — optional local LLM so the entire research stack — search, scrape, and reasoning — runs on your own hardware with nothing leaving the machine.

Step-by-Step Outline

1. Run SearXNG in Docker and enable the JSON format in settings.yml so it returns machine-readable results.
2. Verify it works: curl 'http://localhost:8080/search?q=test&format=json'.
3. In LangChain, configure SearxSearchWrapper(searx_host="http://localhost:8080") and expose it as an agent search tool.
4. Add a scrape tool backed by Firecrawl (or Crawl4AI) to turn result URLs into clean markdown.
5. Build the agent loop (LangChain/LangGraph): search → scrape → decide if more is needed → synthesize a cited answer.
6. Point the model at Ollama (or any provider) and run queries; tune which SearXNG engines are enabled and add caching/rate limits as needed.

Why This Shape Works

Most research-agent tutorials assume a paid search API, which is fine until queries are sensitive or volume makes per-query billing painful. Swapping in self-hosted SearXNG keeps the agent’s exact same architecture while making search local, free, and configurable — and pairing it with a self-hostable scraper and a local model means a fully private research pipeline where no token leaves your environment.

Source

Gabriel Rodewald, Run SearxNG Locally To Keep Your AI Data Private & Free — Create Custom Agentic Tools (Docker setup + Python integration): https://medium.com/@gabrielrodewald/run-searxng-locally-to-keep-your-ai-data-private-free-create-custom-agentic-tools-e8f4b5592082

LangChain reference — searx_search utility for SearXNG: https://reference.langchain.com/python/langchain-community/utilities/searx_search