GPT Crawler

Use it when

• I need to turn documentation pages from one site into a local JSON knowledge file.
• I need a configured site crawl with a URL-match rule and content selector for downstream knowledge upload.

What it solves

Not the fit when

• Crawling login-protected content.
• Avoiding configuration of a site-specific CSS selector.
• Extracting arbitrary fields as a schema-guided JSON API.
• selector-rot
• auth-wall-session
• bot-detection-challenge

Install

git clone https://github.com/BuilderIO/gpt-crawler && cd gpt-crawler && npm i

Invoke

Set url, match, selector, and outputFileName in config.ts, then run npm start.

Alternatives

No reviewed alternatives recorded yet.