ScrapeGraphAI is an open-source Python library that reads a web page with an AI model and returns the exact fields you ask for. Paired with a local model and a daily schedule, it can add new companies to a lead sheet every morning. This guide gives you the full setup, built to collect public business contact details only.
What you need
ScrapeGraphAI: github.com/ScrapeGraphAI/Scrapegraph-ai, MIT-licensed, about 29,500 stars. Free. The company also sells a paid hosted API, which you don't need here.
Python 3.10 or newer.
Ollama (free) to run a free local model, so you don't pay per page. Hosted models like OpenAI, Gemini or Groq also work with an API key.
A coding agent (optional), such as Claude Code or OpenAI's Codex, to set it up and run it for you.
A list of pages you're allowed to crawl: trade association member lists, local business directories whose terms permit it, and companies' own contact pages.
Step by step
Install.
python3 -m venv .venv
source .venv/bin/activate
pip install scrapegraphai
playwright install
ollama pull llama3.2List your source pages. Create
seeds.txt, one URL per line. Read each site's terms first and leave out any site that forbids automated collection.Save this script as
harvest.py. It checks robots.txt before every page, asks only for company-level contact details, drops personal email addresses, skips companies already in the sheet and waits between pages.
import csv, os, time, datetime, urllib.robotparser
from urllib.parse import urlparse
from scrapegraphai.graphs import SmartScraperGraph
BOT_NAME = "MyLeadResearchBot"
CONFIG = {"llm": {"model": "ollama/llama3.2", "model_tokens": 8192, "format": "json"},
"headless": True}
PROMPT = ("List the businesses on this page. For each return: company_name, website, "
"business_email (role addresses only, like info@, sales@, hello@, contact@), "
"business_phone, city, category. Leave a field empty if it is not on the page. "
"Do not return personal names, personal emails or personal phone numbers. "
'Return JSON as {"companies": [...]}.')
FIELDS = ["company_name", "website", "business_email", "business_phone",
"city", "category", "source_url", "date_found"]
ROLE = ("info@", "sales@", "hello@", "contact@", "enquiries@", "support@", "office@")
def allowed(url):
p = urlparse(url)
rp = urllib.robotparser.RobotFileParser(f"{p.scheme}://{p.netloc}/robots.txt")
try:
rp.read()
except Exception:
return False
return rp.can_fetch(BOT_NAME, url) and rp.can_fetch("*", url)
def domain(url):
url = url if "://" in url else "https://" + url
return urlparse(url).netloc.lower().removeprefix("www.")
def companies(result):
if isinstance(result, dict):
if isinstance(result.get("companies"), list):
return result["companies"]
inner = result.get("content")
if isinstance(inner, dict) and isinstance(inner.get("companies"), list):
return inner["companies"]
return []
def main():
is_new = not os.path.exists("leads.csv")
seen = set()
if not is_new:
with open("leads.csv", newline="", encoding="utf-8") as f:
seen = {domain(r["website"]) for r in csv.DictReader(f) if r.get("website")}
with open("seeds.txt", encoding="utf-8") as f:
seeds = [l.strip() for l in f if l.strip() and not l.startswith("#")]
today = datetime.date.today().isoformat()
with open("leads.csv", "a", newline="", encoding="utf-8") as out:
w = csv.DictWriter(out, fieldnames=FIELDS)
if is_new:
w.writeheader()
for url in seeds:
if not allowed(url):
print("Skipped by robots.txt:", url)
continue
try:
result = SmartScraperGraph(prompt=PROMPT, source=url, config=CONFIG).run()
except Exception as e:
print("Failed:", url, e)
continue
for c in companies(result):
if not isinstance(c, dict) or not c.get("website"):
continue
d = domain(c["website"])
if d in seen:
continue
email = (c.get("business_email") or "").strip().lower()
if not email.startswith(ROLE):
email = ""
w.writerow({"company_name": c.get("company_name", ""), "website": c["website"],
"business_email": email, "business_phone": c.get("business_phone", ""),
"city": c.get("city", ""), "category": c.get("category", ""),
"source_url": url, "date_found": today})
seen.add(d)
time.sleep(10)
if __name__ == "__main__":
main()Or let your agent set it up. In Claude Code or Codex, paste:
In this folder, set up harvest.py (pasted below) with a Python virtual environment, ScrapeGraphAI and Ollama's llama3.2 model. Before running, open each URL in seeds.txt, read the site's terms and robots.txt, and remove any site that forbids automated collection; list what you removed and why. Then run the script once and show me the first 10 rows of leads.csv. Do not collect personal names or personal contact details, and do not send any emails or messages.Run it every morning. On Mac or Linux, add a cron job with
crontab -e:
30 7 * * * cd /path/to/leads && /path/to/leads/.venv/bin/python harvest.py >> harvest.log 2>&1Codex automations can also run a task on a schedule, but local tasks need your computer on and the ChatGPT desktop app running.
Review before you reach out. Check each new row by hand. Write to a few companies with a personal, relevant message, not a bulk blast.
Good to know
Robots.txt and terms come first. robots.txt is a standard way for sites to tell crawlers what they may access (RFC 9309). A site's terms can still forbid scraping even where robots.txt allows it.
India (DPDP Act): Section 3(c)(ii) excludes personal data the person made public themselves, or that someone had a legal duty to publish. "Findable online" isn't the test, and the exclusion doesn't override site terms or other laws. The DPDP Rules were notified on 13 November 2025, with the main duties starting 13 May 2027.
UK (UK GDPR and PECR): you can email limited companies without prior consent, but you must say who you are and give an opt-out. Sole traders and some partnerships need consent or the soft opt-in. A named work email like jane.smith@company.com is personal data, so UK GDPR applies: you need a lawful basis and must tell the person where you got their details within one month. Screen calls against the TPS and CTPS.
No bulk unsolicited messaging. Keep a suppression list and honour every opt-out.
Accuracy: a small local model can miss or garble fields. Check every row.
ScrapeGraphAI's README says it's meant for data exploration and research, and the maintainers disclaim responsibility for misuse.
Sources
Found this useful? Follow @kevinsingh.me on Instagram for a new AI tool every day, and subscribe so you don't miss the next one.