What may be scraped, what may not, and which tools do it safely
1. What is allowed
Public facts: prices, availability, schedules, open registries. Facts are not copyrightable — their presentation is (texts, photos).
2. What to avoid
Personal data without a legal basis (GDPR), content behind login, bypassing technical protection, databases protected by sui generis right.
3. robots.txt and ToS
robots.txt is a technical convention, not law — but violating it plus the site's ToS strengthens claims against you. Read ToS before mass collection.
4. Technical etiquette
Rate-limit (1–2 req/s), identifiable User-Agent with contact, response caching, run heavy jobs off-peak. DDoS-like scraping causes real harm.
5. Prefer APIs
Before scraping, look for an official API or dataset (data.gov.ua, partner APIs). Faster, stabler, legally cleaner.
6. Storing results
Keep fetch date, source URL and raw data separate from processed output. Enables re-processing and provenance proof.
Tools