Data processing planning: extract → transform → load — 6 stages
1. Source inventory
List ALL sources: CRM, Excel, site, bank, API. For each — format, refresh rate, owner, volume. Without this list the pipeline is never finished.
2. Extract
API/export/dump — pick a method per source. Rule: pull RAW data unchanged — the staging zone keeps the original for debugging.
3. Transform
Normalize to a shared schema: types, date formats, keys, dedup. Business logic goes here too (categories, computed fields). Each transform is a separate logged step.
4. Validation
Checks before loading: row counts didn't jump unexpectedly, key metrics within bounds, required fields non-empty. A pipeline without validation silently wrecks reports.
5. Load
Full reload or incremental? Start with full — simpler. Incremental when volumes grow. Store last-refresh timestamp in metadata.
6. Monitoring & docs
Pipeline-failure alert + brief docs: what comes from where, what each field means, who to ask. ETL without docs becomes a black box within a quarter.
Tools by level