Automated Data Ingestion & Sanitization Engine
A modular Python application designed to ingest chaotic raw datasets (CSV, JSON, XML), validate schemas against strict data types, clean anomalies and output sanitized records.
- Pluggable parser architecture using Python abstract base classes
- Deterministic type coercion with schema validation reporting
- Configurable export pipeline supporting SQL bulk insert and normalized JSON
View code preview
import csv
from dataclasses import dataclass
from typing import List
@dataclass
class RecordItem:
id: int
label: str
value: float
status: str
def sanitize_batch(raw_filepath: str) -> List[RecordItem]:
clean_items = []
with open(raw_filepath, 'r', encoding='utf-8') as f:
reader = csv.DictReader(f)
for row in reader:
try:
clean_items.append(RecordItem(
id=int(row['id']),
label=row['label'].strip(),
value=float(row['value']),
status=row.get('status', 'PENDING').upper()
))
except (ValueError, KeyError):
continue
return clean_items