Area: Data Intelligence

Problem: ETL pipelines are usually written per pipeline. That makes them slow to add and hard to audit. The client needed pipelines defined by configuration, with security and validation that could not be skipped.

Approach: We built DataFlow (published on PyPI as etl-framework v1.0.1), a JSON-driven framework. A pipeline is a configuration file. Extraction, validation, transformation, and loading are handled by the framework, and every stage validates its input.

Technical details:

  • Pipelines defined in JSON, with no code change required to add one
  • PDF-first extraction with several output formats
  • RBAC, encryption at rest, and audit logging
  • pandas transformation engine with pluggable transformers
  • PostgreSQL and MySQL targets
  • Published to PyPI with a CI/CD pipeline
  • 125+ source files in a modular layout

Outcome: Adding a pipeline means writing a config file. Every record is traceable through the audit log.

Stack: Python, pandas, PostgreSQL, PyPI, CI/CD

Page rendered by SimpleCMS — powered by the Plugin System.

Sandboxed plugin ran for projects/dataflow-etl and read “DataFlow ETL Framework” through the mediated API.