Data Engineering

Industrial Data Cleaning Pipeline

End-to-end Python pipeline that transformed 170,000+ rows of messy industrial production data into a clean, analysis-ready dataset.

PythonPandasNumPyRapidFuzzSQL ServerSSIS

Hire Me for Similar Work See All Projects

170,241

Rows Processed

3,847

Duplicates Removed

844 → 668

Defect Names

87.6% → 97.6%

Data Completeness

About This Project

Built for CASCO Automotive Tunisia (Amphenol Group), this project involved cleaning 5 years of production quality data spread across 159 Excel batch files from 3 ZIP archives. The pipeline handles missing values, duplicate removal, fuzzy string matching for defect name normalization, machine ID standardization, and date format unification. Result: a fully clean, structured dataset loaded into SQL Server and ready for Power BI and ML modeling.

Tools & Technologies

Python + Pandas

Core cleaning logic — null handling, deduplication, type casting

RapidFuzz

Fuzzy string matching to normalize 844 defect name variants

SQL Server

Final storage with M-XXX machine ID schema and unified taxonomy

SSIS ETL

Automated data flow from raw Excel sources to SQL staging tables