The pipeline behind a recommendation model

The problem

The marketplace's product recommendations ran on a model the data science team retrained on shopper behavior. Training it meant turning raw clickstream into a feature set at a scale the existing SQL Server Integration Services jobs could not reach, on a cadence that let the team iterate, with predictions landing somewhere analysts could read them the same hour.

The pipeline

Training data on EMR

Airflow DAGs ran PySpark and SparkSQL on AWS EMR to build the training set from a billion clickstream rows over a rolling ten-month lookback. The window moved with each run, so the model always trained on the most recent ten months of behavior and never on a frozen snapshot.

Post-processing through SageMaker

Model output came back through SageMaker for post-processing in the same DAG family, so training and scoring shared one orchestration path and one set of alerts.

Predictions to Redshift, hourly

A Python DAG wrote the model's prediction outputs to Redshift tables every hour, which gave analysts current data for trend analysis without asking the data science team for an export.

Data-quality DAGs in front of the model

Separate DAGs scanned for anomalies in the inputs, logged what they found, and alerted, so a bad day of clickstream was caught before it trained the model that chose what shoppers saw. The resume line says it plainly: this improved the recommendations.

Around it

• Two critical SSIS workbooks migrated into Python pipelines in Airflow, and about a dozen more DAGs built alongside them.
• Airflow subject-matter expert for the team: runbooks, data definitions, and training that raised adoption and shortened onboarding. Mentored two junior engineers.
• Advised the incoming manager on modernizing the platform, including AWS integration strategy and adopting dbt, which set the roadmap after the turnover.

Stack

Apache Airflow, PySpark and SparkSQL on AWS EMR, SageMaker, Redshift, Python, and the SSIS estate being retired.