There is a version of this project that takes two days. Train Prophet on the Kaggle Store Sales dataset, compute MAPE, upload the notebook, call it done.
I built a different version.
The two-day version answers one question: can a model learn to forecast retail sales? The version I built answers six business questions, exposes a REST API with what-if simulation, runs drift detection against a Prometheus/Grafana stack, orchestrates automated retraining via Prefect, and deploys as a six-service Docker Compose stack. It achieves 8.4% MAPE with 95.6% prediction interval coverage on a 90-day hold-out.
The difference between those two projects is the difference between an ML experiment and an ML system. That gap is where most of the real engineering work lives.
The Problem Worth Solving
The Kaggle Store Sales dataset is 3 million rows of real Ecuadorian retail transaction data of 54 stores, 33 product families, spanning years of promotions, national holidays, and oil price fluctuations (Ecuador's economy is tightly coupled to oil prices, which creates fascinating demand signals in grocery data).
Most teams using this dataset ask: what will sales be next week? That is a useful question. But it is not the question a business intelligence platform needs to answer.
The questions that drive real retail decisions are different:
- What will Store 14's sales be for the next four weeks, and what's the confidence interval?
- Which product categories will decline next quarter?
- What's the revenue impact of running a 10% discount in Region A?
- Which stores need restocking before Friday?
- What's the risk range around our Q4 revenue forecast?
These six questions shaped every architectural decision in the system. The dashboard, API endpoints, feature engineering, and even the choice of ensemble architecture were all derived from the requirement to answer them reliably.
The Ensemble Architecture Decision
The central technical question was: why an ensemble of Prophet and XGBoost, rather than either alone?
Prophet is Facebook's time-series library built on additive decomposition. It excels at modeling trend, seasonality, and holiday effects — exactly the signals present in retail data. But it treats the forecasting problem as purely temporal. It has no way to incorporate the rich cross-series behavioral features that characterize a store's demand pattern.
XGBoost approaches the problem differently. Given a tabular feature matrix — lag values, rolling statistics, cyclical calendar encodings — it learns complex non-linear relationships between those features and the target. It has no inherent understanding of time structure, but it is extraordinarily good at exploiting feature interactions.
Neither model alone solves the problem well. Prophet's standalone MAPE on this dataset was 23.2% reasonable for a baseline with no feature engineering. XGBoost's standalone performance was 11.3% with a proper feature matrix. The ensemble a weighted combination with a Ridge meta-learner reached 8.4% MAPE with 95.6% prediction interval coverage.
The coverage figure matters as much as MAPE. A 90% prediction interval that actually contains the true value 95.6% of the time is a well-calibrated system. It means the uncertainty estimates are honest which is what makes the risk-range visualizations on the dashboard analytically meaningful rather than decorative.
Feature Engineering: 32 Features Across 4 Groups
The feature engineering layer was built to be comprehensive enough to capture the full signal structure of retail time series while rigorous enough to prevent cross-series data leakage.
All 32 features were generated per (store, family) group a critical design decision. Generating lag features across the entire dataset without group stratification would allow information from Store 1's sales to influence Store 2's features, creating a subtle but severe form of data leakage that inflates validation metrics and produces unreliable production forecasts.
The four feature groups serve distinct purposes. The 16 time features capture cyclical and calendar structure: day of week, month, quarter, year, week-of-year boundaries, and crucially the sin/cos cyclical encodings that preserve the circular continuity of time-based signals. The 5 lag features capture autocorrelation at operationally meaningful intervals: 1-day, 7-day, 14-day, 28-day, and 364-day lags. The 8 rolling statistics capture local trend and volatility. The 3 derived features momentum ratio, trend delta, EWMA capture higher-order dynamics that neither lags nor rolling statistics represent well.
The feature pipeline is implemented as a scikit-learn Pipeline wrapper, making it fully serializable, testable, and reusable across training and inference without code duplication.
The What-If Simulation Engine
This component is what separates a forecasting tool from a decision support platform.
The API's /simulate endpoint accepts a scenario definition promotion flag, discount percentage, holiday flag, oil price delta and returns the baseline forecast, the scenario forecast, the absolute uplift, the percentage uplift, and a plain-language recommendation.
{
"baseline_forecast": 31583.35,
"scenario_forecast": 34200.1,
"uplift_pct": 8.28,
"uplift_abs": 2616.75,
"recommendation": "Moderate uplift scenario is likely worthwhile"
}
The simulation engine works by modifying the feature values corresponding to the scenario parameters, running the ensemble against the modified feature matrix, and computing the difference against the unmodified baseline. The approach is mechanistically sound because the model's features directly encode the variables being simulated promotion flags, holiday indicators, and oil price signals are all explicit input features, not latent learned representations.
This matters for trust. When a retail operations team uses the simulation to evaluate a promotion scenario, they need to understand why the model predicts an 8.28% uplift, not just that it predicts it. The feature architecture makes the causal chain from scenario assumption to forecast output transparent and auditable.
Drift Monitoring: The Part Everyone Skips
Most ML portfolios end at deployment. This one doesn't.
The monitoring layer uses Population Stability Index (PSI) and Jensen-Shannon divergence to detect distributional shifts between a 2013 baseline and 2017 production data. The results are revealing.
Oil price PSI was 7.85 catastrophically high, correctly identifying the Ecuador oil crash of 2014–2016 as a major distributional event. Promotion frequency PSI was 0.45 the retailer's promotional behavior changed substantially over the four-year period. Rolling mean sales PSI was 0.21 underlying demand patterns shifted. Store transaction volume, by contrast, showed near-zero drift (0.01) store traffic stayed stable while everything around it changed.
These drift signals are exactly what a production MLOps team needs to make retraining decisions. Not "the model's MAPE degraded" that's a lagging indicator, discovered after the damage is done. PSI-based drift monitoring is a leading indicator: it tells you that the input distribution is shifting before forecast quality has visibly declined.
The PSI metrics feed into Prometheus, visualized in Grafana dashboards, with configurable threshold alerts. This closes the monitoring loop that most ML deployments leave open.
The Six-Service Production Stack
The Docker Compose stack reflects what a production ML platform actually requires:
- FastAPI - the inference API, with Pydantic v2 schema validation, lifespan-managed model loading, and structured prediction responses
- Streamlit - the four-tab BI dashboard answering the six business questions
- MLflow - experiment tracking and model registry, version-controlling every training run's hyperparameters, metrics, and artifacts
- Prefect - orchestrating the three automated pipelines: scheduled data ingestion, automated retraining, and batch prediction
- Prometheus - collecting drift metrics, API latency, and prediction counters
- Grafana - visualizing the monitoring stack with configurable alert thresholds
Each service communicates over Docker's internal network. Each has independent health checks. The entire stack starts with docker compose up --build and is CI/CD-gated through GitHub Actions (lint → test → build) with 4/4 tests passing.
What This System Demonstrates
Every architectural decision in this project was made against a real constraint. The ensemble was chosen because neither model alone satisfied the coverage requirement. The feature groups were separated to prevent leakage. The simulation engine was built because point forecasts without scenario analysis cannot support operational decisions. The drift monitoring was included because production models degrade in ways that accuracy metrics on historical validation sets cannot detect.
The result is a system where you can ask: what will happen to our Pichincha region revenue if we run a 10% promotion across all 36 stores next month, and what's the 90% confidence interval around that projection? and get a defensible, auditable answer.
That is the distance between an RMSE score and a production BI platform. It is a long way. But it is the distance that matters in practice.
Full source code, architecture documentation, and deployment instructions are available on GitHub.