Forecast Accuracy Measurement Methods

Look at the image attached to this post. Which forecast seems more appropriate: 1) the red straight line, 2) or the purple wavy line? Many demand planners might choose (2), thinking it better captures the ups and downs. But, in many cases, the straight line is just fine. Here’s why. In a previous post on Structure vs. Noise (https://lnkd.in/ekZA__aE), we talked about how a time series is made up of different components (such as level, trend, and seasonality), and how the main goal of a point forecast is to capture the structure of the data, not the noise. Noise is unpredictable, and it should be treated by capturing uncertainty around the point forecasts (e.g., prediction intervals). So the answer to the question above comes to understanding what sort of structure we have in the data. In the attached image, the only structure we have is the level (average sales). There's no obvious trend, seasonality, no apparent outliers, and we do not have promotional information or any explanatory variables. The best you can do in that situation is capture the level correctly and produce a straight line for the next 10 observations. In this case, we used our judgment to decide what’s appropriate. That works well when you’re dealing with just a few time series. Petropoulos et al. (2018, https://lnkd.in/eVXQBjh9) showed that humans are quite good at selecting models in such a task as above. But what do you do when you have thousands or even millions of time series? The standard approach today is to apply several models or methods and choose the one that performs best on a holdout sample using an error measure, like RMSE (Root Mean Squared Error, see this: https://lnkd.in/easRx9KX). In our example, the red line produced a forecast with an RMSE of 10.33, while the purple line had an RMSE of 10.62, suggesting that the red line is more accurate. However, relying only on one evaluation can be misleading because just by chance, we can get a better forecast with a model that overfits the data. To address this, we can use a technique called "rolling origin evaluation" (Tashman, 2000: https://lnkd.in/eTQp8djX). The idea is to fit the model to the training data, evaluate its performance on a test set over a specific horizon (e.g., the next 10 days), then add one observation from the test set to the training set and repeat the process. This way, we gather a distribution of RMSEs, leading to a more reliable conclusion about a model’s performance. Nikos Kourentzes has created a neat visualization of this process (second image). For more details with examples in R, you can check out this section of my book: https://lnkd.in/ePJW-6UZ. After doing a rolling origin evaluation, you might find that the straight line is indeed the best option for your data. That’s perfectly fine - sometimes, simplicity is all you need. But then the real question becomes: what will you do with the point forecasts you’ve produced? #forecasting #datascience #machinelearning #businessanalytics

𝐬𝐌𝐀𝐏𝐄 (Symmetric MAPE) – Is it a better alternative to 𝐌𝐀𝐏𝐄? (explained with Examples and Calculations) You’ve likely come across MAPE as a standard for measuring accuracy. While MAPE is widely used, it has a significant weakness—it tends to get distorted when dealing with zero or very low demand values. Enter sMAPE, an alternative that addresses this limitation by including both Actuals & Forecasts in the denominator. This small tweak can make a big difference in forecasting accuracy, especially when you're dealing with volatile demand. Let’s break it down with a couple of examples: 𝐒𝐜𝐞𝐧𝐚𝐫𝐢𝐨 𝟏: Low Actual Demand - Actual Demand: 1 - Forecast: 100 Using MAPE: MAPE = {|Forecast - Actual| / |Actual|}*100 = {|100 - 1|/ 1}*100 = 9900% You get a massively inflated error, which can mislead you into thinking your forecasting model is worse than it actually is. Using sMAPE: sMAPE = [(|Forecast - Actual|) / ((|Actual| + |Forecast|)/2)]*100 = [(|100 - 1|) / ((|1| + |100|)/2)]*100 = 196% In contrast, sMAPE provides a much more realistic error by considering both the actual demand & the forecast in its calculation. 𝐒𝐜𝐞𝐧𝐚𝐫𝐢𝐨 𝟐: Zero Actual Demand - Actual Demand: 0 - Forecast: 200 Using MAPE: MAPE = {|Forecast - Actual| / |Actual|}*100 = {|200 - 0|/ 0}*100 = infinite MAPE results in an undefined (infinite) value when actual demand is zero, which isn’t useful for performance evaluation. Using sMAPE: sMAPE = [(|Forecast - Actual|) / ((|Actual| + |Forecast|)/2)]*100 = [(|200 - 0|) / ((|0| + |200|)/2)]*100 = 200% Here, sMAPE provides a reasonable & finite error value, making it much more practical when dealing with zero-demand situations. 𝐒𝐜𝐞𝐧𝐚𝐫𝐢𝐨 𝟑: Large Forecast Errors (Where MAPE Might Perform Better) - Actual Demand: 500 - Forecast: 50 Using MAPE: MAPE = {|Forecast - Actual| / |Actual|}*100 = {|50 - 500|/ 500}*100 = 90% Here, MAPE gives a clear sense of how far off the forecast is from the actual demand. Using sMAPE: sMAPE = [(|Forecast - Actual|) / ((|Actual| + |Forecast|)/2)]*100 = [(|50 - 500|) / ((|500| + |50|)/2)]*100 = 163% In this case, sMAPE almost doubles the error, which might not always be necessary when you're only interested in how far off your forecast is from actual demand. MAPE provides a simpler, more intuitive error percentage when both the actual & forecast values are significant, making it easier to interpret in certain business scenarios. 𝐖𝐡𝐞𝐧 𝐭𝐨 𝐂𝐡𝐨𝐨𝐬𝐞 𝐖𝐡𝐢𝐜𝐡? - Use sMAPE when dealing with low or zero demand values. It offers more balanced error measurements & avoids inflated percentages. - Use MAPE when you’re forecasting products or services where demand is relatively stable & significant. MAPE gives a clear indication of how far off the forecast is, & the simplicity of its calculation can be more intuitive for business decisions. 𝐓𝐚𝐤𝐞𝐚𝐰𝐚𝐲: Choosing the right metric depends on the context of your business & forecast patterns.

Bad forecast = Bad inventory + Bad losses + Bad cash This infographic shows 7 measures for forecast accuracy & bias for demand planners : 1️⃣ MAPE (Mean Absolute Percentage Error) ↳ Pros: easy to explain; allows to compare SKUs of any size ↳ Cons: explodes when actuals ≈ 0; over‑penalizes low‑volume items 2️⃣ WAPE / WMAPE (Weighted APE) ↳ Pros: volume‑weighted; tiny SKUs don’t distort the big picture ↳ Cons: still collapses when actuals are zero; masks big misses on slow movers 3️⃣ MAE / MAD (Mean Absolute Error/Deviation) ↳ Pros: clear “units‑off” view; less sensitive to outliers ↳ Cons: hard to compare across products with very different scales 4️⃣ RMSE (Root Mean Squared Error) ↳ Pros: heavily penalizes large misses; great for high‑value SKUs ↳ Cons: extremely sensitive to outliers; a single spike skews results 5️⃣ MFE / Bias (Mean Forecast Error) ↳ Pros: shows direction (over‑ vs. under‑forecast); crucial for fixing systematic bias ↳ Cons: positive and negative errors cancel out; hides magnitude 6️⃣ sMAPE (Symmetric MAPE) ↳ Pros: reduces MAPE’s inflation on low volumes; bounded between 0 % and 200 %. ↳ Cons: still undefined when both forecast and actual are zero; less intuitive than plain MAPE 7️⃣ MASE (Mean Absolute Scaled Error) ↳ Pros: scale‑free; compares across SKUs and time series; benchmark: values < 1 beat a naïve forecast ↳ Cons: requires a “naïve” benchmark to compute; harder to communicate to non‑analysts Any others to add?

𝗔𝗿𝗲 𝗬𝗼𝘂𝗿 𝗙𝗼𝗿𝗲𝗰𝗮𝘀𝘁 𝗔𝗱𝗷𝘂𝘀𝘁𝗺𝗲𝗻𝘁𝘀 𝗛𝗲𝗹𝗽𝗶𝗻𝗴 𝗼𝗿 𝗛𝘂𝗿𝘁𝗶𝗻𝗴?  In demand planning, we often tweak forecasts based on market intelligence, gut feel, or stakeholder inputs. But do these adjustments actually improve accuracy? 𝗙𝗼𝗿𝗲𝗰𝗮𝘀𝘁 𝗩𝗮𝗹𝘂𝗲 𝗔𝗱𝗱 (𝗙𝗩𝗔) is a quantitative metric that measures whether manual or system-driven adjustments enhance or degrade forecast accuracy. The goal? 𝗘𝗹𝗶𝗺𝗶𝗻𝗮𝘁𝗲 𝘂𝗻𝗻𝗲𝗰𝗲𝘀𝘀𝗮𝗿𝘆 𝗯𝗶𝗮𝘀 𝗮𝗻𝗱 𝗶𝗺𝗽𝗿𝗼𝘃𝗲 𝗱𝗲𝗺𝗮𝗻𝗱 𝗽𝗹𝗮𝗻𝗻𝗶𝗻𝗴 𝗲𝗳𝗳𝗶𝗰𝗶𝗲𝗻𝗰𝘆. How to Calculate FVA? FVA compares the Mean Absolute Percentage Error (MAPE) before and after forecast adjustments: 𝗙𝗩𝗔= ((𝗠𝗔𝗣𝗘 𝗼𝗳 𝗦𝘁𝗮𝘁𝗶𝘀𝘁𝗶𝗰𝗮𝗹 𝗙𝗼𝗿𝗲𝗰𝗮𝘀𝘁-𝗠𝗔𝗣𝗘 𝗼𝗳 𝗔𝗱𝗷𝘂𝘀𝘁𝗲𝗱 𝗙𝗼𝗿𝗲𝗰𝗮𝘀𝘁))/𝗠𝗔𝗣𝗘 𝗼𝗳 𝗦𝘁𝗮𝘁𝗶𝘀𝘁𝗶𝗰𝗮𝗹 𝗙𝗼𝗿𝗲𝗰𝗮𝘀𝘁 𝗫 𝟭𝟬𝟬 𝗜𝗻𝘁𝗲𝗿𝗽𝗿𝗲𝘁𝗮𝘁𝗶𝗼𝗻: > 𝗣𝗼𝘀𝗶𝘁𝗶𝘃𝗲 𝗙𝗩𝗔 (%) → Adjustments improved accuracy > 𝗡𝗲𝗴𝗮𝘁𝗶𝘃𝗲 𝗙𝗩𝗔 (%) → Adjustments worsened accuracy > 𝗭𝗲𝗿𝗼 𝗙𝗩𝗔 → No impact (waste of effort) Let’s say: Statistical Forecast MAPE = 15% Final Adjusted Forecast MAPE = 10% FVA = (15−10)/15×100=33.3%  𝗔 𝗽𝗼𝘀𝗶𝘁𝗶𝘃𝗲 𝗙𝗩𝗔 𝗼𝗳 𝟯𝟯.𝟯% 𝗺𝗲𝗮𝗻𝘀 𝗺𝗮𝗻𝘂𝗮𝗹 𝗶𝗻𝗽𝘂𝘁𝘀 𝘀𝗶𝗴𝗻𝗶𝗳𝗶𝗰𝗮𝗻𝘁𝗹𝘆 𝗶𝗺𝗽𝗿𝗼𝘃𝗲𝗱 𝗳𝗼𝗿𝗲𝗰𝗮𝘀𝘁 𝗮𝗰𝗰𝘂𝗿𝗮𝗰𝘆. Why Should You Track FVA? > Helps differentiate useful vs. biased forecast changes > Reduces forecasting inefficiencies > Strengthens data-driven decision-making Track FVA by planner, product category, or forecast horizon to identify which inputs add value! #SupplyChain #DemandPlanning #Forecasting #InventoryManagement #Analytics #SafetyStock #CostOptimization #Logistics #Procurement #InventoryControl #LeanSixSigma #Cost #OperationalExcellence #BusinessExcellence #ContinuousImprovement #ProcessExcellence #Lean #OperationsManagement