Mean absolute scaled error

The mean absolute scaled error (MASE) is a scale-independent metric for assessing the accuracy of time series forecasts by comparing the absolute errors of a proposed forecasting method to those of a baseline naive forecast.^[1] Proposed by Rob J. Hyndman and Anne B. Koehler in 2006 as a superior alternative to traditional measures like mean absolute percentage error (MAPE), which can suffer from degeneracy in cases involving zeros or small values, MASE normalizes errors relative to in-sample one-step predictions, enabling fair comparisons across diverse datasets and forecasting horizons.^[1] For non-seasonal time series, MASE is calculated as the average absolute forecast error divided by the average absolute error of a naive one-step-ahead forecast from the training data.^[1] Specifically, if e_t denotes the forecast error at time t and y_t the observed value, then MASE = \frac{1}{h} \sum_{t=1}^h |e_t| / \frac{1}{n-1} \sum_{i=2}^n |y_i - y_{i-1}|, where h is the forecast horizon and n is the number of observations in the training period.^[1] This scaling factor, derived from the in-sample naive errors, ensures the measure is unit-free and avoids issues like infinite values in percentage-based metrics.^[1] In seasonal time series, the denominator adjusts to reflect the seasonal naive forecast, using the average absolute change over the seasonal period m (e.g., m=12 for monthly data).^[1] The formula becomes MASE = \frac{1}{h} \sum_{t=1}^h |e_t| / \frac{1}{n-m} \sum_{i=m+1}^n |y_i - y_{i-m}|, providing a robust benchmark that accounts for periodicity without assuming stationarity.^[1] MASE's key advantages include its interpretability—a value of 1 indicates the forecast performs as well as the naive method, while values below 1 signify improvement—and its applicability to intermittent demand or non-time-series data by adapting the scaling.^[1] It has been endorsed for use in forecast competitions and statistical testing frameworks, such as the Diebold-Mariano test, due to its favorable statistical properties like finite moments, making it preferable over squared-error metrics like RMSE for comparative analysis.^[2] Widely implemented in software libraries for forecasting, MASE remains a standard tool in fields like economics, supply chain management, and machine learning for evaluating model performance.^[3]

Introduction

Definition

The mean absolute scaled error (MASE) is a scale-independent measure of forecast accuracy, defined as the mean absolute forecast error divided by the mean absolute error of a benchmark naive forecast. This approach normalizes the errors relative to a simple one-step-ahead naive method, which uses the previous observation as the forecast for the next period. MASE was proposed by Rob J. Hyndman and Anne B. Koehler in 2006 as a standard for evaluating forecasting performance across diverse time series. For the non-seasonal case, the MASE is calculated using the formula

\text{MASE} = \frac{\frac{1}{n} \sum_{t=1}^{n} |e_t|}{\frac{1}{T-1} \sum_{t=2}^{T} |y_t - y_{t-1}|},

where e_t = y_t - \hat{y}_t denotes the forecast error at time t, with y_t as the actual value and \hat{y}_t as the forecast value; n is the number of out-of-sample forecast periods; and T is the length of the in-sample period used to compute the benchmark. The denominator serves as the scaling factor, representing the average in-sample error of the naive benchmark, which makes MASE unit-free and facilitates direct comparisons of forecast accuracy across datasets with varying scales or units.

History

The mean absolute scaled error (MASE) was proposed in 2006 by Rob J. Hyndman and Anne B. Koehler as a robust alternative to traditional forecast accuracy measures in their seminal paper "Another look at measures of forecast accuracy," published in the International Journal of Forecasting.^[1] This development emerged amid growing concerns in early 2000s forecasting research following the M3-Competition (2000), which highlighted persistent problems with metrics like the mean absolute percentage error (MAPE)—such as its tendency to become infinite or undefined in the presence of zeros or negative values—and the root mean squared error (RMSE), which is scale-dependent and overly sensitive to outliers.^[1] Hyndman and Koehler aimed to create a scale-independent measure suitable for comparing forecasts across diverse time series, drawing on real-world examples from prior competitions to demonstrate the shortcomings of existing approaches.^[1] Following its introduction, MASE gained rapid traction in the forecasting community, with Hyndman prominently recommending it as a standard evaluation tool in his subsequent works, including the widely used textbook Forecasting: Principles and Practice (first edition 2013, with ongoing updates).^[4] In this text, Hyndman emphasizes MASE's utility for cross-series comparisons due to its favorable properties, such as avoiding the pitfalls of percentage-based errors, and integrates it into practical forecasting workflows using software like R.^[4] This endorsement helped establish MASE as a benchmark in academic and applied forecasting, influencing its inclusion in forecasting competitions and software packages by the mid-2010s. Since its inception, MASE has seen no major methodological updates, remaining largely unchanged from the 2006 formulation, though later analyses have examined its statistical properties in specific contexts. For instance, Philip Hans Franses (2015) noted in a commentary that while MASE possesses desirable moment conditions making it compatible with significance testing frameworks like Diebold-Mariano tests.^[5]

Properties

Advantages

The mean absolute scaled error (MASE) offers scale invariance, enabling direct comparisons of forecast accuracy across time series with different units or magnitudes, unlike scale-dependent metrics such as mean absolute error (MAE) or root mean squared error (RMSE).^[6] This property arises from scaling the forecast errors by the in-sample MAE of a naive benchmark forecast, normalizing the metric to a unit-free scale.^[6] MASE effectively handles series with values near zero or intermittent demand, avoiding the division-by-zero problems and infinite values that plague percentage-based metrics like mean absolute percentage error (MAPE).^[6] By relying on absolute errors rather than relative ones, it remains well-defined even when actual values approach zero, providing reliable evaluations in such cases.^[6] The metric treats over- and under-forecasting symmetrically through its use of absolute errors, penalizing deviations equally regardless of direction, in contrast to squared-error measures like RMSE that disproportionately weight larger errors.^[6] This symmetry ensures balanced assessment of forecast performance without bias toward one type of error.^[6] MASE enhances interpretability by benchmarking against a simple naive forecast, where values less than 1 indicate superior accuracy to the naive method, a value of 1 matches it, and values greater than 1 signify poorer performance.^[6] This intuitive scale facilitates quick judgments of model efficacy relative to a baseline.^[6] Furthermore, the scaled errors in MASE exhibit asymptotic normality, assuming finite first and second moments, which supports the application of statistical tests such as the Diebold-Mariano test for comparing forecast accuracies across multiple models. This property aligns MASE with established inferential frameworks, enabling rigorous hypothesis testing.^[5] As a benchmark-based measure, MASE uses in-sample errors from the naive forecast for scaling, promoting robustness to out-of-sample variations and trends or seasonality in the data.^[6] This approach ensures consistent evaluation across diverse series without sensitivity to specific out-of-sample characteristics.^[6]

Limitations and interpretations

The mean absolute scaled error (MASE) depends critically on the quality of the benchmark naive forecast used for scaling, which can lead to misleading evaluations if an inappropriate benchmark is selected. For instance, in highly seasonal time series, applying a non-seasonal naive forecast (repeating the previous observation) as the benchmark may produce a poor scaling factor, undervaluing the performance of models that properly account for seasonality; instead, a seasonal naive forecast (repeating the observation from the same season in the prior cycle) is recommended to ensure fair comparison.^[4]^[7] MASE is particularly effective for evaluating short-term forecasts but may require adjustments for longer horizons, as the scaling is based on in-sample one-step naive errors, which do not directly capture the accumulation of uncertainty over extended periods. While MASE can be applied to multi-step forecasts, the one-step benchmark may not fully reflect the relative difficulty of longer-term predictions, potentially leading to less intuitive comparisons across varying forecast lengths.^[8]^[4] Interpreting MASE values lacks universal thresholds beyond the benchmark of 1, where values less than 1 indicate superior performance to the in-sample one-step naive forecast, and values greater than 1 signify inferior performance.^[8]^[9] Franses (2016) highlights the reliance of MASE on in-sample scaling for its desirable statistical properties, such as finite moments that support asymptotic normality in tests like the Diebold-Mariano framework, but notes that this in-sample focus may not always perfectly align with pure out-of-sample performance assessments.^[5] MASE's properties, including its scaling stability, assume sufficiently large samples for reliable estimation; in small datasets, the in-sample naive error denominator can become unstable or even undefined if historical observations are identical, leading to volatile MASE values.^[8]^[10]

Calculation Methods

Non-seasonal time series

For non-seasonal time series, the mean absolute scaled error (MASE) is computed using a simple naive benchmark that assumes the forecast for each period is the value from the immediately preceding observation. This benchmark, denoted as \hat{y}_t = y_{t-1}, represents a one-step-ahead prediction based on the random walk model without drift, suitable for data lacking strong trends or periodic patterns.^[6] The scaling factor for MASE in this context is the mean absolute change observed in the in-sample training data, calculated as \frac{1}{T-1} \sum_{t=2}^{T} |y_t - y_{t-1}|, where T is the number of observations in the training period. This denominator captures the average one-step deviation in the historical series, providing a unit-free scale that reflects the inherent variability under the naive approach. By dividing forecast errors by this factor, MASE becomes comparable across different series and units.^[6] The complete formula for MASE over a forecast horizon of h periods is given by

\text{MASE} = \frac{\frac{1}{h} \sum_{t=1}^{h} |y_{T+t} - \hat{y}_{T+t}|}{\frac{1}{T-1} \sum_{t=2}^{T} |y_t - y_{t-1}|},

where the numerator is the mean absolute error (MAE) of the out-of-sample forecasts, and \hat{y}_{T+t} are the predicted values for the holdout period. This ratio yields a value of 1 when the forecasting method performs equivalently to the in-sample naive benchmark; values below 1 indicate superior accuracy, while values above 1 suggest poorer performance. The approach assumes the data exhibits random walk-like behavior without pronounced seasonality, ensuring the naive one-lag shift serves as an appropriate baseline.^[6] To illustrate, consider a non-seasonal monthly sales series of 24 observations, such as retail units sold without evident cycles: the first 18 values form the training set (T=18). Compute the scaling factor by averaging the absolute differences between consecutive in-sample values, e.g., |y_2 - y_1| + \cdots + |y_{18} - y_{17}| divided by 17. For the remaining 6 values (h=6), generate naive forecasts (\hat{y}_{19} = y_{18}, \hat{y}_{20} = y_{19}, etc.), calculate the out-of-sample MAE, and divide by the scaling factor to obtain MASE. This process highlights how MASE normalizes errors relative to baseline variability, aiding model evaluation in stable, non-periodic contexts.^[6]

Seasonal time series

For time series data exhibiting strong seasonal patterns, the mean absolute scaled error (MASE) is adapted by using a seasonal naïve forecast as the benchmark, rather than the simple one-step naïve forecast appropriate for non-seasonal data.^[6] The seasonal naïve forecast predicts each future value \hat{y}_{T+t} as the value from the same season in the previous cycle, specifically \hat{y}_{T+t} = y_{T+t-m}, where m is the seasonal period (e.g., m=12 for monthly data or m=4 for quarterly data).^[4] This benchmark captures the inherent periodicity, making it suitable for evaluating forecasts against a simple method that replicates observed seasonal behavior.^[6] The scaling factor for seasonal MASE is the mean absolute error (MAE) of the seasonal naïve forecast applied in-sample to the training data, computed as \frac{1}{T-m} \sum_{t=m+1}^{T} |y_t - y_{t-m}|, where T is the length of the training series.^[4] This denominator normalizes the out-of-sample forecast errors relative to the average one-season-ahead error observed within the historical data, ensuring the metric is scale-independent and comparable across series with similar seasonality.^[6] The full formula for seasonal MASE, based on h out-of-sample forecasts, is then:

\text{MASE} = \frac{\frac{1}{h} \sum_{t=1}^{h} |y_{T+t} - \hat{y}_{T+t}|}{\frac{1}{T-m} \sum_{t=m+1}^{T} |y_t - y_{t-m}|}

This yields a value of 1 when the forecast accuracy matches that of the seasonal naïve benchmark; values below 1 indicate superior performance, while values above 1 suggest poorer accuracy.^[4] For instance, in quarterly data with m=4, forecast errors are scaled against the average absolute deviations between observations one season apart in the training period, such as comparing a model's errors to the typical quarterly fluctuations in historical sales or production data.^[6] This seasonal adaptation assumes the presence of strong, consistent periodic patterns in the data; applying the non-seasonal version to such series would yield misleading results by ignoring the dominant cyclical structure.^[4]

Non-time series data

The mean absolute scaled error (MASE) can be adapted for non-time series data, such as cross-sectional observations, where there is no inherent temporal ordering or structure. In this context, the data are treated as a collection of independent observations, and the benchmark forecast is the simple mean of the training data, denoted as \hat{y} = \bar{y}, which serves as a naive predictor without exploiting any sequential dependencies.^[11] This adaptation maintains the scale-independent property of MASE, allowing comparisons across datasets with varying units or magnitudes.^[11] The scaling factor for non-time series MASE is the mean absolute deviation (MAD) from the sample mean in the training data, calculated as \frac{1}{T} \sum_{t=1}^{T} |y_t - \bar{y}|, where T is the number of training observations and \bar{y} is the mean of the training set. The full MASE formula then becomes

\text{MASE} = \frac{\frac{1}{n} \sum_{i=1}^{n} |y_i - \hat{y}_i|}{\frac{1}{T} \sum_{t=1}^{T} |y_t - \bar{y}|},

where n is the number of out-of-sample predictions, y_i are the actual values, and \hat{y}_i are the forecasted values. A MASE value less than 1 indicates that the forecasting method outperforms the naive mean benchmark, while values greater than 1 suggest inferior performance. This formulation aligns with implementations in statistical software for non-temporal data analysis.^[11] For example, in predicting house prices based on cross-sectional features like location and size, the training data might consist of historical sales without time ordering. Here, prediction errors are scaled by the average absolute deviation from the mean training price, providing a unitless measure to evaluate model accuracy relative to simply predicting the average price for all houses. This approach is particularly useful in scenarios like demand forecasting across product categories or regression tasks on static datasets, where temporal autocorrelation is absent.^[11]

Applications and Extensions

Forecasting evaluation

The mean absolute scaled error (MASE) plays a key role in model selection for time series forecasting by providing a scale-independent metric to rank competing models on the same dataset. For instance, it enables direct comparison of autoregressive integrated moving average (ARIMA) and exponential smoothing state space (ETS) models, where lower MASE values indicate superior out-of-sample performance.^[12] In cross-validation procedures, MASE is computed on hold-out sets or through rolling-origin time series cross-validation to assess model robustness across multiple forecast origins. This approach evaluates one-step-ahead forecasts iteratively, using in-sample naïve errors for scaling, which ensures consistent measurement of generalization error without scale biases.^[13]^[6] The scale-free property of MASE facilitates multi-series comparisons by allowing aggregation of errors across diverse datasets, such as averaging individual MASE values to gauge overall model efficacy in large-scale applications. This aggregation is standard for hierarchical or grouped time series, enabling benchmarks like those in the M-competition datasets where mean MASE summarizes performance over hundreds of series.^[14]^[6] In energy demand forecasting, MASE helps identify superior models for production planning; for example, across 25 electricity load series, machine learning hierarchies achieved average MASE of 0.72, outperforming benchmarks and informing grid optimization.^[15] For statistical significance, differences in MASE between models can be tested using the Diebold-Mariano framework, which leverages the asymptotic normality of scaled errors to determine if one forecasting method significantly outperforms another.^[5]^[16]

Intermittent demand forecasting

Intermittent demand patterns, characterized by frequent zero observations interspersed with sporadic non-zero demands, pose significant challenges for forecast accuracy evaluation. Traditional metrics such as the mean absolute percentage error (MAPE) often fail in these scenarios because division by zero observations results in infinite or undefined values, rendering them unusable for series with sparsity.^[8] The mean absolute scaled error (MASE) addresses this issue through its absolute scaling approach, which divides forecast errors by the in-sample mean absolute error of a naive benchmark forecast, avoiding division by actual values and ensuring finite results even in highly sparse data.^[8] For intermittent series, MASE typically employs the non-seasonal benchmark, calculating the scale as the average one-step-ahead absolute differences from historical data, which naturally accommodates zeros without requiring special adjustments. However, in zero-inflated contexts, evaluations may emphasize errors on non-zero periods to better capture demand variability, though standard MASE computations include all periods for a comprehensive assessment. This adaptation maintains MASE's scale-independence, allowing consistent comparisons across lumpy or erratic intermittent patterns.^[8]^[17] A practical example arises in inventory management for slow-moving spare parts, where MASE is used to assess methods like Croston's approach— which separately forecasts demand sizes and intervals—and the Syntetos-Boylan approximation (SBA), an unbiased variant, against simple naive baselines such as single exponential smoothing. In empirical studies of enterprise data with over 16,000 intermittent items, MASE revealed that while Croston and SBA often underperform relative to optimized methods like the Teunter-Syntetos-Babai (TSB) approach (with scaled MASE values indicating higher errors for Croston and SBA), they provide viable baselines for lumpy demands, outperforming naive methods in erratic subsets.^[17] The benefits of MASE in this domain include delivering stable, interpretable measures for lumpy demand series, where traditional metrics fluctuate wildly due to irregularity, thereby supporting more reliable supply chain decisions such as stock level optimization and reorder policies.^[8] This application was highlighted in foundational work by Hyndman (2006), who advocated MASE as a robust standard for intermittent demand evaluation, with subsequent research confirming its utility in ranking forecasters and handling bias in sparse inventory contexts.^[8]^[17]