Experiments were conducted on four widely used emotional speech corpora (CREMA-D, RAVDESS, SAVEE, and TESS), under within-corpus evaluation settings. Performance was measured using Accuracy (ACC), Macro-averaged F1-score (Macro-F1), and Unweighted Average Recall (UAR) to account for class imbalance and per-class robustness (Table 1).

Baseline models were constructed using the complete handcrafted acoustic feature set (296 features), without applying any feature selection. This baseline serves as a full-feature reference model to evaluate whether the proposed explainability-guided feature selection strategy improves performance beyond the original feature space. For fair comparison, identical classifier configurations and train-validation splits were maintained between the baseline and proposed methods; the only difference was whether all features were used or a Top-K subset selected using SHAP and permutation-based ranking was applied. Such full-feature reference models are commonly used in feature selection studies to assess the effectiveness of dimensionality reduction methods [35].

Table 2 summarizes performance for the baseline and proposed methods. The proposed feature-selection framework generally improves performance across the evaluated datasets, with larger gains on CREMA-D and SAVEE and more modest improvements on RAVDESS, while reducing the feature space by up to 40–70%. To address potential model dependency, the proposed feature selection framework was additionally evaluated using multiple classifiers, including XGBoost (XGB), Support Vector Machine (SVM), and Random Forest (RF). Table 3 present class-wise precision, recall, and F1-score for the proposed model across the four datasets.

Across four speech emotion recognition corpora (CREMA, RAVDESS, SAVEE, and TESS), we evaluated the proposed SHAP- and permutation-guided feature selection framework using all classifiers. On CREMA-D (6 classes, 7,441 samples), XGBoost showed consistent improvement after feature selection, with accuracy increasing from 0.593 to 0.600 and macro-F1 from 0.591 to 0.598. Random Forest also exhibited modest gains, improving accuracy from 0.534 to 0.540 and macro-F1 from 0.522 to 0.530. In contrast, SVM performance remained nearly unchanged, indicating that feature reduction had limited impact on this corpus for kernel-based classification.

On TESS (7 classes, 2,800 balanced samples), all classifiers achieved near-ceiling performance. XGBoost improved slightly from 0.995 to 0.998 accuracy, while SVM and RF maintained approximately 0.999 and 0.998 accuracy, respectively. These results suggest that the TESS corpus is highly separable even without aggressive feature reduction, leaving limited room for measurable improvement.

Larger improvements were observed on more challenging or smaller datasets. On SAVEE (7 classes, 480 samples), XGBoost improved from 0.708 to 0.726 accuracy, with macro-F1 increasing from 0.680 to 0.696. SVM also benefited slightly, improving from 0.761 to 0.764 accuracy and from 0.744 to 0.747 macro-F1. Random Forest demonstrated moderate gains, with accuracy increasing from 0.708 to 0.715 and macro-F1 from 0.683 to 0.692.

On RAVDESS (7 classes, 1,440 samples), XGBoost again showed consistent improvement, increasing accuracy from 0.744 to 0.752 and macro-F1 from 0.728 to 0.737. However, SVM experienced a slight performance decrease after feature selection, with accuracy dropping from 0.834 to 0.821 and macro-F1 from 0.826 to 0.813, suggesting that SVM may rely more heavily on the complete feature representation for this corpus. Random Forest showed marginal reductions as well, although overall trends remained relatively stable.

Overall, the results indicate that the proposed feature selection framework generalizes across multiple classifiers while providing the most consistent benefits when combined with XGBoost. The improvements are particularly noticeable in datasets with limited sample sizes or higher class confusability, whereas gains remain minimal in highly separable datasets where baseline performance is already saturated.

The class-wise evaluation results in Table 3 demonstrate the effectiveness of the proposed feature selection framework when combined with the XGBoost classifier across four benchmark speech emotion recognition datasets. On the CREMA-D dataset, the model shows relatively balanced performance, with the highest F1-score observed for angry (0.72), followed by sad (0.61) and neutral (0.59). However, the model exhibits comparatively lower performance for disgust (0.53) and fear (0.54), indicating that these emotions remain more challenging to distinguish due to their acoustic similarity with other emotional states. Among the evaluated classifiers, XGBoost provided the most consistent overall improvements after feature selection and therefore serves as the primary model for detailed class-wise analysis. For the RAVDESS dataset, the model achieves improved overall performance, with strong results for fear (F1 = 0.79) and surprise (F1 = 0.79), while neutral achieves the highest F1-score of 0.82 due to its very high recall (0.95). In contrast, sad shows relatively lower recall (0.44), resulting in a lower F1-score (0.54).

On the SAVEE dataset, the model demonstrates strong recognition capability, particularly for neutral (F1 = 0.86) and sad (F1 = 0.77), while disgust remains the most difficult class with the lowest F1-score (0.47). The higher recall for happy (0.78) also contributes to a competitive F1-score of 0.68. Finally, the TESS dataset shows near-perfect classification performance across all emotion categories, with F1-scores reaching 1.00 for most classes and 0.99 for surprise. This indicates that the proposed model generalizes extremely well on datasets with clearer emotional articulation and less acoustic variability. Overall, the results highlight the robustness of the proposed approach while also revealing that certain emotions, particularly disgust and fear, remain comparatively harder to recognize in more complex datasets.

The confusion matrices further illustrate the class-wise behavior of the proposed XGBoost-based model across the four datasets, as shown in Fig. 3. For CREMA-D, most correct predictions are concentrated along the diagonal, indicating reasonable classification performance. However, several misclassifications are observed, particularly between angry and disgust (0.246), suggesting that these emotions share similar acoustic characteristics. Additionally, sad is frequently confused with neutral (0.368), which explains the relatively lower recall for the sad class. Despite these confusions, neutral (0.953) and surprise (0.845) demonstrate strong recognition rates.

Confusion matrices of the proposed XGBoost-based model on four speech emotion recognition datasets: a CREMA-D, b RAVDESS, c SAVEE, and d TESS

In the RAVDESS dataset, the confusion matrix shows generally strong class separability. The fear (0.793) and disgust (0.810) classes achieve high correct classification rates, while neutral also demonstrates strong recognition performance. Some confusion still occurs between angry and disgust, as well as between happy and surprise, indicating overlapping vocal expressions in these emotional states. Overall, most values remain concentrated along the diagonal, reflecting stable model performance across the majority of emotion classes.

For SAVEE, the confusion matrix reveals more noticeable misclassification among certain emotions due to the smaller dataset size and speaker variability. For example, fear is often confused with happy (0.222), and disgust is split between disgust and neutral (0.389 each). Nevertheless, neutral is perfectly recognized (1.000), and happy (0.778) and sad (0.667) maintain relatively strong recognition rates compared to other classes.

Finally, the TESS confusion matrix demonstrates near-perfect classification performance. Almost all predictions lie on the diagonal, with most classes achieving 1.000 correct recognition, and only very minor confusion appearing in a few instances (e.g., disgust and happy at 0.992). This result indicates that the proposed model can effectively capture discriminative emotional features when the dataset contains clearer and more consistent speech patterns. Overall, the confusion matrix analysis confirms the robustness of the proposed model while highlighting the remaining challenges in distinguishing acoustically similar emotions in more complex datasets. Due to space limitations, class-wise results for SVM and Random Forest are not included, although overall performance comparisons are provided in Table 2.

The selected feature sets show strong consistency across datasets and seeds. Table 4 summarizes the selected feature subset sizes (\(K\)), dominant feature groups, and representative top-ranked descriptors identified across the evaluated corpora.

The feature ranking analysis demonstrates that the proposed framework consistently prioritizes acoustically meaningful descriptors across datasets and selected feature subset sizes (K). Spectral and MFCC-based features were frequently selected in all corpora, while prosodic and temporal-dynamics descriptors became more prominent in RAVDESS and TESS. Voice-quality measures such as HNR and glottal-noise-related features were also repeatedly selected, indicating their importance for emotion characterization. The results further show that different datasets favor different compact feature subset sizes, reflecting variations in corpus complexity, speaker diversity, and class separability.

Notably, even in TESS, where baseline performance is near ceiling, the method still achieves marginal but consistent improvements, indicating that the selection process does not harm performance even in highly controlled settings.