Purpose
An automated classifier that sorts incoming bug reports into Defects, Tasks and Enhancements, so that triage begins the moment a report is filed.
A machine learning study · Mozilla Bugzilla
Ten thousand bug reports, three categories, one question: can a model tell a defect from a task or an enhancement by its words alone? A stacking ensemble answers with 94.34% accuracy.
Begin with the dataAn automated classifier that sorts incoming bug reports into Defects, Tasks and Enhancements, so that triage begins the moment a report is filed.
Ensemble learning over carefully processed text, paired with deliberate class balancing, rather than reliance on any single algorithm.
Ten thousand reports exported from Mozilla’s Bugzilla, spanning Firefox, Core and the wider platform, complete and without missing values.
The fields supplied with every report. Identifiers and post-triage fields were later removed to avoid leakage.
Four of ten thousand rows, shown as exported.
| Bug ID | Type | Summary |
|---|---|---|
| 1949668 | Defect | Race condition between clearing mIsDeferredPurgePending… |
| 1948993 | Task | Remove Nightly condition for vertical tabs checkboxes… |
| 1947536 | Defect | QR code image is not exposed to assistive technology… |
| 1947606 | Task | Add Nimbus support for calculator |
Four stages turn raw exports into a balanced, model-ready feature matrix.
Irrelevant and leakage-prone columns removed; missing values handled.
Temporal features extracted and sparse categories grouped.
Bug summaries vectorised with TF-IDF.
SMOTE applied to equalise the three classes.
Ten classifiers, from a single perceptron to a stacked ensemble, trained on identical features.
94.34%
Ensemble of base learners
91.73%
Bagged decision trees
83.81%
Single interpretable tree
Precision, recall and F1 for the three strongest models, with the confusion matrix of the winner.
| Model | Accuracy | Precision | Recall | F1 |
|---|---|---|---|---|
| Stacking Classifier | 94.34% | 94.20% | 94.10% | 94.15% |
| Random Forest | 91.73% | 91.50% | 91.80% | 91.65% |
| Decision Tree | 83.81% | 83.50% | 83.90% | 83.70% |
Stacking several algorithms reached 94.34%, more than two and a half points clear of the best single model.
TF-IDF features from bug summaries supplied the most predictive power; each class has a distinctive vocabulary.
Without SMOTE, the dominant defect class swamps the minority classes. Balancing lifted performance markedly.
Careful feature engineering and preprocessing improved both accuracy and generalisation across every model.