Bug Classification Study
IOverview IIData IIIMethod IVModels VPerformance VIFindings

A machine learning study · Mozilla Bugzilla

Reading the language of software defects.

Ten thousand bug reports, three categories, one question: can a model tell a defect from a task or an enhancement by its words alone? A stacking ensemble answers with 94.34% accuracy.

Begin with the data
Fig. 1 Distribution of report types across the dataset.
Bug reports
10,000
Best accuracy
94.34%
Models evaluated
10
Engineered features
500+
I

The project in brief

Purpose

An automated classifier that sorts incoming bug reports into Defects, Tasks and Enhancements, so that triage begins the moment a report is filed.

Outcomes

  • 94.34% accuracy with a stacking classifier
  • TF-IDF text features with SMOTE balancing
  • Ten algorithms evaluated side by side
  • An interactive report of the results

Approach

Ensemble learning over carefully processed text, paired with deliberate class balancing, rather than reliance on any single algorithm.

II

The dataset

Ten thousand reports exported from Mozilla’s Bugzilla, spanning Firefox, Core and the wider platform, complete and without missing values.

Records
10,000
Original columns
9
Missing values
0
Data quality
High

Schema

The fields supplied with every report. Identifiers and post-triage fields were later removed to avoid leakage.

Bug ID
Unique identifier for each report
Type
Classification target — defect, task or enhancement
Summary
Free-text description of the issue
Product
Software product, e.g. Firefox or Core
Component
Specific component within the product
Assignee
Developer assigned to the fix
Status
Current state, e.g. RESOLVED or OPEN
Resolution
How the report was closed, e.g. FIXED
Updated
Timestamp of the last update

Sample records

Four of ten thousand rows, shown as exported.

Bug ID Type Summary Product Component
1949668 Defect Race condition between clearing mIsDeferredPurgePending… Core Memory Allocator
1948993 Task Remove Nightly condition for vertical tabs checkboxes… Firefox Sidebar
1947536 Defect QR code image is not exposed to assistive technology… Firefox Messaging System
1947606 Task Add Nimbus support for calculator Firefox Address Bar
Fig. 2 Reports filed per year.
Fig. 3 Reports by product, top five.
III

Method

Four stages turn raw exports into a balanced, model-ready feature matrix.

  1. 01

    Cleaning

    Irrelevant and leakage-prone columns removed; missing values handled.

  2. 02

    Feature engineering

    Temporal features extracted and sparse categories grouped.

  3. 03

    Text processing

    Bug summaries vectorised with TF-IDF.

  4. 04

    Class balancing

    SMOTE applied to equalise the three classes.

Fig. 4 Class counts before SMOTE — heavily skewed toward defects.
Fig. 5 Class counts after SMOTE — balanced at 6,712 each.
IV

The models

Ten classifiers, from a single perceptron to a stacked ensemble, trained on identical features.

  1. First

    Stacking Classifier

    94.34%

    Ensemble of base learners

  2. Second

    Random Forest

    91.73%

    Bagged decision trees

  3. Third

    Decision Tree

    83.81%

    Single interpretable tree

Fig. 6 Test accuracy of all ten models.
V

Performance

Precision, recall and F1 for the three strongest models, with the confusion matrix of the winner.

Metrics

Model Accuracy Precision Recall F1
Stacking Classifier 94.34% 94.20% 94.10% 94.15%
Random Forest 91.73% 91.50% 91.80% 91.65%
Decision Tree 83.81% 83.50% 83.90% 83.70%

Confusion matrix

Fig. 7 Stacking classifier, actual versus predicted class.
VI

Findings

  1. Ensembles outperform

    Stacking several algorithms reached 94.34%, more than two and a half points clear of the best single model.

  2. Language carries the signal

    TF-IDF features from bug summaries supplied the most predictive power; each class has a distinctive vocabulary.

  3. Balance is essential

    Without SMOTE, the dominant defect class swamps the minority classes. Balancing lifted performance markedly.

  4. Preparation compounds

    Careful feature engineering and preprocessing improved both accuracy and generalisation across every model.

In practice

Automated triage

  • Reports classified the moment they are filed
  • Substantially less manual sorting
  • Faster response to critical defects
  • Better allocation of work across teams

Quality insight

  • Trends monitored over time
  • High-risk components identified early
  • Proactive quality improvement
  • Decisions grounded in data