Application of Ensemble Learning in CXR Classification for Enhancing COVID-19 Diagnosis

This study delves into the vital task of classifying chest X-ray (CXR) samples, particularly those related to respiratory ailments, using advanced clinical image analysis and computer-aided radiology techniques. Its primary focus is on developing a classifier to accurately identify COVID-19 cases. Through the application of machine learning and computer vision methodologies, the research aims to enhance the precision of COVID-19 detection. It investigates the effectiveness of Histogram of Oriented Gradients (HOG) feature extraction techniques in conjunction with various classifiers, such as Support Vector Machine (SVM), Decision Tree (DT), Naive Bayes (NB), K-nearest neighbor (KNN), and Tree Bagger (TB), alongside an innovative ensemble learning approach. Results indicate impressive accuracy rates, with KNN, SVM, DT, NB, and TB all surpassing the 90% mark. However, the ensemble learning method emerges as the standout performer. By leveraging HOG features extracted from CXR images, this approach presents a robust solution for COVID-19 diagnosis, offering a powerful tool to address the diagnostic challenges posed by the pandemic.


Introduction
Radiology, a medical discipline employing radiation and imaging technology, aids in diagnosing and treating diseases.Computer-aided diagnosis (CAD) serves as a valuable adjunct, offering radiologists a "second opinion" in interpreting chest X-rays (CXRs) to detect illness [1].CAD assists in diagnosing various conditions such as atelectasis, consolidation, pneumothorax, and pneumonia, critical in infectious respiratory disorders diagnosis [2].Enhancing CAD capabilities aims to automate illness identification and categorization during CXR interpretation, improving diagnostic precision and consistency.This advancement streamlines radiological workflows, allowing radiologists to work more effectively and efficiently [3].Computer processing of medical images encompasses acquisition, generation, analysis, and visualization.Figure (1) illustrates the fundamental steps of image processing, underscoring its pivotal role in modern medical diagnostics.

Image acquisition Preprocessing Feature Extraction Classification
Pattern recognition, commonly known as image mining and machine learning, has emerged as a recent technique for analyzing clinical images, aiding in the automatic detection and classification of various diseases through Computer-Aided Diagnosis (CAD).With the World Health Organization declaring COVID-19 a pandemic on March 11, 2020, the urgent need for efficient screening and rapid clinical intervention for infected individuals became paramount.While real-time reverse transcription polymerase chain reaction (RT-PCR) remains the primary diagnostic method for COVID-19, its costliness and timeconsuming nature necessitate the exploration of alternative diagnostic approaches.Chest X-ray (CXR) imaging provides a timely means of assessing suspected cases, yet the overlapping features of viral pneumonia with other lung infections underscore the need for improved diagnostic methods.This study aims to address this challenge by developing an advanced classifier capable of swiftly categorizing COVID-19 cases based on early X-ray findings as either positive or negative.Such a method holds promise for expediting patient treatment and ensuring accurate disease diagnosis, vital for saving lives amidst the ongoing pandemic.[32].Ershadi and Seifi proposed a dynamic multi-classifier method for disease diagnosis, combining feature reduction techniques and clustering selection to enhance accuracy and computation time efficiency [33].Other authors proposed a multi-objective optimization model for pharmaceutical supply chain logistics in pandemic situations, aiming to minimize unsatisfied requests and transportation costs while considering various factors and employing a hybrid optimization approach [34].Rahimi Rise et al. explored COVID-19 outbreak scenarios in Iran using system dynamics modeling, emphasizing the transportation system's impact and proposing strategies for government decision-making amid varying mortality rates and recovery scenarios [35].Ershadi and Seifi represented dynamic feature selection and clustering methods to enhance medical diagnosis.Their novel approach combines feature selection, clustering, and deep learning to improve classification performance significantly [36].They developed an efficient Bayesian network for differential diagnosis, integrating expert knowledge and data-driven methods, achieving up to 87% accuracy, as well [37].

Proposed Methodology
This method utilizes machine learning to efficiently identify and categorize COVID-19 from X-rays, streamlining the labor-intensive process of diagnosis.Employing a two-stage strategy, it extracts features in the first stage and classifies images in the second.Various classifiers, including KNN, SVM, decision tree (DT), Tree Bagger (TB), and ensemble learning, are evaluated post feature extraction.Histogram of Oriented Gradients (HOG) is employed for this purpose.The subsequent sections detail the techniques employed in this method to enhance COVID-19 detection from chest X-ray images.

Histogram of Oriented Gradients
Histogram of Oriented Gradients (HOG) is a popular method for extracting features from image data, focusing on object structure and shape.HOG identifies edge features by determining pixel edges and their directions, calculating gradients and edge orientations within localized sections of the image.These sections create histograms based on gradient orientations, producing distinct histograms for each region.Each image block overlaps by 50% and is divided into cells, with cells potentially appearing in multiple blocks due to overlap.For each pixel in each cell, x and y gradients (Gx and Gy) are computed.This process illustrates how gradients represent edges in two directions across an image (see Equation 1).Additionally, Figure (2) illustrates standard HOG and viral HOG.Gradients' magnitudes and phases are then determined accordingly.
where r is the magnitude, and θ is the angle.where Nv denotes all training patterns.The input test vector, up, is compared with the training data to determine its category, denoted by the class labels, i, and compared with the example vectors, mik, to ascertain the exact category (see Equation 2).
In this context, mik signifies the example vector, while the input test vector is represented as up.We consider a collection of metric space points labeled 0 or 1.Given a query (S, T) and samples (S1, T1), (S2, T2), ... represented as (Sn, Tn), the k-nearest neighbor classifier determines the label of the query based on the class with the highest prevalence among the k nearest points to s in the labeled sample.We employ an odd integer for k to avoid ties.Ties can occur either when multiple points at the same distance from the query fail to provide distinct answers or when multiple classes occur with the same frequency among the query's k-nearest neighbors.To prevent distance ties, we demonstrate universal consistency without assuming density distributions.Various techniques, including random selection, are discussed in the literature to resolve ties in the voting process.

Support Vector Machine Classifier
The Support Vector Machine (SVM) operates by segmenting the search space to maximize distance to data points.It excels in text data analysis, allowing flexible feature selection.Its linear method suits highdimensional text classification.However, excessive parameters hinder performance, mitigated by parameter reduction and focused feature selection.SVM, a prominent kernel algorithm, employs hyperplane separation for classification based on maximizing margins between classes and nearest points.

Decision Tree Classifier
Decision trees (DT) serve as a versatile non-parametric supervised learning method for classification and regression tasks.Each internal node in a DT evaluates a specific attribute, with branches representing test outcomes and leaf nodes signifying examined features.The tree comprises decision nodes, chance nodes, and end nodes, with leaf nodes containing the final outcome.The path from root to leaf forms conjunctions in decision tree conditions, enabling the generation of decision rules.These rules can elucidate causal or temporal relationships, aiding in association rule building.DT's transparency as a white box model renders it easily interpretable, and it demonstrates efficacy even with limited training data, making it a valuable tool for various analytical tasks.Decision tree methods, renowned for their widespread use in supervised learning, predict model accuracy.However, ensemble methods, such as bagging, boosting, and random forest, surpass individual decision trees.These ensemble techniques combine multiple decision trees to enhance predictive performance.Decision trees serve as graphical representations of complex decision scenarios, extracting knowledge from vast data.They efficiently classify new data and offer a concise and easily storable format.

Naive Bayes (NB) Classifier
The Naive Bayes classifier, rooted in Bayesian statistics, assumes strong independence between features, simplifying classification.It models each class feature independently, aiding in fruit classification, for instance.Trained via supervised learning, it estimates parameters using maximum likelihood, facilitating application with minimal training data.By assuming independence, only variable variances need be determined, not the entire covariance matrix.The classifier employs the maximal a posteriori choice rule, selecting the hypothesis with the greatest likelihood.This process involves increasing conditional probabilities of features given the class label for each potential label.Overall, Naive Bayes classifiers offer efficient classification, particularly suitable for scenarios with limited training data (see Equation 3).
) where p(Cj) is the conditional probability label, and p (Ti, Cj) represents every label and feature.As a result, it appears that the only requirement to construct the classifier is to calculate every conditional probability, p (Ti, Cj), for every label and feature before multiplying the results by the prior probability for that label, p(Cj).The label for which the classifier gets best product is returned by the classifier.

Tree Bagger Classifier
In the decision-making process of a decision tree, progression occurs from a root node to a leaf node, with each step predicting the input variable.However, a single tree may overfit the model.To mitigate this, bootstrap aggregation, a bagging-based technique, is employed.It generates multiple learners by creating additional data points following the same uniform probability distribution.Typically, N learners are averaged to determine the final learning error (see Equation 4).Components of the tree are drawn using a bootstrap replica of the ensemble, growing independently."Out of bag" observations refer to data elements excluded from computation.This approach helps reduce overfitting and enhances the robustness of the model.where N is the learner, and e is the final error.

The proposed ensemble learning model
The proposed ensemble method combines predictions from these five classifiers through majority voting to derive the final prediction.The steps involved in the proposed ensemble learning algorithm are outlined as follows: Step 1: Load the Dataset • Specify the number of train-test splits to perform (numsplits).
• Initialize an empty list to collect evaluation results (results).
• Iterate over a range of numsplits for repeated train-test splits.
• Employ stratified sampling to divide the data into training and test sets.
• Train the ensemble classifier on the training data and predict on the test data.
• Append the evaluation results to the results list.
Step 6: Construct a Results DataFrame • Create a DataFrame (resultsdf) to store the evaluation results, encompassing metrics, confusion matrices, and timing details.

Step 7: Save the Results
This algorithm iterates through Steps 5 to 7 for each train-test split, yielding multiple sets of evaluation metrics and confusion matrices.By leveraging predictions from various classification models, this ensemble method enhances the robustness and accuracy of classifications.

Used dataset
Data collection is paramount in machine learning research, especially in medical imaging.This study requires a diverse set of chest X-ray (CXR) images encompassing pneumonia, COVID-19 positive and negative cases, and normal cases.Unfortunately, standalone datasets representing each category independently do not exist.Instead, samples are gathered from two sources: a dataset provided by Dr.Joseph Paul, a postdoctoral scholar, and CXR datasets available on Kaggle [20].Dr. Paul's dataset includes CXR and CT scan samples not only of COVID-19 cases but also of other respiratory viruses such as ARDS, SARS, and MERS [21,22].For this research, COVID-19 image samples from Dr. Paul's dataset are utilized.Additionally, Kaggle provides free access to relevant data for research purposes [23].The collected images undergo organization, preprocessing, and conversion into NumPy arrays to facilitate the training process.Notably, the Kaggle dataset offers a variety of CXR images showcasing different chest perspectives of patients afflicted with COVID-19, ARDS, SARS, MERS, and other disorders [23].Figure 3 presents an illustrative example featuring CXR images of both normal individuals and those affected by viral respiratory conditions.By leveraging these datasets, researchers can access a rich pool of CXR images crucial for training machine learning models.This comprehensive dataset not only aids in pneumonia and COVID-19 diagnosis but also contributes to advancing the broader field of medical imaging research.

Results and Discussion
Prior to presenting the results, the evaluation metrics are outlined as follows.

Evaluation Criteria
The confusion matrix, a pivotal metric in assessing machine learning algorithms [24], juxtaposes system outputs with reference data.Derived from it are accuracy, sensitivity, specificity, precision, recall, F-Measure, and G-Mean [25].True positive (TP), true negative (TN), false positive (FP), and false negative (FN) are key statistical indices [25].Figure 4 depicts a sample confusion matrix.

Figure (4). Confusion matrix for binary classification
A classifier's accuracy is measured as the ratio proportion of positive measures to all measures.It determines the degree of accuracy [24] (see Equation 5).
Accuracy = (TP + TN)/(TP + TN + FN + FP) (5) The sensitivity of a classifier is evaluated as a ratio proportion of true positive measures to all positive measures (see Equation 6).
Sensitivity = TP/ (TP + FN) = TPR (6) The specificity of a classifier is measured by the ratio of true negative measures to all negative measures [24] (see Equation 7).Specificity = TN/(FP + TN) = TNR (7) The way in which the percent of all positives were correctly classified is by precision [26] (see Equation 8).

Real
True False True TP FN

False FP TN
Despite the fact that negative situations are classified properly, a low G-Mean specifies poor performance in categorizing the positive data [26] (see Equation 11).G − Mean = sqrt(TPR × TNR)

. Analysis and Discussion
After preprocessing the data and employing classifiers through 10-fold cross-validation, the average results of 10 distinct test sets are shown in Table 1.We conducted these experiments using Python 3.11.5 and Anaconda3 2023.03 on a personal computer equipped with an Intel® Core™ i5-11400H CPU running at 2.70GHz and 16.00 GB of RAM for all executions.Related results with 60% training data samples with 40% testing data samples are represented in Table (1) and Figure (5).Moreover, the scope of this study extends beyond COVID-19 detection alone.There exists the potential to expand the capabilities of the existing model to not only ascertain the presence of COVID-19 but also to identify other infectious diseases.This broader application could significantly contribute to the medical field's diagnostic capabilities, facilitating prompt and accurate identification of various illnesses.

Step 2 : 3 : 5 :
Prepare the Dataset • Perform data preprocessing, including removing non-numeric columns, converting columns to a numeric format, and handling missing values.Step Define the Ensemble Classifier Function • Create a function, 'ensemble_classifier,' which inputs the training data (Xtrain, ytrain) and test data (Xtest).• Within this function, obtain predictions from five classifiers • Combine these predictions using majority voting and return the ultimate ensemble prediction.Step 4: Define Classifier Functions • Establish separate functions for each classifier • Each classifier possesses its unique architecture and hyperparameters.• Compile and train each classifier on the training data, returning predictions for the test data.Step Execute Ensemble Learning and Evaluation

Figure ( 3 ).
Figure (3).Sample image of normal (left picture) and viral (right picture) CXR It is shown in Tables(1)(2)(3)(4) and Figures(5)(6)(7)(8) that the proposed ensemble classifier has better performance among other classification methods.Figure9represents a radar chart to understand it better.The performances of the proposed ensemble learning classifier cover other performances in this chart and it is superior among other classifiers.

Figure ( 9 ).
Figure (9).Radar chart of different metrics for various training/testing data samples Observing a gap in existing literature, this study addresses the scarcity of research utilizing KNN, SVM, DT, TB, NB, and ensemble learning techniques for COVID-19 detection.It endeavors to construct a classifier employing these methods to discern COVID-19 cases as positive or negative.The underlying points are contributions of recommended technique:v Introducing a novel classifier utilizing diverse machine learning techniques for COVID-19 classification.v Leveraging X-ray imaging to detect COVID-19, focusing on the lungs as the primary site of infection.v Proposing a machine learning-based approach using clinical data to identify COVID-19 in suspected cases.
[31]veloping a computer-aided design technique to analyze records from COVID-19 patients or suspects, employing machine learning for enhanced processing speed and accuracy.2.Related workArtificial intelligence is revolutionizing the detection of medical conditions like breast cancer, brain tumors, and COVID-19 using deep learning methods on CXR images.Yet, many studies rely on limited COVID-19 datasets, making it hard to generalize results and ensure prototype efficacy on larger samples.Related works in this fields are reviewed as follows.Instead of traditional convolutional neural networks (CNNs) for COVID-19 identification, Afshar et al. proposed the use of COVID-CAPS capsule network[1].Arman et al. introduced a Bayesian optimization method, achieving a 94% accuracy in COVID-19 detection[2].Apostolopoulos recommended transfer learning and CNNs for identifying COVID-19 from limited datasets[3].Ranganath et al. suggested a pivot distribution approach for COVID-19 identification from chest X-ray (CXR) images[4].Das et al. proposed the velocity-enhanced whale optimization algorithm hybridized by artificial neural networks for medical data classification[5].Han suggested a support vector machine (SVM) Kör et al. utilized transfer learning to develop a multi-class convolutional neural network model for automatic pneumonia identification and distinguishing between pneumonia with and without COVID-19[11].Meanwhile, Mahin et al. proposed a deep deep features and correlation coefficient.Their approach, tested on extensive datasets, outperformed previous methods, highlighting the potential of early detection via chest X-ray images[26].Murugesan and Muthurajkumar proposed a deep learning approach for product recommendation in social networks, achieving a 92.22% positive score out of 2033 reviews, surpassing traditional methods in accuracy and quality[27].COVID-19 is not only analyzed by machine learning approaches.Researchers explore the use of uncertain SEIAR system dynamics modeling for community health management, focusing on COVID-19.Their study employs Ensemble Kalman Filter and Metropolis-Hastings algorithms, offering insights into outbreak control scenarios and mortality rates[28].Ershadi et al. introduce a hierarchical machine learning model for analyzing treatment plans of Glioblastoma Multiforme patients, integrating clinical, biomedical, and image data to improve decisionmaking efficacy.They employ Fuzzy C-mean clustering, Wrapper feature selection, and twelve classifiers to optimize outcomes[29].Rahimi Rise et al. advocate for stronger environmental considerations post-COVID-19 to transition the global economy towards renewable energy and resilient public-health systems, emphasizing the need for institutional reforms within the United Nations System[30].Rise et al. propose a hierarchical model combining expert knowledge, FCM clustering, and ANFIS classification to detect severity levels of hospitalized symptomatic COVID-19 patients, achieving high accuracy using both clinical and image data[31].Rahimi Rise et al. analyze socioeconomic impacts of infectious diseases using an uncertain SEIAR model with scenario-based analysis, emphasizing future GDP and social impact predictions for policymaking

3.2. K-Nearest Neighbor Classifier
The k-nearest neighbor (k-NN) supervised classification technique is employed for sample categorization.It operates by categorizing new data based on their features and labeled training data, without the need to fit a model, making it memory-based.Utilizing Euclidean distance, it identifies the k training points nearest to a query point, u0.The new data point is assigned to a group based on the majority of its neighbors.The nearest neighbor classifier requires a dataset for accurate classification, with the training sample representing the existing dataset.Each training vector, utp, represents a point in the N-dimensional space,