BIO215 Session 4 - Practicals: Machine Learning Part 2
1 Practical 4: Introduction
In this practical, you will build a random forest machine learning classification model based on an unbalanced m6A dataset.
You will train the random forest classifier using a full set of genomic features, and compute comprehensive evaluation metrics using the trained classifier on the testing data.
Afterward, you will interpret the machine learning classifier using two established techniques: feature importance and SHAP values.
Note that this practical is assessed, meaning you need to submit the completed practical as an HTML file knitted from the practical RMD file. The submission will account for 10% of the final score in this module. To knit the practical into an HTML file, you need to click the "Knit" button in the RStudio API and select "Knit to HTML". Once the rendering process is complete (and without code errors), an HTML file will be successfully generated.
To download a knitted HTML report to your computer, click "Files" in the right panel. Navigate the folders, select the generated HTML file, and then click the little gear icon > Export...>Download. This will trigger the download of the HTML file. Before uploading the downloaded HTML file to the LMO submission box, ensure that you double-check the content of the file. The file size should be between 1.6 MB and 2 MB. If your file is too small, it indicates an issue with the export / download process, and the file may be corrupted.
The detailed mark weight for each section is provided at the topic of that section in today's practical. Your assessment will be based on the correctness of your code and outputs. If a short-answer question (SAQ) is included in a section, part of the marks will also be awarded based on your written response under Your answer: .
2 Task 1: Build random forest classifier for unballanced m6A data (30%)
2.1 Data loading and pre-processing (5%)
1. Load the CSV file named "m6A_unbalanced.csv" into R and store it in the variable data_df.
2. Use the same procedure as in the last practical to one-hot encode the 5-mer sequence feature by creating 5 additional categorical variables named nt_pos1, nt_pos2,., nt_pos5.You may used cbind() to combine these as additional columns in data_df.
3. Convert the classification target variable m6A_status to a factor using factor(),ensuring that the negative class is the reference level (the first element in the level attribute of the factor).
4. Check the ratio of negative to positive using table().
#Enter your code here
##
## Negative Positive
## 36807 11248
Question: What is the positive to negative ratio of this data set?
Your answer:
2.2 Fit a random forest classification model (10%)
1. Next, randomly split the data into 80% training and 20% testing, setting the seed to 123 to ensure reproducibility. Store the 80% training data as train_df and the 20% testing data as test_df.
2. Using therandomForest package, fit a random forest classification model with m6A_status as the target and gc_content, RNA_type, RNA_region, exon_length, distance_to_junction, evolutionary_conservation, nt_pos1, nt_pos2, nt_pos4,and nt_pos5 as input features. Store the model fit as _rf_fit.
3. Check the content of rf_fit.
#Enter your code here
## randomForest 4.7-1.2
## Type rfNews() to see new features/changes/bug fixes.
##
## Call:
## randomForest(formula = m6A_status-gc_content + RNA_type + RNA_region + exon_length + distance_to_junction + evolutionary conservation + nt_posl + nt_pos2 + nt_ pos4 + nt_pos5, data= train_df)
## Type of random forest: classification
## Number of trees: 500
## No. of variables tried at each split: 3
##
## 00B estimate of error rate: 16.16%
## Confusion matrix:
## Negative Positive class.error
## Negative 27376 2093 0.07102379
## Positive 4120 4855 0.45905292
2.3 Compute accuracy, precision, recall, and FPR (15%)
1. Using the fitted random forest model, predict the class probabilities on the testing data in test_df. Store the class probabilities in the variable prob_test . Hint: You should check ?predict.randomForest to understand how to retrieve class probabilities rather than binary outcomes.
2. Use head() to check the content of prob_test.
#Enter your code here
## Negative Positive
## 1 0.146 0.854
## 9 0.874 0.126
## 20 0.470 0.530
## 23 0.198 0.802
## 27 0.556 0.444
## 35 0.178 0.822
3. Using the predicted probabilities and the ground truth class label ind test df, calculate a confusion matrix. The positive class prediction should be decided using a cutoff of positive class probabilities greater than 0.5. Save the confusion matrix in a variable named cm, and display the content of cm.
#Enter your code here
## predicted_cut05
## ground_truth Negative Positive
## Negative 6813 525
## Positive 1048 1225
4. Using the same confusion matrix as cm, compute the accuracy, precision, recall, and FPR. Display their values together using cbind(), or show them one by one.
#Enter your code here
## accuracy precision recall FPR
## [1,] 0.8363334 0.7 0.5389353 0.07154538
5. Compute the same set of performance evaluation metrics accuracy, precision, recall, and FPR-now using predictions with a probability cutoff of 0.1. Display all the values of the metrics.
#Enter your code here
## accuracy precision recall FPR
## [1,] 0.6854646 0.4256248 0.9441267 0.3946579