DTS002TC Essentials of Big Data

Coursework 2

Submission deadline: 5 pm China Time (UTC+8 Beijing) on Sun. 26th May 2024

Percentage in final mark: 50%

Maximum score: 100 marks (100 individual marks)

Learning outcomes assessed:

E: Demonstrate the ability to write code to obtain numerical solutions to mathematical problems

F: Demonstrate the ability to display computational results in tabulated or graphical forms

Late policy: 5% of the total marks available for the assessment shall be deducted from the assessment mark for each working day after the submission date, up to a maximum of five working days.

Risks:

•    Please read the coursework instructions and requirements carefully. Not following these instructions and requirements may result in loss of marks.

•     Plagiarism results in award of ZERO mark.

•    The formal procedure for submitting coursework atXJTLU is strictly followed. Submission link on Learning Mall will be provided in due course. The submission timestamp on Learning Mall will be used to check late submission.

Overview

This coursework consists of part A and B. Part A consists of 3 questions and Part B consists of 4 questions. In this coursework, you are  expected to explore the given Citrus dataset by using Python. The dataset contains the measurements of four key features (leaf length, leaf width, fruit length, and fruit diameter) from four distinct citrus species (grapefruit, lemon, lime and orange). Please make sure you have installed required libraries before you start.

Data Introduction

You need to download the raw data set named “citrus.csv” from LMO. The data set includes 5 columns of numerical and text data that describe the characteristics (leaf length, leaf width, fruit length, and fruit diameter) and species of citrus.

Task 1 Data Processing and Analysis (40 marks)

In big data analysis, we will first need to load a large number of data. This involves importing essential libraries and insert the dataset into a format that Python can understand, called data frame. The resulting data frame can be stored in a variable named “citrus” . Answer the questions below to write your programming environment and perform analysis on the given data by using Python programming. You are required to write/demonstrate clearly and neatly the command/output in the given space.

Question 1. Initialization of Model: It is vital for us to define the library and load the dataset to Python environment at the initialization stage. After load the data, we can get information about the data frame.    (10m)

(a)  Write a command to import the required open source data analysis and manipulation library aspd. (2m)

(b)  Write a command to read the dataset and store the data frame as “citrus” .                                        (2m)

•     Show the first five rows (from index 0 to 4) of the citrus data frame.                                         (2m)

•     Show the last five rows (from index 195 to 199) of the citrus data frame.                                  (2m)

(c)   Show the data frame information, which shall consist of (i) its dimensions (number of entries and data columns), (ii) details (column, non-null count, and data type) of each feature (leaf length, leaf width, fruit length, fruit diameter, and species) in a table form, (iii) data types indicating four columns have “float64”, and a column has “object”, (iv) memory usage, and (v) end of the output “None” .                           (2m)

Question 2. Data Processing and Calculations: After successfully load the dataset, it is important to perform process the data and perform preliminary statistical analysis on the dataset.                                                               (14m)

(a)  Write a command to count the occurrences of each species of citrus and store it as “species_count”. Then, show the results of species names (grapefruit, lemon, lime, and orange) with theirrespective counts, including the name and data type.                                                             (4m)

(b)  Write a command to calculate the leaf areas in the data frame (assume the leaf shape is equivalent to circular, and the leaf length and width makeup the diameter of circular). During the computation, use math.pi, and store the area as “leaf_areas”. Then, show the results of values for the leaf areas from first 5 indexes and last 5 indexes, including the length and data type.     (4m)

(c)  Write commands to calculate the average fruit length for each species and store it as “average_fruit_length” .  During the process, data will be re-organized according to species and will be stored as “grouped_data”. Then, show the results of species names (grapefruit, lemon, lime, and orange) with their respective average fruit  length, including the name and data type.                                     (6m)

Question 3. Data Visualization: After extract preliminary statistical information about the dataset, we can describe the relationships, and distributions of different data variables to extract the underlying pattern through visualization methods. We will create informative visualizations using two relevant Python's powerful statistical data visualization library and python plotting libraries.                     (16m)

(a)  Write commands to import necessary Python data visualization libraries. Then, count each species in the Citrus dataset with parameters x=“species” and data =“citrus” and display the result in bar chart. (8m)

(b)  We are interested in knowing the distribution of the fruit diameter of the Citrus. Write commands to create the  histogram  of  fruit  diameter  and  display  the  results.  Then,  by  using  the  parameter   x=“species”, y=“fruit_length”, and data=“citrus”, write commands to create the boxplot of fruit length for all species (grapefruit, lemon, lime, and orange) and display the results.                           (8m)

Task 2 Data Classification (60 marks)

More advanced big data analysis can be used to support numerous computer vision tasks in the field of machine learning. For instance, classification is one of the most common computer vision tasks. Answer the questions below to develop your classification model by using Python programming. Data cleaning is not required since the given dataset has a balanced class and no missing value. You are required to write/demonstrate clearly and neatly the command/output in the given space.

Question  1.  Data  Preparation  for  Model  Learning: Conventional  machine  learning  models  thrive  on  well- structured and organized data. To ensure optimal performance, we will need to prepare the data carefully. This involve separating the feature matrix for the classifier from target variables. Converting the labels into numeric so they are in machine-readable form, and splitting the features data and target data into train and test cohort.                   (18m)

(a)  Define  a  features  matrix  (X)  that  contains  the  “leaf_length”,  “leaf_width”,  “fruit_length”,  and  “fruit _diameter” and a target variable (Y) that contains the label of each species in numerical format (grapefruit = 0, lemon = 1, lime = 2, and orange = 3) by using LabelEncoder.                                     (14m)

(b)  Split the data into training and testing sets using a 70:30 ratio. To do so, write commands to split X and Y into training and testing sets with parameters:

test_size represents a 70:30 ratio and random_state of zero shuffling, so the results do not fluctuate and store them as X_train, Y_train, X_test, and Y_test.                                             (4m)

Question 2. Logistic Regression Classifier Building: After prepare our data meticulously, it's time to construct a classification model capable of predicting Citrus species based on their measurements. First, we explore the process of creating and utilizing a logistic regression classifier. We will use ‘liblinear’ as the solver for optimization since the Citrus dataset is small in size.                (10m)

(a)   To start the classification, write commands to import and instantiate the logistic regression classifier, and then train the logistic regression classifier by using training sets.                    (6m)

(b)  Write  command  to predict the species using the feature matrix  (X) of the testing sets. Then, show the prediction results, consisting of a list of 60 integer values that can only be 0, 1, 2 and 3.                 (4m)

Question 3. Model Evaluation and Interpretation: After build-up the classification model, it's crucial to evaluate its performance and understand its strengths and weaknesses. We will use accuracy to measure the performance and confusion matrix to visualize the performance of the logistic regression classifier.                  (12m)

(a)  Write command to calculate the accuracy score of the logistic regression classifier using the testing sets and store it as “lr_accuracy” . Then, show the accuracy results in decimal value format.                       (4m)

(b)  Write commands to create a confusion matrix to visualize the classifier's performance on each species by using necessary libraries. The matrix shall compare the actual target values (Y_test) with those predicted (lr_Y_predict) by the classifier. Then, display the confusion matrix with correct results by using parameter annot=True.                      (8m)

Question  4.  Comparison  with  Decision  Trees:  In  the  Machine  Learning  domain,  there  are  diverse  array  of classification algorithms, each with its unique strengths and perspectives. It is common to explore different classifiers in performing the same task. During the exploration, we can compare their performance. Let's introduce a new classifier—decision trees—and see how it fares against logistic regression.                               (20m)

(a)  Write commands to import and instantiate the decision tree classifier. Store it as “dt” .                   (4m)

(b)  Write commands to train the decision tree classifier using the training sets and then predict the species by using testing sets. Store the prediction as “dt_Y_predict” and show the results, which shall consist of a list of 60 integer values that can only be 0, 1, 2 and 3.                                (6m)

(c)  Write commands to calculate the accuracy score of the decision tree classifier using the testing sets and store it as “dt_accuracy” . Then, show the accuracy results in decimal value format.                                  (4m)

(d)  Write commands to create a confusion matrix to visualize the decision tree classifier's performance on each species by using “Y_test” and “dt_Y_predict” and store it as “dt_cm” . Then, display the confusion matrix with correct results by using parameter annot=True.                                             (6m)