161.777 Practical Data Mining

Project 2025

General information

This project is assessed. Your work must be submitted by 11:59pm on 9th June 2025.

There are questions on prediction and classification, as well as clustering and association rules.

For the prediction and classification problems you should provide both the writeup of your methodology in the space provided in the project.Rmd file, as well as a .csv file for each exercise. In each case these should be the provided test set, with an additional column containing your predictions or classifications.

Make sure that your predictions or classifications make sense. e.g. that your .csv file has the correct number of rows, and there is a clear column with your predictions.

You may include your code to do the prediction and classification in the project.Rmd provided if you wish. Alternatively you may do your modelling work in a separate R script. or markdown file. This may save re-running code whenever you Knit your project.Rmd as you writeup your methodology and answer the exercises on clustering and association rules.

Start by downloading the project file by Right clicking and Save File As… here:

https://www.massey.ac.nz/~jcmarsha/161777/assessment/project.Rmd

Then loading it into RStudio.

Exercise 1: Predicting Arctic Ice Thickness [25 marks]

This exercise is concerned with measurements of arctic ice thickness collected by scientific vessels in the Arctic Ocean. Each record corresponds to data collected over a single track (voyage between two points).

The aim is to predict the target variable Thickness using information from the other 9 variables (the predictors). The training dataset contains 10,000 records (on all variables) while the test dataset contains 1,085 records (on all variables except for Thickness).

You are to produce predictions for the Thickness variable on the 1,085 records in the test data, adding it as a column to the ice.test data frame. and writing to a .csv file. You can do this with

write_csv(ice.test.with.predictions, "ex1_studentid.csv")

replacing studentid with your student id code.

Please make sure that the column with your predictions is named .pred. This is the default if you use tidymodels.

Make sure you attach your ex1_studentid.csv file to your stream submission.

You will be scored based on root mean squared error on the test set, worth a total of 15 marks. A score of 0 will correspond to utilising the mean of the target on the training set as your predictions.

You should also writeup your methodology for answering this question in the section provided in the project.Rmd file. This writeup should include:

· What, if any, data processing you performed.

· What exploratory analysis you undertook.

· Which modelling techniques you tried, including information on predictors chosen and any tuning performed.

· EITHER: A statement that you did not use information from outside of the course, OR: A statement detailing what information from outside the course was used, where it was from, and how you learnt from that.

· A clear statement of which model you chose, and why you chose it.

Your methodology will be marked out of 10 marks.

Exercise 2: Classifying Campylobacter [25 marks]

This exercise is concerned with some Campylobacter isolates that were collected as part of the Source Assigned Case Control Study in New Zealand (SACNZ) project.

There are 400 isolates of Campylobacter isolated from Beef, Sheep or Poultry sources (from faeces or meat). Each isolate was whole-genome sequenced with 1343 ‘core’ genes identified. For each isolate and each gene, sequences were scored based on sequence similarity, projecting each gene into two dimensional space, and a subset of 25 genes were selected.

The training data thus consists of 50 numeric predictors named as CAMP0003.V1, representing the first score of the CAMP0003 gene, and the target Source, which takes the values Beef, Sheep or Poultry for these 400 isolates.

In addition, we have a test dataset with 160 additional isolates where the Source is unknown.

Based on previous work, we would expect that Poultry isolates are likely to differ compared to Beef and Sheep, but distinguishing Beef and Sheep may be difficult.

You are to produce classifications for the Source variable on the 160 records in the test data, adding it as a column to the campy.test data frame. and writing to a .csv file. You can do this with

write_csv(campy.test.with.classifications, "ex2_studentid.csv")

replacing studentid with your student id code.

Please make sure your classifications are named .pred_class. This is the default if you use tidymodels.

Make sure you attach your ex2_studentid.csv file to your stream submission.

You will be scored based on classification rate, worth a total of 15 marks. A score of 0 will correspond to picking sources at random.

You should also writeup your methodology for answering this question in the section provided in the project.Rmd file. This writeup should include:

· What, if any, data processing you performed.

· What exploratory analysis you undertook.

· Which modelling techniques you tried, including information on predictors chosen and any tuning performed.

· EITHER: A statement that you did not use information from outside of the course, OR: A statement detailing what information from outside the course was used, where it was from, and how you learnt from that.

· A clear statement of which model you chose, and why you chose it.

Your methodology will be marked out of 10 marks.

Exercise 3: Clustering the New Zealand World Values Survey [32 marks]

We consider some data from the New Zealand version of the 2014 World Values Survey. The variables we consider are:

All other answers to these questions are coded as missing, NA.

1. 

Do an initial exploration of the dataset:

2. 

o 

Showing plots and summary tables to examining the distributions of values of each variable, commenting on the distributions. 6 marks

o 

o 

Some sorts of people may be more/less willing to do a Values survey. Comment on how this may bias the survey. 2 marks

o 

o 

Which three variables have the most missing values? Suggest possible reasons for these. 2 marks

o 

3. 

Prepare a new version of the dataset with all variables normalised. Comment on why these this step is appropriate prior to a cluster analysis for these data. 3 marks

4. 

5. 

Produce a dendrogram of the sample, selecting the variables Age, Female, MaritalStatus, StateofHealth, HighestEducation and LifeSatisfaction, after normalisation. Remove all missing values. Use hierarchical clustering with complete linkage. What do you think is the best number of clusters to divide the sample into. 4 marks

6. 

7. 

Perform. a kk-means cluster analysis of the new reduced, normalised dataset, based on Euclidean distances. Choose an appropriate value of kk within 1,2,…,10, and explain why you chose it. Include graphs of tot.withinSS and average silhouette width and silhouettes. Note there is no one perfect answer. 6 marks

8. 

9. 

Regardless of your answer to part 3, show a fviz_cluster graph, based on 4 clusters and complete hierarchical linkage. Does the graph show sufficient separation between the clusters? Explain. 3 marks

10. 

11. 

Assume four clusters, and use kmeans. Explore the differences among the clusters using boxplots. For each cluster, write a sentence or two characterizing the cluster in terms of the variables. 6 marks

12. 

Exercise 4: Association Rules [18 marks]

A vexillologist is a person who studies flags. In this exercise we will consider the colours of 211 flags of different countries. Certain colours seem to go together.

1. 

Make a plot showing how frequently the various colours have been used. What are the top three most popular colours? 3 marks

2. 

3. 

Consider only rules where there are at least 5 countries have used the same colour combination (i.e. those with support parameter 0.023) and with the confidence cutoff set to confidence = 0.5. How many rules are found? 2 marks

4. 

5. 

Find the 5 rules with the highest support. Take the third one in this list and (i) explain the rule itself (i.e. what ‘itemset => itemset’ represents), and (ii) interpret the values of support, confidence, and lift for this rule. 6 marks

6. 

7. 

Find the rules for which lift is greater than 2. How many are there? Explain what these rules tell you. 3 marks

8. 

9. 

Suppose the island of Choiseul declares its independence. The leaders solicit ideas for a flag. Their main stipulation is that it must have black as a prominent colour. Answer the following, making sure you justify your answers from the data:

10. 

· 

If there is going to be at least one other colour, then on the basis of the data, which colour do you think is most likely to be chosen?

· 

· 

If the leaders were tending towards a four-colour flag including black, blue and red, what would be the most likely extra colour? 4 marks

·