BUSAN 300: Data Wrangling

Semester 1, 2025

Wrangling Project Specifications

1.    Purpose

The purpose of the project is for you to achieve all learning outcomes of this course. The process of working through it is more valuable than the mark received.

In attempting and completing this project correctly, you will:

•    experience sourcing real, raw data

•    refine your data wrangling skills

•    encounter real data problems and employ the tools/techniques learned in this class and beyond to solve them

•    improve your data-based problem-solving skills

•   be patient when completing your work

•    explore data in the context of important global issues such as sustainability and equity in society

1.1  Project Theme

The overarching theme for the project is Supporting a Sustainable and Equitable Future through Data- Driven Citizenship with a focus on sustainability and just, ethical and equitable societies. This involves understanding how data can inform. practices and policies for a better future, and you are encouraged to explore datasets that reveal societal dynamics, biases, and inequalities. For example, projects under this theme might investigate how biases in data can lead to unintended consequences for certain minority groups and/or trends in sustainability.

However, you may also choose to propose any theme and topic that you might find compelling to work on. We will review this during the project proposal phase described below.

2.    Guidelines

2.1  Submission

Project Proposal: Submit a single document in PDF format containing your project proposal to Canvas and have it approved by the stated deadline. Attach links to data files (e.g. Google Drive) when needed. Follow the Project Proposal Template.

Project Report: Submit a single document in PDF format containing all the contents of your project report to Canvas by the stated deadline (23:59 NZDT, Sunday, 15th  of June 2025). Attach links to data files (e.g. Google Drive) when needed. Follow the Project Report requirements stated in Section 3.

2.2  Weighting

The Project Report submission is worth 15% of your final grade.

2.3  Academic honesty and integrity

This project is an individual assessment. You must complete all the work yourself. Do not submit work that you did not produce. Do not work in a way which could result in parties producing the same or very similar work.

In attempting this assignment you agree to adhere to all the principles and practices of academic honesty and integrity for the University of Auckland outlined here:https://www.auckland.ac.nz/en/students/forms-policies- and-guidelines/student-policies-and-guidelines/academic-integrity-copyright.html.   Any   form.    of   cheating, plagiarism, assistance in cheating, unfair collaboration, or other behaviour deemed to be academic misconduct will not be tolerated. Academic misconduct will be dealt with according to the University’sStudent Academic Conduct Statute.

3.    Tasks

This wrangling project is an exploration of public data with an intention to discover insights of interest to New Zealand or a global audience. The project is split into two parts: project proposal and project report. You should follow the steps described in the following sections in the order that they are listed.

3.1  Project Proposal

Have your project proposal approved by the stated deadline.

Each student’s project should be sufficiently interesting (i.e. worth doing), doable, and be of similar complexity to other projects (to ensure fairness in marking and learning experience). To ensure your project meets this standard, please spend sufficient time to read the project requirements, explore potential datasets, and plan your project.

Summarise your plan in a project proposal document which briefly states:

1.    The two datasets you will use in the project:

o At least one dataset must be from the list provided under the Project Datasets page in Canvas, which have been specifically chosen to include the themes of ethics, equity, sustainability, and responsible citizenship. Use these datasets to form the basis for your project. We ask you to use at least one provided dataset to simulate a real working environment, where you may be tasked with analysing data on a particular initial topic and must then find additional resources to support your findings.

o Explain how the two datasets interrelate and support the exploration of your chosen theme. Your second dataset does not have to directly relate to the given themes and exploration outside of these themes is fine, provided you can demonstrate how your chosen datasets align with the principles of data-driven citizenship and contribute to a sustainable, ethical and equitable future.

o Clearly state the sources: what are the data, their file types and where do they come from?

o Provide links to the source of the files.

2.    How you plan to combine the two datasets:

o Consider what attribute(s) they have in common.

o Consider what technique(s) you plan to use.

3.    The intended final format of your combined dataset (e.g. Excel, MongoDB, etc.)

4.    A backup plan for completing this project

o Consider what you could do to avoid forfeiting all 15% of your final grade due to poor forethought or underestimation of the requirements of this project etc.

o You should enact this plan if you are unable to successfully combine your two datasets by Week 11.

Copy and use the template provided in the Project Proposal Template page in Canvas within a text editor and write no more than 300 words. Submit your proposal as a PDF file to the “Project Proposal” Canvas assignment and ensure it is approved by the stated deadline.

You will receive feedback on your proposal. If your proposal is approved (marked as “complete” on Canvas) you should proceed with your proposed datasets. If your proposal is rejected (marked as “incomplete” on Canvas) you should meet with the teaching staff to discuss the reason for rejection, then revise andresubmit. You are encouraged to repeat this process until your proposal is approved, and hence it is recommended that you submit your initial proposal as early as possible.

There are no marks for completing this task, but your proposal must be approved foryour project report tobe accepted and marked.

3.2  Project Report: Source and Audit Two (2) Datasets

Source and audit two disparate datasets from separate sources. You will be required to combine the datasets into a single dataset, so ensure the chosen datasets are suitably related.

Remember that the datasets will be used to attempt to discover insights of interest to the New Zealand or the global public.

For learning purposes, your datasets must be:

•    sufficiently different – in format/file type from one another, and

•    sufficiently complex – complexity can arise from

o large data size (e.g. tens of thousands of rows/instances, or tens of columns)

o non-uniform data structures (e.g. the data is an amalgamation from multiple sources, or different attributes exist for different instances)

o dirty data

Summarise your sourcing and auditing activities in the report. See the requirements in the “Wrangling Details” section below.

3.3  Project Report: Pose Questions

Pose three (3) meaningful questions that could only be answered if the two datasets were combined. These questions should  be  impossible  to  answer  if the  datasets  were  not  combined.  You  can  assume  any  problem  or situation/scenario under which these questions are posed. You will be required to attempt to answer the questions in your report, whether you obtain correct answers is not important as long as your attempt is sensible.

Document your questions in the report. See the requirements in the “Project Summary” section and “Questions and Answers” section below.

3.4  Project Report: Combine Datasets to be Stored in a Single Format

Wrangle your two datasets to clean and combine them into a single dataset. Store the combined dataset in a single data store (i.e. its “final storage format”). Data stores could be a file (e.g. Excel workbook) or a database (e.g. Microsoft SQL Server, Microsoft Access, SQLite, MongoDB). You must not use any file converter or automation tool to transform. data.

If your datasets are too large for the tools used in this course to handle, you may use a sufficiently large subset of the data to build a proof-of-concept that the data can in fact be combined.

Document your wrangling processes in the report. See the requirements in the “Wrangling Details” section below.

3.5  Project Report: Answer Questions

Using your cleaned and combined dataset in its “final storage format”, attempt to answer the three questions you posed in any way you can. Answering is likely to involve the use of one or many of the following: PivotTables, XPath, MongoDB queries, visualisations, SQL queries etc.

You are advised to keep your exploration simple. Bear in mind that the learning outcomes ofthis assessment relate to data wrangling process and technique but not statistical analysis or data mining.

Document your answers in the report. See the requirements in the “Project Summary” section and “Questions and Answers” section below.

3.6  Project Report: Write Report

Document the previous tasks in one report. Lay out your report in three sections in this order:

1.   Project Summary

2.   Wrangling Details

3.    Questions and Answers

The requirements of each section are as follows.

3.7  Project Summary

Write this section for a business audience (e.g. a senior business decision maker). Summarise your entire project. Include the three questions you posed, each with a short summary and discussion of your answer/conclusion for it. Provide some insight about your findings.

3.8  Wrangling Details

Write this section for a “data expert” audience (e.g. a classmate).

Detail the following wrangling processes in your report, specifically:

For each dataset

•    Its origin. Provide a direct link to the data, or if that is not possible, explain how you obtained the data.

•    Its general characteristics. What format is it in; what is the structure; how many columns/fields; how many records? etc.

•    An initial audit. What observations did you make about the data? Are there any obvious or potential problems that may have to be dealt with before combining it with the other dataset?

Steps you performed to combine the datasets

•    The level of detail should be so that any of your classmates can replicate the process.

o Stating what you did is more important than stating how you did it. For example, rather than explaining what menu items you clicked on, it is sufficient to state that you “removed duplicates on the user id and timestamp fields”.

o Include relevant screenshots of intermediate steps etc.

•    State all transformations performed on each dataset individually before they were combined.

•    State all transformations performed to combine the datasets.

•    State the steps performed to store the combined dataset in its “final storage format”.

Provide a link to your final combined data store so that it can be inspected. To do this, please upload your combined data store to a platform. such as Google Drive and set the viewing permissions to “Anyone with this link can view”. Please provide the link to your data store clearly in your project report. Marks will be deducted if the final data store is not provided or is inaccessible.

3.8.2      Questions and Answers

Write this section for a “data expert” audience (e.g. a classmate).

Detail how you used your combined dataset, in its “final storage format”, to reach the answers/conclusions to your questions.

Specifically, for each question:

•    State the question and your answer/conclusion.

•    State the steps performed to produce the answer. Include what tools/software you used and what queries you executed etc.

o The goal here is not so that the process can be replicated, but to show your answer/conclusion is sound or otherwise reasonable.

o Even if no exact answer was found, the steps to reach that result still need tobe documented. You should also state what would be needed to reach an answer.

o Include relevant screenshots of intermediate steps etc.

Submit the report as a single PDF file to the “Project Report” Canvas assignment by the stated deadline.

4.    Marking Guide

The experience of completing the project is intended to be more important than the results you produce.

No template for the report will be provided to not limit anyone’s style. and creativity. The content and structure you are required to provide are explicitly given in Section 3.

The table below summarises how your project will generally be marked. Where tasks are completed exceptionally well (demonstrated by its execution or documentation or both), marks for those tasks can compensate for tasks which are not completed as well.