MA2608: MATHEMATICS OF DEEP LEARNING, 70% MAJOR PROJECT  (2023–24): MNIST

In this assignment you will have the opportunity to demonstrate your proiciency in:

. implementing and training a deep learning network for handwritten digit recognition . classifying digits according to whether or not they belong to your student ID

. detailing true and false positives as a function of a decision threshold

. producing a written report detailing your achievement

. producing a recorded audio-video presentation showcasing your work.

The requirements for this assignment fall into four main deliverables: CODE; PDF; MP4 and ROC, all as detailed below.  The grades awarded depend strongly on the quality of your sub- mission across these deliverables. You are strongly advised to base your submission on the code ann08demo.m and to absorb the relevant material from ann09demo.m.

From MATLAB Grader Assignment 7 on Brightspace use your student ID to obtain your person- alized data consisting of positive integers Nh, Nep, learning rate, u, vand w. This step carries no marks but it will be recorded: zero participation = zero marks.

This submission has three components: a MATLAB source ile (which must be suitable for direct loading into MATLAB — a template is provided on Brightspace); awritten report (in PDF); and an audio-visual presentation in an MP4 audio-video ile.  A discussion of how to use Teams for this is footnoted. The brief and requirements for each component are now given, with the Grade Descriptors following.

. . . CODE: Obtain  the skeleton source code  annIDtemplate.m and the training and testing datasets MNIST_train_1000.csv and MNIST_test_100.csv from the Brightspace assign- ment page.  Rename annIDtemplate.m to ann#######.m with the hashes replaced by your Brunel student ID. This is the MATLAB ile you should populate with your responses to this assignment and then submit.

In what follows you should use the sigmoid activation function on all layers, and total squared error (TSE) as the performance index.

1. Alter ann08demo.m or ann09demo.m so that it trains and tests using the data sets referred to above and produces a single scalar output.  The y_train and y_test arrays should contain 1 (for Positive, P) if the corresponding input digit is in your student ID, and 0 (for Negative, N) if not. There should be Nh neurons (or nodes) on the hidden layer and you should use Nep epochs. This entire code should be in the function ann1HL in the annID.mile.

2. Evaluate the trained network’s predictions on the test data by specifying a decision threshold, t ∈ (0,1), such that an output y is considered positive (P), the input belongs to your ID, if y > t, and negative (N) otherwise. The user should be able to choose between heuristic unscaled back propagation and calculus-based back propagation (e.g. by setting the variable bp).

3. Enhance this code by increasing the number of hidden layers to three with theirst layer con- taining u neurons, the second containing v neurons and the third containing w.  Place this new code in the function ann3HL in the annID.mile. This code should also implement the decision threshold and have the backprop choice enabled.

. . . PDF: The written report should consist of a cover page plus no more than ive additional A4 sides, be written in English with in a font size of no less than 10 point, and have margins of no less than 2cm all around.  The report may be handwritten or typed but it should be well- structured and legible. Alligures must be included in the body of this report.

1. Give complete mathematical descriptions of your three and ive layer neural networks.  This should not be a code listing, but should use proper mathematical notation. You should detail the back propagation methods used — but you do not need to derive the formulae.

2. Give tables showing the True Positive Rate, TPR  =  TP/P, and False Positive Rate FPR  = FP/N, for each of the thresholds t ∈ {0.25,0.5,0.75}. This should be done for heuristic unscaled backprop as well as calculus-based backprop for both the three andive layer networks.

3. Give the plots that you generated in the ROC task. These should be well annotated and cap- tioned with an appropriate legend.

4. Comment on this use of TPR and FPR as a means to determine the quality of the classiier. Comment on the concept of ‘AUC’, and give the values.

. . . ROC (for Receiver Operating Characteristic): For each choice of backprop, and each network (3 or 5 layer) plot, on the same axes, the TPR vertically against the FPR horizontally for the deci- sion threshold varying as t ∈ (0,1) — you can use a discrete range and interpolate with straight lines.  Each of these four curves is called a ROC curve.  In each case calculate (using a suitable quadrature rule) the Area Under the Curve, or AUC.

. . . MP4: The audio-visual presentation should be no longer than 5 minutes (be accurate and concise). The presentation should contain the following.

1. An introduction in which you appear live and which contains a clear picture of your student ID card. This is to verify authenticity of authorship and is not optional.

2. A screen share showing your report and code (as submitted). You should go through your re- port and cross reference it to your MATLAB ile. You should explain where and how the algorith- mic and mathematical details are implemented in the code. You must cover forward propagation, back propagation, gradient descent and performance index (TSE).

3. A screen share where you show and discuss how you obtain the TPR and FPR results.  You should also explain where and how you calculated the AUC.

Grade Descriptors: to achieve agrade in the range of an . . .

E (30-39%):  you will have submitted a MATLAB code (following the supplied template) which has content clearly evidencing an attempt to meet requirement 1 in CODE.

D (40-49%): you will have achieved at E grade level, with that version of your code being substantially correct, and also having content clearly evidencing an attempt to meet requirement 2 in CODE. In addition you will have submitted a written report which has content clearly evidencing an attempt to meet requirement 1 (for three layers) in PDF.

C (50-59%): you will have achieved at D grade level, with that version of your report being substantially correct and complete, and with content clearly evidencing an attempt to meet requirements  1 and 2 in PDF in full.  Your code will run and have content clearly evidencing an attempt to meet requirement 3 in CODE. In addition you will have submitted a video presentation which has content clearly evidencing an attempt to meet requirements  1 and 2 in MP4.

B (60-69%): you will have achieved at C grade level and that version of your code should run, be substantially correct, with only minor errors, and meet all requirements in CODE. Your report will be complete, with only minor omissions and/or typos, well structured, with good grammar and layout.  Your presentation, while not perfect, will substantially address both of requirements 1 and 2 in MP4.

A (70-89%): you will have achieved at B grade level. Your code, report and presentation will have substantially and cogently met all requirements, with speciic grading depending particularly on your response to those requirements in and relating to ROC. Conceptual, implementation or reporting errors should be of very high quality in terms of content and form, containing no more than minor errors which in noway detract from the overall impression of comprehension and achievement.

A* (90-100%): Every aspect of your submission will be complete, correct and of near or full professional quality.  It will meet all requirements in full.