ECE5470 Prelim 3 Fall 2023
1. Classifier Evaluation 20 pts
A. (a) A two-class task has the following outcomes for a test set. Carefully sketch the Receiver Operating Characteristic (ROC) on the right.
Only in exams, or theory (or in printing too few digits) will two outcomes have the same probability (0.65), special conditions require special considerations.
(b) Calculate the AUC: 0.4 x 0.2 + 0.2 x 0.7 + 0.2 x 0.8 +0.2 X 1 = 0.57
(0.56 – 0.58)
(c) Based on the above information, would you consider this model to be
random, weak, good, strong, excellent, or outstanding? Weak
B. Consider a classifier with three class outputs A, B, and C.
What method would you use to visualize/evaluate classifier performance _Confusion Matrix
C. If the classes were A, B and ‘insufficient image quality’ would this change your analysis approach? If so, how?
The analysis may now be considered as two two-class problems. The first “is the quality good enough?” is independent of the second; (class A or class B?). (If the image is of low quality, then the second classifier is not used). Thus, rather than a confusion matrix this may be better considered as two two-class (e.g. ROC or PRC) analysis.
2. A Convolution. 20 pts.
a. Convolve the following matrix with the following kernel, centered at the upper-left position of the kernel.
b. Perform. a transposed convolution on the following matrix with the following kernel with a stride of 2.
B Machine Learning: True or False
a. T Stratified sampling helps alleviate class imbalance issue.
b. A In interval validation, the final model is trained on the set that gives the best results.
c. F ROC curves are used for multiclass-classifications tasks and Precision-Recall curves are used for binary-classification tasks.
d. T The purpose of the softmax activation function in the output layer of a neural network is to convert raw output scores into probabilities for classification.
e. F Random hyperparameter search is less efficient than grid searc f. A Binary cross-entropy loss can be used for a regression task
A = possibly ambiguous question
3. CNN classification model 20 pts.
A. Given the following examples for a convolution block and a linear layer, modify the code accordingly to so that has an additional convolution block with 5x5 convolution that maintains the input size, and three total linear layers that have shapes 1028, 514, and 64. The input is a color image with shape 28x28x3 and the number of output classes are 2.
class CustomNet(nn.Module):
def init (self, num_classes=2):
super(CustomNet, self). init ()
self.layer1 = nn.Sequential(
nn.Conv2d(3, 16, kernel_size=3, stride=1, padding=1),
nn.BatchNorm2d(16),
nn.ReLU(),
nn.MaxPool2d(kernel_size=2, stride=2))
self.layer2 = nn.Sequential(
nn.Conv2d(16, 16, kernel_size=5, stride=1, padding=2),
nn.BatchNorm2d(16),
nn.ReLU(),
self.fc2 = nn.Linear(14x14x16, 1028)
self.fc3 = nn.Linear(1028, 514)
self.fc4 = nn.Linear(514, 64)
self.fc1 = nn.Linear(64, num_classes)
def forward(self, x):
ut = self.layer1(x)
ut = self.layer2(out)
ut = out.reshape(out.size(0), -1)
ut = self.fc2(out)
ut = self.fc3(out)
ut = self.fc4(out)
ut = self.fc1(out) return out
B. Calculate the total number of weights in this network. (give the number for each layer in turn)
L1=(3x3x3+1)16 L2=(5x5x16 +1)16 fc2=(14x14x16 +1)1028 fc3=1029x514 fc4=515x64 fc1=65x2
C. What is an example loss function and optimizer you would use for this model?
Loss: Binary cross-entropy, Optimizer: Adam
4. Miscellaneous 20 pts
A. List 2 practical situations when accuracy is not a good evaluation metric.
i. class imbalance
ii. different misclassification costs
B. Floating Point
Given that FP32 is a traditional 32-bit floating point number with an 8-bit exponent and a signed 24-bit mantissa. What are the following formats:
TF32 8-bit exponent and 10-bit mantissa
BF16 8-bit exponent and 7-bit mantissa
Where are they primarily used? GPU
C. Binary Classification
(a) Give an equation for accuracy in terms of T, P, TP, FP, TN, etc.
(TP+TN)/(TP+FP+TN+FN)
(b) Give an equation for specificity in terms of T, P, TP, FP, TN, etc.
_TN/TN+FP
D. Name three different methods for reducing overfitting in a CNN system.
1. Early stopping
2. Regularization
3. Dropout
E. What is Balanced Cross Entropy (BCE) Loss and when is it used? (and can you define b?)
BCE is used for imbalanced dataset. ß is the weighting factor and is used to weigh loss of one class more heavily than the other.
5. Image Segmentation and object identification (20 pts.)
A. Indicate which one of the following statements that best matches the listed models. (a) Object classifications are made by grouping outputs into rectangular regions.
(b) A U-net is used to provide a segmented object region
(c) Outputs from several spatially arragned different classifiers are combined to idetify an object
(d) Scans an object classifier over an image
(e) First model to use a region proposal network
|
(i) Masked RCNN b
|
(iv) Region-based Fully Convolutional Network (R-FCN) c
|
|
(ii) Region-Based CNN (R-CNN) e
|
(iv) Faster Region-based CNN (Faster-RCNN) e
|
|
(iii)You Only Look Once (YOLO) a
|
|
B. A partial U-net design for 100 x 100 color images, is given below. Complete the design and list the parameters for the functional units that you add (i.e., function unit name and number of channels, also mark the bunmber of channels of intermediate data structures as shown) Assume all regular convolution layers are 3x3 with padding 1
Examples: A. conv. + BNorm + Relu (40) . B Copy
|
C.
|
D. . E
|
|
F. .
|
G. . H
|
|
I.
|
J
|