QTM 347 HW1 (2599822, 2539356, 2453794, 2396043)

February 25, 2024

[ ]:

import numpy as np

import pandas as pd

from ISLP import load_data

from sklearn.model_selection import train_test_split

[ ]:

import warnings

warnings.filterwarnings('ignore')

Promblem 1

(a)When the sample size n is extremely large and the number of predictors p is small, the perfor-mance of a flexible statistical learning method is expected to be better than an inflexible method. With a large sample size, a flexible method can better capture the underlying patterns between predictors and dependent variables, leading to improved predictive accuracy. However, a inflexible method may have the risk of overfitting.

(b)When the number of predictors p is extremely large and the number of observations n is small, the performance of a inflexible statistical learning method is expected to be better than an flexible method. With a small sample size relative to the number of predictors, flexible methods may largely be influenced by noise in the data, confusing it with true underlying patterns. Inflexible methods, on the other hand, are less susceptible to overfitting and may provide more stable and reliable predictions.

(c) In the case where the relationship between the predictors and response is highly non-linear, we would generally expect the performance of a flexible statistical learning method to be better than an inflexible method. Flexible methods have the ability to capture complex non-linear relationships in the data. They can adapt their complexity to fit the intricacies of the non-linear relationship, resulting in improved predictive accuracy. In contrast, inflexible methods may struggle to capture the nuances of the non-linear relationship and may result in poorer performance.

Promblem 2

a) lstat = 25 - When k = 1, we find the nearest point to be (24,39, 8.3), medv = 8.3 - When k = 5, we find the 5 nearest points to be (23.6, 11.3), (24.16, 14), (24.39, 8.3), (25.68, 9.7), (26.4, 17.2), medv = (11.3+14+8.3+9.7+17.2)/5 = 12.1

(b) lstat = 27

• When k = 1, we find the nearest point to be (26.82, 13.4), medv = 13.4

• When k = 5, we find the 5 nearest points to be (24.39, 8.3), (25.68, 9.7), (26.4, 17.2), (26.82, 13.4), (29.68, 8.1), medv = (8.3+9.7+17.2+13.4+8.1)/5 = 11.34

(c) lstat = 25

• When k = 1, we find the nearest point to be (24.39, 0), medv_cat = 0

• When k = 5, we find the 5 nearest points to be (23.6, 1), (24.16, 2), (24.39, 0), (25.68, 0), (26.4, 2), medv_cat = (1+2+0+0+2)/5 = 1

(d) lstat = 27

• When k = 1, we find the nearest point to be (26.82, 1), medv_cat = 1

• When k = 5, we find the 5 nearest points to be (24.39, 0), (25.68, 0), (26.4, 2), (26.82, 1), (29.68, 0), medv_cat = (0+0+2+1+0)/5 = 0.6

(e)

• In KNN, the model is less felxible with larger k. With a small k, the model relies on a smaller number of neighbors to make predictions, and so it can capture local patterns in the data. However, as k increases, the model becomes less flexible because it considers a larger number of neighbors, leading to a smoother function and a more generalized model.

• Bias: As k increases, the bias increases. With a larger k, the model averages over more data points to make predictions. This results in a smoother function that may not capture the underlying true relationship between the predictors and the response as accurately, leading to higher bias.

• Variance: As k increases, the variance decreases. With a larger k, the predictions are based on more data points, resulting in a more stable and robust model that is less sensitive to variations in the training data. This leads to lower variance because the predictions are less affected by small changes in the training data.

• Training MSE: As k increases, the training MSE increase. With a larger k, the model becomes less flexible and the predictions tend to be further away from the true values, leading to higher errors in the training data.

• Test MSE: As k increase, the test MSE initially decreases, reaches a minimum, and then starts to increase again. As k increases, the bias increases while the variance decreases. Initially, the decrease in variance dominates, leading to lower test MSE. However, as k continues to increase, the increase in bias starts to dominate, leading to higher test MSE.

Promblem 3

(a)

• The number of observations n is 100, and the number of features is 3 (different powers of x correspond to different features). The model is: y = x^5 - 2x^4 + x^3 (y = ff1x^5 - ff2x^4 + ff3x^3 + ff)

(b)

[ ]:

def f(x):

return x ** 5 - 2 * x ** 4 + x ** 3

def get_sim_data(f, sample_size=100, std=0.01):

x = np.random.uniform(0, 1, sample_size)

y = f(x) + np.random.normal(0, std, sample_size)

df = pd.DataFrame({"x": x, "y": y})

return df

(c)

[ ]:

from sklearn.preprocessing import PolynomialFeatures

from sklearn.linear_model import LinearRegression

import matplotlib.pyplot as plt

[ ]:

sim_data = get_sim_data(f)

degrees = np.arange(16)

for degree in degrees:

polynomial_features = PolynomialFeatures(degree=degree)

X_poly = polynomial_features.fit_transform(sim_data[['x']])

model = LinearRegression()

model.fit(X_poly, sim_data['y'])

[ ]:

x_fixed = np.array([0.18]).reshape(-1,1)

predictions = pd.DataFrame(columns=[f'Degree {i}' for i in range(16)],␣

↪index=[0])

sim_data = get_sim_data(f, sample_size=100, std=0.01)

X_train = np.array(sim_data['x']).reshape(-1, 1)

y_train = np.array(sim_data['y']).reshape(-1, 1)

for degree in range(16):

polynomial_features = PolynomialFeatures(degree=degree)

X_poly = polynomial_features.fit_transform(X_train)

model = LinearRegression().fit(X_poly, y_train)

x_fixed_poly = polynomial_features.transform(x_fixed)

predictions.loc[0, f'Degree {degree}'] = model.predict(x_fixed_poly).item()

predictions

[ ]:

Degree 0 Degree 1 Degree 2 Degree 3 Degree 4 Degree 5 Degree 6 \

0 0.016151 0.009605 0.006157 0.003526 0.00013 0.000486 0.000333

Degree 7 Degree 8 Degree 9 Degree 10 Degree 11 Degree 12 Degree 13 \

0 -0.000021 -0.000999 -0.001182 -0.000977 -0.002414 -0.001029 -0.001101

Degree 14 Degree 15

0 -0.001405 0.000981

(d)

[ ]:

predictions = pd.DataFrame(columns=[f'Degree {i}' for i in range(16)],␣

↪index=range(250))

for i in range(250):

sim_data = get_sim_data(f, sample_size=100, std=0.01)

X_train = np.array(sim_data['x']).reshape(-1, 1)

y_train = np.array(sim_data['y']).reshape(-1, 1)

for degree in range(16):

polynomial_features = PolynomialFeatures(degree=degree)

X_poly = polynomial_features.fit_transform(X_train)

model = LinearRegression().fit(X_poly, y_train)

x_fixed_poly = polynomial_features.transform(x_fixed)

predictions.loc[i, f'Degree {degree}'] = model.predict(x_fixed_poly).

↪item()

predictions.head()

[ ]:

Degree 0 Degree 1 Degree 2 Degree 3 Degree 4 Degree 5 Degree 6 \

0 0.017343 0.015916 0.010586 0.007302 0.003962 0.004943 0.004888

1 0.015583 0.010303 0.00762 0.005239 0.00154 0.001006 0.000951

2 0.016678 0.014221 0.0091 0.007389 0.003267 0.00375 0.003649

3 0.013869 0.010774 0.008704 0.007213 0.004782 0.005209 0.005071

4 0.011751 0.008241 0.007625 0.00356 0.000319 0.000681 0.000473

Degree 7 Degree 8 Degree 9 Degree 10 Degree 11 Degree 12 Degree 13 \

0 0.004635 0.005485 0.005357 0.004927 0.00464 0.005533 0.005648

1 0.000508 0.000307 0.000626 0.000169 0.00103 -0.001761 -0.001532

2 0.004055 0.003242 0.003486 0.003251 0.003898 0.004619 0.00462

3 0.005546 0.005243 0.00505 0.005669 0.006693 0.006152 0.006053

4 0.002626 0.003598 0.00365 0.00316 0.006308 0.005549 0.005766

Degree 14 Degree 15

0 0.005964 0.005617

1 -0.001252 -0.002004

2 0.004486 0.005213

3 0.005498 0.004036

4 0.006062 0.007607

(e) The Square of Bias

[ ]:

mean_predictions = predictions.mean()

true_value = f(0.18)

square_of_bias = (mean_predictions - true_value) ** 2

print("The Square of Bias")

square_of_bias

The Square of Bias

[ ]:

Degree 0 1.594870e-04

Degree 1 6.967211e-05

Degree 2 2.812878e-05

Degree 3 4.343497e-06

Degree 4 1.049161e-06

Degree 5 2.969633e-08

Degree 6 2.674706e-08

Degree 7 2.533310e-08

Degree 8 1.673926e-08

Degree 9 1.127903e-08

Degree 10 2.466444e-08

Degree 11 2.047126e-08

Degree 12 3.051183e-08

Degree 13 2.633141e-08

Degree 14 2.652020e-08

Degree 15 2.428047e-09

dtype: float64

(f) Variance

[ ]:

variance_of_predictions = predictions.var()

print("Variance")

variance_of_predictions

Variance

[ ]:

Degree 0 0.000003

Degree 1 0.000005

Degree 2 0.000003

Degree 3 0.000003

Degree 4 0.000004

Degree 5 0.000005

Degree 6 0.000006

Degree 7 0.000006

Degree 8 0.000008

Degree 9 0.000008

Degree 10 0.000008

Degree 11 0.000010

Degree 12 0.000012

Degree 13 0.000012

Degree 14 0.000013

Degree 15 0.000015

dtype: float64

(g) Irreducible Error

[ ]:

std = 0.01

irreducible_error = std ** 2

print("Irreducible Error")

irreducible_error

Irreducible Error

[ ]:

0.0001

The irreducible error represents the inherent variability in the data that cannot be reduced by any model. It is specified in the data generating process when we set the std to be 0.01.

(h) MSE

[ ]:

mse = square_of_bias + variance_of_predictions + irreducible_error

print("MSE")

mse

MSE

[ ]:

Degree 0 0.000262

Degree 1 0.000174

Degree 2 0.000131

Degree 3 0.000108

Degree 4 0.000105

Degree 5 0.000105

Degree 6 0.000106

Degree 7 0.000106

Degree 8 0.000108

Degree 9 0.000108

Degree 10 0.000108

Degree 11 0.000110

Degree 12 0.000112

Degree 13 0.000112

Degree 14 0.000113

Degree 15 0.000115

dtype: float64

(i) Plot

[ ]:

import matplotlib.pyplot as plt

degrees = range(16)

plt.figure(figsize=(14, 8))

plt.plot(degrees, square_of_bias, label='Square of Bias', marker='o')

plt.plot(degrees, variance_of_predictions, label='Variance', marker='x')

plt.plot(degrees, [irreducible_error]*16, label='Irreducible Error',␣

↪linestyle='--')

plt.plot(degrees, mse, label='MSE', marker='^')

plt.xlabel('Degree of Polynomial')

plt.ylabel('Error')

plt.title('Bias-Variance-MSE vs Degree')

plt.legend()

plt.show()

• Square of Bias: It decreases significantly as the degree of the polynomial increases from 0 to around 4 or 5, reflecting that higher-degree polynomials fit the data more closely and thus have a lower bias. However, beyond a certain point, the reduction in bias becomes marginal.

• Variance: Initially, the variance is relatively low for simpler models (low-degree polynomials) but starts to increase as the model complexity grows. This increase indicates that more complex models are more sensitive to the training data, leading to higher variability in their predictions across different datasets.

• Irreducible Error: It remains constant across all degrees of polynomial, as expected. This component of the error is not affected by the model choice or complexity because it represents the noise inherent in the data generation process.

• MSE: The Mean Squared Error decreases initially as the degree of the polynomial increases, benefiting from the reduced bias. However, as the model becomes more complex (especially beyond degree 4 or 5), the decrease in MSE slows down, and it eventually stabilizes or slightly increases. This pattern is due to the trade-off between bias and variance; while higher-degree polynomials reduce bias, their increased variance eventually offsets these gains, leading to a stabilization or slight increase in MSE.

• Conclusion: The plot and analysis underscore the importance of balancing model complex-ity to minimize overall error. While increasing the polynomial degree reduces bias, it also increases variance, which can harm the model’s generalization ability. The optimal model complexity (in terms of polynomial degree) appears to be in the range where the decrease in bias is balanced by the increase in variance, minimizing the MSE. This balance point is crucial for developing models that generalize well to unseen data.