代做data编程、代写Python设计程序
Question 1. Explore Model-Based Feature Importance
Throughout this question, you may only use Python. For each sub-question, provide commentary (if
needed) along with screenshots of the code used. Please also provide a copy of the code in your solu tions.py file. For fitting models, always use a random seed (or random state) of 4 for reproducibility.
(a) Generate a dataset of two classes using sklearn.datasets.make classification. It should
have 1000 observations, 20 features. Set 5 of those features to be informative (important), and the
rest as redundant. Be sure to set the shuffle parameter to False, so that the informative features
are listed first. Normalize your data using sklearn.StandardScaler(). Then, fit a decision
tree (using entropy as the criteria for splits) to a shuffled version of the data1 using sklearn.tree
model DecisionTreeClassifier, and using its feature importances method, report how
many of the actually important features are found in the top 5 important features by the deci sion tree. Plot a histogram with x-axis showing the features ranked in decreasing order of im portance, and the y-axis showing the feature importance score. Use a random seed of 0 when
generating the data for reproducibility. Use a random seed of 0 when shuffling the data, you can
use shuffled idxs = np.random.default rng(seed=0).permutation(X.shape[1]).
(b) Provide a detailed explanation of how the feature importance (a.k.a. Gini importance) in the pre vious question are computed; use formulas to explain the exact calculation. Further, answer the
following:
1. What feature importance score is assigned to a feature that is not used for any splits of the tree.
Why?
2. What does a feature importance of 0.15 mean?
(c) In order to obtain a more accurate picture of how good decision trees are at finding important
features, we will repeat the experiment in part (a) a large number of times. Repeat the experiment
a total of 1000 times. In the i-th experiment, use a random seed of i when creating the data set,
where i = 1, 2, . . . . , 1000. For each trial, record how many of the actually important features are
identified. Provide a histogram of this metric over the 1000 trials. What do you think about the
ability of decision trees to pick out the top features? Report the average number of good features
recovered over the 1000 trials.
(d) Repeat part (c), but now use logistic regression with no penalty. Do this once with and once without
scaling the feature matrix. As a feature importance metric, use the absolute value of the coefficient
of that feature. Plot a histogram as before and report the average number of features recovered
over the 1000 trials. Compare the scaled and non-scaled versions. How does logistic regression
compare to decision trees?
(e) Does scaling features affect the result for decision trees? Explain.
(f) We now want to assess how often the two models (Decision trees and logistic regression (with
scaling)) identify the same features as being important. Using the set-up of part (c), for each trial,
record the number of overlaps for the top-5 ranked features for each of the two models. Plot a
histogram of the number of overlaps over all trials. For example, if on a particular trial, DT has
[1, 2, 3, 4, 5] in its top-5, and Logistic regression has [1, 2, 6, 7, 8], the number of overlaps for this trial
is 2.
(g) The approaches considered so far are called ”model-based” feature importance methods, since
they define importance with respect to a particular algorithm/model being used. Discuss some
1The reason we do not shuffle the data when creating it is that we want to be able to know which of the features are the most
important (first 5). We do not want to give the algorithm the ordered features as this may inflate the algorithm’s ability to find
important features, it may just break ties by looking at which features come first.
Page 3
potential disadvantages of using a model-based approach if your goal is to uncover truly impor tant features, referring to the previous exercises for evidence. For example, suppose that you are
studying a rare genetic disease and that the 20 features represent specific genetic features, only 5 of
which are truly associated with the disease. Further, discuss the effect of the number of redundant
features used when creating the data set.
Question 2. Greedy Feature Selection
We now consider a different approach to feature selection known as backward selection. In backward
selection, we:
1. start with all features in the model
2. at each round, we remove the j-th feature from the model based on the drop in the value of a
certain metric. We eliminate the feature corresponding to the smallest drop in the metric.
3. we repeat step 2 until there are no features left.
(a) Why do you think this is referred to as a greedy feature importance algorithm? What do you think
are some of the pitfalls of greedy algorithms in this context?
(b) Using the same set-up as in Question 1 part Q1 (a) write code implementing the backward elimina tion algorithm. Use a logistic regression model with no penalty, and the same metric as in Question
1 part (d). Be sure to generate the data without shuffling but then to shuffle the data before fitting
the model. Report the remaining features at round 15 (that is, when only 5 features are left). How
many of these are actually important features?
(c) Repeat part (a) for 1000 trials (similar to what is done in Q1 (c)). Plot a histogram of the number of
important features recovered, and report the average number of recovered features.
(d) Another approach is called best subset selection. This model generates all possible subsets, trains
a model on each subset, evaluates the performance and returns the subset with the highest per formance. For example, at the t-th round, we consider all subsets with t features. How does this
algorithm compare to backward selection? Will it always outperform backward elimination? What
are some disadvantages of this approach?
(e) Implement best subset selection in code. Repeat part (c) using your best subset implementation.
For computational reasons, set all parameters as in Q1 part (a), but with only 7 features, 3 of which
are to be taken to be informative, and the rest to be redundant. Plot a histogram as before and
report the average number of recoveries. Comment on your results.
(f) An alternative approach to feature importance is known as the Permutation Feature Importance
score, implemented in sklearn.inspection.permutation importance. Read the docu mentation and provide a detailed explanation of how permutation importance works. Compare it
to the techniques studied so far in this homework, and explain why we refer to this as a model independent metric. Do you think it’s more or less fair to compare logistic regression and decision
trees using this metric? Finally, using the sklearn implementation, re-do part Q2(c) using this new
feature importance metric. Similar to before, use 20 features, with 5 to be set as informative and
the rest as redundant.
Page 4