Homework 2: Advanced Data Mining Techniques

Objective:

Enhance your data exploration and preprocessing techniques by incorporating advanced dimensionality reduction methods such as PCA and LDA.
Strengthen your understanding of unsupervised learning by generating clustering-based pseudo-labels for further supervised analysis.
Develop critical thinking skills by comparing different data representations and evaluating their impact on predictive modeling.
Foster the ability to derive actionable insights and data-driven recommendations based on comprehensive analysis.

Instructions

This is an individual assignment. No collaboration is allowed.

In Homework 2 you will extend your analysis from Homework 1

Provide reference/citation to any method, metric, function that had not been covered in class. For any methods/code covered in class Labs, you do not need to provide references.

1. PCA (Principal Component Analysis) 30pts

Standardize/normalize your dataset if you have not already done so in Homework 1
Apply PCA to your dataset
Plot the explained variance ratio for the principal components (e.g., a scree plot)
Choose the number of components to retain and explain your choice
Visualize your data projected onto the first two principal components. Discuss any interesting observation (recall how we interpreted the PCA plots in class)

2. LDA (Linear Discriminant Analysis) 30pts

If your dataset contains a categorical variable, use it for LDA

If you dataset does not contain any features that can be used as classes, revisit your cluster analysis and create a new column with cluster labels - these will be your classes for LDA

HINTS (example):
kmeans = KMeans(n_clusters=3, random_state=42)
cluster_labels = kmeans.fit_predict(scaled_data)
# Use cluster_labels as your y:
lda.fit_transform(scaled_data, cluster_labels)

Show the graph with the labels/classes distribution - is it balanced or unbalanced?
Apply LDA. Hint: use sklearn examples for plotting LDA as scatterplot
Discuss any interesting observations and the model prediction

3. Enhanced Predictive Modeling 30pts

Integrate your PCA into your predictive modeling framework from Homework 1 (you only need one regression model)
Evaluate whether the reduced feature space leads to improved model performance

4. Reflections 10pts

Did you find the dimensionality techniques helpful to improve the predicting models?
Summarize the key insights obtained from applying PCA and LDA (for example, in your report to the client, what would you share as insights from this data)
What did you learn from this homework? Share your experience and any challenges you faced.

Submission Instructions

Submit your homework as a URL with clearly labeled sections corresponding to the tasks above (continue your homework 1 colab)
Use text cells for providing written explanations
Ensure your code is clean and warnings are suppressed (e.g., using import warnings; warnings.filterwarnings('ignore')).

Grading Summary

Section	Points
Dimensionality Reduction (PCA)	30
LDA	30
Predictive Modeling Enhancement	30
Conclusion and Reflections	10
Total	100

Good luck with Homework 2! If you have any questions or need clarification, please reach out.