CSSE 386 Syllabus Schedule Worksheets/Quizzes Labs Resources

Homework 1. Data-Driven Insights: Transforming Raw Data into Business Solutions

Objective:

• Provide hands-on experience with exploring and preprocessing datasets
• Develop skills in applying descriptive and predictive data mining methods
• Encourage critical thinking about data insights and their potential applications
• Foster the ability to create business-relevant recommendations based on data analysis

Instructions

This is an individual assignment. No collaboration is allowed.

Students are required to choose a dataset of their own interest and apply data mining techniques as outlined below. The dataset must be appropriate for Data Mining tasks and selected from the provided repositories below.

Provide reference/citation to any method, metric, function that had not been covered in class. For any methods/code covered in class Labs, you do not need to provide references.

1. Dataset Description and Exploratory Data Analysis (20 points)

• Helpful Dataset Resources
  • Kaggle Datasets
  • Scikit-learn Datasets
  • UCI Machine Learning Repository
• Dataset Documentation
  • Provide the origin of the dataset (who collected it, when, and for what purpose)
  • Include a link to the dataset
• Data Description
  • Describe the dataset size, types of columns, and any missing data
  • Identify the target variable and features
  • Discuss multicollinearity among features (e.g., correlation)
  • Analyze the target variable's distribution (e.g., skewness, imbalance)
  • Provide your conclusion on what dta amining techniques will be appropriate for this dataset. Recall that some methods (e.g. Linear regression, decision trees) require certain assumptions about data distribution and balance.
• Written Explanation: Provide clear written answers to the above questions supported by visualizations (e.g., histograms, correlation heatmaps).

2. Data Preprocessing (15 points)

• Normalization and Cleaning:
  • Describe the normalization methods used
  • Explain how missing data was cleaned
  • List any additional cleaning steps, such as dropping unnecessary columns (e.g., IDs, invoice numbers) or renaming columns
• Feature Selection: Justify whether all features were used or if any were excluded.
• Data Splitting: Explain how the data was split for analysis (e.g., training and testing sets for predictive methods). Note: Splitting may not be necessary for descriptive methods like clustering.

3. Descriptive Data Mining Methods (20 points)

• Clustering Analysis:
  • Choose a clustering method (K-means or Hierarchical Clustering)
  • Visualize the clusters using appropriate plots (e.g., scatterplots, dendrograms)
  • Analyze and describe the characteristics of the clusters
• Optional: Reflect on whether cluster labels could be used as additional features in predictive methods.

4. Predictive Methods (35 points)

a) Classification (20 points)

• Apply at least two classification methods from the following: KNN, Naive Bayes, SVM, Decision Tree
• Evaluation Metrics: Provide metrics like accuracy, precision, recall
• Parameter Tuning: Attempt to adjust model parameters to improve results (for example, modify k in KNN, choose a differnet kernel in SVM)
• Reflection: Discuss the results and insights gained

b) Regression (15 points)

• Perform at least two types of regression, with at least one being a regularization method (Lasso or Ridge)
• Evaluation Metrics: Provide metrics like MSE, ROC-AUC
• Parameter Tuning: Attempt to adjust parameters to improve results (e.g. alpha)
• Reflection: Discuss the results and insights gained

5. Conclusion (10 points)

• Overall Insights: Summarize key findings and what you learned from the homework
• Business Recommendation: Provide actionable suggestions for decision-makers based on your analysis. Example: "For a wine dataset, the prediction allows identification of factors influencing wine quality, helping producers optimize their processes."
• References: Include citations and references for the dataset and any tools or libraries used

Submission Instructions

Submit your homework as a URL with clearly labeled sections. Include visualizations, written explanations, and any code snippets where relevant. Ensure your analysis and recommendations are thorough and well-supported. Note the penalty will apply if you print the entire dataset table (please restrict to the top 5 columns).

To make it clean, consider removing any warnings during pip instalation: import warnings warnings.filterwarnings('ignore')

Grading Summary

Section	Points
Dataset Description & Exploratory Analysis	20
Data Preprocessing	15
Descriptive Data Mining Methods	20
Predictive Methods	35
Conclusion, Code, Writing	10
Total	100

Good luck! If you have any questions or need clarification, please reach out during office hours or via email or teams.