Chapter.06 실전 머신러닝 프로젝트_Template Code.ipynb

{"cells":[{"cell_type":"markdown","id":"8f3119aa","metadata":{"id":"8f3119aa"},"source":["## Machine Learning 프로젝트 수행을 위한 코드 구조화\n","\n","`(분류, 회귀 Task)`"]},{"cell_type":"markdown","id":"9f7610ea","metadata":{"id":"9f7610ea"},"source":["- ML project를 위해서 사용하는 템플릿 코드를 만듭니다.\n","\n","1. **필요한 라이브러리와 데이터를 불러옵니다.**\n","\n","\n","2. **EDA를 수행합니다.** 이 때 EDA의 목적은 풀어야하는 문제를 위해서 수행됩니다.\n","\n","\n","3. **전처리를 수행합니다.** 이 때 중요한건 **feature engineering**을 어떻게 하느냐 입니다.\n","\n","\n","4. **데이터 분할을 합니다.** 이 때 train data와 test data 간의 분포 차이가 없는지 확인합니다.\n","\n","\n","5. **학습을 진행합니다.** 어떤 모델을 사용하여 학습할지 정합니다. 성능이 잘 나오는 GBM을 추천합니다.\n","\n","\n","6. **hyper-parameter tuning을 수행합니다.** 원하는 목표 성능이 나올 때 까지 진행합니다. 검증 단계를 통해 지속적으로 **overfitting이 되지 않게 주의**하세요.\n","\n","\n","7. **최종 테스트를 진행합니다.** 데이터 분석 대회 포맷에 맞는 submission 파일을 만들어서 성능을 확인해보세요."]},{"cell_type":"markdown","id":"bd2f7530","metadata":{"id":"bd2f7530"},"source":["## 1. 라이브러리, 데이터 불러오기"]},{"cell_type":"code","execution_count":null,"id":"125fc348","metadata":{"id":"125fc348"},"outputs":[],"source":["# 데이터분석 4종 세트\n","import numpy as np\n","import pandas as pd\n","import matplotlib.pyplot as plt\n","import seaborn as sns\n","\n","# 모델들, 성능 평가\n","# (저는 일반적으로 정형데이터로 머신러닝 분석할 때는 이 2개 모델은 그냥 돌려봅니다. 특히 RF가 테스트하기 좋습니다.)\n","from sklearn.ensemble import RandomForestClassifier\n","from sklearn.ensemble import RandomForestRegressor\n","from lightgbm.sklearn import LGBMClassifier\n","from lightgbm.sklearn import LGBMRegressor\n","\n","# 상관관계 분석, VIF : 다중공선성 제거\n","from statsmodels.stats.outliers_influence import variance_inflation_factor\n","\n","# KFold(CV), partial : optuna를 사용하기 위함\n","from sklearn.model_selection import KFold\n","from functools import partial\n","\n","# hyper-parameter tuning을 위한 라이브러리, optuna\n","import optuna"]},{"cell_type":"code","execution_count":null,"id":"0e4d49ce","metadata":{"id":"0e4d49ce"},"outputs":[],"source":["# flag setting\n","data_reducing = False ## memory reducing technique\n","feature_reducing = False ## feature extraction (curse of dimensionality)"]},{"cell_type":"code","execution_count":null,"id":"3615c24a","metadata":{"id":"3615c24a"},"outputs":[],"source":["# 데이터를 불러옵니다.\n","train =\n","test = "]},{"cell_type":"markdown","id":"c9c9acb8","metadata":{"id":"c9c9acb8"},"source":["## 2. EDA"]},{"cell_type":"markdown","id":"6fdf620b","metadata":{"id":"6fdf620b"},"source":["- 데이터에서 찾아야 하는 기초적인 내용들을 확인합니다.\n","\n","\n","- class imbalance, target distribution, outlier, correlation을 확인합니다."]},{"cell_type":"code","execution_count":null,"id":"adb06474","metadata":{"id":"adb06474"},"outputs":[],"source":["## On your Own\n","data.column.value_counts()\n","sns.countplot()\n","sns.histplot()\n","..."]},{"cell_type":"markdown","id":"f2c602bd","metadata":{"id":"f2c602bd"},"source":["이런 식으로 여러가지 그래프를 그려가며, 데이터에 대한 인사이트를 얻습니다!"]},{"cell_type":"markdown","id":"9dbb8802","metadata":{"id":"9dbb8802"},"source":["### 3. 전처리"]},{"cell_type":"markdown","id":"b79a6f0a","metadata":{"id":"b79a6f0a"},"source":["#### 결측치 처리"]},{"cell_type":"code","execution_count":null,"id":"bbafdcd0","metadata":{"id":"bbafdcd0"},"outputs":[],"source":["# 결측치가 있는 column\n"]},{"cell_type":"code","execution_count":null,"id":"c4d33e12","metadata":{"scrolled":true,"id":"c4d33e12"},"outputs":[],"source":["# 중복정보가 있는 column 제거하기 위해 상관계수를 확인해봅니다.\n","correlated_features = "]},{"cell_type":"markdown","id":"0b757dc0","metadata":{"id":"0b757dc0"},"source":["#### 다중공선성 처리"]},{"cell_type":"code","execution_count":null,"id":"702d47e9","metadata":{"id":"702d47e9"},"outputs":[],"source":["# 상관계수가 threshold(e.g. 0.8)를 넘기는 feature들을 제거합니다.\n","threshold = "]},{"cell_type":"markdown","source":["#### Encoding Categorical Features"],"metadata":{"id":"JdO_MINum6pR"},"id":"JdO_MINum6pR"},{"cell_type":"code","source":["# dtype이 object인 변수들 처리.\n"],"metadata":{"id":"tNT9eZB0nAmq"},"id":"tNT9eZB0nAmq","execution_count":null,"outputs":[]},{"cell_type":"markdown","id":"606493f0","metadata":{"id":"606493f0"},"source":["#### feature extraction\n","\n","- 차원의 저주를 해결하거나, 데이터의 feature 조합을 이용하는 새로운 feature를 생성할 때, PCA를 사용합니다.\n","\n","- 분석에 사용할 feature를 선택하는 과정도 포함합니다."]},{"cell_type":"code","execution_count":null,"id":"4a137c94","metadata":{"id":"4a137c94"},"outputs":[],"source":["# PCA 적용\n","from sklearn.decomposition import PCA\n","\n","if feature_reducing:\n","    pca = PCA(n_components=0.9) # PCA(n_components=6)\n","    pca_data = pca.fit_transform(X)"]},{"cell_type":"markdown","id":"f497a2d8","metadata":{"id":"f497a2d8"},"source":["### 4. 학습 데이터 분할"]},{"cell_type":"code","execution_count":null,"id":"47306aaf","metadata":{"id":"47306aaf"},"outputs":[],"source":["# 첫번째 테스트용으로 사용하고, 실제 학습시에는 K-Fold CV를 사용합니다.\n","# train : test = 8 : 2\n","from sklearn.model_selection import train_test_split\n","\n","X = \n","y =\n","\n","X_train, X_test, y_train, y_test = train_test_split(X, y)"]},{"cell_type":"code","execution_count":null,"id":"67efd2ee","metadata":{"id":"67efd2ee"},"outputs":[],"source":["print(X_train.shape, y_train.shape, X_test.shape, y_test.shape)"]},{"cell_type":"markdown","id":"58056e51","metadata":{"id":"58056e51"},"source":["### 5. 학습 및 평가"]},{"cell_type":"code","execution_count":null,"id":"39fd2515","metadata":{"id":"39fd2515"},"outputs":[],"source":["# 간단하게 LightGBM 테스트\n","# 적당한 hyper-parameter 조합을 두었습니다. (항상 best는 아닙니다. 예시입니다.)\n","\n","param_grid = {\n","    \"max_bin\" : 20,\n","    \"learning_rate\" : 0.0025,\n","    \"objective\" : \"regression\",\n","    \"boosting_type\" : \"gbdt\",\n","    \"metric\" : \"mae\",\n","    \"sub_feature\" : 0.345,\n","    \"bagging_fraction\" : 0.85,\n","    \"bagging_freq\" : 40,\n","    \"num_leaves\" : 512,\n","    \"min_data\" : 500,\n","    \"min_hessian\" : 0.05,\n","    \"verbose\" : 2,\n","    \"feature_fraction_seed\" : 2,\n","    \"bagging_seed\" : 3\n","}\n","\n","model = LGBMRegressor(**param_grid)"]},{"cell_type":"code","execution_count":null,"id":"ddffa474","metadata":{"scrolled":true,"id":"ddffa474"},"outputs":[],"source":["print(\"\\nFitting LightGBM...\")\n","model.fit(X_train, y_train)"]},{"cell_type":"code","execution_count":null,"id":"6c8b0259","metadata":{"id":"6c8b0259"},"outputs":[],"source":["# metric은 그때마다 맞게 바꿔줘야 합니다.\n","evaluation_metric = "]},{"cell_type":"code","execution_count":null,"id":"a6b39be5","metadata":{"id":"a6b39be5"},"outputs":[],"source":["print(\"Prediction\")\n","pred_train = model.predict(x_train)\n","pred_test = model.predict(x_test)\n","\n","\n","train_score = evaluation_metric(y_train, pred_train)\n","test_score = evaluation_metric(y_test, pred_test)\n","\n","print(\"Train Score : %.4f\" % train_score)\n","print(\"Test Score : %.4f\" % test_score)"]},{"cell_type":"markdown","id":"bc755b17","metadata":{"id":"bc755b17"},"source":["### 6. Hyper-parameter Tuning"]},{"cell_type":"markdown","id":"60070d0e","metadata":{"id":"60070d0e"},"source":["> GridSearchCV"]},{"cell_type":"markdown","id":"2bf886a9","metadata":{"id":"2bf886a9"},"source":["** LightGBM의 hyperparameter **\n","\n","[Official Documentation] https://lightgbm.readthedocs.io/en/latest/Parameters-Tuning.html \n","\n","[Blog 1] https://smecsm.tistory.com/133\n","\n","[Blog 2] https://towardsdatascience.com/kagglers-guide-to-lightgbm-hyperparameter-tuning-with-optuna-in-2021-ed048d9838b5\n","\n","[Blog 3] https://nurilee.com/2020/04/03/lightgbm-definition-parameter-tuning/"]},{"cell_type":"code","execution_count":null,"id":"815dcbef","metadata":{"scrolled":true,"id":"815dcbef"},"outputs":[],"source":["# GridSearchCV를 이용하여 가장 좋은 성능을 가지는 모델을 찾아봅시다. (이것은 첫번째엔 선택입니다.)\n","# Lightgbm은 hyper-parameter의 영향을 많이 받기 때문에, 저는 보통 맨처음에 한번 정도는 가볍게 GCV를 해봅니다.\n","# 성능 향상이 별로 없다면, lightgbm으로 돌린 대략적인 성능이 이 정도라고 생각하면 됩니다.\n","# 만약 성능 향상이 크다면, 지금 데이터는 hyper-parameter tuning을 빡빡하게 하면 성능 향상이 많이 이끌어 낼 수 있습니다.\n","\n","from sklearn.model_selection import GridSearchCV\n","\n","param_grid = {\n","    \"max_depth\" : [8, 16, None],\n","    \"n_estimators\" : [100, 300, 500],\n","    \"max_bin\" : [20],\n","    \"learning_rate\" : [0.001, 0.0025, 0.003],\n","    \"objective\" : [\"regression\"],\n","    \"boosting_type\" : [\"gbdt\"],\n","    \"metric\" : [\"mae\"],\n","    \"sub_feature\" : [0.345],\n","    \"bagging_fraction\" : [0.7, 0.75, 0.85],\n","    \"bagging_freq\" : [40],\n","    \"num_leaves\" : [256, 512],\n","    \"min_data\" : [500],\n","    \"verbose\" : [-1], # 필수\n","    \"min_hessian\" : [0.05],\n","    \"feature_fraction_seed\" : [2],\n","    \"bagging_seed\" : [3]\n","}\n","\n","\n","gcv = GridSearchCV(estimator=model, param_grid=param_grid, cv=5,\n","                  n_jobs=-1, verbose=1)\n","\n","gcv.fit(X_train, y_train)\n","print(\"Best Estimator : \", gcv.best_estimator_)"]},{"cell_type":"code","execution_count":null,"id":"9ec6ac14","metadata":{"id":"9ec6ac14"},"outputs":[],"source":["print(\"Prediction with Best Estimator\")\n","gcv_pred_train = gcv.predict(X_train)\n","gcv_pred_test = gcv.predict(x_test)\n","\n","gcv_train_score = evaluation_metric(y_train, gcv_pred_train)\n","gcv_test_score = evaluation_metric(y_test, gcv_pred_test)\n","\n","print(\"Train MAE Score : %.4f\" % gcv_train_score)\n","print(\"Test MAE Score : %.4f\" % gcv_test_score)"]},{"cell_type":"code","execution_count":null,"id":"3b852da7","metadata":{"id":"3b852da7"},"outputs":[],"source":["print(\"Performance Gain\") # 이걸로 성능 향상 확인.\n","print(\"in train : \", (train_score - gcv_train_score))\n","print(\"in test : \", (test_score - gcv_test_score))"]},{"cell_type":"markdown","id":"97e5302d","metadata":{"id":"97e5302d"},"source":["> optuna를 사용해봅시다 !"]},{"cell_type":"code","execution_count":null,"id":"34ce4986","metadata":{"id":"34ce4986"},"outputs":[],"source":["def optimizer(trial, X, y, K):\n","    # 조절할 hyper-parameter 조합을 적어줍니다.\n","    n_estimators = \n","    max_depth = \n","    max_features = \n","    \n","    \n","    # 원하는 모델을 지정합니다, optuna는 시간이 오래걸리기 때문에 저는 보통 RF로 일단 테스트를 해본 뒤에 LGBM을 사용합니다.\n","    model = RandomForestRegressor(n_estimators=n_estimators,\n","                                 max_depth=max_depth,\n","                                 max_features=max_features)\n","    \n","    \n","    # K-Fold Cross validation을 구현합니다.\n","    folds = KFold(n_splits=K)\n","    losses = []\n","    \n","    for train_idx, val_idx in folds.split(X, y):\n","        X_train = X.iloc[train_idx, :]\n","        y_train = y.iloc[train_idx]\n","        \n","        X_val = X.iloc[val_idx, :]\n","        y_val = y.iloc[val_idx]\n","        \n","        model.fit(X_train, y_train)\n","        preds = model.predict(X_val)\n","        loss = mean_absolute_error(y_val, preds)\n","        losses.append(loss)\n","    \n","    \n","    # K-Fold의 평균 loss값을 돌려줍니다.\n","    return np.mean(losses)"]},{"cell_type":"code","execution_count":null,"id":"7150b210","metadata":{"scrolled":false,"id":"7150b210"},"outputs":[],"source":["K = # Kfold 수\n","opt_func = partial(optimizer, X=X_train, y=y_train, K)\n","\n","study = optuna.create_study(direction=\"minimize\") # 최소/최대 어느 방향의 최적값을 구할 건지.\n","study.optimize(opt_func, n_trials=5)"]},{"cell_type":"code","execution_count":null,"id":"72d0a118","metadata":{"id":"72d0a118"},"outputs":[],"source":["# optuna가 시도했던 모든 실험 관련 데이터\n","study.trials_dataframe()"]},{"cell_type":"code","execution_count":null,"id":"a805da05","metadata":{"id":"a805da05"},"outputs":[],"source":["print(\"Best Score: %.4f\" % study.best_value) # best score 출력\n","print(\"Best params: \", study.best_trial.params) # best score일 때의 하이퍼파라미터들"]},{"cell_type":"code","execution_count":null,"id":"051ae1eb","metadata":{"id":"051ae1eb"},"outputs":[],"source":["# 실험 기록 시각화\n","optuna.visualization.plot_optimization_history(study)"]},{"cell_type":"code","execution_count":null,"id":"efbf8f65","metadata":{"scrolled":false,"id":"efbf8f65"},"outputs":[],"source":["# hyper-parameter들의 중요도\n","optuna.visualization.plot_param_importances(study)"]},{"cell_type":"markdown","id":"24b360ec","metadata":{"id":"24b360ec"},"source":["### 7. 테스트 및 제출 파일 생성"]},{"cell_type":"code","execution_count":null,"id":"a0daf54e","metadata":{"id":"a0daf54e"},"outputs":[],"source":["model = RandomForestRegressor(n_estimators=study.best_trial.params[\"n_estimators\"],\n","                                 max_depth=study.best_trial.params[\"max_depth\"],\n","                                 max_features=study.best_trial.params[\"max_features\"])\n","\n","model.fit(X_train, y_train)\n","preds = model.predict(X_test)\n","preds"]},{"cell_type":"code","execution_count":null,"id":"dfa0db70","metadata":{"id":"dfa0db70"},"outputs":[],"source":["X_test # 원본 데이터랑 id가 맞는지 확인 해보기!"]},{"cell_type":"code","execution_count":null,"id":"8ff2070c","metadata":{"id":"8ff2070c"},"outputs":[],"source":["submission = pd.DataFrame() # submission을 생성합니다.\n","submission"]},{"cell_type":"code","execution_count":null,"id":"55a2c13f","metadata":{"id":"55a2c13f"},"outputs":[],"source":["submission.reset_index(drop=True).to_csv(\"submission.csv\", index=False)"]}],"metadata":{"kernelspec":{"display_name":"Python 3 (ipykernel)","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.8.12"},"colab":{"name":"Chapter.06 실전 머신러닝 프로젝트_Template Code.ipynb","provenance":[],"collapsed_sections":[]}},"nbformat":4,"nbformat_minor":5}