赛题以金融风控中的个人信贷为背景，要求选手根据贷款申请人的数据信息预测其是否有违约的可能，以此判断是否通过此项贷款，这是一个典型的分类问题。通过这道赛题来引导大家了解金融风控中的一些业务背景，解决实际问题，帮助竞赛新人进行自我练习、自我提高。
赛题以预测金融风险为任务，数据集报名后可见并可下载，该数据来自某信贷平台的贷款记录，总数据量超过120w，包含47列变量信息，其中15列为匿名变量。为了保证比赛的公平性，将会从中抽取80万条作为训练集，20万条作为测试集A，20万条作为测试集B，同时会对employmentTitle、purpose、postCode和title等信息进行脱敏。
比赛地址：https://tianchi.aliyun.com/competition/entrance/531830/introduction
(资料图片仅供参考)
赛题以预测用户贷款是否违约为任务，数据集报名后可见并可下载，该数据来自某信贷平台的贷款记录，总数据量超过120w，包含47列变量信息，其中15列为匿名变量。为了保证比赛的公平性，将会从中抽取80万条作为训练集，20万条作为测试集A，20万条作为测试集B，同时会对employmentTitle、purpose、postCode和title等信息进行脱敏。
一般而言，对于数据在比赛界面都有对应的数据概况介绍（匿名特征除外），说明列的性质特征。了解列的性质会有助于我们对于数据的理解和后续分析。 Tip:匿名特征，就是未告知数据列所属的性质的特征列。
train.csv
id为贷款清单分配的唯一信用证标识loanAmnt贷款金额term贷款期限（year）interestRate贷款利率installment分期付款金额grade贷款等级subGrade贷款等级之子级employmentTitle就业职称employmentLength就业年限（年）homeOwnership借款人在登记时提供的房屋所有权状况annualIncome年收入verificationStatus验证状态issueDate贷款发放的月份purpose借款人在贷款申请时的贷款用途类别postCode借款人在贷款申请中提供的邮政编码的前3位数字regionCode地区编码dti债务收入比delinquency_2years借款人过去2年信用档案中逾期30天以上的违约事件数ficoRangeLow借款人在贷款发放时的fico所属的下限范围ficoRangeHigh借款人在贷款发放时的fico所属的上限范围openAcc借款人信用档案中未结信用额度的数量pubRec贬损公共记录的数量pubRecBankruptcies公开记录清除的数量revolBal信贷周转余额合计revolUtil循环额度利用率，或借款人使用的相对于所有可用循环信贷的信贷金额totalAcc借款人信用档案中当前的信用额度总数initialListStatus贷款的初始列表状态applicationType表明贷款是个人申请还是与两个共同借款人的联合申请earliesCreditLine借款人最早报告的信用额度开立的月份title借款人提供的贷款名称policyCode公开可用的策略_代码=1新产品不公开可用的策略_代码=2n系列匿名特征匿名特征n0-n14，为一些贷款人行为计数特征的处理1.2预测指标竞赛采用AUC作为评价指标。AUC（Area Under Curve）被定义为ROC曲线下与坐标轴围成的面积。
1.2.1 分类算法常见的评估指标如下：1、混淆矩阵（Confuse Matrix）
（1）若一个实例是正类，并且被预测为正类，即为真正类TP(True Positive )（2）若一个实例是正类，但是被预测为负类，即为假负类FN(False Negative )（3）若一个实例是负类，但是被预测为正类，即为假正类FP(False Positive )（4）若一个实例是负类，并且被预测为负类，即为真负类TN(True Negative )2、准确率（Accuracy）准确率是常用的一个评价指标，但是不适合样本不均衡的情况。$Accuracy = \frac{TP + TN}{TP + TN + FP + FN}$
3、精确率（Precision）又称查准率，正确预测为正样本（TP）占预测为正样本(TP+FP)的百分比。$Precision = \frac{TP}{TP + FP}$
4、召回率（Recall）又称为查全率，正确预测为正样本（TP）占正样本(TP+FN)的百分比。$Recall = \frac{TP}{TP + FN}$
5、F1 Score精确率和召回率是相互影响的，精确率升高则召回率下降，召回率升高则精确率下降，如果需要兼顾二者，就需要精确率、召回率的结合F1 Score。$F1-Score = \frac{2}{\frac{1}{Precision} + \frac{1}{Recall}}$
6、P-R曲线（Precision-Recall Curve）P-R曲线是描述精确率和召回率变化的曲线
7、ROC（Receiver Operating Characteristic）
ROC空间将假正例率（FPR）定义为 X 轴，真正例率（TPR）定义为 Y 轴。TPR：在所有实际为正例的样本中，被正确地判断为正例之比率。$TPR = \frac{TP}{TP + FN}$FPR：在所有实际为负例的样本中，被错误地判断为正例之比率。$FPR = \frac{FP}{FP + TN}$
8、AUC(Area Under Curve)AUC（Area Under Curve）被定义为ROC曲线下与坐标轴围成的面积，显然这个面积的数值不会大于1。又由于ROC曲线一般都处于y=x这条直线的上方，所以AUC的取值范围在0.5和1之间。AUC越接近1.0，检测方法真实性越高;等于0.5时，则真实性最低，无应用价值。
1.2.2 对于金融风控预测类常见的评估指标如下:1、KS(Kolmogorov-Smirnov)KS统计量由两位苏联数学家A.N. Kolmogorov和N.V. Smirnov提出。在风控中，KS常用于评估模型区分度。区分度越大，说明模型的风险排序能力（ranking ability）越强。K-S曲线与ROC曲线类似，不同在于
ROC曲线将真正例率和假正例率作为横纵轴K-S曲线将真正例率和假正例率都作为纵轴，横轴则由选定的阈值来充当。公式如下：$KS=max(TPR-FPR)$KS不同代表的不同情况，一般情况KS值越大，模型的区分能力越强，但是也不是越大模型效果就越好，如果KS过大，模型可能存在异常，所以当KS值过高可能需要检查模型是否过拟合。以下为KS值对应的模型情况，但此对应不是唯一的，只代表大致趋势。| KS（%） | 好坏区分能力 |
|---|---|
| 20以下 | 不建议采用 |
| 20-40 | 较好 |
| 41-50 | 良好 |
| 51-60 | 很强 |
| 61-75 | 非常强 |
| 75以上 | 过于高，疑似存在问题 |
2、ROC
3、AUC
1.3 项目流程介绍1.4 代码示例本部分为对于数据读取和指标评价的示例。
1.4.1 数据读取pandasimport pandas as pdtrain = pd.read_csv("train.csv")testA = pd.read_csv("testA.csv")print("Train data shape:",train.shape)print("TestA data shape:",testA.shape)Train data shape: (800000, 47)TestA data shape: (200000, 48)train.head()## 混淆矩阵import numpy as npfrom sklearn.metrics import confusion_matrixy_pred = [0, 1, 0, 1]y_true = [0, 1, 1, 0]print("混淆矩阵:\n",confusion_matrix(y_true, y_pred))混淆矩阵: [[1 1] [1 1]]## accuracyfrom sklearn.metrics import accuracy_scorey_pred = [0, 1, 0, 1]y_true = [0, 1, 1, 0]print("ACC:",accuracy_score(y_true, y_pred))ACC: 0.5## Precision,Recall,F1-scorefrom sklearn import metricsy_pred = [0, 1, 0, 1]y_true = [0, 1, 1, 0]print("Precision",metrics.precision_score(y_true, y_pred))print("Recall",metrics.recall_score(y_true, y_pred))print("F1-score:",metrics.f1_score(y_true, y_pred))Precision 0.5Recall 0.5F1-score: 0.5## P-R曲线import matplotlib.pyplot as pltfrom sklearn.metrics import precision_recall_curvey_pred = [0, 1, 1, 0, 1, 1, 0, 1, 1, 1]y_true = [0, 1, 1, 0, 1, 0, 1, 1, 0, 1]precision, recall, thresholds = precision_recall_curve(y_true, y_pred)plt.plot(precision, recall)[] ## ROC曲线from sklearn.metrics import roc_curvey_pred = [0, 1, 1, 0, 1, 1, 0, 1, 1, 1]y_true = [0, 1, 1, 0, 1, 0, 1, 1, 0, 1]FPR,TPR,thresholds=roc_curve(y_true, y_pred)plt.title("ROC")plt.plot(FPR, TPR,"b")plt.plot([0,1],[0,1],"r--")plt.ylabel("TPR")plt.xlabel("FPR")Text(0.5, 0, "FPR")## AUCimport numpy as npfrom sklearn.metrics import roc_auc_scorey_true = np.array([0, 0, 1, 1])y_scores = np.array([0.1, 0.4, 0.35, 0.8])print("AUC socre:",roc_auc_score(y_true, y_scores))AUC socre: 0.75## KS值 在实际操作时往往使用ROC曲线配合求出KS值from sklearn.metrics import roc_curvey_pred = [0, 1, 1, 0, 1, 1, 0, 1, 1, 1]y_true = [0, 1, 1, 0, 1, 0, 1, 1, 1, 1]FPR,TPR,thresholds=roc_curve(y_true, y_pred)KS=abs(FPR-TPR).max()print("KS值：",KS)KS值： 0.5238095238095237评分卡是一张拥有分数刻度会让相应阈值的表。信用评分卡是用于用户信用的一张刻度表。以下代码是一个非标准评分卡的代码流程，用于刻画用户的信用评分。评分卡是金融风控中常用的一种对于用户信用进行刻画的手段哦！
#评分卡 不是标准评分卡def Score(prob,P0=600,PDO=20,badrate=None,goodrate=None): P0 = P0 PDO = PDO theta0 = badrate/goodrate B = PDO/np.log(2) A = P0 + B*np.log(2*theta0) score = A-B*np.log(prob/(1-prob)) return score目的：
1.EDA价值主要在于熟悉了解整个数据集的基本情况（缺失值，异常值），对数据集进行验证是否可以进行接下来的机器学习或者深度学习建模.
2.了解变量间的相互关系、变量与预测值之间的存在关系。
3.为特征工程做准备
2.1 代码示例2.1.1 导入数据分析及可视化过程需要的库import pandas as pdimport numpy as npimport matplotlib.pyplot as pltimport seaborn as snsimport datetimeimport warningswarnings.filterwarnings("ignore")/Users/exudingtao/opt/anaconda3/lib/python3.7/site-packages/statsmodels/tools/_testing.py:19: FutureWarning: pandas.util.testing is deprecated. Use the functions in the public API at pandas.testing instead. import pandas.util.testing as tm以上库都是pip install 安装就好，如果本机有python2,python3两个python环境傻傻分不清哪个的话,可以pip3 install 。或者直接在notebook中"!pip3 install ****"安装。
2.1.2 读取文件data_train = pd.read_csv("./train.csv")data_test_a = pd.read_csv("./testA.csv")data_train_sample = pd.read_csv("./train.csv",nrows=5)#设置chunksize参数，来控制每次迭代数据的大小chunker = pd.read_csv("./train.csv",chunksize=5)for item in chunker: print(type(item)) # print(len(item)) #5 data_test_a.shape(200000, 48)data_train.shape(800000, 47)data_train.columnsIndex(["id", "loanAmnt", "term", "interestRate", "installment", "grade", "subGrade", "employmentTitle", "employmentLength", "homeOwnership", "annualIncome", "verificationStatus", "issueDate", "isDefault", "purpose", "postCode", "regionCode", "dti", "delinquency_2years", "ficoRangeLow", "ficoRangeHigh", "openAcc", "pubRec", "pubRecBankruptcies", "revolBal", "revolUtil", "totalAcc", "initialListStatus", "applicationType", "earliesCreditLine", "title", "policyCode", "n0", "n1", "n2", "n2.1", "n4", "n5", "n6", "n7", "n8", "n9", "n10", "n11", "n12", "n13", "n14"], dtype="object")查看一下具体的列名，赛题理解部分已经给出具体的特征含义，这里方便阅读再给一下：
id为贷款清单分配的唯一信用证标识loanAmnt贷款金额term贷款期限（year）interestRate贷款利率installment分期付款金额grade贷款等级subGrade贷款等级之子级employmentTitle就业职称employmentLength就业年限（年）homeOwnership借款人在登记时提供的房屋所有权状况annualIncome年收入verificationStatus验证状态issueDate贷款发放的月份purpose借款人在贷款申请时的贷款用途类别postCode借款人在贷款申请中提供的邮政编码的前3位数字regionCode地区编码dti债务收入比delinquency_2years借款人过去2年信用档案中逾期30天以上的违约事件数ficoRangeLow借款人在贷款发放时的fico所属的下限范围ficoRangeHigh借款人在贷款发放时的fico所属的上限范围openAcc借款人信用档案中未结信用额度的数量pubRec贬损公共记录的数量pubRecBankruptcies公开记录清除的数量revolBal信贷周转余额合计revolUtil循环额度利用率，或借款人使用的相对于所有可用循环信贷的信贷金额totalAcc借款人信用档案中当前的信用额度总数initialListStatus贷款的初始列表状态applicationType表明贷款是个人申请还是与两个共同借款人的联合申请earliesCreditLine借款人最早报告的信用额度开立的月份title借款人提供的贷款名称policyCode公开可用的策略_代码=1新产品不公开可用的策略_代码=2n系列匿名特征匿名特征n0-n14，为一些贷款人行为计数特征的处理通过info()来熟悉数据类型
data_train.info()RangeIndex: 800000 entries, 0 to 799999Data columns (total 47 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 id 800000 non-null int64 1 loanAmnt 800000 non-null float64 2 term 800000 non-null int64 3 interestRate 800000 non-null float64 4 installment 800000 non-null float64 5 grade 800000 non-null object 6 subGrade 800000 non-null object 7 employmentTitle 799999 non-null float64 8 employmentLength 753201 non-null object 9 homeOwnership 800000 non-null int64 10 annualIncome 800000 non-null float64 11 verificationStatus 800000 non-null int64 12 issueDate 800000 non-null object 13 isDefault 800000 non-null int64 14 purpose 800000 non-null int64 15 postCode 799999 non-null float64 16 regionCode 800000 non-null int64 17 dti 799761 non-null float64 18 delinquency_2years 800000 non-null float64 19 ficoRangeLow 800000 non-null float64 20 ficoRangeHigh 800000 non-null float64 21 openAcc 800000 non-null float64 22 pubRec 800000 non-null float64 23 pubRecBankruptcies 799595 non-null float64 24 revolBal 800000 non-null float64 25 revolUtil 799469 non-null float64 26 totalAcc 800000 non-null float64 27 initialListStatus 800000 non-null int64 28 applicationType 800000 non-null int64 29 earliesCreditLine 800000 non-null object 30 title 799999 non-null float64 31 policyCode 800000 non-null float64 32 n0 759730 non-null float64 33 n1 759730 non-null float64 34 n2 759730 non-null float64 35 n2.1 759730 non-null float64 36 n4 766761 non-null float64 37 n5 759730 non-null float64 38 n6 759730 non-null float64 39 n7 759730 non-null float64 40 n8 759729 non-null float64 41 n9 759730 non-null float64 42 n10 766761 non-null float64 43 n11 730248 non-null float64 44 n12 759730 non-null float64 45 n13 759730 non-null float64 46 n14 759730 non-null float64dtypes: float64(33), int64(9), object(5)memory usage: 286.9+ MB 总体粗略的查看数据集各个特征的一些基本统计量
data_train.describe()data_train.head(3).append(data_train.tail(3))查看缺失值
print(f"There are {data_train.isnull().any().sum()} columns in train dataset with missing values.")There are 22 columns in train dataset with missing values.上面得到训练集有22列特征有缺失值，进一步查看缺失特征中缺失率大于50%的特征
have_null_fea_dict = (data_train.isnull().sum()/len(data_train)).to_dict()fea_null_moreThanHalf = {}for key,value in have_null_fea_dict.items(): if value > 0.5: fea_null_moreThanHalf[key] = valuefea_null_moreThanHalf{}具体的查看缺失特征及缺失率
#nan可视化missing = data_train.isnull().sum()/len(data_train)missing = missing[missing > 0]missing.sort_values(inplace=True)missing.plot.bar()Tips:比赛大杀器lgb模型可以自动处理缺失值，Task4模型会具体学习模型了解模型哦！
查看训练集测试集中特征属性只有一值的特征
one_value_fea = [col for col in data_train.columns if data_train[col].nunique() <= 1]one_value_fea_test = [col for col in data_test_a.columns if data_test_a[col].nunique() <= 1]one_value_fea["policyCode"]one_value_fea_test["policyCode"]print(f"There are {len(one_value_fea)} columns in train dataset with one unique value.")print(f"There are {len(one_value_fea_test)} columns in test dataset with one unique value.")There are 1 columns in train dataset with one unique value.There are 1 columns in test dataset with one unique value.47列数据中有22列都缺少数据，这在现实世界中很正常。‘policyCode’具有一个唯一值（或全部缺失）。有很多连续变量和一些分类变量。
2.1.4 查看特征的数值类型有哪些，对象类型有哪些特征一般都是由类别型特征和数值型特征组成，而数值型特征又分为连续型和离散型。类别型特征有时具有非数值关系，有时也具有数值关系。比如‘grade’中的等级A，B，C等，是否只是单纯的分类，还是A优于其他要结合业务判断。数值型特征本是可以直接入模的，但往往风控人员要对其做分箱，转化为WOE编码进而做标准评分卡等操作。从模型效果上来看，特征分箱主要是为了降低变量的复杂性，减少变量噪音对模型的影响，提高自变量和因变量的相关度。从而使模型更加稳定。numerical_fea = list(data_train.select_dtypes(exclude=["object"]).columns)category_fea = list(filter(lambda x: x not in numerical_fea,list(data_train.columns)))numerical_feacategory_fea["grade", "subGrade", "employmentLength", "issueDate", "earliesCreditLine"]data_train.grade0 E1 D2 D3 A4 C ..799995 C799996 A799997 C799998 A799999 BName: grade, Length: 800000, dtype: object#过滤数值型类别特征def get_numerical_serial_fea(data,feas): numerical_serial_fea = [] numerical_noserial_fea = [] for fea in feas: temp = data[fea].nunique() if temp <= 10: numerical_noserial_fea.append(fea) continue numerical_serial_fea.append(fea) return numerical_serial_fea,numerical_noserial_feanumerical_serial_fea,numerical_noserial_fea = get_numerical_serial_fea(data_train,numerical_fea)numerical_serial_feanumerical_noserial_fea["term", "homeOwnership", "verificationStatus", "isDefault", "initialListStatus", "applicationType", "policyCode", "n11", "n12"]data_train["term"].value_counts()#离散型变量3 6069025 193098Name: term, dtype: int64data_train["homeOwnership"].value_counts()#离散型变量0 3957321 3176602 863093 1855 814 33Name: homeOwnership, dtype: int64data_train["verificationStatus"].value_counts()#离散型变量1 3098102 2489680 241222Name: verificationStatus, dtype: int64data_train["initialListStatus"].value_counts()#离散型变量0 4664381 333562Name: initialListStatus, dtype: int64data_train["applicationType"].value_counts()#离散型变量0 7845861 15414Name: applicationType, dtype: int64data_train["policyCode"].value_counts()#离散型变量，无用，全部一个值1.0 800000Name: policyCode, dtype: int64data_train["n11"].value_counts()#离散型变量，相差悬殊，用不用再分析0.0 7296821.0 5402.0 244.0 13.0 1Name: n11, dtype: int64data_train["n12"].value_counts()#离散型变量，相差悬殊，用不用再分析0.0 7573151.0 22812.0 1153.0 164.0 3Name: n12, dtype: int64#每个数字特征得分布可视化f = pd.melt(data_train, value_vars=numerical_serial_fea)g = sns.FacetGrid(f, col="variable", col_wrap=2, sharex=False, sharey=False)g = g.map(sns.distplot, "value")#Ploting Transaction Amount Values Distributionplt.figure(figsize=(16,12))plt.suptitle("Transaction Values Distribution", fontsize=22)plt.subplot(221)sub_plot_1 = sns.distplot(data_train["loanAmnt"])sub_plot_1.set_title("loanAmnt Distribuition", fontsize=18)sub_plot_1.set_xlabel("")sub_plot_1.set_ylabel("Probability", fontsize=15)plt.subplot(222)sub_plot_2 = sns.distplot(np.log(data_train["loanAmnt"]))sub_plot_2.set_title("loanAmnt (Log) Distribuition", fontsize=18)sub_plot_2.set_xlabel("")sub_plot_2.set_ylabel("Probability", fontsize=15)Text(0, 0.5, "Probability")category_fea["grade", "subGrade", "employmentLength", "issueDate", "earliesCreditLine"]data_train["grade"].value_counts()B 233690C 227118A 139661D 119453E 55661F 19053G 5364Name: grade, dtype: int64data_train["subGrade"].value_counts()C1 50763B4 49516B5 48965B3 48600C2 47068C3 44751C4 44272B2 44227B1 42382C5 40264A5 38045A4 30928D1 30538D2 26528A1 25909D3 23410A3 22655A2 22124D4 21139D5 17838E1 14064E2 12746E3 10925E4 9273E5 8653F1 5925F2 4340F3 3577F4 2859F5 2352G1 1759G2 1231G3 978G4 751G5 645Name: subGrade, dtype: int64data_train["employmentLength"].value_counts()10+ years 2627532 years 72358< 1 year 642373 years 641521 year 524895 years 501024 years 479856 years 372548 years 361927 years 354079 years 30272Name: employmentLength, dtype: int64data_train["issueDate"].value_counts()2016-03-01 290662015-10-01 255252015-07-01 244962015-12-01 232452014-10-01 21461 ... 2007-08-01 232007-07-01 212008-09-01 192007-09-01 72007-06-01 1Name: issueDate, Length: 139, dtype: int64data_train["earliesCreditLine"].value_counts()Aug-2001 5567Sep-2003 5403Aug-2002 5403Oct-2001 5258Aug-2000 5246 ... May-1960 1Apr-1958 1Feb-1960 1Aug-1946 1Mar-1958 1Name: earliesCreditLine, Length: 720, dtype: int64data_train["isDefault"].value_counts()0 6403901 159610Name: isDefault, dtype: int64plt.figure(figsize=(8, 8))sns.barplot(data_train["employmentLength"].value_counts(dropna=False)[:20], data_train["employmentLength"].value_counts(dropna=False).keys()[:20])plt.show()train_loan_fr = data_train.loc[data_train["isDefault"] == 1]train_loan_nofr = data_train.loc[data_train["isDefault"] == 0]fig, ((ax1, ax2), (ax3, ax4)) = plt.subplots(2, 2, figsize=(15, 8))train_loan_fr.groupby("grade")["grade"].count().plot(kind="barh", ax=ax1, title="Count of grade fraud")train_loan_nofr.groupby("grade")["grade"].count().plot(kind="barh", ax=ax2, title="Count of grade non-fraud")train_loan_fr.groupby("employmentLength")["employmentLength"].count().plot(kind="barh", ax=ax3, title="Count of employmentLength fraud")train_loan_nofr.groupby("employmentLength")["employmentLength"].count().plot(kind="barh", ax=ax4, title="Count of employmentLength non-fraud")plt.show()fig, ((ax1, ax2)) = plt.subplots(1, 2, figsize=(15, 6))data_train.loc[data_train["isDefault"] == 1] \ ["loanAmnt"].apply(np.log) \ .plot(kind="hist", bins=100, title="Log Loan Amt - Fraud", color="r", xlim=(-3, 10), ax= ax1)data_train.loc[data_train["isDefault"] == 0] \ ["loanAmnt"].apply(np.log) \ .plot(kind="hist", bins=100, title="Log Loan Amt - Not Fraud", color="b", xlim=(-3, 10), ax=ax2)total = len(data_train)total_amt = data_train.groupby(["isDefault"])["loanAmnt"].sum().sum()plt.figure(figsize=(12,5))plt.subplot(121)##1代表行，2代表列，所以一共有2个图，1代表此时绘制第一个图。plot_tr = sns.countplot(x="isDefault",data=data_train)#data_train‘isDefault’这个特征每种类别的数量**plot_tr.set_title("Fraud Loan Distribution \n 0: good user | 1: bad user", fontsize=14)plot_tr.set_xlabel("Is fraud by count", fontsize=16)plot_tr.set_ylabel("Count", fontsize=16)for p in plot_tr.patches: height = p.get_height() plot_tr.text(p.get_x()+p.get_width()/2., height + 3, "{:1.2f}%".format(height/total*100), ha="center", fontsize=15) percent_amt = (data_train.groupby(["isDefault"])["loanAmnt"].sum())percent_amt = percent_amt.reset_index()plt.subplot(122)plot_tr_2 = sns.barplot(x="isDefault", y="loanAmnt", dodge=True, data=percent_amt)plot_tr_2.set_title("Total Amount in loanAmnt \n 0: good user | 1: bad user", fontsize=14)plot_tr_2.set_xlabel("Is fraud by percent", fontsize=16)plot_tr_2.set_ylabel("Total Loan Amount Scalar", fontsize=16)for p in plot_tr_2.patches: height = p.get_height() plot_tr_2.text(p.get_x()+p.get_width()/2., height + 3, "{:1.2f}%".format(height/total_amt * 100), ha="center", fontsize=15) #转化成时间格式 issueDateDT特征表示数据日期离数据集中日期最早的日期（2007-06-01）的天数data_train["issueDate"] = pd.to_datetime(data_train["issueDate"],format="%Y-%m-%d")startdate = datetime.datetime.strptime("2007-06-01", "%Y-%m-%d")data_train["issueDateDT"] = data_train["issueDate"].apply(lambda x: x-startdate).dt.days#转化成时间格式data_test_a["issueDate"] = pd.to_datetime(data_train["issueDate"],format="%Y-%m-%d")startdate = datetime.datetime.strptime("2007-06-01", "%Y-%m-%d")data_test_a["issueDateDT"] = data_test_a["issueDate"].apply(lambda x: x-startdate).dt.daysplt.hist(data_train["issueDateDT"], label="train");plt.hist(data_test_a["issueDateDT"], label="test");plt.legend();plt.title("Distribution of issueDateDT dates");#train 和 test issueDateDT 日期有重叠 所以使用基于时间的分割进行验证是不明智的#透视图 索引可以有多个，“columns（列）”是可选的，聚合函数aggfunc最后是被应用到了变量“values”中你所列举的项目上。pivot = pd.pivot_table(data_train, index=["grade"], columns=["issueDateDT"], values=["loanAmnt"], aggfunc=np.sum)pivotimport pandas_profilingpfr = pandas_profiling.ProfileReport(data_train)pfr.to_file("./example.html")数据探索性分析是我们初步了解数据，熟悉数据为特征工程做准备的阶段，甚至很多时候EDA阶段提取出来的特征可以直接当作规则来用。可见EDA的重要性，这个阶段的主要工作还是借助于各个简单的统计量来对数据整体的了解，分析各个类型变量相互之间的关系，以及用合适的图形可视化出来直观观察。希望本节内容能给初学者带来帮助，更期待各位学习者对其中的不足提出建议。
3.特征工程import pandas as pdimport numpy as npimport matplotlib.pyplot as pltimport seaborn as snsimport datetimefrom tqdm import tqdmfrom sklearn.preprocessing import LabelEncoderfrom sklearn.feature_selection import SelectKBestfrom sklearn.feature_selection import chi2from sklearn.preprocessing import MinMaxScalerimport xgboost as xgbimport lightgbm as lgbfrom catboost import CatBoostRegressorimport warningsfrom sklearn.model_selection import StratifiedKFold, KFoldfrom sklearn.metrics import accuracy_score, f1_score, roc_auc_score, log_losswarnings.filterwarnings("ignore")data_train =pd.read_csv("../train.csv")data_test_a = pd.read_csv("../testA.csv")首先我们查找出数据中的对象特征和数值特征
numerical_fea = list(data_train.select_dtypes(exclude=["object"]).columns)category_fea = list(filter(lambda x: x not in numerical_fea,list(data_train.columns)))label = "isDefault"numerical_fea.remove(label)在比赛中数据预处理是必不可少的一部分，对于缺失值的填充往往会影响比赛的结果，在比赛中不妨尝试多种填充然后比较结果选择结果最优的一种；比赛数据相比真实场景的数据相对要“干净”一些，但是还是会有一定的“脏”数据存在，清洗一些异常值往往会获得意想不到的效果。
把所有缺失值替换为指定的值0
data_train = data_train.fillna(0)
向用缺失值上面的值替换缺失值
data_train = data_train.fillna(axis=0,method="ffill")
纵向用缺失值下面的值替换缺失值,且设置最多只填充两个连续的缺失值
data_train = data_train.fillna(axis=0,method="bfill",limit=2)
#查看缺失值情况data_train.isnull().sum()#按照平均数填充数值型特征data_train[numerical_fea] = data_train[numerical_fea].fillna(data_train[numerical_fea].median())data_test_a[numerical_fea] = data_test_a[numerical_fea].fillna(data_train[numerical_fea].median())#按照众数填充类别型特征data_train[category_fea] = data_train[category_fea].fillna(data_train[category_fea].mode())data_test_a[category_fea] = data_test_a[category_fea].fillna(data_train[category_fea].mode())data_train.isnull().sum()#查看类别特征category_fea["grade", "subGrade", "employmentLength", "issueDate", "earliesCreditLine"]#转化成时间格式for data in [data_train, data_test_a]: data["issueDate"] = pd.to_datetime(data["issueDate"],format="%Y-%m-%d") startdate = datetime.datetime.strptime("2007-06-01", "%Y-%m-%d") #构造时间特征 data["issueDateDT"] = data["issueDate"].apply(lambda x: x-startdate).dt.daysdata_train["employmentLength"].value_counts(dropna=False).sort_index()1 year 5248910+ years 2627532 years 723583 years 641524 years 479855 years 501026 years 372547 years 354078 years 361929 years 30272< 1 year 64237NaN 46799Name: employmentLength, dtype: int64def employmentLength_to_int(s): if pd.isnull(s): return s else: return np.int8(s.split()[0])for data in [data_train, data_test_a]: data["employmentLength"].replace(to_replace="10+ years", value="10 years", inplace=True) data["employmentLength"].replace("< 1 year", "0 years", inplace=True) data["employmentLength"] = data["employmentLength"].apply(employmentLength_to_int)data["employmentLength"].value_counts(dropna=False).sort_index()0.0 159891.0 131822.0 182073.0 160114.0 118335.0 125436.0 93287.0 88238.0 89769.0 759410.0 65772NaN 11742Name: employmentLength, dtype: int64data_train["earliesCreditLine"].sample(5)519915 Sep-2002564368 Dec-1996768209 May-2004453092 Nov-1995763866 Sep-2000Name: earliesCreditLine, dtype: objectfor data in [data_train, data_test_a]: data["earliesCreditLine"] = data["earliesCreditLine"].apply(lambda s: int(s[-4:]))# 部分类别特征cate_features = ["grade", "subGrade", "employmentTitle", "homeOwnership", "verificationStatus", "purpose", "postCode", "regionCode", \ "applicationType", "initialListStatus", "title", "policyCode"]for f in cate_features: print(f, "类型数：", data[f].nunique())grade 类型数： 7subGrade 类型数： 35employmentTitle 类型数： 79282homeOwnership 类型数： 6verificationStatus 类型数： 3purpose 类型数： 14postCode 类型数： 889regionCode 类型数： 51applicationType 类型数： 2initialListStatus 类型数： 2title 类型数： 12058policyCode 类型数： 1像等级这种类别特征，是有优先级的可以labelencode或者自映射
for data in [data_train, data_test_a]: data["grade"] = data["grade"].map({"A":1,"B":2,"C":3,"D":4,"E":5,"F":6,"G":7})# 类型数在2之上，又不是高维稀疏的,且纯分类特征for data in [data_train, data_test_a]: data = pd.get_dummies(data, columns=["subGrade", "homeOwnership", "verificationStatus", "purpose", "regionCode"], drop_first=True)在统计学中，如果一个数据分布近似正态，那么大约 68% 的数据值会在均值的一个标准差范围内，大约 95% 会在两个标准差范围内，大约 99.7% 会在三个标准差范围内。
def find_outliers_by_3segama(data,fea): data_std = np.std(data[fea]) data_mean = np.mean(data[fea]) outliers_cut_off = data_std * 3 lower_rule = data_mean - outliers_cut_off upper_rule = data_mean + outliers_cut_off data[fea+"_outliers"] = data[fea].apply(lambda x:str("异常值") if x > upper_rule or x < lower_rule else "正常值") return datadata_train = data_train.copy()for fea in numerical_fea: data_train = find_outliers_by_3segama(data_train,fea) print(data_train[fea+"_outliers"].value_counts()) print(data_train.groupby(fea+"_outliers")["isDefault"].sum()) print("*"*10)#删除异常值for fea in numerical_fea: data_train = data_train[data_train[fea+"_outliers"]=="正常值"] data_train = data_train.reset_index(drop=True) 特征分箱的目的：
从模型效果上来看，特征分箱主要是为了降低变量的复杂性，减少变量噪音对模型的影响，提高自变量和因变量的相关度。从而使模型更加稳定。数据分桶的对象：
将连续变量离散化将多状态的离散变量合并成少状态分箱的原因：
数据的特征内的值跨度可能比较大，对有监督和无监督中如k-均值聚类它使用欧氏距离作为相似度函数来测量数据点之间的相似度。都会造成大吃小的影响，其中一种解决方法是对计数值进行区间量化即数据分桶也叫做数据分箱，然后使用量化后的结果。分箱的优点：
处理缺失值：当数据源可能存在缺失值，此时可以把null单独作为一个分箱。处理异常值：当数据中存在离群点时，可以把其通过分箱离散化处理，从而提高变量的鲁棒性（抗干扰能力）。例如，age若出现200这种异常值，可分入“age > 60”这个分箱里，排除影响。业务解释性：我们习惯于线性判断变量的作用，当x越来越大，y就越来越大。但实际x与y之间经常存在着非线性关系，此时可经过WOE变换。特别要注意一下分箱的基本原则：
（1）最小分箱占比不低于5%（2）箱内不能全部是好客户（3）连续箱单调固定宽度分箱当数值横跨多个数量级时，最好按照 10 的幂（或任何常数的幂）来进行分组：09、1099、100999、10009999，等等。固定宽度分箱非常容易计算，但如果计数值中有比较大的缺口，就会产生很多没有任何数据的空箱子。
#通过除法映射到间隔均匀的分箱中，每个分箱的取值范围都是loanAmnt/1000data["loanAmnt_bin1"] = np.floor_divide(data["loanAmnt"], 1000)##通过对数函数映射到指数宽度分箱data["loanAmnt_bin2"] = np.floor(np.log10(data["loanAmnt"]))data["loanAmnt_bin3"] = pd.qcut(data["loanAmnt"], 10, labels=False)for col in ["grade", "subGrade"]: temp_dict = data_train.groupby([col])["isDefault"].agg(["mean"]).reset_index().rename(columns={"mean": col + "_target_mean"}) temp_dict.index = temp_dict[col].values temp_dict = temp_dict[col + "_target_mean"].to_dict() data_train[col + "_target_mean"] = data_train[col].map(temp_dict) data_test_a[col + "_target_mean"] = data_test_a[col].map(temp_dict)# 其他衍生变量 mean 和 stdfor df in [data_train, data_test_a]: for item in ["n0","n1","n2","n2.1","n4","n5","n6","n7","n8","n9","n10","n11","n12","n13","n14"]: df["grade_to_mean_" + item] = df["grade"] / df.groupby([item])["grade"].transform("mean") df["grade_to_std_" + item] = df["grade"] / df.groupby([item])["grade"].transform("std")这里给出一些特征交互的思路，但特征和特征间的交互衍生出新的特征还远远不止于此，抛砖引玉，希望大家多多探索。请学习者尝试其他的特征交互方法。
3.5 特征编码3.5.1labelEncode 直接放入树模型中#label-encode:subGrade,postCode,title#高维类别特征需要进行转换for col in tqdm(["employmentTitle", "postCode", "title","subGrade"]): le = LabelEncoder() le.fit(list(data_train[col].astype(str).values) + list(data_test_a[col].astype(str).values)) data_train[col] = le.transform(list(data_train[col].astype(str).values)) data_test_a[col] = le.transform(list(data_test_a[col].astype(str).values))print("Label Encoding 完成")100%|██████████| 4/4 [00:08<00:00, 2.04s/it]Label Encoding 完成# 举例归一化过程#伪代码for fea in [要归一化的特征列表]： data[fea] = ((data[fea] - np.min(data[fea])) / (np.max(data[fea]) - np.min(data[fea])))特征选择的方法：
1 Filter方差选择法相关系数法（pearson 相关系数）卡方检验互信息法2 Wrapper （RFE）递归特征消除法3 Embedded基于惩罚项的特征选择法基于树模型的特征选择3.6.1Filter（方差选择法、相关系数法）基于特征间的关系进行筛选方差选择法
方差选择法中，先要计算各个特征的方差，然后根据设定的阈值，选择方差大于阈值的特征from sklearn.feature_selection import VarianceThreshold#其中参数threshold为方差的阈值VarianceThreshold(threshold=3).fit_transform(train,target_train)相关系数法
Pearson 相关系数皮尔森相关系数是一种最简单的，可以帮助理解特征和响应变量之间关系的方法，该方法衡量的是变量之间的线性相关性。结果的取值区间为 [-1，1] ， -1 表示完全的负相关， +1表示完全的正相关，0 表示没有线性相关。from sklearn.feature_selection import SelectKBestfrom scipy.stats import pearsonr#选择K个最好的特征，返回选择特征后的数据#第一个参数为计算评估特征是否好的函数，该函数输入特征矩阵和目标向量，#输出二元组（评分，P值）的数组，数组第i项为第i个特征的评分和P值。在此定义为计算相关系数#参数k为选择的特征个数SelectKBest(k=5).fit_transform(train,target_train)卡方检验
经典的卡方检验是用于检验自变量对因变量的相关性。 假设自变量有N种取值，因变量有M种取值，考虑自变量等于i且因变量等于j的样本频数的观察值与期望的差距。 其统计量如下： χ2=∑(A−T)2T，其中A为实际值，T为理论值(注：卡方只能运用在正定矩阵上，否则会报错Input X must be non-negative)from sklearn.feature_selection import SelectKBestfrom sklearn.feature_selection import chi2#参数k为选择的特征个数SelectKBest(chi2, k=5).fit_transform(train,target_train)互信息法
经典的互信息也是评价自变量对因变量的相关性的。 在feature_selection库的SelectKBest类结合最大信息系数法可以用于选择特征，相关代码如下：from sklearn.feature_selection import SelectKBestfrom minepy import MINE#由于MINE的设计不是函数式的，定义mic方法将其为函数式的，#返回一个二元组，二元组的第2项设置成固定的P值0.5def mic(x, y): m = MINE() m.compute_score(x, y) return (m.mic(), 0.5)#参数k为选择的特征个数SelectKBest(lambda X, Y: array(map(lambda x:mic(x, Y), X.T)).T, k=2).fit_transform(train,target_train)from sklearn.feature_selection import RFEfrom sklearn.linear_model import LogisticRegression#递归特征消除法，返回特征选择后的数据#参数estimator为基模型#参数n_features_to_select为选择的特征个数RFE(estimator=LogisticRegression(), n_features_to_select=2).fit_transform(train,target_train)from sklearn.feature_selection import SelectFromModelfrom sklearn.linear_model import LogisticRegression#带L1惩罚项的逻辑回归作为基模型的特征选择SelectFromModel(LogisticRegression(penalty="l1", C=0.1)).fit_transform(train,target_train)from sklearn.feature_selection import SelectFromModelfrom sklearn.ensemble import GradientBoostingClassifier#GBDT作为基模型的特征选择SelectFromModel(GradientBoostingClassifier()).fit_transform(train,target_train)本数据集中我们删除非入模特征后，并对缺失值填充，然后用计算协方差的方式看一下特征间相关性，然后进行模型训练
# 删除不需要的数据for data in [data_train, data_test_a]: data.drop(["issueDate","id"], axis=1,inplace=True)"纵向用缺失值上面的值替换缺失值"data_train = data_train.fillna(axis=0,method="ffill")x_train = data_train.drop(["isDefault","id"], axis=1)#计算协方差data_corr = x_train.corrwith(data_train.isDefault) #计算相关性result = pd.DataFrame(columns=["features", "corr"])result["features"] = data_corr.indexresult["corr"] = data_corr.values# 当然也可以直接看图data_numeric = data_train[numerical_fea]correlation = data_numeric.corr()f , ax = plt.subplots(figsize = (7, 7))plt.title("Correlation of Numeric Features with Price",y=1,size=16)sns.heatmap(correlation,square = True, vmax=0.8)features = [f for f in data_train.columns if f not in ["id","issueDate","isDefault"] and "_outliers" not in f]x_train = data_train[features]x_test = data_test_a[features]y_train = data_train["isDefault"]def cv_model(clf, train_x, train_y, test_x, clf_name): folds = 5 seed = 2020 kf = KFold(n_splits=folds, shuffle=True, random_state=seed) train = np.zeros(train_x.shape[0]) test = np.zeros(test_x.shape[0]) cv_scores = [] for i, (train_index, valid_index) in enumerate(kf.split(train_x, train_y)): print("************************************ {} ************************************".format(str(i+1))) trn_x, trn_y, val_x, val_y = train_x.iloc[train_index], train_y[train_index], train_x.iloc[valid_index], train_y[valid_index] if clf_name == "lgb": train_matrix = clf.Dataset(trn_x, label=trn_y) valid_matrix = clf.Dataset(val_x, label=val_y) params = { "boosting_type": "gbdt", "objective": "binary", "metric": "auc", "min_child_weight": 5, "num_leaves": 2 ** 5, "lambda_l2": 10, "feature_fraction": 0.8, "bagging_fraction": 0.8, "bagging_freq": 4, "learning_rate": 0.1, "seed": 2020, "nthread": 28, "n_jobs":24, "silent": True, "verbose": -1, } model = clf.train(params, train_matrix, 50000, valid_sets=[train_matrix, valid_matrix], verbose_eval=200,early_stopping_rounds=200) val_pred = model.predict(val_x, num_iteration=model.best_iteration) test_pred = model.predict(test_x, num_iteration=model.best_iteration) # print(list(sorted(zip(features, model.feature_importance("gain")), key=lambda x: x[1], reverse=True))[:20]) if clf_name == "xgb": train_matrix = clf.DMatrix(trn_x , label=trn_y) valid_matrix = clf.DMatrix(val_x , label=val_y) params = {"booster": "gbtree", "objective": "binary:logistic", "eval_metric": "auc", "gamma": 1, "min_child_weight": 1.5, "max_depth": 5, "lambda": 10, "subsample": 0.7, "colsample_bytree": 0.7, "colsample_bylevel": 0.7, "eta": 0.04, "tree_method": "exact", "seed": 2020, "nthread": 36, "silent": True, } watchlist = [(train_matrix, "train"),(valid_matrix, "eval")] model = clf.train(params, train_matrix, num_boost_round=50000, evals=watchlist, verbose_eval=200, early_stopping_rounds=200) val_pred = model.predict(valid_matrix, ntree_limit=model.best_ntree_limit) test_pred = model.predict(test_x , ntree_limit=model.best_ntree_limit) if clf_name == "cat": params = {"learning_rate": 0.05, "depth": 5, "l2_leaf_reg": 10, "bootstrap_type": "Bernoulli", "od_type": "Iter", "od_wait": 50, "random_seed": 11, "allow_writing_files": False} model = clf(iterations=20000, **params) model.fit(trn_x, trn_y, eval_set=(val_x, val_y), cat_features=[], use_best_model=True, verbose=500) val_pred = model.predict(val_x) test_pred = model.predict(test_x) train[valid_index] = val_pred test = test_pred / kf.n_splits cv_scores.append(roc_auc_score(val_y, val_pred)) print(cv_scores) print("%s_scotrainre_list:" % clf_name, cv_scores) print("%s_score_mean:" % clf_name, np.mean(cv_scores)) print("%s_score_std:" % clf_name, np.std(cv_scores)) return train, testdef lgb_model(x_train, y_train, x_test): lgb_train, lgb_test = cv_model(lgb, x_train, y_train, x_test, "lgb") return lgb_train, lgb_testdef xgb_model(x_train, y_train, x_test): xgb_train, xgb_test = cv_model(xgb, x_train, y_train, x_test, "xgb") return xgb_train, xgb_testdef cat_model(x_train, y_train, x_test): cat_train, cat_test = cv_model(CatBoostRegressor, x_train, y_train, x_test, "cat")lgb_train, lgb_test = lgb_model(x_train, y_train, x_test)************************************ 1 ************************************Training until validation scores don"t improve for 200 rounds[200]training"s auc: 0.749225valid_1"s auc: 0.729679[400]training"s auc: 0.765075valid_1"s auc: 0.730496[600]training"s auc: 0.778745valid_1"s auc: 0.730435Early stopping, best iteration is:[455]training"s auc: 0.769202valid_1"s auc: 0.730686[0.7306859913754798]************************************ 2 ************************************Training until validation scores don"t improve for 200 rounds[200]training"s auc: 0.749221valid_1"s auc: 0.731315[400]training"s auc: 0.765117valid_1"s auc: 0.731658[600]training"s auc: 0.778542valid_1"s auc: 0.731333Early stopping, best iteration is:[407]training"s auc: 0.765671valid_1"s auc: 0.73173[0.7306859913754798, 0.7317304414673989]************************************ 3 ************************************Training until validation scores don"t improve for 200 rounds[200]training"s auc: 0.748436valid_1"s auc: 0.732775[400]training"s auc: 0.764216valid_1"s auc: 0.733173Early stopping, best iteration is:[386]training"s auc: 0.763261valid_1"s auc: 0.733261[0.7306859913754798, 0.7317304414673989, 0.7332610441015461]************************************ 4 ************************************Training until validation scores don"t improve for 200 rounds[200]training"s auc: 0.749631valid_1"s auc: 0.728327[400]training"s auc: 0.765139valid_1"s auc: 0.728845Early stopping, best iteration is:[286]training"s auc: 0.756978valid_1"s auc: 0.728976[0.7306859913754798, 0.7317304414673989, 0.7332610441015461, 0.7289759386807912]************************************ 5 ************************************Training until validation scores don"t improve for 200 rounds[200]training"s auc: 0.748414valid_1"s auc: 0.732727[400]training"s auc: 0.763727valid_1"s auc: 0.733531[600]training"s auc: 0.777489valid_1"s auc: 0.733566Early stopping, best iteration is:[524]training"s auc: 0.772372valid_1"s auc: 0.733772[0.7306859913754798, 0.7317304414673989, 0.7332610441015461, 0.7289759386807912, 0.7337723979789789]lgb_scotrainre_list: [0.7306859913754798, 0.7317304414673989, 0.7332610441015461, 0.7289759386807912, 0.7337723979789789]lgb_score_mean: 0.7316851627208389lgb_score_std: 0.0017424259863954693testA_result = pd.read_csv("../testA_result.csv")roc_auc_score(testA_result["isDefault"].values, lgb_test)0.7290917729487896特征工程是机器学习，甚至是深度学习中最为重要的一部分，在实际应用中往往也是所花费时间最多的一步。各种算法书中对特征工程部分的讲解往往少得可怜，因为特征工程和具体的数据结合的太紧密，很难系统地覆盖所有场景。本章主要是通过一些常用的方法来做介绍，例如缺失值异常值的处理方法详细对任何数据集来说都是适用的。但对于分箱等操作本章给出了具体的几种思路，需要读者自己探索。在特征工程中比赛和具体的应用还是有所不同的，在实际的金融风控评分卡制作过程中，由于强调特征的可解释性，特征分箱尤其重要。
项目链接以及码源数据挖掘专栏
数据挖掘实践（金融风控）：金融风控之贷款违约预测挑战赛(上篇)
数据挖掘实践（金融风控）：金融风控之贷款违约预测挑战赛(下篇)
标签：
