Sklearn for preprocessing¶
sklearn can be a good option for preprocessing data
In [1]:
import numpy as np
import pandas as pd
from sklearn.preprocessing import MinMaxScaler
Normalizing¶
(x-x_min)/(x_max - x_min)
In [2]:
data = np.random.randint(0,100,(10,2))
data
Out[2]:
In [3]:
scale_model = MinMaxScaler()
scale_model
Out[3]:
In [4]:
#fit and transform are usually separate, because we fit on training data and tranform on test data
scale_model.fit_transform(data)
Out[4]:
Splitting data¶
In [5]:
features = np.random.randint(0,101,(50,3))
labels = np.random.randint(0,5,(50,1))
data = np.concatenate((features,labels),axis=1)
In [6]:
df = pd.DataFrame(data,columns = ['f1','f2','f3','labels'])
df[:10]
Out[6]:
In [7]:
from sklearn.model_selection import train_test_split
In [8]:
X_train, X_test, y_train, y_test = train_test_split(features,labels,test_size=0.33,random_state=42)
In [9]:
X_train.shape, y_train.shape, X_test.shape, y_test.shape
Out[9]:
Generating random datasets¶
In [10]:
from sklearn.datasets import make_blobs
In [13]:
data = make_blobs(n_samples=20,n_features=2,centers=2,random_state=75) #centers=[number of classes]
data
Out[13]:
In [14]:
import matplotlib.pyplot as plt
%matplotlib inline
plt.scatter(data[0][:,0],data[0][:,1],c=data[1])
Out[14]:
One Hot Encoding¶
In [4]:
diabetes = pd.read_csv("files/pima-indians-diabetes.csv")
diabetes.head()
Out[4]:
Depricated:
Use Label encoder to change String labels to Integer. And then use OneHotEncoder
In [2]:
from sklearn.preprocessing import LabelEncoder, OneHotEncoder
Change characerts in Groups to Integers with LabelEncoder
In [5]:
cols = diabetes.columns
arr = diabetes.values
le = LabelEncoder()
le_fit = le.fit(arr[:,9]) #fit is used on test data
arr[:,9] = le_fit.transform(arr[:,9])
pd.DataFrame(arr,columns=cols).head()
Out[5]:
Change Age and Group to OneHotEncoding
In [6]:
oe = OneHotEncoder(categorical_features=[7,9]) #Column numbers on which transformation should be applied
oe_fit = oe.fit(arr) #fir is used on test data
arr_new = oe_fit.transform(arr).toarray()
pd.DataFrame(arr_new).head()
Out[6]:
PROBLEM: The Categories considered for AGE are only the ones encountered in training data. If we get a new age, the code breaks
In [7]:
oe_fit.categories_
Out[7]:
Prefered method :
Using Column Transformers. Also eliminated the need for Label encoders
In [8]:
from sklearn.compose import ColumnTransformer
In [9]:
cols = diabetes.columns
arr = diabetes.values
#Specify OneHOtEncoders separately for each column. To manually specify categories, use categories option
#9 ==> Group
#7 ==> Age
colT = ColumnTransformer([("1",OneHotEncoder(),[9]),("2",OneHotEncoder(categories=[np.arange(0,100,1)]),[7])])
In [11]:
colT_fit = colT.fit(arr)
Now you can see all the specified categories have been encoded
In [13]:
np.array(colT_fit.get_feature_names())
Out[13]:
In [14]:
tr = colT_fit.transform(arr).toarray() #Returns just the transformed cols
tr.shape, arr.shape
Out[14]:
ColumnTransforms returns only the transformed columns. So we have to drop the old columns from the original array and concatenate it with the transformed columns
In [15]:
arr_new = np.concatenate((arr[:,[0,1,2,3,4,5,6,8]],tr),axis=1) #specify all columns excetp 7 and 9
arr_new.shape
Out[15]:
In [16]:
pd.DataFrame(arr_new).head()
Out[16]: