Reference: [Scientific Python Stanford University](https://colab.research.google.com/drive/188vQVvDIH-9kbd4Un6BcPERKRztxIORe)"
],
"id": "_kK4RhV_IVn5"
},
{
"cell_type": "markdown",
"metadata": {
"id": "JAZVbXHdIpCT"
},
"source": [
"[Scikit-learn](https://scikit-learn.org/stable/) is a library that allows you to do machine learning, that is, make predictions from data, in Python. There are four basic machine learning tasks:\n",
"\n",
" 1. Regression: predict a number from datapoints, given datapoints and corresponding numbers\n",
" 2. Classification: predict a category from datapoints, given datapoints and corresponding numbers\n",
" 3. Clustering: predict a category from datapoints, given only datapoints\n",
" 4. Dimensionality reduction: make datapoints lower-dimensional so that we can visualize the data\n",
"\n",
"Here is a [handy flowchart](https://scikit-learn.org/stable/tutorial/machine_learning_map/index.html) of when to use each technique."
],
"id": "JAZVbXHdIpCT"
},
{
"cell_type": "markdown",
"metadata": {
"id": "iCkDyIHpJF3-"
},
"source": [
""
],
"id": "iCkDyIHpJF3-"
},
{
"cell_type": "code",
"metadata": {
"scrolled": true,
"id": "UvOyrbRBIOix"
},
"source": [
"import pandas as pd\n",
"import numpy as np\n",
"import matplotlib.pyplot as plt"
],
"id": "UvOyrbRBIOix",
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "j6k2NWBgIOiy"
},
"source": [
"#### Regression\n",
"Abalone are a type of edible marine snail, and they have internal rings that correspond to their age (like trees). In the following, we will use a dataset of [abalone measurements](https://archive.ics.uci.edu/ml/datasets/abalone). It has the following fields:\n",
"\n",
" Sex / nominal / -- / M, F, and I (infant) \n",
" Length / continuous / mm / Longest shell measurement \n",
" Diameter\t/ continuous / mm / perpendicular to length \n",
" Height / continuous / mm / with meat in shell \n",
" Whole weight / continuous / grams / whole abalone \n",
" Shucked weight / continuous\t/ grams / weight of meat \n",
" Viscera weight / continuous / grams / gut weight (after bleeding) \n",
" Shell weight / continuous / grams / after being dried \n",
" Rings / integer / -- / +1.5 gives the age in years \n",
"\n",
"Suppose we are interested in predicting the age of the abalone given their measurements. This is an example of a regression problem."
],
"id": "j6k2NWBgIOiy"
},
{
"cell_type": "code",
"metadata": {
"scrolled": true,
"id": "QYFz8g72IOi1"
},
"source": [
"df = pd.read_csv('http://archive.ics.uci.edu/ml/machine-learning-databases/abalone/abalone.data',\n",
" header=None, names=['sex', 'length', 'diameter', 'height', 'weight', 'shucked_weight',\n",
" 'viscera_weight', 'shell_weight', 'rings'])"
],
"id": "QYFz8g72IOi1",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"scrolled": false,
"id": "LBuF5ZJ8IOi1"
},
"source": [
"df.head()"
],
"id": "LBuF5ZJ8IOi1",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"scrolled": true,
"id": "ahefjnACIOi2"
},
"source": [
"df.describe()"
],
"id": "ahefjnACIOi2",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"scrolled": true,
"id": "Q3o2m9qmIOi3"
},
"source": [
"df['rings'].plot(kind='hist')"
],
"id": "Q3o2m9qmIOi3",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"scrolled": false,
"id": "P_G245IbIOi3"
},
"source": [
"df.plot('weight', 'rings', kind='scatter')"
],
"id": "P_G245IbIOi3",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"scrolled": true,
"id": "BDgvb4nlIOi4"
},
"source": [
"X = df[['weight']].to_numpy()\n",
"y = df['rings'].to_numpy()"
],
"id": "BDgvb4nlIOi4",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"scrolled": true,
"id": "mDk7dTMJIOi4"
},
"source": [
"X"
],
"id": "mDk7dTMJIOi4",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"scrolled": true,
"id": "wOEmW5I8IOi5"
},
"source": [
"y"
],
"id": "wOEmW5I8IOi5",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"scrolled": true,
"id": "eyxohibHIOi5"
},
"source": [
"from sklearn import linear_model\n",
"model = linear_model.LinearRegression()\n",
"model.fit(X, y)\n",
"print(model.coef_, model.intercept_)"
],
"id": "eyxohibHIOi5",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"scrolled": true,
"id": "l5FPsT5aIOi6"
},
"source": [
"print(model.score(X, y))"
],
"id": "l5FPsT5aIOi6",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"scrolled": true,
"id": "92R_i04dIOi6"
},
"source": [
"model.predict(np.array([[1.5], [2.2]]))"
],
"id": "92R_i04dIOi6",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"scrolled": true,
"id": "fgM_9fdIIOi7"
},
"source": [
"df.plot('weight', 'rings', kind='scatter')\n",
"\n",
"weight = np.linspace(0, 3, 10).reshape(-1, 1)\n",
"plt.plot(weight, model.predict(weight), 'r')"
],
"id": "fgM_9fdIIOi7",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "BzvMWHgmIOi7"
},
"source": [
"df['root_weight'] = np.sqrt(df['weight'])"
],
"id": "BzvMWHgmIOi7",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "-4pmQ1XQIOi8"
},
"source": [
"X = df[['weight','root_weight']].to_numpy()\n",
"y = df['rings'].to_numpy()\n",
"model = linear_model.LinearRegression()\n",
"model.fit(X, y)"
],
"id": "-4pmQ1XQIOi8",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "xI1bvJryIOi8"
},
"source": [
"weight = np.linspace(0, 3, 100).reshape(-1, 1)\n",
"root_weight = np.sqrt(weight)\n",
"\n",
"features = np.hstack((weight,root_weight))"
],
"id": "xI1bvJryIOi8",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "Wk3lrU6qIOi8"
},
"source": [
"df.plot('weight', 'rings', kind='scatter')\n",
"\n",
"plt.plot(weight, model.predict(features), 'r')"
],
"id": "Wk3lrU6qIOi8",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "h_bz_h27IOi9"
},
"source": [
"model.score(X,y)"
],
"id": "h_bz_h27IOi9",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "Aqszb3RuIOi9"
},
"source": [
"plt.hist2d(df['weight'],df['rings'],bins=(50,30));"
],
"id": "Aqszb3RuIOi9",
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "lJYf3H2NKXWi"
},
"source": [
"#### Classification\n",
"\n",
"Another example of a machine learning problem is classification. Here we will use a dataset of flower measurements from three different flower species of *Iris* (*Iris setosa*, *Iris virginica*, and *Iris versicolor*). We aim to predict the species of the flower. Because the species is not a numerical output, it is not a regression problem, but a classification problem."
],
"id": "lJYf3H2NKXWi"
},
{
"cell_type": "code",
"metadata": {
"id": "nv_ER6FDKXWi"
},
"source": [
"from sklearn import datasets\n",
"iris = datasets.load_iris()\n",
"print(iris.DESCR)"
],
"id": "nv_ER6FDKXWi",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "jDkdMhtdKXWj"
},
"source": [
"X = iris.data[:, 2:]\n",
"y = iris.target_names[iris.target]\n",
"for name in iris.target_names:\n",
" plt.scatter(X[y == name, 0], X[y == name, 1], label=name)\n",
"plt.xlabel('Petal length')\n",
"plt.ylabel('Petal width')\n",
"plt.legend();"
],
"id": "jDkdMhtdKXWj",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"scrolled": true,
"id": "PfJrRE_UKXWj"
},
"source": [
"from sklearn.model_selection import train_test_split\n",
"X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)\n",
"print(X_train.shape, y_train.shape)\n",
"print(X_test.shape, y_test.shape)"
],
"id": "PfJrRE_UKXWj",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"scrolled": false,
"id": "kiTMK74oKXWk"
},
"source": [
"from sklearn.neighbors import KNeighborsClassifier\n",
"model = KNeighborsClassifier()\n",
"model.fit(X_train, y_train)"
],
"id": "kiTMK74oKXWk",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"scrolled": true,
"id": "XX1bV5TsKXWk"
},
"source": [
"X_test"
],
"id": "XX1bV5TsKXWk",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"scrolled": true,
"id": "PIe9aQKMKXWk"
},
"source": [
"model.predict(X_test)"
],
"id": "PIe9aQKMKXWk",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"scrolled": true,
"id": "vilx5oh3KXWl"
},
"source": [
"import sklearn.metrics as metrics\n",
"metrics.accuracy_score(model.predict(X_test), y_test)"
],
"id": "vilx5oh3KXWl",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"scrolled": true,
"id": "649AqvvYKXWl"
},
"source": [
"print(metrics.classification_report(model.predict(X_test), y_test))"
],
"id": "649AqvvYKXWl",
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "D33vfL2xKXWn"
},
"source": [
"#### Clustering\n",
"\n",
"Clustering is useful if we don't have a dataset labelled with the categories we want to predict, but we nevertheless expect there to be a certain number of categories. For example, suppose we have the previous dataset, but we are missing the labels. We can use a clustering algorithm like k-means to *cluster* the datapoints. Because we don't have labels, clustering is what is called an **unsupervised learning** algorithm."
],
"id": "D33vfL2xKXWn"
},
{
"cell_type": "code",
"metadata": {
"id": "lf7UIAXoMjPM"
},
"source": [
"from sklearn.datasets import make_blobs\n",
"X,y = make_blobs(centers=4, n_samples=200, random_state=0, cluster_std=0.7)\n",
"print(X[:10],y[:10])"
],
"id": "lf7UIAXoMjPM",
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "Lbo31hO8MjPN"
},
"source": [
"Now we plot these points, but without coloring the points using the labels:"
],
"id": "Lbo31hO8MjPN"
},
{
"cell_type": "code",
"metadata": {
"id": "WalBVWi0MjPN"
},
"source": [
"plt.scatter(X[:,0],X[:,1]);"
],
"id": "WalBVWi0MjPN",
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "4CV0HTccMjPO"
},
"source": [
"We can still discern four clusters in the data set. Let's see if the k-means algorithm can recover these clusters. First we create the instance of the k-means model by giving it the number of clusters 4 as a hyperparameter."
],
"id": "4CV0HTccMjPO"
},
{
"cell_type": "code",
"metadata": {
"id": "dSC5t9sJMjPO"
},
"source": [
"from sklearn.cluster import KMeans\n",
"model = KMeans(4)\n",
"model.fit(X)\n",
"print(model.cluster_centers_)"
],
"id": "dSC5t9sJMjPO",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "dqqVQU2eMjPO"
},
"source": [
"plt.scatter(X[:,0],X[:,1], c=model.labels_);\n",
"plt.scatter(model.cluster_centers_[:,0], model.cluster_centers_[:,1], s=100, color=\"red\"); # Show the centres"
],
"id": "dqqVQU2eMjPO",
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "Qhr6EyGuKXWs"
},
"source": [
"#### Dimensionality reduction\n",
"\n",
"Dimensionality reduction is another unsupervised learning problem (that is, it does not require labels). It aims to project datapoints into a lower dimensional space while preserving distances between datapoints."
],
"id": "Qhr6EyGuKXWs"
},
{
"cell_type": "markdown",
"metadata": {
"id": "tQ446lmW3kLM"
},
"source": [
"---"
],
"id": "tQ446lmW3kLM"
},
{
"cell_type": "markdown",
"metadata": {
"id": "260I-jj1N_jN"
},
"source": [
"### ![](\"https://www.tensorflow.org/images/tf_logo_32px.png\")
Tensorflow\n",
"\n",
"TensorFlow is an end-to-end open source platform for machine learning.\n",
"\n",
"TensorFlow was originally created by Google as an internal machine learning tool, but an implementation of it was open sourced under the Apache 2.0 License in November 2015."
],
"id": "260I-jj1N_jN"
},
{
"cell_type": "markdown",
"metadata": {
"id": "baA63HqQSDXE"
},
"source": [
"Few reason for the popluarity of tensorflow\n",
"\n",
"* Python API\n",
"* Portability: deploy computation to one or more CPUs or GPUs in a esktop, server, or mobile device with a single API\n",
"* Flexibility: from Raspberry Pi, Android, Windows, iOS, Linux to server farms\n",
"* Visualization\n",
"* Checkpoints (for managing experiments)\n",
"* Large community (> 10,000 commits and > 3000 TF-related repos in one year)\n",
"* Awesome projects already using TensorFlow"
],
"id": "baA63HqQSDXE"
},
{
"cell_type": "markdown",
"metadata": {
"id": "F9QR6fyVQ-wl"
},
"source": [
"Reference: [Tensorflow beginner](https://colab.research.google.com/github/tensorflow/docs/blob/master/site/en/tutorials/quickstart/beginner.ipynb)"
],
"id": "F9QR6fyVQ-wl"
},
{
"cell_type": "code",
"metadata": {
"id": "0trJmd6DjqBZ"
},
"source": [
"import tensorflow as tf"
],
"id": "0trJmd6DjqBZ",
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "7NAbSZiaoJ4z"
},
"source": [
"Load and prepare the [MNIST dataset](http://yann.lecun.com/exdb/mnist/). Convert the samples from integers to floating-point numbers:"
],
"id": "7NAbSZiaoJ4z"
},
{
"cell_type": "code",
"metadata": {
"id": "7FP5258xjs-v"
},
"source": [
"mnist = tf.keras.datasets.mnist\n",
"\n",
"(x_train, y_train), (x_test, y_test) = mnist.load_data()\n",
"x_train, x_test = x_train / 255.0, x_test / 255.0"
],
"id": "7FP5258xjs-v",
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "BPZ68wASog_I"
},
"source": [
"Build the `tf.keras.Sequential` model by stacking layers. Choose an optimizer and loss function for training:"
],
"id": "BPZ68wASog_I"
},
{
"cell_type": "code",
"metadata": {
"id": "h3IKyzTCDNGo"
},
"source": [
"model = tf.keras.models.Sequential([\n",
" tf.keras.layers.Flatten(input_shape=(28, 28)),\n",
" tf.keras.layers.Dense(128, activation='relu'),\n",
" tf.keras.layers.Dropout(0.2),\n",
" tf.keras.layers.Dense(10)\n",
"])"
],
"id": "h3IKyzTCDNGo",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "9foNKHzTD2Vo"
},
"source": [
"model.compile(optimizer='adam',\n",
" loss='sparse_categorical_crossentropy',\n",
" metrics=['accuracy'])"
],
"id": "9foNKHzTD2Vo",
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "ix4mEL65on-w"
},
"source": [
"The `Model.fit` method adjusts the model parameters to minimize the loss: "
],
"id": "ix4mEL65on-w"
},
{
"cell_type": "code",
"metadata": {
"id": "y7suUbJXVLqP"
},
"source": [
"model.fit(x_train, y_train, epochs=5)"
],
"id": "y7suUbJXVLqP",
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "4mDAAPFqVVgn"
},
"source": [
"The `Model.evaluate` method checks the models performance, usually on a \"[Validation-set](https://developers.google.com/machine-learning/glossary#validation-set)\" or \"[Test-set](https://developers.google.com/machine-learning/glossary#test-set)\"."
],
"id": "4mDAAPFqVVgn"
},
{
"cell_type": "code",
"metadata": {
"id": "F7dTAzgHDUh7"
},
"source": [
"model.evaluate(x_test, y_test, verbose=2)"
],
"id": "F7dTAzgHDUh7",
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "T4JfEh7kvx6m"
},
"source": [
"The image classifier is now trained to ~98% accuracy on this dataset. To learn more, read the [TensorFlow tutorials](https://www.tensorflow.org/tutorials/)."
],
"id": "T4JfEh7kvx6m"
},
{
"cell_type": "markdown",
"metadata": {
"id": "V9NZH-k4TwGz"
},
"source": [
"### **Pytorch**\n",
"\n",
"PyTorch is a machine learning framework that is used in both academia and industry for various applications. An alternative to tensorflow"
],
"id": "V9NZH-k4TwGz"
},
{
"cell_type": "markdown",
"metadata": {
"id": "2KdKKG8qEp4P"
},
"source": [
"# **---**"
],
"id": "2KdKKG8qEp4P"
},
{
"cell_type": "markdown",
"metadata": {
"id": "d94d2871"
},
"source": [
"3. Approaching a Tabular Problem
\n",
" Titanic Survivor Prediction Challenge
\n",
"\n",
"
"
],
"id": "d94d2871"
},
{
"cell_type": "markdown",
"metadata": {
"id": "1f9dca13"
},
"source": [
"3.1 Understanding the Problem
\n",
"\n",
"This challenge is kind of a \"hello world\" program in the whole data Science and Machine Learning community.
The task is simple,use machine learning to create a model that predicts which passengers survived the Titanic shipwreck. In this challenge, we perform a basic EDA and build a predictive model that answers the question: “what sorts of people were more likely to survive?” using passenger data (ie name, age, gender, socio-economic class, etc).
While there was some element of luck involved in surviving, it seems some groups of people were more likely to survive than others.
"
],
"id": "1f9dca13"
},
{
"cell_type": "markdown",
"metadata": {
"papermill": {
"duration": 0.098282,
"end_time": "2021-05-19T14:07:57.257046",
"exception": false,
"start_time": "2021-05-19T14:07:57.158764",
"status": "completed"
},
"tags": [],
"id": "exact-peninsula"
},
"source": [
"Contents
\n",
"\n",
"1. Data Description
\n",
"2. Exploratory Data Analysis
\n",
"3. Findings From EDA
\n",
"4. Data Preprocessing
\n",
"5. Feature Engineering and Feature Selection
\n",
"6. Data Modeling and Evaluation
\n",
"* Logistic Regression
* Gradient Boosting Classifier
* XgBoost
* SGB Classifier
\n",
"7. Models Comparison
\n"
],
"id": "exact-peninsula"
},
{
"cell_type": "markdown",
"metadata": {
"papermill": {
"duration": 0.09873,
"end_time": "2021-05-19T14:07:57.455074",
"exception": false,
"start_time": "2021-05-19T14:07:57.356344",
"status": "completed"
},
"tags": [],
"id": "legal-network"
},
"source": [
"Data Description
\n",
"\n",
"\n",
" - Survival : 0 = No, 1 = Yes
\n",
" - pclass(Ticket Class) : 1 = 1st, 2 = 2nd, 3 = 3rd
\n",
" - Sex(Gender) : Male, Female
\n",
" - Age : Age in years
\n",
" - SibSp : Number of siblings/spouses abroad the titanic
\n",
" - Parch : Number of parents/children abrod the titanic
\n",
" - Ticket : Ticket Number
\n",
" - Fare : Passenger fare
\n",
" - Cabin : Cabin Number
\n",
" - Embarked : Port of Embarkation, C = Cherbourg, Q = Queenstown, S = Southampton
\n",
"
"
],
"id": "legal-network"
},
{
"cell_type": "code",
"metadata": {
"execution": {
"iopub.execute_input": "2021-05-19T14:07:57.663692Z",
"iopub.status.busy": "2021-05-19T14:07:57.663108Z",
"iopub.status.idle": "2021-05-19T14:07:58.439813Z",
"shell.execute_reply": "2021-05-19T14:07:58.439195Z"
},
"papermill": {
"duration": 0.884349,
"end_time": "2021-05-19T14:07:58.439994",
"exception": false,
"start_time": "2021-05-19T14:07:57.555645",
"status": "completed"
},
"tags": [],
"id": "cardiac-environment"
},
"source": [
"import pandas as pd\n",
"import numpy as np\n",
"import matplotlib.pyplot as plt\n",
"import seaborn as sns\n",
"\n",
"import warnings\n",
"warnings.filterwarnings('ignore')\n",
"\n",
"plt.style.use('fivethirtyeight')\n",
"%matplotlib inline"
],
"id": "cardiac-environment",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"execution": {
"iopub.execute_input": "2021-05-19T14:07:58.654824Z",
"iopub.status.busy": "2021-05-19T14:07:58.654308Z",
"iopub.status.idle": "2021-05-19T14:07:58.668669Z",
"shell.execute_reply": "2021-05-19T14:07:58.668231Z"
},
"papermill": {
"duration": 0.12592,
"end_time": "2021-05-19T14:07:58.668772",
"exception": false,
"start_time": "2021-05-19T14:07:58.542852",
"status": "completed"
},
"tags": [],
"id": "crude-humanitarian"
},
"source": [
"#loading a CSV file using pandas dataframe\n",
"train_df = pd.read_csv('./inputs/tab/train.csv')\n",
"train_df.head()"
],
"id": "crude-humanitarian",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"execution": {
"iopub.execute_input": "2021-05-19T14:07:58.969748Z",
"iopub.status.busy": "2021-05-19T14:07:58.969314Z",
"iopub.status.idle": "2021-05-19T14:07:59.002179Z",
"shell.execute_reply": "2021-05-19T14:07:59.001772Z"
},
"papermill": {
"duration": 0.102772,
"end_time": "2021-05-19T14:07:59.002280",
"exception": false,
"start_time": "2021-05-19T14:07:58.899508",
"status": "completed"
},
"tags": [],
"id": "growing-feelings"
},
"source": [
"#Fetching some statistical information of the data\n",
"train_df.describe()"
],
"id": "growing-feelings",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"execution": {
"iopub.execute_input": "2021-05-19T14:07:59.283610Z",
"iopub.status.busy": "2021-05-19T14:07:59.283183Z",
"iopub.status.idle": "2021-05-19T14:07:59.299024Z",
"shell.execute_reply": "2021-05-19T14:07:59.299406Z"
},
"papermill": {
"duration": 0.085522,
"end_time": "2021-05-19T14:07:59.299547",
"exception": false,
"start_time": "2021-05-19T14:07:59.214025",
"status": "completed"
},
"tags": [],
"id": "least-native"
},
"source": [
"#Information about a DataFrame including the index dtype and columns, non-null values and memory usage\n",
"train_df.info()"
],
"id": "least-native",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"execution": {
"iopub.execute_input": "2021-05-19T14:07:59.437540Z",
"iopub.status.busy": "2021-05-19T14:07:59.437091Z",
"iopub.status.idle": "2021-05-19T14:07:59.445367Z",
"shell.execute_reply": "2021-05-19T14:07:59.445002Z"
},
"papermill": {
"duration": 0.077892,
"end_time": "2021-05-19T14:07:59.445467",
"exception": false,
"start_time": "2021-05-19T14:07:59.367575",
"status": "completed"
},
"tags": [],
"id": "difficult-graduation"
},
"source": [
"# Checking for null values\n",
"train_df.isna().sum()"
],
"id": "difficult-graduation",
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"papermill": {
"duration": 0.066897,
"end_time": "2021-05-19T14:07:59.580266",
"exception": false,
"start_time": "2021-05-19T14:07:59.513369",
"status": "completed"
},
"tags": [],
"id": "abroad-planner"
},
"source": [
"Exploratory Data Analysis (EDA)
\n",
"\n",
"Exploratory Data Analysis refers to the critical process of performing initial investigations on data so as to discover patterns,to spot anomalies,to test hypothesis and to check assumptions with the help of summary statistics and graphical representations
\n"
],
"id": "abroad-planner"
},
{
"cell_type": "code",
"metadata": {
"execution": {
"iopub.execute_input": "2021-05-19T14:07:59.718588Z",
"iopub.status.busy": "2021-05-19T14:07:59.718152Z",
"iopub.status.idle": "2021-05-19T14:08:00.528732Z",
"shell.execute_reply": "2021-05-19T14:08:00.529096Z"
},
"papermill": {
"duration": 0.881894,
"end_time": "2021-05-19T14:08:00.529225",
"exception": false,
"start_time": "2021-05-19T14:07:59.647331",
"status": "completed"
},
"tags": [],
"id": "muslim-choice"
},
"source": [
"# visualizing null values\n",
"import missingno as msno\n",
"\n",
"msno.bar(train_df)\n",
"plt.show()"
],
"id": "muslim-choice",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"execution": {
"iopub.execute_input": "2021-05-19T14:08:00.667813Z",
"iopub.status.busy": "2021-05-19T14:08:00.667394Z",
"iopub.status.idle": "2021-05-19T14:08:00.975656Z",
"shell.execute_reply": "2021-05-19T14:08:00.976126Z"
},
"papermill": {
"duration": 0.378582,
"end_time": "2021-05-19T14:08:00.976255",
"exception": false,
"start_time": "2021-05-19T14:08:00.597673",
"status": "completed"
},
"tags": [],
"id": "contemporary-baker"
},
"source": [
"# Correlation and mulitcolinarity\n",
"plt.figure(figsize = (18, 8))\n",
"corr = train_df.corr()\n",
"\n",
"sns.heatmap(corr, annot = True, fmt = '.2f', linewidths = 1, annot_kws = {'size' : 15})\n",
"plt.show()"
],
"id": "contemporary-baker",
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"papermill": {
"duration": 0.069434,
"end_time": "2021-05-19T14:08:01.256086",
"exception": false,
"start_time": "2021-05-19T14:08:01.186652",
"status": "completed"
},
"tags": [],
"id": "functional-republican"
},
"source": [
"Survived Column - Target
"
],
"id": "functional-republican"
},
{
"cell_type": "code",
"metadata": {
"execution": {
"iopub.execute_input": "2021-05-19T14:08:01.398982Z",
"iopub.status.busy": "2021-05-19T14:08:01.398534Z",
"iopub.status.idle": "2021-05-19T14:08:01.516340Z",
"shell.execute_reply": "2021-05-19T14:08:01.516769Z"
},
"papermill": {
"duration": 0.190554,
"end_time": "2021-05-19T14:08:01.516919",
"exception": false,
"start_time": "2021-05-19T14:08:01.326365",
"status": "completed"
},
"tags": [],
"id": "caring-irish"
},
"source": [
"plt.figure(figsize = (12, 7))\n",
"\n",
"sns.countplot(y = 'Survived', data = train_df)\n",
"plt.show()"
],
"id": "caring-irish",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"execution": {
"iopub.execute_input": "2021-05-19T14:08:01.660335Z",
"iopub.status.busy": "2021-05-19T14:08:01.659911Z",
"iopub.status.idle": "2021-05-19T14:08:01.786932Z",
"shell.execute_reply": "2021-05-19T14:08:01.786549Z"
},
"papermill": {
"duration": 0.199188,
"end_time": "2021-05-19T14:08:01.787035",
"exception": false,
"start_time": "2021-05-19T14:08:01.587847",
"status": "completed"
},
"tags": [],
"id": "renewable-apparel"
},
"source": [
"#pie chart to show Survived vs Not in percentage\n",
"values = train_df['Survived'].value_counts()\n",
"labels = ['Not Survived', 'Survived']\n",
"\n",
"fig, ax = plt.subplots(figsize = (5, 5), dpi = 100)\n",
"explode = (0, 0.06)\n",
"\n",
"patches, texts, autotexts = ax.pie(values, labels = labels, autopct = '%1.2f%%', shadow = True,\n",
" startangle = 90, explode = explode)\n",
"\n",
"plt.setp(texts, color = 'grey')\n",
"plt.setp(autotexts, size = 12, color = 'white')\n",
"autotexts[1].set_color('black')\n",
"plt.show()"
],
"id": "renewable-apparel",
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"papermill": {
"duration": 0.07013,
"end_time": "2021-05-19T14:08:01.927411",
"exception": false,
"start_time": "2021-05-19T14:08:01.857281",
"status": "completed"
},
"tags": [],
"id": "selective-investing"
},
"source": [
"Pclass Column
"
],
"id": "selective-investing"
},
{
"cell_type": "code",
"metadata": {
"execution": {
"iopub.execute_input": "2021-05-19T14:08:02.077802Z",
"iopub.status.busy": "2021-05-19T14:08:02.077135Z",
"iopub.status.idle": "2021-05-19T14:08:02.079866Z",
"shell.execute_reply": "2021-05-19T14:08:02.080375Z"
},
"papermill": {
"duration": 0.081457,
"end_time": "2021-05-19T14:08:02.080524",
"exception": false,
"start_time": "2021-05-19T14:08:01.999067",
"status": "completed"
},
"tags": [],
"id": "narrative-validation"
},
"source": [
"#individual count of each class\n",
"train_df.Pclass.value_counts()"
],
"id": "narrative-validation",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"execution": {
"iopub.execute_input": "2021-05-19T14:08:02.298537Z",
"iopub.status.busy": "2021-05-19T14:08:02.297748Z",
"iopub.status.idle": "2021-05-19T14:08:02.305355Z",
"shell.execute_reply": "2021-05-19T14:08:02.304839Z"
},
"papermill": {
"duration": 0.117418,
"end_time": "2021-05-19T14:08:02.305476",
"exception": false,
"start_time": "2021-05-19T14:08:02.188058",
"status": "completed"
},
"tags": [],
"id": "accredited-garlic"
},
"source": [
"#Individual counts of Pclass based on Target\n",
"train_df.groupby(['Pclass', 'Survived'])['Survived'].count()"
],
"id": "accredited-garlic",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"execution": {
"iopub.execute_input": "2021-05-19T14:08:02.517775Z",
"iopub.status.busy": "2021-05-19T14:08:02.517163Z",
"iopub.status.idle": "2021-05-19T14:08:02.749045Z",
"shell.execute_reply": "2021-05-19T14:08:02.748420Z"
},
"papermill": {
"duration": 0.341842,
"end_time": "2021-05-19T14:08:02.749169",
"exception": false,
"start_time": "2021-05-19T14:08:02.407327",
"status": "completed"
},
"tags": [],
"id": "administrative-blogger"
},
"source": [
"#bar chart showing Pclass counts based on Targets\n",
"plt.figure(figsize = (16, 8))\n",
"\n",
"sns.countplot('Pclass', hue = 'Survived', data = train_df)\n",
"plt.show()"
],
"id": "administrative-blogger",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"execution": {
"iopub.execute_input": "2021-05-19T14:08:02.987043Z",
"iopub.status.busy": "2021-05-19T14:08:02.985056Z",
"iopub.status.idle": "2021-05-19T14:08:03.080904Z",
"shell.execute_reply": "2021-05-19T14:08:03.081279Z"
},
"papermill": {
"duration": 0.223682,
"end_time": "2021-05-19T14:08:03.081435",
"exception": false,
"start_time": "2021-05-19T14:08:02.857753",
"status": "completed"
},
"tags": [],
"id": "associate-colorado"
},
"source": [
"#Pie chart showing Pclass counts percentage based on Targets\n",
"values = train_df['Pclass'].value_counts()\n",
"labels = ['Third Class', 'Second Class', 'First Class']\n",
"explode = (0, 0, 0.08)\n",
"fig, ax = plt.subplots(figsize = (5, 6), dpi = 100)\n",
"patches, texts, autotexts = ax.pie(values, labels = labels, autopct = '%1.2f%%', shadow = True,\n",
" startangle = 90, explode = explode)\n",
"plt.setp(texts, color = 'grey')\n",
"plt.setp(autotexts, size = 13, color = 'white')\n",
"autotexts[2].set_color('black')\n",
"plt.show()"
],
"id": "associate-colorado",
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"papermill": {
"duration": 0.074364,
"end_time": "2021-05-19T14:08:03.704338",
"exception": false,
"start_time": "2021-05-19T14:08:03.629974",
"status": "completed"
},
"tags": [],
"id": "wired-breach"
},
"source": [
"Name Column
"
],
"id": "wired-breach"
},
{
"cell_type": "code",
"metadata": {
"execution": {
"iopub.execute_input": "2021-05-19T14:08:03.859190Z",
"iopub.status.busy": "2021-05-19T14:08:03.858702Z",
"iopub.status.idle": "2021-05-19T14:08:03.861999Z",
"shell.execute_reply": "2021-05-19T14:08:03.862375Z"
},
"papermill": {
"duration": 0.08402,
"end_time": "2021-05-19T14:08:03.862517",
"exception": false,
"start_time": "2021-05-19T14:08:03.778497",
"status": "completed"
},
"tags": [],
"id": "blank-somewhere"
},
"source": [
"#Count of uniqes names\n",
"train_df.Name.value_counts()"
],
"id": "blank-somewhere",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"execution": {
"iopub.execute_input": "2021-05-19T14:08:04.017576Z",
"iopub.status.busy": "2021-05-19T14:08:04.017123Z",
"iopub.status.idle": "2021-05-19T14:08:04.020719Z",
"shell.execute_reply": "2021-05-19T14:08:04.021142Z"
},
"papermill": {
"duration": 0.082248,
"end_time": "2021-05-19T14:08:04.021269",
"exception": false,
"start_time": "2021-05-19T14:08:03.939021",
"status": "completed"
},
"tags": [],
"id": "otherwise-decline"
},
"source": [
"len(train_df.Name.unique()), train_df.shape"
],
"id": "otherwise-decline",
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"papermill": {
"duration": 0.075265,
"end_time": "2021-05-19T14:08:04.173597",
"exception": false,
"start_time": "2021-05-19T14:08:04.098332",
"status": "completed"
},
"tags": [],
"id": "normal-breast"
},
"source": [
"Gender Column
"
],
"id": "normal-breast"
},
{
"cell_type": "code",
"metadata": {
"execution": {
"iopub.execute_input": "2021-05-19T14:08:04.362959Z",
"iopub.status.busy": "2021-05-19T14:08:04.362484Z",
"iopub.status.idle": "2021-05-19T14:08:04.365310Z",
"shell.execute_reply": "2021-05-19T14:08:04.364950Z"
},
"papermill": {
"duration": 0.084562,
"end_time": "2021-05-19T14:08:04.365425",
"exception": false,
"start_time": "2021-05-19T14:08:04.280863",
"status": "completed"
},
"tags": [],
"id": "naval-helena"
},
"source": [
"#Indvidual count of Male vs Female\n",
"train_df.Sex.value_counts()"
],
"id": "naval-helena",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"execution": {
"iopub.execute_input": "2021-05-19T14:08:04.520314Z",
"iopub.status.busy": "2021-05-19T14:08:04.519862Z",
"iopub.status.idle": "2021-05-19T14:08:04.525904Z",
"shell.execute_reply": "2021-05-19T14:08:04.526259Z"
},
"papermill": {
"duration": 0.084636,
"end_time": "2021-05-19T14:08:04.526378",
"exception": false,
"start_time": "2021-05-19T14:08:04.441742",
"status": "completed"
},
"tags": [],
"id": "moral-mount"
},
"source": [
"#Count of Male vs Female grouped on Target\n",
"train_df.groupby(['Sex', 'Survived'])['Survived'].count()"
],
"id": "moral-mount",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"execution": {
"iopub.execute_input": "2021-05-19T14:08:04.680592Z",
"iopub.status.busy": "2021-05-19T14:08:04.680157Z",
"iopub.status.idle": "2021-05-19T14:08:04.801786Z",
"shell.execute_reply": "2021-05-19T14:08:04.802135Z"
},
"papermill": {
"duration": 0.199724,
"end_time": "2021-05-19T14:08:04.802266",
"exception": false,
"start_time": "2021-05-19T14:08:04.602542",
"status": "completed"
},
"tags": [],
"id": "third-hawaiian"
},
"source": [
"#Individual counts of Gender based on Target\n",
"plt.figure(figsize = (16, 7))\n",
"\n",
"sns.countplot('Sex', hue = 'Survived', data = train_df)\n",
"plt.show()"
],
"id": "third-hawaiian",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"execution": {
"iopub.execute_input": "2021-05-19T14:08:04.957938Z",
"iopub.status.busy": "2021-05-19T14:08:04.957474Z",
"iopub.status.idle": "2021-05-19T14:08:05.566849Z",
"shell.execute_reply": "2021-05-19T14:08:05.567369Z"
},
"papermill": {
"duration": 0.688623,
"end_time": "2021-05-19T14:08:05.567525",
"exception": false,
"start_time": "2021-05-19T14:08:04.878902",
"status": "completed"
},
"tags": [],
"id": "victorian-sunday"
},
"source": [
"#Individual counts of Gender based on Target for each Pclass Category\n",
"sns.catplot(x = 'Sex', y = 'Survived', data = train_df, kind = 'bar', col = 'Pclass')\n",
"plt.show()"
],
"id": "victorian-sunday",
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"papermill": {
"duration": 0.079778,
"end_time": "2021-05-19T14:08:06.827068",
"exception": false,
"start_time": "2021-05-19T14:08:06.747290",
"status": "completed"
},
"tags": [],
"id": "extended-summary"
},
"source": [
"Age Column
"
],
"id": "extended-summary"
},
{
"cell_type": "code",
"metadata": {
"execution": {
"iopub.execute_input": "2021-05-19T14:08:07.001326Z",
"iopub.status.busy": "2021-05-19T14:08:07.000866Z",
"iopub.status.idle": "2021-05-19T14:08:07.201761Z",
"shell.execute_reply": "2021-05-19T14:08:07.201309Z"
},
"papermill": {
"duration": 0.295487,
"end_time": "2021-05-19T14:08:07.201868",
"exception": false,
"start_time": "2021-05-19T14:08:06.906381",
"status": "completed"
},
"tags": [],
"id": "charitable-investigation"
},
"source": [
"#Age distribution\n",
"plt.figure(figsize = (15, 6))\n",
"plt.style.use('ggplot')\n",
"\n",
"sns.distplot(train_df['Age'])\n",
"plt.show()"
],
"id": "charitable-investigation",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"execution": {
"iopub.execute_input": "2021-05-19T14:08:07.371299Z",
"iopub.status.busy": "2021-05-19T14:08:07.370810Z",
"iopub.status.idle": "2021-05-19T14:08:07.592930Z",
"shell.execute_reply": "2021-05-19T14:08:07.592448Z"
},
"papermill": {
"duration": 0.3099,
"end_time": "2021-05-19T14:08:07.593034",
"exception": false,
"start_time": "2021-05-19T14:08:07.283134",
"status": "completed"
},
"tags": [],
"id": "northern-offset"
},
"source": [
"#Checking outliers in Age grouped by Gender\n",
"sns.catplot(x = 'Sex', y = 'Age', kind = 'box', data = train_df, height = 5, aspect = 2)\n",
"plt.show()"
],
"id": "northern-offset",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"execution": {
"iopub.execute_input": "2021-05-19T14:08:07.762989Z",
"iopub.status.busy": "2021-05-19T14:08:07.762510Z",
"iopub.status.idle": "2021-05-19T14:08:08.271980Z",
"shell.execute_reply": "2021-05-19T14:08:08.271484Z"
},
"papermill": {
"duration": 0.598278,
"end_time": "2021-05-19T14:08:08.272090",
"exception": false,
"start_time": "2021-05-19T14:08:07.673812",
"status": "completed"
},
"tags": [],
"id": "silver-commitment"
},
"source": [
"#Checking outliers in Age grouped by Gender and Pclass\n",
"sns.catplot(x = 'Sex', y = 'Age', kind = 'box', data = train_df, col = 'Pclass')\n",
"plt.show()"
],
"id": "silver-commitment",
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "8dcfff41"
},
"source": [
"Cabin Column
\n",
"\n",
"\n",
"
\n"
],
"id": "8dcfff41"
},
{
"cell_type": "markdown",
"metadata": {
"papermill": {
"duration": 0.080796,
"end_time": "2021-05-19T14:08:08.435523",
"exception": false,
"start_time": "2021-05-19T14:08:08.354727",
"status": "completed"
},
"tags": [],
"id": "confirmed-italy"
},
"source": [
"Fare Column
"
],
"id": "confirmed-italy"
},
{
"cell_type": "code",
"metadata": {
"execution": {
"iopub.execute_input": "2021-05-19T14:08:08.618643Z",
"iopub.status.busy": "2021-05-19T14:08:08.607416Z",
"iopub.status.idle": "2021-05-19T14:08:08.854098Z",
"shell.execute_reply": "2021-05-19T14:08:08.853658Z"
},
"papermill": {
"duration": 0.33511,
"end_time": "2021-05-19T14:08:08.854203",
"exception": false,
"start_time": "2021-05-19T14:08:08.519093",
"status": "completed"
},
"tags": [],
"id": "rolled-coverage"
},
"source": [
"#Histogram showing Fare counts\n",
"plt.figure(figsize = (14, 6))\n",
"\n",
"plt.hist(train_df.Fare, bins = 60, color = 'orange')\n",
"plt.xlabel('Fare')\n",
"plt.show()"
],
"id": "rolled-coverage",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"execution": {
"iopub.execute_input": "2021-05-19T14:08:09.192273Z",
"iopub.status.busy": "2021-05-19T14:08:09.190393Z",
"iopub.status.idle": "2021-05-19T14:08:09.859094Z",
"shell.execute_reply": "2021-05-19T14:08:09.858623Z"
},
"papermill": {
"duration": 0.756902,
"end_time": "2021-05-19T14:08:09.859198",
"exception": false,
"start_time": "2021-05-19T14:08:09.102296",
"status": "completed"
},
"tags": [],
"id": "powerful-diesel"
},
"source": [
"#outliers in Fare based on Gender and Pclass\n",
"sns.catplot(x = 'Sex', y = 'Fare', data = train_df, kind = 'box', col = 'Pclass')\n",
"plt.show()"
],
"id": "powerful-diesel",
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"papermill": {
"duration": 0.076815,
"end_time": "2021-05-19T14:08:10.013683",
"exception": false,
"start_time": "2021-05-19T14:08:09.936868",
"status": "completed"
},
"tags": [],
"id": "based-specific"
},
"source": [
"SibSp Column
"
],
"id": "based-specific"
},
{
"cell_type": "code",
"metadata": {
"execution": {
"iopub.execute_input": "2021-05-19T14:08:10.174969Z",
"iopub.status.busy": "2021-05-19T14:08:10.174369Z",
"iopub.status.idle": "2021-05-19T14:08:10.176899Z",
"shell.execute_reply": "2021-05-19T14:08:10.177365Z"
},
"papermill": {
"duration": 0.086483,
"end_time": "2021-05-19T14:08:10.177502",
"exception": false,
"start_time": "2021-05-19T14:08:10.091019",
"status": "completed"
},
"tags": [],
"id": "brazilian-proof"
},
"source": [
"#Count of people with Siblings or spouse \n",
"train_df['SibSp'].value_counts()"
],
"id": "brazilian-proof",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"execution": {
"iopub.execute_input": "2021-05-19T14:08:10.415478Z",
"iopub.status.busy": "2021-05-19T14:08:10.414985Z",
"iopub.status.idle": "2021-05-19T14:08:10.589368Z",
"shell.execute_reply": "2021-05-19T14:08:10.588937Z"
},
"papermill": {
"duration": 0.295952,
"end_time": "2021-05-19T14:08:10.589471",
"exception": false,
"start_time": "2021-05-19T14:08:10.293519",
"status": "completed"
},
"tags": [],
"id": "stuck-contributor"
},
"source": [
"#count of people with Sibilings or Spouse based on Target\n",
"plt.figure(figsize = (16, 5))\n",
"\n",
"sns.countplot(x = 'SibSp', data = train_df, hue = 'Survived')\n",
"plt.show()"
],
"id": "stuck-contributor",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"execution": {
"iopub.execute_input": "2021-05-19T14:08:10.751377Z",
"iopub.status.busy": "2021-05-19T14:08:10.750773Z",
"iopub.status.idle": "2021-05-19T14:08:11.087843Z",
"shell.execute_reply": "2021-05-19T14:08:11.088226Z"
},
"papermill": {
"duration": 0.421161,
"end_time": "2021-05-19T14:08:11.088352",
"exception": false,
"start_time": "2021-05-19T14:08:10.667191",
"status": "completed"
},
"tags": [],
"id": "weird-italy"
},
"source": [
"#Count of people who survived based on SibSp counts\n",
"sns.catplot(x = 'SibSp', y = 'Survived', kind = 'bar', data = train_df, height = 5, aspect =2)\n",
"plt.show()"
],
"id": "weird-italy",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"execution": {
"iopub.execute_input": "2021-05-19T14:08:11.263589Z",
"iopub.status.busy": "2021-05-19T14:08:11.262899Z",
"iopub.status.idle": "2021-05-19T14:08:11.812454Z",
"shell.execute_reply": "2021-05-19T14:08:11.812803Z"
},
"papermill": {
"duration": 0.639754,
"end_time": "2021-05-19T14:08:11.812958",
"exception": false,
"start_time": "2021-05-19T14:08:11.173204",
"status": "completed"
},
"tags": [],
"id": "magnetic-laundry"
},
"source": [
"#Count of people who survived based on SibSp counts Gender wise\n",
"sns.catplot(x = 'SibSp', y = 'Survived', kind = 'bar', hue = 'Sex', data = train_df, height = 6, aspect = 2)\n",
"plt.show()"
],
"id": "magnetic-laundry",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"execution": {
"iopub.execute_input": "2021-05-19T14:08:11.991611Z",
"iopub.status.busy": "2021-05-19T14:08:11.991129Z",
"iopub.status.idle": "2021-05-19T14:08:12.665522Z",
"shell.execute_reply": "2021-05-19T14:08:12.665114Z"
},
"papermill": {
"duration": 0.767073,
"end_time": "2021-05-19T14:08:12.665626",
"exception": false,
"start_time": "2021-05-19T14:08:11.898553",
"status": "completed"
},
"tags": [],
"id": "appointed-housing"
},
"source": [
"sns.catplot(x = 'SibSp', y = 'Survived', kind = 'bar', col = 'Sex', data = train_df)\n",
"plt.show()"
],
"id": "appointed-housing",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"execution": {
"iopub.execute_input": "2021-05-19T14:08:12.843748Z",
"iopub.status.busy": "2021-05-19T14:08:12.842459Z",
"iopub.status.idle": "2021-05-19T14:08:13.546099Z",
"shell.execute_reply": "2021-05-19T14:08:13.545548Z"
},
"papermill": {
"duration": 0.793747,
"end_time": "2021-05-19T14:08:13.546200",
"exception": false,
"start_time": "2021-05-19T14:08:12.752453",
"status": "completed"
},
"tags": [],
"id": "independent-ranch"
},
"source": [
"#Count of people who survived based on SibSp counts Gender and Pclass wise\n",
"sns.catplot(x = 'SibSp', y = 'Survived', col = 'Pclass', kind = 'bar', data = train_df)\n",
"plt.show()"
],
"id": "independent-ranch",
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"papermill": {
"duration": 0.086248,
"end_time": "2021-05-19T14:08:14.483835",
"exception": false,
"start_time": "2021-05-19T14:08:14.397587",
"status": "completed"
},
"tags": [],
"id": "superior-corner"
},
"source": [
"Parch Column
"
],
"id": "superior-corner"
},
{
"cell_type": "code",
"metadata": {
"execution": {
"iopub.execute_input": "2021-05-19T14:08:14.664702Z",
"iopub.status.busy": "2021-05-19T14:08:14.664121Z",
"iopub.status.idle": "2021-05-19T14:08:14.666585Z",
"shell.execute_reply": "2021-05-19T14:08:14.666975Z"
},
"papermill": {
"duration": 0.094913,
"end_time": "2021-05-19T14:08:14.667096",
"exception": false,
"start_time": "2021-05-19T14:08:14.572183",
"status": "completed"
},
"tags": [],
"id": "beneficial-distance"
},
"source": [
"#Count of people with Parents/Childrens\n",
"train_df.Parch.value_counts()"
],
"id": "beneficial-distance",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"execution": {
"iopub.execute_input": "2021-05-19T14:08:14.847921Z",
"iopub.status.busy": "2021-05-19T14:08:14.847267Z",
"iopub.status.idle": "2021-05-19T14:08:15.322517Z",
"shell.execute_reply": "2021-05-19T14:08:15.322924Z"
},
"papermill": {
"duration": 0.567021,
"end_time": "2021-05-19T14:08:15.323053",
"exception": false,
"start_time": "2021-05-19T14:08:14.756032",
"status": "completed"
},
"tags": [],
"id": "strange-blake"
},
"source": [
"sns.catplot(x = 'Parch', y = 'Survived', data = train_df, hue = 'Sex', kind = 'bar', height = 6, aspect = 2)\n",
"plt.show()"
],
"id": "strange-blake",
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"papermill": {
"duration": 0.087479,
"end_time": "2021-05-19T14:08:15.501074",
"exception": false,
"start_time": "2021-05-19T14:08:15.413595",
"status": "completed"
},
"tags": [],
"id": "familiar-advocacy"
},
"source": [
"Ticket Column
"
],
"id": "familiar-advocacy"
},
{
"cell_type": "code",
"metadata": {
"execution": {
"iopub.execute_input": "2021-05-19T14:08:15.685683Z",
"iopub.status.busy": "2021-05-19T14:08:15.682848Z",
"iopub.status.idle": "2021-05-19T14:08:15.688739Z",
"shell.execute_reply": "2021-05-19T14:08:15.689241Z"
},
"papermill": {
"duration": 0.098836,
"end_time": "2021-05-19T14:08:15.689404",
"exception": false,
"start_time": "2021-05-19T14:08:15.590568",
"status": "completed"
},
"tags": [],
"id": "acting-venture"
},
"source": [
"#count based on ticket types\n",
"train_df.Ticket.value_counts()"
],
"id": "acting-venture",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"execution": {
"iopub.execute_input": "2021-05-19T14:08:15.960680Z",
"iopub.status.busy": "2021-05-19T14:08:15.959860Z",
"iopub.status.idle": "2021-05-19T14:08:15.963096Z",
"shell.execute_reply": "2021-05-19T14:08:15.963551Z"
},
"papermill": {
"duration": 0.139887,
"end_time": "2021-05-19T14:08:15.963709",
"exception": false,
"start_time": "2021-05-19T14:08:15.823822",
"status": "completed"
},
"tags": [],
"id": "unauthorized-southwest"
},
"source": [
"len(train_df.Ticket.unique())"
],
"id": "unauthorized-southwest",
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"papermill": {
"duration": 0.132593,
"end_time": "2021-05-19T14:08:16.228860",
"exception": false,
"start_time": "2021-05-19T14:08:16.096267",
"status": "completed"
},
"tags": [],
"id": "exclusive-netscape"
},
"source": [
"Embarked Column
"
],
"id": "exclusive-netscape"
},
{
"cell_type": "code",
"metadata": {
"execution": {
"iopub.execute_input": "2021-05-19T14:08:16.502417Z",
"iopub.status.busy": "2021-05-19T14:08:16.501652Z",
"iopub.status.idle": "2021-05-19T14:08:16.506262Z",
"shell.execute_reply": "2021-05-19T14:08:16.505647Z"
},
"papermill": {
"duration": 0.143835,
"end_time": "2021-05-19T14:08:16.506389",
"exception": false,
"start_time": "2021-05-19T14:08:16.362554",
"status": "completed"
},
"tags": [],
"id": "listed-pocket"
},
"source": [
"train_df['Embarked'].value_counts()"
],
"id": "listed-pocket",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"execution": {
"iopub.execute_input": "2021-05-19T14:08:16.699174Z",
"iopub.status.busy": "2021-05-19T14:08:16.698675Z",
"iopub.status.idle": "2021-05-19T14:08:16.832975Z",
"shell.execute_reply": "2021-05-19T14:08:16.832317Z"
},
"papermill": {
"duration": 0.231563,
"end_time": "2021-05-19T14:08:16.833160",
"exception": false,
"start_time": "2021-05-19T14:08:16.601597",
"status": "completed"
},
"tags": [],
"id": "given-parameter"
},
"source": [
"#Count of people based on Port Embarked Gender wise\n",
"plt.figure(figsize = (14, 6))\n",
"\n",
"sns.countplot('Embarked', hue = 'Survived', data = train_df)\n",
"plt.show()"
],
"id": "given-parameter",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"execution": {
"iopub.execute_input": "2021-05-19T14:08:17.109835Z",
"iopub.status.busy": "2021-05-19T14:08:17.109022Z",
"iopub.status.idle": "2021-05-19T14:08:17.491860Z",
"shell.execute_reply": "2021-05-19T14:08:17.491166Z"
},
"papermill": {
"duration": 0.523038,
"end_time": "2021-05-19T14:08:17.491999",
"exception": false,
"start_time": "2021-05-19T14:08:16.968961",
"status": "completed"
},
"tags": [],
"id": "welsh-lunch"
},
"source": [
"sns.catplot(x = 'Embarked', y = 'Survived', kind = 'bar', data = train_df, col = 'Sex')\n",
"plt.show()"
],
"id": "welsh-lunch",
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"papermill": {
"duration": 0.091561,
"end_time": "2021-05-19T14:08:17.676978",
"exception": false,
"start_time": "2021-05-19T14:08:17.585417",
"status": "completed"
},
"tags": [],
"id": "ahead-teacher"
},
"source": [
"\n",
"Findings From EDA :-
\n",
"\n",
"\n",
" - Females Survived more than Males.
\n",
" - Passengers Travelling in Higher Class Survived More than Passengers travelling in Lower Class.
\n",
" - Name column is having all unique values so this column is not suitable for prediction, we have to drop it.
\n",
" - In First Class Females were more than Males, that's why Fare of Females Passengers were high.
\n",
" - Survival Rate is higher for those who were travelling with siblings or spouses.
\n",
" - Passengers travelling with parents or children have higher survival rate.
\n",
" - Ticket column is not useful and does not have an impact on survival.
\n",
" - Cabin column have a lot of null values , it will be better to drop this column.
\n",
" - Passengers travelling from Cherbourg port survived more than passengers travelling from other two ports.
\n",
"
"
],
"id": "ahead-teacher"
},
{
"cell_type": "markdown",
"metadata": {
"papermill": {
"duration": 0.136998,
"end_time": "2021-05-19T14:08:17.906660",
"exception": false,
"start_time": "2021-05-19T14:08:17.769662",
"status": "completed"
},
"tags": [],
"id": "polished-spoke"
},
"source": [
"3.2 Data Preprocessing
"
],
"id": "polished-spoke"
},
{
"cell_type": "code",
"metadata": {
"execution": {
"iopub.execute_input": "2021-05-19T14:08:18.180601Z",
"iopub.status.busy": "2021-05-19T14:08:18.180027Z",
"iopub.status.idle": "2021-05-19T14:08:18.181997Z",
"shell.execute_reply": "2021-05-19T14:08:18.181508Z"
},
"papermill": {
"duration": 0.141156,
"end_time": "2021-05-19T14:08:18.182109",
"exception": false,
"start_time": "2021-05-19T14:08:18.040953",
"status": "completed"
},
"tags": [],
"id": "incredible-orientation"
},
"source": [
"# dropping useless columns with noise, missing data ..etc\n",
"train_df.drop(['PassengerId', 'Name', 'Ticket', 'Cabin'], axis = 1, inplace = True)"
],
"id": "incredible-orientation",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"execution": {
"iopub.execute_input": "2021-05-19T14:08:18.464810Z",
"iopub.status.busy": "2021-05-19T14:08:18.461226Z",
"iopub.status.idle": "2021-05-19T14:08:18.468929Z",
"shell.execute_reply": "2021-05-19T14:08:18.468533Z"
},
"papermill": {
"duration": 0.152173,
"end_time": "2021-05-19T14:08:18.469039",
"exception": false,
"start_time": "2021-05-19T14:08:18.316866",
"status": "completed"
},
"tags": [],
"id": "classical-julian"
},
"source": [
"train_df.head()"
],
"id": "classical-julian",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"execution": {
"iopub.execute_input": "2021-05-19T14:08:18.663050Z",
"iopub.status.busy": "2021-05-19T14:08:18.662614Z",
"iopub.status.idle": "2021-05-19T14:08:18.665092Z",
"shell.execute_reply": "2021-05-19T14:08:18.665490Z"
},
"papermill": {
"duration": 0.101591,
"end_time": "2021-05-19T14:08:18.665593",
"exception": false,
"start_time": "2021-05-19T14:08:18.564002",
"status": "completed"
},
"tags": [],
"id": "killing-hands"
},
"source": [
"train_df.isna().sum()"
],
"id": "killing-hands",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"execution": {
"iopub.execute_input": "2021-05-19T14:08:18.870155Z",
"iopub.status.busy": "2021-05-19T14:08:18.869679Z",
"iopub.status.idle": "2021-05-19T14:08:18.876665Z",
"shell.execute_reply": "2021-05-19T14:08:18.876021Z"
},
"papermill": {
"duration": 0.108189,
"end_time": "2021-05-19T14:08:18.876837",
"exception": false,
"start_time": "2021-05-19T14:08:18.768648",
"status": "completed"
},
"tags": [],
"id": "french-resistance"
},
"source": [
"# replacing Zero values of \"Fare\" column with mean of column\n",
"train_df['Fare'] = train_df['Fare'].replace(0, train_df['Fare'].mean())"
],
"id": "french-resistance",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"execution": {
"iopub.execute_input": "2021-05-19T14:08:19.154160Z",
"iopub.status.busy": "2021-05-19T14:08:19.153539Z",
"iopub.status.idle": "2021-05-19T14:08:19.158789Z",
"shell.execute_reply": "2021-05-19T14:08:19.158271Z"
},
"papermill": {
"duration": 0.139395,
"end_time": "2021-05-19T14:08:19.158933",
"exception": false,
"start_time": "2021-05-19T14:08:19.019538",
"status": "completed"
},
"tags": [],
"id": "included-dayton"
},
"source": [
"# filling null values of \"Age\" column with mean value of the column\n",
"\n",
"train_df['Age'].fillna(train_df['Age'].mean(), inplace = True)"
],
"id": "included-dayton",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"execution": {
"iopub.execute_input": "2021-05-19T14:08:19.440186Z",
"iopub.status.busy": "2021-05-19T14:08:19.439725Z",
"iopub.status.idle": "2021-05-19T14:08:19.441366Z",
"shell.execute_reply": "2021-05-19T14:08:19.441730Z"
},
"papermill": {
"duration": 0.143508,
"end_time": "2021-05-19T14:08:19.441853",
"exception": false,
"start_time": "2021-05-19T14:08:19.298345",
"status": "completed"
},
"tags": [],
"id": "social-least"
},
"source": [
"# filling null values of \"Embarked\" column with mode value of the column\n",
"\n",
"train_df['Embarked'].fillna(train_df['Embarked'].mode()[0], inplace = True)"
],
"id": "social-least",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"execution": {
"iopub.execute_input": "2021-05-19T14:08:19.631739Z",
"iopub.status.busy": "2021-05-19T14:08:19.631315Z",
"iopub.status.idle": "2021-05-19T14:08:19.639022Z",
"shell.execute_reply": "2021-05-19T14:08:19.638621Z"
},
"papermill": {
"duration": 0.103346,
"end_time": "2021-05-19T14:08:19.639127",
"exception": false,
"start_time": "2021-05-19T14:08:19.535781",
"status": "completed"
},
"tags": [],
"id": "modified-implement"
},
"source": [
"# checking for null values after filling null values\n",
"\n",
"train_df.isna().sum()"
],
"id": "modified-implement",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"execution": {
"iopub.execute_input": "2021-05-19T14:08:19.838717Z",
"iopub.status.busy": "2021-05-19T14:08:19.838251Z",
"iopub.status.idle": "2021-05-19T14:08:19.842184Z",
"shell.execute_reply": "2021-05-19T14:08:19.841777Z"
},
"papermill": {
"duration": 0.109337,
"end_time": "2021-05-19T14:08:19.842281",
"exception": false,
"start_time": "2021-05-19T14:08:19.732944",
"status": "completed"
},
"tags": [],
"id": "opposed-bolivia"
},
"source": [
"train_df.head()"
],
"id": "opposed-bolivia",
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "8af949f1"
},
"source": [
"3.4 Feature Engineering and Feature Selection
"
],
"id": "8af949f1"
},
{
"cell_type": "code",
"metadata": {
"execution": {
"iopub.execute_input": "2021-05-19T14:08:20.035372Z",
"iopub.status.busy": "2021-05-19T14:08:20.034831Z",
"iopub.status.idle": "2021-05-19T14:08:20.037832Z",
"shell.execute_reply": "2021-05-19T14:08:20.037398Z"
},
"papermill": {
"duration": 0.102114,
"end_time": "2021-05-19T14:08:20.037956",
"exception": false,
"start_time": "2021-05-19T14:08:19.935842",
"status": "completed"
},
"tags": [],
"id": "coordinate-stretch"
},
"source": [
"'''\n",
"Not all Machine Leanrning Algorithms like categorical fearures as strings, \n",
"so they need to me encoded to numeric formats so they these alorithms can ingest this data without throwing error \n",
"labelEncoding : Converting String to numeric format'''\n",
"train_df['Sex'] = train_df['Sex'].apply(lambda val: 1 if val == 'male' else 0)\n",
"train_df.head()"
],
"id": "coordinate-stretch",
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "2d85e3c1"
},
"source": [
"OneHot Encoding : It is the representation of categorical variables as binary vectors
\n",
"\n",
"\n",
"
\n",
"\n"
],
"id": "2d85e3c1"
},
{
"cell_type": "code",
"metadata": {
"execution": {
"iopub.execute_input": "2021-05-19T14:08:20.230040Z",
"iopub.status.busy": "2021-05-19T14:08:20.229556Z",
"iopub.status.idle": "2021-05-19T14:08:20.231842Z",
"shell.execute_reply": "2021-05-19T14:08:20.231489Z"
},
"papermill": {
"duration": 0.100606,
"end_time": "2021-05-19T14:08:20.231980",
"exception": false,
"start_time": "2021-05-19T14:08:20.131374",
"status": "completed"
},
"tags": [],
"id": "forced-thong"
},
"source": [
"# Get one hot encoding of columns 'Embarked'\n",
"one_hot = pd.get_dummies(train_df['Embarked'])\n",
"# Drop column 'Embarked' as it is now encoded\n",
"train_df = train_df.drop('Embarked',axis = 1)\n",
"# Join the encoded train_df\n",
"train_df = train_df.join(one_hot)"
],
"id": "forced-thong",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"execution": {
"iopub.execute_input": "2021-05-19T14:08:20.428374Z",
"iopub.status.busy": "2021-05-19T14:08:20.427750Z",
"iopub.status.idle": "2021-05-19T14:08:20.431280Z",
"shell.execute_reply": "2021-05-19T14:08:20.430917Z"
},
"papermill": {
"duration": 0.105792,
"end_time": "2021-05-19T14:08:20.431378",
"exception": false,
"start_time": "2021-05-19T14:08:20.325586",
"status": "completed"
},
"tags": [],
"id": "several-dragon"
},
"source": [
"train_df.head()"
],
"id": "several-dragon",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"execution": {
"iopub.execute_input": "2021-05-19T14:08:20.640766Z",
"iopub.status.busy": "2021-05-19T14:08:20.639039Z",
"iopub.status.idle": "2021-05-19T14:08:20.664309Z",
"shell.execute_reply": "2021-05-19T14:08:20.663919Z"
},
"papermill": {
"duration": 0.135244,
"end_time": "2021-05-19T14:08:20.664415",
"exception": false,
"start_time": "2021-05-19T14:08:20.529171",
"status": "completed"
},
"tags": [],
"id": "chronic-senator"
},
"source": [
"train_df.describe()"
],
"id": "chronic-senator",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"execution": {
"iopub.execute_input": "2021-05-19T14:08:20.859452Z",
"iopub.status.busy": "2021-05-19T14:08:20.858851Z",
"iopub.status.idle": "2021-05-19T14:08:20.862394Z",
"shell.execute_reply": "2021-05-19T14:08:20.861945Z"
},
"papermill": {
"duration": 0.103584,
"end_time": "2021-05-19T14:08:20.862506",
"exception": false,
"start_time": "2021-05-19T14:08:20.758922",
"status": "completed"
},
"tags": [],
"id": "considerable-wrapping"
},
"source": [
"#checking for feature variance is very important\n",
"#High variance can affect the model performance, espcially for Linear models\n",
"#so we need to normalize high variance features\n",
"train_df.var()"
],
"id": "considerable-wrapping",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"execution": {
"iopub.execute_input": "2021-05-19T14:08:21.248518Z",
"iopub.status.busy": "2021-05-19T14:08:21.247996Z",
"iopub.status.idle": "2021-05-19T14:08:21.250726Z",
"shell.execute_reply": "2021-05-19T14:08:21.249803Z"
},
"papermill": {
"duration": 0.102114,
"end_time": "2021-05-19T14:08:21.250864",
"exception": false,
"start_time": "2021-05-19T14:08:21.148750",
"status": "completed"
},
"tags": [],
"id": "minor-audio"
},
"source": [
"#log normalization\n",
"train_df['Age'] = np.log(train_df['Age'])\n",
"train_df['Fare'] = np.log(train_df['Fare'])\n"
],
"id": "minor-audio",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"execution": {
"iopub.execute_input": "2021-05-19T14:08:21.453576Z",
"iopub.status.busy": "2021-05-19T14:08:21.453089Z",
"iopub.status.idle": "2021-05-19T14:08:21.456130Z",
"shell.execute_reply": "2021-05-19T14:08:21.455646Z"
},
"papermill": {
"duration": 0.109325,
"end_time": "2021-05-19T14:08:21.456237",
"exception": false,
"start_time": "2021-05-19T14:08:21.346912",
"status": "completed"
},
"tags": [],
"id": "naval-milwaukee"
},
"source": [
"train_df.var()"
],
"id": "naval-milwaukee",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"execution": {
"iopub.execute_input": "2021-05-19T14:08:21.846619Z",
"iopub.status.busy": "2021-05-19T14:08:21.846167Z",
"iopub.status.idle": "2021-05-19T14:08:21.858729Z",
"shell.execute_reply": "2021-05-19T14:08:21.859119Z"
},
"papermill": {
"duration": 0.110671,
"end_time": "2021-05-19T14:08:21.859254",
"exception": false,
"start_time": "2021-05-19T14:08:21.748583",
"status": "completed"
},
"tags": [],
"id": "inside-fight"
},
"source": [
"test_df = pd.read_csv('./inputs/tab/test.csv')\n",
"test_df.head()"
],
"id": "inside-fight",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"execution": {
"iopub.execute_input": "2021-05-19T14:08:22.452860Z",
"iopub.status.busy": "2021-05-19T14:08:22.452437Z",
"iopub.status.idle": "2021-05-19T14:08:22.458183Z",
"shell.execute_reply": "2021-05-19T14:08:22.457771Z"
},
"papermill": {
"duration": 0.104201,
"end_time": "2021-05-19T14:08:22.458288",
"exception": false,
"start_time": "2021-05-19T14:08:22.354087",
"status": "completed"
},
"tags": [],
"id": "micro-velvet"
},
"source": [
"# dropping useless columns\n",
"test_df.drop(['PassengerId', 'Name', 'Ticket', 'Cabin'], axis = 1, inplace = True)\n",
"# replacing Zero values of \"Fare\" column with mean of column\n",
"test_df['Fare'] = test_df['Fare'].replace(0, test_df['Fare'].mean())\n",
"# filling null values of \"Age\" column with mean value of the column\n",
"test_df['Age'].fillna(test_df['Age'].mean(), inplace = True)\n",
"# filling null values of \"Embarked\" column with mode value of the column\n",
"test_df['Embarked'].fillna(test_df['Embarked'].mode()[0], inplace = True)"
],
"id": "micro-velvet",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"execution": {
"iopub.execute_input": "2021-05-19T14:08:22.655972Z",
"iopub.status.busy": "2021-05-19T14:08:22.655464Z",
"iopub.status.idle": "2021-05-19T14:08:22.658269Z",
"shell.execute_reply": "2021-05-19T14:08:22.657866Z"
},
"papermill": {
"duration": 0.104291,
"end_time": "2021-05-19T14:08:22.658370",
"exception": false,
"start_time": "2021-05-19T14:08:22.554079",
"status": "completed"
},
"tags": [],
"id": "fitted-corporation"
},
"source": [
"test_df.isna().sum()"
],
"id": "fitted-corporation",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"execution": {
"iopub.execute_input": "2021-05-19T14:08:23.460928Z",
"iopub.status.busy": "2021-05-19T14:08:23.460290Z",
"iopub.status.idle": "2021-05-19T14:08:23.464053Z",
"shell.execute_reply": "2021-05-19T14:08:23.464413Z"
},
"papermill": {
"duration": 0.102972,
"end_time": "2021-05-19T14:08:23.464538",
"exception": false,
"start_time": "2021-05-19T14:08:23.361566",
"status": "completed"
},
"tags": [],
"id": "nervous-nelson"
},
"source": [
"# filling null values of \"Fare\" column with mean value of the column\n",
"test_df['Fare'].fillna(test_df['Fare'].mean(), inplace = True)\n",
"test_df['Sex'] = test_df['Sex'].apply(lambda val: 1 if val == 'male' else 0)\n",
"\n",
"# Get one hot encoding of columns 'Embarked'\n",
"one_hot_test = pd.get_dummies(test_df['Embarked'])\n",
"# Drop column 'Embarked' as it is now encoded\n",
"test_df = test_df.drop('Embarked',axis = 1)\n",
"# Join the encoded train_df\n",
"test_df = test_df.join(one_hot_test)\n",
"# Log Normalization\n",
"test_df['Age'] = np.log(test_df['Age'])\n",
"test_df['Fare'] = np.log(test_df['Fare'])\n",
"test_df.head()"
],
"id": "nervous-nelson",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"execution": {
"iopub.execute_input": "2021-05-19T14:08:24.487582Z",
"iopub.status.busy": "2021-05-19T14:08:24.485670Z",
"iopub.status.idle": "2021-05-19T14:08:24.490531Z",
"shell.execute_reply": "2021-05-19T14:08:24.490868Z"
},
"papermill": {
"duration": 0.10742,
"end_time": "2021-05-19T14:08:24.491029",
"exception": false,
"start_time": "2021-05-19T14:08:24.383609",
"status": "completed"
},
"tags": [],
"id": "exempt-toyota"
},
"source": [
"test_df.var()"
],
"id": "exempt-toyota",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"execution": {
"iopub.execute_input": "2021-05-19T14:08:24.696985Z",
"iopub.status.busy": "2021-05-19T14:08:24.696440Z",
"iopub.status.idle": "2021-05-19T14:08:24.700799Z",
"shell.execute_reply": "2021-05-19T14:08:24.700398Z"
},
"papermill": {
"duration": 0.111132,
"end_time": "2021-05-19T14:08:24.700934",
"exception": false,
"start_time": "2021-05-19T14:08:24.589802",
"status": "completed"
},
"tags": [],
"id": "mysterious-convert"
},
"source": [
"test_df.isna().any()"
],
"id": "mysterious-convert",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"execution": {
"iopub.execute_input": "2021-05-19T14:08:24.901650Z",
"iopub.status.busy": "2021-05-19T14:08:24.901020Z",
"iopub.status.idle": "2021-05-19T14:08:24.910365Z",
"shell.execute_reply": "2021-05-19T14:08:24.910812Z"
},
"papermill": {
"duration": 0.109853,
"end_time": "2021-05-19T14:08:24.910949",
"exception": false,
"start_time": "2021-05-19T14:08:24.801096",
"status": "completed"
},
"tags": [],
"id": "circular-venue"
},
"source": [
"test_df.head()"
],
"id": "circular-venue",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"execution": {
"iopub.execute_input": "2021-05-19T14:08:25.310340Z",
"iopub.status.busy": "2021-05-19T14:08:25.309841Z",
"iopub.status.idle": "2021-05-19T14:08:25.312407Z",
"shell.execute_reply": "2021-05-19T14:08:25.312021Z"
},
"papermill": {
"duration": 0.106042,
"end_time": "2021-05-19T14:08:25.312510",
"exception": false,
"start_time": "2021-05-19T14:08:25.206468",
"status": "completed"
},
"tags": [],
"id": "arbitrary-external"
},
"source": [
"# Dividing data to features and targets\n",
"X = train_df.drop('Survived', axis = 1)\n",
"y = train_df['Survived']"
],
"id": "arbitrary-external",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"execution": {
"iopub.execute_input": "2021-05-19T14:08:25.514022Z",
"iopub.status.busy": "2021-05-19T14:08:25.513575Z",
"iopub.status.idle": "2021-05-19T14:08:25.883200Z",
"shell.execute_reply": "2021-05-19T14:08:25.883570Z"
},
"papermill": {
"duration": 0.471693,
"end_time": "2021-05-19T14:08:25.883709",
"exception": false,
"start_time": "2021-05-19T14:08:25.412016",
"status": "completed"
},
"tags": [],
"id": "ecological-honolulu"
},
"source": [
"# splitting data intp training and test set\n",
"\n",
"from sklearn.model_selection import train_test_split\n",
"\n",
"X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.30, random_state = 0)"
],
"id": "ecological-honolulu",
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"papermill": {
"duration": 0.099773,
"end_time": "2021-05-19T14:08:26.084073",
"exception": false,
"start_time": "2021-05-19T14:08:25.984300",
"status": "completed"
},
"tags": [],
"id": "adolescent-township"
},
"source": [
"Data Modeling
\n",
"\n",
"Training Machine Learning models to predict survivors
"
],
"id": "adolescent-township"
},
{
"cell_type": "markdown",
"metadata": {
"papermill": {
"duration": 0.099202,
"end_time": "2021-05-19T14:08:26.283145",
"exception": false,
"start_time": "2021-05-19T14:08:26.183943",
"status": "completed"
},
"tags": [],
"id": "mounted-windows"
},
"source": [
"1. Logistic Regression
"
],
"id": "mounted-windows"
},
{
"cell_type": "code",
"metadata": {
"execution": {
"iopub.execute_input": "2021-05-19T14:08:26.486843Z",
"iopub.status.busy": "2021-05-19T14:08:26.486406Z",
"iopub.status.idle": "2021-05-19T14:08:26.616797Z",
"shell.execute_reply": "2021-05-19T14:08:26.616355Z"
},
"papermill": {
"duration": 0.2332,
"end_time": "2021-05-19T14:08:26.616918",
"exception": false,
"start_time": "2021-05-19T14:08:26.383718",
"status": "completed"
},
"tags": [],
"id": "gross-scotland"
},
"source": [
"from sklearn.linear_model import LogisticRegression\n",
"\n",
"lr = LogisticRegression()\n",
"lr.fit(X_train, y_train)\n",
"\n",
"# accuracy score, confusion matrix and classification report of logistic regression\n",
"\n",
"from sklearn.metrics import accuracy_score, confusion_matrix, classification_report\n",
"\n",
"lr_acc = accuracy_score(y_test, lr.predict(X_test))\n",
"\n",
"print(f\"Training Accuracy of Logistic Regression is {accuracy_score(y_train, lr.predict(X_train))}\")\n",
"print(f\"Test Accuracy of Logistic Regression is {lr_acc}\")\n",
"\n",
"print(f\"Confusion Matrix :- \\n {confusion_matrix(y_test, lr.predict(X_test))}\")\n",
"print(f\"Classofocation Report : -\\n {classification_report(y_test, lr.predict(X_test))}\")"
],
"id": "gross-scotland",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"execution": {
"iopub.execute_input": "2021-05-19T14:08:26.822702Z",
"iopub.status.busy": "2021-05-19T14:08:26.822115Z",
"iopub.status.idle": "2021-05-19T14:08:28.570110Z",
"shell.execute_reply": "2021-05-19T14:08:28.569687Z"
},
"papermill": {
"duration": 1.853953,
"end_time": "2021-05-19T14:08:28.570217",
"exception": false,
"start_time": "2021-05-19T14:08:26.716264",
"status": "completed"
},
"tags": [],
"id": "casual-orientation"
},
"source": [
"# hyper parameter tuning of logistic regression\n",
"from sklearn.model_selection import GridSearchCV\n",
"\n",
"grid_param = {\n",
" 'penalty': ['l1', 'l2'],\n",
" 'C' : [0.001, 0.01, 0.1, 0.005, 0.5, 1, 10]\n",
"}\n",
"\n",
"grid_search_lr = GridSearchCV(lr, grid_param, cv = 5, n_jobs = -1, verbose = 1)\n",
"grid_search_lr.fit(X_train, y_train)"
],
"id": "casual-orientation",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"execution": {
"iopub.execute_input": "2021-05-19T14:08:28.777354Z",
"iopub.status.busy": "2021-05-19T14:08:28.776665Z",
"iopub.status.idle": "2021-05-19T14:08:28.780007Z",
"shell.execute_reply": "2021-05-19T14:08:28.779604Z"
},
"papermill": {
"duration": 0.109751,
"end_time": "2021-05-19T14:08:28.780116",
"exception": false,
"start_time": "2021-05-19T14:08:28.670365",
"status": "completed"
},
"tags": [],
"id": "motivated-control"
},
"source": [
"# best parameters and best score\n",
"\n",
"print(grid_search_lr.best_params_)\n",
"print(grid_search_lr.best_score_)"
],
"id": "motivated-control",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"execution": {
"iopub.execute_input": "2021-05-19T14:08:28.990744Z",
"iopub.status.busy": "2021-05-19T14:08:28.988940Z",
"iopub.status.idle": "2021-05-19T14:08:29.002490Z",
"shell.execute_reply": "2021-05-19T14:08:29.002118Z"
},
"papermill": {
"duration": 0.12053,
"end_time": "2021-05-19T14:08:29.002595",
"exception": false,
"start_time": "2021-05-19T14:08:28.882065",
"status": "completed"
},
"tags": [],
"id": "spiritual-puppy"
},
"source": [
"# best estimator\n",
"\n",
"lr = grid_search_lr.best_estimator_\n",
"\n",
"# accuracy score, confusion matrix and classification report of logistic regression\n",
"\n",
"lr_acc = accuracy_score(y_test, lr.predict(X_test))\n",
"\n",
"print(f\"Training Accuracy of Logistic Regression is {accuracy_score(y_train, lr.predict(X_train))}\")\n",
"print(f\"Test Accuracy of Logistic Regression is {lr_acc}\")\n",
"\n",
"print(f\"Confusion Matrix :- \\n {confusion_matrix(y_test, lr.predict(X_test))}\")\n",
"print(f\"Classofocation Report : -\\n {classification_report(y_test, lr.predict(X_test))}\")"
],
"id": "spiritual-puppy",
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"papermill": {
"duration": 0.153113,
"end_time": "2021-05-19T14:08:41.327013",
"exception": false,
"start_time": "2021-05-19T14:08:41.173900",
"status": "completed"
},
"tags": [],
"id": "parliamentary-constitution"
},
"source": [
"2. Random Forest
"
],
"id": "parliamentary-constitution"
},
{
"cell_type": "code",
"metadata": {
"execution": {
"iopub.execute_input": "2021-05-19T14:08:41.642248Z",
"iopub.status.busy": "2021-05-19T14:08:41.641568Z",
"iopub.status.idle": "2021-05-19T14:08:41.971499Z",
"shell.execute_reply": "2021-05-19T14:08:41.970991Z"
},
"papermill": {
"duration": 0.489435,
"end_time": "2021-05-19T14:08:41.971628",
"exception": false,
"start_time": "2021-05-19T14:08:41.482193",
"status": "completed"
},
"tags": [],
"id": "dangerous-price"
},
"source": [
"from sklearn.ensemble import RandomForestClassifier\n",
"\n",
"rd_clf = RandomForestClassifier()\n",
"rd_clf.fit(X_train, y_train)\n",
"\n",
"# accuracy score, confusion matrix and classification report of random forest\n",
"\n",
"rd_clf_acc = accuracy_score(y_test, rd_clf.predict(X_test))\n",
"\n",
"print(f\"Training Accuracy of Decision Tree Classifier is {accuracy_score(y_train, rd_clf.predict(X_train))}\")\n",
"print(f\"Test Accuracy of Decision Tree Classifier is {rd_clf_acc} \\n\")\n",
"\n",
"print(f\"Confusion Matrix :- \\n{confusion_matrix(y_test, rd_clf.predict(X_test))}\\n\")\n",
"print(f\"Classification Report :- \\n {classification_report(y_test, rd_clf.predict(X_test))}\")"
],
"id": "dangerous-price",
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"papermill": {
"duration": 0.097421,
"end_time": "2021-05-19T14:08:59.699400",
"exception": false,
"start_time": "2021-05-19T14:08:59.601979",
"status": "completed"
},
"tags": [],
"id": "vulnerable-prediction"
},
"source": [
"3. Gradient Boosting
"
],
"id": "vulnerable-prediction"
},
{
"cell_type": "code",
"metadata": {
"execution": {
"iopub.execute_input": "2021-05-19T14:08:59.901067Z",
"iopub.status.busy": "2021-05-19T14:08:59.900604Z",
"iopub.status.idle": "2021-05-19T14:08:59.997423Z",
"shell.execute_reply": "2021-05-19T14:08:59.997079Z"
},
"papermill": {
"duration": 0.199501,
"end_time": "2021-05-19T14:08:59.997521",
"exception": false,
"start_time": "2021-05-19T14:08:59.798020",
"status": "completed"
},
"tags": [],
"id": "mounted-smell"
},
"source": [
"from sklearn.ensemble import GradientBoostingClassifier\n",
"\n",
"gb = GradientBoostingClassifier()\n",
"gb.fit(X_train, y_train)\n",
"\n",
"# accuracy score, confusion matrix and classification report of gradient boosting classifier\n",
"\n",
"gb_acc = accuracy_score(y_test, gb.predict(X_test))\n",
"\n",
"print(f\"Training Accuracy of Decision Tree Classifier is {accuracy_score(y_train, gb.predict(X_train))}\")\n",
"print(f\"Test Accuracy of Decision Tree Classifier is {gb_acc} \\n\")\n",
"\n",
"print(f\"Confusion Matrix :- \\n{confusion_matrix(y_test, gb.predict(X_test))}\\n\")\n",
"print(f\"Classification Report :- \\n {classification_report(y_test, gb.predict(X_test))}\")"
],
"id": "mounted-smell",
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"papermill": {
"duration": 0.097718,
"end_time": "2021-05-19T14:09:00.193371",
"exception": false,
"start_time": "2021-05-19T14:09:00.095653",
"status": "completed"
},
"tags": [],
"id": "sonic-cliff"
},
"source": [
"4. Stochastic Gradient Boosting (SGB)
"
],
"id": "sonic-cliff"
},
{
"cell_type": "code",
"metadata": {
"execution": {
"iopub.execute_input": "2021-05-19T14:09:00.399307Z",
"iopub.status.busy": "2021-05-19T14:09:00.398815Z",
"iopub.status.idle": "2021-05-19T14:09:00.493810Z",
"shell.execute_reply": "2021-05-19T14:09:00.493418Z"
},
"papermill": {
"duration": 0.202453,
"end_time": "2021-05-19T14:09:00.493927",
"exception": false,
"start_time": "2021-05-19T14:09:00.291474",
"status": "completed"
},
"tags": [],
"id": "stunning-dream"
},
"source": [
"sgb = GradientBoostingClassifier(subsample = 0.90, max_features = 0.70)\n",
"sgb.fit(X_train, y_train)\n",
"\n",
"# accuracy score, confusion matrix and classification report of stochastic gradient boosting classifier\n",
"\n",
"sgb_acc = accuracy_score(y_test, sgb.predict(X_test))\n",
"\n",
"print(f\"Training Accuracy of Decision Tree Classifier is {accuracy_score(y_train, sgb.predict(X_train))}\")\n",
"print(f\"Test Accuracy of Decision Tree Classifier is {sgb_acc} \\n\")\n",
"\n",
"print(f\"Confusion Matrix :- \\n{confusion_matrix(y_test, sgb.predict(X_test))}\\n\")\n",
"print(f\"Classification Report :- \\n {classification_report(y_test, sgb.predict(X_test))}\")"
],
"id": "stunning-dream",
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"papermill": {
"duration": 0.107703,
"end_time": "2021-05-19T14:09:00.706867",
"exception": false,
"start_time": "2021-05-19T14:09:00.599164",
"status": "completed"
},
"tags": [],
"id": "intended-karma"
},
"source": [
"5. XGboost
"
],
"id": "intended-karma"
},
{
"cell_type": "code",
"metadata": {
"execution": {
"iopub.execute_input": "2021-05-19T14:09:00.931572Z",
"iopub.status.busy": "2021-05-19T14:09:00.931140Z",
"iopub.status.idle": "2021-05-19T14:09:01.173931Z",
"shell.execute_reply": "2021-05-19T14:09:01.174459Z"
},
"papermill": {
"duration": 0.361064,
"end_time": "2021-05-19T14:09:01.174641",
"exception": false,
"start_time": "2021-05-19T14:09:00.813577",
"status": "completed"
},
"tags": [],
"id": "employed-expansion"
},
"source": [
"from xgboost import XGBClassifier\n",
"\n",
"xgb = XGBClassifier(booster = 'gbtree', learning_rate = 0.1, max_depth = 5, n_estimators = 180)\n",
"xgb.fit(X_train, y_train)\n",
"\n",
"# accuracy score, confusion matrix and classification report of xgboost\n",
"\n",
"xgb_acc = accuracy_score(y_test, xgb.predict(X_test))\n",
"\n",
"print(f\"Training Accuracy of Decision Tree Classifier is {accuracy_score(y_train, xgb.predict(X_train))}\")\n",
"print(f\"Test Accuracy of Decision Tree Classifier is {xgb_acc} \\n\")\n",
"\n",
"print(f\"Confusion Matrix :- \\n{confusion_matrix(y_test, xgb.predict(X_test))}\\n\")\n",
"print(f\"Classification Report :- \\n {classification_report(y_test, xgb.predict(X_test))}\")"
],
"id": "employed-expansion",
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"papermill": {
"duration": 0.108607,
"end_time": "2021-05-19T14:09:06.914940",
"exception": false,
"start_time": "2021-05-19T14:09:06.806333",
"status": "completed"
},
"tags": [],
"id": "cross-tourism"
},
"source": [
"Model Comparison
"
],
"id": "cross-tourism"
},
{
"cell_type": "code",
"metadata": {
"execution": {
"iopub.execute_input": "2021-05-19T14:09:07.140641Z",
"iopub.status.busy": "2021-05-19T14:09:07.140186Z",
"iopub.status.idle": "2021-05-19T14:09:07.145890Z",
"shell.execute_reply": "2021-05-19T14:09:07.145469Z"
},
"papermill": {
"duration": 0.122396,
"end_time": "2021-05-19T14:09:07.146007",
"exception": false,
"start_time": "2021-05-19T14:09:07.023611",
"status": "completed"
},
"tags": [],
"id": "certain-accessory"
},
"source": [
"models = pd.DataFrame({\n",
" 'Model' : ['Logistic Regression', 'Gradient Boosting Classifier', 'Stochastic Gradient Boosting', 'XgBoost'], \n",
" 'Score' : [lr_acc,gb_acc, sgb_acc, xgb_acc]\n",
"})\n",
"\n",
"models.sort_values(by = 'Score', ascending = False)"
],
"id": "certain-accessory",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"execution": {
"iopub.execute_input": "2021-05-19T14:09:07.379088Z",
"iopub.status.busy": "2021-05-19T14:09:07.378391Z",
"iopub.status.idle": "2021-05-19T14:09:07.625087Z",
"shell.execute_reply": "2021-05-19T14:09:07.624605Z"
},
"papermill": {
"duration": 0.366988,
"end_time": "2021-05-19T14:09:07.625188",
"exception": false,
"start_time": "2021-05-19T14:09:07.258200",
"status": "completed"
},
"tags": [],
"id": "optimum-handling"
},
"source": [
"plt.figure(figsize = (15, 10))\n",
"\n",
"sns.barplot(x = 'Score', y = 'Model', data = models)\n",
"plt.show()"
],
"id": "optimum-handling",
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"papermill": {
"duration": 0.166379,
"end_time": "2021-05-19T14:09:08.207799",
"exception": false,
"start_time": "2021-05-19T14:09:08.041420",
"status": "completed"
},
"tags": [],
"id": "executive-implement"
},
"source": [
"Real Data Model Inference
"
],
"id": "executive-implement"
},
{
"cell_type": "code",
"metadata": {
"id": "277219f8"
},
"source": [
"pred_class = {0: 'Not Survived', 1: 'Survived'}\n",
"def predict(test_input, test_target):\n",
" final_results = gb.predict([test_input])\n",
" test_input['Age'] = np.exp(test_input['Age'])\n",
" test_input['Fare'] = np.exp(test_input['Fare'])\n",
" print(f\"Features:\\n{test_input}\\n\\nClassifier prediction: {pred_class[final_results.item()]}\\nActual Target: {pred_class[test_target]}\")\n",
" \n"
],
"id": "277219f8",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "5f5011eb"
},
"source": [
"test_input = X.iloc[0,:]\n",
"test_target = y.iloc[0]\n",
"predict(test_input, test_target)"
],
"id": "5f5011eb",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "9d34da40"
},
"source": [
"test_input = X.iloc[1,:]\n",
"test_target = y.iloc[1]\n",
"predict(test_input, test_target)"
],
"id": "9d34da40",
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "006866b9"
},
"source": [
"4. Approaching a Text (NLP) Problem
\n",
"\n",
"
\n"
],
"id": "006866b9"
},
{
"cell_type": "markdown",
"metadata": {
"id": "bc5ec3ec"
},
"source": [
"## 4.1 Importance of solving NLP"
],
"id": "bc5ec3ec"
},
{
"cell_type": "markdown",
"metadata": {
"id": "m4V2swA5V6t6"
},
"source": [
"NLP — also known as computational linguistics — is the combination of AI and linguistics that allows us to talk to machines as if they were human.\n",
"\n",
"\n",
"**In other words, NLP is an approach to process, analyze and understand large amount of text data.**"
],
"id": "m4V2swA5V6t6"
},
{
"cell_type": "markdown",
"metadata": {
"id": "jM_IQ5lHWg5c"
},
"source": [
""
],
"id": "jM_IQ5lHWg5c"
},
{
"cell_type": "markdown",
"metadata": {
"id": "f3VVBg0hWv_X"
},
"source": [
"### **1. Handling large volumes of text data**\n",
"\n",
"With the big data technology, NLP has entered the mainstream as this approach can now be applied to handle large volumes of text data via cloud/distributed computing at an unprecedented speed.\n"
],
"id": "f3VVBg0hWv_X"
},
{
"cell_type": "markdown",
"metadata": {
"id": "ctqdYrZ9XF5M"
},
"source": [
"### 2. Structuring highly unstructured data source\n",
"\n",
"Human language is astoundingly complex and diverse.\n",
"\n",
"NLP is important because it helps resolve ambiguity in language and adds useful numeric structure to the data for many downstream applications, such as speech recognition or text analytics."
],
"id": "ctqdYrZ9XF5M"
},
{
"cell_type": "markdown",
"metadata": {
"id": "pgTP_KTxYb-n"
},
"source": [
"## Components of NLP\n",
"\n",
"NLP can be divided into two basic components.\n",
"\n",
"* Natural Language Understanding\n",
"* Natural Language Generation\n"
],
"id": "pgTP_KTxYb-n"
},
{
"cell_type": "markdown",
"metadata": {
"id": "g0vcgPzhaC0P"
},
"source": [
"#### Natural Language Understanding (NLU)\n",
"\n",
"There are lot of ambiguity while learning or trying to interpret a language.\n",
"\n",
"* Lexical Ambiguity can occur when a word carries different sense\n",
"* Syntactical Ambiguity means when we see more than one meaning in a sequence of words. \n",
"* Referential Ambiguity: Very often a text mentions as entity (something/someone),"
],
"id": "g0vcgPzhaC0P"
},
{
"cell_type": "markdown",
"metadata": {
"id": "tYtjYCaNaZKJ"
},
"source": [
"### Natural Language Generation (NLG)\n",
"\n",
"It involves :\n",
"\n",
"* Text planning − It includes retrieving the relevant content from knowledge base.\n",
"* Sentence planning − It includes choosing required words, forming meaningful phrases, setting tone of the sentence.\n",
"* Text Realization − It is mapping sentence plan into sentence structure."
],
"id": "tYtjYCaNaZKJ"
},
{
"cell_type": "markdown",
"metadata": {
"id": "eceGeHrBdO30"
},
"source": [
"## **Applications of NLP**"
],
"id": "eceGeHrBdO30"
},
{
"cell_type": "markdown",
"metadata": {
"id": "vJuqd3lrdbbt"
},
"source": [
"### Search Autocorrect and Autocomplete\n",
"\n",
""
],
"id": "vJuqd3lrdbbt"
},
{
"cell_type": "markdown",
"metadata": {
"id": "QGSNgu8mdjP2"
},
"source": [
"### Language Translator\n",
"\n",
""
],
"id": "QGSNgu8mdjP2"
},
{
"cell_type": "markdown",
"metadata": {
"id": "GqLfpY43dr2Y"
},
"source": [
"### Chatbots\n",
"\n",
""
],
"id": "GqLfpY43dr2Y"
},
{
"cell_type": "markdown",
"metadata": {
"id": "LWDzqBmSd3O9"
},
"source": [
"# Sentiment analysis\n",
"\n",
""
],
"id": "LWDzqBmSd3O9"
},
{
"cell_type": "markdown",
"metadata": {
"id": "acdab842"
},
"source": [
"4.3 Basics of NLP
\n",
"\n",
"Corpus
\n",
"A corpus is a large and structured set of machine-readable texts that have been produced in a natural communicative setting. Its plural is corpora. They can be derived in different ways like text that was originally electronic, transcripts of spoken language and optical character recognition, etc.
\n",
"\n",
"Tokens
\n",
"The raw text is a sequence of characters (bytes), but most times it is useful to group those characters into contiguous units called tokens. In English, tokens correspond to words and numeric sequences separated by white-space characters or punctuation.
\n",
"\n",
"Tokenization
\n",
"The process of breaking a text down into tokens is called tokenization.It can become more complicated than simply splitting text based on nonalphanumeric characters.
\n",
"\n",
"N-grams
\n",
"They are basically a set of co-occurring words within a given window and when computing the n-grams you typically move one word forward. Trigram has three token, bigram has two tokens, a unigram one.
\n",
"\n",
"\n",
"\n"
],
"id": "acdab842"
},
{
"cell_type": "markdown",
"metadata": {
"id": "7bafa6b8"
},
"source": [
"Stemming
\n",
"Stemming algorithms work by cutting off the end or the beginning of the word, taking into account a list of common prefixes and suffixes that can be found in an inflected word.Stemming just removes the last few characters, often leading to incorrect meanings and spelling errors
\n",
"\n",
"Lemmatization
\n",
"Lemmatization is similar to stemming but it brings context to the words. So it links words with similar meaning to one word.
"
],
"id": "7bafa6b8"
},
{
"cell_type": "markdown",
"metadata": {
"id": "c48cf800"
},
"source": [
"4.4 NLP using Python
\n",
"\n",
"NLTK
\n",
"It is a leading platform for building Python programs to work with human language data. It provides easy-to-use interfaces to over 50 corpora and lexical resources such as WordNet, along with a suite of text processing libraries for classification, tokenization, stemming, tagging, parsing, and semantic reasoning, wrappers for industrial-strength NLP libraries
\n",
"\n",
"spaCy
\n",
"It is an open-source software library for advanced natural language processing, written in the programming languages Python and Cython.Unlike NLTK, which is widely used for teaching and research, spaCy focuses on providing software for production usage.spaCy also supports deep learning workflows that allow connecting statistical models trained by popular machine learning libraries like TensorFlow, PyTorch or MXNet through its own machine learning library Thinc
"
],
"id": "c48cf800"
},
{
"cell_type": "markdown",
"metadata": {
"id": "4ec1df71"
},
"source": [
"4.5 Approaching real life NLP problem
\n",
"\n",
"Intro to Twitter tweets processing using Python
\n",
"\n",
"
\n"
],
"id": "4ec1df71"
},
{
"cell_type": "markdown",
"metadata": {
"id": "2b2caa89"
},
"source": [
"Problem Statement
\n",
"\n",
"With all of the tweets circulating every second it is hard to tell whether the sentiment behind a specific tweet will impact a company, or a person's, brand for being viral (positive), or devastate profit because it strikes a negative tone. Capturing sentiment in language is important in these times where decisions and reactions are created and updated in seconds. In order to get the sentiments out of tweets they need to by processed to tweaked in ways so that it could be fed into a algorithm to perform to sentiment inference. So we will take a look at pythonic approach of processing tweets
"
],
"id": "2b2caa89"
},
{
"cell_type": "code",
"metadata": {
"id": "41075983"
},
"source": [
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"import seaborn as sns\n",
"import numpy as np\n",
"from nltk.corpus import stopwords\n",
"from nltk.util import ngrams\n",
"from sklearn.feature_extraction.text import CountVectorizer\n",
"from sklearn.feature_extraction.text import TfidfVectorizer\n",
"from wordcloud import WordCloud, STOPWORDS\n",
"from nltk.stem.wordnet import WordNetLemmatizer\n",
"from nltk import word_tokenize\n",
"from collections import defaultdict\n",
"from collections import Counter\n",
"plt.style.use('ggplot')\n",
"stop=set(stopwords.words('english'))\n",
"import re\n",
"from nltk.tokenize import word_tokenize\n",
"import gensim\n",
"import string\n",
"from tqdm import tqdm\n"
],
"id": "41075983",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "7e030a21"
},
"source": [
"tweet=pd.read_csv(\"./inputs/nlp/train.csv\")\n",
"test=pd.read_csv(\"./inputs/nlp/test.csv\")"
],
"id": "7e030a21",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "784f9a32"
},
"source": [
"tweet.head(100)"
],
"id": "784f9a32",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "927e46dc"
},
"source": [
"print('There are {} rows and {} columns in train'.format(tweet.shape[0],tweet.shape[1]))\n",
"print('There are {} rows and {} columns in train'.format(test.shape[0],test.shape[1]))"
],
"id": "927e46dc",
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "5c3b6dd1"
},
"source": [
"Class Distribution
\n",
"\n",
"Before we begin with anything else,let's check the class distribution.There are only two classes 0 and 1.
\n"
],
"id": "5c3b6dd1"
},
{
"cell_type": "code",
"metadata": {
"id": "f369049e"
},
"source": [
"x=tweet.target.value_counts()\n",
"sns.barplot(x.index,x)\n",
"plt.gca().set_ylabel('samples')"
],
"id": "f369049e",
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "a5dfef16"
},
"source": [
"Exploratory Data Analysis (EDA)
"
],
"id": "a5dfef16"
},
{
"cell_type": "code",
"metadata": {
"id": "e9f68ec8"
},
"source": [
"#First,we will do very basic analysis,that is character level,word level and sentence level analysis.\n",
"fig,(ax1,ax2)=plt.subplots(1,2,figsize=(10,5))\n",
"tweet_len=tweet[tweet['target']==1]['text'].str.len()\n",
"ax1.hist(tweet_len,color='red')\n",
"ax1.set_title('disaster tweets')\n",
"tweet_len=tweet[tweet['target']==0]['text'].str.len()\n",
"ax2.hist(tweet_len,color='green')\n",
"ax2.set_title('Not disaster tweets')\n",
"fig.suptitle('Characters in tweets')\n",
"plt.show()\n",
"\n",
"print(\"The distribution of both seems to be almost same.120 t0 140 characters in a tweet are the most common among both.\")"
],
"id": "e9f68ec8",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "15e192cb"
},
"source": [
"# Number of words in tweet\n",
"fig,(ax1,ax2)=plt.subplots(1,2,figsize=(10,5))\n",
"tweet_len=tweet[tweet['target']==1]['text'].str.split().map(lambda x: len(x))\n",
"ax1.hist(tweet_len,color='red')\n",
"ax1.set_title('disaster tweets')\n",
"tweet_len=tweet[tweet['target']==0]['text'].str.split().map(lambda x: len(x))\n",
"ax2.hist(tweet_len,color='green')\n",
"ax2.set_title('Not disaster tweets')\n",
"fig.suptitle('Words in a tweet')\n",
"plt.show()\n",
"\n",
"print(\"Most of disaster tweets are around between 10 -20 word counts\\nand Non disaster tweet areound 15 to 20\")"
],
"id": "15e192cb",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "2ae980eb"
},
"source": [
"# Average word length in a tweet\n",
"fig,(ax1,ax2)=plt.subplots(1,2,figsize=(10,5))\n",
"word=tweet[tweet['target']==1]['text'].str.split().apply(lambda x : [len(i) for i in x])\n",
"sns.distplot(word.map(lambda x: np.mean(x)),ax=ax1,color='red')\n",
"ax1.set_title('disaster')\n",
"word=tweet[tweet['target']==0]['text'].str.split().apply(lambda x : [len(i) for i in x])\n",
"sns.distplot(word.map(lambda x: np.mean(x)),ax=ax2,color='green')\n",
"ax2.set_title('Not disaster')\n",
"fig.suptitle('Average word length in each tweet')\n"
],
"id": "2ae980eb",
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "a26f0c7c"
},
"source": [
"Creating Corpus
\n",
"\n",
"We need to perform further EDA we need corpus
\n"
],
"id": "a26f0c7c"
},
{
"cell_type": "code",
"metadata": {
"id": "7fa39f5c"
},
"source": [
"#creating text corpus based on target\n",
"#corpus of Disaster vs Non Disaster Tweet\n",
"\n",
"def build_corpus(target):\n",
" corpus=[]\n",
" \n",
" for x in tweet[tweet['target']==target]['text'].str.split():\n",
" for i in x:\n",
" corpus.append(i)\n",
" return corpus"
],
"id": "7fa39f5c",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "4a33faed"
},
"source": [
"#wordcloud of non disaster tweets\n",
"comment_words = ''\n",
"stopwords = set(STOPWORDS)\n",
"comment_words += \" \".join(build_corpus(0))+\" \"\n",
"\n",
"wordcloud = WordCloud(width = 800, height = 800,\n",
" background_color ='white',\n",
" stopwords = stopwords,\n",
" min_font_size = 10).generate(comment_words)\n",
"\n",
"plt.figure(figsize = (8, 8), facecolor = None)\n",
"plt.imshow(wordcloud)\n",
"plt.axis(\"off\")\n",
"plt.tight_layout(pad = 0)\n",
"plt.show()"
],
"id": "4a33faed",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "4b5f89cf"
},
"source": [
"#wordcloud of disaster tweets\n",
"comment_words = ''\n",
"stopwords = set(STOPWORDS)\n",
"comment_words += \" \".join(build_corpus(1))+\" \"\n",
"\n",
"wordcloud = WordCloud(width = 800, height = 800,\n",
" background_color ='black',\n",
" stopwords = stopwords,\n",
" min_font_size = 10).generate(comment_words)\n",
"\n",
"plt.figure(figsize = (8, 8), facecolor = None)\n",
"plt.imshow(wordcloud)\n",
"plt.axis(\"off\")\n",
"plt.tight_layout(pad = 0)\n",
" \n",
"plt.show()"
],
"id": "4b5f89cf",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "143d08ef"
},
"source": [
"#creating corpus based of Non Disaster Tweet\n",
"corpus=build_corpus(0)\n",
"\n",
"#build dicr based on NoN disaster tweet\n",
"dic=defaultdict(int)\n",
"for word in corpus:\n",
" if word in stop:\n",
" dic[word]+=1\n",
" \n",
"#choosing top tweets\n",
"top=sorted(dic.items(), key=lambda x:x[1],reverse=True)[:10] \n",
"top"
],
"id": "143d08ef",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "443c251d"
},
"source": [
"#bar chart showing common words in NoN disaster tweets\n",
"x,y = zip(*top)\n",
"plt.figure(figsize=(15,8))\n",
"plt.bar(x, y);"
],
"id": "443c251d",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "f776c8cf"
},
"source": [
"#creating the same for Diaster Tweet\n",
"corpus=build_corpus(1)\n",
"\n",
"dic=defaultdict(int)\n",
"for word in corpus:\n",
" if word in stop:\n",
" dic[word]+=1\n",
" \n",
"top=sorted(dic.items(), key=lambda x:x[1],reverse=True)[:10] \n",
" \n",
"plt.figure(figsize=(15,8))\n",
"x,y=zip(*top)\n",
"plt.bar(x,y)\n",
"plt.show()\n",
"\n",
"print(\"Words like 'the','in','of' dominates, but these words have not much meaning or prediction capabilities\")"
],
"id": "f776c8cf",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "349c8bd4"
},
"source": [
"#Analysing punctuations for Disaster Tweets\n",
"plt.figure(figsize=(15,8))\n",
"corpus=build_corpus(1)\n",
"\n",
"dic=defaultdict(int)\n",
"import string\n",
"special = string.punctuation\n",
"for i in (corpus):\n",
" if i in special:\n",
" dic[i]+=1\n",
" \n",
"x,y=zip(*dic.items())\n",
"plt.title(\"Analysing punctuations for Disaster Tweets\")\n",
"plt.bar(x,y,color='orange');\n"
],
"id": "349c8bd4",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "e954149c"
},
"source": [
"#Analysing punctuations for Non-Disaster Tweets\n",
"plt.figure(figsize=(15,8))\n",
"corpus=build_corpus(0)\n",
"\n",
"dic=defaultdict(int)\n",
"import string\n",
"special = string.punctuation\n",
"for i in (corpus):\n",
" if i in special:\n",
" dic[i]+=1\n",
" \n",
"x,y=zip(*dic.items())\n",
"plt.title(\"Analysing punctuations for Non-Disaster Tweets\")\n",
"plt.bar(x,y,color='red');"
],
"id": "e954149c",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "1c946180"
},
"source": [
"#performing the same for complete corpus\n",
"complete_corpus = []\n",
"complete_corpus.extend(build_corpus(0))\n",
"complete_corpus.extend(build_corpus(1))\n",
"counter=Counter(corpus)\n",
"most=counter.most_common()\n",
"x=[]\n",
"y=[]\n",
"for word,count in most[:40]:\n",
" if (word not in stop) :\n",
" x.append(word)\n",
" y.append(count)\n",
"plt.figure(figsize=(15,8))\n",
"sns.barplot(x=y,y=x).set_title(\"Complete corpus common words\")\n",
"plt.show()"
],
"id": "1c946180",
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "7f68c536"
},
"source": [
"N-gram Analysis
\n",
"\n",
"We can perform Bigram and Trigram analysis on this data to understand word combinations in the dataset
\n",
"\n"
],
"id": "7f68c536"
},
{
"cell_type": "code",
"metadata": {
"id": "94e5b1c0"
},
"source": [
"def get_top_tweet_bigrams(corpus, gram=2, n=None):\n",
" vec = CountVectorizer(ngram_range=(gram, gram)).fit(corpus)\n",
" bag_of_words = vec.transform(corpus)\n",
" sum_words = bag_of_words.sum(axis=0) \n",
" words_freq = [(word, sum_words[0, idx]) for word, idx in vec.vocabulary_.items()]\n",
" words_freq =sorted(words_freq, key = lambda x: x[1], reverse=True)\n",
" return words_freq[:n]"
],
"id": "94e5b1c0",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "ad2b33c9"
},
"source": [
"#count of common bigrams\n",
"plt.figure(figsize=(15,8))\n",
"top_tweet_bigrams=get_top_tweet_bigrams(tweet['text'], gram=2)[:10]\n",
"x,y=map(list,zip(*top_tweet_bigrams))\n",
"sns.barplot(x=y,y=x).set_title(\"Common Bigram Count\")\n",
"plt.show()"
],
"id": "ad2b33c9",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "ddcd245e"
},
"source": [
"#count of common trigrams\n",
"plt.figure(figsize=(15,8))\n",
"top_tweet_trigrams=get_top_tweet_bigrams(tweet['text'], gram=3)[:10]\n",
"x,y=map(list,zip(*top_tweet_trigrams))\n",
"sns.barplot(x=y,y=x).set_title(\"Common Trigram Count\")\n",
"plt.show()"
],
"id": "ddcd245e",
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "7c7f5e82"
},
"source": [
"After performing Bigram and Trigram analysis on this data clearly shows the need of data cleaning
"
],
"id": "7c7f5e82"
},
{
"cell_type": "markdown",
"metadata": {
"id": "4199317e"
},
"source": [
"Data Cleaning
\n",
"\n",
"Raw twitter data contains a lot of noise like stopwords, irrelavant punctuations, emojis ..etc which could kill its prediction capabilities. So let's clean them
\n",
"\n"
],
"id": "4199317e"
},
{
"cell_type": "code",
"metadata": {
"id": "d91d7534"
},
"source": [
"df=pd.concat([tweet,test])\n",
"df.shape"
],
"id": "d91d7534",
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "80618224"
},
"source": [
"Lemmatization
"
],
"id": "80618224"
},
{
"cell_type": "code",
"metadata": {
"id": "85f6917a"
},
"source": [
"def lemma(text):\n",
" text = text.lower()\n",
" lmtzr = WordNetLemmatizer()\n",
" text = \" \".join([lmtzr.lemmatize(word) for word in word_tokenize(text)])\n",
" return text\n",
"\n",
"ex = \"Our Deeds are the Reason of this #earthquake\"\n",
"lemma(ex)"
],
"id": "85f6917a",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "57fb679f"
},
"source": [
"#apply lemmatization\n",
"df['text']=df['text'].apply(lambda x : lemma(x))"
],
"id": "57fb679f",
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "22b2f525"
},
"source": [
"Removing URL
"
],
"id": "22b2f525"
},
{
"cell_type": "code",
"metadata": {
"id": "3ab272e6"
},
"source": [
"#Regex to remove URL\n",
"def remove_URL(text):\n",
" url = re.compile(r'https?://\\S+|www\\.\\S+')\n",
" return url.sub(r'',text)\n",
"\n",
"sample_URL = \"please find the titanic project at https://www.kaggle.com/c/titanic/overview\"\n",
"remove_URL(sample_URL)"
],
"id": "3ab272e6",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "1a77ce09"
},
"source": [
"#Apply the cleaning process\n",
"df['text']=df['text'].apply(lambda x : remove_URL(x))"
],
"id": "1a77ce09",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "0df25952"
},
"source": [
"df[['text','target']].head()"
],
"id": "0df25952",
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "3d530aa8"
},
"source": [
"Removing HTML
"
],
"id": "3d530aa8"
},
{
"cell_type": "code",
"metadata": {
"id": "90cb2b2b"
},
"source": [
"#remove html\n",
"def remove_html(text):\n",
" html=re.compile(r'<.*?>')\n",
" return html.sub(r'',text)\n",
"\n",
"sample_HTML = '''\n",
"\n",
"\n",
"
Welcome to HTML\n",
"kaggle\n",
"
test paragraph
\n",
"
Removing Emojis 😔
\n",
"\n",
"Emoji's have great prediction capabilities but they need to be encoded to some numeric problem, Lets just remove it for simplicity sake
\n",
"\n"
],
"id": "e7e0769b"
},
{
"cell_type": "code",
"metadata": {
"id": "5a4ebe7d"
},
"source": [
"def remove_emoji(text):\n",
" emoji_pattern = re.compile(\"[\"\n",
" u\"\\U0001F600-\\U0001F64F\" # emoticons\n",
" u\"\\U0001F300-\\U0001F5FF\" # symbols & pictographs\n",
" u\"\\U0001F680-\\U0001F6FF\" # transport & map symbols\n",
" u\"\\U0001F1E0-\\U0001F1FF\" # flags (iOS)\n",
" u\"\\U00002702-\\U000027B0\"\n",
" u\"\\U000024C2-\\U0001F251\"\n",
" \"]+\", flags=re.UNICODE)\n",
" return emoji_pattern.sub(r'', text)\n"
],
"id": "5a4ebe7d",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "8eaf1eb0"
},
"source": [
"\n",
"remove_emoji(\"its flooding i am scared 😔😔\")\n",
"\n"
],
"id": "8eaf1eb0",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "a6f0d4cb"
},
"source": [
"df['text']=df['text'].apply(lambda x: remove_emoji(x))\n",
"df[['text','target']].head()"
],
"id": "a6f0d4cb",
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "82c2b58a"
},
"source": [
"Removing punctuations
\n",
"\n",
"Just like emojis punctuation also posses some kind of meaning like :) :( ..etc ,\n",
"But its hard to encode such punctation to numeric formats when vectorizing text. So we can remove it
\n",
"\n"
],
"id": "82c2b58a"
},
{
"cell_type": "code",
"metadata": {
"id": "4eec6b24"
},
"source": [
"def remove_punct(text):\n",
" table=str.maketrans('','',string.punctuation)\n",
" return text.translate(table)\n",
"\n",
"example=\"There are #earthquakes\"\n",
"print(remove_punct(example))"
],
"id": "4eec6b24",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "3df59cc1"
},
"source": [
"df['text']=df['text'].apply(lambda x : remove_punct(x))\n",
"df[['text','target']].head()"
],
"id": "3df59cc1",
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "3313ced9"
},
"source": [
"Vectorization
\n",
"\n",
"Text needs to be encoded to numerical format before we feed it into a machine learning algorithm
\n",
"\n",
"Bag-Of-Words - The model is simple in that it throws away all of the order information in the words and focuses on the occurrence of words in a document.
\n",
"\n",
"\n",
"
\n"
],
"id": "3313ced9"
},
{
"cell_type": "markdown",
"metadata": {
"id": "5d9c77f5"
},
"source": [
"\n",
"BOW vectorization techniques
\n",
"\n",
"1. convert text to word count vectors with CountVectorizer
2. convert text to word frequency vectors with TfidfVectorizer.
\n",
"\n"
],
"id": "5d9c77f5"
},
{
"cell_type": "markdown",
"metadata": {
"id": "8b09721c"
},
"source": [
"Word Counts with CountVectorizer
\n",
"\n",
"\n",
"The CountVectorizer provides a simple way to both tokenize a collection of text documents and build a vocabulary of known words, but also to encode new documents using that vocabulary.
\n",
"\n"
],
"id": "8b09721c"
},
{
"cell_type": "code",
"metadata": {
"id": "40024021"
},
"source": [
"# list of text documents\n",
"text = [\"heard about the earthquake is different city stay safe everyone and hope the city fast recovery\"]\n",
"# create the transform\n",
"vectorizer = CountVectorizer()\n",
"# tokenize and build vocab\n",
"vectorizer.fit(text)\n",
"# summarize\n",
"print(vectorizer.vocabulary_)\n",
"# encode document\n",
"vector = vectorizer.transform(text)\n",
"# summarize encoded vector\n",
"print(vector.shape)\n",
"print(type(vector))\n",
"print(vector.toarray())"
],
"id": "40024021",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "4ba4c155"
},
"source": [
"#vectorizing the whole tweet\n",
"vectorizer = CountVectorizer()\n",
"vectorizer.fit(df.text.values)\n",
"vector = vectorizer.transform(df.text.values)\n",
"print(vector.shape)"
],
"id": "4ba4c155",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "69ef10ea"
},
"source": [
"#to see the vocab dictionary \n",
"# print(vectorizer.vocabulary_)\n"
],
"id": "69ef10ea",
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "3f078b9d"
},
"source": [
"Word Frequencies with TfidfVectorizer
\n",
"\n",
"TF-IDF are word frequency scores that try to highlight words that are more interesting, e.g. frequent in a document but not across documents.
\n",
"\n",
"* Term Frequency: This summarizes how often a given word appears within a document.
* Inverse Document Frequency: This downscales words that appear a lot across documents.
\n",
"\n"
],
"id": "3f078b9d"
},
{
"cell_type": "code",
"metadata": {
"id": "fc48325d"
},
"source": [
"# list of text documents\n",
"text = [\"The quick brown fox jumped over the lazy dog.\",\"The dog.\",\"The fox\"]\n",
"# create the transform\n",
"vectorizer = TfidfVectorizer()\n",
"# tokenize and build vocab\n",
"vectorizer.fit(text)\n",
"# summarize\n",
"print(vectorizer.vocabulary_)\n",
"print(vectorizer.idf_)\n",
"# encode document\n",
"vector = vectorizer.transform([text[0]])\n",
"# summarize encoded vector\n",
"print(vector.shape)\n",
"print(vector.toarray())"
],
"id": "fc48325d",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "ad881af0"
},
"source": [
"vectorizer = TfidfVectorizer()\n",
"# tokenize and build vocab\n",
"vectorizer.fit(df.text.values);"
],
"id": "ad881af0",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "3767faee"
},
"source": [
"# print(vectorizer.vocabulary_)"
],
"id": "3767faee",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "3f05a6f7"
},
"source": [
"print(vectorizer.idf_.shape)\n",
"print(vectorizer.idf_)"
],
"id": "3f05a6f7",
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "2ee8db5c"
},
"source": [
"vector = vectorizer.transform([text[0]])\n",
"# summarize encoded vector\n",
"print(vector.shape)\n",
"print(vector.toarray())"
],
"id": "2ee8db5c",
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "_2Pcu3EIMtQL"
},
"source": [
"------"
],
"id": "_2Pcu3EIMtQL"
},
{
"cell_type": "markdown",
"metadata": {
"id": "YLNGW22-lOYv"
},
"source": [
"# **What is Computer Vision**\n",
"\n",
"Two definitions of computer vision:\n",
"\n",
"**Computer vision can be defined as\n",
"a scientific field that extracts information out of digital images**. The type of\n",
"information gained from an image can vary from identification, space\n",
"measurements for navigation, or augmented reality applications.\n",
"
\n",
"Another way to define computer vision is through its applications.\n",
"\n",
"**Computer vision is building algorithms that can understand the content\n",
"of images and use it for other applications**.\n",
"\n",
"\n",
"![](\"https://media.geeksforgeeks.org/wp-content/uploads/python_working.png\")
"
],
"id": "42f285bf"
},
{
"cell_type": "markdown",
"metadata": {
"id": "891471fd"
},
"source": [
"
"
],
"id": "891471fd"
},
{
"cell_type": "markdown",
"metadata": {
"id": "6d884875"
},
"source": [
"
"
],
"id": "b1474823"
},
{
"cell_type": "markdown",
"metadata": {
"id": "15402732"
},
"source": [
"
"
],
"id": "86b00819"
},
{
"cell_type": "markdown",
"metadata": {
"id": "e03cd9e6"
},
"source": [
"
\n"
],
"id": "7d2f77c2"
},
{
"cell_type": "markdown",
"metadata": {
"id": "562393ef"
},
"source": [
"## 2.Pythonic way of data science"
],
"id": "562393ef"
},
{
"cell_type": "markdown",
"metadata": {
"id": "kK70oQo5Uzqx"
},
"source": [
"## **Brief Intro to Python Programming Language**\n",
"\n",
"**Python is an interpreted, high-level, general-purpose programming language. It is is dynamically typed and garbage-collected.**\n",
"\n",
"One of the most popular and fastest growing languages programming language in the world (StackOverflow survey 2020). First appeared on February 1991 (30 years ago). \n",
"\n",
"Highly efficient to write and programs tends to be clear and readable. well suitble for application like data science and web development.\n",
"\n",
"Used in a wide range of fields, from healthcare to finance to VFX and AI.\n",
"\n",
"The Father Of Python: Guido Van Rossum \n",
"\n"
],
"id": "kK70oQo5Uzqx"
},
{
"cell_type": "markdown",
"metadata": {
"id": "BrQydd54dW33"
},
"source": [
"\n",
"**Internal working of Python**\n",
"![](\"https://i.imgur.com/kPyXqOP.png\")
![](\"https://i.imgur.com/vmPJvQJ.png\")
![](\"https://i.imgur.com/bLju6gy.png\")
![](\"https://i.imgur.com/oKVvY5M.png2\")
\n",
"![](\"https://i.imgur.com/Doocv8r.png\")
![](\"https://i.imgur.com/37DIND1.png\")
![](\"https://i.imgur.com/RSxdapm.png\")
\n",
"![](\"https://i.imgur.com/p39Feyn.png\")
\n",
"![](\"https://i.imgur.com/2gl7tnY.png\")
\n",
"![](\"https://i.imgur.com/fvFL0C9.png\")
\n",
"![](\"https://i.imgur.com/QTzIFQe.png\")
\n",
"![](\"https://i.imgur.com/ULW8fxT.png\")
\n",
"![](\"https://i.imgur.com/ZGbRVlh.png\")
\n",
"![](\"https://upload.wikimedia.org/wikipedia/commons/thumb/d/d3/Waymo_Chrysler_Pacifica_in_Los_Altos%2C_2017.jpg/1280px-Waymo_Chrysler_Pacifica_in_Los_Altos%2C_2017.jpg\")
\n",
"![](\"https://i.imgur.com/DEklcEf.png\")
![](\"https://i.imgur.com/Kyue7jm.png\")
![](\"https://i.imgur.com/MSxkURR.png\")
![](\"https://i.imgur.com/y8H8jBT.png\")
\n",
"![](\"https://i.imgur.com/Q0CRBg8.png\")
![](\"https://www.pyimagesearch.com/wp-content/uploads/2015/03/pyramid_example.png\")
![](\"https://docs.opencv.org/4.5.2/histogram_sample.jpg\")