13_Dummy Variables AKA One-Hot Encoding.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 13_Dummy Variables AKA One-Hot Encoding\n",
    "\n",
    "In machine learning algorithms for regression or classification it is often necessary to encode categorical data so that it can be evaluated by the learning algorithm. This can be done by encoding each categorical value as a unique boolean feature (i.e. 1 for True, 0 for False). For example, if there was a feature of \"Gender\" with values of \"Male\", \"Female\", or \"Other\"; this can be encoded into 3 separate boolean columns; as below:\n",
    "\n",
    "\n",
    "| Gender | Male   | Female | Other |\n",
    "|------|------|------|------|\n",
    "|   Male  | 1 | 0 | 0 |\n",
    "|   Female  | 0 |1 | 0 |\n",
    "|   Other  | 0 | 0 | 1 |\n",
    "\n",
    "\n",
    "#### Example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Date</th>\n",
       "      <th>feature_1</th>\n",
       "      <th>feature_2</th>\n",
       "      <th>feature_3</th>\n",
       "      <th>feature_4</th>\n",
       "      <th>categorical_feature</th>\n",
       "      <th>weekday</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>01/02/2017</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>37</td>\n",
       "      <td>0</td>\n",
       "      <td>foo</td>\n",
       "      <td>Wednesday</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>02/02/2017</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>168</td>\n",
       "      <td>0</td>\n",
       "      <td>foo</td>\n",
       "      <td>Thursday</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>03/02/2017</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>157</td>\n",
       "      <td>0</td>\n",
       "      <td>other</td>\n",
       "      <td>Friday</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>04/02/2017</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>720</td>\n",
       "      <td>0</td>\n",
       "      <td>other</td>\n",
       "      <td>Saturday</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>05/02/2017</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>721</td>\n",
       "      <td>0</td>\n",
       "      <td>bar</td>\n",
       "      <td>Sunday</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         Date  feature_1  feature_2  feature_3  feature_4 categorical_feature  \\\n",
       "0  01/02/2017          0          0         37          0                 foo   \n",
       "1  02/02/2017          0          0        168          0                 foo   \n",
       "2  03/02/2017          0          0        157          0               other   \n",
       "3  04/02/2017          0          0        720          0               other   \n",
       "4  05/02/2017          0          0        721          0                 bar   \n",
       "\n",
       "     weekday  \n",
       "0  Wednesday  \n",
       "1   Thursday  \n",
       "2     Friday  \n",
       "3   Saturday  \n",
       "4     Sunday  "
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "# Load the example timeseries_daily.csv dataset:\n",
    "df = pd.read_csv(\"./data_etc/timeseries_daily.csv\")\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Use pandas <code>get_dummies()</code> to create the boolean columns:\n",
    "Running the function returns a DataFrame with the same length as the categorical feature, and number of columns equal to the number of unique values in the categorical feature."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>bar</th>\n",
       "      <th>foo</th>\n",
       "      <th>other</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>315</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>316</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>317</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>318</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>319</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     bar  foo  other\n",
       "315    0    1      0\n",
       "316    0    0      1\n",
       "317    1    0      0\n",
       "318    0    1      0\n",
       "319    0    0      1"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dummies = pd.get_dummies(df[\"categorical_feature\"])\n",
    "dummies.tail()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### This can be joined to the original DataFrame:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Date</th>\n",
       "      <th>feature_1</th>\n",
       "      <th>feature_2</th>\n",
       "      <th>feature_3</th>\n",
       "      <th>feature_4</th>\n",
       "      <th>categorical_feature</th>\n",
       "      <th>weekday</th>\n",
       "      <th>bar</th>\n",
       "      <th>foo</th>\n",
       "      <th>other</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>315</th>\n",
       "      <td>13/12/2017</td>\n",
       "      <td>657</td>\n",
       "      <td>1079</td>\n",
       "      <td>2128</td>\n",
       "      <td>454</td>\n",
       "      <td>foo</td>\n",
       "      <td>Wednesday</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>316</th>\n",
       "      <td>14/12/2017</td>\n",
       "      <td>968</td>\n",
       "      <td>1155</td>\n",
       "      <td>2116</td>\n",
       "      <td>163</td>\n",
       "      <td>other</td>\n",
       "      <td>Thursday</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>317</th>\n",
       "      <td>15/12/2017</td>\n",
       "      <td>820</td>\n",
       "      <td>1212</td>\n",
       "      <td>1934</td>\n",
       "      <td>290</td>\n",
       "      <td>bar</td>\n",
       "      <td>Friday</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>318</th>\n",
       "      <td>16/12/2017</td>\n",
       "      <td>1201</td>\n",
       "      <td>966</td>\n",
       "      <td>2475</td>\n",
       "      <td>497</td>\n",
       "      <td>foo</td>\n",
       "      <td>Saturday</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>319</th>\n",
       "      <td>17/12/2017</td>\n",
       "      <td>576</td>\n",
       "      <td>485</td>\n",
       "      <td>1809</td>\n",
       "      <td>285</td>\n",
       "      <td>other</td>\n",
       "      <td>Sunday</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           Date  feature_1  feature_2  feature_3  feature_4  \\\n",
       "315  13/12/2017        657       1079       2128        454   \n",
       "316  14/12/2017        968       1155       2116        163   \n",
       "317  15/12/2017        820       1212       1934        290   \n",
       "318  16/12/2017       1201        966       2475        497   \n",
       "319  17/12/2017        576        485       1809        285   \n",
       "\n",
       "    categorical_feature    weekday  bar  foo  other  \n",
       "315                 foo  Wednesday    0    1      0  \n",
       "316               other   Thursday    0    0      1  \n",
       "317                 bar     Friday    1    0      0  \n",
       "318                 foo   Saturday    0    1      0  \n",
       "319               other     Sunday    0    0      1  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_df = pd.concat([df, dummies], axis=1)\n",
    "new_df.tail()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### For clarity, you can add prefixes to the dummy columns:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>category: bar</th>\n",
       "      <th>category: foo</th>\n",
       "      <th>category: other</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>315</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>316</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>317</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>318</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>319</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     category: bar  category: foo  category: other\n",
       "315              0              1                0\n",
       "316              0              0                1\n",
       "317              1              0                0\n",
       "318              0              1                0\n",
       "319              0              0                1"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dummies_prefixed = pd.get_dummies(df[\"categorical_feature\"], prefix=\"category\", prefix_sep=\": \")\n",
    "dummies_prefixed.tail()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Don't fall for the \"<a href=\"http://www.algosome.com/articles/dummy-variable-trap-regression.html\" target=\"_blank\">Dummy Variable Trap</a>\":\n",
    "For machine learning purposes it is often necessary to drop one category from the dummy features created, and leave this as the \"default\" option; otherwise we run into problems with regression models where some of the categories may be correlated. Pandas <code>get_dummies</code> includes the option to exclude the first category:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>foo</th>\n",
       "      <th>other</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>315</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>316</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>317</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>318</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>319</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     foo  other\n",
       "315    1      0\n",
       "316    0      1\n",
       "317    0      0\n",
       "318    1      0\n",
       "319    0      1"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dummies_drop_first = pd.get_dummies(df[\"categorical_feature\"], drop_first=True)\n",
    "dummies_drop_first.tail()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Wrap this all up in one line of code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Date</th>\n",
       "      <th>feature_1</th>\n",
       "      <th>feature_2</th>\n",
       "      <th>feature_3</th>\n",
       "      <th>feature_4</th>\n",
       "      <th>categorical_feature</th>\n",
       "      <th>weekday</th>\n",
       "      <th>category_foo</th>\n",
       "      <th>category_other</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>315</th>\n",
       "      <td>13/12/2017</td>\n",
       "      <td>657</td>\n",
       "      <td>1079</td>\n",
       "      <td>2128</td>\n",
       "      <td>454</td>\n",
       "      <td>foo</td>\n",
       "      <td>Wednesday</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>316</th>\n",
       "      <td>14/12/2017</td>\n",
       "      <td>968</td>\n",
       "      <td>1155</td>\n",
       "      <td>2116</td>\n",
       "      <td>163</td>\n",
       "      <td>other</td>\n",
       "      <td>Thursday</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>317</th>\n",
       "      <td>15/12/2017</td>\n",
       "      <td>820</td>\n",
       "      <td>1212</td>\n",
       "      <td>1934</td>\n",
       "      <td>290</td>\n",
       "      <td>bar</td>\n",
       "      <td>Friday</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>318</th>\n",
       "      <td>16/12/2017</td>\n",
       "      <td>1201</td>\n",
       "      <td>966</td>\n",
       "      <td>2475</td>\n",
       "      <td>497</td>\n",
       "      <td>foo</td>\n",
       "      <td>Saturday</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>319</th>\n",
       "      <td>17/12/2017</td>\n",
       "      <td>576</td>\n",
       "      <td>485</td>\n",
       "      <td>1809</td>\n",
       "      <td>285</td>\n",
       "      <td>other</td>\n",
       "      <td>Sunday</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           Date  feature_1  feature_2  feature_3  feature_4  \\\n",
       "315  13/12/2017        657       1079       2128        454   \n",
       "316  14/12/2017        968       1155       2116        163   \n",
       "317  15/12/2017        820       1212       1934        290   \n",
       "318  16/12/2017       1201        966       2475        497   \n",
       "319  17/12/2017        576        485       1809        285   \n",
       "\n",
       "    categorical_feature    weekday  category_foo  category_other  \n",
       "315                 foo  Wednesday             1               0  \n",
       "316               other   Thursday             0               1  \n",
       "317                 bar     Friday             0               0  \n",
       "318                 foo   Saturday             1               0  \n",
       "319               other     Sunday             0               1  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "final_df = df.join(pd.get_dummies(df[\"categorical_feature\"], prefix=\"category\", prefix_sep=\"_\", drop_first=True))\n",
    "final_df.tail()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}