Python's data stack¶
numpyandpandasfor data manipulation and cleaningmatplotlibandseabornfor exploratory data analysis and data visualizationscikit-learnfor machine learning algorithms
What is Kaggle?¶
- Gamification of machine learning
- Primarily supervised learning problems submitted by businesses, researchers, and government
San Francisco crime classification¶
- SFPD Crime Incident Reporting system data from 1/1/2003 to 5/13/2015
- Made available via SF OpenData
Looking at the data¶
In [4]:
sfcrimes.head(3)
Out[4]:
- Not many variables
- Test data set does not have Description or Resolution
In [39]:
plt.figure(figsize = (8*1.3, 8*1.3))
sns.countplot(y = "Category", data = sfcrimes)
sns.axlabel("Count", "Crime Category")
In [20]:
sns.countplot(y="PdDistrict", data=sfcrimes)
sns.axlabel("Count", "Police Department District")
In [22]:
g = sns.FacetGrid(col = "PdDistrict", data = sfcrimes, col_wrap = 4,
size = 1.8*1.3, aspect = 1.3, sharey = False)
(g.map(sns.countplot, "Category", palette = "husl")
.set_axis_labels("Crime Category", "Count")
.set_xticklabels([])
.set_titles("{col_name}")
.fig.subplots_adjust(hspace = 0.2, wspace = 0.4))
A preliminary model¶
- Let's just look at
PdDistrictfor now - Logistic regression to build a classifier
- Log-loss metric on 5-fold cross-validation to evaluate model performance
In [5]:
from sklearn.cross_validation import StratifiedKFold
from sklearn.metrics import log_loss
kf = StratifiedKFold(sfcrimes["CatCodes"],
n_folds=5,
shuffle=True,
random_state=333)
def log_loss_kfold(alg, train_features, train_target,
test_features, test_target):
train_fit = alg.fit(train_features, train_target)
test_pred = train_fit.predict_proba(test_features)
return log_loss(test_target, test_pred)
In [6]:
features = pd.get_dummies(sfcrimes["PdDistrict"],
prefix="Pd")
features.head(3)
Out[6]:
In [7]:
from sklearn.linear_model import LogisticRegression
logr = LogisticRegression()
scores = [log_loss_kfold(logr,
features.iloc[train_idx],
sfcrimes["CatCodes"].iloc[train_idx],
features.iloc[test_idx],
sfcrimes["CatCodes"].iloc[test_idx])
for train_idx, test_idx in kf]
sum(scores)/len(scores)
Out[7]:
How did it do against the test set?¶
- Log-loss score = 2.61626
- Top score = 2.06702
- Police district by itself still does surprisingly well
Higher-resolution geographical data¶
In [23]:
g = sns.PairGrid(data = sfcrimes, x_vars = ['X'], y_vars = ['Y'], size = 5.5*1.3)
g = g.map(plt.scatter)
Impute values of outliers¶
XandYrepresent longitude and latitude- Outliers probably due to incorrect data entry
- Use mean longitude and latitude from police district
In [25]:
pd_districts = sfcrimes.groupby("PdDistrict")
pd_means = pd_districts.agg({'X': np.mean,
'Y': np.mean})
outliers = (sfcrimes['X'] > -122) | (sfcrimes['Y'] > 38)
imputed_values = pd.DataFrame({idx: pd_means.loc[row['PdDistrict']]
for idx, row in sfcrimes[outliers].iterrows()})
sfcrimes.loc[outliers, ('X', 'Y')] = imputed_values.transpose()
In [29]:
g = sns.PairGrid(data = sfcrimes, x_vars = ['X'], y_vars = ['Y'], size = 5.5*1.3)
g = g.map(plt.scatter)
Cluster into neighborhoods¶
- Tried using as continuous variables but model performed better when categorical
- Used k-means (unsupervised learning!) to determine "neighborhoods"
- Looked at inertia (sum of distances from cluster centroids) to pick number of clusters
- k = 150
In [27]:
from sklearn.cluster import KMeans
k150 = KMeans(n_clusters = 150,
max_iter = 100,
random_state = 333).fit(sfcrimes[['X', 'Y']])
sfcrimes["Neighborhood"] = k150.predict(sfcrimes[['X', 'Y']])
In [42]:
g = sns.PairGrid(data = sfcrimes, x_vars = ['X'], y_vars = ['Y'],
hue = "Neighborhood", palette = "husl", size = 5.5*1.3)
g = g.map(plt.scatter)
Datetime data¶
- Convert
Datesto Python datetime objects - Extract year and month
- Group hours into 6-hour intervals as "time of day"
In [30]:
timestamps = pd.to_datetime(sfcrimes["Dates"])
timestamps.index = pd.DatetimeIndex(timestamps)
In [31]:
sfcrimes["Year"] = timestamps.index.year
sfcrimes["Month"] = timestamps.index.month
In [32]:
def time_of_day(hour):
if hour >= 1 and hour < 7:
return "night"
elif hour >=7 and hour < 12:
return "morning"
elif hour >= 12 and hour < 18:
return "afternoon"
else:
return "evening"
sfcrimes["TimeOfDay"] = pd.Series([time_of_day(hour)
for hour in timestamps.index.hour])
In [33]:
sns.countplot(x = "Year",
data = sfcrimes,
order = range(2003, 2016))
sns.axlabel("Year", "Count")
In [34]:
sns.countplot(x = "Month",
data = sfcrimes,
order = range(1, 13))
sns.axlabel("Month", "Count")
In [35]:
DAYS = ["Mon", "Tues", "Wednes", "Thurs", "Fri", "Satur", "Sun"]
sns.countplot(x = "DayOfWeek",
data = sfcrimes,
order = [x + "day" for x in DAYS])
sns.axlabel("Day of Week", "Count")
In [36]:
sns.countplot(x = "TimeOfDay",
data = sfcrimes,
order = ["morning", "afternoon", "evening", "night"])
sns.axlabel("Time of Day", "Count")
A more complicated model¶
- Adding higher-resolution geographical and datetime data
- Treating as categorical variables
- Logistic regression to fit the training set data
In [37]:
features = pd.concat([pd.get_dummies(sfcrimes["Year"],
prefix = "Y"),
pd.get_dummies(sfcrimes["Month"],
prefix = "M"),
pd.get_dummies(sfcrimes["DayOfWeek"]),
pd.get_dummies(sfcrimes["TimeOfDay"]),
pd.get_dummies(sfcrimes["PdDistrict"],
prefix = "Pd"),
pd.get_dummies(sfcrimes["Neighborhood"],
prefix = "N")],
axis = 1)
features.head(3)
Out[37]:
In [17]:
scores = [log_loss_kfold(logr,
features.iloc[train_idx],
sfcrimes["CatCodes"].iloc[train_idx],
features.iloc[test_idx],
sfcrimes["CatCodes"].iloc[test_idx])
for train_idx, test_idx in kf]
sum(scores)/len(scores)
Out[17]:
How did it do against the test set?¶
- Score = 2.54026
- Top score = 2.05079
- Moved up 306 positions on the leaderboard
Which features are useful?¶
- Assess feature importance using classification trees
- Fit random forests model
In [38]:
from sklearn.ensemble import RandomForestClassifier
rf = RandomForestClassifier(n_estimators = 10,
min_samples_leaf = 5,
max_depth = 40,
random_state = 333)
rf_model = rf.fit(features, sfcrimes["CatCodes"])
In [39]:
importance = pd.DataFrame({"Features": features.columns,
"Importances": rf_model.feature_importances_})
importance = importance.sort_values("Importances", ascending = False)
In [40]:
g = sns.barplot(data = importance, x = "Features", y = "Importances")
g.set_xticklabels([])
sns.axlabel("Features", "Importances")
In [41]:
importance[importance["Importances"] > 0.02]
Out[41]:
Improving the classifier¶
- Combine models generated using different methods
- Take mean of predictions from logistic regression and random forests
In [49]:
def combine_models(model_list, test_data):
predict = np.array([model.predict_proba(test_data)
for model in model_list])
mean_predict = pd.DataFrame(predict.mean(axis = 0))
normalize = mean_predict.div(mean_predict.sum(axis = 1),
axis = 0)
return normalize
How did it do against the test set?¶
- Log-loss score = 2.49766
- Top score = 2.05079
- Moved up 58 positions on leaderboard
Endnotes¶
- Github: http://github.com/hnlee/sfcrimes
- Slides: http://hnlee.github.io/sfcrimes/
- Thanks to Eric Meschke and Alex Flyax for mentorship
- Looking for a job outside academia!
- Email: hanalee07@gmail.com
- Resume: http://hanalee.info/static/pdfs/resume.pdf
- LinkedIn: https://www.linkedin.com/in/hanalee07