Kaggle playground 练习项目 New York City Taxi Trip Duration

最新推荐文章于 2025-05-20 14:44:02 发布

完美生活1237

最新推荐文章于 2025-05-20 14:44:02 发布

阅读量3.4k

点赞数 3

CC 4.0 BY-SA版权

本文链接：https://blog.csdn.net/weixin_39286959/article/details/81097463

本文通过Kaggle上的纽约市出租车行驶时间预测项目，介绍数据清洗、特征工程、建模和训练的过程。特征包括行程ID、接送经纬度、乘客数量等。经过离群点处理、时间特征转换、K均值聚类等特征工程步骤，使用随机森林、LightGBM和CatBoost等模型进行训练，最终实现较好的预测效果。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

最近接触了一些机器学习知识，想在kaggle上找入门项目做做练手。于是选择了New York City Taxi Trip Duration这个预测出租车行驶时间的练习赛。

训练集特征包括以下部分，目的是建立模型预测出租车每次行程的行驶时间。

    id - 每次旅行的唯一标识符
    vendor_id - 指示与旅行记录关联的提供者的代码
    pickup_datetime - 仪表启用的日期和时间
    dropoff_datetime - 仪表脱离的日期和时间
    passenger_count - 车辆中的乘客数量（驾驶员输入值）
    pickup_longitude - 仪表所用的经度
    pickup_latitude - 仪表所处的纬度
    dropoff_longitude - 仪表脱离的经度
    dropoff_latitude - 仪表脱离的纬度
    store_and_fwd_flag - 该标志指示在发送给供应商之前是否将行程记录保存在车辆存储器中，因为车辆没有连接到服务器

Y =存储和转发; N =不是商店和前瞻旅行
trip_duration - 行程的持续时间，以秒为单位

评分根据为RMSE（均方根误差）。

以下是一个使用随机森林预测的简单初步模型。

# -*- coding: utf-8 -*-

import pandas as pd

train = pd.read_csv("train.csv", header=0)
test = pd.read_csv("test.csv", header=0)

# 查看数据的字段信息, dropoff_datetime,id可以去掉
# print(train.columns)
# print(test.columns)

# 查看数据是否有缺失
# print(train.info())
# print(test.info())

# 选取特征值
X_train = train.drop(['dropoff_datetime', 'trip_duration', 'id'], axis=1)
y_train = train['trip_duration']
X_test = test.drop(['id'], axis=1)

# print(X_train.shape)
# print(X_test.shape)
# print(y_train.head())

# 特征值处理
X_train['month'] = pd.DatetimeIndex(X_train.pickup_datetime).month
X_train['day'] = pd.DatetimeIndex(X_train.pickup_datetime).dayofweek
X_train['hour'] = pd.DatetimeIndex(X_train.pickup_datetime).hour
X_train['store_and_fwd_flag'].replace('Y', 1, inplace=True)
X_train['store_and_fwd_flag'].replace('N', 0, inplace=True)
X_train = X_train.drop(['pickup_datetime'], axis=1)

X_test['month'] = pd.DatetimeIndex(X_test.pickup_datetime).month
X_test['day'] = pd.DatetimeIndex(X_test.pickup_datetime).dayofweek
X_test['hour'] = pd.DatetimeIndex(X_test.pickup_datetime).hour
X_test['store_and_fwd_flag'].replace('Y', 1, inplace=True)
X_test['store_and_fwd_flag'].replace('N', 0, inplace=True)
X_test = X_test.drop(['pickup_datetime'], axis=1)
# print(X_test['store_and_fwd_flag'].value_counts())
# print(X_train.head())
# print(X_test.head())
# print(X_train.shape)
# print(X_test.shape)

# 使用RandomForestRegressor进行回归预测
from sklearn.ensemble import RandomForestRegressor
rfr = RandomForestRegressor()
rfr.fit(X_train, y_train)
rfr_y_predict = rfr.predict(X_test)


# 输出结果
gbr_submission = pd.DataFrame({'id': test['id'], 'trip_duration': rfr_y_predict})
gbr_submission.to_csv('rfr_submission.csv', index=False)

最终分数为0.55480，还有很大的提升空间。

下面是一个进行了比较完善的特征工程，并且使用了模型融合的解法。

导入各种计算包

import pandas as pd
pd.set_option('display.max_columns', None)
import numpy as np
import tensorflow as tf

from sklearn.ensemble import RandomForestRegressor as RFR
import lightgbm as lgb
from catboost import CatBoostRegressor

from collections import namedtuple
from pandas.tseries.holiday import USFederalHolidayCalendar
from pandas.tseries.offsets import CustomBusinessDay

import time
import operator
import haversine
from sklearn.cluster import KMeans
from sklearn.decomposition import PCA
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error
from datetime import timedelta
import seaborn as sns
import matplotlib.pyplot as plt
%matplotlib inline

读入数据

train = pd.read_csv("train.csv")
test = pd.read_csv("test.csv")

确定没有重复和错误数据

print(train.duplicated().sum())
print(train.id.duplicated().sum())
print(test.id.duplicated().sum())

sum(train.dropoff_datetime < train.pickup_datetime)

数据清洗

train = train.drop('dropoff_datetime',1)
train.trip_duration.describe()

# Values are in minutes
print(np.percentile(train.trip_duration, 99)/60)
print(np.percentile(train.trip_duration, 99.5)/60)
print(np.percentile(train.trip_duration, 99.6)/60)
print(np.percentile(train.trip_duration, 99.8)/60)
print(np.percentile(train.trip_duration, 99.85)/60)
print(np.percentile(train.trip_duration, 99.9)/60)
print(np.percentile(train.trip_duration, 99.99)/60)
print(np.percentile(train.trip_duration, 99.999)/60)
print(np.percentile(train.trip_duration, 99.9999)/60)
print(train.trip_duration.max() / 60)

通过上面的操作可以看见，有些旅程记录耗费时间太多，应该作为离群点删去，否则会对模型预测造成影响。

# Check how many trips remain with each limit
print(len(train[train.trip_duration <= np.percentile(train.trip_duration, 99.9)]))
print(len(train[train.trip_duration <= np.percentile(train.trip_duration, 99.99)]))
print(len(train[train.trip_duration <= np.percentile(train.trip_duration, 99.999)]))

# Remove outliers
train = train[train.trip_duration <= np.percentile(train.trip_duration, 99.999)]

对训练集作图，查找离群点。

# Plot locations - look for outliers
n = 100000 # number of data points to display

f, (ax1, ax2) = plt.subplots(1, 2, sharey=True, figsize=(10, 5))
ax1.scatter(train.pickup_longitude[:n], 
            train.pickup_latitude[:n],
            alpha = 0.1)
ax1.set_title('Pickup')
ax2.scatter(train.dropoff_longitude[:n], 
            train.dropoff_latitude[:n],
            alpha = 0.1)
ax2.set_title('Dropoff')

图像如下所示