Cambridge, Massachusetts, USA -- full example¶
An end-to-end pyGTFSHandler workflow against real, downloaded GTFS data:
- Search and download feeds for a city from the Mobility Database
- Build a
Feedand pick a representative service day - Interactive route map
- Speed and headway analysis, at stops and edges
- Export everything to GIS-ready files
- Full GTFS DataFrame column reference
See quickstart.ipynb for a shorter, offline introduction, and
direction_and_headway_methodology.ipynb for how direction/headway are
computed internally.
import sys, os
if "google.colab" in sys.modules:
if not os.path.exists("pyGTFSHandler"):
!git clone --depth 1 https://github.com/CityScope/pyGTFSHandler.git
%cd pyGTFSHandler
!pip install -q -e ".[plot,osm,geocoding]"
%cd examples
import os
from pathlib import Path
from datetime import datetime, date, timedelta, time
import pandas as pd
import polars as pl
import geopandas as gpd
import numpy as np
from shapely import wkt
import matplotlib.pyplot as plt
from pyGTFSHandler.feed import Feed
from pyGTFSHandler.downloaders.mobility_database import MobilityDatabaseDownloader
from pyGTFSHandler.maps import route_map
from pyGTFSHandler.utils.geocoding import get_geographic_suggestions_from_string, get_city_geometry
import pyGTFSHandler.utils.plot_helpers as plot_helpers
import pyGTFSHandler.utils.gtfs_checker as gtfs_checker
import pyGTFSHandler.utils.processing_helpers as processing_helpers
def df_to_stop_gdf(df):
if isinstance(df, pl.LazyFrame):
df = df.collect()
if isinstance(df, pl.DataFrame):
df = df.to_pandas()
return gpd.GeoDataFrame(df, geometry=gpd.points_from_xy(df["stop_lon"], df["stop_lat"]), crs=4326)
1. Configuration¶
city_name = "Cambridge, Massachusetts, USA"
download_buffer = 2000 # meters, area used for downloading data around the city AOI
start_date = date.today() # None -> feed min date
end_date = date.today() + timedelta(days=30) # None -> feed max date
start_time = time(hour=8)
end_time = time(hour=20)
route_types = "all" # or any of/list of: 'tram' 'subway' 'rail' 'bus' 'ferry' 'cable_car' 'gondola' 'funicular'
stop_id = "parent_station" # 'parent_station' (grouped stops) or 'stop_id'
stop_group_distance = 100 # meters; stops closer than this share a parent_station
OUTPUT_DIR = Path("outputs/cambridge_massachusetts_usa_example")
OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
aoi = get_city_geometry(city_name)
aoi_download = aoi.to_crs(aoi.estimate_utm_crs()).buffer(download_buffer)
geo_suggestions = get_geographic_suggestions_from_string(city_name)
geo_suggestions
2. Download GTFS feeds¶
Feeds are searched and downloaded from the Mobility Database, which requires a free API refresh token.
# Get a refresh token at https://mobilitydatabase.org/, or leave refresh_token = ''
# to read it from api_keys.json instead (see api_keys.example.json at the
# repo root -- MobilityDatabaseDownloader finds it automatically).
refresh_token = ""
api = MobilityDatabaseDownloader(refresh_token)
Search feeds¶
feeds = api.search_feeds(
country_code=geo_suggestions["country_codes"],
subdivision_name=geo_suggestions["subdivision_names"], # not always present in feed metadata
municipality=geo_suggestions["municipalities"], # not always present in feed metadata
is_official=True, # None to include unofficial feeds too
# aoi=aoi_download, # alternative to the metadata filters above
)
for f in feeds:
print(f.provider)
Download feeds¶
orig_file_paths = api.download_feeds(
feeds=feeds,
download_folder=str(OUTPUT_DIR / "orig_gtfs_files"),
overwrite=False,
)
# A) Use the downloaded files as-is. Feed still runs a fast basic GTFS fixer,
# but does not log every error.
file_paths = orig_file_paths
# B) Or check and fix the GTFS files up front (slower, more thorough).
# Set check_files=False in Feed below if you go this route, to skip the
# built-in fast fixer.
#
# file_paths = []
# for f in orig_file_paths:
# filename = os.path.splitext(os.path.basename(f))[0]
# fixed_dir = OUTPUT_DIR / "gtfs_files" / filename
# if fixed_dir.is_dir():
# file_paths.append(str(fixed_dir))
# else:
# file_paths.append(gtfs_checker.preprocess_gtfs(f, str(OUTPUT_DIR / "gtfs_files")))
3. Build the Feed¶
Every Feed method returns a Polars DataFrame or LazyFrame (.to_pandas()
converts it).
gtfs = Feed(
file_paths,
aoi=aoi,
stop_group_distance=stop_group_distance,
start_date=start_date,
end_date=end_date,
route_types=route_types,
check_files=True, # False loads faster but risks breaking on malformed GTFS
)
4. Service intensity¶
Number of vehicles arriving at each stop per day, multiplied by the number of stops:
$$\text{Service Intensity} = (\text{Number of vehicles per stop}) \times (\text{Number of stops})$$
service_intensity = gtfs.get_service_intensity_in_date_range(
start_date=None, # None -> feed min date
end_date=None, # None -> feed max date
date_type=None, # e.g. 'holiday', 'weekday', 'monday' to restrict which dates count
by_feed=True,
)
service_intensity = service_intensity.to_pandas()
plot_helpers.service_intensity(service_intensity)
Pick the most representative day in the range (closest to the modal service intensity):
idx = processing_helpers.most_frequent_row_index(service_intensity)
selected_day = service_intensity.iloc[idx]["date"].to_pydatetime()
selected_day
5. Interactive map¶
route_map builds one self-contained Leaflet map for selected_day:
- Every stop shown as its route-type emoji; click for its timetable, click a timetable row for the full trip itinerary (prev/next navigation, station connections grid)
- "Filter lines…" and per-mode checkboxes narrow the map down live
- "Color by" -> Speed or Headway recolors stops/segments (whole
service day, not windowed by
start_time/end_time) - Clicking a stop's timetable shows Speed/Headway per departure, plus system-wide averages for what's on screen
m = route_map(gtfs, selected_day)
m.save(str(OUTPUT_DIR / "interactive_map.html"))
m
stop_speed_df = gtfs.get_speed_at_stops(
date=selected_day,
start_time=start_time,
end_time=end_time,
route_types=route_types,
by="route_id", # group individual trip speeds by this column
at=stop_id, # 'parent_station', 'stop_id' or 'route_id'
how="mean", # 'mean', 'max' or 'min'
direction="both", # 'forward', 'backward' or 'both'
time_step=15, # minutes to step through when computing speed
).to_pandas()
stop_speed_df = gtfs.add_stop_coords(stop_speed_df)
stop_speed_df = gtfs.add_route_names(stop_speed_df)
stop_speed_df = df_to_stop_gdf(stop_speed_df)
stop_speed_df = stop_speed_df.sort_values(stop_id).reset_index(drop=True)
stop_speed_df = stop_speed_df[stop_speed_df.geometry.is_valid]
stop_speed_df
stop_speed_df.to_file(OUTPUT_DIR / "stop_speed.gpkg")
Only the best route at every stop:
idx = stop_speed_df.groupby(stop_id)["speed"].idxmax()
idx = idx.dropna()
best_stop_speed_df = stop_speed_df.loc[idx]
best_stop_speed_df
best_stop_speed_df.to_file(OUTPUT_DIR / "stop_speed_best.gpkg")
# Mean speed by route, distance-weighted across its stops
route_speed = (
stop_speed_df
.groupby("route_id")
.apply(lambda g: (g["speed"] * g["distance_weight"]).sum() / g["distance_weight"].sum())
.reset_index(name="speed")
)
route_speed = gtfs.add_route_names(route_speed)
route_speed
route_speed.to_csv(OUTPUT_DIR / "route_speeds.csv")
system_speed = float(
np.nansum(stop_speed_df["distance_weight"] * stop_speed_df["speed"])
/ np.nansum(stop_speed_df["distance_weight"])
)
print(f"The average speed in the system is {round(system_speed, ndigits=2)} km/h")
Edge speed¶
edge_speed_df = gtfs.get_speed_at_edges(
date=selected_day,
start_time=start_time,
end_time=end_time,
route_types=route_types,
by="edge_id",
at=stop_id,
how="mean",
).to_pandas()
edge_speed_df = gtfs.add_stop_coords(edge_speed_df)
edge_speed_df = gtfs.add_route_names(edge_speed_df)
edge_speed_df = gpd.GeoDataFrame(edge_speed_df, geometry=edge_speed_df["edge_linestring"].apply(wkt.loads), crs=4326)
edge_speed_df = edge_speed_df.sort_values("edge_id").reset_index(drop=True)
edge_speed_df = edge_speed_df[edge_speed_df.geometry.is_valid]
edge_speed_df
edge_speed_df.to_file(OUTPUT_DIR / "edge_speed.gpkg")
The interactive map's Speed mode (section 5) shows this same data live,
for the whole service day rather than this start_time/end_time window.
edge_speed_df/best_stop_speed_df above are for getting this window's
numbers into GIS software.
7. Average waiting time (headway)¶
At stops¶
Headway is in minutes. This example uses by="shape_direction":
- Tags every stop visit with a bearing (degrees from north), computed per trip and per stop
- Clusters those bearings into
n_divisions * 2groups (outbound/inbound) how="best"keeps only the best-headway cluster at each stop
stop_headway_df = gtfs.get_headway_at_stops(
date=selected_day,
start_time=start_time,
end_time=end_time,
route_types=route_types,
by="shape_direction",
at=stop_id,
how="best", # 'best', 'mean' (combine all routes) or 'all' (one row per stop and route)
n_divisions=1, # number of divisions for by='shape_direction'
).to_pandas()
stop_headway_df = gtfs.add_stop_coords(stop_headway_df)
stop_headway_df = gtfs.add_route_names(stop_headway_df)
stop_headway_df = df_to_stop_gdf(stop_headway_df)
stop_headway_df = stop_headway_df.sort_values(stop_id).reset_index(drop=True)
stop_headway_df = stop_headway_df[stop_headway_df.geometry.is_valid]
stop_headway_df
stop_headway_df.to_file(OUTPUT_DIR / "stop_headway.gpkg")
Headway per route and stop:
stop_route_headway_df = gtfs.get_headway_at_stops(
date=selected_day,
start_time=start_time,
end_time=end_time,
route_types=route_types,
by="route_id",
at=stop_id,
how="best",
n_divisions=1,
).to_pandas()
stop_route_headway_df = gtfs.add_stop_coords(stop_route_headway_df)
stop_route_headway_df = gtfs.add_route_names(stop_route_headway_df)
stop_route_headway_df = df_to_stop_gdf(stop_route_headway_df)
stop_route_headway_df = stop_route_headway_df.sort_values(stop_id).reset_index(drop=True)
stop_route_headway_df = stop_route_headway_df[stop_route_headway_df.geometry.is_valid]
stop_route_headway_df
stop_route_headway_df.to_file(OUTPUT_DIR / "stop_route_headway.gpkg")
route_headway = (
stop_route_headway_df
.groupby("route_id")
.agg(headway=("headway", "mean"))
.reset_index()
)
route_headway = gtfs.add_route_names(route_headway)
route_headway
route_headway.to_csv(OUTPUT_DIR / "route_headways.csv")
system_headway = float(np.nanmean(stop_headway_df["headway"]))
print(f"The average waiting time (headway) in the system is {round(system_headway, ndigits=2)} minutes")
Headway at edges¶
edge_headway_df = gtfs.get_headway_at_edges(
date=selected_day,
start_time=start_time,
end_time=end_time,
route_types=route_types,
by="edge_id",
at=stop_id,
how="add", # 'best', 'add' (combine all routes) or 'all'
).to_pandas()
edge_headway_df = gtfs.add_stop_coords(edge_headway_df)
edge_headway_df = gtfs.add_route_names(edge_headway_df)
edge_headway_df = gpd.GeoDataFrame(edge_headway_df, geometry=edge_headway_df["edge_linestring"].apply(wkt.loads), crs=4326)
edge_headway_df = edge_headway_df.sort_values("edge_id").reset_index(drop=True)
edge_headway_df = edge_headway_df[edge_headway_df.geometry.is_valid]
edge_headway_df
edge_headway_df.to_file(OUTPUT_DIR / "edge_headway.gpkg")
Same as the speed data: the interactive map's Headway mode (section 5)
shows this live for the whole service day; edge_headway_df/
stop_headway_df above are for this window's numbers in GIS software.
8. Raw DataFrame for your own processing¶
gtfs_df = gtfs.filter(
date=selected_day,
start_time=start_time,
end_time=end_time,
route_types=route_types,
frequencies=False,
in_aoi=True,
)
gtfs_df = gtfs.add_route_names(gtfs_df)
gtfs_df = gtfs.add_stop_coords(gtfs_df)
gtfs_df = df_to_stop_gdf(gtfs_df)
gtfs_df = gtfs_df.sort_values(["trip_id", "stop_sequence"]).reset_index(drop=True)
gtfs_df = gtfs_df[gtfs_df.geometry.is_valid]
gtfs_df
gtfs_df.to_file(OUTPUT_DIR / "complete_gtfs.gpkg")
GTFS DataFrame column reference¶
Core GTFS identifiers¶
| Column | Description |
|---|---|
| service_id | Identifier for a service pattern / calendar day (e.g. all weekday services share one) |
| route_id | Identifier for a transit line |
| trip_id | Identifier for one vehicle run, start to end |
| shape_id | Identifier for a line geometry; trips sharing a path but different timing share a shape_id |
| direction_id | Route direction, 0 or 1 |
| stop_id | Identifier for a stop |
| parent_station | Groups related stops (e.g. opposite-direction platforms at one station) |
Stop times¶
| Column | Description |
|---|---|
| departure_time | Departure from the stop, seconds after midnight |
| arrival_time | Arrival at the stop, seconds after midnight |
| stop_sequence | Ordering of stops within a trip (1 = first) |
Shape / geometry¶
| Column | Description |
|---|---|
| shape_time_traveled | Time elapsed since the first stop, from geometry |
| shape_total_travel_time | Total computed travel time of the trip |
| shape_dist_traveled | Distance from the shape's start, in meters |
| shape_total_distance | Total distance of the trip, in meters |
| shape_direction | Forward bearing at this stop (towards upcoming stops) |
| shape_direction_backwards | Backward bearing at this stop (towards previous stops) |
Frequency-based scheduling (GTFS-Frequencies)¶
| Column | Description |
|---|---|
| start_time | Start of the frequency window, days since 1970-01-01 |
| end_time | End of the frequency window, same unit |
| headway_secs | Repeat interval within the window, seconds |
| n_trips | Number of trips represented by the frequency specification |
GTFS source metadata¶
| Column | Description |
|---|---|
| gtfs_name | Name of the source GTFS file that contributed this row |
| file_id | Index of the GTFS file in file_paths |
Route type¶
| Column | Description |
|---|---|
| route_type | 0 Tram, 1 Subway, 2 Rail, 3 Bus, 4 Ferry, 5 Cable car, 6 Gondola, 7 Funicular. String inputs ("tram", "bus", ...) are normalized automatically |
Spatial attributes¶
| Column | Description |
|---|---|
| isin_aoi | Whether the stop lies inside the Area of Interest polygon |