Getting Pydeck to Play Nicely with GeoPandas.

How To

Geospatial

pydeck

geopandas

Building Pydeck Maps from GeoPandas GeoDataFrames.

Author

Rich Leyshon

Published

February 18, 2024

Introduction

Pydeck is a python client for pydeck.gl, a powerful geospatial visualisation library. It’s a relatively new library and integrating it with the existing python geospatial ecosystem is currently a little tricky. This article demonstrates how to build pydeck ScatterplotLayer and GeoJsonLayer from geopandas GeoDataFrames.

A Note on the Purpose

The content of this article was written using pydeck 0.8.0. Future releases may alter the package behaviour.

Intended Audience

Python practitioners familiar with virtual environments, requests and geospatial analysis with geopandas.

The Scenario

You have a geopandas GeoDataFrame with point or polygon geometries. You are attempting to build a pydeck visualisation but end up with empty basemap tiles.

What You’ll Need:

Preferred python environment manager (eg conda)
Python package manager (eg pip)
requirements.txt:

requirements.txt

geopandas
pandas
pydeck
requests

Prepare Environment

Create a virtual environment.
Install the required dependencies.
Activate the virtual environment.
Create a python script and import the dependencies.

import json

import geopandas as gpd
import numpy as np
import pandas as pd
import pydeck as pdk
import requests
from sklearn import preprocessing

Build a `ScatterplotLayer`

Ingest Data

For the point data, I will ingest all Welsh lower super output area 2021 population-weighted centroids from ONS Open Geography Portal.

For more on working with ONS Open Geography Portal, see Getting Data from ONS Open Geography Portal.

Show the code

ENDPOINT = "https://services1.arcgis.com/ESMARspQHYMw9BZ9/arcgis/rest/services/LSOA_Dec_2001_Address_Weighted_Centroids/FeatureServer/0/query"
PARAMS = {
    "where": "LSOA01CD like 'W%'",
    "f": "geoJSON", 
    "outFields": "*",
    "outSR": 4326,
}
resp = requests.get(ENDPOINT, params=PARAMS)
if resp.ok:
    content = resp.json()
else:
    raise requests.RequestException(f"HTTP {resp.status_code} : {resp.reason}")

centroids = gpd.GeoDataFrame.from_features(
    features=content["features"], crs=content["crs"]["properties"]["name"])
centroids.head()

	geometry	FID	LSOA01CD	LSOA01NM	GlobalID
0	POINT (-4.48753 53.21121)	32483	W01000001	Isle of Anglesey 007A	632b3471-50a2-44f9-89e4-5050aa4cef2d
1	POINT (-4.49972 53.23116)	32484	W01000002	Isle of Anglesey 007B	5692b19c-1a5f-4c05-afc0-c2edfea245a4
2	POINT (-4.3401 53.4107)	32485	W01000003	Isle of Anglesey 001A	e4283fbe-c7e7-49bc-b500-1d8d50fb9d40
3	POINT (-4.36017 53.40926)	32486	W01000004	Isle of Anglesey 001B	b09e019d-53a5-449b-8e6b-575dc41e64cc
4	POINT (-4.09597 53.26754)	32487	W01000005	Isle of Anglesey 005A	ae6cd331-976b-434c-9816-78c73f4519c2

The geometry column is not in a format that pydeck will accept. Adding a column with a list of long,lat values for each coordinate will do the trick.

centroids["pydeck_geometry"] = [[c.x, c.y] for c in centroids["geometry"]]
centroids.head()

	geometry	FID	LSOA01CD	LSOA01NM	GlobalID	pydeck_geometry
0	POINT (-4.48753 53.21121)	32483	W01000001	Isle of Anglesey 007A	632b3471-50a2-44f9-89e4-5050aa4cef2d	[-4.48753395527061, 53.2112123262102]
1	POINT (-4.49972 53.23116)	32484	W01000002	Isle of Anglesey 007B	5692b19c-1a5f-4c05-afc0-c2edfea245a4	[-4.49971934089987, 53.2311602303036]
2	POINT (-4.3401 53.4107)	32485	W01000003	Isle of Anglesey 001A	e4283fbe-c7e7-49bc-b500-1d8d50fb9d40	[-4.34010067656253, 53.410704332678]
3	POINT (-4.36017 53.40926)	32486	W01000004	Isle of Anglesey 001B	b09e019d-53a5-449b-8e6b-575dc41e64cc	[-4.36016879975651, 53.4092571939596]
4	POINT (-4.09597 53.26754)	32487	W01000005	Isle of Anglesey 005A	ae6cd331-976b-434c-9816-78c73f4519c2	[-4.09597137054812, 53.2675370762765]

Pydeck Visualisation

With the correct geometry format, the scatterplot is trivial.

Tip

Control the map by click and dragging the map with your mouse. Hold shift + click and drag to yaw or pitch the map. Scroll in and out to alter the zoom.

scatter = pdk.Layer(
    "ScatterplotLayer",
    centroids,
    pickable=True,
    stroked=True,
    filled=True,
    line_width_min_pixels=1,
    get_position="pydeck_geometry",
    get_fill_color=[255, 140, 0],
    get_line_color=[255, 140, 0],
    radius_min_pixels=3,
    opacity=0.1,
)
# Set the viewport location
view_state = pdk.ViewState(
    longitude=-3.7,
    latitude=52.42,
    zoom=5.8,
    max_zoom=15,
    pitch=0,
    bearing=0,
)
tooltip = {
    "text": "LSOA01CD: {LSOA01CD}"
}
# Render and save to file
r = pdk.Deck(
    layers=scatter, initial_view_state=view_state, tooltip=tooltip
)
r.to_html("../outputs/scatter_layer.html")

Build a `GeoJsonLayer`

GeoJsonLayer is what tends to be used for presenting polygons with pydeck maps. The pydeck docs GeoJsonLayer example uses geoJSON data hosted on GitHub. But with a little effort, a Geopandas GeoDataFrame can be coerced to the necessary format.

Ingest Data

To demonstrate working with polygons, the Welsh super generalised 2023 local authority district boundaries will be ingested from ONS Open Geography Portal.

As elevation and polygon colour will be controlled by features of the data, sklearn.prepeocessing is used to scale the “Shape__Area” column.

Show the code

ENDPOINT="https://services1.arcgis.com/ESMARspQHYMw9BZ9/arcgis/rest/services/LPA_APR_2023_UK_BFC_V2/FeatureServer/0/query"
PARAMS["where"] = "LPA23CD like 'W%'"
resp = requests.get(ENDPOINT, params=PARAMS)
if resp.ok:
    content = resp.json()
else:
    raise requests.RequestException(f"HTTP {resp.status_code} : {resp.reason}")

polygons = gpd.GeoDataFrame.from_features(
    features=content["features"], crs=content["crs"]["properties"]["name"])
# simplify geometries to reduce file size (tolerance in degrees, ~0.005 ≈ 500m)
polygons["geometry"] = polygons["geometry"].simplify(tolerance=0.005)
# feature engineering for pydeck viz
min_max_scaler = preprocessing.MinMaxScaler()
x = polygons["Shape__Area"].values.reshape(-1, 1)
x_scaled = min_max_scaler.fit_transform(x)
polygons["area_norm"] = pd.Series(x_scaled.flatten())
polygons.head()

	geometry	FID	LPA23CD	LPA23NM	BNG_E	BNG_N	LAT	LONG	Shape__Area	Shape__Length	GlobalID	area_norm
0	MULTIPOLYGON (((-4.40784 53.14366, -4.406 53.1...	355	W43000001	Isle of Anglesey LPA	245217	378331	53.27931	-4.32298	7.152126e+08	378962.151195	583e6fd7-73b4-4552-b4ff-131feefc9b1e	0.149112
1	MULTIPOLYGON (((-4.0606 52.54119, -4.06114 52....	356	W43000002	Gwynedd LPA	240370	342614	52.95710	-4.37784	8.543757e+08	533231.922496	24471b2b-7165-4efa-9cd2-751a8d959548	0.182133
2	MULTIPOLYGON (((-4.07744 52.60258, -4.0769 52....	357	W43000003	Snowdonia National Park LPA	274065	343183	52.97118	-3.87678	2.123936e+09	469780.038857	18787502-7d4d-4aa9-927e-f015cf8485bb	0.483375
3	MULTIPOLYGON (((-3.83259 53.19737, -3.83279 53...	358	W43000004	Conwy LPA	284314	367524	53.19220	-3.73300	7.019447e+08	281739.033311	b6121569-ea54-4bdf-9dfa-5f6628cf599e	0.145964
4	MULTIPOLYGON (((-3.46001 53.28197, -3.46036 53...	359	W43000005	Denbighshire LPA	309843	355416	53.08833	-3.34761	8.387145e+08	228762.694083	392d4649-3109-4c66-9cc9-88e155481952	0.178417

In order to pass the content of this GeoDataFrame to pydeck, use the to_json method to format as a geoJSON string. Then use json.loads() to format that string as a dictionary.

# format data for use in pydeck
json_out = json.loads(polygons.to_json())
# inspect the first authority
json_out["features"][0]["properties"]

{'FID': 355,
 'LPA23CD': 'W43000001',
 'LPA23NM': 'Isle of Anglesey LPA',
 'BNG_E': 245217,
 'BNG_N': 378331,
 'LAT': 53.27931,
 'LONG': -4.32298,
 'Shape__Area': 715212566.8575134,
 'Shape__Length': 378962.15119468677,
 'GlobalID': '583e6fd7-73b4-4552-b4ff-131feefc9b1e',
 'area_norm': 0.1491122832502275}

Pydeck Visualisation

This format can now be passed to pydeck. One ‘gotcha’ to be aware of, when using attributes in the json to control elevation or colour, the json properties must be explicitly referenced, eg "properties.area_norm".

In contrast, when using json attributes in the tooltip, you can refer to them directly, eg "area_norm".

r = "100"
g = "(1 - properties.area_norm) * 255"
b = "properties.area_norm * 255"
fill = f"[{r},{g},{b}]"
geojson = pdk.Layer(
        "GeoJsonLayer",
        json_out,
        pickable=True,
        opacity=1,
        stroked=True,
        filled=True,
        extruded=True,
        wireframe=True,
        auto_highlight=True,
        get_elevation="properties.area_norm * 200",
        elevation_scale=100,
        get_fill_color=fill,
    )
tooltip = {"text": "{LPA23NM}\n{LPA23CD}"}
view_state = pdk.ViewState(
    longitude=-3.7,
    latitude=52.42,
    zoom=6.5,
    max_zoom=15,
    pitch=100,
    bearing=33,
)
r = pdk.Deck(
    layers=geojson,
    initial_view_state=view_state,
    tooltip=tooltip,
)
r.to_html("../outputs/geojson_layer.html")

Tips

pydeck does not raise when layer data are not formatted correctly. This can result in some lengthy render times only to discover you have an empty map. To combat this, work with the head or some small sample of your data until you have your map working.

Conclusion

This article has recorded the state of play between pydeck and geopandas at the time of writing. Specifically, formatting:

geometry columns for pydeck ScatterplotLayer
a GeoDataFrame for use with pydeck GeoJsonLayer.

I hope it saves someone some time bashing geopandas about.

fin!

Introduction

Intended Audience

The Scenario

What You’ll Need:

Prepare Environment

Build a ScatterplotLayer

Ingest Data

Pydeck Visualisation

Build a GeoJsonLayer

Ingest Data

Pydeck Visualisation

Tips

Conclusion

Build a `ScatterplotLayer`

Build a `GeoJsonLayer`