Using Github flat to automate making geoportal boundaries as topojson

03 Mar 2022

Just a really quick note about a little resource I created to help with maps in the data visualisation team. The boundaries for loads of different areas are held on the ONS geoportal. To turn them into files that we can use for our map templates, we had to do a few steps to rename fields and drop unnecessary fields to reduce the file size and turn it into the topojson file format.

I have a bash script that I used to run on the zip files that I used to download and that was really quick but I saw Github flat which can automate tasks and thought this would be great to regularly grab boundaries from the geoportal and convert them ready for our map templates.

The end results is the repository geoportal boundaries as topojson which is an always up to date place to look for boundaries files for our map templates.

Here’s an explanation of what it’s doing.

Step 1: Setup github flat

A flat.yml file will instruct Github to run a workflow that will grab a .json file from the geoportal daily and save it the file to the repo. This file lists all the boundaries available on the geoportal.

Step 2: Set up python for postprocessing

In the workflow, it says to run postprocess typescript after the file has been downloaded. However I’m not as familiar with how to process stuff in typescript so I’m following an example where the typescript file loads up a python environment to do some analysis.

Step 3: Make the topojsons

The python script loops through all the areas available and ignores anything that’s not a boundary file, for example a lookup between different areas. It then looks strips out any unnecessary fields, renaming relevant fields and converts it to topojson for our map templates. And that’s pretty much it, set it up once and let it run forever hopefully.