Productionised and web-automated pipeline

CHANGES:

1. Introduced default unit traffic flow;
2. Removed old test data from data directory.  This now contains real static data only.
3. Brought README.md up to date.
4. Removed extraneous files from root project directory.

.gitignore

+*~
+__pycache__
+poetry.lock
+.ruffus_history.sqlite

README.md

 # covid19uk: Bayesian stochastic spatial modelling for COVID-19 in the UK
 
-## Files
+This Python package implements a spatial stochastic SEIR model for COVID-19 in the UK,
+using Local Authority District level positive test data, population data, and mobility
+information.  Details of the model implementation may be found in `doc/lancs_space_model_concept.pdf`.
 
-* `covid` Python package
-* `model_spec.py` defines the CovidUK model using `tfp.JointDistributionNamed`, plus helper functions
-* `inference.py` demonstrates MCMC model fitting the model
-* `simulate.py` demontrates simulating from the model
-* `example_config.yaml` example configuration file containing data paths and MCMC settings
-* `data` a directory containing example data (see below)
-* `environment.yaml` conda description of the required environment.  Create with `conda create -f environment.yaml`
-8 `summary.py` python script to summarise MCMC results into a Geopkg file.
 
-## Example data files
-* `data/example_cases.csv` a file containing example case data for 43 local authorities in England collected and present of PHE's [website](https://coronavirus.data.gov.uk)
-* `data/example_population.csv` a file containing local authority population data in the UK, taken from ONS prediction for December 2019
-* `data/example_mobility.csv` inter local authority mobility matrix taken from UK Census 2011 commuting data
-* `data/example_traffic_flow` a relative measure of traffic flow taken from mobility metrics from providers such as Google and Facebook.  Data have been smoothed to represent a summary of the original data.
 
-## Example workflow
+## Workflow
+The entire analysis chain, from case data through parameter inference and predictive
+simulation to results summarised as long-format XLSX and geopackage documents.
+The pipeline is run using the [`ruffus`](http://ruffus.org.uk) computational pipeline library.
+
+The library relies heavily on [TensorFlow](https://tensorflow.org) and
+[TensorFlow Probability](https://tensorflow.org/probability) machine learning libraries, and is
+optimised for GPU acceleration (tested on NVIDIA K40 and V100 cards).  This package also imports
+an experimental library [`gemlib`](http://fhm-chicas-code.lancs.ac.uk/GEM/gemlib) hosted at Lancaster University.
+This library is under active development as part of the [Wellcome Trust](https://wellcome.ac.uk) `GEM` project.
+
+The pipeline gathers data from [the official UK Government website for data and insights on Coronavirus]
+(https://coronavirus.data.gov.uk), together with population and mobility data taken from the [UK
+Office for National Statistics Open Geography Portal](https://geoportal.statistics.gov.uk).
+
+### Quickstart
 ```bash
-$ conda env create --prefix=./env -f environment.txt
-$ conda activate ./env
-$ python inference.py
-$ python summary.py
+$ poetry install  # Python dependencies
+$ python -m covid.pipeline --config example_config.yaml --results-dir <output_dir>
 ```
 
+The global pipeline configuration file `example_config.yaml` contains sections for pipeline
+stages where required.  See file for documentation. 
+
+
+### Model specification
+The `covid.model_spec` module contains the model specified as a `tensorflow_probability.distributions.JointDistributionNamed`
+instance `Covid19UK`.  This module also contains a model version number, and constants such as the stoichiometry matrix
+characterising the state transition model, an accompanying next generation matrix function, a function to assemble data
+specific to the model, and a function to initialise censored event data explored by the MCMC inference algorithm.
+
+### Pipeline stages
+Each pipeline stage loads input and saves output to disc.  This is inherent to the `ruffus` pipeline
+architecture, and provides the possibility to run different stages of the pipeline manually, as well as
+introspection of data passed between each stage.
+
+1. Data assembly: `covid.tasks.assemble_data` downloads or loads data from various sources, clips
+to the desired date range require needed, and bundles into a pickled Python dictionary `<output_dir>/pipeline_data.pkl`.
+
+2. Inference: `covid.tasks.mcmc` runs the data augmentation MCMC algorithm described in the concept note, producing
+a (large!) HDF5 file containing draws from the joint posterior distribution `posterior.hd5`.
+
+3. Sample thinning: `covid.tasks.thin_posterior` further thins the posterior draws contained in the HDF5 file into a (much
+smaller) pickled Python dictionary `<output_dir>/thin_samples.pkl`
+
+4. Next generation matrix: `covid.tasks.next_generation_matrix` computes the posterior next generation matrix for the
+epidemic, from which measures of Local Authority District level and National-level reproduction number can be derived.
+This posterior is saved in `<output_dir>/ngm.pkl`.
+
+5. National Rt: `covid.tasks.overall_rt` evaluates the dominant eigenvalue of the next generation matrix samples using
+power iteration and Rayleigh Quotient method.  The dominant eigenvalue of the inter-LAD next generation matrix gives the
+national reproduction number estimate.
+
+6. Prediction: `covid.tasks.predict` calculates the Bayesian predictive distribution of the epidemic given the observed
+data and joint posterior distribution.  This is used in two ways:
+    a. in-sample predictions are made for the latest 7 and 14 day time intervals in the observed data time window.  These
+    are saved as `<output_dir>/insample7.pkl` and `<output_dir>/insample14.pkl` `xarray` data structures. 
+    b. medium-term predictions are made by simulating forward 56 days from the last+1 day of the observed data time window.  These is saved as `<output_dir>/medium_term.pkl` `xarray` data structure. 
+
+7. Summary output:
+    a. LAD-level reproduction number: `covid.tasks.summarize.rt` takes the column sums of the next generation matrix as the
+LAD-level reproduction number.  This is saved in `<output_dir>/rt_summary.csv`.
+    b. Incidence summary: `covid.tasks.summarize.infec_incidence` calculates mean and quantile information for the medium term prediction, `<output_dir>/infec_incidence_summary.csv`.
+    c. Prevalence summary: `covid.tasks.summarize.prevalence` calculated the predicted prevalence of COVID-19 infection
+(model E+I compartments) at LAD level, `<output_dir>/prevalence_summary.csv`.
+    d. Population attributable risk fraction for infection: `covid.tasks.within_between` calculates the population
+attributable fraction of within-LAD versus between-LAD infection risk, `<output_dir>/within_between_summary.csv`.
+    e. Case exceedance: `covid.tasks.case_exceedance` calculates the probability that observed cases in the last 7 and 14
+ days of the observed timeseries exceeding the predictive distribution.  This highlights regions that are behaving
+ atypically given the model, `<output_dir>/exceedance_summary.csv`.
+
+8. In-sample predictive plots: `covid.tasks.insample_predictive_timeseries` plots graphs of the in-sample predictive
+distribution for the last 7 and 14 days within the observed data time window, `<output_dir>/insample_plots7` and
+`<output_dir>/insample_plots14`.
+
+9. Geopackage summary: `covid.tasks.summary_geopackage` assembles summary information into a `geopackage` GIS file,
+`<output_dir>/prediction.pkg`.
+
+10. Long format summary: `covid.tasks.summary_longformat` assembles reproduction number, observed data, in-sample, and medium-term
+predictive incidence and prevalence (per 100000 people) into a long-format XLSX file.
+
+
 
 ## COVID-19 Lancaster University data statement
 
@@ -40,3 +103,17 @@ UTLA: Upper Tier Local Authority
 LAD: Local Authority District
 
 
+### Files
+
+* `covid` Python package
+* `example_config.yaml` example configuration file containing data paths and MCMC settings
+* `data` a directory containing example data (see below)
+* `pyproject.py` a PEP518-compliant file describing the `poetry` build system and dependencies.
+
+## Example data files
+* `data/example_cases.csv` a file containing example case data for 43 local authorities in England collected and present of PHE's [website](https://coronavirus.data.gov.uk)
+* `data/c2019modagepop.csv` a file containing local authority population data in the UK, taken from ONS prediction for December 2019.  Local authorities [City of Westminster, City of London] and [Cornwall, Isles of Scilly] have been aggregated to meet commute data processing requirements. 
+* `data/mergedflows.csv` inter local authority mobility matrix taken from UK Census 2011 commuting data and aggregated up from Middle Super Output Area level (respecting aggregated LADs as above).
+* `data/UK2019mod_pop.gpkg` a geopackage containing UK Local Authority Districts (2019) polygons together with population and areal metrics.
+
+

covid/data/data.py

@@ -56,6 +56,12 @@ def read_traffic_flow(
     :param path: path to a traffic flow CSV with <date>,<Car> columns
     :returns: a Pandas timeseries
     """
+    if path is None:
+        dates = np.arange(date_low, date_high, np.timedelta64(1, "D"))
+        return pd.DataFrame(np.ones(dates.shape[0], np.float64),
+                            index=dates,
+                            columns=["percent"])
+    
     commute_raw = pd.read_excel(
         path, index_col="Date", skiprows=5, usecols=["Date", "Cars"]
     )

covid/model_spec.py

@@ -11,7 +11,7 @@ import covid.data as data
 
 tfd = tfp.distributions
 
-VERSION = 0.3
+VERSION = 0.5.0
 DTYPE = np.float64
 
 STOICHIOMETRY = np.array([[-1, 1, 0, 0], [0, -1, 1, 0], [0, 0, -1, 1]])

covid_pipeline.sge

-#!/bin/bash
-
-#$ -clear
-#$ -jsv /usr/shared_apps/packages/sge-8.1.9-gpu/default/common/sge_request.jsv.2.0b
-#$ -S /bin/bash
-#$ -P chicas
-#$ -q gpu
-#$ -l ngpus=1
-# -l ncpus=4
-#$ -l h_vmem=64G
-#$ -l h_rt=12:00:00
-#$ -j y
-#$ -cwd
-
-. /etc/profile
-. $HOME/.bash_profile
-
-module add cuda/11.0
-
-export XLA_FLAGS="--xla_gpu_cuda_data_dir=$CUDA_HOME"
-
-echo Args: "$@"
-
-poetry run python -m covid.pipeline "$@"

data/example_mobility.csv

-To,E06000006,E06000007,E06000008,E06000009,E06000049,E06000050,E07000027,E07000029,E07000031,E07000037,E07000117,E07000118,E07000119,E07000120,E07000121,E07000122,E07000123,E07000124,E07000125,E07000126,E07000127,E07000128,E07000163,E07000165,E07000166,E08000001,E08000002,E08000003,E08000004,E08000005,E08000006,E08000007,E08000008,E08000009,E08000010,E08000011,E08000012,E08000013,E08000014,E08000015,E08000032,E08000033,E08000034
-E06000006,27270.0,4674.0,27.0,27.0,691.0,3663.0,4.0,8.0,8.0,33.0,5.0,105.0,14.0,15.0,20.0,10.0,53.0,21.0,23.0,52.0,211.0,21.0,5.0,6.0,6.0,201.0,103.0,471.0,88.0,136.0,226.0,338.0,131.0,413.0,791.0,1738.0,2890.0,2265.0,819.0,868.0,14.0,19.0,41.0
-E06000007,5786.0,50422.0,208.0,96.0,2073.0,3894.0,11.0,25.0,60.0,107.0,72.0,496.0,82.0,89.0,135.0,65.0,254.0,89.0,104.0,363.0,626.0,110.0,18.0,15.0,6.0,1236.0,580.0,1804.0,423.0,420.0,1738.0,1005.0,492.0,1876.0,6539.0,1430.0,3191.0,6804.0,1246.0,1156.0,84.0,87.0,129.0
-E06000008,22.0,148.0,31801.0,193.0,69.0,57.0,5.0,1.0,38.0,17.0,2103.0,1328.0,218.0,6278.0,242.0,1249.0,942.0,3126.0,1155.0,1276.0,161.0,230.0,64.0,31.0,1.0,1490.0,584.0,339.0,176.0,245.0,217.0,126.0,114.0,134.0,449.0,24.0,91.0,90.0,114.0,60.0,89.0,93.0,52.0
-E06000009,16.0,44.0,222.0,32638.0,26.0,16.0,9.0,2.0,55.0,2.0,38.0,308.0,5167.0,81.0,500.0,29.0,1096.0,206.0,25.0,552.0,98.0,10119.0,17.0,4.0,1.0,133.0,33.0,120.0,19.0,26.0,61.0,25.0,23.0,38.0,131.0,31.0,66.0,47.0,74.0,12.0,23.0,7.0,13.0
-E06000049,513.0,2005.0,51.0,25.0,94009.0,9041.0,1.0,7.0,20.0,1709.0,17.0,147.0,21.0,28.0,43.0,20.0,96.0,39.0,60.0,94.0,117.0,25.0,9.0,15.0,7.0,319.0,317.0,4162.0,394.0,220.0,729.0,8560.0,1130.0,2786.0,651.0,150.0,499.0,312.0,185.0,423.0,59.0,54.0,95.0
-E06000050,2680.0,2462.0,28.0,19.0,7996.0,80360.0,2.0,5.0,21.0,77.0,8.0,76.0,29.0,11.0,32.0,21.0,47.0,20.0,20.0,47.0,112.0,22.0,9.0,5.0,11.0,107.0,91.0,767.0,130.0,92.0,280.0,492.0,165.0,686.0,462.0,471.0,1767.0,569.0,495.0,10189.0,21.0,21.0,43.0
-E07000027,0.0,5.0,11.0,12.0,8.0,3.0,22590.0,526.0,3324.0,0.0,2.0,2.0,9.0,2.0,198.0,1.0,18.0,5.0,6.0,7.0,6.0,15.0,15.0,3.0,1.0,5.0,4.0,15.0,8.0,3.0,13.0,6.0,3.0,7.0,4.0,7.0,19.0,3.0,17.0,5.0,1.0,1.0,9.0
-E07000029,7.0,27.0,5.0,10.0,15.0,23.0,705.0,22371.0,347.0,1.0,1.0,10.0,3.0,5.0,47.0,1.0,5.0,7.0,3.0,12.0,13.0,10.0,3.0,5.0,2.0,7.0,5.0,24.0,2.0,4.0,5.0,6.0,3.0,14.0,20.0,15.0,27.0,16.0,45.0,8.0,8.0,0.0,6.0
-E07000031,3.0,19.0,29.0,38.0,20.0,26.0,2695.0,409.0,29454.0,2.0,8.0,31.0,43.0,14.0,3046.0,11.0,107.0,34.0,10.0,55.0,20.0,153.0,413.0,5.0,25.0,37.0,9.0,38.0,14.0,18.0,17.0,19.0,10.0,6.0,27.0,4.0,34.0,18.0,12.0,12.0,46.0,8.0,10.0
-E07000037,7.0,13.0,5.0,5.0,858.0,28.0,1.0,0.0,3.0,19288.0,3.0,3.0,1.0,2.0,1.0,0.0,7.0,2.0,6.0,4.0,9.0,1.0,0.0,2.0,0.0,23.0,52.0,286.0,172.0,54.0,68.0,1060.0,1287.0,76.0,22.0,4.0,17.0,18.0,2.0,10.0,5.0,9.0,43.0
-E07000117,6.0,27.0,1331.0,24.0,16.0,15.0,1.0,2.0,9.0,21.0,17395.0,190.0,32.0,2149.0,47.0,5159.0,172.0,1173.0,1588.0,222.0,53.0,47.0,131.0,14.0,1.0,176.0,267.0,144.0,108.0,196.0,102.0,40.0,64.0,61.0,85.0,5.0,25.0,21.0,28.0,15.0,149.0,419.0,22.0
-E07000118,25.0,149.0,739.0,176.0,55.0,64.0,1.0,2.0,43.0,11.0,103.0,17280.0,235.0,275.0,173.0,101.0,1374.0,249.0,103.0,4071.0,942.0,226.0,16.0,6.0,1.0,1468.0,184.0,218.0,98.0,103.0,209.0,82.0,48.0,118.0,2048.0,75.0,131.0,204.0,400.0,56.0,21.0,18.0,29.0
-E07000119,29.0,71.0,367.0,7312.0,45.0,46.0,7.0,4.0,102.0,12.0,57.0,818.0,15100.0,123.0,536.0,73.0,3320.0,358.0,58.0,2112.0,282.0,3353.0,15.0,7.0,1.0,252.0,81.0,88.0,66.0,67.0,97.0,83.0,34.0,42.0,336.0,32.0,331.0,71.0,267.0,36.0,10.0,10.0,32.0
-E07000120,8.0,32.0,3351.0,60.0,28.0,15.0,0.0,1.0,8.0,4.0,2151.0,338.0,72.0,13194.0,68.0,1226.0,307.0,1433.0,1330.0,381.0,56.0,90.0,50.0,5.0,1.0,232.0,271.0,95.0,76.0,170.0,52.0,48.0,42.0,54.0,150.0,9.0,38.0,25.0,43.0,11.0,46.0,81.0,37.0
-E07000121,7.0,35.0,131.0,354.0,23.0,25.0,177.0,31.0,2007.0,3.0,38.0,213.0,343.0,65.0,39800.0,31.0,542.0,261.0,15.0,250.0,51.0,1452.0,516.0,15.0,11.0,81.0,47.0,113.0,24.0,79.0,49.0,33.0,13.0,40.0,118.0,13.0,53.0,21.0,44.0,19.0,26.0,12.0,10.0
-E07000122,2.0,20.0,636.0,35.0,24.0,9.0,0.0,2.0,9.0,9.0,4692.0,113.0,19.0,851.0,29.0,18170.0,96.0,645.0,701.0,114.0,37.0,30.0,596.0,20.0,3.0,101.0,177.0,74.0,32.0,103.0,39.0,28.0,33.0,19.0,80.0,5.0,18.0,30.0,17.0,12.0,340.0,155.0,44.0
-E07000123,74.0,199.0,2315.0,1897.0,108.0,141.0,17.0,7.0,227.0,30.0,468.0,4770.0,2904.0,726.0,1657.0,386.0,34082.0,2155.0,256.0,13492.0,1298.0,2404.0,80.0,13.0,1.0,1015.0,242.0,426.0,109.0,125.0,279.0,125.0,100.0,198.0,1222.0,77.0,266.0,227.0,886.0,69.0,82.0,87.0,118.0
-E07000124,13.0,37.0,1942.0,256.0,37.0,29.0,4.0,4.0,31.0,7.0,1187.0,637.0,412.0,1923.0,275.0,1108.0,1891.0,9896.0,224.0,1298.0,127.0,451.0,196.0,5.0,2.0,273.0,94.0,124.0,147.0,139.0,81.0,66.0,54.0,56.0,281.0,10.0,37.0,44.0,86.0,12.0,55.0,41.0,13.0
-E07000125,7.0,34.0,648.0,9.0,28.0,15.0,4.0,0.0,6.0,11.0,825.0,116.0,19.0,1337.0,17.0,399.0,57.0,208.0,10863.0,68.0,13.0,17.0,14.0,3.0,4.0,313.0,1281.0,130.0,161.0,892.0,93.0,67.0,98.0,53.0,68.0,0.0,12.0,13.0,21.0,12.0,56.0,176.0,35.0
-E07000126,55.0,193.0,1278.0,504.0,95.0,87.0,15.0,4.0,66.0,13.0,278.0,6537.0,611.0,472.0,457.0,197.0,5186.0,564.0,145.0,17478.0,1177.0,534.0,17.0,9.0,2.0,658.0,155.0,214.0,86.0,81.0,175.0,93.0,76.0,125.0,1405.0,58.0,161.0,233.0,619.0,65.0,24.0,17.0,31.0
-E07000127,192.0,396.0,110.0,68.0,101.0,180.0,3.0,3.0,20.0,6.0,25.0,1077.0,42.0,36.0,74.0,25.0,353.0,43.0,34.0,849.0,20637.0,64.0,4.0,6.0,0.0,287.0,78.0,164.0,59.0,43.0,146.0,70.0,39.0,113.0,4763.0,992.0,1461.0,1775.0,5220.0,285.0,13.0,20.0,17.0
-E07000128,7.0,24.0,139.0,4902.0,11.0,16.0,6.0,5.0,49.0,1.0,32.0,274.0,1310.0,59.0,1040.0,19.0,1299.0,308.0,16.0,404.0,62.0,17703.0,14.0,1.0,1.0,71.0,34.0,51.0,14.0,28.0,23.0,11.0,13.0,15.0,108.0,11.0,37.0,26.0,49.0,12.0,14.0,7.0,4.0
-E07000163,0.0,8.0,42.0,12.0,3.0,4.0,2.0,3.0,106.0,2.0,158.0,11.0,3.0,69.0,340.0,1713.0,23.0,198.0,40.0,23.0,12.0,17.0,11763.0,297.0,45.0,22.0,11.0,32.0,17.0,24.0,22.0,4.0,6.0,9.0,25.0,3.0,12.0,6.0,6.0,5.0,4279.0,135.0,71.0
-E07000165,17.0,27.0,17.0,12.0,22.0,327.0,5.0,5.0,23.0,6.0,13.0,13.0,1.0,6.0,22.0,79.0,17.0,21.0,9.0,16.0,7.0,9.0,384.0,45408.0,593.0,23.0,12.0,37.0,8.0,19.0,16.0,20.0,28.0,21.0,43.0,11.0,27.0,14.0,29.0,143.0,1485.0,79.0,290.0
-E07000166,17.0,19.0,27.0,18.0,35.0,48.0,22.0,11.0,27.0,17.0,15.0,18.0,10.0,17.0,38.0,16.0,19.0,5.0,8.0,14.0,9.0,7.0,55.0,399.0,13801.0,22.0,11.0,28.0,8.0,14.0,14.0,33.0,35.0,1.0,45.0,9.0,19.0,21.0,16.0,59.0,82.0,30.0,30.0
-E08000001,70.0,525.0,1827.0,105.0,286.0,163.0,9.0,5.0,47.0,80.0,199.0,2453.0,118.0,298.0,110.0,120.0,506.0,160.0,515.0,633.0,280.0,146.0,21.0,11.0,5.0,62464.0,4786.0,1579.0,610.0,925.0,3627.0,565.0,455.0,897.0,9110.0,93.0,257.0,418.0,194.0,87.0,63.0,116.0,137.0
-E08000002,56.0,228.0,554.0,35.0,241.0,71.0,4.0,4.0,25.0,87.0,243.0,248.0,28.0,313.0,56.0,172.0,94.0,120.0,2176.0,106.0,64.0,43.0,16.0,11.0,0.0,5080.0,32936.0,2490.0,1220.0,4868.0,2154.0,558.0,568.0,644.0,894.0,26.0,115.0,135.0,58.0,33.0,88.0,169.0,165.0
-E08000003,852.0,4232.0,766.0,177.0,9445.0,2886.0,45.0,24.0,128.0,3314.0,365.0,1099.0,235.0,426.0,259.0,361.0,553.0,321.0,1511.0,566.0,629.0,248.0,64.0,89.0,13.0,7017.0,12130.0,108658.0,11712.0,9269.0,18919.0,25549.0,17483.0,24760.0,5415.0,473.0,1706.0,1416.0,797.0,891.0,658.0,940.0,1125.0
-E08000004,36.0,204.0,157.0,31.0,342.0,138.0,1.0,1.0,7.0,328.0,100.0,122.0,22.0,113.0,38.0,86.0,54.0,36.0,605.0,47.0,57.0,24.0,11.0,4.0,2.0,806.0,1911.0,4525.0,45747.0,8003.0,993.0,1389.0,4872.0,746.0,509.0,29.0,120.0,111.0,44.0,53.0,98.0,455.0,831.0
-E08000005,52.0,209.0,220.0,23.0,191.0,90.0,3.0,2.0,16.0,131.0,235.0,154.0,25.0,232.0,32.0,143.0,72.0,74.0,2587.0,87.0,46.0,25.0,13.0,16.0,3.0,1270.0,4095.0,2394.0,5869.0,40260.0,965.0,622.0,1025.0,540.0,506.0,35.0,101.0,123.0,48.0,49.0,144.0,1131.0,475.0
-E08000006,358.0,2155.0,331.0,89.0,1273.0,696.0,16.0,11.0,55.0,445.0,136.0,559.0,72.0,181.0,104.0,139.0,243.0,97.0,573.0,268.0,278.0,95.0,15.0,25.0,12.0,6459.0,5689.0,10090.0,2178.0,2279.0,39355.0,3203.0,2576.0,6496.0,5845.0,286.0,724.0,767.0,420.0,328.0,127.0,219.0,235.0
-E08000007,215.0,801.0,112.0,27.0,6808.0,701.0,6.0,10.0,23.0,3324.0,79.0,115.0,31.0,56.0,31.0,50.0,68.0,37.0,180.0,65.0,80.0,36.0,3.0,17.0,2.0,767.0,961.0,11713.0,1673.0,894.0,1640.0,55275.0,8518.0,3998.0,737.0,80.0,302.0,219.0,101.0,148.0,84.0,95.0,175.0
-E08000008,57.0,226.0,65.0,13.0,611.0,166.0,3.0,2.0,13.0,2735.0,59.0,65.0,22.0,53.0,52.0,36.0,40.0,27.0,190.0,28.0,42.0,17.0,7.0,5.0,1.0,461.0,815.0,3729.0,4606.0,1041.0,751.0,4392.0,41324.0,985.0,298.0,38.0,101.0,78.0,47.0,44.0,56.0,135.0,317.0
-E08000009,523.0,3226.0,265.0,65.0,3510.0,1670.0,6.0,7.0,38.0,575.0,130.0,435.0,63.0,136.0,72.0,95.0,194.0,91.0,449.0,231.0,249.0,84.0,12.0,28.0,6.0,3097.0,2652.0,17100.0,2005.0,1928.0,12461.0,5780.0,2996.0,41677.0,3532.0,264.0,738.0,768.0,350.0,408.0,172.0,199.0,249.0
-E08000010,365.0,2000.0,254.0,82.0,323.0,340.0,6.0,2.0,45.0,36.0,81.0,1912.0,80.0,125.0,153.0,51.0,356.0,73.0,117.0,536.0,2483.0,87.0,12.0,6.0,2.0,4476.0,647.0,942.0,380.0,360.0,1658.0,328.0,291.0,676.0,68238.0,378.0,739.0,4073.0,772.0,206.0,69.0,41.0,100.0
-E08000011,2043.0,1124.0,39.0,18.0,187.0,502.0,3.0,1.0,3.0,13.0,32.0,115.0,9.0,27.0,27.0,16.0,36.0,15.0,20.0,78.0,1149.0,20.0,3.0,3.0,1.0,153.0,60.0,177.0,66.0,70.0,145.0,80.0,46.0,135.0,967.0,18500.0,11549.0,5725.0,3886.0,1305.0,12.0,16.0,15.0
-E08000012,4518.0,2628.0,69.0,113.0,604.0,3754.0,12.0,6.0,49.0,22.0,28.0,299.0,65.0,42.0,66.0,15.0,171.0,39.0,36.0,223.0,3042.0,57.0,1.0,9.0,2.0,297.0,206.0,666.0,115.0,113.0,381.0,294.0,119.0,493.0,1584.0,19655.0,118413.0,5053.0,24208.0,18094.0,60.0,39.0,60.0
-E08000013,1500.0,2516.0,39.0,16.0,208.0,413.0,5.0,1.0,21.0,17.0,22.0,251.0,21.0,20.0,26.0,19.0,99.0,19.0,26.0,147.0,1177.0,30.0,1.0,5.0,0.0,397.0,133.0,330.0,176.0,108.0,347.0,148.0,95.0,253.0,5787.0,2593.0,2213.0,32661.0,1324.0,431.0,28.0,15.0,26.0
-E08000014,474.0,406.0,43.0,29.0,92.0,449.0,2.0,6.0,17.0,10.0,11.0,360.0,57.0,14.0,41.0,14.0,186.0,18.0,19.0,402.0,5476.0,45.0,3.0,4.0,2.0,162.0,34.0,166.0,30.0,32.0,95.0,45.0,22.0,98.0,800.0,2966.0,11542.0,1411.0,55569.0,1918.0,64.0,10.0,20.0
-E08000015,348.0,382.0,15.0,6.0,203.0,6135.0,1.0,0.0,8.0,13.0,3.0,32.0,9.0,8.0,15.0,8.0,30.0,4.0,9.0,24.0,174.0,6.0,2.0,2.0,2.0,42.0,27.0,132.0,25.0,22.0,71.0,55.0,35.0,105.0,173.0,1065.0,4195.0,259.0,1194.0,74137.0,6.0,8.0,20.0
-E08000032,18.0,42.0,76.0,19.0,70.0,59.0,2.0,3.0,47.0,24.0,138.0,49.0,13.0,58.0,26.0,739.0,46.0,76.0,82.0,36.0,25.0,14.0,3963.0,1202.0,50.0,308.0,159.0,210.0,228.0,255.0,103.0,98.0,107.0,75.0,126.0,18.0,56.0,43.0,26.0,77.0,129611.0,8096.0,8897.0
-E08000033,19.0,54.0,49.0,5.0,60.0,427.0,1.0,2.0,7.0,22.0,289.0,21.0,14.0,57.0,10.0,116.0,27.0,30.0,254.0,18.0,26.0,14.0,137.0,114.0,14.0,121.0,144.0,168.0,322.0,689.0,106.0,78.0,135.0,85.0,73.0,11.0,40.0,27.0,33.0,110.0,6164.0,52014.0,10196.0
-E08000034,19.0,62.0,19.0,13.0,58.0,47.0,1.0,3.0,19.0,78.0,45.0,24.0,21.0,16.0,28.0,51.0,25.0,20.0,75.0,15.0,23.0,15.0,78.0,213.0,21.0,102.0,147.0,279.0,543.0,378.0,112.0,96.0,201.0,102.0,70.0,14.0,55.0,28.0,29.0,29.0,4949.0,7387.0,102258.0

data/example_population.csv

-lad19cd,n
-E06000006,128432
-E06000007,209547
-E06000008,148942
-E06000009,139305
-E06000049,380790
-E06000050,340502
-E07000027,67137
-E07000029,68424
-E07000031,104532
-E07000037,92221
-E07000117,88527
-E07000118,116821
-E07000119,79770
-E07000120,80815
-E07000121,144246
-E07000122,91405
-E07000123,141818
-E07000124,60057
-E07000125,70895
-E07000126,110527
-E07000127,113949
-E07000128,111223
-E07000163,56832
-E07000165,160533
-E07000166,53244
-E08000001,285372
-E08000002,190108
-E08000003,547627
-E08000004,235623
-E08000005,220001
-E08000006,254408
-E08000007,291775
-E08000008,225197
-E08000009,236370
-E08000010,326088
-E08000011,149571
-E08000012,494814
-E08000013,180049
-E08000014,275396
-E08000015,323235
-E08000032,537173
-E08000033,210082
-E08000034,438727

data/example_traffic_flow.csv

-,percent
-2020-03-07,1.0141949023033037
-2020-03-08,1.0163138381018992
-2020-03-09,1.0154243371338507
-2020-03-10,1.012667236918105
-2020-03-11,1.0102245691633625
-2020-03-12,1.0072472930149845
-2020-03-13,1.0006052034095325
-2020-03-14,0.9894735231045187
-2020-03-15,0.9743146763253316
-2020-03-16,0.9667143086495742
-2020-03-17,0.9482593539057927
-2020-03-18,0.9211688842569309
-2020-03-19,0.8945880229363807
-2020-03-20,0.8668549265559783
-2020-03-21,0.834094197759868
-2020-03-22,0.7944294269057416
-2020-03-23,0.7485219130879249
-2020-03-24,0.6881917786239129
-2020-03-25,0.6272480573684523
-2020-03-26,0.5638999936371897
-2020-03-27,0.5017736835776103
-2020-03-28,0.44053703375605185
-2020-03-29,0.3788461079627759
-2020-03-30,0.3341435334996338
-2020-03-31,0.31711273833696974
-2020-04-01,0.3093778772598053
-2020-04-02,0.3050779396556957
-2020-04-03,0.30357166376512984
-2020-04-04,0.3053762271323346
-2020-04-05,0.30836912502357244
-2020-04-06,0.3095403749979955
-2020-04-07,0.31038880266200736
-2020-04-08,0.312908922511469
-2020-04-09,0.3192703337581285
-2020-04-10,0.30937316435697937
-2020-04-11,0.3103370612678039
-2020-04-12,0.3063989018487852
-2020-04-13,0.29163189758827296
-2020-04-14,0.2945949983630953
-2020-04-15,0.2956020838775612
-2020-04-16,0.29294065125050756
-2020-04-17,0.3052128656981978
-2020-04-18,0.3075817012367629
-2020-04-19,0.3173484601073278
-2020-04-20,0.33788004382453807
-2020-04-21,0.3409055546103967
-2020-04-22,0.34508475168716385
-2020-04-23,0.34953521184458974
-2020-04-24,0.354561276040548
-2020-04-25,0.36117771558387785
-2020-04-26,0.36568302824490456
-2020-04-27,0.36895491999738966
-2020-04-28,0.36958157558654453
-2020-04-29,0.3706472434718878
-2020-04-30,0.372268223592728
-2020-05-01,0.3744898160129474
-2020-05-02,0.3779201629036545
-2020-05-03,0.3802682476631814
-2020-05-04,0.3832100649517966
-2020-05-05,0.38907118850677813
-2020-05-06,0.3956371147725478
-2020-05-07,0.40485464522199793
-2020-05-08,0.3949601107211625
-2020-05-09,0.3960848708619175
-2020-05-10,0.3973984044925996
-2020-05-11,0.40128892649396564
-2020-05-12,0.406395550865998
-2020-05-13,0.41460815947438656
-2020-05-14,0.42095961118228625
-2020-05-15,0.4460967021141616
-2020-05-16,0.465790890683463
-2020-05-17,0.48796051372524507
-2020-05-18,0.49996813258729034
-2020-05-19,0.5124237759659191
-2020-05-20,0.5229196073174462
-2020-05-21,0.5298486744873399
-2020-05-22,0.536455061589467
-2020-05-23,0.5405761627161707
-2020-05-24,0.5475760810901222
-2020-05-25,0.5439442717209233
-2020-05-26,0.5507774049599644
-2020-05-27,0.555184617773803
-2020-05-28,0.5623133092097337
-2020-05-29,0.5712402880017464
-2020-05-30,0.5869253875799384
-2020-05-31,0.6036901519822301
-2020-06-01,0.6242584126187974
-2020-06-02,0.6330852690126978
-2020-06-03,0.6327816409012493
-2020-06-04,0.6332087107416108
-2020-06-05,0.6329739880025216
-2020-06-06,0.6278607772393378
-2020-06-07,0.6225269204841496
-2020-06-08,0.6196878264651079
-2020-06-09,0.6189777673253793
-2020-06-10,0.6226422166917362
-2020-06-11,0.6249469589232389
-2020-06-12,0.6276954908811366
-2020-06-13,0.6419962738264654
-2020-06-14,0.6567377363850088
-2020-06-15,0.66626490016722
-2020-06-16,0.6706755259454577
-2020-06-17,0.6791090463690199
-2020-06-18,0.6842440190927955
-2020-06-19,0.693966748684767
-2020-06-20,0.7086620502201545
-2020-06-21,0.7195221838539997
-2020-06-22,0.7235945419776973
-2020-06-23,0.7322095681413321
-2020-06-24,0.7448391931584826
-2020-06-25,0.7606888030211947
-2020-06-26,0.7662017670693281
-2020-06-27,0.7555793052618869
-2020-06-28,0.7487156874756987
-2020-06-29,0.7480846411531049
-2020-06-30,0.7468395920389107
-2020-07-01,0.7427868934569296
-2020-07-02,0.7389505102043152
-2020-07-03,0.740022933460794
-2020-07-04,0.7487756642333434
-2020-07-05,0.7592279153234384
-2020-07-06,0.7692580149482262
-2020-07-07,0.773629051657819
-2020-07-08,0.7765973304808143
-2020-07-09,0.7802527206210943
-2020-07-10,0.7912548357629738
-2020-07-11,0.8036866244598093
-2020-07-12,0.8195193779848889
-2020-07-13,0.8215328636955671
-2020-07-14,0.826460377210391
-2020-07-15,0.8320311244634923
-2020-07-16,0.8394188383553345
-2020-07-17,0.844618181598814
-2020-07-18,0.8456580590907689
-2020-07-19,0.8454482497829038
-2020-07-20,0.85161570004657
-2020-07-21,0.8569941036745623
-2020-07-22,0.8609362296635616
-2020-07-23,0.8604941832487933
-2020-07-24,0.8623984502080819
-2020-07-25,0.860612579109952
-2020-07-26,0.8629527876854617
-2020-07-27,0.855724087708701
-2020-07-28,0.8546424817210854
-2020-07-29,0.8571790750942304
-2020-07-30,0.8641798240180493
-2020-07-31,0.8697064037859719
-2020-08-01,0.87961529909642
-2020-08-02,0.88404939078419
-2020-08-03,0.8954430142086799
-2020-08-04,0.8968076476985818
-2020-08-05,0.897247412052961
-2020-08-06,0.8970150341600986
-2020-08-07,0.8938761453721832
-2020-08-08,0.8963333470795762
-2020-08-09,0.899193741884797
-2020-08-10,0.9020367572330367
-2020-08-11,0.9061738244351799
-2020-08-12,0.9097634501187778
-2020-08-13,0.9090486519331283
-2020-08-14,0.9092000478381734
-2020-08-15,0.9052407777543285
-2020-08-16,0.8961101939019586
-2020-08-17,0.8939821690074837
-2020-08-18,0.8943217210103628
-2020-08-19,0.8903278664441101
-2020-08-20,0.8965534995549805
-2020-08-21,0.8955941929211452
-2020-08-22,0.896889517460918
-2020-08-23,0.9028768621917251
-2020-08-24,0.9082211756741755
-2020-08-25,0.9019709540365788
-2020-08-26,0.9112299332397649
-2020-08-27,0.9078194724763309
-2020-08-28,0.9118754989620624
-2020-08-29,0.9145255119934619
-2020-08-30,0.9181219677249194
-2020-08-31,0.9100946743771358
-2020-09-01,0.9234547873294586
-2020-09-02,0.9186062404613251
-2020-09-03,0.9191809161987761
-2020-09-04,0.9211935741340714
-2020-09-05,0.9228555407058023
-2020-09-06,0.9211025233262198
-2020-09-07,0.9271148750081811
-2020-09-08,0.923082694192357
-2020-09-09,0.9265546805119476
-2020-09-10,0.9286829888452542
-2020-09-11,0.9297269702531256
-2020-09-12,0.9278012039567614
-2020-09-13,0.9310832240030181
-2020-09-14,0.935277989034406
-2020-09-15,0.9358299507636685
-2020-09-16,0.9352239495359997
-2020-09-17,0.9369969714231999
-2020-09-18,0.9385499772021334
-2020-09-19,0.939943034741184
-2020-09-20,0.9384636877072061
-2020-09-21,0.9368837349686725

enqueue_pipeline.sh

-#!/bin/bash
-
-# Enqueues COVID-19 pipelines
-
-COMMUTE_VOL_FILE="data/210122_OFF_SEN_COVID19_road_traffic_national_table.xlsx"
-DATE_LOW="2020-10-30"
-DATE_HIGH="2021-01-22"
-
-TEMPLATE_CONFIG=template_config.yaml
-JSV_SCRIPT=/usr/shared_apps/packages/sge-8.1.9-gpu/default/common/sge_request.jsv.2.0b
-
-# Job submisison
-switch-gpu
-RESULTS_DIR=$global_scratch/covid19/scotland_${DATE_HIGH}_notier
-JOB_NAME="covid_scotland_${DATE_HIGH}_notier"	
-qsub -N $JOB_NAME covid_pipeline.sge \
-     --commute-volume "$COMMUTE_VOL_FILE" \
-     --date-range $DATE_LOW $DATE_HIGH \
-     --results-dir "$RESULTS_DIR" \
-     --config $TEMPLATE_CONFIG

template_config.yaml → example_config.yaml

@@ -6,43 +6,40 @@ ProcessData:
     - 2021-01-01
   mobility_matrix: data/mergedflows.csv
   population_size: data/c2019modagepop.csv
-  commute_volume: data/201231_OFF_SEN_COVID19_road_traffic_national_table.xlsx
+  commute_volume:   # Can be replaced by DfT traffic flow data - contact authors <c.jewell@lancaster.ac.uk>
 
   CasesData:
     input: url
     address: https://api.coronavirus.data.gov.uk/v2/data?areaType=ltla&metric=newCasesBySpecimenDate&format=json
-    pillars:
-      - Pillar 1
-      - Pillar 2
-    measure: specimen
+    pillars: None  # Capability to filter Pillar 1 and 2 testing data from PHE confidential line listing
+    measure: None  # Capability to filter date of test report from PHE confidential line listing
     format: json
  
   AreaCodeData:
     input: json
     address:  "https://services1.arcgis.com/ESMARspQHYMw9BZ9/arcgis/rest/services/LAD_APR_2019_UK_NC/FeatureServer/0/query?where=1%3D1&outFields=LAD19CD,LAD19NM&returnGeometry=false&returnDistinctValues=true&orderByFields=LAD19CD&outSR=4326&f=json"
     format: ons
-    regions: ["S"]
-
-  TierData:
-    input: api
-    address: None # Rip from gov.uk
-    format: api
+    regions:
+      - S  # Scotland
+      - E  # England
+      - W  # Wales
+      - NI # Northern Ireland
 
 Mcmc:
-  dmax: 84
-  nmax: 50
-  m: 1
-  occult_nmax: 15
-  num_event_time_updates: 35
-  num_bursts: 200
-  num_burst_samples: 50
-  thin: 20
+  dmax: 84  # Max distance to move events
+  nmax: 50  # Max num events per metapopulation/time to move
+  m: 1      # Number of metapopulations to move
+  occult_nmax: 15  # Max number of occults to add/delete per metapop/time
+  num_event_time_updates: 35  # Num event and occult updates per sweep of Gibbs MCMC sampler.
+  num_bursts: 200  # Number of MCMC bursts of `num_burst_samples` 
+  num_burst_samples: 50  # Number of MCMC samples per burst
+  thin: 20  # Thin MCMC samples every `thin` iterations
 
-ThinPosterior:
+ThinPosterior:  # Post-process further chain thinning HDF5 -> .pkl.
   start: 6000
   end: 10000
   by: 10
 
-Geopackage:
+Geopackage:  # covid.tasks.summary_geopackage
   base_geopackage: data/UK2019mod_pop.gpkg
   base_layer: UK2019mod_pop_xgen
\ No newline at end of file

model_spec_marginal.py

-"""Implements a marginalised COVID SEIR model as a TFP Joint Distribution"""
-
-import numpy as np
-import tensorflow as tf
-import tensorflow_probability as tfp
-
-from gemlib.distributions import DiscreteTimeStateTransitionModel
-from gemlib.util import compute_state
-from covid.util import impute_previous_cases
-import covid.data as data
-
-tfd = tfp.distributions
-DTYPE = np.float64
-
-STOICHIOMETRY = tf.constant([[-1, 1, 0, 0], [0, -1, 1, 0], [0, 0, -1, 1]])
-TIME_DELTA = 1.0
-XI_FREQ = 14  # baseline transmission changes every 14 days
-NU = tf.constant(0.5, dtype=DTYPE)  # E->I rate assumed known.
-
-
-def read_covariates(paths, date_low, date_high):
-    """Loads covariate data
-
-    :param paths: a dictionary of paths to data with keys {'mobility_matrix',
-                  'population_size', 'commute_volume'}
-    :returns: a dictionary of covariate information to be consumed by the model
-              {'C': commute_matrix, 'W': traffic_flow, 'N': population_size}
-    """
-    mobility = data.read_mobility(paths["mobility_matrix"])
-    popsize = data.read_population(paths["population_size"])
-    commute_volume = data.read_traffic_flow(
-        paths["commute_volume"], date_low=date_low, date_high=date_high
-    )
-
-    return dict(
-        C=mobility.to_numpy().astype(DTYPE),
-        W=commute_volume.to_numpy().astype(DTYPE),
-        N=popsize.to_numpy().astype(DTYPE),
-    )
-
-
-def impute_censored_events(cases):
-    """Imputes censored S->E and E->I events using geometric
-       sampling algorithm in `impute_previous_cases`
-
-    There are application-specific magic numbers hard-coded below,
-    which reflect experimentation to get the right lag between EI and
-    IR events, and SE and EI events respectively.  These were chosen
-    by experimentation and examination of the resulting epidemic
-    trajectories.
-
-    :param cases: a MxT matrix of case numbers (I->R)
-    :returns: a MxTx3 tensor of events where the first two indices of
-              the right-most dimension contain the imputed event times.
-    """
-    ei_events, lag_ei = impute_previous_cases(cases, 0.25)
-    se_events, lag_se = impute_previous_cases(ei_events, 0.5)
-    ir_events = np.pad(cases, ((0, 0), (lag_ei + lag_se - 2, 0)))
-    ei_events = np.pad(ei_events, ((0, 0), (lag_se - 1, 0)))
-    return tf.stack([se_events, ei_events, ir_events], axis=-1)
-
-
-def CovidUK(covariates, initial_state, initial_step, num_steps, priors):