Commit a6146241 authored by Chris Jewell's avatar Chris Jewell
Browse files

Productionised and web-automated pipeline

CHANGES:

1. Introduced default unit traffic flow;
2. Removed old test data from data directory.  This now contains real static data only.
3. Brought README.md up to date.
4. Removed extraneous files from root project directory.
parent 6388c3c2
*~
__pycache__
poetry.lock
.ruffus_history.sqlite
# covid19uk: Bayesian stochastic spatial modelling for COVID-19 in the UK # covid19uk: Bayesian stochastic spatial modelling for COVID-19 in the UK
## Files This Python package implements a spatial stochastic SEIR model for COVID-19 in the UK,
using Local Authority District level positive test data, population data, and mobility
information. Details of the model implementation may be found in `doc/lancs_space_model_concept.pdf`.
* `covid` Python package
* `model_spec.py` defines the CovidUK model using `tfp.JointDistributionNamed`, plus helper functions
* `inference.py` demonstrates MCMC model fitting the model
* `simulate.py` demontrates simulating from the model
* `example_config.yaml` example configuration file containing data paths and MCMC settings
* `data` a directory containing example data (see below)
* `environment.yaml` conda description of the required environment. Create with `conda create -f environment.yaml`
8 `summary.py` python script to summarise MCMC results into a Geopkg file.
## Example data files
* `data/example_cases.csv` a file containing example case data for 43 local authorities in England collected and present of PHE's [website](https://coronavirus.data.gov.uk)
* `data/example_population.csv` a file containing local authority population data in the UK, taken from ONS prediction for December 2019
* `data/example_mobility.csv` inter local authority mobility matrix taken from UK Census 2011 commuting data
* `data/example_traffic_flow` a relative measure of traffic flow taken from mobility metrics from providers such as Google and Facebook. Data have been smoothed to represent a summary of the original data.
## Example workflow ## Workflow
The entire analysis chain, from case data through parameter inference and predictive
simulation to results summarised as long-format XLSX and geopackage documents.
The pipeline is run using the [`ruffus`](http://ruffus.org.uk) computational pipeline library.
The library relies heavily on [TensorFlow](https://tensorflow.org) and
[TensorFlow Probability](https://tensorflow.org/probability) machine learning libraries, and is
optimised for GPU acceleration (tested on NVIDIA K40 and V100 cards). This package also imports
an experimental library [`gemlib`](http://fhm-chicas-code.lancs.ac.uk/GEM/gemlib) hosted at Lancaster University.
This library is under active development as part of the [Wellcome Trust](https://wellcome.ac.uk) `GEM` project.
The pipeline gathers data from [the official UK Government website for data and insights on Coronavirus]
(https://coronavirus.data.gov.uk), together with population and mobility data taken from the [UK
Office for National Statistics Open Geography Portal](https://geoportal.statistics.gov.uk).
### Quickstart
```bash ```bash
$ conda env create --prefix=./env -f environment.txt $ poetry install # Python dependencies
$ conda activate ./env $ python -m covid.pipeline --config example_config.yaml --results-dir <output_dir>
$ python inference.py
$ python summary.py
``` ```
The global pipeline configuration file `example_config.yaml` contains sections for pipeline
stages where required. See file for documentation.
### Model specification
The `covid.model_spec` module contains the model specified as a `tensorflow_probability.distributions.JointDistributionNamed`
instance `Covid19UK`. This module also contains a model version number, and constants such as the stoichiometry matrix
characterising the state transition model, an accompanying next generation matrix function, a function to assemble data
specific to the model, and a function to initialise censored event data explored by the MCMC inference algorithm.
### Pipeline stages
Each pipeline stage loads input and saves output to disc. This is inherent to the `ruffus` pipeline
architecture, and provides the possibility to run different stages of the pipeline manually, as well as
introspection of data passed between each stage.
1. Data assembly: `covid.tasks.assemble_data` downloads or loads data from various sources, clips
to the desired date range require needed, and bundles into a pickled Python dictionary `<output_dir>/pipeline_data.pkl`.
2. Inference: `covid.tasks.mcmc` runs the data augmentation MCMC algorithm described in the concept note, producing
a (large!) HDF5 file containing draws from the joint posterior distribution `posterior.hd5`.
3. Sample thinning: `covid.tasks.thin_posterior` further thins the posterior draws contained in the HDF5 file into a (much
smaller) pickled Python dictionary `<output_dir>/thin_samples.pkl`
4. Next generation matrix: `covid.tasks.next_generation_matrix` computes the posterior next generation matrix for the
epidemic, from which measures of Local Authority District level and National-level reproduction number can be derived.
This posterior is saved in `<output_dir>/ngm.pkl`.
5. National Rt: `covid.tasks.overall_rt` evaluates the dominant eigenvalue of the next generation matrix samples using
power iteration and Rayleigh Quotient method. The dominant eigenvalue of the inter-LAD next generation matrix gives the
national reproduction number estimate.
6. Prediction: `covid.tasks.predict` calculates the Bayesian predictive distribution of the epidemic given the observed
data and joint posterior distribution. This is used in two ways:
a. in-sample predictions are made for the latest 7 and 14 day time intervals in the observed data time window. These
are saved as `<output_dir>/insample7.pkl` and `<output_dir>/insample14.pkl` `xarray` data structures.
b. medium-term predictions are made by simulating forward 56 days from the last+1 day of the observed data time window. These is saved as `<output_dir>/medium_term.pkl` `xarray` data structure.
7. Summary output:
a. LAD-level reproduction number: `covid.tasks.summarize.rt` takes the column sums of the next generation matrix as the
LAD-level reproduction number. This is saved in `<output_dir>/rt_summary.csv`.
b. Incidence summary: `covid.tasks.summarize.infec_incidence` calculates mean and quantile information for the medium term prediction, `<output_dir>/infec_incidence_summary.csv`.
c. Prevalence summary: `covid.tasks.summarize.prevalence` calculated the predicted prevalence of COVID-19 infection
(model E+I compartments) at LAD level, `<output_dir>/prevalence_summary.csv`.
d. Population attributable risk fraction for infection: `covid.tasks.within_between` calculates the population
attributable fraction of within-LAD versus between-LAD infection risk, `<output_dir>/within_between_summary.csv`.
e. Case exceedance: `covid.tasks.case_exceedance` calculates the probability that observed cases in the last 7 and 14
days of the observed timeseries exceeding the predictive distribution. This highlights regions that are behaving
atypically given the model, `<output_dir>/exceedance_summary.csv`.
8. In-sample predictive plots: `covid.tasks.insample_predictive_timeseries` plots graphs of the in-sample predictive
distribution for the last 7 and 14 days within the observed data time window, `<output_dir>/insample_plots7` and
`<output_dir>/insample_plots14`.
9. Geopackage summary: `covid.tasks.summary_geopackage` assembles summary information into a `geopackage` GIS file,
`<output_dir>/prediction.pkg`.
10. Long format summary: `covid.tasks.summary_longformat` assembles reproduction number, observed data, in-sample, and medium-term
predictive incidence and prevalence (per 100000 people) into a long-format XLSX file.
## COVID-19 Lancaster University data statement ## COVID-19 Lancaster University data statement
...@@ -40,3 +103,17 @@ UTLA: Upper Tier Local Authority ...@@ -40,3 +103,17 @@ UTLA: Upper Tier Local Authority
LAD: Local Authority District LAD: Local Authority District
### Files
* `covid` Python package
* `example_config.yaml` example configuration file containing data paths and MCMC settings
* `data` a directory containing example data (see below)
* `pyproject.py` a PEP518-compliant file describing the `poetry` build system and dependencies.
## Example data files
* `data/example_cases.csv` a file containing example case data for 43 local authorities in England collected and present of PHE's [website](https://coronavirus.data.gov.uk)
* `data/c2019modagepop.csv` a file containing local authority population data in the UK, taken from ONS prediction for December 2019. Local authorities [City of Westminster, City of London] and [Cornwall, Isles of Scilly] have been aggregated to meet commute data processing requirements.
* `data/mergedflows.csv` inter local authority mobility matrix taken from UK Census 2011 commuting data and aggregated up from Middle Super Output Area level (respecting aggregated LADs as above).
* `data/UK2019mod_pop.gpkg` a geopackage containing UK Local Authority Districts (2019) polygons together with population and areal metrics.
...@@ -56,6 +56,12 @@ def read_traffic_flow( ...@@ -56,6 +56,12 @@ def read_traffic_flow(
:param path: path to a traffic flow CSV with <date>,<Car> columns :param path: path to a traffic flow CSV with <date>,<Car> columns
:returns: a Pandas timeseries :returns: a Pandas timeseries
""" """
if path is None:
dates = np.arange(date_low, date_high, np.timedelta64(1, "D"))
return pd.DataFrame(np.ones(dates.shape[0], np.float64),
index=dates,
columns=["percent"])
commute_raw = pd.read_excel( commute_raw = pd.read_excel(
path, index_col="Date", skiprows=5, usecols=["Date", "Cars"] path, index_col="Date", skiprows=5, usecols=["Date", "Cars"]
) )
......
...@@ -11,7 +11,7 @@ import covid.data as data ...@@ -11,7 +11,7 @@ import covid.data as data
tfd = tfp.distributions tfd = tfp.distributions
VERSION = 0.3 VERSION = 0.5.0
DTYPE = np.float64 DTYPE = np.float64
STOICHIOMETRY = np.array([[-1, 1, 0, 0], [0, -1, 1, 0], [0, 0, -1, 1]]) STOICHIOMETRY = np.array([[-1, 1, 0, 0], [0, -1, 1, 0], [0, 0, -1, 1]])
......
#!/bin/bash
#$ -clear
#$ -jsv /usr/shared_apps/packages/sge-8.1.9-gpu/default/common/sge_request.jsv.2.0b
#$ -S /bin/bash
#$ -P chicas
#$ -q gpu
#$ -l ngpus=1
# -l ncpus=4
#$ -l h_vmem=64G
#$ -l h_rt=12:00:00
#$ -j y
#$ -cwd
. /etc/profile
. $HOME/.bash_profile
module add cuda/11.0
export XLA_FLAGS="--xla_gpu_cuda_data_dir=$CUDA_HOME"
echo Args: "$@"
poetry run python -m covid.pipeline "$@"
This diff is collapsed.
To,E06000006,E06000007,E06000008,E06000009,E06000049,E06000050,E07000027,E07000029,E07000031,E07000037,E07000117,E07000118,E07000119,E07000120,E07000121,E07000122,E07000123,E07000124,E07000125,E07000126,E07000127,E07000128,E07000163,E07000165,E07000166,E08000001,E08000002,E08000003,E08000004,E08000005,E08000006,E08000007,E08000008,E08000009,E08000010,E08000011,E08000012,E08000013,E08000014,E08000015,E08000032,E08000033,E08000034
E06000006,27270.0,4674.0,27.0,27.0,691.0,3663.0,4.0,8.0,8.0,33.0,5.0,105.0,14.0,15.0,20.0,10.0,53.0,21.0,23.0,52.0,211.0,21.0,5.0,6.0,6.0,201.0,103.0,471.0,88.0,136.0,226.0,338.0,131.0,413.0,791.0,1738.0,2890.0,2265.0,819.0,868.0,14.0,19.0,41.0
E06000007,5786.0,50422.0,208.0,96.0,2073.0,3894.0,11.0,25.0,60.0,107.0,72.0,496.0,82.0,89.0,135.0,65.0,254.0,89.0,104.0,363.0,626.0,110.0,18.0,15.0,6.0,1236.0,580.0,1804.0,423.0,420.0,1738.0,1005.0,492.0,1876.0,6539.0,1430.0,3191.0,6804.0,1246.0,1156.0,84.0,87.0,129.0
E06000008,22.0,148.0,31801.0,193.0,69.0,57.0,5.0,1.0,38.0,17.0,2103.0,1328.0,218.0,6278.0,242.0,1249.0,942.0,3126.0,1155.0,1276.0,161.0,230.0,64.0,31.0,1.0,1490.0,584.0,339.0,176.0,245.0,217.0,126.0,114.0,134.0,449.0,24.0,91.0,90.0,114.0,60.0,89.0,93.0,52.0
E06000009,16.0,44.0,222.0,32638.0,26.0,16.0,9.0,2.0,55.0,2.0,38.0,308.0,5167.0,81.0,500.0,29.0,1096.0,206.0,25.0,552.0,98.0,10119.0,17.0,4.0,1.0,133.0,33.0,120.0,19.0,26.0,61.0,25.0,23.0,38.0,131.0,31.0,66.0,47.0,74.0,12.0,23.0,7.0,13.0
E06000049,513.0,2005.0,51.0,25.0,94009.0,9041.0,1.0,7.0,20.0,1709.0,17.0,147.0,21.0,28.0,43.0,20.0,96.0,39.0,60.0,94.0,117.0,25.0,9.0,15.0,7.0,319.0,317.0,4162.0,394.0,220.0,729.0,8560.0,1130.0,2786.0,651.0,150.0,499.0,312.0,185.0,423.0,59.0,54.0,95.0
E06000050,2680.0,2462.0,28.0,19.0,7996.0,80360.0,2.0,5.0,21.0,77.0,8.0,76.0,29.0,11.0,32.0,21.0,47.0,20.0,20.0,47.0,112.0,22.0,9.0,5.0,11.0,107.0,91.0,767.0,130.0,92.0,280.0,492.0,165.0,686.0,462.0,471.0,1767.0,569.0,495.0,10189.0,21.0,21.0,43.0
E07000027,0.0,5.0,11.0,12.0,8.0,3.0,22590.0,526.0,3324.0,0.0,2.0,2.0,9.0,2.0,198.0,1.0,18.0,5.0,6.0,7.0,6.0,15.0,15.0,3.0,1.0,5.0,4.0,15.0,8.0,3.0,13.0,6.0,3.0,7.0,4.0,7.0,19.0,3.0,17.0,5.0,1.0,1.0,9.0
E07000029,7.0,27.0,5.0,10.0,15.0,23.0,705.0,22371.0,347.0,1.0,1.0,10.0,3.0,5.0,47.0,1.0,5.0,7.0,3.0,12.0,13.0,10.0,3.0,5.0,2.0,7.0,5.0,24.0,2.0,4.0,5.0,6.0,3.0,14.0,20.0,15.0,27.0,16.0,45.0,8.0,8.0,0.0,6.0
E07000031,3.0,19.0,29.0,38.0,20.0,26.0,2695.0,409.0,29454.0,2.0,8.0,31.0,43.0,14.0,3046.0,11.0,107.0,34.0,10.0,55.0,20.0,153.0,413.0,5.0,25.0,37.0,9.0,38.0,14.0,18.0,17.0,19.0,10.0,6.0,27.0,4.0,34.0,18.0,12.0,12.0,46.0,8.0,10.0
E07000037,7.0,13.0,5.0,5.0,858.0,28.0,1.0,0.0,3.0,19288.0,3.0,3.0,1.0,2.0,1.0,0.0,7.0,2.0,6.0,4.0,9.0,1.0,0.0,2.0,0.0,23.0,52.0,286.0,172.0,54.0,68.0,1060.0,1287.0,76.0,22.0,4.0,17.0,18.0,2.0,10.0,5.0,9.0,43.0
E07000117,6.0,27.0,1331.0,24.0,16.0,15.0,1.0,2.0,9.0,21.0,17395.0,190.0,32.0,2149.0,47.0,5159.0,172.0,1173.0,1588.0,222.0,53.0,47.0,131.0,14.0,1.0,176.0,267.0,144.0,108.0,196.0,102.0,40.0,64.0,61.0,85.0,5.0,25.0,21.0,28.0,15.0,149.0,419.0,22.0
E07000118,25.0,149.0,739.0,176.0,55.0,64.0,1.0,2.0,43.0,11.0,103.0,17280.0,235.0,275.0,173.0,101.0,1374.0,249.0,103.0,4071.0,942.0,226.0,16.0,6.0,1.0,1468.0,184.0,218.0,98.0,103.0,209.0,82.0,48.0,118.0,2048.0,75.0,131.0,204.0,400.0,56.0,21.0,18.0,29.0
E07000119,29.0,71.0,367.0,7312.0,45.0,46.0,7.0,4.0,102.0,12.0,57.0,818.0,15100.0,123.0,536.0,73.0,3320.0,358.0,58.0,2112.0,282.0,3353.0,15.0,7.0,1.0,252.0,81.0,88.0,66.0,67.0,97.0,83.0,34.0,42.0,336.0,32.0,331.0,71.0,267.0,36.0,10.0,10.0,32.0
E07000120,8.0,32.0,3351.0,60.0,28.0,15.0,0.0,1.0,8.0,4.0,2151.0,338.0,72.0,13194.0,68.0,1226.0,307.0,1433.0,1330.0,381.0,56.0,90.0,50.0,5.0,1.0,232.0,271.0,95.0,76.0,170.0,52.0,48.0,42.0,54.0,150.0,9.0,38.0,25.0,43.0,11.0,46.0,81.0,37.0
E07000121,7.0,35.0,131.0,354.0,23.0,25.0,177.0,31.0,2007.0,3.0,38.0,213.0,343.0,65.0,39800.0,31.0,542.0,261.0,15.0,250.0,51.0,1452.0,516.0,15.0,11.0,81.0,47.0,113.0,24.0,79.0,49.0,33.0,13.0,40.0,118.0,13.0,53.0,21.0,44.0,19.0,26.0,12.0,10.0
E07000122,2.0,20.0,636.0,35.0,24.0,9.0,0.0,2.0,9.0,9.0,4692.0,113.0,19.0,851.0,29.0,18170.0,96.0,645.0,701.0,114.0,37.0,30.0,596.0,20.0,3.0,101.0,177.0,74.0,32.0,103.0,39.0,28.0,33.0,19.0,80.0,5.0,18.0,30.0,17.0,12.0,340.0,155.0,44.0
E07000123,74.0,199.0,2315.0,1897.0,108.0,141.0,17.0,7.0,227.0,30.0,468.0,4770.0,2904.0,726.0,1657.0,386.0,34082.0,2155.0,256.0,13492.0,1298.0,2404.0,80.0,13.0,1.0,1015.0,242.0,426.0,109.0,125.0,279.0,125.0,100.0,198.0,1222.0,77.0,266.0,227.0,886.0,69.0,82.0,87.0,118.0
E07000124,13.0,37.0,1942.0,256.0,37.0,29.0,4.0,4.0,31.0,7.0,1187.0,637.0,412.0,1923.0,275.0,1108.0,1891.0,9896.0,224.0,1298.0,127.0,451.0,196.0,5.0,2.0,273.0,94.0,124.0,147.0,139.0,81.0,66.0,54.0,56.0,281.0,10.0,37.0,44.0,86.0,12.0,55.0,41.0,13.0
E07000125,7.0,34.0,648.0,9.0,28.0,15.0,4.0,0.0,6.0,11.0,825.0,116.0,19.0,1337.0,17.0,399.0,57.0,208.0,10863.0,68.0,13.0,17.0,14.0,3.0,4.0,313.0,1281.0,130.0,161.0,892.0,93.0,67.0,98.0,53.0,68.0,0.0,12.0,13.0,21.0,12.0,56.0,176.0,35.0
E07000126,55.0,193.0,1278.0,504.0,95.0,87.0,15.0,4.0,66.0,13.0,278.0,6537.0,611.0,472.0,457.0,197.0,5186.0,564.0,145.0,17478.0,1177.0,534.0,17.0,9.0,2.0,658.0,155.0,214.0,86.0,81.0,175.0,93.0,76.0,125.0,1405.0,58.0,161.0,233.0,619.0,65.0,24.0,17.0,31.0
E07000127,192.0,396.0,110.0,68.0,101.0,180.0,3.0,3.0,20.0,6.0,25.0,1077.0,42.0,36.0,74.0,25.0,353.0,43.0,34.0,849.0,20637.0,64.0,4.0,6.0,0.0,287.0,78.0,164.0,59.0,43.0,146.0,70.0,39.0,113.0,4763.0,992.0,1461.0,1775.0,5220.0,285.0,13.0,20.0,17.0
E07000128,7.0,24.0,139.0,4902.0,11.0,16.0,6.0,5.0,49.0,1.0,32.0,274.0,1310.0,59.0,1040.0,19.0,1299.0,308.0,16.0,404.0,62.0,17703.0,14.0,1.0,1.0,71.0,34.0,51.0,14.0,28.0,23.0,11.0,13.0,15.0,108.0,11.0,37.0,26.0,49.0,12.0,14.0,7.0,4.0
E07000163,0.0,8.0,42.0,12.0,3.0,4.0,2.0,3.0,106.0,2.0,158.0,11.0,3.0,69.0,340.0,1713.0,23.0,198.0,40.0,23.0,12.0,17.0,11763.0,297.0,45.0,22.0,11.0,32.0,17.0,24.0,22.0,4.0,6.0,9.0,25.0,3.0,12.0,6.0,6.0,5.0,4279.0,135.0,71.0
E07000165,17.0,27.0,17.0,12.0,22.0,327.0,5.0,5.0,23.0,6.0,13.0,13.0,1.0,6.0,22.0,79.0,17.0,21.0,9.0,16.0,7.0,9.0,384.0,45408.0,593.0,23.0,12.0,37.0,8.0,19.0,16.0,20.0,28.0,21.0,43.0,11.0,27.0,14.0,29.0,143.0,1485.0,79.0,290.0
E07000166,17.0,19.0,27.0,18.0,35.0,48.0,22.0,11.0,27.0,17.0,15.0,18.0,10.0,17.0,38.0,16.0,19.0,5.0,8.0,14.0,9.0,7.0,55.0,399.0,13801.0,22.0,11.0,28.0,8.0,14.0,14.0,33.0,35.0,1.0,45.0,9.0,19.0,21.0,16.0,59.0,82.0,30.0,30.0
E08000001,70.0,525.0,1827.0,105.0,286.0,163.0,9.0,5.0,47.0,80.0,199.0,2453.0,118.0,298.0,110.0,120.0,506.0,160.0,515.0,633.0,280.0,146.0,21.0,11.0,5.0,62464.0,4786.0,1579.0,610.0,925.0,3627.0,565.0,455.0,897.0,9110.0,93.0,257.0,418.0,194.0,87.0,63.0,116.0,137.0
E08000002,56.0,228.0,554.0,35.0,241.0,71.0,4.0,4.0,25.0,87.0,243.0,248.0,28.0,313.0,56.0,172.0,94.0,120.0,2176.0,106.0,64.0,43.0,16.0,11.0,0.0,5080.0,32936.0,2490.0,1220.0,4868.0,2154.0,558.0,568.0,644.0,894.0,26.0,115.0,135.0,58.0,33.0,88.0,169.0,165.0
E08000003,852.0,4232.0,766.0,177.0,9445.0,2886.0,45.0,24.0,128.0,3314.0,365.0,1099.0,235.0,426.0,259.0,361.0,553.0,321.0,1511.0,566.0,629.0,248.0,64.0,89.0,13.0,7017.0,12130.0,108658.0,11712.0,9269.0,18919.0,25549.0,17483.0,24760.0,5415.0,473.0,1706.0,1416.0,797.0,891.0,658.0,940.0,1125.0
E08000004,36.0,204.0,157.0,31.0,342.0,138.0,1.0,1.0,7.0,328.0,100.0,122.0,22.0,113.0,38.0,86.0,54.0,36.0,605.0,47.0,57.0,24.0,11.0,4.0,2.0,806.0,1911.0,4525.0,45747.0,8003.0,993.0,1389.0,4872.0,746.0,509.0,29.0,120.0,111.0,44.0,53.0,98.0,455.0,831.0
E08000005,52.0,209.0,220.0,23.0,191.0,90.0,3.0,2.0,16.0,131.0,235.0,154.0,25.0,232.0,32.0,143.0,72.0,74.0,2587.0,87.0,46.0,25.0,13.0,16.0,3.0,1270.0,4095.0,2394.0,5869.0,40260.0,965.0,622.0,1025.0,540.0,506.0,35.0,101.0,123.0,48.0,49.0,144.0,1131.0,475.0
E08000006,358.0,2155.0,331.0,89.0,1273.0,696.0,16.0,11.0,55.0,445.0,136.0,559.0,72.0,181.0,104.0,139.0,243.0,97.0,573.0,268.0,278.0,95.0,15.0,25.0,12.0,6459.0,5689.0,10090.0,2178.0,2279.0,39355.0,3203.0,2576.0,6496.0,5845.0,286.0,724.0,767.0,420.0,328.0,127.0,219.0,235.0
E08000007,215.0,801.0,112.0,27.0,6808.0,701.0,6.0,10.0,23.0,3324.0,79.0,115.0,31.0,56.0,31.0,50.0,68.0,37.0,180.0,65.0,80.0,36.0,3.0,17.0,2.0,767.0,961.0,11713.0,1673.0,894.0,1640.0,55275.0,8518.0,3998.0,737.0,80.0,302.0,219.0,101.0,148.0,84.0,95.0,175.0
E08000008,57.0,226.0,65.0,13.0,611.0,166.0,3.0,2.0,13.0,2735.0,59.0,65.0,22.0,53.0,52.0,36.0,40.0,27.0,190.0,28.0,42.0,17.0,7.0,5.0,1.0,461.0,815.0,3729.0,4606.0,1041.0,751.0,4392.0,41324.0,985.0,298.0,38.0,101.0,78.0,47.0,44.0,56.0,135.0,317.0
E08000009,523.0,3226.0,265.0,65.0,3510.0,1670.0,6.0,7.0,38.0,575.0,130.0,435.0,63.0,136.0,72.0,95.0,194.0,91.0,449.0,231.0,249.0,84.0,12.0,28.0,6.0,3097.0,2652.0,17100.0,2005.0,1928.0,12461.0,5780.0,2996.0,41677.0,3532.0,264.0,738.0,768.0,350.0,408.0,172.0,199.0,249.0
E08000010,365.0,2000.0,254.0,82.0,323.0,340.0,6.0,2.0,45.0,36.0,81.0,1912.0,80.0,125.0,153.0,51.0,356.0,73.0,117.0,536.0,2483.0,87.0,12.0,6.0,2.0,4476.0,647.0,942.0,380.0,360.0,1658.0,328.0,291.0,676.0,68238.0,378.0,739.0,4073.0,772.0,206.0,69.0,41.0,100.0
E08000011,2043.0,1124.0,39.0,18.0,187.0,502.0,3.0,1.0,3.0,13.0,32.0,115.0,9.0,27.0,27.0,16.0,36.0,15.0,20.0,78.0,1149.0,20.0,3.0,3.0,1.0,153.0,60.0,177.0,66.0,70.0,145.0,80.0,46.0,135.0,967.0,18500.0,11549.0,5725.0,3886.0,1305.0,12.0,16.0,15.0
E08000012,4518.0,2628.0,69.0,113.0,604.0,3754.0,12.0,6.0,49.0,22.0,28.0,299.0,65.0,42.0,66.0,15.0,171.0,39.0,36.0,223.0,3042.0,57.0,1.0,9.0,2.0,297.0,206.0,666.0,115.0,113.0,381.0,294.0,119.0,493.0,1584.0,19655.0,118413.0,5053.0,24208.0,18094.0,60.0,39.0,60.0
E08000013,1500.0,2516.0,39.0,16.0,208.0,413.0,5.0,1.0,21.0,17.0,22.0,251.0,21.0,20.0,26.0,19.0,99.0,19.0,26.0,147.0,1177.0,30.0,1.0,5.0,0.0,397.0,133.0,330.0,176.0,108.0,347.0,148.0,95.0,253.0,5787.0,2593.0,2213.0,32661.0,1324.0,431.0,28.0,15.0,26.0
E08000014,474.0,406.0,43.0,29.0,92.0,449.0,2.0,6.0,17.0,10.0,11.0,360.0,57.0,14.0,41.0,14.0,186.0,18.0,19.0,402.0,5476.0,45.0,3.0,4.0,2.0,162.0,34.0,166.0,30.0,32.0,95.0,45.0,22.0,98.0,800.0,2966.0,11542.0,1411.0,55569.0,1918.0,64.0,10.0,20.0
E08000015,348.0,382.0,15.0,6.0,203.0,6135.0,1.0,0.0,8.0,13.0,3.0,32.0,9.0,8.0,15.0,8.0,30.0,4.0,9.0,24.0,174.0,6.0,2.0,2.0,2.0,42.0,27.0,132.0,25.0,22.0,71.0,55.0,35.0,105.0,173.0,1065.0,4195.0,259.0,1194.0,74137.0,6.0,8.0,20.0
E08000032,18.0,42.0,76.0,19.0,70.0,59.0,2.0,3.0,47.0,24.0,138.0,49.0,13.0,58.0,26.0,739.0,46.0,76.0,82.0,36.0,25.0,14.0,3963.0,1202.0,50.0,308.0,159.0,210.0,228.0,255.0,103.0,98.0,107.0,75.0,126.0,18.0,56.0,43.0,26.0,77.0,129611.0,8096.0,8897.0
E08000033,19.0,54.0,49.0,5.0,60.0,427.0,1.0,2.0,7.0,22.0,289.0,21.0,14.0,57.0,10.0,116.0,27.0,30.0,254.0,18.0,26.0,14.0,137.0,114.0,14.0,121.0,144.0,168.0,322.0,689.0,106.0,78.0,135.0,85.0,73.0,11.0,40.0,27.0,33.0,110.0,6164.0,52014.0,10196.0
E08000034,19.0,62.0,19.0,13.0,58.0,47.0,1.0,3.0,19.0,78.0,45.0,24.0,21.0,16.0,28.0,51.0,25.0,20.0,75.0,15.0,23.0,15.0,78.0,213.0,21.0,102.0,147.0,279.0,543.0,378.0,112.0,96.0,201.0,102.0,70.0,14.0,55.0,28.0,29.0,29.0,4949.0,7387.0,102258.0
lad19cd,n
E06000006,128432
E06000007,209547
E06000008,148942
E06000009,139305
E06000049,380790
E06000050,340502
E07000027,67137
E07000029,68424
E07000031,104532
E07000037,92221
E07000117,88527
E07000118,116821
E07000119,79770
E07000120,80815
E07000121,144246
E07000122,91405
E07000123,141818
E07000124,60057
E07000125,70895
E07000126,110527
E07000127,113949
E07000128,111223
E07000163,56832
E07000165,160533
E07000166,53244
E08000001,285372
E08000002,190108
E08000003,547627
E08000004,235623
E08000005,220001
E08000006,254408
E08000007,291775
E08000008,225197
E08000009,236370
E08000010,326088
E08000011,149571
E08000012,494814
E08000013,180049
E08000014,275396
E08000015,323235
E08000032,537173
E08000033,210082
E08000034,438727
,percent
2020-03-07,1.0141949023033037
2020-03-08,1.0163138381018992
2020-03-09,1.0154243371338507
2020-03-10,1.012667236918105
2020-03-11,1.0102245691633625
2020-03-12,1.0072472930149845
2020-03-13,1.0006052034095325
2020-03-14,0.9894735231045187
2020-03-15,0.9743146763253316
2020-03-16,0.9667143086495742
2020-03-17,0.9482593539057927
2020-03-18,0.9211688842569309
2020-03-19,0.8945880229363807
2020-03-20,0.8668549265559783
2020-03-21,0.834094197759868
2020-03-22,0.7944294269057416
2020-03-23,0.7485219130879249
2020-03-24,0.6881917786239129
2020-03-25,0.6272480573684523
2020-03-26,0.5638999936371897
2020-03-27,0.5017736835776103
2020-03-28,0.44053703375605185
2020-03-29,0.3788461079627759
2020-03-30,0.3341435334996338
2020-03-31,0.31711273833696974
2020-04-01,0.3093778772598053
2020-04-02,0.3050779396556957
2020-04-03,0.30357166376512984
2020-04-04,0.3053762271323346
2020-04-05,0.30836912502357244
2020-04-06,0.3095403749979955
2020-04-07,0.31038880266200736
2020-04-08,0.312908922511469
2020-04-09,0.3192703337581285
2020-04-10,0.30937316435697937
2020-04-11,0.3103370612678039
2020-04-12,0.3063989018487852
2020-04-13,0.29163189758827296
2020-04-14,0.2945949983630953
2020-04-15,0.2956020838775612
2020-04-16,0.29294065125050756
2020-04-17,0.3052128656981978
2020-04-18,0.3075817012367629
2020-04-19,0.3173484601073278
2020-04-20,0.33788004382453807
2020-04-21,0.3409055546103967
2020-04-22,0.34508475168716385
2020-04-23,0.34953521184458974
2020-04-24,0.354561276040548
2020-04-25,0.36117771558387785
2020-04-26,0.36568302824490456
2020-04-27,0.36895491999738966
2020-04-28,0.36958157558654453
2020-04-29,0.3706472434718878
2020-04-30,0.372268223592728
2020-05-01,0.3744898160129474
2020-05-02,0.3779201629036545
2020-05-03,0.3802682476631814
2020-05-04,0.3832100649517966
2020-05-05,0.38907118850677813
2020-05-06,0.3956371147725478
2020-05-07,0.40485464522199793
2020-05-08,0.3949601107211625
2020-05-09,0.3960848708619175
2020-05-10,0.3973984044925996
2020-05-11,0.40128892649396564
2020-05-12,0.406395550865998
2020-05-13,0.41460815947438656
2020-05-14,0.42095961118228625
2020-05-15,0.4460967021141616
2020-05-16,0.465790890683463
2020-05-17,0.48796051372524507
2020-05-18,0.49996813258729034
2020-05-19,0.5124237759659191
2020-05-20,0.5229196073174462
2020-05-21,0.5298486744873399
2020-05-22,0.536455061589467
2020-05-23,0.5405761627161707
2020-05-24,0.5475760810901222
2020-05-25,0.5439442717209233
2020-05-26,0.5507774049599644
2020-05-27,0.555184617773803
2020-05-28,0.5623133092097337
2020-05-29,0.5712402880017464
2020-05-30,0.5869253875799384
2020-05-31,0.6036901519822301
2020-06-01,0.6242584126187974
2020-06-02,0.6330852690126978
2020-06-03,0.6327816409012493
2020-06-04,0.6332087107416108
2020-06-05,0.6329739880025216
2020-06-06,0.6278607772393378
2020-06-07,0.6225269204841496
2020-06-08,0.6196878264651079
2020-06-09,0.6189777673253793
2020-06-10,0.6226422166917362
2020-06-11,0.6249469589232389
2020-06-12,0.6276954908811366
2020-06-13,0.6419962738264654
2020-06-14,0.6567377363850088
2020-06-15,0.66626490016722
2020-06-16,0.6706755259454577
2020-06-17,0.6791090463690199
2020-06-18,0.6842440190927955
2020-06-19,0.693966748684767
2020-06-20,0.7086620502201545
2020-06-21,0.7195221838539997
2020-06-22,0.7235945419776973
2020-06-23,0.7322095681413321
2020-06-24,0.7448391931584826
2020-06-25,0.7606888030211947
2020-06-26,0.7662017670693281
2020-06-27,0.7555793052618869
2020-06-28,0.7487156874756987
2020-06-29,0.7480846411531049
2020-06-30,0.7468395920389107
2020-07-01,0.7427868934569296
2020-07-02,0.7389505102043152
2020-07-03,0.740022933460794
2020-07-04,0.7487756642333434
2020-07-05,0.7592279153234384
2020-07-06,0.7692580149482262
2020-07-07,0.773629051657819
2020-07-08,0.7765973304808143
2020-07-09,0.7802527206210943
2020-07-10,0.7912548357629738
2020-07-11,0.8036866244598093
2020-07-12,0.8195193779848889
2020-07-13,0.8215328636955671
2020-07-14,0.826460377210391
2020-07-15,0.8320311244634923
2020-07-16,0.8394188383553345
2020-07-17,0.844618181598814
2020-07-18,0.8456580590907689
2020-07-19,0.8454482497829038
2020-07-20,0.85161570004657
2020-07-21,0.8569941036745623
2020-07-22,0.8609362296635616
2020-07-23,0.8604941832487933
2020-07-24,0.8623984502080819
2020-07-25,0.860612579109952
2020-07-26,0.8629527876854617
2020-07-27,0.855724087708701
2020-07-28,0.8546424817210854
2020-07-29,0.8571790750942304
2020-07-30,0.8641798240180493
2020-07-31,0.8697064037859719
2020-08-01,0.87961529909642
2020-08-02,0.88404939078419
2020-08-03,0.8954430142086799
2020-08-04,0.8968076476985818
2020-08-05,0.897247412052961
2020-08-06,0.8970150341600986
2020-08-07,0.8938761453721832
2020-08-08,0.8963333470795762
2020-08-09,0.899193741884797
2020-08-10,0.9020367572330367
2020-08-11,0.9061738244351799
2020-08-12,0.9097634501187778
2020-08-13,0.9090486519331283
2020-08-14,0.9092000478381734
2020-08-15,0.9052407777543285
2020-08-16,0.8961101939019586
2020-08-17,0.8939821690074837
2020-08-18,0.8943217210103628
2020-08-19,0.8903278664441101
2020-08-20,0.8965534995549805
2020-08-21,0.8955941929211452
2020-08-22,0.896889517460918
2020-08-23,0.9028768621917251
2020-08-24,0.9082211756741755
2020-08-25,0.9019709540365788
2020-08-26,0.9112299332397649
2020-08-27,0.9078194724763309
2020-08-28,0.9118754989620624
2020-08-29,0.9145255119934619
2020-08-30,0.9181219677249194
2020-08-31,0.9100946743771358
2020-09-01,0.9234547873294586
2020-09-02,0.9186062404613251
2020-09-03,0.9191809161987761
2020-09-04,0.9211935741340714
2020-09-05,0.9228555407058023
2020-09-06,0.9211025233262198
2020-09-07,0.9271148750081811
2020-09-08,0.923082694192357
2020-09-09,0.9265546805119476
2020-09-10,0.9286829888452542
2020-09-11,0.9297269702531256
2020-09-12,0.9278012039567614
2020-09-13,0.9310832240030181
2020-09-14,0.935277989034406
2020-09-15,0.9358299507636685
2020-09-16,0.9352239495359997
2020-09-17,0.9369969714231999
2020-09-18,0.9385499772021334
2020-09-19,0.939943034741184
2020-09-20,0.9384636877072061
2020-09-21,0.9368837349686725
This diff is collapsed.
#!/bin/bash
# Enqueues COVID-19 pipelines
COMMUTE_VOL_FILE="data/210122_OFF_SEN_COVID19_road_traffic_national_table.xlsx"
DATE_LOW="2020-10-30"
DATE_HIGH="2021-01-22"
TEMPLATE_CONFIG=template_config.yaml
JSV_SCRIPT=/usr/shared_apps/packages/sge-8.1.9-gpu/default/common/sge_request.jsv.2.0b
# Job submisison
switch-gpu
RESULTS_DIR=$global_scratch/covid19/scotland_${DATE_HIGH}_notier
JOB_NAME="covid_scotland_${DATE_HIGH}_notier"
qsub -N $JOB_NAME covid_pipeline.sge \
--commute-volume "$COMMUTE_VOL_FILE" \
--date-range $DATE_LOW $DATE_HIGH \
--results-dir "$RESULTS_DIR" \
--config $TEMPLATE_CONFIG
...@@ -6,43 +6,40 @@ ProcessData: ...@@ -6,43 +6,40 @@ ProcessData:
- 2021-01-01 - 2021-01-01
mobility_matrix: data/mergedflows.csv mobility_matrix: data/mergedflows.csv
population_size: data/c2019modagepop.csv population_size: data/c2019modagepop.csv
commute_volume: data/201231_OFF_SEN_COVID19_road_traffic_national_table.xlsx commute_volume: # Can be replaced by DfT traffic flow data - contact authors <c.jewell@lancaster.ac.uk>
CasesData: CasesData:
input: url input: url
address: https://api.coronavirus.data.gov.uk/v2/data?areaType=ltla&metric=newCasesBySpecimenDate&format=json address: https://api.coronavirus.data.gov.uk/v2/data?areaType=ltla&metric=newCasesBySpecimenDate&format=json
pillars: pillars: None # Capability to filter Pillar 1 and 2 testing data from PHE confidential line listing
- Pillar 1 measure: None # Capability to filter date of test report from PHE confidential line listing
- Pillar 2
measure: specimen
format: json format: json
AreaCodeData: AreaCodeData:
input: json input: json
address: "https://services1.arcgis.com/ESMARspQHYMw9BZ9/arcgis/rest/services/LAD_APR_2019_UK_NC/FeatureServer/0/query?where=1%3D1&outFields=LAD19CD,LAD19NM&returnGeometry=false&returnDistinctValues=true&orderByFields=LAD19CD&outSR=4326&f=json" address: "https://services1.arcgis.com/ESMARspQHYMw9BZ9/arcgis/rest/services/LAD_APR_2019_UK_NC/FeatureServer/0/query?where=1%3D1&outFields=LAD19CD,LAD19NM&returnGeometry=false&returnDistinctValues=true&orderByFields=LAD19CD&outSR=4326&f=json"
format: ons format: ons
regions: ["S"] regions:
- S # Scotland
TierData: - E # England
input: api - W # Wales
address: None # Rip from gov.uk - NI # Northern Ireland
format: api
Mcmc: Mcmc:
dmax: 84 dmax: 84 # Max distance to move events
nmax: 50 nmax: 50 # Max num events per metapopulation/time to move
m: 1 m: 1 # Number of metapopulations to move
occult_nmax: 15 occult_nmax: 15 # Max number of occults to add/delete per metapop/time
num_event_time_updates: 35 num_event_time_updates: 35 # Num event and occult updates per sweep of Gibbs MCMC sampler.
num_bursts: 200 num_bursts: 200 # Number of MCMC bursts of `num_burst_samples`
num_burst_samples: 50 num_burst_samples: 50 # Number of MCMC samples per burst
thin: 20 thin: 20 # Thin MCMC samples every `thin` iterations
ThinPosterior: ThinPosterior: # Post-process further chain thinning HDF5 -> .pkl.
start: 6000 start: 6000
end: 10000 end: 10000
by: 10 by: 10
Geopackage: Geopackage: # covid.tasks.summary_geopackage
base_geopackage: data/UK2019mod_pop.gpkg base_geopackage: data/UK2019mod_pop.gpkg
base_layer: UK2019mod_pop_xgen base_layer: UK2019mod_pop_xgen
\ No newline at end of file
"""Implements a marginalised COVID SEIR model as a TFP Joint Distribution"""
import numpy as np
import tensorflow as tf
import tensorflow_probability as tfp
from gemlib.distributions import DiscreteTimeStateTransitionModel
from gemlib.util import compute_state
from covid.util import impute_previous_cases
import covid.data as data
tfd = tfp.distributions
DTYPE = np.float64
STOICHIOMETRY = tf.constant([[-1, 1, 0, 0], [0, -1, 1, 0], [0, 0, -1, 1]])
TIME_DELTA = 1.0
XI_FREQ = 14 # baseline transmission changes every 14 days
NU = tf.constant(0.5, dtype=DTYPE) # E->I rate assumed known.
def read_covariates(paths, date_low, date_high):
"""Loads covariate data
:param paths: a dictionary of paths to data with keys {'mobility_matrix',
'population_size', 'commute_volume'}
:returns: a dictionary of covariate information to be consumed by the model
{'C': commute_matrix, 'W': traffic_flow, 'N': population_size}
"""
mobility = data.read_mobility(paths["mobility_matrix"])
popsize = data.read_population(paths["population_size"])
commute_volume = data.read_traffic_flow(
paths["commute_volume"], date_low=date_low, date_high=date_high
)
return dict(
C=mobility.to_numpy().astype(DTYPE),