index_methodology.md

Methodology for calculating indices

Index methodology version 3.1
25 May 2020

The Oxford Covid-19 Government Response Tracker (GitHub repo, university website) tracks individual policy measures across 17 indicators. We also calculate several indices to give an overall impression of government activity, and this page describes how these indices are calculated. Changes to this methodology are recorded in the changelog below.

All of our indices are simple averages of the individual component indicators. This is described in equation 1 below where k is the number of component indicators in an index and I_j is the sub-index score for an individual indicator.

The different indices are comprised as follows:

Index name	k	C1	C2	C3	C4	C5	C6	C7	C8	E1	E2	E3	E4	H1	H2	H3	H4	H5	M1
Government response index	13	x	x	x	x	x	x	x	x	x	x			x	x	x
Containment and health index	11	x	x	x	x	x	x	x	x					x	x	x
Stringency index	9	x	x	x	x	x	x	x	x					x
Economic support index	2									x	x
Legacy stringency index	7	x	x	?	?	x	?	?	x					x

Two versions of each indicator are present in the database. A regular version which will return null values if there is not enough data to calculate the index, and a "display" version which will extraploate to smooth over the last seven days of the index based on the most recent complete data. This is explained below.

Calculating sub-index scores for each indicator

All of the indices use ordinal indicators where policies a ranked on a simple numerical scale. The project also records five non-ordinal indicators – E3, E4, H4, H5 and M1 – but these are not used in our index calculations.

Some indicators – C1-C7, E1 and H1 – have an additional binary flag variable that can be either 0 or 1. For C1-C7 and H1 this corresponds to the geographic scope of the policy. For E1, this flag variable corresponds to the sectoral scope of income support.

The codebook has details about each indicator and what the different values represent.

Because different indicators (j) have different maximum values (N_j) in their ordinal scales, and only some have flag variables, each sub-index score must be calculated separately. The different indicators are:

Indicator	Max. value (Nj)	Flag? (Fj)
C1	3 (0, 1, 2, 3)	yes=1
C2	3 (0, 1, 2, 3)	yes=1
C3	2 (0, 1, 2)	yes=1
C4	4 (0, 1, 2, 3, 4)	yes=1
C5	2 (0, 1, 2)	yes=1
C6	3 (0, 1, 2, 3)	yes=1
C7	2 (0, 1, 2)	yes=1
C8	4 (0, 1, 2, 3, 4)	no=0
E1	2 (0, 1, 2)	yes=1
E2	2 (0, 1, 2)	no=0
H1	2 (0, 1, 2)	yes=1
H2	3 (0, 1, 2, 3)	no=0
H3	2 (0, 1, 2)	no=0

Each sub-index score (I) for any given indicator (j) on any given day (t), is calculated by the function described in equation 2 based on the following parameters:

• the maximum value of the indicator (N_j)
• whether that indicator has a flag (F_j=1 if the indicator has a flag variable, or 0 if the indicator does not have a flag variable)
• the recorded policy value on the ordinal scale (v_j,t)
• the recorded binary flag for that indicator (f_j,t)

This normalises the different ordinal scales to produce a sub-index score between 0 and 100 where each full point on the ordinal scale is equally spaced. For indicators that do have a flag variable, if this flag is recorded as 0 (ie if the policy is geographically targeted or for E1 if the support only applies to informal sector workers) then this is treated as a half-step between ordinal values.

Note that the database only contains flag values if the indicator has a non-zero value. If a government has no policy for a given indicator (ie the indicator equals zero) then the corresponding flag is blank/null in the database. For the purposes of calculating the index, this is equivalent to a sub-index score of zero. In other words, I_j,t=0 if v_j,t=0.

(if v_j,t=0 then the function F_j-f_j,t is also treated as 0, see paragraph above.)

Our data is not always fully compelte and sometimes indicators are missing. We make the conservative assumption that an absence of data corresponds to a sub-index score (I_j,t) of zero.

Here is an explicit example of the calculation for a given country on a single day:

Indicator	vj,t	fj,t		Nj	Fj		Ij,t
C1	2	1		3	yes=1		66.67
C2	no data	no data		3	yes=1		0.00
C3	2	0		2	yes=1		75.00
C4	2	0		4	yes=1		37.50
C5	0	null		2	yes=1		0.00
C6	1	0		3	yes=1		16.67
C7	1	1		2	yes=1		50.00
C8	3	N/A		4	no=0		75.00
E1	2	0		2	yes=1		75.00
E2	2	N/A		2	no=0		100.00
H1	2	0		2	yes=1		75.00
H2	3	N/A		3	no=0		100.00
H3	2	N/A		2	no=0		100.00
Index
Government response							59.29
Containment and health							54.17
Stringency							43.98
Economic support							87.50

Dealing with gaps in the data for display purposes

Because data are updated on twice-weekly cycles, but not every country is updated in every cycle,recent dates may be prone to missing data. If fewer than k-1 indicators are present for an index on any given day, the index calculation is rejected and no value is returned. For the economic support indicator, where k=2, the index calculation is rejected if either of the two indicators are missing.

To increase consistency of recent data points which are perhaps mid contribution, index values pertaining to the past seven days are rejected if they have fewer policy indicators than another day in the past seven days, ie if there is another recent data point with all k indicators included, then no index will be calculated for dates with k-1.

Further, we produce two versions of each index. One with the raw calculated index values, plus we produce a "display" version which will "smooth" over gaps in the last seven days, populating each date with the last available "good" data point.

For example, the date at the time of writing was 22 May. The table below gives an example of which index calculations would be rejected based on the number of policy indicators with data on each data. In this table, we will consider the overall government response index where k=13.

Date	No. of valid indicators	No. of indicators in index (k)	Raw index	"Display" index
10/05/2020	11	13	null	null
11/05/2020	12	13	60	60
12/05/2020	10	13	null	null
13/05/2020	13	13	65	65
14/05/2020	10	13	null	null
15/05/2020	10	13	null	null
16/05/2020	10	13	null	65
17/05/2020	13	13	70	70
18/05/2020	13	13	75	75
19/05/2020	12	13	null	75
20/05/2020	12	13	null	75
21/05/2020	6	13	null	75
22/05/2020 (today)	4	13	null	75

Legacy stringency index

We also report a legacy stringency index that approximates the logic of the first version of the Stringency Index, which only had seven components under our old database structure with the old indicators S1-S7. We generally do not recommend using this legacy index, but it may be useful for continuity purposes.

The legacy indicator only uses seven indicators, and it chooses a single indicator between C3 and C4, and between C6 and C7, selecting whichever of those pairs provides a higher sub-index score. This is because C3 and C4 aim to measure the information previously measured by S3, and similarly for C6, C7 and the old S6. This method, shown in equation 3, faithfully recreates the logic of the old stringency index.

The individual sub-index scores for the legacy index are calculated through a slightly different formula to the one described in equation 2 above. This formula is described in equation 4 below (with a seperate formula for C8, the only indicator in this index without a flagged variable).

Index methodology changelog

• 25 May 2020: implemented v3 of index methodology. Altered sub-index forumula, created new indices (overall government response, containment & health, and economic support) and moved documentation to GitHub here