Abstract

Malcomb, Weaver and Krakowka (2014) published one of the first sub-national geographic climate change vulnerability models for a developing country (1.4). The authors intended for the study to be replicable across space (other African countries with similar data available) (7.1), time (when new survey data is published) (4.5 and 7.1), and vulnerability stimuli (7.1). The study’s social impacts are to address extreme vulnerability to climate change (1.3) and assisting in the allocation and evaluation of foreign aid (1.2). The methodology was designed to be “transparent and easily replicable” (2.1) in its use of “locally derived indicators and granular data” (2.1). The study was designed to address critiques of vulnerability models aimed at their uncertainty and sensitivity due to problems of scale and spatial aggregation, normative and subjective modelling decisions, and data availability, and challenges in model comparability (2.1). The model uses household adaptive capacity data from the United States Agency for International Development (USAID) Demographic and Health Surveys (DHS) (1.4 and 4.1) available in 44 African countries (7.1), livelihood sensitivity data from the USAID / Famine Early Warning Systems Network (FEWSnet) livelihood zones baseline surveys available in 23 African countries (3.6), and global physical exposure data from the United Nations Environment Programme (UNEP) Global Risk Data Platform.

This replication study is motivated by three factors. First, there is an urgent need to evaluate the reproducibility of research in human-environment and geographical sciences (HEGS) and to establish protocols and infrastructure for conducting and publishing reproduction/replication studies and reproducible research in HEGS. Second, a fully reproducible publication can be more readily replicated in new geographic, temporal, and thematic contexts, and tested for uncertainty due to data constraints and subjective modelling decisions. Third, climate change is causing increasingly severe in Africa. Improving the reproducibility and replicability of climate vulnerability research will hopefully enhance the potential for research to inform policy and reduce harm caused by climate change.

Malcomb et al (2014) produce two models of interest for Malawi. Figure 4, labelled “Malawi Household Resilience”, visualizes the average adaptive capacity score of households in each traditional authority. Figure 5, labelled “Malawi Composite Vulnerability Index”, visualizes vulnerability scores by locations (cells) in a continuous raster grid. In this study, we will attempt to identically reproduce figure 4 (adaptive capacity by traditional authority) and figure 5 (vulnerability grid) using The R Project for Statistical Computing and the same data sources cited in the original publication. We will visually compare our resulting reproduction figures with the original figures. Comparison will be aided by digitizing and joining the original figure results to the reproduction results for each model, and then calculating any differences between them. Differences will be visualized with thematic maps for both models, a confusion matrix for figure 4 (adaptive capacity by traditional authority), and a scatterplot for figure 5 (vulnerability grid). An exact reproduction should produce exact replicas of the rank order of traditional authorities by adaptive capacity and grid cells by vulnerability. We will test this with the Spearman’s Rho Correlation Coefficient, expecting values of 1 for perfect correlation.

The original study is a descriptive geographic multi-criteria analysis based on local expert opinion, and therefore has no testable hypotheses or effects.

The replication study data and code will be made available in a GitHub repository to the greatest extent that licensing and file sizes permit. The repository will be made public at github.com/HEGSRR/RPr-Malcomb-2014

Malcomb, D. W., E. A. Weaver, and A. R. Krakowka. 2014. Vulnerability modeling for sub-Saharan Africa: An operationalized approach in Malawi. Applied Geography 48:17–30. DOI:[10.1016/j.apgeog.2014.01.004](https://doi.org/10.1016/j.apgeog.2014.01.004).

Keywords

Reproducibility, Vulnerability, GIS, Climate Change, Africa

Study design

The reproduction study design will first implement the original study as closely as possible to reproduce the 2010 Household Resilience map (F4) and Malawi Vulnerability Map (F5). Our two confirmatory hypotheses are that we will be able to independently reproduce results for both maps.

The working hypotheses are therefore:

H1: There is no perfect positive correlation between Malcomb et al’s ranking of traditional authorities by household resilience and our reproduction study’s ranking of traditional authorities by household resilience.

H2: There is no perfect positive between Malcom et al’s ranking of locations by climate vulnerability and our reproduction study’s ranking of locations by climate vulnerability.

We will evaluate each of these hypotheses using a Spearman’s Rho Correlation. A failure to reject these hypotheses would indicate that our results do not exactly match those of the original authors. A positive correlation approaching 1 would indicate a partial reproduction

Original study design

The original study is observational and descriptive, with no hypotheses or effect sizes. The study is a multi-criteria analysis using geographic information systems (GIS) to implement a hierarchical geographic model of climate change vulnerability model in Malawi.

The spatial extent of the study was the country of Malawi. The spatial scale of the study was the third administrative level (traditional authorities) and a raster grid of unknown spatial resolution. The temporal extent of the study was explicitly 2004—2010 (4.5), but the contains secondary data collected earlier (3.6 and F5).

The model themes, indicators, and weights were selected based upon 70 interviews and 11 village focus groups from field trips to Malawi in March and August of 2011 (1.4, 4.2 and A1). Themes and indicators were also contextualized in literature (3.3 through 3.7) and adjusted based on redundancy and representativeness across the country (4.3). The model and weights were adjusted through “several iterations of the model using alternative weighting schemes” (4.3) to produce a “final product that reflects Malawi’s contextual and perceptual vulnerability” (4.3). Each theme was constructed of indicators from a single data provider: adaptive capacity is measured with USAID DHS surveys, livelihood sensitivity is measured with FEWSnet/Malawi Vulnerability Assessment Committee (MVAC) livelihood zones baseline data, and physical exposure is measured with UNEP Global Risk Data Platform data (T1 and T2). Although the authors emphasize a grounded local evidence-based selection of indicators and weights (2.1, 4.2, 5.1 and 7.1), other evidence in the publication suggests a model design based on a more pragmatic combination of factors including expert local opinion, deductive theory, and the availability and characteristics of data.

The study did not use any randomization.

The original study was conducted using STATA™ (4.4) and ArcGIS™ (4.6, F3 and F4) with unspecified software versions, by 2012 according to creation dates on map figures (F3, F4 and F5).

Computational environment

The study was originally conducted using ArcGIS and unspecified statistical software. This reproduction study uses R, including the rdhs package for DHS survey data, the sf package for vector analysis, the stars package for raster analysis, and the tmap package for cartography.

# set up default knitr parameters
knitr::opts_chunk$set(
  echo = FALSE,
  fig.width = 8,
  fig.path = paste0(here("results", "figures"), "/")
)

# these values allow you to access private and public raw data more efficiently
private_r <- here("data", "raw", "private")
public_r <- here("data", "raw", "public")
public_d <- here("data", "derived", "public")
scratch <- here("data", "scratch")

Data

Lakes

Major lakes were downloaded from MASDAP, the Malawi Spatial Data Platform.

Lakes data transformations

Dissolve lakes into a single multi-part feature with one field EA containing the value Lake.

Livelihood zones

Livelihood zones geographic data may be downloaded from the FEWS NET Data Center at https://fews.net/fews-data/335.

Livelihood zones attribute data was provided by FEWS NET in the form of one three spreadsheets describing typical livelihood profiles for each zone, with one sheet for poor households, one for middle income households, and one for rich households. This data was based on focus groups with stakeholders in each livelihood zone. The authors have summarized the individual poor household spreadsheets into one comprehensive table of variables relevant to the study.

Livelihood zone data transformations

In order to prepare geographic livelihood zone data for analysis, geometry errors are fixed, national parks are removed, and the coordinate reference system is transformed to EPSG:4326 (WGS 1984) geographic coordinates. Livelihood zone attribute data is then joined to the geographic data by livelihood zone code LZCODE.

Traditional authorities

Traditional authorities (TAs) data can be downloaded from Database of Global Administrative Areas (GADM) version 2.8 at https://gadm.org/download_country_v2.html and unzipped. This data must be downloaded directly from GADM. While the data license permits free use of data for research purposes and publication, it does not permit redistribution.

Traditional authorities (TAs)

Load traditional authorities (TA) data, fix geometry data, and count types of areas.

Type	N
City	4
Headquarter	16
National Park	6
Reserve	8
Sub-chief	66
Town	6
Traditional Authority	134
Urban	3
Water body	13

Visualize Lakes, Livelihood Zones, and TAs

TA data transformations

TA data includes conservation areas (reserves and national parks) and water bodies which do not contain populated villages. Extract conservation areas (forests and parks) to a new ta_cons_v layer.

Several of the Lake Malawi water body features in TA data erroneously include populated areas of land. Extract these features as ta_lake_malawi. Likoma island is incorrectly labelled as Lake Malawi, so do not include it as an error for extraction.

Remove conservation areas and water bodies from TAs.

## [1] "256 features in original traditional authorities"

## [1] "230 features after removing conservation areas and water bodies"

Find areas of Lake Malawi features that are actually land by buffering lakes by 500 meters and clipping the Lake Malawi TA features. Calculate new unique second level ID’s as 1000 times the row number. Remove splinter polygons by selecting polygons over 4 km^2 with centroids intersecting livelihood zones.

Merge fixed TA errors back into TA data and save results as derived ta_v.gpkg.

## [1] 9 features created by fixing errors on Lake Malawi shore

## [1] 239 features in final corrected traditional authorites

Drought risk and flood risk

The UNEP Global Risk Data Platform used for this research is no longer available online. The data is provided with the research compendium.

Household DHS data

Geographic USAID Demographic and Health Survey (DHS) data requires pre-approved access clearance and login credentials from the DHS Program. For this reproduction study, the following procedure was used to gain access:

Go to https://dhsprogram.com/Data/
Create an account, ideally with an education or government e-mail address
Within the Datasets menu, Create a new project
Enter the following information: Project Title: Reproducing a Vulnerability Model of Malawi Description of Study: The purpose of this study is to reproduce the methods of a published research article: Malcomb, D. W., E. A. Weaver, and A. R. Krakowka. 2014. Vulnerability modeling for sub-Saharan Africa: An operationalized approach in Malawi. Applied Geography 48:17–30. https://doi.org/10.1016/j.apgeog.2014.01.004. The authors of this paper used geocoded DHS surveys for Malawi in 2004 and 2010, in combination with FEWSnet livelihood data and UNEP flood and drought risk data. Following the author’s methodology, we plan to download the data using the rdhs package for R and aggregate the data at Malawi’s 2nd administrative level: districts. We will be working with a GitHub repository that stores the raw data locally in a directory ignored by the .gitignore file, and only moves the data into a shared and version-controlled directory once it has been aggregated to the District level. This will ensure that the privacy of survey respondents and requirements of data partners are protected, because all of the data will be aggregated into district polygons, as already shown and published in Malcomb et al (2014).
Choose Region: Sub-Saharan Africa
Click Show GPS Datasets at the top-left of the country tables
Check Survey and GPS data for Malawi
Save selection
Read and agree to the conditions of use for the DHS Program datasets and save these conditions for your metadata records.
Enter a Justification for using DHS Program Geographic Datasets: The research aim is to reproduce Malcomb et al (2014) in which GPS Datasets are used to spatially join DHS Survey data to Malawi’s Districts for the purpose of sub-national climate change vulnerability mapping. Therefore, the research will not be reproducible without the geographic datasets.

The rdhs package can be used to download the data, provided a login email and project name via console and password via pop-up dialogue.

Download the Malawi 2010 survey data and geographic points.

Load tabular data of household surveys

Load geographic data of household survey clusters. Some household survey points are erroneously placed at the WGS 1984 coordinate reference system origin (Equator and Prime Meridian).

DHS Data Transformations

In order to simultaneously maximize reproducibility while avoiding direct redistribution of DHS GPS data, we spatially join the GPS data to the Traditional Authority enumeration areas. Adaptive capacity is ultimately mapped by traditional authority, but the data comes from household-level surveys. Surveys are grouped into clusters with one geographic point. Therefore, the traditional authority to which each survey will be assigned must be spatially joined to the cluster point, and then joined by attribute to the household survey. The adaptive capacity calculation at the household level also requires urban/rural status, which is stored in the cluster.

Many household surveys contain inconclusive answers (e.g. “I don’t know”) or are missing data for survey questions used in the adaptive capacity calculation. The livestock variable will be calculated as a sum of four livestock types, so we remove any household with uncertain answers about any of the livestock types and remove households with missing data for all livestock types. Households with answers about some livestock types and missing data for others are still included in the data.

We remove incomplete household surveys.

Analysis

Adaptive Capacity

Rescale adaptive capacity indicators

Calculate percent rank for each component of household adaptive capacity. We had to make many assumptions about calculating individual components, e.g. about how to aggregate different forms of livestock, and which values to invert such that high numbers correspond to low capacity (e.g. number of orphans or sick members of the household). Rescaling to a quintile rank as described in the original study is unclear, especially considering the number of discrete or even binary inputs. We have made a judgement call to do this by calculating percent rank and multiplying by 4, producing a theoretical domain of 0 to 4 similar to that of quintiles.

Household adaptive capacity

Calculate household-level adaptive capacity scores based on original study Table 2 weights. The indicators have already been rescaled to a possible domain of 0 to 4, and the weights sum to 0.4, giving a possible domain of adaptive capacity scores from 0.0 to 1.6.

Summary statistics of adaptive capacity and its components at the household level.

Traditional authority adaptive capacity

Aggregate household adaptive capacity scores to traditional authorities. The original paper found adaptive capacity scores for 203 TAs, of which we found 6 TAs were conservation areas, leaving 197 meaningful TA scores. We created an additional 9 TAs from errors from three features on Lake Malawi, so if the original authors did not notice those errors, we could expect scores for 206 TAs.

Now that household adaptive capacity data has been aggregated, they may be saved to the data\derived\public directory.

Load aggregated public adaptive capacity data.

## Reading layer `ta_v' from data source 
##   `/Users/gracesokolow/Desktop/GitHub/RPr-Malcomb-2014/data/derived/public/ta_v.gpkg' 
##   using driver `GPKG'
## Simple feature collection with 239 features and 15 fields
## Geometry type: MULTIPOLYGON
## Dimension:     XY
## Bounding box:  xmin: 32.67152 ymin: -17.12721 xmax: 35.91505 ymax: -9.363796
## Geodetic CRS:  WGS 84

## # A tibble: 6 × 18
##   ta_id capacity_avg capacity_min capacity_max capacity_sd  n_hh livestock_avg
##   <dbl>        <dbl>        <dbl>        <dbl>       <dbl> <int>         <dbl>
## 1     1        0.667       0.384         0.855       0.139    20         0.120
## 2     2        0.380       0.100         0.846       0.146   315         0.982
## 3     3        0.390       0.0960        0.846       0.152   450         1.05 
## 4     4        0.635       0.227         0.993       0.163   301         0.404
## 5     5        0.370       0.164         0.761       0.158    24         1.41 
## 6     6        0.431       0.203         0.741       0.137    54         0.524
## # ℹ 11 more variables: sick_avg <dbl>, land_avg <dbl>, wealth_avg <dbl>,
## #   orphans_avg <dbl>, water_avg <dbl>, electricity_avg <dbl>,
## #   cooking_avg <dbl>, femalehh_avg <dbl>, cellphone_avg <dbl>,
## #   radio_avg <dbl>, urban_avg <dbl>

Count TAs with adaptive capacity data.

## [1] 215 TAs have adaptive capacity data

Finding scores for 215 traditional authorities is surprising, and most likely relates to differences in discovery and treatment of geometry errors and missing data. The reason(s) for these differences cannot be determined with the content of the original manuscript.

Mapping adaptive capacity

Join adaptive capacity data to geographic TAs and rescale in attempt to match original publication. The original publication figure 4 shows ranges from 11.48 to 25.77, but after rescaling indicators to domains of 0 to 4 and multiplying by percentages in table 2 (which sum to 0.4), the theoretical domain is only 0 to 1.6. We might suppose that the authors had rescaled adaptive capacity to a possible domain of 0 to 40 in accordance with the 40% weight of adaptive capacity in the overall vulnerability model. Therefore, we may multiply our possible domain of 0 to 1.6 by 25 to achieve a possible domain of 0 to 40.

	rpac_unscaled	rpac
nbr.val	215.00	215.00
nbr.na	24.00	24.00
min	0.30	7.41
max	0.68	16.90
range	0.38	9.48
median	0.43	10.66
mean	0.44	10.99
std.dev	0.07	1.80

The original publication uses the Jenks Natural Breaks method to classify the data.

rpac_class	n
1	67
2	80
3	53
4	15
NA	24

Reproduction figure 4

Map reproduction results for comparison to figure 4.

Digitize original study figure 4

Ordinal data from figure 4 was digitized in QGIS with the following procedure:

Copy image from the original publication pdf file using Adobe Acrobat Pro
Paste the image and save as a .png file with pixel dimensions 1982 by 2811
Use QGIS 3.26.3 Georeference the map image to match ta_v.gpkg using WGS 84 geographic coordinates (epsg:4326). Use linear georeferencing with points in metadata\malcomb_fig4.png.points
Make internal buffer to reduce the noise from boundary line symbology.
1. Project ta_v to UTM 36S epsg:32736: ta_v_fig4.gpkg:utm36s.
2. Calculate an internal buffer of -600m: ta_v_fig4.gpkg:utm36s.
3. Project back to WGS 84 epsg:4326: ta_v_fig4.gpkg:buffer_wgs84.
Extract the average and standard deviation of the original map’s red, green, and blue bands for each traditional authority using the zonal statistics algorithm: ta_v_fig4.gpkg:r, ta_v_fig4.gpkg:rb and ta_v_fig4.gpkg:rbg
Join the zonal statistics results to the ta_v layer by the ID_2 attribute: ta_v_fig4.gpkg:ta_v_fig4
Classify the results in a new field orac (original adaptive capacity) using the field calculator and CASE statements, choosing break points that classify most traditional authorities correctly.
Visually inspect results and edit the orac attribute for any mis-classified area.
The original map contains data in six conservation areas, noted with digitized point features in ta_v_fig4.gpkg:fig4_errors. Other areas are coded as follows:

code	description
-3	polygon too small to discern color or pattern fill
-2	white fill not matching any legend item
-1	pattern fill for “missing DHS data”
1	lowest adaptive capacity
2	…
3	…
4	highest adaptive capacity

Original study figure 4

Load digitized figure 4 data and display counts of results. Convert all forms of missing data to NA to be excluded from mapping and statistics. Join original figure 4 adaptive capacity results to ta_v.

orac	n
-3	3
-2	30
-1	3
1	38
2	56
3	72
4	37

Map original figure 4.

Compare adaptive capacity result

Calculate and map difference between the two maps.

##    
##      1  2  3  4
##   1 34 27  6  0
##   2  4 26 44  5
##   3  0  0 19 29
##   4  0  0  0  3

## 
##  Spearman's rank correlation rho
## 
## data:  ta_v$rpac_class and ta_v$orac
## S = 268637, p-value < 2.2e-16
## alternative hypothesis: true rho is greater than 0
## sample estimates:
##       rho 
## 0.7891711

Vulnerability

Extent and spatial resolution

Create bounding box representing the spatial extent of Malawi. Create a raster grid frame matching the extent of the bounding box and the spatial resolution of the drought exposure raster, which is 0.041667 decimal degrees. Although the flood risk raster has a coarser spatial resolution, visual inspection of the original figure 5 suggests that the finer spatial resolution of drought exposure was used for the original analysis.

Adaptive capacity

Convert adaptive capacity to raster grid.

Drought exposure

Clip and warp drought exposure to match our extent and spatial resolution.

Create a mask with the adaptive capacity results so that lakes, conservation areas, and traditional authorities with no data will not skew the classification / rescaling of drought exposure. Apply this mask to drought exposure. Masking is our own decision based on intuition: it is not specified in the original publication.

Classify drought exposure into quintile classes (0 to 4) Then rescale to 20% by multiplying by 4.

Flood risk

Clip and warp flood risk to match our extent and spatial resolution.

Mask and rescale flood. Since flood is already on scale from 0 to 4, simply multiply by 5 to achieve the 20% weight.

Livelihood sensitivity

Calculate livelihood sensitivity indicators from FEWSnet livelihood zone baseline profiles of poor households according to table 2.

Rescale livelihood sensitivity indicators into quantiles.

##              pctOwnCrop pctIncWage pctIncCashCrops pctDisasterCope ownCrop
## nbr.val            18.0       18.0            18.0            18.0    18.0
## nbr.null            0.0        0.0            13.0             1.0     1.0
## nbr.na              0.0        0.0             0.0             0.0     0.0
## min                29.4        9.7             0.0             0.0     0.0
## max                88.0       50.3            75.1            71.9     4.0
## range              58.6       40.6            75.1            71.9     4.0
## sum              1059.3      489.6           171.8           236.5    36.0
## median             55.0       24.7             0.0             8.8     2.0
## mean               58.9       27.2             9.5            13.1     2.0
## SE.mean             3.1        2.6             5.3             3.7     0.3
## CI.mean.0.95        6.6        5.5            11.2             7.9     0.6
## var               176.8      121.3           507.8           251.0     1.6
## std.dev            13.3       11.0            22.5            15.8     1.3
## coef.var            0.2        0.4             2.4             1.2     0.6
##              wageIncome cashCropIncome disasterCope
## nbr.val            18.0           18.0         18.0
## nbr.null            1.0            1.0          1.0
## nbr.na              0.0            0.0          0.0
## min                 0.0            0.0          0.0
## max                 4.0            1.2          4.0
## range               4.0            1.2          4.0
## sum                36.0           17.6         36.0
## median              2.0            1.2          2.0
## mean                2.0            1.0          2.0
## SE.mean             0.3            0.1          0.3
## CI.mean.0.95        0.6            0.2          0.6
## var                 1.6            0.1          1.6
## std.dev             1.3            0.4          1.3
## coef.var            0.6            0.4          0.6

Calculate aggregate livelihood sensitivity score

##              sensitivity
## nbr.val            18.00
## nbr.null            0.00
## nbr.na              0.00
## min                 5.88
## max                14.00
## range               8.12
## sum               161.65
## median              8.65
## mean                8.98
## SE.mean             0.56
## CI.mean.0.95        1.18
## var                 5.65
## std.dev             2.38
## coef.var            0.26

Convert livelihood sensitivity into raster grid

Vulnerability score

Calculate an aggregated vulnerability score by adding low adaptive capacity (invert adaptive capacity by subtracting from the maximum score of 40), livelihood sensitivity, drought exposure, and flood risk.

\[ Vulnerability = (40 - Adaptive Capacity) + Livelihood Sensitivity + Drought Exposure + Flood Risk \]

Reproduction figure 5

Original study figure 5

Comparing the reproduction of figure 5 with the original figure 5 requires first digitizing the original figure 5 (unclassified choropleth map with yellow to red gradient) in QGIS as follows:

Copy image of figure 5 from the original publication pdf file using Adobe Acrobat Pro
Paste the image and save as a .png file with pixel dimensions 1949 by 2811
Use QGIS 3.26.3 Georeference the map image to match ta_v.gpkg using WGS 84 geographic coordinates (epsg:4326). Use linear georeferencing with points in ...
Convert ta_capacity.tif raster to vector polygons
Extract the average blue and green bands from the georeferenced map image using zonal statistics
Save results as georef_bg.gpkg.

To approximate data values from the yellow to red gradient of the original map, the blue and green bands are then added, inverted, and rescaled to a range from 0 to 100.

Compare vulnerability result

## 
##  Spearman's rank correlation rho
## 
## data:  vulnerability_p$orv and vulnerability_p$rpv
## S = 7087504387, p-value < 2.2e-16
## alternative hypothesis: true rho is not equal to 0
## sample estimates:
##       rho 
## 0.1974578

Map differences in Figure 5

Create Alternate Vulnerability Score

Additive aggregation of adaptive capacity (assets and access), livelihood sensitivity, and physical exposure is insensitive to outliers. However, the purpose of a vulnerability index is to identify high (and low) priority areas for intervention: essentially, outliers.

Calculate vulnerability score using geometric aggregation. \[ Vulnerability = (40 - Adaptive Capacity) * Livelihood Sensitivity * Drought Exposure * Flood Risk \]

Figure 6: Geometric Vulnerability

Discussion

Figure Four

We expect a Spearman’s rank correlation rho for a successful reproduction to have a value of 1. Thus, the Spearman’s rho of 0.7891711 generated for Figure 4 indicates a partially successful reproduction.

Some of the difference between the original study and the reproduction may be due to the different numbers of Traditional Authorities found in each study. The original paper found 203 traditional authorities, but we determined that 6 of these were in conservation areas. However, we also found an additional 9 TAs hidden by the features of Lake Malawi, leading us to expect a result of 206 TAs. In our reproduction, though, we found 215. The different number of TAs found in our study may have contributed to the imperfect reproduction of Figure 4.

However, this cannot be the sole cause of the difference because in the error map we generated for Figure 4, we distinguish between TAs that are missing and TAs that have diverging values. One cause of the difference in values for TAs that both studies included may be the lack of information provided in the original study about how the adaptive capacity data was scaled. In this reproduction, we supposed that the authors may have rescaled the domain from 0 to 1.6 to 0 to 40, but this cannot be confirmed by the original manuscript.

Figure 5

We expect a Spearman’s rank correlation rho for a successful reproduction to have a value of 1. Thus, the Spearman’s rho of 0.1974578 generated for Figure 5 indicates a failed reproduction. By mapping the differences in the original Figure 5 compared with the reproduction, we see that the reproduction largely underestimated the vulnerability scores throughout Malawi generated by the original paper.

This may be due in part to the lack of information provided in the original study about how the drought data, flood data, and livelihood sensitivity scores were transformed into rasters at a uniform scale. In particular, it appears that the the coarser flood data may have been transformed to the finer scale of the drought dataset. This could be misleading, as it communicates flood risk with more detail than is available. Even so, this does not explain the discrepancy between the original study and the reproduction given that we identified and followed this unusual decision in our reproduction.

In addition to replicating Figure 5 as presented in the original study, we also made an alternate version using geometric, rather than additive, aggregation. Additive aggregation is insensitive to outliers, but one use for adaptive capacity mapping is precisely to identify outliers: areas of high vulnerability may present pressing cases for aid, while areas of low vulnerability may represent areas that do not need additional aid at this time. Geometric aggregation multiplies index factors rather than adding them, so if there is a very low or very high score, it is unlikely to be cancelled out by another score in the opposite direction. In geometric aggregation, only areas with moderate scores across all factors can be considered to have moderate risk, whereas in additive aggregation, it is easier for a very low score for one factor and a very high score for another factor to cancel out and give the impression of moderate vulnerability, when this may not be the case functionally.

Conclusions

In this study, we were partially able to reproduce the original study’s map of household resilience as a function of assets and access, but we were unable to reproduce their map of overall vulnerability. One of the goals of the original paper was to “determine geographic areas most susceptible to hazards, climatic change and vulnerability, and degraded livelihood security” in order to inform pragmatic allocation of climate change adaptation funding. Our failure to fully reproduce this study indicates that the original study could be made stronger by providing further explanation of its methods and decision-making process. In turn, this would allow for future reproduction and replication studies to test the precision of the original study as well as develop new methods to improve its accuracy.

In addition to making the results more robust, increasing the reproducibility of the original study is vital to maintaining its relevance in the application of climate adaptation. The original study was published in 2014, using data from fieldwork conducted in 2011, DHS data from 2004 and 2010, MVAC data from 2003-2005, and UNEP/GRID data from 2011-2012. Malawi’s rapidly growing population was cited as one of the original drivers of this research. It also underscores the importance of being able to easily update the vulnerability model and identify new priority areas for assistance as the country’s population grows and changes. Indeed, if this model is successful in channeling funds to where they are needed most, it would necessitate frequent re-evaluation to identify new priority areas for aid as existing needs in previously identified areas are hopefully met.

Integrity Statement

This report and its preregistration were written after already attempting the reproduction study, including acquisition and analysis of all of the secondary data sources required. However, the preregistered analysis plan was written as if we had no prior knowledge of the data other than what is documented in the study. Holler has previously reviewed and compared other climate vulnerability models for Malawi, and conducted a scoping study in the Lilongwe and Mangochi districts of Malawi in 2015, including meeting with the Regional Centre for Mapping of Resources for Development (RCMRD) consultants who created the Malawi Hazards and Vulnerability Atlas (2015).

References

Referencing the original paper

Sections

Introduction
Complex vulnerability
Evidence-based Indicators
Methodology
Results
Discussion
Conclusion

Tables, figures, other elements

T1 Evidence-based complex vulnerability indicators
T2 Weighted indicators by metatheme
F1 Map of Malawi
F2 Vulnerability web
F3 Malawi Household Resilience (2004)
F4 Malawi Household Resilience (2010)
F5 Malawi Composite Vulnerability Index
A1 Appendix 1
R References

Other References

Barrett, S. 2014. Subnational Climate Justice? Adaptation Finance Distribution and Climate Vulnerability. World Development 58:130–142. DOI: 10.1016/j.worlddev.2014.01.014. Gallopín, G. C. 2006. Linkages Between Vulnerability, Resilience, and Adaptive Capacity. Global Environmental Change 16 (3):293–303. DOI: 10.1016/j.gloenvcha.2006.02.004. Rufat, S., E. Tate, C. G. Burton, and A. S. Maroof. 2015. Social vulnerability to floods: Review of case studies and implications for measurement. International Journal of Disaster Risk Reduction 14:470–486. DOI: 10.1016/j.ijdrr.2015.09.013. Smit, B., and J. Wandel. 2006. Adaptation, adaptive capacity and vulnerability. Global Environmental Change 16 (3):282–292. DOI: 10.1016/j.gloenvcha.2006.03.008.

Reproduction Analysis of Malcomb et al 2014

Joseph Holler, Kufre Udoh, Drew An-Pham, Middlebury Open GIScience Classes

2023-10-16