-
Notifications
You must be signed in to change notification settings - Fork 8
/
Copy pathGuidelines-for-Customisation.Rmd
executable file
·113 lines (64 loc) · 13 KB
/
Guidelines-for-Customisation.Rmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
---
title: "Guidelines for questionnaire customisation"
---
```{r options_communes, include=FALSE}
source("options_communes.R")
```
<div class="note">
</div>
## Introduction
A great deal of effort has been devoted to ensure that the questions can be used by interviewers to obtain answers from respondents that are both reliable and valid. Thus the questions are drafted in a clear simple language and follow clearly and logically from one to the other, while the layout is designed to make it easy for interviewers to administer the questionnaires. The wording and question sequence are designed to motivate respondents and help them recall information on past events. Using the model questionnaires verbatim is most likely to ensure that the results of the surveys are comparable across participating countries.
It is therefore important that even if adaption and contextualisation are made, the wording of the model questionnaires is not drastically changed.
## What is "customisation"?
Customisation (or adaptation) refers to the process during which the proposed Protection Assessment questionnaires are tailored to the population/context where a the Assessment is being conducted (that is, a national assessment, or a assessment conducted for a population group or for a selected area within a country), using standard principles and approaches, while maintaining global comparability of the indicators that will be derived from the collected data.
The customisation process is by no means an easy and straightforward one. Without a detailed understanding of all the standard tools and of the general principles and recommendations, customisation of questionnaires should not be attempted at country level without the assistance of an expert. During the customisation process, it is also absolutely critical that lessons learned from previous data collection activities are used effectively, and wherever necessary, tools are tested before final decisions are made. Testing may include organized pre-testing, field testing, piloting, and in some cases, cognitive testing. Analysis of raw data from previous assessment and data collection activities, as well as results from these efforts should also be undertaken for successful customisation of standard questionnaires.
## What kind of "Customisation"?
Customisation of the "Questionnaire Modules", "Questions" and "Response Categories" are necessary for at least two basic reasons:
* No single country/survey is expected or recommended to use all of the modules in proposed questionnaires
* No single standard questionnaire can accurately represent all human experience around the globe
Customisation covers the following types of changes to the standard protection questionnaires:
* Country-/assessment-specific modifications to already existing standard questions and response codes,
* Deletions from the standard questionnaires, and
* Additions to the proposed questionnaires.
## Modification
Certain parts of the proposed questionnaires must be modified. Indeed, in several instances, the proposed questionnaires include clear directives that a change or modification needs to be made. These cases are indicated using text such as “insert local name". Similarly, response categories that require customisation are also indicated.
## Deletion
No assessment is recommended to retain all of the modules and questions of the proposed protection questionnaires. First, there will always be some topics that will not be relevant in certain countries or regions. Second, decisions on the content of any assessment will ideally be made as a result of a thorough data gap assessment, generally based on the required analysis, and, for example, when information is available from other recent data sources, certain modules or sets of questions will be dropped. The process and analysis involved in a comprehensive data needs assessment will vary, but is a crucial step in determining the content of the assessment.
Determining what to exclude from the assessment is a balancing act that should take data needs into account, but also learn from countless experiences of data quality issues as a result of overloaded questionnaires. Country priorities will guide decisions, but may also work against achieving an optimum questionnaire size if negotiations turn more political than technical.
A final consideration will also rest with the ability to implement an adequate sample size, as this is often constrained by budget on one hand and on the other the known data quality issues associated with large sample sizes. For instance, some indicators are difficult to measure in low fertility settings, demanding higher sample sizes or complicated sample designs. Unless such issues can be technically addressed, the exclusion of such indicators may be necessary.
## Addition
Some Protection Asessment may also add topics, modules and questions which are not already in the proposed questionnaires. These could include additions that the proposed questionnaires already point to (for example, adding household assets to the list already in the questionnaires), or additions of modules or sets of questions that are not covered in the proposed questionnaires.
From the onset of considerations of what could be added in, you should know that this will affect the technical support available as well as require changes and considerations throughout the package of tools available, from sampling, training, instructions, and data entry application to tabulations and reporting.
As with the above exercise of deleting from the questionnaires, your entry point should be the indicator list or, alternatively, the tabulation plan. Questionnaire design is secondary to the need for precise information on what such proposed additions would be measuring and how such would be presented.
Only questions that are previously well-tested and validated should be included. Questions are often imported from other household surveys that have been conducted in the country. This does not necessarily mean that they are validated nor does it mean that such questions can work within the frame of a Protection Assessment.
If additions are made, please ensure that formatting and coding follow the rules in place for the proposed questionnaires. For entirely new topics it may be useful to build a new module and in other cases you will need to append to an existing module or insert within the existing flow. If you create new questions, [submit them for addition in the library here](https://github.com/unhcr-mena/protection-assessment-toolkit/issues).
## Rules and Useful Tips for Customisation
Customise but do not compromise global comparability:
* Assess the implications of changes;
* Check that all required questions to calculate the indicators are included;
* Check previous surveys to see how the customisation was done;
* Consider translations for all major languages spoken among the survey population – Arabic, Kurdish,etc..
Pre-test rigorously to make sure that :
* The questions are understood and the response categories are meaningful;
* The language style that can be understood by everyone;
* The skip-logic functions within the form are working well;
## Translation
Translation should be planned as an integral part of the study design.
### Importance of translation
Mistaken translation can greatly jeopardize research findings. As reported in the article [World values lost in translation](https://www.washingtonpost.com/news/monkey-cage/wp/2014/09/02/world-values-lost-in-translation/?utm_term=.57bbf105f87e), many translated terms showed different associations than the term used in English. It also shows the changes of translation in later waves of the survey made trend analysis impossible. It thus prevents the analysis on the stability of change in values, which is one of the main goal of many survey.
Translation costs will make up a very small part of a survey budget and cannot reasonably be looked at as a place to cut costs. Experience gained in organizing translation projects and selecting strong translators and other experts is likely to streamline even these costs.
A professional translator does 300 words/hour, __2500 words/day__ and a 40 minutes long questionnaire can be around 4000 characters long meaning that the initial translation could theoritically be done in 2 days. Translation cost for freelancer can vary but a benchmark of 10 cents/word can be used (meaning that the outsourced translation of a 4000 characters long questionnaire would cost about 400 USD).
### Avoid unwritten translation
When the same questionnaire is runned to different population group from different linguistic background, it may become necessary to have bilingual interviewers as they will have to present and read the questions in multiple language.
Sometimes, in addition of presenting and reading in multiple language, bilingual interviewers translate for respondents as they conduct the interview acting as interpreters. In other words, there is a written source questionnaire that the interviewers look at but there is never a written translation, only what they produce orally on the spot. This is sometimes called "on sight", "on the fly" or "oral"" translation.
Evidence available from recent investigations suggests that these modes of translation must be avoided whenever possible for the following reasons:
* There will be inevitabily some variance in translation performance resulting in potential misinterpretation of results.
* What might be gained in saving from translator will be lost with the extensive training and briefing that will need take place in order to disambiguate all complex terms and concept in the survey.
### Organising translation process
Translation is an important step when finalising the survey. The important elements to considerer are summarised as TRAPD (Translation, Review, Adjudication, Pretesting, and Documentation):
* __Translators__ produce, independently from each other, initial translations, (they provide the draft materials for the first discussion). Each translator may prepare a full translation (double/parallel translation) or the material to be translated may be divided among the translators ( split translation). Review of the first 10% of the initial translation (in case you are working with a new translator) may indicate that a given translator is not suitable for the project because it is unlikely that serious deficiencies in translation quality can be remedied by more training or improved instructions. If this is the case, it is probably better to start over with a new translator.
* __Reviewers__ review translations with the translators. Translators should not simply "hand over" the finished assignment abut should be included in the review discussion. Note relying on one person to provide the initial questionnaire translation is particularly problematic if the review is also undertaken by individuals rather than a team (these are reasons for working in teams rather than working with individuals). Even if only one translator can be hired, one or more persons with strong bilingual skills could be involved in the review process. Part of the review is to check for general tone consistency: this means that it is important to use the same style in the entire survey instrument, in terms of language register, politeness norms or level of difficulty. Some projects rely on procedures variously called "back translation" to check that their survey translations are adequate: the two source language versions are compared to try to find out if there are problems in the target language text. In practical and theoretical terms, it is recommended to focus attention on first producing the best possible translation and then directly evaluating the translation produced in the target language, rather than indirectly through a back translation. Comparisons of an original source text and a back-translated source text provide only limited and potentially misleading insight into the quality of the target language.
* One (or more) __adjudicator__ decides whether the translation is ready to move to detailed pretesting and also decides when the translation can be considered to be finalized and ready for fielding. Official approval emphasizes the importance of this step and the significance of translation procedures in the project.
* __Documentation__ of each step is used as a quality assurance and monitoring tool. It is also important if the survey has to be reproduced.
### Reference
To know more about the subject, one can consult this [Bibliography on the transaltion of survey](http://dgim.ucsf.edu/cadc/mm/translationguide.pdf), some important reference includes the [Lessons from the European Social Survey (ESS)](http://ccsg.isr.umich.edu/index.php/chapters/translation-chapter?id=215), the [Census Bureau Guideline for the Translation of Data Collection Instruments and Supporting Materials](https://www.census.gov/srd/papers/pdf/rsm2005-06.pdf) or this [Procedure to prevent differences in translated survey items using Survey Quality Prediction program](https://www.upf.edu/documents/3966940/3986764/RECSM_wp038.pdf).