Cleaning process and adjustments of collected data

Hi Survey Solutions team,

I am analyzing a process to verify the information collected by the survey teams in the field, after the interviewers visited some homes for ten days, there should be a process to review the data that was collected in the field, if the data collected has any detail that needs to be adjusted, it is necessary to make an adjustment to correct the answer obtained in the field.

Based on your experience as users of Survey Solutions conducting surveys around the world, how do you go about this process of fine-tuning and cleaning the collected data? Do you use Survey Solutions to correct your data?

Thank you very much for helping me and guiding me.

Hi Kevin,

I am a consultant working for the World Bank and National Statistical Offices in many countries on surevys and censuses.
I usually use the kind of process you describe in two situations:

  1. During the survey:
    to monitor the incoming interviews on a regular basis. This is to identify mistakes during the interviews and take appropriate measures, s.a. adjusting the questionnaire, adding additional validations, and instructing enumerators.

  2. After the survey: (seems to be your situation?)
    to merge the data from various versions of the questionnaire and to treat outliers in the variables, handle missing values (imputations), etc.

For this I write programs in R which process the exported data from the server.
Since R is a full blown programming language (not just a statistical package) there are no limits in what you can do, including accessing the server via the SuSo API.
For example, for a census with 10,000 incoming interviews per day it may not be possible to manually inspect every interview and decide to approve or reject it. So this can be done by a program instead.
Of course, other languages can also be used, but R is specifically designed to process large datasets of the type produced by a survey.