Use case: The paradata currently includes all events related to answers being provided (e.g., initially set, changed, deleted, etc.) but includes only the last event related to answer being valid/invalid. In other words, the paradata describe whether the last answer given is valid/invalid, but not whether previous answers given were valid/invalid.
One potential use of paradata might be to investigate whether interviewers change answers in response to validation errors, and whether those changes suggest correcting an accidental error (e.g., changing 5000, which contains an extra 0, to 500) or editing data so that it fulfills a validation condition (e.g., entering 5000, seeing an error, changing to 4000, seeing an error, ⌠, and changing to a final answer that satisfies the validation condition but may be inaccurate data).
But current paradata do not allow this to be done easily. While the analyst could, in principle, reconstruct from the paradata whether each answer change result in a valid/invalid answer, the task would be substantially easier if the valid/invalid state for each answer change were part of paradata.
Potential changes:
Record these answer valid/invalid events in the data sent from the tablet to the server. This could result in large data files that might be difficult to send over poor networks.
Reconstruct the answer valid/invalid events on the server based on data received. This would be a computationally costly exercise that may not be a good use of system resources.
Thanks Arthur. I would like to second this feature request. Being able to track error messages flagged during the interview would be a very useful feature for QA purposes.
Use cases:
There is increasing concern that the validation rules added to questionnaires might help interviewers to fake individuals answers, sections or questionnaires. With the data of such interviews being consistent with checking rules, we currently have no means to identify and monitor this. Some users start arguing in favour of not using validation checks in interviews for that very purpose. Having records of error messages flagged during the interviewing process would provide us with a means to identify and monitor this type of behaviour.
For QA work of any survey it is important to check the functionality of all questions and sections and ensure interviewersâ correct understanding. With interviewers tendency to âsolveâ errors by fiddling around with the answer until the error messages disappears and providing limited feedback up, and as data user to only being able to see the output (data) but not the process, one cannot identify which questions are difficult/not understood or not functioning. If we had records in para of errors and warnings flagging, together with the variable name and ideally the validation number, one can relatively easily monitor errors by interviewer, and over time, identifying those who did not understand something so they can be retrained, and to identify questions that do not function or validations that fire wrongly. Currently there is a trade off between the use of validations and motoring of errors over time for targeted feedback. The feature would remove the trade-off.
Since e.g. count of errors and warnings flagged during an interview is survey independent, there is potential to develop one or more standard, off-the-shelf QA indicators that can be used for every Survey Solution survey to assess the quality of the interviews. One could test them in experiments, as well as the usefulness of in questionnaire validation checks. There is potentially a bit of publications etc that one could produce to demonstrate the benefits of Survey Solutions QA system.
Paradata in its current format is almost too large to be useful. It contains many passive events that make up 3/4 of the para data, make exporting and downloading time prohibitively long, but are not at all needed from a survey quality point of view (I spent quite some time trying to get something useful out of them). I end up using para data only sporadically, because it is taking too long. If it were to be faster it could be more comprehensively integrated into QA systems that are being built by some of the Survey Solutions users.
Changes required to have such functionality:
To shorten para data and make it more usable, the following actions can be dropped. VariableDisabled, VariableEnabled, QuestionDeclaredValid, QuestionDeclaredInvalid (at the end of an interview). There is no recognizable use for them in a survey QA monitoring exercise and they slow down export and downloading. If they are needed for internal purposes, they could be dropped upon para data export. I am happy to go into more details about any of them if of help.
Add entries into the paradata every time a warning or error message is fired, with the variable name, and if possible with indication if it is a warning or error message, and the validation number (the one that is displayed in the interviewer, to know which validation rule fired).
Touching base on this suggestion, is there any chance this could be addressed/included in future releases?
If I see it correctly, the current paradata file format still does not allow to easily identify if a validation condition was triggered subsequently to an âAnswerSetâ.
Would be great to hear opionions of other users as well!
I donât understand this. Whether a validation condition was evaluated depends on whether the question that was answered is mentioned at all in that condition. If yes, it will be re-evaluated. This dependency tree is done at compile time and is independent of the interview data. It is a questionnaire property,
Variable/Question: Q1 of type numeric. In the Designer I add one VC at Q1 with self>10.
During Interview Process:
Event 1: Interviewer enters value 8. Error message will be displayed in UI.
Event 2: Interviewer clears answers at Q1
Event 3: Interviewers enters value 11 at Q1. No more error messages appear.
Event 4 Interviewer Completes (& subsequently syncs the Interview)
In the paradata, there would be no event âQuestionDeclaredInvalidâ as at compile time (=Completion?) there are no errors left? (Correct me if I am wrong).
Current Result:
Event 1 - Answer Set - Q1||8||
Event 2 - AnswerRemoved - Q1||
Event 3 - Answer Set - Q1||11||
Desired Result:
Event 1 - Answer Set - Q1||8||
Event 2 - Question Declared Invalid (or similar) - probably same timestamp as Event 1 Q1||
Event 3 - AnswerRemoved Q1||
Event 4 - Answer Set Q1||11|
As highlighted by @arthurshaw2002 above, we could reconstruct that the value entered at Event 1 was conflicting existing VC.
If this is not possible since it is done at compile time, then obviously totally fine!
Whether the validation condition should be re-evaluated or not is decided at compile time. But the evaluation itself will occur at run time, correspondingly the result will be determined at run time as well.
The intermediate events are removed for efficiency.
I like your proposal to include whether the question was declared valid or invalid, but I have my reservations about it, for example I am trying to imagine how efficient it could be to investigate whether a question was declared invalid or valid in the paradata file. I think adding this intermediate event could be useful but paradata files are generally very large files.
I remember that we had the same fear with our interviewers in Honduras, we also feared that our interviewers might enter less information than the one provided by the informant. That fear was always present in the few weeks we had the interview in the field, to mitigate this risk our HQ users compared the amount of information collected and time allocated by one team of interviewers with the amount of information collected and time allocated by another team of interviewers, both teams in similar conditions.
A question triggering an error is supposed to be corrected. So I would virtually always expect the answer to be changed in response to validation errors. Therefore the additional recorded event would be trivial. It would always be there.
If I need this information explicitely, it will probably be only for a number of key questions. If I want to examine the paradata programmatically I can deduce from the answer whether a validation error was generated because I know the condition under which that happens from the questionnaire.
So this would just mean another line of progam code.
@peter_brueck , are you thinking of checking whether interviewers are engaged in this behavior globally or for specific key questions, as @klaus suggests?
As you may remember, we did have all the detailed events, including these some time ago, but cut them out to save time/space of synchronizations for the cases when large questionnaire(lot of events) was used in the survey and the survey team wasnât really interested in the detailed paradata analysis.
It would be useful to come up with use case(s) when weâd want to know about to events happening during the survey data collection and hopefully react immediately to affect data-collection. For these cases we definitely need the (para)data to be coming âliveâ. Detecting that a specific question doesnât really work /has errors and must be immediately corrected; or an interviewer doing completely unacceptable job and must be replaced would be examples.
For the post-collection tasks, or for those that we donât really have time/possibility to react while in the field lwe could have a dedicated step of âregeneratingâ and exporting ALL events so that we do richer analysis.
In general globally, as an âeasyâ way to identify the behavior & issue that you and Andreas raised in the OP above. Not only for a particular key question (that I might monitor exhaustively anyway) but to identify issues with questions OR Interviewers that I would not see that easily otherwise (having said that, a current proxy is the # of times an answer was changed/removed by question/interviewer).
But also for specific key questions it would be an âeasierâ way for users with less technical skills that might have a hard time to reconstruct a VC for a particular question.
As a naive enduser I could see this as a âlow-hangingâ fruit as it would âjustâ imply that the paradata gets another X number of rows per interview, depending on the amount of VCs in QX (+ their quality) & how often it was triggered. (Maybe in exchange the events âQuestionDeclaredValidâ can be removed from paradata, as @andreaskutka mentioned above ). Obviously, might be very different from developer standpoint if this causes 100 other issues, especially re system resources!
Subsequent question would be: What about if a subsequent answer set fulfills VC, so error message disappears. Including in paradata as well?
). Obviously, might be very different from developer standpoint if this causes 100 other issues, especially re system resources!