If anyone has processed paradata for question-timing purposes, how have you dealt with the multiple-select questions?
Consider a numeric question for an illustration first.
Suppose the interviewer was setting the answers as follows:
t1 71 t2 65 t3 . t4 72
We can easily interpret this as three distinct answers. And the time to set the “finally standing answer” is (
Now consider a multiselect question with options
D and the interviewer setting:
t1 A t2 AB t3 ABC
So we treat it as one single answer to the multiselect question (since every option selection is recorded separately and immediately).
But in the case of
t1 A t2 AB t3 A t4 . t5 C
one would argue that this is 2 distinct answers , because the question ended up in an un-answered state after the option
A was unselected and before option
C was selected.
Yet, if the interviewer selected option
C before unselecting option
A, the question would not be in an unanswered state:
t1 A t2 AB t3 A t4 AC t5 C
The practical situation of this could be:
Interviewer: which options apply to you? A,B,C,D?
Respondent: A and B!
(interviewer marks A and B, notices an error, repeats the question)
Respondent: no, scratch that, it’s C!
(interviewer unselects earlier given answers and marks C).
Assuming the time to give the “finally standing answer” is needed, and that the final answer doesn’t contain either of
B I am hesitant to include period from
And if the “finally standing answer” sounds too exotic, consider for example, that you are interested in the number of changes to the answer to questions. You will get an equivalent task of determining spells in the sequence of selection of options of multiselect questions.
Another, but related question is
If the total time spent on the question includes all the earlier given and corrected answers, then, in theory this is affected by presence of the other questions and even more subtly, by presence of validations in other questions. Indeed, in the scenario above, if there was no other question with validation, then the interviewer would not have been alerted about the error and would not have repeated the question and not spent additional time on it.
Is it still valid to prescribe the time spent solely to this one question then?
Your thoughts and experience on these questions are very welcomed and appreciated!