Audio auditing (also called CARI, computer assisted interview recording) is increasingly being used as a quality assurance tool in CAPI interviews, e.g. Weststat has used it for years now, Blaise and SurveyCTO packages have random audio auditing functionality. Broadly speaking it helps designers to test functionality of their instruments and quality assurance to find out if questions are administered correctly and if falsification is happening, which traditionally could only be achieved through costly (and often impossible) interview observations and re-interviews. It would be a particular useful tool for quality assurance of data collection the development context where data users and stakeholders are often far away from the agencies or firms that collect data, and where re-interviews and observations are impossible due to budget constraints, travel time or security reasons (as e.g. the case in NLSS). It would be great to see functionality in Survey Solutions that allows audio auditing to some degree.
In audio auditing, parts of the interview are audio recorded using the tablet after consent is given without an interviewer knowing when a recording is happening.
To set up audio editing a questionnaire designer needs:
- a mechanism to select whole interview, sections or certain questions that can be audio recorded
- a mechanism to select the probability of interviews and interview items of being recorded
- a mechanism to group items that are recorded, e.g. if a question on mobile phone ownership is recorded also record assets
- a condition that allows and disables automatic audio recording (to be able to close functionality if consent was not given)
To review audio audits one needs:
- a mechanism to order/assign to reviewers the recordings by interview/interviewer/item. Reviewers may be supervisors, but agencies like Westat more often use a parallel structure with centralized reviewers (to reduce/be able to manage reviewer effect).
*the ability to play the audio in the context of the interview answers, i.e. while being able to look at the filled in questionnaire
*the ability to codify the audio by answering a few (customizable) questions
A possible way of building basic audio auditing functionality into Survey Solutions in line with current functionality would be:
- for audio questions, allow an audio auditing options that:
- hides the question in the interviewer mode
- allows to specify a trigger to start the audio recording (max flexibility would be the first time a customized syntax validates as true, e.g. IsAnswered(AQuestion), or AQuestion=12)
- allows to specify an end trigger (either duration in seconds or syntax)
- the enabling condition would allow the audio recording (so one can make it conditional on the consent to audio recoding and select the probability using the system random number)
- displays in the supervisor and HQ mode, with the ability to listen to the audio
- the audio files are exported with the interview__id, variable name and roster row in the name/folder location (already the case I believe)
A user that wants to use random audio auditing in Survey Solutions could then:
- add an audio question to all the relevant parts of the questionnaire, and customize triggers
- either codify the audio files by adding supervisor questions after each audio auditing question (audible, question asked correctly, answer recorded correctly, leading, etc). and have supervisor or HQ fill them in. Importantly, I think often HQ would want to fill them in to allow for specialization in skills and tasks, so HQ should also be able to answer supervisor questions (I think this applies generally to supervisor questions)
- or codify the questions by having a separate questionnaire and organize themselves by using the file name and folder path of the audio recording.
Overall it might be a bit more cumbersome to set up compared to a new functionality tailored to audio auditing, but it would allow users to build most of the key functionality of audio auditing into the questionnaires and data management systems.