Proposal
Testing scenarios are a wonderful and criminally under-used feature, but in practice they are very often too fragile. To use them, one may need to do a lot of clicking. To make them no longer applicable, one simply needs to make changes such that data from the scenario can no longer be captured. And that is where I, personally, stopped using scenarios.
To make testing scenarios more useful, I would suggest two changes:
- Allow exporting, editing and updating scenarios (outside of Designer).
- Allow scenarios to be created by importing a scenario file (composed outside of Designer, imported into Designer) (nice to have)
- Provide an interface or domain-specific language to generate the JSON files that underlie Designer’s testing scenarios (stretch goal)
Already, one can see the the test as some sort of JSON. However, to my knowledge, there’s no way to edit and update scenarios if some part of them is broken/invalidated by changes to the questionnaire.
Some inspiration drawn from shinytest2, an R package for creating by recording usage of webapp (e.g., provide input in field x, etc) and editing those tests as needed.
Affected subsystems
Designer
Users
Designers–particularly are large surveys–who are sick of re-entering fake data endlessly to test things.
Alternatives
The traditional types of “testing”:
- Manual in-office testing
- Scripted whitebox testing: a questionnaire developer runs
- Unscripted testing: get some colleagues or interviewers to punch in data
- Manual “field” testing
- Use interviewer testing to “test” by getting participants to fill out the survey
- Use pre-test/pilot to test with interviewers/respondents who provide real and (mostly) full information in an app that doesn’t have good “test coverage” through in-office testing