Beta-release of the new SurveySolutionsAPIv2 (httr2 based) R package

This is a (beta - release1) new edition of the Survey Solutions R API package, based on the recently (as stable) released httr2 package. Most of the syntax is the same as in the existing httr based SurveySolutionsAPI package, and therefore it should be easy to integrate it in your existing workflow, if you have used the previous edition so far.

But besides a few syntax changes as well as several other updates like more meaningful error messages using the cli and the rlang packages there are also quite a bunch of new features. Therefore i decided to release this package as a completely new one, such that users who do not want to switch to the new package or make use of the new features, can still work with the old package as usual. Nevertheless a transition to the new package is recommended.

Additionally the package also seeks a deeper integration with the updated susographql package, which now also is purely based on the httr2 package.

New features in this package specific to one or several functions are:

  • suso_get_stats_interview, suso_get_assignments, suso_createUSER, suso_createASS, suso_getSV and suso_getINT with potentially long running queries now use httr2’s parallel request feature, which reduces processing time substantially. For example 10k assignments (suso_createASS) can now be created in about 3 minutes, a list of all interviewers (suso_getINT) in a workspace for 18K interviewers and 5k supervisors now only takes around 1.5 minutes. In particular in large scale surveys and censuses, this feature may facilitate working with the API considerably.
  • suso_set_key: workspace can now be set as an environment variable, like it was already the case for server, user and password.
  • suso_export: now has the option to merge all the data export files into a single data table and also add survey weights, for analysis-ready data sets.
  • suso_export: of spatial (Survey Solutions Geography Questions) data (i.e. polygons, point locations, lines) can be processed into sf (simple feature) objects directly, by setting process_mapquestions = TRUE. For example a type polygons question, collected manually or automatic, will result in an sf polygon, which can be stored directly as a shape file with st_write.
  • suso_export: now also processes available value labels and applies them to categorical variables, as well as variable labels. In case you have one or several translations for your questionnaire, these can be used as well, such that the labels applied are in the required language. The (ExportClass) specific methods, subsequently make use of these additional attributes, resulting in publication-ready tables and graphs.
  • suso_export_paradata: now uses milliseconds for all time based calculations, and also adds comprehensive questionnaire information to the data, like i.e. question type.
  • suso_export_paradata: now also identifies GPS variable in your questionnaire, and uses it, to add GPS location data to the paradata, which facilitates spatial analysis of paradata considerably.
  • suso_mapupload, suso_mapassign, suso_maps, suso_mapreport and suso_deletemap now also enable complete map management with a single package, and the same syntax/outputs2.
  • suso_assignWorkspace: now also supports workspace assignment of multiple supervisors and interviewers in one go. In addition it has the argument keep_old_workspace . If TRUE , then the existing workspaces the user is already assigned to are automatically added. With this new feature, all users from one workspace can be assigned to a new workspace with a single function call (requires admin credentials!).

New general features in this package are:

  • New classes and methods, i.e.: AssignmentClass, ExportClass, UserClass, and methods like summaryTable.exportClass (DT based) or boxplot_summary.exportClass (ggplot2 based).
  • Http error messages are translated into R errors, and use the error codes (and messages) as provided by the Survey Solutions API response), which makes debugging easier.
  • Several options, which allows the user to customize processing to their environment, like: suso.maxpar.req for the maximum number of parallel requests, suso.para.break for customizing breaks when calculating paradata response times, suso.para.tz for setting the time zone or suso.para.maxcore for the number of cores used in parallel processing.
  • User notifications including progress bars when used in shiny applications, for potentially long running processes.

This is just a very rough overview of the most prominent features, for details please see the individual functions’ documentation. An extensive documentation is still in preparation and will follow over the next weeks, as well as some more class specific methods. For now, until the new documentation (i.e vignettes, gihub.io website) is released please use the original SurveySolutionsAPI package documentation, which is still valid for most of the functions.

For more details please see the corresponding GitHub repository: GitHub - michael-cw/SurveySolutionsAPIv2: A comprehensive set of R functions to access the Survey Solutions REST/GraphQL API (httr2 based version).

Since this is a beta release, feedback, bug reports and feature requests are very welcome. You can either use the standard GitHub approach by filing a bug report/feature request here or respond directly to this post.

  1. The package is still under testing and also development of some features, therefore it is only recommended for experienced users. :leftwards_arrow_with_hook:
  2. All functions use the susographql functions, but are adjusted to the syntax of this package. :leftwards_arrow_with_hook:

As a small teaser for the new package, I have put below as short script for a shiny application with the following components:

  1. A modal dialogue for the authentication
  2. A selectinput to select from the list of questionnaires.
  3. A summary DT package datatable as produced by summaryTable S3 method.

That’s how the app looks like after authentication:

And this is the code.

# Load necessary libraries
library(shiny)
library(DT)
library(SurveySolutionsAPIv2)

# Define UI
ui <- fluidPage(titlePanel("Survey Solutions Export Summary Example"),
    sidebarLayout(sidebarPanel(actionButton("setcredentials", "Set Credentials"),
        selectInput("questionnaire", "Select Questionnaire:", choices = c(`Load Options` = 0))),
        mainPanel(DTOutput("my_datatable"))))
# Define Server
server <- function(input, output, session) {

    onStop(
      function() suso_clear_keys()
    )
    # simple credentials modal !! TO DO LATER
    observeEvent(input$setcredentials, {
        AUTH(NULL)
        showModal(modalDialog(title = "Survey Solutions Credentials", textInput("srv",
            "Server Name"), textInput("user", "User Name"), passwordInput("pass",
            "Password"), textInput("ws", "Workspace"), actionButton("auth",
            "Connect to Server")))

    })

    # Connect
    AUTH <- reactiveVal(NULL)
    observeEvent(input$auth, {
        removeModal()
        # req(suso_get_default_key('susoPass')!='')
        suso_set_key(input$srv, input$user, input$pass, input$ws)
        authresult <- suso_PwCheck()
        if (authresult == 200)
            AUTH(authresult)
    })

    # load the questionnaires on the server
    questlist <- reactive({
        req(AUTH())
        suso_getQuestDetails()
    })
    observe({
        req(AUTH())
        q <- req(questlist())
        questselect <- sprintf("%s %s", q$Title, q$Version)
        questselect <- stats::setNames(1:length(questselect), questselect)

        updateSelectInput(session = session, "questionnaire", "Select Questionnaire:",
            choices = questselect)

    })

    # Render the DT datatable
    output$my_datatable <- DT::renderDT({
        req(AUTH())
        q <- req(questlist())
        sel <- req(input$questionnaire)
        sel <- as.numeric(sel)
        req(sel > 0)
        exp <- suso_export(questID = q$QuestionnaireId[sel], version = q$Version[sel],
            workStatus = "All", process_mapquestions = T, combineFiles = T,
            addTranslation = T, reloadTimeDiff = 3, translationLanguage = "italian")

        suppressWarnings(tab <- summaryTable(exp))
        return(tab)
    })

}

shinyApp(ui = ui, server = server)

I hope this short example gives you an idea of how to work with this package more examples will follow and also be included in the extended documentation

ATTENTION: please be aware that this authentication mode is not safe, and should only be used in interactive mode on your own device.

And here is another small teaser. Similar to the previous one, but this time using summaryTable for the userClass, the app has the following components

  1. A modal dialogue for the authentication
  2. A selectizeInput to select from the list of workspaces.
  3. A radioButton input to select the user role.
  4. A summary DT package datatable with all users in the specific role, core id data, plus a column for the number of assignments, and the number of completed interviews.

The apps UI looks prettry similar to the previous one:

and this is the code for it:

# Load necessary libraries
library(shiny)
library(DT)
library(SurveySolutionsAPIv2)

# Define UI
ui <- fluidPage(titlePanel("Survey Solutions User Summary Example"),
    sidebarLayout(sidebarPanel(actionButton("setcredentials", "Set Credentials"),
        selectizeInput("workspace", "Select workspace:", choices = c(`Load Options` = 0)),
        radioButtons("role", "Which user  role?", c("Supervisor", "Interviewer"))),
        mainPanel(DTOutput("my_datatable"))))
# Define Server
server <- function(input, output, session) {
    # clear credentials when the app stops
    onStop(
      function() suso_clear_keys()
    )
    # simple credentials modal !! only for local environments
    observeEvent(input$setcredentials, {
        AUTH(NULL)
        showModal(modalDialog(title = "Survey Solutions Credentials", textInput("srv",
            "Server Name"), textInput("user", "User Name"), passwordInput("pass",
            "Password"), textInput("ws", "Workspace"), actionButton("auth",
            "Connect to Server")))

    })

    # Connect
    AUTH <- reactiveVal(NULL)
    observeEvent(input$auth, {
        removeModal()
        # req(suso_get_default_key('susoPass')!='')
        suso_set_key(input$srv, input$user, input$pass, input$ws)
        authresult <- suso_PwCheck()
        if (authresult == 200)
            AUTH(authresult)
    })

    # load the workspace available with current auth
    ws <- reactive({
      req(AUTH())
      suso_getWorkspace()
    })
    # update selectize input workspace
    observe({
        req(AUTH())
        ws1 <- req(ws())
        updateSelectizeInput(session = session, "workspace", "Select workspace:",
            choices = ws1$Name,
            options = list(
              placeholder = "Select workspace",
              onInitialize = I('function() { this.setValue(""); }')
            )
            )

    })

    # Retrieve select users
    USR<-reactiveVal(NULL)
    realist<-reactive({
      USR(NULL)
      list(input$workspace, input$role)
    })
    observeEvent(realist(), {
      req(AUTH())
      ws1 <- req(ws())
      req(input$workspace!="")

      usr<-switch(input$role,
             Supervisor = suso_getSV(workspace = input$workspace),
             Interviewer = suso_getINT(workspace = input$workspace)
             )
      USR(usr)

    }, ignoreInit = T)

    # Render the DT datatable
    output$my_datatable <- DT::renderDT({
        req(AUTH())
        usr <- req(USR())
        suppressWarnings(tab <- summaryTable(usr))
        return(tab)
    }, server = T)

}

shinyApp(ui = ui, server = server)

A useful colateral of the switch to the cli package is also, that progress bars are displayed in shiny apps too and not only in interactive mode. To customize the content of basic shiny notifications, the package also contains some options for these notifications, i.e. in case you want to use a translation. And if you don’t want to use these widgets at all, you can just deactivate them globally.

And now for the final teaser! This time the app will use the automatic processing of Survey Solutions Geography Questions as well as GPS questions, during export, when process_mapquestions=TRUE in the suso_export function. The app has the following components:

  1. A modal dialogue for the authentication
  2. A selectizeInput to select from the list of questionnaires.
  3. A selectizeInput input to select the roster levels (see details in the suso_export documentation).
  4. A selectizeInput input to select the gps or map question.
  5. A Leaflet map showing the results of the selected question, already processed into ansf (Simple Features) polygons, or an sf points data.frame ready to be displayed on the map.

This is how the map looks like for the polygons:

And this is for the points data:

And finally here is the app code:

library(shiny)
library(leaflet)
library(sf)
library(SurveySolutionsAPIv2)

selectizeOptions<-list(
  placeholder = "Non Loaded",
  onInitialize = I('function() { this.setValue(""); }')
)

# Define UI
ui <- fluidPage(titlePanel("Survey Solutions Map Export Example"),
                sidebarLayout(sidebarPanel(actionButton("setcredentials", "Set Credentials"),
                                           selectizeInput("questionnaire", "Select Questionnaire:", choices = c(`Load Options` = 0),
                                                          options = selectizeOptions),
                                           selectizeInput("roster", "Select Roster Level:", choices = c(`Load Options` = 0),
                                                          options = selectizeOptions),
                                           selectizeInput("int", "Select Map Question:", choices = c(`Load Options` = 0),
                                                          options = selectizeOptions)),
                              mainPanel(leafletOutput("map", height = "80vh"))))



server <- function(input, output, session) {

  onStop(
    function() suso_clear_keys()
  )
  # simple credentials modal !! TO DO LATER
  observeEvent(input$setcredentials, {
    AUTH(NULL)
    showModal(modalDialog(title = "Survey Solutions Credentials", textInput("srv", "Server Name"),
                          textInput("user", "User Name"), passwordInput("pass","Password"), textInput("ws", "Workspace"),
                          actionButton("auth","Connect to Server")))

  })

  # Connect
  AUTH <- reactiveVal(NULL)
  observeEvent(input$auth, {
    removeModal()
    # req(suso_get_default_key('susoPass')!='')
    suso_set_key(input$srv, input$user, input$pass, input$ws)
    authresult <- suso_PwCheck()
    if (authresult == 200)
      AUTH(authresult)
  })

  # load the questionnaires on the server
  questlist <- reactive({
    req(AUTH())
    suso_getQuestDetails()
  })

  # 1. update questionnaire selection
  observe({
    req(AUTH())
    q <- req(questlist())
    questselect <- sprintf("%s %s", q$Title, q$Version)
    questselect <- stats::setNames(1:length(questselect), questselect)

    updateSelectizeInput(session = session, "questionnaire", "Select Questionnaire:",
                         choices = questselect,
                         options = list(
                           placeholder = "Select Questionnaire",
                           onInitialize = I('function() { this.setValue(""); }')
                         )
    )
  })

  # 2. Get Export file
  expfile <- reactive({
    req(AUTH())
    q <- req(questlist())
    req(input$questionnaire!="")
    sel <- req(input$questionnaire)
    sel <- as.numeric(sel)
    req(sel > 0)
    exp <- suso_export(questID = q$QuestionnaireId[sel], version = q$Version[sel], reloadTimeDiff = 0,
                       workStatus = "All", process_mapquestions = T, combineFiles = F)

    return(exp)
  })

  # 4. Update Roster Level Selection
  observe({
    xport<-req(expfile())
    rlevels<-unlist(sapply(names(xport), function(name) {
      if (length(xport[[name]]) > 0) {
        return(name)
      } else {return(NULL)}
    }, USE.NAMES = F, simplify = F))

    rlevels <- stats::setNames(rlevels, rlevels)
    updateSelectizeInput(session = session, "roster", "Select Roster Level:",
                         choices = rlevels,
                         options = list(
                           placeholder = "Select Roster Level",
                           onInitialize = I('function() { this.setValue(""); }')
                         )
    )
  })

  # 4. Update Map Selection
  observe({
    xport<-req(expfile())
    req(input$roster!="")
    intid<-names(xport[[input$roster]])
    intsel <- stats::setNames(intid, intid)
    # get sf elements
    intsel<-intsel[grepl("^sf_", intsel)]
    updateSelectizeInput(session = session, "int", "Select Map Question:",
                         choices = intsel,
                         options = list(
                           placeholder = "Select Map Question",
                           onInitialize = I('function() { this.setValue(""); }')
                         )
    )
  })

  # 5. Leflet map
  output$map <- renderLeaflet({
    leaflet() %>%
      addProviderTiles(providers$Esri.WorldImagery) %>%
      leaflet::setView(lat = 38.899063, lng = -77.042386, zoom = 12)
  })

  # 6. Create leaflet proxy
  lfp <- leafletProxy("map", session = session)
  # 7. Update map
  observe({
    xport<-req(expfile())
    req(input$int!="")
    sf_poly<- xport[[input$roster]][input$int][[1]]
    req(sf_poly)
    # to latlong if not
    if (!sf::st_is_longlat(sf_poly)) {
      sf_poly <- sf_poly %>%
        sf::st_transform(4326)
    }

    bounds <- sf::st_bbox(sf_poly) %>% as.character()

    # check geometry type
    if(sf::st_geometry_type(sf_poly, by_geometry = F)=="POLYGON"){
      lfp %>%
        clearShapes() %>%
        fitBounds(bounds[1], bounds[2], bounds[3], bounds[4]) %>%
        addPolygons(
          data = sf_poly,
          # fillColor = ~filcol(sf_poly[[filcolvar]]),
          # color = ~filcol(sf_poly[[filcolvar]]),
          fillOpacity = 0.8
        )
    } else if(sf::st_geometry_type(sf_poly, by_geometry = F)=="MULTIPOINT"){
      # cast to point, as multi point is not supported
      sf_poly<-sf_poly |> st_cast("POINT", warn = F)
      lfp %>%
        clearMarkers() %>%
        fitBounds(bounds[1], bounds[2], bounds[3], bounds[4]) %>%
        addMarkers(
          data = sf_poly
        )

    }
  })
}

# Run the application
shinyApp(ui = ui, server = server)

That’s it for now. More will follow in the official documentation. And in case you have any ideas on what other convenience functions could be useful, then please let me know.