R package for Survey Solutions GraphQL API released

michael_wild · January 31, 2023, 8:13pm

I have just released a new R package for the most important Survey Solutions GraphQL queries & mutations. The package is purely based on the httr and jsonlite package, and does not require any other client. It also includes all the map related queries and mutations, and in particular the latest addition, the map upload. Using the suso_gql_uploadmap() function allows you to upload a zip file of maps (ESRI shape files, GeoTIFF or TPK) directly from R (or from your shiny application). Thanks to @vitalii for his support in this regard.

The package is released as a beta version, as it is still under active testing and development, including the documentation. Therefore it is only recommended for experienced users at this stage.

Further information on the different types and fields can be found here: https://demo.mysurvey.solutions/graphql/

The package can be installed directly from GitHub via devtools (or remotes): https://github.com/michael-cw/susographql

Please also note, that this package is currently completely independent from the Survey Solutions REST API package. There are plans for a future integration, but this will take a bit. Also be aware, that the return types may change in the future. Currently a list is returned containing the returned JSON response, however this may be changed to more convenient return types like data.frame or data.table.. This means if you use it in a Shiny application, you will need to change this in the future.

Feedback, feature proposals & bug reports are very welcome, either here or on GitHub directly. If you want to use parts of the code on your own, or create a fork, feel free to do so. If useful for the majority of Survey Solutions users, happy to include it in the next version.

sergiy · January 31, 2023, 9:58pm

Hello @michael_wild ,

thank you for the package. From the log below is there anything critical? or will it still work without updating these dependencies?

Best, Sergiy

michael_wild · January 31, 2023, 10:08pm

Hi @sergiy ,

no, i don’t think so. This may have to do with some read/write permission. However to me it looks like the package installed cleanly. Since it uses pretty basic functions, you should be fine even with an older version of the required packages.

Try library(susographql) if it loads cleanly, then you are good to go. Thanks also for testing. Always good to see if it works on different systems.

sergiy · January 31, 2023, 10:27pm

@michael_wild , thank you for confirming!

I see no error messages after typing: library(susographql).
Will let you know if see anything else worth bringing up.

Best, Sergiy

ashwinikalantri · February 1, 2023, 5:06am

Thank you @michael_wild.

From my initial look, I have a few suggestions:

A function for setting the credentials. Having to set these for every function can be cumbersome.

suso_gql_setcreds(
  endpoint = "https://www.url.com/graphql",
  workspace = "workspace",
  user = "apiuser",
  password = "password"
)

Instead of endpoint, why not ask for the server URL? From a user point-of-view this is easier and natural. The /graphql could be added in the function.
suso_gql_assignments only returns 100 assignments.

I look forward to seeing this package grow.

michael_wild · February 1, 2023, 11:41am

Thank you very much @ashwinikalantri for your feedback. Yes, this a function which will be introduced very soon, similar to what is already used in the Survey Solutions REST API package.

And thanks also for checking the number of assignments. I only had a demo server with only a few test assignments, but will check on this too.

michael_wild · February 1, 2023, 8:50pm

Since i came across this solution proposed by @sergiy How to know if data have been updated since given date-time? - Survey Management - Survey Solutions user community (mysurvey.solutions)

Given that this solution also demonstrates their usefulness for monitoring purposes, I have now also added certain sort parameters as well as the take and skip parameters to the interviews and maps types.

The solution proposed by Sergiy would then look like this:

suso_gql_interviews(endpoint = [your server] user = [your api user], password = [your password], sortby_updateDateUtc = “DESC”, take = 1)

Other parameters in the interviews type will follow during the next days.

michael_wild · February 2, 2023, 2:21pm

@ashwinikalantri point 3 in your comment has now been addressed, by implementing the skip and take parameters. Since the graphql API only exports a 100 items, you have to apply the solutions as suggested by @sergiy in this posting: Api GraphQl interviews bring 100 records only - #2 by sergiy

This means in the case of lets say 120 assignments, you would do the following:

suso_gql_assignments(endpoint = [your server] user = [your api user], password = [your password], take = 100)
suso_gql_assignments(endpoint = [your server] user = [your api user], password = [your password], skip = 100, take = 20)

In case you need to do this programmatically, you would first get the total number, divide it by 100, and then loop over it. And certainly the first take above (100) would not be necessary, i only did it for demonstration purposes.

ashwinikalantri · February 2, 2023, 3:59pm

@michael_wild this does the trick. But wouldn’t it be better to build this in the function? The default requirement will always be to get all assignments.

michael_wild · February 2, 2023, 4:43pm

@ashwinikalantri thank you very much for your feedback. This is definitely a good idea, However for now, this package uses the GraphQL API “native” queries & mutations without any significant data wrangling. Nevertheless with a (planned) deeper integration with the Survey Solutions API package this will most likely happen. But at this stage i can not tell you when this will be the case.

In any case you can already achieve this (i.e. getting all assignments with a single call), by using the suso_get_assignments() function from said API package. This function uses the REST API, and will return all assignments in a single data.table.

The latter package was also build with a more application oriented approach, such that it is easier to use i.e. in a shiny application.

michael_wild · February 3, 2023, 3:02pm

To facilitate operator selection, the package now also has some transformation functions, which allows you to use the large variety of operators provided by the API. Currently this is only implemented for numeric values, as outlined under the input types ComparableInt64OperationFilterInput and ComparableNullableOfInt32OperationFilterInput. The following examples demonstrate how to use them:

suso_gql_interviews(endpoint = [srv], user = [user], password = [pass], workspace = [ws], errorsCount = eq(0))
suso_gql_interviews(endpoint = [srv], user = [user], password = [pass], workspace = [ws], errorsCount = gt(0))
suso_gql_interviews(endpoint = [srv], user = [user], password = [pass], workspace = [ws], errorsCount = inbetw(c(0, 5))

If no transformer function is used, the call defaults to eq() as in the initial version.

suso_gql_interviews(endpoint = [srv], user = [user], password = [pass], workspace = [ws], errorsCount = 0)