Access API using R

API
#1

Dear all:

Inspired by a previous post on accessing the SuSo API, I published a brief tutorial on how to access it using R.

The tutorial is available here:

http://rpubs.com/mrahija/SuSoApiAccess

Thanks, Michael

7 Likes
#2

Michael, a belated, but no less sincere, thanks for putting this R code walk-through out there for people to use and access.

Since your code got me working with the API via R, I wanted to contribute to the conversation that you started, by pointing out a few omissions from your script:

  • Checking the status of download file creation after asking the server to start creating a file
  • Reacting to the file creation status (e.g., if the file is not ready, need to wait to download; if the file is ready, download it)
  • Capturing errors that may happen at any stage in the process

Inspired by your example, some of my colleagues and I have been writing R scripts to fetch data from Survey Solutions as well. The most mature scripts are a family of functions that Lena developed and posted on GitHub. They deal with some of the issues above as well as some obscure corner cases (e.g., when tab files are created and their creation status reverts back to “NotStarted” because the whole process is so fast).

We may have other scripts hitting the public domain in the coming months too. When they’re ready for public consumption, we’ll share. But when they are released, please consider this important caveat: these scripts should not be construed as Survey Solutions products, but rather than as code shared by individual authors on an as-is basis and maintained as time allows. That means that, while we may announce scripts on this forum, we’ll host conversations and issue reports elsewhere.

Retrieving Interviews through API
Export data from multiple questionnaires
#3

Hey Arthur,

Great! I’m excited to hear someone is using the code.

For R users, a package designed to interact with SuSo (i.e. API), make basic tabulations from standard output files (paradata, comments, etc.), and some kind of complementary functions (e.g. automated emails for CAWI, markdown files with basic reports, etc.) would be really useful. We’re working on scripts for automated emails for CAWI surveys right now. I’d be happy to share these functions and older ones with Lena. I even added Roxygen headings in hopes of one day having the time to package them all up.

Best, Michael

#4

I was looking for a way to automate data import in R and then I stumbled upon this!! Now I never have to generate, download, unzip the data then import it in R.

Thanks for the scripts!!

Ashwini

#5

Will it be possible to turn these into a R package? Should be very useful.

#6

It always makes me happy to hear that someone else is using the functions. I am working to expand the functions past the export of data but it’s a slow process. Feel free to submit issues on the github if you hit any bugs.

We agree that turning all these into an R package would be very useful and I think it’s probably not so hard to turn it all into a package. @arthurshaw2002 and I definitely have the ambition of doing that. We have to just set aside time to do it.

1 Like
#7

Waiting, impatiently… :smiley:

#8

I am facing issues while using the scripts. The data that is downloaded is incomplete.

The actual interviews are 18489 with a roster with 70108 records. But when I download using the script, the number is 17917 and 67804. This number changes every time I run the script to download the data (17813 and 67429 on try no 2).

#9

I was able to trace the issue down to the unzipping process. When I manually unzip the files all the data is present. But when the script unzips the files, I get this warning in R and there is random loss of data.

Warning messages:
1: In unzip(zip_name, exdir = zip_path) :
  error -1 in extracting from zip file
2: In unzip(zip_name, exdir = zip_path) :
  error -1 in extracting from zip file
3: In unzip(zip_name, exdir = zip_path) :
  error -1 in extracting from zip file
4: In unzip(zip_name, exdir = zip_path) :
  error -1 in extracting from zip file
5: In unzip(zip_name, exdir = zip_path) :
  error -1 in extracting from zip file
6: In unzip(zip_name, exdir = zip_path) :
  error -1 in extracting from zip file
7: In unzip(zip_name, exdir = zip_path) :
  error -1 in extracting from zip file
8: In unzip(zip_name, exdir = zip_path) :
  error -1 in extracting from zip file
9: In unzip(zip_name, exdir = zip_path) :
  error -1 in extracting from zip file
10: In unzip(zip_name, exdir = zip_path) :
  error -1 in extracting from zip file
#10

That’s strange that utils::unzip is throwing errors.

If that’s the case, set unzip = FALSE.

Also, please post issues with the API scripts to the GitHub repo. That way, we hit two goals simultaneously: first, keeping this forum on SuSo only; second, ensuring that issues like this don’t get lost in improvement and maintenance of our the scripts Lena and I put together.

Thanks so much for flagging this issue.

Stata package 'sursol' for Survey Solutions
#11

Yeah, I have had this problem as well but I have not managed to google my way to a satisfactory solution. I think it may stem from it trying to unzip a large zip file. You can read data files in a zip file R without unzipping it so that may be a good workaround so you don’t have to leave the R environment. But like Arthur suggested, please put it as an issue on the github repo so we do not forget about it.

#12

check that your software supports the zip64 compression/decompression: