Old export files

We export survey data every 6 hours for updating our dashboard for monitoring the survey. This has led to a buildup of old exports that have started to take up a lot of space on the server.

Can I manually delete the old exports without breaking anything? I don’t find a way to do this in the Web UI.

The Survey Solutions server components should be smart enough to realize that the data file is no longer available and not to crash in this case, but please, proceed with caution as usual:

  1. stop the Survey Solutions server.
  2. move (not delete) the unwanted older files to another folder
  3. restart Survey Solutions server.
  4. revisit the data export page and create an export file manually.
  5. download and inspect the file.

Since the older jobs were completed, there will be records for them, so the system may still show the cards, albeit without the download button. For exports pre-dating the introduction of the v2-export (job-id based export of data), the cards will probably not appear at all.

If all goes well, you can purge the files you’ve moved in #2, or move them to another location for preservation, freeing up the storage space in your server.

Clearly, if the data export page doesn’t open or doesn’t allow you to generate new export files, reverse your changes.

The nature of the problem is that there is nothing in our data export protocol that would indicate WHEN the export file is no longer needed. So there is no automatic clean-up. We have discussed this several times among ourselves, but couldn’t reach an agreement on what is a reasonable expiration policy for such stored data.

PS: doing a new export through the web-UI is important in step#4. The API-based exports only care about the presence of the one file you are dealing with in the query, so will continue to work, but the data export web-page will try to render the cards for all the earlier export jobs, along with some attributes, like file size and production date/time, and this is where it may be vulnerable to missing old data files.

Cool! I was able to move the old exported files out and was able to generate a new export via web UI.

The old export cards continue to remain without the download button. Even for the newer exports (v2) the cards still exist.

I understand that different people will use the service differently. But these export files will take up significant space if they contain media files, audio recordings etc. In our case, we don’t have any media, but exported more frequently. Not having a cleanup process will start becoming a problem for these surveys.

How about letting the user (HQ) choose the expiration of old data? This could be [a] never/manual, [b] x days/months/years or [c] automatic (which could be 1 year and/or export takes up x GB space). This choice can even at the level of workspace or questionnaire.

1 Like