How to know if data have been updated since given date-time?

[arthurshaw2002]

Let’s suppose that one only wanted to export data if any data have changed (e.g., interviews inserted, inteviews updated, etc.). How would could one know that data have been updated since the last export? Put another way: how can one know the last date-time that data changed?

Here’s my quick, crude answer is:

1. Request interview metadata for all interviews via the GraphQL API
2. Identify the most recent date-time in interview metadata
3. Get the date-time from last export job (for a given template) or from files exported (e.g., date in export__readme.txt)
4. Compare that date-time against the date time of the last export job

For step 1, I’m envisioning something like this:

{
  interviews (
    workspace: "some_workspace"
    ) 
  {
    nodes {
      id
      updateDateUtc
    } 
  }
}

For simplicity, assume there’s only one questionnaire template and only as many interviews on the server as would fit in a single request.

Does anyone see a better solution?

If there’s not a better solution, would it make sense to provide users with some mechanism to get the last date-time the DB tables associated with a particular questionnaire template were updated (e.g., new interviews received, any interviews modified in content/status, etc)?

[sergiy]

@arthurshaw2002

how can one know the last date-time that data changed?

That translates to : “The date for update for the most recently updated interview is after the time export was requested (for that specific questionnaire).”

Best, Sergiy

[arthurshaw2002]

Wonderful–and wonderfully elegant !

To refresh my memory, what events will make updateDateUtc change? Would it be any action that changes that data or metadata of the interview such as … ?

• Interview posted to server
• Supervisor/Headquarters approves/unapproves/rejects interview
• Supervisor/Headquarters comments interview or its questions

Separate but related question: is there a date anywhere in the system that identifies when an interview is sent to the server (e.g., interviewer sends interview for the first time, interviewer sends rejected interview back to the server)? Would this only come from the action logs? Here, I’m looking for a way to identify dates where the interviewer could have changed data in the interview.

[vitalii]

Export API supports starting date.
You are trying to get required interviews by comparing last change with some previously saved date.
If you have this date just supply it to export request.

[peter_brueck]

@arthurshaw2002

Separate but related question: is there a date anywhere in the system that identifies when an interview is sent to the server (e.g., interviewer sends interview for the first time, interviewer sends rejected interview back to the server)?

I think this info is (for CAPI interviews) not accessible, correct @vitalii @sergiy?

See related discussion here.

[sergiy]

@arthurshaw2002

… date anywhere in the system that identifies when an interview is sent to the server…

See event ReceivedBySupervisor in Survey Solutions paradata.

[arthurshaw2002]

Thanks!

Here I veer into a feature request territory… Would it make sense to make this event more visible in the system, outside of paradata (e.g., add as an attribute of interviews in GraphQL, add event to interview__actions, etc.)?

For my own part, I think this event date-time helps with a few tasks:

• Know when teams have last sent interviews.
• Know whether interview needs to be reviewed.

Using the paradata is a viable solution, but may not be the most convenient.

[sergiy]

@arthurshaw2002

this event date-time helps with a few tasks:

• Know when teams have last sent interviews.

Not exactly. It reveals when this particular interview (each one interview) was sent by the interviewer. To determine when was the last communication - you would need to go across all interviews and process them to pick the latest. Instead, inspect the last synchronization timestamp property in the interviewer profile. This is a more direct way of getting there. It will even tell you whether the last sync was successful or not.

@arthurshaw2002

Know whether interview needs to be reviewed.

Interviews in the status ‘completed’ need to be reviewed.

[arthurshaw2002]

@sergiy

Instead, inspect the last synchronization timestamp property in the interviewer profile. This is a more direct way of getting there. It will even tell you whether the last sync was successful or not.

Is there a way to access this information via the API or exported data?

[michael_wild]

Hi @arthurshaw2002,

I think you could use the Rest API action-log for this, which gives you the following data:

452 2023-02-03T13:37:05.710081 Interview 87-57-29-96 created from assignment 1
453 2023-02-03T13:36:55.591392 Sync completed: 1 interview(s) rejected
454 2023-02-03T13:36:38.985343 Sync started (Online)
455 2023-02-03T13:35:38.041052 Sync completed: 5 interview(s) uploaded
456 2023-02-03T13:34:56.08893 Sync started (Online)
457 2023-02-03T13:34:51.556203 Interview 45-71-42-89 closed
458 2023-02-03T13:34:44.611288 Interview 45-71-42-89 opened
459 2023-02-03T13:34:43.367227 Interview 45-71-42-89 created from assignment 1

[arthurshaw2002]

Thanks, Michael. I’ve been doing exactly that. I was just looking for an alternative–ideally one that gives a “posted-to-server” date per interview.