Synchronization failed

Hello!
We’re having some troubles with synchronization. Users synchronize and sometimes it fails and others it succeeds but the interviews are not sent.
The interviewer has several completed interviews but they remain on the device.
We couldn’t identify the problem in the actions log (attached) and the tablet information package.
Thank you in advance for your help
actions_log.pdf (286.9 KB)

This is normal, if there are any disruptions during the communication the synchronization will not succeed, but you can retry.

Should they? Which interviews were not sent?

You really need the technical log, not the audit log for troubleshooting this issue.

Hi Sergiy, looking at the log from the support package, beside the connection problem, we see an error code 404. But the user (TU4Miranda) is in the workspace (Tucuman). Are we missing something?

01:44:48[EnumeratorAuditLogService][Info][OpenInterview {“InterviewId”:“ad8b0fc5-1eb3-4da9-8874-cf9fc163205a”,“InterviewKey”:“06-36-47-40”,“AssignmentId”:172,“Type”:“OpenInterview”}]
01:44:57[EnumeratorAuditLogService][Info][CloseInterview {“InterviewId”:“ad8b0fc5-1eb3-4da9-8874-cf9fc163205a”,“InterviewKey”:“06-36-47-40”,“Type”:“CloseInterview”}]
01:45:04[EnumeratorAuditLogService][Info][SynchronizationStarted {“SynchronizationType”:“Online”,“Type”:“SynchronizationStarted”}]
01:45:04[EnumeratorAuditLogService][Info][SynchronizationFailed {“ExceptionMessage”:“No hay red. Compruebe su configuración de red”,“StackTrace”:" at WB.Core.SharedKernels.Enumerator.Implementation.Services.EnumeratorSynchronizationService.TryGetRestResponseOrThrowAsync[T] (System.Func1[TResult] restRequestTask) [0x0008f] in <a140d0468a264935a25c90f86bb24ce8>:0 \n at WB.Core.BoundedContexts.Interviewer.Implementation.Services.InterviewerOnlineSynchronizationProcess.RefreshUserInfo (System.Threading.CancellationToken cancellationToken) [0x00096] in <093578007a104e72ba05ecd1888dcbb9>:0 \n at WB.Core.SharedKernels.Enumerator.Implementation.Services.Synchronization.AbstractSynchronizationProcess.SynchronizeAsync (System.IProgress1[T] progress, System.Threading.CancellationToken cancellationToken) [0x0029a] in :0 ",“Type”:“SynchronizationFailed”}]
01:45:11[EnumeratorAuditLogService][Info][SynchronizationStarted {“SynchronizationType”:“Online”,“Type”:“SynchronizationStarted”}]
01:45:15[InterviewerUploadInterviews][Error][Failed to synchronize interview]WB.Core.SharedKernels.Enumerator.Implementation.Services.SynchronizationException: La dirección del sitio web del supervisor no es válida. Asegúrese de que ha introducido la dirección url correcta. —> WB.Core.Infrastructure.HttpServices.HttpClient.RestException: Not Found —> WB.Core.Infrastructure.HttpServices.HttpClient.ExtendedMessageHandlerException: Request POST http://demo-dnecv.indec.gob.ar/tucuman/api/enumerator/v3/interviews/efe26e1f-c28b-4b42-993e-9f6348f35ac9/getSyncInfoPackage failed with status code 404 (Not Found).
at WB.Core.Infrastructure.HttpServices.HttpClient.ExtendedMessageHandler.SendAsync (System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken) [0x0011e] in :0
at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered (System.Threading.Tasks.Task1[TResult] sendTask, System.Net.Http.HttpRequestMessage request, System.Threading.CancellationTokenSource cts, System.Boolean disposeCts) [0x000b3] in <3ab97adc469048029b91f4dbf8384ccb>:0 at WB.Core.Infrastructure.HttpServices.Services.RestService.ExecuteRequestAsync (System.String url, System.Net.Http.HttpMethod method, System.Object queryString, System.Net.Http.HttpContent httpContent, WB.Core.Infrastructure.HttpServices.HttpClient.RestCredentials credentials, System.Boolean forceNoCache, System.Collections.Generic.Dictionary2[TKey,TValue] customHeaders, System.Nullable1[T] userCancellationToken) [0x003cd] in <c3633effcc3a49a2ada4de700f826ce5>:0 --- End of inner exception stack trace --- at WB.Core.Infrastructure.HttpServices.Services.RestService.ExecuteRequestAsync (System.String url, System.Net.Http.HttpMethod method, System.Object queryString, System.Net.Http.HttpContent httpContent, WB.Core.Infrastructure.HttpServices.HttpClient.RestCredentials credentials, System.Boolean forceNoCache, System.Collections.Generic.Dictionary2[TKey,TValue] customHeaders, System.Nullable1[T] userCancellationToken) [0x005a2] in <c3633effcc3a49a2ada4de700f826ce5>:0 at WB.Core.Infrastructure.HttpServices.Services.RestService.PostAsync[T] (System.String url, System.IProgress1[T] transferProgress, System.Object request, WB.Core.Infrastructure.HttpServices.HttpClient.RestCredentials credentials, System.Nullable1[T] token) [0x000a6] in <c3633effcc3a49a2ada4de700f826ce5>:0 at WB.Core.SharedKernels.Enumerator.Implementation.Services.EnumeratorSynchronizationService.TryGetRestResponseOrThrowAsync[T] (System.Func1[TResult] restRequestTask) [0x00080] in :0
— End of inner exception stack trace —
at WB.Core.SharedKernels.Enumerator.Implementation.Services.EnumeratorSynchronizationService.TryGetRestResponseOrThrowAsync[T] (System.Func`1[TResult] restRequestTask) [0x0008f] in :0
at WB.Core.SharedKernels.Enumerator.Implementation.Services.Synchronization.Steps.UploadInterviews.ExecuteAsync () [0x0029d] in :0 |MoveNextRunner.InvokeMoveNext => d__8.MoveNext => NLogLogger.Error

I add one thing: the user can recieve new assigments, but cannot send Complete interviews to the server.
Please let me konw if there is any other info needed for diagnosis.
Thanks

Hello @vicarin ,

Thank you. Interviews in the status “Completed” must be sent to the server during the synchronization session.

Yes, these are the proper logs for this task. Thank you for sharing the relevant portions.

I believe you are trying to follow up on the error message contained in the technical log, specifically this one:

Barring the cases when the site has genuinely been unavailable, this should be interpreted as “This particular request was not responded to by that site”.

This can very well mean that something else has intercepted this request and responded INSTEAD of Survey Solutions. Please refer to the server installation article:

As a user you may not even know if WAF is involved. You may need to get in touch with your network admin or another security admin to conclusively find out if it may be involved.

The following will help:

  • if you can synchronize the tablet that has joined the intranet (connect to local network where the server is placed) then the problem is indeed very likely related to WAF or firewall rules.
  • if the synchronization problem is repeated all the time, not just sometimes (which would be indicative of communication disruptions).

Note that the following will not disqualify this explanation:

  1. if the user still can’t synchronize from the intranet, as the blocking rules can be implemented within the local network too, though this is rare, or
  2. if other users can synchronize from internet, since it may be triggered by a particular combination of factors, such as the length of the package.

Best, Sergiy

What versions do you have?
HQ version and Interviewer app version that experiencing this issue?

Thank you Sergiy for your answer. I have not yet been able to reach the network admin.

Vitalii,
We are using version 22.02.7 (build 32498) and the interviewer is working with version 22.06.7 (build 30204) of the application. The weird thing is that other users can sync without problems (using the same version). We have about 200 interviewers and only 3 have had this issue.

This should never be happening. The interviewers should only be using the version that is smaller or equal to the version on the server where they send the data to.

  1. Please check the version numbers that you’ve posted. Version 22.06.7 (current) has a build number 32810, unless you’ve acquired it from somewhere else and not from the public dissemination page.

  2. Please explain, how the interviewers acquired a version newer then the version on the server, thank you.

Best, Sergiy

Hi Sergiy,

  1. The issue is related to versions 22.06.6 (build 30191) and 22.06.7 (build 30204).
  2. The interviewers installled the app by themselves. So far, we have no further details on where they got it from. I guess they downloaded it from https://demo.mysurvey.solutions/ at different times.

I would like to konw if there is a way to recover the interviews from this devices and how can we do it?
Thank you so much for your assitance.
Best, Victoria

Please update your server to the latest version of Survey Solutions and try to sync again.

Thank you, Vitalii.
Since we are conducting a survey, we prefer not to update the server during fieldwork.
Can we get the interviews from the tablet information package of these users?

This is not the interviewers’ fault then. If they were not told where to get it from and had to google up where to download the app, I salute their creativity and dedication.

Instead the survey coordinator should have instructed to:
1. install the interviewer app from that particular server which is used for collection and not from any other place.
2. have any other version removed from the tablet even if it is newer that the one they are getting in ^1 above.

See also this post.

Sergiy, the tablets already had the app from our server installed. We don’t know why they did the installation again.

On the tablets that have this issue please download the Interviewer apk from your server and install it on top of the current version.

Hello @vicarin, we are committed to improving the software and need to understand what went wrong. Could you please follow up with the coordinators of the survey and get a description of:

  1. What was done exactly?
  2. Why it was done in exactly that way?

We do need specifics. Was some message from the program not clear/misunderstood? Was some instruction not possible to carry out in your environment? Was there any contradictions in the documentation?

What are your recommendations to the developers?
What are your recommendations to other users preparing for a survey?

Thank you, Sergiy

Hi Sergiy,
I took the time to consult with the field work team, but I did not get very precise information. The interviewers said that they did the installation without asking their superiors, because they had trouble logging in (we assume they mistyped the password). So they searched the web and found the demo server download link.

They were able to connect to the server and receive assignments with this version of the app. The problem arose when they tried to synchronize the completed interviews.

Obviously, this is an organizational problem. This survey is carried out at the national level and different levels of managers are involved. The interviewers report to sub-coordinators who in turn report to a provincial coordinator. These structures depend on a national coordination, which is the team which we have contact with.

We are working on improving communication channels to report technical incidents and provide responses.

Regarding the recommendations, I think that the system could alert on the first login in the app that there is a difference between app and server versions.

At another survey we worked with the blocking of the tablet so that the interviewers could not install anything, but that did not work because in cases of technical problems the circuit to recover the device generated many delays, since it is centralized at the national level.

I want to thank you for your help and follow-up on this issue.
Best, Victoria

Hello dear members of Survey Solutions team,

I have been notified that a similar situation is happening with a CAPI survey going on in the field now.

My situation is:

  • HQ rejects an interview.
  • Our interviewers open the rejected interview, review and complete the rejected interview.
  • In the Interviewer App the interview goes to the Complete tab.
  • Then our interviewers synchronize their devices.
  • Synchronization was successful, but the rejected interview just completed by the interviewer does not appear as a completed interview in HQ.

This was a test, and it seems that the status of the rejected interview is not updated, because in this test there are no more rejected interviews but in the image you can see that a rejected interview was synchronized as well.
image

Our server has the version 22.06.1 (build 32717) and all the interviewers have the same version installed in the app Interviewer. And this situation occurs with all interviewers.

What can I do to resolve this incident?

Thanks for your help.

It looks like this issue has been fixed already. There were numerous fixes to the buggy synchronization in the 22.06 release. See e.g.:

Please update and check again.