Synchronization issues between interviewer/supervisor tablets

Hello,

After several days of successful data collection and interview assignment/synchronization, our project has been unable to synchronize between the supervisor and interviewer/enumerator tablets. Each time we try to synchronize we are getting the message “Synchronization completed with errors”, but nothing moves in either direction. This issue is occurring for all 5 tablets we are using.

Here is our information:

  • Full server name: https://50539.lwssd.mathematica-intl.org

  • User name (login) of the user(s) experiencing the problem: Vmiti, vphiri, mbernard, hellenk, amwanza

  • Date and time when the problem was last experienced: Feb 28, 2022 at 19:00 GMT. Problem first experienced on Friday Feb 25th, 2022

  • Version of the Android operating system on the tablet of the affected interviewer: Android 11

  • Survey Solutions Interviewer App version on the tablet of the affected interviewer: 21.09.7 (build 29901)

I am including a snippet of the log from user vmiti from 2/28, though let us know if any other information would be helpful to provide:

01:31:36[EnumeratorAuditLogService][Info][OpenApplication {“Type”:“OpenApplication”}]
01:31:36[InterviewerAppStart][Info][Application started. Version: 21.09.7 (build 29901)]
01:31:58[EnumeratorAuditLogService][Info][Login {“UserName”:“vmiti”,“Type”:“Login”}]
01:34:28[NearbyConnection][Info][[STOP ALL]]
01:34:28[EnumeratorAuditLogService][Info][Logout {“UserName”:“vmiti”,“Type”:“Logout”}]
01:34:42[EnumeratorAuditLogService][Info][Login {“UserName”:“vmiti”,“Type”:“Login”}]
01:35:48[NearbyConnection][Info][[STOP ALL]]
01:35:48[NearbyConnection][Info][[START DISCOVERY] of 3da8ee06fe0c4f5ea0a5d9eebd0a2239primary]
01:35:48[NearbyConnection][Info][[START DISCOVERY] Result - Status: StatusOk Code: 0 Message: ]
01:35:53[NearbyConnection][Info][[STOP DISCOVERY]]
01:35:55[NearbyConnection][Info][[ACCEPT CONNECTION] to endpoint: W0SF. Status: Status: StatusOk Code: 0 Message: ]
01:35:55[NearbyConnectionClient][Error][(W0SF)]Android.Gms.Common.Apis.ApiException: 8009: STATUS_OUT_OF_ORDER_API_CALL
at WB.UI.Shared.Enumerator.OfflineSync.Services.Implementation.NearbyConnectionClient.AcceptConnectionAsync (System.String endpoint) [0x000bb] in <61505fa62d694de38a410a2925184ee8>:0
— End of managed Android.Gms.Common.Apis.ApiException stack trace —
com.google.android.gms.common.api.ApiException: 8009: STATUS_OUT_OF_ORDER_API_CALL
at com.google.android.gms.internal.nearby.zzby.setFailedResult(Unknown Source:4)
at com.google.android.gms.internal.nearby.zzba.zzc(Unknown Source:18)
at com.google.android.gms.internal.nearby.zzea.dispatchTransaction(Unknown Source:7)
at com.google.android.gms.internal.nearby.zzb.onTransact(Unknown Source:22)
at android.os.Binder.execTransactInternal(Binder.java:1159)
at android.os.Binder.execTransact(Binder.java:1123)
|MoveNextRunner.InvokeMoveNext => d__29.MoveNext => NLogLogger.Error
01:35:55[NearbyConnection][Info][[ACCEPT CONNECTION] to endpoint: W0SF. Status: Status: StatusOutOfOrderApiCall Code: 8009 Message: 8009: STATUS_OUT_OF_ORDER_API_CALL]
01:35:55[NearbyConnection][Info][[STOP DISCOVERY]]
01:35:55[NearbyConnection][Info][[STOP DISCOVERY]]
01:35:56[EnumeratorAuditLogService][Info][SynchronizationStarted {“SynchronizationType”:“Offline”,“Type”:“SynchronizationStarted”}]
01:36:03[NearbyCommunicator][Error][Got exception. System.Exception: Exception of type ‘System.Exception’ was thrown.
at WB.Core.SharedKernels.Enumerator.OfflineSync.Services.Implementation.NearbyCommunicator.SendAsync[TRequest,TResponse] (WB.Core.SharedKernels.Enumerator.OfflineSync.Services.INearbyConnection connection, System.String endpoint, TRequest message, System.IProgress1[T] progress, System.Threading.CancellationToken cancellationToken) [0x00311] in <58aad36136cc47d0892eecec2cd4dcfa>:0 ] 01:36:04[InterviewerUploadInterviews][Error][Failed to synchronize interview]WB.Core.SharedKernels.Enumerator.OfflineSync.Services.CommunicationException: Error while sending <UploadInterviewRequest, OkResponse> to endpoint 'W0SF' ---> System.Exception: Exception of type 'System.Exception' was thrown. at WB.Core.SharedKernels.Enumerator.OfflineSync.Services.Implementation.NearbyCommunicator.SendAsync[TRequest,TResponse] (WB.Core.SharedKernels.Enumerator.OfflineSync.Services.INearbyConnection connection, System.String endpoint, TRequest message, System.IProgress1[T] progress, System.Threading.CancellationToken cancellationToken) [0x00311] in <58aad36136cc47d0892eecec2cd4dcfa>:0
— End of inner exception stack trace —
at WB.Core.SharedKernels.Enumerator.OfflineSync.Services.Implementation.NearbyCommunicator.SendAsync[TRequest,TResponse] (WB.Core.SharedKernels.Enumerator.OfflineSync.Services.INearbyConnection connection, System.String endpoint, TRequest message, System.IProgress1[T] progress, System.Threading.CancellationToken cancellationToken) [0x003c3] in <58aad36136cc47d0892eecec2cd4dcfa>:0 at WB.Core.SharedKernels.Enumerator.Implementation.Services.Synchronization.Steps.UploadInterviews.ExecuteAsync () [0x00566] in <58aad36136cc47d0892eecec2cd4dcfa>:0 error: WB.Core.SharedKernels.DataCollection.Exceptions.InterviewException: Questionnaire '3b4cc40bb64548d69220b0c8a62d3e38$7' was not found. InterviewId 908e5f35-0322-4a87-b372-8ac0ac633727 at WB.Core.SharedKernels.DataCollection.Implementation.Aggregates.Interview.GetQuestionnaireOrThrow (System.String language) [0x0005a] in <7c575880e61e47ffb4cdbe58f90d5425>:0 at WB.Core.SharedKernels.DataCollection.Implementation.Aggregates.Interview.GetQuestionnaireOrThrow () [0x00007] in <7c575880e61e47ffb4cdbe58f90d5425>:0 at WB.Core.SharedKernels.DataCollection.Implementation.Aggregates.Interview.SynchronizeInterviewEvents (WB.Core.SharedKernels.DataCollection.Commands.Interview.SynchronizeInterviewEventsCommand command) [0x00036] in <7c575880e61e47ffb4cdbe58f90d5425>:0 at WB.Core.Infrastructure.CommandBus.CommandRegistry+AggregateSetup1+<>c__DisplayClass0_01[TAggregate,TCommand].<InitializesWith>b__0 (TCommand command, TAggregate aggregate) [0x0000c] in <00839cd14742427b8ae84a7d9082ef48>:0 at WB.Core.Infrastructure.CommandBus.CommandRegistry+<>c__DisplayClass5_02[TCommand,TAggregate].b__1 (WB.Core.Infrastructure.CommandBus.ICommand command, WB.Core.Infrastructure.Aggregates.IAggregateRoot aggregate) [0x00000] in <00839cd14742427b8ae84a7d9082ef48>:0
at WB.Core.Infrastructure.CommandBus.Implementation.CommandExecutor.ExecuteEventSourcedCommand (WB.Core.Infrastructure.CommandBus.ICommand command, System.String origin, System.Type aggregateType, System.Guid aggregateId, System.Collections.Generic.IEnumerable1[T] validators, System.Collections.Generic.IEnumerable1[T] preProcessors, System.Collections.Generic.IEnumerable1[T] postProcessors, System.Action2[T1,T2] commandHandler, System.Threading.CancellationToken cancellationToken) [0x00115] in <00839cd14742427b8ae84a7d9082ef48>:0
at WB.Core.Infrastructure.CommandBus.Implementation.CommandExecutor.ExecuteCommand (WB.Core.Infrastructure.CommandBus.ICommand command, System.String origin, System.Threading.CancellationToken cancellationToken, System.Guid aggregateId) [0x00056] in <00839cd14742427b8ae84a7d9082ef48>:0
at WB.Core.Infrastructure.CommandBus.Implementation.CommandService+<>c__DisplayClass16_0.b__1 (WB.Core.Infrastructure.CommandBus.Implementation.ICommandExecutor ce) [0x00000] in <00839cd14742427b8ae84a7d9082ef48>:0
at WB.Core.Infrastructure.Extensions+<>c__DisplayClass0_01[T].<ExecuteInScope>b__0 (WB.Core.GenericSubdomains.Portable.ServiceLocation.IServiceLocator scope) [0x0000c] in <00839cd14742427b8ae84a7d9082ef48>:0 at WB.Core.SharedKernels.Enumerator.Implementation.Services.ExecuteInWorkspaceService.Execute (System.Action1[T] action, System.String workspace) [0x0000f] in <58aad36136cc47d0892eecec2cd4dcfa>:0
at WB.Core.Infrastructure.Extensions.ExecuteInScope[T] (WB.Core.GenericSubdomains.Portable.ServiceLocation.IServiceLocator sl, System.Action1[T] action) [0x00013] in <00839cd14742427b8ae84a7d9082ef48>:0 at WB.Core.Infrastructure.CommandBus.Implementation.CommandService+<>c__DisplayClass16_0.<ExecuteImpl>b__0 () [0x00000] in <00839cd14742427b8ae84a7d9082ef48>:0 at WB.Core.GenericSubdomains.Portable.NamedLocker.RunWithLock (System.String name, System.Action body) [0x00036] in <e7d93a8945d54e2c85f135ae5ea2b524>:0 at WB.Core.Infrastructure.Implementation.Aggregates.AggregateLock.RunWithLock (System.String aggregateGuid, System.Action run) [0x00000] in <00839cd14742427b8ae84a7d9082ef48>:0 at WB.Core.Infrastructure.CommandBus.Implementation.CommandService.ExecuteImpl (WB.Core.Infrastructure.CommandBus.ICommand command, System.String origin, System.Threading.CancellationToken cancellationToken) [0x000a1] in <00839cd14742427b8ae84a7d9082ef48>:0 at WB.Core.Infrastructure.CommandBus.Implementation.SequentialCommandService.ExecuteImpl (WB.Core.Infrastructure.CommandBus.ICommand command, System.String origin, System.Threading.CancellationToken cancellationToken) [0x00048] in <00839cd14742427b8ae84a7d9082ef48>:0 at WB.Core.Infrastructure.CommandBus.Implementation.CommandService.Execute (WB.Core.Infrastructure.CommandBus.ICommand command, System.String origin) [0x00008] in <00839cd14742427b8ae84a7d9082ef48>:0 at WB.Core.BoundedContexts.Supervisor.Services.Implementation.OfflineSyncHandlers.SupervisorInterviewsHandler.UploadInterview (WB.Core.SharedKernels.Enumerator.OfflineSync.Messages.UploadInterviewRequest request) [0x0021d] in <a08d0bbd2e9b4e7f8b87117885ad9540>:0 at WB.Core.SharedKernels.Enumerator.OfflineSync.Services.Implementation.NearbyConnectionsRequestHandler+<>c__DisplayClass4_02[TReq,TResp].b__0 (WB.Core.SharedKernels.Enumerator.OfflineSync.Messages.ICommunicationMessage r) [0x000e4] in <58aad36136cc47d0892eecec2cd4dcfa>:0
at WB.Core.SharedKernels.Enumerator.OfflineSync.Services.Implementation.NearbyCommunicator.HandlePayloadContent (WB.Core.SharedKernels.Enumerator.OfflineSync.Services.INearbyConnection nearbyConnection, System.String endpoint, WB.Core.SharedKernels.Enumerator.OfflineSync.Services.Implementation.NearbyCommunicator+PayloadContent payloadContent) [0x000a2] in <58aad36136cc47d0892eecec2cd4dcfa>:0
failedPayload:
errorMessage: Questionnaire ‘3b4cc40bb64548d69220b0c8a62d3e38$7’ was not found. InterviewId 908e5f35-0322-4a87-b372-8ac0ac633727
requestType: UploadInterviewRequest
message:

Did you uninstall the supervisor application and reset it?
Did you delete questionnaire on the server?

5 tablets means 5 Interviewers and one Supervisor or 5 supervisors?

The problem here is that the Supervisor app doesn’t have questionnaire for some reasons.

@vitalii I am working with @efantozzi on this and no we did not uninstall the application on the supervisor tablet neither did we delete the questionnaire. We do have 8 versions of the questionnaire which was necessitated by some evolving changes we have been having to the survey instrument. 5 tablets refers to 4 enumerators and 1 supervisor tablet. We have already done over 300 interviews which were able to synchronize just fine until this error popped up last week on Friday.