An unexpected error occurred during export

Good morning everyone,

The Mozambique NSO has just started a survey and are unable to export any data (“An unexpected error occurred during export” message).
Synchronization, viewing interviews, etc. all works fine.

Background information:

Version: 21.09.5 (build 31862)

They had two questionnaires on the server for training purposes and export was working fine.
They then deleted these training questionnaires, they are now shown greyed-out in the Survey Setup.
Then they imported new qx versions and started data collection two days ago. When they try to export they get the error.

When I try to start the export through the API I get a “403 Forbidden” response.

The logs show a number of errors of type “Questionnaire was deleted”:

2022-01-12 04:44:23.038 -08:00 [ERR] Error when dispatching ‘OnConnectedAsync’ on hub.
WB.Core.SharedKernels.DataCollection.Exceptions.InterviewException: Questionnaire was deleted
at WB.Core.BoundedContexts.Headquarters.Interview.Validators.QuestionnaireStateForInterviewValidator.CheckQuestionnaireAndThrow(QuestionnaireIdentity questionnaireIdentity)
— End of stack trace from previous location —
at WB.Core.Infrastructure.CommandBus.CommandRegistry.<>c__DisplayClass18_0.b__0(IAggregateRoot aggregate, ICommand command)
at WB.Core.Infrastructure.CommandBus.Implementation.CommandExecutor.ExecuteEventSourcedCommand(ICommand command, String origin, Type aggregateType, Guid aggregateId, IEnumerable1 validators, IEnumerable1 preProcessors, IEnumerable1 postProcessors, Action2 commandHandler, CancellationToken cancellationToken)
at WB.Core.Infrastructure.CommandBus.Implementation.CommandExecutor.ExecuteCommand(ICommand command, String origin, CancellationToken cancellationToken, Guid aggregateId)
at WB.Core.Infrastructure.CommandBus.Implementation.CommandService.<>c__DisplayClass16_0.b__1(ICommandExecutor ce)
at WB.Core.BoundedContexts.Headquarters.Implementation.UnitOfWorkInScopeExecutor.Execute(Action1 action, String workspace) at WB.Core.GenericSubdomains.Portable.NamedLocker.RunWithLock(String name, Action body) at WB.Core.Infrastructure.CommandBus.Implementation.CommandService.ExecuteImpl(ICommand command, String origin, CancellationToken cancellationToken) at WB.Core.Infrastructure.CommandBus.Implementation.CommandService.Execute(ICommand command, String origin) at WB.UI.Headquarters.Code.WebInterview.Pipeline.PauseResumePipelineModule.OnConnected(Hub hub) at WB.Enumerator.Native.WebInterview.WebInterview.OnConnectedAsync() at Microsoft.AspNetCore.SignalR.Internal.DefaultHubDispatcher1.OnConnectedAsync(HubConnectionContext connection)
at Microsoft.AspNetCore.SignalR.Internal.DefaultHubDispatcher1.OnConnectedAsync(HubConnectionContext connection) at Microsoft.AspNetCore.SignalR.HubConnectionHandler1.RunHubAsync(HubConnectionContext connection)

Note: We are not trying to export data for one of the deleted questionnaires.

Thank you,

Klaus

This message is probably unrelated to the export problem. The log may contain other error messages. See what else is there.

@benj is also reporting strange deletion behavior where the questionnaires being deleted get stuck for a long time:

Dear Sergiy,
You are right, the deleted questionnaire message doesn’t seem to be the cause of the export problem.
Apparently there is a single export job which does not terminate since days now. It repeatedly throws an error:

Npgsql.PostgresException (0x80004005): 42703: column “e9ce9f37-eaee-560f-c890-d63912348c4e” of relation “IOF_NUTRICAO$1-e” does not exist

@vitalii is looking at the logs, we are waiting to see how to get rid of this blocking export job.

Thanks,
Klaus

Thank you for sharing the logs.

Please try the maintenance M1 procedure according to Procedures for maintenance .

Dear Sergiy, thank you for the link to the procedure. We only removed the temporary schema for the workspace that had the problem and now everything works perfectly again.
Klaus

Great!
Thank you for confirming!