An unexpected error occurred during export

Good morning everyone,

The Mozambique NSO has just started a survey and are unable to export any data (“An unexpected error occurred during export” message).
Synchronization, viewing interviews, etc. all works fine.

Background information:

Version: 21.09.5 (build 31862)

They had two questionnaires on the server for training purposes and export was working fine.
They then deleted these training questionnaires, they are now shown greyed-out in the Survey Setup.
Then they imported new qx versions and started data collection two days ago. When they try to export they get the error.

When I try to start the export through the API I get a “403 Forbidden” response.

The logs show a number of errors of type “Questionnaire was deleted”:

2022-01-12 04:44:23.038 -08:00 [ERR] Error when dispatching ‘OnConnectedAsync’ on hub.
WB.Core.SharedKernels.DataCollection.Exceptions.InterviewException: Questionnaire was deleted
at WB.Core.BoundedContexts.Headquarters.Interview.Validators.QuestionnaireStateForInterviewValidator.CheckQuestionnaireAndThrow(QuestionnaireIdentity questionnaireIdentity)
— End of stack trace from previous location —
at WB.Core.Infrastructure.CommandBus.CommandRegistry.<>c__DisplayClass18_0.b__0(IAggregateRoot aggregate, ICommand command)
at WB.Core.Infrastructure.CommandBus.Implementation.CommandExecutor.ExecuteEventSourcedCommand(ICommand command, String origin, Type aggregateType, Guid aggregateId, IEnumerable1 validators, IEnumerable1 preProcessors, IEnumerable1 postProcessors, Action2 commandHandler, CancellationToken cancellationToken)
at WB.Core.Infrastructure.CommandBus.Implementation.CommandExecutor.ExecuteCommand(ICommand command, String origin, CancellationToken cancellationToken, Guid aggregateId)
at WB.Core.Infrastructure.CommandBus.Implementation.CommandService.<>c__DisplayClass16_0.b__1(ICommandExecutor ce)
at WB.Core.BoundedContexts.Headquarters.Implementation.UnitOfWorkInScopeExecutor.Execute(Action1 action, String workspace) at WB.Core.GenericSubdomains.Portable.NamedLocker.RunWithLock(String name, Action body) at WB.Core.Infrastructure.CommandBus.Implementation.CommandService.ExecuteImpl(ICommand command, String origin, CancellationToken cancellationToken) at WB.Core.Infrastructure.CommandBus.Implementation.CommandService.Execute(ICommand command, String origin) at WB.UI.Headquarters.Code.WebInterview.Pipeline.PauseResumePipelineModule.OnConnected(Hub hub) at WB.Enumerator.Native.WebInterview.WebInterview.OnConnectedAsync() at Microsoft.AspNetCore.SignalR.Internal.DefaultHubDispatcher1.OnConnectedAsync(HubConnectionContext connection)
at Microsoft.AspNetCore.SignalR.Internal.DefaultHubDispatcher1.OnConnectedAsync(HubConnectionContext connection) at Microsoft.AspNetCore.SignalR.HubConnectionHandler1.RunHubAsync(HubConnectionContext connection)

Note: We are not trying to export data for one of the deleted questionnaires.

Thank you,

Klaus

This message is probably unrelated to the export problem. The log may contain other error messages. See what else is there.

@benj is also reporting strange deletion behavior where the questionnaires being deleted get stuck for a long time:

Dear Sergiy,
You are right, the deleted questionnaire message doesn’t seem to be the cause of the export problem.
Apparently there is a single export job which does not terminate since days now. It repeatedly throws an error:

Npgsql.PostgresException (0x80004005): 42703: column “e9ce9f37-eaee-560f-c890-d63912348c4e” of relation “IOF_NUTRICAO$1-e” does not exist

@vitalii is looking at the logs, we are waiting to see how to get rid of this blocking export job.

Thanks,
Klaus

Thank you for sharing the logs.

Please try the maintenance M1 procedure according to Procedures for maintenance .

Dear Sergiy, thank you for the link to the procedure. We only removed the temporary schema for the workspace that had the problem and now everything works perfectly again.
Klaus

Great!
Thank you for confirming!

Dear Sergiy and Support,

We finished our 2021 survey quite some time ago (Jan 2022). But we kept the server live as some interviews with commercial agriculture holdings needed postponing and because it was easier for interviewers to get back to the original interview cases when supervisors or data cleaning staff identified issues. Now is the time to close the instance and we successfully downloaded all data from the two questionnaire versions we had, except for the paradata for the version with the highest number of cases. We would like to try and access this paradata as it could help answering some questions we have about data quality.

It seems like we have the same error as described in this thread, which is why I am following up on it. We tried to update the SuSo version to see if it could help solve the issue (it did not), and I downloaded the logs, as suggested (access them at https://drive.google.com/drive/folders/1AZ8sHox-hqVXTmi0tpXDQzMEUrJwIdhg?usp=share_link). I am not sure about what I should be looking for here. Can you spot any issues? What should my next move be?
Thanks a lot.

Dear Sergiy and Support,

I also tried M1 procedure but still having problem in exporting paradata from the server. Would you please guide us on option to export paradata?

Thanks a lot.

According to the latest log you are using version 21.09.2, but it’s not the latest available version.

Dear Vitalli, I copied the wrong logs (before we updated SuSo), sorry for that. Just updated them.
Thank you.

Logs are mentioning this: “System.IO.IOException: There is not enough space on the disk.”

Dear Vitali,

Thank you for your assistance for resolving the issue. We manage to download the paradata.

The server storage was full due to multiple exported files during the survey (197 exported files available at C:\Survey Solutions\Data_Site\export. Will it be possible to add the delete button next to the download on the data export page? Administrator can delete those files after downloading and never run into storage full issue.

Thanking you once again for the support.