Data export is not working 2

Dear Survey Solutions Support Team,

I would like to ask for your help to eliminate the technical error that appears when I am trying to download a dataset from the server https://smesurvey.intracen.org/.

When I click on “Add to queue” my download request, I keep receiving a message “An unexpected error occurred during export. We are sorry for inconvenience. Please contact support team support@mysurvey.solutions with the log files *.log that are stored in the application folder.”

Here is the list of steps I follow after logging into the system:

  1. Go to “Data Export” and choose a “survey template” (any dataset that is available on the server)

  2. I tick the following options that define the format of the dataset that I want to download

  3. I click on “Add to queue” and wait until the dataset is prepared. I noticed that it takes up to 20-30 seconds for the system to reply, which is very rare. It usually takes 5-10 seconds.

  4. Then, instead of receiving a message that the dataset is prepared, and I can download it, I receive a message “An unexpected error occurred during export. We are sorry for inconvenience. Please contact support team support@mysurvey.solutions with the log files *.log that are stored in the application folder.”

I saw that some new observations were uploaded to the server recently, meaning that it works for some functions, but not entirely. Unfortunately, it is not related to any specific dataset – I tried several ones, and the problem persists. I also tried to change the browser (Google Chrome and Microsoft Edge) – no result.

We use this functionality weekly. As shown below, the last time we successfully exported was on Jan 24th at 1:40 pm local server hour. And since then, the application has not been able to write any more on the folder.

We have upgraded the app to the latest version, rebooted the webserver and recycled the app pool several times to test some theories. We also stopped the IIS and then ran a Repair on the installation, and everything was completed successfully, but the problem persists.

I would appreciate it if you could help us to solve this technical issue as soon as possible to access the data collected.

Kind regards,
Antonina

Log files:

2022-02-03 12:37:19.336 +00:00 [INF] Start job execution [hq {“ExportSettings”:{“JobId”:null,“ExportFormat”:2,“QuestionnaireId”:{“Id”:“88f9bb923beb4e818894102030add6eb$5”},“Status”:null,“FromDate”:null,“ToDate”:null,“Tenant”:{“BaseUrl”:https://smesurvey.intracen.org/primary,“Id”:{“Id”:“44863722-fb50-45a6-aa39-92e28b8826e5”},“Name”:“hq”,“ShortName”:“hq”,“Workspace”:“primary”},“Translation”:null,“IncludeMeta”:true,“NaturalId”:"44863722-fb50-45a6-aa39-92e28b8826e5$All$STATA$88f9bb923beb4e818894102030add6eb$5$EMPTY FROM DATE$EMPTY TO DATE$No translation"},“Status”:{“CreatedDate”:“0001-01-01T00:00:00”,“BeginDate”:null,“EndDate”:null,“LastUpdateDate”:“0001-01-01T00:00:00”,“Status”:2,“JobStatus”:0,“ProgressInPercents”:0,“IsRunning”:false,“TimeEstimation”:null,“Error”:null},“ArchivePassword”:null,“NaturalId”:“NoExternal$44863722-fb50-45a6-aa39-92e28b8826e5$All$STATA$88f9bb923beb4e818894102030add6eb$5$EMPTY FROM DATE$EMPTY TO DATE$No translation”,“AccessToken”:null,“RefreshToken”:null,“StorageType”:null,“ProcessId”:0,“ShouldDropTenantSchema”:false}]
2022-02-03 12:37:19.518 +00:00 [FTL] Encountered interview id 64b3b427-358e-400d-9faa-3f5f6d42be9a without corresponding InterviewCreated or InterviewOnClientCreated events
System.Exception: Encountered interview id 64b3b427-358e-400d-9faa-3f5f6d42be9a without corresponding InterviewCreated or InterviewOnClientCreated events
at WB.Services.Export.Questionnaire.QuestionnaireEventFilter.FilterAsync(ICollection`1 feed, CancellationToken cancellationToken)
at WB.Services.Export.Events.EventsHandler.HandleEventsFeedAsync(EventsFeed feed, CancellationToken token)
at WB.Services.Export.Events.EventsHandler.HandleEventsFeedAsync(EventsFeed feed, CancellationToken token)
2022-02-03 12:37:19.518 +00:00 [FTL] Exception Data : WB:Event : YesNoQuestionAnswered
2022-02-03 12:37:19.518 +00:00 [FTL] Exception Data : WB:GlobalSequence : 261568
2022-02-03 12:37:19.518 +00:00 [FTL] Exception Data : WB:EventSourceId : 64b3b427-358e-400d-9faa-3f5f6d42be9a
2022-02-03 12:37:19.518 +00:00 [FTL] Exception Data : WB:Events : 260785:264999
2022-02-03 12:37:19.519 +00:00 [ERR] Export job failed

2022-02-03 10:55:03.005 +00:00 [ERR] HTTP GET /primary/api/DataExportApi/ExportStatus responded 500 in 1663.7482 ms
Autofac.Core.DependencyResolutionException: An exception was thrown while activating WB.Core.BoundedContexts.Headquarters.DataExport.Services.DataExportStatusReader → λ:WB.Core.BoundedContexts.Headquarters.DataExport.IExportServiceApi.
—> System.UriFormatException: Invalid URI: The format of the URI could not be determined.
at System.Uri.CreateThis(String uri, Boolean dontEscape, UriKind uriKind)
at WB.UI.Headquarters.Services.ExportServiceApiConfigurator.ConfigureHttpClient(HttpClient hc)
at WB.UI.Headquarters.Code.Workspaces.WorkspaceExtension.<>c__DisplayClass2_03.<AddWorkspaceAwareHttpClient>b__0(IServiceProvider s) at Autofac.Core.Activators.Delegate.DelegateActivator.ActivateInstance(IComponentContext context, IEnumerable1 parameters)
at Autofac.Core.Resolving.InstanceLookup.Activate(IEnumerable1 parameters, Object& decoratorTarget) --- End of inner exception stack trace --- at Autofac.Core.Resolving.InstanceLookup.Activate(IEnumerable1 parameters, Object& decoratorTarget)
at Autofac.Core.Resolving.InstanceLookup.Execute()
at Autofac.Core.Resolving.ResolveOperation.GetOrCreateInstance(ISharingLifetimeScope currentOperationScope, IComponentRegistration registration, IEnumerable1 parameters) at Autofac.Core.Resolving.ResolveOperation.Execute(IComponentRegistration registration, IEnumerable1 parameters)
at Autofac.ResolutionExtensions.TryResolveService(IComponentContext context, Service service, IEnumerable1 parameters, Object& instance) at Autofac.ResolutionExtensions.ResolveOptionalService(IComponentContext context, Service service, IEnumerable1 parameters)
at Microsoft.Extensions.DependencyInjection.ActivatorUtilities.GetService(IServiceProvider sp, Type type, Type requiredBy, Boolean isDefaultParameterRequired)
at lambda_method1174(Closure , IServiceProvider , Object[] )
at Microsoft.AspNetCore.Mvc.Controllers.ControllerFactoryProvider.<>c__DisplayClass5_0.g__CreateController|0(ControllerContext controllerContext)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync()
— End of stack trace from previous location —

This is not normal as the DB seems to contain an interview without its beginning. This blocks the export of all interviews.

The issue is clearly not related to the browser from which you are starting the export of the data, but to the state of the DB.

It is not clear from your message what version was running there before the update, and whether the update was the cause of the interruption or an attempted remedy.

Perhaps, you could try [with caution] the maintenance procedure M1.

Best, Sergiy

Good morning Sergiy,
We have upgraded to the latest version from SurveySolutions.21.9.3.31749 and we tried also with a clean version of the software and we still have this issue. Yesterday I tried the maintenance M1 and deleted in cascade the only one hq_ schema we have in the db the when I started the IIS and the site the same hq_ schema gets created automatically and when I try to export the data I get the same output.
I think that the issue might be related with some missing/corrupt information at db level as you mentioned – see the logs below:


2022-02-04 16:50:44.262 +00:00 [FTL] Encountered interview id 64b3b427-358e-400d-9faa-3f5f6d42be9a without corresponding InterviewCreated or InterviewOnClientCreated events
System.Exception: Encountered interview id 64b3b427-358e-400d-9faa-3f5f6d42be9a without corresponding InterviewCreated or InterviewOnClientCreated events
at WB.Services.Export.Questionnaire.QuestionnaireEventFilter.FilterAsync(ICollection`1 feed, CancellationToken cancellationToken)
at WB.Services.Export.Events.EventsHandler.HandleEventsFeedAsync(EventsFeed feed, CancellationToken token)
at WB.Services.Export.Events.EventsHandler.HandleEventsFeedAsync(EventsFeed feed, CancellationToken token)
2022-02-04 16:50:44.297 +00:00 [FTL] Exception Data : WB:Event : YesNoQuestionAnswered
2022-02-04 16:50:44.298 +00:00 [FTL] Exception Data : WB:GlobalSequence : 261568
2022-02-04 16:50:44.298 +00:00 [FTL] Exception Data : WB:EventSourceId : 64b3b427-358e-400d-9faa-3f5f6d42be9a
2022-02-04 16:50:44.298 +00:00 [FTL] Exception Data : WB:Events : 216925:267404
2022-02-04 16:50:44.303 +00:00 [ERR] Export job failed



2022-02-04 16:49:44.043 +00:00 [ERR] HTTP GET /primary/api/DataExportApi/ExportStatus responded 500 in 2429.6196 ms
Autofac.Core.DependencyResolutionException: An exception was thrown while activating WB.Core.BoundedContexts.Headquarters.DataExport.Services.DataExportStatusReader → λ:WB.Core.BoundedContexts.Headquarters.DataExport.IExportServiceApi.
—> System.UriFormatException: Invalid URI: The format of the URI could not be determined.
at System.Uri.CreateThis(String uri, Boolean dontEscape, UriKind uriKind)
at WB.UI.Headquarters.Services.ExportServiceApiConfigurator.ConfigureHttpClient(HttpClient hc)
at WB.UI.Headquarters.Code.Workspaces.WorkspaceExtension.<>c__DisplayClass2_03.<AddWorkspaceAwareHttpClient>b__0(IServiceProvider s) at Autofac.Core.Activators.Delegate.DelegateActivator.ActivateInstance(IComponentContext context, IEnumerable1 parameters)
at Autofac.Core.Resolving.InstanceLookup.Activate(IEnumerable`1 parameters, Object& decoratorTarget)
— End of inner exception stack trace —


What else can we try here?
Thank you,

This is the correct behavior. The hope was that if there were any problems with filling the table for export earlier, they can be rectified by rebuilding it. Apparently the rebuilding process has hit the same obstacle.

However, this doesn’t explain the other exception that is present in your log:

This seems to be a configuration problem of some sort.

yes this is really odd but we haven’t changed the config files - as mentioned before I even done a clean installation and we still have this issue. let me know where/what to look for or if you need additional file/log to have a better idea of where the issue may lie.

Also note that the URI could not be determined error is no longer present.
Check the log below:

2022-02-07 08:57:09.058 +00:00 [INF] Application is shutting down…
2022-02-07 08:57:46.378 +00:00 [WRN] HQ application starting. Version 21.09.9 (build 32035)
2022-02-07 08:57:51.899 +00:00 [INF] Now listening on: http://127.0.0.1:18471
2022-02-07 08:57:51.927 +00:00 [INF] Starting Export Service host
2022-02-07 08:57:51.930 +00:00 [INF] Configuring export service to use http://127.0.0.1:18471 as tenant url for hq
2022-02-07 08:57:54.109 +00:00 [INF] Application started. Press Ctrl+C to shut down.
2022-02-07 08:57:54.110 +00:00 [INF] Hosting environment: Production
2022-02-07 08:57:54.110 +00:00 [INF] Content root path: C:\Survey Solutions\Site
2022-02-07 08:57:54.556 +00:00 [INF] Headquarters reconfigured to use http://127.0.0.1:55005 address for Export Service
2022-02-07 08:59:39.987 +00:00 [INF] Application is shutting down…
2022-02-07 09:00:40.086 +00:00 [WRN] HQ application starting. Version 21.09.9 (build 32035)
2022-02-07 09:00:46.424 +00:00 [INF] Now listening on: http://127.0.0.1:40030
2022-02-07 09:00:46.452 +00:00 [INF] Starting Export Service host
2022-02-07 09:00:46.455 +00:00 [INF] Configuring export service to use http://127.0.0.1:40030 as tenant url for hq
2022-02-07 09:00:48.579 +00:00 [INF] Application started. Press Ctrl+C to shut down.
2022-02-07 09:00:48.580 +00:00 [INF] Hosting environment: Production
2022-02-07 09:00:48.580 +00:00 [INF] Content root path: C:\Survey Solutions\Site
2022-02-07 09:00:49.807 +00:00 [INF] Headquarters reconfigured to use http://127.0.0.1:55073 address for Export Service
2022-02-07 09:00:51.889 +00:00 [INF] Application is shutting down…
2022-02-07 09:01:46.918 +00:00 [WRN] HQ application starting. Version 21.09.9 (build 32035)
2022-02-07 09:01:53.140 +00:00 [INF] Now listening on: http://127.0.0.1:27372
2022-02-07 09:01:53.167 +00:00 [INF] Starting Export Service host
2022-02-07 09:01:53.169 +00:00 [INF] Configuring export service to use http://127.0.0.1:27372 as tenant url for hq
2022-02-07 09:01:55.055 +00:00 [INF] Application started. Press Ctrl+C to shut down.
2022-02-07 09:01:55.055 +00:00 [INF] Hosting environment: Production
2022-02-07 09:01:55.055 +00:00 [INF] Content root path: C:\Survey Solutions\Site
2022-02-07 09:01:55.792 +00:00 [INF] Headquarters reconfigured to use http://127.0.0.1:55128 address for Export Service
2022-02-07 10:06:34.196 +00:00 [INF] Return 10000 events in “00:00:00.4314348” with 4.1172075271606445 MB size
2022-02-07 10:06:34.463 +00:00 [INF] Return 1225 events in “00:00:00.0149684” with 526.052734375 KB size
2022-02-07 10:06:45.342 +00:00 [INF] Return 10000 events in “00:00:00.1220596” with 4.1172075271606445 MB size
2022-02-07 10:06:45.605 +00:00 [INF] Return 1225 events in “00:00:00.0133384” with 526.052734375 KB size
2022-02-07 10:06:56.118 +00:00 [INF] Return 10000 events in “00:00:00.1699751” with 4.1172075271606445 MB size
2022-02-07 10:06:56.298 +00:00 [INF] Return 1225 events in “00:00:00.0163673” with 526.052734375 KB size
2022-02-07 10:07:06.684 +00:00 [INF] Return 10000 events in “00:00:00.1031184” with 4.1172075271606445 MB size
2022-02-07 10:07:06.928 +00:00 [INF] Return 1225 events in “00:00:00.0213253” with 526.052734375 KB size
2022-02-07 10:15:20.369 +00:00 [INF] Return 10000 events in “00:00:00.1050238” with 4.1172075271606445 MB size
2022-02-07 10:15:20.585 +00:00 [INF] Return 1225 events in “00:00:00.0195335” with 526.052734375 KB size
2022-02-07 10:15:31.464 +00:00 [INF] Return 10000 events in “00:00:00.0956441” with 4.1172075271606445 MB size
2022-02-07 10:15:31.677 +00:00 [INF] Return 1225 events in “00:00:00.0141970” with 526.052734375 KB size
2022-02-07 10:15:42.055 +00:00 [INF] Return 10000 events in “00:00:00.1054064” with 4.1172075271606445 MB size
2022-02-07 10:15:42.480 +00:00 [INF] Return 1225 events in “00:00:00.0331910” with 526.052734375 KB size
2022-02-07 10:15:53.024 +00:00 [INF] Return 10000 events in “00:00:00.1410030” with 4.1172075271606445 MB size
2022-02-07 10:15:53.184 +00:00 [INF] Return 1225 events in “00:00:00.0120795” with 526.052734375 KB size
2022-02-07 10:31:41.896 +00:00 [INF] Return 10000 events in “00:00:00.2133704” with 4.1172075271606445 MB size
2022-02-07 10:31:42.256 +00:00 [INF] Return 1225 events in “00:00:00.0344555” with 526.052734375 KB size
2022-02-07 10:31:53.362 +00:00 [INF] Return 10000 events in “00:00:00.1865026” with 4.1172075271606445 MB size
2022-02-07 10:31:53.966 +00:00 [INF] Return 1225 events in “00:00:00.0470117” with 526.052734375 KB size
2022-02-07 10:32:04.607 +00:00 [INF] Return 10000 events in “00:00:00.1100832” with 4.1172075271606445 MB size
2022-02-07 10:32:04.844 +00:00 [INF] Return 1225 events in “00:00:00.0182724” with 526.052734375 KB size
2022-02-07 10:32:15.437 +00:00 [INF] Return 10000 events in “00:00:00.0907313” with 4.1172075271606445 MB size
2022-02-07 10:32:15.623 +00:00 [INF] Return 1225 events in “00:00:00.0133697” with 526.052734375 KB size


So the only error present is the one from the Export file complaining about the data missing in the db

2022-02-07 10:32:15.306 +00:00 [INF] Start job execution [hq {“ExportSettings”:{“JobId”:null,“ExportFormat”:2,“QuestionnaireId”:{“Id”:“88f9bb923beb4e818894102030add6eb$5”},“Status”:null,“FromDate”:null,“ToDate”:null,“Tenant”:{“BaseUrl”:“https://smesurvey.intracen.org/primary",“Id”:{“Id”:“44863722-fb50-45a6-aa39-92e28b8826e5”},“Name”:“hq”,“ShortName”:“hq”,“Workspace”:“primary”},“Translation”:null,“IncludeMeta”:true,“NaturalId”:"44863722-fb50-45a6-aa39-92e28b8826e5$All$STATA$88f9bb923beb4e818894102030add6eb$5$EMPTY FROM DATE$EMPTY TO DATE$No translation”},“Status”:{“CreatedDate”:“0001-01-01T00:00:00”,“BeginDate”:null,“EndDate”:null,“LastUpdateDate”:“0001-01-01T00:00:00”,“Status”:2,“JobStatus”:0,“ProgressInPercents”:0,“IsRunning”:false,“TimeEstimation”:null,“Error”:null},“ArchivePassword”:null,“NaturalId”:“NoExternal$44863722-fb50-45a6-aa39-92e28b8826e5$All$STATA$88f9bb923beb4e818894102030add6eb$5$EMPTY FROM DATE$EMPTY TO DATE$No translation”,“AccessToken”:null,“RefreshToken”:null,“StorageType”:null,“ProcessId”:0,“ShouldDropTenantSchema”:false}]
2022-02-07 10:32:15.720 +00:00 [FTL] Encountered interview id 64b3b427-358e-400d-9faa-3f5f6d42be9a without corresponding InterviewCreated or InterviewOnClientCreated events
System.Exception: Encountered interview id 64b3b427-358e-400d-9faa-3f5f6d42be9a without corresponding InterviewCreated or InterviewOnClientCreated events
at WB.Services.Export.Questionnaire.QuestionnaireEventFilter.FilterAsync(ICollection`1 feed, CancellationToken cancellationToken)
at WB.Services.Export.Events.EventsHandler.HandleEventsFeedAsync(EventsFeed feed, CancellationToken token)
at WB.Services.Export.Events.EventsHandler.HandleEventsFeedAsync(EventsFeed feed, CancellationToken token)
2022-02-07 10:32:15.720 +00:00 [FTL] Exception Data : WB:Event : YesNoQuestionAnswered
2022-02-07 10:32:15.720 +00:00 [FTL] Exception Data : WB:GlobalSequence : 261568
2022-02-07 10:32:15.720 +00:00 [FTL] Exception Data : WB:EventSourceId : 64b3b427-358e-400d-9faa-3f5f6d42be9a
2022-02-07 10:32:15.720 +00:00 [FTL] Exception Data : WB:Events : 256925:266925
2022-02-07 10:32:15.721 +00:00 [ERR] Export job failed


Please advise,

Can you please help me to obtain the Interview__key from the below error message.
with this info we can delete the interview in question and try again the ExportData option.

Encountered interview id 64b3b427-358e-400d-9faa-3f5f6d42be9a without corresponding InterviewCreated or InterviewOnClientCreated events

System.Exception: Encountered interview id 64b3b427-358e-400d-9faa-3f5f6d42be9a without corresponding InterviewCreated or InterviewOnClientCreated events

  1. There is no formula to obtain interview__key from interview__id.
  2. Interview key may be modified (once) during the lifetime of an interview.
  3. There is no ‘delete interview’ functionality in Survey Solutions.

The correspondence between the interview id and key is established at the creation of the interview when a special event KeyAssigned (nr 27) is fired with a new key value. See Paradata file format. And presumably this is exactly the information that is missing in your DB.

I see,
what can we do in this case?
is there a way to delete the wrong/incomplete interview? or do we have to go back and restore the db from any backup dated prior to Jan 24th (this is the day when the last successful ExportData was triggered)?

thank you

@iulian_mihai

Is it happening with all the formats to export the data?

yes, we have tried with all the formats and the same error occurs

The issue is solved thanks to the timely help of the Survey Solutions Support team.