Export service suddenly stopped working

stanislaso · January 4, 2021, 4:29am

Hello

Been running a SSol version 20.09 server for a couple of months, interviewers synchronizing their tablets, running exports daily, all fine… until an hour ago: Export service stopped working.

Restarted postgresql and IIS, which usually fixes this kind of issue.
But not this time. Error still occuring.
Restarted the whole Windows server. No luck either.

No changes have been made to the server recently.
No windows updates, no changes in configuration (not that I am aware of anyway), nothing.

Please don’t just tell me to update Survey Solutions to the latest version.
I won’t do it as the survey is in progress and the risk of data loss during an upgrade is still not clear to me.

Diagnostics

.
LOGS SINCE LAST SERVICES RESTART:

Export-service.20210104.log
2021-01-04 03:45:35.400 +00:00 [INF] Started background service: JobWorkersManageService
2021-01-04 03:45:35.402 +00:00 [INF] Export service starting. version 20.09.2.2371
2021-01-04 03:45:35.405 +00:00 [INF] JobProgressReporterBackgroundService started
2021-01-04 03:45:35.417 +00:00 [INF] Now listening on: http://127.0.0.1:49786
2021-01-04 03:45:35.417 +00:00 [INF] Application started. Press Ctrl+C to shut down.

… and that’s it, nothing else.

Headquarters.20210104.log
2021-01-04 03:45:35.170 +00:00 [INF] Application started. Press Ctrl+C to shut down.
2021-01-04 03:45:35.171 +00:00 [INF] Hosting environment: Production
2021-01-04 03:45:35.171 +00:00 [INF] Content root path: C:\Survey Solutions\Site
2021-01-04 03:45:35.188 +00:00 [ERR] Connection id “0HM5GCCOJ3K5K”, Request id “0HM5GCCOJ3K5K:00000001”: An unhandled exception was thrown by the application.
Autofac.Core.DependencyResolutionException: An exception was thrown while activating WB.Core.BoundedContexts.Headquarters.DataExport.Services.DataExportStatusReader -> Î»:WB.Core.BoundedContexts.Headquarters.DataExport.IExportServiceApi.
—> System.UriFormatException: Invalid URI: The format of the URI could not be determined.
at System.Uri.CreateThis(String uri, Boolean dontEscape, UriKind uriKind)
at System.Uri…ctor(String uriString)
at WB.UI.Headquarters.Services.ExportServiceApiConfigurator.ConfigureHttpClient(HttpClient hc)
at WB.Infrastructure.AspNetCore.HttpClientFactoryExtensions.<>c__12.<AddHttpClientWithConfigurator>b__1_0(IServiceProvider sp, HttpClient hc) at Microsoft.Extensions.DependencyInjection.HttpClientBuilderExtensions.<>c__DisplayClass1_1.<ConfigureHttpClient>b__2(HttpClient client) at Microsoft.Extensions.Http.DefaultHttpClientFactory.CreateClient(String name) at Microsoft.Extensions.DependencyInjection.HttpClientBuilderExtensions.<>c__DisplayClass16_01.b__0(IServiceProvider s)
at Autofac.Extensions.DependencyInjection.AutofacRegistration.<>c__DisplayClass3_0.b__0(IComponentContext context, IEnumerable1 parameters) at Autofac.Core.Activators.Delegate.DelegateActivator.ActivateInstance(IComponentContext context, IEnumerable1 parameters)
at Autofac.Core.Resolving.InstanceLookup.Activate(IEnumerable1 parameters, Object& decoratorTarget) --- End of inner exception stack trace --- at Autofac.Core.Resolving.InstanceLookup.Activate(IEnumerable1 parameters, Object& decoratorTarget)
at Autofac.Core.Resolving.InstanceLookup.Execute()
at Autofac.Core.Resolving.ResolveOperation.GetOrCreateInstance(ISharingLifetimeScope currentOperationScope, IComponentRegistration registration, IEnumerable1 parameters) at Autofac.Core.Resolving.ResolveOperation.ResolveComponent(IComponentRegistration registration, IEnumerable1 parameters)
at Autofac.Core.Resolving.ResolveOperation.Execute(IComponentRegistration registration, IEnumerable1 parameters) at Autofac.Core.Lifetime.LifetimeScope.ResolveComponent(IComponentRegistration registration, IEnumerable1 parameters)
at Autofac.ResolutionExtensions.TryResolveService(IComponentContext context, Service service, IEnumerable1 parameters, Object& instance) at Autofac.ResolutionExtensions.ResolveOptionalService(IComponentContext context, Service service, IEnumerable1 parameters)
at Autofac.ResolutionExtensions.ResolveOptional(IComponentContext context, Type serviceType, IEnumerable1 parameters) at Autofac.ResolutionExtensions.ResolveOptional(IComponentContext context, Type serviceType) at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetService(Type serviceType) at Microsoft.Extensions.DependencyInjection.ActivatorUtilities.GetService(IServiceProvider sp, Type type, Type requiredBy, Boolean isDefaultParameterRequired) at lambda_method(Closure , IServiceProvider , Object[] ) at Microsoft.AspNetCore.Mvc.Controllers.ControllerActivatorProvider.<>c__DisplayClass4_0.<CreateActivator>b__0(ControllerContext controllerContext) at Microsoft.AspNetCore.Mvc.Controllers.ControllerFactoryProvider.<>c__DisplayClass5_0.<CreateControllerFactory>g__CreateController|0(ControllerContext controllerContext) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync() --- End of stack trace from previous location where exception was thrown --- at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|24_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context) at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeFilterPipelineAsync() --- End of stack trace from previous location where exception was thrown --- at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope) at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger) at Microsoft.AspNetCore.Localization.RequestLocalizationMiddleware.Invoke(HttpContext context) at HotChocolate.AspNetCore.HttpGetSchemaMiddleware.InvokeAsync(HttpContext context) at HotChocolate.AspNetCore.QueryMiddlewareBase.InvokeAsync(HttpContext context) at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext) at Anemonis.AspNetCore.RequestDecompression.RequestDecompressionMiddleware.InvokeAsync(HttpContext context, RequestDelegate next) at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.<>c__DisplayClass5_1.<<UseMiddlewareInterface>b__1>d.MoveNext() --- End of stack trace from previous location where exception was thrown --- at Microsoft.AspNetCore.ResponseCompression.ResponseCompressionMiddleware.Invoke(HttpContext context) at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context) at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context) at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider) at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context) at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context) at Serilog.AspNetCore.RequestLoggingMiddleware.Invoke(HttpContext httpContext) at Microsoft.AspNetCore.Builder.Extensions.MapMiddleware.Invoke(HttpContext context) at WB.UI.Headquarters.Metrics.HeadquartersHttpMetricsMiddleware.<>c__DisplayClass5_0.<<Invoke>g__Next|0>d.MoveNext() --- End of stack trace from previous location where exception was thrown --- at WB.UI.Headquarters.Metrics.HeadquartersHttpMetricsMiddleware.Invoke(HttpContext context) at StackExchange.Exceptional.ExceptionalMiddleware.Invoke(HttpContext context) at StackExchange.Exceptional.ExceptionalMiddleware.Invoke(HttpContext context) at Microsoft.AspNetCore.Server.IISIntegration.IISMiddleware.Invoke(HttpContext httpContext) at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication1 application)
2021-01-04 03:45:35.418 +00:00 [INF] Headquarters reconfigured to use http://127.0.0.1:49786 address for Export Service
2021-01-04 03:45:37.942 +00:00 [ERR] HTTP GET /api/DataExportApi/ExportStatus responded 500 in 18.3022 ms
Autofac.Core.DependencyResolutionException: An exception was thrown while activating WB.Core.BoundedContexts.Headquarters.DataExport.Services.DataExportStatusReader -> Î»:WB.Core.BoundedContexts.Headquarters.DataExport.IExportServiceApi.
—> System.UriFormatException: Invalid URI: The format of the URI could not be determined.
at System.Uri.CreateThis(String uri, Boolean dontEscape, UriKind uriKind)
at System.Uri…ctor(String uriString)
at WB.UI.Headquarters.Services.ExportServiceApiConfigurator.ConfigureHttpClient(HttpClient hc)
at WB.Infrastructure.AspNetCore.HttpClientFactoryExtensions.<>c__12.<AddHttpClientWithConfigurator>b__1_0(IServiceProvider sp, HttpClient hc) at Microsoft.Extensions.DependencyInjection.HttpClientBuilderExtensions.<>c__DisplayClass1_1.<ConfigureHttpClient>b__2(HttpClient client) at Microsoft.Extensions.Http.DefaultHttpClientFactory.CreateClient(String name) at Microsoft.Extensions.DependencyInjection.HttpClientBuilderExtensions.<>c__DisplayClass16_01.b__0(IServiceProvider s)
at Autofac.Extensions.DependencyInjection.AutofacRegistration.<>c__DisplayClass3_0.b__0(IComponentContext context, IEnumerable1 parameters) at Autofac.Core.Activators.Delegate.DelegateActivator.ActivateInstance(IComponentContext context, IEnumerable1 parameters)
at Autofac.Core.Resolving.InstanceLookup.Activate(IEnumerable1 parameters, Object& decoratorTarget) --- End of inner exception stack trace --- at Autofac.Core.Resolving.InstanceLookup.Activate(IEnumerable1 parameters, Object& decoratorTarget)
at Autofac.Core.Resolving.InstanceLookup.Execute()
at Autofac.Core.Resolving.ResolveOperation.GetOrCreateInstance(ISharingLifetimeScope currentOperationScope, IComponentRegistration registration, IEnumerable1 parameters) at Autofac.Core.Resolving.ResolveOperation.ResolveComponent(IComponentRegistration registration, IEnumerable1 parameters)
at Autofac.Core.Resolving.ResolveOperation.Execute(IComponentRegistration registration, IEnumerable1 parameters) at Autofac.Core.Lifetime.LifetimeScope.ResolveComponent(IComponentRegistration registration, IEnumerable1 parameters)
at Autofac.ResolutionExtensions.TryResolveService(IComponentContext context, Service service, IEnumerable1 parameters, Object& instance) at Autofac.ResolutionExtensions.ResolveOptionalService(IComponentContext context, Service service, IEnumerable1 parameters)
at Autofac.ResolutionExtensions.ResolveOptional(IComponentContext context, Type serviceType, IEnumerable1 parameters) at Autofac.ResolutionExtensions.ResolveOptional(IComponentContext context, Type serviceType) at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetService(Type serviceType) at Microsoft.Extensions.DependencyInjection.ActivatorUtilities.GetService(IServiceProvider sp, Type type, Type requiredBy, Boolean isDefaultParameterRequired) at lambda_method(Closure , IServiceProvider , Object[] ) at Microsoft.AspNetCore.Mvc.Controllers.ControllerActivatorProvider.<>c__DisplayClass4_0.<CreateActivator>b__0(ControllerContext controllerContext) at Microsoft.AspNetCore.Mvc.Controllers.ControllerFactoryProvider.<>c__DisplayClass5_0.<CreateControllerFactory>g__CreateController|0(ControllerContext controllerContext) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync() --- End of stack trace from previous location where exception was thrown --- at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|24_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context) at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeFilterPipelineAsync() --- End of stack trace from previous location where exception was thrown --- at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope) at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger) at Microsoft.AspNetCore.Localization.RequestLocalizationMiddleware.Invoke(HttpContext context) at HotChocolate.AspNetCore.HttpGetSchemaMiddleware.InvokeAsync(HttpContext context) at HotChocolate.AspNetCore.QueryMiddlewareBase.InvokeAsync(HttpContext context) at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext) at Anemonis.AspNetCore.RequestDecompression.RequestDecompressionMiddleware.InvokeAsync(HttpContext context, RequestDelegate next) at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.<>c__DisplayClass5_1.<<UseMiddlewareInterface>b__1>d.MoveNext() --- End of stack trace from previous location where exception was thrown --- at Microsoft.AspNetCore.ResponseCompression.ResponseCompressionMiddleware.Invoke(HttpContext context) at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context) at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context) at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider) at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context) at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context) at Serilog.AspNetCore.RequestLoggingMiddleware.Invoke(HttpContext httpContext) 2021-01-04 03:45:37.947 +00:00 [ERR] An unhandled exception has occurred, logging to Exceptional Autofac.Core.DependencyResolutionException: An exception was thrown while activating WB.Core.BoundedContexts.Headquarters.DataExport.Services.DataExportStatusReader -> Î»:WB.Core.BoundedContexts.Headquarters.DataExport.IExportServiceApi. ---> System.UriFormatException: Invalid URI: The format of the URI could not be determined. at System.Uri.CreateThis(String uri, Boolean dontEscape, UriKind uriKind) at System.Uri..ctor(String uriString) at WB.UI.Headquarters.Services.ExportServiceApiConfigurator.ConfigureHttpClient(HttpClient hc) at WB.Infrastructure.AspNetCore.HttpClientFactoryExtensions.<>c__12.b__1_0(IServiceProvider sp, HttpClient hc)
at Microsoft.Extensions.DependencyInjection.HttpClientBuilderExtensions.<>c__DisplayClass1_1.b__2(HttpClient client)
at Microsoft.Extensions.Http.DefaultHttpClientFactory.CreateClient(String name)
at Microsoft.Extensions.DependencyInjection.HttpClientBuilderExtensions.<>c__DisplayClass16_01.<AddTypedClientCore>b__0(IServiceProvider s) at Autofac.Extensions.DependencyInjection.AutofacRegistration.<>c__DisplayClass3_0.<Register>b__0(IComponentContext context, IEnumerable1 parameters)
at Autofac.Core.Activators.Delegate.DelegateActivator.ActivateInstance(IComponentContext context, IEnumerable1 parameters) at Autofac.Core.Resolving.InstanceLookup.Activate(IEnumerable1 parameters, Object& decoratorTarget)
— End of inner exception stack trace —
at Autofac.Core.Resolving.InstanceLookup.Activate(IEnumerable1 parameters, Object& decoratorTarget) at Autofac.Core.Resolving.InstanceLookup.Execute() at Autofac.Core.Resolving.ResolveOperation.GetOrCreateInstance(ISharingLifetimeScope currentOperationScope, IComponentRegistration registration, IEnumerable1 parameters)
at Autofac.Core.Resolving.ResolveOperation.ResolveComponent(IComponentRegistration registration, IEnumerable1 parameters) at Autofac.Core.Resolving.ResolveOperation.Execute(IComponentRegistration registration, IEnumerable1 parameters)
at Autofac.Core.Lifetime.LifetimeScope.ResolveComponent(IComponentRegistration registration, IEnumerable1 parameters) at Autofac.ResolutionExtensions.TryResolveService(IComponentContext context, Service service, IEnumerable1 parameters, Object& instance)
at Autofac.ResolutionExtensions.ResolveOptionalService(IComponentContext context, Service service, IEnumerable1 parameters) at Autofac.ResolutionExtensions.ResolveOptional(IComponentContext context, Type serviceType, IEnumerable1 parameters)
at Autofac.ResolutionExtensions.ResolveOptional(IComponentContext context, Type serviceType)
at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetService(Type serviceType)
at Microsoft.Extensions.DependencyInjection.ActivatorUtilities.GetService(IServiceProvider sp, Type type, Type requiredBy, Boolean isDefaultParameterRequired)
at lambda_method(Closure , IServiceProvider , Object[] )
at Microsoft.AspNetCore.Mvc.Controllers.ControllerActivatorProvider.<>c__DisplayClass4_0.b__0(ControllerContext controllerContext)
at Microsoft.AspNetCore.Mvc.Controllers.ControllerFactoryProvider.<>c__DisplayClass5_0.g__CreateController|0(ControllerContext controllerContext)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync()
— End of stack trace from previous location where exception was thrown —
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Awaited|24_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeFilterPipelineAsync()
— End of stack trace from previous location where exception was thrown —
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
at Microsoft.AspNetCore.Routing.EndpointMiddleware.g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
at Microsoft.AspNetCore.Localization.RequestLocalizationMiddleware.Invoke(HttpContext context)
at HotChocolate.AspNetCore.HttpGetSchemaMiddleware.InvokeAsync(HttpContext context)
at HotChocolate.AspNetCore.QueryMiddlewareBase.InvokeAsync(HttpContext context)
at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext)
at Anemonis.AspNetCore.RequestDecompression.RequestDecompressionMiddleware.InvokeAsync(HttpContext context, RequestDelegate next)
at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.<>c__DisplayClass5_1.<b__1>d.MoveNext()
— End of stack trace from previous location where exception was thrown —
at Microsoft.AspNetCore.ResponseCompression.ResponseCompressionMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)
at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider)
at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
at Serilog.AspNetCore.RequestLoggingMiddleware.Invoke(HttpContext httpContext)
at Microsoft.AspNetCore.Builder.Extensions.MapMiddleware.Invoke(HttpContext context)
at WB.UI.Headquarters.Metrics.HeadquartersHttpMetricsMiddleware.<>c__DisplayClass5_0.<g__Next|0>d.MoveNext()
— End of stack trace from previous location where exception was thrown —
at WB.UI.Headquarters.Metrics.HeadquartersHttpMetricsMiddleware.Invoke(HttpContext context)
at StackExchange.Exceptional.ExceptionalMiddleware.Invoke(HttpContext context)

… and this goes on and on with similar errors…

alfeg · January 4, 2021, 9:11am

@stanislaso

As far as I can see You have malformed URI in the following configuration:

[Headquraters]
BaseUrl=

This issue is actually fixed in 20.10 version.
Please specify in BaseUrl You real public address, this will solve an issue

sergiy · January 4, 2021, 9:37am

The above two statements appear contradictive. Either the configuration is wrong and was wrong before (meaning the exports were not running earlier) or someone has had changed the configuration indeed, perhaps without notifying @stanislaso .

stanislaso · January 4, 2021, 11:07pm

Hi there,

Just checked. No changes have been made to the appsettings.production.ini file since the launch in October 2020. And it worked fine until yesterday.

And the BaseUrl is correct (partially hidden below).

[Headquarters]
BaseUrl=https://xxxxxx.surveys.spc.int
TenantName=hq

stanislaso · January 5, 2021, 12:52am

Hi again,

I found out the PostgreSQL server had some issues writing to a log file.

Postgres logs dislayed some errors like these:

2021-01-04 23:00:40.983 UTC [5192] LOG:  could not rename file "pg_wal/000000010000001D000000DD": Permission denied
[...]
2021-01-04 23:53:30.321 UTC [996] LOG:  could not rename temporary statistics file "pg_stat_tmp/global.tmp" to "pg_stat_tmp/global.stat": Permission denied

Checking the permissions on those files showed that indeed nobody could make any changes to those files.

While restarting the server yesterday didn’t help, now that a new day has started simply restarting postgres again did the trick. New log files have indeed be re-created with proper permissions.

I am not a postgres expert but that’s the only explanation I can find from the logs. Still doesn’t explain the malformed URI error, but I haven’t made any change (and nobody else did).

Anyway and most importantly, it’s all working again, including the export service.

vitalii · January 5, 2021, 1:01am

Hi @stanislaso,

It looks like Export Schema was corrupted and system was not able to recover from it.

Do you have an option to drop schema manually?

Could you do the following:

Stop the SurveySolutions site in IIS
Open your DB editor (pgAdmin by default) and find your database (SurveySolution by default, you could check your connection string in config file)
Find schema(s) that starts with hq_ (something like hq_104043a3-8323-…)
Drop this schema(s)
Start site

It would take some time to generate export from scratch.

Sorry for inconvenience

stanislaso · January 5, 2021, 1:43am

Hi @vitalii

Well an export is currently in progress and looks like it’s working.
Will wait for it to finish first.
Do you think I still need to do this if everything looks good?

Stan

sergiy · January 5, 2021, 3:59pm

A similar case is discussed here:

A quote from that discussion:

Either somebody or something locked or modified the permissions on this file. Perhaps it was anti-virus software (which should always be prevented from messing with the data directory).

Glad that the issue is resolved.

vitalii · January 5, 2021, 6:21pm

Hi @stanislaso,

Since you found the cause of the problem and it was resolved Export Service was recovered and started working properly.
No need to make any changes.

stanislaso · January 5, 2021, 9:28pm

A similar case is discussed here:
(link to stackoverflow)

Yes @sergiy, I read that one when I discovered the error from the postgres logs.
As explained there, same happened to me. File eventually disappeared once I stopped Postgres.