Hello
Been running a SSol version 20.09 server for a couple of months, interviewers synchronizing their tablets, running exports daily, all fine… until an hour ago: Export service stopped working.
Restarted postgresql and IIS, which usually fixes this kind of issue.
But not this time. Error still occuring.
Restarted the whole Windows server. No luck either.
No changes have been made to the server recently.
No windows updates, no changes in configuration (not that I am aware of anyway), nothing.
Please don’t just tell me to update Survey Solutions to the latest version.
I won’t do it as the survey is in progress and the risk of data loss during an upgrade is still not clear to me.
Diagnostics
.
LOGS SINCE LAST SERVICES RESTART:
Export-service.20210104.log
2021-01-04 03:45:35.400 +00:00 [INF] Started background service: JobWorkersManageService
2021-01-04 03:45:35.402 +00:00 [INF] Export service starting. version 20.09.2.2371
2021-01-04 03:45:35.405 +00:00 [INF] JobProgressReporterBackgroundService started
2021-01-04 03:45:35.417 +00:00 [INF] Now listening on: http://127.0.0.1:49786
2021-01-04 03:45:35.417 +00:00 [INF] Application started. Press Ctrl+C to shut down.
… and that’s it, nothing else.
Headquarters.20210104.log
2021-01-04 03:45:35.170 +00:00 [INF] Application started. Press Ctrl+C to shut down.
2021-01-04 03:45:35.171 +00:00 [INF] Hosting environment: Production
2021-01-04 03:45:35.171 +00:00 [INF] Content root path: C:\Survey Solutions\Site
2021-01-04 03:45:35.188 +00:00 [ERR] Connection id “0HM5GCCOJ3K5K”, Request id “0HM5GCCOJ3K5K:00000001”: An unhandled exception was thrown by the application.
Autofac.Core.DependencyResolutionException: An exception was thrown while activating WB.Core.BoundedContexts.Headquarters.DataExport.Services.DataExportStatusReader -> λ:WB.Core.BoundedContexts.Headquarters.DataExport.IExportServiceApi.
—> System.UriFormatException: Invalid URI: The format of the URI could not be determined.
at System.Uri.CreateThis(String uri, Boolean dontEscape, UriKind uriKind)
at System.Uri…ctor(String uriString)
at WB.UI.Headquarters.Services.ExportServiceApiConfigurator.ConfigureHttpClient(HttpClient hc)
at WB.Infrastructure.AspNetCore.HttpClientFactoryExtensions.<>c__12.<AddHttpClientWithConfigurator>b__1_0(IServiceProvider sp, HttpClient hc) at Microsoft.Extensions.DependencyInjection.HttpClientBuilderExtensions.<>c__DisplayClass1_1.<ConfigureHttpClient>b__2(HttpClient client) at Microsoft.Extensions.Http.DefaultHttpClientFactory.CreateClient(String name) at Microsoft.Extensions.DependencyInjection.HttpClientBuilderExtensions.<>c__DisplayClass16_0
1.b__0(IServiceProvider s)
at Autofac.Extensions.DependencyInjection.AutofacRegistration.<>c__DisplayClass3_0.b__0(IComponentContext context, IEnumerable1 parameters) at Autofac.Core.Activators.Delegate.DelegateActivator.ActivateInstance(IComponentContext context, IEnumerable
1 parameters)
at Autofac.Core.Resolving.InstanceLookup.Activate(IEnumerable1 parameters, Object& decoratorTarget) --- End of inner exception stack trace --- at Autofac.Core.Resolving.InstanceLookup.Activate(IEnumerable
1 parameters, Object& decoratorTarget)
at Autofac.Core.Resolving.InstanceLookup.Execute()
at Autofac.Core.Resolving.ResolveOperation.GetOrCreateInstance(ISharingLifetimeScope currentOperationScope, IComponentRegistration registration, IEnumerable1 parameters) at Autofac.Core.Resolving.ResolveOperation.ResolveComponent(IComponentRegistration registration, IEnumerable
1 parameters)
at Autofac.Core.Resolving.ResolveOperation.Execute(IComponentRegistration registration, IEnumerable1 parameters) at Autofac.Core.Lifetime.LifetimeScope.ResolveComponent(IComponentRegistration registration, IEnumerable
1 parameters)
at Autofac.ResolutionExtensions.TryResolveService(IComponentContext context, Service service, IEnumerable1 parameters, Object& instance) at Autofac.ResolutionExtensions.ResolveOptionalService(IComponentContext context, Service service, IEnumerable
1 parameters)
at Autofac.ResolutionExtensions.ResolveOptional(IComponentContext context, Type serviceType, IEnumerable1 parameters) at Autofac.ResolutionExtensions.ResolveOptional(IComponentContext context, Type serviceType) at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetService(Type serviceType) at Microsoft.Extensions.DependencyInjection.ActivatorUtilities.GetService(IServiceProvider sp, Type type, Type requiredBy, Boolean isDefaultParameterRequired) at lambda_method(Closure , IServiceProvider , Object[] ) at Microsoft.AspNetCore.Mvc.Controllers.ControllerActivatorProvider.<>c__DisplayClass4_0.<CreateActivator>b__0(ControllerContext controllerContext) at Microsoft.AspNetCore.Mvc.Controllers.ControllerFactoryProvider.<>c__DisplayClass5_0.<CreateControllerFactory>g__CreateController|0(ControllerContext controllerContext) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync() --- End of stack trace from previous location where exception was thrown --- at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|24_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context) at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeFilterPipelineAsync() --- End of stack trace from previous location where exception was thrown --- at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope) at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger) at Microsoft.AspNetCore.Localization.RequestLocalizationMiddleware.Invoke(HttpContext context) at HotChocolate.AspNetCore.HttpGetSchemaMiddleware.InvokeAsync(HttpContext context) at HotChocolate.AspNetCore.QueryMiddlewareBase.InvokeAsync(HttpContext context) at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext) at Anemonis.AspNetCore.RequestDecompression.RequestDecompressionMiddleware.InvokeAsync(HttpContext context, RequestDelegate next) at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.<>c__DisplayClass5_1.<<UseMiddlewareInterface>b__1>d.MoveNext() --- End of stack trace from previous location where exception was thrown --- at Microsoft.AspNetCore.ResponseCompression.ResponseCompressionMiddleware.Invoke(HttpContext context) at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context) at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context) at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider) at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context) at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context) at Serilog.AspNetCore.RequestLoggingMiddleware.Invoke(HttpContext httpContext) at Microsoft.AspNetCore.Builder.Extensions.MapMiddleware.Invoke(HttpContext context) at WB.UI.Headquarters.Metrics.HeadquartersHttpMetricsMiddleware.<>c__DisplayClass5_0.<<Invoke>g__Next|0>d.MoveNext() --- End of stack trace from previous location where exception was thrown --- at WB.UI.Headquarters.Metrics.HeadquartersHttpMetricsMiddleware.Invoke(HttpContext context) at StackExchange.Exceptional.ExceptionalMiddleware.Invoke(HttpContext context) at StackExchange.Exceptional.ExceptionalMiddleware.Invoke(HttpContext context) at Microsoft.AspNetCore.Server.IISIntegration.IISMiddleware.Invoke(HttpContext httpContext) at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication
1 application)
2021-01-04 03:45:35.418 +00:00 [INF] Headquarters reconfigured to use http://127.0.0.1:49786 address for Export Service
2021-01-04 03:45:37.942 +00:00 [ERR] HTTP GET /api/DataExportApi/ExportStatus responded 500 in 18.3022 ms
Autofac.Core.DependencyResolutionException: An exception was thrown while activating WB.Core.BoundedContexts.Headquarters.DataExport.Services.DataExportStatusReader -> λ:WB.Core.BoundedContexts.Headquarters.DataExport.IExportServiceApi.
—> System.UriFormatException: Invalid URI: The format of the URI could not be determined.
at System.Uri.CreateThis(String uri, Boolean dontEscape, UriKind uriKind)
at System.Uri…ctor(String uriString)
at WB.UI.Headquarters.Services.ExportServiceApiConfigurator.ConfigureHttpClient(HttpClient hc)
at WB.Infrastructure.AspNetCore.HttpClientFactoryExtensions.<>c__12.<AddHttpClientWithConfigurator>b__1_0(IServiceProvider sp, HttpClient hc) at Microsoft.Extensions.DependencyInjection.HttpClientBuilderExtensions.<>c__DisplayClass1_1.<ConfigureHttpClient>b__2(HttpClient client) at Microsoft.Extensions.Http.DefaultHttpClientFactory.CreateClient(String name) at Microsoft.Extensions.DependencyInjection.HttpClientBuilderExtensions.<>c__DisplayClass16_0
1.b__0(IServiceProvider s)
at Autofac.Extensions.DependencyInjection.AutofacRegistration.<>c__DisplayClass3_0.b__0(IComponentContext context, IEnumerable1 parameters) at Autofac.Core.Activators.Delegate.DelegateActivator.ActivateInstance(IComponentContext context, IEnumerable
1 parameters)
at Autofac.Core.Resolving.InstanceLookup.Activate(IEnumerable1 parameters, Object& decoratorTarget) --- End of inner exception stack trace --- at Autofac.Core.Resolving.InstanceLookup.Activate(IEnumerable
1 parameters, Object& decoratorTarget)
at Autofac.Core.Resolving.InstanceLookup.Execute()
at Autofac.Core.Resolving.ResolveOperation.GetOrCreateInstance(ISharingLifetimeScope currentOperationScope, IComponentRegistration registration, IEnumerable1 parameters) at Autofac.Core.Resolving.ResolveOperation.ResolveComponent(IComponentRegistration registration, IEnumerable
1 parameters)
at Autofac.Core.Resolving.ResolveOperation.Execute(IComponentRegistration registration, IEnumerable1 parameters) at Autofac.Core.Lifetime.LifetimeScope.ResolveComponent(IComponentRegistration registration, IEnumerable
1 parameters)
at Autofac.ResolutionExtensions.TryResolveService(IComponentContext context, Service service, IEnumerable1 parameters, Object& instance) at Autofac.ResolutionExtensions.ResolveOptionalService(IComponentContext context, Service service, IEnumerable
1 parameters)
at Autofac.ResolutionExtensions.ResolveOptional(IComponentContext context, Type serviceType, IEnumerable1 parameters) at Autofac.ResolutionExtensions.ResolveOptional(IComponentContext context, Type serviceType) at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetService(Type serviceType) at Microsoft.Extensions.DependencyInjection.ActivatorUtilities.GetService(IServiceProvider sp, Type type, Type requiredBy, Boolean isDefaultParameterRequired) at lambda_method(Closure , IServiceProvider , Object[] ) at Microsoft.AspNetCore.Mvc.Controllers.ControllerActivatorProvider.<>c__DisplayClass4_0.<CreateActivator>b__0(ControllerContext controllerContext) at Microsoft.AspNetCore.Mvc.Controllers.ControllerFactoryProvider.<>c__DisplayClass5_0.<CreateControllerFactory>g__CreateController|0(ControllerContext controllerContext) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync() --- End of stack trace from previous location where exception was thrown --- at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|24_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context) at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeFilterPipelineAsync() --- End of stack trace from previous location where exception was thrown --- at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope) at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger) at Microsoft.AspNetCore.Localization.RequestLocalizationMiddleware.Invoke(HttpContext context) at HotChocolate.AspNetCore.HttpGetSchemaMiddleware.InvokeAsync(HttpContext context) at HotChocolate.AspNetCore.QueryMiddlewareBase.InvokeAsync(HttpContext context) at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext) at Anemonis.AspNetCore.RequestDecompression.RequestDecompressionMiddleware.InvokeAsync(HttpContext context, RequestDelegate next) at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.<>c__DisplayClass5_1.<<UseMiddlewareInterface>b__1>d.MoveNext() --- End of stack trace from previous location where exception was thrown --- at Microsoft.AspNetCore.ResponseCompression.ResponseCompressionMiddleware.Invoke(HttpContext context) at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context) at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context) at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider) at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context) at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context) at Serilog.AspNetCore.RequestLoggingMiddleware.Invoke(HttpContext httpContext) 2021-01-04 03:45:37.947 +00:00 [ERR] An unhandled exception has occurred, logging to Exceptional Autofac.Core.DependencyResolutionException: An exception was thrown while activating WB.Core.BoundedContexts.Headquarters.DataExport.Services.DataExportStatusReader -> λ:WB.Core.BoundedContexts.Headquarters.DataExport.IExportServiceApi. ---> System.UriFormatException: Invalid URI: The format of the URI could not be determined. at System.Uri.CreateThis(String uri, Boolean dontEscape, UriKind uriKind) at System.Uri..ctor(String uriString) at WB.UI.Headquarters.Services.ExportServiceApiConfigurator.ConfigureHttpClient(HttpClient hc) at WB.Infrastructure.AspNetCore.HttpClientFactoryExtensions.<>c__1
2.b__1_0(IServiceProvider sp, HttpClient hc)
at Microsoft.Extensions.DependencyInjection.HttpClientBuilderExtensions.<>c__DisplayClass1_1.b__2(HttpClient client)
at Microsoft.Extensions.Http.DefaultHttpClientFactory.CreateClient(String name)
at Microsoft.Extensions.DependencyInjection.HttpClientBuilderExtensions.<>c__DisplayClass16_01.<AddTypedClientCore>b__0(IServiceProvider s) at Autofac.Extensions.DependencyInjection.AutofacRegistration.<>c__DisplayClass3_0.<Register>b__0(IComponentContext context, IEnumerable
1 parameters)
at Autofac.Core.Activators.Delegate.DelegateActivator.ActivateInstance(IComponentContext context, IEnumerable1 parameters) at Autofac.Core.Resolving.InstanceLookup.Activate(IEnumerable
1 parameters, Object& decoratorTarget)
— End of inner exception stack trace —
at Autofac.Core.Resolving.InstanceLookup.Activate(IEnumerable1 parameters, Object& decoratorTarget) at Autofac.Core.Resolving.InstanceLookup.Execute() at Autofac.Core.Resolving.ResolveOperation.GetOrCreateInstance(ISharingLifetimeScope currentOperationScope, IComponentRegistration registration, IEnumerable
1 parameters)
at Autofac.Core.Resolving.ResolveOperation.ResolveComponent(IComponentRegistration registration, IEnumerable1 parameters) at Autofac.Core.Resolving.ResolveOperation.Execute(IComponentRegistration registration, IEnumerable
1 parameters)
at Autofac.Core.Lifetime.LifetimeScope.ResolveComponent(IComponentRegistration registration, IEnumerable1 parameters) at Autofac.ResolutionExtensions.TryResolveService(IComponentContext context, Service service, IEnumerable
1 parameters, Object& instance)
at Autofac.ResolutionExtensions.ResolveOptionalService(IComponentContext context, Service service, IEnumerable1 parameters) at Autofac.ResolutionExtensions.ResolveOptional(IComponentContext context, Type serviceType, IEnumerable
1 parameters)
at Autofac.ResolutionExtensions.ResolveOptional(IComponentContext context, Type serviceType)
at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetService(Type serviceType)
at Microsoft.Extensions.DependencyInjection.ActivatorUtilities.GetService(IServiceProvider sp, Type type, Type requiredBy, Boolean isDefaultParameterRequired)
at lambda_method(Closure , IServiceProvider , Object[] )
at Microsoft.AspNetCore.Mvc.Controllers.ControllerActivatorProvider.<>c__DisplayClass4_0.b__0(ControllerContext controllerContext)
at Microsoft.AspNetCore.Mvc.Controllers.ControllerFactoryProvider.<>c__DisplayClass5_0.g__CreateController|0(ControllerContext controllerContext)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync()
— End of stack trace from previous location where exception was thrown —
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Awaited|24_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeFilterPipelineAsync()
— End of stack trace from previous location where exception was thrown —
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
at Microsoft.AspNetCore.Routing.EndpointMiddleware.g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
at Microsoft.AspNetCore.Localization.RequestLocalizationMiddleware.Invoke(HttpContext context)
at HotChocolate.AspNetCore.HttpGetSchemaMiddleware.InvokeAsync(HttpContext context)
at HotChocolate.AspNetCore.QueryMiddlewareBase.InvokeAsync(HttpContext context)
at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext)
at Anemonis.AspNetCore.RequestDecompression.RequestDecompressionMiddleware.InvokeAsync(HttpContext context, RequestDelegate next)
at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.<>c__DisplayClass5_1.<b__1>d.MoveNext()
— End of stack trace from previous location where exception was thrown —
at Microsoft.AspNetCore.ResponseCompression.ResponseCompressionMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)
at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider)
at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
at Serilog.AspNetCore.RequestLoggingMiddleware.Invoke(HttpContext httpContext)
at Microsoft.AspNetCore.Builder.Extensions.MapMiddleware.Invoke(HttpContext context)
at WB.UI.Headquarters.Metrics.HeadquartersHttpMetricsMiddleware.<>c__DisplayClass5_0.<g__Next|0>d.MoveNext()
— End of stack trace from previous location where exception was thrown —
at WB.UI.Headquarters.Metrics.HeadquartersHttpMetricsMiddleware.Invoke(HttpContext context)
at StackExchange.Exceptional.ExceptionalMiddleware.Invoke(HttpContext context)
… and this goes on and on with similar errors…