Syncing problem when synchronization of test interviews is done simultaneously by >10 interviewers

Deployment
#1

We have 2 Survey Solutions apps deployed on our on-premises server. They are accessible from internet via a CNAME record that was configured in our firewall settings.

Today we had a connectivity problem during the training stage with one of the web apps whenever >10 interviewers tried to sync with the server (from various networks in different countries). We expect that this could be a problem during the production stage, because our use case involves syncing of even larger amounts of data and this syncing could happen simultaneously. Can you please advise if the issue could be related to:

  • firewall settings
  • on-prem server settings
  • internet bandwidth limitations
  • virtual machine settings incl. processor capacity (each sync requires about 100 requests per second)
  • anything else?

Thank you!

#2

Yes, positively, the issue could be related to the mentioned factors.

Survey Solutions software doesn’t have a limit of 10 interviewers, but other elements of your infrastructure may have. So the total throughput of the system will be determined by the weakest element (the bottleneck).

#3

Thank you for your response, Sergiy.
One more question - are there any recommended IIS settings (max. number of connections, timeout etc.) or should the default IIS values be used?
Thank you!

#4

IIS comes with reasonable defaults. Some fine-tuning may be required, but not in such trivial situations.

What is the setup you are using? Ten is suspiciously round to be caused by work overload, more like an imposed limit.

#5

Sorry for a slight misrepresentation of the issue. Ten was just a rough number of simultaneous connections. It’s actually more than ten. Around ~20 users are trying to sync with the on-premises server located in Germany from their locations both in Germany and Malawi during the training session. We are wondering if that would cause problems during the real data collection. The firewall that we are using has about 250Mbit/s throughput.

#6

2021-03-30 00:20:59.647 +02:00 [INF] Application is shutting down…
2021-03-30 07:38:36.524 +02:00 [WRN] HQ application starting. Version 21.01.1 (build 30301)
2021-03-30 07:38:37.406 +02:00 [INF] Metrics server started
2021-03-30 07:38:39.365 +02:00 [INF] Now listening on: http://127.0.0.1:16696
2021-03-30 07:38:39.374 +02:00 [INF] Starting Export Service host
2021-03-30 07:38:39.375 +02:00 [INF] Configuring export service to use http://127.0.0.1:16696 as tenant url for hq
2021-03-30 07:38:39.486 +02:00 [INF] User import job: Finished. Elapsed time: “00:00:00.0692893”
2021-03-30 07:38:39.861 +02:00 [INF] Application started. Press Ctrl+C to shut down.
2021-03-30 07:38:39.861 +02:00 [INF] Hosting environment: Production
2021-03-30 07:38:39.861 +02:00 [INF] Content root path: C:\Survey Solutions\Site_P2012
2021-03-30 07:38:40.145 +02:00 [INF] Headquarters reconfigured to use http://127.0.0.1:53737 address for Export Service
2021-03-30 08:00:35.395 +02:00 [INF] Application is shutting down…
2021-03-30 08:02:35.209 +02:00 [WRN] HQ application starting. Version 21.01.1 (build 30301)
2021-03-30 08:02:36.085 +02:00 [INF] Metrics server started
2021-03-30 08:02:37.852 +02:00 [INF] Now listening on: http://127.0.0.1:30099
2021-03-30 08:02:37.861 +02:00 [INF] Starting Export Service host
2021-03-30 08:02:37.861 +02:00 [INF] Configuring export service to use http://127.0.0.1:30099 as tenant url for hq
2021-03-30 08:02:37.968 +02:00 [INF] User import job: Finished. Elapsed time: “00:00:00.0773456”
2021-03-30 08:02:38.329 +02:00 [INF] Application started. Press Ctrl+C to shut down.
2021-03-30 08:02:38.329 +02:00 [INF] Hosting environment: Production
2021-03-30 08:02:38.329 +02:00 [INF] Content root path: C:\Survey Solutions\Site_P2012
2021-03-30 08:02:38.490 +02:00 [INF] Headquarters reconfigured to use http://127.0.0.1:53881 address for Export Service
2021-03-30 08:27:34.474 +02:00 [INF] Application is shutting down…
2021-03-30 08:27:47.439 +02:00 [WRN] HQ application starting. Version 21.01.1 (build 30301)
2021-03-30 08:27:48.321 +02:00 [INF] Metrics server started
2021-03-30 08:27:50.175 +02:00 [INF] Now listening on: http://127.0.0.1:13577
2021-03-30 08:27:50.185 +02:00 [INF] Starting Export Service host
2021-03-30 08:27:50.185 +02:00 [INF] Configuring export service to use http://127.0.0.1:13577 as tenant url for hq
2021-03-30 08:27:50.277 +02:00 [INF] User import job: Finished. Elapsed time: “00:00:00.0747567”
2021-03-30 08:27:50.651 +02:00 [INF] Application started. Press Ctrl+C to shut down.
2021-03-30 08:27:50.651 +02:00 [INF] Hosting environment: Production
2021-03-30 08:27:50.651 +02:00 [INF] Content root path: C:\Survey Solutions\Site_P2012
2021-03-30 08:27:50.812 +02:00 [INF] Headquarters reconfigured to use http://127.0.0.1:54004 address for Export Service
2021-03-30 09:21:46.725 +02:00 [INF] Application is shutting down…
2021-03-30 09:45:42.533 +02:00 [WRN] HQ application starting. Version 21.01.1 (build 30301)
2021-03-30 09:45:43.518 +02:00 [INF] Metrics server started
2021-03-30 09:45:45.384 +02:00 [INF] Now listening on: http://127.0.0.1:21570
2021-03-30 09:45:45.393 +02:00 [INF] Starting Export Service host
2021-03-30 09:45:45.394 +02:00 [INF] Configuring export service to use http://127.0.0.1:21570 as tenant url for hq
2021-03-30 09:45:45.501 +02:00 [INF] User import job: Finished. Elapsed time: “00:00:00.0733365”
2021-03-30 09:45:45.864 +02:00 [INF] Application started. Press Ctrl+C to shut down.
2021-03-30 09:45:45.864 +02:00 [INF] Hosting environment: Production
2021-03-30 09:45:45.864 +02:00 [INF] Content root path: C:\Survey Solutions\Site_P2012
2021-03-30 09:45:46.023 +02:00 [INF] Headquarters reconfigured to use http://127.0.0.1:54410 address for Export Service
2021-03-30 10:11:41.786 +02:00 [INF] Application is shutting down…
2021-03-30 11:43:40.581 +02:00 [WRN] HQ application starting. Version 21.01.1 (build 30301)
2021-03-30 11:43:41.470 +02:00 [INF] Metrics server started
2021-03-30 11:43:43.344 +02:00 [INF] Now listening on: http://127.0.0.1:44432
2021-03-30 11:43:43.357 +02:00 [INF] Starting Export Service host
2021-03-30 11:43:43.358 +02:00 [INF] Configuring export service to use http://127.0.0.1:44432 as tenant url for hq
2021-03-30 11:43:43.474 +02:00 [INF] User import job: Finished. Elapsed time: “00:00:00.0759559”
2021-03-30 11:43:43.832 +02:00 [INF] Application started. Press Ctrl+C to shut down.
2021-03-30 11:43:43.832 +02:00 [INF] Hosting environment: Production
2021-03-30 11:43:43.832 +02:00 [INF] Content root path: C:\Survey Solutions\Site_P2012
2021-03-30 11:43:43.993 +02:00 [INF] Headquarters reconfigured to use http://127.0.0.1:54781 address for Export Service
2021-03-30 12:06:39.813 +02:00 [INF] Application is shutting down…
2021-03-30 14:19:26.601 +02:00 [WRN] HQ application starting. Version 21.01.1 (build 30301)
2021-03-30 14:19:27.478 +02:00 [INF] Metrics server started
2021-03-30 14:19:29.338 +02:00 [INF] Now listening on: http://127.0.0.1:30673
2021-03-30 14:19:29.347 +02:00 [INF] Starting Export Service host
2021-03-30 14:19:29.347 +02:00 [INF] Configuring export service to use http://127.0.0.1:30673 as tenant url for hq
2021-03-30 14:19:29.459 +02:00 [INF] User import job: Finished. Elapsed time: “00:00:00.0750843”
2021-03-30 14:19:29.814 +02:00 [INF] Application started. Press Ctrl+C to shut down.
2021-03-30 14:19:29.814 +02:00 [INF] Hosting environment: Production
2021-03-30 14:19:29.814 +02:00 [INF] Content root path: C:\Survey Solutions\Site_P2012
2021-03-30 14:19:29.980 +02:00 [INF] Headquarters reconfigured to use http://127.0.0.1:55263 address for Export Service
2021-03-30 14:21:05.735 +02:00 [INF] Downloading Questionnaire Backup.
2021-03-30 14:21:05.737 +02:00 [INF] Requesting pdf generator to start working for questionnaire 26163b76-3f16-40e2-969f-567c58043dcd
2021-03-30 14:21:16.311 +02:00 [INF] Loading pdf for default language
2021-03-30 14:21:17.476 +02:00 [INF] loading pdf for translation WB.Core.SharedKernels.SurveySolutions.Documents.Translation
2021-03-30 14:21:18.560 +02:00 [INF] Saving translation.
2021-03-30 14:21:19.230 +02:00 [INF] Saving lookup table.
2021-03-30 14:21:19.234 +02:00 [INF] Saving lookup table.
2021-03-30 14:21:52.545 +02:00 [INF] Requesting pdf generator to start working for questionnaire 9ba6eaea-13c1-4cf0-8da5-b2a3f17a020d
2021-03-30 14:21:52.545 +02:00 [INF] Downloading Questionnaire Backup.
2021-03-30 14:21:56.638 +02:00 [INF] Loading pdf for default language
2021-03-30 14:21:57.144 +02:00 [INF] loading pdf for translation WB.Core.SharedKernels.SurveySolutions.Documents.Translation
2021-03-30 14:21:57.683 +02:00 [INF] Saving translation.
2021-03-30 14:21:57.712 +02:00 [INF] Saving lookup table.
2021-03-30 14:22:09.014 +02:00 [INF] Downloading Questionnaire Backup.
2021-03-30 14:22:09.014 +02:00 [INF] Requesting pdf generator to start working for questionnaire a88b06da-3436-4472-93f7-246c52d15764
2021-03-30 14:22:12.792 +02:00 [INF] Loading pdf for default language
2021-03-30 14:22:13.446 +02:00 [INF] loading pdf for translation WB.Core.SharedKernels.SurveySolutions.Documents.Translation
2021-03-30 14:22:13.981 +02:00 [INF] Saving translation.
2021-03-30 14:22:14.005 +02:00 [INF] Saving lookup table.
2021-03-30 14:22:14.005 +02:00 [INF] Saving lookup table.
2021-03-30 14:22:29.937 +02:00 [INF] Downloading Questionnaire Backup.
2021-03-30 14:22:29.937 +02:00 [INF] Requesting pdf generator to start working for questionnaire 3123be5c-f07b-44b3-a01d-fc8075f0b1eb
2021-03-30 14:22:37.374 +02:00 [INF] Loading pdf for default language
2021-03-30 14:22:38.370 +02:00 [INF] loading pdf for translation WB.Core.SharedKernels.SurveySolutions.Documents.Translation
2021-03-30 14:22:38.916 +02:00 [INF] Saving translation.
2021-03-30 14:22:38.970 +02:00 [INF] Saving lookup table.
2021-03-30 14:22:38.970 +02:00 [INF] Saving lookup table.
2021-03-30 14:22:52.631 +02:00 [INF] Failed to import questionnaire from designer. RestException
WB.Core.Infrastructure.HttpServices.HttpClient.RestException: Your questionnaire has errors. Please verify the questionnaire on Designer.
—> WB.Core.Infrastructure.HttpServices.HttpClient.ExtendedMessageHandlerException: Request GET https://designer.mysurvey.solutions/api/hq/v3/questionnaires/f65faf10-3bd2-49d3-a761-155559553257?clientQuestionnaireContentVersion=32&minSupportedQuestionnaireVersion=32 failed with status code 412 (Your questionnaire has errors. Please verify the questionnaire on Designer.).
at WB.UI.Headquarters.Services.Impl.DesignerRestServiceHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
— End of inner exception stack trace —
at WB.UI.Headquarters.Services.Impl.DesignerRestServiceHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.SendAsyncCore(HttpRequestMessage request, HttpCompletionOption completionOption, Boolean async, Boolean emitTelemetryStartStop, CancellationToken cancellationToken)
at Refit.RequestBuilderImplementation.<>c__DisplayClass14_0`2.<b__0>d.MoveNext() in /_/Refit/RequestBuilderImplementation.cs:line 283
— End of stack trace from previous location —
at WB.Core.BoundedContexts.Headquarters.Implementation.Services.QuestionnaireImportService.ImportImpl(RestCredentials credentials, IServiceLocator serviceLocator, Guid userId, String userName, Guid questionnaireId, QuestionnaireImportResult questionnaireImportResult, String name, Boolean isCensusMode, String comment, String requestUrl, Boolean shouldMigrateAssignments, QuestionnaireIdentity migrateFrom, Boolean includePdf)
2021-03-30 14:23:41.423 +02:00 [INF] Requesting pdf generator to start working for questionnaire f65faf10-3bd2-49d3-a761-155559553257
2021-03-30 14:23:41.423 +02:00 [INF] Downloading Questionnaire Backup.
2021-03-30 14:23:48.860 +02:00 [INF] Loading pdf for default language
2021-03-30 14:23:49.232 +02:00 [INF] loading pdf for translation WB.Core.SharedKernels.SurveySolutions.Documents.Translation
2021-03-30 14:23:49.754 +02:00 [INF] loading pdf for translation WB.Core.SharedKernels.SurveySolutions.Documents.Translation
2021-03-30 14:23:50.141 +02:00 [INF] Saving translation.
2021-03-30 14:23:50.175 +02:00 [INF] Saving translation.
2021-03-30 14:23:50.209 +02:00 [INF] Saving lookup table.
2021-03-30 14:23:50.210 +02:00 [INF] Saving lookup table.
2021-03-30 14:23:50.210 +02:00 [INF] Saving lookup table.
2021-03-30 14:24:02.908 +02:00 [INF] Requesting pdf generator to start working for questionnaire ae24db4c-8d69-4118-8de4-786abfc0fe55
2021-03-30 14:24:02.908 +02:00 [INF] Downloading Questionnaire Backup.
2021-03-30 14:24:06.528 +02:00 [INF] Loading pdf for default language
2021-03-30 14:24:06.892 +02:00 [INF] Saving lookup table.
2021-03-30 14:24:06.892 +02:00 [INF] Saving lookup table.
2021-03-30 14:24:17.168 +02:00 [INF] Downloading Questionnaire Backup.
2021-03-30 14:24:17.169 +02:00 [INF] Requesting pdf generator to start working for questionnaire faf7f5ad-49f5-4b60-9acd-5bbad60033b0
2021-03-30 14:24:21.059 +02:00 [INF] Loading pdf for default language
2021-03-30 14:24:21.590 +02:00 [INF] Saving lookup table.
2021-03-30 14:24:21.590 +02:00 [INF] Saving lookup table.
2021-03-30 14:24:35.296 +02:00 [INF] Requesting pdf generator to start working for questionnaire e087d661-4912-4caa-b6f3-6cdfd978c78a
2021-03-30 14:24:35.296 +02:00 [INF] Downloading Questionnaire Backup.
2021-03-30 14:24:39.168 +02:00 [INF] Loading pdf for default language
2021-03-30 14:24:39.694 +02:00 [INF] Saving lookup table.
2021-03-30 14:24:39.694 +02:00 [INF] Saving lookup table.
2021-03-30 14:24:49.822 +02:00 [INF] Downloading Questionnaire Backup.
2021-03-30 14:24:49.822 +02:00 [INF] Requesting pdf generator to start working for questionnaire 4200f748-0e36-4a65-815c-426e98bcf414
2021-03-30 14:24:53.703 +02:00 [INF] Loading pdf for default language
2021-03-30 14:24:54.218 +02:00 [INF] Saving lookup table.
2021-03-30 14:24:54.218 +02:00 [INF] Saving lookup table.
2021-03-30 14:25:18.929 +02:00 [INF] Requesting pdf generator to start working for questionnaire ab283ba1-dbe8-48bb-9f67-627e8d2cc5d2
2021-03-30 14:25:18.929 +02:00 [INF] Downloading Questionnaire Backup.
2021-03-30 14:25:22.810 +02:00 [INF] Loading pdf for default language
2021-03-30 14:25:23.653 +02:00 [INF] Saving lookup table.
2021-03-30 14:25:23.653 +02:00 [INF] Saving lookup table.
2021-03-30 14:25:34.836 +02:00 [INF] Requesting pdf generator to start working for questionnaire 4f3254ca-45e7-4c46-bad2-ee0b0632bd1e
2021-03-30 14:25:34.836 +02:00 [INF] Downloading Questionnaire Backup.
2021-03-30 14:25:38.410 +02:00 [INF] Loading pdf for default language
2021-03-30 14:25:39.063 +02:00 [INF] Saving lookup table.
2021-03-30 14:25:39.063 +02:00 [INF] Saving lookup table.
2021-03-30 14:51:25.857 +02:00 [INF] Application is shutting down…
2021-03-30 15:02:32.657 +02:00 [WRN] HQ application starting. Version 21.01.1 (build 30301)
2021-03-30 15:02:33.758 +02:00 [INF] Metrics server started
2021-03-30 15:02:35.619 +02:00 [INF] Now listening on: http://127.0.0.1:6366
2021-03-30 15:02:35.629 +02:00 [INF] Starting Export Service host
2021-03-30 15:02:35.630 +02:00 [INF] Configuring export service to use http://127.0.0.1:6366 as tenant url for hq
2021-03-30 15:02:35.851 +02:00 [INF] User import job: Finished. Elapsed time: “00:00:00.1804798”
2021-03-30 15:02:36.109 +02:00 [INF] Application started. Press Ctrl+C to shut down.
2021-03-30 15:02:36.109 +02:00 [INF] Hosting environment: Production
2021-03-30 15:02:36.109 +02:00 [INF] Content root path: C:\Survey Solutions\Site_P2012
2021-03-30 15:02:36.269 +02:00 [INF] Headquarters reconfigured to use http://127.0.0.1:55509 address for Export Service
2021-03-30 15:23:31.904 +02:00 [INF] Application is shutting down…

#7

2021-03-30 16:35:16.359 +02:00 [WRN] HQ application starting. Version 21.01.1 (build 30301)
2021-03-30 16:35:17.231 +02:00 [INF] Metrics server started
2021-03-30 16:35:19.098 +02:00 [INF] Now listening on: http://127.0.0.1:16374
2021-03-30 16:35:19.106 +02:00 [INF] Starting Export Service host
2021-03-30 16:35:19.107 +02:00 [INF] Configuring export service to use http://127.0.0.1:16374 as tenant url for hq
2021-03-30 16:35:19.349 +02:00 [INF] User import job: Finished. Elapsed time: “00:00:00.2069584”
2021-03-30 16:35:19.572 +02:00 [INF] Application started. Press Ctrl+C to shut down.
2021-03-30 16:35:19.572 +02:00 [INF] Hosting environment: Production
2021-03-30 16:35:19.572 +02:00 [INF] Content root path: C:\Survey Solutions\Site_P2012
2021-03-30 16:35:19.953 +02:00 [INF] Headquarters reconfigured to use http://127.0.0.1:55803 address for Export Service
2021-03-30 17:44:51.721 +02:00 [ERR] HTTP POST /primary/api/interviewer/v3/interviews/184f0d06-6396-4633-aadb-6947f34f6072/image responded 500 in 5732.0967 ms
Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException: Reading the request body timed out due to data arriving too slowly. See MinRequestBodyDataRate.
at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelBadHttpRequestException.Throw(RequestRejectionReason reason)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1ChunkedEncodingMessageBody.PumpAsync()
at System.IO.Pipelines.PipeCompletion.ThrowLatchedException()
at System.IO.Pipelines.Pipe.GetReadResult(ReadResult& result)
at System.IO.Pipelines.Pipe.GetReadAsyncResult()
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1ChunkedEncodingMessageBody.ReadAsyncInternal(CancellationToken cancellationToken)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpRequestStream.ReadAsyncInternal(Memory1 destination, CancellationToken cancellationToken) at System.IO.Compression.DeflateStream.FinishReadAsyncMemory(ValueTask1 readTask, Memory1 buffer, CancellationToken cancellationToken) at Microsoft.AspNetCore.WebUtilities.FileBufferingReadStream.ReadAsync(Memory1 buffer, CancellationToken cancellationToken)
at Microsoft.AspNetCore.WebUtilities.StreamHelperExtensions.DrainAsync(Stream stream, ArrayPool1 bytePool, Nullable1 limit, CancellationToken cancellationToken)
at Microsoft.AspNetCore.Mvc.Formatters.NewtonsoftJsonInputFormatter.ReadRequestBodyAsync(InputFormatterContext context, Encoding encoding)
at Microsoft.AspNetCore.Mvc.ModelBinding.Binders.BodyModelBinder.BindModelAsync(ModelBindingContext bindingContext)
at Microsoft.AspNetCore.Mvc.ModelBinding.ParameterBinder.BindModelAsync(ActionContext actionContext, IModelBinder modelBinder, IValueProvider valueProvider, ParameterDescriptor parameter, ModelMetadata metadata, Object value, Object container)
at Microsoft.AspNetCore.Mvc.Controllers.ControllerBinderDelegateProvider.<>c__DisplayClass0_0.<g__Bind|0>d.MoveNext()
— End of stack trace from previous location —
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Awaited|24_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Awaited|19_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
at Microsoft.AspNetCore.Routing.EndpointMiddleware.g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext)
at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.<>c__DisplayClass6_1.<b__1>d.MoveNext()
— End of stack trace from previous location —
at Microsoft.AspNetCore.ResponseCompression.ResponseCompressionMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)
at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider)
at Microsoft.AspNetCore.Authorization.Policy.AuthorizationMiddlewareResultHandler.HandleAsync(RequestDelegate next, HttpContext context, AuthorizationPolicy policy, PolicyAuthorizationResult authorizeResult)
at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
at WB.UI.Headquarters.Code.Workspaces.WorkspaceRedirectMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Builder.Extensions.MapMiddleware.Invoke(HttpContext context)
at WB.UI.Headquarters.Metrics.HeadquartersHttpMetricsMiddleware.<>c__DisplayClass5_0.<g__Next|0>d.MoveNext()
— End of stack trace from previous location —
at WB.UI.Headquarters.Metrics.HeadquartersHttpMetricsMiddleware.Invoke(HttpContext context)
at WB.UI.Headquarters.Code.Workspaces.WorkspaceMiddleware.<>c__DisplayClass2_0.<g__InvokeNextWithScope|0>d.MoveNext()
— End of stack trace from previous location —
at WB.UI.Headquarters.Code.Workspaces.WorkspaceMiddleware.Invoke(HttpContext context)
at Serilog.AspNetCore.RequestLoggingMiddleware.Invoke(HttpContext httpContext)
2021-03-30 17:44:51.741 +02:00 [ERR] An unhandled exception has occurred, logging to Exceptional
Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException: Reading the request body timed out due to data arriving too slowly. See MinRequestBodyDataRate.
at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelBadHttpRequestException.Throw(RequestRejectionReason reason)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1ChunkedEncodingMessageBody.PumpAsync()
at System.IO.Pipelines.PipeCompletion.ThrowLatchedException()
at System.IO.Pipelines.Pipe.GetReadResult(ReadResult& result)
at System.IO.Pipelines.Pipe.GetReadAsyncResult()
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1ChunkedEncodingMessageBody.ReadAsyncInternal(CancellationToken cancellationToken)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpRequestStream.ReadAsyncInternal(Memory1 destination, CancellationToken cancellationToken) at System.IO.Compression.DeflateStream.FinishReadAsyncMemory(ValueTask1 readTask, Memory1 buffer, CancellationToken cancellationToken) at Microsoft.AspNetCore.WebUtilities.FileBufferingReadStream.ReadAsync(Memory1 buffer, CancellationToken cancellationToken)
at Microsoft.AspNetCore.WebUtilities.StreamHelperExtensions.DrainAsync(Stream stream, ArrayPool1 bytePool, Nullable1 limit, CancellationToken cancellationToken)
at Microsoft.AspNetCore.Mvc.Formatters.NewtonsoftJsonInputFormatter.ReadRequestBodyAsync(InputFormatterContext context, Encoding encoding)
at Microsoft.AspNetCore.Mvc.ModelBinding.Binders.BodyModelBinder.BindModelAsync(ModelBindingContext bindingContext)
at Microsoft.AspNetCore.Mvc.ModelBinding.ParameterBinder.BindModelAsync(ActionContext actionContext, IModelBinder modelBinder, IValueProvider valueProvider, ParameterDescriptor parameter, ModelMetadata metadata, Object value, Object container)
at Microsoft.AspNetCore.Mvc.Controllers.ControllerBinderDelegateProvider.<>c__DisplayClass0_0.<g__Bind|0>d.MoveNext()
— End of stack trace from previous location —
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Awaited|24_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Awaited|19_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
at Microsoft.AspNetCore.Routing.EndpointMiddleware.g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext)
at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.<>c__DisplayClass6_1.<b__1>d.MoveNext()
— End of stack trace from previous location —
at Microsoft.AspNetCore.ResponseCompression.ResponseCompressionMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)
at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider)
at Microsoft.AspNetCore.Authorization.Policy.AuthorizationMiddlewareResultHandler.HandleAsync(RequestDelegate next, HttpContext context, AuthorizationPolicy policy, PolicyAuthorizationResult authorizeResult)
at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
at WB.UI.Headquarters.Code.Workspaces.WorkspaceRedirectMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Builder.Extensions.MapMiddleware.Invoke(HttpContext context)
at WB.UI.Headquarters.Metrics.HeadquartersHttpMetricsMiddleware.<>c__DisplayClass5_0.<g__Next|0>d.MoveNext()
— End of stack trace from previous location —
at WB.UI.Headquarters.Metrics.HeadquartersHttpMetricsMiddleware.Invoke(HttpContext context)
at WB.UI.Headquarters.Code.Workspaces.WorkspaceMiddleware.<>c__DisplayClass2_0.<g__InvokeNextWithScope|0>d.MoveNext()
— End of stack trace from previous location —
at WB.UI.Headquarters.Code.Workspaces.WorkspaceMiddleware.Invoke(HttpContext context)
at Serilog.AspNetCore.RequestLoggingMiddleware.Invoke(HttpContext httpContext)
at Microsoft.AspNetCore.Localization.RequestLocalizationMiddleware.Invoke(HttpContext context)
at StackExchange.Exceptional.ExceptionalMiddleware.Invoke(HttpContext context)
2021-03-30 17:44:51.817 +02:00 [ERR] Connection id “0HM7JH45FN2PP”, Request id “0HM7JH45FN2PP:00000020”: An unhandled exception was thrown by the application.
Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException: Reading the request body timed out due to data arriving too slowly. See MinRequestBodyDataRate.
at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelBadHttpRequestException.Throw(RequestRejectionReason reason)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1ChunkedEncodingMessageBody.PumpAsync()
at System.IO.Pipelines.PipeCompletion.ThrowLatchedException()
at System.IO.Pipelines.Pipe.GetReadResult(ReadResult& result)
at System.IO.Pipelines.Pipe.GetReadAsyncResult()
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1ChunkedEncodingMessageBody.ReadAsyncInternal(CancellationToken cancellationToken)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpRequestStream.ReadAsyncInternal(Memory1 destination, CancellationToken cancellationToken) at System.IO.Compression.DeflateStream.FinishReadAsyncMemory(ValueTask1 readTask, Memory1 buffer, CancellationToken cancellationToken) at Microsoft.AspNetCore.WebUtilities.FileBufferingReadStream.ReadAsync(Memory1 buffer, CancellationToken cancellationToken)
at Microsoft.AspNetCore.WebUtilities.StreamHelperExtensions.DrainAsync(Stream stream, ArrayPool1 bytePool, Nullable1 limit, CancellationToken cancellationToken)
at Microsoft.AspNetCore.Mvc.Formatters.NewtonsoftJsonInputFormatter.ReadRequestBodyAsync(InputFormatterContext context, Encoding encoding)
at Microsoft.AspNetCore.Mvc.ModelBinding.Binders.BodyModelBinder.BindModelAsync(ModelBindingContext bindingContext)
at Microsoft.AspNetCore.Mvc.ModelBinding.ParameterBinder.BindModelAsync(ActionContext actionContext, IModelBinder modelBinder, IValueProvider valueProvider, ParameterDescriptor parameter, ModelMetadata metadata, Object value, Object container)
at Microsoft.AspNetCore.Mvc.Controllers.ControllerBinderDelegateProvider.<>c__DisplayClass0_0.<g__Bind|0>d.MoveNext()
— End of stack trace from previous location —
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Awaited|24_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Awaited|19_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
at Microsoft.AspNetCore.Routing.EndpointMiddleware.g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext)
at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.<>c__DisplayClass6_1.<b__1>d.MoveNext()
— End of stack trace from previous location —
at Microsoft.AspNetCore.ResponseCompression.ResponseCompressionMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)
at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider)
at Microsoft.AspNetCore.Authorization.Policy.AuthorizationMiddlewareResultHandler.HandleAsync(RequestDelegate next, HttpContext context, AuthorizationPolicy policy, PolicyAuthorizationResult authorizeResult)
at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
at WB.UI.Headquarters.Code.Workspaces.WorkspaceRedirectMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Builder.Extensions.MapMiddleware.Invoke(HttpContext context)
at WB.UI.Headquarters.Metrics.HeadquartersHttpMetricsMiddleware.<>c__DisplayClass5_0.<g__Next|0>d.MoveNext()
— End of stack trace from previous location —
at WB.UI.Headquarters.Metrics.HeadquartersHttpMetricsMiddleware.Invoke(HttpContext context)
at WB.UI.Headquarters.Code.Workspaces.WorkspaceMiddleware.<>c__DisplayClass2_0.<g__InvokeNextWithScope|0>d.MoveNext()
— End of stack trace from previous location —
at WB.UI.Headquarters.Code.Workspaces.WorkspaceMiddleware.Invoke(HttpContext context)
at Serilog.AspNetCore.RequestLoggingMiddleware.Invoke(HttpContext httpContext)
at Microsoft.AspNetCore.Localization.RequestLocalizationMiddleware.Invoke(HttpContext context)
at StackExchange.Exceptional.ExceptionalMiddleware.Invoke(HttpContext context)
at StackExchange.Exceptional.ExceptionalMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Server.IISIntegration.IISMiddleware.Invoke(HttpContext httpContext)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)

#8

2021-03-30 17:48:50.878 +02:00 [ERR] HTTP POST /primary/api/interviewer/v3/interviews/184f0d06-6396-4633-aadb-6947f34f6072/image responded 500 in 128104.2541 ms
Microsoft.AspNetCore.Connections.ConnectionResetException: Eine vorhandene Verbindung wurde vom Remotehost geschlossen.
—> System.Net.Sockets.SocketException (10054): Eine vorhandene Verbindung wurde vom Remotehost geschlossen.
at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.Internal.SocketAwaitableEventArgs.g__ThrowSocketException|7_0(SocketError e)
at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.Internal.SocketAwaitableEventArgs.GetResult()
at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.Internal.SocketConnection.ProcessReceives()
at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.Internal.SocketConnection.DoReceive()
— End of inner exception stack trace —
at System.IO.Pipelines.PipeCompletion.ThrowLatchedException()
at System.IO.Pipelines.Pipe.GetReadResult(ReadResult& result)
at System.IO.Pipelines.Pipe.GetReadAsyncResult()
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1ChunkedEncodingMessageBody.PumpAsync()
at System.IO.Pipelines.PipeCompletion.ThrowLatchedException()
at System.IO.Pipelines.Pipe.GetReadResult(ReadResult& result)
at System.IO.Pipelines.Pipe.GetReadAsyncResult()
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1ChunkedEncodingMessageBody.ReadAsyncInternal(CancellationToken cancellationToken)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpRequestStream.ReadAsyncInternal(Memory1 destination, CancellationToken cancellationToken) at System.IO.Compression.DeflateStream.FinishReadAsyncMemory(ValueTask1 readTask, Memory1 buffer, CancellationToken cancellationToken) at Microsoft.AspNetCore.WebUtilities.FileBufferingReadStream.ReadAsync(Memory1 buffer, CancellationToken cancellationToken)
at Microsoft.AspNetCore.WebUtilities.StreamHelperExtensions.DrainAsync(Stream stream, ArrayPool1 bytePool, Nullable1 limit, CancellationToken cancellationToken)
at Microsoft.AspNetCore.Mvc.Formatters.NewtonsoftJsonInputFormatter.ReadRequestBodyAsync(InputFormatterContext context, Encoding encoding)
at Microsoft.AspNetCore.Mvc.ModelBinding.Binders.BodyModelBinder.BindModelAsync(ModelBindingContext bindingContext)
at Microsoft.AspNetCore.Mvc.ModelBinding.ParameterBinder.BindModelAsync(ActionContext actionContext, IModelBinder modelBinder, IValueProvider valueProvider, ParameterDescriptor parameter, ModelMetadata metadata, Object value, Object container)
at Microsoft.AspNetCore.Mvc.Controllers.ControllerBinderDelegateProvider.<>c__DisplayClass0_0.<g__Bind|0>d.MoveNext()
— End of stack trace from previous location —
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Awaited|24_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Awaited|19_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
at Microsoft.AspNetCore.Routing.EndpointMiddleware.g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext)
at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.<>c__DisplayClass6_1.<b__1>d.MoveNext()
— End of stack trace from previous location —
at Microsoft.AspNetCore.ResponseCompression.ResponseCompressionMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)
at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider)
at Microsoft.AspNetCore.Authorization.Policy.AuthorizationMiddlewareResultHandler.HandleAsync(RequestDelegate next, HttpContext context, AuthorizationPolicy policy, PolicyAuthorizationResult authorizeResult)
at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
at WB.UI.Headquarters.Code.Workspaces.WorkspaceRedirectMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Builder.Extensions.MapMiddleware.Invoke(HttpContext context)
at WB.UI.Headquarters.Metrics.HeadquartersHttpMetricsMiddleware.<>c__DisplayClass5_0.<g__Next|0>d.MoveNext()
— End of stack trace from previous location —
at WB.UI.Headquarters.Metrics.HeadquartersHttpMetricsMiddleware.Invoke(HttpContext context)
at WB.UI.Headquarters.Code.Workspaces.WorkspaceMiddleware.<>c__DisplayClass2_0.<g__InvokeNextWithScope|0>d.MoveNext()
— End of stack trace from previous location —
at WB.UI.Headquarters.Code.Workspaces.WorkspaceMiddleware.Invoke(HttpContext context)
at Serilog.AspNetCore.RequestLoggingMiddleware.Invoke(HttpContext httpContext)
2021-03-30 17:48:50.886 +02:00 [ERR] An unhandled exception has occurred, logging to Exceptional
Microsoft.AspNetCore.Connections.ConnectionResetException: Eine vorhandene Verbindung wurde vom Remotehost geschlossen.
—> System.Net.Sockets.SocketException (10054): Eine vorhandene Verbindung wurde vom Remotehost geschlossen.
at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.Internal.SocketAwaitableEventArgs.g__ThrowSocketException|7_0(SocketError e)
at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.Internal.SocketAwaitableEventArgs.GetResult()
at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.Internal.SocketConnection.ProcessReceives()
at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.Internal.SocketConnection.DoReceive()
— End of inner exception stack trace —
at System.IO.Pipelines.PipeCompletion.ThrowLatchedException()
at System.IO.Pipelines.Pipe.GetReadResult(ReadResult& result)
at System.IO.Pipelines.Pipe.GetReadAsyncResult()
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1ChunkedEncodingMessageBody.PumpAsync()
at System.IO.Pipelines.PipeCompletion.ThrowLatchedException()
at System.IO.Pipelines.Pipe.GetReadResult(ReadResult& result)
at System.IO.Pipelines.Pipe.GetReadAsyncResult()
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1ChunkedEncodingMessageBody.ReadAsyncInternal(CancellationToken cancellationToken)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpRequestStream.ReadAsyncInternal(Memory1 destination, CancellationToken cancellationToken) at System.IO.Compression.DeflateStream.FinishReadAsyncMemory(ValueTask1 readTask, Memory1 buffer, CancellationToken cancellationToken) at Microsoft.AspNetCore.WebUtilities.FileBufferingReadStream.ReadAsync(Memory1 buffer, CancellationToken cancellationToken)
at Microsoft.AspNetCore.WebUtilities.StreamHelperExtensions.DrainAsync(Stream stream, ArrayPool1 bytePool, Nullable1 limit, CancellationToken cancellationToken)
at Microsoft.AspNetCore.Mvc.Formatters.NewtonsoftJsonInputFormatter.ReadRequestBodyAsync(InputFormatterContext context, Encoding encoding)
at Microsoft.AspNetCore.Mvc.ModelBinding.Binders.BodyModelBinder.BindModelAsync(ModelBindingContext bindingContext)
at Microsoft.AspNetCore.Mvc.ModelBinding.ParameterBinder.BindModelAsync(ActionContext actionContext, IModelBinder modelBinder, IValueProvider valueProvider, ParameterDescriptor parameter, ModelMetadata metadata, Object value, Object container)
at Microsoft.AspNetCore.Mvc.Controllers.ControllerBinderDelegateProvider.<>c__DisplayClass0_0.<g__Bind|0>d.MoveNext()
— End of stack trace from previous location —
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Awaited|24_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Awaited|19_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
at Microsoft.AspNetCore.Routing.EndpointMiddleware.g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext)
at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.<>c__DisplayClass6_1.<b__1>d.MoveNext()
— End of stack trace from previous location —
at Microsoft.AspNetCore.ResponseCompression.ResponseCompressionMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)
at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider)
at Microsoft.AspNetCore.Authorization.Policy.AuthorizationMiddlewareResultHandler.HandleAsync(RequestDelegate next, HttpContext context, AuthorizationPolicy policy, PolicyAuthorizationResult authorizeResult)
at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
at WB.UI.Headquarters.Code.Workspaces.WorkspaceRedirectMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Builder.Extensions.MapMiddleware.Invoke(HttpContext context)
at WB.UI.Headquarters.Metrics.HeadquartersHttpMetricsMiddleware.<>c__DisplayClass5_0.<g__Next|0>d.MoveNext()
— End of stack trace from previous location —
at WB.UI.Headquarters.Metrics.HeadquartersHttpMetricsMiddleware.Invoke(HttpContext context)
at WB.UI.Headquarters.Code.Workspaces.WorkspaceMiddleware.<>c__DisplayClass2_0.<g__InvokeNextWithScope|0>d.MoveNext()
— End of stack trace from previous location —
at WB.UI.Headquarters.Code.Workspaces.WorkspaceMiddleware.Invoke(HttpContext context)
at Serilog.AspNetCore.RequestLoggingMiddleware.Invoke(HttpContext httpContext)
at Microsoft.AspNetCore.Localization.RequestLocalizationMiddleware.Invoke(HttpContext context)
at StackExchange.Exceptional.ExceptionalMiddleware.Invoke(HttpContext context)
2021-03-30 17:48:50.895 +02:00 [ERR] Connection id “0HM7JH45FN2PQ”, Request id “0HM7JH45FN2PQ:00000014”: An unhandled exception was thrown by the application.
Microsoft.AspNetCore.Connections.ConnectionResetException: Eine vorhandene Verbindung wurde vom Remotehost geschlossen.
—> System.Net.Sockets.SocketException (10054): Eine vorhandene Verbindung wurde vom Remotehost geschlossen.
at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.Internal.SocketAwaitableEventArgs.g__ThrowSocketException|7_0(SocketError e)
at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.Internal.SocketAwaitableEventArgs.GetResult()
at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.Internal.SocketConnection.ProcessReceives()
at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.Internal.SocketConnection.DoReceive()
— End of inner exception stack trace —
at System.IO.Pipelines.PipeCompletion.ThrowLatchedException()
at System.IO.Pipelines.Pipe.GetReadResult(ReadResult& result)
at System.IO.Pipelines.Pipe.GetReadAsyncResult()
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1ChunkedEncodingMessageBody.PumpAsync()
at System.IO.Pipelines.PipeCompletion.ThrowLatchedException()
at System.IO.Pipelines.Pipe.GetReadResult(ReadResult& result)
at System.IO.Pipelines.Pipe.GetReadAsyncResult()
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1ChunkedEncodingMessageBody.ReadAsyncInternal(CancellationToken cancellationToken)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpRequestStream.ReadAsyncInternal(Memory1 destination, CancellationToken cancellationToken) at System.IO.Compression.DeflateStream.FinishReadAsyncMemory(ValueTask1 readTask, Memory1 buffer, CancellationToken cancellationToken) at Microsoft.AspNetCore.WebUtilities.FileBufferingReadStream.ReadAsync(Memory1 buffer, CancellationToken cancellationToken)
at Microsoft.AspNetCore.WebUtilities.StreamHelperExtensions.DrainAsync(Stream stream, ArrayPool1 bytePool, Nullable1 limit, CancellationToken cancellationToken)
at Microsoft.AspNetCore.Mvc.Formatters.NewtonsoftJsonInputFormatter.ReadRequestBodyAsync(InputFormatterContext context, Encoding encoding)
at Microsoft.AspNetCore.Mvc.ModelBinding.Binders.BodyModelBinder.BindModelAsync(ModelBindingContext bindingContext)
at Microsoft.AspNetCore.Mvc.ModelBinding.ParameterBinder.BindModelAsync(ActionContext actionContext, IModelBinder modelBinder, IValueProvider valueProvider, ParameterDescriptor parameter, ModelMetadata metadata, Object value, Object container)
at Microsoft.AspNetCore.Mvc.Controllers.ControllerBinderDelegateProvider.<>c__DisplayClass0_0.<g__Bind|0>d.MoveNext()
— End of stack trace from previous location —
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Awaited|24_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Awaited|19_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
at Microsoft.AspNetCore.Routing.EndpointMiddleware.g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext)
at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.<>c__DisplayClass6_1.<b__1>d.MoveNext()
— End of stack trace from previous location —
at Microsoft.AspNetCore.ResponseCompression.ResponseCompressionMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)
at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider)
at Microsoft.AspNetCore.Authorization.Policy.AuthorizationMiddlewareResultHandler.HandleAsync(RequestDelegate next, HttpContext context, AuthorizationPolicy policy, PolicyAuthorizationResult authorizeResult)
at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
at WB.UI.Headquarters.Code.Workspaces.WorkspaceRedirectMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Builder.Extensions.MapMiddleware.Invoke(HttpContext context)
at WB.UI.Headquarters.Metrics.HeadquartersHttpMetricsMiddleware.<>c__DisplayClass5_0.<g__Next|0>d.MoveNext()
— End of stack trace from previous location —
at WB.UI.Headquarters.Metrics.HeadquartersHttpMetricsMiddleware.Invoke(HttpContext context)
at WB.UI.Headquarters.Code.Workspaces.WorkspaceMiddleware.<>c__DisplayClass2_0.<g__InvokeNextWithScope|0>d.MoveNext()
— End of stack trace from previous location —
at WB.UI.Headquarters.Code.Workspaces.WorkspaceMiddleware.Invoke(HttpContext context)
at Serilog.AspNetCore.RequestLoggingMiddleware.Invoke(HttpContext httpContext)
at Microsoft.AspNetCore.Localization.RequestLocalizationMiddleware.Invoke(HttpContext context)
at StackExchange.Exceptional.ExceptionalMiddleware.Invoke(HttpContext context)
at StackExchange.Exceptional.ExceptionalMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Server.IISIntegration.IISMiddleware.Invoke(HttpContext httpContext)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)
2021-03-30 19:10:25.770 +02:00 [INF] Application is shutting down…

#9

image
image1413×169 35.1 KB

This shows that you may have experienced an error 500 due to a very slow connection. You have a bunch of these in the log.

And then you have forced connection close also with error code 500:

image
image1380×251 44 KB

(in English: An existing connection was forcibly closed by the remote host).

Both mention /image in URL. Are you collecting any images? What size are they?

#10

I talked to our survey manager - we do not have images in the assignments or questionnaires and we do not collect them. The other 3 log files from yesterday did not have error 500, but the issue was still there.

The log file for PostgreSQL shows multiple “Permission denied” errors:
“2021-03-31 09:51:34.899 CEST [5508] LOG: konnte Datei »pg_wal/0000000100000000000000BE« nicht umbenennen: Permission denied” although the data folder is included into exceptions in Windows Defender settings. Do you think this could be the source of the problem? Could you advise how to fix it?

The interesting thing is that mostly the issue occurs if the teams need to sync (~9) new assignments! That is, we removed all existing assignments and new ones were assigned (during training).
Apparently during normal synchronization of interviews the issue is far less pronounced.

#11

Well, if there is anything else that is installed on your server that intervenes with it’s operation then it must be removed or reconfigured.

In this case it is not Survey Solutions that is not working, on the contrary, it tries to do its business, but then the OS tells it “No, I deny you permission to proceed” and Survey Solutions has no other choice then to obey. (in this case actually PostgreSQL, as you found the error logged there, which is a DB that the current versions of Survey Solutions use). I see nothing that we can do to “overcome” the security or other policies of access to folders or files imposed by your system.

But it also could be that the message “…konnte Datei … nicht umbenennen: permission denied” (could not rename the file, permission denied) indicates temporary unavailability, such as when the file is being used by another application. So if you have any smart eyes program that scans the files make sure that it doesn’t block them, or does it when the server is stopped. Because merely by looking at the file it may intervene with the program’s operation.

I imagine that there could be a rare twist of events where Survey Solutions could somehow deadlock itself by locking own files, but this is not something that would be consistently happening on one machine and not happening on others. Yet if there is evidence that this is what’s happening or a particular scenario how this can be reproduced, we need to be notified about this.

If you remove ALL the assignments, then the questionnaire is also removed and to receive a new assignment the questionnaire is also sent. So sending the first assignment may take multiple megabytes and the second assignment only a few bytes, precisely because a questionnaire is required. An interview is also rather small, unless it has images, audio audit (binary data).