GraphQL error: Object reference not set to an instance of an object

On a docker deployment of 21.1.8 that shows all green on the diagnostics screen I am getting a lot of red error pop-ups in the top-right corner, telling me that there was a “GraphQL error: Object reference not set to an instance of an object”. The corresponding POST actually returns as HTTP 200 with a structure of:

{
    "errors": [
        {
            "message": "Object reference not set to an instance of an object."
        },
        {
            "message": "Object reference not set to an instance of an object."
        },
        {
            "message": "Object reference not set to an instance of an object."
        },
        {
            "message": "Object reference not set to an instance of an object."
        },
        {
            "message": "Object reference not set to an instance of an object."
        },
        {
            "message": "Object reference not set to an instance of an object."
        },
        {
            "message": "Object reference not set to an instance of an object."
        },
        {
            "message": "Object reference not set to an instance of an object."
        },
        {
            "message": "Object reference not set to an instance of an object."
        },
        {
            "message": "Object reference not set to an instance of an object."
        },
        {
            "message": "Object reference not set to an instance of an object."
        },
        {
            "message": "Object reference not set to an instance of an object."
        }
    ],
    "data": {
        "interviews": {
            "totalCount": 17,
            "filteredCount": 17,
            "nodes": [
                …
            ],
            "__typename": "IPagedConnectionOfInterview"
        }
    }
}

This is from the Interviews screen. The data insode the nodes array actually looks OK.This seems to happen erratically and reloading or triggering another search by modifying the text in the search box will often eventually work. It feels a bit like a race condition.

The server log says:

hq_1        | [13:03:49 ERR] Headquarters /interviews/nodes[2]/identifyingData[0]/entityUnexpected Execution Error
hq_1        | 
hq_1        | System.NullReferenceException: Object reference not set to an instance of an object.
hq_1        |    at HotChocolate.Execution.Processing.MiddlewareContext.Service[T]()
hq_1        |    at WB.UI.Headquarters.Controllers.Api.PublicApi.Graphql.Interviews.AnswerObjectType.<>c__DisplayClass0_0.<<Configure>b__4>d.MoveNext() in /src/UI/WB.UI.Headquarters.Core/Controllers/Api/PublicApi/Graphql/Interviews/AnswerObjectType.cs:line 39
hq_1        | --- End of stack trace from previous location ---
hq_1        |    at HotChocolate.DataLoader.BatchDataLoader`2.FetchAsync(IReadOnlyList`1 keys, CancellationToken cancellationToken)
hq_1        |    at GreenDonut.DataLoaderBase`2.<>c__DisplayClass25_0.<<DispatchBatchAsync>b__0>d.MoveNext()
hq_1        | --- End of stack trace from previous location ---
hq_1        |    at HotChocolate.Types.ResolverObjectFieldDescriptorExtensions.<>c__DisplayClass3_0`1.<<Resolver>b__0>d.MoveNext()
hq_1        | --- End of stack trace from previous location ---
hq_1        |    at HotChocolate.Types.FieldMiddlewareCompiler.<>c__DisplayClass3_0.<<CreateResolverMiddleware>b__0>d.MoveNext()
hq_1        | --- End of stack trace from previous location ---
hq_1        |    at WB.UI.Headquarters.Controllers.Api.PublicApi.Graphql.WorkspaceGraphQlMiddleware.InvokeAsync(IMiddlewareContext context, IWorkspaceContextAccessor workspaceContextAccessor, IWorkspaceContextSetter workspaceContextSetter) in /src/UI/WB.UI.Headquarters.Core/Controllers/Api/PublicApi/Graphql/WorkspaceGraphQlMiddleware.cs:line 35
hq_1        |    at HotChocolate.Utilities.MiddlewareCompiler`1.ExpressionHelper.AwaitTaskHelper(Task task)
hq_1        |    at HotChocolate.Execution.Processing.ResolverTask.ExecuteResolverPipelineAsync(CancellationToken cancellationToken)
hq_1        |    at HotChocolate.Execution.Processing.ResolverTask.TryExecuteAsync(CancellationToken cancellationToken)
hq_1        | [13:03:49 ERR] Headquarters nullObject reference not set to an instance of an object.

Any hints?

Hello @tschoel ,and sorry to hear you are experiencing troubles.

For the moment I can only confirm that this has been a second report of this kind with the GraphQL error reported on a page, which is supposed to show the list of the interviews. The other report mentions the same version 21.01.8 and the same setup (in Docker). However, no systematic pattern has been revealed to date on what may be causing it.

Based on the exception texts that you’ve sent, I don’t recollect seeing entityUnexpected mentioned earlier. Anything specific about the preloaded data?

Hi Sergiy,

It is data from the questionnaire 441498506a5a4b05b895dd83feff65d8. Not sure if you can access it. If you cannot, please let me know where I can send the export. This questionnaire has quite a few computed variables and a translation.

Also, as I mentioned earlier, this happens very erratically, sometimes all goes well, sometimes all interviews fail to load, sometimes an arbitrary number fail to load while the others succeed.

Hope this helps.

Dear @tschoel,

unfortunately I couldn’t observe the described GraphQL-related error with this questionnaire in a controlled environment. This doesn’t prove anything, since you are describing that the error appears only episodically.

If anyone is also experiencing the same symptoms, please jump into the conversation with your details.

Best, Sergiy

Here are a few additional observations from a colleague who does some more day-to-day work with the system:

It appears to be isolated to the Interviews menu. The pattern seems to be roughly this:

  1. Click on Interviews menu (new session)
  2. Intermittent appearance of GraphQL error (approx. 40% of the time) accompanied by ‘No data available in the table’ message
  3. Usually fixed by selecting any of the filters (table loads again normally) but very rarely this will trigger another GraphQL error message
  4. Further activity within the menu does not appear to trigger further errors (i.e. you can go wild on filters back and forth and no problems)
  5. Click on any other menu (e.g. Assignments) then return to Interviews and the error appears at the same intermittent frequency

Additionally, one slightly different error message I have observed infrequently seems to refer to a questionnaire (ref. attached 2.jpg):

He is referring to the “could not execute query” messages starting about halfway through the screen.

And here is the log for one of these instances with the SQL error:

[16:02:27 ERR] Headquarters /interviews/nodes[0]/identifyingData[0]/entityUnexpected Execution Error

System.NullReferenceException: Object reference not set to an instance of an object.
   at HotChocolate.Execution.Processing.MiddlewareContext.Service[T]()
   at WB.UI.Headquarters.Controllers.Api.PublicApi.Graphql.Interviews.AnswerObjectType.<>c__DisplayClass0_0.<<Configure>b__9>d.MoveNext() in /src/UI/WB.UI.Headquarters.Core/Controllers/Api/PublicApi/Graphql/Interviews/AnswerObjectType.cs:line 41
--- End of stack trace from previous location ---
   at HotChocolate.DataLoader.BatchDataLoader`2.FetchAsync(IReadOnlyList`1 keys, CancellationToken cancellationToken)
   at GreenDonut.DataLoaderBase`2.<>c__DisplayClass25_0.<<DispatchBatchAsync>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at WB.UI.Headquarters.Controllers.Api.PublicApi.Graphql.Interviews.AnswerObjectType.<>c.<<Configure>b__0_2>d.MoveNext() in /src/UI/WB.UI.Headquarters.Core/Controllers/Api/PublicApi/Graphql/Interviews/AnswerObjectType.cs:line 43
--- End of stack trace from previous location ---
   at HotChocolate.Types.FieldMiddlewareCompiler.<>c__DisplayClass3_0.<<CreateResolverMiddleware>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at WB.UI.Headquarters.Controllers.Api.PublicApi.Graphql.WorkspaceGraphQlMiddleware.InvokeAsync(IMiddlewareContext context, IWorkspaceContextAccessor workspaceContextAccessor, IWorkspaceContextSetter workspaceContextSetter) in /src/UI/WB.UI.Headquarters.Core/Controllers/Api/PublicApi/Graphql/WorkspaceGraphQlMiddleware.cs:line 35
   at HotChocolate.Utilities.MiddlewareCompiler`1.ExpressionHelper.AwaitTaskHelper(Task task)
   at HotChocolate.Execution.Processing.ResolverTask.ExecuteResolverPipelineAsync(CancellationToken cancellationToken)
   at HotChocolate.Execution.Processing.ResolverTask.TryExecuteAsync(CancellationToken cancellationToken)
[16:02:27 ERR] Headquarters /interviews/nodes[2]/identifyingData[5]/entityUnexpected Execution Error

NHibernate.Exceptions.GenericADOException: could not execute query
[ select questionna0_.Id as id1_30_, questionna0_.QuestionnaireIdentity as questionnaireidentity2_30_, questionna0_.EntityId as entityid3_30_, questionna0_.ParentId as parentid4_30_, questionna0_.question_type as question5_30_, questionna0_.Featured as featured6_30_, questionna0_.question_scope as question7_30_, questionna0_.entity_type as entity8_30_, questionna0_.is_filtered_combobox as is9_30_, questionna0_.cascade_from_question_id as cascade10_30_, questionna0_.linked_to_roster_id as linked11_30_, questionna0_.linked_to_question_id as linked12_30_, questionna0_.stata_export_caption as stata13_30_, questionna0_.question_text as question14_30_, questionna0_.variable_label as variable15_30_, questionna0_.variable_type as variable16_30_, questionna0_.included_in_reporting_at_utc as included17_30_ from questionnaire_entities questionna0_ where questionna0_.Id in (:p0 , :p1 , :p2 , :p3 , :p4 , :p5) ]
  Name:p1_0_ - Value:2578  Name:p1_1_ - Value:2577  Name:p1_2_ - Value:2576  Name:p1_3_ - Value:2575  Name:p1_4_ - Value:2574  Name:p1_5_ - Value:2573
[SQL: select questionna0_.Id as id1_30_, questionna0_.QuestionnaireIdentity as questionnaireidentity2_30_, questionna0_.EntityId as entityid3_30_, questionna0_.ParentId as parentid4_30_, questionna0_.question_type as question5_30_, questionna0_.Featured as featured6_30_, questionna0_.question_scope as question7_30_, questionna0_.entity_type as entity8_30_, questionna0_.is_filtered_combobox as is9_30_, questionna0_.cascade_from_question_id as cascade10_30_, questionna0_.linked_to_roster_id as linked11_30_, questionna0_.linked_to_question_id as linked12_30_, questionna0_.stata_export_caption as stata13_30_, questionna0_.question_text as question14_30_, questionna0_.variable_label as variable15_30_, questionna0_.variable_type as variable16_30_, questionna0_.included_in_reporting_at_utc as included17_30_ from questionnaire_entities questionna0_ where questionna0_.Id in (:p0 , :p1 , :p2 , :p3 , :p4 , :p5)]
 ---> Npgsql.NpgsqlOperationInProgressException (0x80004005): A command is already in progress: select questionna0_.Id as id1_30_, questionna0_.QuestionnaireIdentity as questionnaireidentity2_30_, questionna0_.EntityId as entityid3_30_, questionna0_.ParentId as parentid4_30_, questionna0_.question_type as question5_30_, questionna0_.Featured as featured6_30_, questionna0_.question_scope as question7_30_, questionna0_.entity_type as entity8_30_, questionna0_.is_filtered_combobox as is9_30_, questionna0_.cascade_from_question_id as cascade10_30_, questionna0_.linked_to_roster_id as linked11_30_, questionna0_.linked_to_question_id as linked12_30_, questionna0_.stata_export_caption as stata13_30_, questionna0_.question_text as question14_30_, questionna0_.variable_label as variable15_30_, questionna0_.variable_type as variable16_30_, questionna0_.included_in_reporting_at_utc as included17_30_ from questionnaire_entities questionna0_ where questionna0_.Id in (:p0 , :p1 , :p2)
   at Npgsql.NpgsqlConnector.<StartUserAction>g__DoStartUserAction|233_0(<>c__DisplayClass233_0& )
   at Npgsql.NpgsqlConnector.StartUserAction(ConnectorState newState, NpgsqlCommand command, CancellationToken cancellationToken, Boolean attemptPgCancellation)
   at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken)
   at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken)
   at Npgsql.NpgsqlCommand.ExecuteDbDataReaderAsync(CommandBehavior behavior, CancellationToken cancellationToken)
   at NHibernate.AdoNet.AbstractBatcher.DoExecuteReaderAsync(DbCommand cmd, CancellationToken cancellationToken)
   at NHibernate.AdoNet.AbstractBatcher.ExecuteReaderAsync(DbCommand cmd, CancellationToken cancellationToken)
   at NHibernate.Loader.Loader.GetResultSetAsync(DbCommand st, QueryParameters queryParameters, ISessionImplementor session, IResultTransformer forcedResultTransformer, CancellationToken cancellationToken)
   at NHibernate.Loader.Loader.DoQueryAsync(ISessionImplementor session, QueryParameters queryParameters, Boolean returnProxies, IResultTransformer forcedResultTransformer, QueryCacheResultBuilder queryCacheResultBuilder, CancellationToken cancellationToken)
   at NHibernate.Loader.Loader.DoQueryAndInitializeNonLazyCollectionsAsync(ISessionImplementor session, QueryParameters queryParameters, Boolean returnProxies, IResultTransformer forcedResultTransformer, QueryCacheResultBuilder queryCacheResultBuilder, CancellationToken cancellationToken)
   at NHibernate.Loader.Loader.DoListAsync(ISessionImplementor session, QueryParameters queryParameters, IResultTransformer forcedResultTransformer, QueryCacheResultBuilder queryCacheResultBuilder, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at NHibernate.Loader.Loader.DoListAsync(ISessionImplementor session, QueryParameters queryParameters, IResultTransformer forcedResultTransformer, QueryCacheResultBuilder queryCacheResultBuilder, CancellationToken cancellationToken)
   at NHibernate.Loader.Loader.ListIgnoreQueryCacheAsync(ISessionImplementor session, QueryParameters queryParameters, CancellationToken cancellationToken)
   at NHibernate.Hql.Ast.ANTLR.QueryTranslatorImpl.ListAsync(ISessionImplementor session, QueryParameters queryParameters, CancellationToken cancellationToken)
   at NHibernate.Engine.Query.HQLQueryPlan.PerformListAsync(QueryParameters queryParameters, ISessionImplementor session, IList results, CancellationToken cancellationToken)
   at NHibernate.Impl.SessionImpl.ListAsync(IQueryExpression queryExpression, QueryParameters queryParameters, IList results, Object filterConnection, CancellationToken cancellationToken)
   at NHibernate.Impl.SessionImpl.ListAsync(IQueryExpression queryExpression, QueryParameters queryParameters, IList results, Object filterConnection, CancellationToken cancellationToken)
   at NHibernate.Impl.AbstractSessionImpl.ListAsync[T](IQueryExpression query, QueryParameters parameters, CancellationToken cancellationToken)
   at NHibernate.Impl.AbstractQueryImpl2.ListAsync[T](CancellationToken cancellationToken)
   at NHibernate.Linq.DefaultQueryProvider.ExecuteListAsync[TResult](Expression expression, CancellationToken cancellationToken)
   at NHibernate.Linq.LinqExtensionMethods.<>c__DisplayClass59_0`1.<<ToListAsync>g__InternalToListAsync|0>d.MoveNext()
--- End of stack trace from previous location ---
   at WB.UI.Headquarters.Controllers.Api.PublicApi.Graphql.Interviews.AnswerObjectType.<>c__DisplayClass0_0.<<Configure>b__9>d.MoveNext() in /src/UI/WB.UI.Headquarters.Core/Controllers/Api/PublicApi/Graphql/Interviews/AnswerObjectType.cs:line 41
--- End of stack trace from previous location ---
   at HotChocolate.DataLoader.BatchDataLoader`2.FetchAsync(IReadOnlyList`1 keys, CancellationToken cancellationToken)
   at GreenDonut.DataLoaderBase`2.<>c__DisplayClass25_0.<<DispatchBatchAsync>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at WB.UI.Headquarters.Controllers.Api.PublicApi.Graphql.Interviews.AnswerObjectType.<>c.<<Configure>b__0_2>d.MoveNext() in /src/UI/WB.UI.Headquarters.Core/Controllers/Api/PublicApi/Graphql/Interviews/AnswerObjectType.cs:line 43
--- End of stack trace from previous location ---
   at HotChocolate.Types.FieldMiddlewareCompiler.<>c__DisplayClass3_0.<<CreateResolverMiddleware>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at WB.UI.Headquarters.Controllers.Api.PublicApi.Graphql.WorkspaceGraphQlMiddleware.InvokeAsync(IMiddlewareContext context, IWorkspaceContextAccessor workspaceContextAccessor, IWorkspaceContextSetter workspaceContextSetter) in /src/UI/WB.UI.Headquarters.Core/Controllers/Api/PublicApi/Graphql/WorkspaceGraphQlMiddleware.cs:line 35
   at HotChocolate.Utilities.MiddlewareCompiler`1.ExpressionHelper.AwaitTaskHelper(Task task)
   at HotChocolate.Execution.Processing.ResolverTask.ExecuteResolverPipelineAsync(CancellationToken cancellationToken)
   at HotChocolate.Execution.Processing.ResolverTask.TryExecuteAsync(CancellationToken cancellationToken)
   at WB.UI.Headquarters.Controllers.Api.PublicApi.Graphql.Interviews.AnswerObjectType.<>c.<<Configure>b__0_2>d.MoveNext() in /src/UI/WB.UI.Headquarters.Core/Controllers/Api/PublicApi/Graphql/Interviews/AnswerObjectType.cs:line 43
--- End of stack trace from previous location ---
   at HotChocolate.Types.FieldMiddlewareCompiler.<>c__DisplayClass3_0.<<CreateResolverMiddleware>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at WB.UI.Headquarters.Controllers.Api.PublicApi.Graphql.WorkspaceGraphQlMiddleware.InvokeAsync(IMiddlewareContext context, IWorkspaceContextAccessor workspaceContextAccessor, IWorkspaceContextSetter workspaceContextSetter) in /src/UI/WB.UI.Headquarters.Core/Controllers/Api/PublicApi/Graphql/WorkspaceGraphQlMiddleware.cs:line 35
   at HotChocolate.Utilities.MiddlewareCompiler`1.ExpressionHelper.AwaitTaskHelper(Task task)
   at HotChocolate.Execution.Processing.ResolverTask.ExecuteResolverPipelineAsync(CancellationToken cancellationToken)
   at HotChocolate.Execution.Processing.ResolverTask.TryExecuteAsync(CancellationToken cancellationToken)
[16:02:27 ERR] Headquarters nullObject reference not set to an instance of an object.

[16:02:27 ERR] Headquarters /interviews/nodes[2]/identifyingData[4]/entityUnexpected Execution Error

NHibernate.Exceptions.GenericADOException: could not execute query
[ select questionna0_.Id as id1_30_, questionna0_.QuestionnaireIdentity as questionnaireidentity2_30_, questionna0_.EntityId as entityid3_30_, questionna0_.ParentId as parentid4_30_, questionna0_.question_type as question5_30_, questionna0_.Featured as featured6_30_, questionna0_.question_scope as question7_30_, questionna0_.entity_type as entity8_30_, questionna0_.is_filtered_combobox as is9_30_, questionna0_.cascade_from_question_id as cascade10_30_, questionna0_.linked_to_roster_id as linked11_30_, questionna0_.linked_to_question_id as linked12_30_, questionna0_.stata_export_caption as stata13_30_, questionna0_.question_text as question14_30_, questionna0_.variable_label as variable15_30_, questionna0_.variable_type as variable16_30_, questionna0_.included_in_reporting_at_utc as included17_30_ from questionnaire_entities questionna0_ where questionna0_.Id in (:p0 , :p1 , :p2 , :p3 , :p4 , :p5) ]
  Name:p1_0_ - Value:2578  Name:p1_1_ - Value:2577  Name:p1_2_ - Value:2576  Name:p1_3_ - Value:2575  Name:p1_4_ - Value:2574  Name:p1_5_ - Value:2573
[SQL: select questionna0_.Id as id1_30_, questionna0_.QuestionnaireIdentity as questionnaireidentity2_30_, questionna0_.EntityId as entityid3_30_, questionna0_.ParentId as parentid4_30_, questionna0_.question_type as question5_30_, questionna0_.Featured as featured6_30_, questionna0_.question_scope as question7_30_, questionna0_.entity_type as entity8_30_, questionna0_.is_filtered_combobox as is9_30_, questionna0_.cascade_from_question_id as cascade10_30_, questionna0_.linked_to_roster_id as linked11_30_, questionna0_.linked_to_question_id as linked12_30_, questionna0_.stata_export_caption as stata13_30_, questionna0_.question_text as question14_30_, questionna0_.variable_label as variable15_30_, questionna0_.variable_type as variable16_30_, questionna0_.included_in_reporting_at_utc as included17_30_ from questionnaire_entities questionna0_ where questionna0_.Id in (:p0 , :p1 , :p2 , :p3 , :p4 , :p5)]
 ---> Npgsql.NpgsqlOperationInProgressException (0x80004005): A command is already in progress: select questionna0_.Id as id1_30_, questionna0_.QuestionnaireIdentity as questionnaireidentity2_30_, questionna0_.EntityId as entityid3_30_, questionna0_.ParentId as parentid4_30_, questionna0_.question_type as question5_30_, questionna0_.Featured as featured6_30_, questionna0_.question_scope as question7_30_, questionna0_.entity_type as entity8_30_, questionna0_.is_filtered_combobox as is9_30_, questionna0_.cascade_from_question_id as cascade10_30_, questionna0_.linked_to_roster_id as linked11_30_, questionna0_.linked_to_question_id as linked12_30_, questionna0_.stata_export_caption as stata13_30_, questionna0_.question_text as question14_30_, questionna0_.variable_label as variable15_30_, questionna0_.variable_type as variable16_30_, questionna0_.included_in_reporting_at_utc as included17_30_ from questionnaire_entities questionna0_ where questionna0_.Id in (:p0 , :p1 , :p2)
   at Npgsql.NpgsqlConnector.<StartUserAction>g__DoStartUserAction|233_0(<>c__DisplayClass233_0& )
   at Npgsql.NpgsqlConnector.StartUserAction(ConnectorState newState, NpgsqlCommand command, CancellationToken cancellationToken, Boolean attemptPgCancellation)
   at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken)
   at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken)
   at Npgsql.NpgsqlCommand.ExecuteDbDataReaderAsync(CommandBehavior behavior, CancellationToken cancellationToken)
   at NHibernate.AdoNet.AbstractBatcher.DoExecuteReaderAsync(DbCommand cmd, CancellationToken cancellationToken)
   at NHibernate.AdoNet.AbstractBatcher.ExecuteReaderAsync(DbCommand cmd, CancellationToken cancellationToken)
   at NHibernate.Loader.Loader.GetResultSetAsync(DbCommand st, QueryParameters queryParameters, ISessionImplementor session, IResultTransformer forcedResultTransformer, CancellationToken cancellationToken)
   at NHibernate.Loader.Loader.DoQueryAsync(ISessionImplementor session, QueryParameters queryParameters, Boolean returnProxies, IResultTransformer forcedResultTransformer, QueryCacheResultBuilder queryCacheResultBuilder, CancellationToken cancellationToken)
   at NHibernate.Loader.Loader.DoQueryAndInitializeNonLazyCollectionsAsync(ISessionImplementor session, QueryParameters queryParameters, Boolean returnProxies, IResultTransformer forcedResultTransformer, QueryCacheResultBuilder queryCacheResultBuilder, CancellationToken cancellationToken)
   at NHibernate.Loader.Loader.DoListAsync(ISessionImplementor session, QueryParameters queryParameters, IResultTransformer forcedResultTransformer, QueryCacheResultBuilder queryCacheResultBuilder, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at NHibernate.Loader.Loader.DoListAsync(ISessionImplementor session, QueryParameters queryParameters, IResultTransformer forcedResultTransformer, QueryCacheResultBuilder queryCacheResultBuilder, CancellationToken cancellationToken)
   at NHibernate.Loader.Loader.ListIgnoreQueryCacheAsync(ISessionImplementor session, QueryParameters queryParameters, CancellationToken cancellationToken)
   at NHibernate.Hql.Ast.ANTLR.QueryTranslatorImpl.ListAsync(ISessionImplementor session, QueryParameters queryParameters, CancellationToken cancellationToken)
   at NHibernate.Engine.Query.HQLQueryPlan.PerformListAsync(QueryParameters queryParameters, ISessionImplementor session, IList results, CancellationToken cancellationToken)
   at NHibernate.Impl.SessionImpl.ListAsync(IQueryExpression queryExpression, QueryParameters queryParameters, IList results, Object filterConnection, CancellationToken cancellationToken)
   at NHibernate.Impl.SessionImpl.ListAsync(IQueryExpression queryExpression, QueryParameters queryParameters, IList results, Object filterConnection, CancellationToken cancellationToken)
   at NHibernate.Impl.AbstractSessionImpl.ListAsync[T](IQueryExpression query, QueryParameters parameters, CancellationToken cancellationToken)
   at NHibernate.Impl.AbstractQueryImpl2.ListAsync[T](CancellationToken cancellationToken)
   at NHibernate.Linq.DefaultQueryProvider.ExecuteListAsync[TResult](Expression expression, CancellationToken cancellationToken)
   at NHibernate.Linq.LinqExtensionMethods.<>c__DisplayClass59_0`1.<<ToListAsync>g__InternalToListAsync|0>d.MoveNext()
--- End of stack trace from previous location ---
   at WB.UI.Headquarters.Controllers.Api.PublicApi.Graphql.Interviews.AnswerObjectType.<>c__DisplayClass0_0.<<Configure>b__9>d.MoveNext() in /src/UI/WB.UI.Headquarters.Core/Controllers/Api/PublicApi/Graphql/Interviews/AnswerObjectType.cs:line 41
--- End of stack trace from previous location ---
   at HotChocolate.DataLoader.BatchDataLoader`2.FetchAsync(IReadOnlyList`1 keys, CancellationToken cancellationToken)
   at GreenDonut.DataLoaderBase`2.<>c__DisplayClass25_0.<<DispatchBatchAsync>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at WB.UI.Headquarters.Controllers.Api.PublicApi.Graphql.Interviews.AnswerObjectType.<>c.<<Configure>b__0_2>d.MoveNext() in /src/UI/WB.UI.Headquarters.Core/Controllers/Api/PublicApi/Graphql/Interviews/AnswerObjectType.cs:line 43
--- End of stack trace from previous location ---
   at HotChocolate.Types.FieldMiddlewareCompiler.<>c__DisplayClass3_0.<<CreateResolverMiddleware>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at WB.UI.Headquarters.Controllers.Api.PublicApi.Graphql.WorkspaceGraphQlMiddleware.InvokeAsync(IMiddlewareContext context, IWorkspaceContextAccessor workspaceContextAccessor, IWorkspaceContextSetter workspaceContextSetter) in /src/UI/WB.UI.Headquarters.Core/Controllers/Api/PublicApi/Graphql/WorkspaceGraphQlMiddleware.cs:line 35
   at HotChocolate.Utilities.MiddlewareCompiler`1.ExpressionHelper.AwaitTaskHelper(Task task)
   at HotChocolate.Execution.Processing.ResolverTask.ExecuteResolverPipelineAsync(CancellationToken cancellationToken)
   at HotChocolate.Execution.Processing.ResolverTask.TryExecuteAsync(CancellationToken cancellationToken)
   at WB.UI.Headquarters.Controllers.Api.PublicApi.Graphql.Interviews.AnswerObjectType.<>c.<<Configure>b__0_2>d.MoveNext() in /src/UI/WB.UI.Headquarters.Core/Controllers/Api/PublicApi/Graphql/Interviews/AnswerObjectType.cs:line 43
--- End of stack trace from previous location ---
   at HotChocolate.Types.FieldMiddlewareCompiler.<>c__DisplayClass3_0.<<CreateResolverMiddleware>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at WB.UI.Headquarters.Controllers.Api.PublicApi.Graphql.WorkspaceGraphQlMiddleware.InvokeAsync(IMiddlewareContext context, IWorkspaceContextAccessor workspaceContextAccessor, IWorkspaceContextSetter workspaceContextSetter) in /src/UI/WB.UI.Headquarters.Core/Controllers/Api/PublicApi/Graphql/WorkspaceGraphQlMiddleware.cs:line 35
   at HotChocolate.Utilities.MiddlewareCompiler`1.ExpressionHelper.AwaitTaskHelper(Task task)
   at HotChocolate.Execution.Processing.ResolverTask.ExecuteResolverPipelineAsync(CancellationToken cancellationToken)
   at HotChocolate.Execution.Processing.ResolverTask.TryExecuteAsync(CancellationToken cancellationToken)
[16:02:27 ERR] Headquarters nullcould not execute query
[ select questionna0_.Id as id1_30_, questionna0_.QuestionnaireIdentity as questionnaireidentity2_30_, questionna0_.EntityId as entityid3_30_, questionna0_.ParentId as parentid4_30_, questionna0_.question_type as question5_30_, questionna0_.Featured as featured6_30_, questionna0_.question_scope as question7_30_, questionna0_.entity_type as entity8_30_, questionna0_.is_filtered_combobox as is9_30_, questionna0_.cascade_from_question_id as cascade10_30_, questionna0_.linked_to_roster_id as linked11_30_, questionna0_.linked_to_question_id as linked12_30_, questionna0_.stata_export_caption as stata13_30_, questionna0_.question_text as question14_30_, questionna0_.variable_label as variable15_30_, questionna0_.variable_type as variable16_30_, questionna0_.included_in_reporting_at_utc as included17_30_ from questionnaire_entities questionna0_ where questionna0_.Id in (:p0 , :p1 , :p2 , :p3 , :p4 , :p5) ]
  Name:p1_0_ - Value:2578  Name:p1_1_ - Value:2577  Name:p1_2_ - Value:2576  Name:p1_3_ - Value:2575  Name:p1_4_ - Value:2574  Name:p1_5_ - Value:2573
[SQL: select questionna0_.Id as id1_30_, questionna0_.QuestionnaireIdentity as questionnaireidentity2_30_, questionna0_.EntityId as entityid3_30_, questionna0_.ParentId as parentid4_30_, questionna0_.question_type as question5_30_, questionna0_.Featured as featured6_30_, questionna0_.question_scope as question7_30_, questionna0_.entity_type as entity8_30_, questionna0_.is_filtered_combobox as is9_30_, questionna0_.cascade_from_question_id as cascade10_30_, questionna0_.linked_to_roster_id as linked11_30_, questionna0_.linked_to_question_id as linked12_30_, questionna0_.stata_export_caption as stata13_30_, questionna0_.question_text as question14_30_, questionna0_.variable_label as variable15_30_, questionna0_.variable_type as variable16_30_, questionna0_.included_in_reporting_at_utc as included17_30_ from questionnaire_entities questionna0_ where questionna0_.Id in (:p0 , :p1 , :p2 , :p3 , :p4 , :p5)]

[16:02:27 ERR] Headquarters /interviews/nodes[1]/identifyingData[0]/entityUnexpected Execution Error

… and so on (sorry, the forum won’t let me post the full thing).

Hi

Facing similar issue using docker containers as well:

  • Survey Solution version 21.06.3
  • Postgres version 13.2 (with max_connections = 200 and shared_buffers = 512MB)
  • Requests are being passed through by nginx

From the server logs:

[03:40:00 DBG] Headquarters AuthenticationScheme: AuthToken was successfully authenticated.
[03:40:00 ERR] Headquarters /interviews/nodes[9]/identifyingData[3]/entityUnexpected Execution Error

System.NullReferenceException: Object reference not set to an instance of an object.
   at HotChocolate.Execution.Processing.MiddlewareContext.Service[T]()
   at WB.UI.Headquarters.Controllers.Api.PublicApi.Graphql.Interviews.AnswerObjectType.<>c__DisplayClass0_0.<<Configure>b__9>d.MoveNext() in /src/UI/WB.UI.Headquarters.Core/Controllers/Api/PublicApi/Graphql/Interviews/AnswerObjectType.cs:line 41
--- End of stack trace from previous location ---
   at HotChocolate.DataLoader.BatchDataLoader`2.FetchAsync(IReadOnlyList`1 keys, CancellationToken cancellationToken)
   at GreenDonut.DataLoaderBase`2.<>c__DisplayClass25_0.<<DispatchBatchAsync>b__0>d.MoveNext()
   at WB.UI.Headquarters.Controllers.Api.PublicApi.Graphql.Interviews.AnswerObjectType.<>c.<<Configure>b__0_2>d.MoveNext() in /src/UI/WB.UI.Headquarters.Core/Controllers/Api/PublicApi/Graphql/Interviews/AnswerObjectType.cs:line 43
--- End of stack trace from previous location ---
   at WB.UI.Headquarters.Controllers.Api.PublicApi.Graphql.Interviews.AnswerObjectType.<>c.<<Configure>b__0_2>d.MoveNext() in /src/UI/WB.UI.Headquarters.Core/Controllers/Api/PublicApi/Graphql/Interviews/AnswerObjectType.cs:line 43
--- End of stack trace from previous location ---
   at WB.UI.Headquarters.Controllers.Api.PublicApi.Graphql.Interviews.AnswerObjectType.<>c.<<Configure>b__0_2>d.MoveNext() in /src/UI/WB.UI.Headquarters.Core/Controllers/Api/PublicApi/Graphql/Interviews/AnswerObjectType.cs:line 43
   at WB.UI.Headquarters.Controllers.Api.PublicApi.Graphql.Interviews.AnswerObjectType.<>c.<<Configure>b__0_2>d.MoveNext() in /src/UI/WB.UI.Headquarters.Core/Controllers/Api/PublicApi/Graphql/Interviews/AnswerObjectType.cs:line 43
--- End of stack trace from previous location ---
   at HotChocolate.Types.FieldMiddlewareCompiler.<>c__DisplayClass3_0.<<CreateResolverMiddleware>b__0>d.MoveNext()
   at WB.UI.Headquarters.Controllers.Api.PublicApi.Graphql.Interviews.AnswerObjectType.<>c.<<Configure>b__0_2>d.MoveNext() in /src/UI/WB.UI.Headquarters.Core/Controllers/Api/PublicApi/Graphql/Interviews/AnswerObjectType.cs:line 43
--- End of stack trace from previous location ---
   at WB.UI.Headquarters.Controllers.Api.PublicApi.Graphql.WorkspaceGraphQlMiddleware.InvokeAsync(IMiddlewareContext context, IWorkspaceContextAccessor workspaceContextAccessor, IWorkspaceContextSetter workspaceContextSetter) in /src/UI/WB.UI.Headquarters.Core/Controllers/Api/PublicApi/Graphql/WorkspaceGraphQlMiddleware.cs:line 58
   at HotChocolate.Types.FieldMiddlewareCompiler.<>c__DisplayClass3_0.<<CreateResolverMiddleware>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at WB.UI.Headquarters.Controllers.Api.PublicApi.Graphql.WorkspaceGraphQlMiddleware.InvokeAsync(IMiddlewareContext context, IWorkspaceContextAccessor workspaceContextAccessor, IWorkspaceContextSetter workspaceContextSetter) in /src/UI/WB.UI.Headquarters.Core/Controllers/Api/PublicApi/Graphql/WorkspaceGraphQlMiddleware.cs:line 58
   at WB.UI.Headquarters.Controllers.Api.PublicApi.Graphql.WorkspaceGraphQlMiddleware.InvokeAsync(IMiddlewareContext context, IWorkspaceContextAccessor workspaceContextAccessor, IWorkspaceContextSetter workspaceContextSetter) in /src/UI/WB.UI.Headquarters.Core/Controllers/Api/PublicApi/Graphql/WorkspaceGraphQlMiddleware.cs:line 58
   at WB.UI.Headquarters.Controllers.Api.PublicApi.Graphql.Interviews.AnswerObjectType.<>c.<<Configure>b__0_2>d.MoveNext() in /src/UI/WB.UI.Headquarters.Core/Controllers/Api/PublicApi/Graphql/Interviews/AnswerObjectType.cs:line 43
   at WB.UI.Headquarters.Controllers.Api.PublicApi.Graphql.Interviews.AnswerObjectType.<>c.<<Configure>b__0_2>d.MoveNext() in /src/UI/WB.UI.Headquarters.Core/Controllers/Api/PublicApi/Graphql/Interviews/AnswerObjectType.cs:line 43
--- End of stack trace from previous location ---
   at WB.UI.Headquarters.Controllers.Api.PublicApi.Graphql.Interviews.AnswerObjectType.<>c.<<Configure>b__0_2>d.MoveNext() in /src/UI/WB.UI.Headquarters.Core/Controllers/Api/PublicApi/Graphql/Interviews/AnswerObjectType.cs:line 43
--- End of stack trace from previous location ---
   at WB.UI.Headquarters.Controllers.Api.PublicApi.Graphql.WorkspaceGraphQlMiddleware.InvokeAsync(IMiddlewareContext context, IWorkspaceContextAccessor workspaceContextAccessor, IWorkspaceContextSetter workspaceContextSetter) in /src/UI/WB.UI.Headquarters.Core/Controllers/Api/PublicApi/Graphql/WorkspaceGraphQlMiddleware.cs:line 58
   at HotChocolate.Types.FieldMiddlewareCompiler.<>c__DisplayClass3_0.<<CreateResolverMiddleware>b__0>d.MoveNext()
   at HotChocolate.Utilities.MiddlewareCompiler`1.ExpressionHelper.AwaitTaskHelper(Task task)
   at WB.UI.Headquarters.Controllers.Api.PublicApi.Graphql.WorkspaceGraphQlMiddleware.InvokeAsync(IMiddlewareContext context, IWorkspaceContextAccessor workspaceContextAccessor, IWorkspaceContextSetter workspaceContextSetter) in /src/UI/WB.UI.Headquarters.Core/Controllers/Api/PublicApi/Graphql/WorkspaceGraphQlMiddleware.cs:line 58
   at HotChocolate.Utilities.MiddlewareCompiler`1.ExpressionHelper.AwaitTaskHelper(Task task)
   at WB.UI.Headquarters.Controllers.Api.PublicApi.Graphql.WorkspaceGraphQlMiddleware.InvokeAsync(IMiddlewareContext context, IWorkspaceContextAccessor workspaceContextAccessor, IWorkspaceContextSetter workspaceContextSetter) in /src/UI/WB.UI.Headquarters.Core/Controllers/Api/PublicApi/Graphql/WorkspaceGraphQlMiddleware.cs:line 58
   at HotChocolate.Execution.Processing.ResolverTask.ExecuteResolverPipelineAsync(CancellationToken cancellationToken)
   at WB.UI.Headquarters.Controllers.Api.PublicApi.Graphql.WorkspaceGraphQlMiddleware.InvokeAsync(IMiddlewareContext context, IWorkspaceContextAccessor workspaceContextAccessor, IWorkspaceContextSetter workspaceContextSetter) in /src/UI/WB.UI.Headquarters.Core/Controllers/Api/PublicApi/Graphql/WorkspaceGraphQlMiddleware.cs:line 58
   at HotChocolate.Execution.Processing.ResolverTask.ExecuteResolverPipelineAsync(CancellationToken cancellationToken)
   at HotChocolate.Execution.Processing.ResolverTask.TryExecuteAsync(CancellationToken cancellationToken)
   at HotChocolate.Execution.Processing.ResolverTask.TryExecuteAsync(CancellationToken cancellationToken)
   at HotChocolate.Execution.Processing.ResolverTask.ExecuteResolverPipelineAsync(CancellationToken cancellationToken)
   at HotChocolate.Execution.Processing.ResolverTask.TryExecuteAsync(CancellationToken cancellationToken)
   at HotChocolate.Utilities.MiddlewareCompiler`1.ExpressionHelper.AwaitTaskHelper(Task task)
   at HotChocolate.Execution.Processing.ResolverTask.ExecuteResolverPipelineAsync(CancellationToken cancellationToken)
   at HotChocolate.Execution.Processing.ResolverTask.TryExecuteAsync(CancellationToken cancellationToken)
   at HotChocolate.Utilities.MiddlewareCompiler`1.ExpressionHelper.AwaitTaskHelper(Task task)
   at HotChocolate.Execution.Processing.ResolverTask.ExecuteResolverPipelineAsync(CancellationToken cancellationToken)
   at HotChocolate.Execution.Processing.ResolverTask.TryExecuteAsync(CancellationToken cancellationToken)
[03:40:00 ERR] Headquarters /interviews/nodes[0]/identifyingData[3]/entityUnexpected Execution Error

System.NullReferenceException: Object reference not set to an instance of an object.
   at HotChocolate.Execution.Processing.MiddlewareContext.Service[T]()
   at WB.UI.Headquarters.Controllers.Api.PublicApi.Graphql.Interviews.AnswerObjectType.<>c__DisplayClass0_0.<<Configure>b__9>d.MoveNext() in /src/UI/WB.UI.Headquarters.Core/Controllers/Api/PublicApi/Graphql/Interviews/AnswerObjectType.cs:line 41
--- End of stack trace from previous location ---
   at HotChocolate.DataLoader.BatchDataLoader`2.FetchAsync(IReadOnlyList`1 keys, CancellationToken cancellationToken)
   at GreenDonut.DataLoaderBase`2.<>c__DisplayClass25_0.<<DispatchBatchAsync>b__0>d.MoveNext()
   at WB.UI.Headquarters.Controllers.Api.PublicApi.Graphql.Interviews.AnswerObjectType.<>c.<<Configure>b__0_2>d.MoveNext() in /src/UI/WB.UI.Headquarters.Core/Controllers/Api/PublicApi/Graphql/Interviews/AnswerObjectType.cs:line 43
--- End of stack trace from previous location ---
   at WB.UI.Headquarters.Controllers.Api.PublicApi.Graphql.Interviews.AnswerObjectType.<>c.<<Configure>b__0_2>d.MoveNext() in /src/UI/WB.UI.Headquarters.Core/Controllers/Api/PublicApi/Graphql/Interviews/AnswerObjectType.cs:line 43
--- End of stack trace from previous location ---
   at WB.UI.Headquarters.Controllers.Api.PublicApi.Graphql.Interviews.AnswerObjectType.<>c.<<Configure>b__0_2>d.MoveNext() in /src/UI/WB.UI.Headquarters.Core/Controllers/Api/PublicApi/Graphql/Interviews/AnswerObjectType.cs:line 43
   at WB.UI.Headquarters.Controllers.Api.PublicApi.Graphql.Interviews.AnswerObjectType.<>c.<<Configure>b__0_2>d.MoveNext() in /src/UI/WB.UI.Headquarters.Core/Controllers/Api/PublicApi/Graphql/Interviews/AnswerObjectType.cs:line 43
--- End of stack trace from previous location ---
   at HotChocolate.Types.FieldMiddlewareCompiler.<>c__DisplayClass3_0.<<CreateResolverMiddleware>b__0>d.MoveNext()
   at WB.UI.Headquarters.Controllers.Api.PublicApi.Graphql.Interviews.AnswerObjectType.<>c.<<Configure>b__0_2>d.MoveNext() in /src/UI/WB.UI.Headquarters.Core/Controllers/Api/PublicApi/Graphql/Interviews/AnswerObjectType.cs:line 43
--- End of stack trace from previous location ---
   at WB.UI.Headquarters.Controllers.Api.PublicApi.Graphql.WorkspaceGraphQlMiddleware.InvokeAsync(IMiddlewareContext context, IWorkspaceContextAccessor workspaceContextAccessor, IWorkspaceContextSetter workspaceContextSetter) in /src/UI/WB.UI.Headquarters.Core/Controllers/Api/PublicApi/Graphql/WorkspaceGraphQlMiddleware.cs:line 58
   at HotChocolate.Types.FieldMiddlewareCompiler.<>c__DisplayClass3_0.<<CreateResolverMiddleware>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at WB.UI.Headquarters.Controllers.Api.PublicApi.Graphql.WorkspaceGraphQlMiddleware.InvokeAsync(IMiddlewareContext context, IWorkspaceContextAccessor workspaceContextAccessor, IWorkspaceContextSetter workspaceContextSetter) in /src/UI/WB.UI.Headquarters.Core/Controllers/Api/PublicApi/Graphql/WorkspaceGraphQlMiddleware.cs:line 58
   at WB.UI.Headquarters.Controllers.Api.PublicApi.Graphql.WorkspaceGraphQlMiddleware.InvokeAsync(IMiddlewareContext context, IWorkspaceContextAccessor workspaceContextAccessor, IWorkspaceContextSetter workspaceContextSetter) in /src/UI/WB.UI.Headquarters.Core/Controllers/Api/PublicApi/Graphql/WorkspaceGraphQlMiddleware.cs:line 58
   at WB.UI.Headquarters.Controllers.Api.PublicApi.Graphql.Interviews.AnswerObjectType.<>c.<<Configure>b__0_2>d.MoveNext() in /src/UI/WB.UI.Headquarters.Core/Controllers/Api/PublicApi/Graphql/Interviews/AnswerObjectType.cs:line 43
   at WB.UI.Headquarters.Controllers.Api.PublicApi.Graphql.Interviews.AnswerObjectType.<>c.<<Configure>b__0_2>d.MoveNext() in /src/UI/WB.UI.Headquarters.Core/Controllers/Api/PublicApi/Graphql/Interviews/AnswerObjectType.cs:line 43
--- End of stack trace from previous location ---
   at WB.UI.Headquarters.Controllers.Api.PublicApi.Graphql.Interviews.AnswerObjectType.<>c.<<Configure>b__0_2>d.MoveNext() in /src/UI/WB.UI.Headquarters.Core/Controllers/Api/PublicApi/Graphql/Interviews/AnswerObjectType.cs:line 43
--- End of stack trace from previous location ---
   at WB.UI.Headquarters.Controllers.Api.PublicApi.Graphql.WorkspaceGraphQlMiddleware.InvokeAsync(IMiddlewareContext context, IWorkspaceContextAccessor workspaceContextAccessor, IWorkspaceContextSetter workspaceContextSetter) in /src/UI/WB.UI.Headquarters.Core/Controllers/Api/PublicApi/Graphql/WorkspaceGraphQlMiddleware.cs:line 58
   at HotChocolate.Types.FieldMiddlewareCompiler.<>c__DisplayClass3_0.<<CreateResolverMiddleware>b__0>d.MoveNext()
   at HotChocolate.Utilities.MiddlewareCompiler`1.ExpressionHelper.AwaitTaskHelper(Task task)
   at WB.UI.Headquarters.Controllers.Api.PublicApi.Graphql.WorkspaceGraphQlMiddleware.InvokeAsync(IMiddlewareContext context, IWorkspaceContextAccessor workspaceContextAccessor, IWorkspaceContextSetter workspaceContextSetter) in /src/UI/WB.UI.Headquarters.Core/Controllers/Api/PublicApi/Graphql/WorkspaceGraphQlMiddleware.cs:line 58
   at HotChocolate.Utilities.MiddlewareCompiler`1.ExpressionHelper.AwaitTaskHelper(Task task)
   at WB.UI.Headquarters.Controllers.Api.PublicApi.Graphql.WorkspaceGraphQlMiddleware.InvokeAsync(IMiddlewareContext context, IWorkspaceContextAccessor workspaceContextAccessor, IWorkspaceContextSetter workspaceContextSetter) in /src/UI/WB.UI.Headquarters.Core/Controllers/Api/PublicApi/Graphql/WorkspaceGraphQlMiddleware.cs:line 58
   at HotChocolate.Execution.Processing.ResolverTask.ExecuteResolverPipelineAsync(CancellationToken cancellationToken)
   at WB.UI.Headquarters.Controllers.Api.PublicApi.Graphql.WorkspaceGraphQlMiddleware.InvokeAsync(IMiddlewareContext context, IWorkspaceContextAccessor workspaceContextAccessor, IWorkspaceContextSetter workspaceContextSetter) in /src/UI/WB.UI.Headquarters.Core/Controllers/Api/PublicApi/Graphql/WorkspaceGraphQlMiddleware.cs:line 58
   at HotChocolate.Execution.Processing.ResolverTask.ExecuteResolverPipelineAsync(CancellationToken cancellationToken)
   at HotChocolate.Execution.Processing.ResolverTask.TryExecuteAsync(CancellationToken cancellationToken)
   at HotChocolate.Execution.Processing.ResolverTask.TryExecuteAsync(CancellationToken cancellationToken)
   at HotChocolate.Execution.Processing.ResolverTask.ExecuteResolverPipelineAsync(CancellationToken cancellationToken)
   at HotChocolate.Execution.Processing.ResolverTask.TryExecuteAsync(CancellationToken cancellationToken)
   at HotChocolate.Utilities.MiddlewareCompiler`1.ExpressionHelper.AwaitTaskHelper(Task task)
   at HotChocolate.Execution.Processing.ResolverTask.ExecuteResolverPipelineAsync(CancellationToken cancellationToken)
   at HotChocolate.Execution.Processing.ResolverTask.TryExecuteAsync(CancellationToken cancellationToken)
   at HotChocolate.Utilities.MiddlewareCompiler`1.ExpressionHelper.AwaitTaskHelper(Task task)
   at HotChocolate.Execution.Processing.ResolverTask.ExecuteResolverPipelineAsync(CancellationToken cancellationToken)
   at HotChocolate.Execution.Processing.ResolverTask.TryExecuteAsync(CancellationToken cancellationToken)

[…]

Hello @stanislaso ,

please update to the most recent version, see

Hi @sergiy

We haven’t upgraded because

  1. we use only one workspace
  2. interviewers are in the field and updating their tablets would be extremely difficult.

You can update just the server, without forcing the tablets to update. See the relevant setting mentioned here:

Dear @stanislaso , in addition to updating the server please investigate:

  1. Who is affected? What are the roles (supervisor, interviewer, etc) of the users observing the error message?

  2. How many are affected? All? Some? One? If not all, is there any obvious difference between the affected and unaffected groups? E.g. only the supervisors that started working today, only interviewers that have more than 100 assignments, etc.

  3. How does the problem manifest itself? Does it appear all the time when visiting a particular page? Always when doing a certain action (like approval or assignment). Or sometimes? @tschoel was writing about ~40% chance of getting an error message.

  4. The message indicates a problem related to values of identifying fields. Are all identifying fields in your questionnaire genuine questions? or do you have any calculated variables? Can you share the questionnaire?

  5. While you are seeing the message on the screen, does it actually prevent you from doing anything in the system? Like opening an interview or rejecting it? Or is it completely harmless and just annoying? If it does prevent something, what specifically? For how many users? (be specific e.g. for 40 out of 90 supervisors) (if we can’t solve the error message issue, we might advise on a workaround to perform the blocked action).

  6. When did the issue first appear? Anything special that was done at the server before that?

  7. Try to restart (graciously reboot) the server in a quiet period when field staff is not working. See if anyone reports the same issue reappearing.

  8. Try to update as advised already above.

  9. Do you see anything strange in the health check page? Any other exceptions logged?

  10. If the problem appears on the pages using filters, try to see if there is anything special set up in the filters, clear the filters or change the filters.@tschoel wrote above that modifying the search criteria will eventually make it work. Try this and see whether it has any effect on the situation.

@tschoel , if you are running any survey now, are you still getting this error message? Have you updated to the most recent version?

  1. The issue reported by @tschoel (and another user directly) was related to a docker installation of Survey Solutions. Please, provide as many details as possible that would characterize your setup.

Last, but not least, don’t panic. Export your data now. Have a copy stored reliably. Inspect the exported data, see if you have recent interviews (interviews from today). Try to do selective checks, especially if you know that Supevisor X had reported seeing the message, is the data of team of Supervisor X there? If they claim they collected N interviews today and synchronized, do you see N? If the interviewers don’t experience any such problem (and I believe they should not) and the data comes out complete (and I believe it would) then by all means carry on, or elaborate on #5 above.

If nothing from the above helps, I am afraid the only way to proceed would be to request a direct access to your web server and database server, so that someone could get directly under the hood of it.

PS: if the sticker attached to the monitor on your photo contains a user’s password to access Survey Solutions (or any other system for that matter) please do remove it and store in some place safe.

Hi @sergiy

I’ve just upgraded the server to version 21.09.4 (latest version available as docker image) after backing up the database beforehand.

So far so good. I can see interviews getting synched.
No error appearing on the web frontend or in the logs so far.
We are trying to request an export overnight, will let you know how this goes.

PS: as for the post-it note no worries, it’s just some random notes :slight_smile:

Well hi again @sergiy,

Didn’t last very long before getting the same error again.

To answer some of your questions:

  • I can see this error when connected as admin, and I know other hq users get that too
  • I myself see these errors on the Interviews page
  • This occurs quite randomly. Reloading the page often fixes it and loads interviews fine again.
  • It started after a while, when we reached around 8000 interviews in the server (maybe more)

I need to investigate to answer other questions.

For now (after the upgrade) the diagnotisc page reports no error.
Before that, and after a while, the “health status” section stayed blank.
I’ll let you know if this happens again.

Users report same errors as before.
Health check is stuck again.
Export is not working (I created another topic for this)

Hi @sergiy ,

So, up updated to 21.09.4 (the latest Docker image) and the issue perists. To add some info, here are the relevant parts from the docker-compose .yml:

version: '3.3'

services:
    hq:
        image: 'surveysolutions/surveysolutions:21.9'
        depends_on:
            - "postgres"
        environment:
            ASPNETCORE_ENVIRONMENT: Production
            HQ_ConnectionStrings__DefaultConnection: "Persist Security Info=true;Server=postgres;Port=5432;User Id=postgres;Password=${PG_PASSWORD};Database=SurveySolutions;"
            HQ_Headquarters__BaseUrl: http://hq.${HOST}
            HQ_Headquarters__TenantName: default
            Export_ExportSettings__DirectoryPath: /app/AppData/Export
            HQ_Logging__LogsLocation: /app/AppData/logs
        command: []
        volumes:
            - hq_data:/app/AppData
            - hq_keys:/root/.aspnet/DataProtection-Keys
        ports:
            - 9700:80
        restart: always

    postgres:
        image: "postgres:13"
        environment:
            POSTGRES_PASSWORD: ${PG_PASSWORD}
            ANALYTICS_PASSWORD: ${ANALYTICS_PASSWORD}
            PERSONAL_INFO_PASSWORD: ${PERSONAL_INFO_PASSWORD}
        volumes:
            - pg_data:/var/lib/postgresql/data
            - ./postgres/init:/docker-entrypoint-initdb.d
        expose:
            - 5432
        restart: always

volumes:
    hq_data: {}
    hq_keys: {}
    pg_data: {}

We then have a world-facing NGINX reverse proxy with the following config for the site:

server {

	server_name <THE EXTERNAL HOST NAME>;

	root /var/www/example.com;
	index index.html;

	location / {
		proxy_pass http://localhost:9700;
		proxy_set_header HOST $server_name;
		proxy_set_header Referer https://$server_name;
		proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
		proxy_set_header X-Forwarded-Host $server_name;
		
		# WebSocket support
		proxy_http_version 1.1;
		proxy_set_header Upgrade $http_upgrade;
    		proxy_set_header Connection $http_connection;
	}

    listen [::]:443 ssl; # managed by Certbot
    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/<host name>/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/<host name>/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

}

server {
    if ($host = <host name>) {
        return 301 https://$host$request_uri;
    } # managed by Certbot


	listen 80;
	listen [::]:80;

	server_name <host name>;
    return 404; # managed by Certbot


}

Both are virtually identical to another server, though, where we are not facing this issue.

One of the questionnaires with which this happens would be 3d540184-7e28-4e95-84b4-51ff9a15f266, but we have also seen this in a workspace with a3b5d650-6504-4ae8-9b44-cb2b9068aba9 and b6e48a21-69c1-4f2a-ae9c-a837b5fed54c.

Hope this can get you closer to it.

Hello Thorsten,

thank you very much for providing the additional information and confirming that the issue still persists with the newest version.

At this moment we can’t be 100% certain that the issue with export not being produced and the GraphQL error are results of the same original cause.

  • Have you had any issues with the export so far?
  • Can you give an approximate magnitude of the workforce? (tens, hundreds, thousands of interviewers/supervisors).
  • Have you had to adjust the maximum permitted number of connections to the server? Postgresql database?
  • Are you using CloudFlare for hosting, by any chance?

Thank you, Sergiy

To expand, I’ve asked @tschoel about Cloudflare for a reason.

@stanislaso has mentioned that the health check page doesn’t render itself correctly.
There is another page which shows the health status /.hc, briefly mentioned here:

Under normal operation the page should indicate a healthy status.
image

But for @stanislaso server it displays an error (while another endpoint displaying a version still works).

What concerns me is that the response came from the Cloudflare hosting, not the Survey Solutions server, as if it didn’t wait to give enough time for the server to check its status and responded instead of it.

No, sorry, no cloudflare, just the nginx. And .hc and .version are showing just fine. We have 162 users in that installation.

Exports were never an issue on that specific system. When I mentinoed an export back in May that meant an export of the questionnaire from designer or so (not an issue either, just a suggestion to get you that questionnaire). In fact, we are automatically triggering the export process via API at regular intervals (on this machine as well as on others where we do not see this GraphQL issue) and have never observed any issues with that.

Hi @sergiy

  1. Who is affected? What are the roles (supervisor, interviewer, etc) of the users observing the error message?

It affects both Supervisor and HQ. Supervisors and headquarters received some error message saying: This page isn’t working. hq305.surveys.spc.int (server name) is currently unable to handle this request. In the interviewer app some validation condition message shows up when it shouldn’t be shown but on the tester it works well.

  1. How many are affected? All? Some? One? If not all, is there any obvious difference between the affected and unaffected groups? E.g. only the supervisors that started working today, only interviewers that have more than 100 assignments, etc.

Affecting almost all the HQs and Supervisors. The above error affected all headquarters, supervisors and interviewer. The interviewer confuse warning happens right at the start of the data collection. The message shown when supervisors and headquarters try to login after a new version of the questionnaire was upgraded from version 5 to 6.

  1. How does the problem manifest itself? Does it appear all the time when visiting a particular page? Always when doing a certain action (like approval or assignment). Or sometimes? @tschoel was writing about ~40% chance of getting an error message.

It appear on random times not on particular task but using INTERVIEWS filtering responsible people and while checking of forms 5. While you are seeing the message on the screen, does it actually prevent you from doing anything in the system? Like opening an interview or rejecting it? Or is it completely harmless and just annoying? If it does prevent something, what specifically? For how many users? (be specific e.g. for 40 out of 90 supervisors) (if we can’t solve the error message issue, we might advise on a workaround to perform the blocked action).

No data displayed while filtering interviews

Appear for less than a minute than disappear. Mainly when approving a form (appears while it’s loading) and Adding comments. Though not affected.

A few reports that while it appears the page is locked.

  1. Who is affected? What are the roles (supervisor, interviewer, etc) of the users observing the error message?

It affects both Supervisor and HQ. Supervisors and headquarters received some error message saying: This page isn’t working. hq305.surveys.spc.int (server name) is currently unable to handle this request. In the interviewer app some validation condition message shows up when it shouldn’t be shown but on the tester it works well.

  1. How many are affected? All? Some? One? If not all, is there any obvious difference between the affected and unaffected groups? E.g. only the supervisors that started working today, only interviewers that have more than 100 assignments, etc.

Affecting almost all the HQs and Supervisors. The above error affected all headquarters, supervisors and interviewer. The interviewer confuse warning happens right at the start of the data collection. The message shown when supervisors and headquarters try to login after a new version of the questionnaire was upgraded from version 5 to 6.

  1. How does the problem manifest itself? Does it appear all the time when visiting a particular page? Always when doing a certain action (like approval or assignment). Or sometimes? @tschoel was writing about ~40% chance of getting an error message.

It appear on random times not on particular task but using INTERVIEWS filtering responsible people and while checking of forms 5. While you are seeing the message on the screen, does it actually prevent you from doing anything in the system? Like opening an interview or rejecting it? Or is it completely harmless and just annoying? If it does prevent something, what specifically? For how many users? (be specific e.g. for 40 out of 90 supervisors) (if we can’t solve the error message issue, we might advise on a workaround to perform the blocked action).

No data displayed while filtering interviews

Appear for less than a minute than disappear. Mainly when approving a form (appears while it’s loading) and Adding comments. Though not affected.

A few reports that while it appears the page is locked.

  1. When did the issue first appear? Anything special that was done at the server before that? - In general it started appearing on Wednesday the 1st of November.

Dear @tschoel , @stanislaso ,

sorry for the back-and-forth. My colleagues are trying to reproduce the issue in the controlled environment. So far with no success. Could you please specify the exact versions of the:

  • host operating system in which you run Docker?
  • exact version of the Docker application?