Hi all,
I have Survey Solutions running in a clustered deployment with 3 instances in Microsoft Azure App Service. In the log I often encounter an error like below:
[06:59:05 WRN] Headquarters This scheduler instance (<instance_id>) is still active but was recovered by another instance in the cluster. This may cause inconsistent behavior.
What exactly does this mean? Should I be worried? How can I cure it?
Thanks
While I am not sure, but the above might be related to the following error I encounter in the logs.
2022-06-14T06:58:50.766056228Z [06:58:50 ERR] Headquarters Couldn't rollback ADO.NET connection. Cannot access a disposed object.
2022-06-14T06:58:50.766108829Z Object name: 'NpgsqlTransaction'.
2022-06-14T06:58:50.766116029Z System.ObjectDisposedException: Cannot access a disposed object.
2022-06-14T06:58:50.766121329Z Object name: 'NpgsqlTransaction'.
2022-06-14T06:58:50.766126229Z at Npgsql.NpgsqlTransaction.CheckDisposed()
2022-06-14T06:58:50.766131029Z at Npgsql.NpgsqlTransaction.get_Connection()
2022-06-14T06:58:50.766136729Z at Npgsql.NpgsqlTransaction.get_DbConnection()
2022-06-14T06:58:50.766142229Z at System.Data.Common.DbTransaction.get_Connection()
2022-06-14T06:58:50.766147329Z at Quartz.Impl.AdoJobStore.ConnectionAndTransactionHolder.CheckNotZombied()
2022-06-14T06:58:50.766152430Z at Quartz.Impl.AdoJobStore.ConnectionAndTransactionHolder.Rollback(Boolean transientError)
2022-06-14T06:58:50.809193575Z [06:58:50 ERR] Headquarters Error managing cluster: Failure identifying failed instances when checking-in: Exception while reading from stream
2022-06-14T06:58:50.809236575Z Quartz.JobPersistenceException: Failure identifying failed instances when checking-in: Exception while reading from stream
2022-06-14T06:58:50.809243575Z ---> Npgsql.NpgsqlException (0x80004005): Exception while reading from stream
2022-06-14T06:58:50.809249076Z ---> System.TimeoutException: Timeout during reading attempt
2022-06-14T06:58:50.809256076Z at Npgsql.NpgsqlReadBuffer.<Ensure>g__EnsureLong|40_0(NpgsqlReadBuffer buffer, Int32 count, Boolean async, Boolean readingNotifications)
2022-06-14T06:58:50.809261676Z at Npgsql.NpgsqlConnector.<ReadMessage>g__ReadMessageLong|194_0(NpgsqlConnector connector, Boolean async, DataRowLoadingMode dataRowLoadingMode, Boolean readingNotifications, Boolean isReadingPrependedMessage)
2022-06-14T06:58:50.809268076Z at Npgsql.NpgsqlConnector.<ReadMessage>g__ReadMessageLong|194_0(NpgsqlConnector connector, Boolean async, DataRowLoadingMode dataRowLoadingMode, Boolean readingNotifications, Boolean isReadingPrependedMessage)
2022-06-14T06:58:50.809273476Z at Npgsql.NpgsqlDataReader.NextResult(Boolean async, Boolean isConsuming, CancellationToken cancellationToken)
2022-06-14T06:58:50.809278776Z at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken)
2022-06-14T06:58:50.809284476Z at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken)
2022-06-14T06:58:50.809303676Z at Npgsql.NpgsqlCommand.ExecuteDbDataReaderAsync(CommandBehavior behavior, CancellationToken cancellationToken)
2022-06-14T06:58:50.809309176Z at Quartz.Impl.AdoJobStore.StdAdoDelegate.SelectSchedulerStateRecords(ConnectionAndTransactionHolder conn, String instanceName, CancellationToken cancellationToken)
2022-06-14T06:58:50.809313777Z at Quartz.Impl.AdoJobStore.JobStoreSupport.FindFailedInstances(ConnectionAndTransactionHolder conn, CancellationToken cancellationToken)
2022-06-14T06:58:50.809318377Z --- End of inner exception stack trace ---
2022-06-14T06:58:50.809322677Z at Quartz.Impl.AdoJobStore.JobStoreSupport.FindFailedInstances(ConnectionAndTransactionHolder conn, CancellationToken cancellationToken)
2022-06-14T06:58:50.809329577Z at Quartz.Impl.AdoJobStore.JobStoreSupport.ClusterCheckIn(ConnectionAndTransactionHolder conn, CancellationToken cancellationToken)
2022-06-14T06:58:50.809335077Z at Quartz.Impl.AdoJobStore.JobStoreSupport.DoCheckin(Guid requestorId, CancellationToken cancellationToken)
2022-06-14T06:58:50.809341077Z at Quartz.Impl.AdoJobStore.JobStoreSupport.DoCheckin(Guid requestorId, CancellationToken cancellationToken)
2022-06-14T06:58:50.809346477Z at Quartz.Impl.AdoJobStore.ClusterManager.Manage() [See nested exception: Npgsql.NpgsqlException (0x80004005): Exception while reading from stream
2022-06-14T06:58:50.809352377Z ---> System.TimeoutException: Timeout during reading attempt
2022-06-14T06:58:50.809358477Z at Npgsql.NpgsqlReadBuffer.<Ensure>g__EnsureLong|40_0(NpgsqlReadBuffer buffer, Int32 count, Boolean async, Boolean readingNotifications)
2022-06-14T06:58:50.809364077Z at Npgsql.NpgsqlConnector.<ReadMessage>g__ReadMessageLong|194_0(NpgsqlConnector connector, Boolean async, DataRowLoadingMode dataRowLoadingMode, Boolean readingNotifications, Boolean isReadingPrependedMessage)
2022-06-14T06:58:50.809369877Z at Npgsql.NpgsqlConnector.<ReadMessage>g__ReadMessageLong|194_0(NpgsqlConnector connector, Boolean async, DataRowLoadingMode dataRowLoadingMode, Boolean readingNotifications, Boolean isReadingPrependedMessage)
2022-06-14T06:58:50.809375777Z at Npgsql.NpgsqlDataReader.NextResult(Boolean async, Boolean isConsuming, CancellationToken cancellationToken)
2022-06-14T06:58:50.809381078Z at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken)
2022-06-14T06:58:50.809386678Z at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken)
2022-06-14T06:58:50.809392478Z at Npgsql.NpgsqlCommand.ExecuteDbDataReaderAsync(CommandBehavior behavior, CancellationToken cancellationToken)
2022-06-14T06:58:50.809398278Z at Quartz.Impl.AdoJobStore.StdAdoDelegate.SelectSchedulerStateRecords(ConnectionAndTransactionHolder conn, String instanceName, CancellationToken cancellationToken)
2022-06-14T06:58:50.809404378Z at Quartz.Impl.AdoJobStore.JobStoreSupport.FindFailedInstances(ConnectionAndTransactionHolder conn, CancellationToken cancellationToken)]
Any ideas?