Scheduler instance active but recovered by another instance in the cluster meaning?

Hi all,

I have Survey Solutions running in a clustered deployment with 3 instances in Microsoft Azure App Service. In the log I often encounter an error like below:

[06:59:05 WRN] Headquarters This scheduler instance (<instance_id>) is still active but was recovered by another instance in the cluster.  This may cause inconsistent behavior.

What exactly does this mean? Should I be worried? How can I cure it?

Thanks

While I am not sure, but the above might be related to the following error I encounter in the logs.

2022-06-14T06:58:50.766056228Z [06:58:50 ERR] Headquarters Couldn't rollback ADO.NET connection. Cannot access a disposed object.
2022-06-14T06:58:50.766108829Z Object name: 'NpgsqlTransaction'.
2022-06-14T06:58:50.766116029Z System.ObjectDisposedException: Cannot access a disposed object.
2022-06-14T06:58:50.766121329Z Object name: 'NpgsqlTransaction'.
2022-06-14T06:58:50.766126229Z    at Npgsql.NpgsqlTransaction.CheckDisposed()
2022-06-14T06:58:50.766131029Z    at Npgsql.NpgsqlTransaction.get_Connection()
2022-06-14T06:58:50.766136729Z    at Npgsql.NpgsqlTransaction.get_DbConnection()
2022-06-14T06:58:50.766142229Z    at System.Data.Common.DbTransaction.get_Connection()
2022-06-14T06:58:50.766147329Z    at Quartz.Impl.AdoJobStore.ConnectionAndTransactionHolder.CheckNotZombied()
2022-06-14T06:58:50.766152430Z    at Quartz.Impl.AdoJobStore.ConnectionAndTransactionHolder.Rollback(Boolean transientError)
2022-06-14T06:58:50.809193575Z [06:58:50 ERR] Headquarters Error managing cluster: Failure identifying failed instances when checking-in: Exception while reading from stream
2022-06-14T06:58:50.809236575Z Quartz.JobPersistenceException: Failure identifying failed instances when checking-in: Exception while reading from stream
2022-06-14T06:58:50.809243575Z  ---> Npgsql.NpgsqlException (0x80004005): Exception while reading from stream
2022-06-14T06:58:50.809249076Z  ---> System.TimeoutException: Timeout during reading attempt
2022-06-14T06:58:50.809256076Z    at Npgsql.NpgsqlReadBuffer.<Ensure>g__EnsureLong|40_0(NpgsqlReadBuffer buffer, Int32 count, Boolean async, Boolean readingNotifications)
2022-06-14T06:58:50.809261676Z    at Npgsql.NpgsqlConnector.<ReadMessage>g__ReadMessageLong|194_0(NpgsqlConnector connector, Boolean async, DataRowLoadingMode dataRowLoadingMode, Boolean readingNotifications, Boolean isReadingPrependedMessage)
2022-06-14T06:58:50.809268076Z    at Npgsql.NpgsqlConnector.<ReadMessage>g__ReadMessageLong|194_0(NpgsqlConnector connector, Boolean async, DataRowLoadingMode dataRowLoadingMode, Boolean readingNotifications, Boolean isReadingPrependedMessage)
2022-06-14T06:58:50.809273476Z    at Npgsql.NpgsqlDataReader.NextResult(Boolean async, Boolean isConsuming, CancellationToken cancellationToken)
2022-06-14T06:58:50.809278776Z    at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken)
2022-06-14T06:58:50.809284476Z    at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken)
2022-06-14T06:58:50.809303676Z    at Npgsql.NpgsqlCommand.ExecuteDbDataReaderAsync(CommandBehavior behavior, CancellationToken cancellationToken)
2022-06-14T06:58:50.809309176Z    at Quartz.Impl.AdoJobStore.StdAdoDelegate.SelectSchedulerStateRecords(ConnectionAndTransactionHolder conn, String instanceName, CancellationToken cancellationToken)
2022-06-14T06:58:50.809313777Z    at Quartz.Impl.AdoJobStore.JobStoreSupport.FindFailedInstances(ConnectionAndTransactionHolder conn, CancellationToken cancellationToken)
2022-06-14T06:58:50.809318377Z    --- End of inner exception stack trace ---
2022-06-14T06:58:50.809322677Z    at Quartz.Impl.AdoJobStore.JobStoreSupport.FindFailedInstances(ConnectionAndTransactionHolder conn, CancellationToken cancellationToken)
2022-06-14T06:58:50.809329577Z    at Quartz.Impl.AdoJobStore.JobStoreSupport.ClusterCheckIn(ConnectionAndTransactionHolder conn, CancellationToken cancellationToken)
2022-06-14T06:58:50.809335077Z    at Quartz.Impl.AdoJobStore.JobStoreSupport.DoCheckin(Guid requestorId, CancellationToken cancellationToken)
2022-06-14T06:58:50.809341077Z    at Quartz.Impl.AdoJobStore.JobStoreSupport.DoCheckin(Guid requestorId, CancellationToken cancellationToken)
2022-06-14T06:58:50.809346477Z    at Quartz.Impl.AdoJobStore.ClusterManager.Manage() [See nested exception: Npgsql.NpgsqlException (0x80004005): Exception while reading from stream
2022-06-14T06:58:50.809352377Z  ---> System.TimeoutException: Timeout during reading attempt
2022-06-14T06:58:50.809358477Z    at Npgsql.NpgsqlReadBuffer.<Ensure>g__EnsureLong|40_0(NpgsqlReadBuffer buffer, Int32 count, Boolean async, Boolean readingNotifications)
2022-06-14T06:58:50.809364077Z    at Npgsql.NpgsqlConnector.<ReadMessage>g__ReadMessageLong|194_0(NpgsqlConnector connector, Boolean async, DataRowLoadingMode dataRowLoadingMode, Boolean readingNotifications, Boolean isReadingPrependedMessage)
2022-06-14T06:58:50.809369877Z    at Npgsql.NpgsqlConnector.<ReadMessage>g__ReadMessageLong|194_0(NpgsqlConnector connector, Boolean async, DataRowLoadingMode dataRowLoadingMode, Boolean readingNotifications, Boolean isReadingPrependedMessage)
2022-06-14T06:58:50.809375777Z    at Npgsql.NpgsqlDataReader.NextResult(Boolean async, Boolean isConsuming, CancellationToken cancellationToken)
2022-06-14T06:58:50.809381078Z    at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken)
2022-06-14T06:58:50.809386678Z    at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken)
2022-06-14T06:58:50.809392478Z    at Npgsql.NpgsqlCommand.ExecuteDbDataReaderAsync(CommandBehavior behavior, CancellationToken cancellationToken)
2022-06-14T06:58:50.809398278Z    at Quartz.Impl.AdoJobStore.StdAdoDelegate.SelectSchedulerStateRecords(ConnectionAndTransactionHolder conn, String instanceName, CancellationToken cancellationToken)
2022-06-14T06:58:50.809404378Z    at Quartz.Impl.AdoJobStore.JobStoreSupport.FindFailedInstances(ConnectionAndTransactionHolder conn, CancellationToken cancellationToken)]

Any ideas?