Export fails with extremely large number in double typed calculated variable

We have a questionnaire that has an calculated variable of double type. The calculation itself is trivial and apparently works well. it is of the form (q1 ?? 0) + (q2 ?? 0) + (q3 ?? 0) + … + (qn ?? 0). All questions in the formula are of “numeric” type. In the Web view, the correct result is displayed just fine. For one particular interview it comes out as 12833144223594760000000.

Now, when running any export from the workspace that contains that interview, the export will fail with the message “An unexpected error occurred during export”. The log does appear to be helpful here with the message:

hq_1        | [15:17:26 FTL] export-service Unable to cast object of type 'System.Numerics.BigInteger' to type 'System.IConvertible'.
hq_1        | System.InvalidCastException: Unable to cast object of type 'System.Numerics.BigInteger' to type 'System.IConvertible'.
hq_1        |    at System.Convert.ToDouble(Object value)
hq_1        |    at WB.Services.Export.InterviewDataStorage.InterviewDataDenormalizer.UpdateVariableValue(Guid interviewId, Guid entityId, RosterVector rosterVector, Object value, CancellationToken token) in /src/Services/Export/WB.Services.Export/InterviewDataStorage/InterviewDataDenormalizer.cs:line 566
hq_1        |    at WB.Services.Export.InterviewDataStorage.InterviewDataDenormalizer.Handle(PublishedEvent`1 event, CancellationToken token) in /src/Services/Export/WB.Services.Export/InterviewDataStorage/InterviewDataDenormalizer.cs:line 360
hq_1        |    at WB.Services.Infrastructure.EventSourcing.FunctionalHandlerExtensions.Handle(IStatefulDenormalizer eventsHandler, Event ev, CancellationToken token) in /src/Services/Core/WB.Services.Infrastructure/EventSourcing/FunctionalHandlerExtensions.cs:line 91
hq_1        |    at WB.Services.Export.Events.EventsHandler.HandleEventsFeedAsync(EventsFeed feed, CancellationToken token) in /src/Services/Export/WB.Services.Export/Events/EventsHandler.cs:line 61
hq_1        |    at WB.Services.Export.Events.EventsHandler.HandleEventsFeedAsync(EventsFeed feed, CancellationToken token) in /src/Services/Export/WB.Services.Export/Events/EventsHandler.cs:line 89
hq_1        | [15:17:26 FTL] export-service Exception Data : WB:Event : VariablesChanged
hq_1        | [15:17:26 FTL] export-service Exception Data : WB:GlobalSequence : 2716
hq_1        | [15:17:26 FTL] export-service Exception Data : WB:EventSourceId : b33b920e-6266-4fbf-af1d-36dba0c83c58
hq_1        | [15:17:26 FTL] export-service Exception Data : WB:Events : 1:4760
hq_1        | [15:17:26 ERR] export-service Export job failed
hq_1        | System.InvalidCastException: Unable to cast object of type 'System.Numerics.BigInteger' to type 'System.IConvertible'.
hq_1        |    at System.Convert.ToDouble(Object value)
hq_1        |    at WB.Services.Export.InterviewDataStorage.InterviewDataDenormalizer.UpdateVariableValue(Guid interviewId, Guid entityId, RosterVector rosterVector, Object value, CancellationToken token) in /src/Services/Export/WB.Services.Export/InterviewDataStorage/InterviewDataDenormalizer.cs:line 566
hq_1        |    at WB.Services.Export.InterviewDataStorage.InterviewDataDenormalizer.Handle(PublishedEvent`1 event, CancellationToken token) in /src/Services/Export/WB.Services.Export/InterviewDataStorage/InterviewDataDenormalizer.cs:line 360
hq_1        |    at WB.Services.Infrastructure.EventSourcing.FunctionalHandlerExtensions.Handle(IStatefulDenormalizer eventsHandler, Event ev, CancellationToken token) in /src/Services/Core/WB.Services.Infrastructure/EventSourcing/FunctionalHandlerExtensions.cs:line 91
hq_1        |    at WB.Services.Export.Events.EventsHandler.HandleEventsFeedAsync(EventsFeed feed, CancellationToken token) in /src/Services/Export/WB.Services.Export/Events/EventsHandler.cs:line 61
hq_1        |    at WB.Services.Export.Events.EventsHandler.HandleEventsFeedAsync(EventsFeed feed, CancellationToken token) in /src/Services/Export/WB.Services.Export/Events/EventsHandler.cs:line 89
hq_1        |    at WB.Services.Export.Events.EventsProcessor.<>c__DisplayClass10_1.<<HandleNewEventsImplementation>b__2>d.MoveNext() in /src/Services/Export/WB.Services.Export/Events/EventsProcessor.cs:line 92
hq_1        | --- End of stack trace from previous location ---
hq_1        |    at WB.Services.Export.Events.EventsProcessor.<>c__DisplayClass10_0.<<HandleNewEventsImplementation>g__ExecuteAndEstimate|1>d.MoveNext() in /src/Services/Export/WB.Services.Export/Events/EventsProcessor.cs:line 141
hq_1        | --- End of stack trace from previous location ---
hq_1        |    at WB.Services.Export.Events.EventsProcessor.HandleNewEventsImplementation(Int64 exportProcessId, Int64 sequenceToStartFrom, CancellationToken token) in /src/Services/Export/WB.Services.Export/Events/EventsProcessor.cs:line 76
hq_1        |    at WB.Services.Export.Events.EventsProcessor.HandleNewEvents(Int64 exportProcessId, CancellationToken token) in /src/Services/Export/WB.Services.Export/Events/EventsProcessor.cs:line 56
hq_1        |    at WB.Services.Export.Jobs.ExportJob.ExecuteAsync(DataExportProcessArgs pendingExportProcess, CancellationToken cancellationToken) in /src/Services/Export/WB.Services.Export/Jobs/ExportJob.cs:line 82
hq_1        | [15:17:26 ERR] export-service Error during job run [ {"ExportSettings":{"JobId":null,"ExportFormat":1,"QuestionnaireId":{"Id":"1155dc9f7a3241ca8589b0b021d65b3f$1"},"Status":null,"FromDate":null,"ToDate":null,"Tenant":{"BaseUrl":"http://hq.giae-monitoring.de/primary","Id":{"Id":"00ce7a0c-5af9-4be5-940b-5a19c7655558"},"Name":"default","ShortName":"default","Workspace":"primary"},"Translation":null,"IncludeMeta":true,"NaturalId":"00ce7a0c-5af9-4be5-940b-5a19c7655558$All$Tabular$1155dc9f7a3241ca8589b0b021d65b3f$1$EMPTY FROM DATE$EMPTY TO DATE$No translation"},"Status":{"CreatedDate":"0001-01-01T00:00:00","BeginDate":null,"EndDate":null,"LastUpdateDate":"0001-01-01T00:00:00","Status":2,"JobStatus":0,"ProgressInPercents":0,"IsRunning":false,"TimeEstimation":null,"Error":null},"ArchivePassword":null,"NaturalId":"NoExternal$00ce7a0c-5af9-4be5-940b-5a19c7655558$All$Tabular$1155dc9f7a3241ca8589b0b021d65b3f$1$EMPTY FROM DATE$EMPTY TO DATE$No translation","AccessToken":null,"RefreshToken":null,"StorageType":null,"ProcessId":0,"ShouldDropTenantSchema":false} ]
hq_1        | System.InvalidCastException: Unable to cast object of type 'System.Numerics.BigInteger' to type 'System.IConvertible'.
hq_1        |    at System.Convert.ToDouble(Object value)
hq_1        |    at WB.Services.Export.InterviewDataStorage.InterviewDataDenormalizer.UpdateVariableValue(Guid interviewId, Guid entityId, RosterVector rosterVector, Object value, CancellationToken token) in /src/Services/Export/WB.Services.Export/InterviewDataStorage/InterviewDataDenormalizer.cs:line 566
hq_1        |    at WB.Services.Export.InterviewDataStorage.InterviewDataDenormalizer.Handle(PublishedEvent`1 event, CancellationToken token) in /src/Services/Export/WB.Services.Export/InterviewDataStorage/InterviewDataDenormalizer.cs:line 360
hq_1        |    at WB.Services.Infrastructure.EventSourcing.FunctionalHandlerExtensions.Handle(IStatefulDenormalizer eventsHandler, Event ev, CancellationToken token) in /src/Services/Core/WB.Services.Infrastructure/EventSourcing/FunctionalHandlerExtensions.cs:line 91
hq_1        |    at WB.Services.Export.Events.EventsHandler.HandleEventsFeedAsync(EventsFeed feed, CancellationToken token) in /src/Services/Export/WB.Services.Export/Events/EventsHandler.cs:line 61
hq_1        |    at WB.Services.Export.Events.EventsHandler.HandleEventsFeedAsync(EventsFeed feed, CancellationToken token) in /src/Services/Export/WB.Services.Export/Events/EventsHandler.cs:line 89
hq_1        |    at WB.Services.Export.Events.EventsProcessor.<>c__DisplayClass10_1.<<HandleNewEventsImplementation>b__2>d.MoveNext() in /src/Services/Export/WB.Services.Export/Events/EventsProcessor.cs:line 92
hq_1        | --- End of stack trace from previous location ---
hq_1        |    at WB.Services.Export.Events.EventsProcessor.<>c__DisplayClass10_0.<<HandleNewEventsImplementation>g__ExecuteAndEstimate|1>d.MoveNext() in /src/Services/Export/WB.Services.Export/Events/EventsProcessor.cs:line 141
hq_1        | --- End of stack trace from previous location ---
hq_1        |    at WB.Services.Export.Events.EventsProcessor.HandleNewEventsImplementation(Int64 exportProcessId, Int64 sequenceToStartFrom, CancellationToken token) in /src/Services/Export/WB.Services.Export/Events/EventsProcessor.cs:line 76
hq_1        |    at WB.Services.Export.Events.EventsProcessor.HandleNewEvents(Int64 exportProcessId, CancellationToken token) in /src/Services/Export/WB.Services.Export/Events/EventsProcessor.cs:line 56
hq_1        |    at WB.Services.Export.Jobs.ExportJob.ExecuteAsync(DataExportProcessArgs pendingExportProcess, CancellationToken cancellationToken) in /src/Services/Export/WB.Services.Export/Jobs/ExportJob.cs:line 82
hq_1        |    at WB.Services.Scheduler.Services.Implementation.JobExecutor.ExecuteAsync(JobItem job, CancellationToken token) in /src/Services/Core/WB.Services.Scheduler/Services/Implementation/JobExecutor.cs:line 86
hq_1        |    at WB.Services.Scheduler.Services.Implementation.JobExecutor.ExecuteAsync(JobItem job, CancellationToken token) in /src/Services/Core/WB.Services.Scheduler/Services/Implementation/JobExecutor.cs:line 86
hq_1        | [15:17:26 VRB] export-service WB.Services.Scheduler.Model.Events.FailJobEvent

I realize that the value we have in that variable is artificially high in the specific case, but in any case, it should not fail all exports.

Hello Thorsten,

thank you for reporting the issue. I am afraid I will need more details.
The value like this can’t be accommodated by a numeric integer question (If you try to copy-paste it there it will be entered as zero).

So,

  1. what is the number of components being added? (q1,…qn)
  2. what is the type of each? (numeric integer, numeric with decimals), etc
  3. what are the specific values of q1…qn?

I assume the error you are getting occurs for every file format (tab-delimited, Stata, SPSS), if not, please clarify.

PS: just in case, an alternative to your expression could be: new {q1,q2,q3,q4}.Sum()

Thank you, Sergiy

Hi Sergiy,

Thanks for taking your time.

So, the exact expression is: (vet_cos_1 ?? 0) + (hay_bou_1 * hay_pri_1) + (straw_bou_1 * straw_pri_1) + (gra_bou_1 * gra_pri_1) + (prot_bou_1 * prot_pri_1) + (mais_bou_1 * mais_pri_1) + (soy_bou_1 * soy_pri_1) + (lick_bou_1 * lick_pri_1) + (pas_mai_1 ?? 0) + (dai_shed_1 ?? 0) + (li_trans_1 ?? 0) + (li_elect_1 ?? 0). It is the variable “dairy_inp_cost” in the questionnaire 2b8c7f3e1475417f9a02fef3695bac94.

All inputs are of type “NumericQuestion” with isInteger set to false. Except for “dai_shed_1”, they all have CountOfDecimalPlaces set to 2. In “dai_shed_1” that setting is missing.

The error happens for all exports indeed, even including Paradata. It appears to me that the export service tries to rebuild all interviews from the events and then fails when it comes upon a certain, problematic event. The value column for the event with the globalsequence from the stack reads:

{"originDate": "2021-05-10T15:10:03.8224685+02:00", "changedVariables": [{"identity": {"id": "00f75083-d378-4325-96b9-41
5468c79111"}, "newValue": 12833144223594760000000}]}

That also means that our attempt to work around the error by simply modifying the inputs for the variable does not work, because that event will always remain.

That attempt also means I cannot reconstruct the original values for the components that led that high number.

Regards, Thorsten

P.S.: Thanks for the tip. I haven’t written any C# for some eight years and it is interesting to see what it can do these days.

What is the type of this variable? Double? Long? (as declared in the Designer).

Strange. Then it is possibly integer??

This is correct.

Very likely this is the case.

The high number is likely coming from the multiplication, such as here:
hay_bou_1 * hay_pri_1
if both components are very very large.

Is this one of the PDS (personal demo servers)? If yes, please send the server name to support.

@tschoel , the problem is reproducible and will be fixed in a future update.