Incomplete multiple assignments creation

There are 20 headquarters users uploading assignments into SuSo by using multiple assignment creation features (importing tab files). Each file contains 2,700 assignments in average for a team of 5 interviewers (approximately 540 assignments per interviewer).

During the upload process, some files are accepted and processed but not all the assignments are created in the SuSo database. When this problem occurs from the 2,700 assignments that contain the file, 2% to 10% of the assignments are not created in SuSo.

NSO census platform configuration is the following: one load balancer distributing the load into 10 servers with SuSo. The load balancer has configured sticky sessions for up to 15 minutes.

Alta disponibilidad 2

Did you get any errors displayed after assignments preloading file was processed?

You might also try use API to create assignments as well.

No errors were displayed by SoSo, in fact the messages presented were referred to as a successful upload process. We discovered after that not all the assignments containing the file were created in the SuSo database.

Right now we are creating a program that connects to Suso API and determines which assignments were not created. This is a massive problem because from 5M cases we don’t know how many assignments we need to create in Suso.

This program is not that efficient because first we have to call the endpoint /api/v1/assignments and it does not return the identifying data, just the assignment id. Then for each assignment returned in the first call, we have to call the endpoint /api/v1/assignments/{id} to collect the identifying data and then compare the households included in the file. For those households that do not have assignments created in SuSo, we will create it by using the API.

Hello,
I read your post and I am very interested in your experience with multiple nodes. I am testing with only 3 nodes and I am losing transactions. Were you able to resolve the incident?
Any help I would greatly appreciate.

Regards

Cristian

Hi Cristian,

My personal conclusion is that the process of uploading files to create very large workloads (in the case I described in Nov 22 we had to create 6.4M assignments ) is not efficient and important percentage of the transactions are lost. If I have to face that problem again I would use the API but I recommend an implementation of multithreaded programming for it. We built processes that handled multiple threads to address challenges like what this thread describes.
Hi Cristian,

My personal conclusion is that the process of uploading files to create very large workloads (in the case I described in Nov 2022 we had to create 6.4M assignments ) is not efficient it loses transactions. If I have to face this challenge again I would use the API but I recommend an implementation of multithreaded programming for it. In that project, we end developing several processes using multiple threads approach to address similar challenges like the one described in this thread.

Vladimir
Vladimir

First, thank you very much for your response.
Regarding the point indicated “Nov 2022 we had to create 6.4M assignments…”, that number of transactions, how long was it? Was the deployment of multiple nodes only with a load balancer?

Thank you in advance for sharing your experience.

Best Regards!!

PD: “Do you speak Spanish?”

Hola Cristian,
Si, hablo español. No entendí tu pregunta sobre el load balancer.