Entity update submission blocked in backlog after connection issues

[srujner]

Problem description
While checking PR: 815 a strange behavior happened by accident. We were trying to create a Soft Conflict and just as I had the Entity Upgrade form in the backlog and I Wiktor used command to pushed it, I lost my internet connection and the submission locked up with information in the Activity Feed that I had not seen before link: https://staging.getodk.cloud/#/projects/130/forms/color_update/submissions/uuid%3A4ca11b30-1078-413e-81a1-c122ddbd3b7f
.(visible on the screenshot below).
I waited an hour to see if the problem would resolve itself, then tried another push backlog and no submission was sent.
I then added another Entity Update form and it ended up in the same state, but with one thing different (https://staging.getodk.cloud/#/projects/130/forms/color_update/submissions/uuid%3A61faebb3-2ab1-47a0-b23e-acdc73f39c6b)
As you can see at the second screenshot there was 12 minutes between two entries about offline update chain that might be because at 13:25 I sent the registration entity submission and after that pushed backlog one more time (this time with information: "Submissions processed: 1".
The entity registration submission although did not create any Entity: https://staging.getodk.cloud/#/projects/130/forms/color_registration/submissions/uuid%3A94d86448-ba66-4e2a-821a-0f39621a0d73

Let me know if you need any additional information about this case or maybe we can catch on some late meeting.

URL of the page
https://staging.getodk.cloud/#/projects/101/app-users](https://staging.getodk.cloud/#/projects/130/forms/color_update/submissions/uuid%3A4ca11b30-1078-413e-81a1-c122ddbd3b7f)

Steps to reproduce the problem
It would be hard to provide steps in this case, because it was random connection lost.
Probably problem description should be enough.

Expected behavior
Probably the entity should not be blocked in the backlog

"Other information"
First Screenshot

Second Screenshot

Central version shown in version.txt
515bfb9 (v2024.2.1-8-g515bfb9)
+55217eae8a7273b1003833a6c414a576f11855a6 client (v2024.2.1-51-g55217eae)
+38fd6226e8c93bc9bebcc211d19a067db2aa7b69 server (v2024.2.0-110-g38fd6226)

[matthew-white]

@ktuite, this might be a good one for you to look into if you have time.

[ktuite]

First off, sorry for not warning you all better about the messages in the activity feed! We snuck those in over the weekend and they are meant to show a submission going into and out of the backlog and make things less confusing, but they might have made things more confusing in this case.

Second, any time I see a submission go into the backlog and then come out of the backlog again without any entity being touched and no error, my first guess is that the dataset has "approvalRequired" set on it. I think that is all that is happening here!

https://staging.getodk.cloud/#/projects/130/entity-lists/colors/settings

Because of this issue (#682) a submission that would be an-update-applied-as-create does NOT create the entity when it is force processed from the backlog. This is because it didn't seem right to have a back door for creating entities without approval.

If dataset is configured for approvalRequired of submissions, then an entity update that is force applied should never result in a new entity. Applying an update-as-create should not happen in this case.

I think everything is working as expected here, but I did create an issue about showing more information about the underlying entity referenced in a submission that is stuck in the backlog. I think that could help clarify these scenarios #836

[srujner]

Yeah, because of the new entries in activity feed I completely forgot that there is approvalRequired set for that entity.

Will this entry be visible for all Entities in backlog or after creating an entity it will disappear from Activity feed?

[ktuite]

For now, these backlog events will always be visible. But we might change how they show up in the future or hide some of the events sometimes if it seems like they're not necessary.