[Feature] Enable cancelation of stuck "in progress" mass firmware updates

[tricksol]

On the mass update history tab where you see the running updates, there should be a button to cancel updates that seem to be stuck.

[kosli]

I have run into the same "issue" :-)

[nemesifier]

This is a rare corner case which shouldn't happen thanks to OPENWISP_FIRMWARE_UPGRADER_TASK_TIMEOUT (see relevant section in the readme).

It can happen though, if while an upgrade is in progress, a cold shut down is performed.

Now, how can we deal with this is the question.

Allow canceling or abort an in-progress upgrade can create other problems and more corner cases to handle.
The upgrade can be canceled if the firmware is not being flashed yet, but once the flashing starts, there's no point in aborting, it's too late.

I think the problem here is that while an upgrade is in progress users cannot do new upgrades, which is very annoying.
A quick fix is to open the django shell and execute this SQL query, just make sure there's no other real upgrade in progress!

UpgradeOperation.objects.filter(status='in-progress').update(status='failed')

A definitive solution could be the following: when a new upgraded operation is created if we detect another one in progress, let's check if it's old enough to be timed out, if it is we assume this corner case is happening, flag it as failed automatically and move on.

Now, I cannot promise we'll invest our resources in fixing this since we can easily work around it with 1 minute and it never happens to the OpenWISP instances we manage, but if anyone wants to contribute, feel free to implement this solution and send a patch.

[ghost]

Hey, I opened a draft PR, haven't tested it yet though as I don't have my env set up. https://github.com/openwisp/openwisp-firmware-upgrader/pull/238/files