juggernaut server push is broken

[s-edwards]

Tony reported that server push notifications are no longer working. This affects simultaneous viewing of live instructor examples, where edits by the instructor would be pushed to update content on all browsers viewing the example, and also affects submission of code for grading, where test results from server-side checks would be pushed to the browser for viewing by the student. This is an extremely critical bug that needs to be fixed immediately.

[s-edwards]

OK, two issues. First, apparently the juggernaut service does not automatically restart on server boot (still needs to be fixed).

Second, juggernaut appears to be working fine on the production server now. In particular, the juggernaut log does not show any errors, and the server-push features used in examples works. I can create a new example, open two browsers on it, and see the push in action.

However, server-push of grading results is not working. From the juggernaut log, it is clear that the pub/sub requests are being made. However, in most cases "results" for assignments are only being subscribed to, but never published. After poking around, I was able to get grading to work on the staging instance of Pythy by manually restarting sidekiq.

It looks/feels like the correct startup for sidekiq is currently only accomplished via capistrano. I don't see how to make it work separately. We need to find a way to have it auto-start when the server comes up. In the mean time, I'll see if I can get it going on the production server. ... It worked!

[ArturAguiar]

So at the moment juggernaut is working perfectly in production?
Is it the same in development (pythydev)?

[s-edwards]

Yes, both production and pythydev are both working correctly for grading results and other juggernaut-based messaging. I believe the issue is how to correctly ensure that juggernaut and sidekiq are both started correctly when the app is restarted or the server is restarted (the same is true of redis, but redis does not restart with the app, and is correctly configured to auto-start when the server starts). Basically, we have three separate daemon-style services that need to run alongside the pythy rails app:

1. redis (runs once on the server, serves both pythy and pythydev, currently starts automatically)
2. juggernaut (runs once on the server, serves both pythy and pythydev, currently does not start automatically)
3. sidekiq (need a separate running instance for each application, currently does not start automatically unless it is happening through capistrano-based deploy/restart)

Fixing (2) is fairly easy, I think. I'm not sure about (3), and will have to look into it ...

[s-edwards]

It turns out that CentOS 6 supports Upstart (borrowed from Ubuntu), so we can probably use the Upstart-based solution here:

https://github.com/mperham/sidekiq/tree/master/examples/upstart/manage-many