Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Gem will break after Sidekiq 6.0 has been released #224

Open
Flauschbaellchen opened this issue Jan 31, 2019 · 29 comments
Open

Gem will break after Sidekiq 6.0 has been released #224

Flauschbaellchen opened this issue Jan 31, 2019 · 29 comments

Comments

@Flauschbaellchen
Copy link

Flauschbaellchen commented Jan 31, 2019

After updating sidekiq to version 5.2.5 the following warning is shown:

WARNING: PID file creation will be removed in Sidekiq 6.0, see #4045. Please use a proper process supervisor to start and manage your services
WARNING: Logfile redirection will be removed in Sidekiq 6.0, see #4045. Sidekiq will only log to STDOUT
WARNING: Daemonization mode will be removed in Sidekiq 6.0, see #4045. Please use a proper process supervisor to start and manage your services

The ticket: sidekiq/sidekiq#4045

As the gemspec denotes a requirement to sidekiq >= 3.4 but no upper limit, this gem will most likely break after the new version of sidekiq has been released.

The quickfix would be to additionally require sidekiq <= 6.0 and release a new version.
What needs to be done to fix this issue in the long run?

PS: I already had gem 'sidekiq', '~> 5.1' within my gemfile so I'm not directly affected but will prevent the update to any newer version of sidekiq.

@lufutu
Copy link

lufutu commented Mar 2, 2019

Any update or solution ?

@vilusa
Copy link

vilusa commented Mar 4, 2019

Hi everyone,
I am using sidekiq 5.2.5 version

Error: WARNING: Daemonization mode will be removed in Sidekiq 6.0, see #4045. Please use a proper process supervisor to start and manage your services
WARNING: PID file creation will be removed in Sidekiq 6.0, see #4045. Please use a proper process supervisor to start and manage your services

screen shot 2019-03-04 at 5 10 08 pm

This errors puts console, Is there any solution or resource for this problem ?

@Paprikas
Copy link
Contributor

Paprikas commented Apr 7, 2019

For those who use rbenv like me, I did quick manual on how to switch to systemd. https://gist.github.com/Paprikas/11fc850f81b687d9cbb7a8efb5ead208
You have to remove old monitoring\ running scripts to avoid conflict.

@JoeDupuis
Copy link

I think adding a sidekiq <= 6.0 restriction on the last version is a bad idea.
The right thing to do would be what the creator of sidekiq advocate.
https://www.mikeperham.com/2014/09/22/dont-daemonize-your-daemons/
Deprecating the daemonization and force using a service manager.

capistrano-sidekiq already works with sidekiq 6 if you use the systemd init feature.

If somebody want to keep deamonizing sidekiq they can still force the version of sidekiq and capistrano-sidekiq in their gemfile. If you lock the sidekiq version in the gemspec, you force people who do the right thing to fork capistrano-sidekiq if they want the latest version of sidekiq.
It's a lot easier and faster to lock the sidekiq version in your gemfile than it is to fork a gem.

Though, forcing systemd restrict deployment to linux servers. One way to deal with this would be to abstract the call to systemd in a class that we could substitute for other class if we want to use different service manager. Then adding a new service manager would be trivial.

@leoarnold
Copy link

For systemd-based deployments, it's basically:

set :sidekiq_roles, -> { :app }
set :sidekiq_systemd_unit_name, "sidekiq@#{fetch(:stage)}"

namespace :sidekiq do
  desc 'Quiet sidekiq (stop fetching new tasks from Redis)'
  task :quiet do
    on roles fetch(:sidekiq_roles) do
      # See: https://github.com/mperham/sidekiq/wiki/Signals#tstp
      execute :systemctl, '--user', 'kill', '-s', 'SIGTSTP', fetch(:sidekiq_systemd_unit_name), raise_on_non_zero_exit: false
    end
  end

  desc 'Stop sidekiq (graceful shutdown within timeout, put unfinished tasks back to Redis)'
  task :stop do
    on roles fetch(:sidekiq_roles) do
      # See: https://github.com/mperham/sidekiq/wiki/Signals#tstp
      execute :systemctl, '--user', 'kill', '-s', 'SIGTERM', fetch(:sidekiq_systemd_unit_name), raise_on_non_zero_exit: false
    end
  end

  desc 'Start sidekiq'
  task :start do
    on roles fetch(:sidekiq_roles) do
      execute :systemctl, '--user', 'start', fetch(:sidekiq_systemd_unit_name)
    end
  end

  desc 'Restart sidekiq'
  task :restart do
    on roles fetch(:sidekiq_roles) do
      execute :systemctl, '--user', 'restart', fetch(:sidekiq_systemd_unit_name)
    end
  end
end

after 'deploy:starting', 'sidekiq:quiet'
after 'deploy:updated', 'sidekiq:stop'
after 'deploy:published', 'sidekiq:start'
after 'deploy:failed', 'sidekiq:restart'

@lucascaton
Copy link

lucascaton commented Sep 2, 2019

Sidekiq 6 has been released and invalid option: --index error messages are now showing up during deploys:

05:40 sidekiq:start
      01 /usr/local/rvm/bin/rvm default do bundle exec sidekiq --index 0 --pidfile /app/shared/tmp/pids/sidekiq-0.pid --environment production --logfile /app/shared/log/sidekiq.log …
      01 invalid option: --index
      01 /app/shared/bundle/ruby/2.6.0/gems/sidekiq-6.0.0/lib/sidekiq/cli.rb:345:in `parse_options'
      01 /app/shared/bundle/ruby/2.6.0/gems/sidekiq-6.0.0/lib/sidekiq/cli.rb:201:in `setup_options'
      01 /app/shared/bundle/ruby/2.6.0/gems/sidekiq-6.0.0/lib/sidekiq/cli.rb:24:in `parse'
      01 /app/shared/bundle/ruby/2.6.0/gems/sidekiq-6.0.0/bin/sidekiq:11:in `<top (required)>'
      01 /app/shared/bundle/ruby/2.6.0/bin/sidekiq:23:in `load'
      01 /app/shared/bundle/ruby/2.6.0/bin/sidekiq:23:in `<main>'
      01 /app/shared/bundle/ruby/2.6.0/bin/ruby_executable_hooks:24:in `eval'
      01 /app/shared/bundle/ruby/2.6.0/bin/ruby_executable_hooks:24:in `<main>'
#<Thread:0x00007fd9f96195b0@~/.rvm/gems/ruby-2.6.4/gems/sshkit-1.20.0/lib/sshkit/runners/parallel.rb:10 run> terminated with exception (report_on_exception is true):
Traceback (most recent call last):
	19: from ~/.rvm/gems/ruby-2.6.4/gems/sshkit-1.20.0/lib/sshkit/runners/parallel.rb:12:in `block (2 levels) in execute'
	18: from ~/.rvm/gems/ruby-2.6.4/gems/sshkit-1.20.0/lib/sshkit/backends/abstract.rb:31:in `run'
	17: from ~/.rvm/gems/ruby-2.6.4/gems/sshkit-1.20.0/lib/sshkit/backends/abstract.rb:31:in `instance_exec'
	16: # etc...

mjgiarlo added a commit to sul-dlss/was-registrar-app that referenced this issue Sep 3, 2019
Now that Sidekiq 6 has been released, which removes the ability to daemonize
sidekiq, we must pin sidekiq until we are daemonizing sidekiq via systemd (vs.
the sidekiq executable in the bundle). Note the tracking issue in
capistrano-sidekiq: seuros/capistrano-sidekiq#224
@ragesoss
Copy link

ragesoss commented Sep 5, 2019

Thanks @leoarnold! Your example code was extremely useful.

@JoeDupuis
Copy link

JoeDupuis commented Sep 5, 2019

This gem does support user level systemd deployment already. With customizable service template included.

You just have to set the correct parameters.

https://github.com/seuros/capistrano-sidekiq/blob/master/README.md#integration-with-systemd

@leoarnold
Copy link

@twistedjoe True that, yet most users do not require the versatile support for the whole variety of init systems. Since this project is not very active (see screenshot below), it seems advisable (c.f. CVE-2019-15224) to replace it with four simple and maintainable customized Capistrano tasks rather than hoping for a fresh breeze in this project.

Screenshot_20190906_132153

@seuros
Copy link
Owner

seuros commented Sep 6, 2019

I'm going to release the new version this weekend, that support only sidekiq 6+.
As @twistedjoe noted, the current workaround is to use the upstart/systemd integration that is optional for now.
In the next version, it will be the only way to do it.

@jules-w2
Copy link

Hey @seuros,

Do you have an ETA for this release? (sidekiq 6+)
Thanks :)

@tcd
Copy link

tcd commented Oct 5, 2019

@seuros ping

@rusikf
Copy link

rusikf commented Oct 8, 2019

+1, I have

 /gems/sshkit-1.20.0/lib/sshkit/runners/parallel.rb:15:in `rescue in block (2 levels) in execute': Exception while executing as [email protected]: sidekiqctl exit status: 1 (SSHKit::Runner::ExecuteError)                                    
sidekiqctl stdout: Nothing written
sidekiqctl stderr: /home/deploy/.rvm/rubies/ruby-2.5.3/lib/ruby/site_ruby/2.5.0/bundler/rubygems_integration.rb:462:in `block in replace_bin_path': can't find executable sidekiqctl for gem sidekiq (Gem::Exception)                                                                      
        from /home/deploy/.rvm/rubies/ruby-2.5.3/lib/ruby/site_ruby/2.5.0/bundler/rubygems_integration.rb:482:in `block in replace_bin_path' 
        from /home/deploy/app_staging/shared/bundle/ruby/2.5.0/bin/sidekiqctl:23:in `<main>'                                                

after updating to sidekiq6 and deploying app

@tcd
Copy link

tcd commented Oct 14, 2019

@seuros ping ping

@ghost
Copy link

ghost commented Oct 14, 2019

Any news on this?

@ghost
Copy link

ghost commented Oct 30, 2019

gif

(this was just for fun btw, it's easier to complain than help)

@jules-w2
Copy link

jules-w2 commented Nov 6, 2019

@seuros ping

@rwojnarowski
Copy link

This gem does support systemd and sidekiq 6.0 but currently sidekiq is locked to version < 6, so I've created fork which allow to use newest sidekiq 6.0.3: https://github.com/rwojnarowski/capistrano-sidekiq

In short, to get this gem working with sidekiq 6.0:
In Gemfile:
gem "capistrano-sidekiq", git: "https://github.com/rwojnarowski/capistrano-sidekiq.git"
In deploy.rb:
set :init_system, :systemd
On your server:
loginctl enable-linger USERACCOUNT
From repo directory run:
bundle exec cap sidekiq:install
You may need to enable new service on your server, so:
systemctl --user enable sidekiq-production
systemctl --user start sidekiq-production
Check the status:
systemctl --user status sidekiq-production
and deploy new app as usual with Capistrano

@krtschmr
Copy link

@rwojnarowski well, my bundler path doesn't seem to be there tho

@ziaulrehman40
Copy link

ziaulrehman40 commented Nov 29, 2019

@rwojnarowski Worked like a charm, hopefully main gem will be updated soon.

UPDATE:

status is giving me this:

● sidekiq-production.service - sidekiq for simplypo (production)
   Loaded: loaded (/home/user/.config/systemd/user/sidekiq-production.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Fri 2019-11-29 10:55:32 UTC; 15min ago
 Main PID: 8450 (code=exited, status=203/EXEC)

Nov 29 10:55:32 ip-172-30-0-142 systemd[1430]: sidekiq-production.service: Service hold-off time over, scheduling restart.
Nov 29 10:55:32 ip-172-30-0-142 systemd[1430]: sidekiq-production.service: Scheduled restart job, restart counter is at 5.
Nov 29 10:55:32 ip-172-30-0-142 systemd[1430]: Stopped sidekiq for MYAPP (production).
Nov 29 10:55:32 ip-172-30-0-142 systemd[1430]: sidekiq-production.service: Start request repeated too quickly.
Nov 29 10:55:32 ip-172-30-0-142 systemd[1430]: sidekiq-production.service: Failed with result 'exit-code'.
Nov 29 10:55:32 ip-172-30-0-142 systemd[1430]: Failed to start sidekiq for MYAPP (production).

Update 2:

I tracked it down to the path of bundler in generated service was not relative to rvm. I change that line to ExecStart=/home/deployer/.rvm/gems/ruby-2.6.5/wrappers/bundle exec sidekiq -e production -C config/sidekiq.yml and boom.

For anyone playing around with systemctl for sidekiq or anything, i used following sequence of commands for hit and try:

sudo nano /home/USER/.config/systemd/user/sidekiq-production.service
# make your changes

# do everything to restart :dagger:  
sudo systemctl daemon-reload && systemctl --user reenable sidekiq-production.service && systemctl --user stop sidekiq-production.service && systemctl --user start sidekiq-production.service && sleep 3 && systemctl --user status sidekiq-production.service

Commenting this to help others as well as my log for next time.

@mihir-kumar-thakur
Copy link

mihir-kumar-thakur commented Dec 12, 2019

https://curationexperts.github.io/playbook/production/sidekiq_in_production.html

  1. gem 'capistrano-sidekiq' , group: :development

  2. Load the following in your Capfile
    require 'capistrano/sidekiq'

  3. Install sidekiq as a systemd service on your server(s)
    Start by adding Sidekiq specific settings to your config/deploy.rb file

  • Sidekiq service defaults
    set :init_system, :systemd
    set :service_unit_name, "sidekiq.service"
  1. sudo loginctl enable-linger deploy

  2. bundle exec cap $STAGE_NAME sidekiq:install
    after this you may go to server and check rbenv path
    /home/deploy/.config/systemd/user/sidekiq.service and check rbenv path

acoffman added a commit to griffithlab/civic-server that referenced this issue Jan 14, 2020
@agatheblues
Copy link

agatheblues commented May 7, 2020

Newbies corner for those out here who are struggling with this 🤼‍♀️
I got it to work by re-reading that line and giving it the proper attention:

after this you may go to server and check rbenv path
/home/deploy/.config/systemd/user/sidekiq.service and check rbenv path

This means: open that file and check that the ExecStart is using the correct path to your ruby bundle. If you're using rbenv, chances are, it is not the correct path.

ExecStart=/bin/bash -lc 'exec /home/deploy/.rbenv/shims/bundle exec sidekiq -e production'

Then, capistrano will tell you (check the logs, I'm learning that it's a healthy habit 🧘‍♀️ ) when deploying that this file got changed and you have to systemctl --user daemon-reload. Do that. Then ps aux | grep '[s]idekiq' should tell you that capistrano managed to start sidekiq like a breeze 🎉

@kannancet
Copy link

I struggled with this a lot. Downgraded to 5.2.7 without wasting time on systemctl and things worked!

@avvari-da
Copy link

avvari-da commented May 10, 2020

This & This (together) worked for me! 🎉

@krthi
Copy link

krthi commented Aug 28, 2020

Any update on the support for Sidekiq 6?

@ramonrails
Copy link

ramonrails commented Sep 21, 2020

This & This (together) worked for me! 🎉

This still fails for me :(
rbenv, ruby 2.7.1, sidekiq 6.1.2, capistrano-sidekiq master@/rwojnarowski, ExecStart updated to match rbenv bundle

sidekiq-production.service - sidekiq for commerce (production)
     Loaded: loaded (/home/deploy/.config/systemd/user/sidekiq-production.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Mon 2020-09-21 13:50:08 IST; 1s ago
    Process: 582875 ExecStart=/usr/local/rbenv/bin/rbenv exec bundle exec sidekiq -e production (code=exited, status=1/FAILURE)
   Main PID: 582875 (code=exited, status=1/FAILURE)

Sep 21 13:50:08 e2e-70-40 systemd[1829]: sidekiq-production.service: Scheduled restart job, restart counter is at 5.
Sep 21 13:50:08 e2e-70-40 systemd[1829]: Stopped sidekiq for commerce (production).
Sep 21 13:50:08 e2e-70-40 systemd[1829]: sidekiq-production.service: Start request repeated too quickly.
Sep 21 13:50:08 e2e-70-40 systemd[1829]: sidekiq-production.service: Failed with result 'exit-code'.
Sep 21 13:50:08 e2e-70-40 systemd[1829]: Failed to start sidekiq for commerce (production).

working directory is also set

WorkingDirectory=/var/www/<project>/commerce/current
ExecStart=/usr/local/rbenv/bin/rbenv exec bundle exec sidekiq -e production

but works when I run it from current_path manually at the server

deploy@e2e-70-40:/var/www/<project>/commerce/current$ /usr/local/rbenv/bin/rbenv exec bundle exec sidekiq -e production
2020-09-21T08:31:14.057Z pid=583216 tid=ck34 INFO: Booting Sidekiq 6.1.2 with redis options {:url=>"redis://localhost:6379/0"}
2020-09-21T08:31:14.403Z pid=583216 tid=ck34 INFO: Booted Rails 6.0.3.3 application in production environment
2020-09-21T08:31:14.403Z pid=583216 tid=ck34 INFO: Running in ruby 2.7.1p83 (2020-03-31 revision a0c7c23c9c) [x86_64-linux]
2020-09-21T08:31:14.403Z pid=583216 tid=ck34 INFO: See LICENSE and the LGPL-3.0 for licensing details.
2020-09-21T08:31:14.404Z pid=583216 tid=ck34 INFO: Upgrade to Sidekiq Pro for more features and support: https://sidekiq.org

@gauravm31
Copy link

@leoarnold I followed your steps. Created a sidekiq.service file and tried to use that in deploy.rb.

I cant seem to get this working. While deploying, capistrano cant find the service file.

On my server, i can start sidekiq manually with systemctl start sidekiq. It asks for user authentication but then works. If I do systemctl --user start sidekiq, it says "Failed to restart sidekiq.service: Unit sidekiq.service not found".

Any idea about this.

@leoarnold
Copy link

@gauravm31 That is because you installed the systemd unit in system space, not in user space. To run services in user space, enable-linger for the unprivileged user and then follow e.g. https://computingforgeeks.com/how-to-run-systemd-service-without-root-sudo/

@gauravm31
Copy link

@leoarnold Followed the steps. Then there was another issue regarding user/group. If we user systemctl with --user, we need to remove the user and group setting from service file as it is not needed. Otherwise it breaks. explained here https://unix.stackexchange.com/questions/438064/

after removing user and group settings, it works fine. Thanks for the help.

DevneyHamilton added a commit to seedcommons/madeline that referenced this issue Mar 9, 2022
sourcefilter added a commit to UCLALibrary/ursus that referenced this issue Apr 26, 2022
sidekiq 6 is not compatible with the way we are using capistrano-sidekiq, resulting in failed deployments even though everything installed successfully in development and CI.

More info:
seuros/capistrano-sidekiq#224
pghorpade pushed a commit to UCLALibrary/ursus that referenced this issue Apr 27, 2022
sidekiq 6 is not compatible with the way we are using capistrano-sidekiq, resulting in failed deployments even though everything installed successfully in development and CI.

More info:
seuros/capistrano-sidekiq#224
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests