rm_rf throws ZK::Exception::NotEmpty

[lukaszx0]

There seems to be a race condition somewhere in rm_rf implementation.

I have a rspec tests with after(:each) block calling zk.rm_rf('/LOCAL') which basically wipes all data in ZK before next test for me (I'm not creating any other nodes at root level). This however results in a random flakiness with exceptions which doesn't quite make sense to me:

     ZK::Exceptions::NotEmpty:
       inputs: {:path=>"/LOCAL/test/foobar/private/groups/foobar", :version=>-1}
     # /Users/lukasz/.rvm/gems/ruby-2.2.1/gems/zk-1.9.5/lib/zk/client/base.rb:1083:in `check_rc'
     # /Users/lukasz/.rvm/gems/ruby-2.2.1/gems/zk-1.9.5/lib/zk/client/base.rb:1072:in `call_and_check_rc'
     # /Users/lukasz/.rvm/gems/ruby-2.2.1/gems/zk-1.9.5/lib/zk/client/threaded.rb:584:in `call_and_check_rc'
     # /Users/lukasz/.rvm/gems/ruby-2.2.1/gems/zk-1.9.5/lib/zk/client/base.rb:792:in `delete'
     # /Users/lukasz/.rvm/gems/ruby-2.2.1/gems/zk-1.9.5/lib/zk/client/unixisms.rb:57:in `block in rm_rf'
     # /Users/lukasz/.rvm/gems/ruby-2.2.1/gems/zk-1.9.5/lib/zk/client/unixisms.rb:51:in `each'
     # /Users/lukasz/.rvm/gems/ruby-2.2.1/gems/zk-1.9.5/lib/zk/client/unixisms.rb:51:in `rm_rf'
     # /Users/lukasz/.rvm/gems/ruby-2.2.1/gems/zk-1.9.5/lib/zk/client/unixisms.rb:54:in `block (2 levels) in rm_rf'
     # /Users/lukasz/.rvm/gems/ruby-2.2.1/gems/zk-1.9.5/lib/zk/client/unixisms.rb:53:in `each'
     # /Users/lukasz/.rvm/gems/ruby-2.2.1/gems/zk-1.9.5/lib/zk/client/unixisms.rb:53:in `block in rm_rf'
     # /Users/lukasz/.rvm/gems/ruby-2.2.1/gems/zk-1.9.5/lib/zk/client/unixisms.rb:51:in `each'
     # /Users/lukasz/.rvm/gems/ruby-2.2.1/gems/zk-1.9.5/lib/zk/client/unixisms.rb:51:in `rm_rf'
     # /Users/lukasz/.rvm/gems/ruby-2.2.1/gems/zk-1.9.5/lib/zk/client/unixisms.rb:54:in `block (2 levels) in rm_rf'
     # /Users/lukasz/.rvm/gems/ruby-2.2.1/gems/zk-1.9.5/lib/zk/client/unixisms.rb:53:in `each'
     # /Users/lukasz/.rvm/gems/ruby-2.2.1/gems/zk-1.9.5/lib/zk/client/unixisms.rb:53:in `block in rm_rf'
     # /Users/lukasz/.rvm/gems/ruby-2.2.1/gems/zk-1.9.5/lib/zk/client/unixisms.rb:51:in `each'
     # /Users/lukasz/.rvm/gems/ruby-2.2.1/gems/zk-1.9.5/lib/zk/client/unixisms.rb:51:in `rm_rf'
     # /Users/lukasz/.rvm/gems/ruby-2.2.1/gems/zk-1.9.5/lib/zk/client/unixisms.rb:54:in `block (2 levels) in rm_rf'
     # /Users/lukasz/.rvm/gems/ruby-2.2.1/gems/zk-1.9.5/lib/zk/client/unixisms.rb:53:in `each'
     # /Users/lukasz/.rvm/gems/ruby-2.2.1/gems/zk-1.9.5/lib/zk/client/unixisms.rb:53:in `block in rm_rf'
     # /Users/lukasz/.rvm/gems/ruby-2.2.1/gems/zk-1.9.5/lib/zk/client/unixisms.rb:51:in `each'
     # /Users/lukasz/.rvm/gems/ruby-2.2.1/gems/zk-1.9.5/lib/zk/client/unixisms.rb:51:in `rm_rf'
     # /Users/lukasz/.rvm/gems/ruby-2.2.1/gems/zk-1.9.5/lib/zk/client/unixisms.rb:54:in `block (2 levels) in rm_rf'
     # /Users/lukasz/.rvm/gems/ruby-2.2.1/gems/zk-1.9.5/lib/zk/client/unixisms.rb:53:in `each'
     # /Users/lukasz/.rvm/gems/ruby-2.2.1/gems/zk-1.9.5/lib/zk/client/unixisms.rb:53:in `block in rm_rf'
     # /Users/lukasz/.rvm/gems/ruby-2.2.1/gems/zk-1.9.5/lib/zk/client/unixisms.rb:51:in `each'
     # /Users/lukasz/.rvm/gems/ruby-2.2.1/gems/zk-1.9.5/lib/zk/client/unixisms.rb:51:in `rm_rf'
     # /Users/lukasz/.rvm/gems/ruby-2.2.1/gems/zk-1.9.5/lib/zk/client/unixisms.rb:54:in `block (2 levels) in rm_rf'
     # /Users/lukasz/.rvm/gems/ruby-2.2.1/gems/zk-1.9.5/lib/zk/client/unixisms.rb:53:in `each'
     # /Users/lukasz/.rvm/gems/ruby-2.2.1/gems/zk-1.9.5/lib/zk/client/unixisms.rb:53:in `block in rm_rf'
     # /Users/lukasz/.rvm/gems/ruby-2.2.1/gems/zk-1.9.5/lib/zk/client/unixisms.rb:51:in `each'
     # /Users/lukasz/.rvm/gems/ruby-2.2.1/gems/zk-1.9.5/lib/zk/client/unixisms.rb:51:in `rm_rf'
     # ./spec/spec_helper.rb:20:in `block (2 levels) in <top (required)>'

Is it a sign of a bug in my code or rather race condition in ZK's rm_rf? Not sure if it matters but I'm using zk-server for testing.

@slyphon any ideas?

[slyphon]

rm_rf doesn't do retries if some other thread/connection/client happens to write to a path it's trying to recursively delete.

if you have

/foo/a
/foo/b

rm_rf will remove /foo/a, /foo/b (in either order, i can't recall) and then remove /foo. If you have a callback listening for events on /foo's children that creates node /foo/wat then the removal of /foo may fail if the creation beats the removal of /foo. This is a classic read-then-write concurrency issue, but one that's documented.

One thing to try may be to close your connection at the end of the test, and do cleanup with a fresh connection to ensure any registered watchers on the original connection are no longer active. (I'm pretty sure that's what the zk unittests do, fwiw).

[lukaszx0]

Thanks! This makes sense. I'll make sure I'm always closing ZK connection
after tests and do clean up after that with fresh one.

On Sat, Jul 18, 2015 at 6:58 AM, Jonathan Simms [email protected]
wrote:

rm_rf doesn't do retries if some other thread/connection/client happens
to write to a path it's trying to recursively delete.

if you have

/foo/a
/foo/b

rm_rf will remove /foo/a, /foo/b (in either order, i can't recall) and
then remove /foo. If you have a callback listening for events on /foo's
children that creates node /foo/wat then the removal of /foo will fail.
This is a classic read-then-write concurrency issue, but one that's
documented.

One thing to try may be to close your connection at the end of the test,
and do cleanup with a fresh connection to ensure any registered watchers
have been freed. (I'm pretty sure that's what the zk unittests do, fwiw).

—
Reply to this email directly or view it on GitHub
#88 (comment).