Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Gazebo Plugin: Spawning URDF in Gazebo with large # of HEBI Actuators causes hebiros_node failure or Gazebo Plugin memory error (race condition?) #70

Open
josephcoombe opened this issue Mar 28, 2018 · 2 comments

Comments

@josephcoombe
Copy link
Contributor

josephcoombe commented Mar 28, 2018

Using HEBI-ROS (v.1.3.0).

Encountered some issues simulating a fairly large HEBI Hexapod URDF with 16 HEBI Actuators in Gazebo.

I've been able to generate 4 outcomes pretty consistently (on my machine):

  1. hebiros_node fails silently after creating group
    a) Launch Gazebo/spawn Hexapod URDF
    b) Start simulation
    c) Launch hebiros_node 5+ seconds later
    d) Launch my_node / call AddGroupFromNamesSrv service
    ❌ : hebiros_node fails silently after creating group
joseph@joseph-Aspire-VN7-591G:~$ roslaunch hebiros hebiros_gazebo.launch 
... logging to /home/joseph/.ros/log/6424bbc2-32cd-11e8-ad3d-3065ec753340/roslaunch-joseph-Aspire-VN7-591G-1217.log
Checking log directory for disk usage. This may take awhile.
Press Ctrl-C to interrupt
WARNING: disk usage in log directory [/home/joseph/.ros/log] is over 1GB.
It's recommended that you use the 'rosclean' command.

started roslaunch server http://joseph-Aspire-VN7-591G:42407/

SUMMARY
========

PARAMETERS
 * /rosdistro: kinetic
 * /rosversion: 1.12.13

NODES
  /
    hebiros_node (hebiros/hebiros_node)

ROS_MASTER_URI=http://localhost:11311

process[hebiros_node-1]: started with pid [1244]
Broadcasting on:
  192.168.1.255
[ INFO] [1522271853.884137512]: Using Gazebo
[ INFO] [1522271853.889273936]: Parameters:
[ INFO] [1522271853.889300673]: /hebiros/node_frequency=200
[ INFO] [1522271853.889320614]: /hebiros/action_frequency=200
[ INFO] [1522271853.889337735]: /hebiros/feedback_frequency=100
[ INFO] [1522271853.889352443]: /hebiros/command_lifetime=100
[ INFO] [1522271857.693422949, 3.986000000]: Created group [hexapod]:
[ INFO] [1522271857.693461425, 3.986000000]: /hexapod/Leg1/Hip
[ INFO] [1522271857.696260478, 3.987000000]: /hexapod/Leg1/Knee
[ INFO] [1522271857.699193837, 3.988000000]: /hexapod/Leg1/Ankle
[ INFO] [1522271857.701563500, 3.989000000]: /hexapod/Leg2/Hip
[ INFO] [1522271857.704214415, 3.990000000]: /hexapod/Leg2/Knee
[ INFO] [1522271857.706693322, 3.991000000]: /hexapod/Leg2/Ankle
[ INFO] [1522271857.708986570, 3.992000000]: /hexapod/Leg3/Hip
[ INFO] [1522271857.711288946, 3.993000000]: /hexapod/Leg3/Knee
[ INFO] [1522271857.714586326, 3.994000000]: /hexapod/Leg3/Ankle
[ INFO] [1522271857.717103043, 3.995000000]: /hexapod/Leg4/Hip
[ INFO] [1522271857.719666942, 3.996000000]: /hexapod/Leg4/Knee
[ INFO] [1522271857.722359279, 3.997000000]: /hexapod/Leg4/Ankle
[ INFO] [1522271857.724945592, 3.998000000]: /hexapod/Leg5/Hip
[ INFO] [1522271857.727315932, 3.999000000]: /hexapod/Leg5/Knee
[ INFO] [1522271857.729805885, 4.000000000]: /hexapod/Leg5/Ankle
[ INFO] [1522271857.732199354, 4.001000000]: /hexapod/Leg6/Hip
[ INFO] [1522271857.734937289, 4.002000000]: /hexapod/Leg6/Knee
[ INFO] [1522271857.737532682, 4.003000000]: /hexapod/Leg6/Ankle
[hebiros_node-1] process has died [pid 1244, exit code -11, cmd /home/joseph/tto-robotics/behavior_library_catkin_ws/catkin_ws/devel/lib/hebiros/hebiros_node -use_gazebo true __name:=hebiros_node __log:=/home/joseph/.ros/log/6424bbc2-32cd-11e8-ad3d-3065ec753340/hebiros_node-1.log].
log file: /home/joseph/.ros/log/6424bbc2-32cd-11e8-ad3d-3065ec753340/hebiros_node-1*.log
all processes on machine have died, roslaunch will exit
shutting down processing monitor...
... shutting down processing monitor complete
done
  1. hebiros_node doesn't crash and successfully handles AddGroupFromNamesSrv service call
    a) Launch Gazebo/spawn Hexapod URDF
    b) Launch hebiros_node
    c) Start simulation
    d) Launch my_node / call AddGroupFromNamesSrv service
    ✔️ Everything seems to work! I can even close my_node and restart it (calling AddGroupFromNamesSrv gain) w/o any problems
    or
    a) Launch Gazebo/spawn Hexapod URDF
    b) Start simulation
    c) Launch hebiros_node immediately after step b) (~ 2-3 seconds later)
    d) Launch my_node / call AddGroupFromNamesSrv service
    ✔️ Everything seems to work! I can even close my_node and restart it (calling AddGroupFromNamesSrv gain) w/o any problems

  2. hebiros_node doesn't crash but handles AddGroupFromNamesSrv call incorrectly
    a) ??? (I can't consistently reproduce this one)
    b) Launch my_node / call AddGroupFromNamesSrv service
    the hebiros_node won't crash and will appear to handle the AddGroupFromNameSrv service call correctly but will only publish joint_state feedback for 1-2 of the HEBI Actuators instead of the entire set of 16.
    ✔️ If I close my_node and restart it, the hebiros_node will handle the AddGroupFromNamesSrv service call successfully

  3. Gazebo Plugin crashes with memory error
    a) ??? (I can't consistently reproduce this one)
    Gazebo Plugin will crash with a memory error and dump the following to the terminal
    hebiros_memory_error.zip

@josephcoombe
Copy link
Contributor Author

An additional note:
Everything usually works if I launch everything (gazebo_ros, hebiros, nodes, etc) as part of a single .launch file (though it's worth noting that roslaunch by design makes no guarantees regarding launch order or timing).

I have a .launch file that will load everything successfully when my laptop is plugged into a power supply, but the hebiros node / Gazebo Plugin will crash when my laptop is running off battery, confirming suspicions that at least part of the problem is in fact a race condition.

@dcconner
Copy link

dcconner commented Feb 6, 2020

I saw similar issue upgrading to Melodic running x-demo
Seems to be fixed by initializing node handle in plugin loader. Early in validation stage.
https://github.com/CNURobotics/HEBI-ROS/tree/melodic-devel

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants