[ROS2] Pausing Cameras

[InvincibleRMC]

Hello is it possible to pause a flir camera using the ros-driver? Currently cannot run three flir cameras on my network. I can run two just fine but, am being limited by a 1 gig network. I only need 2 feeds at the same time so I am curios about somehow pausing a camera to free up network bandwidth and cycling through which cameras I display. I messed around with trigger_mode = "Software" but, did not get anywhere with it.

[berndpfrommer]

Are you using the ROS1 or the ROS2 driver?

[InvincibleRMC]

Using the ROS2 driver with Iron.

[berndpfrommer]

So at the moment this is not possible, but if you commit to testing, I will implement it. We have 3 options:

1. Service call to start/stop the camera. Not my favorite because pretty inconvenient.
2. Stop the camera the when the last subscriber is gone. The cleanest but with some potential to cause crashes. Could be difficult to implement but from a technical point my favorite
3. Have a boolean rosparam "run" that toggles start/stop.
   What are your thoughts?

[InvincibleRMC]

To me I believe a service or rosparam makes more sense. Option number 2 requires that a subscriber be destroyed then recreated to pause and unpause wouldn't that cause lots of unnecessary overhead/latency?

[berndpfrommer]

True, latency is there, but I'm more worried about the stability of the system. A Spinnaker start/stop is a pretty heavy weight operation and I'm starting to lose track of all the threads that are running. I did a draft implementation of 2) and in the process hosed my laptop such that none of the spinnaker cameras would start again, not even with SpinView, after replugging the cameras. I fear implementing a service call or a toggle parameter won't be much safer. The only upside there is that the code path is less frequently exercised than subscribe/unsubscribe. But that also means it's a feature that is likely to be not well tested and causes driver crashes every now and then.

[InvincibleRMC]

That makes sense. My use case doesn't require lots of toggles so if there is worse latency I could live with that. Is there anything else you want from me like the testing you had mentioned?

[berndpfrommer]

I need a few days to finish up another project I'm on. Hope to get to this one on the weekend.

[berndpfrommer]

Looking busy so must push this by another week, but not forgotten!

[berndpfrommer]

Please test the driver in this branch and let me know how it goes. It has a feature connect_while_subscribed that you must set to True in the launch file (default is False).

[InvincibleRMC]

So having tested the branch it definitely destroys and creates the node/publisher. However, my network usage remains the same. So this sadly does not seem to be helping with my network bandwidth limitations.

[berndpfrommer]

What do you mean with "destroys and creates the node/publisher"? The ROS node should be left intact, it is just that when you unsubscribe it should disconnect the ROS node from the camera, then reconnect it when you subscribe to the image again. Can you positively assure with "ros2 node info" and "ros2 topic info" that there are no more subscribers to the image?
When testing this with my USB3 cameras the CPU usage of the ROS node dropped to completely zero upon unsubscribing, indicating that there were no more images coming from the SDK to the ROS node.

[InvincibleRMC]

Sorry I was mistaken the ROS node was not destroyed. However it is still not closing the publisher.

ros2 topic info /surface/bottom_cam/image_raw
Type: sensor_msgs/msg/Image
Publisher count: 1
Subscription count: 0

My launch files looks like this. Do I have to do something in the .yaml file I am using as well?

import os

from ament_index_python.packages import get_package_share_directory
from launch.launch_description import LaunchDescription
from launch_ros.actions import Node
from launch_ros.parameter_descriptions import Parameter


def generate_launch_description() -> LaunchDescription:

    parameter_file = os.path.join(get_package_share_directory('rov_flir'), 'config',
                                  'blackfly_s.yaml')

    parameters = {
        'debug': False,
        'compute_brightness': False,
        'adjust_timestamp': True,
        'dump_node_map': False,
        # set parameters defined in blackfly_s.yaml
        'gain_auto': 'Continuous',
        'pixel_format': 'BayerRG8',
        'exposure_auto': 'Continuous',
        # These are useful for GigE cameras
        'device_link_throughput_limit': 125000000,
        'gev_scps_packet_size': 9000,
        # ---- to reduce the sensor width and shift the crop
        #   'image_width': 1280,
        #   'image_height': 720,
        # 'offset_x': 16,
        # 'offset_y': 0,
        'binning_x': 3,
        'binning_y': 3,
        'connect_while_subscribed': True,
        'frame_rate_auto': 'Off',
        'frame_rate': 30.0,
        'frame_rate_enable': True,
        'buffer_queue_size': 1,
        'trigger_mode': 'Off',
        'chunk_mode_active': True,
        'chunk_selector_frame_id': 'FrameID',
        'chunk_enable_frame_id': True,
        'chunk_selector_exposure_time': 'ExposureTime',
        'chunk_enable_exposure_time': True,
        'chunk_selector_gain': 'Gain',
        'chunk_enable_gain': True,
        'chunk_selector_timestamp': 'Timestamp',
        'chunk_enable_timestamp': True
    }

    # launches node to run front flir camera
    front_cam = Node(
        package='spinnaker_camera_driver',
        executable='camera_driver_node',
        name='front_cam',
        emulate_tty=True,
        output='screen',
        parameters=[Parameter('serial_number', '23473577'),
                    Parameter('parameter_file', parameter_file),
                    parameters]
    )

    # launches node to run bottom flir camera
    bottom_cam = Node(
        package='spinnaker_camera_driver',
        executable='camera_driver_node',
        name='bottom_cam',
        emulate_tty=True,
        output='screen',
        parameters=[Parameter('serial_number', '23473566'),
                    Parameter('parameter_file', parameter_file),
                    parameters]
    )

    return LaunchDescription([
        front_cam,
        bottom_cam
    ])

Another thing to note is I am using the POE mode for the cameras not usb.

[berndpfrommer]

Your launch file has the connect_while_subscribed flag set correctly, so that should work. The yaml file needs no adjustment. The publisher should also be there, and nobody is subscribing to it, so it should definitely disconnect.

For good measure I suggest you put a LOG_INFO("stopping camera") into stopCamera() in camera.cpp, just to be really sure the camera is stopped.

Also you could move the call to deinitCamera() from stop() to stopCamera() and see if that stops the stream of data. It will break the reconnect then but at least we'd learn if that shuts down the network stream.

Apparently GigE vision is based on UDP. You should be able to see a UDP listener appear (netstat -ap) as the camera connects, then see a stream of udp packets coming (tcpdump is helpful here). The UDP listener should go away when the driver exits (or successfully disconnects from the SDK).

[InvincibleRMC]

Ok it seems to be working great now. Apparently when I cloned with branch command I goofed and cloned the wrong branch.

[berndpfrommer]

Great! Please use it for maybe a week or two, and if you don't see crashes/hangs I'll merge it into the main development branch.

[InvincibleRMC]

An unrelated question but is there a way to turn off Camera::PrintStatus()? I see that the Camera constructor takes a useStatus parameter but, I can't see if there is a way to set it from the launch file.

[berndpfrommer]

The answer is: No. Please open a new issue if you want this implemented. I don't want to conflate the necessary changes with the connect_while_subscribed pull request.

[InvincibleRMC]

Bad news sometimes while still subscribed the image stream stops and doesn't resume without re-listening on the topic. This new issue was quite intermittent issue and took between 5-10 minutes to show up.

[berndpfrommer]

The only thing I can suggest is to put debug printout statements into checkSubscriptions() to print out the subscription count. Also put statements into startCamera() and stopCamera() so you know if the camera is actually running (meaning the SDK is sending images).

I've had my share of negative experiences with ROS2 transports aka rmw. Weird stuff like this happens to me sometimes and I blame it on myself because hey, <sarcasm>such basic stuff would be noticed by everyone else and it would be fixed, right? </sarcasm>. Bottom line: if this is not a problem with the driver, this may well be a rmw feature. The log you are showing seems to indicate that packets are coming in from the SDK. Why would the driver not publish? Quite possibly your listener somehow can't connect to the topic.

[berndpfrommer]

I don't really want to understand the details of this, but from what I understand others have experienced similar problems, too. How this can ever be acceptable, I don't know. I guess it's an engineering tradeoff.

ros2/rmw_fastrtps#677

[InvincibleRMC]

Ok as a solution I just added a watch dog that just checks every second to see if the frame updated on my subscriber so if it gets stuck it resets the subscriber. So this branch has solved my use case.

[InvincibleRMC]

@berndpfrommer Would it be possible to get this merged into iron-devel aswell? That is the ros distro I was testing on.

[berndpfrommer]

I'll merge it into iron but it requires an update to rosdistro (just submitted the PR for it) because I also added the new synchronized camera driver package. It could take weeks/months until this feature finds its way into the apt repository.

[InvincibleRMC]

Thanks so much. I don't mind building from source in the meantime.