Too many bugs, can not run any code with digit sensor

[DavidYaonanZhu]

No description provided.

[DavidYaonanZhu]

More detailed documentation and tutorial is needed

[robertocalandra]

Hi @DavidYaonanZhu,
Could you be a bit more specific about what is not working for you and what bugs are you encountering?
Have you already tried to follow the tutorials here: https://facebookresearch.github.io/PyTouch/docs/tutorials/intro
We would be happy to assist and improve the documentation wherever needed.

[DavidYaonanZhu]

Thank you for replying, I have been trying to run surface_3d in example folder

First thing is that where can I get data set? It requires path to data set

[AlphaBetaPhi]

Hi @DavidYaonanZhu,

Are you using DIGIT sensor hardware or are you looking to run examples with a dataset you've previously collected?

The surface_3d can use a dataset collected from physical DIGIT hardware, or datasets generated by TACTO

Are you specifically looking for the dataset files? I'll provide an update to the documentation on how to collect your own dataset.

[DavidYaonanZhu]

@AlphaBetaPhi Hi, I am trying to test pytouch with DIGIT sensor (hardware).
I have already assembled and tested basics functions of DIGIT.
I have not collect any dataset yet, it will be much better to have a tutorial about how to collect data.

[DavidYaonanZhu]

I have also tried to run the contact_area code.
I saved some frames from DIGIT sensor and set base_img_path and sample_img_path as:
base_img_path = "/home/ur/Downloads/digit/digit-interface/data/digit_1.png" sample_img_path = "/home/ur/Downloads/digi
However, there is an error saying:

File "/home/ur/anaconda3/envs/plot/lib/python3.9/site-packages/pytouch/tasks/contact_area.py", line 107, in _compute_contact_area
return poly, major_axis, major_axis_end, minor_axis, minor_axis_end
UnboundLocalError: local variable 'poly' referenced before assignment

[DavidYaonanZhu]

For the touch_detect.py
the error is:
cannot find model touchdetect_resnet18_DigitSensor.pth

[DavidYaonanZhu]

I guess I might have missed some process

[AlphaBetaPhi]

For the touch_detect.py the error is: cannot find model touchdetect_resnet18_DigitSensor.pth

The example contains various methods of initializing tasks,

Initialize defaults specified by the task,

    pt = pytouch.PyTouch(DigitSensor, tasks=[TouchDetect])
    is_touching, certainty = pt.TouchDetect(source.image)

Initialize with a custom configuration, including model,

 touch_detect = TouchDetect(DigitSensor, zoo_model="touchdetect_resnet18")

This can be specified to any model for touch detection providing the model has been submitted to the model zoo, touch_detect = TouchDetect(DigitSensor, zoo_model="touchdetect_my_model_name_here

Example:

import pytouch
from pytouch.handlers import ImageHandler
from pytouch.sensors import DigitSensor
from pytouch.tasks import TouchDetect

def touch_detect():
    source = ImageHandler("/path/to/image")

    # initialize with task defaults
    pt = pytouch.PyTouch(DigitSensor, tasks=[TouchDetect])
    is_touching, certainty = pt.TouchDetect(source.image)
    print(f"Is touching? {is_touching}, {certainty}")


if __name__ == "__main__":
    touch_detect()

[AlphaBetaPhi]

I have also tried to run the contact_area code. I saved some frames from DIGIT sensor and set base_img_path and sample_img_path as: base_img_path = "/home/ur/Downloads/digit/digit-interface/data/digit_1.png" sample_img_path = "/home/ur/Downloads/digi However, there is an error saying:

File "/home/ur/anaconda3/envs/plot/lib/python3.9/site-packages/pytouch/tasks/contact_area.py", line 107, in _compute_contact_area return poly, major_axis, major_axis_end, minor_axis, minor_axis_end UnboundLocalError: local variable 'poly' referenced before assignment

Can you provide the code?

[DavidYaonanZhu]

This is basically the code I use to save frames, I just modified some part of digit_demo.py from digit-interface

rom digit_interface.digit import Digit
from digit_interface.digit_handler import DigitHandler

digit = Digit("D00046", "Left Gripper")
digit.connect()

for rgb in rgb_list:
    digit.set_intensity_rgb(*rgb)
    time.sleep(1)
digit.set_intensity(15)

for i in range(10):
    digit.save_frame(path="./digit_%d.png"%i)
    time.sleep(1)

digit.disconnect()

And the contact_area code is:

import pytouch
from pytouch.handlers import ImageHandler
from pytouch.sensors import DigitSensor
from pytouch.tasks import ContactArea


def extract_surface_contact():
    base_img_path = "/home/ur/Downloads/digit/digit-interface/data/digit_1.png"
    sample_img_path = "/home/ur/Downloads/digit/digit-interface/data/digit_5.png"

    base_img = ImageHandler(base_img_path).nparray
    sample_img = ImageHandler(sample_img_path).nparray
    sample_img_2 = sample_img.copy()

    # initialize with default configuration of ContactArea task
    pt = pytouch.PyTouch(DigitSensor, tasks=[ContactArea])
    major, minor = pt.ContactArea(sample_img, base=base_img)

    print("Major Axis: {0}, minor axis: {1}".format(*major, *minor))
    ImageHandler.save("surface_contact_1.png", sample_img)

    # initialize with custom configuration of ContactArea task
    contact_area = ContactArea(base=base_img, contour_threshold=10)
    major, minor = contact_area(sample_img_2)

    print("Major Axis: {0}, minor axis: {1}".format(*major, *minor))
    ImageHandler.save("surface_contact_2.png", sample_img_2)

if __name__ == "__main__":
    extract_surface_contact()

[DavidYaonanZhu]

The example contains various methods of initializing tasks,

Initialize defaults specified by the task,

pt = pytouch.PyTouch(DigitSensor, tasks=[TouchDetect])
is_touching, certainty = pt.TouchDetect(source.image)

Initialize with a custom configuration, including model,

touch_detect = TouchDetect(DigitSensor, zoo_model="touchdetect_resnet18")
This can be specified to any model for touch detection providing the model has been submitted to the model zoo, > touch_detect = TouchDetect(DigitSensor, zoo_model="touchdetect_my_model_name_here

I commented out resnet18 part as suggested.
I modified the code as follows to use realtime digit stream.

def touch_detect():
    #source = ImageHandler("/home/ur/Downloads/digit/digit-interface/data/digit_1.png")

    digit = Digit("D00002", "Left Gripper")
    digit.connect()

    rgb_list = [(15, 0, 0), (0, 15, 0), (0, 0, 15)]

    # Cycle through colors R, G, B
    for rgb in rgb_list:
        digit.set_intensity_rgb(*rgb)
        time.sleep(1)

    digit.set_intensity(15)
    qvga_res = Digit.STREAMS["QVGA"]
    digit.set_resolution(qvga_res)

    # # Change DIGIT FPS to 15fps
    fps_30 = Digit.STREAMS["QVGA"]["fps"]["30fps"]
    digit.set_fps(fps_30)

pt = pytouch.PyTouch(DigitSensor, tasks=[TouchDetect])

    while True:
        frame = digit.get_frame()
        cv2.imshow('frame', frame)
        sample = ImageHandler(frame)
        is_touching, certainty = pt.TouchDetect(sample)
        print(f"Is touching? {is_touching}, {certainty}")
        if cv2.waitKey(1) == 27:
            break

But now the error says:
File "/home/ur/anaconda3/envs/plot/lib/python3.9/site-packages/PIL/Image.py", line 2957, in open
fp.seek(0)
AttributeError: 'numpy.ndarray' object has no attribute 'seek'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/ur/Downloads/digit/PyTouch/examples/touch_detect.py", line 59, in
touch_detect()
File "/home/ur/Downloads/digit/PyTouch/examples/touch_detect.py", line 46, in touch_detect
sample = ImageHandler(frame)
File "/home/ur/anaconda3/envs/plot/lib/python3.9/site-packages/pytouch/handlers/image.py", line 11, in init
self.img = Image.open(img_path).convert(convert)
File "/home/ur/anaconda3/envs/plot/lib/python3.9/site-packages/PIL/Image.py", line 2959, in open
fp = io.BytesIO(fp.read())
AttributeError: 'numpy.ndarray' object has no attribute 'read'

[DavidYaonanZhu]

Now I have solved the error, by referencing here

But, now the detection is always 1 even without touching the sensor, I guess it is the model issue?

[DavidYaonanZhu]

In addition, I am trying to run contact_area.py, the error says:

File "/home/ur/anaconda3/envs/plot/lib/python3.9/site-packages/pytouch/tasks/contact_area.py", line 107, in _compute_contact_area
return poly, major_axis, major_axis_end, minor_axis, minor_axis_end
UnboundLocalError: local variable 'poly' referenced before assignment

I am checking the error code
I guess "poly" is related to polyline drawing, and this error indicates something is wrong with contours detection?
Is it related to base_img?

UPDATE:
I have checked the code, the error is related to contours, if the sensor image contains fewer contours, then the "poly and other variables" are not initialized in the original code.
I will suggest implementing some checking mechanism.

[DavidYaonanZhu]

Now I will try with surface_3d.py

[AlphaBetaPhi]

@AlphaBetaPhi Hi, I am trying to test pytouch with DIGIT sensor (hardware). I have already assembled and tested basics functions of DIGIT. I have not collect any dataset yet, it will be much better to have a tutorial about how to collect data.

There is a dataset available through,

wget https://www.dropbox.com/s/19jmxxoobe3ixpl/local.zip -P $PKG_DIR

unzip $PKG_DIR/local.zip -d $PKG_DIR
rm -r $PKG_DIR/local.zip

The surface_3d example expects a dataset with the following structure:

root
-- 0000
---- image1
---- image2
-- 0001
-- abcd

Running with the above dataset the following is produced,

[AlphaBetaPhi]

Can you provide a few examples of your DIGIT outputs you are using?

[DavidYaonanZhu]

I mistakenly closed

[DavidYaonanZhu]

[DavidYaonanZhu]

For surface_3d.py:
The error I get is:

File "/home/ur/Downloads/digit/PyTouch/examples/surface_3d.py", line 103, in visualize_surface_3d
output = surface3d.point_cloud_3d(img_color=img)
File "/home/ur/anaconda3/envs/plot/lib/python3.9/site-packages/pytouch/tasks/surface_3d.py", line 115, in point_cloud_3d
normal_pred = self.normals(img_color)
File "/home/ur/anaconda3/envs/plot/lib/python3.9/site-packages/pytouch/tasks/surface_3d.py", line 62, in normals
img_normal_pred = self.model.color_to_normal(img_input).to("cpu")
File "/home/ur/anaconda3/envs/plot/lib/python3.9/site-packages/pytouch/models/pix2pix/pix2pix.py", line 93, in color_to_normal
img = self.preprocess_image(img_input)
File "/home/ur/anaconda3/envs/plot/lib/python3.9/site-packages/pytouch/models/pix2pix/pix2pix.py", line 120, in preprocess_image
img = Image.fromarray(img)
File "/home/ur/anaconda3/envs/plot/lib/python3.9/site-packages/PIL/Image.py", line 2815, in fromarray
raise TypeError("Cannot handle this data type: %s, %s" % typekey) from e
TypeError: Cannot handle this data type: (1, 1, 320), |u1

Seems to be related to pix2pix.py

[kimanton]

Now I have solved the error, by referencing here

But, now the detection is always 1 even without touching the sensor, I guess it is the model issue?

I also always get 1, with certainty always greater that 98-98%.
Do we need to train model for new sensors?
I am trying to use DIGITs

My code is almost the same as [DavidYaonanZhu](#22 (comment))

[vocdex]

Yes, the classification model, namely, "touchdetect_resnet_DigitSensor.pth" was trained for that particular sensor by PyTouch developers.
I guess you can fine-tune this pre-trained model with RGB images from your sensor, or train ResNet classification from scratch. It shows always above 90% for our sensors too.
On that note, pix2pix model, which is used for surface3d task, needs to be trained from scratch too. Both provided models are sensor specific and factors such as RGB lighting, gel properties differ for each sensor.

[kimanton]

Yes, the classification model, namely, "touchdetect_resnet_DigitSensor.pth" was trained for that particular sensor by PyTouch developers. I guess you can fine-tune this pre-trained model with RGB images from your sensor, or train ResNet classification from scratch. It shows always above 90% for our sensors too. On that note, pix2pix model, which is used for surface3d task, needs to be trained from scratch too. Both provided models are sensor specific and factors such as RGB lighting, gel properties differ for each sensor.

Do you know how to train or tune the model with the provided code?

[vocdex]

I am not sure how to do that with the provided code but you can refer to this repo for training pix2pix, CycleGan, MLP model for RGB to Surface3d mapping.
https://github.com/psodhi/tactile-in-hand/tree/main/inhandpy/scripts/img_translation
Again, it's not straight forward and you may have to do some tweaking for dataset creation for training.

[robertokcanale]

I have to agree. The repo i extremely buggy. I am trying to fix it myself but there is a large inconsistency from:
-the documentation
-the examples
-the the actual code and what actual works.

Let me say this clearly: this documentation PyTouchDocumentation is really BADLY MADE and things DO NOT WORK

Since DIGIT has become a commercial product with GelSight, this should be fixed. Half of the functionalities do not work. I do not know how they accepted a paper about a repo that does not work. (or maybe only through some hidden gimmicks known by the developers)