Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

IndexError: index 8 is out of bounds for axis 0 with size 7 #33

Open
robmarkcole opened this issue Mar 11, 2022 · 10 comments
Open

IndexError: index 8 is out of bounds for axis 0 with size 7 #33

robmarkcole opened this issue Mar 11, 2022 · 10 comments

Comments

@robmarkcole
Copy link

Code:

#Specify path to the coco.json file
path_to_annotations = "data/coco.json"
#Specify the path to the images (if they are in a different folder than the annotations)
path_to_images = "data/image_patches/"

#Import the dataset into the pylable schema 
dataset = importer.ImportCoco(path_to_annotations, path_to_images=path_to_images, name="tanks_coco")
dataset.df.head(5)

Error:

---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
/Users/robin/Documents/GitHub/oil-storage-tank/pre-processing.ipynb Cell 7' in <cell line: 7>()
      [4](vscode-notebook-cell:/Users/robin/Documents/GitHub/oil-storage-tank/pre-processing.ipynb#ch0000010?line=3)[ path_to_images = "data/image_patches/"
      ]()[6](vscode-notebook-cell:/Users/robin/Documents/GitHub/oil-storage-tank/pre-processing.ipynb#ch0000010?line=5)[ #Import the dataset into the pylable schema 
----> ]()[7](vscode-notebook-cell:/Users/robin/Documents/GitHub/oil-storage-tank/pre-processing.ipynb#ch0000010?line=6)[ dataset = importer.ImportCoco(path_to_annotations, path_to_images=path_to_images, name="tanks_coco")
      ]()[8](vscode-notebook-cell:/Users/robin/Documents/GitHub/oil-storage-tank/pre-processing.ipynb#ch0000010?line=7)[ dataset.df.head(5)

File ~/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/pylabel/importer.py:94, in ImportCoco(path, path_to_images, name)
     ]()[89](file:///Users/robin/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/pylabel/importer.py?line=88)[ df.ann_category_id = df.ann_category_id.astype(str)
     ]()[91](file:///Users/robin/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/pylabel/importer.py?line=90)[ df[
     ]()[92](file:///Users/robin/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/pylabel/importer.py?line=91)[     ["ann_bbox_xmin", "ann_bbox_ymin", "ann_bbox_width", "ann_bbox_height"]
     ]()[93](file:///Users/robin/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/pylabel/importer.py?line=92)[ ] = pd.DataFrame(df.ann_bbox.tolist(), index=df.index)
---> ]()[94](file:///Users/robin/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/pylabel/importer.py?line=93)[ df.insert(8, "ann_bbox_xmax", df["ann_bbox_xmin"] + df["ann_bbox_width"])
     ]()[95](file:///Users/robin/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/pylabel/importer.py?line=94)[ df.insert(10, "ann_bbox_ymax", df["ann_bbox_ymin"] + df["ann_bbox_height"])
     ]()[97](file:///Users/robin/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/pylabel/importer.py?line=96)[ # debug print(df.info())
     ]()[98](file:///Users/robin/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/pylabel/importer.py?line=97)[ 
     ]()[99](file:///Users/robin/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/pylabel/importer.py?line=98)[ # Join the annotions with the information about the image to add the image columns to the dataframe

File ~/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/pandas/core/frame.py:4439, in DataFrame.insert(self, loc, column, value, allow_duplicates)
   ]()[4436](file:///Users/robin/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/pandas/core/frame.py?line=4435)[     raise TypeError("loc must be int")
   ]()[4438](file:///Users/robin/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/pandas/core/frame.py?line=4437)[ value = self._sanitize_column(value)
-> ]()[4439](file:///Users/robin/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/pandas/core/frame.py?line=4438)[ self._mgr.insert(loc, column, value)]()
File ~/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/pandas/core/internals/managers.py:1230, in BlockManager.insert(self, loc, item, value)
   [1220](file:///Users/robin/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/pandas/core/internals/managers.py?line=1219)[ """
   ]()[1221](file:///Users/robin/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/pandas/core/internals/managers.py?line=1220)[ Insert item at selected position.
   ]()[1222](file:///Users/robin/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/pandas/core/internals/managers.py?line=1221)[ 
   (...)
   ]()[1227](file:///Users/robin/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/pandas/core/internals/managers.py?line=1226)[ value : np.ndarray or ExtensionArray
   ]()[1228](file:///Users/robin/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/pandas/core/internals/managers.py?line=1227)[ """
   ]()[1229](file:///Users/robin/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/pandas/core/internals/managers.py?line=1228)[ # insert to the axis; this could possibly raise a TypeError
-> ]()[1230](file:///Users/robin/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/pandas/core/internals/managers.py?line=1229)[ new_axis = self.items.insert(loc, item)
   ]()[1232](file:///Users/robin/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/pandas/core/internals/managers.py?line=1231)[ if value.ndim == 2:
   ]()[1233](file:///Users/robin/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/pandas/core/internals/managers.py?line=1232)[     value = value.T

File ~/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/pandas/core/indexes/base.py:6602, in Index.insert(self, loc, item)
   ]()[6595](file:///Users/robin/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/pandas/core/indexes/base.py?line=6594)[ if arr.dtype != object or not isinstance(
   ]()[6596](file:///Users/robin/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/pandas/core/indexes/base.py?line=6595)[     item, (tuple, np.datetime64, np.timedelta64)
   ]()[6597](file:///Users/robin/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/pandas/core/indexes/base.py?line=6596)[ ):
   ]()[6598](file:///Users/robin/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/pandas/core/indexes/base.py?line=6597)[     # with object-dtype we need to worry about numpy incorrectly casting
   ]()[6599](file:///Users/robin/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/pandas/core/indexes/base.py?line=6598)[     # dt64/td64 to integer, also about treating tuples as sequences
   ]()[6600](file:///Users/robin/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/pandas/core/indexes/base.py?line=6599)[     # special-casing dt64/td64 https://github.com/numpy/numpy/issues/12550
   ]()[6601](file:///Users/robin/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/pandas/core/indexes/base.py?line=6600)[     casted = arr.dtype.type(item)
-> ]()[6602](file:///Users/robin/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/pandas/core/indexes/base.py?line=6601)[     new_values = np.insert(arr, loc, casted)
   ]()[6604](file:///Users/robin/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/pandas/core/indexes/base.py?line=6603)[ else:
   ]()[6605](file:///Users/robin/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/pandas/core/indexes/base.py?line=6604)[     # No overload variant of "insert" matches argument types
   ]()[6606](file:///Users/robin/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/pandas/core/indexes/base.py?line=6605)[     # "ndarray[Any, Any]", "int", "None"  [call-overload]
   ]()[6607](file:///Users/robin/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/pandas/core/indexes/base.py?line=6606)[     new_values = np.insert(arr, loc, None)  # type: ignore[call-overload]

File <__array_function__ internals>:180, in insert(*args, **kwargs)

File ~/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/numpy/lib/function_base.py:5280, in insert(arr, obj, values, axis)
   ]()[5278](file:///Users/robin/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/numpy/lib/function_base.py?line=5277)[ index = indices.item()
   ]()[5279](file:///Users/robin/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/numpy/lib/function_base.py?line=5278)[ if index < -N or index > N:
-> ]()[5280](file:///Users/robin/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/numpy/lib/function_base.py?line=5279)[     raise IndexError(f"index {obj} is out of bounds for axis {axis} "
   ]()[5281](file:///Users/robin/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/numpy/lib/function_base.py?line=5280)[                      f"with size {N}")
   ]()[5282](file:///Users/robin/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/numpy/lib/function_base.py?line=5281)[ if (index < 0):
   ]()[5283](file:///Users/robin/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/numpy/lib/function_base.py?line=5282)[     index += N

IndexError: index 8 is out of bounds for axis 0 with size 7]()
@alexheat
Copy link
Contributor

Hello @robmarkcole , could you share your dataset with me? That will help me to debug the issue

@robmarkcole
Copy link
Author

Hi @alexheat
it is https://www.kaggle.com/towardsentropy/oil-storage-tanks

@alexheat
Copy link
Contributor

alexheat commented Mar 11, 2022 via email

@alexheat alexheat transferred this issue from pylabel-project/samples Mar 13, 2022
@alexheat
Copy link
Contributor

alexheat commented Mar 13, 2022 •
edited
Loading

Hello @robmarkcole I figured out what the problem is. The .json file for that dataset is missing some of the fields that are usually in a COCO annotation file. For example it doesn't have the height or width of the images:

   {
        "id": 5142,
        "file_name": "52_4_2.jpg"
    },

It also doesn't have some of the usual metadata about the annotations.

Unfortunately this library won't work with that dataset. If you are really stuck I can try and help work around the issue. (Which would require some more Python code to either augment the JSON or borrow some code from this library or add some additional logic to this library.)

@robmarkcole
Copy link
Author

robmarkcole commented Mar 13, 2022 •
edited
Loading

@alexheat do you know of any validators that could be run against the file? Did a brief search but did not find one, although https://github.com/levan92/cocojson looks useful here

@alexheat
Copy link
Contributor

I am not aware of any validation tool or even an official place where the schema is documented. But the image height and width are included in the original COCO dataset. The height and width are required information to convert to other formats like YOLO because it requires calculating the size of the bounding box relative to the size of the image.

@robmarkcole
Copy link
Author

this looks like the spec https://cocodataset.org/#format-data

for this issue, I suggest raising an exception if the keys are missing.

@robmarkcole
Copy link
Author

robmarkcole commented Mar 14, 2022 •
edited
Loading

I have updated the json file but still get the error... file on https://gist.github.com/robmarkcole/286951efe0830d3e205666f68c2b490e

Possibly due to mismatch of keys using id and image_id.
Update, the following did not resolve:

for entry in in_json['annotations']:
    id = entry['image_id']
    entry['id'] = id

Possibly some conflict as raised in cocodataset/cocoapi#95

Alternatively, might be due to some images having no annotations? Nope, remove those and still error

@robmarkcole
Copy link
Author

OK the issue was missing area in the annotations

@alexheat
Copy link
Contributor

this looks like the spec https://cocodataset.org/#format-data

for this issue, I suggest raising an exception if the keys are missing.

Thank you. Glad you got it working. I will keep the issue open as a reminder to add the better error handling.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@robmarkcole @alexheat