Tool to scan a (Mac) Photos library for duplicates, thumbnails etc.
The photosdup tool is available from PyPI:
pip install photosdup
This version can also scan arbitrary directories with images that are not in a Mac Photos library. In this case, the --no-thumbs option is assumed and the --tag options is ignored.
The newest version of Photos Duplicates supports all the image formats of OpenCV plus the HEIC format produced by iPhones.
After installation, photosdup can be run by providing it with the directory of the Photos library you would like to scan.
python -m photosdup Pictures/Photos.photoslibrary
By default, the command line version prints a list of lists of images, with each interior list representing an original as its first element and its likely duplicates as the remaining elements.
By default, photsdup uses the thumbnails pre-computed by Photos. If this causes any issues or you want to use larger scales, you scan scale originals instead:
python -m photosdup Pictures/Photos.photoslibrary --no-thumbs
To tag originals and duplicates with keywords and create albums for them instead, you can add the following parameter:
python -m photosdup Pictures/Photos.photoslibrary --tag
You can instantiate a DuplicateFinder object by providing the path of the library to scan and using the scan convenience function:
from photosdup import DuplicateFinder df = DuplicateFinder("Picturs/Photos Library.photoslibrary") print(df.scan())
Finer control is available through the functions load, represent, find, and tag. See the implementation of scan for the typical usage.
There is an experimental graphical user interface, which attempts to aid in locating the Photos library and set the parameters (see next section).
python -m photosdup --gui
The graphical user interface always tags originals and duplicates with keywords and creates albums for them.
Several parameters such as the dimensions of the scaled down image used for comparison can be provided in both the command line and the graphical user interface. For an explanation and overview, just use the help function.
python -m photosdup --help
If the graphical user interface has stability problems, force single core code using 0 for the cores parameter.
The result of the scan is stored by photosdup in two ways:
- each time a duplicate is found, the higher-quality image (as judged by total file size) is tagged with the keyword photosdup-duplicate while the lower-quality duplicates are tagged with the keyword photosdup-original.
- each set of original and duplicates is tagged with the UUID of the original and put in an album called photosdup-UUID.
The approach used for scaling images is inspired by the approach taken in difPy. Unfortunately, difPy could not be used as it does not integrate with the Photos database (minor nuisance regarding updating the database) and uses a quadratic algorithm that compares each image to all other images, i.e., N*(N-1) comparisons for N images. Nevertheless, the approach described here was a great inspiration: https://towardsdatascience.com/finding-duplicate-images-with-python-71c04ec8051
The search for duplicates and near duplicates uses a radius query on a KD tree. The SciPy implementation was used as it supports parallelization via multiple threaded workers: https://docs.scipy.org/doc/scipy/reference/generated/scipy.spatial.KDTree.query_ball_point.html#scipy.spatial.KDTree.query_ball_point
Last, but not least, iterative deepening in the form of lower-resolution scanning is used to eliminate likely non-duplicates from the more costly higher-resolution scanning. This aspect can be controlled by the --xdims, --ydims, and --radiuses parameters (cf. the output of --help).