Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Syncing watchlist between Trakt and Plex always adds back removed items #1247

Open
3 tasks done
rfgamaral opened this issue Dec 1, 2022 · 33 comments
Open
3 tasks done
Labels
enhancement New feature or request

Comments

@rfgamaral
Copy link

rfgamaral commented Dec 1, 2022

Confirmation

  • I have read the README.md on the project homepage
  • I have checked if identical issue already exists
  • I have tried downgrading to find version that can be used as a workaround

The problem

Syncing my watchlist between Trakt and Plex does not preserve remove state, that is:

  • If I remove an item from my Trakt watchlist, I expected the item to be removed from Plex on the next sync
  • If I remove an item from my Plex watchlist, I expected the item to be removed from Trakt on the next sync

Error trace / logs

I removed 3 movies from my Trakt Watchlist and ran plextraktsync sync --sync=movies:

2022-12-01 12:19:58,946 INFO[PlexTraktSync]:PlexTraktSync [0.24.5]
2022-12-01 12:19:59,275 INFO[PlexTraktSync]:Sync Movie sections: [<PlexLibrarySection:movie:Concerts>, <PlexLibrarySection:movie:Movies>]
2022-12-01 12:20:01,890 INFO[PlexTraktSync]:Concerts processed in 2.6 seconds
2022-12-01 12:20:09,803 INFO[PlexTraktSync]:Movies processed in 7.9 seconds
2022-12-01 12:20:17,613 INFO[PlexTraktSync]:Adding 'Amsterdam' to Trakt watchlist
2022-12-01 12:20:31,097 INFO[PlexTraktSync]:Adding 'Black Panther: Wakanda Forever' to Trakt watchlist
2022-12-01 12:20:35,505 INFO[PlexTraktSync]:Adding 'Deepwater Horizon' to Trakt watchlist
2022-12-01 12:20:48,794 INFO[PlexTraktSync]:Skipping 'Untitled Ghostbusters: Afterlife Sequel' from Trakt watchlist because not found in Plex Discover
2022-12-01 12:20:52,170 INFO[PlexTraktSync]:Skipping 'The King' from Trakt watchlist because not found in Plex Discover
2022-12-01 12:20:52,185 INFO[PlexTraktSync]:Updated watchlist and/or liked list in 40.7 seconds
2022-12-01 12:20:52,322 INFO[PlexTraktSync]:Completed full sync in 53.1 seconds

I removed 3 movies from my Plext Watchlist and ran plextraktsync sync --sync=movies:

2022-12-01 12:21:57,098 INFO[PlexTraktSync]:PlexTraktSync [0.24.5]
2022-12-01 12:21:57,184 INFO[PlexTraktSync]:Sync Movie sections: [<PlexLibrarySection:movie:Concerts>, <PlexLibrarySection:movie:Movies>]
2022-12-01 12:21:58,890 INFO[PlexTraktSync]:Concerts processed in 1.7 seconds
2022-12-01 12:22:06,670 INFO[PlexTraktSync]:Movies processed in 7.8 seconds
2022-12-01 12:22:59,716 INFO[PlexTraktSync]:Skipping 'Untitled Ghostbusters: Afterlife Sequel' from Trakt watchlist because not found in Plex Discover
2022-12-01 12:23:04,296 INFO[PlexTraktSync]:Skipping 'The King' from Trakt watchlist because not found in Plex Discover
2022-12-01 12:23:04,919 INFO[PlexTraktSync]:Adding 'Amsterdam' to Plex watchlist
2022-12-01 12:23:06,450 INFO[PlexTraktSync]:Adding 'Black Panther: Wakanda Forever' to Plex watchlist
2022-12-01 12:23:07,845 INFO[PlexTraktSync]:Adding 'Deepwater Horizon' to Plex watchlist
2022-12-01 12:23:08,483 INFO[PlexTraktSync]:Updated watchlist and/or liked list in 1 min 0.1 seconds
2022-12-01 12:23:08,486 INFO[PlexTraktSync]:Completed full sync in 1 min 11.3 seconds

Expected behavior

  • Removed items from Plex Watchlist should be removed from Trakt Watchlist on the next sync
  • Removed items from Trakt Watchlist should be removed from Plex Watchlist on the next sync

Steps to reproduce the behavior

Any sync command results in the issue.

Inspect of problematic items

No response

Workarounds

No response

Install method

docker-compose

Config file contents

# Config File: /app/config/config.yml
cache:
  path: /app/config/.cache/trakt
config:
  dotenv_override: true
excluded-libraries:
- Samples
logging:
  append: true
  console_time: true
  debug: false
  filename: plextraktsync.log
  filter: null
plex:
  timeout: 30
sync:
  plex_to_trakt:
    collection: false
    ratings: false
    watched_status: true
    watchlist: true
  trakt_to_plex:
    liked_lists: false
    ratings: true
    watched_status: true
    watchlist: true
    watchlist_as_playlist: false
watch:
  add_collection: true
  remove_collection: true
  scrobble_threshold: 80
  username_filter: true
xbmc-providers:
  movies: imdb
  shows: tvdb

Version

0.24.5

Python Version

3.11.0

Operating System and Version

Synology (Linux)

@simonc56
Copy link
Collaborator

simonc56 commented Dec 1, 2022

When there is a Movie in one watchlist but not in the other watchlist, how could the script kow if it should remove it from the first watchlist or add it in the second wtchlist ?

@rfgamaral
Copy link
Author

I can't work out the details right now because I don't know how exactly each API works, but it should be possible by keeping some local state for the previous sync for each watchlist, and do some condition checks.

Right now, the way it's implemented, it acts more like a "mirror" than a "sync".

@glensc

This comment was marked as off-topic.

@simonc56

This comment was marked as off-topic.

@rfgamaral

This comment was marked as off-topic.

@simonc56
Copy link
Collaborator

simonc56 commented Dec 1, 2022

Workaround is to sync watchlist only one-way (plex_to_trakt or trakt_to_plex).

@rfgamaral
Copy link
Author

Not really a workaround for me, I have no use for having just one of them enabled. Either 2-way sync works taking into account removed items, or I might as well just turn off both plex_to_trakt and trakt_to_plex 😔

@simonc56 simonc56 added the enhancement New feature or request label Dec 1, 2022
@glensc
Copy link
Collaborator

glensc commented Dec 27, 2022

I noticed the trakt response includes list: updated_at, could that be useful?

2022-12-27 09:39:21,429 DEBUG[PlexTraktSync]:Updated Trakt watchlist: {'deleted': {'movies': 3, 'shows': 1, 'seasons': 0, 'episodes': 0}, 'list': {'updated_at': '2022-12-27T07:39:20.000Z', 'item_count': 29}}

also, trakt side/sync/watchlist and /users/id/watchlist have listed_at timestamp. maybe useful?

    "listed_at": "2014-09-01T09:10:11.000Z",

@glensc
Copy link
Collaborator

glensc commented Jan 2, 2023

so, the trakt response has:

{'id': 773647533,
  'listed_at': '2022-12-07T10:34:20.000Z',
  'movie': {'ids': {'imdb': 'tt0070948',
                    'slug': 'zardoz-1974',
                    'tmdb': 4923,
                    'trakt': 2858},
            'title': 'Zardoz',
            'year': 1974},
  'notes': None,
  'rank': 8,
  'type': 'movie'}

but all data that's not inside movie gets lost, because only movie data is collected:

so would need new data class to pytrakt that has listed_at, notes, movie, rank, type properties?

@glensc
Copy link
Collaborator

glensc commented Jan 2, 2023

actually, watchlist_shows would pass the extra data to TVShow:

but I guess a new class with extra fields would be a cleaner solution.

@simonc56
Copy link
Collaborator

simonc56 commented Jan 2, 2023

I noticed the trakt response includes list: updated_at, could that be useful?

Explain how useful it is please.

@glensc
Copy link
Collaborator

glensc commented Jan 2, 2023

I noticed the trakt response includes list: updated_at, could that be useful?

Explain how useful it is please.

please note the question form of the sentence.

the idea was to compare plex side watchlistedAt vs trakt side listed_at. for this to be useful, need the third timestamp, when was the last sync ran, such timestamp is not available.

do you see a solution how to do with only two timestamps? something like compare newest entry in plex = last update in plex side, and same for trakt?

@rfgamaral
Copy link
Author

@glensc I have a Trakt Pro subscription, and I could make a feature request on the Trakt VIP forums to improve the API to allow this use case, I just need to know the exact requirements. Let me know how I can help.

@simonc56
Copy link
Collaborator

simonc56 commented Jan 2, 2023

the idea was to compare plex side watchlistedAt vs trakt side listed_at. for this to be useful, need the third timestamp, when was the last sync ran, such timestamp is not available.

Explain how comparing listedAt dates could help here.
What would you do if plex_listed_at > trakt_listed_at for example ?

@glensc
Copy link
Collaborator

glensc commented Jan 2, 2023

Explain how comparing listedAt dates could help here. What would you do if plex_listed_at > trakt_listed_at for example ?

You omitted the part where I said sync timestamp is needed.

in my mind, the algo would be:

  1. if item is missing in trakt watchlist, compare last_sync_date and plex_listed_at. if plex_listed_at > last_sync_date then add it to trakt otherwise drop from plex

the same for other direction.

@glensc
Copy link
Collaborator

glensc commented Jan 2, 2023

yes. this could work

    def plex_watchlist_updated_at(self):
        from datetime import datetime, timezone
        from plexapi import utils
        updated_at = datetime(1970, 1, 1)
        for pm in self.plex_wl.values():
            listed_at = utils.toDatetime(pm._data.get("watchlistedAt"))
            updated_at = max(updated_at, listed_at)
        return updated_at.astimezone(timezone.utc)

    def trakt_watchlist_updated_at(self, watchlist):
        from datetime import datetime, timezone
        updated_at = datetime(1970, 1, 1).astimezone(timezone.utc)
        for tm in watchlist.values():
            listed_at = datetime.strptime(tm.listed_at, "%Y-%m-%dT%H:%M:%S.%f%z")
            updated_at = max(updated_at, listed_at)

        return updated_at

    def watchlist_sync_item(self, m: Media, dry_run=False):
        if not self.sync_wl:
            return

        plex_watchlist_updated_at = self.plex_watchlist_updated_at()
        print(plex_watchlist_updated_at)
        trakt_movies_watchlist_updated_at = self.trakt_watchlist_updated_at(self.trakt_wl_movies)
        print(trakt_movies_watchlist_updated_at)
        if plex_watchlist_updated_at > trakt_movies_watchlist_updated_at:
            print(f"Plex watchlist is newer")
        else:
            print(f"Trakt watchlist is newer")

@glensc
Copy link
Collaborator

glensc commented Jan 3, 2023

extra thing to consider:

  • trakt removes automatically when item is played, so the "last timestamp from items" might be wrong.

https://trakt.docs.apiary.io/#reference/sync/get-watchlist/get-watchlist:

Auto Removal

When an item is watched, it will be automatically removed from the watchlist. For shows and seasons, watching 1 episode will remove the entire show or season.

could maybe compare "played_on_trakt" state and make an extra decision.

@simonc56
Copy link
Collaborator

simonc56 commented Jan 3, 2023

You omitted the part where I said sync timestamp is needed.

How do you get it ?

        if plex_watchlist_updated_at > trakt_movies_watchlist_updated_at:
            print(f"Plex watchlist is newer")
        else:
            print(f"Trakt watchlist is newer")

I think those print() assertions are wrong because it forgets to read the "remove" actions. It only reads "add" actions on watchlists.

@glensc
Copy link
Collaborator

glensc commented Jan 3, 2023

You omitted the part where I said sync timestamp is needed.

How do you get it ?

read last sentence:

        if plex_watchlist_updated_at > trakt_movies_watchlist_updated_at:
            print(f"Plex watchlist is newer")
        else:
            print(f"Trakt watchlist is newer")

I think those print() assertions are wrong because it forgets to read the "remove" actions. It only reads "add" actions on watchlists.

also must consider trakt removes automatically from Watchlist:

@simonc56
Copy link
Collaborator

simonc56 commented Jan 3, 2023

Removed automatically or manually from watchlist doesn't change the problem.

@glensc
Copy link
Collaborator

glensc commented Jan 3, 2023

Perhaps one thing to do is not to add to watchlist items that are played.

@glensc
Copy link
Collaborator

glensc commented Jan 20, 2023

found that trakt offers a response that includes when watchlist was last updated:

@aleksandarmomic
Copy link

This feature would be a great addition as I would like to use the watchlists in this way:

  • Add a movie to Plex Watchlist (I expect it to be added to Trakt)
  • I add a movie to Trakt Watchlist (I expect it to be added to Plex)
  • I watch a movie on Plex and it detects the movie is watched and auto removes the movie from the Plex Watchlist (I expect that movie to also be removed from Trakt Watchlist)
  • I watch a movie somewhere else and I manually mark the movie as watched in Trakt and I manually remove it from Trakt Watchlist (I expect that movie removed from Plex Watchlist also)

@KoenBoone

This comment was marked as off-topic.

@glensc

This comment was marked as off-topic.

@KoenBoone

This comment was marked as off-topic.

@glensc

This comment was marked as off-topic.

@KoenBoone

This comment was marked as off-topic.

@Jpinilla712
Copy link

This feature would be a great addition as I would like to use the watchlists in this way:

  • Add a movie to Plex Watchlist (I expect it to be added to Trakt)
  • I add a movie to Trakt Watchlist (I expect it to be added to Plex)
  • I watch a movie on Plex and it detects the movie is watched and auto removes the movie from the Plex Watchlist (I expect that movie to also be removed from Trakt Watchlist)
  • I watch a movie somewhere else and I manually mark the movie as watched in Trakt and I manually remove it from Trakt Watchlist (I expect that movie removed from Plex Watchlist also)

I agree with this!

@RileyXX
Copy link
Contributor

RileyXX commented Jun 4, 2023

This feature would be a great addition as I would like to use the watchlists in this way:

  • Add a movie to Plex Watchlist (I expect it to be added to Trakt)
  • I add a movie to Trakt Watchlist (I expect it to be added to Plex)
  • I watch a movie on Plex and it detects the movie is watched and auto removes the movie from the Plex Watchlist (I expect that movie to also be removed from Trakt Watchlist)
  • I watch a movie somewhere else and I manually mark the movie as watched in Trakt and I manually remove it from Trakt Watchlist (I expect that movie removed from Plex Watchlist also)

Alternatively you could use a script specifically to remove watched items from plex/trakt watchlists.

Edit: After thinking about it this would be a great separate feature to add as an optional config setting. I opened a separate issue for it here #1494

@Miladiir
Copy link

Miladiir commented Jun 27, 2024

Similar problem to synchronizing two filesystems, that do not know about each other. Take a look at how rclone solved with for the bisync feature: https://rclone.org/bisync/.
Initial run:
Get list from trakt and save the list with timestamp. Get list from plex and save the list with timestamp.
Decide, which list to keep. Remove missing from plex? Remove missing from trakt? Keep both, as in, add missing items to the other list?
Perform synchronisation action and save result (the two lists, with two timestamps)

Subsequent runs can then decide based on timestamp, which items where removed from each list or added to each list and replicate this action to the other list.

The only problem with this approach, is that the index needs to be protected. If the local copy of the list with the timestamps is tampered with, lost or in some other way corrupted, chaos can ensue. Worst case is, both lists get pruned of all items. Another failure case is that updates are lost, e.g. all removed items from last run are added back.

There is a lot of fun computer sciency topics to think about, e.g. https://en.wikipedia.org/wiki/Write%E2%80%93write_conflict. Don't think implementing this well would be trivial. Or maybe I am overthinking it.

@glensc
Copy link
Collaborator

glensc commented Jul 5, 2024

Without adding local database conclusion made already earlier. Some attempts already exists:

Perhaps first step would be add plugin to fill the database using new plugin infrastructure:

@RileyXX
Copy link
Contributor

RileyXX commented Jul 6, 2024

If the local copy of the list with the timestamps is tampered with, lost or in some other way corrupted, chaos can ensue. Worst case is, both lists get pruned of all items. Another failure case is that updates are lost, e.g. all removed items from last run are added back.

I was thinking about ways you could implement this too and this was my conclusion as well. There is a lot of risk associated with trying to determine which items to delete. Trakt API goes down pretty frequently which can cause issues. Local database can be corrupted, deleted, etc. Sometimes Trakt API can also omit results when things are not working correctly (multiple pagination pages missing). I would assume the same risks for Plex API as well. There would need to be a lot of fail proof checks in order to accomplish this safely.

Some things I was considering in one of my projects (IMDB-Trakt-Syncer) that would apply here as well, schedule items to be removed. Periodically check all lists to verify the item should be deleted (to ensure no API errors), after a certain amount of checks or time passed then approve the scheduled items for deletion. Additionally, you could add a check so that if more than 50% of a list is to be deleted then stop the deletion (in order to prevent accidental deletion due to unforeseen issues). As you can see it would get very complicated and there is still risk of chaos ensuing as you mentioned.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

8 participants