duplicate dataset comes back after dedupe

[FuhuXia]

After dedupe, a duplicate of https://catalog.data.gov/dataset/national-settlement-service-data keeps coming back after next harvest job.

How to reproduce

1. harvest https://catalog.data.gov/harvest/federal-reserve
2. Run dedupe script on org https://catalog.data.gov/organization/board-of-governors-of-the-federal-reserve-system
3. reharvest

Expected behavior

The dataset number should not change.

Actual behavior

One duplicate created for dataset https://catalog.data.gov/dataset/national-settlement-service-data

Sketch

Three approaches to fix the issue

1. Clear the harvest source then reharvest. This will lose tracking stats for all datasets in this source.
2. Exam the state of the affected dataset in DB and SOLR and figure out why duplicate occurs. Could be a new bug in ckanext-datajson.
3. Could be a bug in the dedupe process that an edge case is not handled well.

[FuhuXia]

Did a harvest source clear and reharvest. The issue is back but the duplicate dataset changed to
international-summary-statistics
international-summary-statistics-2ddc8

Trying to replicate it in other environments.

[FuhuXia]

Could not replicate on develop or staging.