fix: Remove unnecessary calls to the DB during DataValueSet import [D…

[david-mackessy]

…HIS2-16159]

Summary

During a DataValueSet import, calls are made to the DB to get OrganisationUnits when the AttributeOptionCombo has links to OrganisationUnits. When there are large amounts of OrganisationUnits this can lead to the import taking a long time.
v38 did not experience these issues.

Cause

During a large refactor PR the implementation that was working well was changed.

In v38 code we check OrganisationUnits that are already in-memory (ImportContext).
Post v38 we check OrganisationUnits using a service call which in turn calls the DB.

Fix

Revert back to original implementation, checking the OrganisationUnits in memory.
Needs backporting to:

• 2.39
• 2.40

Testing

The data below is based on following the steps provided in the Jira issue.
Running locally on a Mac.

Version	DB	Org Check	Import time
2.38	SL	In memory	~5 seconds
2.39	SL	In memory	~5 seconds
2.39	SL	DB	~5 minutes
2.40	SL	In memory	~5 seconds
2.40	SL	DB	~5 minutes
dev (master)	SL	In memory	~5 seconds
dev (master)	SL	DB	~5 minutes

[codecov]

Codecov Report

Merging #15688 (9e1ded8) into master (520136b) will increase coverage by 16.82%.
The diff coverage is 0.00%.

Additional details and impacted files

@@              Coverage Diff              @@
##             master   #15688       +/-   ##
=============================================
+ Coverage     49.39%   66.21%   +16.82%     
- Complexity    23283    31280     +7997     
=============================================
  Files          3483     3483               
  Lines        129911   129910        -1     
  Branches      15182    15181        -1     
=============================================
+ Hits          64166    86019    +21853     
+ Misses        59684    36809    -22875     
- Partials       6061     7082     +1021     

Flag	Coverage Δ
integration	49.93% <0.00%> (?)
integration-h2	32.28% <0.00%> (-0.01%)	⬇️
unit	30.28% <0.00%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
...dxf2/datavalueset/DataValueSetImportValidator.java	92.74% <0.00%> (+4.11%)	⬆️

... and 1226 files with indirect coverage changes

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 520136b...9e1ded8. Read the comment docs.

[sonarqubecloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

0.0% Coverage
0.0% Duplication