Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf(ftp): reduce number of ftp commands run in install preparation #63

Merged
merged 8 commits into from
May 5, 2024

Conversation

mircearoata
Copy link
Member

When connecting to an FTP server not on the same network, the latency for every FTP command being run greatly increases, slowing down the install process, which SMM frequently triggers.

Testing this using SirDigby's server, the latency was around 0.3s, and I observed 20 seconds being spent before even starting to extract mods. This PR reduces that time to ~5 seconds.

Improvements made:

  • ftp.Exists - consider the error codes returned by MLST or LIST to determine whether a path exists
    • previously it would check for every path segment whether it exists, resulting in longer wait times the deeper the server is located
    • improved install by ~10s
  • ftp.MkDir - find last existing segment by checking up rather than down
    • it is very likely that only one or two directories have to be created, so only a couple of path existence checks will be made
    • improved install by ~1s
    • also greatly improves mod extraction (not benchmarked, but previously the throughput would drop to 100B/s average at time, since a lot of directories have to be created)
  • parallelize mod removal
    • mod extraction was done in parallel, so there's no reason why deletion shouldn't too
    • one mod removal takes around 6s (not included in the 20s of the benchmark)
  • i.Validate - parallelize
    • it is not performed concurrently with any other FTP operation, so it has the whole connection pool available
    • improved install by ~2.5s
  • fewer Validate and GetPlatform calls - obtain platform once and pass it to other function calls
    • Validate was being called 6 times, and GetPlatform 4 times
    • improved install by ~1.5s

Additionally, . and .. are ignored in ReadDir results, as non-standard FTP servers that use LIST -a would return those too, leaving the possibility of the game dir being deleted if it contained a .smm file

@mircearoata mircearoata requested a review from Vilsol April 28, 2024 12:29
cli/disk/ftp.go Outdated Show resolved Hide resolved
@mircearoata mircearoata merged commit d051b58 into master May 5, 2024
10 checks passed
@budak7273 budak7273 deleted the ftp-performance branch October 10, 2024 20:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants