Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

imports/uploads slow due to reading files into memory due to schema validation and exif (for images) #167

Open
djay opened this issue Nov 26, 2024 · 0 comments · May be fixed by #166
Open

Comments

@djay
Copy link
Member

djay commented Nov 26, 2024

Currently the schema validation checks .data exists but not it's contents for INamedFile and INamedImage.
But since this is implemented as a property decorator this validation will load the whole file into memory. If this is a large file or you are doing a big import, this can slow down the process considerably as often the data is transferred using fs cp.

I have a PR that avoids this check. #166

This might also be able to be fixed in zope.schema itself - zopefoundation/zope.schema#127

In addition for images the file is being read for unnessarily for exif and content-type extraction. The attached PR also fixes this.

@djay djay linked a pull request Nov 26, 2024 that will close this issue
@djay djay changed the title imports/uploads slow due to reading files into memory due to schema validation imports/uploads slow due to reading files into memory due to schema validation and exif (for images) Dec 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant