Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix cache rebuild #169

Open
wants to merge 4 commits into
base: main
Choose a base branch
from
Open

Conversation

badri
Copy link

@badri badri commented Aug 1, 2023

Summary

Resolve #107

Use Cases

Certain frameworks, most notably Drupal, place modules and themes downloaded from composer install in directories other than the vendor directory. During a cache restore phase for the composer install buildpack, this will get erased because of which the final image won't serve the web application.

This is fixed by running composer install even if the cached layer is being reused. This places the already downloaded and cached dependencies in the appropriate non-vendor directories. This can be toggled by setting the BP_RUN_COMPOSER_INSTALL env var to 1.

Checklist

  • I have viewed, signed, and submitted the Contributor License Agreement.
  • I have linked issue(s) that this PR should close using keywords or the Github UI (See docs)
  • I have added an integration test, if necessary.
  • I have reviewed the styleguide for guidance on my code quality.
  • I'm happy with the commit history on this PR (I have rebased/squashed as needed).

badri added 2 commits August 1, 2023 07:56
Specifically designed to re-run composer install in a cached build.
@badri badri requested a review from a team as a code owner August 1, 2023 02:43
@linux-foundation-easycla
Copy link

linux-foundation-easycla bot commented Aug 1, 2023

CLA Signed

The committers listed above are authorized under a signed CLA.

@badri badri closed this Aug 1, 2023
@badri
Copy link
Author

badri commented Aug 1, 2023

Didn't add labels.

@badri badri reopened this Aug 1, 2023
@arjun024
Copy link
Member

arjun024 commented Aug 1, 2023

Thanks for the PR @badri
I'm not familiar with Drupal at all, but instead of ignoring caching to run "composer install" again, can we cache the web directory as well, and (I'm just think out aloud here) do you think we can make use of composer.json/installer-paths to figure out if web has to be restored as well?

@sophiewigmore @effulgentsia - thoughts?

@badri
Copy link
Author

badri commented Aug 2, 2023

@arjun024 the web directory will contain custom business logic in addition to housing the downloaded composer modules. So, caching it and restoring will override the custom logic. Instead, what we can do is, re-run composer install in that phase so that composer will place the already downloaded modules from that layer into the appropriate locations under web.
This is only needed for Drupal and other PHP frameworks which follow this practice. Hence controlling it through a build time env var.
This won't re-download the files or nullify the benefits of caching the composer layer in any way.

Of course, I'm eager to hear opinions of others :)

I've been using paketo buildpacks since Drupal 8 and I had to resort to creating my own buildpacks to circumvent this problem where I was essentially doing the same thing.

@sophiewigmore
Copy link
Member

sophiewigmore commented Aug 21, 2023

the web directory will contain custom business logic in addition to housing the downloaded composer modules. So, caching it and restoring will override the custom logic

@badri can you explain a bit more about what you mean by this? What type of errors do you encounter when you do this?
I'm also interested if you ever looked into @effulgentsia's idea of checking if there's a way to keep track of directories created by composer install?

Overall, if re-running composer install will still be able to leverage the cached dependencies, then I'm open to this approach for sure. I'd ideally like to find a way for us to cache the other directories that composer installs if we can safely find a way to do that

P.S. thanks for the contribution!

@@ -268,6 +268,30 @@ func runComposerInstall(
logger.Debug.Subprocess(fmt.Sprintf("- %s", f.Name()))
}
}
if os.Getenv("BP_RUN_COMPOSER_INSTALL") == "1" {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we do end up going with the environment variable, we usually use booleans for this type of environment variable, so I'd want to see this changed to true/false. I'd also want to see this env. var documented in the README with the rest of the configurations.

Let's not make these changes until we (you and @paketo-buildpacks/php-maintainers) are in agreement about how to do this

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Composer-managed directories other than "vendor" get lost from workspace during a rebuild from cache
3 participants