Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

composer install skipping patch process #583

Open
5 tasks done
arderyp opened this issue Jun 19, 2024 · 9 comments
Open
5 tasks done

composer install skipping patch process #583

arderyp opened this issue Jun 19, 2024 · 9 comments
Labels
bug Inconsistencies or issues which will cause a problem for users or implementors.

Comments

@arderyp
Copy link

arderyp commented Jun 19, 2024

Verification

  • I have updated Composer to the most recent stable release (composer self-update)
  • I have updated Composer Patches to the most recent stable release (composer update cweagans/composer-patches)
  • I am using one of the supported PHP versions (8.0+)
  • I have searched existing issues and discussions for my problem.
  • My problem is not addressed in the troubleshooting guide.

What were you trying to do (and why)?

I am deploying Drupal to Pantheon, a popular hosting service.

My Pantheon and local are on the latest version of all dependencies and are in sync. I'm running 2.0.0-beta2 of composer-patches.

I then introduce a new patch on my local. While composer install does nothing, skipping the patch process with Nothing to install, update or remove, I can hack around this by deleting the target web/modules/composer/page_manager composer managed directory and then running composer update. This re-installs page_manager and applies the patch without issue. Likewise, composer patches-repatch works.

But Pantheon doesn't do this. Their deployment process runs "composer --no-interaction --no-progress --prefer-dist --ansi install". As noted above on my local, the output of this command also says Nothing to install, update or remove. The patching mechanism isn't triggered at all.

I'm happy to provide more info if you tell me what will be helpful.

What happened? What did you expect to happen?

The patch process was not executed.

I expected the patch to be applied, as it is successfully applied during composer patches-repatch

Contents of composer.json

{
    "name": "XXX",
    "type": "project",
    "license": "MIT",
    "repositories": [
        {
            "type": "composer",
            "url": "https://packages.drupal.org/8"
        },
        {
            "type": "path",
            "url": "upstream-configuration"
        },
        {
            "type": "package",
            "package": {
                "name": "jackmoore/colorbox",
                "version": "1.6.4",
                "type": "drupal-library",
                "dist": {
                    "url": "https://github.com/jackmoore/colorbox/archive/1.6.4.zip",
                    "type": "zip"
                }
            }
        },
        {
            "type": "package",
            "package": {
                "name": "ckeditor-plugin/autoembed",
                "version": "4.11.0",
                "type": "drupal-library",
                "dist": {
                    "url": "https://download.ckeditor.com/autoembed/releases/autoembed_4.11.0.zip",
                    "type": "zip"
                }
            }
        },
        {
            "type": "package",
            "package": {
                "name": "ckeditor-plugin/autolink",
                "version": "4.11.0",
                "type": "drupal-library",
                "dist": {
                    "url": "https://download.ckeditor.com/autolink/releases/autolink_4.11.0.zip",
                    "type": "zip"
                }
            }
        },
        {
            "type": "package",
            "package": {
                "name": "ckeditor-plugin/embed",
                "version": "4.11.0",
                "type": "drupal-library",
                "dist": {
                    "url": "https://download.ckeditor.com/embed/releases/embed_4.11.0.zip",
                    "type": "zip"
                }
            }
        },
        {
            "type": "package",
            "package": {
                "name": "ckeditor-plugin/embedbase",
                "version": "4.11.0",
                "type": "drupal-library",
                "dist": {
                    "url": "https://download.ckeditor.com/embedbase/releases/embedbase_4.11.0.zip",
                    "type": "zip"
                }
            }
        },
        {
            "type": "package",
            "package": {
                "name": "ckeditor-plugin/embedsemantic",
                "version": "4.11.0",
                "type": "drupal-library",
                "dist": {
                    "url": "https://download.ckeditor.com/embedsemantic/releases/embedsemantic_4.11.0.zip",
                    "type": "zip"
                }
            }
        },
        {
            "type": "package",
            "package": {
                "name": "ckeditor-plugin/notification",
                "version": "4.11.0",
                "type": "drupal-library",
                "dist": {
                    "url": "https://download.ckeditor.com/notification/releases/notification_4.11.0.zip",
                    "type": "zip"
                }
            }
        },
        {
            "type": "package",
            "package": {
                "name": "ckeditor-plugin/notificationaggregator",
                "version": "4.11.0",
                "type": "drupal-library",
                "dist": {
                    "url": "https://download.ckeditor.com/notificationaggregator/releases/notificationaggregator_4.11.0.zip",
                    "type": "zip"
                }
            }
        },
        {
            "type": "package",
            "package": {
                "name": "ckeditor-plugin/textmatch",
                "version": "4.11.0",
                "type": "drupal-library",
                "dist": {
                    "url": "https://download.ckeditor.com/textmatch/releases/textmatch_4.11.0.zip",
                    "type": "zip"
                }
            }
        },
        {
            "type": "package",
            "package": {
                "name": "jquery/cycle",
                "version": "3.0.3",
                "type": "drupal-library",
                "extra": {
                    "installer-name": "jquery.cycle"
                },
                "dist": {
                    "url": "https://github.com/malsup/cycle/archive/master.zip",
                    "type": "zip"
                }
            }
        }
    ],
    "require": {
        "ckeditor-plugin/autoembed": "^4.11",
        "ckeditor-plugin/autolink": "^4.11",
        "ckeditor-plugin/embed": "^4.11",
        "ckeditor-plugin/embedbase": "^4.11",
        "ckeditor-plugin/embedsemantic": "^4.11",
        "ckeditor-plugin/notification": "^4.11",
        "ckeditor-plugin/notificationaggregator": "^4.11",
        "ckeditor-plugin/textmatch": "^4.11",
        "composer/installers": "^2.2",
        "cweagans/composer-patches": "^2.0@BETA",
        "drupal/acl": "^2.0@beta",
        "drupal/admin_toolbar": "^3.1.0",
        "drupal/aggregator": "^2.1",
        "drupal/auto_entitylabel": "^3.0-beta4",
        "drupal/bootstrap": "^3.23",
        "drupal/cas": "^2.0",
        "drupal/colorbox": "^2.0",
        "drupal/content_access": "^2.0",
        "drupal/core-composer-scaffold": "^10",
        "drupal/core-project-message": "^10.0",
        "drupal/core-recommended": "^10",
        "drupal/crop": "^2.1",
        "drupal/ctools": "^4.1",
        "drupal/editor_advanced_link": "^2.0.0",
        "drupal/exclude_node_title": "^1.2",
        "drupal/externalauth": "^2.0",
        "drupal/field_permissions": "^1.1",
        "drupal/google_tag": "^2.0",
        "drupal/image_widget_crop": "^2.3",
        "drupal/imce": "^3.0",
        "drupal/jquery_ui": "^1.4",
        "drupal/jquery_ui_draggable": "^2.1",
        "drupal/jquery_ui_droppable": "^2.1",
        "drupal/menu_block": "^1.7",
        "drupal/override_node_options": "^2.6",
        "drupal/page_manager": "^4.0-beta6",
        "drupal/panels": "^4.6",
        "drupal/redirect": "^1.6",
        "drupal/smtp": "^1.2",
        "drupal/token": "^1.9",
        "drupal/upgrade_status": "^4.0@alpha",
        "drupal/view_unpublished": "^1.1",
        "drupal/views_field_view": "^1.0-beta3",
        "drush/drush": "^11 || ^12",
        "jackmoore/colorbox": "^1.6",
        "jquery/cycle": "^3.0",
        "mglaman/composer-drupal-lenient": "^1.0",
        "mglaman/phpstan-drupal": "^1.1",
        "pantheon-systems/drupal-integrations": "^10",
        "pantheon-upstreams/upstream-configuration": "dev-main",
        "phpstan/phpstan": "^1.9",
        "phpstan/phpstan-deprecation-rules": "^1.1"
    },
    "require-dev": {
        "drupal/core-dev": "^10.0"
    },
    "conflict": {
        "drupal/drupal": "*"
    },
    "minimum-stability": "stable",
    "prefer-stable": true,
    "extra": {
        "drupal-scaffold": {
            "locations": {
                "web-root": "web/"
            },
            "allowed-packages": ["pantheon-systems/drupal-integrations"],
            "file-mapping": {
                "[project-root]/.editorconfig": false,
                "[project-root]/pantheon.upstream.yml": false,
                "[project-root]/.gitattributes": false
            }
        },
        "installer-paths": {
            "web/core": ["type:drupal-core"],
            "web/libraries/ckeditor/plugins/{$name}": ["vendor:ckeditor-plugin"],
            "web/libraries/{$name}": ["type:drupal-library"],
            "web/modules/composer/{$name}": ["type:drupal-module"],
            "web/profiles/composer/{$name}": ["type:drupal-profile"],
            "web/themes/composer/{$name}": ["type:drupal-theme"],
            "drush/Commands/contrib/{$name}": ["type:drupal-drush"],
            "web/private/scripts/quicksilver/{$name}/": ["type:quicksilver-script"]
        },
        "drupal-lenient": {},
        "enable-patching": true,
        "composer-exit-on-patch-failure": true,
        "patchLevel": {
            "drupal/core": "-p2"
        }
    },
    "autoload": {
        "classmap": ["upstream-configuration/scripts/ComposerScripts.php"]
    },
    "scripts": {
        "pre-update-cmd": ["DrupalComposerManaged\\ComposerScripts::preUpdate"],
        "post-update-cmd": ["DrupalComposerManaged\\ComposerScripts::postUpdate"]
    },
    "config": {
        "preferred-install": "dist",
        "sort-packages": true,
        "platform": {
            "php": "8.1.0"
        },
        "allow-plugins": {
            "composer/installers": true,
            "drupal/core-composer-scaffold": true,
            "drupal/core-project-message": true,
            "mglaman/composer-drupal-lenient": true,
            "zaporylie/composer-drupal-optimizations": true,
            "cweagans/composer-patches": true,
            "dealerdirect/phpcodesniffer-composer-installer": true,
            "phpstan/extension-installer": true,
            "php-http/discovery": true
        }
    }
}


### Contents of `patches.lock.json`

```shell
{
    "_hash": "7a8e0acb4f81ccf52f494fcf7da0b176e2aa60d0356efc1f6062a64a7a30ed6d",
    "patches": {
        "drupal/page_manager": [
            {
                "package": "drupal/page_manager",
                "description": "make __sleep() function compatible with new type hinted return in upstream ctools: https://www.drupal.org/project/page_manager/issues/3455521",
                "url": "patches/page_manager.3455521-0.patch",
                "sha256": "51f641d189d3da8ea19db5cdd82c960be4b0d6804f87248bb3ae0c896cbebbf8",
                "depth": 1,
                "extra": {
                    "provenance": "patches-file:patches.json"
                }
            }
        ]
    }
}



### Full output of `composer patches-doctor`

```shell
$ composer patches-doctor

System information
================================================================================
Composer version:                                                          2.7.7
PHP version:                                                              8.2.20

Available patchers
================================================================================
cweagans\Composer\Patcher\GitPatcher usable:                                 yes
cweagans\Composer\Patcher\GitInitPatcher usable:                             yes
cweagans\Composer\Patcher\FreeformPatcher usable:                            yes
Has usable patchers:                                                         yes

Common configuration issues
================================================================================
has plain http patch URLs:                                                    no


### Full output of relevant Composer command with the `-vvv` flag added

```shell
$ composer install -vvv
Running 2.7.7 (2024-06-10 22:11:12) with PHP 8.2.20 on Linux / 5.4.0-186-generic
Reading ./composer.json (/home/arderyp/drupal/composer.json)
Loading config file /home/arderyp/.config/composer/config.json
Loading config file /home/arderyp/.config/composer/auth.json
Loading config file ./composer.json (/home/arderyp/drupal/composer.json)
Checked CA file /etc/pki/tls/certs/ca-bundle.crt does not exist or it is not a file.
Checked directory /etc/pki/tls/certs/ca-bundle.crt does not exist or it is not a directory.
Checked CA file /etc/ssl/certs/ca-certificates.crt: valid
Executing command (/home/arderyp/drupal): 'git' 'branch' '-a' '--no-color' '--no-abbrev' '-v'
Reading /home/arderyp/.config/composer/composer.json
Loading config file /home/arderyp/.config/composer/config.json
Loading config file /home/arderyp/.config/composer/auth.json
Loading config file /home/arderyp/.config/composer/composer.json (/home/arderyp/.config/composer/composer.json)
Loading config file /home/arderyp/.config/composer/auth.json
Reading /home/arderyp/.config/composer/auth.json
Reading /home/arderyp/drupal/vendor/composer/installed.json
Reading /home/arderyp/.config/composer/vendor/composer/installed.json
Loading plugin Composer\Installers\Plugin (from composer/installers)
Loading plugin cweagans\Composer\Plugin\Patches (from cweagans/composer-patches)
Loading plugin Http\Discovery\Composer\Plugin (from php-http/discovery)
Loading plugin PHPCSStandards\Composer\Plugin\Installers\PHPCodeSniffer\Plugin (from dealerdirect/phpcodesniffer-composer-installer)
Loading plugin PHPStan\ExtensionInstaller\Plugin (from phpstan/extension-installer)
Loading plugin Drupal\Composer\Plugin\Scaffold\Plugin (from drupal/core-composer-scaffold)
Loading plugin Drupal\Composer\Plugin\ProjectMessage\MessagePlugin (from drupal/core-project-message)
Loading plugin ComposerDrupalLenient\Plugin (from mglaman/composer-drupal-lenient)
Reading ./composer.lock (/home/arderyp/drupal/composer.lock)
> command: Drupal\Composer\Plugin\Scaffold\Plugin->onCommand
Installing dependencies from lock file (including require-dev)
Verifying lock file contents can be installed on current platform.
Reading ./composer.lock (/home/arderyp/drupal/composer.lock)
> pre-pool-create: ComposerDrupalLenient\Plugin->modifyPackages
Built pool.
Generating rules
Resolving dependencies through SAT
Looking at all rules.

Dependency resolution completed in 0.001 seconds
Nothing to install, update or remove
Generating autoload files
> pre-autoload-dump: Http\Discovery\Composer\Plugin->preAutoloadDump
98 packages you are using are looking for funding.
Use the `composer fund` command to find out more!
> post-install-cmd: PHPCSStandards\Composer\Plugin\Installers\PHPCodeSniffer\Plugin->onDependenciesChangedEvent
Running PHPCodeSniffer Composer Installer
Executing command (/home/arderyp/drupal/vendor/squizlabs/php_codesniffer): '/usr/bin/php8.2' -d allow_url_fopen='1' -d disable_functions='' -d memory_limit='-1' ./bin/phpcs --config-show
No PHPCS standards to install or update
> post-install-cmd: PHPStan\ExtensionInstaller\Plugin->process
phpstan/extension-installer: Extensions installed
> post-install-cmd: Drupal\Composer\Plugin\Scaffold\Plugin->postCmd
Scaffolding files for pantheon-systems/drupal-integrations:
  - Skip [web-root]/sites/default/settings.php because it already exists and overwrite is false.
Executing command (/home/arderyp/drupal/web): git ls-files --error-unmatch '/home/arderyp/drupal/web//autoload.php'
Executing command (/home/arderyp/drupal): git rev-parse --show-toplevel
Executing command (/home/arderyp/drupal): git check-ignore 'vendor'
Executing command (/home/arderyp/drupal): git check-ignore '/home/arderyp/drupal/web/sites/default/settings.php'
Executing command (/home/arderyp/drupal): git ls-files --error-unmatch '/home/arderyp/drupal/web/sites/default/settings.php'
> post-install-cmd: Drupal\Composer\Plugin\ProjectMessage\MessagePlugin->displayPostCreateMessage

@arderyp arderyp added the bug Inconsistencies or issues which will cause a problem for users or implementors. label Jun 19, 2024
@cweagans
Copy link
Owner

I then introduce a new patch on my local.

How are you doing this? Just adding it manually to composer.json and then ___ ? You should not have to manually delete anything or run composer update to get composer-patches to work. I recommend following this process: https://docs.cweagans.net/composer-patches/usage/recommended-workflows/#add-a-patch-to-your-project

I know for sure that Pantheon will install patches properly. If you want to see how it'll behave, commit your changes to composer.json, composer.lock, and patches.lock.json and push it to the repo. Then, clone a completely fresh copy of the Git repo and run a composer install (you can add the same arguments that they add if you want, but it doesn't matter).

@arderyp
Copy link
Author

arderyp commented Jun 20, 2024

I created patches.json and put the patch in there. I already had composer-patches installed and enabled from previous usage. I did bump the version to the latest 2.x beta from 1.x.

When I ran composer update it created patches.lock.json after implementing rhe patch.

No changes were ever established on composer.lock. Maybe that's the issue? Why would changes be made to that file? I'm not actually updating the patched module (page_manager), I'm patching the existing installed version that composer.lock already knows about. Hence a basic install sees the lock version is already installed and decides "do nothing", bypassing the patch process.

I will try your linked workflow and report back. I thought I did something similar but can't be totally sure so I will try again.

@arderyp
Copy link
Author

arderyp commented Jun 20, 2024

@cweagans Here is what I did:

  1. remove patches.json
  2. remove web/modules/composer
  3. remove patches.lock.json
  4. move my patch from patches.json (deleted) to composer.json in order to follow the documentation exactly (all steps below)
  5. Define a patch in your composer.json (see step 4 above)
  6. Run composer patches-relock to regenerate patches.lock.json with your new patch.
  7. Run composer patches-repatch to delete patched dependencies and reinstall them with any defined patches
  8. run composer update --lock to update the content hash in composer.lock
  9. Commit any related changes to your external patches file (if configured), composer.json, composer.lock, and patches.lock.json.

It did not work.

$ terminus composer:logs my-site.dev
Composer version 2.7.6 2024-05-04 23:03:15
PHP version 8.1.29 (/opt/pantheon/php8.1/php)
Run the "diagnose" command to get more detailed diagnostics output.
Cache directory does not exist (cache-vcs-dir):
Cache directory does not exist (cache-repo-dir):
Cache directory does not exist (cache-files-dir):
Clearing cache (cache-dir): /home/pantheon-app/.composer/cache
All caches cleared.
Running composer install...
composer --no-interaction --no-progress --prefer-dist --ansi install
Installing dependencies from lock file (including require-dev)
Verifying lock file contents can be installed on current platform.
Warning: The lock file is not up to date with the latest changes in composer.json. You may be getting outdated dependencies. It is recommended that you run `composer update` or `composer update `.
Nothing to install, update or remove
Generating autoload files
98 packages you are using are looking for funding.
Use the `composer fund` command to find out more!
phpstan/extension-installer: Extensions installed
Scaffolding files for pantheon-systems/drupal-integrations:
  - Skip [web-root]/sites/default/settings.php because it already exists and overwrite is false.

In fact, this procedure didnt really change the state of my repo in meaningful ways. Step 8 is confusing. No changes were made to composer.lock and I have no idea why we should expect there to be. As I mentioned, the installed version is the latest. I want to patch it. Its not being updated, so composer.lock should not (IMO) bve updated. Step 8 mentions the hash changing, but the hash stored in composer.lock is associated with the unpatched version pulled by composer, right? patches.lock.json has a hash associated with the patch, but I don't see how/why composer.lock should/would be touched.

As the output above shows, an unchanged composer.lock is basically telling the system "there is nothing to do here", and since there is nothing to do, there is nothing to patch. Again, maybe I'm misunderstanding something. If I clone the repo and did a composer install, I am sure it would work, because there would be no vendor dependencies in place, so they'd be freshly installed, which triggers the patch process. But on Pantheon dev, since the dependencies are already installed, there is no installation, as noted in the output above, just as there is no patching on my local, where the vendor dependencies are already installed , unless I delete the vendor install directory first.

Does any of this make sense? The issue is not with Pantheon. There may not even be an issue with composer or composer-patches... it may be a workflow thing that simply doesn't trigger the patching process when the installed vendor set matches composer.lock, and this composer doesn't think there is anything to do, and this the patching process doesnt run, and thus any new patches I try to create for the current composer.lock simply will not be applied unless I run the extra step of composer patches-repatch, which unfortunately isn't an options on a restricted hosting service like Pantheon.

I may try dropping the dependency and patch, commiting/deploying that change, then adding back the dependency and patch and the generated files from the documentation steps, and see if that causes Pantheon to trigger a fresh install of the dependency in question, which will then trigger the patching.

If this understanding of the situation is correct, it should probably be detailed in the documentation. I will report back when I test tomorrow. But I am really eager to test any further ideas you have or provide any further info that could help debug the situation.

@arderyp
Copy link
Author

arderyp commented Jun 20, 2024

My analysis seems to be correct. Once I removed the package then re-added it, the patch was triggered. However, I had to break this into two separate commits to make it work on Pantheon, the reason being what the same was what I've explained above---if I remove it and re-add it, then the state it the same, there is nothing to commit, and composer determines there is nothing to do, including patching, because the installed version is the locked version.

This should probably be in the documentation somewhere, but it's a rather complex issue to put into words. Maybe something like:

If you are patching a dependency whose version is in sync with composer.lock, you patch will not be applied with a composer install command because composer will see that the installed version matches the locked version, and thus that there is nothing to do. If your environment allows, you can use composer update or composer patches-repatch to trigger the patch. If you are deploying to a limited environment that only used composer install, you will need to drop the dependency or temporarily require a different version, commit/deploy that change, then re-require the desired version, which will then be re-installed and patched by composer install.

@arderyp
Copy link
Author

arderyp commented Jun 21, 2024

@cweagans do you want me to close this? Does any of this make sense? If so, do you think it should be documented a la my comment above?

@cweagans
Copy link
Owner

Need some time to grok it all. I think you might be doing something wrong but I want to understand more fully before I make that assertion :)

@arderyp
Copy link
Author

arderyp commented Jun 26, 2024

K. I think the question is: does patching get triggered or skipped if "composer install" determines there is nothing to do?

If triggered: I must be doing something wrong or have some environmental issue.

If skipped: this should be documented

@cweagans
Copy link
Owner

If composer install determines that there are no packages to install, it's likely that nothing would happen (since patches are applied after installation of each package). In theory, if the dependency has been installed already, it would have already been patched. If you updated the dependency (i.e. composer.lock has updated data for that dependency), that dependency would be removed and re-installed, thereby repatching it.

One thing that you could do in your project as a test is to set up a post-install script that runs composer patches-repatch:

{
  "extra": {
    "scripts": {
      "post-install-cmd": [
        "@composer patches-repatch"
      ]
    }
  }
}

That should trigger on Pantheon too. It might add a little bit of extra time to your deploys, but it shouldn't be too much. If that makes things behave a little closer to what you're expecting, it might be worth investigating adding something that subscribes to the "post-install-cmd" event in the plugin itself that somehow checks if patches are applied. Or maybe there's a setting that makes composer patches more aggressive about re-applying patches?

If this is specific to how Pantheon handles their environments, you might be able to set up a Quicksilver script (https://docs.pantheon.io/guides/quicksilver/install-script) that just runs composer patches-repatch during your deploys.

In the past, there were a lot of complaints about composer patches just blowing away an entire directory (which could have contained work-in-progress if somebody was e.g. contributing to a Drupal module). This was done in the interest of ensuring that patches were always applied no matter what. I'm not really sure what to do here - it seems like re-patching automatically doesn't work for some people and not re-patching automatically doesn't work for others. Maybe this is just a config option that needs to be exposed - if true, then composer patches-repatch after every install command?

@arderyp
Copy link
Author

arderyp commented Jun 26, 2024

It sounds like my analysis is correct then, and maybe it should be documented. The situation being: if you want to patch your current locked dependency, without updating, it will not work unless you remove and then reinstall the dependency, or use patches-repatch

I tried a similar post script, but it caused an infinite loop.

I'm beyond the issue on my end. I removed the dependency, deployed that to pantheon, then re-added it, and deployed that. The patch was trigger and all is well

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Inconsistencies or issues which will cause a problem for users or implementors.
Projects
None yet
Development

No branches or pull requests

2 participants