Running PITest and overwriting configuration

Gradle

Most common is the gradle-pitest-plugin. Running the gradle task pitest is already implemented in the GradleTaskExecutor.kt. Pitest is configured within the pitest plugin configuration block.

The drawback is that we have no direct way of overwriting the configurations inside that block except for additionalFeatures and targetTests for which the gradle-pitest-plugin has implemented the @Option() property, making them overwritable. To overwrite all the other settings, we need to provide a separate gradle plugin. The gradle-pitest-plugin itself mentions the gradle-override-plugin to override settings via the command line, but it does not support complex datatypes like lists/sets/directories/... Thus as a proof-of-concept plugin was added here to demonstrate how to override the gradle-pitest-plugin's configuration.

As of now it is able to overwrite

• reportDir as Directory
• targetClasses as a Set
• targetTests as a Set
• timestampedReports as a Boolean
• verbose as bool
• outputFormats as a Set
• threads as an Integer

Another benefit of providing a dedicated gradle-plugin is that it can handle extracting the current pitest configuration from gradle by providing a task.

Drawbacks

• Need to maintain an additional plugin
• Users of PITmutationmate plugin need to include another plugin in their build.gradle[.kts]

Advantages

• No need to parse pitest configuration
• No need to deal with classpaths etc (as opposed to running directly from the command line)
• Dedicated gradle-plugin can handle extracting the current pitest configuration from gradle

Command line

Running PITest directly from the cli can be done like this:

java -cp <your classpath including pit jar and dependencies> \
    org.pitest.mutationtest.commandline.MutationCoverageReport \
    --reportDir <outputdir> \
    --targetClasses com.your.package.tobemutated* \
    --targetTests com.your.package.*
    --sourceDirs <pathtosource>

As noted by the PITest documentation though:

The command line jar, core pitest jar and either JUnit or TestNG will need to be on the classpath.

and

It recommended that you use one of the provided build system plugins instead of the command-line tool. The build plugins are easier to use and you are much less likely to encounter configuration problems.

Please only use the command-line tool if you have a really good reason to do so.

The issue with this approach is getting PITest's jars and its dependencies for the classpath. Typically when using the gradle/maven plugins they take care of all the dependencies and where all those jars are located. The issue for us is that there is no easy programmatic way to query gradle for a task's or a plugin's dependencies AND get their jar's locations (i haven't looked at maven for this matter).

For the projects classes and even tests, intellij provides an api to assemble classpaths but gradle just does not provide the same capabilities.

To get around this we need to either

• search the filesystem for the correct pitest dependency files (and we dont really know which versions are needed),
• have the user provide the paths to pitest and the dependencies (for example by manually adding a custom task we provide to their build.gradle),
• or provide a custom gradle plugin which can register those tasks we could use to get those dependencies.

TLDR on classpath: We dont know where the pitest jar is and there is straight forward way to get this information.

Drawbacks

• How to get a proper classpath for the testes classes and used tests?
• How to get the complete pitest configuration?
  • -> We would have to read and convert existing settings from - say gradle-pitest-plugin - to a valid commandline call.

Advantages

• No need for extra plugin at least one option does require an extra plugin
• Might be easier to adapt to other build tools
• Easier way to give the user the ability to change detailed pitest run configuration options

Recommendation

Given the classpath issues i would recommend using a gradle-plugin for overwriting the command line arguments to gradle-pitest-plugin.

Maven

Maven might require changes to the POM to allow overwriting the configuration as it is mentioned here that it does not support overwriting a set configuration in the POM. It does provide an API but that has not yet been looked at in detail.