SOLR-16781: Remove solrconfig.xml <lib> directives

[gerlowskija]

https://issues.apache.org/jira/browse/SOLR-16781

Description

Solr offers a number of ways for users to add JARs and resources to their classpath, including:

• solr.xml entries
• SOLR_MODULES env-var/support
• core and install-level "lib/" directories
• the package manager
• direct classpath modification

In addition to being largely redundant with the methods above, solrconfig.xml's directive has been a pain point and source of security concerns in the past.

Solution

This commit removes references and support from 'main'. Most of the functional changes live in SolrConfig.java, where we now check for tags only to log out a warning about them being no longer supported. The remainder of the PR consists of updates to our ref-guide, to recommend other means of adding JARs to Solr's classpath. Most of these are for first-party modules, which can use the SOLR_MODULES system.

This PR is only intended for main: I'll open a separate more forgiving PR for branch_9x that disables <lib> by default but allows it to be re-enabled via sysprop/envvar.

Still TODO:

• decide whether vestigial trusted/untrusted configset support should be removed entirely
• ensure that Solr examples use the necessary SOLR_MODULES for the libraries they rely on.
• CHANGES.txt and Upgrade Notes

Tests

Some modifications made to existing tests to ensure that tags no longer function (particularly TestConfigSetsAPI).

Checklist

Please review the following and check all that apply:

• I have reviewed the guidelines for How to Contribute and my code conforms to the standards described there to the best of my ability.
• I have created a Jira issue and added the issue ID to my pull request title.
• I have given Solr maintainers access to contribute to my PR branch. (optional but recommended, not available for branches on forks living under an organisation)
• I have developed this patch against the main branch.
• I have run ./gradlew check.
• I have added tests for my changes.
• I have added documentation for the Reference Guide

[epugh]

@gerlowskija how do you want to handle #2861? Should it go in first, or after?

[epugh]

@gerlowskija FYI I merged and backported https://issues.apache.org/jira/browse/SOLR-17556 to clear the decks, but which may impact this a bit. I think you plan for beefing up bin/solr start -e techproducts is a "good thing".

[gerlowskija]

how do you want to handle #2861? Should it go in first, or after?

Sorry - missed this comment the other day. I'm glad I didn't prevent you from merging 2861 - I'm 👍 to incorporate any fallout into this PR.

[gerlowskija]

Alright - I've fixed merge conflicts, and tested out the examples, so this should be good to merge shortly. Thanks for the reviews all!

I've created a corresponding change for branch_9x that will disable <lib> by default, but still leave the feature available for users that really want it. Would appreciate any feedback there as well!

[HoustonPutman]

Maybe this file doesn't need to exist at all?

[gerlowskija]

Good catch - removed!

[gerlowskija]

One thing we'll probably want/need to come up with a better story around is model-serving.

Several places in our docs described using <lib> as a way not of loading new JARs, but of accessing model files. Users can still raise the ZK filesize limit and access their models from there, so we're not breaking those usecases. But all the same, it'd be great if there was a way to do this that didn't require modifying the ZK limit. (Does solr.xml's <sharedLib> tag already get us this maybe?)

[HoustonPutman]

One thing we'll probably want/need to come up with a better story around is model-serving.

Several places in our docs described using <lib> as a way not of loading new JARs, but of accessing model files. Users can still raise the ZK filesize limit and access their models from there, so we're not breaking those usecases. But all the same, it'd be great if there was a way to do this that didn't require modifying the ZK limit. (Does solr.xml's <sharedLib> tag already get us this maybe?)

Yeah, it should, but there is also the $SOLR_ROOT_DIR/lib option as well

[gerlowskija]

Alright - merged! Thanks all for the reviews!

We'll still need to decide whether to remove the last remnants of the "trusted" configset code, which this PR mostly leaves in tact since several components (e.g. ScriptUpdateProcessorFactory, XSLTUpdateRequestHandler) still rely on this distinction even with the <lib> code removed.

My vote would be to tear it out. All the places it's still used require the toggling of a module at startup (SOLR_MODULES=scripting), and I'm not sure that "trusted"-ness adds any value on top of that. Particularly given how many gaps have been found over time in the "trusted-config" code. But I'll make that case in its own JIRA ticket and we can discuss there...

[dsmiley]

My vote would be to tear it out.

+1
I'm glad to hear you suggest this. I thought I might have been alone in this thinking. I don't think we should "support" this idea; it's too burdensome for us to uphold this concept to a good standard.