Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SOLR-17023: Use Modern NLP Models via ONNX and Apache OpenNLP with Solr #1999

Open
wants to merge 30 commits into
base: main
Choose a base branch
from
+1,126 −4
Open

SOLR-17023: Use Modern NLP Models via ONNX and Apache OpenNLP with Solr #1999

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
f19f660
first version w bats test
epugh Oct 10, 2023
48fd37b
not positive I need this
epugh Oct 10, 2023
cecd4f6
new name, and OpenNLP is kind of a implmentation detail ;-).
epugh Oct 10, 2023
10abbd0
baby steps, found that packagesstore blows up killing solr when i pos…
epugh Oct 10, 2023
dd4cc89
remove resolutionStraregy force from gradle build..
epugh Oct 10, 2023
af18d0c
match name in the underlying OpenNLP project. bikeshedding!
epugh Oct 10, 2023
6950629
tidy
epugh Oct 10, 2023
1adba22
reorder params
epugh Oct 11, 2023
bcbf16d
log formatting
epugh Oct 11, 2023
e634b1c
regenerate...
epugh Oct 11, 2023
778e3f4
dynamically grab the models from hugging face.
epugh Oct 11, 2023
405153b
use logging structure for stack traces
epugh Oct 11, 2023
4e54090
download the correct jar, and document the work to remove this need i…
epugh Oct 13, 2023
9e30c29
Upgrade to OpenNLP 2.3.1 and related dependencies.
epugh Nov 27, 2023
ef9364a
no longer need workarounds for gpu/cpu issues with updated OpenNLP.
epugh Nov 27, 2023
08b3de8
Merge remote-tracking branch 'upstream/main' into SOLR-17023
epugh Nov 27, 2023
2c106e9
We cleaned up the name ;-)
epugh Dec 5, 2023
8112229
Merge branch 'main' into SOLR-17023
epugh Jan 9, 2024
8d6187f
prompted to update the locks
epugh Jan 9, 2024
50887bb
Add in required license files
epugh Jan 9, 2024
a31ef8e
lint
epugh Jan 9, 2024
4b7528a
precommit warning
epugh Jan 9, 2024
3a760da
Merge remote-tracking branch 'origin/main' into SOLR-17023
cpoerschke Feb 13, 2024
bdee568
upgrade OpenNLP from 2.3.1 to 2.3.2 (to match Lucene main branch)
cpoerschke Feb 13, 2024
541dceb
tentative: minimum Java17 for this PR
cpoerschke Feb 13, 2024
633391e
Update gradle-precommit.yml - Java 11 --> 17
cpoerschke Feb 13, 2024
e2fe1fe
Update solrj-test.yml - Java 11 --> 17
cpoerschke Feb 13, 2024
855b5ea
Update docker-test.yml - Java 11 --> 17
cpoerschke Feb 13, 2024
9c16a8b
Update bin-solr-test.yml - Java 11 --> 17
cpoerschke Feb 13, 2024
476a7d4
undo 'tentative: minimum Java17 for this PR' -- see PR 1510 instead
cpoerschke Feb 13, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions solr/modules/analysis-extras/build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,13 @@ apply plugin: 'java-library'

description = 'Additional analysis components'

configurations.all {
resolutionStrategy {
force 'org.apache.opennlp:opennlp-tools:2.2.0'
force 'org.apache.opennlp:opennlp-dl:2.2.0'
}
}

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do you need for force a resolution? This shouldn't be necessary at all...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need apache/lucene#448 to be merged to get us to 2.2, or we ahve to work around Lucene... Is there a better way? This is what Jeff and I hacked in..

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

try org.apache.opennlp:opennlp*=2.2.0 in versions.props and remove this from here. It should force the upgrade to 2.2.0.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lots of errors when I took out the resolutionStrategy:

* What went wrong:
Could not determine the dependencies of task ':solr:modules:analysis-extras:compileJava'.
> Could not resolve all task dependencies for configuration ':solr:modules:analysis-extras:compileClasspath'.
   > Could not resolve org.apache.opennlp:opennlp-tools:2.2.0.
     Required by:
         project :solr:modules:analysis-extras
         project :solr:modules:analysis-extras > org.apache.opennlp:opennlp-dl:2.2.0
      > Cannot find a version of 'org.apache.opennlp:opennlp-tools' that satisfies the version constraints:
           Dependency path 'org.apache.solr:analysis-extras:10.0.0-SNAPSHOT' --> 'org.apache.opennlp:opennlp-tools:2.2.0'
           Constraint path 'org.apache.solr:analysis-extras:10.0.0-SNAPSHOT' --> 'org.apache.solr:core:10.0.0-SNAPSHOT' (apiElements) --> 'org.apache.opennlp:opennlp-tools:1.9.4' because of the following reason: Computed from com.palantir.consistent-versions' versions.lock in solr-root
           Constraint path 'org.apache.solr:analysis-extras:10.0.0-SNAPSHOT' --> 'org.apache.solr:solrj:10.0.0-SNAPSHOT' (apiElements) --> 'org.apache.opennlp:opennlp-tools:1.9.4' because of the following reason: Computed from com.palantir.consistent-versions' versions.lock in solr-root
           Dependency path 'org.apache.solr:analysis-extras:10.0.0-SNAPSHOT' --> 'org.apache.lucene:lucene-analysis-opennlp:9.8.0' (compile) --> 'org.apache.opennlp:opennlp-tools:1.9.1'

also, feel free to push up changes to this branch ;-)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

make sure to run ./gradlew --write-locks to make sure it takes into account versions.props changes.

dependencies {
api project(':solr:core')

Expand All @@ -33,6 +40,7 @@ dependencies {
implementation 'org.apache.lucene:lucene-core'
// NOTE: Need to stay on same version of opennlp-tools as lucene-analysis-opennlp
implementation 'org.apache.opennlp:opennlp-tools'
implementation 'org.apache.opennlp:opennlp-dl'
implementation 'org.slf4j:slf4j-api'

testImplementation project(':solr:test-framework')
Expand Down
Loading
Loading