references stop working; index file not found errors; deadlocks with autobuild.enabled = false and maxConcurrentBuilds=8

[mfussenegger]

I unfortunately don't have a good reproduction for this, but after one of the recent eclipse.core updates I started noticing that listing references for a symbol occassionally stops working.

The logs then show errors like:

.metadata/.plugins/org.eclipse.jdt.core/744288969.index (No such file or directory)

Java Index broken - will be automatically deleted to repair: 

Failed to update qualified index.

And I'm also seeing errors like:

Deadlock detected. All locks owned by thread Worker-2: Update project crate will be suspended.
null
java.lang.IllegalStateException
\tat org.eclipse.core.internal.jobs.DeadlockDetector.reportDeadlock(DeadlockDetector.java:624)
\tat org.eclipse.core.internal.jobs.DeadlockDetector.lockWaitStart(DeadlockDetector.java:413)
\tat org.eclipse.core.internal.jobs.LockManager.addLockWaitThread(LockManager.java:143)
\tat org.eclipse.core.internal.jobs.ThreadJob.waitForRun(ThreadJob.java:319)
\tat org.eclipse.core.internal.jobs.ThreadJob.joinRun(ThreadJob.java:214)
\tat org.eclipse.core.internal.jobs.ImplicitJobs.begin(ImplicitJobs.java:95)
\tat org.eclipse.core.internal.jobs.JobManager.beginRule(JobManager.java:341)
\tat org.eclipse.core.internal.resources.WorkManager.checkIn(WorkManager.java:124)
\tat org.eclipse.core.internal.resources.Workspace.prepareOperation(Workspace.java:2398)
\tat org.eclipse.core.internal.resources.Resource.refreshLocal(Resource.java:1595)
\tat org.eclipse.jdt.apt.core.internal.generatedfile.GeneratedSourceFolderManager.createOnDisk(GeneratedSourceFolderManager.java:241)
\tat org.eclipse.jdt.apt.core.internal.generatedfile.GeneratedSourceFolderManager.ensureFolderExists(GeneratedSourceFolderManager.java:190)
\tat org.eclipse.jdt.apt.core.internal.generatedfile.GeneratedSourceFolderManager.ensureFolderExists(GeneratedSourceFolderManager.java:226)
\tat org.eclipse.jdt.apt.core.internal.generatedfile.GeneratedResourceChangeListener.addGeneratedSrcFolderTo(GeneratedResourceChangeListener.java:128)
\tat org.eclipse.jdt.apt.core.internal.generatedfile.GeneratedResourceChangeListener.resourceChanged(GeneratedResourceChangeListener.java:96)
\tat org.eclipse.jdt.internal.core.DeltaProcessingState$1.run(DeltaProcessingState.java:496)
\tat org.eclipse.core.runtime.SafeRunner.run(SafeRunner.java:47)
\tat org.eclipse.jdt.internal.core.DeltaProcessingState.resourceChanged(DeltaProcessingState.java:489)
\tat org.eclipse.core.internal.events.NotificationManager$1.run(NotificationManager.java:321)
\tat org.eclipse.core.runtime.SafeRunner.run(SafeRunner.java:47)
\tat org.eclipse.core.internal.events.NotificationManager.notify(NotificationManager.java:311)
\tat org.eclipse.core.internal.events.NotificationManager.broadcastChanges(NotificationManager.java:174)
\tat org.eclipse.core.internal.resources.Workspace.broadcastBuildEvent(Workspace.java:444)
\tat org.eclipse.core.internal.resources.Workspace.aboutToBuild(Workspace.java:387)
\tat org.eclipse.core.internal.resources.Project$1.run(Project.java:595)
\tat org.eclipse.core.internal.resources.Workspace.run(Workspace.java:2448)
\tat org.eclipse.core.internal.resources.Project.internalBuild(Project.java:642)
\tat org.eclipse.core.internal.resources.Project.build(Project.java:154)
\tat org.eclipse.m2e.core.internal.project.ProjectConfigurationManager.lambda$5(ProjectConfigurationManager.java:447)
\tat java.base/java.util.Collection.removeIf(Collection.java:581)
\tat org.eclipse.m2e.core.internal.project.ProjectConfigurationManager.updateProjectConfiguration0(ProjectConfigurationManager.java:438)
\tat org.eclipse.m2e.core.internal.project.ProjectConfigurationManager.lambda$3(ProjectConfigurationManager.java:339)
\tat org.eclipse.m2e.core.internal.embedder.MavenExecutionContext.executeBare(MavenExecutionContext.java:458)
\tat org.eclipse.m2e.core.internal.embedder.MavenExecutionContext.execute(MavenExecutionContext.java:339)
\tat org.eclipse.m2e.core.internal.embedder.MavenExecutionContext.execute(MavenExecutionContext.java:278)
\tat org.eclipse.m2e.core.internal.embedder.MavenImpl.execute(MavenImpl.java:1053)
\tat org.eclipse.m2e.core.internal.project.ProjectConfigurationManager.updateProjectConfiguration(ProjectConfigurationManager.java:338)
\tat org.eclipse.m2e.core.internal.project.ProjectConfigurationManager.updateProjectConfiguration(ProjectConfigurationManager.java:325)
\tat org.eclipse.jdt.ls.core.internal.managers.MavenBuildSupport.update(MavenBuildSupport.java:86)
\tat org.eclipse.jdt.ls.core.internal.managers.ProjectsManager$3.runInWorkspace(ProjectsManager.java:490)
\tat org.eclipse.core.internal.resources.InternalWorkspaceJob.run(InternalWorkspaceJob.java:43)
\tat org.eclipse.core.internal.jobs.Worker.run(Worker.java:63)

Setting maxConcurrentBuilds = 1, seems to resolve the problem

I'd suspect there's some missing locking and synchronization issues. Due to that I haven't found a better reproduction other than "use it for some time with maxConcurrentBuilds = 8".

[mfussenegger]

Actually, this was probably a false alarm. There appears to be a race condition in the client that causes more than 1 client/server to be started - where both try to operate on the same workdir.

[appleseedexm]

This saved me from tons of debugging, old jdtls instance was lingering in the background which caused the same behaviour.