Using LLMSharpEmbeddings to setup a rag from a local LLM File

[mzand111]

Description

What I want to achieve

I'm trying to use LLMSharpEmbeddings to use a SqlLite db to setup a RAG:

 var modelPath = @"D:\LLMModels\Models\thespis-13b-v0.5.Q2_K.gguf";
 //Also tested with Meta-Llama-3.1-8B-Instruct-Q6_K

 var vectorDatabase = new SqLiteVectorDatabase(dataSource: @"D:\LLMModels\DBs\hp.db");
 var embeddingModel = LLamaSharpEmbeddings.FromPath(modelPath);

 var fileAddress = @"D:\LLMModels\RAGResources\Books\Harry-Potter-and-the-Philosophers-Stone.pdf";
 var dataSource = DataSource.FromPath(fileAddress);


 var vectorCollection = await vectorDatabase.AddDocumentsFromAsync<PdfPigPdfLoader>(
     embeddingModel, 
     dimensions: 5120, // Not sure about this param!
     dataSource: dataSource,
     collectionName: "harrypotter", 
     textSplitter: null,
     behavior: AddDocumentsToDatabaseBehavior.OverwriteExistingCollection);


 const string question = "What is Harry's Address?";
 var similarDocuments = await vectorCollection.GetSimilarDocuments(embeddingModel, question, amount: 5);

 Console.WriteLine($"Similar Doc Count: {similarDocuments.Count}");

However at this line:

var embeddingModel = LLamaSharpEmbeddings.FromPath(modelPath);

the following exception is thrown:
System.MissingMethodException HResult=0x80131513 Message=Method not found: '!!0 Microsoft.Extensions.AI.IEmbeddingGenerator2.GetService(System.Object)'.
Source=LangChain.Providers.LLamaSharp
StackTrace:
at LangChain.Providers.LLamaSharp.LLamaSharpEmbeddings..ctor(LLamaSharpConfiguration configuration)
`

I also tried this:

 var embeddingModel = new LLamaSharpEmbeddings(new LLamaSharpConfiguration
 {
     PathToModelFile = modelPath,
     EmbeddingMode = true,
 });

Steps to reproduce the use the smple code

1. Create a console app and add required packages
2. Copy and paste the given code to the main method
3. Adjust the address of the model to your local address
4. Run the project

NuGet package version

<PackageReference Include="LangChain" Version="0.15.3-dev.14" /> <PackageReference Include="LangChain.Databases.Sqlite" Version="0.15.4-dev.48" /> <PackageReference Include="LangChain.DocumentLoaders.Pdf" Version="0.15.3-dev.14" /> <PackageReference Include="LangChain.Providers.Abstractions" Version="0.15.3-dev.93" /> <PackageReference Include="LangChain.Providers.HuggingFace" Version="0.15.3-dev.93" /> <PackageReference Include="LangChain.Providers.LLamaSharp" Version="0.15.3-dev.93" /> <PackageReference Include="LLamaSharp" Version="0.19.0" /> <PackageReference Include="LLamaSharp.Backend.Cuda12.Windows" Version="0.19.0" />

[biapar]

Have you tried to use the Microsoft KernelMemory package?

[bturski]

I just went through this too. Commit [972e5ae] will fix it for 9.0.1-preview.1.24570.5 of Microsoft.Extensions.AI.Abstractions when it gets released. If you clone the repo, the examples work with latest code.

Could downgrade to v0.18.0 for now and should be good to go. The fun side to pre-release versions and interface breakage.