Skip to content

Commit

Permalink
.Net: StepwisePlanner Chat support (#2504)
Browse files Browse the repository at this point in the history 
### Motivation and Context
This pull request introduces a series of improvements and updates to the
StepwisePlanner, StepwisePlannerTests, and WebSearchEngineSkill. The
changes include
better handling of action invocation, function retrieval, logging, chat
history management, execution helpers, and the addition of an offset
parameter to the
WebSearchEngineSkill. Additionally, the StepwisePlannerTests have been
enhanced with more test cases and refined expected minimum steps, while
the
ChatRequestSettings class has been improved with a method to create a
new settings object from another settings object.

Resolves #2367
Fixes #2466 
Fixes #2553 

### Description
1. Improved the StepwisePlanner class with better action invocation,
function retrieval, logging, chat history management, and execution
helpers.
2. Enhanced the StepwisePlannerTests with more test cases and refined
expected minimum steps.
3. Added an offset parameter to the WebSearchEngineSkill class for
improved search functionality.
4. Improved the ChatRequestSettings class with a method to create a new
settings object from another settings object.
5. Updated the Example51_StepwisePlanner.cs file with new questions,
improved result handling, and additional output for better analysis.
6. Enhanced the ParseResultTests with new test cases for the ParseResult
class.
7. Updated the WebSearchEngineSkill to include parameter descriptions
for the SearchAsync method.
8. Added project references and minor changes to the
SemanticKernel.MetaPackage project.

### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄

Co-authored-by: Andrew Hesky @andhesky
  • Loading branch information
@lemillermicrosoft
lemillermicrosoft authored Sep 6, 2023
1 parent 7313258 commit f6fb9f9
Show file tree
Hide file tree
Showing 20 changed files with 892 additions and 367 deletions.
9 changes: 9 additions & 0 deletions dotnet/SK-dotnet.sln
View file
Original file line number Diff line number Diff line change
Expand Up @@ -158,6 +158,8 @@ Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Reliability.Polly", "src\Ex
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Reliability.Basic", "src\Extensions\Reliability.Basic\Reliability.Basic.csproj", "{3DC4DBD8-20A5-4937-B4F5-BB5E24E7A567}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Extensions.StepwisePlanner.UnitTests", "src\Extensions\Extensions.StepwisePlanner.UnitTests\Extensions.StepwisePlanner.UnitTests.csproj", "{6F651E87-F16E-407B-AF7F-B3475F850E9A}"
EndProject
Global
GlobalSection(SolutionConfigurationPlatforms) = preSolution
Debug|Any CPU = Debug|Any CPU
Expand Down Expand Up @@ -393,6 +395,12 @@ Global
{3DC4DBD8-20A5-4937-B4F5-BB5E24E7A567}.Publish|Any CPU.Build.0 = Publish|Any CPU
{3DC4DBD8-20A5-4937-B4F5-BB5E24E7A567}.Release|Any CPU.ActiveCfg = Release|Any CPU
{3DC4DBD8-20A5-4937-B4F5-BB5E24E7A567}.Release|Any CPU.Build.0 = Release|Any CPU
{6F651E87-F16E-407B-AF7F-B3475F850E9A}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{6F651E87-F16E-407B-AF7F-B3475F850E9A}.Debug|Any CPU.Build.0 = Debug|Any CPU
{6F651E87-F16E-407B-AF7F-B3475F850E9A}.Publish|Any CPU.ActiveCfg = Debug|Any CPU
{6F651E87-F16E-407B-AF7F-B3475F850E9A}.Publish|Any CPU.Build.0 = Debug|Any CPU
{6F651E87-F16E-407B-AF7F-B3475F850E9A}.Release|Any CPU.ActiveCfg = Release|Any CPU
{6F651E87-F16E-407B-AF7F-B3475F850E9A}.Release|Any CPU.Build.0 = Release|Any CPU
EndGlobalSection
GlobalSection(SolutionProperties) = preSolution
HideSolutionNode = FALSE
Expand Down Expand Up @@ -449,6 +457,7 @@ Global
{10E4B697-D4E8-468D-872D-49670FD150FB} = {078F96B4-09E1-4E0E-B214-F71A4F4BF633}
{D4540A0F-98E3-4E70-9093-1948AE5B2AAD} = {078F96B4-09E1-4E0E-B214-F71A4F4BF633}
{3DC4DBD8-20A5-4937-B4F5-BB5E24E7A567} = {078F96B4-09E1-4E0E-B214-F71A4F4BF633}
{6F651E87-F16E-407B-AF7F-B3475F850E9A} = {078F96B4-09E1-4E0E-B214-F71A4F4BF633}
EndGlobalSection
GlobalSection(ExtensibilityGlobals) = postSolution
SolutionGuid = {FBDC56A3-86AD-4323-AA0F-201E59123B83}
Expand Down
16 changes: 8 additions & 8 deletions dotnet/samples/KernelSyntaxExamples/Example15_TextMemorySkill.cs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -146,7 +146,7 @@ private static async Task RunWithStoreAsync(IMemoryStore memoryStore, Cancellati
.WithOpenAITextEmbeddingGenerationService(TestConfiguration.OpenAI.EmbeddingModelId, TestConfiguration.OpenAI.ApiKey)
.Build();

// Create an embedding generator to use for semantic memory.
// Create an embedding generator to use for semantic memory.
var embeddingGenerator = new OpenAITextEmbeddingGeneration(TestConfiguration.OpenAI.EmbeddingModelId, TestConfiguration.OpenAI.ApiKey);

// The combination of the text embedding generator and the memory store makes up the 'SemanticTextMemory' object used to
Expand All @@ -155,7 +155,7 @@ private static async Task RunWithStoreAsync(IMemoryStore memoryStore, Cancellati

/////////////////////////////////////////////////////////////////////////////////////////////////////
// PART 1: Store and retrieve memories using the ISemanticTextMemory (textMemory) object.
//
//
// This is a simple way to store memories from a code perspective, without using the Kernel.
/////////////////////////////////////////////////////////////////////////////////////////////////////
Console.WriteLine("== PART 1a: Saving Memories through the ISemanticTextMemory object ==");
Expand All @@ -175,12 +175,12 @@ private static async Task RunWithStoreAsync(IMemoryStore memoryStore, Cancellati
// Retrieve a memory
Console.WriteLine("== PART 1b: Retrieving Memories through the ISemanticTextMemory object ==");
MemoryQueryResult? lookup = await textMemory.GetAsync(MemoryCollectionName, "info1", cancellationToken: cancellationToken);
Console.WriteLine("Memory with key 'info3':" + lookup?.Metadata.Text ?? "ERROR: memory not found");
Console.WriteLine("Memory with key 'info1':" + lookup?.Metadata.Text ?? "ERROR: memory not found");
Console.WriteLine();

/////////////////////////////////////////////////////////////////////////////////////////////////////
// PART 2: Create TextMemorySkill, store and retrieve memories through the Kernel.
//
//
// This enables semantic functions and the AI (via Planners) to access memories
/////////////////////////////////////////////////////////////////////////////////////////////////////

Expand Down Expand Up @@ -212,8 +212,8 @@ private static async Task RunWithStoreAsync(IMemoryStore memoryStore, Cancellati

/////////////////////////////////////////////////////////////////////////////////////////////////////
// PART 3: Recall similar ideas with semantic search
//
// Uses AI Embeddings for fuzzy lookup of memories based on intent, rather than a specific key.
//
// Uses AI Embeddings for fuzzy lookup of memories based on intent, rather than a specific key.
/////////////////////////////////////////////////////////////////////////////////////////////////////

Console.WriteLine("== PART 3: Recall (similarity search) with AI Embeddings ==");
Expand Down Expand Up @@ -260,7 +260,7 @@ private static async Task RunWithStoreAsync(IMemoryStore memoryStore, Cancellati

/////////////////////////////////////////////////////////////////////////////////////////////////////
// PART 3: TextMemorySkill Recall in a Semantic Function
//
//
// Looks up related memories when rendering a prompt template, then sends the rendered prompt to
// the text completion model to answer a natural language query.
/////////////////////////////////////////////////////////////////////////////////////////////////////
Expand Down Expand Up @@ -300,7 +300,7 @@ END FACTS

/////////////////////////////////////////////////////////////////////////////////////////////////////
// PART 5: Cleanup, deleting database collection
//
//
/////////////////////////////////////////////////////////////////////////////////////////////////////

Console.WriteLine("== PART 5: Cleanup, deleting database collection ==");
Expand Down
223 changes: 168 additions & 55 deletions dotnet/samples/KernelSyntaxExamples/Example51_StepwisePlanner.cs
View file
Original file line number Diff line number Diff line change
@@ -1,9 +1,12 @@
// Copyright (c) Microsoft. All rights reserved.

using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Linq;
using System.Threading.Tasks;
using Microsoft.SemanticKernel;
using Microsoft.SemanticKernel.Orchestration;
using Microsoft.SemanticKernel.Planning;
using Microsoft.SemanticKernel.Skills.Core;
using Microsoft.SemanticKernel.Skills.Web;
Expand All @@ -12,77 +15,216 @@
using RepoUtils;

/**
* This example shows how to use Stepwise Planner to create a plan for a given goal.
* This example shows how to use Stepwise Planner to create and run a stepwise plan for a given goal.
*/

// ReSharper disable once InconsistentNaming
public static class Example51_StepwisePlanner
{
// Used to override the max allowed tokens when running the plan
internal static int? ChatMaxTokens = null;
internal static int? TextMaxTokens = null;

// Used to quickly modify the chat model used by the planner
internal static string? ChatModelOverride = null; //"gpt-35-turbo";
internal static string? TextModelOverride = null; //"text-davinci-003";

internal static string? Suffix = null;

public static async Task RunAsync()
{
string[] questions = new string[]
{
"Who is the current president of the United States? What is his current age divided by 2",
// "Who is Leo DiCaprio's girlfriend? What is her current age raised to the (his current age)/100 power?",
// "What is the capital of France? Who is that city's current mayor? What percentage of their life has been in the 21st century as of today?",
// "What is the current day of the calendar year? Using that as an angle in degrees, what is the area of a unit circle with that angle?"
"What color is the sky?",
"What is the weather in Seattle?",
"What is the tallest mountain on Earth? How tall is it divided by 2?",
"What is the capital of France? Who is that city's current mayor? What percentage of their life has been in the 21st century as of today?",
"What is the current day of the calendar year? Using that as an angle in degrees, what is the area of a unit circle with that angle?",
"If a spacecraft travels at 0.99 the speed of light and embarks on a journey to the nearest star system, Alpha Centauri, which is approximately 4.37 light-years away, how much time would pass on Earth during the spacecraft's voyage?"
};

foreach (var question in questions)
{
var kernel = GetKernel();
await RunWithQuestion(kernel, question);
for (int i = 0; i < 1; i++)
{
await RunTextCompletion(question);
await RunChatCompletion(question);
}
}

PrintResults();
}

// print out summary table of ExecutionResults
private static void PrintResults()
{
Console.WriteLine("**************************");
Console.WriteLine("Execution Results Summary:");
Console.WriteLine("**************************");

foreach (var question in ExecutionResults.Select(s => s.question).Distinct())
{
Console.WriteLine("Question: " + question);
Console.WriteLine("Mode\tModel\tAnswer\tStepsTaken\tIterations\tTimeTaken");
foreach (var er in ExecutionResults.OrderByDescending(s => s.model).Where(s => s.question == question))
{
Console.WriteLine($"{er.mode}\t{er.model}\t{er.stepsTaken}\t{er.iterations}\t{er.timeTaken}\t{er.answer}");
}
}
}

private static async Task RunWithQuestion(IKernel kernel, string question)
private struct ExecutionResult
{
public string mode;
public string? model;
public string? question;
public string? answer;
public string? stepsTaken;
public string? iterations;
public string? timeTaken;
}

private static List<ExecutionResult> ExecutionResults = new();

private static async Task RunTextCompletion(string question)
{
Console.WriteLine("RunTextCompletion");
ExecutionResult currentExecutionResult = default;
currentExecutionResult.mode = "RunTextCompletion";
var kernel = GetKernel(ref currentExecutionResult);
await RunWithQuestion(kernel, currentExecutionResult, question, TextMaxTokens);
}

private static async Task RunChatCompletion(string question, string? model = null)
{
Console.WriteLine("RunChatCompletion");
ExecutionResult currentExecutionResult = default;
currentExecutionResult.mode = "RunChatCompletion";
var kernel = GetKernel(ref currentExecutionResult, true, model);
await RunWithQuestion(kernel, currentExecutionResult, question, ChatMaxTokens);
}

private static async Task RunWithQuestion(IKernel kernel, ExecutionResult currentExecutionResult, string question, int? MaxTokens = null)
{
currentExecutionResult.question = question;
var bingConnector = new BingConnector(TestConfiguration.Bing.ApiKey);
var webSearchEngineSkill = new WebSearchEngineSkill(bingConnector);

kernel.ImportSkill(webSearchEngineSkill, "WebSearch");
kernel.ImportSkill(new LanguageCalculatorSkill(kernel), "advancedCalculator");
kernel.ImportSkill(new LanguageCalculatorSkill(kernel), "semanticCalculator");
kernel.ImportSkill(new TimeSkill(), "time");

// StepwisePlanner is instructed to depend on available functions.
// We expose this function to increase the flexibility in it's ability to answer
// given the relatively small number of functions we have in this example.
// This seems to be particularly helpful in these examples with gpt-35-turbo -- even though it
// does not *use* this function. It seems to help the planner find a better path to the answer.
kernel.CreateSemanticFunction(
"Generate an answer for the following question: {{$input}}",
functionName: "GetAnswerForQuestion",
skillName: "AnswerBot",
description: "Given a question, get an answer and return it as the result of the function");

Console.WriteLine("*****************************************************");
Stopwatch sw = new();
Console.WriteLine("Question: " + question);

var plannerConfig = new Microsoft.SemanticKernel.Planning.Stepwise.StepwisePlannerConfig();
plannerConfig.ExcludedFunctions.Add("TranslateMathProblem");
plannerConfig.ExcludedFunctions.Add("DaysAgo");
plannerConfig.ExcludedFunctions.Add("DateMatchingLastDayName");
plannerConfig.MinIterationTimeMs = 1500;
plannerConfig.MaxTokens = 4000;
plannerConfig.MaxIterations = 25;

StepwisePlanner planner = new(kernel, plannerConfig);
sw.Start();
var plan = planner.CreatePlan(question);
if (!string.IsNullOrEmpty(Suffix))
{
plannerConfig.Suffix = $"{Suffix}\n{plannerConfig.Suffix}";
currentExecutionResult.question = $"[Assisted] - {question}";
}

var result = await plan.InvokeAsync(kernel.CreateNewContext());
Console.WriteLine("Result: " + result);
if (result.Variables.TryGetValue("stepCount", out string? stepCount))
if (MaxTokens.HasValue)
{
Console.WriteLine("Steps Taken: " + stepCount);
plannerConfig.MaxTokens = MaxTokens.Value;
}

if (result.Variables.TryGetValue("skillCount", out string? skillCount))
SKContext result;
sw.Start();

try
{
Console.WriteLine("Skills Used: " + skillCount);
StepwisePlanner planner = new(kernel: kernel, config: plannerConfig);
var plan = planner.CreatePlan(question);

result = await plan.InvokeAsync(kernel.CreateNewContext());

if (result.Result.Contains("Result not found, review _stepsTaken to see what", StringComparison.OrdinalIgnoreCase))
{
Console.WriteLine("Could not answer question in " + plannerConfig.MaxIterations + " iterations");
currentExecutionResult.answer = "Could not answer question in " + plannerConfig.MaxIterations + " iterations";
}
else
{
Console.WriteLine("Result: " + result.Result);
currentExecutionResult.answer = result.Result;
}

if (result.Variables.TryGetValue("stepCount", out string? stepCount))
{
Console.WriteLine("Steps Taken: " + stepCount);
currentExecutionResult.stepsTaken = stepCount;
}

if (result.Variables.TryGetValue("skillCount", out string? skillCount))
{
Console.WriteLine("Skills Used: " + skillCount);
}

if (result.Variables.TryGetValue("iterations", out string? iterations))
{
Console.WriteLine("Iterations: " + iterations);
currentExecutionResult.iterations = iterations;
}
}
#pragma warning disable CA1031
catch (Exception ex)
{
Console.WriteLine("Exception: " + ex);
}

Console.WriteLine("Time Taken: " + sw.Elapsed);
currentExecutionResult.timeTaken = sw.Elapsed.ToString();
ExecutionResults.Add(currentExecutionResult);
Console.WriteLine("*****************************************************");
}

private static IKernel GetKernel()
private static IKernel GetKernel(ref ExecutionResult result, bool useChat = false, string? model = null)
{
var builder = new KernelBuilder();
var maxTokens = 0;
if (useChat)
{
builder.WithAzureChatCompletionService(
model ?? ChatModelOverride ?? TestConfiguration.AzureOpenAI.ChatDeploymentName,
TestConfiguration.AzureOpenAI.Endpoint,
TestConfiguration.AzureOpenAI.ApiKey,
alsoAsTextCompletion: true,
setAsDefault: true);

maxTokens = ChatMaxTokens ?? (new Microsoft.SemanticKernel.Planning.Stepwise.StepwisePlannerConfig()).MaxTokens;
result.model = model ?? ChatModelOverride ?? TestConfiguration.AzureOpenAI.ChatDeploymentName;
}
else
{
builder.WithAzureTextCompletionService(
model ?? TextModelOverride ?? TestConfiguration.AzureOpenAI.DeploymentName,
TestConfiguration.AzureOpenAI.Endpoint,
TestConfiguration.AzureOpenAI.ApiKey);

builder.WithAzureChatCompletionService(
TestConfiguration.AzureOpenAI.ChatDeploymentName,
TestConfiguration.AzureOpenAI.Endpoint,
TestConfiguration.AzureOpenAI.ApiKey,
alsoAsTextCompletion: true,
setAsDefault: true);
maxTokens = TextMaxTokens ?? (new Microsoft.SemanticKernel.Planning.Stepwise.StepwisePlannerConfig()).MaxTokens;
result.model = model ?? TextModelOverride ?? TestConfiguration.AzureOpenAI.DeploymentName;
}

Console.WriteLine($"Model: {result.model} ({maxTokens})");

var kernel = builder
.WithLoggerFactory(ConsoleLogger.LoggerFactory)
Expand All @@ -97,32 +239,3 @@ private static IKernel GetKernel()
return kernel;
}
}

// *****************************************************
// Question: Who is the current president of the United States? What is his current age divided by 2
// Result: The current president of the United States is Joe Biden. His current age divided by 2 is 40.5.
// Steps Taken: 9
// Skills Used: 7 (WebSearch.Search(4), time.Year(1), time.Date(1), advancedCalculator.Calculator(1))
// Time Taken: 00:01:13.3766860
// *****************************************************
// *****************************************************
// Question: Who is Leo DiCaprio's girlfriend? What is her current age raised to the (his current age)/100 power?
// Result: Leo DiCaprio's girlfriend is Camila Morrone. Her current age raised to the power of (his current age)/100 is approximately 4.94.
// Steps Taken: 9
// Skills Used: 5 (WebSearch.Search(3), time.Year(1), advancedCalculator.Calculator(1))
// Time Taken: 00:01:17.6742136
// *****************************************************
// *****************************************************
// Question: What is the capital of France? Who is that cities current mayor? What percentage of their life has been in the 21st century as of today?
// Result: The capital of France is Paris. The current mayor of Paris is Anne Hidalgo, who was born on June 19, 1959. As of today, she has lived for 64 years, with 23 of those years in the 21st century. Therefore, 35.94% of her life has been spent in the 21st century.
// Steps Taken: 14
// Skills Used: 12 (WebSearch.Search(8), time.Year(1), advancedCalculator.Calculator(3))
// Time Taken: 00:02:06.6682909
// *****************************************************
// *****************************************************
// Question: What is the current day of the calendar year? Using that as an angle in degrees, what is the area of a unit circle with that angle?
// Result: The current day of the year is 177. Using that as an angle in degrees (approximately 174.58), the area of a unit circle with that angle is approximately 1.523 square units.
// Steps Taken: 11
// Skills Used: 9 (time.Now(1), time.DayOfYear(1), time.DaysBetween(1), time.MonthNumber(1), time.Day(1), advancedCalculator.Calculator(4))
// Time Taken: 00:01:41.5585861
// *****************************************************
Loading

0 comments on commit f6fb9f9

Please sign in to comment.