[Add] Added function calling support to GeminiClient

[arjun-g]

Why are these changes needed?

I have added function calling ability to GeminiClient so that we can use autogen tools with Gemini as well.

Related issue number

Completes "Function Calling" roadmap item in #2387

Checks

• I've included any doc changes needed for https://microsoft.github.io/autogen/. See https://microsoft.github.io/autogen/docs/Contribute#documentation to build and test documentation locally.
• I've added tests (if relevant) corresponding to the changes introduced in this PR.
• I've made sure all auto checks have passed.

[arjun-g]

I am learning python just for the past few days. Kindly give me feedback on what I can improve in this.

[BeibinLi]

Thanks for your great PR @arjun-g !! There are two tiny stuff missing: test cases and tutorial notebook.
Can you add test cases for the new features? I can help to update the Gemini notebook after the test cases are done.

[arjun-g]

@BeibinLi added a test case. Let me know if this is suffice

[r4881t]

@arjun-g - A notebook example of using this would be great to have.

[arjun-g]

@BeibinLi added a notebook as well.

[BeibinLi]

Thanks @arjun-g for updating. Take a look at @luxzoli PR #2805 , and we might need to slightly update this PR to be consistent with vertexai package. I will edit the notebook after luxzoli's PR is merged. Then, I will edit this PR and merge it.

[arjun-g]

@BeibinLi got it. I will wait for #2805 to be merged. The function call is a bit different in vertex ai. I will handle that and give the PR.

[gitguardian]

⚠️ GitGuardian has uncovered 1 secret following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

Since your pull request originates from a forked repository, GitGuardian is not able to associate the secrets uncovered with secret incidents on your GitGuardian dashboard.
Skipping this check run and merging your pull request will create secret incidents on your GitGuardian dashboard.

🔎 Detected hardcoded secret in your pull request

GitGuardian id	GitGuardian status	Secret	Commit	Filename
11616921	Triggered	Generic High Entropy Secret	6563248	notebook/agentchat_agentops.ipynb	View secret

🛠 Guidelines to remediate hardcoded secrets

1. Understand the implications of revoking this secret by investigating where it is used in your code.
2. Replace and store your secret safely. Learn here the best practices.
3. Revoke and rotate this secret.
4. If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider

• following these best practices for managing and storing secrets including API keys and other credentials
• install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.

---

^{🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.}

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 43.75000% with 72 lines in your changes missing coverage. Please review.

Project coverage is 25.59%. Comparing base (03259b2) to head (6563248).
Report is 123 commits behind head on main.

Files	Patch %	Lines
autogen/oai/gemini.py	43.75%	56 Missing and 16 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2793      +/-   ##
==========================================
- Coverage   32.45%   25.59%   -6.87%     
==========================================
  Files          93       99       +6     
  Lines       10109    10261     +152     
  Branches     2172     2358     +186     
==========================================
- Hits         3281     2626     -655     
- Misses       6544     7325     +781     
- Partials      284      310      +26     

Flag	Coverage Δ
unittest	12.35% <2.34%> (?)
unittests	24.81% <43.75%> (-7.65%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[robraux]

Is there anything else you need here @arjun-g? This is an important component for the gemini client I'd love to make use of in a more official capacity. I'm happy to help wrap up.

[arjun-g]

@robraux I am held up in a work. Will update the PR in couple of days,

[luxzoli]

It would be really nice to have this feature 🙂

[MO7YW4NG]

Looking forward to having this feature 🙂

[luxzoli]

@arjun-g I have prepared a pull request to your repo that would resolve the merge conflicts, synchronize updates from the upstream autogen repo, and also add support for tool config with function calling.

Only changes to gemini.py , test_gemini.py , and openai_utils.py need to be reviewed, as the rest is changed due to synchronization with the official autogen repo except for agentchat_gemini.ipynb, where the difference is only automated formatting, which is not visible to the user anyway and thus irrelevant.

I hope this would save you some effort in case you still do not have time to finish this PR 🙂

Link to the PR: arjun-g#1

[ekzhu]

@luxzoli would you like to make a PR with your changes directly to this repo? Unless @arjun-g you would like to continue with this?

[luxzoli]

@ekzhu yes, sure. I can make a PR directly to the repo if @arjun-g will not finalize this PR

[ekzhu]

@arjun-g please let us know this week. Thanks!

[ekzhu]

Closing this as work continue in #3726