OCI Provder: Update OCIPromptTokensDetails #14092
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Fixes #14063 |
|
I'm trying to use OCI Generative AI's Llama via LiteLLM Proxy with Hugging Face smolagents and n8n, and I'm facing the same issue (#14063 ). I'm eagerly awaiting the merge of this PR. |
|
According to the Oracle Docs: https://docs.oracle.com/en-us/iaas/tools/python/2.159.0/api/generative_ai_inference/models/oci.generative_ai_inference.models.Usage.html#oci.generative_ai_inference.models.Usage, completionTokensDetails and promptTokensDetails are not required. Therefore, OCI Generative AI may not return these two values. However, in the current OCIResponseUsage class, these are mandatory, causing pydantic validation errors. It is appropriate to make these optional, and the fix in this PR is valid. Note that the failing Mock Test is related to groq/llama and is unrelated to this fix, which only pertains to the OCI provider. I look forward to the review being completed and the PR being merged soon. |
|
Hey @gotsysdba we should always have usage - if the usage is missing, let's use this will ensure we always have some estimate of cost |
Title
Fix parsing failure due to a validation error because
completionTokensDetailsandpromptTokensDetailsare missing in the completion endpoint response with certain models.Relevant issues
Fixes #14090
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
tests/litellm/directory, Adding at least 1 test is a hard requirement - see detailsmake test-unitType
🐛 Bug Fix
✅ Test
Changes
Update the OCIResponseUsage model:
add test.