fix(ollama): improve context window detection and parameter parsing #2210
+65
−15
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Add parseOllamaParametersToJSON function to properly parse model parameters - Enhance context window detection by checking model_info keys ending with '.context_length' - Prioritize environment variable OLLAMA_CONTEXT_LENGTH over model defaults - Update default context window from 200K to 128K to match common Ollama models - Fix maxTokens assignment to use defined context window This change ensures more accurate context window detection for Ollama models and provides better handling of model parameters.
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This change ensures more accurate context window detection for Ollama models and provides better handling of model parameters.
Context
Previous effort didn't quite work with models that had a large default context (like qwen3:8b) of 256K.
Implementation
New method does more careful parsing of the parameters field returned by Ollama. If thats set, it uses that.
Other wise, it uses the default context that is set in the model architecture section (
qwen3.context_windoworgemma3._context_window).If the env var is sat, that overrides all.
Lastly, it uses the default hardcoded, which is changed to be 128K, matching the top 3 models on ollama library.
Get in Touch
mcowger