I am hoping to see a way to add in Custom LLM (Models) via API. Just add in a URL, Token and then refresh to find the active models. Then select the appropriate model. It would really help when I want to use a specific model that is not on one of the selections.
Got it! We have currently added OpenAi, Anthropic, Perplexity and OpenRouter. It covers almost 90% of the providers and models. Which ones are you looking for?
Hello Advait, Thanks! for a quick response. I am not looking for one specific provider to be added in. I am looking for a way to add in my own, just a custom API provider. Require (assume) the provider should support OpenAI API format. One example - I have a AI Server that runs on my lab network, but I also tunnel it out - with secure access_key. Also, might be a use case - if someone gets a Deepseek API Key (cause they are cheap inference), so easy for them to add in a model, URL, and get running. Typically - when you add in the first 2 items and query /models API endpoint you get a list of all the active models. Then you can select one. Similar to when OpenAI account is validated, it exposes all the models so you can select 1. HTH. Thanks!! & keep up the good work. I bought my license just a bit ago. e.g. # Dan-AI-Cluster # LITELLM_API_KEY=your_optional_api_key_here # LITELLM_API_URL=https://api.privatesite.com/v1 # LITELLM_MODEL=model-name # DeepSeek # DEEPSEEK_API_KEY=your_deepseek_api_key_here # DEEPSEEK_API_URL=https://api.deepseek.com/v1 # DEEPSEEK_MODEL=deepseek-chat
Advait, Thanks!! a ton. The update you made, works perfectly. I haven't tested it extensively but just did a little bit of AI text clean up, after switching over to using Custom AI Endpoint. I was able to setup my network inference cluster. I also was able to test with a few models I serve locally through LiteLLM proxy. I am new to MacOS in general, catching on slowly to some of the uniqueness in the OS. Thanks for helping make the transition smoother. Regards,