<type>documentation</type>
<domain>ai</domain>
<id>local</id>
<title>Running AI Models Locally</title>
<date>$Date: 2006-05-24 22:53:07 -0400 (Wed, 24 May 2006) $</date>
<version>$Revision: 13098 $</version>
<product>pro+</product>

It is also possible to run AI models locally, using `ollama <https://ollama.com/>`__ or similar
solution. In this case, you download the model and all your requests and code are sent to the
locally running model and thus are never uploaded to any third party AI provider.

.. note::
  **Note:** Although this is one way to use AI in Wing, be warned that locally run models tend
  to produce results that are considerably lower in quality and usability than those you 
  can obtain from other AI providers.  This may change over time, as more powerful models can
  be run locally, but for now we recommend against trying to do serious work with locally 
  run models.

**Ollama**

To use `ollama <https://ollama.com/>`__ to run models locally, all you need to do is:

* Download and install ollama
* Run ``ollama pull <model>`` with ``<model>`` replaced with the model you want to try.
* Run ``ollama run <model>``
* Configure an AI provider in Wing, setting ``API Host`` to ``Use Base URL`` with url
  ``http://localhost:11434/v1/``.  Set ``Key`` to ``Use Key`` and enter ``unused`` (the
  key is required but ignored), enter the ``Model`` identifier you selected, and set
  costs to 0.0.
* Select this new provider configuration as the current provider.
* In ``AI Chat`` select ``New Conversation`` from its ``Options`` menu
* Try a request like "Can you write a Python hello world?"

.. comment:

  [This doesn't work since apparently it's not really OpenAI compatible.  Saving in case
  this can be fixed later.]
  
  **text-generation-webui**
  
  To use `text-generation-webui <https://github.com/oobabooga/text-generation-webui>`__ to run models
  locally:
  
  * Download and install text-generation-webui
  * Point your browser to http://localhost:7860/
  * Under ``Models`` download a model
  * Quit and restart the ``start_*`` script with the ``--api`` command line argument
  * Configure an AI provider in Wing, setting ``API Host`` to ``Use Base URL`` with url
    ``http://localhost:5000/v1/``.  Set ``Key`` to ``Use Key`` and enter ``unused`` (the
    key is required but ignored), enter the ``Model`` identifier for the model you
    downloaded, and set costs to 0.0.
  * Select this new provider configuration as the current provider.
  * In ``AI Chat`` select ``New Conversation`` from its ``Options`` menu
  * Try a request like "Can you write a Python hello world?"

