MerchantryTidbits

llm-inference / library

Llama Cpp Python

Capability: Llama Cpp Python

Use it when

  • run GGUF models in-process via llama.cpp
  • CPU/GPU local inference without a server

What it solves

cost-per-call-too-high

Install

pip install llama-cpp-python

Invoke

# see https://github.com/abetlen/llama-cpp-python
pip install llama-cpp-python

Alternatives

No reviewed alternatives recorded yet.