Experimenting with LLMs

Using the llama-cpp inference runtime with smaller GGUF transformer models on old dusty hardware.

Context

• Lenovo Desktop
• Intel(R) Core(TM) i3-6100T CPU @ 3.20GHz
• 3.8GB RAM
• Debian 12

Install llama-cpp

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp

Build without GPU

mkdir build
cd build
cmake .. -DLLAMA_CUBLAS=OFF

Get model that will work with low resources

• tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf
• Model weights ~0.8 GB
• KV cache (1024 ctx) ~0.5–0.8 GB
• Overhead ~0.3 GB
• Total ~1.8–2.2 GB

Start interface

./build/bin/llama-cli \
  -m models/tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf \
  -c 1024 \
  -n 256 \
  -t 2

Prompt & Output

Function works but examples are wrong.