What VPS do you need for AI? API calls or a model on the server

Two needs, two server sizes

Most applications that "use AI" (assistants, agents, summaries, data extraction, document search) call a model hosted by its provider, through an API. The computing happens there; your server builds the request, waits for the answer and processes it. It is sized like any web application: by its language, its database and its number of concurrent users.

The other case is a model running on the server itself, with Ollama, llama.cpp or a Python library. The reasons are familiar: data that stays on your own infrastructure, a fixed cost whatever the number of requests, background processing of large volumes, a specialised model. Here the whole model has to fit in memory, and it sets the size.

Through an API: size the application, not the model

A call to a hosted model takes from one to several tens of seconds, especially when the answer is streamed. During that time your server computes almost nothing: it waits. What matters is its ability to keep many requests waiting without blocking the rest:

• An asynchronous server: Node.js, FastAPI with Uvicorn, or async Gunicorn workers. A synchronous worker waiting 20 seconds for an answer is a worker blocked for every other visitor.
• A job queue for long tasks (summarising a document, analysing a batch): the visitor’s request returns straight away and a worker does the job in the background.
• Timeouts set on the reverse proxy: by default, Nginx drops a response that stays silent for more than 60 seconds (proxy_read_timeout).

The memory to plan for is that of the application and its database: 4 GB covers a Python or Node.js app with PostgreSQL at launch. A common setup, a Next.js front end, a FastAPI backend and a PostgreSQL database calling a model through an API, fits in 6 to 8 GB, front-end build included.

Embeddings and vector search: memory for vectors

An app that searches your documents (RAG, retrieval-augmented generation) turns each passage into a vector, then finds the ones closest to a question. Two things to size:

• The embedding model, if it runs on your server: common models weigh 100 MB to 1.5 GB and run on CPU without trouble. all-MiniLM-L6-v2, for example, takes under 100 MB.
• The vectors: in pgvector, a vector takes 4 bytes per dimension, plus 8. One million passages at 1,536 dimensions come to about 6 GB of data, plus the HNSW index that speeds up search. For fast queries, the index and the vectors you query need to fit in memory.

pgvector’s halfvec type stores each dimension in 2 bytes instead of 4: half the memory, with search quality that stays very close.

A model on the server: memory comes first

A language model is a file of weights, loaded entirely into memory to answer. Its size depends on the number of parameters and on quantisation, the precision each weight is stored at. The GGUF format used by llama.cpp and Ollama offers quantisations from 2 to 16 bits; Q4_K_M, at around 4 bits, is the usual trade-off between size and quality.

On top of the weights come two more items:

• Context (the KV cache): every token of the current conversation takes memory. For Llama 3.1 8B, allow about 128 KB per token at 16-bit precision, so 1 GB for an 8,000-token context, and as much again for every conversation handled in parallel.
• The rest of the server: operating system, application, database, as for any web app.

Worked example. Llama 3.1 8B at 4 bits, two concurrent 8,000-token conversations and a small application: 5 GB of model, 2 GB of context, 2 GB for the app and the OS, so 9 GB, and about 12 GB with 25% headroom.

On CPU: what sets the speed

To produce each token, the model reads all of its weights from memory. On CPU, generation speed is therefore bounded by memory bandwidth more than by core count: a 5 GB model on a machine that reads 25 GB per second tops out around 5 tokens per second, before any computation is counted.

Two phases behave differently:

• Reading the prompt (prompt processing) is compute-heavy: more vCPUs make it faster, which matters for long documents.
• Generating the answer depends mostly on memory bandwidth and model size: a model half the size answers about twice as fast.

In practice, on a few vCPUs, a 7 to 8 billion parameter model at 4 bits produces a few tokens per second. That is comfortable for background work (sorting messages, extracting fields from documents, summarising overnight what came in during the day) and for an internal assistant. Interactive use at scale, and models with tens of billions of parameters, run on GPUs.

Measure to be sure: llama-bench, shipped with llama.cpp, reports the speed of both phases on your server, for your model.

llama-bench -m model-Q4_K_M.gguf -p 512 -n 128

Disk: models, versions, vectors

A model takes roughly as much disk as memory: 5 GB for an 8 billion parameter model at 4 bits. Leave room for two or three models or versions, vector indexes, source documents and the database. An NVMe SSD loads a model in seconds when the service starts, which counts every time it restarts.

As a rule of thumb: 50 GB for an app that calls a model through an API, 80 to 120 GB once a model and its documents live on the server.

Setting it up safely

With Ollama, getting started takes two commands:

curl -fsSL https://ollama.com/install.sh | sh
ollama run llama3.1:8b

By default Ollama listens on 127.0.0.1:11434, which means only from the server itself. Keep it that way: its API has no authentication, and an instance left open to the internet serves anyone, on your resources. Your application, on the same server, talks to it locally; if another server needs it, go through a proxy that authenticates requests, or a private network.

Then set the context size (num_ctx) to what your requests actually need: it is what makes memory grow, conversation by conversation.