uncloseai.

How We Run Inference

How we run inference

This section is optional. This is only if you wanted to try to contribute idle GPU time to the project or if you wanted to reproduce everything in your own cluster.

vLLM Setup — Hermes (3090)

We use vLLM to run models, generally with full f16 safetensors. We make sure to use a virtualenv to hold the dependencies.

Note: For Hermes, we use an FP8 quant by adamo1139 (adamo1139/Hermes-3-Llama-3.1-8B-FP8-Dynamic), which is optimized for 4090 and 3090 GPUs.

Tool Calling: The --enable-auto-tool-choice and --tool-call-parser hermes flags are required for Hermes 3 to support function calling via the OpenAI-compatible API. Without these flags, tool calling requests will fail.

GPU Memory: The --max-num-seqs 32 and --gpu-memory-utilization 0.85 flags cap concurrent sequences and leave VRAM headroom. Without these, vLLM defaults to 256 concurrent sequences, which OOMs during sampler warmup on a 24 GiB card at 82k context (error: torch.OutOfMemoryError: CUDA out of memory ... when warming up sampler with 256 dummy requests). 32 is a sensible concurrency for an 8B model on a single 3090.

Stand up a replica cluster on a new domain.

sudo apt-get install gcc python3.12-dev python3.12-venv
cd ~
python3 -m venv env
source env/bin/activate
pip install vllm
python -m vllm.entrypoints.openai.api_server \
    --model adamo1139/Hermes-3-Llama-3.1-8B-FP8-Dynamic \
    --host 0.0.0.0 --port 18888 \
    --max-model-len 82000 \
    --max-num-seqs 32 \
    --gpu-memory-utilization 0.85 \
    --enable-auto-tool-choice \
    --tool-call-parser hermes

vLLM Setup — Qwen (4090)

TL;DR: On a single RTX 3090 or 4090 (24 GiB VRAM), you can run the INT4 quant of Qwen3.6-27B with fp8 KV cache to fit a 64k token context window.

Qwen3.6-27B is the multimodal flagship from the Qwen team — vision + text in a single model (capable of OCR, document parsing, screenshot reading, charts). We run it on a single RTX 4090 (24 GiB VRAM) via vLLM.

Model choice: We serve Lorbus/Qwen3.6-27B-int4-AutoRound — 17.45 GiB weights including the MTP draft layer and vision encoder. The AWQ alternative (cyankiwi/Qwen3.6-27B-AWQ-INT4) is heavier at 19.04 GiB on disk; AutoRound is the lighter pick for a 24 GiB card.

fp8 KV cache: --kv-cache-dtype fp8 halves KV memory on Ada Lovelace (sm_89, RTX 4090). Critical for fitting 64k context with a 27B model on a 24 GiB card. Quality cost is near zero for chat — KV in fp8 alongside already-INT4 weights stays far below the quant noise floor.

GPU memory budget: --gpu-memory-utilization 0.95 leaves just enough KV pool (~2.3 GiB) for one active 64k sequence. 0.92 reports only 1.62 GiB pool because vit (vision transformer) warmup eats more activation memory than expected. 0.95 works because vLLM accounts for actual peak measured during warmup.

Tool calling: Lorbus's chat template emits the qwen3_coder tool format (<tool_call><function=...><parameter=...>). Use --tool-call-parser qwen3_coder so vLLM parses these into proper OpenAI tool_calls JSON. Without it, agent frameworks like hermes-agent and goose can't drive tool use against the endpoint.

Context window: --max-model-len 65536 matches the 64k minimum that hermes-agent and similar frameworks require. vLLM mixes prompt lengths in continuous batching — short chat requests (8k each, ~0.29 GiB KV with fp8) still squeeze alongside one long 64k sequence (2.3 GiB).

Salt state + pillar (foxhop-states, foxhop-pillar) drive the deployment; the systemd unit below is the rendered ExecStart.

sudo apt-get install gcc python3.12-dev python3.12-venv ninja-build
cd ~
python3 -m venv env
source env/bin/activate
pip install vllm
vllm serve Lorbus/Qwen3.6-27B-int4-AutoRound \
    --host 0.0.0.0 --port 18888 \
    --max-model-len 65536 \
    --max-num-seqs 16 \
    --gpu-memory-utilization 0.95 \
    --kv-cache-dtype fp8 \
    --dtype auto \
    --quantization auto_round \
    --enable-prefix-caching \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_coder

Why ninja-build: FlashInfer (vLLM's CUDA sampling backend) JIT-compiles its top-k/top-p kernels on first request and shells out to ninja. The pip wheel doesn't bundle the binary, so without ninja-build installed the engine crashes at startup with FileNotFoundError: 'ninja'.

Proxy Setup

If you want to see how we setup the proxy, check out /etc/caddy/Caddyfile

hermes.ai.unturf.com {
    reverse_proxy <removed>:18888
    log {
        output file /var/log/caddy/hermes.ai.unturf.com.log {
            roll_size 50mb
            roll_keep 5
        }
    }
    tls {
        on_demand
    }
}

qwen.ai.unturf.com {
    reverse_proxy <removed>:18889
    log {
        output file /var/log/caddy/qwen.ai.unturf.com.log {
            roll_size 50mb
            roll_keep 5
        }
    }
    tls {
        on_demand
    }
}

speech.ai.unturf.com {
    reverse_proxy <removed>:8000
    log {
        output file /var/log/caddy/speech.ai.unturf.com.log {
            roll_size 50mb
            roll_keep 5
        }
    }
    tls {
        on_demand
    }
}

Model Discovery

vLLM provides an OpenAI-compatible API with built-in documentation. You can discover available models and explore the full API using these endpoints:

Swagger Documentation

Access the interactive API docs at the /docs endpoint:

hermes.ai.unturf.com/docs - Hermes endpoint Swagger docs

qwen.ai.unturf.com/docs - Qwen endpoint Swagger docs

The Swagger UI lets you explore all available endpoints, see request/response schemas, and test API calls directly in your browser.

Model Discovery

To get the current model ID being hosted, query the /v1/models endpoint:

Or via curl:

curl https://hermes.ai.unturf.com/v1/models
curl https://qwen.ai.unturf.com/v1/models

Example response:

{
  "object": "list",
  "data": [
    {
      "id": "adamo1139/Hermes-3-Llama-3.1-8B-FP8-Dynamic",
      "object": "model",
      "created": 1735689600,
      "owned_by": "vllm",
      "root": "adamo1139/Hermes-3-Llama-3.1-8B-FP8-Dynamic",
      "max_model_len": 82000
    }
  ]
}

The id field contains the model name you should use in your API calls. The max_model_len field tells you the maximum context length supported.

Tip: Always query /v1/models programmatically rather than hardcoding model names. This ensures your code works even when models are updated or swapped.

Rate Limiting

Rate limiting is configured based on client IP address: 3 requests per second per IP per endpoint.

Next Steps

Ready to add text-to-speech to your application?

📖 Read the Text-to-Speech documentation →