Dockerised vLLM OpenAI endpoint
Overview
I wrote this to answer one question: what does it take to replace a hosted AI provider with my own GPU?
The answer turned out to be small. An official vLLM image, a few settings that I spelled out, and a compose file for the GPU. Then a short script with the normal OpenAI client, pointed at my own machine.
I tested with the official SDK on purpose. Claims of compatibility are cheap, and a normal client that just works is the best proof.
I do not use this setup today. My platform now uses hosted models behind a gateway, and a local model only for embeddings. The conclusions from this example still shaped that design.
Problem
I wanted to know what it really takes to swap a hosted AI provider for my own GPU. Many tools say they are compatible with the OpenAI API. I wanted to test that claim with a small, working example.
Approach
- I started from the official vLLM image for the OpenAI compatible server.
- I wrote down the settings that matter on a small GPU, so nothing depends on defaults.
- I put the GPU and memory rules in a compose file, because a Dockerfile cannot express them.
- I tested it with the official OpenAI Python client and not with a custom client.
Architecture and design decisions
- The Dockerfile uses the official image, exposes port 8000 and sets the model as the start command.
- The model is Meta-Llama-3-8B-Instruct, run in half precision with a context window of 2048 tokens to keep memory low. It also has 2 GB of swap space and a fixed API key.
- The compose file adds a GPU reservation, a 4 GB memory reservation, the visible GPU setting, a restart policy and an optional Hugging Face token for gated models.
- A short Python script uses the normal OpenAI client with the base URL pointed at localhost:8000/v1. It uses the same model name and key and has no vLLM code in it.
Reliability and performance
- The model name and context length are fixed in the setup, so they cannot change by accident.
- The API key is kept in the service and not in the client.
- The container has a restart policy.
Outcomes
- A local Llama 3 8B model answers chat requests the same way a hosted API would.
- The whole example is two files and a script of about twenty lines.
- The test with the official SDK showed that the endpoint really is compatible.
- The lessons carried into my current platform: pin the model name and context length, keep the key in the service and always test with the official SDK.
Known gaps
- This was a learning exercise. I do not run it today, because my platform now uses hosted models behind a gateway plus a local embedding model.
- I have no numbers for speed, memory use or how many requests it handled.
- I do not say which GPU I used.
- I did not test it with more than one user or under load.
- I did not compare the answer quality of the local model with a hosted one.