MAX Inference Engine

1 · What MAX Is

The platform in one sentence

MAX (Modular Accelerated Xecution) is Modular's AI inference platform. You give it a supported model — commonly from Hugging Face — and it compiles, optimises, and serves it with an OpenAI-compatible HTTP endpoint across NVIDIA, AMD, Apple, and CPU targets, without changing your application code.

Three components

MAX is made of three interlocking layers:

• MAX Serve — the serving layer. A single max serve command downloads a supported model, compiles it for your hardware, and exposes an OpenAI-compatible REST endpoint. Existing OpenAI clients can generally point at the local endpoint.
• MAX Engine — the inference runtime. The core execution engine that loads a model and runs forward passes programmatically via a Python API. Used when you need tighter integration than an HTTP endpoint provides.
• MAX Graph API + Mojo kernels — the extensibility layer. Define custom computation graphs and write hardware-specific kernels in Mojo. This is how you go below what a standard model offers — custom attention variants, quantisation schemes, or entirely novel operators.

Why it exists

Inference is pulled by two forces: models keep getting larger, and the hardware landscape keeps fragmenting. CUDA remains the performance leader on NVIDIA, but AMD data-centre GPUs are now credible alternatives, Apple Silicon runs many models locally, and edge devices span every architecture imaginable. Maintaining a separate serving stack per backend is expensive. MAX's pitch is one stack across all of them — write and deploy once, and let the compiler retarget it per machine.

2 · Installing MAX

Recommended: pixi

pixi gives you a reproducible Conda environment with MAX pinned. This is the path Modular officially recommends for production setups:

# Install pixi
curl -fsSL https://pixi.sh/install.sh | sh

# Create a new project pulling from the Modular channel
pixi init quickstart \
  -c https://conda.modular.com/max/ \
  -c conda-forge
cd quickstart

pixi add max-all      # MAX Serve, benchmark, Mojo, and all optional dependencies
pixi shell            # activate the environment

Alternative: uv

If you already use uv for Python project management, install the current MAX extras:

uv init quickstart && cd quickstart
uv venv && source .venv/bin/activate
uv add "max[all]"

Set your Hugging Face token

MAX downloads gated models from Hugging Face. Export your token before serving:

export HF_TOKEN="hf_your_token_here"

3 · Serving Your First Model

Start the endpoint with max serve

One command downloads the model, compiles it for your GPU, and starts the server:

max serve --model google/gemma-3-4b-it

MAX will log progress as it fetches weights, runs compilation, and binds the port:

Fetching model weights...
Compiling for NVIDIA H100 (CUDA 12.x)...
Server ready at http://localhost:8000

Query the endpoint

The endpoint is OpenAI API-compatible. Any code already targeting OpenAI works unchanged — just swap the base URL:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="EMPTY"    # MAX does not require an API key locally
)

completion = client.chat.completions.create(
    model="google/gemma-3-4b-it",
    messages=[{"role": "user", "content": "Explain SIMD in one paragraph."}]
)
print(completion.choices[0].message.content)

Benchmarking

MAX ships a built-in benchmarking command to measure throughput and latency against your running endpoint:

max benchmark \
  --model google/gemma-3-4b-it \
  --backend modular \
  --num-prompts 200 \
  --max-concurrency 32

Serving a larger model with tensor parallelism

For models that don't fit on a single GPU, MAX shards them across devices with tensor parallelism. Pass --num-gpus to set how many it uses:

max serve \
  --model meta-llama/Llama-3.3-70B-Instruct \
  --num-gpus 4

4 · MAX Engine Python API

When to use the Engine API vs MAX Serve

MAX Serve is a standalone HTTP server — right for production deployments and microservices. The Engine API is a Python library you embed directly in your application — right for batch pipelines, custom pre/post-processing, or when you want inference in-process without an HTTP hop.

Loading and running a model

from max.engine import InferenceSession
from transformers import AutoTokenizer

MODEL = "meta-llama/Llama-3.1-8B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(MODEL)

# Load the model — MAX compiles it on first load
session = InferenceSession()
model = session.load(MODEL)

# Run a forward pass
input_ids = tokenizer.encode("Hello, world!")
outputs = model.execute(input_ids=input_ids)
print(tokenizer.decode(outputs["logits"].argmax(-1)))

Custom MAX Graph operators in Mojo

The MAX Graph API lets you define a computation graph and drop in Mojo-written custom ops at any node. The graph compiler fuses adjacent ops and selects the optimal hardware path automatically:

# Python side — define the graph
from max.graph import Graph, TensorType, ops

with Graph("my_model", input_types=[TensorType(dtype, shape)]) as graph:
    x = graph.inputs[0]
    y = ops.matmul(x, weight)        # built-in op
    z = ops.custom["my_mojo_op"](y)  # Mojo kernel wired in
    graph.output(z)

# Mojo side — the custom kernel
@register_mojo_op("my_mojo_op")
def my_op(x: Tensor[DType.float32]) -> Tensor[DType.float32]:
    # runs on whatever hardware MAX targets
    ...

5 · Vulkan — Cross-Platform GPU Compute

What Vulkan is

Vulkan is an open, vendor-neutral GPU API maintained by the Khronos Group — the same standards body behind OpenGL and WebGL. Where CUDA is an NVIDIA proprietary SDK, Vulkan is an industry standard implemented by every major GPU vendor: NVIDIA, AMD, Intel, ARM, Qualcomm, Apple (via the MoltenVK translation layer).

Vulkan was designed primarily for graphics (rasterisation, ray tracing), but it exposes a compute pipeline — arbitrary parallel programs that run on the GPU's shader cores, with no graphics output required. This compute path is what matters for AI inference.

Vulkan vs CUDA — key differences

SPIR-V — Vulkan's portable bytecode

Vulkan compute kernels are distributed as SPIR-V — a binary intermediate representation, analogous to LLVM IR or CUDA PTX. You write a shader in GLSL or HLSL, compile it to SPIR-V offline, and ship the SPIR-V. The GPU driver JITs it to native machine code on the target device. This is how a single binary runs on an AMD RDNA card and an Intel Arc card without recompilation.

# Compile a GLSL compute shader to SPIR-V
glslc matmul.comp -o matmul.spv

# The .spv file is what you distribute — hardware-agnostic

Vulkan for AI inference

Several production inference engines use Vulkan compute as their portability path:

• llama.cpp — uses Vulkan for GPU acceleration on any Vulkan-capable card, including consumer AMD and Intel GPUs where ROCm is unavailable or unsupported.
• GGML — the tensor library underlying llama.cpp; its Vulkan backend runs on mobile, Windows, and Linux without CUDA.
• WebGPU — the browser GPU API. It is an abstraction over whatever the host provides: Vulkan on Linux and Android, Metal on Apple, D3D12 on Windows. Inference in-browser via transformers.js runs through this path.

6 · Deploying on Non-CUDA Hardware

How MAX abstracts hardware backends

MAX does not require you to select a backend manually. When you run max serve, the MAX compiler inspects available hardware and picks the optimal execution path:

• NVIDIA GPU present → compiles to CUDA PTX, runs via the CUDA runtime.
• AMD GPU present → compiles to AMDGPU ISA via ROCm / HIP. Supported data-centre cards: MI300X, MI325X, MI355X.
• Apple Silicon → compiles to Metal shaders and runs via the Metal compute API.
• CPU fallback → lowered through LLVM IR to native machine code; slower but functional on any machine.

MAX reaches AMD through ROCm and Apple through Metal directly — Vulkan is not one of its documented inference backends, and max serve will never select it. Vulkan matters here as the portability layer for engines outside MAX, such as llama.cpp, which is how you cover hardware MAX does not target at all.

Deploying on AMD without a CUDA box

# On a machine with AMD MI300X and the supported AMD driver installed
uv venv && source .venv/bin/activate
uv pip install "max[serve]"

export HF_TOKEN="hf_..."
max serve --model meta-llama/Llama-3.3-70B-Instruct

# MAX detects the AMD GPU automatically — no --device flag needed
# Compiling for AMD gfx942 (MI300X)...

Deploying on a Vulkan-only device (edge / consumer GPU)

For hardware not covered by MAX (for example, some Intel Arc, consumer AMD, or edge devices), use a separate Vulkan-backed inference engine such as llama.cpp:

# Build llama.cpp with Vulkan support
cmake -B build -DGGML_VULKAN=1
cmake --build build --config Release -j

# Run inference via Vulkan on any Vulkan-capable GPU
./build/bin/llama-cli \
  -m Llama-3.1-8B-Instruct.Q4_K_M.gguf \
  --gpu-layers 99 \
  -p "Explain what Vulkan is"

Combining MAX Serve with a Vulkan pre-processing sidecar

A practical production pattern: run MAX Serve for the LLM inference core (NVIDIA or AMD), and a lightweight Vulkan-accelerated sidecar for pre-processing — image tokenisation with CLIP, audio feature extraction. You are wiring two processes together; MAX has no built-in Vulkan mode.

# MAX handles the LLM
max serve --model meta-llama/Llama-3.3-70B-Instruct &

# Vulkan sidecar handles vision tokenisation on an Intel Arc GPU
./clip-encoder --vulkan --input /tmp/image.jpg --output /tmp/tokens.bin

# Application glues them together via HTTP / shared memory

7 · vLLM + CUDA vs MAX

What vLLM is

vLLM is an open-source LLM inference and serving library from UC Berkeley. Its core innovation is PagedAttention — a KV-cache management algorithm inspired by OS virtual memory paging that dramatically reduces memory fragmentation during concurrent inference. vLLM is the most widely deployed open-source LLM serving solution in production today.

Installation and serving (vLLM)

# Install (NVIDIA GPU assumed)
uv venv --python 3.12 --seed
source .venv/bin/activate
uv pip install vllm --torch-backend=auto

# Serve a model — blocks in the foreground
vllm serve Qwen/Qwen2.5-7B-Instruct

Then, from a second terminal, query it with the same OpenAI-compatible client MAX uses:

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
completion = client.chat.completions.create(
    model="Qwen/Qwen2.5-7B-Instruct",
    messages=[{"role": "user", "content": "Hello"}]
)

Head-to-head comparison

When to choose vLLM

• Your entire fleet is NVIDIA and you want the most battle-tested, community-supported solution.
• You need PagedAttention's specific memory behaviour — it is still the reference implementation.
• You are writing custom CUDA kernels with Triton and want tight integration.
• You need a large catalogue of advanced features: speculative decoding, LoRA, multi-modal models — vLLM has shipped these longer.

When to choose MAX

• Your hardware fleet is mixed (NVIDIA + AMD + Apple) and you want one serving stack rather than three.
• You are writing inference kernels in Mojo and want them to target all supported hardware without maintaining a CUDA C++ version.
• You want to evaluate AMD MI300X as a cost alternative to H100 — MAX is purpose-built for this.
• You are deploying on non-CUDA hardware — AMD data-centre GPUs or Apple Silicon — where vLLM support is still experimental.

Further reading

• MAX release notes — release changes and breaking changes
• MAX Get Started — official quickstart
• MAX packages and compatibility — installation and hardware requirements
• vLLM Quickstart — serving your first model with vLLM
• Khronos Vulkan Samples — compute shader examples
• llama.cpp — Vulkan-backed inference for consumer and edge hardware
• modular/modular on GitHub — MAX source and examples