From AI Exploration to Production Deployment

Master inference optimization on Jetson Thor with vLLM

Welcome! In this hands-on workshop, you’ll unlock truly high-performance, on-device generative AI using the new NVIDIA Jetson Thor .
You’ll start by unleashing Thor's full potential with a state-of-the-art 120B model, then step through practical optimizations -- FP8 , FP4 , and speculative decoding -- measuring speed vs. quality at each stage.

Workshop overview

🚀 Experience: Thor's Raw Power with 120B Intelligence

Exercise : Launch Your First 120B Model

Ready to experience Thor's raw power?
In this exercise, you'll launch OpenAI's GPT-OSS-120B model locally using vLLM. Follow the steps below to get your local AI powerhouse running 💪.

1️⃣ Starting vLLM container

Start running the vLLM container (provided by NVIDIA on NGC).
We mount some local (on host) directories for making models downloaded persist and re-using cache:

docker run --rm -it \
  --network host \
  --shm-size=16g \
  --ulimit memlock=-1 \
  --ulimit stack=67108864 \
  --runtime=nvidia \
  --name=vllm \
  -v $HOME/data/models/huggingface:/root/.cache/huggingface \
  -v $HOME/data/vllm_cache:/root/.cache/vllm \
  nvcr.io/nvidia/vllm:25.09-py3

Key mount points:

2️⃣ Set Tokenizer Encodings

Configure the required tokenizer files for GPT-OSS models:

mkdir /etc/encodings
wget https://openaipublic.blob.core.windows.net/encodings/cl100k_base.tiktoken -O /etc/encodings/cl100k_base.tiktoken
wget https://openaipublic.blob.core.windows.net/encodings/o200k_base.tiktoken -O /etc/encodings/o200k_base.tiktoken
export TIKTOKEN_ENCODINGS_BASE=/etc/encodings

3️⃣ Verify Pre-downloaded Model

Inside the container, check if the model is available:

# Check if model is available
ls -la /root/.cache/huggingface/hub/models--openai--gpt-oss-120b/
du -h /root/.cache/huggingface/hub/models--openai--gpt-oss-120b/
# Should show ~122GB - no download needed!

Even if you don't find the pre-downloaded models you can skip ahead to the next step. vLLM downloads the model automatically.

4️⃣ Launch vLLM Server

Start the vLLM inference server:

vllm serve openai/gpt-oss-120b

What happens next:

The vLLM serve command will take approximately 2.5 minutes to complete startup on Thor. Watch for the final "Application startup complete" message to know when it's ready!

Parse safetensors files: 100%|████████████████████| 15/15 [00:01<00:00, 14.70it/s]
Loading safetensors checkpoint shards: 100%|████████████████████| 15/15 [00:24<00:00,  1.64s/it]
INFO: Loading weights took 24.72 seconds

INFO: torch.compile takes 45.2s in total for 1 model(s)

INFO: Graph capturing finished in 21.0 secs

(APIServer pid=92) INFO:     Waiting for application startup.
(APIServer pid=92) INFO:     Application startup complete.

Test the API endpoints (Optional)

Test your vLLM server is working:

# Check available models
curl http://localhost:8000/v1/models

# Test chat completion
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai/gpt-oss-120b",
    "messages": [{"role": "user", "content": "Hello! Tell me about Jetson Thor."}],
    "max_tokens": 100
  }'

5️⃣ Launch Open WebUI

Start the web interface for easy interaction:

docker run -d \
  --network=host \
  -v ${HOME}/open-webui:/app/backend/data \
  -e OPENAI_API_BASE_URL=http://0.0.0.0:8000/v1 \
  --name open-webui \
  ghcr.io/open-webui/open-webui:main

Access the interface:

1. Open your browser to http://localhost:8080
2. Create an account (stored locally)
3. Start chatting with your local 120B model!

You → Open WebUI (Browser) → vLLM Server → GPT-OSS-120B → Response

6️⃣ Evaluate

Interact with OpenAI's gpt-oss-120b model!

This is your chance to evaluate the accuracy, generalizability, and the performance of the model.
Use Open WebUI to test different scenarios and see how Thor handles this massive 120B parameter model.

# Inside container
vllm serve openai/gpt-oss-20b

7️⃣ Stop vLLM serving

You can continue to the next section "Optimize" by just stopping the current model serving by hitting Ctrl + C and keeping the container running.

However, even when just stopping the vLLM serving process , you need to clear GPU memory to make it available for the next model.

1. Stop the vLLM serving:

   Press Ctrl + C on the terminal where you ran vllm serve command.

2. ⚠️ CRITICAL: Clear GPU memory cache (Workaround)

   Due to the vLLM limitation, run this command on the HOST (outside the container):

   sudo sysctl -w vm.drop_caches=3
   

3. Verify memory cleared:

   jtop
   # GPU memory should drop to baseline (~3-6GB)
   

🔧 Optimize: Precision Engineering (FP16 → FP8 → FP4)

Now that you've experienced Thor's raw power with the 120B model, it's time to dive into the real engineering work of model deployment optimization.
In this section, we'll systematically explore how to balance performance vs. quality through precision engineering—taking a production-ready model through FP16 → FP8 → FP4 quantization while measuring the impact at each step.

We'll use Llama-3.1-8B-Instruct (FP16) as our baseline.

1️⃣ Test FP16 model

1.0 Check model cache and size

First, let's examine the FP16 model to understand its memory footprint:

# Check the cached model location and size
ls -la $HOME/data/models/huggingface/hub/models--meta-llama--Llama-3.1-8B-Instruct/

# Get detailed size breakdown
du -h $HOME/data/models/huggingface/hub/models--meta-llama--Llama-3.1-8B-Instruct/

# Check individual model weight files (stored as hash-named blobs)
ls -lh $HOME/data/models/huggingface/hub/models--meta-llama--Llama-3.1-8B-Instruct/blobs/

# Alternative: Check the actual model files in snapshots directory
find $HOME/data/models/huggingface/hub/models--meta-llama--Llama-3.1-8B-Instruct/snapshots/ \
  -name "*.safetensors" -exec ls -lh {} \;

Expected output:

• Total model size : ~30GB (15GB actual + 15GB in snapshots - HuggingFace cache structure)
• Actual model weights : ~15GB in FP16 precision
• 4 safetensors files : Each ~4GB containing the neural network weights (stored as hash-named blobs)
• Memory bandwidth impact : Every token generation requires accessing these 15GB of weights

1.1 Serve

vllm serve meta-llama/Llama-3.1-8B-Instruct

Once loaded, select the model in Open WebUI.

1.2 Baseline prompt & measurements

Prompt (copy/paste):

Write a 5-sentence paragraph explaining the main benefit of using Jetson Thor for an autonomous robotics developer.

Observe:

• Time-to-First-Token (TTFT) — perceived latency
• Tokens/sec — throughput (use Open WebUI stats or API logs)
• Answer quality — coherence, accuracy, task fit

curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "meta-llama/Llama-3.1-8B-Instruct",
        "messages": [
            {
                "role": "user",
                "content": "Write a 5-sentence paragraph explaining the main benefit of using Jetson Thor for an autonomous robotics developer."
            }
        ],
        "max_tokens": 256,
        "temperature": 0.7
      }'

2️⃣ FP8 Quantization

FP8 reduces memory bandwidth/footprint and often matches FP16 quality for many tasks.

vllm serve nvidia/Llama-3.1-8B-Instruct-FP8

Select this FP8 variant in Open WebUI and repeat the same prompt. Compare TTFT , tokens/sec , and answer quality vs. FP16.

3️⃣ FP4 Quantization

Let's push further!
FP4 halves memory again vs. FP8 and is much faster , but may introduce noticeable quality drift (hallucinations, repetition).

3.1 Relaunch in FP4

vllm serve nvidia/Llama-3.1-8B-Instruct-FP4

3.2 Evaluate FP4 perf and accuracy

Run the same prompt and evaluate:

• 🚀 Speed : should be visibly faster than FP16/FP8 both in terms of TTFT and generation speed.

• 🎯 Quality : check fidelity to the prompt and coherence

  The Quality Question

  Now, read the answer very carefully. Is it still as high-quality? Does it answer the prompt accurately? Do you see any strange wording, repetition, or nonsensical statements? This is the trade-off in action, and you are now experiencing the core challenge of on-device AI optimization.

# Using AutoGPTQ to quantize your own model
from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig

quantize_config = BaseQuantizeConfig(
    bits=4,  # FP4 quantization
    group_size=128,
    desc_act=False,
)

model = AutoGPTQForCausalLM.from_pretrained(
    "your-model-name",
    quantize_config
)
model.quantize(calibration_dataset)
model.save_quantized("./quantized-model")

4️⃣ Speculative Decoding

In Part 3, we pushed our model to FP4. We got a massive boost in speed and a huge reduction in memory , but we also hit the trade-off: the quality can start to degrade .

So, the next logical question is: Can we get even faster than our FP4 model, but without sacrificing any more quality?

The answer is yes. We can do this by using Speculative Decoding .

4.1 Relaunch with Speculative Decoding

Let's stop the last server. This time, our command is a bit more complex. We're telling vLLM to serve our FP4 model, but to also load a specific "draft" model (EAGLE3) to help it.

# Stop current FP4 serving (Ctrl+C)
# Then launch with speculative decoding:
vllm serve nvidia/Llama-3.1-8B-Instruct-FP4 \
  --trust_remote_code \
  --speculative-config '{"method":"eagle3","model":"yuhuili/EAGLE3-LLaMA3.1-Instruct-8B","num_speculative_tokens":5}'

4.2 Test Performance and Quality

Once it's loaded, go back to Open WebUI . The model name will be the same as before ( nvidia/Llama-3.1-8B-Instruct-FP4 ), but it's now running with its new speculative assistant .

Now let's prompt it again and pay very close attention to these two things:

1. 🚀 Generation Speed : Look at the throughput. The words should appear to fly onto the screen . This should be the fastest version we've seen today .

   Performance Recap: The Complete Optimization Journey

   🎯 From 10.7 to 25.6+ tokens/second - A 2.4x Performance Breakthrough!

   Configuration	Memory	Generation (Short)	Generation (Long)	vs FP16	Best Case
   FP16 (Baseline)	14.99 GiB	10.7 tok/s	~10.7 tok/s	Baseline	Baseline
   FP8	8.49 GiB	14.2 tok/s	~14.2 tok/s	+33%	+33%
   FP4	6.07 GiB	19.1 tok/s	~19.1 tok/s	+78%	+78%
   FP4 + Speculative	6.86 GiB	14.3 tok/s	25.6 tok/s	+139%	+139%

   🔍 Key Engineering Insights:

   • Memory bandwidth bottleneck confirmed : Thor's performance scales directly with memory reduction
   • Context matters for speculation : Short responses show modest gains, long responses show dramatic improvements
   • Speculative decoding sweet spot : Most effective for structured, longer content generation
   • Production decision framework : Choose based on your specific use case and content length

   🚀 Ultimate Result : 25.6 tokens/second for long content - nearly 2.4x faster than our FP16 baseline!

2. 🎯 Quality : Compare this answer very closely to the one you got in Part 3 (FP4). The quality should be essentially unchanged . This proves that speculative decoding is purely a performance optimization and does not change the model's final output.

🚑 Troubleshooting

# Run this command on the HOST after stopping vLLM container
sudo sysctl -w vm.drop_caches=3

# More aggressive memory clearing
sudo systemctl restart docker
# Or as last resort:
sudo reboot

cat /etc/docker/daemon.json

{
    "runtimes": {
        "nvidia": {
            "args": [],
            "path": "nvidia-container-runtime"
        }
    }
}
// ❌ Missing "default-runtime": "nvidia" !

sudo nano /etc/docker/daemon.json

{
    "runtimes": {
        "nvidia": {
            "args": [],
            "path": "nvidia-container-runtime"
        }
    },
    "default-runtime": "nvidia"
}

# Restart Docker daemon
sudo systemctl restart docker

# Verify Docker is running
sudo systemctl status docker

# Test NVIDIA runtime (Thor-compatible CUDA 13)
docker run --rm --runtime=nvidia nvcr.io/nvidia/cuda:13.0.0-runtime-ubuntu24.04 nvidia-smi

# Restart your vLLM container
docker stop vllm  # if running
# Then relaunch with the corrected Docker configuration

sudo reboot

docker run --rm --runtime=nvidia nvcr.io/nvidia/cuda:13.0.0-runtime-ubuntu24.04 nvidia-smi

ValueError: Free memory on device (14.45/122.82 GiB) on startup is less than
desired GPU memory utilization (0.7, 85.98 GiB)

# Check current GPU memory usage
jtop
# Expected baseline: ~3-6GB system usage
# Problem: 25GB+ or 100GB+ unexplained usage

# 1. Stop all containers
docker stop $(docker ps -q) 2>/dev/null
docker rm $(docker ps -aq) 2>/dev/null

# 2. Restart Docker daemon
sudo systemctl restart docker

# 3. Check if memory cleared
jtop

# 4. If memory still high (>10GB baseline), reboot system
sudo reboot

groups $USER  # Should include 'docker'

nvidia-smi
# Or test via Docker:
docker run --rm --runtime=nvidia nvcr.io/nvidia/cuda:13.0.0-runtime-ubuntu24.04 nvidia-smi

OSError: You are trying to access a gated repo.
Make sure to have access to it at https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct.
401 Client Error.
Cannot access gated repo for url https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct/resolve/main/config.json.
Access to model meta-llama/Llama-3.1-8B-Instruct is restricted. You must have access to it and be authenticated to access it.

# 1. Get HuggingFace token from https://huggingface.co/settings/tokens
# 2. Inside the container, authenticate:
pip install huggingface_hub
huggingface-cli login
# Enter your token when prompted

# 3. Or set environment variable:
export HUGGING_FACE_HUB_TOKEN="your_token_here"

What to do next

• Try a 70B FP4 model with speculative decoding and compare UX to 8B
• Add observability: latency histograms , p95 TTFT , tokens/sec

Appendix

Pre-Workshop Setup (For Organizers)

/home/jetson/data/                    177G total
├── models/huggingface/               176G (all workshop models)
│   ├── hub/                          167G
│   │   ├── models--openai--gpt-oss-120b                   122G (120B main model)
│   │   ├── models--meta-llama--Llama-3.1-8B-Instruct      30G (FP16 baseline)
│   │   ├── models--nvidia--Llama-3.1-8B-Instruct-FP8      8.5G (FP8 quantized)
│   │   ├── models--nvidia--Llama-3.1-8B-Instruct-FP4      5.7G (FP4 quantized)
│   │   └── models--yuhuili--EAGLE3-LLaMA3.1-Instruct-8B   811M (Speculative draft)
│   ├── xet/                          9.3G (HuggingFace cache)
│   └── modules/                      4KB
└── vllm_cache/torch_compile_cache/   628M (6 pre-warmed compilation caches)
    ├── 248df3d3e5/ (97M), 71b2b46784/ (78M), 92deee5901/ (141M)
    ├── b7ef42f749/ (92M), cd8c499fb2/ (116M), fce60ae060/ (107M)
    └── (optimized for all model variants - instant startup!)

Model Pre-download Process:

# 1. Download model once (takes 30-60 minutes depending on network)
sudo docker run --rm -it --runtime=nvidia --name=vllm-download \
  nvcr.io/nvidia/vllm:25.09-py3

# Inside container, trigger model download:
python -c "
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained('openai/gpt-oss-120b')
print('Model downloaded successfully!')
"

# 2. Copy model cache to host (jetson-containers structure)
mkdir -p $ROOT/data/models
docker cp vllm-download:/root/.cache/huggingface $ROOT/data/models/

# 3. Verify model size
du -h $ROOT/data/models/huggingface/hub/models--openai--gpt-oss-120b/
# Should show ~122GB

Distribute to all workshop units:

# Copy to each Thor unit (adjust IPs/hostnames)
for unit in thor-{01..60}; do
  rsync -av --progress $ROOT/data/models/ ${unit}:$ROOT/data/models/
done