Qwen3.6:35B on DGX Spark

Model Overview

Qwen3.6:35B is Qwen's 35-billion parameter model — excellent reasoning, coding, and multilingual capabilities at a parameter count that fits in the DGX Spark's 128 GB unified memory even without quantization. The GB10's coherent unified memory means both the model weights and the KV cache live in the same pool, accessible by both CPU and GPU cores at full bandwidth.

Setup — Using Ollama

The easiest way to run Qwen3.6:35B on the Spark is via Ollama:

# Pull the model (ARM64 optimized binary)
ollama pull qwen3.6:35b

# Run it
ollama run qwen3.6:35b

# Verify GPU is being used:
ollama ps
# Should show the model running on GPU (Blackwell) not just CPU

Setup — Using vLLM (Higher Throughput)

For serving multiple concurrent requests, vLLM gives better throughput than Ollama's default:

# Install vLLM for ARM64
pip install vllm

# Run with FP8 quantization
vllm serve Qwen/Qwen3.6-35B-A3B \
    --quantization fp8 \
    --tensor-parallel-size 1 \
    --max-model-len 8192 \
    --gpu-memory-utilization 0.85

# Test it
curl http://localhost:8000/v1/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "Qwen/Qwen3.6-35B-A3B",
        "prompt": "Explain quantum computing in simple terms.",
        "max_tokens": 256,
        "temperature": 0.7
    }'

Performance Benchmarks

All numbers below are rough ranges, not exact measurements. Your results will vary based on model, quantization, context length, batch size, and system load.

Throughput by Quantization

Lower quantization = more throughput but slightly lower quality. The GB10's 128 GB unified memory means you have plenty of room to experiment:

Context Length Impact

Longer context = more KV cache memory = slightly lower throughput:

First-Token Latency

Time from query to first word of response. Affected by whether the model is already loaded and your system state:

Multi-Request Performance

With vLLM, concurrent requests increase total throughput but raise per-request latency:

At high concurrency (32+ concurrent requests), per-request throughput drops significantly but total throughput increases. This is the tradeoff: more requests served per second, but each request takes longer individually.

Quality Assessment

Beyond raw speed, Qwen3.6:35B is genuinely competitive with larger models in many categories. The quality difference between Q4_K_M and FP16 is typically about 2-3 percentage points across most benchmarks — small for the VRAM savings.

Based on reported results from the Qwen3.6:35B family across similar parameter ranges (your actual results may vary):

The quantization cost is small. For most real-world use cases (chat, code assistance, document analysis), you won't notice the difference between Q4_K_M and FP16.

Tips and Optimizations

1. Use GGUF format with llama.cpp backend for the best ARM64 performance on this hardware.
2. Pin CPU cores to the high-performance cores if your system supports core pinning: taskset -c 8-27 ollama serve
3. Set OOM score so the GPU process is protected from being killed under memory pressure: echo -1000 > /proc/$(pgrep ollama)/oom_score_adj
4. Pre-load the model on startup so cold-latency doesn't hit your first request.
5. Use NVMe storage for model weights. Load times are significantly faster than on SATA.
6. For batch serving, set --gpu-memory-utilization 0.9 in vLLM to pack more concurrent requests.
7. Monitor memory — the GB10's 128 GB is generous but not infinite. Keep an eye on total memory usage: model + KV cache + system + other services.