Share feedback
Answers are generated based on the documentation.

NVIDIA NIM

Table of contents

Use NVIDIA NIM models with Docker Agent.

Overview

NVIDIA provides access to Nemotron and many other open-weight models through build.nvidia.com (with a free tier) via an OpenAI-compatible API. Docker Agent includes built-in support for NVIDIA as an alias provider. The same alias also works against a self-hosted NVIDIA NIM deployment by overriding base_url.

Setup

  1. Get an API key from build.nvidia.com

  2. Set the environment variable:

    export NVIDIA_API_KEY=your-api-key

Usage

Inline Syntax

The simplest way to use NVIDIA NIM:

agents:
  root:
    model: nvidia/nvidia/nemotron-3-super-120b-a12b
    description: Assistant using NVIDIA NIM
    instruction: You are a helpful assistant.

Named Model

For more control over parameters:

models:
  nemotron:
    provider: nvidia
    model: nvidia/nemotron-3-super-120b-a12b
    temperature: 0.7
    max_tokens: 8192

agents:
  root:
    model: nemotron
    description: Assistant using NVIDIA NIM
    instruction: You are a helpful assistant.

Available Models

NVIDIA NIM hosts Nemotron alongside many other open models (Llama, Qwen, DeepSeek, Mistral, ...). Check the NVIDIA API catalog for the current model list.

Model Description
nvidia/nemotron-3-super-120b-a12b Nemotron 3 Super, reasoning + tool calling
nvidia/nemotron-3-nano-30b-a3b Nemotron 3 Nano, smaller/faster
meta/llama-3.3-70b-instruct Llama 3.3 70B instruction-tuned
qwen/qwen3-coder-480b-a35b-instruct Qwen3 Coder, code-focused

On-Prem / Self-Hosted NIM

For self-hosted NIM deployments, point base_url at your own endpoint instead of the hosted integrate.api.nvidia.com API:

models:
  local_nim:
    provider: nvidia
    model: meta/llama-3.3-70b-instruct
    base_url: http://localhost:8000/v1

How It Works

NVIDIA is implemented as a built-in alias in Docker Agent:

  • API Type: OpenAI-compatible (openai_chatcompletions)
  • Base URL: https://integrate.api.nvidia.com/v1
  • Token Variable: NVIDIA_API_KEY

Because NIM fronts open-weight models whose chat templates often only accept a single system message, Docker Agent coalesces the agent instruction and any toolset instructions into one leading system message before sending the request.

Example: Code Assistant 

agents:
  coder:
    model: nvidia/nvidia/nemotron-3-super-120b-a12b
    description: Code assistant using Nemotron
    instruction: |
      You are an expert programmer using NVIDIA Nemotron.
      Write clean, well-documented code.
      Follow best practices for the language being used.
    toolsets:
      - type: filesystem
      - type: shell
      - type: think