Manage Secrets

NeMo Platform provides an interface to manage user-defined secrets. This allows you to securely store API keys to integrate with other providers. For example, you might want to store a Hugging Face API token as a secret to access models, or a Weights & Biases API key for experiment tracking.

Concepts

A secret is a named resource that holds sensitive information, such as API keys or tokens.

• Secrets are unique within a workspace, meaning they exist within a specific workspace in NeMo Platform. Operations which utilize secrets must specify the workspace in which the secret resides.

• All secrets are encrypted at rest. Once a secret is created, its value cannot be retrieved through the API—not even by workspace Admins. Users can view secret metadata (name, description, timestamps) but can only update or delete the secret, never read its value back.

• Only Platform Administrators can retrieve secret values. This restricted access ensures that sensitive credentials remain protected even from users with full workspace administrative privileges.

• Platform services access secrets internally as needed to facilitate user-requested operations, for example, when a job requires access to a Hugging Face model, or when reporting metrics to Weights & Biases.

• Secrets referenced by other resources (for example, customization configs) are referenced by their name and workspace.

• A reference to a secret by another resource is stored in the format workspace/secret_name.

Access Control

When authentication is enabled, secrets follow role-based access control:

Naming Requirements

Secret names must follow these rules:

• Allowed characters: letters (a-z, A-Z), digits (0-9), hyphens (-), underscores (_), and dots (.)
• Maximum length: 255 characters
• Must contain at least one character (cannot be empty)
• Pattern: ^[\w\-.]+$ (word characters, dots, and hyphens)

Example valid names: hf-token, nim-api-key, wandb.key.production, my_secret_1

Understanding Workspaces

In NeMo Platform, a workspace is used to organize and isolate resources. Workspaces function similarly to namespaces - they provide logical separation between different sets of resources.

• The default workspace is named default
• Most API operations require specifying a workspace parameter
• Secrets, model providers, jobs, and other resources exist within a workspace
• Resources in one workspace cannot directly access resources in another workspace

Operations

To manage secrets, you can use the NeMo Platform SDK. Below are examples of how to create, list, update, and delete secrets.

To work with secrets, you can use the following import and initialization:

1
import os
2
from nemo_platform import NeMoPlatform
3
4
client = NeMoPlatform(
5
    base_url=os.environ.get("NMP_BASE_URL", "http://localhost:8080"),
6
    workspace="default",
7
)

Creating Secrets

Creating a secret involves specifying a name, workspace, and the initial secret data.

1
secret = client.secrets.create(
2
    name="my-secret", workspace="default", value="<your-secret-value>"
3
)

Listing Secrets

You can list all secrets within a specific workspace. This returns metadata only.

1
# List all secrets in the "default" workspace
2
secrets = client.secrets.list(workspace="default")
3
for secret in secrets.data:
4
    print(secret.name)

Retrieving a Secret

To retrieve a single secret by name, use the retrieve method. This returns the secret’s metadata only.

1
# Get a specific secret's metadata
2
secret = client.secrets.retrieve(name="my-secret", workspace="default")
3
print(f"Secret: {secret.name}, Created: {secret.created_at}")

Updating Secrets

To update a secret, you can change its value field.

1
# Update the content of the secret
2
updated_secret = client.secrets.update(
3
    name="my-secret", workspace="default", value="<your-new-secret-value>"
4
)

Deleting Secrets

Deleting a secret will remove the secret from the platform.

1
# Delete a secret
2
client.secrets.delete(name="my-secret", workspace="default")

Use Cases

Hugging Face API Tokens

A Hugging Face API token secret allows NeMo Platform to authenticate with the Hugging Face Hub for accessing models and repositories.

How are Hugging Face API Tokens Used?

NeMo Platform uses Hugging Face api tokens to authenticate requests to the Hugging Face Hub. This enables users to download private models and upload customized models to their Hugging Face repositories.

Creating a Hugging Face Secret

To create a Hugging Face API key secret, use the NeMo Platform SDK to create a new secret with your Hugging Face token.

1
import os
2
from nemo_platform import NeMoPlatform
3
4
client = NeMoPlatform(
5
    base_url=os.environ.get("NMP_BASE_URL", "http://localhost:8080"),
6
    workspace="default",
7
)
8
9
hf_secret = client.secrets.create(
10
    name="my-hugging-face-token", workspace="default", value="<your-hugging-face-token>"
11
)

Weights & Biases Keys

NeMo Platform users can integrate with Weights & Biases by providing their API key as a secret. This allows the NeMo Customizer service to report training metrics to the user’s W&B account.

How are Weights & Biases Keys Used?

When a Weights & Biases key is provided, the NeMo Customizer service sends telemetry data to W&B including:

• Job ID
• Training loss
• Validation loss
• Other relevant metrics

All metrics are stored under the user’s nvidia-nemo-customizer project in their W&B account.

Creating a Weights & Biases Secret

To create a W&B secret, use the NeMo Platform SDK to create a new secret with your W&B API key.

1
import os
2
from nemo_platform import NeMoPlatform
3
4
client = NeMoPlatform(
5
    base_url=os.environ.get("NMP_BASE_URL", "http://localhost:8080"),
6
    workspace="default",
7
)
8
9
wandb_secret = client.secrets.create(
10
    name="my-wandb-key", workspace="default", value="<your-wandb-api-key>"
11
)

External Inference Provider API Keys

When you register an external inference provider (such as NVIDIA Build or OpenAI), you store the provider’s API key as a secret and reference it when creating the model provider.

How are External Provider Keys Used?

NeMo Platform attaches the stored API key to every inference request forwarded through the Inference Gateway to the external provider. This lets you manage credentials centrally and rotate keys without recreating providers.

Creating an External Provider Secret

$
echo "$NVIDIA_API_KEY" | nemo secrets create "nvidia-api-key" --from-file -

After creating the secret, reference it by name when registering the provider.

Encryption Configuration

Rotating or Migrating Encryption Keys

NeMo Platform supports rotation of encryption keys used to secure secrets at rest. Key rotation can be performed by Platform Administrators to comply with security policies or to migrate between encryption providers.

To rotate or migrate encryption keys:

1. Add a new provider configuration to the platform config’s secrets encryption settings. You can name the new provider as desired (for example, v2) and set it as the current_provider.

1
secrets:
2
encryption:
3
current_provider: new_provider_name
4
providers:
5
secret_key:
6
# Keep the existing provider for decryption of old secrets
7
v1:
8
value: "<old-encryption-key>"
9
# Add the new provider for encrypting new secrets
10
v2:
11
value: "<new-encryption-key>"
12
# from_env: "NEW_ENCRYPTION_KEY_ENV_VAR" # Optional alternative to load from environment variable

2. Re-deploy the platform services to apply the new configuration.

3. As the platform administrator role, call the /v2/secrets/rotate-encryption-keys endpoint to initiate the rotation process. This will re-encrypt all existing secrets using the new provider. You can also initiate the rotation using the NeMo Platform SDK:

1
import os
2
from nemo_platform import NeMoPlatform
3
4
client = NeMoPlatform(
5
    base_url=os.environ.get("NMP_BASE_URL", "http://localhost:8080"),
6
    workspace="default",
7
)
8
9
# Initiate the key rotation process
10
client.secrets.admin.rotate_encryption_keys()

This operation can take some time depending on the number of secrets stored. During this process, both the old and new encryption providers must be available to ensure seamless access to secrets. Do not remove the old provider until rotation is confirmed completed.

If any errors occur during rotation, they will be logged for review. You can retry the rotation operation as needed until all secrets are successfully re-encrypted.

4. To validate the rotation completed successfully, you can check the logs for the following confirmation message:

Completed rotation of encryption keys for all secrets

5. After rotation is confirmed, you can optionally remove the old provider configuration from the platform settings to prevent its further use.