NeMo Auditor NeMo Platform SDK Resources

The NeMo Auditor plugin mounts a Python SDK surface on the nemo_platform client at client.auditor. This page documents that surface: how to manage audit configurations and targets in the entity store, and how to run an audit in-process using the local execution path.

The CRUD methods exposed on client.auditor.configs and client.auditor.targets are 1:1 mirrors of the audit configuration and audit target lifecycle and use the same AuditConfig and AuditTarget pydantic schemas the entity store persists.

AuditorPluginResource

The AuditorPluginResource is the sync SDK object for working with the NeMo Auditor plugin. It is accessed directly from a NeMoPlatform instance:

1 import os
2 from nemo_platform import NeMoPlatform
3 
4 
5 client = NeMoPlatform(
6     base_url=os.environ.get("NMP_BASE_URL", "http://localhost:8080"),
7     workspace="default",
8 )
9 auditor = client.auditor  # AuditorPluginResource

Method or property	Description	Returns
`plugin_status()`	Returns auditor plugin health information from the service.	`dict[str, object]`
`configs`	Sub-resource for `AuditConfig` CRUD operations.	`_ConfigResource`
`targets`	Sub-resource for `AuditTarget` CRUD operations.	`_TargetResource`
`run()`	Runs one audit locally, in-process, against a configured target.	`dict`

`configs` sub-resource

Five CRUD methods for AuditConfig entities. The full field reference is in Configuration Schema.

Method	Description	Returns
`create(*, workspace, name, description=None, system=None, run=None, plugins=None, reporting=None)`	Persists a new `AuditConfig`. Sub-blocks default to their `AuditSystemData` / `AuditRunData` / `AuditPluginsData` / `AuditReportData` defaults when omitted.	`AuditConfig`
`list(*, workspace, page=1, page_size=20, sort="-created_at")`	Lists audit configurations in `workspace`.	`dict` with `data`, `pagination`, `sort` keys
`get(*, workspace, name)`	Retrieves a single audit configuration.	`AuditConfig`
`update(*, workspace, name, description=None, system=None, run=None, plugins=None, reporting=None)`	Replaces a configuration’s fields. The PUT semantics replace every sub-block; omitted sub-blocks reset to their defaults.	`AuditConfig`
`delete(*, workspace, name)`	Deletes a configuration.	`None`

`targets` sub-resource

Five CRUD methods for AuditTarget entities. The full field reference is in Target Schema.

Method	Description	Returns
`create(*, workspace, name, type, model, options=None, description=None)`	Persists a new `AuditTarget`. `type` is a garak generator class (for example `nim.NVOpenAIChat`), `model` is the provider’s model identifier, `options` is the generator-specific options dict.	`AuditTarget`
`list(*, workspace, page=1, page_size=20, sort="-created_at")`	Lists audit targets in `workspace`.	`dict` with `data`, `pagination`, `sort` keys
`get(*, workspace, name)`	Retrieves a single audit target.	`AuditTarget`
`update(*, workspace, name, type, model, options=None, description=None)`	Replaces a target’s fields.	`AuditTarget`
`delete(*, workspace, name)`	Deletes a target.	`None`

`run()` arguments

run() invokes garak locally, in-process, against a configured target. The work happens entirely on the host running the SDK call — there is no remote job submission.

Argument	Type	Required	Description
`config`	`AuditConfig \	str`	Yes
`target`	`AuditTarget \	str`	Yes
`workspace`	`str \	None`	No

`run()` return value

run() returns a dict with the following keys:

Key	Type	Description
`status`	`str`	`"completed"` when garak exits with `0`, otherwise `"failed"`.
`returncode`	`int`	The garak subprocess exit code.
`stdout_tail`	`str`	Last ~4 KB of garak’s stdout, useful for diagnostics.
`stderr_tail`	`str`	Last ~4 KB of garak’s stderr.
`results`	`dict[str, dict]`	One entry per produced report artifact. Each value is a `ResultRef` (`{"name": str, "artifact_url": str}`). For local runs, `artifact_url` is a `file://` URL under the scheduler’s temporary results directory.

The results dict can contain up to three keys, each present only if the corresponding file was produced:

report-jsonl — line-delimited JSON probe-by-probe report.
report-html — rendered HTML summary.
report-hitlog-jsonl — line-delimited JSON of every detected hit (failure).

Run an audit locally

1 from nemo_auditor.entities import (
2     AuditSystemData,
3     AuditRunData,
4     AuditPluginsData,
5     AuditReportData,
6 )
7 
8 
9 # Persist a configuration.
10 config = auditor.configs.create(
11     workspace="default",
12     name="quick-scan",
13     description="Lite garak scan, 3 generations per probe.",
14     system=AuditSystemData(lite=True, parallel_attempts=4),
15     run=AuditRunData(generations=3),
16     plugins=AuditPluginsData(probe_spec="latentinjection", detector_spec="auto"),
17     reporting=AuditReportData(report_prefix="quick-scan"),
18 )
19 
20 # Persist a target.
21 target = auditor.targets.create(
22     workspace="default",
23     name="llama-31-8b",
24     type="nim.NVOpenAIChat",
25     model="meta/llama-3.1-8b-instruct",
26     options={
27         "nim": {
28             "nmp_uri_spec": {
29                 "inference_gateway": {"workspace": "default", "provider": "build"},
30             },
31         },
32     },
33 )
34 
35 # Run locally — name strings resolve via the entity store.
36 result = auditor.run(config="quick-scan", target="llama-31-8b", workspace="default")
37 
38 print(result["status"], result["returncode"])
39 for name, ref in result["results"].items():
40     print(f"  {name}: {ref['artifact_url']}")

Alternatively, pass inline AuditConfig and AuditTarget instances directly — useful for ad-hoc runs that should not be persisted:

1 result = auditor.run(config=config, target=target, workspace="default")

AsyncAuditorPluginResource

The AsyncAuditorPluginResource provides the same surface for AsyncNeMoPlatform. Async methods must be awaited.

1 import os
2 from nemo_platform import AsyncNeMoPlatform
3 
4 
5 client = AsyncNeMoPlatform(
6     base_url=os.environ.get("NMP_BASE_URL", "http://localhost:8080"),
7     workspace="default",
8 )
9 auditor = client.auditor  # AsyncAuditorPluginResource

Method or property	Description	Returns
`plugin_status()`	Returns auditor plugin health information from the service.	`dict[str, object]`
`configs`	Sub-resource for `AuditConfig` CRUD operations.	`_AsyncConfigResource`
`targets`	Sub-resource for `AuditTarget` CRUD operations.	`_AsyncTargetResource`
`run()`	Runs one audit locally, in-process, against a configured target.	`dict`

AsyncAuditorPluginResource.run() and the async configs / targets sub-resource methods accept the same arguments as their sync counterparts above. Because the local execution path is synchronous (garak runs in a subprocess), the async run() dispatches the scheduler call through asyncio.to_thread so the caller’s event loop is not blocked.

1 import asyncio
2 
3 
4 async def main() -> None:
5     result = await auditor.run(
6         config="quick-scan",
7         target="llama-31-8b",
8         workspace="default",
9     )
10     for name, ref in result["results"].items():
11         print(f"  {name}: {ref['artifact_url']}")
12 
13 
14 asyncio.run(main())

AuditorPluginResource

configs sub-resource

targets sub-resource

run() arguments