This guide documents a working, tested process for routing NVIDIA Base Command Manager (BCM) monitoring alerts — both health checks and metric thresholds — into ServiceNow as incidents. It's based on a live build-out and includes the gotchas that cost the most time, so future setups can skip them.

BCM raises alerts through CMDaemon's monitoring framework: measurables (metrics or health checks) get evaluated by triggers (conditions), which fire actions (scripts) when the condition is met. This guide wires a trigger's action to a script that authenticates to ServiceNow via OAuth2 client credentials and creates an incident via the Table API.

Auth model confirmed working: ServiceNow OAuth2 client_credentials grant against /oauth_token.do, then a Bearer token against /api/now/table/incident.

Confirm with your ServiceNow admin before starting:

1. OAuth Application Registry exists with grant type Client Credentials, and you have the client_id / client_secret.
2. The system property glide.oauth.inbound.client.credential.grant_type.enabled is set to true (off by default).
3. The OAuth Application User tied to that registry has write access to the incident table (or whichever table you're targeting). A missing grant here produces a 403 on incident creation even though the token request succeeds.

Test both halves independently before touching BCM:

# 1. Get a token
curl -s -X POST 'https://YOUR_INSTANCE.service-now.com/oauth_token.do' \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  --data-urlencode 'grant_type=client_credentials' \
  --data-urlencode 'client_id=YOUR_CLIENT_ID' \
  --data-urlencode 'client_secret=YOUR_CLIENT_SECRET' | jq
 
# 2. Use the token to create a test incident
curl -s -w '\nHTTP_STATUS:%{http_code}\n' -X POST \
  'https://YOUR_INSTANCE.service-now.com/api/now/table/incident' \
  -H 'Authorization: Bearer PASTE_ACCESS_TOKEN_HERE' \
  -H 'Content-Type: application/json' \
  -d '{"short_description": "Test incident", "description": "Manual curl test"}' | jq

Expect HTTP_STATUS:201 on the second call. Gotcha: don't accidentally POST the incident payload to the token URL (/oauth_token.do) — it silently returns a 401 that looks like an auth failure but is actually a wrong-endpoint error. Always double check the URL, not just the credentials, when debugging a 401.

sudo mkdir -p /cm/local/apps/cmd/scripts/actions
sudo nano /cm/local/apps/cmd/scripts/actions/servicenow-incident.sh

#!/bin/bash
# BCM -> ServiceNow incident creation (OAuth2 client_credentials)
SN_INSTANCE="https://YOUR_INSTANCE.service-now.com"
SN_CLIENT_ID="your_client_id"
SN_CLIENT_SECRET="your_client_secret"
 
# Confirmed real CMDaemon env vars (see "Environment Variables" section below)
NODE="${CMD_ENTITY_NAME:-unknown-node}"
CHECK="${CMD_MEASURABLE_NAME:-unknown-check}"
SEVERITY="${CMD_SEVERITY:-unknown}"
VALUE="${CMD_VALUE:-N/A}"
 
# Step 1: get access token
TOKEN_RESPONSE=$(curl -s -X POST "${SN_INSTANCE}/oauth_token.do" \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  --data-urlencode 'grant_type=client_credentials' \
  --data-urlencode "client_id=${SN_CLIENT_ID}" \
  --data-urlencode "client_secret=${SN_CLIENT_SECRET}")
 
ACCESS_TOKEN=$(echo "${TOKEN_RESPONSE}" | jq -r '.access_token')
 
if [ -z "${ACCESS_TOKEN}" ] || [ "${ACCESS_TOKEN}" = "null" ]; then
  echo "$(date) - Failed to get access token: ${TOKEN_RESPONSE}" >> /var/log/cmd-servicenow-status.log
  exit 1
fi
 
# Step 2: create the incident
curl -s -w "\nHTTP_STATUS:%{http_code}\n" \
  -H "Authorization: Bearer ${ACCESS_TOKEN}" \
  -H "Content-Type: application/json" \
  -X POST "${SN_INSTANCE}/api/now/table/incident" \
  -d "{
    \"short_description\": \"BCM alert: ${CHECK} on ${NODE}\",
    \"description\": \"Health check/metric ${CHECK} on ${NODE} reported severity ${SEVERITY}, value ${VALUE}\",
    \"category\": \"Infrastructure\",
    \"urgency\": \"2\",
    \"impact\": \"2\"
  }" >> /var/log/cmd-servicenow-status.log 2>&1

CMDaemon runs as root (not a dedicated cmdaemon user — that user doesn't exist by default). Match BCM's own convention for scripts under /cm/local/apps/cmd/scripts/:

sudo chown root:root /cm/local/apps/cmd/scripts/actions/servicenow-incident.sh
sudo chmod 700 /cm/local/apps/cmd/scripts/actions/servicenow-incident.sh

700 keeps the embedded client_secret from being group/world-readable. For anything beyond a validation exercise, split credentials into a separate chmod 600 file sourced by the script, so the secret isn't mixed into logic you might paste elsewhere for debugging.

sudo /cm/local/apps/cmd/scripts/actions/servicenow-incident.sh
sudo cat /var/log/cmd-servicenow-status.log

Confirm HTTP_STATUS:201 and a real incident in ServiceNow (values will show as “unknown-node”/“unknown-check” here since the CMD_* vars are only populated when CMDaemon itself invokes the script).

% monitoring action
% add ServiceNowIncident
% set script /cm/local/apps/cmd/scripts/actions/servicenow-incident.sh
% set runon "Active head node"
% set timeout 30
% commit

Gotchas:

• The parameter is script, not command.
• There's no runlocally parameter — it's runon, and it takes one of exactly three quoted strings: Node, “Active head node”, or “Monitoring node”. Bare ACTIVE will fail.
• Set an explicit timeout so a hung ServiceNow call doesn't block other monitoring actions.

Verify:

% use ServiceNowIncident
% show

BCM's data model: every metric or health check is a measurable, each tied to a producer. The monitoring setup; list view truncates the measurables column — don't rely on it. Instead, use the top-level measurable command:

% measurable
% list

This gives the full table with Type (Metric / HealthCheck / Enum), Name, Parameter (e.g. which GPU, which disk), Class, and Producer. Use this to find your exact target, e.g.:

Gotcha — fractions, not percentages: Metrics like CPUUsage and gpu_utilization are stored as a 0–1 fraction, not 0–100. Check Minimum/Maximum via measurable; use <name>; show before writing any threshold — 80 in an expression means 8000%, not 80%. Use .8, not 80.

Gotcha — device scope: latestmetricdata and measurable show data for whatever device you're currently use'd into under device mode. If a node doesn't appear in device; list, BCM has no visibility into it at all — its hardware (including any GPU) is invisible to monitoring until it's provisioned/added as a managed device. Don't assume a machine you SSH into by one hostname is the same device BCM refers to by another name — confirm by comparing IPs (show on the device vs. ip a on the box).

Triggers are a separate top-level mode from both measurable and action — they reference a measurable via an Expression string and fire actions on state transitions.

% trigger
% list
% use "Failing health checks"
% show

This reveals the real field names, most importantly:

• Expression — pattern format (entity, measurable, parameter) <operator> <value>. Wildcards are *.
• Enter actions / During actions / Leave actions — fire respectively on entering the condition, while it persists, and on leaving it. Use Enter actions for one-shot alerting (don't fire repeatedly while still in the failed/high state).
• State flapping period / State flapping count — built-in noise suppression; a real sustained breach still fires normally, this just protects against a value rapidly oscillating across the threshold.
• Mark entity as failed — set yes for real failure conditions (health checks), no for a metric threshold that isn't itself a device failure (e.g. high CPU).

% trigger
% add ServiceNowManagedServicesFail
% set expression "(*, ManagedServicesOk, *) == FAIL"
% set enteractions ServiceNowIncident
% commit

Gotcha — field order: The expression format is (entity, measurable, parameter), not (measurable, parameter, entity). Putting the measurable name in the first slot produces a silent-ish warning: No known entity matches the specified regex.

% trigger
% add ServiceNowCPUUsageHigh
% set expression "(*, CPUUsage, *) > .8"
% set enteractions ServiceNowIncident
% set markentityasfailed no
% commit

The following names are confirmed correct (captured via env > /tmp/debug.txt inside the action script during a live trigger firing). Guessed names like CMD_NODE / CMD_HEALTHCHECK do not exist — using them silently falls through to blank/default values with no error.

To re-capture this list on a different BCM version (field names may drift), temporarily add env > /tmp/cmd-env-dump.txt as the first line of the action script, trigger a real alert, then cat/grep the dump — don't rely on documentation or guessing.

To stop BCM from auto-restarting a service you're trying to fail intentionally for testing:

% device
% use <device-name>
% services
% list

Gotcha: use <servicename> frequently fails with Unable to find, even for a service visibly in the list — this submode expects add, which is idempotent (re-adds/re-enters edit mode on an existing entry without duplicating it):

% add <servicename>
% set autostart no
% commit

Stop the service at the systemd level using its real unit name, which may differ from BCM's internal key (e.g. BCM calls it ntpd, systemd calls it ntpsec):

sudo systemctl stop <real-unit-name>

Always revert after testing:

% add <servicename>
% set autostart yes
% commit

sudo systemctl start <real-unit-name>

Caution: stopping certain services (auth, DNS, firewall, NFS) even briefly can disrupt your own session or other systems depending on it. Prefer inert services (e.g. ntpd/time sync) for test purposes, and check what depends on a service before stopping it.

1. Confirm ServiceNow OAuth prerequisites (Part 1) with a manual curl test.
2. Deploy and standalone-test the script (Part 2.3) — confirms ServiceNow half works independent of BCM.
3. Register the action in cmsh (Part 3).
4. Identify the exact measurable via measurable; list (Part 4) — confirm fraction vs. percentage before setting any threshold.
5. Create the trigger with Enter actions pointed at the action (Part 5).
6. Trigger a real condition (service stop, load generation) and confirm:
   • latesthealthdata / latestmetricdata shows the expected state on the correct device
   • /var/log/cmd-servicenow-status.log shows a fresh HTTP_STATUS:201
   • The incident in ServiceNow shows real node/check values, not “unknown” (confirms Part 6 env vars are wired correctly)
7. Revert any test-only changes (service autostart, temporary thresholds, debug lines in the script).
8. Restore the trigger to a realistic production threshold if it was temporarily lowered for faster testing.

• Don't guess cmsh parameter/value names — set ? and tab-completion resolve this in seconds versus multiple failed attempts.
• Don't assume a metric is 0–100 — check Minimum/Maximum on the measurable first.
• Don't assume the machine you're SSHed into is the device BCM refers to by the same name — confirm via device; list and IP comparison before spending time debugging “missing” metrics.
• Don't guess CMDaemon's passed environment variable names — dump env once inside a real trigger firing and read the real names directly.
• use vs add vs use “quoted role-tagged key” varies by submode in cmsh — when use <name> fails with “Unable to find,” try add <name> before assuming the object doesn't exist.