BCM → ServiceNow Alert Integration: Deployment Guide

This guide documents a working, tested process for routing NVIDIA Base Command Manager (BCM) monitoring alerts — both health checks and metric thresholds — into ServiceNow as incidents. It's based on a live build-out and includes the gotchas that cost the most time, so future setups can skip them.

BCM raises alerts through CMDaemon's monitoring framework: measurables (metrics or health checks) get evaluated by triggers (conditions), which fire actions (scripts) when the condition is met. This guide wires a trigger's action to a script that authenticates to ServiceNow via OAuth2 client credentials and creates an incident via the Table API.

Auth model confirmed working: ServiceNow OAuth2 client_credentials grant against /oauth_token.do, then a Bearer token against /api/now/table/incident.

Confirm with your ServiceNow admin before starting:

1. OAuth Application Registry exists with grant type Client Credentials, and you have the client_id / client_secret.
2. The system property glide.oauth.inbound.client.credential.grant_type.enabled is set to true (off by default).
3. The OAuth Application User tied to that registry has write access to the incident table (or whichever table you're targeting). A missing grant here produces a 403 on incident creation even though the token request succeeds.

Test both halves independently before touching BCM:

# 1. Get a token
curl -s -X POST 'https://YOUR_INSTANCE.service-now.com/oauth_token.do' \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  --data-urlencode 'grant_type=client_credentials' \
  --data-urlencode 'client_id=YOUR_CLIENT_ID' \
  --data-urlencode 'client_secret=YOUR_CLIENT_SECRET' | jq
 
# 2. Use the token to create a test incident
curl -s -w '\nHTTP_STATUS:%{http_code}\n' -X POST \
  'https://YOUR_INSTANCE.service-now.com/api/now/table/incident' \
  -H 'Authorization: Bearer PASTE_ACCESS_TOKEN_HERE' \
  -H 'Content-Type: application/json' \
  -d '{    
    \"short_description\": \"Test incident from BCM integration testing\",
    \"description\": \"Manual curl test, ignore/close this\",
    \"assignment_group\": \"SO-NOC-CLDAI\",
    \"priority\": \"3\",
    \"severity\": \"3\",
    \"company\": \"cdw\",
    \"configuration_item\": \"CDW-PLACEHOLDER\"
  }' | jq

Expect HTTP_STATUS:201 on the second call. Gotcha: don't accidentally POST the incident payload to the token URL (/oauth_token.do) — it silently returns a 401 that looks like an auth failure but is actually a wrong-endpoint error. Always double check the URL, not just the credentials, when debugging a 401.

sudo mkdir -p /cm/local/apps/cmd/scripts/actions
sudo nano /cm/local/apps/cmd/scripts/actions/servicenow-incident.sh

#!/bin/bash
# BCM -> ServiceNow incident creation (OAuth2 client_credentials)
SN_INSTANCE="https://YOUR_INSTANCE.service-now.com"
SN_CLIENT_ID="your_client_id"
SN_CLIENT_SECRET="your_client_secret"
 
# Confirmed real CMDaemon env vars (see "Environment Variables" section below)
NODE="${CMD_ENTITY_NAME:-unknown-node}"
CHECK="${CMD_MEASURABLE_NAME:-unknown-check}"
SEVERITY="${CMD_SEVERITY:-unknown}"
VALUE="${CMD_VALUE:-N/A}"
 
# Step 1: get access token
TOKEN_RESPONSE=$(curl -s -X POST "${SN_INSTANCE}/oauth_token.do" \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  --data-urlencode 'grant_type=client_credentials' \
  --data-urlencode "client_id=${SN_CLIENT_ID}" \
  --data-urlencode "client_secret=${SN_CLIENT_SECRET}")
 
ACCESS_TOKEN=$(echo "${TOKEN_RESPONSE}" | jq -r '.access_token')
 
if [ -z "${ACCESS_TOKEN}" ] || [ "${ACCESS_TOKEN}" = "null" ]; then
  echo "$(date) - Failed to get access token: ${TOKEN_RESPONSE}" >> /var/log/cmd-servicenow-status.log
  exit 1
fi
 
# Step 2: create the incident
curl -s -w "\nHTTP_STATUS:%{http_code}\n" \
  -H "Authorization: Bearer ${ACCESS_TOKEN}" \
  -H "Content-Type: application/json" \
  -X POST "${SN_INSTANCE}/api/now/table/incident" \
  -d "{
    \"short_description\": \"Test incident from BCM integration testing\",
    \"description\": \"Manual curl test, ignore/close this\",
    \"assignment_group\": \"SO-NOC-CLDAI\",
    \"priority\": \"3\",
    \"severity\": \"3\",
    \"company\": \"cdw\",
    \"configuration_item\": \"CDW-PLACEHOLDER\"
  }" >> /var/log/cmd-servicenow-status.log 2>&1

CMDaemon runs as root (not a dedicated cmdaemon user — that user doesn't exist by default). Match BCM's own convention for scripts under /cm/local/apps/cmd/scripts/:

sudo chown root:root /cm/local/apps/cmd/scripts/actions/servicenow-incident.sh
sudo chmod 700 /cm/local/apps/cmd/scripts/actions/servicenow-incident.sh

700 keeps the embedded client_secret from being group/world-readable. For anything beyond a validation exercise, split credentials into a separate chmod 600 file sourced by the script, so the secret isn't mixed into logic you might paste elsewhere for debugging.

sudo /cm/local/apps/cmd/scripts/actions/servicenow-incident.sh
sudo cat /var/log/cmd-servicenow-status.log

Confirm HTTP_STATUS:201 and a real incident in ServiceNow (values will show as “unknown-node”/“unknown-check” here since the CMD_* vars are only populated when CMDaemon itself invokes the script).

% monitoring action
% add ServiceNowIncident
% set script /cm/local/apps/cmd/scripts/actions/servicenow-incident.sh
% set runon "Active head node"
% set timeout 30
% commit

Gotchas:

• The parameter is script, not command.
• There's no runlocally parameter — it's runon, and it takes one of exactly three quoted strings: Node, “Active head node”, or “Monitoring node”. Bare ACTIVE will fail.
• Set an explicit timeout so a hung ServiceNow call doesn't block other monitoring actions.

Verify:

% use ServiceNowIncident
% show

BCM's data model: every metric or health check is a measurable, each tied to a producer. The monitoring setup; list view truncates the measurables column — don't rely on it. Instead, use the top-level measurable command:

% measurable
% list

This gives the full table with Type (Metric / HealthCheck / Enum), Name, Parameter (e.g. which GPU, which disk), Class, and Producer. Use this to find your exact target, e.g.:

Gotcha — fractions, not percentages: Metrics like CPUUsage and gpu_utilization are stored as a 0–1 fraction, not 0–100. Check Minimum/Maximum via measurable; use <name>; show before writing any threshold — 80 in an expression means 8000%, not 80%. Use .8, not 80.

Gotcha — device scope: latestmetricdata and measurable show data for whatever device you're currently use'd into under device mode. If a node doesn't appear in device; list, BCM has no visibility into it at all — its hardware (including any GPU) is invisible to monitoring until it's provisioned/added as a managed device. Don't assume a machine you SSH into by one hostname is the same device BCM refers to by another name — confirm by comparing IPs (show on the device vs. ip a on the box).

Triggers are a separate top-level mode from both measurable and action — they reference a measurable via an Expression string and fire actions on state transitions.

% trigger
% list
% use "Failing health checks"
% show

This reveals the real field names, most importantly:

• Expression — pattern format (entity, measurable, parameter) <operator> <value>. Wildcards are *.
• Enter actions / During actions / Leave actions — fire respectively on entering the condition, while it persists, and on leaving it. Use Enter actions for one-shot alerting (don't fire repeatedly while still in the failed/high state).
• State flapping period / State flapping count — built-in noise suppression; a real sustained breach still fires normally, this just protects against a value rapidly oscillating across the threshold.
• Mark entity as failed — set yes for real failure conditions (health checks), no for a metric threshold that isn't itself a device failure (e.g. high CPU).

% trigger
% add ServiceNowManagedServicesFail
% set expression "(*, ManagedServicesOk, *) == FAIL"
% set enteractions ServiceNowIncident
% commit

Gotcha — field order: The expression format is (entity, measurable, parameter), not (measurable, parameter, entity). Putting the measurable name in the first slot produces a silent-ish warning: No known entity matches the specified regex.

% trigger
% add ServiceNowCPUUsageHigh
% set expression "(*, CPUUsage, *) > .8"
% set enteractions ServiceNowIncident
% set markentityasfailed no
% commit

The following names are confirmed correct (captured via env > /tmp/debug.txt inside the action script during a live trigger firing). Guessed names like CMD_NODE / CMD_HEALTHCHECK do not exist — using them silently falls through to blank/default values with no error.

To re-capture this list on a different BCM version (field names may drift), temporarily add env > /tmp/cmd-env-dump.txt as the first line of the action script, trigger a real alert, then cat/grep the dump — don't rely on documentation or guessing.

To stop BCM from auto-restarting a service you're trying to fail intentionally for testing:

% device
% use <device-name>
% services
% list

Gotcha: use <servicename> frequently fails with Unable to find, even for a service visibly in the list — this submode expects add, which is idempotent (re-adds/re-enters edit mode on an existing entry without duplicating it):

% add <servicename>
% set autostart no
% commit

Stop the service at the systemd level using its real unit name, which may differ from BCM's internal key (e.g. BCM calls it ntpd, systemd calls it ntpsec):

sudo systemctl stop <real-unit-name>

Always revert after testing:

% add <servicename>
% set autostart yes
% commit

sudo systemctl start <real-unit-name>

Caution: stopping certain services (auth, DNS, firewall, NFS) even briefly can disrupt your own session or other systems depending on it. Prefer inert services (e.g. ntpd/time sync) for test purposes, and check what depends on a service before stopping it.

1. Confirm ServiceNow OAuth prerequisites (Part 1) with a manual curl test.
2. Deploy and standalone-test the script (Part 2.3) — confirms ServiceNow half works independent of BCM.
3. Register the action in cmsh (Part 3).
4. Identify the exact measurable via measurable; list (Part 4) — confirm fraction vs. percentage before setting any threshold.
5. Create the trigger with Enter actions pointed at the action (Part 5).
6. Trigger a real condition (service stop, load generation) and confirm:
   • latesthealthdata / latestmetricdata shows the expected state on the correct device
   • /var/log/cmd-servicenow-status.log shows a fresh HTTP_STATUS:201
   • The incident in ServiceNow shows real node/check values, not “unknown” (confirms Part 6 env vars are wired correctly)
7. Revert any test-only changes (service autostart, temporary thresholds, debug lines in the script).
8. Restore the trigger to a realistic production threshold if it was temporarily lowered for faster testing.

#!/bin/bash
# BCM -> ServiceNow incident creation (OAuth2 client_credentials)
#
# Config files (both under /cm/local/apps/cmd/scripts/actions/, chmod 600, root-owned):
#   servicenow-shared.env   - SN_INSTANCE / SN_CLIENT_ID / SN_CLIENT_SECRET
#                             (same across every client deployment)
#   servicenow-company.env  - SN_COMPANY_NAME
#                             (the one value that differs per client deployment)

SHARED_CONFIG_FILE="/cm/local/apps/cmd/scripts/actions/servicenow-shared.env"
COMPANY_CONFIG_FILE="/cm/local/apps/cmd/scripts/actions/servicenow-company.env"
LOG=/var/log/cmd-servicenow-status.log

log() {
  echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" >> "$LOG"
}

# ---------------------------------------------------------------------------
# Load and validate config
# ---------------------------------------------------------------------------
if [ ! -f "$SHARED_CONFIG_FILE" ]; then
  log "FATAL: shared config file not found at $SHARED_CONFIG_FILE"
  exit 1
fi
if [ ! -f "$COMPANY_CONFIG_FILE" ]; then
  log "FATAL: company config file not found at $COMPANY_CONFIG_FILE"
  exit 1
fi
# shellcheck disable=SC1090
source "$SHARED_CONFIG_FILE"
# shellcheck disable=SC1090
source "$COMPANY_CONFIG_FILE"

if [ -z "$SN_INSTANCE" ] || [ -z "$SN_CLIENT_ID" ] || [ -z "$SN_CLIENT_SECRET" ]; then
  log "FATAL: SN_INSTANCE / SN_CLIENT_ID / SN_CLIENT_SECRET missing from $SHARED_CONFIG_FILE"
  exit 1
fi

if [ -z "$SN_COMPANY_NAME" ]; then
  log "FATAL: SN_COMPANY_NAME is required but not set in $COMPANY_CONFIG_FILE — refusing to create an unattributed incident"
  exit 1
fi

# ---------------------------------------------------------------------------
# Alert context from CMDaemon
# ---------------------------------------------------------------------------
NODE="${CMD_ENTITY_NAME:-unknown-node}"
CHECK="${CMD_MEASURABLE_NAME:-unknown-check}"
SEVERITY="${CMD_SEVERITY:-unknown}"
VALUE="${CMD_VALUE:-N/A}"

# ---------------------------------------------------------------------------
# Step 1: get access token
# ---------------------------------------------------------------------------
TOKEN_RESPONSE=$(curl -s -X POST "${SN_INSTANCE}/oauth_token.do" \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  --data-urlencode 'grant_type=client_credentials' \
  --data-urlencode "client_id=${SN_CLIENT_ID}" \
  --data-urlencode "client_secret=${SN_CLIENT_SECRET}")

ACCESS_TOKEN=$(echo "${TOKEN_RESPONSE}" | jq -r '.access_token')

if [ -z "${ACCESS_TOKEN}" ] || [ "${ACCESS_TOKEN}" = "null" ]; then
  log "Failed to get access token: ${TOKEN_RESPONSE}"
  exit 1
fi

# ---------------------------------------------------------------------------
# Step 2: resolve Company Name -> sys_id (core_company table)
# ---------------------------------------------------------------------------
COMPANY_SYSID=""
COMPANY_LOOKUP_FAILED=no

COMPANY_RESPONSE=$(curl -s -w "\n%{http_code}" \
  -H "Authorization: Bearer ${ACCESS_TOKEN}" \
  -H "Content-Type: application/json" \
  -G "${SN_INSTANCE}/api/now/table/core_company" \
  --data-urlencode "sysparm_query=name=${SN_COMPANY_NAME}" \
  --data-urlencode "sysparm_fields=sys_id,name" \
  --data-urlencode "sysparm_limit=1")

COMPANY_HTTP_STATUS=$(echo "$COMPANY_RESPONSE" | tail -n1)
COMPANY_BODY=$(echo "$COMPANY_RESPONSE" | sed '$d')

if [ "$COMPANY_HTTP_STATUS" = "200" ]; then
  COMPANY_SYSID=$(echo "$COMPANY_BODY" | jq -r '.result[0].sys_id // empty')
fi

if [ -z "$COMPANY_SYSID" ]; then
  COMPANY_LOOKUP_FAILED=yes
  log "WARNING: could not resolve company '${SN_COMPANY_NAME}' to a sys_id (HTTP ${COMPANY_HTTP_STATUS}). Incident will be created without a company link and tagged for review."
fi

# ---------------------------------------------------------------------------
# Step 3: build a dedup identity for logging purposes. We no longer write to
# correlation_id (blocked by an ACL on this instance) — instead we match on
# short_description containing the check name and node name, scoped by
# company. NOTE: short_description also contains ${VALUE}, which changes
# every time the check fires, so we deliberately match on CHECK and NODE as
# SUBSTRINGS rather than an exact title match — an exact match would treat
# every differing value as a "new" issue and defeat deduplication entirely.
# ---------------------------------------------------------------------------
DEDUP_LABEL="check=${CHECK} node=${NODE} company=${SN_COMPANY_NAME}"

# ---------------------------------------------------------------------------
# Step 4: check for an existing OPEN incident matching this check+node
# (+ company, if resolved). Retries a few times before treating the check
# as failed.
# ---------------------------------------------------------------------------
DEDUP_CHECK_FAILED=no
EXISTING_SYSID=""
EXISTING_NUMBER=""
MATCH_COUNT=0

DEDUP_QUERY="short_descriptionLIKE[BCM-ALERT]^short_descriptionLIKECheck: ${CHECK} |^short_descriptionLIKENode: ${NODE} |"
if [ -n "$COMPANY_SYSID" ]; then
  DEDUP_QUERY="${DEDUP_QUERY}^company=${COMPANY_SYSID}"
fi
DEDUP_QUERY="${DEDUP_QUERY}^ORDERBYDESCsys_updated_on"

for attempt in 1 2 3; do
  DEDUP_RESPONSE=$(curl -s -w "\n%{http_code}" \
    -H "Authorization: Bearer ${ACCESS_TOKEN}" \
    -H "Content-Type: application/json" \
    -G "${SN_INSTANCE}/api/now/table/incident" \
    --data-urlencode "sysparm_query=${DEDUP_QUERY}" \
    --data-urlencode "sysparm_fields=sys_id,number,sys_updated_on,state" \
    --data-urlencode "sysparm_display_value=true" \
    --data-urlencode "sysparm_limit=1")

  DEDUP_HTTP_STATUS=$(echo "$DEDUP_RESPONSE" | tail -n1)
  DEDUP_BODY=$(echo "$DEDUP_RESPONSE" | sed '$d')

  if [ "$DEDUP_HTTP_STATUS" = "200" ]; then
    MATCH_COUNT=$(echo "$DEDUP_BODY" | jq '.result | length')
    if [ "$MATCH_COUNT" -gt 0 ]; then
      TOP_MATCH_STATE=$(echo "$DEDUP_BODY" | jq -r '.result[0].state')
      if [ "$TOP_MATCH_STATE" = "Resolved" ]; then
        # Most recent matching incident is already resolved — this is a
        # fresh recurrence of the issue, not a duplicate. Let it create new.
        log "Most recent matching incident is Resolved — treating this as a new occurrence, not a duplicate."
      else
        EXISTING_SYSID=$(echo "$DEDUP_BODY" | jq -r '.result[0].sys_id')
        EXISTING_NUMBER=$(echo "$DEDUP_BODY" | jq -r '.result[0].number')
      fi
    fi
    DEDUP_CHECK_FAILED=no
    break
  else
    DEDUP_CHECK_FAILED=yes
    log "WARNING: dedup check attempt ${attempt}/3 failed (HTTP ${DEDUP_HTTP_STATUS}), retrying..."
    sleep 2
  fi
done

if [ "$DEDUP_CHECK_FAILED" = "yes" ]; then
  log "WARNING: dedup check failed after 3 attempts for ${DEDUP_LABEL}. Failing open — will create a new incident tagged as unverified for duplicates."
fi

# ---------------------------------------------------------------------------
# Case A: a genuine duplicate was found -> bump it, don't create a new one.
# ---------------------------------------------------------------------------
if [ -n "$EXISTING_SYSID" ]; then
  NOTE_TEXT="Recurred again at $(date '+%Y-%m-%d %H:%M:%S %Z'). Node: ${NODE}, Check: ${CHECK}, Severity: ${SEVERITY}, Value: ${VALUE}."

  curl -s -o /dev/null \
    -H "Authorization: Bearer ${ACCESS_TOKEN}" \
    -H "Content-Type: application/json" \
    -X PATCH "${SN_INSTANCE}/api/now/table/incident/${EXISTING_SYSID}" \
    -d "{\"work_notes\": \"${NOTE_TEXT}\"}" >> "$LOG" 2>&1

  log "Duplicate suppressed for ${DEDUP_LABEL} — bumped existing incident ${EXISTING_NUMBER} with a work note instead of creating a new one."
  exit 0
fi

# ---------------------------------------------------------------------------
# Case B: no duplicate (or dedup check failed open) -> create a new incident.
# ---------------------------------------------------------------------------
SHORT_DESC="[BCM-ALERT] Check: ${CHECK} | Node: ${NODE} | Value: ${VALUE}"
DESC="Health check/metric ${CHECK} on ${NODE}, THIS IS A TEST INCIDENT"
CONFIG_ITEM="${SN_COMPANY_NAME} - ${NODE}"

if [ "$DEDUP_CHECK_FAILED" = "yes" ]; then
  DESC="${DESC} [DEDUP CHECK UNVERIFIED — ServiceNow query failed after retries; this may be a duplicate, please review.]"
fi
if [ "$COMPANY_LOOKUP_FAILED" = "yes" ]; then
  DESC="${DESC} [COMPANY LOOKUP FAILED — no company record matched '${SN_COMPANY_NAME}'; incident created without a company link.]"
fi

# Build the JSON payload, only including "company" if we actually resolved one.
if [ -n "$COMPANY_SYSID" ]; then
  PAYLOAD=$(jq -n \
    --arg short_description "$SHORT_DESC" \
    --arg description "$DESC" \
    --arg company "$COMPANY_SYSID" \
    --arg configuration_item "$CONFIG_ITEM" \
    '{short_description: $short_description, description: $description, assignment_group: "SO-NOC-CLDAI", priority: "3", severity: "3", company: $company, configuration_item: $configuration_item}')
else
  PAYLOAD=$(jq -n \
    --arg short_description "$SHORT_DESC" \
    --arg description "$DESC" \
    --arg configuration_item "$CONFIG_ITEM" \
    '{short_description: $short_description, description: $description, assignment_group: "SO-NOC-CLDAI", priority: "3", severity: "3", configuration_item: $configuration_item}')
fi

curl -s -w "\nHTTP_STATUS:%{http_code}\n" \
  -H "Authorization: Bearer ${ACCESS_TOKEN}" \
  -H "Content-Type: application/json" \
  -X POST "${SN_INSTANCE}/api/now/table/incident" \
  -d "$PAYLOAD" >> "$LOG" 2>&1

log "Created new incident for ${DEDUP_LABEL}"

• Don't guess cmsh parameter/value names — set ? and tab-completion resolve this in seconds versus multiple failed attempts.
• Don't assume a metric is 0–100 — check Minimum/Maximum on the measurable first.
• Don't assume the machine you're SSHed into is the device BCM refers to by the same name — confirm via device; list and IP comparison before spending time debugging “missing” metrics.
• Don't guess CMDaemon's passed environment variable names — dump env once inside a real trigger firing and read the real names directly.
• use vs add vs use “quoted role-tagged key” varies by submode in cmsh — when use <name> fails with “Unable to find,” try add <name> before assuming the object doesn't exist.