====== BCM → ServiceNow Alert Integration: Deployment Guide ======

This guide documents a working, tested process for routing NVIDIA Base Command Manager (BCM) monitoring alerts — both health checks and metric thresholds — into ServiceNow as incidents. It's based on a live build-out and includes the gotchas that cost the most time, so future setups can skip them.


===== Overview =====

BCM raises alerts through CMDaemon's monitoring framework: **measurables** (metrics or health checks) get evaluated by **triggers** (conditions), which fire **actions** (scripts) when the condition is met. This guide wires a trigger's action to a script that authenticates to ServiceNow via OAuth2 client credentials and creates an incident via the Table API.

**Auth model confirmed working:** ServiceNow OAuth2 ''client_credentials'' grant against ''/oauth_token.do'', then a Bearer token against ''/api/now/table/incident''.


===== Part 1: ServiceNow-Side Prerequisites =====

Confirm with your ServiceNow admin before starting:

  - **OAuth Application Registry** exists with grant type ''Client Credentials'', and you have the ''client_id'' / ''client_secret''.
  - The system property ''glide.oauth.inbound.client.credential.grant_type.enabled'' is set to ''true'' (off by default).
  - The **OAuth Application User** tied to that registry has write access to the ''incident'' table (or whichever table you're targeting). A missing grant here produces a 403 on incident creation even though the token request succeeds.

**Test both halves independently before touching BCM:**

<code bash>
# 1. Get a token
curl -s -X POST 'https://YOUR_INSTANCE.service-now.com/oauth_token.do' \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  --data-urlencode 'grant_type=client_credentials' \
  --data-urlencode 'client_id=YOUR_CLIENT_ID' \
  --data-urlencode 'client_secret=YOUR_CLIENT_SECRET' | jq

# 2. Use the token to create a test incident
curl -s -w '\nHTTP_STATUS:%{http_code}\n' -X POST \
  'https://YOUR_INSTANCE.service-now.com/api/now/table/incident' \
  -H 'Authorization: Bearer PASTE_ACCESS_TOKEN_HERE' \
  -H 'Content-Type: application/json' \
  -d '{    
    \"short_description\": \"Test incident from BCM integration testing\",
    \"description\": \"Manual curl test, ignore/close this\",
    \"assignment_group\": \"SO-NOC-CLDAI\",
    \"priority\": \"3\",
    \"severity\": \"3\",
    \"company\": \"cdw\",
    \"configuration_item\": \"CDW-PLACEHOLDER\"
  }' | jq
</code>

Expect ''HTTP_STATUS:201'' on the second call. **Gotcha:** don't accidentally POST the incident payload to the token URL (''/oauth_token.do'') — it silently returns a 401 that looks like an auth failure but is actually a wrong-endpoint error. Always double check the URL, not just the credentials, when debugging a 401.

----

===== Part 2: The Action Script =====

==== 2.1 Script location and content ====

<code bash>
sudo mkdir -p /cm/local/apps/cmd/scripts/actions
sudo nano /cm/local/apps/cmd/scripts/actions/servicenow-incident.sh
</code>

<code bash>
#!/bin/bash
# BCM -> ServiceNow incident creation (OAuth2 client_credentials)
SN_INSTANCE="https://YOUR_INSTANCE.service-now.com"
SN_CLIENT_ID="your_client_id"
SN_CLIENT_SECRET="your_client_secret"

# Confirmed real CMDaemon env vars (see "Environment Variables" section below)
NODE="${CMD_ENTITY_NAME:-unknown-node}"
CHECK="${CMD_MEASURABLE_NAME:-unknown-check}"
SEVERITY="${CMD_SEVERITY:-unknown}"
VALUE="${CMD_VALUE:-N/A}"

# Step 1: get access token
TOKEN_RESPONSE=$(curl -s -X POST "${SN_INSTANCE}/oauth_token.do" \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  --data-urlencode 'grant_type=client_credentials' \
  --data-urlencode "client_id=${SN_CLIENT_ID}" \
  --data-urlencode "client_secret=${SN_CLIENT_SECRET}")

ACCESS_TOKEN=$(echo "${TOKEN_RESPONSE}" | jq -r '.access_token')

if [ -z "${ACCESS_TOKEN}" ] || [ "${ACCESS_TOKEN}" = "null" ]; then
  echo "$(date) - Failed to get access token: ${TOKEN_RESPONSE}" >> /var/log/cmd-servicenow-status.log
  exit 1
fi

# Step 2: create the incident
curl -s -w "\nHTTP_STATUS:%{http_code}\n" \
  -H "Authorization: Bearer ${ACCESS_TOKEN}" \
  -H "Content-Type: application/json" \
  -X POST "${SN_INSTANCE}/api/now/table/incident" \
  -d "{
    \"short_description\": \"Test incident from BCM integration testing\",
    \"description\": \"Manual curl test, ignore/close this\",
    \"assignment_group\": \"SO-NOC-CLDAI\",
    \"priority\": \"3\",
    \"severity\": \"3\",
    \"company\": \"cdw\",
    \"configuration_item\": \"CDW-PLACEHOLDER\"
  }" >> /var/log/cmd-servicenow-status.log 2>&1
</code>

==== 2.2 Permissions and ownership ====

CMDaemon runs as **root** (not a dedicated ''cmdaemon'' user — that user doesn't exist by default). Match BCM's own convention for scripts under ''/cm/local/apps/cmd/scripts/'':

<code bash>
sudo chown root:root /cm/local/apps/cmd/scripts/actions/servicenow-incident.sh
sudo chmod 700 /cm/local/apps/cmd/scripts/actions/servicenow-incident.sh
</code>

''700'' keeps the embedded ''client_secret'' from being group/world-readable. For anything beyond a validation exercise, split credentials into a separate ''chmod 600'' file sourced by the script, so the secret isn't mixed into logic you might paste elsewhere for debugging.

==== 2.3 Standalone test (before touching BCM at all) ====

<code bash>
sudo /cm/local/apps/cmd/scripts/actions/servicenow-incident.sh
sudo cat /var/log/cmd-servicenow-status.log
</code>

Confirm ''HTTP_STATUS:201'' and a real incident in ServiceNow (values will show as "unknown-node"/"unknown-check" here since the ''CMD_*'' vars are only populated when CMDaemon itself invokes the script).

----

===== Part 3: Registering the Action in BCM =====

<code>
% monitoring action
% add ServiceNowIncident
% set script /cm/local/apps/cmd/scripts/actions/servicenow-incident.sh
% set runon "Active head node"
% set timeout 30
% commit
</code>

**Gotchas:**

  * The parameter is ''script'', not ''command''.
  * There's no ''runlocally'' parameter — it's ''runon'', and it takes one of exactly three quoted strings: ''Node'', ''"Active head node"'', or ''"Monitoring node"''. Bare ''ACTIVE'' will fail.
  * Set an explicit ''timeout'' so a hung ServiceNow call doesn't block other monitoring actions.

Verify:
<code>
% use ServiceNowIncident
% show
</code>

----

===== Part 4: Finding the Right Measurable =====

BCM's data model: every metric or health check is a **measurable**, each tied to a **producer**. The ''monitoring setup; list'' view truncates the measurables column — don't rely on it. Instead, use the top-level ''measurable'' command:

<code>
% measurable
% list
</code>

This gives the full table with ''Type'' (Metric / HealthCheck / Enum), ''Name'', ''Parameter'' (e.g. which GPU, which disk), ''Class'', and ''Producer''. Use this to find your exact target, e.g.:

^ Type ^ Name ^ Parameter ^ Producer ^
| HealthCheck | ManagedServicesOk | — | CMDaemonState |
| Metric | CPUUsage | — | ProcStat |
| Metric | gpu_utilization | gpu0 | GPUSampler |

**Gotcha — fractions, not percentages:** Metrics like ''CPUUsage'' and ''gpu_utilization'' are stored as a **0–1 fraction**, not 0–100. Check ''Minimum''/''Maximum'' via ''measurable; use <name>; show'' before writing any threshold — ''80'' in an expression means 8000%, not 80%. Use ''.8'', not ''80''.

**Gotcha — device scope:** ''latestmetricdata'' and ''measurable'' show data for whatever device you're currently ''use'''d into under ''device'' mode. If a node doesn't appear in ''device; list'', BCM has no visibility into it at all — its hardware (including any GPU) is invisible to monitoring until it's provisioned/added as a managed device. Don't assume a machine you SSH into by one hostname is the same device BCM refers to by another name — confirm by comparing IPs (''show'' on the device vs. ''ip a'' on the box).

----

===== Part 5: Creating a Trigger =====

Triggers are a separate top-level mode from both ''measurable'' and ''action'' — they reference a measurable via an ''Expression'' string and fire actions on state transitions.

==== 5.1 Inspect an existing trigger first ====

<code>
% trigger
% list
% use "Failing health checks"
% show
</code>

This reveals the real field names, most importantly:

  * ''Expression'' — pattern format ''(entity, measurable, parameter) <operator> <value>''. Wildcards are ''*''.
  * ''Enter actions'' / ''During actions'' / ''Leave actions'' — fire respectively on entering the condition, while it persists, and on leaving it. Use **Enter actions** for one-shot alerting (don't fire repeatedly while still in the failed/high state).
  * ''State flapping period'' / ''State flapping count'' — built-in noise suppression; a real sustained breach still fires normally, this just protects against a value rapidly oscillating across the threshold.
  * ''Mark entity as failed'' — set ''yes'' for real failure conditions (health checks), ''no'' for a metric threshold that isn't itself a device failure (e.g. high CPU).

==== 5.2 Health check trigger ====

<code>
% trigger
% add ServiceNowManagedServicesFail
% set expression "(*, ManagedServicesOk, *) == FAIL"
% set enteractions ServiceNowIncident
% commit
</code>

**Gotcha — field order:** The expression format is ''(entity, measurable, parameter)'', not ''(measurable, parameter, entity)''. Putting the measurable name in the first slot produces a silent-ish warning: ''No known entity matches the specified regex''.

==== 5.3 Metric threshold trigger ====

<code>
% trigger
% add ServiceNowCPUUsageHigh
% set expression "(*, CPUUsage, *) > .8"
% set enteractions ServiceNowIncident
% set markentityasfailed no
% commit
</code>

----

===== Part 6: The Real CMDaemon Environment Variables =====

The following names are **confirmed correct** (captured via ''env > /tmp/debug.txt'' inside the action script during a live trigger firing). Guessed names like ''CMD_NODE'' / ''CMD_HEALTHCHECK'' **do not exist** — using them silently falls through to blank/default values with no error.

^ Variable ^ Example value ^ Use for ^
| CMD_ENTITY_NAME | bcm | Node/device hostname |
| CMD_MEASURABLE_NAME | CPUUsage | Which metric/health check fired |
| CMD_MEASURABLE_PARAMETER | (e.g. gpu0) | Sub-parameter, if any |
| CMD_SEVERITY | 10 | Numeric severity |
| CMD_VALUE | 71.8% | Formatted/display value |
| CMD_RAW_VALUE | 0.717588 | Raw numeric value |
| CMD_TRIGGER_NAME | ServiceNowCPUUsageHigh | Which trigger fired |
| CMD_TRIGGER_EXPRESSION | (*, CPUUsage, *) > .8 | The expression that matched |
| CMD_ENTITY_TYPE | HeadNode | Device role type |
| CMD_ACTION_NAME | ServiceNowIncident | Which action is running |

To re-capture this list on a different BCM version (field names may drift), temporarily add ''env > /tmp/cmd-env-dump.txt'' as the first line of the action script, trigger a real alert, then ''cat''/''grep'' the dump — don't rely on documentation or guessing.

----

===== Part 7: Editing Device Services (autostart/monitored) =====

To stop BCM from auto-restarting a service you're trying to fail intentionally for testing:

<code>
% device
% use <device-name>
% services
% list
</code>

**Gotcha:** ''use <servicename>'' frequently fails with ''Unable to find'', even for a service visibly in the list — this submode expects ''add'', which is idempotent (re-adds/re-enters edit mode on an existing entry without duplicating it):

<code>
% add <servicename>
% set autostart no
% commit
</code>

Stop the service at the systemd level using its **real unit name**, which may differ from BCM's internal key (e.g. BCM calls it ''ntpd'', systemd calls it ''ntpsec''):

<code bash>
sudo systemctl stop <real-unit-name>
</code>

**Always revert after testing:**
<code>
% add <servicename>
% set autostart yes
% commit
</code>
<code bash>
sudo systemctl start <real-unit-name>
</code>

**Caution:** stopping certain services (auth, DNS, firewall, NFS) even briefly can disrupt your own session or other systems depending on it. Prefer inert services (e.g. ''ntpd''/time sync) for test purposes, and check what depends on a service before stopping it.

----

===== Part 8: Useful cmsh Commands for Faster Iteration =====

^ Command ^ Purpose ^
| measurable; list | Full list of all metrics/health checks with exact names |
| latestmetricdata (inside device; use <name>) | Current recorded value for every metric on that device |
| latestmetricdata <name> | Filter to one metric |
| latesthealthdata | Current health check states for a device |
| samplenow | Force an immediate sample rather than waiting for the interval (check samplenow ? for exact usage) |
| set <param> ? or set ? | Show valid parameter names/values in current mode — use this liberally, cmsh's field names frequently differ from what you'd guess (e.g. runon not runlocally, values must be exact strings like "Active head node") |

----

===== Part 9: End-to-End Test Checklist =====

  - Confirm ServiceNow OAuth prerequisites (Part 1) with a manual curl test.
  - Deploy and standalone-test the script (Part 2.3) — confirms ServiceNow half works independent of BCM.
  - Register the action in ''cmsh'' (Part 3).
  - Identify the exact measurable via ''measurable; list'' (Part 4) — confirm fraction vs. percentage before setting any threshold.
  - Create the trigger with ''Enter actions'' pointed at the action (Part 5).
  - Trigger a real condition (service stop, load generation) and confirm:
    * ''latesthealthdata'' / ''latestmetricdata'' shows the expected state on the correct device
    * ''/var/log/cmd-servicenow-status.log'' shows a fresh ''HTTP_STATUS:201''
    * The incident in ServiceNow shows **real** node/check values, not "unknown" (confirms Part 6 env vars are wired correctly)
  - Revert any test-only changes (service autostart, temporary thresholds, debug lines in the script).
  - Restore the trigger to a realistic production threshold if it was temporarily lowered for faster testing.

----

===== Part 10: Adding Deduplication Logic =====

<code>
#!/bin/bash
# BCM -> ServiceNow incident creation (OAuth2 client_credentials)
#
# Config files (both under /cm/local/apps/cmd/scripts/actions/, chmod 600, root-owned):
#   servicenow-shared.env   - SN_INSTANCE / SN_CLIENT_ID / SN_CLIENT_SECRET
#                             (same across every client deployment)
#   servicenow-company.env  - SN_COMPANY_NAME
#                             (the one value that differs per client deployment)

SHARED_CONFIG_FILE="/cm/local/apps/cmd/scripts/actions/servicenow-shared.env"
COMPANY_CONFIG_FILE="/cm/local/apps/cmd/scripts/actions/servicenow-company.env"
LOG=/var/log/cmd-servicenow-status.log

log() {
  echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" >> "$LOG"
}

# ---------------------------------------------------------------------------
# Load and validate config
# ---------------------------------------------------------------------------
if [ ! -f "$SHARED_CONFIG_FILE" ]; then
  log "FATAL: shared config file not found at $SHARED_CONFIG_FILE"
  exit 1
fi
if [ ! -f "$COMPANY_CONFIG_FILE" ]; then
  log "FATAL: company config file not found at $COMPANY_CONFIG_FILE"
  exit 1
fi
# shellcheck disable=SC1090
source "$SHARED_CONFIG_FILE"
# shellcheck disable=SC1090
source "$COMPANY_CONFIG_FILE"

if [ -z "$SN_INSTANCE" ] || [ -z "$SN_CLIENT_ID" ] || [ -z "$SN_CLIENT_SECRET" ]; then
  log "FATAL: SN_INSTANCE / SN_CLIENT_ID / SN_CLIENT_SECRET missing from $SHARED_CONFIG_FILE"
  exit 1
fi

if [ -z "$SN_COMPANY_NAME" ]; then
  log "FATAL: SN_COMPANY_NAME is required but not set in $COMPANY_CONFIG_FILE — refusing to create an unattributed incident"
  exit 1
fi

# ---------------------------------------------------------------------------
# Alert context from CMDaemon
# ---------------------------------------------------------------------------
NODE="${CMD_ENTITY_NAME:-unknown-node}"
CHECK="${CMD_MEASURABLE_NAME:-unknown-check}"
SEVERITY="${CMD_SEVERITY:-unknown}"
VALUE="${CMD_VALUE:-N/A}"

# ---------------------------------------------------------------------------
# Step 1: get access token
# ---------------------------------------------------------------------------
TOKEN_RESPONSE=$(curl -s -X POST "${SN_INSTANCE}/oauth_token.do" \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  --data-urlencode 'grant_type=client_credentials' \
  --data-urlencode "client_id=${SN_CLIENT_ID}" \
  --data-urlencode "client_secret=${SN_CLIENT_SECRET}")

ACCESS_TOKEN=$(echo "${TOKEN_RESPONSE}" | jq -r '.access_token')

if [ -z "${ACCESS_TOKEN}" ] || [ "${ACCESS_TOKEN}" = "null" ]; then
  log "Failed to get access token: ${TOKEN_RESPONSE}"
  exit 1
fi

# ---------------------------------------------------------------------------
# Step 2: resolve Company Name -> sys_id (core_company table)
# ---------------------------------------------------------------------------
COMPANY_SYSID=""
COMPANY_LOOKUP_FAILED=no

COMPANY_RESPONSE=$(curl -s -w "\n%{http_code}" \
  -H "Authorization: Bearer ${ACCESS_TOKEN}" \
  -H "Content-Type: application/json" \
  -G "${SN_INSTANCE}/api/now/table/core_company" \
  --data-urlencode "sysparm_query=name=${SN_COMPANY_NAME}" \
  --data-urlencode "sysparm_fields=sys_id,name" \
  --data-urlencode "sysparm_limit=1")

COMPANY_HTTP_STATUS=$(echo "$COMPANY_RESPONSE" | tail -n1)
COMPANY_BODY=$(echo "$COMPANY_RESPONSE" | sed '$d')

if [ "$COMPANY_HTTP_STATUS" = "200" ]; then
  COMPANY_SYSID=$(echo "$COMPANY_BODY" | jq -r '.result[0].sys_id // empty')
fi

if [ -z "$COMPANY_SYSID" ]; then
  COMPANY_LOOKUP_FAILED=yes
  log "WARNING: could not resolve company '${SN_COMPANY_NAME}' to a sys_id (HTTP ${COMPANY_HTTP_STATUS}). Incident will be created without a company link and tagged for review."
fi

# ---------------------------------------------------------------------------
# Step 3: build a dedup identity for logging purposes. We no longer write to
# correlation_id (blocked by an ACL on this instance) — instead we match on
# short_description containing the check name and node name, scoped by
# company. NOTE: short_description also contains ${VALUE}, which changes
# every time the check fires, so we deliberately match on CHECK and NODE as
# SUBSTRINGS rather than an exact title match — an exact match would treat
# every differing value as a "new" issue and defeat deduplication entirely.
# ---------------------------------------------------------------------------
DEDUP_LABEL="check=${CHECK} node=${NODE} company=${SN_COMPANY_NAME}"

# ---------------------------------------------------------------------------
# Step 4: check for an existing OPEN incident matching this check+node
# (+ company, if resolved). Retries a few times before treating the check
# as failed.
# ---------------------------------------------------------------------------
DEDUP_CHECK_FAILED=no
EXISTING_SYSID=""
EXISTING_NUMBER=""
MATCH_COUNT=0

DEDUP_QUERY="short_descriptionLIKE[BCM-ALERT]^short_descriptionLIKECheck: ${CHECK} |^short_descriptionLIKENode: ${NODE} |"
if [ -n "$COMPANY_SYSID" ]; then
  DEDUP_QUERY="${DEDUP_QUERY}^company=${COMPANY_SYSID}"
fi
DEDUP_QUERY="${DEDUP_QUERY}^ORDERBYDESCsys_updated_on"

for attempt in 1 2 3; do
  DEDUP_RESPONSE=$(curl -s -w "\n%{http_code}" \
    -H "Authorization: Bearer ${ACCESS_TOKEN}" \
    -H "Content-Type: application/json" \
    -G "${SN_INSTANCE}/api/now/table/incident" \
    --data-urlencode "sysparm_query=${DEDUP_QUERY}" \
    --data-urlencode "sysparm_fields=sys_id,number,sys_updated_on,state" \
    --data-urlencode "sysparm_display_value=true" \
    --data-urlencode "sysparm_limit=1")

  DEDUP_HTTP_STATUS=$(echo "$DEDUP_RESPONSE" | tail -n1)
  DEDUP_BODY=$(echo "$DEDUP_RESPONSE" | sed '$d')

  if [ "$DEDUP_HTTP_STATUS" = "200" ]; then
    MATCH_COUNT=$(echo "$DEDUP_BODY" | jq '.result | length')
    if [ "$MATCH_COUNT" -gt 0 ]; then
      TOP_MATCH_STATE=$(echo "$DEDUP_BODY" | jq -r '.result[0].state')
      if [ "$TOP_MATCH_STATE" = "Resolved" ]; then
        # Most recent matching incident is already resolved — this is a
        # fresh recurrence of the issue, not a duplicate. Let it create new.
        log "Most recent matching incident is Resolved — treating this as a new occurrence, not a duplicate."
      else
        EXISTING_SYSID=$(echo "$DEDUP_BODY" | jq -r '.result[0].sys_id')
        EXISTING_NUMBER=$(echo "$DEDUP_BODY" | jq -r '.result[0].number')
      fi
    fi
    DEDUP_CHECK_FAILED=no
    break
  else
    DEDUP_CHECK_FAILED=yes
    log "WARNING: dedup check attempt ${attempt}/3 failed (HTTP ${DEDUP_HTTP_STATUS}), retrying..."
    sleep 2
  fi
done

if [ "$DEDUP_CHECK_FAILED" = "yes" ]; then
  log "WARNING: dedup check failed after 3 attempts for ${DEDUP_LABEL}. Failing open — will create a new incident tagged as unverified for duplicates."
fi

# ---------------------------------------------------------------------------
# Case A: a genuine duplicate was found -> bump it, don't create a new one.
# ---------------------------------------------------------------------------
if [ -n "$EXISTING_SYSID" ]; then
  NOTE_TEXT="Recurred again at $(date '+%Y-%m-%d %H:%M:%S %Z'). Node: ${NODE}, Check: ${CHECK}, Severity: ${SEVERITY}, Value: ${VALUE}."

  curl -s -o /dev/null \
    -H "Authorization: Bearer ${ACCESS_TOKEN}" \
    -H "Content-Type: application/json" \
    -X PATCH "${SN_INSTANCE}/api/now/table/incident/${EXISTING_SYSID}" \
    -d "{\"work_notes\": \"${NOTE_TEXT}\"}" >> "$LOG" 2>&1

  log "Duplicate suppressed for ${DEDUP_LABEL} — bumped existing incident ${EXISTING_NUMBER} with a work note instead of creating a new one."
  exit 0
fi

# ---------------------------------------------------------------------------
# Case B: no duplicate (or dedup check failed open) -> create a new incident.
# ---------------------------------------------------------------------------
SHORT_DESC="[BCM-ALERT] Check: ${CHECK} | Node: ${NODE} | Value: ${VALUE}"
DESC="Health check/metric ${CHECK} on ${NODE}, THIS IS A TEST INCIDENT"
CONFIG_ITEM="${SN_COMPANY_NAME} - ${NODE}"

if [ "$DEDUP_CHECK_FAILED" = "yes" ]; then
  DESC="${DESC} [DEDUP CHECK UNVERIFIED — ServiceNow query failed after retries; this may be a duplicate, please review.]"
fi
if [ "$COMPANY_LOOKUP_FAILED" = "yes" ]; then
  DESC="${DESC} [COMPANY LOOKUP FAILED — no company record matched '${SN_COMPANY_NAME}'; incident created without a company link.]"
fi

# Build the JSON payload, only including "company" if we actually resolved one.
if [ -n "$COMPANY_SYSID" ]; then
  PAYLOAD=$(jq -n \
    --arg short_description "$SHORT_DESC" \
    --arg description "$DESC" \
    --arg company "$COMPANY_SYSID" \
    --arg configuration_item "$CONFIG_ITEM" \
    '{short_description: $short_description, description: $description, assignment_group: "SO-NOC-CLDAI", priority: "3", severity: "3", company: $company, configuration_item: $configuration_item}')
else
  PAYLOAD=$(jq -n \
    --arg short_description "$SHORT_DESC" \
    --arg description "$DESC" \
    --arg configuration_item "$CONFIG_ITEM" \
    '{short_description: $short_description, description: $description, assignment_group: "SO-NOC-CLDAI", priority: "3", severity: "3", configuration_item: $configuration_item}')
fi

curl -s -w "\nHTTP_STATUS:%{http_code}\n" \
  -H "Authorization: Bearer ${ACCESS_TOKEN}" \
  -H "Content-Type: application/json" \
  -X POST "${SN_INSTANCE}/api/now/table/incident" \
  -d "$PAYLOAD" >> "$LOG" 2>&1

log "Created new incident for ${DEDUP_LABEL}"

</code>

===== Known Time Sinks to Avoid Next Time =====

  * Don't guess ''cmsh'' parameter/value names — ''set ?'' and tab-completion resolve this in seconds versus multiple failed attempts.
  * Don't assume a metric is 0–100 — check ''Minimum''/''Maximum'' on the measurable first.
  * Don't assume the machine you're SSHed into is the device BCM refers to by the same name — confirm via ''device; list'' and IP comparison before spending time debugging "missing" metrics.
  * Don't guess CMDaemon's passed environment variable names — dump ''env'' once inside a real trigger firing and read the real names directly.
  * ''use'' vs ''add'' vs ''use "quoted role-tagged key"'' varies by submode in cmsh — when ''use <name>'' fails with "Unable to find," try ''add <name>'' before assuming the object doesn't exist.