NCP-AIO Hands-On Lab Environment

Complete Documentation — AWS g4dn Edition

Setup Guide · Daily Operations · Labs 1–3 · Troubleshooting Reference

Consolidated from an actual build-out on AWS, including every real error hit and fixed along the way.

Three EC2 instances, each with one job. Only the driver is launched by hand — cm-cod-aws (BCM's Cluster on Demand tool for AWS) creates the head node and compute node automatically.

Cost (on-demand, us-east-1 — verify current rates before you start):

NOTE: This is the corrected, battle-tested version of the environment — every step reflects a real error hit and fixed while actually building it, not a first-draft assumption. Follow it in order to skip the detours that came up the first time through.

Register at the link below with a corporate email address (personal addresses are rejected). Free tier covers up to 8 accelerators — a single T4 fits easily. Approval can take up to 2 business days, so do this first.

https://enterpriseproductregistration.nvidia.com/?LicType=Eval&ProductFamily=BCM

You'll receive a product key in the format 123456-123456-123456-123456-123456.

Console → Service Quotas → EC2 → “Running On-Demand G and VT instances”. g4dn.xlarge needs 4 vCPUs of this quota. If it's at 0, request an increase now — approval isn't instant.

You need one role (for instances to assume) and one user (for authentication), with specific permissions discovered by trial and error — AmazonEC2FullAccess alone is not sufficient.

Create the role:

• IAM → Roles → Create role → Trusted entity: AWS service → Use case: EC2
• Attach: AmazonEC2FullAccess
• Name: ncp-aio-lab-driver-role
• This role attaches to both the driver instance and, later, the head node itself (via –headnode-instance-profile) — one role, two uses.

Create the user (ncp-aio-lab-user) with these AWS-managed policies:

• AmazonEC2FullAccess
• AWSPriceListServiceFullAccess — required; cm-cod-aws validates EBS volume pricing during creation and fails with an opaque AccessDeniedException on pricing:GetProducts without this.
• AmazonElasticFileSystemReadOnlyAccess — required; both cluster creation and deletion check EFS resources and fail without this.

Plus this inline policy — the single most-missed permission, letting the user hand the role to a new EC2 instance:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "iam:PassRole",
      "Resource": "arn:aws:iam::<YOUR_ACCOUNT_ID>:role/ncp-aio-lab-driver-role"
    }
  ]
}

Generate an access key for this user (Security credentials tab → Create access key → “Command Line Interface (CLI)”). Keep it for step 2.5.2 — it goes into the AWS CLI's own credential store, never into a lab-specific file.

EC2 → Key Pairs → Create key pair, name bcm-lab-key, type ED25519, format .pem. Download it and chmod 400 bcm-lab-key.pem locally.

2.5.1 Launch

• EC2 → Launch instances → name ncp-aio-driver
• AMI: Ubuntu 22.04 or 24.04 LTS
• Type: t3.micro
• Key pair: bcm-lab-key
• Network: default VPC, auto-assign public IP on, new security group with SSH (22) restricted to My IP
• Advanced details → IAM instance profile: ncp-aio-lab-driver-role

ssh -i bcm-lab-key.pem ubuntu@<DRIVER_IP>

2.5.2 Bootstrap

sudo apt update
sudo apt install -y python3-venv python3-pip unzip curl

Install the AWS CLI via the official installer — apt install awscli fails on current Ubuntu (“no installation candidate”):

curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install
aws --version

Configure a named profile — this writes to ~/.aws/credentials (permissions 600), not a lab-specific plaintext file:

aws configure --profile ncp-aio-lab

2.5.3 Install the COD client

python3 -m venv ~/cod-env
source ~/cod-env/bin/activate
pip install cm-cluster-on-demand-aws

2.5.4 Minimal config file — credentials excluded on purpose

mkdir -p ~/cm-cluster-on-demand.d
cat > ~/cm-cluster-on-demand.d/config.ini << 'EOF'
[cluster.create.license]
license_product_key = YOUR_BCM_PRODUCT_KEY
 
[cluster.create.password]
ssh_pub_key_path = /home/ubuntu/.ssh/bcm_lab_key.pub
EOF
chmod 600 ~/cm-cluster-on-demand.d/config.ini

NOTE: The real config.ini format uses dotted section names and underscores — [cluster.create.license], license_product_key — not hyphens or a flat [aws]/[cluster] structure. Getting this wrong produces confusing “unknown parameter” errors.

Get the key's public half onto the driver:

# from your local machine
scp -i bcm-lab-key.pem bcm-lab-key.pem ubuntu@<DRIVER_IP>:~/.ssh/bcm_lab_key
 
# on the driver
chmod 400 ~/.ssh/bcm_lab_key
ssh-keygen -y -f ~/.ssh/bcm_lab_key > ~/.ssh/bcm_lab_key.pub

curl ifconfig.me

NOTE: This IP changes whenever your network changes (home wifi, hotspot, VPN). If SSH ever times out on a later day, recheck this first and update BOTH the security group AND the network ACL — they are two independent firewall layers, and fixing only one still leaves you locked out.

cm-cod-aws cluster create \
  --name ncp-aio-lab \
  --on-error cleanup \
  --wlm slurm \
  --nodes 1 \
  --node-type g4dn.xlarge \
  --head-node-type t3.large \
  --aws-profile ncp-aio-lab \
  --headnode-instance-profile ncp-aio-lab-driver-role \
  --inbound-rule '<YOUR_IP>/32,22:tcp' \
  --inbound-rule '<YOUR_IP>/32,8081:tcp' \
  --inbound-network-acl-rule '<YOUR_IP>/32,22:TCP,ALLOW,100' \
  --inbound-network-acl-rule '<YOUR_IP>/32,8081:TCP,ALLOW,200' \
  --inbound-network-acl-rule '<YOUR_IP>/32,32768-65535:TCP,ALLOW,300'

Flags that aren't obvious, each confirmed necessary by actually hitting the error from omitting it:

• –headnode-instance-profile — without it, the tool demands raw –aws-access-key-id/–aws-secret-key instead.
• –inbound-rule (×2) — security group rules; creation refuses to proceed without at least one.
• –inbound-network-acl-rule (×3) — a SEPARATE, stateless firewall layer. The ephemeral port range (32768-65535) must be allowed too, or SSH's return traffic is silently dropped even with port 22 “open.”

NOTE: The full build, including BCM's first-boot license activation, took roughly 20–30 minutes end to end. A “Wait For CMDaemon: IOError: read operation timed out” error at ~80% progress does not necessarily mean it failed — check cmsh -c “device list” after a few more minutes before assuming you need to delete and retry.

Do this immediately after creation, so the IP never changes across stop/start:

• EC2 → Elastic IPs → Allocate Elastic IP address → Allocate
• Select it → Actions → Associate Elastic IP address → choose the head node instance

ssh -i bcm-lab-key.pem root@<ELASTIC_IP>
cmsh -c "device list"

Expect the HeadNode as [ UP ] and a CloudNode (cnode001) as [ DOWN ] — not yet powered on, that's expected.

cmsh -c "device; power on -n cnode001"

NOTE: BCM's cloud-status polling isn't instant — it checks AWS roughly every 30 minutes on its own schedule. A power status check run too soon may still show FAILED (Instance does not exist) even though the instance is actually running fine. Confirm directly in the EC2 console if in doubt.

cmsh -c "device list"
ssh cnode001 nvidia-smi

Passing –wlm slurm at creation should configure Slurm automatically, but if first-boot was interrupted (see the patience note in 2.7), this can silently not complete. Check:

cmsh -c "wlm; list"

If this returns an empty table, run the wizard manually:

cm-wlm-setup

Choose Slurm, assign roles to cnode001's category, let it finish (~20 minutes). Re-check wlm; list — it should now show an entry.

NOTE: In this environment, sinfo fails with the DNS SRV error below on essentially every session, not just occasionally. Treat the fix below as a standard, required part of setup and daily startup — not an if-this-happens troubleshooting step.

The error looks like this:

sinfo: error: resolve_ctls_from_dns_srv: res_nsearch error: Unknown host
sinfo: error: fetch_config: DNS SRV lookup failed
sinfo: fatal: Could not establish a configuration source

Root cause: Slurm's client tools fall back to DNS-based controller auto-discovery when they can't find a local config source, and BCM's internal DNS zone was never written correctly during first boot in this environment. Rather than re-diagnosing this every session, apply the fix once, permanently:

export SLURM_CONF=/cm/shared/apps/slurm/etc/slurm/slurm.conf
echo 'export SLURM_CONF=/cm/shared/apps/slurm/etc/slurm/slurm.conf' >> ~/.bashrc
source ~/.bashrc
sinfo -N -l

Confirm cnode001 shows real node status (idle, not an error) before moving on. If a future session still prompts for this — e.g. after a fresh cluster rebuild rather than a normal stop/start — re-run the find command below once to confirm the path hasn't changed, then repeat the lines above:

find / -name "slurm.conf" 2>/dev/null

At this point: a reachable head node on a stable IP, cnode001 powered on showing a real GPU via nvidia-smi, wlm list showing Slurm configured, and sinfo returning real node status. Move on to the labs.

Stopping (not terminating) instances does not erase work. BCM's config, Slurm's setup, and Kubernetes' install all persist across stop/start — confirmed by actually doing it repeatedly. What resets:

• Running pods/jobs — runtime state, not disk state. Redeploy them next session.
• Public IP on any instance without an Elastic IP attached — changes every start. (The head node has one, so it doesn't have this problem.)

# 1.
cmsh -c "device; power off -n cnode001"
cmsh -c "device; power status"      # wait for OFF
 
# 2. (run on the head node)
shutdown -h now
 
# 3.
# EC2 Console -> ncp-aio-driver -> Instance state -> Stop instance

All three stop billing for compute immediately. Only idle EBS storage (a few cents/day) and the Elastic IP (~$0.005/hr) keep accruing.

Step 0 — check your current IP before touching anything else:

curl ifconfig.me

NOTE: If your IP changed, update BOTH firewall layers before attempting SSH — this was learned the hard way. Security group alone being correct is not enough if the Network ACL still has the old IP, and vice versa.

• EC2 → Security Groups → head node's SG → Inbound rules → update the 22 and 8081 rules to your new IP
• VPC → Network ACLs → the one on the public subnet → update all three rules (22, 8081, 32768-65535) to your new IP

# 1.
# EC2 Console -> Start ncp-aio-driver and the head node instance
 
# 2. (same IP every time, thanks to the Elastic IP)
ssh -i bcm-lab-key.pem root@<ELASTIC_IP>
 
# 3. (confirm CMDaemon is back; fast, normal reboot)
cmsh -c "device list"
 
# 4.
cmsh -c "device; power on -n cnode001"
 
# 5. Wait a few minutes (BCM's cloud-status polling lags), then:
cmsh -c "device list"
sinfo -N -l
ssh cnode001 nvidia-smi

You're back exactly where you left off — no cluster recreation, no reinstalling Slurm/Kubernetes/BCM, ever, for the life of this environment.

The gap is almost entirely the g4dn.xlarge (~85% of the hourly total) — never leave that one running idle.

# from the driver
cm-cod-aws cluster delete ncp-aio-lab --aws-profile ncp-aio-lab

• Confirm in the console that the head node and cnode001 both actually terminated
• Release the Elastic IP (EC2 → Elastic IPs → Release) — allocated-but-unassociated IPs bill more aggressively
• Terminate ncp-aio-driver if fully done
• Optionally deactivate the IAM user's access key and remove the IAM role/user

Maps to: Troubleshooting & Optimization. Prerequisite: Section 2 complete through 2.12 — you need a working baseline before you can spot what's broken.

Four layers, top to bottom, every time:

1. Cluster-level — does BCM/Slurm even see the node and consider it healthy?

sinfo -N -l
cmsh -c "device status cnode001"

2. Service-level — is the specific daemon that should be running, actually running?

ssh cnode001 systemctl status slurmd
ssh cnode001 journalctl -u slurmd -xe --no-pager | tail -50
ssh cnode001 systemctl status munge

3. Config-level — does the config the daemon reads exist and say what it should?

scontrol show node cnode001
find / -name "slurm.conf" 2>/dev/null
ssh cnode001 cat /cm/local/apps/slurm/var/etc/gres.conf

4. Hardware/dependency-level — is the thing underneath actually working?

ssh cnode001 nvidia-smi
ssh cnode001 'ls -l /dev/nvidia*'
ssh cnode001 dmesg | grep -i nvidia | tail -30

Run all four whenever something's wrong, even if you think you know the answer — the value of this lab is fluency with the sequence, not guessing right on the first try.

Copy break_node.sh to the head node, then run it against cnode001 without reading it first:

# from your local machine
scp -i bcm-lab-key.pem break_node.sh root@<ELASTIC_IP>:/root/
 
# on the head node
ssh cnode001 'bash -s' < /root/break_node.sh

It injects one of five random faults and writes an answer key to /root/.lab1_answer_key on cnode001 (the node the script actually ran on) — only check that if truly stuck.

Work through all four layers above. Common findings: sinfo shows cnode001 as down*/drain/drng; scontrol show node shows a Reason= field; systemctl status shows a daemon inactive (dead); or nvidia-smi errors despite the driver being intact underneath.

NOTE: BCM's own service monitoring can auto-restart a stopped critical service (like slurmd) on its own, sometimes repeatedly if something else keeps re-triggering the underlying condition. Check `grep -i slurmd /var/log/cmdaemon | tail -40` on the node in question if a fault seems to have “fixed itself” — CMDaemon's own log will show “Service X is not running, attempting to restart” entries if this is what happened. A flapping restart loop accompanied by cgroup errors (“No space left on device” on cpuset attributes) is best resolved with a clean node reboot (`cmsh -c “device restart -n cnode001”`) rather than manual cgroup surgery.

# node stuck in a bad state
scontrol update NodeName=cnode001 State=RESUME
 
# a daemon stopped/disabled
ssh cnode001 systemctl enable --now slurmd
ssh cnode001 systemctl enable --now munge
 
# gres.conf GPU line missing/commented — fix the file, then:
ssh cnode001 systemctl restart slurmd
 
# after any daemon restart or config change
scontrol reconfigure

NOTE: Add –mpi=none to srun. Without it, Slurm defaults to a PMIx MPI plugin that fails for any non-MPI command like nvidia-smi, producing a confusing “mpi/pmix: can not load PMIx library” error unrelated to your actual fix.

srun -w cnode001 --gres=gpu:1 --mpi=none -N1 nvidia-smi | tee ~/lab1_validation_output.txt

Capture the job's accounting record as two separate commands, since the second depends on the first's output:

sacct -n -X --format=JobID | tail -1
# take the job ID that returns, then:
sacct --format=JobID,JobName,Partition,State,ExitCode,NodeList -j <job_id> | tee -a ~/lab1_validation_output.txt

Confirm sinfo shows cnode001 back to idle before considering the lab done.

While building this environment, cmsh -c “device status cnode001” returned:

cnode001 ................. [ UP ], health check failed

Slurm's own scontrol show node showed State=IDLE (healthy from Slurm's point of view), but BCM's independent health-check system disagreed. The two systems can and do disagree — never trust just one signal. To see exactly which check failed:

cmsh -c "device latesthealthdata cnode001"

Separately, sinfo itself failed cluster-wide at one point with the DNS SRV error described in Section 2.12. Root cause: Slurm's client fell back to DNS-based controller discovery because it couldn't find a local slurm.conf, combined with an unrelated fault in BCM's internal DNS zone. The SLURM_CONF workaround in 2.12 resolves it. This is a good example of a fault that lives above the individual node — always check whether a symptom on cnode001 is actually a head-node/cluster-wide problem before assuming the fault is local to the worker.

Maps to: Installation & Deployment / Workload Management. Prerequisite: Section 2 complete, cnode001 up and healthy.

cm-kubernetes-setup

This is a multi-screen TUI wizard. The choices below are what was actually selected while building this environment, with the reasoning — accept these unless you have a specific reason to deviate.

NOTE: The deployment pulls container images and installs the full control plane — expect a real wait, similar in order of magnitude to the original cluster creation. Do not power off any instance while this is actively running; interrupting it mid-install risks leaving Kubernetes half-configured, the same way the Slurm WLM setup was left incomplete earlier when its own first-boot process was cut short.

NOTE: At the worker-node/category question, explicitly select cnode001 only. If you've completed Lab 3 and a gpu-compute category with placeholder nodes (cnode002-004) exists, accepting a category default here can sweep those placeholders in as intended worker nodes. Since they're never powered on, the install will later fail trying to reboot them (“Some of the nodes cannot be rebooted: Nodes not up”). If this happens mid-wizard and you're dropped to an Undo/Abort/Skip/Retry prompt, do not let the SSH session drop before answering it — an interrupted answer here can leave a second, half-configured Kubernetes cluster (e.g. default2) behind. See Section 5.6.

module load kubernetes
kubectl get nodes -o wide
kubectl get pods -A
kubectl get pods -n gpu-operator

Confirm the GPU is exposed as a schedulable resource:

kubectl describe node cnode001 | grep -A6 Capacity
# expect: nvidia.com/gpu: 1

NOTE: If nvidia.com/gpu is missing right after a stop/start cycle, don't panic — powering the node back on restarts kubelet, which cascades into container restarts across the node. Recheck a few minutes later before assuming something's actually broken; this resolved itself every time it came up during the real build.

cmsh
% kubernetes
% list
% status
% exit

Subcommands under kubernetes mode vary by BCM version — run help inside that mode if list/status don't match exactly. Also spot-check the underlying services directly:

cmsh -c "device use cnode001; services; status kubelet"
cmsh -c "device use master; services; status kube-apiserver"

The wizard's own automatic validators already prove the GPU stack works end-to-end:

• nvidia-operator-validator pod: 1/1 Running — comprehensive post-install check across driver, toolkit, device plugin, and CUDA
• nvidia-cuda-validator pod: Completed with 0 errors — a one-shot job that runs a real CUDA workload
• kubectl describe node showing nvidia.com/gpu: 1 in Capacity

These three together are sufficient evidence that GPU Operator is correctly installed and the GPU is schedulable — the core thing Lab 2 and the exam domain are actually testing.

A separate, manually-created test pod hit a real, unresolved edge case worth documenting:

cat << 'EOF' | kubectl create -f -
apiVersion: v1
kind: Pod
metadata:
  name: cuda-vectoradd
spec:
  restartPolicy: OnFailure
  containers:
  - name: cuda-vectoradd
    image: "nvidia/samples:vectoradd-cuda11.2.1"
    resources:
      limits:
        nvidia.com/gpu: 1
EOF
kubectl logs cuda-vectoradd

This pod repeatedly hit CrashLoopBackOff / RunContainerError with:

OCI runtime create failed: could not apply required modification to OCI specification:
error modifying OCI spec: failed to inject CDI devices: unresolvable CDI devices
management.nvidia.com/gpu=GPU-0b402f82-3aa6-6bd4-c3bd-21a993b2f670

Root cause: the CDI (Container Device Interface) spec at /run/cdi/management.nvidia.com-gpu.yaml on cnode001 only contained a generic “name: all” device entry, never a per-GPU-UUID entry the container runtime was actually asking for. Restarting the nvidia-container-toolkit-daemonset pod did not regenerate a per-device entry.

Diagnostic/fix path attempted, for reference if you hit this again:

# inspect the current spec
ssh cnode001 cat /run/cdi/management.nvidia.com-gpu.yaml
 
# check whether CDI mode is even the right mode for this install
kubectl get clusterpolicies.nvidia.com -o yaml | grep -B2 -A5 'cdi:'
 
# if enabled: true, falling back to the legacy (non-CDI) runtime mode is a reasonable next step:
kubectl patch clusterpolicies.nvidia.com cluster-policy --type=merge -p '{"spec":{"cdi":{"enabled":false,"default":false}}}'
 
# then recreate the test pod and watch it rather than checking logs immediately:
kubectl delete pod cuda-vectoradd
kubectl get pod cuda-vectoradd -w

NOTE: This specific pod's CDI issue was left open when the session moved on to Lab 3, given the already-strong evidence from the automatic validators and the real AWS cost of continuing to chase a single extra test pod. Treat this pod as optional bonus verification, not a required pass/fail gate for Lab 2 — if you hit the same CDI error, the ClusterPolicy patch above is the most promising untried lead.

Before ending a Lab 2 session, uninstall Kubernetes so the next attempt starts from a clean slate rather than fighting leftover state from a prior run.

NOTE: If a wizard run is ever interrupted mid-operation (a dropped SSH session, a stray Ctrl+C at an Undo/Abort/Skip/Retry prompt), it can leave a second, half-configured cluster behind rather than cleanly updating the first. Check for this before assuming only one cluster exists.

1. Confirm what actually exists:

cmsh -c "kubernetes; list"

Expect one entry (e.g. default) on a clean install. If a second entry shows up (e.g. default2), an interrupted prior run left a partial second cluster behind — remove both in the steps below, not just the broken one.

2. Run the wizard and select the existing cluster name at the AskKubeCluster prompt, rather than typing a new one:

cm-kubernetes-setup

Selecting an existing cluster name (instead of creating a new one) surfaces an action menu rather than jumping straight into configuration questions. Read the exact wording on screen before choosing — it varies by BCM version — and pick whichever option represents removal/undeploy, not redeploy.

3. Repeat for every cluster name step 1 revealed.

4. Verify nothing is left:

cmsh -c "kubernetes; list"
cmsh -c "configurationoverlay; list"

Expect no kube-* entries in either.

5. Check whether any node — especially the Lab 3 placeholders (cnode002-004) — retained a Kubernetes role from a prior run's worker-category sweep-in:

cmsh -c "device use cnode002; roles; list"
cmsh -c "device use cnode003; roles; list"
cmsh -c "device use cnode004; roles; list"

Remove anything kube-related found there (roles; remove <role>; commit) before your next run.

NOTE: At the AskWorkerCategories question on your next install, select cnode001 explicitly rather than accepting a category default — this is what caused cnode002-004 to get swept in as worker nodes previously, which then failed because those placeholders are never actually powered on.

Maps to: Administration / Installation & Deployment. Prerequisite: Section 2 complete.

This lab builds new device placeholders rather than touching the live cnode001 — same clone/category/MAC workflow real onboarding uses, zero risk to the node used in Labs 1 and 2.

A representative customer intake form for 3 new GPU nodes, named cnode002-004 to match the naming convention the real cm-cod-aws-created node (cnode001) already uses:

cmsh
% softwareimage
% clone default-image gpu-compute-image
% commit
% use gpu-compute-image
% kernelmodules
% add nvidia_peermem
% commit
% exit

NOTE: Committing this may return a warning like “Module nvidia_peermem does not exist for specified kernel.” This is expected — the module is part of NVIDIA's GPUDirect RDMA support and isn't present in this image's driver package. The configuration entry still commits; the warning only means the module wouldn't actually load if this image were deployed to a real booting node right now. Since cnode002-004 are placeholders that are never powered on, this doesn't block the lab.

% category
% clone default gpu-compute
% set softwareimage gpu-compute-image
% commit
% exit

% device
% foreach --clone cnode001 -n cnode002..cnode004 ()
% commit
 
% use cnode002
% set category gpu-compute
% interfaces
% set eth0 mac 02:1A:C0:FF:EE:02
% exit
% commit
 
% use cnode003
% set category gpu-compute
% interfaces
% set eth0 mac 02:1A:C0:FF:EE:03
% exit
% commit
 
% use cnode004
% set category gpu-compute
% interfaces
% set eth0 mac 02:1A:C0:FF:EE:04
% exit
% commit

cmsh -c "device list"                              # cnode002-004 present, right category
cmsh -c "softwareimage list"                        # gpu-compute-image shows 3 nodes assigned
cmsh -c "category list"                              # gpu-compute category exists
cmsh -c "device use cnode002; interfaces; list"     # MAC set correctly

The original plan for this lab included a final step: attaching cnode002-004 to Slurm's client/submit roles and adding them to a partition, so they'd appear in sinfo. In practice, this step ran into three layers of real confusion worth documenting, followed by a deliberate decision to stop.

What was learned:

• cmsh's “partition” object is an unrelated BCM concept (cluster/hierarchy management) — not a Slurm partition. The error “Multiple partitions are disabled in this version of Cluster Manager” is a BCM free-tier licensing message about that unrelated object, a red herring for this task.
• The real Slurm partition is called defq (confirmed directly in /cm/shared/apps/slurm/etc/slurm/slurm.conf), and is managed in cmsh via wlm > use slurm — but Client nodes/Submit nodes shown there are read-only computed summaries, not directly settable fields.
• The actual mechanism is assigning the slurmclient/slurmsubmit roles to each device individually (cmsh > device > use <node> > roles > assign slurmclient), which also requires an additional required field, wlmcluster, set to slurm before it will commit.

Why the lab stops before completing this step:

cnode002-004 are placeholders that are never actually powered on — there is no real EC2 instance, no OS, and no slurmd behind them. Even with a fully correct role assignment, sinfo would show them as permanently unavailable, since there's no machine to check in. The exam-relevant skill this lab targets — correctly translating a site survey into category, image, and MAC configuration for new nodes — was already fully demonstrated in Sections 6.2-6.5. Continuing to chase the Slurm attachment for hardware that will never exist is diminishing-returns effort against a verification step that cannot actually succeed, at real ongoing AWS cost.

NOTE: If you want to practice the Slurm role-attachment workflow specifically, the correct commands are documented in Section 4.4/2.11 (roles; assign slurmclient; set wlmcluster slurm; commit) and apply cleanly to a node that is actually powered on and running — such as a genuine additional compute node, rather than a placeholder.

Before ending a Lab 3 session, remove the placeholder nodes, category, and software image so the next attempt starts from a blank slate rather than cloning on top of leftover objects. Remove in this order — devices first, then category, then image — since each step removes the thing referencing the next one, avoiding “still in use” errors.

1. Remove the device placeholders:

cmsh
% device
% remove cnode002
% remove cnode003
% remove cnode004
% commit

2. Remove the category:

% category
% remove gpu-compute
% commit

3. Remove the software image:

% softwareimage
% remove gpu-compute-image
% commit

4. Verify a blank slate:

cmsh -c "device list"
cmsh -c "category list"
cmsh -c "softwareimage list"

Expect device list to show only ncp-aio-lab and cnode001; category list to show only default; softwareimage list to show only the original default image(s) — no gpu-compute/gpu-compute-image anywhere.

NOTE: If removal of the category or image errors claiming it is still referenced, a device or role still points at it — re-check device list's Category column before retrying, rather than guessing at a different removal command.

• Always include -i bcm-lab-key.pem — a bare ssh user@host silently falls back to your default local key and fails with “Permission denied (publickey…)”, which looks like a server-side problem but isn't.
• Check curl ifconfig.me at the start of every session, before attempting SSH — home/mobile IPs change, and a stale rule produces a silent connection timeout rather than a clear error.
• Security groups and Network ACLs are two independent firewall layers. Both must allow your current IP; fixing only one still leaves you locked out. NACLs are stateless and need the ephemeral port range (32768-65535) allowed explicitly, or return traffic is silently dropped.
• An Elastic IP on the head node keeps its address stable across stop/start — allocate one immediately after cluster creation, well worth the trivial ~$0.005/hr.
• “Operation timed out” when even EC2 Instance Connect (which runs from AWS's own infrastructure, not your network) also fails points to a networking layer problem (missing Internet Gateway, missing route table entry) rather than your local network or IP.

• config.ini uses dotted section names and underscore keys: [cluster.create.license] / license_product_key, [cluster.create.password] / ssh_pub_key_path — not [aws]/[cluster] with hyphens.
• –aws-profile authenticates the driver (via ~/.aws/credentials); –headnode-instance-profile is a separate flag that gives the newly-created head node its own AWS identity via an IAM role. Confusing the two produces credential errors that look identical but need different fixes.
• Required IAM permissions beyond AmazonEC2FullAccess: an inline iam:PassRole policy scoped to the specific role being passed, AWSPriceListServiceFullAccess (volume-type validation), and AmazonElasticFileSystemReadOnlyAccess (checked during both create and delete).
• –inbound-rule sets security group rules; –inbound-network-acl-rule sets NACL rules — both are required, and NACL rules need the ephemeral port range included.

• device use <hostname> — extra words (e.g. “device use node cnode001”) fail silently and dump the top-level mode list instead of a clear syntax error.
• show <role> takes the role name directly as an argument — no “use” first.
• “Readonly properties cannot be set” means the field is a computed summary (e.g. Client nodes/Submit nodes on a wlm object), not something to set directly — the real mechanism is usually a role assignment elsewhere.
• When unsure of a valid property/subcommand name inside interactive cmsh, type the start of a word and press Tab rather than guessing blind — faster and free of charge compared to repeated failed attempts.
• cmsh's own “partition” concept and Slurm's partition concept are not the same object — see Section 6.6.

• Add –mpi=none to srun for any non-MPI command (like nvidia-smi) — otherwise Slurm's default PMIx plugin produces an unrelated-looking error.
• sacct -j <id> requires an actual job ID argument; get one first with sacct -n -X –format=JobID | tail -1, run as a separate command.
• sinfo failing with a DNS SRV lookup error happens on essentially every session in this environment, not just occasionally. The permanent fix — export SLURM_CONF=/cm/shared/apps/slurm/etc/slurm/slurm.conf added to ~/.bashrc — is now a standard step in Section 2.12, not a conditional workaround.

Injects one of five random faults into whatever node it's run against, for blind diagnosis practice:

• slurmd stopped and disabled
• munge (Slurm's auth daemon) stopped and disabled
• GPU line in gres.conf commented out
• Node manually set to DRAIN state
• nvidia-smi binary permissions set to 000 (driver itself untouched)

Usage:

scp -i bcm-lab-key.pem break_node.sh root@<ELASTIC_IP>:/root/
ssh cnode001 'bash -s' < /root/break_node.sh

An answer key is written to /root/.lab1_answer_key on cnode001 (wherever the script actually ran) on whichever host runs the script.