Combined Cloud Managed Services

User Tools

Site Tools

wiki:ai:affirm

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
wiki:ai:affirm [2026/05/15 16:06] – sramasamywiki:ai:affirm [2026/05/18 11:29] (current) – sramasamy
Line 19: Line 19:
   - [[#6_procedure_generate_ai_monitoring_performance_report|Procedure — Generate AI Monitoring Performance Report]]   - [[#6_procedure_generate_ai_monitoring_performance_report|Procedure — Generate AI Monitoring Performance Report]]
   - [[#7_procedure_generate_ai_health_report|Procedure — Generate AI Health Report]]   - [[#7_procedure_generate_ai_health_report|Procedure — Generate AI Health Report]]
-  - [[#8_exporting_delivering_the_report|Exporting & Delivering the Report]] +  - [[#8_Support Details| Affirm Support Details]]
-  - [[#9_validation_checklist_before_sending_to_client|Validation Checklist (Before Sending to Client)]] +
-  - [[#10_troubleshooting|Troubleshooting]] +
-  - [[#11_recommended_practices_tips|Recommended Practices & Tips]] +
-  - [[#12_revision_history|Revision History]] +
 ---- ----
  
Line 101: Line 96:
 <note>**Both reports use the same filter pattern.** \\ AI Health Report and AI Monitoring Performance Report share a near-identical filter pane: Organization Name, AI Asset Name, Evaluator Name, Method Name, and a Date / Date Range filter. Mastering the filter pane once applies to both reports. The differences are summarized in Section 7 (performance) and Section 8 (health).</note> <note>**Both reports use the same filter pattern.** \\ AI Health Report and AI Monitoring Performance Report share a near-identical filter pane: Organization Name, AI Asset Name, Evaluator Name, Method Name, and a Date / Date Range filter. Mastering the filter pane once applies to both reports. The differences are summarized in Section 7 (performance) and Section 8 (health).</note>
  
-===== 7. Procedure — Generate AI Monitoring Performance Report =====+===== 6. Procedure — Generate AI Monitoring Performance Report =====
  
 Use this report to evaluate the performance of one or more AI assets across selected evaluators and methods, over a specific reporting period. Use this report to evaluate the performance of one or more AI assets across selected evaluators and methods, over a specific reporting period.
  
-==== 7.1 Open the Report ====+==== 6.1 Open the Report ====
  
   - From the Power BI Home page (after Section 6), click **AI Monitoring Performance Report** under Recommended or Recent.   - From the Power BI Home page (after Section 6), click **AI Monitoring Performance Report** under Recommended or Recent.
   - The report opens on the **Cover** page with the Filters pane on the right.   - The report opens on the **Cover** page with the Filters pane on the right.
  
-{{:affirm:affirm_perf_cover.png?700|Figure 4. AI Monitoring Performance Report — Cover page with Filters pane.}} +{{ :wiki:ai:pic4.png?600 |}}
-//Figure 4. AI Monitoring Performance Report — Cover page with Filters pane.//+
  
-==== 7.2 Apply the Date Filter ====+==== 6.2 Apply the Date Filter ====
  
 The **Date** filter controls the reporting window. Power BI offers four filter types; pick the one that best fits the request. The **Date** filter controls the reporting window. Power BI offers four filter types; pick the one that best fits the request.
  
-=== 7.2.1 Filter Type — Advanced Filtering ===+=== 6.2.1 Filter Type — Advanced Filtering ===
  
 Use when the requestor provides explicit start and end dates. Use when the requestor provides explicit start and end dates.
Line 127: Line 121:
   - Click **Apply filter**.   - Click **Apply filter**.
  
-{{:affirm:affirm_filter_date_advanced.png?350|Figure 5. Advanced filtering.}} +{{ :wiki:ai:pic5.png?400 |}}
-//Figure 5. Advanced filtering — example: 'is on or after 01-04-2026 AND is on or before 30-04-2026'.//+
  
-=== 7.2.2 Filter Type — Basic Filtering ===+=== 6.2.2 Filter Type — Basic Filtering ===
  
 Use when the requestor needs to cherry-pick specific calendar dates. Use when the requestor needs to cherry-pick specific calendar dates.
Line 137: Line 130:
   - Tick the individual dates required. Use **Select all** to include every date in the dataset.   - Tick the individual dates required. Use **Select all** to include every date in the dataset.
  
-{{:affirm:affirm_filter_date_basic.png?400|Figure 6. Basic filtering — select individual calendar dates.}} +{{ :wiki:ai:pic6.png?400 |}}
-//Figure 6. Basic filtering — select individual calendar dates.//+
  
-=== 7.2.3 Filter Type — Relative Date ===+=== 6.2.3 Filter Type — Relative Date ===
  
 Use for rolling windows that should auto-shift each time the report is regenerated. Use for rolling windows that should auto-shift each time the report is regenerated.
Line 151: Line 143:
   - Click **Apply filter**.   - Click **Apply filter**.
  
-{{:affirm:affirm_filter_date_relative.png?400|Figure 7. Relative date — 'is in the last 30 days, Include today'.}} +{{ :wiki:ai:pic7.png?400 |}}
-//Figure 7. Relative date — 'is in the last 30 days, Include today'.//+
  
-=== 7.2.4 Filter Type — Relative Time ===+=== 6.2.4 Filter Type — Relative Time ===
  
 Use only for intra-day reports. Resolution is hours or minutes. Use only for intra-day reports. Resolution is hours or minutes.
Line 164: Line 155:
   - Click **Apply filter**.   - Click **Apply filter**.
  
-{{:affirm:affirm_filter_time_relative.png?400|Figure 8. Relative time.}} +{{ :wiki:ai:pic8.png?400 |}}
-//Figure 8. Relative time — 'is in the last N hours/minutes'.//+
  
 <note warning>**Date filter caveat:** Mixing filter types on the same field is not supported in a single render. If you switch from Advanced to Basic, the previous values are cleared. Confirm the date range with the requestor before switching.</note> <note warning>**Date filter caveat:** Mixing filter types on the same field is not supported in a single render. If you switch from Advanced to Basic, the previous values are cleared. Confirm the date range with the requestor before switching.</note>
  
-==== 7.3 Apply the Organization Name Filter ====+==== 6.3 Apply the Organization Name Filter ====
  
 The **Organization Name** identifies the client whose data is being reported. The document uses "CDW Internal" as the example client. The **Organization Name** identifies the client whose data is being reported. The document uses "CDW Internal" as the example client.
Line 177: Line 167:
   - If "(Blank)" is shown, leave it unchecked unless the requestor explicitly asks for it.   - If "(Blank)" is shown, leave it unchecked unless the requestor explicitly asks for it.
  
-{{:affirm:affirm_filter_organization.png?400|Figure 9. Organization Name filter.}} +{{ :wiki:ai:pic9.png?400 |}}
-//Figure 9. Organization Name — example with 'CDW Internal' selected.//+
  
 <note warning>**Always confirm exactly one organization is selected.** Selecting multiple organizations will mix client data into the same report. This is a data-confidentiality violation. Reviewer must verify in Section 10.</note> <note warning>**Always confirm exactly one organization is selected.** Selecting multiple organizations will mix client data into the same report. This is a data-confidentiality violation. Reviewer must verify in Section 10.</note>
  
-==== 7.4 Apply the AI Asset Name Filter ====+==== 6.4 Apply the AI Asset Name Filter ====
  
 Select the AI Assets in scope. The report renders separate views per asset selected. Select the AI Assets in scope. The report renders separate views per asset selected.
Line 189: Line 178:
   - Tick the assets in scope, or use **Select all**. Available assets include Azure MS Cloud Chatbot, CloudGenie - Chat, CloudGenie - KnowledgeBase, Jenkins AI Copilot, Onboarding Guide, SOW Analyzer, etc.   - Tick the assets in scope, or use **Select all**. Available assets include Azure MS Cloud Chatbot, CloudGenie - Chat, CloudGenie - KnowledgeBase, Jenkins AI Copilot, Onboarding Guide, SOW Analyzer, etc.
  
-{{:affirm:affirm_filter_ai_asset.png?400|Figure 10. AI Asset Name filter.}} +{{ :wiki:ai:pic10.png?400 |}}
-//Figure 10. AI Asset Name — pick one or more assets, or Select all.//+
  
-==== 7.5 Apply the Evaluator Name Filter ====+==== 6.5 Apply the Evaluator Name Filter ====
  
 Choose which evaluators (metric families) should appear in the report. Choose which evaluators (metric families) should appear in the report.
Line 199: Line 187:
   - Tick the evaluators in scope, or use **Select all**. Typical evaluators: Groundedness, Hallucination, Prompt Relevance, Response Relevance, Readability, Latency.   - Tick the evaluators in scope, or use **Select all**. Typical evaluators: Groundedness, Hallucination, Prompt Relevance, Response Relevance, Readability, Latency.
  
-{{:affirm:affirm_filter_evaluator.png?400|Figure 11. Evaluator Name filter.}} +{{ :wiki:ai:pic11.png?400 |}}
-//Figure 11. Evaluator Name — example with all evaluators selected.//+
  
-==== 7.6 Apply the Method Name Filter ====+==== 6.6 Apply the Method Name Filter ====
  
 Each evaluator can be computed by one or more methods. For example, Readability supports FRES and SMOG. Each evaluator can be computed by one or more methods. For example, Readability supports FRES and SMOG.
Line 209: Line 196:
   - Tick the methods in scope, or use **Select all**. Available methods include Fluency, FRES, LLMaJ, Parallel, Sequential, SMOG.   - Tick the methods in scope, or use **Select all**. Available methods include Fluency, FRES, LLMaJ, Parallel, Sequential, SMOG.
  
-{{:affirm:affirm_filter_method.png?400|Figure 12. Method Name filter.}} +{{ :wiki:ai:pic12.png?400 |}}
-//Figure 12. Method Name — choose the methods that match the requested evaluators.//+
  
 <note>**Method–Evaluator pairing:** Some methods only apply to specific evaluators (e.g., FRES and SMOG only apply to Readability). Selecting an irrelevant method has no effect on the report but adds clutter to the filter audit trail. Pick only the methods aligned to the chosen evaluators.</note> <note>**Method–Evaluator pairing:** Some methods only apply to specific evaluators (e.g., FRES and SMOG only apply to Readability). Selecting an irrelevant method has no effect on the report but adds clutter to the filter audit trail. Pick only the methods aligned to the chosen evaluators.</note>
  
-==== 7.7 Review the Rendered Report ====+==== 6.7 Review the Rendered Report ====
  
   - Wait for all visuals on every report page to finish rendering. The status bar should display **Updated**.   - Wait for all visuals on every report page to finish rendering. The status bar should display **Updated**.
Line 220: Line 206:
   - Visually confirm: (a) date range in the cover matches the requested window, (b) organization name is correct, (c) selected evaluators and methods appear on the details pages.   - Visually confirm: (a) date range in the cover matches the requested window, (b) organization name is correct, (c) selected evaluators and methods appear on the details pages.
  
-==== 7.8 Export the Report ====+==== 6.8 Export the Report ====
  
   - Click the **download / export** icon in the top toolbar (highlighted in red in Figure 13 below).   - Click the **download / export** icon in the top toolbar (highlighted in red in Figure 13 below).
Line 227: Line 213:
   - Save the file using the naming convention defined in Section 9.   - Save the file using the naming convention defined in Section 9.
  
-{{:affirm:affirm_perf_export.png?700|Figure 13. Export the report.}} +{{ :wiki:ai:pic13.png?600 |}}
-//Figure 13. Export the report — the export icon is highlighted at the top-right of the toolbar.//+
  
-===== 8. Procedure — Generate AI Health Report =====+===== 7. Procedure — Generate AI Health Report =====
  
 The AI Health Report shares the Organization / AI Asset / Evaluator / Date filter pattern with the Performance Report, plus two additional filters specific to health monitoring: **Trending Days** and **is_normalized**. The AI Health Report shares the Organization / AI Asset / Evaluator / Date filter pattern with the Performance Report, plus two additional filters specific to health monitoring: **Trending Days** and **is_normalized**.
  
-==== 8.1 Open the Report ====+==== 7.1 Open the Report ====
  
   - From the Power BI Home page (after Section 6), click **AI Health Report - CDW**.   - From the Power BI Home page (after Section 6), click **AI Health Report - CDW**.
   - The report opens on the **Cover** page with the Filters pane on the right.   - The report opens on the **Cover** page with the Filters pane on the right.
  
-==== 8.2 Apply Common Filters ====+==== 7.2 Apply Common Filters ====
  
 Apply the following filters in order, using the same procedures as for the AI Monitoring Performance Report: Apply the following filters in order, using the same procedures as for the AI Monitoring Performance Report:
  
-  * **Organization** (or Organization(s)) — see Section 7.3. +  * **Organization** (or Organization(s)) — see Section 6.3. 
-  * **AI Asset** (or AI Asset(s)) — see Section 7.4. +  * **AI Asset** (or AI Asset(s)) — see Section 6.4. 
-  * **Evaluators** — see Section 7.5. +  * **Evaluators** — see Section 6.5. 
-  * **Report Date Range** — see Section 7.2 (use Advanced or Relative date as appropriate).+  * **Report Date Range** — see Section 6.2 (use Advanced or Relative date as appropriate).
  
-==== 8.3 Set Trending Days ====+==== 7.3 Set Trending Days ====
  
-**Trending Days** defines the lookback window used to compute the **Trend %** column shown in the AI Health Snapshots table. The default value is 7.+**Trending Days** defines the lookback window used to compute the **Trend %** column shown in the AI Health Snapshots table. The default value is 0.
  
   - Expand **Trending Days** in the Filters pane.   - Expand **Trending Days** in the Filters pane.
Line 258: Line 243:
   - Click **Apply filter**.   - Click **Apply filter**.
  
-{{:affirm:affirm_filter_trending_days.png?350|Figure 14. Trending Days.}} +{{ :wiki:ai:pic14.png?400 |}}
-//Figure 14. Trending Days — set to 7 for a weekly trend window.//+
  
 <note>**Choosing Trending Days:** Pick a window proportional to the report's date range and the volatility of the metric. 7 days suits high-volume daily monitoring; 30 days suits low-volume monthly review. Avoid windows longer than the date range itself — the trend will be meaningless.</note> <note>**Choosing Trending Days:** Pick a window proportional to the report's date range and the volatility of the metric. 7 days suits high-volume daily monitoring; 30 days suits low-volume monthly review. Avoid windows longer than the date range itself — the trend will be meaningless.</note>
  
-==== 8.4 Set is_normalized ====+==== 7.4 Set is_normalized ====
  
 **is_normalized** determines whether the **Actual Normalized Score** column on the AI Health Snapshots page uses normalized values. **is_normalized** determines whether the **Actual Normalized Score** column on the AI Health Snapshots page uses normalized values.
Line 270: Line 254:
   - Tick **True** to show normalized scores, **False** to show raw scores, or both for comparison.   - Tick **True** to show normalized scores, **False** to show raw scores, or both for comparison.
  
-{{:affirm:affirm_filter_is_normalized.png?300|Figure 15. is_normalized filter.}} +{{ :wiki:ai:pic15.png?400 |}}
-//Figure 15. is_normalized — choose True, False, or both.//+
  
 <note>**When to use normalized scores:** Use normalized scores (True) when comparing different evaluators or metrics that have different native scales (e.g., comparing Latency in milliseconds against Readability scores). Use raw scores (False) when the requestor needs the original measurement values.</note> <note>**When to use normalized scores:** Use normalized scores (True) when comparing different evaluators or metrics that have different native scales (e.g., comparing Latency in milliseconds against Readability scores). Use raw scores (False) when the requestor needs the original measurement values.</note>
  
-==== 8.5 Review the AI Health Snapshots Table ====+==== 7.5 Review the AI Health Snapshots Table ====
  
 Navigate to the **2- AI Health Snapshots** page. The table shows one row per Evaluator × Method × Data Type combination. Navigate to the **2- AI Health Snapshots** page. The table shows one row per Evaluator × Method × Data Type combination.
  
-{{:affirm:affirm_health_snapshots.png?700|Figure 16. AI Health Snapshots.}} +{{ :wiki:ai:pic16.png?400 |}}
-//Figure 16. AI Health Snapshots — example output for AI Asset 'CloudGenie - Chat'.//+
  
 Columns to verify: Columns to verify:
Line 291: Line 273:
   * **Trend % (N Day(s))** — directional change vs. baseline. Green up-arrow indicates improvement; red down-arrow indicates degradation.   * **Trend % (N Day(s))** — directional change vs. baseline. Green up-arrow indicates improvement; red down-arrow indicates degradation.
  
-==== 8.6 Review the Cover and Evaluator Group Pages ==== 
  
-  - Confirm the **Cover** page shows the correct Date Range, Organization, and AI Asset(s). +==== 7.6 Export the Report ====
-  - Page through **3- Evaluator_Group 1** to **17- Evaluator_Group 15** to verify each evaluator group renders without errors.+
  
-{{:affirm:affirm_health_cover.png?700|Figure 17. AI Health Report — Cover page with full filter pane.}} +Follow the same export procedure as Section 6.8. PDF is the default delivery format unless the requestor specifies otherwise.
-//Figure 17. AI Health Report — Cover page with full filter pane (Organization, AI Asset, Evaluators, Report Date Range, Trending Days, is_normalized).//+
  
-==== 8.7 Export the Report ====+==== 8. Affirm Support Details ==== 
 +Following are the details of Affirm support, in case of any Power BI Access Issue or Technical issue. 
 +  * **Teams Channel:** CDW AFFIRM Support Chat
  
-Follow the same export procedure as Section 7.8. PDF is the default delivery format unless the requestor specifies otherwise. 
  
-===== 9. Exporting & Delivering the Report ===== 
- 
-==== 9.1 File Naming Convention ==== 
- 
-Use the following pattern for every exported file: 
- 
-<code>{Organization}_{ReportType}_{StartDate}_{EndDate}_v{N}.{ext}</code> 
- 
-**Example:** ''CDWInternal_AIMonPerf_2026-04-01_2026-04-30_v1.pdf'' 
- 
-  * Use ISO date format (YYYY-MM-DD) for sortability. 
-  * Increment the version (v1, v2, …) only when re-exporting the same report for the same period. 
-  * Avoid spaces and special characters in file names. 
- 
-==== 9.2 Storage Location ==== 
- 
-  * Save the exported file to the shared client delivery folder under the client's subfolder (e.g., ''/ClientDelivery/CDW Internal/2026-04/''). 
-  * Do not store exported reports on personal drives or local desktops. 
- 
-==== 9.3 Delivery ==== 
- 
-  - Hand the file to the Reviewer / QA along with a link to the original report request. 
-  - After Reviewer sign-off, Client Delivery sends the report to the client via the agreed channel (secure email, client portal, etc.). 
-  - Log delivery in the request ticket: file path, delivery date, recipient. 
- 
-===== 10. Validation Checklist (Before Sending to Client) ===== 
- 
-The Reviewer must check every item below before approving the report for delivery. Do not skip items even if they look obvious. 
- 
-^ ✓ ^ Check Item ^ 
-| ☐ | You are in the **Synergist Technologies, LLC** tenant (top-left brand should match). | 
-| ☐ | Exactly **one** Organization is selected, and it matches the ticket. | 
-| ☐ | The Date / Date Range shown on the Cover page matches the requested period. | 
-| ☐ | AI Asset(s) selected match the ticket. | 
-| ☐ | Evaluators and Methods selected match the ticket. | 
-| ☐ | All report pages have rendered fully — no "Working on it…" spinners or blank visuals. | 
-| ☐ | AI Health only: Trending Days value and is_normalized selection match the ticket. | 
-| ☐ | No filter warning banner (e.g., "Filter affects only this page" applied where global was intended). | 
-| ☐ | Exported file opens cleanly in the target application (Adobe Reader / PowerPoint / Excel). | 
-| ☐ | File name follows the convention in Section 9.1. | 
-| ☐ | File has been saved to the client delivery folder, not to a personal drive. | 
- 
-===== 11. Troubleshooting ===== 
- 
-^ Symptom ^ Resolution ^ 
-| Power BI link opens but the report is not visible. | Confirm you switched to **Synergist Technologies, LLC** (Section 6.2). If the tenant option is missing, contact the Tenant Administrator to verify your guest access. | 
-| Filters pane is empty or missing. | Click the **Filters** chevron (>) on the right edge of the report canvas. If still missing, hard-refresh the browser with **Ctrl+F5**. | 
-| Date filter shows no data. | The selected date range may be outside the dataset's refresh window. Check the **Data updated** timestamp at the top of the report; the latest data is bounded by that timestamp. | 
-| Trend % column shows blanks. | Trending Days is greater than the date range. Reduce Trending Days or extend the date range so the baseline window fits inside the dataset. | 
-| Export to PDF cuts off the right edge. | Open the report's **File → Export to PDF** using the "Current view" option, not "Active page". Alternatively, export to PowerPoint, then save as PDF. | 
-| "License upgrade required" banner. | Your Power BI license is Free. Request a Pro or PPU license in the Affirm tenant from the Tenant Administrator. | 
-| Report shows different data on refresh. | The underlying dataset was refreshed between renderings. Note the **Data updated** timestamp in the file name suffix so the delivered file is reproducible. | 
-| Multiple organizations appear selected. | **Stop.** Clear all filters using the eraser icon at the top of the Filters pane, then re-apply Section 7.3 carefully. Do not export until exactly one organization is selected. | 
- 
-===== 12. Recommended Practices & Tips ===== 
- 
-  * Always **clear all filters** (using the eraser icon at the top of the Filters pane) before starting a new client's report. Stale filters from a previous session are the #1 source of mis-delivered reports. 
-  * Apply filters in the order listed in this SOP — Date → Organization → AI Asset → Evaluator → Method — to minimize report re-renders and to keep an auditable trail. 
-  * Take a screenshot of the Filters pane before exporting. Attach it to the request ticket as evidence of the filter set used. 
-  * If the requestor is unclear about Evaluator or Method, default to **Select all** and discuss the resulting report with them, rather than guessing. 
-  * Lock the date range with explicit start/end dates when delivering to a client. Avoid **Relative date** for client deliverables — relative filters produce different numbers on re-run. 
-  * Use **Relative date** for internal recurring monitoring views where the most recent data is wanted on each open. 
-  * Run the report twice in two browser windows when verifying parity (e.g., comparing two date ranges). Use Edge in one window and Chrome in another to detect any browser-specific rendering issues. 
-  * Document any filter combination that produces unexpected results and escalate to Data Engineering with a screenshot and the exact filter set. 
-  * Never share screenshots or exported reports outside the agreed delivery channel. Both reports contain client-confidential AI performance data. 
- 
-===== 13. Revision History ===== 
- 
-^ Version ^ Date         ^ Author                      ^ Changes ^ 
-| 1.0     | Prior to 2026 | Reporting Ops              | Initial draft of the Affirm Power BI report generation procedure. | 
-| 2.0     | 15 May 2026  | Reporting Ops (Enhanced)    | Restructured as a full SOP: added Purpose, Scope, Roles, Definitions, Prerequisites, Validation Checklist, Troubleshooting, and Recommended Practices. Re-illustrated using existing screenshots. | 
- 
----- 
  
-//Powered by Synergist Technology  •  Internal// 
  
  
wiki/ai/affirm.1778861211.txt.gz · Last modified: by sramasamy

Page Tools

Trace: • ncp-aio-lab-training • gpu_firmware_upgrade • affirm
Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com) IP:10.0.157.8