Standard Operating Procedure for generating AI Health and AI Monitoring Performance reports from the Affirm Power BI tenant.

1. Purpose
2. Scope
3. Definitions & Glossary
4. Prerequisites
5. Procedure — Access the Affirm Power BI Tenant
6. Procedure — Generate AI Monitoring Performance Report
7. Procedure — Generate AI Health Report
8. Exporting & Delivering the Report
9. Validation Checklist (Before Sending to Client)
10. Troubleshooting
11. Recommended Practices & Tips
12. Revision History

This Standard Operating Procedure (SOP) defines the end-to-end steps for generating, validating, and delivering Power BI reports for Affirm clients from the Affirm Power BI tenant. It covers two report templates published under the Synergist Technologies, LLC workspace:

• AI Health Report — long-range health snapshots, trend analysis, and normalized scoring for AI assets.
• AI Monitoring Performance Report — performance evaluation of AI assets across configurable evaluators and methods.

This document is intended to standardize the report generation process so that any qualified operator can produce consistent, accurate, audit-ready outputs without ad-hoc decisions.

• Applies to all Power BI report generation performed against the Affirm tenant for internal or external Affirm clients.
• Covers ad-hoc report generation triggered by client request, scheduled delivery cycles, or internal QA review.

Before starting this procedure, confirm ALL of the following are in place. If any item is missing, stop and contact the Tenant Administrator.

• Client / organization name (e.g., CDW Internal, CDW Corporation, Integration).
• Reporting period (start and end date).
• AI Asset(s) in scope (or “All”).
• Evaluator(s) and Method(s) in scope (or “All”).
• For AI Health Report only: Trending Days window and whether normalized scoring is required.

This section is common to both reports. Complete steps 6.1 through 6.4 before moving to Section 7 or Section 8.

1. Open a supported browser and navigate to the Affirm Power BI workspace URL:
   Affirm Power BI Report Link
2. Sign in with your corporate credentials if prompted. Complete MFA if required.

<note important>Important: If the link opens but reports are not visible, you are still in your default tenant. Continue to step 6.2 to switch tenants.</note>

1. Click your profile picture in the top-right corner of Power BI.
2. In the profile flyout, click Switch tenant.

1. In the Switch tenant dialog, select Synergist Technologies, LLC from the dropdown.
2. Click Switch. The Power BI Home page reloads in the Affirm tenant context.

On the Home page you should see the following two reports listed under the Recommended section and in the Recent list:

• AI Health Report - CDW
• AI Monitoring Performance Report

<note>Both reports use the same filter pattern.
AI Health Report and AI Monitoring Performance Report share a near-identical filter pane: Organization Name, AI Asset Name, Evaluator Name, Method Name, and a Date / Date Range filter. Mastering the filter pane once applies to both reports. The differences are summarized in Section 7 (performance) and Section 8 (health).</note>

Use this report to evaluate the performance of one or more AI assets across selected evaluators and methods, over a specific reporting period.

1. From the Power BI Home page (after Section 6), click AI Monitoring Performance Report under Recommended or Recent.
2. The report opens on the Cover page with the Filters pane on the right.

The Date filter controls the reporting window. Power BI offers four filter types; pick the one that best fits the request.

Use when the requestor provides explicit start and end dates.

1. In the Filters pane, expand Date.
2. Set Filter type to Advanced filtering.
3. Configure Show items when the value (top condition). Operators available: is, is not, is after, is on or after, is before, is on or before, is blank, is not blank.
4. Choose And to combine with a second condition (typically is on or before for the end date).
5. Click Apply filter.

Use when the requestor needs to cherry-pick specific calendar dates.

1. Set Filter type to Basic filtering.
2. Tick the individual dates required. Use Select all to include every date in the dataset.

Use for rolling windows that should auto-shift each time the report is regenerated.

1. Set Filter type to Relative date.
2. Choose the operator: is in the last, is in this, or is in the next.
3. Enter the numeric value (e.g., 30).
4. Choose the unit: days, weeks, months, years, calendar months, or calendar years.
5. Tick Include today if the current day must be included.
6. Click Apply filter.

Use only for intra-day reports. Resolution is hours or minutes.

1. Set Filter type to Relative time.
2. Choose the operator: is in the last, is in this, or is in the next.
3. Enter the numeric value (1 to 10000).
4. Choose the unit: hours or minutes.
5. Click Apply filter.

<note warning>Date filter caveat: Mixing filter types on the same field is not supported in a single render. If you switch from Advanced to Basic, the previous values are cleared. Confirm the date range with the requestor before switching.</note>

The Organization Name identifies the client whose data is being reported. The document uses “CDW Internal” as the example client.

1. Expand Organization Name in the Filters pane.
2. Tick the single organization in scope. Untick any others.
3. If “(Blank)” is shown, leave it unchecked unless the requestor explicitly asks for it.

<note warning>Always confirm exactly one organization is selected. Selecting multiple organizations will mix client data into the same report. This is a data-confidentiality violation. Reviewer must verify in Section 10.</note>

Select the AI Assets in scope. The report renders separate views per asset selected.

1. Expand AI Asset Name and set Filter type to Basic filtering.
2. Tick the assets in scope, or use Select all. Available assets include Azure MS Cloud Chatbot, CloudGenie - Chat, CloudGenie - KnowledgeBase, Jenkins AI Copilot, Onboarding Guide, SOW Analyzer, etc.

Choose which evaluators (metric families) should appear in the report.

1. Expand Evaluator Name and set Filter type to Basic filtering.
2. Tick the evaluators in scope, or use Select all. Typical evaluators: Groundedness, Hallucination, Prompt Relevance, Response Relevance, Readability, Latency.

Each evaluator can be computed by one or more methods. For example, Readability supports FRES and SMOG.

1. Expand Method Name and set Filter type to Basic filtering.
2. Tick the methods in scope, or use Select all. Available methods include Fluency, FRES, LLMaJ, Parallel, Sequential, SMOG.

<note>Method–Evaluator pairing: Some methods only apply to specific evaluators (e.g., FRES and SMOG only apply to Readability). Selecting an irrelevant method has no effect on the report but adds clutter to the filter audit trail. Pick only the methods aligned to the chosen evaluators.</note>

1. Wait for all visuals on every report page to finish rendering. The status bar should display Updated.
2. Navigate through the pages in the left-hand Pages pane: Cover, 1- Executive, 2- Evaluator Snapshots, 3- Details (1) … 6- Details (4), Legal Disclaimer, Glossary 1–3.
3. Visually confirm: (a) date range in the cover matches the requested window, (b) organization name is correct, © selected evaluators and methods appear on the details pages.

1. Click the download / export icon in the top toolbar (highlighted in red in Figure 13 below).
2. Choose the requested format: PDF (recommended for client delivery), PowerPoint, or Excel data.
3. Wait for the export job to finish. Power BI shows a notification with a download link.
4. Save the file using the naming convention defined in Section 9.

The AI Health Report shares the Organization / AI Asset / Evaluator / Date filter pattern with the Performance Report, plus two additional filters specific to health monitoring: Trending Days and is_normalized.

1. From the Power BI Home page (after Section 6), click AI Health Report - CDW.
2. The report opens on the Cover page with the Filters pane on the right.

Apply the following filters in order, using the same procedures as for the AI Monitoring Performance Report:

• Organization (or Organization(s)) — see Section 6.3.
• AI Asset (or AI Asset(s)) — see Section 6.4.
• Evaluators — see Section 6.5.
• Report Date Range — see Section 6.2 (use Advanced or Relative date as appropriate).

Trending Days defines the lookback window used to compute the Trend % column shown in the AI Health Snapshots table. The default value is 0.

1. Expand Trending Days in the Filters pane.
2. Set Filter type to Advanced filtering.
3. Set Show items when the value to is, then enter the desired number of days (e.g., 7, 14, 30).
4. Leave the And/Or row empty unless a second condition is required.
5. Click Apply filter.

<note>Choosing Trending Days: Pick a window proportional to the report's date range and the volatility of the metric. 7 days suits high-volume daily monitoring; 30 days suits low-volume monthly review. Avoid windows longer than the date range itself — the trend will be meaningless.</note>

is_normalized determines whether the Actual Normalized Score column on the AI Health Snapshots page uses normalized values.

1. Expand is_normalized in the Filters pane.
2. Tick True to show normalized scores, False to show raw scores, or both for comparison.

<note>When to use normalized scores: Use normalized scores (True) when comparing different evaluators or metrics that have different native scales (e.g., comparing Latency in milliseconds against Readability scores). Use raw scores (False) when the requestor needs the original measurement values.</note>

Navigate to the 2- AI Health Snapshots page. The table shows one row per Evaluator × Method × Data Type combination.

Columns to verify:

• Evaluator and Method — match the filters applied.
• Group — semantic grouping (e.g., Semantic Quality, Relevance & Coverage, Linguistic Quality, Efficiency, Harmful Content).
• Data Type — Manual or Generated.
• Actual Normalized Score — the score for the period (highlighted in Figure 16).
• Baseline Start Date / Baseline End Date — the reference window for trend computation.
• Trend % (N Day(s)) — directional change vs. baseline. Green up-arrow indicates improvement; red down-arrow indicates degradation.

1. Confirm the Cover page shows the correct Date Range, Organization, and AI Asset(s).
2. Page through 3- Evaluator_Group 1 to 17- Evaluator_Group 15 to verify each evaluator group renders without errors.

Figure 17. AI Health Report — Cover page with full filter pane (Organization, AI Asset, Evaluators, Report Date Range, Trending Days, is_normalized).

Follow the same export procedure as Section 6.8. PDF is the default delivery format unless the requestor specifies otherwise.

• Always clear all filters (using the eraser icon at the top of the Filters pane) before starting a new client's report. Stale filters from a previous session are the #1 source of mis-delivered reports.
• Apply filters in the order listed in this SOP — Date → Organization → AI Asset → Evaluator → Method — to minimize report re-renders and to keep an auditable trail.
• Take a screenshot of the Filters pane before exporting. Attach it to the request ticket as evidence of the filter set used.
• If the requestor is unclear about Evaluator or Method, default to Select all and discuss the resulting report with them, rather than guessing.
• Lock the date range with explicit start/end dates when delivering to a client. Avoid Relative date for client deliverables — relative filters produce different numbers on re-run.
• Use Relative date for internal recurring monitoring views where the most recent data is wanted on each open.
• Run the report twice in two browser windows when verifying parity (e.g., comparing two date ranges). Use Edge in one window and Chrome in another to detect any browser-specific rendering issues.
• Document any filter combination that produces unexpected results and escalate to Data Engineering with a screenshot and the exact filter set.
• Never share screenshots or exported reports outside the agreed delivery channel. Both reports contain client-confidential AI performance data.

Powered by Synergist Technology • Internal