• Give explanation for why each monitoring test failed – 'Toxicity test failed due to these words:'
• Functionality for running multiple initial monitoring tests at once or creating a queue.
• Ability to evaluate models against the framework packs, rather than a simple checklist
• Key vault within the tool to store model secrets, reference the secrets when onboarding model
• Automatically pull the AI Model inventory from AWS/Azure/GCP instead of manually onboarding models
• Variety in test prompts so that the same exact prompt is not used every time to yield the same exact result – shouldn’t need to manually create/add every prompt.
• Need more automation in general - manually onboarding models, manually designing prompts, etc. is too much

• 'Create Report' button didn’t always work in AI Models section
• In 'AI Monitoring' → 'Audit Logs': User and User ID are both random character strings
• 'View Audit Log' in the test results: the screen would be blank
• Sending the same prompt over and over would likely lead to the same results

Affirm Knowledge