Which tools can test an AI chatbot? A comparison
For developers, open-source tools such as promptfoo, garak, PyRIT or Giskard let you script attacks and evaluations of AI models and applications. For a small business or a no-code agency the need is different: test the live bot without writing code, understand the result and get fixes; that is what BotArmor does. Manual testing is still useful for a first look.
Updated 25 October 2026
The comparison
| Tool | For whom | What it does | Good to know |
|---|---|---|---|
| Manual testing | Everyone | You play the difficult customer yourself. | Free and telling, but slow, hard to repeat and subjective. |
| promptfoo | Developers | Open-source command-line tool to evaluate and attack prompts and AI applications, configured in YAML. | You write and maintain the test configuration. |
| garak | Technical teams | Open-source vulnerability scanner for language models (injections, leaks, toxic content). | Model- and endpoint-oriented, command line. |
| PyRIT (Microsoft) | Security and AI teams | Open-source Python library to automate red teaming of generative AI. | Requires code and expertise. |
| Giskard | Data scientists, product teams | Testing tools for AI applications (hallucinations, injections, bias), open source and enterprise. | Built for the teams that develop the application. |
| BotArmor | Small businesses, no-code agencies (Make, n8n, Voiceflow, Paperclip) | Attacks the live bot through its webhook or the Voiceflow API: 13 levels, A–F grade, copy-paste fixes, signed report. | Focused on customer service and the AI Act; not an infrastructure or code audit. |
How to choose
- You build an AI application in Python or TypeScript: an open-source tool fits into your code and your CI.
- You deliver chatbots on Make, n8n or Voiceflow, or you have a bot on your website: you need to test the real bot without code and hand over an understandable report.
- Either way, test after every prompt change, over several conversations.
Would your chatbot hold up?
Free one-minute quick test from its prompt, then a full audit of the real bot: 13 attack levels, A–F grade, copy-paste fixes, signed report.
Test my chatbot for free →FAQ
Are these tools competitors?
They complement each other: open-source tools serve teams that code, BotArmor serves businesses and agencies that want a verdict and fixes on the bot in production.
Is BotArmor free?
The quick test is free. During the beta every new account gets 3 free audits; after that, 1 audit costs €9.