Before You Buy an AI Tool

25 Questions Behavioral Health Leaders Should Ask

Cody Saunders, LMSW · Responsible AI for Behavioral Health

Use this checklist with sellers and your team. Record each answer, proof, who checks it, and the next step.

Purpose & fit

  1. What task will this tool help us do?

    Choose one clear task, such as drafting care notes. Measure the time it takes now. Set a goal before you buy.

    Ask for: A task, a starting measure, and a goal.

  2. Does it fit our team and clients?

    Test the tool in your care setting. Include the languages and people you serve. A sales demo alone does not show a good fit.

    Ask for: Tests and examples that match your work.

  3. What should we use it for?

    List allowed tasks and uses to avoid. Staff should know when to use AI and when to use their usual process.

    Ask for: A list of uses, limits, and tasks to avoid.

  4. What proof shows that it works?

    Ask how the tool was tested and who took part. Read results from outside groups when you can. Keep test results separate from sales claims.

    Ask for: Test methods, results, and who was tested.

  5. How will we know it helps?

    Track time saved, fixes, care quality, staff views, and costs. Set goals for a small trial before buying.

    Ask for: A trial plan with clear goals.

Data, privacy & security

  1. Who owns our data and records?

    Put data rights in the contract. Include data you enter, results, notes, and new data made from them. State how you can view and take out your records.

    Ask for: Written terms for each type of data.

  2. Where does our data go?

    List the data types, where they are stored, and who uses them. Include other firms the seller hires. Give the map to your privacy and security teams.

    Ask for: A data map and a list of firms involved.

  3. How long is data kept?

    Set how long recordings, requests, results, logs, and backups stay. Confirm how they will be deleted, even after you stop using the tool.

    Ask for: A written schedule and deletion terms.

  4. Can the seller use our data to train AI?

    Ask if the seller reuses data, results, or logs to improve AI. Decide what you allow. Put clear limits in the contract.

    Ask for: Written data-use rules and settings.

  5. What privacy rules and contracts apply?

    Ask your privacy and legal teams which rules apply. Some uses need a business associate agreement, called a BAA. Get it in place before use if required. A BAA alone does not complete the review.

    Ask for: Required contracts and your team’s review.

  6. How does the tool keep data safe?

    Check who can sign in and see data. Review encryption, which protects data by coding it. Check security tests and support access. Name the tasks your own team must do.

    Ask for: Security reports, access rules, and each team’s duties.

Care quality & human review

  1. How do we find and fix wrong results?

    Test examples from real types of work. Make sure staff can spot and fix mistakes. Set how staff report them and how the seller will help.

    Ask for: Sample results and a clear process for fixes.

  2. Does it work well for different people?

    Ask for tests by language and group. Check ease of use for people with disabilities. One overall score may hide differences.

    Ask for: Group and language tests, access tests, and known limits.

  3. Who checks the AI’s work?

    Name who checks and signs off before results go into care or records. Give staff time to review. Teach each person their role.

    Ask for: A review process with named roles.

  4. Can staff change or stop its work?

    Staff need ways to edit, reject, or stop results. Keep another way to do the task. Test that process for outages or pauses.

    Ask for: A demo of controls and a backup process.

  5. What will clients be told?

    Explain why you use AI, what data it uses, and what choices clients have. Ask the right reviewers which notice and consent rules apply.

    Ask for: Approved client information and consent steps.

Tasks & staff

  1. Does it work with our health record system?

    Test links to your electronic health record, called an EHR. Check what data goes in and out. Name who sets up and maintains the link.

    Ask for: A full test of the links and clear setup duties.

  2. Does it save time after we check the work?

    Measure the full task, including setup, fixes, and repeat typing. Ask staff what helps and what adds work.

    Ask for: Task times before and after use, plus staff views.

  3. What training and help will staff get?

    Plan lessons, examples, refreshers, and help channels. Give staff time to learn. Make sure leads know how to support them.

    Ask for: A training plan and named support contacts.

  4. What is the full cost?

    Count fees, use limits, setup, links to other tools, training, review time, upgrades, and exit costs. Compare that total with the results you measure.

    Ask for: All prices, cost assumptions, and contract terms.

Team rules & ongoing value

  1. Who approves the tool and leads its use?

    Name a lead. Include care, IT, privacy, security, buying, and compliance staff as needed. Record approval and allowed uses.

    Ask for: A lead, review notes, and use limits.

  2. Can we trace what the tool does?

    Check if you can track access, results, edits, and key actions. Keep logs safe. Set how long they stay.

    Ask for: A demo of logs and rules for access and storage.

  3. How will we learn about changes?

    Require notice of key changes to AI models, features, other firms, and data use. Set when your team must review the change.

    Ask for: Written notice terms and review triggers.

  4. What happens when there is a problem?

    Use your team’s usual reporting process. Name seller contacts, response times, and who can pause use. Plan how to check and fix the issue.

    Ask for: A response plan and written seller duties.

  5. How do we review value or leave the seller?

    Set dates to review results. Plan how to take out records, delete data, end the contract, and switch to another process.

    Ask for: Review dates and an exit plan.

Sources & further reading

This checklist is for learning and planning. It does not prove that a tool meets standards. Your care, legal, privacy, and security teams must still review it.

Prepared October 2, 2026 · Cody Saunders, LMSW