A new AI tool lands on your desk one of two ways. A vendor demoed it to the board and there's a "let's move on this" sitting in your inbox — or, far more often, you find out your engineers have been using it for three weeks and it's already in the pull requests. Both feel like a yes/no call under time pressure. Neither one is.
Start from the honest position: the tool is probably already inside. In GitLab's 2023 survey, 90% of respondents said they use AI in software development or plan to, and Cyberhaven, analysing 1.6 million knowledge workers, found that 11% of what employees paste into ChatGPT is confidential company data. Your people aren't waiting for your policy. The only real question is whether anyone decided to let the tool in, on terms you could defend.
The yes/no framing is the trap. Say no and you're the one slowing the company down while competitors move. Say yes and you've handed a third party access to your codebase, your customer data, or your production pipeline on the strength of a demo. The job the AI era actually hands the technology leader is neither. AI creates: it writes the code, drafts the policy, generates the design. The leader governs — not by blocking, not by rubber-stamping, but by deciding, on defensible terms, where a new capability is allowed to touch the business.
That decision is repeatable. Here is the evaluation framework I run before any AI tool gets past "interesting" — six questions, in order, fast enough to keep up with the speed these tools actually arrive at.
Stop evaluating the tool. Evaluate what it reaches.
The demo is designed to make you assess the tool. That's the wrong object. Two tools that both say "AI" can carry wildly different risk: a grammar assistant in a browser tab and an agent with commit access to your infrastructure repo are not in the same universe, even though the pitch decks look alike. So the first question isn't "is it good?" It's "what can it touch on a bad day?"
1. What does it touch? (Blast radius.) Map exactly what systems, data, and code the tool can read and write. Draw the line around its reach when it misbehaves — because it will. Everything downstream flows from this line.
2. Where does your data go — and does it train the vendor's model? This is the question the demo won't volunteer, and the one that turned into headlines. In April 2023, Samsung engineers pasted internal source code into ChatGPT to debug it and leaked it in the process; within weeks Samsung restricted the tools internally — and so did Apple, JPMorgan Chase, and Verizon, all over the same fear of confidential data walking out the door (Wall Street Journal, May 2023). The industry's own policies were already shifting before that: on 1 March 2023, OpenAI changed its policy so that data sent through its API is no longer used to train its models by default — note the boundary, that was the API, not the consumer ChatGPT product. So: "we don't train on your data" belongs in the contract, not the sales deck, and you need to know whether your content leaves your tenant at all.
3. Does it generate, or does it decide? This is the judgment boundary, and vendors blur it on purpose. A tool that drafts and waits for your approval is a multiplier — adopt it fast. A tool that decides and acts has taken over a control, and it needs a named human accountable at the point of action. The sharpest illustration is a courtroom: in June 2023, a New York lawyer was sanctioned $5,000 after filing a brief that cited six court cases ChatGPT had entirely invented, none of which he verified (Mata v. Avianca, S.D.N.Y., 2023). He didn't misuse AI; he mistook a draft for a decision. Know which one you're buying, because the vendor won't tell you.
4. When it's wrong, who catches it — a person, or a gate? AI is wrong confidently, and you will not reliably catch it by eye. A Stanford study (Perry, Srivastava, Kumar & Boneh, 2023) found that developers using an AI coding assistant wrote less secure code while being more confident it was secure. Wrong, and it feels right. So "the engineer will review it" is not a control: review happens at human speed, the tool generates faster than anyone reviews under a deadline, and the review is the first thing to get compressed. (I've argued the longer version elsewhere: safety can't live in the review.) Ask instead: what runs whether or not anyone remembers?
5. Can you leave? (Reversibility.) If the vendor doubles the price, gets acquired, or ships a breaking change next quarter, what is your exit? A tool woven through every workflow with no export and no substitute is not a purchase — it's a dependency, and one day it will be priced like one. Reversible-by-design beats best-in-class-you-can't-leave.
6. What does it actually cost? (Total cost.) Not the licence. The real cost is the integration, the review overhead it adds, the governance you now owe, and the concentration risk if it joins the pile of critical things already running on one provider. A tool that's cheap per seat can be expensive per landscape.
The three ways leaders get this wrong
The rubber stamp. The tool is impressive, everyone wants it, so it goes in — and the six questions get answered later, in an incident review. Speed without a defensible line isn't leadership; it's luck.
The blanket ban. "No AI tools until we have a policy." The tools don't wait for your policy — they arrive as shadow AI, unmanaged, and now you own all the risk and none of the visibility. Samsung, Apple and the rest didn't ban their way out of the problem; they were reacting to leaks that had already happened.
The pilot that never ends. Endless evaluation as a way to avoid deciding, while the org either stalls or routes around you. The framework exists precisely so you can decide quickly — the point is a fast yes, not a slow maybe.
Monday morning
Build one page: a tool-intake sheet that answers the six questions, filled in before any AI tool touches production, and kept in a register you could put in front of a board or an auditor without flinching. This is not bureaucracy — it's the opposite. It's what lets you say yes quickly and often, because every yes is one you can defend.
If you'd rather not build it from a blank page, I've packaged exactly this — the auto-scoring scorecard, the intake register, a decision-memo template, a worked example, and the vendor question bank — as the AI Tool Intake Kit. There's a free one-page scorecard to start with.
The leader who governs adoption this way is not the brake on the organisation. They're the reason it can move fast without driving blind — because "we move fast on AI" and "we know exactly what it touches" stop being a contradiction the moment there's a gate instead of a gut feel.
Common questions
What should you evaluate before adopting an AI tool? Not the tool — what it reaches. Map its blast radius (the data, code, and systems it can touch), where your data goes and whether it trains the vendor's model, whether it drafts or decides, who catches its mistakes, whether you can leave, and the total cost beyond the licence.
What questions should you ask an AI vendor before buying? Lead with: "Does our data leave our tenant, and is it used to train your model?" — and get the answer in the contract, not the demo. Then: what exactly can the tool read and write, does it act autonomously or wait for approval, what is the export and exit path, and what do integration and review really cost.
How do you evaluate an AI tool's security and data handling? Start from the blast radius — what it can reach on a bad day. Confirm data residency and retention, whether your inputs train the vendor's model (OpenAI, for one, stopped training on API data by default in March 2023), and require a gate that validates the tool's output automatically rather than relying on a human to remember.
What are the red flags when evaluating an AI vendor? A demo that won't say where your data goes; "we don't train on your data" offered as a verbal claim rather than a contract term; a tool that acts autonomously with no human accountable at the point of action; and no export path or substitute — lock-in dressed up as convenience.
Sources & further reading
- Global DevSecOps Report on AI Shows Cybersecurity and Privacy Concerns Create an Adoption Dilemma — GitLab, 2023
- 11% of data employees paste into ChatGPT is confidential — Cyberhaven, 2023
- Samsung Bans ChatGPT Among Employees After Sensitive Code Leak — Forbes, 2023
- Apple, Goldman Sachs, Samsung, and 10 others clamp down on ChatGPT at work — Fortune, 2023 (confirms Apple, JPMorgan Chase, and Verizon restrictions; cited in place of the original Wall Street Journal piece, whose exact URL could not be independently verified)
- How your data is used to improve model performance — OpenAI, policy page
- Addressing criticism, OpenAI will no longer use customer data to train its models by default — TechCrunch, 2023
- Do Users Write More Insecure Code with AI Assistants? — Perry, Srivastava, Kumar & Boneh, ACM CCS 2023
- Judge sanctions lawyers whose AI-written filing contained fake citations — CNBC, 2023