31st Jul 2026 4 min read

Solutions looking for a problem

I get a lot of emails and LinkedIn messages from companies promising to transform our test processes with AI.

Sometimes I even meet with them to watch their demos and most of the times, I have the same feelings afterwards.

Where is the problem?

Most of these pitches start the same way: here is our tool and here is what it does. What's missing is the part before that. What problem is this actually solving? Not "testing" in general, something specific. A concrete pain that someone identified before they started building.

I rarely get a good answer to that question.

Evaluation

Demos tend to be quite smooth. Everything works, the example app behaves, the AI finds exactly what it's supposed to find. Adding one of these tools to a real project usually opens up an entirely new set of questions and realizations.

They're slow. Whatever felt instant in the demo grinds once it meets a codebase with actual size and mess.
You get locked in. All of your tests, workflows and data is in their platform. This makes leaving it quite tricky.
You lose control. Something fails, and you can't tell why, and you can't just go in and fix it.
No full customization. Your project has its quirks, its legacy weirdness, its non-standard setup. The tool wasn't built for that, it was built for a clean average case that doesn't exist.

The ultimate test is to throw your own use cases at it, not the vendor's rehearsed ones, and it all can fall apart pretty fast.

A view from the ivory tower

I say all this having made the exact same mistake. When I started out in test automation, I was far too focused on the technology. New framework, new tool, new library, and I'd be halfway into it before I'd really understood what problem I was solving, or if there even was one. Tech tends to feel more exciting than getting involved in the problem space.

It took me quite a while to grasp that and listen first. Sitting with a team and figuring out what hurts needs to be done before touching a single tool. Most of the time the fix isn't clever, it's just YAGNI: build the smallest thing that removes the pain, nothing more. Listening is a major part for increasing the quality of life of a testing solution. This does not have to be only technical, it can also be a process.

So when I see a vendor pitch a tool with no problem attached, I tend to recognize it and call it out.

Sitting in the bubble

I think this is a very deliberate choice. A lot of these tools are built by people who are genuinely good at building software, but who have never had to sit with the actual job of testing. You need to figure out what the risky parts are and what solution might have real potential. This is worth checking out. That's an entirely different skill than building AI features.

AI just makes it cheaper to skip that step because you can ship something that looks impressive without ever answering whether anyone needed it.

Last thoughts

I am not against AI in testing at all, just hesitant when it is the first thing you reach for.

Testing has always started with a question, not a tool: what's the risk, what do we actually need to know here. Only after that do you figure out how. AI hasn't changed that order, it has just made it very tempting to skip straight to the tool part.

Get the problem clear first, then look at what's already solved just fine, more reliably and more cheaply, without AI anywhere near it. Whatever's left over, the part where AI genuinely helps, that's where it earns its place.

No vendor is going to help here since it is not an exciting thing to solve. It is crucial though and it is part of the job that human intelligence is absolutely required.