As product managers, we get a lot of ‘push’ to embed AI within our products. After all, everyone is doing so and we don’t want to stay behind.

But what I often see across the industry is a lack of true understanding about the types of tasks AI excels at. Many people out there believe that a proper AI solution is always better than a traditional algorithmic-driven solution.

But that’s not the case.

Let me explain why.

How AI ‘thinks’

Let’s start with a very important observation:

AI-based solutions are not deterministic. 

Being deterministic means that for the same input, you will always get the same output.

In practice, this means that when AI is involved, repeated similar requests or executions may produce different answers or results.

We can all see it when we ask ChatGPT (or one of the other LLMs out there) a question, and then clicking on the ‘refresh/retry’ icon if we’re not happy with the generated output. A completely different answer – sometimes even contradicting the previous one – will be generated.

But it’s not limited only to GenAI. It happens also with traditional AI. For instance, let’s take an AI application in the field of image recognition. And let’s say that this AI application was trained to identify when people are trying to break in into cars.

And finally, let’s assume the engineers and scientists behind this application did a remarkable job, and the AI engine is highly effective at identifying car break-ins.

Still, even the best AI engine will miss some of the cases. The burglar may take a stance that the engine wasn’t trained on, and the break-in won’t be flagged.

This is why AI solutions usually provide their answers together with a confidence level. It’s like they tell us – “look, this is our answer/observation/analysis for your question, but with a confidence level of 90%”. Meaning, there is a 10% chance that the answer is inaccurate.

As users, we’re not always exposed to the confidence level (for example, ChatGPT doesn’t show its confidence level with the answers it provides), but it doesn’t mean that behind the scenes, such a score doesn’t exist. It’s usually exposed to the engineers behind the model, so they can evaluate it.

Additionally, sometimes, calculating the confidence level can be quite tricky by itself. Hence, the fact that the engine is positive that the answer provided is accurate by 90%, doesn’t mean that this is actually the case.

But for the sake of this post, the accuracy of the confidence score is not that important. What’s important is to understand that given the same challenge, to the same AI engine multiple times, may result in different answers. This is why we say AI is not deterministic.

This is the first thing many people miss:

It’s perfectly ok that it’s not deterministic.

You see, we (well, engineers & scientists around the world) worked so hard to train AI to behave & think like humans, and then, when it finally does it (to some degree), we’re getting annoyed that it’s not 100% accurate.

Why are we so surprised or annoyed by this?

After all, the same applies for humans as well. A human will identify that a person is trying to break into a car the great majority of the time. But sometimes, conditions may not be great, or the human will be distracted and he or she will miss it.

The same applies if I ask you to solve 100 simple arithmetic questions (such as 17 × 80). You’ll get the great majority of them solved properly, maybe even all of them. But some people, especially given aggressive time constraints (which we also apply on AI engines) will get some of them wrong, even if they know basic math quite well.

Hence, if we train AI to think like humans, you should expect the fact that sometimes you won’t get the result you hoped for, because their ‘thinking’ is not deterministic.

Where AI is the wrong tool

Because it’s not deterministic, AI can’t be the answer for everything.

If you recall, we invented computers to solve arithmetic problems in 100% accuracy and in a fraction of the time it would take humans to solve the same problem.

Computers were later evolved to do all sorts of repeated tasks that were very time consuming for humans, and maybe even error-prone.

When you use a calculator on your phone, you know that 100% of the time it will give you the right answer, amazingly fast. Your calculator doesn’t use AI. If it did, achieving 100% accuracy might not be possible.

When you operate your Windows/Mac machine – closing and opening windows, copying files, sending documents to the printer, etc.. – those are all deterministic operations, done with traditional software. You don’t need and don’t want AI replacing that.

Even more advanced software – like reading your email, streaming music to speakers in sync, and robotic arms assembling cars – relies on deterministic software, and you’d want to keep it that way.

Therefore, AI is not suitable for automation tasks where the desired result is known in advance.

Let me give you a real-world example I keep seeing in the hi-tech industry:

Many finance teams need to produce recurring reports every month – budget tracking, expenses, and more.

This is tedious and time-consuming, so naturally, startups rushed to offer AI-powered automation solutions.

In demos, these solutions often look magical. Financial teams see them, get excited, and sign up.

But here’s the catch:

In such scenarios, 100% accuracy is non-negotiable. If your budget report is off by even 1%, that’s a disaster. AI, by nature, can’t guarantee that level of precision. Even when confidence levels are high, mistakes still happen.

So, in reality, there are two options here:

1. Option 1: The startup is using “AI” mostly as a buzzword, and under the hood, the product relies on deterministic, traditional automation. That’s fine – and may solve the problem well.
2. Option 2: The startup actually uses AI to automate these workflows. In that case – Bad idea. They should ask for a refund. AI cannot and should not be trusted to automate processes where perfect accuracy is required.

Financial reporting is deterministic by nature. AI is simply the wrong tool for the job.

What AI is good for

AI based solutions are still great for many other use cases.

I won’t cover each and every use case, though. Instead, I’ll classify these use cases to high level buckets:

1. Automation of tasks where less than 100% accuracy is acceptable. For example – observing cameras for you and spotting suspicious activity. The AI can be trained to deliver great results for many cases. Yet, like a human, it may miss a case or two. Same goes for anomaly detection (which are not mission critical), bots that clean houses, real time translations, summarizing texts, etc..
2. Generative and creative tasks such as generating images, answering questions based on a training set, generating posts, researching, etc..
3. Information retrieval, analysis and summary of data or content. I wouldn’t rely on AI to transform data, when the reliability of this is critical, but I’d definitely use it for more lenient tasks such as summarizing texts, providing facts about events, people and places based on data or the open web, and explaining patterns in a dataset.
4. Building deterministic solutions through code generation. While the AI itself is not deterministic, it can help you build deterministic solutions. For example, AI can help you design and build a deterministic workflow that will automate the financial tasks we mentioned before. It can help build sophisticated rule engines, unique algorithms or advanced data transformations. All of which are 100% deterministic.

Summary

AI is sexy, AI is trendy and AI is the future.

This is all true.

Still, AI is not the answer to every problem.

As a product manager, it’s crucial to understand which problems AI is well-suited for – and which ones it isn’t.

You can let your marketing team advertise that your product leverages sophisticated AI for providing great solutions to your customers’ pain. Just don’t fall into your own marketing buzz.

Does the product you’re responsible for truly need AI, or would a brilliant (deterministic) algorithm be a better fit?

So next time someone says ‘just use AI’ – pause. Think. AI is powerful, but so is the good old algorithm. Choose wisely.

If you liked this post – please ‘like’ it below, and you’re more than welcome to share it with your friends using any of the social options below.