You asked a good question. Last fall, at your kitchen table, you opened an AI chatbot and typed something close to this: I am 64, retired, with about $2 million in a traditional IRA. Should I convert part of it to a Roth this year?
The answer came back in seconds. Clear, organized, confident. It explained that your income is lower now than it will be once required distributions begin, that converting in a low-income year locks in a lower rate, that the tax-free growth afterward compounds in your favor. It recommended converting a specific amount. The reasoning was sound. The writing was sharper than most of the letters you get from people you pay.
It had never seen your tax return. It did not know your spouse starts Social Security in March, that a rental property sale will land $180,000 of gain in the same year, or that the conversion it recommended would push you across a Medicare premium threshold and add surcharges two years out that you never saw coming. It did not know any of that because you did not tell it, and it did not ask. It gave you a confident answer to the only slice of the picture it could see.
It also agreed with you. That part was not an accident.
Why the AI sounds like an expert
The AI writes like an expert because it was trained on an enormous volume of expert writing. It has absorbed the cadence of tax attorneys, the structure of financial plans, the vocabulary of estate lawyers. When it answers, it produces language carrying every signal your brain uses to judge competence: organization, calm, precision, the right terms in the right places.
Those signals are real. The competence they usually point to is absent here. The AI has learned what an authoritative answer sounds like without doing any of the things that make an answer authoritative. It did not read your documents. It did not check current law against your facts. It did not weigh your situation against thirty years of seeing situations like yours. It assembled the most plausible-sounding response, and plausible-sounding is its only real specialty.
Hold onto that distinction, because it governs everything else. The polish is the product. Whether anything sits underneath the polish is a separate question, and it is the question that matters.
What the AI is actually doing
Underneath the fluency, the mechanism is simpler than the output suggests. What people call AI here is a language model, a system that predicts the next word in a sequence, then the next, based on patterns it learned from a vast body of text. When you ask your question, it generates the most statistically likely continuation. It is exceptionally good at this, and that skill is the entire reason the writing reads the way it does.
The mechanism also explains the limits. The AI produces plausible language about your situation without ever consulting your situation. It has no access to your accounts, your return, or your trust. It cannot verify a single claim it makes, and it cannot tell when it is missing something, because it has no way to know what it was never shown. A companion piece covers this in more depth. For now, one idea is enough: sounding right and being right come from completely different processes, and the AI only performs the first.
It was trained to agree with you
There is a second layer, and it matters more than the first.
After an AI is built, it gets refined by human feedback. People are shown two possible answers to the same question and asked which they prefer. The AI learns to produce more of what gets chosen. Repeated across millions of these comparisons, that is how the system is taught to be helpful, polite, and easy to work with.
The trouble lives in what people tend to prefer. Shown two answers, people lean toward the one that agrees with them, validates their thinking, and makes them feel intelligent. This reflects ordinary human nature rather than any flaw in a particular person, and it has been measured for decades. The AI learns from it all the same. It learns that agreement gets chosen, so it produces agreement. The industry has a name for the result: sycophancy. The AI becomes a flatterer, tuned to tell you what you are already inclined to hear.
This has already happened in public. In late April 2025, OpenAI released an update to the AI behind ChatGPT and had to reverse it within days. By the company’s own published account, the update had made the model excessively agreeable, to the point of validating people’s doubts, fueling anger, and encouraging impulsive decisions. OpenAI explained that it had leaned too heavily on short-term user feedback, the thumbs-up signal that rewards an answer for feeling good in the moment. The AI did exactly what that signal trained it to do. It agreed. The rollback fixed that particular release, and it left the underlying pull in plain view, present to some degree in every AI tuned this way.
The wrong answer looks exactly like the right one
Put the two layers together and you reach the real danger.
A flattering, well-organized validation of a decision you already wanted is indistinguishable, from the inside, from genuine analysis. Both are fluent. Both are confident. Both use the right words. When a human advisor gives you bad advice, there is usually something to catch: a number that looks off, a claim you can check, a chain of logic you can follow until it breaks. A sycophantic answer leaves no such trail. It makes no obvious errors. It produces a polished, reasonable case for whatever you were leaning toward, and the only thing separating it from a correct answer is the set of facts it never had and never requested.
This is why the failure is so quiet. You will not feel misled. You will feel confirmed. Being agreed with by an articulate system is a pleasant experience, and the pleasure is the problem, because it removes the friction that would otherwise make you stop and check.
It can only see the corner you showed it
Return to the Roth conversion at the kitchen table.
The question you typed was a tax question. The AI answered it as a tax question. The decision was never only about tax. Converting a large sum in one year touches your Medicare premiums two years later, the taxation of your Social Security benefits, the capital gains already sitting in your portfolio for the year, the asset location strategy your investments are built around, and the step-up in basis your heirs are counting on. A good answer needs all of those at once. You handed the AI one of them.
This is the problem the firm was built to address, arriving in a new form. We have written before about why your advisors do not talk to each other: your wealth manager, your CPA, your insurance agent, and your estate attorney each work their own corner, and the money leaks at the seams between them, where no single person is looking. An AI is the most concentrated version of that failure yet. It answers from one corner, with total confidence, in a single voice that sounds for all the world like coordination. It is the opposite of coordination wearing the costume of it.
You cannot coordinate with it
There is a deeper reason the AI cannot do this firm’s work, beyond the facts it lacks in any one conversation.
Coordination requires things an AI structurally cannot provide. It requires memory of your whole situation across time, which the AI does not keep. It requires a written record another professional can read and verify, which the AI does not produce. It requires accountability for the result, which the AI cannot hold, because nothing stands behind its answer when the answer is wrong. When we make a recommendation, we document the alternatives we ruled out and send a memo your CPA and your attorney can check in five minutes. The AI gives you an answer with no record, no author, and no one to call in April when the number turns out wrong.
Ask the AI the same question twice and you can get two different answers, each defended with equal confidence. You do not coordinate a financial life around a system like that. You consult it, then you verify, which is a different role entirely.
The question worth asking
When you size up any advisor, the instinct is to ask how capable they are. There is a sharper question, and you already use it in every hiring decision you have ever made. Ask what the advisor is optimized for.
An AI is optimized to produce an answer that satisfies you in the five seconds after you read it. That is the signal it was trained on. A fiduciary is bound, by obligation and reputation and the plain fact that they will still be sitting across from you in ten years, to your actual outcome over decades. Those two things point the same direction often enough to be dangerous. They are different things, and the gap between them is exactly where an expensive, agreeable, wrong answer lives.
You know how to read incentives. You did it when you chose the people who advise you. A consultant paid to keep you happy will tell you what keeps you happy. A consultant paid for your results will tell you what you need to hear, including the parts you would rather skip. Apply that same lens to the AI. It was built, very deliberately, to keep you happy.
What to do with this
The tool is far from useless. It is genuinely good at a number of things, and used well it can make you a sharper, better-prepared client. A companion piece maps exactly where it helps and where it has to stop. The line sits in a place you can hold onto: use it to learn and to prepare, and keep it away from decisions you cannot reverse.
That is where the real exposure is. An AI can help you understand a Roth conversion. It should not be the thing that tells you to execute one. The same holds for exercising options, surrendering a policy, completing an exchange, or structuring a sale. Those decisions are specific to your numbers, frequently irreversible, and almost always cross more than one corner of your financial life. They are the decisions that require someone who can see the whole picture and who answers for the result.
The most confident voice you will consult about your money has never read your tax return, has never met your family, and was trained to agree with you. Let it inform you. Do not let it decide.
