Edition #41: The Guardrails Are the Speed

Why the companies slowing down for AI governance are the ones still running

By Mykel Salomon · 2026-10-10

<hr><h3>🔓 Opening Reflection</h3>There is a database.It sits in public, free to read, and almost nobody reads it.As of August 2026 it held 1,618 documented cases of artificial intelligence causing real harm to real people. A hundred and forty-eight of those were added in a single three month stretch. Families that lost their homes. A government that fell. People who died.Most of it never reached you, and not because it was hidden. The stories were reported. They were simply louder things happening that week.My guest this week reads these files every day. <a target="_blank" rel="noopener noreferrer nofollow" class="text-[#4FD1C5] underline hover:opacity-80 ember-view" href="https://www.linkedin.com/in/lathakarthigaa/">Latha Karthigaa, PhD</a> has a PhD in software engineering, runs the Global AI Certification Council, and has set herself the task of publishing one AI incident breakdown every day for 365 days. What went wrong. Why. How it could have been prevented.She did not start out as a governance person. She started out curious. She read about McDonald's shutting down its AI drive-through after three years of trying, with IBM as its partner, and she had the thought that should stop all of us:If a company that size could get it that wrong, this cannot possibly be the only one.It was not.Here is what she found underneath all of it, and it is not the conclusion most people expect.The companies that fail at AI are usually not failing at the technology. They are failing at the thing they decided they could skip in order to move faster. And the thing they skipped is the only reason anyone gets to move fast at all.<hr><h3>🔦 Signal of the Week</h3>Companies are quietly walking away from AI, and the number doubled in a year.S&P Global surveyed more than a thousand organizations across North America and Europe. In 2025, 42% of companies abandoned most of their AI initiatives. The year before, that number was 17%.The average organization now scraps 46% of its AI proofs of concept before they ever reach production.The three reasons they gave for the failures were cost, data privacy, and security risk. Not one of them was "the model was not good enough."Governance policies exist almost everywhere. They are not what is actually running.EY surveyed 202 senior AI decision-makers in mid-2026. Ninety-eight percent said their organization has a formal AI governance policy.Then:<ul><li>47% admitted their organization had bypassed its own governance process to push an urgent AI deployment out the door</li><li>69% said they lack the internal expertise to evolve their controls</li><li>49% have not updated their governance framework for agentic AI, while 91% are already using it</li><li>26% cannot detect unauthorized AI agents running inside their own organization</li></ul>And here is the number that should decide your next quarter.Among the organizations that actually stopped and ran a formal review of their AI systems:<ul><li>64% had to significantly modify a quarter or more of them</li><li>29% had to pause a quarter or more</li><li>25% had to shut down a quarter or more entirely</li></ul>A quarter of what you have running is probably wrong in some way that matters. That is not a prediction. That is what the companies who bothered to look found when they looked.The ones who did not look are not in better shape. They just do not know yet.<hr><h3>🌍 In the Wild</h3><h3>1. The guardrails are why you get to drive seventy</h3>This is Latha's metaphor and it reframed the whole conversation for me.Think about a highway. You drive seventy, sometimes faster, and you do it without gripping the wheel. Why? Because there is a barrier between you and oncoming traffic. Because no pedestrian is going to step out. Because the system around you is designed so that one mistake does not become a catastrophe.Now think about a city street with no barriers, no markings, unpredictable crossings. You drive forty. Maybe thirty. You are tense the entire time.The guardrail is not what slows you down on the highway. The guardrail is the entire reason you are allowed to go fast.She put the business case in terms of time horizon, which is the part most leaders get wrong:If your horizon is three months, governance genuinely does look like a tax. You will ship faster without it. That is true, and pretending otherwise is why governance arguments usually lose in the room.But most of the failures she reads about do not happen in month three. They happen somewhere between year one and year three. That is the window where the thing you built quietly starts producing wrong outputs, and nobody has a mechanism to notice, and by the time someone does the damage is structural.So the honest version of the trade is this. Skipping governance buys you speed in the first quarter and costs you the project somewhere in year two. The S&P numbers are what that trade looks like at scale: 42% walking away, half their pilots scrapped.The companies still running three years in are not the ones who moved carefully. They are the ones who built something that could survive being checked.<h3>2. Fourteen years is how long a system can be wrong before anyone looks</h3>In 2005 the Dutch tax administration began using a self-learning algorithm to decide which parents were likely committing childcare benefits fraud.It ran until 2019.Over those years it developed a pattern. It began flagging applicants as high risk based on nationality and dual citizenship. Immigrant families were disproportionately marked as probable fraudsters. The Dutch data protection authority later found the system had been built in a way that gave "permanent and structural unnecessary negative attention" to exactly those markers.Roughly 26,000 families were wrongly accused.They were ordered to repay benefits they had legitimately received, often years of them, all at once. People lost their homes. People lost their jobs. Marriages ended.The Dutch national statistics office later counted 2,090 children who were removed from their families and placed into state care during that period.The government of the Netherlands resigned on January 15, 2021.Compensation was set at €30,000 per wrongly accused parent, which is a real sum and also a number that tells you something about the limits of money as a remedy. Latha said the part that stayed with her, and it stayed with me too:A student who loses a year can rebuild. A parent who lost four years of their child's childhood cannot buy those back at any price.Fourteen years. A self-learning system, learning in a direction nobody audited, inside a government, in a wealthy country with a free press and functioning courts.Nobody was asleep at the wheel out of malice. There simply was no wheel. There was no mechanism whose job it was to ask, periodically, whether this thing was still doing what we built it to do.That mechanism has a name now. It is the job almost nobody was hiring for in 2005.<h3>3. The systems fail hardest on the people furthest from the pattern</h3>Here is something neither of us said out loud during the conversation, and it only came into focus for me afterward, when I put her stories side by side.Latha told me about a university student accused of using AI to write an essay. The professor pasted student essays into ChatGPT and asked it whether it had written them. ChatGPT said yes. The student was suspended for a year.ChatGPT is not a detection tool. It has no memory of what it generated and no mechanism for recognizing its own output. Asked a question in that shape, it produces a confident answer because producing confident answers is what it does.But look at what the actual detection tools do, which is worse, because those at least claim to work.Stanford researchers tested seven AI detectors against TOEFL essays written by non-native English speakers. The average false positive rate was 61.3%. More than 91% of those essays were flagged as AI-generated by at least one detector.On essays written by native English speakers, the false positive rate was close to zero.The reason is mechanical and almost sad. People writing in a second language tend to use a narrower, more standard vocabulary and more templated sentence construction. That produces exactly the statistical signature these tools read as machine-generated. Writing carefully in a language that is not your first gets you flagged as a cheater.OpenAI quietly killed its own AI detection classifier in July 2023. Its own numbers: it correctly identified about 26% of AI text, while mislabeling roughly 9% of human writing as machine-written.Now put that next to the Dutch algorithm flagging people on nationality and dual citizenship.Different decade. Different continent. Different technology. Same failure.The system learns a dominant pattern, and then it treats distance from that pattern as evidence of wrongdoing. The people furthest from the pattern are, almost by definition, the people with the least institutional power to contest the decision. An immigrant family facing a tax authority. A foreign student facing a disciplinary board.The harm is not distributed randomly. It concentrates on the people who can least afford it and are least likely to be believed.This is why Latha's insistence on empathy as a job requirement is not soft. It is the diagnostic. If nobody on the team has ever been the person furthest from the pattern, nobody is going to think to ask who the system is quietly getting wrong.<h3>4. "The chatbot was a separate legal entity"</h3>That is not my paraphrase. That is the defense Air Canada actually ran in court.A customer asked the airline's chatbot about bereavement fares. The chatbot described a policy that did not exist. When the customer tried to claim it, Air Canada argued, in a tribunal, that the chatbot was effectively its own legal entity and its statements were not binding on the airline.The tribunal rejected it. Air Canada paid.The principle is now plain: you own what your AI says. You cannot deploy a system to speak for you and then disown it when it speaks badly.The same lesson keeps arriving in different costumes.A software engineer opened a Chevrolet dealership's chatbot, instructed it to end every response by agreeing the offer was legally binding, and then asked to buy a Tahoe for one dollar. The chatbot agreed. He screenshotted it. A $76,000 vehicle, one dollar, one sentence of prompt injection.And then the version with no funny side at all. A finance employee at a multinational received a request to transfer a large sum. He was suspicious, so he did the responsible thing and asked for a video call. On the call were the CFO and several colleagues he recognized, faces and voices.Every one of them was synthetic, generated from publicly available recordings of real executives.He made 15 transfers totaling $25 million.Nothing was hacked. No system was breached. No credential was stolen. The company's security held perfectly. What failed was the assumption underneath it, which is that seeing a familiar face on a video call constitutes verification.Latha and I spent a while on where this goes next, and it is the part that keeps me up. A chatbot connected to a knowledge base could only say wrong things. An agent can do things. And organizations are not deploying one agent. They are deploying orchestrations of them, sometimes tens of thousands.Which brings back the EY number: 26% of organizations cannot detect unauthorized AI agents already operating inside them.Not "cannot control." Cannot detect.The question I kept pressing her on, and the one I would put to any leader reading this: if one of your agents started doing something it was never meant to do, what, specifically, would tell you? Not in principle. Which log. Which alert. Which person's inbox.If the answer is a pause, that is the finding.<h3>5. The job this creates, and the skill it actually requires</h3>Latha's framing is the one I keep coming back to, because it runs against the anxiety almost everyone brings to this subject.Every regulation written so far, in every jurisdiction, requires a human being to approve certain decisions. Which means every regulation written so far has created work that cannot be automated by design, because the whole point of the work is that a human did it.The market has moved. Nearly 2,000 US job postings for AI governance roles went up between January and August of 2026. Median compensation sits around $169,000. Seventy-three percent of those postings are senior level or above. Professional services leads the hiring at 35%, with technology and financial services close behind.The skills employers list most: observability tooling at 41% of postings, Python at 28%, the NIST AI frameworks at 27%.But Latha's point about what makes someone good at this is different, and I think she is right.Technical fluency gets you in the door. What makes a governance professional effective is the capacity to imagine harm that has not happened yet, to someone who is not you. That is not a coding skill. That is an empathy skill, and it is the one thing on the list that cannot be picked up in a weekend course.She was blunt about why: you cannot write a rule against a failure you have never heard of. Which is the entire reason she publishes a story a day. She is not collecting disasters. She is building the curriculum that does not exist yet. "We do not know what we do not know" was her refrain through the whole conversation, and the incident database is her answer to it.One clarification, because this sits next to something we covered recently.Two weeks ago this newsletter covered AI evaluators, the people who test whether a model does what it claims. That is a different job from this one. Evaluation asks does this work? Governance asks should this be running, who can override it, and how would we know if it drifted?You need both. Most organizations currently have neither, and are deploying anyway.And one more thing Latha said that I think is underrated: no single governance professional in an organization can find the issues alone. It has to be collaborative. One person knows one failure mode, another knows a different one, and the risks surface in the overlap. The lone compliance hire who gets handed the whole problem is being set up to fail.<hr><h3>💬 The Big Question</h3>What is running inside your organization right now that nobody has checked since the day it went live?And the harder follow-up, the one the Dutch tax authority could not answer for fourteen years:If it were quietly wrong, how exactly would you find out?<hr><h3>🧠 A Small Exercise</h3>If you are an individual:<ol><li>Go read one incident. The AI Incident Database is public and free. Pick an industry you work in and read a single case start to finish. It takes ten minutes and it changes how you look at every tool on your desk.</li><li>Test something that has judged you. If an AI system has scored your resume, your writing, or your application, find out what it actually measures. Most people have never asked.</li><li>Decide your own override rule. Name one category of decision in your life where you will not accept a machine's answer without checking it yourself. Write it down. Medical, financial, anything involving another person's reputation.</li></ol>If you lead a team:<ol><li>Name the system, the human, and the trail. For every AI system you have deployed, answer three questions out loud: what decision does it make, which named human can overrule it, and where is the record of what it has done? Any system that cannot answer all three is running without governance, whatever the policy document says.</li><li>Run the review you have been postponing. Remember what the organizations who did this found. Sixty-four percent had to significantly change a quarter or more of their systems. You are not going to be the exception, and finding out now is cheaper than finding out in court.</li></ol><hr><h3>🎙️ This Week on The Human Protocol Podcast</h3>AI Governance, Real Disasters, and Why Empathy Is a Technical Requirement Guest: Latha Karthigaa, PhD, Founder of the Global AI Certification CouncilLatha has a PhD in software engineering and started coding at eleven. Today she runs the Global AI Certification Council, teaches AI governance, and publishes a daily breakdown of real-world AI incidents: what failed, why, and what would have prevented it.In this episode, you'll learn:<ul><li>What the AI Incident Database is, how to use it, and why most of what is in it never reached the news</li><li>The full story of the Dutch childcare benefits scandal, and how an algorithm ran wrong for fourteen years inside a functioning democracy</li><li>Why "model drift" is the failure mode almost nobody outside the field can name, and the Uber example that makes it click</li><li>How prompt injection turned a $76,000 vehicle into a one dollar offer, and what it says about every customer-facing chatbot</li><li>Why regulation without penalties changes nothing, and what the EU AI Act gets right that most frameworks do not</li><li>The case for AI governance as a career, and why empathy is the hardest requirement on the job description</li><li>What she would tell her younger self: be a human before you are a tech person</li></ul>🎧 Available now on <a target="_self" rel="noopener noreferrer nofollow" class="text-[#4FD1C5] underline hover:opacity-80 tfhTEVcYQadBfYyMtYiEPuRvTquXleKuzgdU " href="https://open.spotify.com/episode/69mTP7lXTjcnvd5d2PwU9l?si=tUr6KPwlQoi4Y_tXMojlUQ">Spotify</a>, <a target="_self" rel="noopener noreferrer nofollow" class="text-[#4FD1C5] underline hover:opacity-80 tfhTEVcYQadBfYyMtYiEPuRvTquXleKuzgdU " href="https://podcasts.apple.com/us/podcast/1-588-ai-disasters-nobody-is-talking-about-latha-karthigaa/id1843402023?i=1000793652238">Apple Podcasts</a>, and <a target="_self" rel="noopener noreferrer nofollow" class="text-[#4FD1C5] underline hover:opacity-80 tfhTEVcYQadBfYyMtYiEPuRvTquXleKuzgdU " href="https://youtu.be/bSErg2F2l-k?si=hzP4hM2xvYicqezb">YouTube</a>.<a target="_blank" rel="noopener noreferrer nofollow" class="text-[#4FD1C5] underline hover:opacity-80" href="https://youtu.be/bSErg2F2l-k?si=hzP4hM2xvYicqezb">https://youtu.be/bSErg2F2l-k?si=hzP4hM2xvYicqezb</a><hr><h3>🧭 Thought to End the Week</h3>Every system in every story above was built by people who were good at their jobs. The Dutch algorithm was built by competent engineers inside a competent government. The detection tools were built by companies with real research teams. The chatbots were built by firms with legal departments.Competence was never the missing ingredient.What was missing, every single time, was someone whose job it was to sit with the question of who this gets wrong, and to keep asking it after launch, when the pressure to move on to the next thing is at its strongest.That is not a technical gap. A technical gap can be closed by hiring a better engineer.This is a gap in who we decided to put in the room.We have spent three years asking what AI will take from us. The more useful question is what it will ask of us. And the answer turns out to be something we already have and have been treating as a soft skill: the ability to imagine being someone else, and to care what happens to them before it happens.Fourteen years. Twenty-six thousand families. Two thousand and ninety children.Not one of those numbers required a breakthrough to prevent. They required somebody to look.Build the guardrails. Not because they will slow you down, but because they are the only reason anyone ever gets to go fast.The code may change. And it will.But our humanity is the constant.— Mykel<hr><h3>📚 Sources</h3><ol><li>AI Incident Database, Incident Roundup, May to July 2026 (1,618 incidents, 148 added in the quarter) <a target="_blank" rel="noopener noreferrer nofollow" class="text-[#4FD1C5] underline hover:opacity-80" href="https://incidentdatabase.ai/blog/incident-report-2026-may-june-july/">https://incidentdatabase.ai/blog/incident-report-2026-may-june-july/</a></li><li>AI Incident Database, main index <a target="_blank" rel="noopener noreferrer nofollow" class="text-[#4FD1C5] underline hover:opacity-80" href="https://incidentdatabase.ai/apps/incidents/">https://incidentdatabase.ai/apps/incidents/</a></li><li>CIO Dive, "AI project failure rates are on the rise," reporting S&P Global Market Intelligence survey of 1,000+ respondents (42% abandonment, up from 17%; 46% of POCs scrapped) <a target="_blank" rel="noopener noreferrer nofollow" class="text-[#4FD1C5] underline hover:opacity-80" href="https://www.ciodive.com/news/AI-project-fail-data-SPGlobal/742590/">https://www.ciodive.com/news/AI-project-fail-data-SPGlobal/742590/</a></li><li>EY, "Survey finds that autonomous AI implementation outpaces oversight, yielding an AI governance gap," September 2026 (202 senior AI decision-makers, fielded May 28 to June 15, 2026) <a target="_blank" rel="noopener noreferrer nofollow" class="text-[#4FD1C5] underline hover:opacity-80" href="https://www.ey.com/en_us/newsroom/2026/09/ey-survey-finds-that-autonomous-ai-implementation-outpaces-oversight-yielding-an-ai-governance-gap">https://www.ey.com/en_us/newsroom/2026/09/ey-survey-finds-that-autonomous-ai-implementation-outpaces-oversight-yielding-an-ai-governance-gap</a></li><li>Wikipedia, Dutch childcare benefits scandal (26,000 families, 2005 to 2019, government resignation January 15 2021, €30,000 compensation) <a target="_blank" rel="noopener noreferrer nofollow" class="text-[#4FD1C5] underline hover:opacity-80" href="https://en.wikipedia.org/wiki/Dutch_childcare_benefits_scandal">https://en.wikipedia.org/wiki/Dutch_childcare_benefits_scandal</a></li><li>Vision Times, reporting Dutch national statistics office figure of 2,090 children taken into state care <a target="_blank" rel="noopener noreferrer nofollow" class="text-[#4FD1C5] underline hover:opacity-80" href="https://www.visiontimes.com/2022/12/08/dutch-children-seized-tax-agency-benefits-scandal.html">https://www.visiontimes.com/2022/12/08/dutch-children-seized-tax-agency-benefits-scandal.html</a></li><li>CASRAI, "AI Detection Accuracy in Higher Education," summarizing Liang et al. (Stanford, 2023): 61.3% average false positive rate on TOEFL essays by non-native English speakers; OpenAI classifier retired July 2023 at 26% accuracy <a target="_blank" rel="noopener noreferrer nofollow" class="text-[#4FD1C5] underline hover:opacity-80" href="https://casrai.org/guides/ai-detection-accuracy-higher-education">https://casrai.org/guides/ai-detection-accuracy-higher-education</a></li><li>Moffatt v. Air Canada case analysis, Civil Resolution Tribunal (the "separate legal entity" defense and its rejection) <a target="_blank" rel="noopener noreferrer nofollow" class="text-[#4FD1C5] underline hover:opacity-80" href="https://www.compelframework.org/articles/casestudy-air-canada-chatbot-moffatt-decision">https://www.compelframework.org/articles/casestudy-air-canada-chatbot-moffatt-decision</a></li><li>The Register, "Hong Kong deepfaked CFO" (US$25 million, 15 transfers, no systems compromised) <a target="_blank" rel="noopener noreferrer nofollow" class="text-[#4FD1C5] underline hover:opacity-80" href="https://www.theregister.com/2024/02/05/hong_kong_deepfaked_cfo">https://www.theregister.com/2024/02/05/hong_kong_deepfaked_cfo</a></li><li><a target="_blank" rel="noopener noreferrer nofollow" class="text-[#4FD1C5] underline hover:opacity-80" href="http://Regulation-AI.eu">Regulation-AI.eu</a>, EU AI Act penalty tiers (€35M/7% for prohibited practices; €15M/3% for transparency obligations; Article 50 transparency applying from August 2, 2026) <a target="_blank" rel="noopener noreferrer nofollow" class="text-[#4FD1C5] underline hover:opacity-80" href="https://www.regulation-ai.eu/en/penalties/">https://www.regulation-ai.eu/en/penalties/</a></li><li>Axial Search, "The State of the AI Governance Job Market in 2026" (1,997 US postings January to August 2026, $169,000 median, 73% senior or above, skills breakdown) <a target="_blank" rel="noopener noreferrer nofollow" class="text-[#4FD1C5] underline hover:opacity-80" href="https://axialsearch.com/insights/ai-governance-jobs">https://axialsearch.com/insights/ai-governance-jobs</a></li><li>Global AI Certification Council <a target="_blank" rel="noopener noreferrer nofollow" class="text-[#4FD1C5] underline hover:opacity-80" href="https://gaicc.org">https://gaicc.org</a></li></ol>