AI Is Great at "How." But Not "Why." | Moving AI From Pilot to Production w/ Ian P Cook | S2E5

Full Episode Transcript

This is the full recorded conversation from The Human Protocol Podcast, published by The Human Protocol Group. © 2026 The Human Protocol Group. Please link to and attribute the original episode when referencing this conversation.

THE HUMAN PROTOCOL (00:01.35)
Hello, everyone. Welcome to the Human Protocol. This is your host, Mykel Salomon. Today's conversation is really for leaders that are tired of slide decks, unlimited demos, pilots, and hype cycles. For those that want to know what it actually takes to make AI work in real business, because here is the hard truth.

Most AI projects, they don't fail because the technology is bad. They do because organizations confuse possibility with production. My guest today is Ian Peekook, and I'm very excited to have you on the show today, Ian. So welcome to the Human Protocol Podcast.

Ian Cook (00:50.968)
Thank you so much for having me, Michael. It's terrific to be here.

THE HUMAN PROTOCOL (00:54.06)
Awesome. To kick things off, Ian, we would welcome if you can give us a brief introduction of who you are, where have you been, and where are you now?

Ian Cook (01:05.422)
Fantastic. So currently my job title is the SVP of AI at a company called Clue. Q-L-O-O for anybody who wants to look it up. We do AI for cultural intelligence. So putting together the world's biggest knowledge graph on cultural assets like books, movies, restaurants, hotels, places around the city, and we cover the entire world. So I encourage people to take a look at it. It's fantastic.

I got here because I have spent the past 15 plus years working in software development specifically for what at the time used to be data science and then became machine learning engineering. And now we talk more about AI and I've spent the past several years working in the new iteration of it called generative AI. I have spent the majority of my career working in and around issues related to data and what we can do with data.

and producing tools that go into the hands of people. So I've done this for companies that do electricity market predictions. I've done it for work at healthcare companies, consumer packaged goods companies, direct to consumer clothing companies, insurance underwriting, and now for cultural information that we use for personalization and recommendation improvements. So it's been a...

I spent a lot of time in different startups and have seen everything from massive rocket ships to things that have petered out over time and have spent a huge amount of time building things that work sometimes, don't work sometimes, work really well that nobody wants or don't work at all that everybody wants.

THE HUMAN PROTOCOL (02:47.418)
That's awesome. Thanks for going over that. And I'm very, very excited to kick this conversation with you because a lot of what we're going to talk today, it's about substance instead of sensation. I know you're going to bring a lot to the table today where it's going to show people what it really make AI to work at the end of the day. Right. So, so I'm really, really excited to get this to hit the ground running with you and show people in reality.

what it makes to kick up these projects and to make really work them in real business. And you have seen Gen.ai go from promise to production reality, right? And I'm curious to hear from you first. Where do you think the hype break first?

Ian Cook (03:34.424)
Hype breaks first, I believe, is in the idea that these technologies can do anything. So people throw a problem at it that is not well spec'd out, not well defined, and doesn't have a huge connection to a business goal. And they just think, well, if I come up with the right set of words to put into the little chat box, it will just immediately give me back exactly what I want.

THE HUMAN PROTOCOL (03:41.168)
Hmm.

Ian Cook (04:01.26)
And unfortunately what happens is it doesn't do that. And then people go, well, this isn't the magic I thought it was. It's not really useful. It's just more of a pain. And they try to go find something else instead of working with the tool as it is and understanding that, you know, this isn't magic. This is a incredible tool that can speed up a lot of things, but it doesn't erase the ability to have those fundamentals.

THE HUMAN PROTOCOL (04:24.654)
And you just say it better than I could. It's not magic, right? And this is what I need everybody and all our listeners always to pay attention to. AI is not magic. It has to solve a problem and you have to plan towards that problem. when the problem also for a lot of people that are not made well versed as we are in technology and especially in AI and automation, you name it, is that they hear a lot of noise out there.

Like AI is going to solve our problem like plug and play. This is not how this works, right? And so there has to be a process between before you get and put something in production. And that happens at the conception of the idea of what you're trying to work and, and, and replace. So it's like, how do you know when you are implementing a project that is not going to work at the end of the day?

Ian Cook (04:52.568)
Mm-hmm.

Ian Cook (05:07.384)
Yes.

Ian Cook (05:20.372)
I can usually tell when I ask a couple of questions. One is what do you expect someone to do with this? And what problem does that person have that this is solving? That does a couple of things. Taking those in reverse order.

I think this is something that a lot of people, especially in the founder space and technology space really do have inculcated into them, which is you have to be solving a business problem. For people to want to use this tool, it has to solve a problem for them. A lot of people will use the cliche like a painkiller, not a vitamin. This isn't just like, it's a little bit better than I had it before. I need a real problem. The issue is, is you get scale up in the size of company that

wants to do these things. Often what they identify as a business problem is really under specified. It's like, well, my customers are just doing this and I don't want them to do that as much. They're turning too fast or they are not spending the time and they aren't doing the kind of process that I want. And let's fix that. The problem with that is you're at a huge company, which means there's a lot of money riding on it, but that isn't a well specified problem. A well specified problem is they leave this particular page

too fast without succeeding at some task. Great. Now we can start talking about the problem that you've got. And then what do you expect people to do with this? So you're creating a tool that somebody's got to work with. Now, tons of software development is for backend things for automating processes, but I would expect even then you would say, I expect this piece of the technology to do this with what I'm creating, but let's keep it with humans for a moment.

I expect you, Michael, to open this up and chat with it and to get back a response that says, you know, what it is that you, that answers the question in one turn or two turns. That's a great way to specify what I expect someone to do with it. If I don't hear that from like the people building the tool or the building, the, um, the prototype, I'm pretty sure that's going to fail. Like, I just know what's going to happen is you're going to build something, you're going to be in a room and a couple of people are going to be like, well,

THE HUMAN PROTOCOL (07:13.958)
Hmm.

Ian Cook (07:27.394)
I didn't expect it to do that. And I had a different thing in mind. And then somebody will say, well, I tried it and it didn't do what I wanted, or it doesn't feel quite right. And then suddenly you just sort of lost trust and lost the momentum.

THE HUMAN PROTOCOL (07:41.36)
the solution will only be as good as their requirements are. And that's where I see also lot of people and a lot of companies struggling with, because people usually are reluctant to spend time on putting down and writing down specific requirements, And for us that work in technology, this is crucial, because we are not here to think what you have in mind, right? We are not here to develop or implement a solution thinking of what you think.

Ian Cook (07:58.734)
Mm-hmm.

THE HUMAN PROTOCOL (08:10.156)
is your problem. We need requirements. And we work and we develop these solutions of technology based on what you are requesting us to do. Right?

Ian Cook (08:18.402)
Yeah, preach it brother. That is terrific. That's great. Yeah, I could not agree more. One of the really interesting things about the fact that code has been sped up so fast and writing code used to be the bottleneck and trying to find time with your developers when they're doing other things has that has become the sorry.

The issue is that is flipped. So the whole thing is code is no longer the bottleneck. The people are the bottleneck. The documentation is the bottleneck, which actually isn't that bad a thing. The word bottleneck shows up and everybody's like, we have to get rid of that. It really should be a moment for you to pause and think about, let's write down what we need. Let's write down specifically what we are trying to achieve. Because the other thing about the tools that are coming out is

Everything you write down is literally, like not just figuratively, literally what you can put into these tools to say, build me this, and it should meet these requirements. And I need one of the big things for me, especially at product companies that have consumer-based things is what is the definition of good and what is the definition of done? And you can feed that directly into things like Claude code or whatever your tools are. know, it used to be that we'd want developers to look at this and to sort of like try it out.

and that's great and we should still evaluate that way. But you can, those documents you're creating that feel like, we're doing some paperwork. That is literally the context that goes into these tools that it will tack onto and say like, I have to build this exactly the way they've said it. The more time and effort you spend in fleshing that out and making it exactly what you expect, not only is code going faster, the results will come faster.

THE HUMAN PROTOCOL (09:58.118)
Yeah, it's a time and an effort that I tell everyone, it's needed. The success of the project and the solution that you will receive and the outcome that you expect will only be as good as the time that you put in front to let us know and tell us what is it that you need and what is it that you expect.

Because we are not magicians, developers are not magicians, and any AI or automation that we can put in place will solve a problem and will provide you an outcome. But we need to know. We really need to know. Everybody wants things quick and easy. Solve my life. Make my life easy. OK, tell me, what does that mean for you? Right?

Ian Cook (10:32.961)
Yes, absolutely.

Ian Cook (10:44.27)
Exactly. And we're at a point now where we can move pretty quickly, but we have to spend that time on defining what it is. I like to be a little controversial when I go into these things and I'm just like, all right, narrow your vision, reduce your dreams, reduce your, not because I don't want people to think big, but what I want is for people to think very pointedly. What is the very first, very specific thing that you think you could get done and tackle?

That is what your prototype should go after because that, once it does it, is gonna get into production so much faster. The bigger you scope something in a prototype and you're like, well, let's rewrite this entire process and let's re-figure everything. It's like, you know, it's gonna hit a mess. You're gonna have too many people trying to figure out too many parts. Narrow it down. Keep your dreams big, your tactics much, much narrower.

THE HUMAN PROTOCOL (11:19.44)
Mm-hmm.

THE HUMAN PROTOCOL (11:37.058)
It's a journey, right? You're just getting it started. Sometimes people want to tackle everything at once. And I cannot agree with you more than narrow it down. Let's get started. And then we'll build on top of it. And we're going to have more opportunity to continue to build on top of it. And that should be kind of the continuous process improvement mentality comes into play. Even with this type of solution, we don't need to solve all problems at once. We will continue to improve.

Ian Cook (11:39.362)
Yeah.

THE HUMAN PROTOCOL (12:05.944)
as we move along. Because once we get started right in, the support and improvement continues. It doesn't stop right there.

Ian Cook (12:09.443)
Yeah.

Ian Cook (12:13.321)
Yeah. Iteration and experimentation. start with a small item, iterate on it, get it to where you need it. Experiment with the next step. Did that work? Did it not work? I actually like seeing failure. it's a thing I encourage among teams that I manage. I want to know what didn't work. Why didn't it work? It's a great teacher. We know now why this failed so we can move on. It's not, you know, getting something wrong. It's, you know, finding one more way that didn't work. And we now can narrow down our scope for what can work.

THE HUMAN PROTOCOL (12:19.494)
Mm.

Ian Cook (12:42.446)
I just, you know, I come from sort of a...

the social science research experimentation background and find that that's just been a huge benefit to the way I approach things, which is what's a small item that I can take, try it out, repeat it, see if I can build on it. What's the next step? What's the next step? Because then it's so much easier to roll back when we have these massive companies worrying about ROI. Well, if I invest this, am I going to get enough out of it? It's like, well, if you only invest small amounts, then you can get that snowball running or you can say, hey, that didn't work. We need to find something else.

THE HUMAN PROTOCOL (13:14.063)
You

Ian Cook (13:16.524)
and nobody is, you know, being terrified that they're going to lose their job because of the P &L sheet.

THE HUMAN PROTOCOL (13:21.784)
It's better when we find things and we do that as much as possible during our testing cycle, right? That's when you push things to break. It's not because we want to find issues. Issue teaches us, and it better find us during the testing cycle than when something is in production. So yeah, I agree with you on that as well, that I think is extremely important that we push things to a point that it breaks, because that's actually what disaster recovery

Ian Cook (13:47.747)
Mm-hmm.

THE HUMAN PROTOCOL (13:50.192)
prepare us for in case we need to fine tune or fix something later on when something is in production, for example. Yeah, absolutely agree with you on that. And Ian, I know that you have, on your previous professional engagements and experience, have worked on companies where there were LL platforms that reduce underwriters' review time by 97%.

Ian Cook (14:02.263)
grid.

THE HUMAN PROTOCOL (14:19.366)
I want to ask you a little bit about that, how you guys got there and what are the things that you can share with all of us on how to approach until you can reach that type of solution.

Ian Cook (14:33.518)
So the company you're talking about is a company called Sixfold and they do AI for insurance underwriting. And I think this is a killer application of LLMs. think they have a fantastic product.

The reason we could get to that stage and they can get to that level of improvement is literally what we're talking about is they have a very particular problem. In general, the idea is this is an insurance underwriter who's looking and possibly underwriting some case for a small business or for health or even individual health.

THE HUMAN PROTOCOL (15:05.915)
Mm-hmm.

Ian Cook (15:10.018)
faces a case situation where they have an enormous amount of information in terms of documents and reports and background and forms that have been filled out by doctors and individuals and legal and all that comes in, they have to review it all and they have to say, they have to pull and affect the diamonds out of the stack. They've got to find those needles in the haystack that are.

aligning with what that insurance company finds, you know, as a finds a useful signal as to whether to provide the underwriting or not provide the underwriting. This is a fantastic use of LLM. So what they've done is they found a very clear problem. The sheer amount of time it takes in those documents. They found a great tool to do this. You know, one of the key things to do with LLMs is to process all of those documents and find the information. And then you spend a great deal of time, which they have been doing.

THE HUMAN PROTOCOL (15:49.542)
Mm-hmm.

Ian Cook (16:03.008)
about making sure the LLM finds good information, the correct information, finds all the information that they need. So for anybody that's done a little bit of statistics work, there's always false positives, false negatives. And that's where they focus their time and focusing on the quality of what comes out rather than, OK, well, the LLM does this or that in terms of how it processes things. But they have to worry about processing the documents and getting the right information out. But I think the reason it was

successful at reducing so much of that time is a very focused problem with the strengths of the tool that they selected and very much in the way we've talked about. They had a very small process at first and said we're going to do this one thing and do it well and see what we can get from it and once they've gotten that they said okay now we can start expanding add a couple more features different types of insurance because you know obviously property and casualty is very different than health

THE HUMAN PROTOCOL (16:51.94)
Right.

THE HUMAN PROTOCOL (16:56.582)
Right. You start adding other flavors to it once you have the solid foundation, right?

Ian Cook (17:01.058)
Yeah. Yeah. So they've been great at it. And, know, I'd expect to see even better stuff coming from them. But I think this just plays directly into what we've been talking about here. The other thing that they focus heavily on is describing that problem. So a clear understanding of what their user problem is. You they've got people who are experts in the insurance world. They have people that are talking constantly with the underwriters that that they

want to use the product and they can tell you immediately, here's the problem we're solving, here's the way we solve it, and here's the, you know, when you see them go, they experiment, they try things, some works, some doesn't, some new tool works, but not as well as you expect, and you evaluate. One thing I will say as we get into these prototypes is always have a plan for evaluation. You know, your definition of good, what does it have to do well to consider it a success?

lot of places will just be like well you know did it reduce costs it's kind of a too large category I'd say yeah yep so those are the things I think that that really drove towards hitting that number of the 97 reduction

THE HUMAN PROTOCOL (18:05.818)
That's vague of a scope, right? Absolutely. Yeah.

THE HUMAN PROTOCOL (18:16.292)
And that was, and that's impressive, Ian. Thanks for sharing that. And that was an in-house developed LLM, right? It's a proprietary product.

Ian Cook (18:30.194)
I don't want share too much. Those are a lot of those are company secrets. I will say this. They really embraced the idea of experimentation. So you can think about a company looking out at the world going, all right, do we make our own? Do we use off the shelf ones? And I think what we're seeing, and I, you know, we can get into this as well. The general trend is there is no one answer.

THE HUMAN PROTOCOL (18:47.893)
Different on Ryan.

Ian Cook (18:57.366)
Right. You can't go to open AI for everything. You can't go to anthropic for everything. There's a huge push towards small language models, highly, highly verticalized, highly specialized. So it's been a little while since I've been inside and that's fine. So, the idea is

THE HUMAN PROTOCOL (18:59.973)
Yeah.

THE HUMAN PROTOCOL (19:04.186)
Yeah.

THE HUMAN PROTOCOL (19:14.414)
I'm so glad you mentioned this because on the end user side, regular human beings, that's why we're also getting notice from people, some LLMs and some product works better for this, other for that. So I think not one size fits all. as you were saying, experimentation is key until you find the right key, the right recipe to the problem that you're trying to solve.

Ian Cook (19:25.006)
Mm-hmm.

THE HUMAN PROTOCOL (19:44.262)
thoroughly.

Ian Cook (19:45.002)
Yeah, and part of that experimentation, I'll just repeat it again, is failure. When I talk about projects, I want to hear what failed. And you have to make sure that your culture is one that people don't hide that. One of the things that at a company that I always worry about is if no one is raising their hand saying, could I get some help? Or this didn't work and I tried it and I'm not even sure why.

THE HUMAN PROTOCOL (19:48.485)
Yeah.

Mmm.

Ian Cook (20:08.438)
You want a culture of people who are willing to say, I tried, it didn't work, let me move on to the next. Because if people are hiding like, well, I'm just a little worried that that's going to make me look like I can't do my job or, you know, I'm trying this new tool. You know, I think one of the things right now with some of these LLMs is people are trying it, but they don't want to admit that they're trying it because they're worried about like somebody is going to say like, well, then what do I need you for? You have to be able to stand up and say, look, this didn't work. This worked well.

THE HUMAN PROTOCOL (20:31.608)
if

Ian Cook (20:36.44)
Like here's what we're doing to move on. And you know, as the phrase goes, fail forward.

THE HUMAN PROTOCOL (20:41.732)
Yeah, it's incentivized that part as part of the culture, right? It is okay. It is not possible that a project will run without any problems. We all know things and problems happen, but I agree to you, it's better to be told and be known on the things that we really need to pay attention to before it is too late. So those are the type of the things that I think any project and any organizations definitely need to work on.

Ian Cook (20:54.317)
Mm-hmm.

THE HUMAN PROTOCOL (21:10.846)
And Ian, when we discuss LLM and AI, there is still a concern out there about hallucination. And I really want to get your thoughts from two points of view. If you can tell us from your understanding, experience, and knowledge what hallucination means in plain words when we're working with LLMs.

Ian Cook (21:23.128)
Right.

THE HUMAN PROTOCOL (21:39.362)
And based on your experience, what are the things that companies have tried to reduce the amount of hallucination on internal systems and et cetera?

Ian Cook (21:52.078)
Certainly. The easiest way to understand a hallucination is when the response that's generated by an LLM, maybe you're just using a chat bot or it could very well be in sort of a production. Well, unfortunately they do show up in production, but even when you're using the LLM as a piece of a pipeline and you're not just say signing into open AI and starting to type where the LLM will

provide an answer that is not part of the data that you've given it or that is just factually incorrect. You know, it reports that unfortunately David or not unfortunately but it says yes David Bowie is an active musician still making music in 2026.

And you think, well, that's not correct. So then what happens is you think, all right, is everything in here not correct? So you start worrying that everything that it's produced is a hallucination. It's generally not that bad. When hallucinations happen, it is often...

because the LLM has been given a very broad ability to respond. It's just like, you know, tell me the best musicians of the past 20 years. And then one of them is listed that you're like, wait, that person stopped producing 25 years ago. And it just, the LLM was just not great at the math and it didn't do the numbers terribly well. The resolution to hallucination is a...

going industry and there are tons of ways that people try to do this. ultimately what we have to face is that the output of these LLMs is what we would call stochastic or non-deterministic. It's just a fancy way to say it's not gonna produce the same thing every time you ask it to do the same thing. So if you put in the same question multiple times, you're likely to get slightly different answers.

Ian Cook (23:49.192)
work on this you can do a number of things. Yeah, very basic is in your prompts if you have documents that you're putting in like please tell me summarize this PDF. You can add to your prompts try to guide it to say do not use your own information refer only to what is in the document. Only summarize what you know about this don't don't infer anything don't attempt to tell me what it means just tell me what's in the document.

That can often help guide it, you know, to keep the guardrails there. People will do things like fine tuning and saying, giving it the chance to work off of just a very narrow and particular set of data that you can help fine tune it on to help, you know, control and narrow it down how much it will try to look other places for answers. You can use...

That's those are, you know, you can use what's called rag. I'm sure people have heard the phrase, but not quite sure what that means. Retrieval augmented generation. This is the idea that the LLM goes to either a database or a set of documents and pulls things out of it to put into its window before it answers the question. So like some people can upload a PDF or an image to the LLM and say only talk about this.

Well, there's a way to get the system to pull in its own information and say only refer to these 10 pieces of information when giving the answer. And it's used some intelligent process, which is called the retrieval, to find those 10 pieces of information. All those attempt to limit hallucinations. On the other end, you can get into detecting the hallucination. So you can imagine, you know, if you happen to know music really, really well and you know David Bowie passed away and says he's working in 2026.

Well, unfortunately, like that's just something as an expert you happen to know. Detecting hallucinations. Now we get into the world of evaluations, tests that you can run. A lot of times these are human in the loops. Here's what it produced, pass it by an expert. know, if it's producing things regularly, use humans to say this output is quite good. This output is not as good or I'm finding errors. That's a human in the loop process. That's one way to go at it. A lot of times people will design their system.

Ian Cook (26:07.628)
to cite its sources. So if anybody's ever used perplexity as a search engine or ChatGPT is doing this a bit more now, if you give it documents, it'll have a little indicator saying, hey, this is where I'm getting the information from. So you can click on the document and say, did you get the correct information? So that's one way to help detect those hallucinations. And as I mentioned, there is an entire world that's very technical, but

THE HUMAN PROTOCOL (26:20.784)
from.

Ian Cook (26:35.448)
really, really useful that gets into evaluations. How to evaluate the output numerically, qualitatively. You know, you'll hear people say something that's the LLM is a judge. So you take what's output by chat GPT and you put it into Claude and say, is anything here incorrect or is it incorrect? You can do things like ask for confidence in the prompts and say like, how confident are you in this answer? And we'll give you a number and it's often not, you know, a hundred percent.

THE HUMAN PROTOCOL (26:39.92)
Hmm.

Ian Cook (27:04.76)
So those are ways to detect hallucinations. Unfortunately, the ability to eradicate hallucinations is right now out of our hand or not something we have in our hands. I think there's a fantastic company, a woman who used to be at OpenAI, her name is Mira Maradi. She started a company called Thinking Machines.

THE HUMAN PROTOCOL (27:24.55)
I read about it, excited about it, right? Exciting stuff.

Ian Cook (27:27.534)
Yeah, they have a really interesting product off the shelf right now called Tinker to help fine tune models, but they released some research and I not much has been done with it. And I, because I, they are hinting at their own internal tools where they said they've been able to eliminate hallucinations and make things actually deterministic so that you get the same answer each time. I'm going to be fascinated when that happens. But of course that's all cutting edge stuff. That's not going to, that's not something that your massive

Fortune 500 company can just go, you know, put a credit card in and get off the shelf and suddenly realize that they'll never get hallucinations.

THE HUMAN PROTOCOL (28:05.658)
Yeah, I know. Thanks for going over all that. It's as long as needed, right? Especially with this type of conversation to explain those that may not be fully familiar with it. There is a lot of work from what we use in the front end to what's happening in the back end. There is a lot of work that I usually call it boring work for some people because they don't understand what's really happening behind the scenes to make all these new technologies and tool work.

Ian Cook (28:09.069)
Sorry if I was a little long-winded.

THE HUMAN PROTOCOL (28:35.244)
It's really people and work needed to make this work and to improve it. And the hallucination is a clear example. As you were saying, humans in the loop, more prompt engineering in the back end, right? To fine tune the outputs that these tools provide to us, all that this company have to invest on to continue to improve it. So it's really fascinating what this new world with AI

it's bringing to us and what's more fascinating still what's happening in the back end that a lot of people don't have access to it, right? To how to make this thing work, right Ian? We don't talk a lot about that. People talk about the magic stuff. No, this is really work happening behind the scenes.

Ian Cook (29:12.109)
Yeah.

Ian Cook (29:23.584)
One of the ways I've seen bigger companies, and I think this has worked really well, and I like it as well, and it's something that comes just from typical data science kind of thoughts is as part of specing out things, when you build your requirements, have...

An example or multiple examples is even better that has a right answer and say this system has to be able to get the right answer repeatedly. So there's something you want to automate. There's some tool that you want to make and to help solve the problem.

THE HUMAN PROTOCOL (29:50.565)
Hmm.

Ian Cook (29:56.546)
Go through the process of saying, you know, here's what it takes to do this and here's what the right answer is. So perhaps it's some analytics, perhaps it's going through claims checking, or perhaps it's some internal process that, you know, you're resolving books and it comes out and says, this is the right answer at the end. Have that right answer written down in those steps so that part of the human evaluation later is to say, I gave it 10 cases. It's getting 10 out of 10, you know, 95 % of the time and we'll work on those other 5%.

that becomes a great way to say, is this doing what I need it to do in my system? And I'm giving it like, I know what the answer should be. Cause a lot of times when people go in, they're not entirely sure what the answer should be. They want it. They want to trust that the LLM just figures out the right answer and this is what it is. A great way to test that you're getting what you want is to have a good answer written. So it's like having the answer key next to you and seeing if your student can reproduce the answers for you. Because then you can also look at the process. How did it get there? Is it getting it there?

interesting way. So as long as it gets to what I believe is the right answer, great. Now we know and can trust, which is a huge part of this, can trust what the LLM is doing.

THE HUMAN PROTOCOL (31:06.02)
Yeah. Awesome. Fascinating. What is it that you are willing to trade off? Right? Because we may not get everything that we want, but at least you need to be able to say, OK, I need to get this as an output. I'm expecting this to call this a successful implementation or a successful output. And I'm willing to give away this. Right? And all that should be part of the conversation as we are talking about

Ian Cook (31:12.481)
Right.

Ian Cook (31:20.12)
Mm-hmm.

Ian Cook (31:28.311)
Mm-hmm.

THE HUMAN PROTOCOL (31:33.776)
How will this impact production changes, for example? And when you're processing a lot of information, a lot of documents, a lot of outputs at once, for example, a day, you need to know what are you willing to trade off.

Ian Cook (31:46.658)
That's a great way to put it. love that you can have that conversation where you say like it just has to do these things. And then you get into, especially when you do this by committee or get larger stakeholders and you get people up, hire up the company.

THE HUMAN PROTOCOL (31:52.23)
Mm-hmm.

Ian Cook (32:00.46)
Well, it'd be great if it also did that. And it'd be great if it also did that. So you have this huge human process of like keeping expectations in line. And then maybe, you know, somebody wants to write down, well, maybe there's some stretch goals for this proof of concept or this prototype has some stretch goals. you know, pretty soon your stretch goals become expectations and, know, just these things tend to grow. So it.

It is often a hard job to sit across from somebody who's like, look, I want to invest time. I'm to let you kind of go off and do this exploration and to tell them like, I can't do what you want because it's adding too much and we won't know how successful it is at one thing if it's got to do these other four things. That's a tough place. But that's to me where the human still comes into this, right? Is, know, I owe the phrase I use is AI is fantastic at how, but not why.

Why am I trying to do this? Why am I solving this problem? Why this problem instead of the 300 other problems?

THE HUMAN PROTOCOL (32:54.64)
that why it's our, it will continue to be our, right?

Ian Cook (32:58.828)
Yes. Yep. That is what taste and expertise and judgment like, you know, the tool is just sitting there and it does not have, and I don't foresee in the future, it being able to say, you know, here is what you should be working on today because it's just more important than this other thing to your business. It's like, well, you know, I disagree. I'm the, I've been the CEO for forever. I've been the CFO or I'm just like, you know, a lead engineer and you know, this thing is

making everybody, it's what we call quality of life improvements. Everybody just complains that this exists. I need to work on that so that I can retain people. Whereas the LLM is just going be like, here's how to reduce your cost by 1%. And that's the most important thing because I've said so.

THE HUMAN PROTOCOL (33:44.614)
Yeah, it's a, they are tough conversations in that happens that have to happen in between. And sometimes it's not easy conversation, but definitely needed conversation. Right.

Ian Cook (33:57.027)
Yeah, so did the I'm sure you have some terrifically senior people that listen to this podcast. My advice to them is when somebody comes to you and wants to try one of these things, listen to them about the limits, listen to them about the scope and accept that this is a quick product and you should see results and help them think through what the best results for that small item are instead of saying like, wow, but I really would love that to radically change all of our legal operations or it'd be great if like I could

just totally change how we do customer success tomorrow. It's like, well, you know, that's just expecting too much out of something as you've said is not magic.

THE HUMAN PROTOCOL (34:29.35)
Mm-hmm.

THE HUMAN PROTOCOL (34:35.634)
It is not magic and people needs to be willing to go through it with us. And there is a cost for organizations just to pick the cool product instead of the right one. And you can only get to the right one if you go and you do the hard work, which is understanding what you need, understanding what you need to work on and telling you what is the capabilities of the tool.

Ian Cook (34:50.497)
Mm-hmm.

THE HUMAN PROTOCOL (35:02.82)
right? What we can get done now and we cannot get done now. Right? It is important and I cannot emphasize that more than in conversations like this and guests like you that brings this clarity to people and organizations. It's definitely needed. It's definitely needed.

Ian Cook (35:20.62)
Yeah, I've definitely seen that in my experience.

THE HUMAN PROTOCOL (35:23.942)
We see it. We see it even today more often than needed because I think also in some organizations and companies, leadership sometimes they're not caught up with what AI means and how they can leverage and what are the capabilities of it. And I think decisions are still being made in a way that may not be fully understood on what is it that we can get from it.

Ian Cook (35:51.276)
I totally agree. And we may still be, I think we're coming out of it a little bit, but we may still be in this era where a bunch of senior leaders who are working in sort of the non-technological fields hear about this and think, well, we need to be doing it. We need to be doing AI. AI should be reshaping everything. And the worst case outcome is the leader who's just like, all right, go find me a third party company that can bring us some AI.

THE HUMAN PROTOCOL (35:59.696)
Yep. Yeah.

Ian Cook (36:16.622)
And it's just like, you know, there might be a ton of third party companies that will agree to that contract. But if you are not thinking through what's there, just because you were afraid to be missing out, then you're just causing for yourself a massive headache and a bunch of costs that you could have avoided. Hopefully. And I think people are understanding now that like, look, you know, this needs to be thought through. This is something that will probably change drastic parts of our lives and how we work. But

We don't want to go at it without focus and without purpose. And that comes from the people that are using it. The people who are going to have to employ this tool all the time and use it. Even if you're attempting to automate some processes, people have to use what goes into it and what comes out of it. If you think that you can just eradicate that without a conversation about how that's going to happen, you're just building a recipe for failure and for high costs.

THE HUMAN PROTOCOL (37:16.76)
There are so many benefits if this is done right. We all know and we all have heard about efficiency gains. And for sure there are efficiency gains, but it requires a mindset shift in organization and it requires leaders that are really willing to understand how they can leverage this in their organizations. It's not plug and play. There is a lot part that it becomes part of your culture that you need to

make sure people embrace this in a way that they understand that if leveraged the right way, it can start providing more efficiency to them. But that needs to come from the leader understanding what are the capabilities, what are the things that can do or cannot do. It is extremely important and organizations, if they want to step up into the future, leaders needs to be AI leaders for sure. I don't think there is any other way right now.

Ian Cook (38:06.638)
Yes.

Yeah, you're going to have to know about these tools and know what they are. When people bring them up and what they can do. You know, there's plenty of jobs that are not going to have to be. Experts in every single tool that's out there exactly what happens, but under having this concept of what it's doing, how it works underneath, having played around with it, having seen what it can do for people at your company. I am a big proponent of AI as a bottom up kind of initiative.

So talk to people about the problems they've got instead of assigning like a center of excellence and say, now AI is going to be everywhere. You know, certainly internal champions are great. Find someone who can talk to the different groups and say, look, there's like 10 different areas where people are trying it out or we think there's a good opportunity for it and picking one or two trying very small tasks, very small targeted tasks. But hearing about that from the people who every day are going through these processes.

ends up working much better than the process of, you know, somebody at the top going, need AI tomorrow. you know, Margaret, you're in charge of the AI initiative report back to me every month about how much AI we have at the company. Yeah. Yep. And like, you know, until this thing brings down our costs, reduces head count and increases sales. It's a failure. And it was like, okay, well, nobody wants that. Like, you know, that's just not a job that I would want to be.

THE HUMAN PROTOCOL (39:14.395)
Yeah.

THE HUMAN PROTOCOL (39:23.354)
These are the KPIs, these are the metrics, right?

THE HUMAN PROTOCOL (39:31.077)
Mm-hmm.

Ian Cook (39:39.022)
pinned with because then I have no idea how to prove success without like radically changing the company. And then you're going to face all of this distrust and all of this headwinds. So

THE HUMAN PROTOCOL (39:50.018)
And there is no better way for human beings, because at the end of the day, we are working with human beings, changing some part of their jobs. There is no better way for someone, a human being, to embrace change than actually going over and understanding how this is solving their problems. Right. And I can agree with you more on that, that I think the bottom up approach is really what it makes sense. If you really want your organization and culture to embrace this change.

is start getting to their level and understanding what we can fix for them and show them how we can improve their lives. And I think it's a much better way for people and organizations and cultures to embrace the change and move on as an organization, right? And they will continue to embrace it because, I mean, we all like doing things that add more value to our daily jobs instead of just the manual tasks that are repetitive and no one like, right?

Ian Cook (40:24.013)
Yes.

Ian Cook (40:45.934)
Exactly.

THE HUMAN PROTOCOL (40:46.736)
So it's just need to start there, but it will take leaders to really understand that how you approach this makes a total difference for the success of the implementation.

Ian Cook (40:58.062)
Couldn't agree with you more.

THE HUMAN PROTOCOL (40:59.31)
Yeah, absolutely. And yes, we talk about leadership and organizations and how that can be implemented successfully. And I think we all need to be part of where this AI maturity is going next. And just to close this cycle and part of the episode that we're talking about leadership, what is

that leadership mindset shift that you propose to leaders so they can embrace AI in a way that you think will make the organization successful.

Ian Cook (41:40.556)
That's a terrific question because I think this is a make or break thing depending on your culture. I would say to leaders that a great place to start on this is to think of it as a as a pole instead of a push.

Talk to people about the work that they do and the places that they think they might be able to utilize these tools. People may be coming to you saying like, hey, there's something out there I really want to try. And am I allowed to do that? Can I get a subscription to something or can I try this set of tools? And to be open to that because that means somebody is looking actively to improve what they're doing, to speed up what they're doing, to deliver better for your company.

natural reaction of like, well, suddenly that's going to leak all of our IP and, know, we're all going to have, you know, all of these companies stealing what we're doing. There are a lot of things you can do to mitigate that. You know, I recommend pouring all of your corporate IP into a chat GPT and just hoping for the best.

But you can do things like getting enterprise accounts with these companies that will come with guarantees that they do not train on your data, that they don't even cache your data for very long. Meaning when you send it to them, it doesn't sit around ephemeral in their systems for very long and see what people do with it. And when they come back to you and say, this is how I used it, and this is what I did, and this is the things that I found from it, and maybe they don't like it, maybe they do like it.

be willing to hear how these people want to improve. There's often the fear, I think, with these tools that people will say, like, well, I did this with OpenAI or I did this with ChatGPT, and I saw that it was doing what it used to take me all day. It did it in five minutes. The answer to that and these leaders is not great. I need fewer people right now.

Ian Cook (43:45.173)
It's how much more could I be doing if the people I have were this effective and efficient and enjoyed their jobs? Like if there are things that are bumps in the road that are just some manual things that they're finding ways to get done quickly and with the quality that you would expect, then terrific. And then encourage them to not outsource what makes them like truly individuals and human. They have developed expertise. They've developed taste over time.

THE HUMAN PROTOCOL (44:12.262)
Hmm.

Ian Cook (44:13.654)
that LLMs will never have that. So why you ask a certain question, how you ask it, what you need to get out of it, reviewing what comes out of it, your time.

It should not just be, can turn this into an automatic process and then forget about it. It's, it used to take me all day to do this. I can now do it in five minutes, but the next hour should be reviewing it. It should be looking through how would I change it? How would I edit it? And then maybe you can help improve how you go through that process, but don't give up what makes your background history and experience special. And for the leaders, cause that's the way you asked the question. And I really liked the way that you asked that is

Talk with people about that aspect of it, not, you know, well, did the LLM come up with a hallucination or did it, you know, answer something wrong one time or getting very particular about it. Hear what the person is saying about how it helps them with their job.

THE HUMAN PROTOCOL (45:09.006)
And I hope that also what this new feature and this new transformational period take us to, I really hope we all as leaders find a way to make their new jobs, right, whatever their new job is, because jobs are going to change, enjoyable. And we continue to find ways to talk to our people and bring them together in a way that they continue to enjoy.

Ian Cook (45:30.946)
Mm-hmm.

THE HUMAN PROTOCOL (45:38.318)
whatever the new task or the new job is, right? Because it's something that we don't discuss a lot, But if the role changes, we may have people that may not like what they are going to do next. So there is going to have to be a lot of conversation with our people, our cultures, to make sure that we continue to find ways to make whatever the new role is enjoyable for our people. I think that's as important as well. Yeah.

Ian Cook (45:41.325)
Yes.

Ian Cook (46:06.467)
That's great.

THE HUMAN PROTOCOL (46:07.878)
Yeah. Ian, you've been there, done that in a way. What would be a piece of advice that you would give to other companies developing, building AI systems today? I'm talking and I'm asking Ian, the one that has, it's in the backend, working, developing, right? Safety in an AI.

Ian Cook (46:13.419)
Mm-hmm.

Ian Cook (46:30.648)
Mm-hmm.

THE HUMAN PROTOCOL (46:34.118)
and so many other initiatives. What would you tell their companies starting today in building these AI systems?

Ian Cook (46:45.496)
We've covered quite a bit that I would use as an answer to that one is never be afraid to experiment. Let people fail, like encourage people to try and fail. And I use the and there specifically try a bunch of times and fail a few times and tell me how you got better after you failed.

I would say do not wait. One of the benefits of finding a very targeted problem to go after is the speed with which you can go after it. So don't wait until the timing seems right or until everything seems perfectly set up. Focus your energy on identifying that very specific problem. Identify exactly how you want to see the result come out and give it a try.

benefit of code happening this fast is that the speed of failure has sped up. The cost of failure has dramatically reduced. So these things are,

adoptable at a speed that I've never seen in my career, but they're also easier to toss away. You tried it, it didn't work, great, you can move on to the next. So do not be afraid of trying it and seeing it not work. The problem with the huge method of going after a ton of things is that the sunk cost when you're that far in gets very, very high.

some very very tactical things is really really focus on output evaluate what you're getting make sure you can find those hallucinations make sure you're doing everything you can to reduce them make sure you've got something that says this is how to know that it's doing what i need it to do but a lot of that's that documentation that you talked about set requirements set specifications set define what is good define what is done and how to how to measure it

THE HUMAN PROTOCOL (48:29.05)
Hmm.

Ian Cook (48:40.406)
right from the outset. you know, just assume that the middle part that is code just happens. Great. What do I have to see on the other end? Don't worry about the whole, like everybody arguing about like, well, it can't do this and it can't do that. Great. You'll, you'll confront those when you're building the thing, focus on what it needs to do, the problem it solves and writing that down in very clear language.

THE HUMAN PROTOCOL (49:01.584)
Thanks for sharing that great piece of advice. What do you want to hear or see from governments as far as regulations or initiatives from the government side on building safer AI systems for all of us?

Ian Cook (49:22.392)
So I've spent quite a bit of time working with the federal government. I think it works best when it sets goalposts or at least measures to say, this is what you have to do in order for us to consider it safe and usable. So this is the sort of process you see at the FDA or at the National Institute of Health or other places that deal with things that could be dangerous. And to a large extent, these things can be dangerous.

I think the danger and the effects of social media over the past few years, especially on kids and mental health, has gone completely unchecked. And we are only now understanding the cost of having left that entirely unregulated. I think, you know, if I could wave a wand, would do age gating, which I know is incredibly controversial among tech folks because they, you know, bring up a lot of

very correct issues with age gating, but having regulation that says, you know, in these situations, this is what we'd expect to see. So again, this could be things, think of it a bit like evaluations. If I go into a chat bot and I put in something I expect to pass regulation, it can not ever say, you know, sure. I, know, there was a great,

article apparently people were able to jailbreak is not a great term but essentially fool the LLM into giving things that it normally wouldn't by doing the prompt in rhyme so for some reason the LLM was fine with like responding with this was this isn't what it was but if you can think of it

Similarly, you know dangerous thing like here's how to make a nuclear bomb but in rhyme like things that if you asked right now to Claude it would say like I'm sorry. I'm not able to do that. You see this in a lot like hey make a picture of this celebrity a lot of times would be like we don't have the capability to do that but there's ways you can get around it so the government should be saying it here's a raft of things that for red teaming it has to absolutely be rock solid therefore, you know, we trust it. I think the

Ian Cook (51:35.0)
process of government trying to get into the number of flops or the amount of data that it was trained on is just sort of a weird horse race that they're always going to be behind on, you know, because the models are going to get bigger and they're going to get faster. Or you get things like from China where they figure out how to train it on far less data and with far less compute, but it's way more capable. So that regulation that says, well, you know, this many flops and this much data is where we need to get involved. Like now we've just found a loophole around that.

I, the other big thing I think that is coming up is synthetic relationships. Those tools where you can essentially define what the other side is, but then people get into the belief that that other side is sentient, is a good replacement for human interaction, is a good replacement for parents, for loved ones, for physical relationships. We are opening a Pandora's box if we just let that go without regulation.

THE HUMAN PROTOCOL (52:24.998)
Hmm.

Ian Cook (52:32.104)
We've seen that with social media. know, amping that up on steroids times 10 just feels like a bad choice.

THE HUMAN PROTOCOL (52:38.116)
Right.

Now to your knowledge, are there any best practices or guidelines, federal guidelines that are provided to companies like yours or working with automations and AI?

Ian Cook (52:51.598)
I can think of very little regulation that it, there is regulation on digital material that applies to people just having, I apologize for bringing up horrible subjects, but photographs of things that they should not have. So there isn't even a clarity of what is acceptable from these tools.

THE HUMAN PROTOCOL (53:08.944)
Right.

Ian Cook (53:16.832)
you happen to be an X user, there is a recent trend on X on having Grok, G-R-O-K, the X version of Grok, not the Grok that just did the deal with Nvidia, to draw, like to draw a or make a photorealistic picture of people without clothes on. There is no regulation for that, even though if that were a real person, depending on the age, that would be, you know, a major felony. We are

THE HUMAN PROTOCOL (53:29.904)
Mmm.

Ian Cook (53:46.318)
in a scary wild west to me. And I feel terrible that I'm ending this on such a downer note. I don't want to do that. These are amazing tools that are going to tremendously improve the work lives of so many.

THE HUMAN PROTOCOL (53:57.52)
But it's a reality we need to face as well, right? We should not be blind towards it. It's the good and the bad. That's what these conversations make it real for people.

Ian Cook (54:00.364)
It is. No.

Ian Cook (54:06.956)
And kudos to you for bringing it up because people can go through this stuff just thinking, you know, it's all bright and shiny and the future is going to be terrific. are exceptional costs to these technologies.

THE HUMAN PROTOCOL (54:10.906)
Yeah. Right.

THE HUMAN PROTOCOL (54:15.726)
Yeah, yeah, yeah, absolutely. And last but not least, what would you tell a younger version of yourself going through this transformational period today? So imagine seeing a younger Ian today. What would you tell that kid or that younger version of yourself today?

Ian Cook (54:36.512)
I would say, hmm, is this a younger version of myself? If you were going through it today, I think I would be motivated or possibly inclined to skip the fundamentals.

THE HUMAN PROTOCOL (54:44.879)
Right.

Ian Cook (54:56.458)
So I think one of the reasons I've been successful at doing a lot of these tools and adopting these tools quickly is because I understand what's going on behind them. I've built tools that were machine learning, that were, you know, basically statistics on steroids and doing, you know, all of these things and turning it into those tools. This is a new and slightly different generation, but I have worked natural language processing, natural language understanding from the early days when vectors first came out.

I'm not saying, you you have to be the expert in all of these things, but you, without the fundamentals, that's when people start thinking of this much more as magic. So for me, getting into this as a technical, a very, very technical level, don't skip the fundamentals. If I were, you know, talking to someone who were, who was facing this world, but not necessarily looking to code or to work on it is do not be afraid. Don't think this is going to turn into a sentient.

AGI monster and repolice all humans soon. Get comfortable with them, use them frequently, understand what they can and cannot do, what are they good and not good at, and try a ton of them. Just keep trying them and figuring out what's going on so that when you do apply for a job and you talk to somebody and they're like, yeah, we use these tools, you can go, that's great. I've used that. You know, I found that it's output on spreadsheets is a little sketchy or whatever the result happens to be.

THE HUMAN PROTOCOL (56:22.522)
Awesome, thanks for sharing that, right? And that's something that I also try to tell everyone, my friends, colleagues, everyone that I can be curious, continue to remain curious. This is a tool, try to learn it, try to leverage as much as possible, learn where it works, where it doesn't work, and keep trying. That's what we do, right? We are human beings at the end of the day. Keep trying and keep going. Yeah, thanks Ian. It has been a pleasure to have you.

Ian Cook (56:44.108)
Yes. Excellent.

Ian Cook (56:50.104)
Thank you, Michael. This has been a fantastic conversation. I really appreciate the opportunity to talk to you.

THE HUMAN PROTOCOL (56:54.982)
No, I appreciate you more and thank you for cutting through the noise, right? And really reminding us what real innovation is in the loud, is in the execution, is in the reality. So if this episode challenged you how you think about AI, share it with the leader, with the friend, with the family who's still measuring success by demos. And follow the human protocol and see you all next week. Thank you all.

Ian Cook (57:19.95)
Thanks so much.