95% of AI Projects Fail Because People Build Them Themselves w/ Issac Hicks | S2E17

Full Episode Transcript

This is the full recorded conversation from The Human Protocol Podcast, published by The Human Protocol Group. © 2026 The Human Protocol Group. Please link to and attribute the original episode when referencing this conversation.

THE HUMAN PROTOCOL (00:01.34)
Welcome to the Human Protocol, the podcast about staying human while building the future. I am your host, Michael Salomon. Today we're talking about the gap that nobody puts in a slide deck, which is the implementation gap, especially in times of AI implementations. The place where AI will look very perfect, right? But no one really knows what it takes to put it in the business and implement it and then run it.

Because really AI in production is not the problem. It is what we do and how we implement it. So this episode will be a practical map for everyone that trying to get from pilot to production, right, without wasting on budget, what it works, what it doesn't work. And with us today, we have our friend, Isaac Higgs. Isaac, welcome to the Human Protocol.

Issac Hicks (00:54.882)
Thank you, happy to be here.

THE HUMAN PROTOCOL (00:56.984)
Awesome. Thanks for taking the time to be with us today. I'll be looking forward to having this conversation. And to kick things off, Isaac, I would like to see if you can give us an overview of who Isaac is.

Issac Hicks (01:12.396)
Yeah, absolutely. So I am a technology implementer and operator, particularly in the AI space. I've been working in the mid market. So that's companies between five million to 100 million for the last five years. I've built dozens of applications, handled software migrations, done pretty much everything under the sun related to operations and mid market organizations. I've also served enterprises, done major technology migrations, and I've also helped a couple of startups build their initial MVC.

and pilots to raise money and get to the next level.

THE HUMAN PROTOCOL (01:46.97)
Awesome, so you are the guy we wanted to talk about. So let's kick things off, Isaac, with getting your thoughts and really why do you think most enterprise AI initiatives, they fail after the demos? What can you tell us about what you have experienced on those implementations with other organizations?

Issac Hicks (01:51.192)
you

Issac Hicks (02:12.044)
Yeah, so the biggest thing I see is that the demo is always perfect, right? That's always what gets polished up. That's always what the most kind of important part is whenever a company is looking to vent to an enterprise, especially because, you know, that's kind of where the deal is won or lost. You either get to the next level or you don't with that. But the thing about it is the demo and the conversation leading up to the demo, that's really only about five to 10 percent of the work that's actually required in order to make

and enterprise rollout work. And once you get into the details, things like requirements gathering, understanding edge cases, understanding exactly what adoption rates are going to look like across the organization, how this is going to be rolled out, and what the real time usage is going to look like until that's really mapped out, schematized out.

and set up in a way where you can respond to unknown unknowns and edge cases that'll come up, the project is very premature. And what I see is a lot of projects that remain in that premature state and then move forward in time and really never get to the level of specification they need to to make something work in runtime in an actual production use case.

THE HUMAN PROTOCOL (03:27.388)
When you and your company are brought in into these conversations, is it at the beginning? Is it at the middle? Because they need you to fix something that is going very wrong. What are the different flavors that you're getting out there?

Issac Hicks (03:44.364)
Yeah, so I get a little bit of both. I would say that we're about 50-50 on either starting from the very beginning and working with people from the idea stage all the way through to go live and then coming into a project that is halfway built, three quarters of the way built, wasn't built properly. There was some kind of falling out with the original vending team and then we come in to save the day, so to speak. And so we're very familiar with both workflows.

THE HUMAN PROTOCOL (04:11.586)
And is there anything in common as far as trends or insights that you have seen? For example, when organizations call you for the first time to go from working on the idea with them, what are the things that you think are important for organizations to have in mind before they reach out as an implementation company like yours, for example?

Issac Hicks (04:41.998)
So I think that you have to have a very good idea of exactly what it is you want to do, who's going to be affected, the processes that are going to be affected. All of this has to be very well understood and very clear.

And a lot of organizations I talk to, they come into the conversation ready to build something. But once we get to talking, it's really more that we need to understand exactly what the problem is that we're trying to solve in the first place and how we're trying to solve it. Because a lot of assumptions get made very early on that seem very obvious. But the thing about artificial intelligence is a lot of the ways that you get value are actually very counterintuitive. So, you know, it's

It's fairly common that we'll get into a conversation where there's just not an understanding of the true level of due diligence that needs to be done on the process side internally to understand what it is that we're doing in the first place and what problem we're trying to solve. Because a lot of the times what I'll see is an organization has a plan for what needs to be done, but that plan is based on assumptions that if they do that, it's going to solve the problem. And then once we interrogate that problem, actually

isn't going to solve that problem or they're trying to solve the problem in the wrong way. And that is a very insidious issue that you'll see blow up much later after a ton of work has been done if you don't catch it very early on is, is this actually the solution fit for the problem that we're having in the first place?

THE HUMAN PROTOCOL (06:14.512)
And that makes me think that's why it's also so important that you find the right partner to all of these kids. The last thing you want, as you are saying, is wasting time, resources, time, energy, money, finding out that what you were trying to accomplish is not really the way to do that. And organizations and projects, especially project teams, as you were saying, I think sometimes they are

trying to find solutions for problems that they have that may not be the right tool or the right technology for that.

Issac Hicks (06:50.688)
Yeah, absolutely. you know, one of the very common thing that we see a lot of actually, and this is the perfect.

the perfect ideation of how something that seems could go right actually does go wrong is a lot of people want to expand their marketing. They want to expand their cold outbound. They want to push cold outbound farther. And so their idea is, okay, we're going to set up an operation where we're able to find whatever trigger points, whatever is going on with our market, and we're able to reach out to these people automatically do follow ups automatically do all this stuff automatically. And that sounds great upfront. Why? Why

spend more time on initial outreach. Bye.

If you don't have your offer dialed in properly, if you don't have your top of funnel dialed in properly, if you are sending messages that are attempting to be personalized, but really have just scraped websites and say, well, I know that you did XYZ thing from your website. Not only does the person who opened that email know that it's AI generated and it's probably not going to respond, but it actually hurts you. It goes in the opposite direction because if you are scaling a bad process, you scale a mistake instead.

And we've done many initiatives where we come in and the problem is, hey, we blew through 10,000 people in our target market and we didn't get a single response, what happened? And then you dig into it and half the emails went to spam because that wasn't set up properly. And of the emails that didn't go to spam, you had 20 % unsubscribers because your targeting wasn't right. There's all sorts of things that go into that.

THE HUMAN PROTOCOL (08:28.85)
So many things to think about, right? And taking into consideration when looking at that thing. It all gets back to the very basic, which is planning. And I talk to these also to everyone. Planning is really what's gonna take you to the next level, right? The more you plan, the more you can anticipate what's gonna happen. And I think organizations...

especially big organizations, I believe sometimes it's not, they're not so good at planning and on trying to lay out how is it that it's gonna happen, how is it that they're gonna get from point A to point B. And especially with these projects, planning is also key to the success.

At Isaac, there is always a lot of conversation and especially in this conversation around AI. And we see companies and organizations cutting costs, cutting costs, cutting costs. Some say AI, others don't say AI, but it's everybody's looking at the return on investment. And I think for any business owner at the end of the day, we all look into what is going to be our ROI in all this thing, right?

What's gonna happen after we implement this in production and and some you know? Some people they look at ROI just as a cost saving and which you know It's a it's usually the short-term gain that people get from the stock markets for the most part right if you cut costs Or if you save it's probably your stock. It's gonna. It's gonna go high, but well, how do you think?

Based on all your experience implementing these solutions, how do you think the conversation around ROI on an AI implementation should go? More as an educational, from your point of view, but mostly to educate business leaders. What is it in it for them that it's not only the number that they could be looking at?

Issac Hicks (10:43.66)
Yeah, so there's two sides to the ROI equation whenever you're looking at AI implementation. And I think most business owners only look at the first side. So the first side, the one that we're very familiar with is, where can I save time? What processes can I improve upon where, you know, a process that took six hours now takes one, something that took 40 hours now takes 10. You know, that's that's pretty straight up. And a lot of business owners, when they're looking at this, the way that they try to associate the ROI is OK. So if I had

30 working hours an average cost of $37 an hour then that $30 times $37 an hour

Let's say that that happens on a monthly clip, so we're looking at a little over a grand a month. Is this worth a little over a grand a month? And then what are they going to charge me to do that? Are they going to charge me 20 grand, 50 grand to do this thing that's going to save me a grand a month? So the payoff period is going to be multiple years. That doesn't make any sense. Why do that? Right? Now the part that they miss.

is where could you reallocate those hours if they're not being spent there, right? Because, you know, the 30 hours to let's say you save 30 hours, well, what are those new 30 hours going to be spent on and how is that going to change the company? And it might be that what you're doing is you're automating the administrative process around your sales team and that 30 hours now goes to closing more deals.

THE HUMAN PROTOCOL (12:13.532)
selling,

Issac Hicks (12:13.71)
Okay, so what's the average amount of time that gets spent on closing a new deal? What's your average deal size? If it takes two and a half working hours on average to close a new deal and that new deal is worth $10,000 in lifetime value to you, then getting 30 hours back actually turns into 100 grand, 100 grand plus.

And so there's where the real ROI is. That second part is typically where the vast majority of the ROI actually lives. And it's not about cutting costs. It's about the reallocation of labor to things that are more efficient and more effective from a revenue generation basis. And that's really the proper way to look at these things. It's not how can I make the current state more efficient. It's what could the future state be and how do I get there?

THE HUMAN PROTOCOL (13:02.854)
And what about maintaining these solutions, Isaac? One thing is go live with this, but this will need maintenance. Basically, to keep these AI solutions running as accurate or efficient as we need, we know that there is documentation, training. mean, this is, the work continues from that point of view, right?

Issac Hicks (13:26.328)
Of

Yeah, of course.

What it really comes down to when it comes to maintenance is first of all, I mean, obviously I'm biased as an AI implementer myself, but I see a lot of organizations that try to do internal builds and the devil is really in the details because the average competent professional, you know, a technical executive, they can probably study up and get 80, 90 % of the way there on something that they're trying to do completely on their own. But it's that last 10 % that's really going to make or break you. And

You know, having a dedicated team of people who have been there, done that, who are able to run the support and able to be there is very, very important. And then it comes down to professional professionalism of management. Right. So it's, know, is everything documented properly? Do you have error codes associated with all of the different failure modes? If it fails in a way that is unexpected, does that get documented? How how does the ticketing system work? How does remediation work? Is there a workflow?

if your AI system is touching customers to ensure that a problem with that system doesn't result in customer churn. You know, there's a lot of different things that you need to plan for and it's really

Issac Hicks (14:42.562)
the whole thing where you plan for the worst but expect the best. And that planning for the worst is super duper important because you never really know what's going to happen. And even if you design a system to be very, very sophisticated, very deterministic, there's only so much determination you can get, especially out of a system that is going to be using something that is generative. So designing around non-determinant systems is, it's very complex. It's not the same as designing a traditional system.

and making sure you have people who understand the difference is very important.

THE HUMAN PROTOCOL (15:19.804)
Yeah, good stuff. And it makes me think, let's say that we're going through that project, right? You and your team is helping that organization to implement it. We are ready to go live. What are those metrics that you and your company are usually looking for to measure the quality of the implementation as far as

what is expected from the AI. I know the businesses that you're working with, probably they have their own goals, but just curious to hear from you how that works, how that interaction works between what you expect and what the customer needs.

Issac Hicks (16:02.572)
Yeah, for sure. So pre-go live, there's a wide battery of tests. Of course, we have user acceptance testing, making sure it works the way that the end user expects it to. But then above that, you have anomaly testing. So if you run the same query over and over and over again, are you going to get the same results or different results? And if you get differing results to any lie outside the range of acceptability, you have

THE HUMAN PROTOCOL (16:20.434)
Hmm.

Issac Hicks (16:27.598)
like adversarial tests, so trying to get the system to do something it's not supposed to do and making sure that the system does not do things that it's not supposed to do. A really funny example that I saw one time is whenever...

the whole AI chat bot on your website was a new thing. There were memes going around of people asking to solve differential equations and asking, where can I order a pizza and stuff on some random customer support website for a software company? like, yeah, you can order pizza over here. That's not what you want. That makes you a meme, right? So that's not what you want at all.

THE HUMAN PROTOCOL (17:02.298)
Absolutely, you don't want.

Issac Hicks (17:07.424)
And then there's also load testing, right? So especially if you're looking at deploying something at scale, if you're going to have 10,000 simultaneous jobs, 100,000 simultaneous jobs, well, did you make sure that you can actually have those many jobs? Did you try it repeatedly? Did you try 10 times that amount to make sure that even an unexpected burst load or even if, you know, there's something unexpected, you have redundancy there. If you have something built on scalable architecture that's serverless,

Did you make sure that every single instance of that serverless architecture is going to spin up? And if you spin up a hundred thousand instances at once and one percent of them fail, is it easy to to go through that one percent that failed that hundred? it able to now? Are you able to navigate that hundred and immediately understand what went wrong and why and write stop gaps around those? And these are the kinds of things that you have to be ready for and that you need to test ahead of go live because all of these things

will happen in the real world and you don't want to get caught with your pants down.

THE HUMAN PROTOCOL (18:12.754)
Yeah, absolutely. It's a quality, quantity, right? And it makes me think also, when projects are running, usually there is a challenge, right? Depending on the implementation of the project, on the amount of data that is used to test any product or system, right? I'm curious to hear from you if in an AI implementation, having or not having

access to the same amount of data that is used in production, it makes a difference or not.

Issac Hicks (18:51.64)
So it depends on the use case. For the most part, you want your data that you're testing and training on to be completely representative of the data that is being used in production. If there's some kind of...

bureaucratic process or some kind of sensitivity around being able to get access to production data, then you do need to make data that behaves in such a way that is similar to production data. has to be, you want to have as close to the real thing as you possibly can if you can't have the real thing. And in general, you want to just try to get the real thing. So let's say, you you have a...

THE HUMAN PROTOCOL (19:14.076)
like dummy data, right?

Issac Hicks (19:28.606)
big customer database, but you can't run anything against the customer database because of the security specifications that you have with those MSAs. Well, could you go to a limited number of those clients and see if you can use their data in an isolated test environment that is not going to be around the real world? Or could you ask those customers if you redacted different information, if you could still use the same information set? There's always room for negotiation around that. And those are the kinds of negotiations

that are worth having.

THE HUMAN PROTOCOL (20:00.806)
Got it. All important information, especially around data. It's extremely important, especially when working with these AI solutions. And let's talk a little bit, Isaac, about change management, And why do you think we humans, even though it's in our nature,

sometimes being reluctant to change. Why do you think in times of AI implementation it's even more important having a good change management in place for the project to be successful?

Issac Hicks (20:39.427)
Well, if you don't have good change management in place and you have people who are trying to do both the old way and the new way at the same time and something goes wrong,

it's going to be much harder to track down exactly what went wrong and why. And every time something goes wrong that adds more justification to the team of people who do not want to change in order to continue to reinforce not changing. And if you are attempting to do a change and you have a mounting amount of evidence against why you should not, then that's going to cause widespread disillusionment, really bad morale problems. You might start to run into employee attrition

problems and if your company is big enough and you're in the public eye enough those problems could be very big. mean we've all seen how like the the safety officers, the AI safety officers leaving OpenAI, what does that make you think of OpenAI? Nothing good right? Or what does it when you see you know like the chief operating officer of a company leave because of AI implementation and they don't feel like that's going well it's like what does that do to your company?

in the image. And if your customer base is really dependent on your image and your customer loyalty also is affected by your image because there's brand toxicity that could go there, this could very quickly spiral into something that gets extraordinarily out of control. So what seems like an internal change management issue is an internal change management issue until it's handled poorly and then it can very quickly become very public.

THE HUMAN PROTOCOL (22:18.278)
Yeah, can't agree more. Can't agree more. And in times that, you know, we talk about data, we talk about change management, and who owns what? And many organizations and everybody's asking this question lately. So who should be responsible for owning the outcome? When it's I come from a customer support background, right? So I know in my team who owns the outcome because we

for the processes that are very human, a human owns the outcome. When it gets tricky, and I'm already hearing people talking about this more and more, an organization having challenges with this is, if you are not running with the process end to end, and you have AI in between 50%, 60%, 70 % of it, at the end of the day, who owns the outcome?

Issac Hicks (23:14.575)
mean, it would be whoever is in charge of that process, right? And the person who is in charge of that process needs to be empowered to own the outcome. They need to understand everything that is required in order to ensure that the outcome is successful. And if they need technical support, then they need that technical support team underneath them. And that's another thing that I see go wrong very often is let's say that you had a process that took 20 people. Maybe it was like an inventory control, accounts payable, you some kind of process that's very repetitive.

very administrative heavy and you moved from 20 people to two people because you automated most of it with artificial intelligence. It's like okay well do those two people know how to deal with things when they go wrong? Are there manual overrides in place where they can go in and handle that? Are there monitoring systems that will proactively alert them when something goes wrong so that they don't find out that something went wrong after it's been too long and there's nothing that can really be done about it and you just have to eat the damage right? So

It's really important to make sure that I mean this goes back to the planning for things to go wrong planning for the worst, right? It's Is there any situation in which there is in exposure? to if something was to go wrong with this process you're in a situation where the Problem cannot be remedied in an amount of time that's acceptable and if the answer to that question is yes then I mean you have an exposure that you need to deal with and you need to deal with it sooner rather than later because anybody

who's been in business for long enough knows that not only will the worst case scenario happen, it is kind of to be expected. And so you have to plan for that.

THE HUMAN PROTOCOL (24:56.678)
Yeah, it's very important. Even when you may not like it, you own it. Even when you may not be responsible for 100 % of it. But at the end of the day, it's like, I see like any other process, right? Even if it's a human being making a mistake. But if you own that process, it's on you to figure things out. You just need to learn how the tool is working and how the automation is behaving and make sure that you plan around it.

Isaac, I have another question for you regarding hallucinations. In your world, when you are implementing these projects, how is that measured? How is that managed? How is that remediated?

I'm to hear your thoughts on this part, which I think is critical for the success of these LLMs.

Issac Hicks (26:02.297)
Yeah, so in general, are two main things that you need to do in order to make sure that you don't have hallucination problems. The first thing is when you're designing the system from the very beginning, you want to make sure that your outcomes are as deterministic as possible. And so even in situations where you're using artificial intelligence to do something such as flag an anomaly or do something such as make a decision in an ambiguous way, it's like, okay, well, the decisions to be made, there should be a finite number

of them and there should be clear understanding and reasoning as to why. One of the little tricks that you can do whenever you design an AI system is for every decision that it makes or for every output that it makes, make it give you the justification as to why it made that decision as well. And even if that isn't something that goes outward, if that gets put in a log or something, this does make a situation where the LLM has to self-correct or self-justify and that can cut down

on hallucination rate. The second one, which is also very important, is you want to ground your models and you want to ensure that anything that you are doing has some kind of source justification. You can do this in the same way that you do the logical chain, but what you want to do is, let's say you have an inventory control management system, right? And what you're trying to do is you are saying, when inventory

gets below a certain level based on the current demand that I've seen over XYZ period this is when the reorder point should hit right. The last thing you want is to not have a reorder hit point like a reorder point hit when you want or have it hit way in advance then you don't even have space in your warehouse right. So

If you have the AISA, not only choose the reorder point, but provide the actual data and justification as to why the trigger was hit, then you can get the information back that's like, oh, the trigger was hit because...

Issac Hicks (28:07.349)
of this reason and it's not enough to have that just go into the logs. You want the LLM itself. It's like, tell me why you are taking action. And if you can have the LLM tell you why it's taking action or where it got the data from or what it went to and have it link back to that data source, that's also very important, right? Because, you know, it is less likely that the AI is going to make up a source

THE HUMAN PROTOCOL (28:29.543)
Hmm.

Issac Hicks (28:38.272)
that you have already dictated, hey, you must pull from these sources because if you design your system right, it's gonna go to that source in the first place to determine whether or not you like whether or not it's going to make a given decision. if you.

require data to be input into the system or some kind of data gathering process to be part of the decision making process before the decision is made and the decision is made based on data that is coming in externally, that's going to greatly reduce the hallucination problem as well.

THE HUMAN PROTOCOL (29:14.236)
Great stuff. Good to hear all about that. And I think those are very good points to always take into consideration if you want to kind of test and pressure test these solutions, right? It's a good way of going in that regard, as you were saying, to ask the AI for an explanation of why that decision or how it came up with that decision.

Good stuff. So I want to talk to you next a little bit, Isaac, about something that companies, founders, everyone at some point will have to make a decision, which is buy it, build it yourself, or a blend between them. Basically, we are trying to implement an AI solution. Some organizations are leaning towards more like building in-house.

or actually buying it and then implement it, or a blend of both. And I'm curious to hear your thoughts and what do you see as working on each one of them and what would be your recommendation to go one way or the other and how to approach it.

Issac Hicks (30:35.437)
Yeah, so it's funny, I actually just made a long form YouTube video about exactly this topic a couple weeks ago. But so in general, the answer is buy it. It's either buy it or buy a of, you know, a couple of different tools and then tie them together with some kind of custom middleware. Building from scratch yourself almost never makes sense. And the reason it almost never makes sense is twofold. So the first part is it typically will only make sense

if the ownership of the intellectual property is very important and the solution that you're creating is truly novel and is giving you some kind of competitive advantage.

that is truly sophisticated, right? It's like, it's something that is part of a longer term strategy. This is something that you're basing your entire organization around, or it's something that you're going to sell. Like you're planning on making this system to sell its outputs or to sell access to the system directly. In those cases, it might make sense to build it, but...

In general, you don't want to build it because you have to go through the software iteration process of making a V1, releasing the V1, having all the problems with the V1, then creating a V2, then having problems with the V2, and then maybe the dynamic changes, etc., etc.

A lot of people that you're going to hire if you're trying to build it yourself, these are people who have done builds inside of corporations and have, you know, they might have 10, 15 years of software develop experience, senior software developer, 300 grand a year, Mac daddy shirt, right? But this character has most likely not built.

Issac Hicks (32:23.285)
many artificial intelligence solutions and is not a specialist in building artificial intelligence solutions for your use case. That's very rare that you're going to find that person. And if you do find that person or that team, it's going to be significantly more expensive than it would be to just hire it outright. So in for most cases, even if you are trying to do something that's relatively novel, what you do is you find all of the components that do the different pieces

that you need and then you build a middleware solution that ties all of those together and a traditional software developer that might have a little bit of AI experience is going to know how to do this in a much easier way than like building a AI system from scratch and if you are building an AI system from scratch and this is really like we're talking about building like a large language model with a transformer from the very beginning right

If you are doing something like that, there are specialist firms that do just that, that have a lot more experience than you're going to be able to hire in the market to do it. So there are some cases in which you want to do like build it in-house, but I would say 95 % of the time the answer is to either buy it or configure something.

THE HUMAN PROTOCOL (33:44.104)
95 % is high.

Issac Hicks (33:46.275)
Well, 95 % of implementations fail because people try to do them themselves.

THE HUMAN PROTOCOL (33:50.236)
There you go. It should make people think. It's like building a product, right, Isaac? It's like, mean, do you really want to spend time, energy, resources on starting from scratch on building these? And as you were saying, it's the technical capabilities. Do you have everything in house? So it's a lot of things to take into consideration. And yep.

Issac Hicks (34:17.143)
And I would say most of the projects we get where we're coming in halfway where it didn't get done was attempts to build it in-house.

THE HUMAN PROTOCOL (34:27.652)
and then they're trying to scrub that and reaching out to you so you can go take them to the other route. Interesting. And that takes me to the next question, right? A decision is made, right? Okay, I'm not going to build any house. It's too much work, you know, based on what I'm hearing today. Okay, let's go to make a decision on...

Issac Hicks (34:36.206)
Yes.

THE HUMAN PROTOCOL (34:56.466)
Who is going to be the vendor? Which LLMs are we going to use? How is that working out there? We have multiple companies, that we all know who they are. But how is that decision come? What is triggering decision going one way or the other? Is it money? Is it a capability? What are the things that you're seeing out there?

Issac Hicks (35:22.231)
Yeah, so there's there's kind of two flavors of this question, right? The first one is how do I choose the right people to build this for me or the right product, right? And then the other one is how do I choose between commercial large language models that all seem the same? So to answer the first one just very, very explicitly, because I've spent thousands of hours in each of them, OpenAI is traditionally overhyped, especially their latest releases. They they just don't really measure

up properly. They are more of a marketing play than anything else and I hate to be like you know the open AI hater but I personally...

THE HUMAN PROTOCOL (36:00.754)
You're crushing, you're crushing my shodgy PT right now.

Issac Hicks (36:04.399)
I'm sorry, I'm sorry. just, personally do not believe in the direction of the organization. And if you look at the benchmarking since GPT 5.2, they have decided that they're going to do all of their benchmarks against previous versions of their own software. And the reason they do all of the benchmarks against previous versions of their own software is because they can no longer compete with other software options. So they, if you look back in the past, it used to be, how

How does our model compare to Anthropix model and compare to Google's model? They don't do that anymore because they can't compete just straight up. It's not as good. Now Gemini, the Google models...

THE HUMAN PROTOCOL (36:43.144)
Wow.

Issac Hicks (36:48.917)
those are the best use case whenever what you're doing is very deterministic and the outputs are not going to be something that is focused on communication. And the reason for this is the way that they have trained their models is they train more based on trying to emulate the sophistication of decision making. And so if it's about

getting some kind of data in and then determining what to do with that data and push it out then that's going to be the best use case for that. Anthropic and the Claude

ecosystem that is primarily around a narrative. So a lot of their stuff is about making communications that actually sound human, the ability to steer the voicing of the model, their coding implementations, they're starting to go in that direction. Those are very impressive as well now. I personally am a big Anthropic fan of all of the stuff that I've seen. think that what they're doing is what I believe in the most. And then

AI, grok, stuff like that, that is the best use case if what you're operates in an area that you may have some kind of censorship issue or run into some kind of guardrails with other AI systems that you're not, like it's not necessarily an actual issue, but they get flagged, you know, overly. So an example of this would be

Let's say that you want to create a system that is supposed to be like a psychological self-rescue system. So if somebody is having a really bad time, they're in a crisis panic type of situation and they need to talk through the issue that they're having. It might make sense to build that on a system like XAI instead of building that on one of the other models for the sheer fact that what you don't want to happen is you don't want an output that could be helpful.

Issac Hicks (38:51.875)
be blocked because it's talking about something that is considered to be, you know, like sensitive. So like if you're, if you're talking somebody down from making a very bad decision, you don't want the output to be like, well, we can't say that because that's not appropriate. It's a, well, it's appropriate in this exact situation. Right. And so you're not going to run into those kinds of problems with X where you might run into those problems with other vendors.

THE HUMAN PROTOCOL (39:12.04)
Hmm.

THE HUMAN PROTOCOL (39:20.498)
Wow, a bunch of good information there. You're making me think about the future of my child GPT. I have to think about, especially with everything going on with OpenAI lately. So thanks for sharing that, Isaac. And let's say we implement, we go with Anthropic. The system is implemented.

And a couple years from now, things are going south. How does this work? Can you switch an LLM and put another one in? It's brand new project. Something goes wrong. For whatever reason, the company is not keeping up with the quality of the LLM. What would happen in those cases?

Issac Hicks (40:12.813)
Yeah, so we always build everything to be model agnostic where what we do is, you know, it's like it's all an API interface. And so we just build our API interface where you can just switch out what you're using. And then it just uses a different code and goes to a different system. And then we'll typically compare systems against each other whenever we're doing testing in order to choose which one's right. In the vast majority of cases, Anthropic ends up being right for most of the use cases we use, because most of our use cases are operational and do require

some type of abstract logic and abstract logic is really where anthropic can you know kind of shine in comparison to something like Gemini. Most, I'd say 90 % of the time you're choosing between those two it's either going to be a Gemini or an Anthropic. Gemini is much better at long contexts so if you have like very very long context that you need to analyze it's going to be better for that and then if you have outputs that are going to be

in non-English and in like non-traditional English lettering format. So it's like if you need to get an output that's going to be in Arabic or in Mandarin or something like that, then Gemini is going to be better for stuff like that.

Anyway, I mean, I could go on and on about these specific use cases of which model is best for which just because I have so much experience with that. But in general, what you want to do is you want to build your back end so that you can swap out very, very easily where it's as easy as changing a variable in your code base. And then if you do get to the point where you're looking at potentially selling this or doing something that requires more complete ownership, then if you have your back end set up in that way, you can switch to an open model.

you can host that open model within your own cloud instance. So you're running like AWS or Azure cloud or you're running Google cloud, you can just put everything up in there and you can build an API interface on top of that that works exactly the same way. So I just say remain flexible so that whatever the latest and greatest is, you can just switch it out when you need to.

THE HUMAN PROTOCOL (42:23.1)
But is there a potential downtime while these LLMs are training in a new data and new processes? If for whatever reason I decided exiting what I have and getting another LLM, is there a learning period, something, a training period that they can just pick up on and it's kind of plug and play in a way?

Issac Hicks (42:49.613)
Yeah, so I'd say most use cases are going to be plug and play. In use cases that are a little bit more nuanced, if you run into that, of course, you want to do a lot of testing on what you're going to move to, and you don't want to do a cut over until you are sure that that's going to work. But.

If you're installing an open model and you're transitioning over to that, there might be more of a lift there just because you have to deal with self-hosting and provisioning of your stack and everything. But in general, you prevent downtime by just not doing the cutover until you're sure that everything's going to work properly. And then if you design your backend where the cutover is changing a variable from one API key to another API key, then I mean, the cutover itself is a matter of a couple seconds.

THE HUMAN PROTOCOL (43:39.196)
Yeah, that makes sense. Absolutely, especially if you need to test it. That's always what the dev environment should be there for. Cloud versus on-prem infrastructure. How the different flavors work lately.

Issac Hicks (43:57.835)
Yeah, so this is something that you run into a lot is a lot of people think that they need an on-prem solution. Unless you're a government contractor or working for the government or you're

just absurdly massive. Like if you're in the tens or hundreds of billions of dollars in annual recurring revenue, maybe you need an on-prem solution. But typically the drive towards an on-prem solution is about security risk and the desire to make sure that the security protocols are in place properly. And you can cover all your bases with like the traditional systems that are out there. know, AWS is FedRAMP approved. They run

top secret clearance government systems in AWS. They run top secret government clearance systems in Google Cloud, right? So there's no real security advantages that you can get on-prem that you can't get on cloud unless it's explicitly required by your client or by your use case. We have to be on-prem. The more common use case, which is still kind

overblown is self-hosted LLM versus commercial LLM. Now back to my chat GPT hate, would not do anything with sensitive data related to chat GPT because especially since they're now doing advertising and stuff, I mean...

I don't know, there's only so much that I'm willing to say on a podcast that could be publicized, but I wouldn't trust putting sensitive data there. That's all I'm going to say. Yeah, I've... Yeah.

THE HUMAN PROTOCOL (45:37.0)
I wouldn't, I wouldn't in any of them. mean, it's a sensitive is sensitive. It should be as private as possible or highly secure. Whatever that is. Yes, I agree.

Issac Hicks (45:48.015)
Yeah. Now, these organizations are willing to do things such as sign business associate agreements with you if you need HIPAA compliance. They are willing to do NDAs and they're willing to do enterprise level agreements, especially if you are, you know, like a power user. I know that on all of the platforms where, you know, like tier five users, we spend tens of thousands of dollars a month with all of them, right? And so,

We have like an enterprise agreement with Anthropic where they guarantee us that we have our own isolated environment if we need to use it and stuff like that, right? But.

You can do those kinds of things if the main issue is legal exposure and you want to make sure that you are covered legally in case something happens and you need that for your insurances. But if it's something where you would be held liable directly regardless of what happens and there is no way for you to get out of that liability, then you would want an on-prem solution.

THE HUMAN PROTOCOL (46:52.508)
Got it. And when it gets to these negotiations, are the contracts between the end customer and directly these AI companies, or sometimes you are in the middle, and how is this negotiation takes place?

Issac Hicks (47:08.983)
Now the negotiation is between the organization that is building the tool and the LLM provider. So as an example, none of my clients are on the enterprise agreement that I have with Anthropic if I'm using the Anthropic API for them. That's between me and Anthropic, right? But, you know, where the devil's in the details is in the MSAs that I sign with my clients, it's like if there is some kind of breach of

THE HUMAN PROTOCOL (47:16.167)
the cross.

Issac Hicks (47:38.879)
data while the data is in our systems, then we are liable. But if there's some kind of, you know, breach of data that would be related to the LLM provider,

THE HUMAN PROTOCOL (47:51.24)
their environment, right?

Issac Hicks (47:53.099)
then the issue is like, okay, well, you can't use our data to train your model. It's like, okay, we can't use our data to train our model. So then I have an agreement with Anthropic, you can't use anything I send you to train your model, right? And then let's say that they do, well, I get sued and then the lawsuit becomes passed through, I get sued and then I sue Anthropic because they, they screwed me and that's why I screwed them, right? And so you absolve yourself from responsibility from a legal

standpoint so long as you have your contracts buttoned up across the board and anything you're offering your clients, you're also making sure you're being offered as the provider.

THE HUMAN PROTOCOL (48:35.016)
Yeah, because sometimes people say, yeah, OK, we have that in a contract, but how do we know that they are not using our data for testing or training their model? Well, we really don't know, right? At the end of the day, but it's a liability that they take on their organization if something goes wrong. Isn't how you feel about this, Isaac?

Issac Hicks (48:57.229)
Yeah, I mean, the way that I feel about it, especially, you know, the further I get in the business world and the more I learn about the way that things actually work as an entrepreneur, it's just, you know, there are the legal liabilities of whether or not something happens and then there's whether or not it actually happens. And the question of whether or not it actually happens, the answer is probably because I mean, you know, what

What kinds of things happen behind closed doors where nobody knows and then people are incentivized financially to do it all sorts of things, right all sorts of things, but What the question really is is is there going to be some kind of financial or legal? Exposure if xyz thing happens and am I protected from that financial or legal exposure? And if the answer is yes, then for all intents and purposes that is that's it right and to say

to say it's anything less than that is to either just be that's that's just kind of like some moral superiority public positioning thing you know it's just it's not real it's not real you know yeah

THE HUMAN PROTOCOL (50:07.634)
Yeah, yeah, especially in the world business, right? good stuff. Thanks for sharing all that, Isaac. And while we approach to the last section of our conversation, I have a few questions for you. It's kind of one question, but for different groups, just to get your thoughts on what you think, and we should go.

And the first one is, what would be your piece of advice for really individuals learning AI today?

Issac Hicks (50:43.471)
Okay, so my biggest piece of advice is the best way to really learn how an AI system works is to just build one because there's a lot of people who have all of these courses and you know there are some things to learn you need to learn about when to expect hallucinations and how the APIs work and how you can root up a system that is going to

you know, be sufficient in order to do things. There is some knowledge to be gained in learning, you know, just system architecture and solution architecture and stuff.

But the only way you're going to really learn how to make an AI system that provides real value is to try and to have it not work and to try again and to have that not work and to see where all of the little nitty gritty things are. that's why I mean, personally, whenever I go out to hire people, I don't even hire based on years of experience anymore. I just say, what projects have you done and what have you learned? And can I see your GitHub essentially? Right. Because that's the only way to really know.

what somebody knows in this space. And yeah, I mean, if you need a way to do that, that isn't going to be high risk, do it on your own business, right? If you're looking to learn AI and you're looking to become an AI provider, then do you need to automate your outreach? Try to automate your outreach, do it poorly and make a lot of people mad, then figure out how to do it right. Automate your finances.

Have that mess up, run into a big problem with that, learn your lesson and do it again, right? That's what it is. Yeah. Exactly. Exactly.

THE HUMAN PROTOCOL (52:23.708)
Don't be afraid on failing, right? Because that's at the end of the day how we learn. Great. Your piece of advice, Isaac, for companies thinking on implementing AI in their organizations.

Issac Hicks (52:37.739)
Okay, so my piece of advice for companies looking to implement AI in their organizations. when, okay, so at the top down from the least technical to the most technical. From the least technical, make sure you really understand what it is you're trying to do and look at the problem and derive the solution from the problem. Do not try to create a solution and fit it to the problem because of the marketing in the AI space is all about

all of these solutions and all of these things that you can do and my God, look at the unlock potential, right? But that unlock potential might not actually matter for you, right? Do you actually need that? So that's something is to like really be problem first focused. The second thing is to be extra, extra careful on anything where...

the outputs of a generative AI system in LLM is going to be directly interfacing with your customer base. So anywhere where the LLM is going to be handling communication channels directly, you have to pay very close attention to that. You have to really test that to death because that is the most likely failure mode.

And then the third is when you're designing the AI system, try to make as few nodes in the tree of the way that your system works actually dependent on the artificial intelligence. So what I mean by this is let's say,

Let's say that you're creating a system that is going to search through all your company data and tell you things based on your company data, right? The wrong way to do that is to just pipe all of your company data into a vector database and then put an LLM on top of it and then just ask the LLM questions. The right way to do this is to ask a question and then have all of the different places that your data is stored

THE HUMAN PROTOCOL (54:28.487)
Yeah.

Issac Hicks (54:38.929)
and what you're looking for and have the model determine, okay, based on this query, what data do I need to get and where do I need to get it from?

and then hit the API calls that bring that data into one place and then actually answer the question based on the data that's been consolidated into the one place. The former version that I spoke of, very easy to set up. It's gonna be right about 50 % of the time. The latter version I set up quite a bit more complex to set up, but it's gonna be right like 95 % of the time. So that's the option that you want.

THE HUMAN PROTOCOL (55:09.744)
Maybe more time upfront, but more reliable and accurate. Yeah. Awesome. And what would be your piece of advice to policymakers and governments navigating through this time? What do you want to see on that area?

Issac Hicks (55:13.919)
Exactly. Exactly.

Issac Hicks (55:28.909)
Hmm.

Issac Hicks (55:36.271)
I would say that it is naive to believe that we are going to be able to reduce the amount of intelligence that an artificial intelligence system has over time because it is incentivized to make a more intelligent system. And so the best way to combat the...

well, what if AI gets super duper powerful and no longer needs us? The right way to handle that doomsday situation is to narrow the scope in which the systems operate instead of attempting to minimize the intelligence or put guardrails around this intelligence. you put, if you minimize the environment in which the operation occurs, that is how you actually contain it. But if you're trying to put guardrails

on something that's not contained that's just not going to work.

THE HUMAN PROTOCOL (56:35.104)
And, you know, the last question that I had is when is efficiency enough? But now that you're mentioning environment and containing there, is that ever going to be possible based on where we are and where we're heading towards?

Issac Hicks (56:53.807)
I think that it is possible.

but I do not think that it is incentivized. And I think that one of the biggest issues that we see related to how to ensure AI alignment moving forward and what's gonna happen with all of this is pretty reminiscent of what we just see in society at large, which is people don't do what is right, so to speak. They do what has the most incentive to be done. And so the question,

to really consider and what policymakers and lawmakers and everybody really needs to consider is, does it make more sense to just...

like essentially break monopolized AI services where there's AI that can do pretty much everything and create portfolios of specialized AIs that operate in a very specific domain. Because if we can't achieve this kind of bifurcation of responsibilities, then it's possible to prevent the worst outcomes or it's more likely to prevent the worst projected outcomes. But the

all-powerful, all-knowing system that can communicate with everybody you know like that. That holy grail of the doom sphere version of this. It's like, well, I am an AI bull rather than an AI bear when it comes to the future, but

Issac Hicks (58:30.763)
as far as the bearish perspective is concerned, the all-powerful, all-knowing system is like that is the system that becomes the bear version, right? And so if we can split that up into subsystems, then we have a better chance of avoiding misalignment.

THE HUMAN PROTOCOL (58:51.28)
Are you seeing anyone going in that direction? I haven't seen it myself. I think everybody's trying to rush so they are not left behind.

Issac Hicks (59:02.125)
Yeah, everyone's trying to rush so they're not left behind. And there's been the emergence of all of these applications of artificial intelligence that are not super duper ethical, but are very profitable. And I think that the leaders of the space have a lot of power on their hands because it is up to them to determine.

what use cases can be used or not. you know, if you look at

what OpenAI has decided to do versus what Anthropic has decided to do versus what Gemini has decided to do versus what X has decided to do. It becomes pretty clear pretty fast where the heads of the leaders of these organizations are at and you know I hope the good guys win.

THE HUMAN PROTOCOL (01:00:04.06)
That's all we can do sometimes. It's hope for the best in a way. But thank you. I know we're just at the end of this, Isaac, and probably we will have another one or two hours to continue talking about all these. So really appreciate all the insights and this conversation that you have brought to us.

Issac Hicks (01:00:09.827)
Yeah.

Issac Hicks (01:00:26.988)
Absolutely.

THE HUMAN PROTOCOL (01:00:28.53)
Perfect. And remember, this is the human protocol. And the code may change, but our humanity will continue to be the constant. Thank you all for tuning in. And talk soon. Take care, everyone.