Brian Bell (00:00.684) Hey everyone, welcome back to the Ignite podcast. Today we are delighted to have Eric Siegel back on the mic. He's the co-founder of Gooder AI and the author of a couple books you may have heard of, the AI Playbook and Predictive Analytics. It's his second time on the show. thanks for coming back, Eric.
Eric Siegel (00:14.541) My pleasure, Brian. Thanks for having me.
Brian Bell (00:16.428) Yeah, so great to see you again. So for anyone who missed your first time on the show, love to start with your origin story, like how'd you get into machine learning? Why are you doing this?
Eric Siegel (00:26.957) been in well I'm a nerd. I I've been in machine learning more than 35 years. and you know the thing that in I when I was a kid growing up I was very interested in AI and all concepts of AI. And then sort of the summer between college and grad school and I went to Columbia for grad school got my PhD I was on the faculty there teaching AI courses machine learning and and this kind of thing. but during that summer, I kind of came to the recognition that machine learning was where it's at, that you know, replicating the human mind is kind of a fool's errand. There's only so much we can tell by introspection. And it's it's there's a it's sort of a level of geeky narcissism of like, I just want to make it do what I do or capable of everything I do. And instead of that, and those sort of philosophical problems with that, wait, the best way to ramp up capability is is machine learning, is learning from example. many, many examples, right? And and certainly we've seen that come to light, deep learning revolution and more recently generative AI and large language models, but still good old fashioned machine learning, which is older but not old school and you know otherwise known as enterprise machine learning, predictive AI, predictive analytics, right? This is the technology you turn to to improve your existing large scale operations, learning from data to predict who's gonna click by, lie, die, commit an act of fraud, any outcome or behavior. So I'm spending a lot of my time these days trying to remind the world that generative AI is not in large language model models are not the only game in town. that most organizations, and this is very subversive of me to say, should should be investing at least as much in in predictive AI types of projects as they are in generative AI, even though it's potentially the fruits much lower hanging and the footprint is much smaller, the investment's much smaller. and it's been around a long time, but Seriously, most of its values still largely untapped. Not that there's not a an extreme amount of of value to be getting gotten from large language models. Both of them are s even though large language models is much newer, they're sort of at a very similar stage of arrested development as far as there being a lot of excitement, enthusiasm, and fetishizing of the core technology. The first of two wows. And the second wow is, wait, this isn't just a great idea on paper. (02:48.995) But it also pays off. We can actually operationalize the deployment and it's deployed. It's practical. It actually delivers returns and improves operations. that's the second while. Hey, this isn't just cool. It's also valuable. And that second wow for most projects, both predictive and generative to this date, usually is not achieved.
Brian Bell (03:09.271) So you must be really excited about, or you you saw the news on JEV last week, kind of the new decision-making LLM. What do you think of that? It's kind of like combining J J E V. it's so it's from the co-creator of Chat GPT. and basically what it does is it it's hundreds of times faster than a large language model, and it only does prediction. So it doesn't write text, it only outputs decisions.
Eric Siegel (03:18.689) No, I'm not f I'm not familiar. Which call? Yeah. (03:26.639) Mm-hmm.
Brian Bell (03:39.331) so you can the I think the context window is up to sixty four thousand tokens and basically it's really, really good at just making like classifications or yes or no, and it it will not output, it will not write for you, but it it's really fast at prediction.
Eric Siegel (03:55.034) Well, I'm ill prepared for that question, but I I can speak to it in general. I mean, you can take a large language model, you said tokens, so I assume that some of the base technology is is similar and the magnitude of of the core model is is on a similar scale. and you can have it predict. You can use that technology to predict. So the difference the difference between predictive AI and generative AI is not so much a difference in technology or technical approach or analytical method or algorithm. The difference is more
Brian Bell (03:57.006) Yeah.
Eric Siegel (04:24.549) in the way you're using it. it's these are these are the two main classifications or types of AI projects. So in predictive, you're managing uncertainty. And that's why these are these two areas are destined to remain distinct. Gender AI is not going to subsume predictive AI, no matter how sophisticated a model you use, you're still going to have You're gonna have very diminishing returns on how well you can predict. We're never gonna have a magic crystal ball. So the the what we do have the next best thing, which is probabilities, right? Which is a number between zero and a hundred. How likely for this particular case is this individual going to click by lie or die or commit an act of fraud, turn out to be a bad debtor or whatever, whatever is valuable for the organizational process. So uncertainty is not going anywhere. We can play the odds with those probabilities. So for improving existing large-scale operations, what's usually called for is predictive AI. that, you know, prediction on that level, the odds for each individual case is is what's called for is the holy grail for improving each per case decision directly, whether to contact, investigate, incarcerate, set up on a date or medicate. So whatever decisions being being run. And although we don't have a magic crystal ball, predicting better than guessing and playing the odds in a more sort of scientific data-driven way turns out to be a windfall for improving those bottom line KPIs and business metrics. so that's all predictive, right? Whereas, I mean, I don't need to lecture you on all the value of generative, that's that's what all of the venture investment well, not all, but mo the vast majority of venture investment and media attention. White hot spotlight is on that. I say very much disproportionately so. The two ideally they d very much compete for attention and resources, but ideally they shouldn't. They solve different problems, they're different value propositions, they serve different roles. They should compete no more than a camera and a telephone. and in fact, just like those two things, they work best when combined together.
Brian Bell (06:41.953) Yeah. Have you are you aware of any studies that compare predictive models in the classical sense that you're describing versus LLMs and their ability to, you know, classify or predict things?
Eric Siegel (06:56.291) Yeah, I mean, I see that I I pay a lot of attention to that because I'm very interested in hey, to what degree do large language models improve the prediction process? But the story overall is has always been the same, even well before large language models came on the scene, which is that there's diminishing returns. So you increase the complexity of the model. You can the simplest model is a decision tree, which is just a bunch of if-then business rules concatenated together, essentially. log linear regression, which is just a weighted sum of all the different factors. with a lot with a certain nonlinear twist to it. But you know, those are very simple. They're very transparent. You can see exactly it's easier to under the lear. Yeah. Right. So you get transparency in the way it operates. And you also have more transparency in how the learning process goes down, right? It's a it's a little it's definitely easier to understand. it's not as rocket science y.
Brian Bell (07:33.537) Weights and biases are, like why is it making that decision? Yeah.
Eric Siegel (07:49.634) And then you go to something complex like an ensemble model. An ensemble is basically just a bunch of what could be very simple models like decision trees, basically voting or having some kind of weighted sum of them. And it turns out that when you make that step, all of a sudden you increase complexity a lot. You you you lose that transparency, it becomes very opaque very quickly. And you can and it definitely tends to win, right? So when you take that type of approach, that level of complexity, it tends to win. you know, the public predictive modeling competitions. but at what cost and is it worth it? So it wins because there's these leaderboards and you're trying to inch up the leaderboard. But in reality, in real life, the thing that tends to pay off more isn't com in increasing the complexity of the analytical method, but rather improving the data. And the world changes so quickly, the business context for any given project changes so quickly that it's relatively rare that it's really that sort of things are stable enough and the project is ongoing, it's been deployed, everything everything it's been adopted, and it's worthwhile investing in that additional complexity because things change so much, and this project kind of has to start over from scratch anyway. So, in reality, it's pretty rare that you need that level of complexity, and that story ends up being just as true. When you go to you know, a model that's dramatically even more complex than than the most c complex typical ensemble model, namely deep neural networks and and large language models.
Brian Bell (09:24.077) So tell me about the new version of the book. So you had a hardcover and now you re released a paperback. Is it kind of a like a semi-new edition? Tell us about it.
Eric Siegel (09:33.166) Yeah, this is the AI playbook, second my second of two books. It's about to come out in paper book. And you excuse me, paperback, you can get the audio book for free. Go to the website bizml.com and pre order the paperback. So you'll there's instructions there. You can pre order the paperback and get the audiobook and also get the audio book at the same time for predictive analytics, my prior book. We're just trying we're just playing the numbers and trying to get more pre orders on Amazon. That's just how that's just what you're supposed to do.
Brian Bell (09:53.336) Why'd you do the audiobook for free? What's the strategy behind that? Yeah. Yeah. if you pre order it you get the free audiobook basically. I see. Okay, gotcha.
Eric Siegel (10:03.703) Yeah, that's the promo we put out there. So biz the reason the the website is BizML is that's the titular playbook, the the paradigm, the framework for running machine learning projects. It's both of my books are about predictive AI specifically. so to run these projects success successfully through to deployment, most of them fail to deploy, but that's an organizational dilemma and sort of a lack of professionalization and maturity of the industry very much stands to be rectified and can be. so it's a six-step practice that I call BizML in in the AI playbook. the paperback edition coming out, and the audiobook already already has this, has this new preface where I'm comparing predictive AI and generative AI and go through the ways sort of that sort of intrinsic industry competition, how it's playing out. you know, the bottom line is that even though generative looks like it's killing predictive AI, in reality. Outside of venture investments and the white hot public spotlight and media attention, what we're seeing inside enterprises that most of the returns we just this is anecdotally it's not real data, but you just constantly hear this. Yeah, most of the returns we're still getting, the things that actually get deployed are predictive. That's been around longer. The there's a lot of pressure to create agents or whatever you want to call it, large language model based projects. And the the demo effect is incredible. The technology is clearly much more sexy and much more AI-ish in the sci-fi concept, much more seemingly human-like. But in terms of achieving returns, it depends on the organization. And there's there's just as many initiatives. So one one piece of of of of data is the conference series I've been running since 2009 called Machine Learning Week. We did an extensive call for speakers. For the last US edition of the conference, which was last spring, 2026. And we didn't say, hey, this is an event about predictive versus generative. We just sort of put it out there. And it was about a 50-50 mashup between presentations that focused mostly on generative AI systems, those that focus more on predictive, and then some other portion that covered concepts that apply to both. (12:17.657) So that's a that's another kind of somewhat informal, but a little more data driven read. So, you know, my my pos my impression is that the investments are disproportional. And the the attention is there's a lot of hype around Gender AI, you know, false promises, bad expectation management, there's gonna be disillusionment. That is me criticizing the hype, not the technology. I think that technology is incredible. I was in natural language processing research for six years. I never thought I'd see what we see today. and it has a lot of value, and there's a lot of different ways to use it. it's funny, it's this very, very general purpose idea. So trad in both cases, we're using machine learning, right? And what machine learning does is it creates models, and what models do is they predict. Right now, the word prediction is a little bit A little bit more general than just predicting the future, because you might be predicting is this transaction fraudulent? Does this healthcare patient have this diagnosis? But in it anyway, you're trying to ascertain or put odds on an unknown, right? So the con and the weird word we generally use is predict. In the case of large language model, what should the next token be? So like, you know, well, we've written this much. What should the next token be? Or what was it in some, you know, prior existing text that was generated by a human, hopefully. Either you're just using it in different ways. But the difference is with a large language model, you're then constructing this thing that's supposed to solve lots of different tasks. Whereas with enterprise machine learning and predictive AI projects, you're constructing a model that's very much specialized for your organization on this one particular project, on the way that it's going to target retention offers to customers who are at risk of defecting or whatever it is. So there's a very big difference in the way it's meant to be used and the range of problems it's supposed to solve. And then that kind of goes hand in hand with the way in which it's probably hype.
Brian Bell (14:15.361) Did you just out of curiosity 'cause I'm about to pub you know, publish my own book, did you read the audio book yourself? Or did you hire a narrator? Yeah, okay.
Eric Siegel (14:21.687) Mm-hmm. I narrated it. Yep. in the first book I only narrated the preface or something.
Brian Bell (14:29.057) Yeah. And you decided on this one I'm gonna do the whole thing.
Eric Siegel (14:31.521) I did the whole thing, yeah, I did a I I I made a little sound booth. Yeah.
Brian Bell (14:35.713) Yeah. Yeah. I think I would just do it sitting here in my office and like hope for the best. You know. Yeah, it'd be better to have a sound booth, I guess. But that's yeah, that's really interesting. So would tell us about the the new edition itself.
Eric Siegel (14:39.343) Yeah, yeah. (14:50.989) Well, the the it's not. It's just a paperback with a new preface. But but but more to the so the the what one of the things that the new preface accomplishes is is the jargon thing. So the book's called the AI Playbook. And it it in within the book it mostly refers to it as machine learning, enterprise machine learning, or just ML, right? But nowadays with the advent of generative AI, you know, we probably call it predictive AI more. It used to be called predictive analytics. So whatever you call it, the message of the book,
Brian Bell (14:54.303) Okay, we preface.
Eric Siegel (15:21.259) pertains as much as ever, right? We have all these potential ways in which we can target and more make more effective and efficient large scale operations. and prediction's the holy grail to do that. That's the definition of a predictive AI project. The value proposition, the the use case is always defined by two things, what's predicted and what's done about it. And we're and it's used to target, you know, marketing, fraud detection, credit scoring, right? And then there's this extremely long tail of application areas. But this is the technology you turn to improve your existing large scale operations. That's not changing. It's still true. And it's unfortunately still true that most of the projects fail to deploy. So my first book, predictive analytics, is about how it works in the sense of yes, the core technology, but also how does it s work to serve value in terms of the nitty-gritty of taking a probability, which again is the best thing we have other than an absolute prediction, which we generally don't have. Right, because we don't have a magic crystal ball. How do you take a probability and actually have that sort of translated into an action? that's the deployment piece. That's the part that usually fail you fail to get to. That's sort of the culmination of of of the project when you're trying to get this system, a new system launched. so my first book, predictive analytics, is how that works, how that serves value. The second is how to capitalize on it. so to the degree that I have faith that value matters. Right, that the world has some rationality to how it turns. I I'm very, very firm footed in believing that gender of AI is can't by definition subsume predictive AI the two are gonna live happily ever after together.
Brian Bell (17:03.501) Yeah, and I I think like I I think back to reading Kevin Kelly's book, What Technology Wants, or How Technology Evolves, I forget. I that was his thesis, is that new technologies come along and they don't replace existing technologies. In fact, you can kind of look back and find all the old technologies still kind of being used used somewhere. And in some cases, it could actually shine a light and and make the technology used even more.
Eric Siegel (17:21.453) Mm-hmm. Yeah, yeah.
Brian Bell (17:30.387) one of one of your core statistics from the book is most ML models never deploy. this is back in 2023 when you published the the the hard hard book, the hardcover edition. Since publication, you know, generative AI has all also built out its own failure record, as you've pointed out. the MIT study fo report found roughly 95% of enterprise gen AI pilots pr produce no measurable PL return. Is is Gen I gen AI failing for the same reason as predictive was failing last.
Eric Siegel (17:41.476) Yeah.
Brian Bell (17:59.704) generation or is are they new reasons?
Eric Siegel (18:03.351) Yeah, it's new reasons. So there's the the story is broadly similar. Predictive AI fails because of a complexity issue. It's not that complex. You don't have to know the real rocket science to understand it, but you need some understanding, a very accessible, semi-technical understanding to get it deployed and participate as a business stakeholder person, the manager or business professional in charge of operations meant to be improved with the all these predictions. You need to have some sense of it. It's very accessible, it's very interesting. And That's the other thing that my books serve is to like serve that up on a platter so you know enough so you can participate in those projects as is needed. If you don't participate from the business side, it won't get deployed. In generative AI, it's it's a reliability problem. So with predictive AI, anyone sort of somewhat familiar to you have other than just a superficial level with predictive AI, everyone knows it's not a magic crystal ball. This idea of c high confidence that turn and phrase. often is used to overpromise exactly how well it predicts. it as far as implying that it has high confidence about both positive and negative predictions of who's pregnant for the target story, right? Who's gonna click by lie or die. and that's that's overpromising. But in general, once you kind of le get a certain level of understanding and expertise in the area, everyone knows, right? It's not a magic crystal ball. So the issue isn't believing it it it it can it can do magical predictions, but rather understanding, hey, if predicting better than guessing is potentially valuable, exactly how valuable is it? And how do we understand the very particular arithmetic? And that's that's the focus of my startup, gooder AI. Whereas with generative AI, people are s generally still expecting it to be right a lot more often than it is. And you you run up against these. So you can get this incredible, exciting demo. Look how well it does on these several cases. and then you even if you do get to deployment, you very quickly backpedal and push pause on deployment because more often than not, it turns out, hey, it makes errors that are grave, right? Hallucination where it gets something wrong, problematic content, something that's unethical or offensive, conducting transactions that aren't proper or authorized, changing an airline (20:25.507) reservation that you know it shouldn't do, whatever it is, something that's that's that's consequential, if it makes mistakes, let's say five percent of the time, the 95% correct rate is incredible. It's dumbfounding, it's unprecedented. We couldn't even have imagined it several years ago. And yet the whole system's not viable for deployment because it's not reliable enough. So the antidote to that, in my opinion, and I think that And I see that the industry's gone this going this way, and it was the focus of our conference, but it's still pretty primordial and emergent, is to have a predictive reliability layer and funnel off the higher risk cases and interactions to a human. So we have to compromise on our overzealous notion of these relatively sophisticated systems being 100% autonomous, and maybe it's 80% autonomous. Some amount of it still goes to that more expensive human in the loop. And then we get a system potentially that is viable. We can deploy it. it, you know, it it realizes 80% of that audacious promise of autonomy, which is a lot better than the zero percent that's typically happens because you can't deploy it. and I'll I I'll just I'll just qualify all that with one thing. There's lots of projects where an LLM is reliable enough. Right. So if you have it do customer support and you're calling in about your healthcare insurance out questions and it's supposed to be able to answer and do no more than answer questions pertaining to let's say 10 or 20 pages worth of FAQ type of information. And then if you want to conduct a transaction or you want to escalate to a human or you want to g work with sensitive data or information. It's not going to handle that. It's going to funnel you off. But for this one sort of well-defined, reasonable scope, right? It it is viable. It does get deployed. maybe not by a large health insurance company. We're not quite seeing that yet, at least in my experience as a consumer. but the problem is that the story, the hype, the ambition, all the excitement comes out of this notion of autonomy, of supreme autonomy, in the most exaggerated case of that is. (22:44.717) Artificial general intelligence, of course, if it's equivalent to a human, that is ultimate autonomy, because you don't need a human if it's equivalent. but even without going that far, we have an intrinsic susceptibility, almost all of us, to mental scopes creep where we're gonna very easily and very quickly enlarge in our scope for what this project's meant to do. Our ambition and hopes for what this autonomous, let's call it agent. I think there's a problem with the term agent, tends to overpromise to non-technical people because we're trying to ascribe agency to a machine rather than considering it a tool. But in any case, you enlarge in what this thing is supposed to be capable of, you let it do some transactions or what have you, and very quickly, and and and again, the whole the hype, right, the excitement, the promise for enterprise value is autonomy. So, to the degree a system like that is supposed to operate without the human in the loop, that very quickly becomes infeasible and unrealistic. So I think we just backpedal a bit as an industry, we say, hey, look, it's not 100% autonomous and we can judiciously play the numbers with predictive AI. It's not really that we do it with predictive AI, it's just that by virtue of doing this, it becomes also a predictive AI project. And we include that human in the loop for those high risk cases, and then all of a sudden we have systems that can get deployed.
Brian Bell (24:14.317) So in the book you talk about six biz ML steps. maybe you could walk us through what those six steps are for implementing a project like this. Which steps do companies skip the most that that you see and which one actually kills the most projects.
Eric Siegel (24:25.411) Yeah, sure. (24:32.653) Yeah, yeah. Let me describe the six-step practice I know that I call BizML, the playbook for running these projects. And again, this is to run a predictive AI project, including if you're using predictive AI as a reliability for a Gen AI system. This the same concept applies and the same deployment challenges exist for any of these millions of different, or that's that's a hyperbole, but these tens of thousands of different kinds of predictive AI projects. So this sip six-step practice is basically three pre-production and three production steps. The first three concern yourself and engenders deep collaboration between biz and tech. And that's absolutely necessary because it's a business endeavor, right? We're changing the way the main operations, such as for targeting, marketing, or credit scoring, or risk management, whatever it is, fraud detection, we're changing the way these operations. act based on a probability and a probability of what exactly? So what you have to concern yourself with is what's predicted, as I mentioned a few minutes ago, what's predicted and what's done about it. That's what defines the project, the purpose, what the technical specification of what you're trying to model, what you're using machine learning to predict. So that sort of defines the project. And then there's a third fundamental which is how well does it predict? So what's predicted How well and what's done about it. the first three steps, not quite in that order. get everybody on the same page to collaborate deeply and get into a very clear, specific, not just hey, I'm gonna predict who would buy if I contact them or who's gonna cancel if I don't, if I don't offer them a retention offer, but it has to be much more specific. Like which customers who've been around for at least six months. Who've spent at least this much and live in this area and are qualified for this and da-da-da-da-da-da-da, and tend to buy low margin, or whatever it is, there's a like there's always a whole bunch of caveats and qualifiers, will not necessarily cancel entirely, but decrease their spend by 80% over the next four months, whatever, like all those parameters that define a yes-no predict, which what often starts out as simply a yes-no prediction goal. And that that's a good way to start these projects. (26:54.311) binary classification that is. So how do you define the thing that's being predicted? So that ends up being step two. Get real specific about that. It's not rocket science, but it is kind of semi-technical. And it every aspect of that needs business stakeholders because it speaks to the it speaks to all the pragmatic and practic practicalities of the operation that's going to be changed by the probabilities or odds. of that exact thing. So that defines the technical specification of what it is you're trying to predict. And then lastly, as I mentioned, how well does it predict how good is it? And this is the question that's generally not answered properly for both predictive AI and generative AI is how good is it? Because what happen so what happens, I'll tell you both stories separately. When predictive AI, and this has been going on this problem has been going on for decades, and this is a main focus of my startup, Gooder AI. Is that almost always the data scientist generates the model and evaluates how well it predicts its predictive performance, how good it is, only in terms of standard technical metrics like precision recall accuracy, area under the curve, F score, the confusion matrix of false positive and false negative rates and all that kind of stuff, right? Which tell you all those things have something in common. They're they're helpful, they give you a sense that the thing is is is kind of good. But only that it's relatively good, that it's improved over some simple baseline like random guessing, which is again usually the most we can hope for because we're not going for a magic crystal ball. and that therefore it's potentially valuable, but they tell you almost nothing about the actual value. The value is just super straightforward, right? It's business metrics, profit, savings, right? Like how much money would it make me or save me? To actually deploy this or whatever KPI or business metric pertains, if it's a healthcare organization, it may not be money for some projects, but whatever it is, it's a business metric. It's it's in the language that's that's intuitive, that matters to the organization, to the business staff at the organization, to the executives, right? So also estimating the potential value of the deployment if I use this to to score transactions and then the ones above this threshold get passed off to, you know, get. (29:17.123) Temporarily held and passed off to a human auditor, whatever the operationalization, the deployment plan is, we must estimate the business value thereof in order to plan for that business value, to maximize the potential business value, and perhaps most importantly, to sell the deployment in business terms, because this is exactly where things usually fail. The project stalls. People this kind of becomes convoluted. People are busy. The failure is the industry is extremely good now at sweeping the failure under the rug. And again, the vast majority of these projects, of these machine learning models that are trained over historical data in order to deploy, fail to deploy. So those are the first three steps pertaining to what's predicted how well and what's done about it. The other three steps. are just the same standard technical steps that have always existed since we started using machine learning models to target credit scoring or targeting marketing back in the 60s, right? So it's just prep the data, train the model over the data, and then deploy it. But all three of those, again, are not there's a rocket science in the training part, but all three of those, everyone's fully capable of getting a very interesting, pertinent, semi-technical understanding. And the business stakeholders, again, really must in order to participate step by step across the project. With all of that in place, you stand a good chance of beating what are generally the failing odds of predictive AI projects and get the darn thing deployed.
Brian Bell (30:56.557) That's awesome. Great, great summary for anybody deploying these these sorts of projects. so tabular foundation models like Tab PF PFN published in Nature last year and and Jev, which just came out like this week, can do pla a classic predictive tasks with little or no training. if foundation models absorb predictive AI, doesn't your predictive versus generative generative distinction collapse into one stack within a few years?
Eric Siegel (31:24.163) That's that's a great question. Definitely not. So again, the difference between these projects isn't the kind of technology, it's just the nature of the project. If you're dealing with uncertainties, or that is to say, degree of certainty, if you're dealing with the odds, if you're dealing with probabilities, if you're dealing with uncertainty, you have to be doing it in terms of probabilities. It's just simply a number. Probabilities sound arcane and it can sound really boring, but look, they're the only game in town. We have uncertainty it's not going anywhere. We don't have a magic crystal ball, but we do have the next best thing, which is a number between zero and one or a zero and hundred and a hundred, right? They can be misunderstood, right? Everybody blame Nate Silver for getting it wrong after Hillary Clinton lost the the election. But come on, she had she had an expected chance according to Nate Silver's model of 70% win. That's closer to a 50-50 toss-up. The main thing to learn from the odds he was putting on at the time was shrug your shoulders, right? 30% is far from a long shot. There were other models that gave Trump only 1% or half a percent or whatever, and and they were probably off, right? But so yes, there's things like that where people need to be sat down and be like, hey, let's spend five minutes dealing with the fact that uncertainty exists in the world and some numbers just say shrug your shoulders because you can't make a really confident bet. But once we get past that, the notion of a probability and how to act on it is not rocket science. The rocket science part is the machine learning part, is learning from data in order to calculate those predictions and how to use that model, how it functions, which in some cases can be pretty opaque to calculate those. But once it calculates them and we've validated that, you know, that that that it calculates them well, Then we get to use them and they are not rocket science. So the act of act the types of projects that integrate probabilities or odds are otherwise known as predictive AI, enterprise machine learning, or predictive analytics projects, whatever you want to call them. That is takes a very particular discipline along the lines of what I just outlined in those six steps, right? It's a very particular type of predictive AI project that requires a very particular organizational practice to actually get. (33:45.988) Successfully deployed. No matter how you calculate the odds. So even if you're using a more sophisticated or new technology and that turns out to calculate odds a bit more effectively, awesome. In fact, in the hybrid approach of using real predictive as the reliability layer, you might not even have to create a separate predictive model. You might just be able to use the large language model itself that's already doing the task. To also estimate its own confidence in whether it's going to get correct or its own sort of threat level of whether it's going to make a misstep or a hallucination. So for example, if it's output putting yes or no, you can look under the hood and be like, well, what's the probability of those two tokens? Right. That would be one way. Or you can ask it, right? And there's a whole bunch of ways you or sort of rejigger. That might be what the JEV system I haven't read about yet does. I know there's other systems that do that. Rejigger the output part. Of a large language model to then actually be doing binary classification. Whatever you're doing, you don't necessarily have to recreate the wheel or create a model specialized. But what you are doing is admitting to yourself and to your colleagues that we have uncertainty, that the best thing we can do is generate probabilities that are as precise as as precise as possible, and then use them systematically to drive these decisions across all this long tail. amazing ways types of predictive AI projects. So when you do that, it's a predictive AI project. Again, it's not a di necessarily different kind of technology. It's just the type of discipline, the type of endeavor.
Brian Bell (35:27.191) So for the new book, who should not read the book?
Eric Siegel (35:32.911) Who should not read the book?
Brian Bell (35:34.859) Yeah. Who's it not for?
Eric Siegel (35:39.196) I think poets and like artists and like
Brian Bell (35:42.253) That's a good answer. Yeah, if you're a poet or art history major, probably you shouldn't read it. if
Eric Siegel (35:49.902) Well, I mean I've a like if you're even if you're even if you're far from tech and business, right? I I think there's a pop science appeal to this stuff. The idea of learning from data to predict, which fans out both as large language models and for any of these enterprise ML use cases, predictive use cases, either way, this idea of learning from data to predict, to to to capture patterns or formulas or somehow learn something. From a limited number of examples. It might be millions of examples, but it's extremely limited and finite number compared to the number of possible cases that could happen, right? So it's always some representative example sample that's relatively small on the scale of things. And learn from this finite number of cases to ascertain truths, right, that pan out, that work, that turn out to hold for unprecedented cases that have never before been encountered or observed, right? That's cool, right? That's in my I've always thought that's the best kind of STEM. you know, engineering, math, science, whatever. I I and turns out to be really valuable. So there's no reason not to learn about it. And and you know, in the books, because it needs to be accessible also to business stakeholders, right? I've bent over backwards to serve it up on a platter so you can understand the gist of what it means to learn from data and different ways of Of ascertaining these patterns and then validating that they do hold in general. I think I so I think that stuff is that stuff's really cool. well, so as far as who the book's not for, if you're a senior hands on data professional and you've, you know, a machine learning professional and and you're and you're already a quant in that respect, what I say in the FAQ at the beginning of the book is like, well. you're probably not gonna wanna really read these chapters, but you should skim th the book because what most technical quants don't know is the business side necessities to get these things deployed.
Brian Bell (37:58.936) Totally makes sense. So if you were to write a a true second edition of the book, what would be the first chapter you would write? The first new chapter? Or rewrite or both?
Eric Siegel (38:09.475) Well, that's a great question. I mean, I I think that if you're gonna write a book right now, you know, re writing a book is generally a silly thing to do, right? I mean it's it's an extraordinary amount of of time investment. and it's kind of that's a high risk. Right, right. I should have told should have I should have told you not to. Like everybody. I'm just I'm being kind of facetious, right? I mean I I've been very happy with the business outcome my books, but that
Brian Bell (38:24.397) I just did it. I'm j I'm publishing a book. It took me a year. I mean, it's a lot of work. It's a lot of work. Yeah.
Eric Siegel (38:38.729) you know I've I I've done 180 paid keynote speeches and and stuff like that and first books adopted at hundreds of universities. So I've been fortunate and that's been great, but that's not why I wrote the books. I wrote the book because I like to geek out in a way that's accessible to my uncle, right? To any to non-technical people. I think this stuff should be shared and should be widely understood in order for it to actually deliver and realize value rather than just being theatrical and really cool tech. along those lines, by the way, I promised something several minutes ago and I didn't deliver it. So let me deliver it now, which is why the bad metrics lead to lack of deployment. I covered that for predictive AI, but as I mentioned, there's a very similar parallel story for large language model-based systems, where also the lack of deployment is because of of False benchmarks, common bench standard benchmarks that are at best only a proxy for value. And that's that's the story that exists for both kinds of AI. the way it goes down in generative AI isn't that it's just focused on a on an arcane technical metric. That's the predictive AI story. With generative AI, what happens is you focus on generic benchmarks. So standardized tests, humanities last exam, LSATs, you know. Tests where you've you've got these leaderboards and all the different bench LLMs are constantly kind of leapfrogging each other and they keep getting a little bit better at these at these tests, or sometimes a lot better. But the tests themselves are generic. There's you know, which is definitely a good idea for humans, right? We have a general sense that the humans that are that are are good at a standardized test that's that somehow is a proxy for how good they be at related tasks in general. But for a enterprise system, for a technology. The only definite reading of how good it is for the task that you intend to use it in any particular project is is actually evaluating it on that task. So the actual metric, which is simply accuracy, how often is it correct? That's a perfectly fine metric in many cases for these LLM type projects. but we've got to move to also measuring how good it is (40:59.147) on a test on a large representative test bed of encounters or situations where you're gonna potentially deploy it. And then once you do that, you may very well get bad news. Hey, 7% of the time it's gonna fail and that's never going to fly with the executives. We can't even deploy the system. That's when you say, hey, wait, instead of scrapping this project, instead of lots of confusion and chaos and disappointment and and heads flying and all that kind of stuff. Let's put a predictive layer, let's put the odds on it, let's triage the risky cases, and maybe then we can get a net failure rate down to one percent or something that would be viable for deployment.
Brian Bell (41:40.334) Mm-hmm. Yeah, it makes sense. So as we record this, you know, a week or two ago, we, you know, we had all the AI doomers out there, you know, we had the guy quit anthropic and, you know, claim that, you know, AI is gonna kill us all. We need to do something. What's what is your personal pea doom versus pea boom as you look out five or ten years?
Eric Siegel (41:45.119) Yeah. (41:53.603) The guy who left. Yeah. (42:05.249) my P doom is zero.
Brian Bell (42:07.732) Wha yeah, why is that?
Eric Siegel (42:09.891) Because that's just another way to hype the technology. It's a ghost story. This is the this is the the the the horror story that Mary Shelley, who wrote Frankenstein, would have written if she knew about algorithms. this is a you know, it's it sounds ironic because it's it's it seems like the opposite narrative, but this is just another way to pump up, you know, if it can kill everybody, it's also probably pretty good at targeting your marketing, right? This is a the the notion that the thing is gonna kind of get its own volition. And ha and and be capable of what humans can do and therefore improve itself just the same as AI research humans, a human AI researchers. right, that that is the way now I'm not s I don't mean that there aren't a lot of people who don't genuine genuinely believe it, but I do question whether some of the more famous talking heads and heads of corporations genuinely believe it. Because it really is a way for the for the industry to prop itself up. And I I don't believe it. I think that it's a fairy tale.
Brian Bell (43:17.207) Yeah, it's it's great Doomer marketing, right? They're able to get a lot of attent free attention, you know, hundreds of millions of views on that tweet, from a from an account that had no tweets before that and no followers before that. I mean, that's pretty incredible.
Eric Siegel (43:31.213) Yeah, I mean he's a successful social media maven, right? I mean they say people who people who the people who are like me talk skeptically not about AI, but about the hype about AI, sound smart and the people who prop up the hype get rich.
Brian Bell (43:34.507) Yeah. (43:50.668) Yeah, that's really well said. So back to RSI really quick, recursive self improvement. I mean, Anthropic and OpenAI and others have said that they're using their models internally to develop their newest models, right? And you have the n recently the Navier Stokes thing that was solved, the millennial prize. You're it sounds like what you're saying is like we won't get to a hard takeoff, basically. there'll be this nice smooth takeoff where we're able to align the tool and it gets just
Eric Siegel (44:02.926) Mm-hmm. (44:16.516) Mm-hmm.
Brian Bell (44:19.841) J you know, gradually smarter and smarter and it won't take
Eric Siegel (44:22.359) Well, what's the wait, so what is the y coordinate in your graph here? That's the real problem, right? What what does intelligence mean? Intelligence is an anthropomorphic term, right? It's if it's nothing but it's it's nothing if not subjective. And it's a term that applies to pr to humans who have a very strange particular set of capabilities. So what are we we're is tech getting better? Yes, no question. But this idea that it's reaching AGI. goes hand in hand with the intelligence explosion narrative, recursive, which recursive self-improvement tends to in the way that it's publicized. We've always had meta-learning for decades. I mean, I remember getting into it in college, and I graduated college in 1991. The notion that you can kind of help improve the parameters of the learning process with meta learning, right? That does pan out, right? And you can use a large language model to help you with large language model improvement research and with a s in a semi-automatic way, which is very different from the notion that technology is gonna reach achieve some threshold, some critical tipping point of quote unquote intelligence, as if intelligence is a clear, specific, well-defined. platonic ideal that exists outside of human beings. And then once you once it kind of gets that, then it can do its own thing and improve indefinitely. But intelligence is not a singular dimension. And so, you know, the narratives are tapping into sci-fi tropes. And Terminator Two's really fun movie and always was. And it creates a very visceral effect and the AI leaders are exploiting that.
Brian Bell (46:19.853) Yeah, I love that. I have here on the notes that you rapped about machine learning some of your keynotes. Do you still remember any of those raps?
Eric Siegel (46:25.795) yeah. predictive analytics learns from the data you provide to predict whether you click bic click by lie or die. It ain't astro it ain't astrological, it's math, it's methodological, so better pay attention 'cause my rap is pedagogical. There. It's been ten it's been ten years. So go to predict this dot org. It's the best three and a half minutes of your life. It's a it's a legitimately
Brian Bell (46:44.353) That's awesome. (46:49.371) wow. wow.
Eric Siegel (46:53.463) educational rap about predictive analytics. Unfortunately, you might not learn anything because you'll be so distracted by the extremely campy video. But close your eyes and listen hard and you'll learn everything you ever wanted to know about predictive analytics.
Brian Bell (46:56.749) That's so cool. (47:10.165) like schoolhouse rock. so what's the one sentence the AI playbook from the AI playbook or about the AI playbook you put on a billboard in San Francisco?
Eric Siegel (47:11.791) Yeah. (47:21.839) well, I would just go to the subtitle. Let me make sure I 'cause that's the whole point. You know, you spent half the time writing a book and the other half writing the title. So it's it's the I playbook, Mastering the Rare Art of Machine Learning Deployment.
Brian Bell (47:29.783) Yeah, I yeah. (47:35.885) Yeah, I love that. and
Eric Siegel (47:36.813) No, I actually w I wanted to I wanted to call the book Unfuck Machine Learning. But MIT Press didn't exactly they didn't exactly agree with that.
Brian Bell (47:45.751) They weren't gonna go along with that. Yeah. Yeah. where's and where can people go find the book?
Eric Siegel (47:52.265) at bizml.com. So business practice for running machine learning projects. That's the titular a playbook of the AI playbook. So B I Z L dot com.
Brian Bell (48:01.933) Awesome. Well, so awesome to catch up, Eric. Congrats on on the new paperback and we'll look forward to having you back on again in two hundred episodes.
Eric Siegel (48:09.963) Awesome. Excellent, Brian. Thanks so much.
Brian Bell (48:13.774) Thanks, Eric.
