L.O.G. 204 Accelerated Learning with Dr Lia DiBello

Okay, well, welcome to the show, Dr. Lia DiBello. We really appreciate you coming on here and talking to us today.

Yeah, I'm sure it'll be a lot of fun.

I think so. And I do want to warn the listeners, and you, Greg, is a little under the weather today, so he's not his usual self. So, maybe he'll only be at like a normal person's game.

What is my usual self?

Yeah, I can tell you this, our viewers and our listeners really don't know the real Greg. So, maybe we'll unpack...

There's no hiding after this many episodes. I think they know exactly who you are, no matter how under the weather I am. I'm so excited about this episode because one of my favorite humans on the face of the planet is on, and we get to talk to her.

Yeah, well, so that is kind of a good place to start, Greg, because, Lia, we reached out to you after we were introduced by a mutual friend, Matt Finnell. We talked to him before on here (The Human Behavior Podcast), and talked about him on the show because of his work up at the Infantry Immersive Trainer on Camp Pendleton.

But some of the main reasons why we wanted to have you on today is: one, you're a cognitive scientist. You're actually the Chief Science Officer for ACSI Labs, which we'll talk about. But, you're in, you do a lot with cognition and decision-making. You're a practitioner, so you work with companies and organizations, not so much in a lab or an academic setting. You're really out there in the real world, applying your trade and doing what you do best. And you're in simulation and VR, and you work with subject matter experts.

A lot of our listeners, kind of like I was briefly mentioning, a lot of them are subject matter experts in their own field. Whether that's some of them are law enforcement, some of them are highly technically skilled people working for major organizations, some are... We have a kind of a wide stretch of folks that listen to our show. But all of them are very interested in the thing that they do, and that's kind of why they like us, because we talk a lot about cognition and sense-making, problem-solving, decision-making, and everything that we do. This is obviously a more informal format than what we do at work, but those, that's kind of the real reasons I wanted to have you on today. And because your whole backstory is amazing, and I'll have links in the details for everyone to check out even more about it. But one of the things that your expertise is in is accelerated learning. So, I figured, could we start there and define what you mean by what is accelerated learning?

Well, it does depend on who you ask. Even the co-authors of the book that I wrote on accelerated learning, we don't all agree on the best definition, so they're all in that book. But my definition of accelerated learning is basically accelerated expertise.

So, let's define an expert. An expert, for me, is a person who has a first-principle's intuitive grasp of the organizing forces of his domain. So, a physicist, for example, sees physics very differently than the rest of us and sees it intuitively. And I would say that that's true of all of us. We all have some form of intuitive expertise. When I was a professor, I used to illustrate this by writing something on the board, like some phrase, and say to my class, "Okay, now look at that. Don't read it." And they couldn't do that, because we're all intuitive experts at decoding the alphabet into some sort of meaning, and we can't not do it now. And so, when you get to the point where you understand a domain where it's a priori, you can't not do it, right? That to me is what we call an intuitive expert.

Now, Hubert Dreyfus has a nice little taxonomy. He has five stages of the development of expertise, a kind of genetic epistemology model. And for him, it'd be Stage 5, right? Whereas before that, you're kind of... but the other stages are interesting too, because they help us understand when you're not an expert, what you're like, right?

And you... one thing, Brian, that I absolutely love about this... We, I'm the comic relief, I guess, on the show. But my job is to "street" things up when things are big and scary, and the concept is too broad for most people to understand because it's out of their field. I like to bring it down into something that they do every day. Well, you just did that in your definition. Because when you're talking about a physicist as an expert in the realm, you're also talking about an expert model that comes from a bass fisherman that takes people out on an afternoon fishing and understands that the color of the water or the color of the lure or structure has something to do with when fish bite or when they don't. And he also understands that fish don't know when it's raining, right? They understand certain things that come together and when they coalesce, that creates a body of knowledge that's important in that person's day-to-day.

And so, what you're doing is not, you know, highbrow for only Chief Science Officers. What you're saying is every single field, no matter what it is, if we broke it down into critical skills, there would be people within those skills that we would deem expert. And we can follow an expert model and mimic some of those good behaviors. Is that an accurate way of expressing that?

Well, that's a... the last thing you said is a bit of a point of debate. Some of my colleagues think that you can become an expert by mimicking experts. I believe there is a developmental trajectory and you may not be able to skip steps.

We know that when kids are learning to talk, even though they never hear anybody say, "We go to the store," they say that because they're starting to understand that putting "-ed" on the end of a verb makes it past (tense), right? So, there are these interim stages in the development of expertise, and there is some décalage at a stage transition. And actually, my dissertation was on that.

But what's really interesting is once you're an expert. I'll give you an example. I was developing expertise in supply chain management technologies among blue-collar workers at New York City Transit. And I thought, "Well, this will be interesting, right?" They were very nice to me. They helped me with my study because they figured I'd never get a job, I didn't have any qualifications. And they thought I was like a community college student, and they were trying to help me. So, and I was Italian, and so were they, so we had a lot of cappuccinos together on the shop floor.

The thing that I noticed was I would use card sorts to see how their operating mental model was working when it came to the work that they did. Was they... they remanufactured air brakes. And, you know, they had a very specific kind of cognitive bias in how they looked at manufacturing that was very counter to the way supply chain management advanced technologies work. Roughly speaking, they start with the end and go backwards. Most people put everything there to need for a recipe on the table and do it bottom-up, and they were no different.

Once we had done some work with them, and some gaming, to transform the way they looked at their work, when I did the card sorts with them again to say, "You know, show me basically what you need to get this started," they couldn't replicate their novice behavior. They said, "Oh," and I did it. I said, "Well, how about this?" They said, "No, nobody would ever do it that way. That's a stupid way to do it. You're gathering all this stuff before you need it." And then I showed them a videotape of them doing it that way, and they're like, "Wow, who is that?" You know?

But I learned two things from that: one, when you are an intuitive expert, or you've reached another stage in your operating mental model, you can't go back. And also, conscious awareness has nothing to do with it, right? And that's when I got really interested. I said, "Maybe what I'm doing with gaming is not teaching people anything. What I'm doing is I'm reorganizing an operating mental model." And then I started looking at what's going on there, right? And it seemed as though — and this is to answer your question, Brian, what's accelerated learning — what's really accelerated learning, in my opinion, is the implicit reorganization of mental models through iterative trial and error cycles.

And I started to notice that, and I guess I should have known this because I worked with Catherine Nelson, who studied how the concept of time is created by language, that the brain has no sense of time. You know, this whole 10,000-hour rule, it's junk.

Yeah, yeah.

It is. Chronological time is just an accident of how education works. But actually, if you have enough cycles of trial and error, especially under a certain kind of time pressure, the brain will adapt. And a story... and we've, with some of our gaming environments, we've really pushed this to the point where we got two to three years of expertise in six hours. That was the fastest we ever gotten. But we had... (phone rings) Right now, answering it all. Sorry, my phone is ringing.

No, no, not a problem. For your fans!

Yeah, that's Malcolm Gladwell calling you right now and saying, "I think he wants advice on for hair, obviously," you know. And he could use it anyway. Don't, don't edit that out.

The point is that what we're, what we've, what our whole mission has been since that insight is we know that the adaptive unconscious is there, and we know that it learns much faster and much more holographically than things in school. And we shouldn't be surprised because human beings have been learning for 170,000 years, and school has been around a couple hundred years for the average person, right? So, it hasn't caught up.

And then if you look at evidence like people who are blindsighted, you know, who can see even though they don't have an occipital cortex, a visual cortex, we know that if you have a goal and you need to adapt, your brain will learn, but you may not experience consciously what it's doing. So, then the challenge is how do we reorganize the adaptive unconscious for the most powerful form of expertise, because it does whatever it wants. It's not like you say, "Okay, now be aware for a few minutes and, you know, try to control all these incredibly powerful and rapid processes in the direction I want you to go." Doesn't happen.

So, what we do, because the adaptive unconscious is very idiosyncratic, it's got ADD, it looks at everything, and it's kind of a free association chaining machine. We create virtual environments. Now, a virtual environment is like a really good movie. It actually isn't capturing everything that's going on. It's actually very specifically focused on what the designer wants you to be overwhelmed by, and everything else is kind of back in the background. But your brain doesn't experience it that way. When you're in a really well-designed virtual world, you experience it as the whole world.

So, you just hit on, we could just do the whole couple hours on everything you just brought up. One, because you, you also validated our approach, our methodology to how we do training as well, and with everything you said. So, before we get into the virtual world part, because I want to get there and I, you know, a simulation training, I, when I call something simulation training, I use that in the most general sense of simulation. That could be two people, not on a computer, just in person simulating some event, right? So, everything, that's what I mean by simulation stuff.

But you, you brought into a lot of everything that we address in terms of the unconscious and how it is. And, and when we specifically, we're getting into human behavior because you brought up different cognitive biases and how the brain actually learns. You know, we come at it from such a different approach because we, we know that those fundamentals are true, and why we use a lot of images and not a lot of words, and, and, you know, let you use a little like we, we teach a lexicon but it's also kind of influenced by you. So, it's your own lexicon, not so much to remember exactly what this term is and what it means. It's what does it mean to you? And, and that sort of, that sort of unconscious free play, you know, when we're talking specifically about human behavior, one of the things we tell people, because you got into the sort of the Dreyfus model of subject matter expertise and there's another one that talks about, you know, being a conscious competent and unconscious competent and what that all means.

And we actually sort of flip it upside down a little bit. And what we tried to do for a little bit is we say, "Alright, when it comes to human behavior, you're actually almost you're an unconscious competent. You, you have the skillset I'm going to teach you. You're not aware of it yet, because maybe you didn't get that language, maybe you didn't know how to articulate something." So, we provide sort of a scientific lexicon that's legal, more unethical, that you can use to articulate. So, you almost become that conscious competent. But then at the same time, we have to explain to people like, "Look, conscious awareness is, don't worry about that part. You, you actually want it to be completely, you want to get to the level where it is unconscious."

So, and you, you brought up the learning process of that specific folks, it was the New York Transit system, I think. And like, you know, we love seeing that because we have those same experiences where someone's like, "Oh, no, yeah, I knew it was this." And we're like, "Yeah, but, but five minutes ago you said it was that." "Like, no, I would never say that." And you're like, "Okay, well, I know learning has occurred, so that's good." And they've taken ownership of it. So, they're like, "No, I got to that answer." I was like, "Well, yes, because you use the framework, so I'm happy with that."

But that sort of, I look at that almost as we kind of explained it like top-down versus bottom-up processing and how people look at things, right? And I tell a story, I had two buddies growing up with, and they're the most mechanically inclined people I've ever met, right? They fix everything. They own apartment buildings, their whole family did, they worked on everything, right? And, and one of them could look at literally like a pile of wood and tell you exactly the size, you know, house and how many bedrooms you could do, you could build with that. Where the other one would look at like a completed house and go, "Alright, if you want to build that, here's everything you need." Right? And it's interesting how people, you know, we, you can use both of those processes, it just depends on where you're at. And I think when you're getting into, you know, maybe you can explain this better, when you get into that area of subject matter expertise, you've sort of created whichever process works best for you and then seen it over and over again. Is that like kind of what you're getting at?

I think what you're, what you're talking about is, we see a lot too where people are already experts, they're just experts in a way that's not working anymore, right? And they have all the content because they've been doing this for decades, but it's reorganized into a sort of autopilot model that's not going to work for the for the changes in this situation. So, what we're doing there is instead of going one to five in a Dreyfus model, we're going five to five, and you're reorganizing the content. But the process is very similar. It's just that the person who's already a subject matter expert has a lot more to work with, and in some ways they're much easier, because they do have these incredibly rigid biases that are really getting in their way. But once you get them into that trial and error activity cycle, what the brain is really doing is unlearning that and reorganizing for something more adaptive with the same content. And it's very interesting to see it happen. Whereas novices going through the same exercise, they're really starting with very little in the closet there, so they don't come up with as good of an outfit.

And we, you know, that's one of our sort of points that we make a lot is that don't give up on the old guys. They actually have quite a bit of content and can be sort of converted to be intuitive expertise if they get the right methods of doing it. And it's gaming and rehearsal in simulations.

And that's one of the things, Brian, I want to hit on. So, one of the reasons that I fell in love with your brain a long time ago is that we're meeting you, Lia, we, Brian and I are in the business of accelerated expertise in extremis. The situations being very kinetic war, or cops on the street involved in a very dangerous situation where lives are hanging in the balance. They come very quickly and there isn't a great degree of learning curve. And so, what we have to create is context-dependent strategic or tactical or operational rehearsals. And we have to reorganize cognitively just a certain amount of detail to create a response. So, our decision point has to come much earlier in the process, and the decision can be okay, not great. And that's another thing that you do in your work, being able to identify those decision points early enough that will have a profound outcome temporarily. Right? And I love that because that's what we do.

So, we deal with body bombers and IEDs and vehicle-borne IEDs and snipers and those types of things. Well, just because you're dealing with the mining industry and you're worried about the tread wear pattern on specific vehicles over time, it's the same issue. That's the beauty of it. Is that as long as you can create a context-dependent world within which to rehearse realistically, I think that's the magic. And am I close on that, or am I way off?

No, no, you're, no, you're right. In fact, a lot of our stuff is very relevant to law enforcement because, you know, and I would say one of the things I've discovered working with a lot of domains is how much overlap there is. And also how like mining is very similar to pharmaceutical drug development because you're both, in both situations, you're intuiting the value of a raw material and transforming it into something valuable. And mining is very similar to law enforcement because it's dangerous, yeah. And one wrong move, everybody is toast. You know, you can, you know, you can neglect tire tread on a multi-ton hauler and if the thing catches fire, the person driving the truck is history. End of story, right? It's not like losing your tire on your Prius. So, I think, but, but at the same time, when you become comfortable driving a multi-ton haul truck, you start to feel like it is your Prius and you may. So, helping people in simulations really show that the consequences are much more dire. And we can kill people, especially virtual world simulations, and it's pretty dramatic for them, even though they're not really dead.

I, and I love the consequences piece as well. So, the beauty of repeated rehearsals within a realm where you can make mistakes and you can try to test hypotheses and you can skin your knee and go back over and over and over and then consider the outcomes. We can't do that on the street, although some people are, right?

Yeah, yeah, there's things, yeah, there's discovery learning going on out there and that's what we call case law. There's no nothing before the Supreme Court.

No, no, no, you get what I mean.

That's that was me. I would say as a kid, I was the one who like grabbed the handle on the stove, burned their hand and then went with the other hand and did it to him, like, "Okay, that's going to be like that every time."

But you're, you're already, you're so, so let's get into exactly what you do then with ACSI Labs and its virtual worlds, and how you got into that. Like, because you guys have, and I'll have the links up in the episode details for folks to check out, just your Future View platform and you've got all these other things you work on, because you can create a virtual world and simulate some of this, right? So, how do you take what you know as your expertise and what works and put it into a gaming engine? What is that like? How does that work?

Well, first of all, I, again, I have to be very clear about this. We, we have tried putting our stuff in gaming engines, like all of OpenSim, et cetera, and we got frustrated because even though there was a lot of value in that, it, it, we're cognitive scientists. We need something that does cognitive science, right? And there's no gaming engine for gaming purposes that does that. So, we said, "Okay, we're going to have to make it."

And how our environment is different, even though it looks like a lot of multiplayer gaming environments, is it's a lot bigger. You can have a whole city in there if you want, if that's your concern for object. In other words, if you need to train law enforcement people to be vigilant about the patterns of activity in a whole city, we can put the whole city in there. The other thing that we do that's I think important is we track every decision, every micro decision by everybody. Where did they walk? When did they do it? How close did they get to an IED or to a bad guy? Did they notice any subtle indicators that the bad guy is doing bad things in that building? The simulation is what we call agent-based. So, it has a self-knowledge that records your response to it. Like, "He just walked by, he was within two feet of what expert law enforcement person would see as an indicator of illegal activity, and he missed it." That's data. Okay?

So, we make these smart environments, and we have people, we give them a goal. I'm designing one right now, and not only do you have to find the bad guy, but you have to decide what to do about it and evaluate all the risks and benefits. And the bad guy and all of his or her environmental constraints know what you should be looking for and are reporting on you. So, it's like a smart world. And then all of that is stored in a very detailed log, and we can use that to drive detailed metrics so that we can give you a heads-up display. "How many misses you had in that, the last five minutes? And how many hits?" And you're going, "You know, I didn't see anything." "Well, it was there to see and you missed it." So, you get that, that feedback.

It's because it's not just what you do, it's what you fail to do.

That's right.

It can sometimes be just as important. Like, I mean, even to tie back to the, to the tire tread example, you know, failing to inspect that tire could lead to a catastrophic meltdown of that vehicle, then therefore that entire operation, and therefore that company. I mean, if that's so significant, Greg always sums it up as, you know, especially with law enforcement, "Every tactical decision you make creates an operational certainty and a strategic unknown." So, you have to balance all of those things out. And I think a lot of people do miss it.

I just want to hit on that point because it's incredible. Measuring what you did do, what you didn't.

And for businesses like mining, we also calculate the financial costs, right? At a very detailed level. Like, you just degraded the value of your asset besides setting yourself on fire. You know?

But those tires are cost prohibitive. Therein lies another form of bias: a fiduciary bias. Because replacing those tires at the optimum level of replacement and sending them back for retread or whatever, that's that's a big cost and that cuts into my bottom line. And that means that if I go another day or another days or another week, then I make more money. And who's really going to notice those little, you know, outside that?

Well, when you go to the gaming engine and you can see the consequences of your actions and you can see that those mistakes not only kill, but it kills productivity, and it costs more in the long run. That's where we are in law enforcement now. Law enforcement thinks that they can fix the dike by sticking a finger in it, and we've got plenty of fingers, we're doing fine. And the idea is, "Okay, but what about tomorrow? What about next week?" And it's not, it's the accumulative effect is so damaging.

And back early on in my experimental phases, I looked at Hicks and Zip because I was working on predictability, what's likely going to occur. So, if a person was going to run from a copper, I could predict which backyards they would choose and which they wouldn't. For example, if there was a shed in the backyard, you're more likely to choose that. If it had wood fence instead of a cyclone fence, you're more likely to choose that. If there was a trailer parked in it and no dog, you could use that to boost yourself over the fence and hide in the shed. So, I was creating these mental models for coppers with no computers and, you know, no internet type of thing. And what was happening is they started becoming a best practice. But you can prove it. You actually prove it in the world, in the game, because I'm immersed in it, and I see the effects over time of all my decisions. That's magic. Tell me about that.

Well, and you make the mistakes. So, in other words, we tell people, "Okay, we're not going to tell you anything except what your goal is, and you got to figure it out, and you've got, you know, 10 minutes or less." And what they do is they do what they would normally do. They make all the mistakes they would normally do, and they suffer the consequences. Or they miss all the cues, and they get to see that. And we don't tell them which ones they miss. We just tell them, "Go back and do it again." It is a safe way to iteratively, you know, rehearse over and over again until you figure it out.

And our event generator, there's a couple things about our platform. Our event generator also changes the context. It like reshuffles the deck, right? Maybe there won't be a shed in the backyard next time. So that you don't just memorize a solution, but you actually start to pay attention to how these things are organized and forward-simulate in your mind what could happen. And then I think that the, we're not sure why, but in businesses when we show the financial cost of these things all the way up to stock price and market cap, and also more locally and what it cost, that really affects people. Yeah, and they really start to understand how they're located in the organization.

It's almost like I used to explain this to students is the DNA of your whole body is in every cell, and that tells your cell what its job is. You have to have that when you're working with an organization. You have to have not just the local consequences, but also how you affect the value or effectiveness of your whole organization.

But you know, cynically speaking, the thing I worry about is there are people making a living on the chaos who don't want it fixed.

Yeah.

Of course, yeah. And you, you just nailed it, Lia. Yeah, with the DNA to the lowest common denominator.

Let's... Greg is going to street it up now with all this cold medicine on board. But one of the things like, we go in and we conduct a vulnerability assessment of a physical structure, let's say. And one of the things that it shows their book and they show us their cameras and they show us their protocols, they even rehearse them for us. And then we say, "Okay, we're going to take a break, have a cup of coffee because we got to catch up on our notes." And they prop open the fire door, disable the alarm so they can step outside and smoke. Right? That's the DNA of the whole animal being influenced by this one cell. And so, we try to teach people that if you don't, a janitor sees more in an hour than most of your executives are going to see in a month. So, if you don't train and heal holistically the whole event, you know, you, you said something, it's very iterative, but not intuitive when you talk about a gaming engine that you build and how yours works compared to a first-person shooter, for example. First-person shooter, "Don't worry, I have unlimited lives and if I really, really, really get killed, there's a place around the corner that I can trade in and get more lives and don't worry if I make a mistake, I can keep trying the thing over and over." You know what they don't do? They don't do the consequences. Life has consequences. So, if you, if you have to map out how I'm going to get better, what's the difference? I can go through the model and never learn anything.

Well, no, because the model is subtly changing your experience. So, if you don't follow good habits, you're never going to succeed.

And you're not showing them something unrealistic. When you're building the model, you could build a model of Mars or some future that you've never been to. But you don't. You build their business, so they can walk around in the closest thing to reality and pick things up and test them and sample hypotheses. Right? That's amazing to me. Talk, talk more about why that was the key. Because, you know, we all know arousal goes up, performance goes down. So, there's not a lot of flash bangs and funny wonderful and shooting and all that other stuff in your game. Your game is much more like life. Right?

Right, right. But it's if it's, if it's your life, it's interesting to you.

Yeah, exactly, yeah, exactly, right.

One of my first experiences with that was, actually two. When we were making physical models, we worked with a uranium refinement company that makes nuclear fuel rods, and they were never profitable. They are now, all because of us. You know, you're welcome. And one of the issues was to make a model of their uranium refinement cascade, which is 72 acres under roof. And we had to make a model of it that worked, and we had to have the little turbines and the cooler sublimers fail after a certain number of hours of operation. So, that's why virtual worlds are nice because these smart objects were very hard to program.

Anyway, when we showed it, these people did not want to be there. We had union management together, and they thought the whole thing was going to be stupid. We brought them into the room, we took the big sheet off the big model, which was 22 feet long and running. You know, they, they couldn't stay seated. They kept, they're like, "Wow!" They got their hands all over it. We're like, "Sit down, we got to give you your onboarding, kids. We're not starting yet." But they did not want to (sit down). It was irresistible to them because it was theirs. And it's amazing. Yeah.

And it's the same thing with mining.

You know, I show people our mining simulations and they go, "Wow, this is really hard to figure out. It's so big." But you put a mining person in there and they go, "Wow!" And they don't want to leave because it's intuitively like a mother tongue to them.

Exactly. There's a concept in the military called "left seat, right seat." And if I'm going to occupy an area of operation that Brian's unit is in, there's a seamless way of Brian and I going together through it: "Hey, this is the mosque, these are the key key players here. This is a community well and, you know, the cattle over there are for the next jurisdiction, you don't have to worry about that because they're covered." And you trade this knowledge. And what we found is high-functioning units do a left-seat, right-seat, that that change of command very well. Low-functioning units bring all of their biases with them. And so, it's just like adversely influencing AI, you know, bad AI, because if you feed the information poorly, you're going to get a poor outcome. We could tell the units very early on where we are going to have trouble with because they followed certain bad patterns. Right? But then when you brought it up to them, they were like, "No, no, it's, you're missing the point. The point is..." So, we had to create a more structured environment where they went through in a series of checks and balances. Now, how do you do that? Because I know you do that, but how do you do that inside the game? What are some of the consequences and how do you recreate them?

Well, the most important thing in our simulations is the goal or the mission, right? Because that has the most powerful effect on the brain. The brain does not like to be wrong. So, if you're not accomplishing the mission and you get minute feedback that you're getting farther away, what you're doing is jeopardizing the success of the mission, even where you walk, right? The brain doesn't like that, and it's going to start to change it, change it up a bit, try new things to get better, get better micro-feedback, right?

Now, in mining, we use just traffic lights. Every time you do something, you get a red, green, or yellow. Even a conversation with a subordinate is scored, whether you walk by a hazard you don't notice, it's scored. So, you're constantly getting these traffic lights at you. And what we find is that after about a 45 minutes to an hour, you're getting mostly green because you don't like, your brain doesn't like getting those reds, and you start being more vigilant and aware. So, what we try to do is we don't have a kind of linear sequence. We have a goal, and then how you get there is kind of up to you. And that allows you to make all the mistakes that you would make when you're on your own and get them reorganized to be more appropriate. But it also accommodates multiple entry points into the problem, right? It's almost a universal solution. And some people are more novice, and some people are expert in the wrong way. It doesn't matter. It's like you're, you're accomplishing the goal, you're not a cop, you're accomplishing the goal, you're not whatever you're doing. Because you're a novice or because you're an expert in the wrong way, you're getting reds. So, you got to figure it out.

And you, I like how you describe it as an expert in the wrong way, and you kind of hit on that at the beginning too. And that's a lot of, we, we've seen that a lot in our work too, especially when you have a lot of someone who's been doing something for a really long time. And, you know, the whole thing, I, I, because I used to make the joke like, "You know, Brian, what do you do for," when someone asks me, "Brian, what do you do for a living?" I go, "Why, I teach old dogs new tricks." And they're like, "You know, what do you mean?" I was like, "Look, you're, you're an expert. I'm never going to tell you how to do your job. Let's just rearrange it."

We had, I have seen it, probably you have too, similarly with some of your clients where, you know, we had, it was a police officer and after our first day of training or something came up, he's like, "I'm now realizing how many situations I've been in where I, I was the contributing factor that caused that situation to happen the way I handled it." And came in just, and they almost feel bad. And I feel like, "Don't feel bad. You were trained. Someone showed you how to do it that way. That's why you did it that way. Now, knowing what, knowing what we know now, like, you can change that. And you already have this knowledge, skills, attitudes, aptitudes, abilities to do that sometimes."

But I'm curious to you because this is sort of, I'll get to my question in a second, because you brought up a little bit about, measurement and assessment is very difficult. I think people don't always know how to measure and assess whatever problem they're trying to solve. You can look at what happened, you know, after George Floyd was killed. That's, that's cost over six billion dollars, people have said. So, we can do that with a monetary amount. But like, what, what's the cost to that neighborhood, to our country, to the trust? And that, that you can, that is hard to measure. And that came down to one person doing something that they shouldn't have been doing and not paying attention to the situation. It's like, when, when you look at at these effects, we, in the moment, we can't measure it. And even when I'm an expert in my domain, it's hard for me to measure it because maybe we have something that our company wants to see, what's the bottom line. But maybe, maybe that's the bottom line this quarter and they're not thinking about over the next three years. And so, that's going to change how I look at it.

And so, what I'm getting to is the cognitive biases we come in with as a SME (Subject Matter Expert). Is there any, you know, similarities that you see across the board in people that come in that way? Like, what are the typical hang-ups, or what is that bias that that person needs to have that lightbulb moment, and how do you get them to there? Like, I would love for someone to be listening to this conversation and go, "Oh, wait a minute, maybe..." You know, and that's hard to do on a podcast, you have to do that during training. But what are those common things that you see with those cognitive biases that get in our way and how do you guys address that in in your training?

Well, I think that there could be ways of measuring the impact of bad behavior on the value of communities by, by indirect things like people don't buy the houses in that neighborhood anymore, right? And one of the things that that we've been able to do, and some of my colleagues are much better at this than I am, is they look at innovation. Companies that have toxic leadership are not innovative. And they have high turnover. And there are pockets of what we call the O-ring problem, where bad news doesn't get passed up and bad things happen as a result. That tends to be a symptom of toxic leadership, for example. And that it's, it's actually for us, kind of easy to measure. We measure it in all these financial metrics that already are kind of used by people who study publicly traded companies.

Now, for something like a neighborhood, I don't think it would be that hard. I think it's the only reason, you know, business seems very mysterious and very volatile and unpredictable. But people who need to predict do it very well. I think that the same kind of attention could be paid. And I think real estate prices, purchasing, businesses that do, you know, businesses do a lot of research on traffic patterns before they decide to open a store on a street. They don't want to be there. You know, those are, and then the tax revenue from the purchases to the city is not there. And those things are very easy to quantify and to put in a game.

Yeah, now that makes so much sense. And let's look at a very, I'd like a least objectionable outcome. And I'm always looking at the street, so I'm thinking of like a teeter-totter or a scales of justice kind of thing. And, and when, when you look at certain games that Brian and I have been privy to, and it's fair to say, I think that Brian and I, and we are working together on a project that hopefully will, will, you know, take off and everybody will finally see it for what it is. Because there's certain things that you can handle notionally, like the constructed, the building, the smells down on the street, unless they're very specific to a meth lab or explosives or something like that. Those are certain things that, that can be gray, we don't need to see them. They're not as impactful to our cognitive brains.

But there's other things, and specifically in mistake genealogy, that are electrochemical essentials. Because whether it's good or it's bad, you get that electrochemical neurotransmitter pumping in your brain, and you want to repeat that behavior. That's how a casino can get us to almost win then come back. Well, mistakes are a lot like that, right? So, when, when I see, and Brian, you brought up George Floyd, so I'm going to go there, and I've never gone to George Floyd on tape, but I'll say this: after George Floyd, administrations looked at it as a situation judgment test, and they were coming in with "if then." And you know what the bottom line on George Floyd was? We had to obviate the poorest decision that was made. What in the greater good of anyone was it incarcerating or arresting at that time George Floyd? Because it was a low-level crime scene, felony counterfeiting. He had pills, he had drugs, he had all these other things. Who was he hurting? He wasn't hurting society. That kind of behavior went on in that society every single day. Somebody chose on this day at this time, "This is going to be the person that's going to jail." And they never once saw the spirals. They didn't see the ripples that were coming out from that major decision.

What I loved about what you're doing as Chief Science Officer — let's remember that — I love what you're doing is you're smacking us in the face with the obvious. And by doing it over and over, a little subtly at the beginning and maybe a little more powerful, we learn. And that's so simple, but it's amazing, isn't it? I mean, it's that simple, isn't it?

It is that simple. But when you think about how our species has survived over 100,000 years with no evolution, right? And we were living in a cave with the same brain that we're now using an iPhone, we are able to learn when we need to to adapt. Yeah. And we don't need school for that. We just need trial and error.

And I also think that the reason the simulations work is because when the brain is wrong at the most primitive level, we see it as a threat to our survival. We are adaptive survival machines. We're preying on the big, you know. And yet we've figured out what we have to do to adapt equals survive from in our primitive brain. So, that's why the games are hard to design, but also very powerful. We have to make sure we don't shut down learning, like activate the fear response, right? But at the same time, we want the brain to think that survival is a little bit threatened here. So, I think that's why games are fun, you know. I think that's why, I think that's why Tetris is fun. There's, there's a level at which if they all crash at the bottom, you think you're going to die. And but you know you're not. So, that tension is very important to activating the most powerful forms of learning. And that's that's a good, that's a well-designed simulation.

And no, and you, you, you, that was a great explanation because what you said too is, you know, you're, you're able to learn when we need to adapt, and that need has to be there, right? There, there, we have, whether that's, you know, we, we made up the need or there is an actual need to be there, right? We, we need to have that kind of electrochemical response from our brain. It has to buy into that, and we have to tie into that survival. But you also sort of put a limit on it. You said, "You know, enough, but we can't create fear." Right? We can't create this, this, "Oh, my gosh," you know, "everything's going to," you know, "I'm, if I do this wrong, you know, everyone's going to die." Right?

Because I've seen the good and the bad and ugly in training. And that was even within the Infantry Immersive Trainer up in, in on Camp Pendleton where I'd see people setting up these scenarios, and at the end of the scenario, it's like they get overwhelmed and they all die. And I'm like, "What are you doing? Like, that, you're training that Marine to think, 'When this occurs, there's no way out and you're all going to die!'" Like, you have to have a way, there has to be a way out. There has to be a win. I have to be able to at least see, taste, touch, smell some part of a victory here to get kind of my brain, and I'm talking about in a training environment, to recreate because I see a lot of what I would say, you know, well-intentioned ways of utilizing technology that are just completely the wrong way to do it. And, and so it's virtual environments, but it turns into almost a first-person shooter game. Or, let's, let's make you do a bunch of, let's run some laps around the building, do some push-ups, and then put the VR goggles on and go through it. It's like, "Wait, wait a minute. Wait, wait. What are we trying to simulate here?"

And so, that's kind of things like, what are those limitations? How do I set that up if I'm listening to this going like, "Hey, we do some virtual training, or I understand simulation a little bit." Like, what, what things need to be there and what shouldn't be there? What should I focus on and what should I not focus on?

Well, you know, we make our world VR accessible with the goggles, but everybody takes their goggles off. And I think it's because when you're trying to learn how your behavior affects the big picture, you do much better seeing yourself in the scene. And you like, you do much better seeing the whole scene. And people intuitively, I think what's happening is their pre-conscious brain says, "You know, how am I going to win? I'm not going to win." And with a first-person view where I can only see four feet in front of me, and I have to exactly. You know, I, I need, I need almost a drone view of my life right now. And we make that available too.

So, but you know, VR and AR have their place. So, I guess what I think about all the time is back in the nineties, the Internet was one thing, and those of us in academe had email, we had Internet. My friends and I cracked, you know, hacked into the Kremlin. It was really fun. But, you know, the idea was that the general public can't have the Internet, it's too dangerous. And I thought the Internet is not one thing, it's whatever you do with it. And VR or the metaverse is the same. Yeah. Some of my colleagues are looking at what Meta's doing with their worlds, for example, and we go, "Wow, that is like so 2005." We made, we made those mistakes, you know. And we're not, you know, we're intuitive experts, we're not even sure what's so stupid about it. Exactly. I don't know if we could explain it to Zuckerberg, but the fact is that nobody's going to feel like they're in that world. No, nobody's, you know, that's like, you know, that's watching cartoons.

Well, yeah, that's such a perfect example of what I mean. And it's like you're, you're trying to recreate a real, you know, existence or real scenario or real experience, I should say, in this virtual world. Why, you don't, you don't need to. That's not, you're, you're exactly, your brain doesn't need that. It doesn't want that. It doesn't care about that. It's these certain elements that allow it to think. And you just brought up one, you know, needing to see yourself in the scene, and you know, correct in the wrong. But you're really talking about perspective, like, and, and if I have literally an overhead perspective, and I always tell people like, "Because, you know, you hear the sayings, 'Hey, you got to walk a mile in that person's shoe, you got to see it through their eyes.'" It's like, "Okay, psychologically, do you know how hard that is for a human being to see the world through another person's? That's, that's next to impossible." Which you can gain a better perspective, both literally and metaphorically, by sometimes like that, like, and what's an overhead view? I see it's funny when I see law enforcement agents I'm working with, and they've got this, they get really nice drones and all this stuff, and like they're using it to film their SWAT raid. I'm like, "Hey, man, like, we're, this, let's, let's, let's open up this, the capabilities on this thing a little bit. You know, really cool camera, but what are we using it for? How do we gain some advantage over the situation just to get more informed awareness here?" Right? I want to make a better decision. And so, that perspective is huge.

And that's also what I, I love about the, the whole idea of what you guys have built is like, "Okay, I, I was not surprised when you said, 'Well, people just end up taking off the goggles and they stare at the screen and they want to be there.'" Like, that does not disappoint me at all.

Okay, physiologically, we have a functional field of view, and we see in certain degrees. Okay, if we're boys or girls, six to eleven degrees of field of view. And we have a peripheral vision to make us orient towards problems. So, AR, not bashing augmented reality or VR or any of the goggles, what they're trying to do is they're trying to say, "You have to walk in these shoes at this time and look around your environment," which is great. But what happens is they lost sight of that, not physiologically how humans are. That's not chemically how we learn.

And you know, Lia, when we were working back in the early days, in, in Iraq, it was kinetic. And people in America didn't know a lot about roadside bombs. What they built and they brought me to evaluate was an IED lane. And I said, "Okay, it's an IED lane. You got to change the name." And they go, "Why?" I said, "Well, you're predisposing everybody for what they're going to find in here. Why don't you call it a Domino's Pizza lane? And then when somebody comes in there, if they find the pizza, they get a free pizza. But every once in a while, you blow somebody the hell up. And that is now when they're starting to pay attention." Right? And the idea was at first I was, I was advised that my methods weren't going to be used. Six months, nine months later, we had Combat Hunter, and it becomes a law of the land. The idea is that people sometimes think that they're doing the best thing to make VR and augmented reality better. But what they don't understand is they're needlessly and laboriously weighting down cognitive load. Is being added where it doesn't need to be. Would you agree with that?

Yeah, in fact, I really like what David Eagleman said, the neuroscientist, that what we're always doing is creating a simulation in our own heads. And we have these two little holes in our head that give us all the visual stuff, and we have some tactile stuff, some of us have, you know, better than others, depending on, you know, how well we move. And we still manage to construct a whole damn world with that very little stuff. Well, we're not paying attention to everything. We're, we're paying attention to, you know, that, that experiment with the gorilla runs across while you're trying to... I always miss the gorilla.

Okay. And I think you're counting basketballs. Yeah, yeah, right. I'm like, "And I say, 'Yeah, I'm focused.'" Like most people who are trying to solve a problem.

And if you design your worlds so that they're easy for people to to reconstruct the experience in their own heads, and we put EEGs on people to make sure that we're activating the motor cortex and stuff. We want people to feel like they're there, and not that they're watching somebody else's vision of what they would like the world to be like. And I think that, you know, for me, I think virtual worlds are the best for, with a big screen, are the best for changing how we make decisions in context with long-term effects. But I also understand that, you know, even I didn't get that 20 years ago. And I think that, you know, we, maybe it's like I had my own trial and error cycles and now we come up with something very powerful. And you also need to track the world has to tell you what, you know, your impact. That's why we use smart worlds. That's why we programmed our worlds to be smart and agent-based.

And because that's how humans learn. I mean, I always look at little kids. How does a kid learn? Like, they fall down, they get back up, they try something a few times, they go back and, you know, I mean, like, you, you just, that, that's what you're trying to, we always try to create in a training scenario. And I know when we have it, I love it, you know, when like you just said, you, you know, you'll actually hook people up to measurement. And we can just see it sometimes where we're doing an observation exercise on two people meeting across the street somewhere. And you've got people on the radio and their voice is up and they're going, "Oh, my God, he's coming over here!" And they're trying to take notes and they're yelling. And like that physiological arousal is happening. And we're like, "Man, this is that to your brain. That might as well be the real thing. Cognitively, your brain doesn't care. It is. It is."

It's cognitively close enough. And you know what we hear? We hear from other trainers, we hear, "Well, we have a crawl, walk, run approach." "Now, your brain processes its environment. So, stop doing it that way." Right? And they're like, "What are you talking about?" "Well, you know, we've done this for years and you guys are coming in here, 'Oh, my God!'" So, we live up to the disruptive name, but it's nowhere on our brand. But sometimes you have to disrupt. Sometimes you have to kick the gosh damn money lenders out of the temple for somebody to take another look at what's going on. Right?

Yeah.

And what, what we love about your stuff is your stuff is all subtle, but it's profound. It's got a profound impact at the end. Our training is specifically, our in-person training is much more in your face, meaning that you've got to say, "Listen, you have to disabuse some people of the idea that they can keep doing it the same way and it's going to somehow change over time." It's like, "If it's a failure today and it was a failure last week, and most of the things associated with it are failures, you're probably going to fail." And it's because they're not training the right part of their brain.

We see people again, we'll start where we ended with the 10,000 repetitions. And they say, "You know, you can practice it." Well, you know what? That reload drill never made you a better decision-maker under stress. Right? And so, we stick to what we know. We're, we're so laser-focused about our lane that we know where the lane markers are, how wide the lane is, and what the speed limit is. And a lot of trainers need to get back in theirs.

How do you, I, because you're expensive, and you're expensive because the stuff, well, because it works too. But I want to ask a question in here. I've had so many people that'll come up and throw something in my face, like they'll go, "Well, in Kahneman's book..." And I'm like, "Have you ever really read anything Kahneman wrote?" Because, yeah, okay, you know what I'm talking about. The idea is that I go out there on the street and actually use this (expletive deleted). So, so don't just throw me a bunch of platitudes, you know, because I know that they work.

And the, the thing that that drives me to that question is you said something that I absolutely love. You, I'll paraphrase, said that scientists learn too, and that when we find a better way, we adapt over time. And that's amazing because you're not who you were 25 years ago, and I'm certainly not that person. How does that strike people that have known you that whole time? I mean, do they come to you and go, "Hey, you're, you're going back on your earlier work," or "You're changing your mind." How does that work?

You know, actually, they don't, which is kind of disturbing. You know, exactly. You know, they, they, what they noticed about me is that I was always a moving target. Like, I was always asking myself, "What am I doing wrong?" You know, "We know that, but just having a human brain, that we're missing a lot." So, we're always asking ourselves, "What are we missing?" And because I was always involved in that kind of self-interrogation, I probably haven't changed in that sense. But what we do changes all the time.

You know, we do, we do expert worlds now where we have a very specific capability that we're designing for, like servant leadership, right? So, I mean, I would have, even five years ago, not been sure that we could have done that, but I'm like, "You know what, how was it, let's try it. Theoretically, it should work." And I think that just being able to say, "We don't know..." I mean, the James Webb Telescope should be telling us, "We don't know anything." I mean, we're, you know, a couple years ago, we thought the world was flat compared to what we know now. So, just always take that attitude, and I think it, it will be a good thing. But I think also, to that point, it does, it, it, we have to have the same social bravery. We're going to make mistakes, we're going to be wrong, we're going to be ridiculous looking in 20 years from now. So what? We got, you know?

Right. You got to get there. And, and take away... Hold on, hold on right there, that's a beautiful takeaway. Brian and I hate to interrupt you, but look, look, Lia, what you're talking about is in police work, police are going to make mistakes. They are going to stumble. They do a lot of work out there. And what we have to do is we have to say, "We're giving them this really, really, really hard job, and we're giving them all these complex situations. Sooner or later, that's going to fail somebody." So, right now what we're trying to do is we're trying to journal our way out of it by making them look like boobs. Or we're trying to prosecute our way out by pointing to them and going, "You're the problem. You're the one that made this mistake," rather than trying to fix it.

And if we could take a more scientific approach, what do we do in science? Okay, when we see that that attack doesn't work, we go back to the hypothesis and we do hypothesis testing until we find a new model that may work, right? And then we test that one. If they would just listen to this broadcast.

You said something else, Brian, just give me a minute. You said something else. You said that in the virtual world, people take off their goggles. We've seen the same. We've seen it over and over. And if you can recreate it on your laptop computer and it's just as powerful, just as impactful, then why spend all of that money, comma? Okay, I'm not going to go there because it will get crap done all over the place. But you also said that immersive environments work, and we know that too. So, it doesn't need to be all bells and whistles all the time. It doesn't have to be so mind-numbingly bright and flashy. What it has to be is it has to be real, but the real that it has to be is cognitively real. Is that a fair assessment?

Yeah, I think so. And I think cognitive reality is relative. You know, if we go back to our uranium refinement cascade, I mean, to anybody else, especially TSA when we were trying to fly with it, so it was bizarre. "Maybe a weapon!" You know? But the people who work with those, that equipment their whole lives, they thought it was gorgeous, and they thought it was wonderful. And I think that that it's all about what's meaningful to you.

And I think the mistake people are making with the metaverse — and now I'm going to get on my soapbox — is there, you know, they're, they're gaming companies that are trying to be cognitive scientists. We're cognitive scientists who are living with the reality that we have to use a game because it's easier to make a smart world in a virtual environment than it is to build a physical smart world, and it's more scalable and reusable. So, it's a, it's a means to another end. But, you know, when you start making stuff too flashy, first of all, it's creepy. Second of all, you're giving people the message that you don't respect them, that you need to entertain them and entice them. And if you make it more like their real life, that's a much more respectful approach.

You know, we, we made a mining world and we were showing it to some mining engineers and they're like, "You, it's too clean." "Yeah, let it be dirtier!" "Yeah!" You know, it's, "It's too cinematic!" So, we made it dirtier. You know, we made it sort of more gross, and they said, "Hey, no, this is, this is what we're talking about. That's the spirit."

You know, we made Hoberman and we said, "We're going to cut out all the the flash and the whistles and everything. We're just going to use students as the actors." And because the actors have to record some pre-recorded responses because it's interactive, there's a time that they'll say, "Yes, pause three, two, no, yes, positive." So, the feedback we get is, "Yeah, but compared to some of the other stuff that's out there, your actors glow." And it's like, "Okay, if you're focused on the actors, you weren't focused on the message. Did you solve the problem?" Then they go back and they go, "Yeah, it was a wicked hard problem, but we were able to solve it." That's the key. The key is, can you, you know, it like, like you could dress, answer me, if you, if I'm wrong, please tell me. You could dress your environment in mining and make it look like The Wizard of Oz. You could have the flying monkeys, you could have all the the bricks would be gold on the street. And your brain will accept all of that and still want to solve the gosh damn problem. That's what they miss. And, and they're this Carnival atmosphere that we're getting. And those folks are at the shows. Brian and I aren't at the shows, we're teaching in the trenches. Right? And you're going to be the show, make sure you tell us about that. But, but Brian and I don't get to go to the show. And at the show, they have wine tasting and, you know, crudités and all that other stuff, and they talk to big thoughts. But you know what they're not doing? They're not moving the dial. And we're saying sometimes you got to take a giant evolutionary step backwards to make your training more cognitive. And they, they don't, they don't get it because they're saying, "Okay, well, ours has a pneumatic device that makes the gun shake and, and we have, you know, the limb immobilizer and we have all of these things." And you know, that's wonderful. But I bet 10 years from now, iPhone's still going to be around. Your stuff's not, you know, it's just a fact. Right?

Well, I mean, D. Andrews, who used to have the human performance labs and is a co-PI with me on a project, said that when they built the flight simulators, and they had all the turbulence and everything, experts don't pay attention to turbulence. They only pay attention to dials. And when I was, yeah, and when I was in a flight simulator for a helicopter, I knew exactly what he meant. You are not focused on any of the other stuff. You're only focused on what the dials are telling you: "Are you going to crash or not?" And I think, I think that, particularly, subject matter experts have, you know, they pick and choose what they focus on. They don't look at everything.

You've got to have that symbolic density there. That's what we call it.

Exactly, exactly.

And, and so, too much cognitive load, bad. Experts, the better they train themselves and experience or experiential tacit, whatever, knowledge they get, they can actually offload some of those things. And Brian and I have always looked at a computer to be just that: I use my computer to help me free up this space so I can think about the hard thoughts that, that, you know, the problems that need to be solved right now that are going to impact our destiny. And, and it's, I, I got to tell you, Brian, favorite episode because when you're talking to somebody that's this smart, and you're, you're a Wile E. Coyote Super Genius, so it's always fun having you on. And you challenge us, and that's good too because there's a lot of people in the industry that are spending a lot of money on VR and billions and trillions of dollars on VR, and that could be better better spent elsewhere and improve training at the same time.

Yeah, no, I agree with you. And I think, you know, one of the reasons in my early career, when my mentor was Sylvia Scribner, she was a pioneer in workplace tech culture. She wouldn't hire anybody that had not worked. She didn't want any peer students in her PhD program. And when we started doing research on workplaces, we had to go to workplaces and be almost professional apprentices. We had to learn what those people are doing. We had to be on the shop floor. Coming from a business family, it was handy because I knew how to do that. But, I decided, you know, I'm never going to do research in a lab again. It's always going to be with real people. Because, you know, you get 30 undergraduates and you think you've learned something about how law enforcement is done from that. It's you just flew in just trying to get tenure on the backs of a lie. You got to go hit the street and you've got to learn how to be there as a researcher without disrupting too much of what's going on. And a lot of our colleagues can't do that. I've brought some, when I was a professor, I would bring people, colleagues to like factories and stuff to show them. They always got themselves thrown out. I said, "You know, you got to realize nobody wants to see you walking around with a clipboard looking down at them from your glasses. You got to say, 'Look, I know nothing. Teach me. Imagine that I got to do your job in a month. What do I need to know?'" You know, exactly.

Yeah.

And and totally understand that I'm never going to get there. But I'd like to know a little bit of what it's like to be you. But at the same time, you know, one of the things that we learned with New York City Transit, for example, we ended up training 3,000 people on a complex technology, and we increased the MTBF (mean distance between failure) of this old equipment. And the first pass yield on repairs was, went way up, and we saved the property hundreds of millions of dollars. And then I realized we don't know how they're doing it. And we started analyzing their data entry patterns. And we realized there was quite a bit of homogeny in how people saw the same problem. But it was way beyond us in terms of what they were seeing. And I said to myself, "We don't need to understand this, we just need to know that they do, and we can, we repeat it."

Right. Yeah.

Exactly. And the only statistics we did was a chi-square test (chafe test) to look for homogeny. And we knew that we know that experts are more alike than like each other than novices are. Right? So, I said, "Let's just look for that in the data." And of course, the financial impact was huge. But you got to have that humility when you're doing this research. You're never going to be that level five. Whatever. Let me just tell these people I'll be right there. Yeah.

Yeah, perfect. Now, and Brian, while she's doing that, I'll tell you right now that Maslow is waiting, he's pissed at us from the grave just so you know, because that's exactly what we're talking about. Is that, you know, when you do in the lab, it's a great theorem. But that has nothing to do with real life. You know, you have to get out (into the world).

And she brought up a good point there too, and Lia, sorry, we'll be respectful of your time here. You know, I know you got, we've all got other stuff too, and I'm pushing off a dreaded call that we have after this which is...

Yeah, that's right.

But, but, the, you know, you brought up a good point there too about sort of that being a good student, right? And, and we've had other folks that invited us, "Hey, we want you to come through our course." "Well, it's not going to be how you guys do it, and you're going to be a little bored." Like, "Don't say that!" Like, I will come in there, like, "I want to be, I'm going to have an empty notepad and I'm going to be staring at you, and I want to what, what is this, you know, let me take it all in." Like, and, and that, and I try to do that almost with everyone I meet too, because you find out if they're, you because you'll find people, like we'll go to an organization, like, "Hey, this, this girl over here, like, she's really good at her job. Like, look at how much she does!" And people don't even, you know, notice that stuff because of personality or whatever that gets overlooked. And, and just, just simply letting people tell you what's important to them.

And, and the other thing you brought up too a little bit ago, which I think is important, you know, about sort of updating your hypothesis and looking at things differently, and how stuff changes over time. Like, my whole thing is that I hope, I hope that someone 25 years from now listens to this podcast and goes, "These people were morons! I can't believe we used to think this way!" You know what I mean? Like, that's the goal. Like, you look back at certain stuff and go, "Phrenology? How did anyone ever think that the bumps on your head told you..."

I was the assistant for phrenology. I put the bumps on a lot of people's heads.

So, so it's, it's, it's, it's good. But, but again, a lot of people have a hard time letting go. "Like, this is how it's worked for me, and this is how it's done." It's like, "But it's a different world. It's a different context. You have to update that. You still have all of this really, really, really great tacit knowledge from which you can draw on. Use that, but conceptualize it in maybe this new context or with this new situation."

Well, what I'd like to see in 25 years is that accelerated learning through iterative trial and error with gaming environments is like a no-brainer. Yeah. It's like nobody is going to listen to a lecture or do a PowerPoint, "death by Death PowerPoint" again. And everybody knows how they really learn, and it's just taken for granted. And, you know, even in my career, it wasn't taken for granted that adults could learn. You know, in my early career, it was assumed you get baked by the age of 21 and you're done. We now know that people change careers, or at least suffer big changes in their current career, at least 12 times before they retire, between 21 and retirement. We got to, you know, that's got to be a given. That that's not hard, it's not threatening. It's going to be fun to be different. And people embrace the journey instead of try to take a course. But we have competition. We have people making a lot of money telling people that you got to do this for four years. You don't. It takes, it takes maybe 20 hours.

Yeah, yeah, that's great. That's good. We love it that because that's our approach to it. Like, one, we're never going to waste your time. Two, you're going to be doing something the entire time I'm here. Like, I'm exhausted after teaching after a few days because we have to constantly engage that student's brain no matter what. Everywhere they turn, there's, there's some stimulus that that relates to the the topic at hand. And we always tell them, "Look, you're not going to remember everything right now. In a week, you're going to come back and go, 'That's exactly what they were talking about.'" You're going to see something and go, "This is one of those things I need to act now versus waiting for this thing." And that's, that's that whole concept, we try to do that accelerated learning, you know, in person as much with virtual as we can with some of the folks that we work with. But, you know, you know how the the tech folks are, it's like, "Well, it needs, you know, to be a better refresh rate on the screen and the graphics need to be better, and I need more." It's like, "No, you really don't. You just, you just don't. You don't need any of that stuff. I just need to engage you cognitively." And then I'll have you eating the popcorn and you'll be watching them.

Yeah, yeah, my my colleague calls it, "picking flies out of the pepper." It's it just keeps you busy. It's not adding any value.

Right, yeah.

Well, Brian, I just got to tell you, my favorite episode when I get to interact with somebody as smart as Lia DiBello, it's a lot of fun. I'm not saying that some of our guests are rocks on the, you know, floor, but some of them are. Some of them are on the show for very different reasons, right?

So, my sweetness of the things they've done wrong. Exactly. So, no, we, but we, we, what a fun time. Yeah, we, we appreciate you hopping on here and talking about this stuff. I could talk for hours on all this. It's really fascinating. I love your approach and just your experience and how you do it. I took a couple pages of notes. And, and also, I did it on, is like I said at the beginning, as a validation of a lot of what we do. Like, we're like, "Tough to have people see, like, I know we put on a show and we're goofy people, but we, we also really know what we're talking about here." Yeah. So, I was looking forward to this. It was really a lot of fun. We appreciate that. And thanks everyone for tuning in. I'll put all of Lia's contact information for the ACSI. She's, remember, you know, the director, excuse me, Chief Science...

Chief Science Officer.

Chief Science Officer. I'll have the links up so you can check out their Future View platform and everything she's got going on, and link up with her on LinkedIn. And I do appreciate you coming on. And everyone out there, don't forget that training changes behavior.