Context is the Real Intelligence:
Why AI & Humans Both Depend on It

00:38Thank you for joining me today for another episode of Data Stakes, where I have conversations with professionals who work directly in the institutional research or effectiveness field or are data adjacent in their support for higher education. My guests today are Larry Blackburn, CEO of Datatelligent, and Robert Miller, Director of Product Development. Today's episode is entitled, Context is the Real Intelligence, Why AI and Humans Both Depend on It. And it's a continuation of a popular AI focused conversation that Robert and I started earlier this year. So Larry and Robert, thanks for being guests on the podcast. I'm pleased we're going to focus today on a critical need for success when implementing AI for data analytics. And that is context. So let's set the scene for our listeners. Think about how often we communicate with very little information. Maybe someone says, well, that didn't go well. And immediately you wonder what didn't go well. Was it the meeting, the presentation, the project? Those four words only make sense if we understand the context. As humans, we rely on context constantly. Our experiences, prior conversations, shared knowledge, tone, and even body language. But what happens when we ask an AI system a question? AI doesn't automatically know our environment, our organization, or even our history, which means something really important. AI systems depend on context even more than humans do. And so today we're going to talk about why context is the real intelligence behind AI systems. and how something called context engineering is becoming one of the most important disciplines in AI development. So let's get started. How about we start with something simple for our listeners? When we talk about context and human communication, what do we actually mean?

02:39Yeah, so I can start the conversation here. Debbie, thanks for having me today. Excited to be part of your podcast. Congrats on getting it launched. I think as part of this series you've been running, right, it's given us this larger context, if you will, for the work we do. So in terms of human communication context, so context has been something I've been really, really interested in since I first started being aware of how it's used to help us use AI better. And so ever since I started looking at it, I also became aware that it also helps us in our human communications. So when we think about context and communications, right, so there's several layers of context that we can talk about, right? So think about when you're talking with somebody, how you're doing that, like what makes you able to have that conversation, right? So you probably bring in maybe prior conversations you've had with a person or other people, right? Shared experiences. So especially if you've had more shared experiences, you get more context with the person. As far as if you're in a business setting, right? You have that organizational knowledge that you're pulling in. And then other sort of human context, right? that we use, body language, tone, emotion, even cultural norms, right? So if you're in different settings, different cultures, sometimes if you're in a culture that you don't have experience with, you may feel like you don't have all that context, right, to be able to have that interaction. So that's a little bit about why it's important in human communication, right? So think about if somebody says to you in a conversation at work, you're talking to maybe when I'm having a conversation with our head of sales, I'll say, hey, I noticed our numbers are down, right? So because of that setting, that context, right, we can have that conversation in sort of a shorthand, right? So you don't have to give a lot of details to it, but immediately both sides have similar context that we can use. for that communication.

04:49So Larry, let me make sure that I understand. It sounds like what you're saying is the more aware you are, the more knowledgeable you are while having a conversation, the more adept you are at having a high quality conversation and a high quality outcome. Is that correct?

05:08Yeah, think that's why I think, you last time you talked about what Robert, right, AI really doesn't have intelligence, right? There's no intelligence. But by adding this context, right, you're able to have that interaction and have that intelligence, right? Now we're adding intelligence to these conversations.

05:31Okay, so that's a good segue to my next question. How does context work with AI?

05:37Right, so as I was saying, right, when we have conversations as humans, right, we bring all that context, right? You have all the years of experience that you bring, that I bring, that Robert brings, right? And nobody has to give it to you, right? You already have it, right? And so something that you just gain over the years of experience. But then when we think about... kids, they're younger, right, they have less context, right? So AI is kind of like that young child, right? It has very little context and you have to keep giving it context, right? So AI doesn't have that context built into it. You have to give it, right? And at the time you're having that interaction with AI, right? The more context you give it, the better the response. will be right and so that context is given in the prompt. Maybe there's conversation history that you've had with your AI chat bot. It can bring that in. It can use data from other systems or documents, right? So without that context, then AI has to do more guessing right? Because it's going to try to guess and with less context then. it's just going to the probabilities of a good response are less and less.

07:00Okay, so this guessing that you refer to, is this what people refer to when they say AI is having a hallucination?

07:09Yeah, that's a good part of it, right? Because again, think about the concept of the more context, the more accurate it can be in those responses, the more it has to go on, right? So again, think about yourself. If you're in a situation that's brand new to you, think about your first day on a new job, right? You're having to figure out things and you may make a lot of guesses that are less accurate than somebody who's worked there for a long, time, right? So AI kind of behaves that way. It's gonna try to guess, right? Because it wants to give a response, but because it doesn't have the full context, then it's gonna guess wrong in many cases, right? So that's what we're calling hallucinations.

07:54Can I break in there for just a sec, Debbie? So Larry, kind of taking this concept that you sort of laid out, if you sort of shift and think about the spaces that Datatelligent and you and your team work on all the time, I'm thinking particularly the industries that while they're not necessarily niche industries, they're wildly underserved in the general marketplace for tailored solutions.

07:56You bet, Robert.

08:19You know, accessible data, things like higher education or nonprofits. How does this context really sort of play into your ability to build out AI solutions in those spaces?

08:31Yeah, that's a great question, Robert. you know, having that industry knowledge, right? So, higher ed, right? And Debbie knows this, right, Debbie? You can probably agree. You have your own language almost, right? So, and Debbie, you know, think about your first day working in higher ed, right? So, what did you have to learn to be able to do that?

08:44Definitely.

08:53Right, so think about that, right? So. So having that industry knowledge, that context of what do things like enrollment and retention, those terms mean, right? And having knowledge about the data that all of the institutions work with, right, makes it so that when we have a conversation with our higher ed customers, they don't have to teach us, right? So you've heard a lot of times right? Consultants, when you start working with consultants, then you have to teach them everything you know. Again, you're giving them context in order for them to help you, right? So our team, we've focused on higher ed. We have then educated our team and given them that context to be able to work in higher ed without them having to teach us about higher ed, about higher ed data, about higher ed outcomes. We come in with that. context.

09:52Okay, so it sounds like what you're saying is we have to provide AI with institutional data definitions, correct? We have to...

10:01Right.

10:03Possibly, like you said, in higher ed, we'd have to give them some historical knowledge. Maybe we want the chat bot to answer questions about enrollment. So we have to teach the chat bot about how enrollment has looked historically, or maybe other types of rules that are unique to our institution. So are there any other things that go into AI context?

10:26Yeah, absolutely. yeah. Yeah, absolutely.

10:2630 tools or?

10:30So I want to give this concept, introduce this concept. in the early days of using the chat bots, right, there was a lot of talk about prompt engineering, right, and like how to properly ask the questions. And so I want to introduce the concept of context engineering, right. So context engineering is designing that information environment around the AI tools. tools, right? So giving it specific context such as what data the AI can access, what tools it can use, what instructions to give it so that it guides its behavior, giving it guardrails, right? Giving it business definitions, like you're saying, right? Even give it examples of what are good responses versus incorrect responses. So for instance, if you're talking about being able to answer questions about enrollment trends, right? It needs those institutional definitions about the data that goes into enrollment trends. You may wanna give it historical enrollment data, right? And so all of that gives it that context, but that has to be engineered into the AI solution in order for it to give those consistent responses that you can trust.

12:01Okay, Robert, you have anything you wanna add there?

12:03Yeah, you know, I mean, you know, Larry's done a great job of really sort of framing out sort of the importance of this and the role it plays and how we think about AI as we move into the future of its use in so many different areas and in particular in data. You know, one of the things I think on a technical level for those of our listeners that really like to get into the nuts and bolts of this is, you know, each of these different solutions out here have come up with their own. you know, sort of niche ways of accomplishing this exact task. But, you know, the key thing here is there's really, it breaks down into three areas. One, it is setting that context through a set of stored prompts. That's essentially you're instructing the model or the tool that you're using to behave a certain way, as Larry mentioned. The other is actually with the data itself, you have to create a semantic layer on top of it. All of our data has been stored in a million different ways, each unique to different platforms that different people use, institutions that built custom data. The semantic layer really operates in a way of almost a map to the data to understand what the data is, how it relates to each other within the data set, and how that way the model when it's looking at the data, as a very clear roadmap of how to navigate through it and understand the underlying data. And the third is really more around setting those guardrails as Larry talked about. It's then limiting the scope of what it's going to do. The models that as most people have experienced them in their personal lives are trained on mass public data. They will by default go out and search online for additional information. and incorporate that into your responses. Well, in an institutional use case, you have to really watch out for that. And so that's a big guardrail is focusing the resources that it can use and the source of its answers down to a very specific set of data. Now that may be very small data set, it may be a huge data set, but it's going to be something that's defined very clearly. to set those borders. This gets back to Debbie a little bit of what you mentioned earlier around hallucinations. Because these things are ultimately probabilistic relation algorithms, that's really all they are. They're just very, very fast calculations, right? Based on both language, understanding, associative aspects of data, et cetera. And so really what you're doing is you're setting you know, ground rules, definitions, and then saying, but I only want answers sourced from these places, the governed, reliable, accurate places to then get to exactly what Larry mentioned there. You want reliable and consistent answers. You know, all of us have had the experience of putting the exact same prompt in multiple times and getting slightly different answers every time. That's not necessarily that the answers are incorrect. and it might not even be hallucinations. It's just because it's running in a wild space, right? It's pulling from different places each time without all of that engineering upfront to focus it in so that it gives that reliable response.

15:20Alright, so let me see. Is context engineering becoming more important than a data model selection?

15:28I would say it's not necessarily more important at this stage. I would say they're roughly equal at this point of the evolution of AI. While a lot of the tools and solutions are starting to move away from needing tailored structured data and allowing you to just point out to raw data sets and things like that, we're not there yet where that... the tools are highly, highly accurate in that environment. By all means, we all know that this is changing weekly, daily, almost hourly. There may come a time where the context engineering is really the only aspect of engineering solutions. I could see that, but we're not there yet. I think at this stage of the development of the process, having that structured data model underneath is a critical aspect of success in, again, the institutional environment where you need such high levels of accuracy and real-time

16:22Larry, have anything to add to that?

16:24Yeah, and I think what we're seeing as we think about. Again, I think we're seeing a trend where institutions want to use AI more, right? Because I think one of the values, and I think you guys talked about it in the last podcast, that it cuts out a lot of layers, right, of getting to the answers, right? And so if you think about the multi-steps, and I know you've built some of these solutions, if we can kind of cut the layers and go from question to response, right? That's the value of AI, right? The challenge is how do you get the accurate response when you ask those questions, right? Because again, while we humans, when we talk, because we have all this context we're talking about, we can sort of infer a lot, right? Like, okay, I heard this question, but maybe in my mind, I'm thinking, well, you really meant this. So let me give you the answer I think you meant, right?

17:20Mm-hmm.

17:21And so you're able to do that because of context, right? So I think what we want to be able to do is to add that context. What we're seeing in these tools, we use Snowflake a lot, right, is that Snowflake now has this semantic layer that Robert just talked about, right? And so I think most of our work is going to be in that context engineering layer, right, because that's sort of the new work, right, is adding that semantic layer, because we get asked a lot, like, why can't I just point my chat bot, such as chat GPT or to my data, right? I think you can, right Robert? You could, but do you want to, right? And so I think our work is going to be working on that context engineering, that semantic layer, because then that will enable that.

18:03Yeah, yeah, absolutely you can.

18:18Question to response being more accurate, right? And then trusting it, right? So we know it's a new thing, so not everybody may trust the use of these AI solutions, but we want that trust, right? So I think it's through this context engineering that we're gonna build that trust. And when we've been talking to customers about this, they seem to respond well, it's like, okay, it's not just gonna be a chat bot connecting to my data. and I don't know where it's going and where it's getting its answers, we can now build this semantic layer which becomes that trusted layer, interpretation that we get to define, not the LLM. So I think that's the key to this and that's why I'm excited when I got this concept in my head. It's like this is sort of the key to building the trust AI solutions, right? And so, and I think it's something everybody can understand. I think everybody understands sort of this, this let's give it context so that we can, we can make sure that our context as humans is consistent with the context the, the AI tools use, right? So I think that's sort of the value here that we're talking about.

19:37Right. So you know in data work, privacy, especially student privacy is paramount. It's on everyone's mind and especially when we start talking about using AI. So what about PII? What about those types of data that are right there in the data environment? What can you do to prevent that?

19:58You know, that's a great segue actually from a phrase that Larry used while ago, Or a concept of just point the chatbot at my data. You know, just as you want controls in order to drive trust and accuracy, et cetera, you know, a big aspect of that is also security. It's, you know, in this environment that we're in, we all know it's evolving. The bad guys are using AI to actually... leap ahead of a lot of our protections that we have in place today. So there's a sort of counter to that as well, which is as organizations, we have to be much more cognizant of how we're exposing our data, where we're exposing it, and to whom, right? So it's not even just the engineering within the solution itself to make sure that any PII data isn't... exposed inappropriately just in general, but also tailoring it down to understanding who's using the chatbot, who is accessing the tool right now, and based on who that is, setting the appropriate level of security just like we would in our classical apps and tools and data solutions. So carrying over those security models is really important. In the environment at... that Datatelligent has built out where they're using Snowflake, this is inherently carried over from the existing security platform. Now, if you do expose this outside of Snowflake, you do have to take that into consideration as you design and build out the solution. But, you know, that is a critical, absolutely critical thing. And it is something that I would strongly encourage that anyone that's heading down this path of using AI on sensitive data, They thoroughly test this, both by different users, both positive and negative tests, really extensively making sure that they've considered this in their design and before they launch it out.

21:56Yeah, and now I'll add the human connection to this because that's probably, I've become probably more interested in context with humans as we communicate, realize how important that is. And when we don't do it well, you see the confusion on the other people's face. So I'll relate this, your question about PII and privacy to a human. And I'm working on this with my son. He's younger, obviously. You know how when kids get older, we don't want them to say, certain words that are inappropriate, right? It's because they have no context like, I just heard this word, I'm going to say it, we don't have a filter. So those filters are part of that context. So think about all those filters we have, that we don't say certain words in certain settings, right? And it's okay in some settings, right? So think about that too. So those filters, and protecting information is also a part of that context engineering. Telling AI what it can't do is also important for that context engineering.

23:02Okay, well, this was really interesting. Just to let our listeners know, we have been working with customers who are beginning to set up semantic views in their Snowflake data lake, and I have found it really fascinating. I have to admit that I was AI, very AI hesitant. when I joined Datatelligent in November. But the more that I learn about it, the more that I see that when you take a responsible approach to it, you train the AI, you teach it how to think about the data, you host it in a secure environment, and then you realize that it takes time. You're not gonna get this done in a month. You might not get it done in six months. And then even when you think you have it built out that semantic layer that you both referenced, you're still going to spend quite a bit of time testing it. And so I find it all fascinating and I hope that we can talk again soon. So in both human conversations and AI systems, meaning doesn't come from words alone. It comes from context. Humans carry context with us automatically. But AI systems require us to engineer that context deliberately. The organizations that understand this will unlock the real power of AI, not by building a bigger model, but by building better context. And so, listeners, thanks for joining us today. I hope you got as much out of this conversation as I did. And... Before we leave, let me ask you this question. If AI is only as good as the context we provide, what context are you giving your AI systems today? Thanks, Steve. Thanks, Larry. Thanks, Robert.

24:49Thanks, Debbie.

24:50Thank you, Debbie.