Charlie Sull – How AI Can Actually Measure Company Culture
36:16 Share

The world of business is more complex than ever. The world of human resources and compensation is also getting more complex. Welcome to the HR Data Labs podcast, your direct source for the latest trends from experts inside and outside the world of human resources.

Listen as we explore the impact that compensation strategy, data, and people analytics can have on your organization. This podcast is sponsored by Salary.com, your source for data, technology, and consulting for compensation and beyond. Now, here are your hosts, David Teretsky and Dwight Brown. Hello and welcome to the HR Data Labs podcast. I am your host, David Teretsky. We're going to have a bloopers reel from that first couple of takes this morning.

I am David Tretsky, your host, alongside my best friend, co-host, and partner at salary.com, Dwight Brown. Dwight, how are you?

David, I'm pretty good. It's a Monday, so as good as it gets on a Monday. You should have had a pistachio muffin like me. Yeah, that's good. Yeah. And we'd really be keyed up. Yes, we would. But we're even more keyed up. You know why? Tell me. We have an exciting guest today. Charlie Saul. Charlie, how are you? I'm doing good, thanks. No muffins for me, but I'm all good. But you've had some really amazing Colombian coffee, I imagine. Yeah.

It's interesting. Actually, Columbia, where I live, by the way, exports all of its best coffee. So it's actually kind of hard to get a great cup of coffee in Columbia. Really? Yeah, I can attest to that. Wow. That stinks.

Well, I would, I would, I would actually go into the jungles to get it if I had to, because I love fresh coffee. Yeah. Not where you want to be going to find fresh coffee. For lots of reasons, I imagine. Yeah. Charlie, why don't you tell us a little bit about yourself?

So over the past 10 years, I've been researching corporate culture at MIT. So we have launched the largest ever research study into corporate culture, at least measured by a number of employee voices heard from MIT.

And in the process, we developed a new way of measuring corporate culture that doesn't use the one to five point scale that a lot of people still use. It uses AI and analyzes textual feedback to understand what employees are actually saying.

Wow. Talk about large language model. Yeah. So do you have any published research that we might be able to have people take a look at? Yeah, a lot of our research is published in the Measuring Culture series on MIT Sloan Management Review. This is actually the most read series in their publication's history, and it goes through

answers a lot of questions. What drives the employee experience the most powerfully? What is toxic culture? How do you measure it? Two companies walk the talk on culture. I think we have about 10 articles in that one now. Oh, wow. Yeah. If you could give us links to it so we can put it in the show notes, that would be really awesome. I think a lot of people would probably love to read those things. Definitely. Perfect. So Charlie, like we do for every one of our guests, what's one fun thing that no one knows about Charlie?

That's one fun thing that no one knows about me. All right. I forget what I wrote in the response I sent to you guys. I guess something I don't, I mean, people know this about me, but I don't widely publicize it, but I'm a, I'm a terrible driver. I grew up in England where admittedly the, the, the driving test is very hard in England because there are all these roundabouts and you're driving on the wrong side of the road and everything. But I failed my driving test five times in England and,

And I didn't learn how to drive until I was 23 years old. So that's a fun fact. Wow. I actually have my driver's license from England. Oh, you got it there. Yes. I lived there for a few years and got it. And you're right. It is very tough. It was especially tough for a kid from the U.S. who had never driven a standard before. And it was all standard at that point still. Yeah.

no automatics and driving up Pembroke Hill with a million cars beeping their horns as loudly as they possibly could. I was with my driver's test exam and trying to pass my exam on a hill. That was the first time, Charlie. So I failed to once, but I got it the second time. But yeah. Yeah. So good. So you have a chauffeur in Columbia then. Yeah.

Uh, we do. Yeah, we have a driver. We don't have a car here. It's all these windy, long and winding roads. Um, so yeah, we, we have a driver. Um, I'm aspiring to have that as well. Although I am a chauffeur to my children and they were, they typically refer to me as, um, David when we're in the car because I'm their, I'm their driver, their Uber driver. Uh, I get no tips and I get less than five stars almost every single time.

Do they just sit in the back and you're the driver in the front? Mm-hmm. And if I got my mother in the car, she'd tell me exactly how to drive. But that's a conversation for another podcast. Today's topic is going to be a really cool one because as you mentioned in your research, you talk about cultural measurement. And we're going to be talking about how to use AI to understand and improve corporate culture. ♪

So, Charlie, our first question is, why would anyone care about cultural measurement? Yeah, that's a great question that I come into a lot. Okay, so cultural measurement is incredibly esoteric. When I tell people I've basically devoted my life to cultural measurement, it's, you know, a hard conversation at parties for the most part. But the reason...

Yeah, yeah. Then I really got your saying insights to come out. No. OK, so maybe you don't care about cultural measurement like most people. That's fine. But there is a good chance if you work for a business or if you're interested in business, you care about culture.

And there are many reasons to compare culture. I mean, it's the number one driver of employee satisfaction, three times as powerful as compensation as we found. I mean, it shapes everything about what you do at work, how your company is treating you, all these different outcomes, all these outcomes that affect strategy execution, like is the company agile, is it innovative? Yeah.

Culture matters, and a lot of people think that. So there was a good study at Duke done, surveyed about 1,400 CEOs and CFOs from big companies. CEOs, CFOs, not CHROs, CEOs, CFOs. And it found that the majority, 54%, thought that culture was the top three driver of financial value creation out of anything imaginable.

And, you know, culture matters. I mean, just go on LinkedIn. There are all these posts about culture. Everyone's talking about culture. Everyone knows on some level that culture matters.

But everyone also knows that culture is incredibly intangible, ethereal even. You know, it's this... People don't even know what it is. If you read the literature reviews of academics and culture academics over the past 40 years, the reality is they've spent almost all their time just arguing about what the definition of culture even is. There's no consensus about that. So what actually is culture? And the reason culture...

Cultural measurement matters is because cultural measurement is the bridge between something that almost everyone could agree matters a tremendous amount. And this ethereality, this intangibility, cultural measurement is what allows you to actually make culture tangible and quantifiable.

And the upside of that is if you can make culture tangible and quantifiable, then it becomes much easier to manage. And my hope is that culture will pretty soon become as easy to manage as any other very important asset in a company that people are already devoting a lot of resources to taking seriously.

You know, to that point, one thing that you said at the beginning there really kind of struck me. You said that CHROs were not part of that study, which is interesting because if you want to look at who's knee deep in culture in a company, it's your CHRO. Yeah, well, I think CHROs, 100% of them would say that that culture matters. Most CHROs think that culture matters. The people you got to convince are the CEOs and CFOs. Okay. Yeah.

But Charlie, isn't a lot of culture marketing as well? I mean, it's branding. It's how people feel about you, not just the employees, but how your customers and stakeholders and owners feel about you.

I mean, a lot of culture nowadays is just, you know, PR bullshit. If I can say that, frankly, it's, it's, uh, you know, going out term you used, it started with a B and then had an S in it. No, but seriously, so that is, that is what a lot of people think it is. It's the BS that we get fed. It's exactly right. It's, it's, it's companies going on their LinkedIn page and saying, Oh, we care about our employees so much. We were this and that we're this and that.

And the reason why such BS is allowed to fester is

is because there isn't any good measurement. If there was some reliable system of measurement and saying, oh, actually, you think you care about your employees, how come employees are two standard deviations less likely to say you care about them than your competitors? And if there was a, you know, a widely regarded metric for that, then this, so much of culture is just complete nonsense. So much of it is just complete nonsense. And a lot of that comes back to measurement. Right.

But before we get into the cultural part and the measurement part, sorry, the measurement part, when we see things like, you know, corporate statements coming from Apple or from Google or from Amazon, let's take those because those are three gigantic organisms in the U.S., right? Not only every household knows those names, they live them on a daily basis. And their culture affects the United States and how we live, right?

probably as well around the world, but much more so in the U.S. And the culture of them have been very much led by the three people, or in the case of Google, it changes a little bit. But isn't that kind of what leads culture is the heads and the leaders of those organizations more so than even the marketing? Because Tim could go to a White House dinner with Donald Trump, and that affects...

how that brand is then judged from a cultural perspective. No positive or negative. I'm not being political there. Yeah, absolutely. I mean, the, the top team of a company, especially the CEO is going to have by far the most outsized influence on the culture. I mean, we, it sounds like you're maybe talking a little bit about brand and external reputation, which is, uh, affected by culture. Sure. But, um,

We think of culture as a, as a company's operating system. So culture is the, the system that's, uh, determining thousands of different important algorithms in the company. So for instance, one algorithm that culture determines is, uh,

uh, my direct report just did something stupid. Is it okay for me to lose my temper and yelled him? Yes or no. Yes or no. And you could see how that would create a very different culture of respect around the company or, or a different algorithm is around, uh, agility. Say you're a distributed manager in Sweden and you just noticed that market conditions have just changed really rapidly in Sweden. So the question is, do I take ownership for that? Do I proactively respond to that? And, and, uh,

handle this on my own or do I report back to headquarters and wait a couple of weeks for their decision to respond to this market condition? So that's an algorithm around agility and ownership.

But lately, wouldn't that stretch into, like we've seen with Facebook and other companies, how they decide on letting people go and how they make those public pronouncements about them? Like, for example, these are performance-based layoffs versus what the reality was, was they were just...

Um, riffing people to, you know, hit better corporate numbers. Yes. I mean, that's, that's definitely, it's definitely going to have an impact on culture. I mean, you can read what meta employees are saying on, uh, on Glassdoor. They're going to be criticizing things like fairness, criticizing the extent to which performance is rewarded.

those criticizing strategy, those will all certainly impact culture.

Why don't we shift now to like one of the more fascinating things to us. I'm sure Dwight is on pins and needles for this one, which is how is it actually measured? How is culture actually measured? And what are you actually trying to change with that measurement? Well, what we're trying to change with this measurement is, you know, my...

This is my Lex Luthor or something. I devoted my life to trying to kill this person. You're menial about it. I'm a little bit crazy about this, I admit. But the enemy is this survey that you're all familiar with. It comes from 1932. It's nearly a century old technology. And how it works is...

You ask employees on a scale of one to five, do you agree with this? Do you agree with this? Do you agree with this? And on and on for dozens of questions, maybe over a hundred questions. And this is how you understand what they think, right? And you can just see off the top of the bat,

This is just a crazy way of understanding ideas, right? You just, you get employees to say five, five, five, five, four, three, five, five, five. That's just not, that's not human. And the only reason we're still doing it that way is because the technology hasn't come around to, you know, allow our humanity to flourish. But these surveys have, they just don't work is the main reason not to do that. So when employees are faced with these questions, they,

They go on autopilot and they answer almost every question with almost the same answer. We did a study of 900,000 of these survey responses. The average respondent is answering about 90% of questions with the same two answers. It's just, you're not getting any information. And if what you're trying to measure is culture,

which is incredibly sophisticated, incredibly multidimensional, incredibly complex. It has all these different moving parts and it's fundamentally human. If you're trying to measure that with this robotic, really repetitive, low quality data, you just can't understand what's going on. And that's the true tragedy right now. Most companies are still doing it this way. They don't even know what their employees are saying.

you know, or they're looking for an answer or they're looking for that five. Yeah. And they're trying to, yeah, how we're trying to guide them across the board. Right. Yeah. That's, that's a great point. They're just looking for validation that the employees are going to write four to fives to everything, which they, which they do. So, yeah. So how we do it is just really simple. I mean, the technology is complicated, but the, the approach is really simple. You just,

Listen to what employees say when they write a review. You listen to what topics they mention and you understand, are they talking about the topics positively or negatively? And you just you hear that message and it's just advanced text analytics. That's all it is.

It is interesting how when you start to put two and two together, how just asking the questions, you do lose so much of what's behind that, you know, and and I think a lot of that is just it's unconscious, you know.

in the minds of the people who are answering the questions. But now being led that technology where you can get sentiment analysis and some of those other pieces that you don't get just asking questions and especially not questions that give you five choices. And so this does seem pretty groundbreaking in terms of what you're doing and how you're measuring it. But let me ask a different question, though, inside of what Dwight just asked.

Where are you getting the data from to be able to do what Dwight said? Like where, Charlie, where's the data coming? Is there a coming from Glassdoor? Because we know that a lot of people fetch, you know, complain, you know, on Glassdoor. There's more emphasis on the fetch than there is on the compliment. So if you think everyone's just going on Glassdoor to complain, what percentage of reviews do you think are negative on Glassdoor? Hmm.

Probably 60%. 95. No, not 95. Really? 95, you think? I would think it's like 65%. The actual answer is 18%. Really? And 63% of reviews are five-star and four-star reviews. Glassdoor is not a negatively biased platform. Interesting. I had always thought the opposite. Yeah.

Everyone thinks the opposite. What about Yelp? Yelp is much more polarized because it doesn't have the... What makes Glassdoor so unpolarized is their give-to-get policy. So if you want to have access to the reviews, you have to leave your own reviews, which is going to incentivize more sensible people to write reviews versus with Yelp.

Oh, I had a really great experience. I'll write a Yelp review. Oh, I had a terrible experience. I'll definitely write a Yelp review. And you're getting mostly extremely polarized reviews. But with Glassdoor, there have been studies done. It's actually one of the least polarized review platforms of any kind on the internet.

And is that the only place that you pull the data from? No, we also work with clients' internal data. So Glassdoor is how we did this huge research that I was telling you about. That's all Glassdoor data. But then to understand what's going on with the company, to understand, for instance, things like where within the company is the issue you care about most pronounced and ideas for how to fix it, then we also do internal data. But you can learn a lot just from Glassdoor.

Interesting. Yeah. Yeah, I totally would have thought the opposite. Yeah, a lot of the organizations I've been a part of, it has been the opposite. It's been where most of the reviews I've seen have been negative in nature, but I won't be mentioning the name of those companies, by the way, but to protect the innocent.

We're not in this case. Well, if there really is a company where 65% of the reviews are negative, then there's information in that. Yeah, that is true. But I guess the question I'm asking you is, you know, you're using natural language models, you're using text analysis, you're doing a lot to try and dig into the sentiment that drives what people believe about their culture, correct? Correct.

is there validation on from, from using those studies you were talking about before that are not as great or you don't, you don't really approve of the, of that, of that methodology. Do you, do you actually ever do an analysis to see what the correlations are? Yeah. So I, I,

I'm wary of getting too much into the weeds of analyzing these old studies, but basically in a 50 question, old fashioned study there for most of the questions, the employees are just answering it completely the same. And you basically can't tell any information, but there are probably going to be three or four questions that,

where the employees care about those questions so much that they break autopilot and they go out of their way to give either an extremely positive or an extremely negative answer. So for, you know, it's an inexpensive way of doing it, but for a 50-question survey, you can get maybe three or four good answers. And then, yeah, then you can compare that to the free text. And sure enough, the free text said, yeah, like, obviously I already knew that.

So the data that you're getting from inside the companies, tell me a little bit more about that. So most companies, when they do an engagement survey, they will... I don't even know why they do this, but they'll include a couple of free text questions that they then don't analyze. It's pretty funny, actually. I've seen cases where huge companies...

they don't even translate the free text data, right? So they don't, it's not even in the same language that, uh, that the analysts can read. That's, that's how unseriously they treat this. Um, but, but nevertheless, they, they do ask those questions and then, or most of the time they do. And then we, we analyze them. I, I remember in a past life, um, when I was a manager and, and even when I wasn't on the manager side, we,

We did employee satisfaction surveys and the surveys always felt like a joke because there was free text where employees could comment. But then what they would do is they would aggregate this huge cloud of information and provide feedback about your service.

managerial unit or whatever it was. And it was it never was useful because it was so overgeneralized. And, you know, and and we knew why they were doing it. They were trying to protect the employee. But it's just really detracted from from the value of doing that survey. And it was really unhelpful. Right. Exactly. Yeah.

They still, you'd be surprised how prevalent word clouds are even today. Oh my God, what a joke. But yeah, they're all over the place. It's awful because you have so much value in so much of that data. And it's such a waste of time that could be so better spent being able to actually get what people really feel. And I think with the advent of a more generative AI and text-based conversational AI, it's

we should be able to ask people more readily, Hey, Dwight, how's your day going? How do you feel about work today? Right. Not in an intrusive way, but in a, you know, how can we make your life better? You know, let's, let's ask them questions that matter to the employee that, that might be able to turn around some usable context. Sorry, that was a, that was a fetch myself. So I apologize.

Yeah, that's the hope. But it comes back to leadership prioritization. If the leadership isn't interested in doing this in the first place, it's never going to happen. Or maybe they'll ask the question, but they won't even care about the results. So it's the technology is there. In 2025, the technology is there. It's just up for people to take it. Hey, are you listening to this and thinking to yourself, man, I wish I could talk to David about this? Well, you're in luck. We have a special offer for listeners of the HR Data Labs podcast.

a free half hour call with me about any of the topics we cover on the podcast or whatever is on your mind. Go to salary.com forward slash HRDL consulting to schedule your free 30 minute call today. Let's transition then to why is AI a better way of measuring culture? We use, we talk about that term AI all the time on this program.

Why is now the time and why is this the correct methodology? Well, AI has a number of advantages over this old-fashioned Likert approach. So for starters, well, for starters, it's a century more advanced technology. But then it doesn't have this autopilot effect. So whenever employees are consciously expressing their language, even if they're just very terse, even if they just say something like,

Pay me more. That's still valuable information because they could have been talking about hundreds of different things. They chose to talk about compensation. They're consciously expressing themselves. Another reason why AI is good is because you can express messages. So say you're taking a survey. The thing you really want to say is my boss is micromanaging me and is driving me crazy with AI. The AI just says, okay, got it. His boss is micromanaging. He doesn't like that. That's the message.

Versus with this traditional survey, maybe one of the questions was about micromanagement, maybe a couple were about the boss. But it's impossible for the analysts on the other end of the survey to actually decipher that message because there are also dozens of other questions being asked.

Another huge advantage of AI is that employees are going out of their way to not only identify issues, but also to explain how issues are breaking down and even to offer actionable solutions for improving the issues. And if you can understand this information, it's much easier to actually fix the problem. So it's just a much more actionable methodology approach.

And finally, everyone hates these long engagement surveys. They take like 20 minutes and they're so clunky. With an AI survey, it can be just what you said, Dwight. It can be asking the employee, what do you like about working here? Take 30 seconds to respond. What don't you like? Take 30 seconds. It's just a lot shorter, a lot more natural. So those are some of the...

And I think this will be the best approach for a while because the only alternative I see to this is pretty dystopian, which is let's analyze employees' email, let's analyze their Slack, let's analyze the video footage and then get passive signals that way. But to me, I think that's just going to make the culture more toxic through the listening mechanism itself.

So I think this kind of consensual AI is a really happy middle ground that can be very effective. So, Charlie, what's the platform by which this happens? Is it a...

Is it an email they get that they click on a link and they go to a website and it's an avatar and they're chatting with the avatar? Is it a text-based where they're just on their phones and they AI asks it three or four questions and they're responding on the phone, hit submit and they're done? Is this a thing that constantly keeps their pulse? What's the technology and how does it manifest?

The exact survey mechanism doesn't really matter that much. You can do it on Slack. You can send a SMS. You can do it by email. You can do a more traditional survey platform. You just basically need some mechanism to ask the question, what do you like about working here? What don't you like about working here? And allow them to respond in free text. It's even possible to have...

a phone call and transcribe the phone call. So that part of it, you can do a lot of different ways. Then you want to analyze it on the AI platform. I'm more concerned about that interruption. Like AI, like HR is bothering me again. You know, I've got to take this survey or I've got to do this. Oh God, it's an HR thing. And your mind and your entire disposition changes because

AI, sorry, HR is forcing you to do something. So it already puts you behind from a feeling perspective that, you know, this is a burden to me, so I'm going to give it, I'm going to give a crappy answer or I'm going to, you know, get angry about it or my emotion changes. Wouldn't we want to do it in the most natural way for that person who's answering? And that actually could be different by generation, no? Yeah, I think there's a lot of room for improvement there. And I mean, you also have to consider, you know,

There are going to be some cultures with elevated levels of toxicity where you have to design the entire thing around confidentiality. I mean, in some cases, we'll explicitly say, look, leadership from your company is never going to analyze your individual response. This is all going to an independent team to ensure psychological safety. So, yeah, I mean, it's...

You just got to get them getting the right stuff down. Then it becomes interesting. It's part of what I think. But of course, you're right. I'm curious some of the findings that you've that you found. I mean, can you compare and contrast like findings from two different companies? And were they were they strikingly different? Were they very much the same? What what does it look like in terms of the output?

Oh boy. Yeah. You, you can, you can compare millions of different companies. And the thing is no, no two cultures are exactly the light, exactly like each culture is like a snowflake or something. It has its own distinct, uh,

You can combine any two cultures head to head and see exactly what employees speak about more favorably, less favorably. You can see how the culture is changing over time. So, for instance, when you're talking about Jeff Bezos' successor, whatever his name is, it's pretty interesting. It's amazing we don't know that, but...

Yeah, I do know it. I just can't think of it. But Amazon is unique on Glassdoor because it has so many reviews. So each year employees are writing maybe like 35,000 reviews. So you can really see at a very granular level exactly how the culture is changing. And what's interesting is Amazon historically, back when Bezos was CEO, it never had a very good culture from an employee experience perspective. Employees never really liked working there that much.

But what it did have a very good culture on was these leadership principles that are very core to Amazon's identity. And you could measure that on Glassdoor. You could say, for instance, Amazon compared to competitors, employees speak to standard deviations more favorably about customer orientation, for instance, what they call customer obsession. And historically, on most of these leadership principles, they demonstrably did a very solid job.

But what's interesting is if you look at how the culture's changed only very recently since Bezos stepped down as CEO, they're still pretty strong. But there's been a notable erosion in these core leadership principles that I would expect at least someone in the company would find interesting.

But anyway, that's an example of one of the many, many things you can do with this data. And I found it was Andy Jassy, by the way. Andy Jassy. Not the household name yet, but... Soon to be. Do you have a follow-up? Yeah, my mind is going right now, which is always dangerous. But I start to think about some of the utility...

kind of business case utilities with this and being able to take the culture, being able to take the information that you're getting from this study, number one, and then number two, replicating that. And you think about mergers, for example, like looking at that cultural aspect is probably a

One good indicator of success or failure of a merger. Yeah, according to research by all the big consulting firms who do most of this merger work, like KPMG, Deloitte, and all those, McKinsey and everyone, there's consensus that culture is one of, if not the thing most likely to derail a merger.

And it's a very interesting use case because now right now what you can do is just take the two companies, see exactly where the areas of culture fit are, exactly where the areas of culture mismatch are, and even how to improve the areas of culture mismatch. And it becomes a lot easier to manage that aspect of the transaction. But it's still in its early days.

But is it easier in a specific industry to call out what those measurements of culture are or what the characteristics of them are through using your technique? Or am I overthinking it a little? It's pretty much industry agnostic, at least how we designed the platform. So how our platform was built is we analyzed...

all this glass door, you know, millions of glass door views. And we built topics so that anytime an employee mentioned something above something like 0.05% of the time or even smaller, you can pick up on it. So it's at least how we built it. You can pretty much cover anything.

any imaginable cultural topic from any industry. And then if there is industry or organization-specific language, the AI can also pick up on that and then you can update the models to accommodate the new language.

Is there any particular cultural characteristic that rises above the others when it comes to employee sentiment? Yeah, toxic culture. Toxic culture is by far the biggest driver of employee satisfaction or dissatisfaction. Wow. What's the least that you found? The least impactful on employee satisfaction. There are a lot of topics. To compare and contrast.

Well, I mean, there are a lot of things that don't really matter. Like we measured pet friendliness that almost never is important. But in terms of more notable cultural measures that don't impact employee satisfaction, things like agility, innovation and execution are vital for strategy execution, but they almost never have a big impact on employees. Interesting. Interesting. Wow.

Well, we could talk to you about this all freaking day, actually, because Dwight and I are total geeks when it comes to being able to use the data to drive business outcomes. That's the reason why we have this podcast and the reason why we actually started it. Charlie, we're going to have to ask you back again, dude, because there is a lot we haven't uncovered yet about this. Yeah, I would love that. Thanks for having me, guys. Thank

Thank you. And Dwight, thank you. Thank you. Thanks for being with us. My head is swirling right now of all kinds of good ideas. This is great. We have to write it all down. I know, exactly. Charlie, thanks a lot. And thank you all for listening. Take care and stay safe. That was the HR Data Labs podcast. If you liked the episode, please subscribe. And if you know anyone that might like to hear it, please send it their way.

Thank you for joining us this week and stay tuned for our next episode. Stay safe. Hello and welcome to the HR tip. Okay, one more time.