OK, I’m going to get started. So hello, everyone. My name is Victoria Mussali. I’m the People Operations Manager here at Long Ridge. And today, we are very excited to have Keith join us. Keith is the CEO of SmartRank, which is a tool that helps hiring organizations stack rank and filter job applicants without using a resume.

And part of the discussion we’re going to be focusing on is ways that we can make hiring more efficient and effective by kind of looking through some of these kind of different strategies and platforms that Keith is going to kind of walk us through. I was saying that I handle, for Long Ridge, all of our kind of entry- level hiring and internship programs. So I’m excited to also be part of a member of this as well as kind of helping to facilitate. And thank you, as always, to OneGuide for also acting as our facilitators.

We have Mary and Catherine here today. So again, this should be very interactive. Feel free to go off mute if you have any questions. Feel free to turn on your camera. We want to facilitate discussion. So Keith, I bring it over to you all.

Excellent.

Yeah, thank you so much for the intro. And it’s nice to meet everybody on this call. Let me go ahead and share my screen real quick, and we will dive right in. It’s the longest title of a webinar ever. Here we go.

And just change one thing here. Great. All right.

Hopefully, everybody can see my screen. Is that everybody good on being able to see that?

OK. Like I said, longest title ever, Improving Hiring Efficiency with Highly Predictive Applicant Screening Questions. Wow, that’s a mouthful.

By way of background, I’m not going to spend a lot of time on this. But I’ve got 23 years of business experience in sales operations and leadership, most of that being in leadership.

So I’ve hired hundreds of people over the last couple of decades, which is what ultimately prompted me to start the company that I’m now co- founder of.

Been a two- time CEO, and then on the personal side, I love to travel, learn, anything related to sports. And I’ve got one wife, one dog, and two kids. So there’s me in a quick 20- second snapshot.

Let’s dive right in. I’m not going to start at the highest level about recruiting overall or the hiring process.

But let’s just say that it’s got an enormous amount of challenges happening right now. As you probably can see, if you just go to LinkedIn, you’re going to see people talking about it from the candidate side, the hiring manager side, and even the talent acquisition side.

But what we’re going to do is focus on specifically the screening part of this. Because to set the stage here, this is primarily where most of the issue arises.

And this is really the very beginning, which is not identifying exactly what is needed from the beginning is only going to lead to an enormous amount of challenges later on. So this is a real job description on the right side and a real resume on the left side. And what you’re going to see is the typical type of language that you’re going to see in these, which are going to be keywords.

Because that’s essentially how a lot of recruiting and screening works today is based upon keywords.

So in this role, it’s a full stack Ruby on Rails software developer. So obviously, Ruby on Rails is a big keyword here. So what is ultimately happening is looking at as many keywords between the job description and the resume.

That’s the primary process. The problem is that almost every single job description on the planet uses the exact same language.

Familiarity with, knowledge of, experience in, expertise with.

What does that mean?

It doesn’t mean anything unless you define it, which makes it 100% subjective for every single person that’s looking at it, the recruiters, the hiring managers even to an extent if they don’t define it, and certainly the candidates. And then when you look at a resume, it’s the same thing.

If I say that I’m extremely proficient in Excel, does anybody on this call know what that means? Anybody?

No. Why?

Because I haven’t defined exactly what that means.

What does that mean that I know how to do in Excel?

And these are usually looked at for a very short period of time. So what we’re dealing with today across the board, it doesn’t matter if you’re Amazon.

I’ve literally talked to Amazon.

I’ve talked to the biggest companies down to the smallest startups.

And these are the main tools that are used in screening, which is resumes, job description, intake meetings.

And unfortunately, this is all surface above the surface kind of root issue of the problem is that we’re only hitting the tip of the iceberg. Everything needed is below the surface. That’s where the real good stuff is.

That’s the stuff the hiring managers really want to know.

And then I’ll just kind of say this.

it all starts with identifying these qualifications. If we don’t get this first part right of identifying and screening, it’s almost like good luck with the rest of it. And I really kind of mean that because it’s gonna lead to more interviews, right? A higher interview to a higher ratio. It’s gonna lead to potentially bad hires that we all know what that feels like. So if you don’t invest your time here, you’re almost certainly going to waste an enormous amount of time on the other side of that. So how do we fix the problems?

Well, the high level, I’ll just give the highest and then we’ll get into the details. The highest level is we have to get away from the surface, have to have hiring managers defining exactly what they need through multiple choice and multi- select questions and answers. I’ll get to this a little bit later if we have time, certainly in our Q& A, to talk about why it should be multiple choice or multi- select. We need to score. This is gonna automate the process for us ultimately. We need to be able to have a delivery mechanism for this.

And there’s multiple ways that you could do this. There are cheaper ways, there’s more expensive ways, but automate the delivery of this because it needs to be fast and it needs to be easy and it needs to be highly relevant to the candidate or they’ll bail. They won’t participate. What do I mean by that? I’ll just give you a quick example. If your questions are, what is your proficiency in Ruby on Rails? What is your proficiency in Python? What is your proficiency? And they have to type all that in, you’re gonna lose candidates.

They’re just not gonna be interested in doing that. And it’s gonna be highly subjective because every answer is gonna be different. You gotta systematically have the same set of answers and for each one of the candidates. And then somehow being able to automate the stack ranking and ultimately filtering of every applicant if that’s possible. That’s ideal, right? If you’re able to do that. But here’s what we’re talking about when we talk about below the surface.

So earlier I gave an example of, what does it mean to have a Ruby on Rails software developer? And again, in the current screening process, we’re looking at keywords. Keywords is just word association, okay? It’s not telling us what they know how to do. No resume that you’ll look at usually will tell you exactly what they know how to do. So we just ask them, right? We just say, which of the following best describes your level of proficiency with Ruby on Rails?

And they may answer this question, which they may say, I can build non- trivial queries with active record model associations and scopes, and I’m comfortable with our spec and dynamic text fixture generation, and I can write HTML, CSS, and JavaScript. Now, to most people, that’s like, wow, that’s a lot of detail. But that’s exactly what the hiring manager really wants to know. In their first interview, chances are they’re trying to vet that out.

So why not vet that out at the very front end and screen your 200 applicants down to the 10 that have the guru or expert levels that you see here below? Because the reality is this may get you only zero points, because this is like bare bones Ruby on Rails knowledge. Sounds good, but this is like bare bones. Okay, great. You just came out of coding bootcamp and they focused on Ruby and you know how to do this, right? So we’re looking for something more.

And to take this a little bit further, screening for the correct qualifications, and to take this back to the original kind of screening process, right? That’s a real example of required qualifications for a job description now, right? So you see there, excellent understanding, experience with working knowledge, right? Again, all the same type of terms that we were talking about. Down below that second kind of image you see on the left there, that is examples of what experts say, these are the questions you should ask in the intake meeting.

But look at them. What would be nice to have skills? You know, e. g. experience in retail, familiarity with X programming language. It’s the same surface level stuff, right? And then down in the bottom left, that’s what ChatGBT told me that you should ask in an intake meeting to get really, really detailed understanding of what’s required for the role. Again, same thing.

Again, same thing.

That’s not it.

What we’re looking for is on the right, where you can have things like understanding, you know, if they know how to work with external libraries such as NumPy and Panda, right? So that’s the type of information we’re looking for.

Okay.

So let’s dive into some general rules and we’ll kind of go into each one of these a little bit more.

Identifying screening versus interview questions. So one thing I would just say up front is this is not meant to replace an interview. We’re talking about screening.

We’re talking about taking 400 candidates and bringing it to the top 10 or 15 that we should really spend our time with and then quickly getting back to the other 400 or 385 and letting them know we’re not interested, right? That’s the point of this.

So we don’t wanna put in interview questions which are gonna be more open- ended, behavioral and situational type of questions.

These, and we also don’t wanna have yes or no.

Yes or no almost never will help you, right? Because it’s just as bad as the keywords.

Are you proficient in Excel? Yes or no. And that got us how much further?

Not much, right? So we need to be specific in our questions and in our answers.

And one of the ways that we can do that and just a quick kind of tip here is give definitions. So use IE, use EG when we’re talking about these things.

So if you say, you know, which of the following best describes your level of proficiency in, and maybe in parentheses you say, and EG, you know, this is an example of what I’m talking about.

We need to give full spectrum answers.

So every applicant can answer every single question. I’ll get into that in a little bit. And then we really need to have a mixture of different types of questions. We don’t wanna have the exact same types of questions for every single one of the roles that we’re hiring for. And I’ll show you what I mean by that.

Okay, so let’s go through a few examples of this.

We have to define those ambiguous phrases, right?

Specific wording. Anywhere you see experience with knowledge of those types of things, we have to define.

So look, if you see on the left- hand side in that top example, general wording question, how many years of experience do you have in enterprise sales? My guess is 80% of the people on this call could probably raise their hand in one way or another and say, because why?

Because what do we mean by enterprise sales?

If I ask 10 different people what enterprise means, I’m gonna get 10 different answers.

Some are gonna base it off of revenue. Some are gonna base it off of employee size.

Some are gonna base it off of their own deal size that they sell into those.

I mean, it’s all across the board.

So what would be a specific way that we would reword that?

How many years of experience do you have directly selling SaaS, so software as a service solutions, in a full cycle?

We’re gonna define what full cycle means here. Conducting your own prospecting, discovery, live demos, objection handling, negotiating, and closing, to enterprise, meaning Fortune 1000.

We define what enterprise means to us in a B2B business to business environment, right?

The one down below, that’s for a seemingly simple thing. That’s for a person that actually does work like custodial type of a work, like a maintenance worker inside of an office building.

So if you see those things like Drain Snake, Drain Pipe, Sloan Valve, it may be like, yeah, that’s good enough.

How much more detail do you really need?

Well, if you go over on the right- hand side, you’ll see things like you have the ability to diagnose and repair Sloan Valve toilets and urinals, meaning you know how to disassemble it to the stem and diagnose automated toilet heads.

Huge difference from knowing how to press a button that flushes the toilet, right? I mean, I know that I’m kind of being a little bit funny there, but it’s not that different, right? If you leave it subjective, it will be subjective to them. And then you’re gonna get four interviews in, and you’re gonna say, how did this person get this far in?

It’s because we never define that from the beginning in specifics.

Okay, there’s almost no reasons. I say almost, because there’s a couple, right? Are you legally eligible to work in the US? Do you require a visa?

Fine, yes or no questions are fine there. But when we start talking about things like, are you proficient in Excel?

Yes or no, that’s completely subjective.

So we have to remove that. And so what we would do is use an example like you see on the right- hand side, where we’re gonna say things like, no, we’re gonna give you four examples, and we need you to pick the one that most closely matches you, meaning you know how to do VLOOKUPs, concatenate, special data validations, conditional formatting, things like that.

So basically, if you’re struggling with this, just try to reverse engineer the question to be more specific.

So asking it like, are you, or taking it from something like, are you, and changing it to which of the following, and then listing those out. Ambiguous language. There is so much ambiguous language in the hiring process. I’ve already mentioned this a few different times, but this is probably the most important. So I’ll just kind of keep mentioning it here. Try to avoid one or two- word answers. So in the example up there in JavaScript, which is the level of proficiency, beginner, intermediate, expert?

No, we need to define what those mean to us.

And by the way, like we’ll work with hiring managers that both hire for software developers, and they have differing opinions on what proficiency means.

Fine, that’s great.

And hopefully if you have data, you’ll be able to go back and show them which one is right and which one’s wrong if you have the data. But the point is that you have to be able to get into that specificity, and

Full- spectrum answers. Everybody that takes this need, that answers these questions, has got to be able to answer every single question. So this is one of the mistakes we sometimes see, is you’ll see like in that example up there, how many years of experience you have performing live software demonstrations in front of customers? One to two, two to four, four to seven. Okay, seems fine, right? What if you have less than one year?

Which one would you answer? What if you have none? What if you have zero experience? That would be really good to know, right? So which one would you answer?

What if you have 12 years?

Which one would you answer?

I guess four to seven, that’s the closest one, but it would be nice to know if you have more than seven. So again, just slightly changing a few of those things. On the right- hand side, you’ll see I have a none answer, less than one year, one to two, three to five, six to 10, and 11 plus.

One other small thing there.

If you have two years on the left- hand side, the ABC options, which one would you choose? Would you select A or would you select B? You’re probably gonna select B because that’s two to four and that makes you look a little bit better, but don’t leave that up to them.

They have one to two and two to four. Two is in both of those answer options. So you wanna remove that.

Okay, different question types.

I would strongly urge that, you know, you’re gonna probably wanna have anywhere between eight to 12 of these questions.

I would put some of our technical companies, like in ML or AI, they kinda, or just in general, like Python, any of these technical coding types of roles, a lot of times once they have ability to do this, they wanna push it and they’ll go up to maybe 15 maximum. But what you wanna have is a whole bunch of different types of questions. So years of experience is fine, but you don’t wanna have 10 questions of how many years of experience do you have in Bootstrap? How many years of experience do you have in Python?

How many years of experience do you have programming?

Like that, we all know that we probably all worked with people that have 10 years of experience in something and they’re not very good at it. Or people that have two years and they’re pretty darn good at it. So that doesn’t, that does not automatically equate to you’re going to be good if you have more time in a field. It just doesn’t.

So great, if you want that as a data point, have it, but I would strongly urge that you have other questions like if certifications, education are important, put that in there.

If it is knowing if they know how to use specific tools, if it’s technical knowledge, skills- based type of questions, a couple of those you see on the right- hand side are actually skills- based screening questions. So let’s go into what these types of questions can be. There’s a lot of different types.

I’ve just kind of narrowed this down to these eight real quickly.

So education, the first thing I would ask is, is education really critical for the role? In some roles it is. So for like mechanical, chemical engineering, those types of electrical engineering, a lot of the engineering type roles, it’s very important. For sales roles, not so much.

So do you really need to have an education question in there? Is it really that important?

If the person has crushed SaaS sales in their career for the first 10 years, for the entire 10 years of their career, do you really care if they have a bachelor’s or not?

Maybe you do, maybe you don’t. That’s not up for me to decide, but I would really strongly urge you to think about that. Sorry, here’s the education one I’m diving into.

You can ask certain things like, what is your highest level? That’s fine. Or you can get into specifics like the question below.

Which field of study do you currently hold a bachelor’s degree in?

Electrical, mechanical, computer. Those are the top three we’re really looking for.

Maybe you have a bachelor’s degree in something else. That’s okay, I still wanna know that.

That’s a fourth option.

Or I don’t have a bachelor’s degree. Great, I know that as well.

So there’s a lot of different ways that you can kind of phrase this, but let’s jump to the next one. Licensure and certification. So here’s an example of required versus preferred type of certifications. I only say that because job descriptions can be bad in so many ways. And one of the ways is that they are so confusing. They’ll say things like required, and then the bullets, that literally the next bullet underneath would be like, you know, CPA is preferred.

And that’s literally underneath the subheading.

is that I would do a select all where they can select multiple, right?

They can choose, maybe they have all three of these, a CPA, CMA, and a CFA, that’s great. So are there really any knockouts that need to occur here? Maybe, maybe there are, probably not. And then letting the hiring manager know what is going to happen if there is a knockout. And what that means is if you do put a knockout in here, then that means that it doesn’t matter how they answer any of the other questions, they’re knocked out, they’re out. So that’s a very serious type of a thing.

And you don’t need to knock people out. This process for highly specific screening questions should not just be to knock people out.

It should also be to knock people in that maybe don’t have a good resume, but they’re absolutely good at what they do, right?

Because there’s a lot of people like that.

I would even argue that some of the best people that I’ve worked with in my career have terrible resumes because they have high standards for what they think are experts and amazing at certain things. And they downplay what they know, which is better than 90% of the other people in their fields.

Okay, so job requirements. This is critical.

Oh, sorry, were you going to say something?

Oh yeah, Keith, we just, someone messaged me a question.

Oh yeah.

So I see that you have a point system on the right- hand side. Do the applicants see the points when they’re filling out the survey?

No, no, I would strongly urge you to not make sure that they don’t think that there are any scores associated with it. I mean, they’re not dumb. They know that by answering these questions in the way that this is structured, that it’s probably not going to help them if they say zero, I’m not willing to travel. I mean, they know that, but I think what’s really helpful is that you just kind of don’t confuse them or get them distracted by the fact that there’s a scoring mechanism. It’s not relevant.

What’s relevant is I’m going to ask you 12 questions, just like I would if we were sitting here on a phone screen or an interview, and I want you to answer those. That’s it. It’s pretty straightforward. Okay, so job requirements. This is critical.

These can be really, really important.

We have clients that will look at their data and we’ll see people that are unbelievably, like the most qualified people you could find for that role, but they’re not willing to do some of the different types of job requirements.

For example, they’re not willing to travel.

Well, look, they can be the best person in the world at that job if they are required to travel, like one of our clients, one of their roles is required to travel 75% of the time. If they’re not willing to travel, it’s not going to work out.

They already know that because they’ve had a lot of turnover for that particular reason. So being able to identify that and just get people out of the process earlier is so much better. And comp is another one. I bet every person on this call has dealt with the situation where you get all the way to the end with the candidate, and now they’re sitting here and you’re negotiating, and you’re like, I thought we told you what the comp, and now you’re telling me you need something that’s way higher.

So getting this out and early is a really good thing to kind of have as a baseline there, as far as like, what is your minimum salary expectation? And just in case anybody’s wondering, you could absolutely ask that. You’re not allowed to ask what they made at the last company in a lot of states, but you can ask what their expectation is that they expect to make for this particular role.

And then years of experience is, I think, an easy question. Again, it’s a good data point to have.

Like I said earlier, it’s not the end all be all.

What I would strongly, strongly suggest is that the wording of the question, though, is highly specific. The answer options aren’t going to be specific.

They’re going to be one to two years, three to four years, five to six, whatever. But the question itself is where you should spend the time on the years in experience.

If I just say, how many people have sales experience in this webinar?

I bet many people could raise their hands.

But if I say, how many years of experience do you have selling SaaS solutions to Fortune 100 financial services firms like JP Morgan, Wells Fargo, and Bank of America? That is a completely different question.

And now you’re putting them on the spot to say, OK, well, when you phrase it like that, I have zero. Well, good, that’s what we need to know, because that’s exactly who you’re going to be selling to.

OK, this is my favorite of all of the question types, and that’s proficiency.

Because like we said, you could have 10 years and still not be very good at the job. You could have a certification, but you still don’t really know 100% about what you’re doing.

But the proficiency really seems to separate the pack. This is where we see in the data, when you have multiple proficiency questions, this really starts to really quickly separate folks.

And this is where we’re getting into actually asking you what you know how to do within these different types of solutions.

Again, these are great for technical type of roles, but they’re great for everything.

I mean, accounting roles, right? Oh, do you know how to use finance? Must know how to be familiar with using financial statements.

Wow, that’s broad. How about, what is your level of expertise as it relates to revenue recognition for a SaaS company with income statements?

And then we have four different levels of what you know how to do and how you know how to analyze an income statement. That’s a lot better.

And then skills assessment. So these are great. A lot of times, again, going back to like SaaS companies, a lot of times really like to use skills assessment. This is not meant to replace a coding assessment. I wanna delineate between the two, okay? Coding assessments are great. This is complimenting those. But what you’re not gonna do is give a coding assessment to 300 applicants. It’s too expensive, takes too much time. They’re not gonna do it, most likely. And it’s just, it’s way too much effort to have to go through and analyze all that.

What you wanna do though is potentially have some skills assessment that basically just ask them, if they’re applying for this role, they should know the answer to this question, right? It’s a basic question. They should know what the answer is. And if they get the right answer, then that should tell you, okay, yeah, they at least kind of know some of that. Okay, so skills assessments are great. And then situational. These are more to get an idea of what, how that person thinks.

So it’s a little more subjective for sure, and starts to blend a little bit into kind of interview. But there is some really good insights you can glean if you do ask questions like this. So for example, let’s look at the top one up there. You have a client that says they’re gonna end their account with you and go with a competitor. What would be your initial reaction? The key here is to give four answers that seemingly all seem like they could be a good answer.

Like one of these answers in here is tell them to please hold, then go get your manager to help you resolve their issues and save the client.

That sounds reasonable, right? Like save the client, go get your manager, right?

Let’s get a team together involved here. The problem is, is that that person may be more likely to come and get their manager for every single issue that arises, right? If they can’t immediately ask this person, okay, hold on, before we do anything, what’s going on? Like, just tell me what the issue is. That’s the first line defense that they should be having. So if they’re gonna run and get their manager every time, eh, that may not be a good thing. So good responses for all of the answers.

I only say that because you can see some situational questions where they’re like, what’s the right answer? And it’s like, tell the client you don’t want their business anyway, blah, blah, blah. Like, of course they know that’s a terrible answer. Nobody’s gonna answer that. Okay, good versus bad outcomes. So very quickly here, I’ll just kind of run through. So if you have a mechanism for scoring and stack ranking, a good outcome would look, or a bad outcome would look like this. You have a lot of the candidates that have the same scores.

That’s the best way I can say that. So if you have like 100 applicants and a third of them have 100% and a third of them have 50% and a third of them have 0%, you probably didn’t do a great job of, you know, really getting a bunch of different types of questions. Maybe you have too few questions. Maybe they’re not specific enough. Maybe they’re all yes or no, but you really want to make sure that you avoid that. And that’s your first indicator that you probably don’t have the right questions in there. The specificity is what is so key to this.

I know I keep harping on this, but this is what is the most important about creating these types of questions. So words that say things like experience and expertise, they don’t mean anything. In fact, they mean something completely different to every person that reads them. People get upset at candidates all the time because they apply to a role, they get into the interview, and then they’re upset because they’re like, what was that person thinking when they came into this interview, thinking that we’d be interested in them based on their skill level?

It’s because everyone’s playing off of a different set of rules.

It was never defined from the beginning. So trying to get very specific, like you see on the right- hand side there, specificity is very, very key here. So what does a good outcome look like? Well, it’s the reverse of the bad outcome. You have a real true stack ranking, ideally. You have scores that are representative of many different types of outcomes and answers to those questions. So you have 86, 83, 79, 71, 70.

That’s great.

That shows you you’ve got a pretty good.

a pretty good spread of how the applicants are answering that.

So what are causes for good outcomes here?

Well, you have about 8 to 12, as I mentioned, pushing 15 max. I wouldn’t do any more than 15. I probably wouldn’t do anything less than 8.

Very specific, different types of questions like we talked about. Absolutely having some level of proficiency questions. And the scoring is evenly distributed.

Now, what do I mean by that?

I mean the weighting of the scoring. If you have one question that has a score that is so high that it outweighs the other 11 questions, that’s not a good thing.

Because seemingly, they could end up at the top of that rank, SAC rank, and that was only based off of one question. So I would really try to keep your weighting in those questions fairly similar.

Here’s an example of what a good outcome may look like.

So this is a real example. 136 applicants, 50 of them, or 37% were knocked out. 15% had a zero match score, meaning they were probably incomplete.

Less than 50% score was 17%.

Effectively, what that means is out of the 136, 94, or almost 70%, you don’t need to really review.

That’s great. That’s awesome.

Who you’re primarily going to review is that top 7% or the top 10 that you’re looking at between the scores of 76 and 93%.

And maybe a few of the ones right below that in the 50 to 72%. In this particular case, they hired the fourth person. And here’s a good example of the difference between looking at…

This is a real example where this particular client had 43% knocked out, 25 incomplete, and then 8% top ranked. This was just one of the resumes that you see in there.

And it’s pretty good, right?

This person’s got a Master of Science in computer science. And engineering from UT. It’s pretty good.

They have all the languages that you would typically see. They have four years of full stack development. They have SQL. These are core qualifications, okay?

That means that they must be pretty good because they’re calling them core, whatever that means.

SQL, Node. js, great.

And they have profound knowledge in managing databases. If anybody on this call can tell me what this person meant by profound knowledge in managing databases, I’ll buy you lunch. The thing is, is that this person ended up being… I mean, pretty good, right? Not bad, right?

This person was ranked dead last. They were actually tied for dead last.

174 out of 175 applicants.

Because when you ask them highly specific questions, you put them in a place where they can either lie to you or they can just tell you what’s going on. And most candidates are not liars.

There’s a myth out there I want to dispel really quickly that candidates are liars. Well, let me tell you something really quickly.

There is no system on the planet that’s going to fix liars.

If they’re liars, they’re going to lie on their resume. They’re going to lie in an interview. They’re going to lie on the job. You’re not going to be able to fix that unless you can somehow suss that out in the interview.

The challenge is being able to put them in a position where they have to give you a specific answer. And we’ve gone off of these broad level questions for so long that when you ask a broad question, you’re going to get a broad answer back.

So when you ask something very specific, you put them in a position where they can either lie or tell the truth.

And what we see and what our data shows us is that most of the people out there are going to tell you the truth if you ask the right question.

Okay, really quickly, I know we’re a couple minutes over here so on kind of the time and we’re going to get into some more Q& A. I’ll just mention this very quickly. LLMs can be really good tools for creating questions and answers. So if you’re sitting there right now thinking, how on earth am I going to create a Python question with that level of specificity? I would never be able to do that. First and foremost, hiring managers need to be involved in the process, but that doesn’t mean they need to create it from scratch.

Here’s an example question created from ChatGPT related to income statements, as I mentioned before.

I didn’t touch any of this. I didn’t create a single word of this. Every single thing you see here was created through ChatGPT 4. But it’s all about the prompt. You have to prompt it in the right ways. So everything that you’ll see there in that prompt, the only thing we may change is the blue text that you see there, like where it says Zendesk. Everything else is the same type of prompt, which is highly specific.

I’m telling the LLM exactly what I want and then they’ll produce it.

So there’s some practice that goes in here. But a couple of tips, be very specific.

Back to that again. Tell it how many answer options you want.

Tell it what person you want it to write this in. First person, third person, things like that, which I would suggest first person.

How you want them to organize it.

So a lot of times ChatGPT will give bullets.

I would suggest telling it you want it in line so that they’re in paragraphs instead of that.

So it’ll just do the work for you.

You just got to prompt it in the right way.

And then you can build off of previous questions. You know, I’m sure you’ve all probably played around with that. A couple of cautionary tales I’ll just mention very quickly. One, and this has been updated by the way, but, and specifically with chatGBT4, but if you’re using chatGBT3, I think it may, but they may have done an update to this, but it could have outdated information as a point.

Two, if you’re not familiar with hallucinations, this is a real thing that exists in AI. This is an example of which are the best applicant tracking systems in the market. I can tell you at least five of these are the most despised applicant tracking systems in the market. Now, why do they show up as the top? Because they have a lot of content around them on the internet. So that’s why they show up as some of the top 10. That does not mean they’re the best. They can provide long- winded answers.

My suggestion is just take the answers and just tell it, take everything you just gave me and shorten it. And then sometimes it can produce things that’ll give you the same problems we already talked about. Like if you look in this example, it says knowledge of fundamental concepts, familiarity with, basic understanding of, that’s not good. You want to get that stripped out and explain what that means through specific examples of what you want it to do, tasks, activities, things like that.

And the last thing I’ll just mention here is that chatGBT, what is it not? It’s not a replacement for the hiring manager. The hiring manager is the subject matter expert, should be the subject matter expert, and should always be consulted in creating, editing, and ultimately approving the questions that we really have that are going to be going to the candidate. If they’re not involved, it’s not a good thing. I won’t go down a long diatribe explaining all the reasons why, unless you really want to know.

But the short answer is, if they’re not involved, then they can always point the finger and say, well, you didn’t give me what I’m looking for. And that’s not what we want to have happen here. All right, I’m going to stop there. Let’s see what questions we have. A lot of content.

Quick question from me. Does this, in your opinion, replace the phone screen, or what sort of remains for that phone screen or first phone call with a candidate prior to the interview process?

I think, yeah, it’s a good question. And I think, you’re not going to like this initial answer, I’m sorry, it depends, obviously. But my question would be, what are you asking in the phone screen? And is there a way that you could automate that through these questions? So you’re like, maybe you’re saying, I don’t know, I’m just going to take a completely random one here. Let’s say it’s, you must be in the office three days a week. Okay, why can’t we just ask that in a question?

Like, I don’t want to wait all the way to a phone screen to find out that you’re not able to come into the office three days a week, if that’s a requirement. So why would we not put that up there? If the phone screen questions are more around, I guess, like open- ended type questions where you want to understand more about certain situations, I think that’s totally fine. But again, what you’re doing is you’re taking a list of 200 and you’re doing phone screens now with the 10 that you already know are pre- qualified.

We’ve texted you.

Do we have another question? Yeah, so Katie, I don’t know if that helps to kind of answer that, but there could be some questions you may want to do, but at least this would limit that down. And by the way, I just want to mention this really quickly. Of those 10, that doesn’t mean all 10 of those people are going to be qualified and be good to hire, right? That’s not what I’m saying by screening questions. We’re getting the qualifications piece and the job requirements piece and all those out of the way.

So that we can then figure out which one’s a good culture fit, which one’s a good team fit. I’m not suggesting that you’re going to find all that out through screening questions. That’s what interviews are for, right?

Yeah, it makes perfect sense. Thank you. Keith, I have a question.

Yeah.

So I don’t do much of the pre- screening. You know, I’m still kind of new to the hiring process on the management side, but we have a specific role that is very hard to fill in terms of problem solving skills. And obviously, the screening part of the process, but is it…

is it important not to get too into the weeds of the screening stuff where you might not get any applicants, if you’re asking the wrong questions or maybe too detailed questions. I guess that’s kind of my question.

So I just want to repeat it back, make sure I understand. Are you wondering if by giving them, let’s say eight to 12 really detailed specific questions, does that push away some applicants potentially from applying?

Possibly, yes, but also trying to find the best applicants to actually interview and get to more specifics, but to maybe avoid getting too detailed or too into the weeds too early on, I guess.

It’s a really good question, Dustin, I like it.

All right, so let me give this audience a quick statistic.

Does anybody on here have a guess as to what, so SHRM is kind of like HR, like standard kind of body for a lot of different types of knowledge and things. According to SHRM, does anybody know what the online job application completion rate is? Any guesses? This is how many people start an online job application and then hit submit and complete it.

40%, I’ll just guess.

Yeah, that’s a good guess.

It’s 8%, 8% of people start an online job application and then actually complete it.

The reality is that this process, the hiring process and candidates more than ever, absolutely despise the hiring process as it stands today.

Because what is, put yourself in their shoes, right? They come in, they put in a bunch of information, they upload a resume, sometimes they have to go through and retype all that stuff again, which they really don’t like to do that. And then they go apply to another job. And let’s say that job even uses the exact same ATS at a different company, what do they end up doing? Same exact thing. All they’re doing is taking their resume, they’re throwing it over this ATS fence and they’re praying that they’re gonna get a call.

But the reality is that most ATSs out there are going to organize those applicants chronologically. Meaning that if there’s 200 people and they applied 199, they could be the best possible person for that role. And they’re probably not even gonna get a call back because they’re not even gonna get looked at.

So they’re very, very, very frustrated.

And then they get these automated emails, like four hours after saying, oh, thanks, but no thanks, we’re not interested. And they’re like, are you kidding me? Did you really review all my information already in four hours? I can see on LinkedIn that 1200 people have applied to this, right? So I’ll just share some data to back to kind of contrast that 8%. So when we first set out, we wondered the same thing you did, Dustin, is like, are they gonna answer this?

What are their thoughts gonna be? In our applicant completion rate, it is 74%. It’s almost 10 times what the average is out there. Now, why is that? It’s because at the end of the day, these people wanna get a job.

And what this allows for them to do is to A, understand more about what the job really is asking for, right? So we’ve gotten notes that they’ve sent to us afterwards saying, I know I’m not qualified, I know I’m not gonna get an interview, but I really appreciated the fact that I finally knew for the first time what this job was actually asking for.

And the second thing is, this is a way for them to showcase what they actually know how to do instead of just how good does my resume look?

I hope I have the right keywords on there. I hope I have the right companies on there. And that’s the big concern they have.

So this helps them. And there’s a lot of data that backs up that candidates love assessments. And it sounds crazy, but on the front end, it has to be fast, it has to be easy, and it has to be highly relevant to that particular role.

If you’re asking them to be like, if you were a fruit, which one would you be? An apple, an orange, or a banana?

Like some sort of behavioral or like personality assessment upfront in the screening process?

Uh- uh, they don’t like that. There’s a lot of data that backs that up.

So I don’t know if that helps at all, it doesn’t.

One other thing I’ll mention is, you mentioned that it’s problem solving.

Some of our clients will actually create problem, like creative problems that they ask the applicants to fill out.

And you’re sitting there and you’re like, well, they’ll just go to chat GPT, or they’ll go somewhere and just throw that in there and get the right answer, right? Nope, there’s a lot of people that are not going to do that. A, they’re not gonna spend the time to do that, and B, they’re just gonna go in there and they’re gonna answer it with what they think, and that may knock out like 50%.

Gotcha, yeah, that’s kind of what I was asking mostly, because could you get those kind of questions in there so that they could even just prove a basic problem solving type thing? Because that’s been our biggest problem right now in terms of one.

Yeah, you bet.

Do you have any other questions? Great. One that someone had messaged me is, Keith, earlier you were talking about the use of AI in the screening process. And we like to use these roundtables as great learning opportunities from each other as well. So does anyone here have experience using AI successfully in the screening process or even at large the hiring process and want to share maybe some specific tools or tips that you found helpful with the group?

I can add something here.

Yeah, yeah, go for it, Keith.

And this is kind of related to a bigger subject, but it’s really important. If there’s anybody on this call that deals with hiring managers right now, and you’re sitting there right now and you’re saying, there’s no way I’m going to be able to get hiring managers to create these questions, right? They won’t even get a job description back to me.

First of all, that’s a whole other topic, right?

The job descriptions are not their favorite thing to do. It’s kind of a waste of time for them, right? But I also would not suggest that you show up and say, hey, I want you to create 10 detailed questions around the role that you’re looking for. Go and please get that done and get back to me. That’s probably not going to go very well. Because hiring managers have their day job and they’re super busy. And they don’t even know where to start.

And I’ll say this because every hiring manager, me included, for many, many years, we’ve been trained that that surface level information is good enough.

It’s OK. So if you ask them to create 10 questions, guess what they’re going to do? They’re going to come, even if they do the exercise, they’re going to come back with 10 questions that are going to be yes or no, single word answers.

Everything I just described not to do is probably how those questions are going to come back because they’ve never been trained differently and they don’t really know exactly what you’re asking them to do.

This is why AI can be a really powerful tool.

So here’s the very specific things that you can do. You can go in and you can take the job description. Fine. Take the job description as your starting point.

Look at what is said as far as what they’re looking for. Chat cheaply. In 10 minutes, you can create 10 amazing questions.

And then you can go to the hiring manager for your normal intake meeting that you would have and say, here’s 10 questions and answer options for what you already have on the job description. Can you please go ahead and just take a quick look and edit what you want to edit and approve? And just let me know if this is where you want to go with it. Once you show them what good looks like, then they know. Oh, I see the level of detail you’re trying to get into here.

OK. Oh, well, that’s awesome.

Are we really going to know this information about these folks before they even come in for an interview?

Before you even send them to me? Great.

I promise you, hiring managers will love it. They’ll be on board knowing that people already have these qualifications before they get to them. But you’re helping get them to the 10- yard line. And they just need to get it across the goal line. Sorry for the football analogy. But it’s the best one I can think of. You got to get them all the way there, show them what good looks like, and then just have them tweak what they need to do.

And AI can be a really, really good tool to use as a part of that. This is the greatest part. You don’t need to be a subject matter expert. There is a tool out there called ChatGBT.

It is a subject matter expert in a lot of these areas.

But you will look so cool and so smart when you show up to that intake meeting.

And they’re going to be like, how on earth did you know this level of expertise in Node. js? I didn’t realize you knew that. And you’re like, yeah, no big deal. It just doesn’t matter. What do you think about the questions? And they’re like, wow, I’m pretty impressed.

One of the things that I am curious about anecdotally on the Long Ridge side, we’ve always been a lean shop. And as we have grown and scaled, figuring out how do we create infrastructure for our hiring and building that out has been part of what I have been working on over the last number of years. And again, this is more related, in my case, more to entry- level hiring. But what I’m curious, do you keep in the group at large, just generally, because right now we don’t have any pre- screening questions, which means we get a huge influx.

And so trying to understand what’s the best software or preferred software as a company is scaling to be able to integrate some of these questions into if you have recommendations or would love to hear from the group at large what people recommend as they themselves have scaled their teams.

I have the same question as you, Victoria. I feel like we use Greenhouse, so we’re able to have pre- screened questions. We’re able to tie them to automatic triggers. But it doesn’t allow us to wait questions. And it also still requires manual review of people’s inputs. There’s also no easy way to export that data. You’d have to manually search for it. So when you get thousands of applicants, it can be really, really challenging to look at 8 to 12 questions and say, OK, here’s how I’m waiting it. And it sort of still takes a lot of time.

So I’m curious as well about software or anything anyone’s done to make that more automated.

Yeah, I can talk to this. This is what I live and breathe for the last four years of my life very intently. The reason we started our company was to solve this very specific problem because almost every ATS out there today fundamentally acts in the same way. And that is, unfortunately, like what you just described is spot on, right? Like chronological list of applicants going through and kind of opening up each one of those.

I did put some additional slides together here just really quickly that are kind of like, this is a very cheap and somewhat the UI itself is easy but the process is not easy. But you can use Google Forms or SurveyMonkey. But the problems here are that it’s time intensive. Creating questions and managing those questions for each hiring manager per each role is a big challenge. The applicant UX is a big consideration here. So how is the applicant, how is the flow going to work for the applicant? Is it going to be smooth?

Or is it going to be like, here’s the link and then click on this. And then once you’re done with this, back into our ATS, like that’s not going to work. It has to be seamless. It has to be totally smooth and seamless. And then the data integration and management piece is the other big piece. And then hiring manager piece of it. Most of the folks on this call probably have something in these right now, right? You either have an HRAS that has an ATS component or you have your own individual like Greenhouse that was just mentioned.

Not to go through all these, but it’s nice. It’s all in one, the HRIS pros. Costs can be lower in some cases than just having a point solution in addition. But it’s kind of a jack of all trades, master of none. Meaning they’re not known for having good ATSs. That’s just how it generally works. And they definitely won’t solve the biggest problems. ATS, they’re better than the HRS ATS modules. I would say hands down, there’s no question about that. But they haven’t changed much in 25 years is the other big problem.

And again, they really won’t solve the core root problems like we’ve just been talking about on this call. And I think Katie was just kind of mentioning. So here’s an example, right? A lot of times they’re yes or no, or fill in the blank. That’s not good. That’s gonna put all the work on the recruiter and the hiring manager. You’re adding time to your process. You’re not saving time. If somebody has to subjectively go through and review all those. So no scoring, no ranking. Well, what’s the point, right? I mean, we need to be able to rank.

And then the applicant UX. If it’s more than 15 minutes, by the way, just as a side note, you’re gonna lose more than 50% of your applicants. So you gotta go through this yourself and make sure that that process is super smooth. And then there’s just kind of some other issues that I’ll go through. But some things to keep in mind. One, organized hiring managers and roles has to be an easy way to manage the questions. This is not something that can be overlooked. The hiring managers need to be able to have access to be able to see that.

And then we need an easy way to bring those questions into every rec. If you have to go and manually pull in questions or create them for each rec.

And then we

you’re going to give up, and I would too, because that just takes too long to do that. It needs to be something where you can just pull those in very quickly. The other thing I would say is creating good questions, not just questions, but good questions. So being able to pull from a library or something like that that you have, or generating with AI. AI, like I keep saying, creates some really cool types of questions.

And then again, being able to have those analytics that I think Katie was kind of talking about earlier, like being able to know how this breakdown is occurring, and then being able to see data down to the question level. I’m telling you right now, this completely changes the game as far as having data like this, because it takes every conversation you’ve ever had with a hiring manager that’s anecdotal and makes it a data- driven conversation, which fundamentally transforms the way that conversation is going to go completely.

Because now it’s not them saying, well, are you really doing this? Are you really comfortable having a conversation? Did you really ask 200 applicants what they’re willing to do?

No, no, no.

This is what the data shows us. 28% are willing to take your salary. This isn’t me. This is what 127 applicants independently told us they want to do this job. And if you want to see those, let’s take a look at those. None of them are qualified to do the job that you want them to do. The highest percent match we had was a 20%. Is that really what you want? Here’s your choices. Lower your qualifications, up your salary, or sit back and wait. Those are your three options. Which one do you want to do? That’s really what it boils down to.

Shortens the conversation. So I don’t know if that helps at all, but I just wanted to give a few things to keep in mind as you’re kind of looking at some of those. I know we’re getting close on time. Sorry.

Oh, I’m so sorry. Would this tool sit on top of an ATS? Or is it in place of?

No, both. I mean, I think that, yes, both. There’s a company that I think you mentioned earlier that is an ATS provider. They’re looking at our technology. That just shows you the extent of how big of a challenge this really is. Even ATS companies that sell ATSs are struggling with the challenges that we were talking about on this call. So yeah, I think. And to your point, I think you mentioned, Katie, just one last thing. This data analysis piece, you cannot underscore how important this is.

Like if you take three hires, everyone wants quality of hire. That’s a post- hire metric, right? It’s a post- hire metric. So if you want to figure out what should we be looking for at the very front end, the only way to do that is to say, OK, well, let’s take our three people. And then tie it back to data that they’ve answered. So OK, question number three, not a correlation. Question number four, kind of a correlation. Question number six, absolute correlation. If they answer that way to answer to question number six, they are going to be a good.

That has to stay. That’s a critical piece that we have to look at. And let’s bail on question number three. It doesn’t really matter. It doesn’t have any correlation here. But this is the type of stuff that allows you to really identify what are the absolute best things that we need to focus on from the very, very, very beginning of the process, right? All right, sorry. Just wanted to go through that since you had some questions about it.

That’s super helpful. Thank you. Does anyone else have any other questions? And if not, I am just going to say I know that Mary had sent out on a different Zoom screen a little survey. So that is helpful for us at Long Ridge as we continue doing these roundtables to see what topics people are interested in for future events. Yeah, it helps us to sort of make these better month over month, year over year. And again, if anyone has anything, I don’t want to cut it too early. So I’m going to give like 30 seconds. OK, great.

So I just want to give a huge thank you to Keith and also for sort of a smart match, smart tools application, which is super helpful and useful just to sort of think about as we are all going through these kind of hiring processes. And thank you to all of you for being here today and tuning in. If anyone has any questions, Keith, what’s the best way to kind of get in touch with you?

Yeah, just my first name, Keith, at smartrank. ai is the best way. Or you can go to our website, which is just www. smartrank. ai.

And everyone will also get an email afterwards with a way to contact Keith, as well as the slides from today, the video, and some recap materials.