Don't Overuse "Tell Me About a Time..." PM Interview Questions
Behavioral interview questions are problematic. Instead, ask questions about actual PM challenges at your company.
TL;DR - A trend in Product Manager interviews is “Tell me about a time when…” questions that test a candidate’s soft skills like self-awareness, collaborative ability, and EQ. Use these questions sparingly—or not at all!—because they’re easy to game, are oversensitive to interview prep, and can lead to hiring good talkers instead of good do-ers. You’ll get better results from interviews that are future-looking and which focus on specific problems your team is having or has had.
After interviewing a few hundred Product Manager candidates over the years, I’ve come to believe that interviews should approximate actual work tasks and conditions. The more you test candidates’ ability to do everyday work, the more you’ll hire people who can succeed at the everyday job. For example, I don’t use “puzzle” questions like “How many ping-pong balls can fit into a 747?” or “Convince me to buy an egg” because they’re too abstract and unrealistic compared to actual PM work.
But pretty soon I’m going to start the long process of finding a startup to join next. For the first time in many years, I’ll be answering PM interview questions as opposed to asking them. So I’ve had to bone up on 2021 fashions for PM interviews. From my perspective as an interviewer, some of these questions look more useful than others. But one type seems really dubious: “behavioral” questions like the ones below that take the form “Tell me about a time when…”
Tell me about a time when you disagreed with a decision of your manager.
Tell me about a time when your team couldn’t meet a deadline.
Tell me about a time when someone on your team was under-performing.
Tell me about a time when you used metrics or data to improve a product.
Tell me about a time when a customer asked for a feature you thought was a bad idea.
These questions *seem* like they should be reflective of actual PM work because they’re talking about actual PM work. But in my experience, talking about a thing and doing a thing require different skills—it’s easy to end up with a good storyteller who can’t do the job well.
I’m not the only one concerned about these questions. According to Glassdoor, these questions are biased against diverse candidates. According to Harvard Business Review, “Be Wary of Historical Questions”. According to Wharton Business School professor Adam Grant:
They're too easy to game--you end up hiring the candidate who's the best talker, not the best contributor.… They're not tailored to your organization or the job--they're stuck in what applicants have encountered in the past, not what they're going to do for you in the future.
Google search users also have an opinion:
Twitter apparently agrees. In a fun coincidence, the 11th most liked job-interview-related tweet of all time was just posted yesterday.
Here’s what to expect in the rest of this post:
Alternative questions to validate soft skills without “Tell me about a time when…”
Details about my specific concerns with storytelling questions
Ideas for going beyond “questions” to also include non-question activities that better approximate real PM work.
How the neuroscience of human memory intersects with interview questions.
Alternative questions with more context
There are many other ways to test the soft skills of PM candidates besides “Tell me about a time…” questions! My preference: ask about real but anonymized interpersonal situations that the team is currently experiencing (or has experienced but are still relevant) and ask the candidate how they’d help.
Providing context gives more flexibility to the candidate to answer in the most intuitive way for them. Folks with good episodic memory can share anecdotes. Folks like me who like discovery and brainstorming can do that. Folks who focus on process and structure can talk about how they’ve engineered ways to prevent problems in the first place. And so on.
Another reason that real situations are helpful: when the candidate asks clarifying questions you won’t have to invent facts on the fly.
Below are example questions that I might ask a SaaS VP Product candidate. They’re adapted from my own experience, with details changed to protect the innocent… or guilty! ;-)
IMPORTANT: Don’t use my questions! They’re from *my* history, and are relevant to SaaS VP Product positions that I specialize in. For your interviews, use questions based on challenges at *your* company and roles you’re recruiting for!
Alternatives to: “Tell me about a time when you disagreed with a decision of your manager.”
“You’ve just finished a refresh of the product roadmap. It’s been approved by the exec team and been vetted with a few trusted customers. You’re planning to present it to the whole company at an all-hands meeting tomorrow. But your manager, who just got back from customer road trip, emails you asking to remove the top two features on the roadmap and replace them with other features you haven’t researched nor costed. What do you do?”
“Your industry’s annual conference is coming up in a month. Your manager is pushing you to release a big new feature in time for the conference. You’re pretty confident that the feature won’t be ready in time, but when you told him your concerns, he said ‘just ship it anyway, we can clean up afterwards’ and left no wiggle room for discussion. When you DM-ed your Engineering counterpart what your manager said, he said “no” and stopped answering your texts or emails. What do you do?”
“Your CEO is sometimes a micro-manager on design issues. One of the designers on your team is having a really hard time with this, specifically with the CEO’s focus on relatively minor design details like specific colors or fonts. You believe your designer is correct that these relatively minor decisions should be left up to the Design team, but you also agree with the CEO that this designer’s visual design instincts aren’t as polished as their overall UX skills. What do you do?”
Alternatives to: “Tell me about a time when your team couldn’t meet a deadline”
“Based on earlier guidance from you, Sales promised several big customers that a particular feature would ship in April. At the time you were very confident about that estimate. But your engineering lead just told you that a technical issue was uncovered and the feature can’t be finished until July: 3 months later than expected. What do you do?”
“Your CEO told you that she’s concerned that features are being shipped at a slower pace than last year, even though the PM and Eng teams have grown by 2x during that time. You agree with her, although you don’t have any hard data to prove this assumption. What do you do?”
“Your QA team just uncovered a serious scalability issue that will cause your service to completely fail when concurrent users grow by about 40%, which is projected to happen in about 6 months. Fixing the problem will require about one-fourth of total roadmap capacity for the year. You know that several huge sales deals are blocked on features that you’re planning to ship this year. What do you do?”
Alternatives to: “Tell me about a time when someone on your team was under-performing.”
“One of the engineers you work with seems to frequently make mistakes that lead to customer-reported bugs. You’ve mentioned this problem to the Engineering lead and she said she’d look into it, although you haven’t seen any improvement. Now this engineer has been assigned to build a high-priority, technically-challenging feature. What do you do?”
“Developers at your company are complaining that one of the PMs on your team writes specs without enough detail. How do you address the issue?”
“The Product Marketing team at your company has been struggling to support the fast pace of product updates. For example, press releases are full of sloppy mistakes and clumsy messaging; the corporate website hasn’t been updated in a year despite a big shift in product strategy; and the latest content-marketing blog posts aren’t aligned with what you believe are customers’ top interests and concerns. What do you do?”
Alternatives to: “Tell me about a time when you used metrics or data to improve a product.”
“After steadily climbing for two years, your company’s NPS score declined by 10 points in the last month. There have been no major changes in pricing or product features, no major outages, nor any other immediately-obvious explanation for the change. What do you do?”
“Our company’s north-star metric is repeat usage: how many people use the product every week. The cohort of customers who signed up last year continues to improve in this metric. But the cohort of customers who signed up this year has been flat. What do you do?”
“Our customers in Mexico are churning 3x as much as customers in other geographies. What do you do?”
Alternatives to: “Tell me about a time when a customer asked for a feature you thought was a bad idea.”
“Your biggest SaaS customer—responsible for 15% of revenue!—is demanding that you add a custom integration with their company’s bespoke inventory software. No other customer uses this same software. What do you do?”
“Your engineering team wants to stop supporting iOS 12 and below, in order to use APIs introduced in iOS 13 that will reduce the cost of new mobile features by about one-third. Only 2% of users use iOS 12 or below, but your largest customer—responsible for 15% of revenue!—has 50 field techs who use your app on old-model iPads which can’t be upgraded past iOS 12. What do you do?”
“Your top competitor just released a new reporting feature that has been well-received in the market. Two large customers have told your salespeople that they are considering switching to that competitor in order to take advantage of this feature. You’re in the middle of a yearlong overhaul of your current reporting features which would make it much easier to deliver features like your competitor’s. But Sales (and the CEO, and your contacts at those customers) are pressuring you to implement this feature on your old reporting system, which will push back the entire roadmap by 3 calendar months and result in 4 engineers spending 3 months on code that will be mostly thrown away when the new reporting engine ships. What do you do?”
For all these alternative questions, here’s what to look for:
The candidate shouldn’t freeze like a deer in the headlights or otherwise show that they’d struggle to face a situation like this.
The candidate should ask thoughtful clarifying questions before launching into a solution.
Good candidates will push back on some requirements or otherwise show that they’re creative about finding win-win alternatives that achieve similar goals via easier means. But they won’t push back forever against an immovable object. Instead they’ll often try to get around the objection in a creative way
When candidates weave in anecdotes about how they’ve solved similar problems in the past, they should be relevant to the context you’ve provided.
It’s often helpful for interviewers to introduce new information into the question midway in order to make the candidate’s initial solution unworkable. Good candidates will appropriately adapt to the new information. Bad candidates will keep going on their initial idea without stepping back to consider if it’s still appropriate.
Candidates should react well to being told that a particular proposed solution won’t work. “Well” usually means “curious”—instead of immediately pushing back or giving up quickly. The best candidates will dig in to understand *why* it won’t work, and will use that additional information to improve their proposed solution.
Good candidates will be comfortable talking about and managing interpersonal dynamics. A red flag for me is a candidate who keeps turning the discussion back to features and metrics, even after I nudge them to talk about people and emotions. I sometimes test this by role-playing a difficult manager, customer, or team-mate and observe how the candidate treats me. Are they warm, collaborative, and steadily pushing for a win-win outcome? Or are they condescending, inflexible (or a pushover), or uncreative in coming up with reasonable compromises?
The best candidates will involve the interviewer in figuring out the solution, so that it feels more like a natural brainstorm session than a Q&A.
These questions and techniques are forward-looking and context-sensitive. They invite the candidate to demonstrate discovery and collaboration skills. They can introduce unpredictable ambiguity and change. In other words, they’re like actual PM work that the candidate will be doing if they get the job.
Why are anecdotal storytelling questions bad?
A few concerns with “Tell me about a time when…” questions:
Too generic / lacking context. Challenges vary by company. One team may constantly squabble while another is too conflict-avoidant. One team’s PMs may write sloppy specs, while another’s are too prescriptive. And so on. But “tell me about a time…” questions are generic. If a candidate picks pick the wrong anecdote, it can look like they don’t have experience with the problems that matter most to your company or team. Good PMs can adapt their problem-solving skills to the situation at hand, but they need context to know what the problem is! For example, instead of “Tell me about a time when there was conflict on your team” consider this instead: “Frequent engineer vs. designer arguments are getting out of hand on one team. How could you help?”
Wastes scarce interview time. Interviews are short, so interview time is precious. But when telling stories, the candidate may have to burn a few minutes providing context to the interviewer about the candidate’s company/product/team so that the story will make sense. This is especially true for candidates who work on obscure products or industries that require more explanation. However, if the interviewer supplies the anecdote, then it’s up to the candidate to interrogate the interviewer to discover context… which is just what real PMs must do every day! Using the “engineer vs. designer arguing” example described above, the candidate can ask “What do they usually argue about? Do their arguments usually lead to compromises, or does one side usually win? Is arguing limited to specific people, or is it team-vs-team?” and other questions that reveal to the interviewer how that candidate approaches interpersonal problem-solving. Instead of wasting scarce time learning about the candidate’s workplace drama, the interviewer can verify an important PM skill: asking good questions and synthesizing answers into a better solution.
Game-able / too sensitive to preparation. I’ve had a few occasions to answer these kinds of questions as a candidate, and when I had a suitable anecdote pre-prepared with talking points, my answer was clear, concise, and convincing. This was true even in areas where I know I’m relatively weak, because like most PMs I’m pretty good at narrative storytelling. It was very easy to fake competence. But when none of my canned anecdotes fit and I had to pick a story on the fly, my answers were rambling even in areas where I know I’m solid. The format seemed to be testing interview prep more than PM skills.
Unrelated to the company’s product & industry. By focusing solely on the candidate’s history, interviewers can’t test my ability to understand or solve problems with *their* product or industry. This is odd, especially for PM roles expected to own the product roadmap. As a candidate, it’d make me worry about screening for other roles besides mine; would my potential colleagues have been vetted for their ability to solve this industry’s problems?
Suggests a dysfunctional culture. So many of these questions focus on conflict or difficulty. It would make me wonder whether this company was an interpersonal mess!
No collaboration. Solving most real-world problems requires teamwork, but story-telling questions leave little space to collaborate with the interviewer. Problem-solving questions, however, encourage back-and-forth with the interviewer to clarify the problem and iterate on a solution. That’s better.
Worse with experience. If you’d asked 16-year-old me “Tell me about a time you drove on the freeway” I’d tell a detailed and compelling story of my first freeway drive because it was just months earlier. Ask me the same question today and you’d either get a blank look, or a story about a flaming car, a terrifying accident, or something else that was unusual enough to stand out from thousands of other freeway drives. Will those unusual stories really help an interviewer know whether I’d be a good delivery driver? The same is true for PMs. Ask a brand-new PM “tell me about a time you handled inter-group conflict” and there may be only one or two occasions to choose from. But an experienced PM may only remember unusual situations that aren’t representative of everyday performance. A better approach is to generalize, e.g. “How do you handle inter-group conflict?” which lets inexperienced candidates narrate their 1-2 times doing it, while senior candidates can talk about general approaches they’ve learned by trial and error after doing it hundreds of times.
Unrealistic. Summarizing the concerns above, “Tell me about a time when…” questions are not like actual PM work. Real PM work generally involves gathering and synthesizing information, solving problems with colleagues, resolving disagreements, building team cohesion, helping teams make tough decisions, and clearly communicating results to multiple audiences. In my experience, *talking* about doing those things requires different skills than actually *doing* those things. When interviewing PMs, it’s better to focus on “doing”.
Company context isn’t just for soft skills
Realistic, context-ful questions are also helpful for “hard” skill interviews focused on product strategy, business issues, UX design, etc. When I interview at a company, I usually spend 2-3 days learning all that I can about that company, its industry, and its competition. I search the web to find users’ opinions about the company and its products. I download and use its product (if possible), and ideally its top competitors too. I try to find and interview a few of its customers (if possible) and get their opinions. I write up prioritized feedback about the product. And when I’m an interviewer, I expect that every candidate should do this… although maybe not to the same obsessive extent that I do. ;-)
Candidates won’t know a lot of insider details after a weekend of research, but being able to quickly research products and industries and to speak intelligently about the results is something that PMs do frequently at work. So in an interview, I think it’s good to ask the candidate the same kinds of tough questions you’d ask them if they were already working there. For example:
Here’s a screenshot of a page in our product. How would you improve it?
Imagine you’re in charge of the Birthdays feature at Facebook. What would you do to make it better? (This is a real Facebook PM interview question, BTW.)
What did you think of our product? How would you improve it?
We’re trying to decide if we should focus more on Enterprise or continue our SMB-centric strategy. What do you think?
A big trend in our industry is XXXXX. What do you think of this?
Our competitor XXXX has been offering our customers to buy out their current contracts if they’ll switch to the competitor’s product. How should we react?
Imagine we could buy our competitor XXXX for $50M. Should we do it?
Like with soft-skill questions, don’t copy my questions above! You should pick your own that match the context of your company and the position you’re interviewing for. It’s also OK to ask “stretch” questions, because you want to hire people who are capable of thinking about strategy beyond their initial position.
A big caveat: don’t judge candidates’ answers using the same yardstick you’d use for existing employees. Candidates will make incorrect assumptions. They will suggest ideas that you’ve previously considered and dismissed. They’ll be completely unaware of important details. This is OK. What you *should* expect is:
Intelligent questions to fill in gaps in their knowledge
Self-awareness of what they know vs. don’t know
Curiosity—not hostility, persistence, or resignation— if you tell them their idea is off-base
Use of analogies or otherwise bringing in knowledge from outside your industry
They teach you something interesting about your product, company, or industry
Insightful feedback about the new-user experience (if they used your product)
A final note: a context-ful approach can also impress candidates because you’re treating them like colleagues, not like test subjects in a psychology experiment. For example, a PM friend of mine shared that context-sensitive interview questions actually helped him decide on his current company:
PM interviews beyond “questions”
Continuing on the theme of making interviews more like real work, keep in mind that “questions” aren’t the only thing you can do when interviewing a candidate. Here’s a few ways to supplement questions with more-realistic work tasks:
Ask for written work. I’ve been burned in the past by candidates who solve problems well in interviews but have trouble turning high-level solutions into more precise written formats that engineers and executives need. I now always ask candidates for examples of their written work, typically one for a technical audience (like a spec or PRD) and one for a non-technical audience (like a conference presentation or competitive analysis). It’s helpful is to ask for these samples *before* scheduling a full round of interviews with the team. Then if the written work isn’t good, you don’t need waste the team’s time on candidates who won’t work out.
Do a team brainstorm. Instead of a group interview where multiple people each ask one question in a rushed interview, consider a group brainstorm session where the group needs to solve a problem and the candidate is expected to help. Send out the topic in advance so the candidate can prepare, and then the candidate can spend an hour or so with the team and hash out a solution.
Present to the team. If you want to test a PMs ability to connect with an audience, try having her present something to your team. It could be the “work product” above or the output of the “team brainstorm” above. Or it could be a topic of the candidate’s choosing. I did this once long ago as a candidate (I presented about my favorite books) which was a nice way to get to know the team and vice versa, while also demonstrating competence at 1-to-many communication. Note that “present” isn’t necessarily slideware—it could be whatever format the company prefers.
Send questions in advance. If you’re committed to “Tell me about a time when…” questions, consider sending them in advance. Then the candidate can jog their memory with old emails & docs so they’ll have a more accurate and less long-winded story to tell you. This can also be good for non-behavioral problem-solving questions because no PM is expected to solve complex work problems without thinking about them for a while. Take-home questions aren’t good for generic questions where the candidate can Google for answers on the web, but if you take my advice and tailor questions to your company and team, then this should be a non-issue.
Take-home projects. For senior PM candidates, another approach that can work well is a take-home assignment where the candidate spends a few hours solving a problem or defining a plan or strategy, and then delivers it writing form and/or as a presentation. Some caveats:
Projects are time-consuming for the candidate, so it’s only appropriate after a successful initial round of interviews with the team.
It’s best if the output is prescribed, e.g. “a 3-5 slide presentation” or “a 2-3 page narrative”. Ideally, send the PM a sample of the kind of output you’re looking for. Good PMs can quickly adapt how they communicate to whatever format the team prefers. DO NOT make the candidate guess what you like!
Be careful about using this approach for junior candidates. I’ve seen much Twitter angst where candidates feel used by companies for free labor. DO NOT use the interview process to get free work. It’s a test, not a contract assignment. If you want free work, make it a paid project.
Epilogue: the neuroscience of PM interview questions
While researching this post I fell down a rabbit hole of neuroscience research about human memory. Some of it was relevant to interviewing, so I’ll share a summary here.
Human memory is composed of multiple, somewhat-independent components. For example: “semantic memory” is for recalling facts, names, concepts, or ideas, “episodic memory” is for remembering details about events that happened, while “procedural memory” is the ability to perform complex, pre-learned skills like riding a bike or running an effective meeting.
These memory systems are complimentary but operate somewhat independently. Neuroscientists learned this from people with brain injuries. For example, some patients have episodic memory amnesia (they can’t remember what happened yesterday) but they can still learn new skills like throwing a frisbee which requires building new procedural memories. Even without brain damage, some people are clearly better or worse at some types of memory vs. others. For example, while watching The Last Dance I was amazed at some basketball players’ ability to recall minute details of decades-ago games amid thousands of other games. That’s genius-level episodic memory. On the other hand, we’re all familiar with “absent-minded professor” types who are brilliant but frequently forget mundane stuff.
Circling back to the topic of this post, PMs who have great episodic memory will be better at answering “Tell me about a time when…” questions. PMs with lousy episodic memory will struggle with these questions. But I’m unconvinced that good episodic memory (especially details from years ago) is important for day-to-day PM work. The ability to accurately remember a bug report or customer meeting doesn’t matter much if we have JIRA or detailed meeting notes. Relying on memory might even be worse because, unlike JIRA or meeting notes, memory can’t easily be shared asynchronously with many people.
This comes back to my original thesis: interviews should try to match actual PM activities. Pulling up accurate and notes-free memory of long-ago events doesn’t seem to be a common PM activity, so I’d worry that “Tell me about a time when…” questions are unnecessarily screening out good PMs with bad episodic memory skills.
Epilogue 2: hilarious quote tweets
Many thanks to Daniel Lu, Joshua Herzig-Marx, Dan Duett, and Ankit Raheja for reading an early draft of this post and giving great feedback. The bad parts are all my fault, though. ;-)