The interviewer skills ladder for high growth companies

(medium.com)

182 points | by leeny 1714 days ago

9 comments

  • choppaface 1712 days ago
    This article has some interesting tips and useful encouragement but the system it outlines has a glaring hole: there is no objective feedback loop based upon outcomes back to the interviewer. The author appears to encourage that interviewers seek feedback, but the message is not to look for data but sentiment:

    “One way you can tell you’re proficient is that recruiters and hiring managers try to find ways to include you in their interview loops because they know you’ll both get the candidate excited and be able to assess their work fairly, even if they take an unusual approach.”

    If recruiters and hiring managers want you on a panel, it signals that you’re unlikely to disrupt their pipelines by rejecting candidates. This is especially true for high-growth companies (e.g. Twitter where ICs would do 8 onsites a week; I did 5 per week for years). They like you not because you’re proficient but because you’re aligned.

    Is it possible to be aligned and “wrong” ? Of course! Invariably, gluttonous hiring practices result in the need for “bar raisers” to restore consistency, entire teams with no real focus, and an explosion of communication channels. Hypergrowth is cancerous. And if everybody is in it together, nobody will be hold another accountable.

    This article outlines very little about proficiency and a lot about how to brainwash new grads into believing alignment itself with a recruiting process is a very special skill. Seek evidence about real outcomes instead. Do your own critical thinking.

    • bsder 1712 days ago
      > If recruiters and hiring managers want you on a panel, it signals that you’re unlikely to disrupt their pipelines by rejecting candidates. This is especially true for high-growth companies (e.g. Twitter where ICs would do 8 onsites a week; I did 5 per week for years). They like you not because you’re proficient but because you’re aligned.

      Even recruiters will cut you a lot of slack if your feedback is occasionally something like: "Well, we passed on that candidate. However, his knowledge of X, Y and Z were quite outstanding. Try and funnel him to a place that needs that--maybe company H?" This shows that you, as an interviewer, really did dedicate the time and effort to interview the candidate properly. Good recruiters consider this to be gold.

      And other hiring managers have never begrudged my negative judgments. Especially because I generally try to find positive points in a candidate and give some details--"Well, I liked X, Y, and Q, however, I am deeply concerned about lack of Z. If anybody else has counter-evidence that shows he does know Z, I'm open for it." Generally, the result is a couple other managers also saying "No, that lack of Z is real, and there are some other holes we discovered."

      Occasionally, you get a surprise. Someone will pipe up: "Huh? We talked about Z for almost 40 minutes and he was phenomenal. You must have had a communication misfire." It happens--then I will defer on that.

      I have had managers probe my consistency by sending me "ringers" that they were obviously going to hire just to see if I was being overly harsh. One manager decided to quit doing that after I threatened to hire the candidate for my own group.

      • choppaface 1712 days ago
        How do you push back when the manager sending the "ringer" is the VP of Engineering? And they're wrong about the candidate? I've had this happen twice with VPs at different companies. VP makes a joke panel, candidate gets hired, then fired within 3 weeks. Then at least 1-2 people want out of the fired person's team the following month.

        There are definitely recruiters and other players who are thoughtful when they get thoughtful feedback. Yes, if you help a recruiter tune their funnel, you'll help their numbers, so of course they're going to listen. Those incentives are clearly aligned. But in a high growth setting, especially around new grad job fair time, the head count quota is what dominates. It's easy for that quota to be unrealistic, and for hiring efforts to become misaligned, or even at odds, with the long-term interests of the engineering team.

        The core problem is that recruiters, hiring managers, executives, etc., deprive ICs of the contextual data (e.g. headcount goals / internal rankings) and outcome data (especially compensation details) that are critical for ICs drawing their own informed conclusions. We must cease accepting the norm where ICs have to dig for glimpses of this data, and share it surreptitiously (e.g. Blind), when the whole picture is readily available to those owning the final decisions.

        • bsder 1712 days ago
          > How do you push back when the manager sending the "ringer" is the VP of Engineering?

          Work someplace better?

          Seriously, anyone I have known who is senior in the engineering chain always made sure to tell people "Look, I'm recommending this person, but you are the hiring manager whom he is going to be reporting to. If you and your team don't agree, then he doesn't get hired. So be it."

          And, they meant it. VP recommendations only have about a 50% success rate when I'm involved. If the VP isn't a dolt, normally it's simple misfit. The tougher discussions go along the lines of "This person is really good at Z and we really need that. The problem is that for him to be useful at this company after this project he needs background in X and Y, and he simply doesn't have that." If you have a good VP, he recognizes that the solution is "short term contract consultant".

      • leggomylibro 1712 days ago
        It does seem like you are usually incentivized to reject people, as an interviewer.

        Nobody ever complains if you pass on someone but say that they were good at X/Y and might be a reasonable choice for role Z.

        But people will jump down your throat if you let a subpar candidate through the funnel; I've been chased away from doing phone screens for being too charitable, because people got sick of their time being wasted every now and then.

        If my experience is anything to go by, it's probably smart to avoid becoming an interviewer if you're overly empathetic. It's a role where trying to help people is usually the wrong thing to do.

  • emtel 1712 days ago
    I don't think "assessing talent" during interviewing is a skill you can learn. The skill you're learning is assessing how well the candidate performed during the interviewing process. The interviewing process itself is what is assessing the talent.

    Put another way, I would rather have a novice interviewer use a good process than have an experienced interviewer use a bad or mediocre process.

    In my experience, FAANG-style whiteboard interviews are a bad process. They don't work for assessing developers, and the skill of the interviewer can't compensate for that. The only thing I have found that works consistently is some form of work-sample interview, where the candidate produces working code in their preferred development environment.

    • braythwayt 1712 days ago
      "I would rather have a novice interviewer use a good process than have an experienced interviewer use a bad or mediocre process."

      This provokes me to ask, is that also true of software development? Would we rather have a novice programmer using a good process than an experienced programmer using a bad or mediocre process?

      My first thought is that the experienced programmer produces good results despite the bad process because they write a lot of tests, and so forth, but that suggest that they are using their own good process inside a bad process.

      An experienced person using a bad or mediocre software development process really ought to be understood to be eschewing source code control and tests. They are patching production directly. They aren't putting thought into their names or software organization.

      And yet they get some good results. But given this false dichotomy of novice with good process versus experienced with bad process... Which do we really prefer, and why?

      • cbanek 1712 days ago
        I would say each developer also has their own process, which may be more or less or different than the organizational process. I find that good devs have a good personal process.

        I think being an expert sometimes means you know which rules should be followed, and which rules can be bent for certain circumstances, possibly even producing a better result. Novice programmers may not know all the consequences of their actions, so it is better for them to follow a strong rigorous process.

        Good results is also subjective. Does good results mean fast? Or that you can look at the code while not having to clean the vomit off your keyboard? Bug count? Getting it right the first time? Good UX?

        Knowing what you're going for and what counts is where the process should be pointing, but sometimes they are just rules for rules sake.

      • wisty 1712 days ago
        A lot of programming has low stakes and immediate feedback. If a "bad" process means you have to fix a bug and recompile it's no biggie. And since there's rapid feedback, a "bad" programmer will usually be doing stuff that actually works.

        Exceptions would be things like security, backups, major data / source code loss where things don't really go wrong until they go spectacularly wrong.

    • PopeDotNinja 1712 days ago
      The real skill is deciding what you need to interview for in the first place. Once you know roughly what you need, the next skill is learning how to assess applicants in an impartial manner. Once you know that, the interview is kind of a formality.
    • endymi0n 1712 days ago
      After more than 500 interviews and having hired about 50 engineers, I have to wholeheartedly disagree. I've A/B tested my interview script and technique over all those years and there's a massive inverse correlation between how hard the answer to a question is to grade and how good of a predictor they are. Deep and open questions are the best ones, period, and learning to interpret the answers to them takes years. Interviewing is really a skill to be learned and it's a hyperdimensional optimization.

      One of my currently best engineers started out with one of the worst coding challenges I've ever received (think JSON marshalling by string concatenation), but he was so smart, empathetic and driven I took the time and energy to build him up. One of the best challenges was by an asshole that was so intelligent he constantly managed to sabotage himself as well as the team.

      TL;DR: These days I don't believe anybody anymore who proposes there's just a single skill to be checked in interviewing. Tech Interviewing is just a hard job, just like engineering itself, but it's equally fulfilling if you take the time to learn it and get good.

      • emtel 1712 days ago
        I'm not sure exactly what your disagreement is. I agree interviewing shouldn't test a single skill. I also agree you can get better at interviewing. But I still think the interviewing process you use is more important than the skill of the interviewer.

        I have done a similar number of interviews as you, and I've found very little correlation between performance on FAANG style interviews and performance on the job.

      • braythwayt 1712 days ago
        I see your answer as boiling down to "That's a false dichotomy. We need experienced interviewers working with a good interview process in the form of deep and open questions that require experience to grade correctly."
  • pkteison 1712 days ago
    Can anyone recommend a good book on becoming a better technical interviewer? I’ve found plenty of books on being a better interview candidate, and a few for hiring ceos or sales people, but I haven’t been able to find anything with useful suggestions for the hiring side of the table in a tech interview.
    • alexallain 1712 days ago
      (original author here) I got inspired to write this post after helping contribute a few chapters to The Holloway Guide to Technical Interviewing and Recruiting, which is in early access and has a more material that might be interesting+relevant to you: https://www.holloway.com/g/technical-recruiting-hiring/about (It goes beyond just interviewing, as well, to cover the whole hiring process end-to-end)
    • jsjohnst 1712 days ago
      While I can’t recommend a specific book off hand, I’d suggest looking into books in the realm of profiling. Technics I’ve read in that area made the biggest improvement in my overall ability to interview engineering candidates.
      • Ozzie_osman 1712 days ago
        Me and over a dozen other contributors are working on a Guide [1] to Technical Hiring and Recruiting that's due out this fall (published by Holloway). Alex Allain (author of this blog post) is actually a contributing author and wrote our section on interviewing. If you're curious enough, and willing to spend time reviewing an early draft and giving feedback, please use the contact us link on Holloway.com.

        [1] https://www.holloway.com/g/technical-recruiting-hiring/about

      • wool_gather 1711 days ago
        Could you be more specific about the kind of profiling you mean? My assumption is software performance profiling, but I'm not sure if that makes sense. The other kind that occurs to me is law enforcement profiling, but that makes even less sense to me. I think there's a use of the word that I'm missing to understand your comment.
  • tignaj 1712 days ago
    Engineer / engineering manager here.

    Nicely articulated article. There is one observation that I respectfully disagree with - the one that says interviews are monotonous and boring.

    I have done hundreds of interviews in my career and never found them to be boring. Perhaps one the reasons is I don't do scripted interviews. It is always an exploration for me. I will start with simple questions and if I get good answers I will go deeper and deeper into the subject matter or into the adjacent area. If I don't get good answers I will retract and probe another direction. This is sort of a depth-first traversal of candidate's skills with a heuristic to stop traversing a particular branch if I see it is not a promising direction.

    My goal is to discover the area where the candidate is the strongest (obviously within the areas I am interested in for the particular role I am hiring for). It is very fun and almost never boring. The experience is always different because people are never the same. Yes, some candidates are just not good, but even bad ones are bad in different ways.

    I know common wisdom for using scripts is to have repeatable and comparable evaluations so that you can choose the best candidate. I understand the desire but I prefer to find the best in each candidate and if it means I have to use a lot of gut feeling and intuition to make the final decision then so be it, I believe it still results in better overall results. The idea to choose the person I want to hire based on how many answers to my standard questionnaire they got right was never appealing to me.

    I can totally see how if you do scripted interviews it can become boring, so don't. Interviews can be fun and it can be a creative process.

    Another way interviews can be fun is if you find someone who has stronger skills than you do in a particular area. I love it when a candidate teaches me something new. Find where their strength is and let them shine.

    • Ozzie_osman 1712 days ago
      I agree that scripted interviews can be boring... but it's not just common wisdom that using structured interviews leads to better assessments, there's plenty of research* that shows that too. Of course, you should always take research in this field with a grain of salt, since things are different from industry to industry and company to company. But I think it's safe to assume that unstructured interviews are more prone to unconscious bias. Research also shows that most people think they are better at assessing candidates in an unstructured environment than they actually are. If you're really experienced, you might be calibrated enough to be immune to that to a large extent, but it's still hard.

      But yeah, I agree that it can be boring and a poor experience for both candidate and interviewer if interviews are too scripted. It also has other failures, e.g. it can fail to capture different talents that candidates might have. I personally think the best way to handle this is to mix both structured and unstructured interviews.

      There are plenty of other ways to make interviewing rewarding. Think of it not as a burden, but as a way to help both the company and the candidate make a possibly life-changing decision as accurately as possible. Empathy can go a long way towards humanizing the process. Honestly, one of the most rewarding things I felt as an interviewer (and then a manager) was seeing someone I interviewed (or hired) later thriving in a role. Being a part of making that happen is really fulfilling.

      * http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.172...

      • tignaj 1712 days ago
        Thanks for the link, I haven't seen this before, going to read it.
        • Ozzie_osman 1712 days ago
          Google has a few links as well: https://rework.withgoogle.com/guides/hiring-use-structured-i...

          There are also some showing that many people choose unstructured interviews over structured ones because they are overconfident in their ability to assess, and I think one or two that show that unstructured and structured if done properly and together, are better than either method alone. But again, these sorts of studies and metanalyses are very general, so up to you on how seriously to take them. I at the very least found it made me much more cautious of my own assessments in unstructured settings.

          It's also true that interviewing isn't just about assessing, it's about giving the candidate information and a good experience. So for instance some studies show that small talk at the beginning of an interview can bias the interviewer. But I still do it because I want interviewee to be as comfortable as possible.

    • choppaface 1712 days ago
      How do you ensure that your reports get objective feedback about their own interviewing?
  • sixtypoundhound 1712 days ago
    My reaction to this article - we've utterly failed when it comes to training effective mid-level software engineering leaders. It feel like we are substituting rule by committee for what should ultimately be a leadership decision. If you cannot select engineers, you should not be leading them.

    Employee Input is certainly welcome. We don't want to hire jerks or idiots. But the manager is responsible and accountable for the quality of hires and employee performance.

    The other half of my life is in manufacturing / distribution. We have engineers too, managing great big piles of machinery. We RUN on engineers and mechanics. Generally led by engineers. Hell, my last CEO was a former engineer and my current COO (who runs the plants) is an engineer himself. It's engineers, all the way up.

    They don't need to shop candidates all over the building to figure out if they can handle the job. They know. And can make an appropriate decision.

    It starts with setting the expectation that engineering centered functions are to be led by engineers. And the price of leading those functions is knowing your stuff...

    No Excuses.

  • the_arun 1712 days ago
    I found this is a great summary of non-technical side of the interviews. Thanks for sharing!
  • neilv 1712 days ago
    The article names "making the candidate want to work with you" as the first of the two critical skillsets, and goes on to reinforce that this is important.

    I've thought for a while that this might be one reason why some large dotcoms (which reportedly study their interview efficacy) put even very experienced candidates through one-way leetcode whiteboard batteries. It seems similar to "negging"[1]. If there's an attraction intent behind it, that would explain a lot.

    If so, then perhaps we have countless other companies mimicing the big dotcoms, or at least being influenced by the practice, without being in on the dark pattern side to it.

    [1] https://xkcd.com/1027/

  • draw_down 1712 days ago
    I noticed that the people who interview a lot don’t really get anything in return; in our quarterly engineering org meetings or whatever there’s a slide thanking the people who interviewed most. I’ve never gotten a thanks when I interviewed more than other times, nor any other indication that this is something my manager values. (And what your manager values is kind of everything, in my experience.)

    So I almost never volunteer when they need someone to cover, don’t try to put more effort into it than necessary, and have blocks in my calendar so they don’t try to schedule when I don’t want them. It may be crucial for the company (and ethically, you owe it to candidates to treat them fairly) but it’s not treated like something they value.

    For what it’s worth, I somehow slipped through the cracks and didn’t have to interview for quite a long time after I started. No one complained or made a note of it at all.

    • csydas 1712 days ago
      Ideally, the trade-off from Interviewing should be that a person has some agency in the direction of the company, that their values can be reflected in the influx of new recruits, and that persons with similar ideals or interesting skillsets can be added to the floor to make it a more interesting place to work.

      I've had the role of global interviewing for our technical team thrust on me, and most of the time, it's just a chore. I'm constantly battling with HR/Managers who simply have a "fill seats" attitude towards hiring, and it can be a political minefield. (Professional tip; if someone asks you to take-over hiring from someone (or a group) that has not fully ceded this responsibility for any reason, don't accept)

      Despite the above though, our global team now looks, works, and acts like I want. Even our technically weaker Engineers have the interest to improve and the drive to get better, and universally our team agrees that the flood of questions from these weaker engineers are not wasted. This is the one bit of satisfaction I get from doing this type of hiring -- I get to work with people that do the type of work I want to see.

    • kinkrtyavimoodh 1712 days ago
      You touch on a very important thing. It feels like no one cares about interviewing UNTIL they can use it like a cudgel to hit you in the head when you ask for a promotion. In the normal course of affairs, interviewing gets you nothing. But if you don't interview, they get a convenient excuse to not promote you.
      • draw_down 1712 days ago
        I’m sure you’re right, but honestly they will always find a reason if they’re looking to find one.