Episode 31:What Nick Does
Kai interviews Nick about what he does, what A/B testing is, how he started Draft, the type of projects he’s worked on in the past, and the types of projects he wants to work on in the future.
Summary
Nick Disabato runs DraftRevise, a retainer A/B testing service under his consultancy Draft, and explains how his process shifted from solo heuristic reviews to research-driven work combining heat maps, Google Analytics, and jobs-to-be-done customer interviews before any test runs. He covers how he qualifies clients, why he refuses to use tests to settle internal design debates, and what it means to treat A/B testing as a design process rather than a marketing tactic.
Highlights
- The industry-wide A/B test success rate is 12%. Nick’s early heuristic-only approach hit 18–19%, a real improvement but still mostly failures.
- Nick now requires research before picking what to test. Heat maps showing zero scroll depth or no clicks on a section mean testing elements there will produce the same indifference to the new variant.
- He won’t run A/B tests to settle internal debates. If Carl wants the button blue and Wendy wants it green, the test can make one of them look like the better designer, which creates internal politics with no clean outcome regardless of which color wins.
- Nick frames DraftRevise as a design process, not a marketing or growth-hacking service. Changes to copy, layout, and color are all design decisions, and his job is to serve the design first, the design serves the customer, the customer serves the client.
- DraftRevise requires at least 1,000 revenue-generating conversions per month. Below that threshold, Nick routes prospects to a one-off teardown called Revise Express instead.
- Winning variants are contractually required to ship. Clients cannot veto the rollout.
- Revise Weekly is his paid newsletter at draft.nu/revise/weekly, with 1,000-plus word actionable A/B testing lessons each week. Kai credits two months of it with making him confident enough to raise A/B testing in prospect calls and know when to refer clients to Nick.
Read the transcript
So, Nick, you run DraftRevise, and I want to call it an A-B testing agency, but what would you call it?
So draft is the agency, draft revise is the service. And draft is I basically call it a consultancy. And I do interaction design for a living. Interaction design is the process of making something easier to use, more effective at communicating, and generally better to complete tasks on. That can be a lot of things, right? That involves fixing the usability of something. That involves interviewing customers and seeing what their motivations are, making sure you’re making something that they actually want to buy. It involves fixing some of the graphic design, involves fixing some of the functionality, involves addressing a lot of edge cases, like what happens when I paste Copied JPEG from a text editor into the credit card field. What happens? Oh, gosh. Where is your God now? Right? I do that sort of stuff. So it’s kind of a QA-ing of the product. It’s a lot of things, right? And Draft has had that under its umbrella for the entire five years that it’s existed. For the past three years, I’ve been running a service called DraftRevise, which basically acts as a A B testing, conversion rate optimization type service, and I operate on a retainer basis. What is A B testing? It is the process of you have control, the existing website, and you want to test a change and see what happens. So you run that change for about 50% of your customers, and you measure the success of the change. Is a way of being very careful and deliberate about your design process. It allows you to measure the impact of a design change in a way that is far more careful, less cavalier than if you just made the change and assessed the step function on Google Analytics, which who does that in practice? Nobody does that. It’s a lot easier to execute. So you have an A-B testing framework that’s usually a WYSIWYG editor, or you can redirect people to a specific page if it’s a more ambitious change. And that’s what I’ve been doing for the past three years. It’s been really interesting. I’ve changed the service in a lot of ways. But overall, that’s kind of the core conceit of it.
So I got a couple of questions here. You start working with a client, client comes in the door, you sit down, they’re like, one A-B testing, and you’re like, sure. How do you know what changes to test when you start that engagement?
So I’ve been doing a more and more involved research process over the past few years. It started out with DraftRevise, where I would basically go in and do what’s called a heuristic evaluation, where I had a bunch of different metrics that I would go and vet your website against. And That was done in a vacuum, right? It was done, I’m an expert, I know what’s going on, but I’m still not matching it to your customer needs, which was bad. It didn’t actually work out well. I ended up beating the industry-wide success rate for A-B testing, but only by a little. So it was 12% is the industry-wide success rate for A-B tests. And I was at like 18% or 19%, something like that. And that’s fine. I mean, you’re paying me to a 50% increase is cool, but that’s still kind of dismal in the grand scheme, right? Like you’re going to be dealing with more failure than success. which sucks. So more and more I’ve been going off like the heat maps that come in as I run A B tests and seeing like, oh, where are people actually like working on the site. Oh, they’re not clicking at all here. So why are we A-B testing it? Right? Like they’re just going to ignore the new thing too. How far are they scrolling down the page? Oh, nobody’s even scrolling there. Okay, well, that’s a problem. Where are they being lost on the page? Oh, they’re being lost up here. Now we need to change this element. And then we find the core thing that needs to be changed. I’ve started looking more and more at Google Analytics because everyone has a horribly broken Google Analytics install. And so just trying to fix that and deal with the Byzantine horror that is that Stupid web application that I’m very grateful is being offered for free. You get what you pay for. And Also, talking to customers. I get people on the phone. I have them write surveys. I have them provide support inquiries. And I listen to your support team and figure out what’s going on that’s wrong with your product and do everything that I can to Capture revenue, reduced churn, and make it so that people are actually buying and getting the benefits of your product.
What I love about your service is there’s such a focus on understanding what the customer actually needs and what they’re trying to accomplish. Like you said in your story about looking at a heat map and looking at how people are interacting with a page, I could imagine Another person who practices A-B testing coming in and being like, We need to test these six things. But there’s no research informing that decision. It’s just these are the six things we test first. But by looking at, like, oh, hey, you know, we could be testing the button, and the button might affect it, or the language around the button might affect it, but nobody scrolls down more than a third of the page. That seems like a more pressing issue. It seems like that’s a real value you contribute to an engagement.
Yeah, I mean, one thing that I used to actually recommend, which I no longer recommend, was, oh, A-B testing can settle a debate. And so Carl over here wants the button to be blue, and Wendy wants it to be green. And they don’t have any reason for it. They just think that blue and green, one of them, is going to generate more revenue. Okay. Let’s run an A-B test. Blue and green. Well, the button was red, so we have to have the control in there also. So several outcomes can happen, right? Red can win. Blue and green can do dismally. Okay, well then everyone is unhappy. Blue wins. All of a sudden, Carl looks like the better designer. And then you’re dealing with internal politics that are actually really toxic and problematic. What happens if nothing wins and it’s completely inconclusive? Everybody throws their hands up and blames me. Right. There’s no positive outcome, even if you have a revenue generating outcome from an A-B test that exists to settle a debate. So, what I do, I always, always, always justify it with research. I’m saying, like, oh, people are already hitting this button. Like, a lot of people are already hitting this button. Maybe we need to look further afield and figure out ways to get more people to hit the button. Maybe there are a ton of outbound links in the header that other people are hitting that we don’t want there. That could actually be potentially problematic. And so let’s look more deeply at this and not focus on the stupid color of the stupid button because you read something from a blog from five years ago that said that tested, interestingly. So when people come in with an agenda for testing, I’m like, great. So I’m going to take this, I’m going to look at it, and then I’m going to print it out, and I’m going to go on a Skype call with you, and I’m going to Set it on
This sort of leads into the next question I was going to ask you. What changes have you made to your processes over the years? Like you said, you used to say A-B testing is solid. You don’t put it on fire. Oh, you just boil it.
I would I would uh fricasy it. No, um it’s such a funny word. Uh so I no, it was I would actually take your opinion seriously, right? I’d be like, oh, yeah, we could test that. That’s an interesting idea. But now it’s like, okay, well, that has to go through the research process. So you have this idea. How are you justifying it? And sometimes they do justify it, and then we’re good, right? But if I have to come in as a consultant and justify every design decision to you, it goes both ways. You don’t get a free pass. I know it’s your business, but you don’t get a free pass. And the reason for that is because both of us are human and we’re both experts and we both know what we’re doing. I come to you with respect and integrity and I inspect expect you to do the same. And so, you know Another thing that matters significantly in the design process, I come at this from a design background. I am not a marketing guy or growth hacker at all. I believe that you are making design decisions when you’re A-B testing, which you are. Changes to the copy are design decisions. They affect the way that your site communicates. Changes to the coloration, obviously design decisions. Changes to the layout, also obviously design decisions. So when you’re coming to me, we’re doing a design process. And I do everything in my power to convey that DraftRevise is a design process and that you’re buying design and it’s in the guise of revenue generating conversion rate optimization, right? I found that works. It took three years to settle on a process that works, and I’m very, very confident about it right now. So, one thing about the design process, the reason I’m spending this whole time harping on this, when you’re in a design critique, it doesn’t matter who ventures the winning idea. It doesn’t matter who critiques the idea in the best way. It’s not a contest. You need to everybody needs to work in the service of the design. There’s another person in the room, and you’re there to serve them, and that person is the design. Okay, I’m not serving the client, I’m serving the design. The design serves the customer, the customer serves the client. And I know that’s a long way to get to that point, but it generates business value and it results in a better outcome for everybody. It also takes your ego out of the process, right? So you’re not just fighting with the designer about matters of taste. And you’re taking it away from like small trivial stuff. So, one thing I love saying, the logo is purple. I hate purple. It’s my least favorite color. I’m like, well, I hate purple too. But does purple fulfill the business goals? And then we have a conversation about the business goals, right? And I’ve immediately just pulled you over to my side of the table. And we’re both attacking purple from a business standpoint. And then it turns out, oh, we have a premium offering. Purple has these royal connotations. Oh, it’s kind of like. A feminine color. Oh, it’s like more of a luxury product. Oh, great. It’s like a beauty product. Now we can use purple. And you’re just going to grin and bear it every time you drop purple on that website, right? Like. And it’s I’m just completely spitballing an example here, but that that’s an example of some something where like, you know I don’t care whether or not the color is purple, only in as much as it fulfills the business goals.
Do you often run into clients or experienced clients who assume that their customers are carbon copies of themselves? Same taste, same preference. How do you overcome that? I research.
I mean, I talk to the customer. I’m like, who are you? How old are you? Where do you live? I get demographic maps. When do you shower? When do you shower? Where were you on the night of January the 16th? Great user experience interview question. No. I do what’s basically a jobs-to-be-done interview. And my colleague Eric White actually wrote a really great thing about this recently that I saw at a conference that I went to in Stockholm with you. Basically, you’re sitting down and you’re asking, well, why did you buy this? Right? Okay. You needed it. Okay. Well, why did you need it? Okay, great. How did you determine to buy this one? I’ve vetted a bunch of competitors. Okay, which competitors? Okay, why did you settle on this one? Well, because it was the prettiest. Okay, well, what what about it is pretty? Okay, this. And then all of a sudden, you have this giant map of the customer’s motivations, right? And you’re sitting there and I’m just asking questions and shutting up. Like I’m not saying, oh, well You know, you’re a, I don’t know, a single mom in New York City, and you just bought this, and you completely love it. And it helps your life in New York City so much. And you really love it as a parent, and it’s really great. Everybody else thought that people in trucker hats in Alabama really like this. It’s like, oh, crap. You know. Why were 85% of your hits coming from Manhattan? You know, like, right? Like. So you get this all the time. And I try and provide, I mean, this is a classic UX research thing. You provide a persona, right? You say, okay, well, here’s Jane. She’s basically. You know, as if Sex in the City were a documentary, and these are her motivations, and these are what she wants, and this is what, you know, et cetera, et cetera. And, um, You end up getting a lot of insights that are really, really valuable as far as how to communicate, right? And how to change the design, right? So maybe it should be purple at that point. Maybe it should be like peach or pink or I don’t know. I hate changing something to pink because feminine, but consider alternate things that aren’t the like red and black and the impact font. That you’re using to pitch to Joe Q public in Alabama, that’s not landing, right? This is the drastic example, but you see the point.
We’re losing listeners incredibly fast.
Everyone is just on track, right?
So I’m curious. I know you’ve changed this over time, and I’m curious, what types of businesses have you found benefit the most from A-B testing? And where are you personally focusing your time and attention within the business. What type of businesses are you focusing on?
Yes. So the businesses I initially created DraftRevise for SaaS businesses. And there’s still a plurality of my customers, but I found a lot of people come in that aren’t that. For two years, I worked with a tech blog called The Wirecutter. Since DraftRevise’s inception, I’ve worked with e-commerce sites a lot. And I found that e-commerce gets more traffic and more sales volume, especially like lower involvement ones that have really high sales transactions. you need at least 500 discrete transactions per month in order to justify having any tests on your website at all. And that is a high number for especially SaaS businesses. A lot of places don’t get that kind of volume, and it doesn’t actually work well for them. So what ends up happening is I just have a lot of like qualifications in my application process that are like, how many sales do you make? How much traffic do you make? Okay, great. You want a one-off teardown. And I can still do design and optimize your funnel a little bit by saying, okay, well. this one browser actually is doing horribly, maybe you should fix it. Bugs are a huge enemy of conversion, and you don’t do enough browser testing, I guarantee you. there’s probably something on mobile that’s failing horribly for you also. So I can outline that and do pretty well by it. But most of the time I’m working with like higher volume SaaS and e-commerce businesses. And they know they need to be testing. They know they’re actively losing money by not doing this. And they don’t know where to start. Hi. I’m here to help you. The number one reason people quit actually and end the engagement is because I’ve taught them everything they need to know, and they just want to do testing in-house. And when that happens, I’m thrilled. But then three months later, I email you, I’m like, How’s your testing practice going? And they’re like, it’s a disaster.
I’ve had the same happen with a number of clients where they’re like, We’d love to spend like the next month you training up somebody internal. They’ve been watching, they understand. I’m like, Yeah, of course, happy to. And then I’m like, How’s the outreach? And they’re like, Wah, fire everywhere, it’s burning. Come to Papa. It’s okay. I’m rubbing your head. It’s okay. Everything will be fine. I love you. Happy place. Happy place. So, how did you get to this point? How did you get to running this type of business? Practice, practice, practice.
No, um, I started designing. I did like wire Frames and like UX strategy and that sort of thing. It was a typical design consultancy. It felt like a Agency, I would do one-off projects, and I actually created DraftRevise because I was wondering what the Venn diagram overlap was of UX work and retainer-based engagements. And I came up with A-B testing. Seemed like a good idea. Launched it on a whim. It blew up. I was not expecting that. And then I hiked the price eight times. Well done. Charger. And now people are taken aback by the price. But I tell you, man, I closed two in the past couple of weeks. Congratulations. Thank you. Thank you. So I feel really good about where we’re at right now and other ways it’s evolved. The role of research has become not just prominent, but essential. And it used to be not really present. It was something that we would do as kind of a hunch. But I was stabbing in the dark too much, and now I’m not doing that. Another thing that happens, I contractually require that revenue-generating variants have to be rolled out. You can’t tell me no. And that’s fun. Another thing that happens is actually, you can’t tell your customers no. That’s really what it is. You can’t tell your business no. Another thing I do, I have put in more and more controls on what it qualifies a good client. So you have to have at least actually one thousand revenue generating conversions every month. And if you have less than a thousand, under every circumstance but one, I will like aggressively nope you and send you off to get a one-off teardown, which is called Revise Express. That one circumstance is because I use your product and love it and know your CEO and have drank him under the table. Then I’ll get you on a call and tell you no and why. But you have to be making more than that to even get me on the phone. Point because I’m too busy working with other stuff and talking with Kai Davis and having a great time and loving my job. The design Has never changed. The role of design has never changed. It has only become more prominent. I’ve only beaten my chest more about the role of design in A-B testing. And not enough people do it. They’re like, let’s change your headline. It’s like, why? Let’s change your value prop. It’s like, why? The only people I know that do A-B tests and don’t have a capital D design process that I respect tremendously are probably Joanna Weeby at Cubby Hackers. Because she is a copywriter, right? And a bit by trade at least. And then she vets her copy using A-B tests, which is fine. But that is also a design decision. And I don’t think she recognizes well, I think she recognizes it’s a design decision, but I don’t think she she says that it’s a design decis Publicly I think she would readily recognize it’s a design decision. But yeah, it’s I I make design decisions by measuring twice and cutting once. That’s a beautiful way to put it. Yeah. Thank you. I’m not a move fast and break things type person.
I once got on a call of a prospect who was like, We’re very much a move fast and break things type organization. And I’m like, and I honestly responded with a, I’m a move slow type of person.
Yeah, you’re going to work out at one point. You ended up closing that deal. Too, right? I did. That’s dope. Hell yeah.
Strategically lazy is how I place it.
I’m like a sloth.
You grab your own arm and fall out of the tree? No, man.
I you know. They persist. It’s high. It’s high.
So, if people want to learn a little more about you and the services you offer, what’s the best entrance point? What’s the best way for them to figure out what the draft experience is? Draft. nu.
Uh that is n as in nick, u as in the letter u. Usability. Usability Unctuousness. Now, draft. neu.
I mean, you have a paid mailing list that’s
That’s if you want to learn everything that I know about AB testing, go to draft. nu slash revise/slash weekly. And every week you will get a 1,000-word plus actionable lesson about A-B testing. Kai is a paid subscriber. What are your thoughts on Revised Weekly, Kai Davis? Give me a testimonial right now.
Revise Weekly has taught me as a marketer the essentials of A-B testing that I didn’t even know I needed to know and has made me confident in discussing. Whether A-B testing makes sense for a business, even though I have never done an A-B test myself, I’ve even added a question: do you do A-B testing? How is that going for you? What are you testing? To my prospect interview questions, because after being a subscriber of Revised Weekly for two months now I could see how important it is to a business. And I feel I’m not doing my just duty to a client. If they come in and they’re like, well, we have 100,000 email subscribers and we get, you know, 150,000 monthly visitors. And I’m like, do you do A-B testing? And they’re like, no. And I’m like, This seems like a situation where it would benefit you. Why don’t you? And now I feel knowledgeable enough to have that conversation. I don’t think I could sell them on it, but I could at least raise the problem, raise ideas around it, and then say, I think you should talk to Nick. I think he could help you make money.
Forward a couple of my lessons too. I mean, just plain text. They’re eminently piratable. You’re probably on the pirate bay or something.
I’ve literally made an e-book that is just a compilation of them. It’s selling for $1. Yeah. Yes. Very popular.
Go to yahoo. com for that. If you want to download all of Revised Weekly for free, go to thepiratebay. org. If you want to actually pay me money for it, go to draft. mu slash revise slash weekly, and I promise you I will give you a high five in person if we ever cross paths.
I’ve seen it. He does. It’s beautiful. It’s majestic.
Look at the elbow. That’s how you have a good high five. I know it sounds counterintuitive. Try it.
Yeah. Now you got to follow through for the elbow, and then it’s just like smack. And they’re like, oh, my hand stinks. And you’re like, yes. Thank you for joining me for this high five. Loving me is painful. Any other things you’ve written or content on your site or resources that people should check out if they’re like, I want to learn more about Nick or learn more about Evie testing?
I have a book. About interaction design called Cadence and Slang. It’s widely considered the best in my field. If you go to cadence. cc. If you want to actually think you’re a good fit for draft revise, take a look at draft. nu/slash revise. It is a fantastic way to begin the conversation. If you just throw your email address at the bottom there and fill out the application, I would love to hear from you if you’re a business that thinks you may benefit from A-B testing. and the kind of research-driven design work that I do. And yeah, that’s it.
Wonderful. Thanks so much. Mm-hmm.