Episode 37:The A/B Testing Manual
We discuss Nick’s new book, The A/B Testing Manual. That’s it. That’s all we do today.
Summary
Nick walks Kai through the A/B Testing Manual, his third book and first PDF-only product, built from lessons he’s been serializing in his Revised Weekly newsletter. The episode covers why he’s structuring it as a three-tier product, how it replaces Revise Express as a standalone offering, and what he’s actually scared about shipping.
Highlights
- The book is PDF-only because the technology topics will go obsolete. Nick said he prints only when content can “durably last for decades.”
- Three tiers: a base PDF with checklists; a second tier adding supplementary docs and a five-hour video course recorded with Patrick McKenzie (flying in from Tokyo); a third tier that bundles everything with a Revise Express engagement at three times what Revise Express used to cost.
- Nick is sunsetting Revise Express as a standalone product. His words: “This is the only way you can get Revised Express from now on.”
- He’s written roughly 28,000 of a target 45,000 words, assembling the book from weekly newsletter lessons. Every Revised Weekly subscriber gets a free copy of the book and the video course.
- The book is a customer acquisition channel, not a cash flow channel. Nick expects most of his direct money from consulting, not the $50 base price, and said he doesn’t care if teams buy one copy and pass it around.
- Nick’s floor for reader results: a 10–25% conversion rate improvement in the first year of running the process. He called it “bare minimum.”
- Pre-orders are live at draft.nu/manual. Buyers get lifetime updates as the book changes.
Read the transcript
So Nick, you’re writing a third book. Tell me a bit about what that book is.
It is a how-to guide for researching and executing A-B tests in an organization of any size able to get statistical significance. That’s it.
So you’ve written two books, Draft Evidence and Cadence in Slang. Why create a third book? Why invest your time there?
I had a lot of conversations at the beginning of the year with you and with a lot of other colleagues that kind of Shown clarity on a considerable amount of my business and made me realize that I should be focusing on research-driven A-B testing for the foreseeable future. So I started a weekly newsletter full of valuable lessons that people can get. We talked about Revised Weekly significantly in a previous episode. And this is kind of a serialization of what’s been happening in Revised Weekly and a Clarification of what my process is. I have never had a book that was so similar to what my job is on a daily basis. I wrote this in my mailing list. You can take this book if you’re a client of mine and fire me. And that’s scary to like point blank say that, but it is important to note. Right, you don’t have to hire me. Also, I can shot myself out, right? So, if I’m too busy or I think you’re a sociopath, I can give you a copy of my book, or I can sell you a copy of my book. So the book is underway, and it will be done in I’m saying March. I think it will actually be much sooner than that. And it’s going to be PDF only, which is kind of new for me. I’ve done print books pretty solely in the past. The reason being because I’m going to be talking about a lot of technological things and they’re going to go obsolete pretty quickly. So I only print when I think that something will durably last for decades, including cadence and slang. So the thing that spurred me to do this was kind of a clarification of my business’s positioning. And the thing that’s spurring me to do it now is I just don’t have it yet and I want it. And I think that we talked about this a lot in the products episode last week, but It will help promote my authority more significantly, and it will help me get in front of better clients, which is not to say my clients are currently bad, but there’s always room for improvement.
Let’s go one level deeper on each of those. So the assumption, which I agree with, is writing the book will promote your authority and help you get better clients. We’ve talked a bit about How writing a book, creating a product gets you more authority, but let’s go a level deeper on getting you better clients. Why or how do you see this book helping you get those better clients?
I think that when you release a book, people respect you more if you are a published author. in any capacity. And it could even just be a matter of them like skimming the book or even the marketing page for the book and seeing, oh, this guy’s legit. So I think that is like a key first point there. But another is kind of word of mouth, like, you know With A-B testing, it has to come from the top. It has to be a mandate from the top because revenue-generating design decisions are still design decisions, and they’re subject to all of the horrible politics that happen within organizations. If you do not have sea level approval of revenue generating design decisions, they don’t ship. And I enforce a ship clause with all of my contracts. I do not do the executive dance to make you feel good. So in terms of clients that actually ship, and I think it’ll help quite a bit, in terms of clients that truly understand what it is that I do and what it is they’re buying. They can spend two and a half hours reading this book and understand what it is they’re buying, right? Or they can read a marketing page and have a vague idea, and then I throw cold water on it and the welcome packet and the introductory Skype call. Spend a lot of time talking about that process. It all just becomes easier when you have a book and you can point somebody to it. I’m not against people pirating the hell out of my book. I fully expect that one person in a team will buy my book and then give it to all of their team members. It’s digital media. It’s going to happen. There’s no DRM. Don’t give a shit. I will probably even like leak it on a torrent track or something like that, where you can just get it there. Or you can pay me, right? I’m not going to be hurting for money on this. I’m not doing it for the direct revenue that I’m getting off of it.
So it’s not writing the book to establish it as a new cash flow channel for the business. It’s writing the book to Solidify your positioning as the A-B testing guy to have something that you’re able to point people to if you’re book solid and they’re like, hey, I want to learn from you. Or if they can’t afford you and they say, I want to learn from you. An opportunity just to make a statement about this is the draft process, for lack of a better phrasing.
Yeah, and actually the revised process was the name, like the working title for the book for a while. It’s now the A-B testing manual. Yeah, I think you’re right. Like, it’s designers love talking about their process and they love thinking incorrectly that their process is different from everybody else. It’s like we’re in a research and then we’re going to synthesize the research. And that’s the, in certain Name of agency process. And I’m like, what the f? That’s every agency’s process. What I do is not a whole lot different from what a lot of other CRO firms do that are white hat, like Conversion Excel is the one that jumps out to me the most. Right, and they’re amazing at this. They have a whole framework that’s oriented around research and revenue capture in various parts of your funnel. And they have a contract that states revenue-generating design decisions have to ship. They contractually have to ship. I don’t know what the penalty is. Getting fired? Suing? I don’t know. Something. But I’m doing, to directly answer your question, I’m doing this book to increase my stature as a consultant. I want money. If you want to pay me money for the book, I’m not against it. The book will be probably around $50. And if that does not make you back four times that amount of money in the first like two weeks that you have that book, and you follow my instructions and you can get it actually shipped to paying customers You’re probably either doing something wrong, or I don’t want your money, and you can get a money-back guarantee from it. Period. This book should be a bonkers ROI for you. I suspect that many of the suggestions in this book will make people seven figures. Strongly believe because I’ve seen it. I see it in my job every single day. Every time I call a test, that it ups your MRR by $44,000. That happened last week. You know, that’s this is a thing that happens in my job. And it is not an outlier. It is a relatively common occurrence. And so I want to be abundantly clear when I say that the information in this book is extremely comically valuable. If you balk at the price of a PDF being that much, go on Gartner’s White Paper site for a moment and take a look at what they’re charging. Or maybe the book just isn’t for you.
No, and I think that’s a very important thing to point out when it comes to this book or really any information product. People might balk between it’s a book you’re charging. I guess. Yeah, people will balk because it’s a book and you’re charging $29, $49, $99, $199 for it. But what buyers need to realize is You aren’t buying a book. You’re buying information. You’re buying experience. You’re buying expertise. You’re buying insight. It might be, let’s say you charge $99 for the book. I honestly don’t know what you’re going to price that at, which will be my next question. But let’s say it’s $99 People might be like, what? $99? That’s insane. What you’re really doing is purchasing a decade of design experience, design insight, and design expertise. that will influence the direction and structure of your business. And so when we take that into account, it’s not that you’re paying $99 for a book, it’s that you’re paying $99 for access to this expertise and insight.
Yeah, you’re buying an outcome, right? You’re buying an outcome. And if you think that you’re buying a book or that that outcome can come from someplace else, don’t buy it. That’s fine. You’re buying business results. Just like you’re buying that from me as a consultant, you know? So
Yeah, yeah. So we talk about it being an info product, a PDF, right but Tell me more about what is included with that. Is it just a PDF or are we looking at a video in addition to it? What does this look like? at least in a sketched out form as a product?
Yeah, that’s an excellent question. So there’s going to be a handful of like to do lists and checklists and stuff like that for you to follow various processes. Because there’s a lot of stuff that like I could write about in a book, but it’s more just like Here’s how you do a heuristic evaluation. Well, that’s pretty simple, right? A handful of like how-to guides. Like, here’s how you actually configure an A-B testing snippet on your website. Well, that’s kind of a development-y concern, and I don’t need to spend 10,000 words on it, right? So Here’s a checklist of what happens if your A-B test is going wrong, right? Valuable, useful, a little bit ancillary from the book itself. Yeah, I mean, that’s kind of supplementary, like appendix-y material. But the book is being launched as a full-blown info product. So there will be tiers to the book. It is not just the PDF. And this is very important. The first bit of the book is the PDF and maybe a couple of the checklists. The second is all of my supplementary documentation and a Full five-hour comprehensive video course with me and my friend Patrick McKenzie, who’s real good doing all this stuff and has a lot of B plus. Patrick, we love you. You’re working with you. He just started listening to the podcast, and I’m, you know, he’s fantastic, and he’s a wonderful human being. He’s flying in from Tokyo in a few weeks to record this with me. And then the third tier is all of that plus a Revise Express engagement. There’s a bunch of reasons for this. Number one is Patrick is recording a video course and he needed somebody to do it with. Great. He’s hired me. And we’re sharing some of the content on that. And so I’m able to sell this. We’ve come to an agreement that we can sell it as part of drafts work, and that’s completely wonderful, and I’m really, really grateful for it. The second and far more important reason is around Revise Express. Right now, Revise Express is $1,000 for teardown of your website. It is booked out comically, doing extremely well, and I don’t make enough money on it. I work too hard for it. Last podcast episode, we even talked about this. We talked a lot about. Putting your offerings into two columns: how much money you’re actually able to charge for it and how much you’re actually working on it. The ultimate goal of this book is to be able to charge more for Revise Express. By triple. So you get the videos, which is literally trivial for me to serve to you, and then you get a Revised Express engagement where I tear apart your website on my time. And that’s it. That’s all you get. And it’s got to be ready or not. And I’m going to just rip apart what you have together. And that’s about it. I was initially just going to launch this as a book and see how it would do, because that’s kind of my thing. But with the videos and knowing my strategic direction with Revise Express, I think this fits me quite well.
Yeah, this is an interesting change. Everything else you’ve launched before, cadence and slang and draft evidence, it took me a second. It really had been like single-tier or almost dartboard pricing model. Get the book or the book and PDF. But this is your first foray into Hey, there are three distinct tiers. There’s additional bonuses with each tier. The top tier gives you part of a consulting engagement. This is a new direction for you to take when it comes to creating products.
Yeah, it’s a classical three-tier end-for-product. It’s not, I’m not. Trodding new ground, right? There’s nothing special about what it is that I’m doing. But it is new for draft a little bit. You know, I would have like other tiers of my Kickstarter or whatever have you that would be like one-off stuff. sponsoring this podcast, or I made a dashboard for you for draft evidence, that sort of stuff. And those worked. They sold pretty well. They converted well. But this is different. This is definitely different. And it’s something that’s going to be going into perpetuity. And this is the only way you can get Revised Express from now on. That’s it. I am ending Revise Express as a concept. And if you want, you can pre-order all the content and get Revise Express. And you have to pay me three times as much money. Now you get a video course. Which is tremendously powerful. It’s so much better than the dumb how-to guide I attached to Revise Express before, right? And I’m proud of that how-to guide, to be clear, but like, in comparison. The video, of course, will give it a wedgie and hang it from a flagpole.
I honestly love the idea of this being intentionally designed to feed people up. I like the idea of sunsetting one of your offerings and replacing it with. Buy the top package of this tier if you want to get an express engagement. Like, I love how this solidifies your positioning in the A-B testing world. adds a new, really a new marketing channel to your business. A book can be a customer acquisition channel rather than a this is something I’m selling to make money channel. And I think Thinking about it in terms of a customer acquisition channel really reframes an understanding of product creation and information products. It’s not I’m going to launch this, and let’s see if I could get sales up to this. I’m launching this, it establishes my authority, it solidifies my positioning even further, and it becomes a marketing channel to move people into consulting engagements.
Yeah, absolutely. Absolutely. I’ve had 39 Revise Express clients since I launched it like two and a half years ago. And that’s a lot, and it’s great. I’ve pretty much proven the value of Revise Express at this point. People really like it. I’ve been hiking the price for it. And it hit a ceiling at one point. And My time is worth more than that. And I need to convey to people that my time is worth more than that. And the best way to do that is to write another dang book. And I’ve been forcing myself to do it by writing weekly lessons that I’m eventually going to stitch together into a more comprehensive book. Everybody who subscribed to Revised Weekly is going to get a free copy of the book and the video course.
Let’s talk about that for a second because I think it bridges two of our two interesting concepts and connects to a previous episode. So you’re essentially writing the book. I wish I could remember the phrase for this, but writing the book in place through your lessons, your weekly newsletter. And then that becomes the book itself. So it lets you approach this iteratively and piece by piece rather than now go create a book. What’s that experience been like as a writer and author?
Great, because you get ongoing feedback from people. I mean, I get replies back to every revised weekly lesson, and they help me refine the content, and they help me make it better for everybody.
What hesitations do you have around this project?
Like, what’s scary about it? The book is not done. That’s the book’s thing.
Is the book available for sale or pre-ordered right now? Or is it scary because it’s an unknown thing that still needs to be assembled?
It’s scary because it’s an unknown thing that needs to be assembled. And yes, the book is available for preorder right now. So if you go to draft. nu/slash manual You can buy either any tier of this, and I promise to you that you will end up receiving it. I will probably be delivering in descending order of value, which means Revised Express reports come first. Then you’re going to get the video course because I have a whole team on the video course that are going to be doing post-production editing on it. Recording of the video is happening the first week of September. We’re recording this like end of August, so you’re probably going to get it in mid-September. So by this point, hopefully, the video course will be fully recorded. Like, I’m scared about the writing aspect of it because that’s the thing that I have to do the most work on, right? I can sit in front of a camera and talk for eight hours. That’s not hard. I can and then I’m farming out all of the editing to somebody else. So I’m just kind of like letting them deal with it. But for me, I’m doing a lot of the editing and then I’m sending it off to my friend Caitlin to do the editing also. So there’s definitely like A lot that needs to be happening on the bookend. It’s the most unfinished part of it, which is why I’m saying it’s going to happen both last and in early 2017. Do not expect to get parts of the book early.
Well, I guess do expect if you sign up for Revised Weekly. Do.
But not all of it, right? There’s still going to be a lot of content that’s exclusive to Revised Weekly. There’s a lot of stuff that doesn’t quite fit in the outline of the book. But the goal is to get about 45,000 words on the page. I’m about 28 in right now. Yeah, and by this point, the time when you hear it, it’s going to be more like 35. So we’re making progress. But I also have three major revised client engagements and a dashboard and a bunch of other stuff to do. And I’m speaking at a conference and I have Patrick flying in and all these things. That is all really scary. I have to write the marketing page. I have not done that yet. And by the time you listen to this, the marketing page will be done. But I am flipshit scared about the marketing page. And I think I have every good reason to be scared about the marketing page. But yeah, that’s kind of like what I’m most worried about right now.
Yes, my next question was going to be in almost classic Kickstarter fashion, what are the risks or uncertainties that surround the project? But I think you’ve preemptively answered that. That’s all the risks.
I am scared of all the risks. I have every right to be scared of the risks. I’m not scared about digital delivery. I’m not scared about putting together Revisa Express reports because I’ve done it for two and a half years. I’m pretty good at putting together Revised Express reports by this point. Um, but uh PDF first book, I’m gonna probably be typesetting it in not InDesign, which is great, but also scary because that’s new That’s new territory. Even doing not in design is new territory for me. And it’s probably going to be in Apple Pages because they do decent enough typography. Yeah, I think the fact that I’m concerned about all of this will mean it will turn out good. That’s how I feel about it. It is healthy and reasonable, and in fact, quite encouraged to be worried about a product being good.
And I think. The fact that you’re creating this in a digital medium, that you’re going to be publishing it as a PDF, gives you more flexibility than you’ve had with. Draft evidence or with cadence and slang, where, oops, there’s an error. Oh, well, we had a thousand copies printed. Yes, we’re stuck with it.
There’s no undo button on the printer, right?
Yeah. Where you could, hey, you know what, version 1. 1 came out a week later because people emailed me hopefully about these issues in it. Great, here’s an updated version for you. It gives you flexibility. And in my mind, at least, I’ve never published a printed book, just electronic books. But it gives you a little more flexibility in terms of course correcting after the fact or adding a chapter, removing a chapter, iterating on it. It could be a living organic book rather than something that’s Fixed as a point in time.
Yes, absolutely. This is a complete situation where I’m happy to upgrade it and make changes to it. And anybody who buys it is going to get updates for life. So there’s that also. You’re going to be notified of new updates. It’s going to work out really well. And the outcome for this is Your organization understands that optimization is the most important thing to be doing on any pages that are generating revenue. Optimization is not something that black hat hackers do in order to steal money from their customers. It is something that you do to fix the bugs on your website and up your value proposition so that it actually communicates to people more effectively and solves the true problems that. That they’re facing in their lives. And you do that by listening to your customers, right? You do that by shutting up and listening to what they actually need. And that is scary and difficult, and I can help. And the ultimate outcome, you should be bumping your conversion rate by at least 10 to 25% in the first year that you do this. That is bare minimum. Bare minimum. I bet you’ll be better. I would love to hear you do better. Gathering a ton of success stories around this, I’m relishing it. I will dine out on it like a steak.
But no, you’re right that the success really is the success of the book is the reader or the readers collectively having success with the information you’re sharing. It’s not just Okay, great, I sold 100 copies. It’s hey, people are implementing these ideas. People are succeeding with them. We could see positive changes in the businesses of people who are buying the book.
Yes, absolutely. I mean, there’s no better way to get a palpable. Numerical economic outcome than by doing A B testing, right? Because what are you measuring? Revenue. What are you trying to do? Make the revenue go up. End of statement. There is no other goal of A B testing. It is entirely focused on revenue generation. And that’s great if you’re a designer, you know, like it works out super, super well. and allows you to have an economic impact to your work. So many people talk about how design is like this very like squishy, qualitative thing. And it is if you’re doing it wrong. If you just make the pretty for a living, it’s not just the pretty. It’s making it work better and communicate better for your paying customers.
That’s why people hire you. Right. And I think that connects to like the fundamental expensive problem that somebody who is a potential draft client or a potential buyer of the book is really looking at. It’s understanding that AP testing isn’t a way to make it look better, it’s a way to make it better solve the problem that a customer is having. And you get there through iteration, through research, through Testing these ideas and seeing what comes out the other side. If a test is a failure, it doesn’t, it’s not a failure across the board. It’s we tested this hypothesis and we learned something from it. There’s a lot of value in that. Even if you run five tests and each one, you know, the null wins, well, okay, we’ve learned that our assumptions were incorrect. There’s a value in that. And Above anything else, I think if your book teaches people to accept this testing process or accept that you need research, you need this methodology, you need to invest time in it, and the outcome is more than it looks better.
Yeah.
That’s a huge win.
If I told you, you know, to pat your head and rub your stomach for three hours and I would hand you $10,000 in cash, no strings attached. Would you do it? Just that’s it. I you’d never hear from me again. Tell me more. Do I get the cash ahead of time? No, you get it after the padding of the head and rubbing on the stomach is complete. That’s it. I’ll just sit there and watch you for three hours. There will be, I’ll just be stone faced. I’ll just be there. I’ll watch you do it. And that’s it. I’m just going to do that for three hours, and you get $10,000. You might be very tempted to say yes. Right. I think most listeners of this podcast would also be tempted to say yes, unless their hourly rate is way higher than I think. And that’s what research feels like. It’s doing something really weird and unfamiliar and a little bit uncomfortable. For a few hours, and then you make a shitload of money. And you do it by kicking your ego to the curb and checking your worldview at the door. And then you end up with something different. You are supposed to be radically questioning your own assumptions when you’re doing this. There is no. One of the things you’re going to learn in the book, there is no call to action that always works in A-B testing. There’s no change your headline to eight different headlines and that’ll work. That doesn’t happen, right? And anybody who tells you that is selling you snake oil, frankly. There is a long, hard, stupid way to A B testing, and I have an entire book ready for you to read that will teach you how to do it. And where can people find that book? Draft. nu slash manual.
Notes
- The A/B Testing Manual
- Periodically Yelling for Profit Note: Nick said there would be a money-back guarantee at one point in the episode. Then he thought better. Now there is no such thing. In fact, if you buy a copy of The A/B Testing Manual, there are no refunds of any kind. We apologize for the error.