Story: One Million Checkboxes - The Viral Game That Defied Convention
52:05 Share

Hello and welcome to Code Recursive. I'm Adam Gordon-Bell. Back in high school, during the dial-up era, my family had a computer, and so did most of my friends, and we all connected through this chat site that was popular in our little circle. One of my friends had this paper with HTML, like handwritten on it in pencil, and if you typed it into the chat app correctly, which was hard because it wasn't clear what all the symbols were,

It would launch a bunch of pop-ups and it would crash your browser. It was like this secret hack where you could take down the chat site. And it was handed around from person to person. Back then, nothing on the computer felt very real or important. It was just like a fun playground for teenage mischief.

There was once a fight that broke out at school over something that someone said in chat, but mostly it was just playing and a little bit of trolling, occasionally crashing people's browsers.

But now the internet is just life, right? And when the internet became serious business, when it became part of reality, did we lose that fun? Today's show is with Nolan Royalty. We'll dive into his wild encounters with the internet's chaos, how his creation sometimes got misused, and also his worry about being hacked and data breaches.

This was happening on Monday evening, and my girlfriend had made dinner. I thought that I had a data validation error that let people stuff strings into this bit set. Because I hadn't really even thought about what it was that my script did at this point. And I had kind of gotten re-obsessed with what is going on here right as dinner finished. And she was like, can we sit down and eat? You've been very busy recently. And I was panicking about...

There are these URLs, I have been hacked, I have been hacked. But before we dive into the hack, let's rewind to a moment in his past that reveals something crucial about him. A moment where he pushed the boundaries, expected punishment, but got something entirely different. Something that might just explain why he's drawn to the chaos of the internet in the first place.

Nolan calls himself a deranged game builder. He dropped out of college, he spent some time at AWS and then Jane Street, a trading firm. But today is not about that, right? It's about the deranged game building part of Nolan's career and his exploration of the bounds of computer systems and what you can do. And it all started back when he was in boarding school where they had something called the student-wide information system. It was like email and it was chat.

But point being, it was, you know, an important part of student life. And also, it was the way that you turned in your homework. It had like a weirdly powerful mail rule system. So you could write rules in the same way that like in Gmail, you can filter your mail or whatever, right? You could write rules that said, like, when I get a message from Richard, apply the Richard tag. The rules could take a lot of actions. And in particular, they could send emails.

And one other quirk, the reason that I kind of got into doing the thing that I ended up doing, was that when you got a new email, it made a pretty loud beep. And I had this friend, Brody, who always had headphones on during our study hall. We had study hall for two hours a night, Sunday through Thursday. And whenever he got an email, he would like jump because the noise was loud and he had headphones on. I thought it was funny. So it's kind of fun to like send him a message and see him jump.

Then, Nolan started tinkering with the email's rule system. So what I did is I wrote a rule that said whenever I get a message from myself with an empty subject, 1. Send a message to my friend Brody, 2. Delete the email that I just got from myself, and 3. Send myself two more emails that also have empty subjects.

And so then I would enable this rule and I would send myself a message. And then it would send Brody two and then four and then eight and then 16. And, you know, pretty quickly, a lot of messages. And he would get like ding, ding, ding, ding, ding, ding, right? And like jump out his chair, fall over. And it was very funny to me.

Brody fought back, setting up rules that would delete Nolan's emails before they could cause a beep. And then he made rules that would send things back. Nolan tweaked his emails to slip past Brody's defenses. And so the nerdiest battle began, each of them writing rules to outsmart the other. And that was all totally fine and fun. And we did this for like, you know, a week or something. And then one day, both of us just because whatever our defenses were good enough, forgot to disable our rules.

So normally you would like enable this for like three minutes and send a lot of emails and then it was fine. But instead we just totally forgot and we like went to bed and we woke up the next day and the school email server was done. No one could send emails. They got the system back up briefly, but the rules were still active, causing it to crash repeatedly over a couple days. Eventually, they traced the problem to the rules and informed the dorm supervisor that Nolan and Brody's battle was behind the email chaos.

And he came to me and was like, what's going on here? And I explained. And he was like, that's hilarious. That's so funny. And he made me a t-shirt. But I was kind of proud of finding this thing. And I think it certainly made me, I don't know, there's like some through line from that to plenty of the stuff that I do now in terms of like, how can I misuse this thing in a way that's kind of fun? And so being encouraged then I think was pretty nice. Imagine expecting detention, expulsion even, and instead being met with laughter and encouragement.

This supervisor didn't just forgive Nolan's digital mischief, he seemed to celebrate it. That moment was a hint of what was to come for Nolan. The New York Times interviews, the media buzz, the Washington Post, the speaking gigs, they all trace back to his knack for bending things outside their normal use. So fast forward, Nolan is an engineer, then a manager at Jane Street, the quant firm. He builds up a bankroll that's large enough to quit and do something that he's always wanted to do, make video games.

And I had this big list I had spent all of 2022. Every time I had a game idea, I would like put it in the list. And so I had this big list of ideas that I thought I would like be working from for a while. The first thing I made was Tetris, just to kind of like get an understanding of how to like put objects on the screen in Giddo.

And a thing I decided pretty early was that the goal was to build lots and lots and lots of small games that I understood that one common failure mode for people who try to make games is that they're like, oh, I really love whatever. This is a silly example, probably, but like World of Warcraft. So I'm going to build a big MMO.

And don't think about how that takes years for a team of hundreds, and it's probably not a good starter project. And you learn so much by finishing a game and seeing people play it or playtest it. And the thing you want to do is get a lot of reps in of that. So it's like, I'm going to make games that take me a week or two as fast as I can and blog about them. And there's a sign that

right above me that says the plan one make games as fast as you can to blog a lot that I put on my wall to kind of I don't know give myself some some goal that sign is still there and so Nolan builds games but it's harder than he thought like it's actually really hard to just like go to your little I don't know second bedroom and and sit at the computer all day and write code and not talk to anybody about it and you can tell your friends about it but like

I don't know, your friends aren't working on it with you and you probably can't talk to them as much as you would talk to your coworkers about what you're working about. So Nolan decided to apply to the Recur Center, a place that's kind of like a writer's retreat, but for coding. You go there and you write code for 12 weeks or six weeks if you do a half batch. And when you join, there are a bunch of people who have been there for six weeks to kind of show you the ropes. Then they leave six weeks in and now you're showing people the ropes.

And the goal is just like write as much code as you can every day according to whatever interests you, right? So like when you apply, I think they do like a basic coding test thing, but mostly they're looking for like, can you write code and do you have an idea of what you want to work on in a very self-directed way? So I applied and I told them I was going to make eight games in 12 weeks and I showed up and I started making games.

The Recur Center wasn't just about solo coding marathons either. They had workshops. Nolan joined one on pair programming, where his task was to create a small game.

We were writing a solver for the game Mastermind. Yeah, with the mines? No, not Minesweeper. Oh, yeah. Mastermind is like the colors? Yeah, the colors. So it's actually very much Wordle. Like Wordle is Mastermind, but there's only one player and you're guessing words. And Mastermind is a two-player version of Wordle where you're guessing the colors of some pegs that somebody has hidden from you. But it has kind of similar mechanics. Yeah.

But they told us we were going to write a solver for Mastermind or a backend for Mastermind or something. And I said that I wanted to work in SQL. And they matched me with someone else that could work in SQL. And so we wrote a Mastermind thing that worked entirely as a big SQL query. And it was really fun. It was a huge blast. And we showed it to a bunch of people. And everyone was like, yes, this is funny. We're glad that you did this. And that was very exciting to me. Nolan got inspired to create a game using OpenSearch.

that is the way that your computer knows how to search a specific site. So if you've ever gone to like, you've typed Wikipedia into your search bar and hit tab and then typed a query and it's like searched directly over Wikipedia, right? The way that works is that under the hood, when your computer visits Wikipedia, Wikipedia serves it a document that says, if a user wants to search my site, hit this endpoint. And the spec happens to include this crazy thing where optionally browsers can choose to support

search suggestions. So just like when you type into the Google search bar, it guesses what you want to search for. Your website can offer those suggestions. So you say, hey, whenever the user types a new character, hit this endpoint and I'll return some ideas of what they might be searching for along with like URLs to directly visit that page. So the whole bit was that you can return whatever you want from that.

So why don't you return Wordle answers based on the Wordle guess that they're typing into the URL bar, if that makes sense. So Nolan took this idea and created a Wordle-inspired project. Then he wrote a blog post to break down how it all worked. It's nice to have a website where you can point to everything you've made. But I think it's also just a really nice way to codify what it was that you learned from a project and explain what you were thinking. And so for this, I made a blog.

Nolan demoed this at a Friday presentation at the Recur Center where anyone could grab a slot and show what they were working on. And I think that was the first time that like I saw that people enjoyed something that I made as I like presented it and the response was like pretty good. Getting positive feedback on that was really fun.

Someone at the Reeker Center shared Nolan's blog post on Hacker News, and then it gained traction. Encouraged by this response, Nolan kept going to the Reeker Center and he just kept making games. I made several more kind of more traditional games, you know, games with like high scores and guys that you controlled and you like shot an enemies. I remember taking a game of mine, a more normal game of mine to a playtest and being there with some friends from Reekers who had brought games that I thought were more much more interesting than what I had made.

And being at this playtest event and getting feedback on my game from some people and just being embarrassed to be showing the game, being like, oh, I really... Not like it was a terrible thing, but just that it so clearly wasn't what I wanted to be doing. I wanted to be one of my friends showing these much weirder, more experimental things. And I was getting feedback on like,

how the numbers in my game could be adjusted and like, oh man, I just really don't want to think about like balancing numbers in games. That's not where, what I'm, what I'm, what I'm interested in and what I'm passionate about. And so I think like the, the big thing there for me was learning that what I wanted to be doing was this kind of weird stuff. And also that I was just better at it. Like I'm very, very into taking a weird bit of technology and pushing it as far as I can.

I have zero patience for picking nice numbers that make a game feel balanced. I don't want to work on games that need nice numbers to feel balanced. I want to work on, like, weird experimental stuff. He wanted to do the stuff that bent the rules, that surprised people, that maybe even broke things a little bit in a fun way. Just like that original boarding school email prank. But now, he was channeling his creation powers into this. And after 12 weeks, Nolan's batch ended.

but he kept showing up at the Rieker Center every day.

I came up with this idea for making a game inside Mac OS Finder by renaming files and doing a bunch of other hacky stuff, which turned into a version of Flappy Bird that you ran inside Mac OS Finder. That, I think, was the first time that I remember making something and realizing after... I demoed it at Recurse. I was no longer in batch, but you can still show up to demos whenever you want.

And like people like lost their minds. Like it was, you know, like the reception was really, really, really good. And a bunch of people wanted to talk to me about it after. And I had also just like loved making this thing. I thought it was so, so fun. And I loved talking about it, like getting to present it and like the story of how to make it faster was really fun. I got a bunch of advice from some friends on like the problems and some people helped me solve that.

But that was the first thing I think I made that I was like, oh, wow, like this is this is I think actually good. Meanwhile, Nolan is wrestling with his direction. I like having a plan. And when I don't feel like I have a plan or a goal, objective function, it's hard to know that I'm making progress. And I do need to be making progress.

I think was really, really challenging at points because I would wake up sometimes and be like, what am I doing? Like, what in the world? What's the goal? Like, what am I trying to do? Does anyone like me in any way? Like what? And I sometimes still feel that way and find that really stressful.

But then Flappy Durr was released and it went to the top of Hacker News and then it was on Daring Fireball and then GitHub. They have a YouTube channel and they covered it on that. From this experience, Nolan learns to refine his taste a bit. There is an appetite out there for these sorts of conceptual weird games. And so he keeps building. And then I made this game called Stranger Video, which I think is one of the things that I'm still probably the proudest of.

And Stranger Video was a website that showed you live video of a stranger's face until one of you blinked. And that's all it did. It used MediaPipe, which can do, it's a library from Google that has face detection, to crop video to only show faces and prevent you from showing anything else. And also to detect whether your eyes were opened or closed. That was it. Like it was just a website with silent video of somebody else on the website.

where all the restrictions were kind of coming from this place of like, I've used chat roulette. I understand what happens when you just have strangers look at each other on the internet and what they choose to show. And I don't want them to be able to show those things. I want just only faces. I was also kind of thinking at the time about like some social anxieties I have around like, when do I make eye contact with people that I've noticed far away and kind of wanted to channel that into a game.

So Nolan releases this game, and for some reason, it went viral in Japan, like really viral for a couple days. And then even some of his friends were very touched by it. Martin, one of his friends, said he never had so many people look at him and smile in his life. And that was like this profound, like, oh, wow, like that is, I've delivered, I've given Martin this like, I think genuinely cool novel experience and kind of opened my eyes to like a new thing, type of thing that I could be building.

People playing with the restrictions was really interesting. So your eyes needed to be opened, but you could put pieces of paper over your eyes that had little eyes drawn on them, and that tricked the camera. And like, if this was a competitive game, that'd be a problem. But for a weird social experiment, that's actually just funny and fine, I think.

I made this thing and I tried to figure out why it worked and why I liked it so much and how to do more things like it. And that was certainly a moment of validation and of like, okay, I'm proud of this and I want to make more things like this and people like it when I make these things and I think it has artistic value.

So the next constrained collaboration game was Talk Paper Scissors, the phone version of Rock Paper Scissors, but with strangers. So you like call a phone number. It asks you to say Rock Paper Scissors, use some voice detection so that you can't say anything but those words, plays your audio to someone else, plays their audio to you.

And that wasn't as much of a hit as Stranger video, but again, got, I don't know, like 50,000 calls or something. Like enough calls that I was kind of validated again. Like this is people like to play with strangers. There are not a lot of environments that are obviously restricted enough that you feel comfortable doing it. Both games thrive on constraints, right? They connect you to strangers, but they have limits on how you can interact.

And I think in particular had this theory of like the trick to this is constraints and the cool things that come from this are how people subvert those constraints that like you add constraints to constrain the assholes, but also constrain.

That gives people material for like funny things to do that wouldn't be possible with more of a sandbox. But since kind of the thesis here is like simple interaction with constraints that prevent jerks, it didn't really feel like something I could force. You need to come up with an interaction that you think is going to be compelling and fun and maybe silly. And you also need to come up with a way to constrain the obvious ways of abusing the action.

Stranger Video and Talk, Paper, Scissors weren't just games. They were experiments in constrained chaos. Nolan was almost designing for playful subversion, for people to find the edges and to push against them, just like he had in the early days. Meanwhile, he craved community. He craved that interaction that comes with building things together. So he went to NYU's ITP summer camp, which stands for like, I actually don't know, you

interactive telecommunications program or something. It's pretty old, so they probably use the word telecommunications. But it's a school that's focused on interesting artistic mixed media stuff, tech art things. It's like a real... You can get a master's there. I don't want to get a master's. That's a lot of time and money. But they have a summer camp that's like hang out there for three weeks with people that think it would be fun to hang out there for three weeks, which like...

I don't know, I probably am gonna get along with most of those people. So I did that this year because I didn't want to do Recurse again. At the summer camp, Nolan dove into face tracking tech. It was fun, but it was a lot of work and he couldn't release anything. And it left him feeling eager to just get something out the door. I think I feel really good when I'm regularly releasing stuff and when I'm not, I get kind of down on myself.

And I got lunch with my friend Neil, who makes the website Neil.fun, which is a really lovely collection of fun different games.

Infinite Craft is a very popular game of his from this year. Feels like you two should know each other. Yeah, we're both in Brooklyn. I think a surprising number of people that build funny stuff like this are from Brooklyn. And at the same time, that scene is not that big. It's feasible to know a lot of those people. But I think we originally met via a friend of mine from Recurse. And then, I don't know, we've got common interests and sometimes get together to bounce ideas off each other.

And I think explicitly the goal was we were bouncing ideas off of each other. We had originally planned to do kind of a Stranger Videos style thing together. And then he made Infinite Craft and that went crazy. Like that went completely insane. And so he was busy for months and I was doing other stuff.

And then that had come down. We're like, okay, we'll come back and we'll collaborate on this thing. And I pitched 1 million checkboxes just as kind of a joke. The pitch to Neil for 1 million checkboxes is simple. It's a website with a million checkboxes on them. If you check one, it's checked for everybody else. That's it. And we chatted about it for a bit. And we're kind of like, eh, this is funny, but I don't think there's that much there yet.

And I don't think either of us thought it was like a bad idea, but we were both kind of like, this is not the thing that we should collaborate on. But I thought it was funny and kind of interesting. And I knew I could build it really fast. Like I knew it was just not that much work. I know how WebSockets work. I know how to put checkboxes on a webpage.

And I was super hungry for like making something quick, right? Like I said, I hadn't shipped something in a while. And so near the end of the conversation, I was like, hey, do you mind if I just go build this? I think it'll take me a few days and then we can like collaborate on one of these other ideas like when that's done. And that was fine. After that Friday lunch, Nolan spent the weekend figuring out how to build 1 million checkboxes in a way that would feel responsive in real time. It's a lot of checkboxes, a million of them, and they need to stay all in sync across all the users.

How do I only render the checkboxes that you're looking at? I've never built an application that uses WebSockets of this scale. Like, can I sketch out an architecture that will handle the couple hundred people that I think will play and that I could scale up if it was a little more popular than that? And so I kind of came out of that weekend with a plan. And then I built the site in two days. I actually wrote the code the next Monday and Tuesday.

So on Wednesday, he launched it, the million checkboxes. And that's when things got interesting. I was excited about it. I thought it was funny. I was proud of like the constraints I had added in particular in ways we can talk about later made it difficult for people to draw because I understand what happens when you let people draw on the internet.

And so I was proud of all of that and thought, you know, it was evidence that I had kind of learned how to make websites, which wasn't true at the start of the year. So I was proud of that too. Nolan thought it was just a curiosity. Unlike Stranger Video, all you could do is check a box. There wasn't a lot to do, but somehow it took off. I think it was first Mastodon.

My theory is it was Mastodon, and then some big names on Mastodon posted about it, and that got several people to post it to Hacker News, and Hacker News collects votes if you repeatedly submit the same link in a quick succession, and that got it onto the front page there. And then I think once it was kind of popular, the fact that it was kind of popular drove the popularity. You went to the site, and...

Oh my god, like all these boxes are changing all the time. Like the site is so much more compelling because it's weirdly popular and that kept it more popular.

I don't know. I think the first three hours were me noticing, like, wow, there's some activity on the site. And then, like, oh, my God, there's really activity on the site. And then, oh, my God, my servers are going to, like, this is going somewhere. And if I don't start working, the server is going to fall over and I need to stop checking the load and start writing code.

And that was, I don't know, that was the morning. But it was a very fun, if still slightly confusing thing to wake up and see. That week marked Nolan's third at summer camp, and it was time to show off what he had made, his new spin on Pac-Man. And the camp show was that Wednesday night. You're supposed to be presenting something that you had worked on there. I was planning on presenting my Pac-Man game. And instead, the game had gone like

pretty viral by that point. And so I was manning my booth, trying to show people my game, but mostly like spinning up more servers, trying to keep the website online. How many people did you reach? So the numbers here are a little fuzzy for two different reasons. The first is that I originally, so I logged everything, but when I wrote the login code, I was like, there's no way that more than a million boxes are checked in a day.

So my logs will be truncated at a million lines a day because like that's an insanely high number. Like that's stupid. And I didn't think about that until like 5 p.m. or something. So I lost all my logs of what boxes people checked for the first several hours, which is a bummer. So my estimate of players from the first few days is from looking at my NGINX logs, which, you know, I have like IPs, etc. And I think there's something like 500,000 unique IPs there.

It's hard to know, you know, sometimes people are going to be like many users are buying the same static IP at a school or whatever. In the first few days, on average, we checked something like 50 million boxes a day, which is just totally, totally nuts and well beyond expectations.

50 million checkboxes, half a million people. These numbers are wild, right? Picture the largest stadium or music venue you've ever been to. It's more people than that. The largest stadium in the US per Wikipedia is Michigan's big house. And that's 100,000 people. We're talking 500 here. That's five times that many people over the course of 48 hours.

And these half a million people fall into two camps, right? There's the checkers who like to check things and watch the score go up, but then there's also the uncheckers, unchecking the boxes and driving the numbers down. I think you can look at media coverage of the site and plenty of the time it'll say something like, "It's a website where the goal is to check all the boxes." And I'm like, "I didn't say that. I just said, here's the check boxes, do whatever you want."

And it's fine that that's your goal, but that's your own goal. But I think that that was part of the appeal here is that I think in general, people are much more attached to goals that they come up with themselves and saying, here's the thing, do whatever you want with it. And you deciding my goal is to prevent all the boxes from getting checked, or my goal is to prevent box number two from being checked ever, or I want to check every box.

pretty quickly people start having their own stories about how they're using the site. So an example of that is I would often hear from someone who would find a place where someone was like checking a bunch of boxes and they would follow that person and uncheck the boxes.

And they'd be like, yeah. And like, they were getting so mad, right? Like I was trolling them and they're getting so bad. And again, it's like, you don't actually know that, right? Like you have no idea whether they're noticing what you're doing. Like you have no idea what's going on with that person. Maybe they find it funny. But because you have such limited information, it's really easy for you to like play pretend in this way. And that's just super fun. Like you get really into that. So I think it really let people tell those stories in fun ways.

Okay, let's get into the weeds for a minute. Imagine managing a million pieces of shared mutable state with tens of thousands of reads and writes per second. It's a scaling challenge. It needs to feel real time to engage users. And plus, it has some kind of aggregate data. You need to have a count of the number of things each user has flipped and then a total. Seriously, if you've ever had to stop and scale a data store or a database, stop and think about how you would handle this for a minute. What

what's the hard part? To me, the issue is the number of writes. If you're going to use a traditional database, it's a lot of concurrent writes. And some of the rows, depending on how you organize the data, we're going to be really hot. Those first hundred checkboxes, I'm sure, are where a lot of people are clicking. So then I start thinking about something in memory, but how do you keep all this state in lockstep? Yeah. Yeah. The architecture was like, Redis has a bit set, just roughly a million bits.

with some nice operations for flipping individual bits. So Redis has that. Redis has a count of the number of boxes checked, stored separately, just so that we don't have to iterate through a million things to determine how many are checked, and the website's supposed to display that.

Side note, Redis also, you can write Lua and invoke it via Redis atomically, which is really sick. So like, you really don't have to worry about races as long as you write your logic in Lua scripts. So it was really easy to say like, oh, we're storing this count separately from the number of checkboxes. There's an obvious race there kind of, you know, depending on the primitives you have. But like, let's not even think about it at all.

Redis has a Pub/Sub feature. You can flip a bit, bump a counter, and then broadcast the change all in one Lua script. And the rough architecture was you had a bunch of workers whose job were to accept WebSocket connections from people on the website. They offered an API that let clients flip bits along with some basic rate limiting. There was also some client rate limiting, but that's obviously trivial to avoid.

And then whenever a client flipped a bit, you would write that to the PubSub. And then every WebSocket worker process was also reading from that PubSub. And whenever something got written there, would update all of the clients that were connected to that individual instance. This is smart, right? State management is atomic and fully outsourced to Redis. But scaling the workers to handle the WebSockets, that's still a challenge. And then there are other things that Nolan didn't even think about.

I think it was night one when I realized I hadn't thought about bandwidth cost. I was managing all my servers myself. And so I had originally decided, okay, like my costs are capped. Like I know how much my servers cost. I'm spinning up a bunch of servers. Like this is probably going to cost me like a couple hundred bucks. I'm getting some donations. That's probably fine. And then remember that, you know, you get charged for bandwidth and I'm sending like

weirdly large amount of data, especially at this point, I hadn't thought about how to reduce the size of these updates that I was sending clients. And so I was pushing out, you know,

hundreds of megabits a second or something. So Nolan started doing the math and estimating costs, and the math gave him an estimate of $10,000 for his bandwidth so far. And like the pit in my stomach, right? Like it just felt terrible. But I called a friend, my friend Greg, who I worked on TalkPaperScissors with, and he talked me through it. And we agreed that my math was wrong. But also I started working on like, I can vastly reduce the amount of data I'm sending. That's going to help a lot.

And two, I happened to be pretty comfortable using the Linux command line utility TC, which stands for traffic control, and is an obscure utility for limiting bandwidth on instances. My job back at Amazon Bud Services a decade ago involved using it a lot. And so I was like, oh, I can just add a rule that drops all outbound traffic above, I think it was 250 megabits a second. And that let me calculate an upper bound, right? And that was what really mattered to me was like,

The upper bound was kind of high, but DigitalOcean's bandwidth pricing is actually quite sane. So that was the one time that I was really scared. Beyond that, I was delighted and baffled and excited and super motivated to keep the site up and keep working on it and improve it in different ways.

I had made a mistake that I often make when I make things like this, which was I had very little attribution to myself or any self-promotion when I made it. I don't even know that there was a link to my website. There was certainly not a link to my mailing list or a way to donate me money or anything like that. Nolan added links to his website and his mailing list, and just like that, the media came calling. And I did the interview with The Washington Post.

which was fun. They asked me how I was monetizing it. I was like, not really. But I think in the question, they were like, are you going to sell this to the New York Times like Wordle? And I made a joke about like, I will sell it to the New York Times for $1 a checkbox.

And like 12 hours later, I got an email from the New York Times, which quickly came with a follow-up email saying, we're so sorry, this is not an offer to buy the site. But it was just someone there who also wanted to talk to me. And I did all that. And that was super fun. That was the first time I'd ever done any media like that. And pretty validating. Like it was very convenient for the family in that they don't really understand what like Hacker News is or whatever. But like, hey, I'm in the New York Times is like that. That is meaningful to them.

Next up, Nolan needs to boost the streaming server's performance. So I had a friend from the startup that I used to work at back in Michigan, who these days is like a performance engineer at Facebook, who is very good at making things fast. And we've been wanting to work on something together for years and hadn't had the time. And I was like, one, you really want to work on this problem. And two, you're not going to be able to do it.

Like, I actually really, really need help here. So I called him up and went over to his apartment at like 2 p.m. on a Sunday, and we worked until 2 a.m. on a Sunday. Part of the big rewrite was moving from an untyped Python flask app to Go. During the rewrite, we noticed at several points we screwed up our rate limits in one way or another.

there were a couple of pretty embarrassing bugs. It was, you know, like we'd just been working on this thing for a long time. It was getting really late. And so with one of the first versions of the Go rewrite we launched, the site went totally crazy because we had broken our rate limits and like Redis briefly broke, like we were sending Redis too many updates. That was crazy. Normally, you could only send a limited number of requests per IP. Otherwise, you could just open developer tools and write a little for loop and start looping through and checking all the boxes.

But were people actually hitting that limit? Were people writing bots and hitting the API? You know, sometimes you'd go somewhere and like 30 boxes would clear all at the same time and you'd be like, that's not a human. But I think the scale of it wasn't clear until we removed the rate limits for a little bit and the bots went crazy. And during the rewrite, I had noticed briefly some data in Redis that looked weird.

And by looked weird, I mean it contained URLs. Nolan had spotted some weird data before. In fact, he had had the site crash due to weird data on the first big night. And the way that it crashed is that I wasn't validating indices. So I had an API that the clients used to checker and check a box, and it took an int. And I didn't validate that the int was in the range 0 to 1 million.

And someone was like, what if I manually call this and send 10 million and check that box? And that totally worked. And Redis's behavior around the data structure that I was using was like, oh, I was told to check bit number 10 million. I have a million bits. I guess I need to add 9 million bits of padding to check bit number 10 million, right? So that happened. The site crashed. They did all sorts of weird things. And I...

woke up to this and quickly figured out what was going wrong, but made a couple of utilities to dump the database. And my goal then was like, I will bring the site back up and I will just truncate the bit set to the first million bits. So I had these scripts that dumped the data in the database. And they happened to, just based on what Reddit does when you ask it to display this data structure, convert that data to ASCII.

So you have 125,000 bytes, right? A million over eight. And each of those bytes is a number and we look it up and decide what ASCII character is. And like that is what Redis prints unless you're asking it to print a character not in the ASCII character range because ASCII is only what like 127. Otherwise you get like a hex digit approximately. Yeah.

While running the script to dump checkboxes, Nolan and his friend Elliot spot the URL in the data. But they have bigger priorities to fix. And I saw that while Elliot and I were working on it. And...

We were in the middle of this rewrite from Python to Go. My Python had no types. It was wildly unsafe in all of these ways. We noticed it briefly and we're like, oh, you have another bug around not validating user input. But I thought that I had a data validation error that let people stuff strings into this bit set because I hadn't really even thought about what it was that my script did at this point.

And so we notice this and we're like, well, we're rewriting this in a much more type safe way. Like this bug will just be fixed. So like, don't, we don't, we don't have that much to worry about. Right. With the new service up and running, they breathe a sigh of relief and they call it a night.

And then it got like hilariously faster with the Go rewrite. I have these CPU graphs that were pegged at 100. And then like we launched the Go version and they're pegged at, you know, one. Like there's just nothing, you know, it's ridiculous. And I woke up the next day and I ran the script again, I guess just because like it was still kind of bugging me. Like I had noticed these URLs. It seemed like maybe I had been not quite hacked necessarily, but like,

Something bad was going on. And I ran the script again. And I noticed that the URLs were still there. And I was like messaging him being like, they're still there. We're going back and forth. And we're like, this shouldn't like we're type safe. Like there's no like obvious like you're just injecting strings into my database bug here. What is going on? And then I remembered something, which is that I had gotten a bunch of DMs on Twitter about OMCB.

And many of them were just kind of like, hey, I like this thing. But one had been from someone who was like, hey, can I help you rewrite the site to make it faster? I am friends with a bunch of people who are good at writing performant code. And I was like, oh, maybe, I don't know. I like collaborating with online strangers and giving you access to my code base. It's a little sketchy, but that is a very sweet offer. And then he had sent me this other message that I had no idea what to make of at the time that was like,

Look at the end of the site or something. A friend of mine left you a message. And I like that message popped back into my head. And I had no like, I think I had sent him like question mark, question mark, question mark at the time. Like, what in the world are you talking about? That's like such a weird thing to say. But realized like, okay, maybe there is something there. This all happened on Monday evening. And Nolan had started to connect the dots in his head, but also his girlfriend had made him dinner.

And I had kind of gotten re-obsessed with what is going on here right as dinner finished. And she was like, can we sit down and eat? You've been very busy recently. And I was panicking about, there are these URLs. I have been hacked. I have been hacked. And had decided, okay, I'll stop thinking about this for a second. And sat down and then had this thought. And it all came together.

And then like rush back to my computer to like, you know, confirm what was going on. And I think like all the pieces kind of came into place, like started like looking at the actual site. Oh my God, there is a pattern here. Oh, holy shit. Like they're not, the data is in my database because what they are doing is checking and unchecking boxes.

to make the binary representation of this when interpreted as ASCII spell out this URL. Like this isn't hacked, this isn't I have a bug, this is like someone who is being very persistent about writing a weird message into my database. The only way this could happen is with a bot, right? Any visitor can check or uncheck a checkbox, but this URL is always there.

And encoding that URL takes a lot of checkboxes in a row, right? 192 for this specific URL. That implies a bot is enforcing the data in this very specific part of the checkbox grid. And the URL linked to a Discord.

It took me like five minutes to convince myself that I could click it since it was catgirls.win slash OMCB. And that seemed super sketchy, but I did everything I could to convince myself that it was a real website, that it was safe to visit, and then decided that I couldn't convince myself of that, but that I would regret it forever if I didn't click the link. So I just had to click the link anyway. And I clicked the link, went to a Discord. Discord is called Checkinboxes. And it was full of teens talking about botting the site.

who had been kind of gathering because they were all looking at this data, right? Like they were all, they had all like figured out how my API worked. They were writing tools that hit the API and the command line, looking at either the base 64 or ASCII representation of the data for one reason or another, and had all noticed these URLs and had joined the discord of, I think, one of the first people to start botting the site.

That's awesome. How many people were in the Discord? I think there were about 20 when I joined, and then by the time I shut the site down, about 60, which is just mind-blowingly cool. You know,

not remotely what I expected. It's kind of impossible to convey the feeling of going from, I have been hacked, this data is still here. I think it felt worse because I thought that I understood, yeah, I was kind of owned, someone had stuffed these strings in, but then we fixed it. And then, oh my god, I haven't fixed it. Maybe somebody has root online boxes, something is really bad. To wait, there's this whole community and they're doing something that is insanely cool.

Nolan created 1 million checkboxes without a fixed grid because he wanted to stop people from drawing offensive images. The number of checkboxes wide was device specific, so you could draw whatever you wanted, but most people wouldn't see it. This tied into his idea of curbing bad behavior. But if you mass check and uncheck boxes and build a CLI app and somehow write to the bit set as ASCII, you find a URL.

And it leads you to a Discord. And just like that, a community forms. And their big goal, in addition to gathering more people, was drawing on the site, which they were doing by treating it as a thousand-by-thousand canvas.

So they would have a bot that like it would take in an image and coordinates and a size. And from that determine the set of checkboxes that would need to be checked to like form that image on this hypothetical thousand by thousand canvas that they were drawing onto. And then listen to events and check and uncheck boxes to make sure that that image stayed drawn on the canvas, which was just insane and so, so cool. So does it farm out the work to multiple of them or are they each? I think within the Discord, people had a sense of who is drawing what, which.

So they solved the problem of like, obviously, if you run two of these bots on the same territory, you just check and uncheck boxes forever and nothing ever gets drawn and my server crashes.

But for the most part, almost everyone who was drawing stuff was in the server. There was one exception of some people that were trying to draw the logo for Osu! the game who weren't in the server and were fighting over territory with the people on the server. And that was pretty funny to see. And then the other big problem for them was I had rate limits, right? Like I rate limited people based on IPs. And you could work around them by getting more IPs.

And so the people that were like the fastest and able to draw the biggest things were the people that could come up with the best ways to get a bunch of IPs to send their traffic from. And these are like kids mostly. They're like mostly, you know, like 18. So they don't necessarily just have like

a bunch of money to like rent a bunch of servers. So they were coming up with clever, like there are ways to route your traffic through residential IPs. They were using a bunch of stuff like that. So that was kind of the big constraint for them was having enough IPs and coordinating which check events you farm out to which IPs you need, whatever, a central server that's like listening for those events and then farming out the work to a bunch of different workers, which it's like kind of an impressive amount of infrastructure to spin up for something like this. So what were they drawing?

When I joined, they were drawing really simple stuff. So they had, I think the first thing they wrote was "be gay, do crime." And then after that, when I joined, they had a QR code that links to the Discord, which I thought was really funny. But they started getting more and more complicated over time. And so one way that they got more complicated is they started coming up with big detailed images to draw. They did a Windows blue screen in death, which I thought was really funny.

But all sorts of things that they would try to draw over the vast majority of the grid, which just took more technical work. And I say vast majority there because they were very careful to not touch the top few or bottom few rows because they figured that's where most people were playing and they knew they were kind of degrading the experience, which I thought was pretty sweet of them. But then the big innovation later was, oh, we should start doing GIFs.

And like, what are animated images that we could put on? So obviously we got, we got some Rick rolls at some points. We got a lot of Jake Gyllenhaal gifts. There's, do you know bad apple? No. So there's this thing, and I don't quite understand this phenomenon to be honest, but there is a video called bad apple. It is a cool, interesting, like anime video thing. But the point is it's a meme to make bad apple run inside stuff.

Kind of like how with computers it's always like, "Can I run Doom on this thing?" It's like, "Can I run Bad Apple inside this thing?" So there was a project, Bad Apple, that a lot of people were devoted to. And I actually hadn't thought that they had succeeded, and then when I looked back through all the data after building some of my own tools to inspect it, I found that someone had successfully run Bad Apple on it. So that's cool and exciting. Now, all that's being concerned is figuring out the right way to connect with this community.

I didn't want to ruin it. I think it was very fun that I was there. And they were excited. They had a lot of questions for me. They had found bugs, whatever. They wanted to talk to me. But at the same time, I did think this is your fun little hangout for you and your mostly teen friends. And I am a 33-year-old software developer. And we're in very different life stages. And you should have your own space.

So I did kind of try to like pop in, answer questions. I would pop in and make announcements like, hey, an interview with me is running in the Washington Post. There will be new users. Please like limit your botting for a little bit, things like that. And they were happy to listen to that, which was great. But they had a bunch of questions about the site, about how I was solving things, and just kind of like about my deal. Like I think they're just like interested and I was happy to talk about that.

In a few cases, people have gotten in touch with me somehow to ask whether any of the teens in the Discord are looking for jobs. And so I have relayed some job offers to them, a couple of things like that. But yeah, we've chatted a fair amount and they're great. I don't know. I'm so proud that they decided to point their talents at my website. And I think they're just crazy talented.

For Nolan, just knowing this group existed changed things. This felt like it really validated this thesis that I had had from back when I made Stranger Video and other things. I had had this belief for a while based on these smaller but still moderately successful stranger interaction things that the way to approach a problem like this, or a way that I think works well, is to have a small, fun, novel interaction with enough constraints that

people can't, you know, trivially ruin it, and that those constraints produce really fun, interesting behavior. And this just felt like the purest possible validation of that thesis, that like, I believed this, I had given some talks where I believed this, and I said it in the talk, and I think the talks were okay. But like, I didn't have enough proof.

And this was like all of the proof I would ever need. This was so much of a reason to continue pursuing projects with this flavor. Yeah, it was validating to kind of like this core thing that I knew to be true but needed to prove in a bigger way. And that was also really, really thrilling. And I don't know, we've talked a couple of times about clarity. Yeah.

of what to pursue and, you know, what I know how to do and what the goal is. And I think that was another step in that direction. It's funny how this direction that Nolan has refined and this direction that he's still on, it all started off with that playful exploration and boundary pushing around email rules and how they were handled back in high school. Because back then, he was the one who was having the trolling behavior that other people saw potential in.

And it wasn't just his dorm supervisor person who created him a t-shirt because he also emailed his high school computer science teacher about the rules he was creating. Like, I don't think I knew what Turing completeness was and I'm not sure that the rules were actually Turing complete, but I was kind of like, this is way too powerful for a school email system. But like my memory of it is basically him being like, don't mess everything up. Like, don't, you know, like that's pretty funny. Like, I'm sure he was like kind of proud.

Yeah, I don't know. I think back to that sometimes in terms of like, wow, I really, you could easily imagine a world where the response by the school was like, you're expelled, you like broke a bunch of stuff. And the fact that instead, I was kind of mischievous, but it was kind of funny. And it was like, ultimately pretty low stakes.

and people just kind of were encouraging was really nice. There are a lot of versions of trolling that are really fun. If you go back to the version of the internet that we had in 2008, I think there are lots of places where people did some kind of like large-scale collective trolling that is remembered very fondly, like 4chan rigging polls to spell out funny things. The

The thing that was the most emotionally satisfying to me was certainly discovering this and then just being so proud and excited of them and of myself for making something that they thought was worth their time. Getting to be proud of them and not mad at them was so meaningful to me. I cried a lot about it, which is not something I normally do.

I think that when people think about trolls now, this is more like, I don't know, people being really mean on Twitter. But providing people a very low stakes way to troll where the worst thing you can do is uncheck a checkbox is great, right? It's fun to do that. It's fun to prank somebody in some small way and it's totally okay. And I don't know, I think people really enjoy it when you give them a chance to do that. That's it. That's the show.

Nolan's had multiple projects spreading around the web since we last talked. He's really good at this. Regex Bad Apple is that bad apple animation he previously mentioned, but playing inside Vim using regular expressions. EveryUUID.com is a website with everyUUID on it.

somehow. So thank you, Nolan, for pushing the boundaries. You're clearly one of the world's experts at twisting things beyond their intended use for our amusement. Go to his website, eieio.games, and he can send you a newsletter whenever his new creations come out. And thank you...

especially to the supporters who keep me at this. The main benefit of being a supporter is literally showing your support. And I do appreciate that. But there is also a Slack channel and some bonus episodes. Really just thank you everybody who joins the Patreon supporters group just to show that they support what I'm doing here. And yeah, we also have a pretty awesome Slack channel that you can find on the website.

So I hope you liked this one. And until next time, thank you so much for listening.