Our anonymous submitter, whom we'll call Craig, worked for GlobalCon. GlobalCon relied on an offshore team on the other side of the world for adding/removing users from the system, support calls, ticket tracking, and other client services. One day at work, an urgent escalated ticket from Martin, the offshore support team lead, fell into Craig's queue. Seated before his cubicle workstation, Craig opened the ticket right away:

The new GlobalCon support website is not working. Appears to have been taken over by ChatGPT. The entire support team is blocked by this.

Instead of feeling any sense of urgency, Craig snorted out loud from perverse amusement.

"What was that now?" The voice of Nellie, his coworker, wafted over the cubicle wall that separated them.

"Urgent ticket from the offshore team," Craig replied.

"What is it this time?" Nellie couldn't suppress her glee.

"They're dead in the water because the new support page was, quote, taken over by ChatGPT."

Nellie laughed out loud.

"Hey! I know humor is important to surviving this job." A level, more mature voice piped up behind Craig from the cube across from his. It belonged to Dana, his manager. "But it really is urgent if they're all blocked. Do your best to help, escalate to me if you get stuck."

"OK, thanks. I got this," Craig assured her.

He was already 99.999% certain that no part of their web domain had gone down or been conquered by a belligerent AI, or else he would've heard of it by now. To make sure, Craig opened support.globalcon.com in a browser tab: sure enough, it worked. Martin had supplied no further detail, no logs or screenshots or videos, and no steps to reproduce, which was sadly typical of most of these escalations. At a loss, Craig took a screenshot of the webpage, opened the ticket, and posted the following: Everything's fine on this end. If it's still not working for you, let's do a screenshare.

Granted, a screensharing session was less than ideal given the 12-hour time difference. Craig hoped that whatever nefarious shenanigans ChatGPT had allegedly committed were resolved by now.

The next day, Craig received an update. Still not working. The entire team is still blocked. We're too busy to do a screenshare, please resolve ASAP.

Craig checked the website again with both laptop and phone. He had other people visit the website for him, trying different operating systems and web browsers. Every combination worked. Two things mystified him: how was the entire offshore team having this issue, and how were they "too busy" for anything if they were all dead in the water? At a loss, Craig attached an updated screenshot to the ticket and typed out the best CYA response he could muster. The new support website is up and has never experienced any issues. With no further proof or steps to reproduce this, I don't know what to tell you. I think a screensharing session would be the best thing at this point.

The next day, Martin parroted his last message almost word for word, except this time he assented to a screensharing session, suggesting the next morning for himself.

It was deep into the evening when Craig set up his work laptop on his kitchen counter and started a call and session for Martin to join. "OK. Can you show me what you guys are trying to do?"

To his surprise, he watched Martin open up Microsoft Teams first thing. From there, Martin accessed a chat to the entire offshore support team from the CPO of GlobalCon. The message proudly introduced the new support website and outlined the steps for accessing it. One of those steps was to visit support.globalcon.com.

The web address was rendered as blue outlined text, a hyperlink. Craig observed Martin clicking the link. A web browser opened up. Lo and behold, the page that finally appeared was www.chatgpt.com.

Craig blinked with surprise. "Hang on! I'm gonna take over for a second."

Upon taking control of the session, Craig switched back to Teams and accessed the link's details. The link text was correct, but the link destination was ChatGPT. It seemed like a copy/paste error that the CPO had tried to fix, not realizing that they'd needed to do more than simply update the link text.

"This looks like a bad link," Craig said. "It got sent to your entire team. And all of you have been trying to access the support site with this link?"

"Correct," Martin replied.

Craig was glad he couldn't be seen frowning and shaking his head. "Lemme show you what I've been doing. Then you can show everyone else, OK?"

After surrendering control of the session, Craig patiently walked Martin through the steps of opening a web browser, typing support.globalcon.com into the header, and hitting Return. The site opened without any issue. From there, Craig taught Martin how to create a bookmark for it.

"Just click on that from now on, and it'll always take you to the right place," Craig said. "In the future, before you click on any hyperlink, make sure you hover your mouse over it to see where it actually goes. Links can be labeled one thing when they actually take you somewhere else. That's how phishing works."

"Oh," Martin said. "Thanks!"

The call ended on a positive note, but left Craig marveling at the irony of lecturing the tech support lead on Internet 101 in the dead of night.

It's a holiday in the US today, so we're taking a long weekend. We flip back to a classic story of a company wanting to fill 15 different positions by hiring only one person. It's okay, Martin handles the database. Original - Remy

A curious email arrived in Phil's Inbox. "Windows Support Engineer required. Must have experience of the following:" and then a long list of Microsoft products.

Phil frowned. The location was convenient; the salary was fine, just the list of software seemed somewhat intimidating. Nevertheless, he replied to the agency saying that he was interested in applying for the position.

A few days later, Phil met Jason, the guy from the recruitment agency, in a hotel foyer. "It's a young, dynamic company", the recruiter explained,"They're growing really fast. They've got tons of funding and their BI Analysis Suite is positioning them to be a leading player in their field."

Phil nodded. "Ummm, I'm a bit worried about this list of products", referring to the job description. "I've never dealt with Microsoft Proxy Server 1.0, and I haven't dealt with Windows 95 OSR2 for a long while."

"Don't worry," Jason assured, "The Director is more an idea man. He just made a list of everything he's ever heard of. You'll just be supporting Windows Server 2003 and their flagship application."

Phil winced. He was a vanilla network administrator – supporting a custom app wasn't quite what he was looking for, but he desperately wanted to get out of his current job.

A few days later, Phil arrived for his interview. The company had rented smart offices on a new business park on the edge of town. He was ushered into the conference room, where he was joined by The Director and The Manager.

"So", said The Manager. "You've seen our brochure?"

"Yeah", said Phil, glancing at the glossy brochure in front of him with bright, Barbie-pink lettering all over it.

"You've seen a demo version of our application – what do you think?"

"Well, I think that it's great!", said Phil. He'd done his research – there were over 115 companies offering something very similar, and theirs wasn't anything special. "I particularly like the icons."

"Wonderful!" The Director cheered while firing up PowerPoint. "These are our servers. We rent some rack space in a data center 100 miles away." Phil looked at the projected picture. It showed a rack of a dozen servers.

"They certainly look nice." said Phil. They did look nice – brand new with green lights.

"Now, we also rent space in another data center on the other side of the country," The Manager added.

"This one is in a former cold-war bunker!" he said proudly. "It's very secure!" Phil looked up at another photo of some more servers.

"What we want the successful applicant to do is to take care of the servers on a day to day basis, but we also need to move those servers to the other data center", said The Director. "Without any interruption of service."

"Also, we need someone to set up the IT for the entire office. You know, email, file & print, internet access – that kind of thing. We've got a dozen salespeople starting next week, they'll all need email."

"And we need it to be secure."

"And we need it to be documented."

Phil was scribbled notes as best he could while the interviewing duo tag teamed him with questions.

"You'll also provide second line support to end users of the application."

"And day-to-day IT support to our own staff. Any questions?"

Phil looked up. "Ah… which back-end database does the application use?" he asked, expecting the answer would be SQL Server or perhaps Oracle, but The Director's reply surprised him.

"Oh, we wrote our own database from scratch. Martin wrote it." Phil realized his mouth was open, and shut it. The Director saw his expression, and explained. "You see, off the shelf databases have several disadvantages – the data gets fragmented, they're not quick enough, and so on. But don't have to worry about that – Martin takes care of the database. Do you have any more questions?"

Phil frowned. "So, to summarize: you want a data center guy to take care of your servers. You want someone to migrate the application from one data center to another, without any outage. You want a network administrator to set up, document and maintain an entire network from scratch. You want someone to provide internal support to the staff. And you want a second line support person to support the our flagship application."

"Exactly", beamed The Director paternally. "We want one person who can do all those things. Can you do that?"

Phil took a deep breath. "I don't know," he replied, and that was the honest answer.

"Right", The Manager said. "Well, if you have any questions, just give either of us a call, okay?"

Moments later, Phil was standing outside, clutching the garish brochure with the pink letters. His head was spinning. Could he do all that stuff? Did he want to? Was Martin a genius or a madman to reinvent the wheel with the celebrated database?

In the end, Phil was not offered the job and decided it might be best to stick it out at his old job for a while longer. After all, compared to Martin, maybe his job wasn't so bad after all.

We talked about singletons a bit last week. That reminded John of a story from the long ago dark ages where we didn't have always accessible mobile Internet access.

At the time, John worked for a bank. The bank, as all banks do, wanted to sell mortgages. This often meant sending an agent out to meet with customers face to face, and those agents needed to show the customer what their future would look like with that mortgage- payment calculations, and pretty little graphs about equity and interest.

Today, this would be a simple website, but again, reliable Internet access wasn't a thing. So they built a client side application. They tested the heck out of it, and it worked well. Sales agents were happy. Customers were happy. The bank itself was happy.

Time passed, as it has a way of doing, and the agents started clamoring for a mobile web version, that they could use on their phones. Now, the first thought was, "Wire it up to the backend!" but the backend they had was a mainframe, and there was a dearth of mainframe developers. And while the mainframe was the source of truth, and the one place where mortgages actually lived, building a mortgage calculator that could do pretty visualizations was far easier- and they already had one.

The client app was in .NET, and it was easy enough to wrap the mortgage calculation objects up in a web service. A quick round of testing of the service proved that it worked just as well as the old client app, and everyone was happy - for awhile.

Sometimes, agents would run a calculation and get absolute absurd results. Developers, putting in exactly the same values into their test environment wouldn't see the bad output. Testing the errors in production didn't help either- it usually worked just fine. There was a Heisenbug, but how could a simple math calculation that had already been tested and used for years have a Heisenbug?

Well, the calculation ran by simulation- it simply iteratively applied payments and interest to generate the entire history of the loan. And as it turns out, because the client application which started this whole thing only ever needed one instance of the calculator, someone had made it a singleton. And in their web environment, this singleton wasn't scoped to a single request, it was a true global object, which meant when simultaneous requests were getting processed, they'd step on each other and throw off the iteration. And testing didn't find it right away, because none of their tests were simulating the effect of multiple simultaneous users.

The fix was simple- stop being a singleton, and ensure every request got its own instance. But it's also a good example of misapplication of patterns- there was no need in the client app to enforce uniqueness via the singleton pattern. A calculator that holds state probably shouldn't be a singleton in the first place.

[Advertisement] Utilize BuildMaster to release your software with confidence, at the pace your business demands. Download today!

Barry rolled into work at 8:30AM to see the project manager waiting at the door, wringing her hands and sweating. She paced a bit while Barry badged in, and then immediately explained the issue:

Today was a major release of their new features. This wasn't just a mere software change; the new release was tied to major changes to a new product line- actual widgets rolling off an assembly line right now. And those changes didn't work.

"I thought we tested this," Barry said.

"We did! And Stu called in sick today!"

Stu was the senior developer on the project, who had written most of the new code.

"I talked to him for a few minutes, and he's convinced it's a data issue. Something in the metadata or something?"

"I'll take a look," Barry said.

He skipped grabbing a coffee from the carafe and dove straight in.

Prior to the recent project, the code had looked something like this:

if (IsProduct1(_productId))
	_programId = 1;
elseif (IsProduct2(_productId))
	_programId = 2;
elseif (IsProduct3(_productId))
	_programId = 3;

Part of the project, however, was about changing the workflow for "Product 3". So Stu had written this code:

if (IsProduct1(_productId))
	_programId = 1;
else if (IsProduct2(_productId))
	_programId = 2;
else if (IsProduct3(_productId))
	_programId = 3;
	DoSomethingProductId3Specific1();
	DoSomethingProductId3Specific2();
	DoSomethingProductId3Specific3();

Since this is C# and not Python, it took Barry all of 5 seconds to spot this and figure out what the problem was and fix it:

if (IsProduct1(_productId))
{
	_programId = 1;
}
else if (IsProduct2(_productId))
{
	_programId = 2;
}
else if (IsProduct3(_productId))
{
	_programId = 3;
	DoSomethingProductId3Specific1();
	DoSomethingProductId3Specific2();
	DoSomethingProductId3Specific3();
}

This brings us to about 8:32. Now, given the problems, Barry wasn't about to just push this change- in addition to running pipeline tests (and writing tests that Stu clearly hadn't), he pinged the head of QA to get a tester on this fix ASAP. Everyone worked quickly, and that meant by 9:30 the fix was considered good and ready to be merged in and pushed to production. Sometime in there, while waiting for a pipeline to complete, Barry managed to grab a cup of coffee to wake himself up.

While Barry was busy with that, Stu had decided that he wasn't feeling that sick after all, and had rolled into the office around 9:00. Which meant that just as Barry was about to push the button to run the release pipeline, an "URGENT" email came in from Stu.

"Hey, everybody, I fixed that bug. Can we get this released ASAP?"

Barry went ahead and released the version that he'd already tested, but out of morbid curiosity, went and checked Stu's fix.

if (IsProduct1(_productId))
	_programId = 1;
else if (IsProduct2(_productId))
	_programId = 2;
else if (IsProduct3(_productId))
{
	_programId = 3;
}

if (IsProduct3(_productId))
{
	DoSomethingProductId3Specific1();
	DoSomethingProductId3Specific2();
	DoSomethingProductId3Specific3();
}

At least this version would have worked, though I'm not sure Stu fully understands what "{}"s mean in C#. Or in most programming languages, if we're being honest.

With Barry's work, the launch went off just a few minutes later than the scheduled time. Since the launch was successful, at the next company "all hands", the leadership team made sure to congratulate the people instrumental in making it happen: that is to say, the lead developer of the project, Stu.

Today is holiday in the US, where we celebrate a cosplay version of history with big meals and getting frustrated with our family. It's also a day where we are thankful- usually to not be at work, but also, thankful to not work with Brad. Original --Remy

Anita parked outside the converted garage, the printed graphic reading Global Entertainment Strategies (GES) above it. When the owner, an old man named Brad, had offered her a position after spotting her in a student computer lab, she thought he was crazy, but a background check confirmed everything he said. Now she wondered if her first intuition was correct.

“Anita, welcome!” Brad seemed to bounce like a toddler as he showed Anita inside. The walls of the converted garage were bare drywall; the wall-mounted AC unit rattled and spat in the corner. In three corners of the office sat discount computer desks. Walls partitioned off Brad’s office in the fourth corner.

He practically shoved Anita into an unoccupied desk. The computer seemed to be running an unlicensed version of Windows 8, with no Office applications of any kind. “Ross can fill you in!” He left the office, slamming the door shut behind him.

“Hi.” Ross rolled in his chair from his desk to Anita’s. “Brad’s a little enthusiastic sometimes.”

“I noticed. Uh, he never told me what game we’re working on, or what platform. Not even a title.”

Ross’s voice lowered to a whisper. “None of us know, either. We’ve been coding in Unity for now. He hired you as a programmer, right? Well, right now we just need someone to manage our documentation. I suggest you prepare yourself.”

Ross led Anita into Brad’s office. Above a cluttered desk hung a sagging whiteboard. Every square inch was covered by one, sometimes several, overlapping sticky notes. Each had a word or two written in Brad’s scrawl.

“We need more than just random post-its with ‘big guns!’ and ‘more action!’” Ross said. “We don’t even know what the title is! We’re going crazy without some kind of direction.”

Anita stared at the wall of sticky notes, feeling her sanity slipping from her mind like a wet noodle. “I’ll try.”

Sticky Escalation

Brad, can we switch to Word for our documentation? It’s getting harder
to read your handwriting, and there’s a lot of post-its that have
nothing to do with the game. This will make it easier to proceed with
development. -Anita

Two minutes after she sent the email, Brad barged out of his office. “Anita, why spend thousands of dollars on software licenses when this works just fine? If you can’t do your job with the tools you have, what kind of a programmer does that make you?”

“Brad, this isn’t going to work forever. Your whiteboard is almost out of room, and you won’t take down any of your non-game stickies!”

“I can’t take any of them down, Anita! Any of them!” He slammed the door to his office behind him.

The next day, Anita was greeted at the door by the enthusiastic Brad she had met before the interview. “I listened to reason, Anita. I hope this is enough for you to finish this documentation and get coding again!”

Brad led Anita into his office. On every wall surface, over the door, even covering part of the floor, were whiteboards. Sticky notes dotted nearly a third of the new whiteboard space.

“Now, Anita, if I don’t see new code from you soon, I may just have to let you go! Now get to work!”

Anita went to sit at her desk, then stopped. Instead, she grabbed a bright red sticky note, wrote the words “I QUIT” with a sharpy, barged into Brad’s office, and stuck it to his monitor. Brad was too stunned to talk as she left the converted garage.

The Avalanche

“Are you doing better?” Jason called Anita a few weeks later. Their short time together at GES has made them comrades-in-arms, and networking was crucial in the business.

“Much,” she said. “I got a real job with an indie developer in Santa Monica. We even have a wiki for our framework!”

“Well, listen to this. The day after you quit, the AC unit in the garage broke. I came into work to see Brad crying in a corner in his office. All of the sticky notes had curled in the humidity and fallen to the floor. The day after he got us all copies of Word.

“Too bad we still don’t know what the title of the game is.”

[Advertisement] ProGet’s got you covered with security and access controls on your NuGet feeds. Learn more.

One of the clients for Rudolf's company was getting furious with them. The dev team was in constant firefighting mode. No new features ever shipped, because the code-base was too fragile to add new features to without breaking something else. What few tests existed were broken. Anyone put on the project burned out and fled in months, sometimes weeks, and rarely after only a few days.

Rudolf wasn't too pleased when management parachuted him into the project to save it. But when he pulled the code and started poking around, it looked bad but not unsalvageable. The first thing he noticed is that, when following the instructions in the README, he couldn't build and run the application. Or maybe he wasn't following the instructions in the README, because the README was a confusing and incoherent mess, which included snippets from unresolved merges. Rudolf's first few days on the project were spent just getting it building and running locally, then updating the README. Once that was done, he started in on fixing the broken tests. There was a lot of work to be done, but it was all doable work. Rudolf could lay out a plan of how to get the project back on track and start delivering new features.

It's about then that Steve, the product owner, called Rudolf in to his office. "What the hell do you think you're doing?"

Rudolf blinked. "Um… what I was asked to do?"

"Three days and you just commit a README update? A couple of unit tests?"

"Well, it was out of date and meant I couldn't-"

"Our client is crazy about their business," Steve said. "Not about READMEs. Not about unit tests. None of that actually helps their business."

Rudolf bit back a "well, actually," while Steve ranted.

"Next thing you're going to tell me is that we should waste time on refactoring, like everybody else did. Time is money, time is new features, and new features are money!"

Suddenly, Rudolf realized that the reason the project had such a high burnout rate had nothing to do with the code itself. And while Rudolf could fix the code, he couldn't fix Steve. So, he did what everyone else had done: kept his head down and struggled through for a few months, and kept poking his manager to get him onto another project. In the meantime, he made this code slightly better for the next person, despite Steve's ranting. Rudolf eventually moved on, and Steve told everyone he was the worst developer that had ever touched the project.

The customer continued to be unhappy.

[Advertisement] Continuously monitor your servers for configuration changes, and report when there's configuration drift. Get started with Otter today!

Back in the heady days of the DotCom Bubble, startups were thick on the ground, and Venture Capital money was a flood- lifting startups atop a tsunami only to crash them back into the ground a short time later. Taliesyn once worked for one such startup.

Taliesyn's manager, Irving, was an expert in AI. In the age of the DotCom Bubble, this meant Irving knew LISP. Knowing LISP was valuable here, because their core product was a database system built on LISP- specifically the Common LISP Object System, an object-oriented bolt-on for LISP.

It was an object-oriented database system, akin to the modern NoSQL databases, but its architecture left a few things to be desired. First, since disk reads and writes were expensive operations, the system avoided them. All updates to data were done in memory, and someday, at some point, when the program felt like it, the changes would be written to disk. This meant that any failures or crashes could lose potentially days of data. Worse, the data was stored in one gigantic text file, which meant corruption could easily take out the entire database.

These were legitimate problems, and due to the architecture, they were going to be challenging to resolve. That was the startup life, however- they had a minimum viable product, and just needed to pour energy into making it something worth using.

Everyone looked to Irving, the AI and LISP expert, to guide them through this.

Irving saw where the real problems lay. "Your database doesn't support SQL," Irving said.

"Well, sure," Taliesyn said. "That's our selling point."

Irving nodded, and then, speaking slowly, as if to a particularly dense child, said, "A database needs to support SQL."

"I mean, a relational database, sure," Taliesyn said, "but we're using an object oriented data model which means we don't need to do joins or-" Taliesyn kept talking, explaining why their database didn't support SQL, why it shouldn't support SQL, and why SQL support was not only off the roadmap, but so far off the roadmap that it was labelled "Here there be dragons."

Irving nodded along, and ended the conversation with a, "Sure, that makes sense."

Everything was fine for a few weeks, until one of Taliesyn's peers on a different team, Angela, shot him an email: "Hey, marketing is getting antsy about a SQL demo, and I've got half a dozen features blocked until you get me a build with that functionality. What's the timeline like?"

Taliesyn was uncertain about what the timeline was like, since he had now clearly slipped into a parallel universe. He politely informed his peer that he had no idea what was going on, but would find out. It didn't take a great sleuth to figure out that Irving had started appending his own features to the roadmap.

Taliesyn tracked Irving down and demanded to know what was going on.

"A customer is already using it!" Irving protested. "They wrote it themselves! So we should be able to do it easily. Frankly, it's embarrassing to say that we can't do something with our own tools that the customer is already doing!"

Taliesyn knew that Irving was either wrong or lying, and asked to talk to the customer. Irving was, in fact, wrong. The customer had used LISP to write an extension to their object database (another one of those selling point features), and this extension used ODBC to open a connection to a relational database. This let them move data between the two different database systems.

"Irving," Taliesyn said, "they're not using SQL on our database, they're connecting to a relational database and using SQL there. SQL doesn't make sense for our data structure! We don't have tables, or keys, or relationships. It's objects! You're promising features that we can't deliver."

Irving was unmoved. "We are making a database. A database must have SQL capability. It's structured query language- it's right there in the name! Our structure should be queried by structured query language."

Taliesyn tried going over Irving's head, but upper management had no interest in actually solving personnel problems. Irving's buzzword laden ideas about why SQL was required seemed to jive with what their customers wanted, and the idea of shipping a non-SQL database was getting lost in the endless quest for the next round of VC funding.

The result was a bolted on monster of broken implementations of SQL that infuriated the few customers who tried it, Irving got promoted, and Taliesyn ran as far from the trainwreck as possible before the company flamed out.

[Advertisement] Continuously monitor your servers for configuration changes, and report when there's configuration drift. Get started with Otter today!

Alan worked on a website which had a weekly event which unlocked at 9PM, Saturday, Eastern Time. The details of the event didn't matter, but this little promotion had been running for about a year and a half, without any issues.

Until one day, someone emailed Alan: "Hey, I checked the site on Sunday, and the countdown timer displays 00:00:00."

Alan didn't check their email on Sunday, and when they checked the site, everything was fine, so he set himself a reminder to check things out next Sunday, and left things alone for a week.

Well, Alan forgot on Sunday. It was his day off after all, but he did remember to check first thing on Monday. When he came in, the timer read 00:00:00. Alan had a twelve o'clock flasher. Oddly, when he checked it after grabbing some coffee, the timer now showed the correct value.

It was time to dig through the code. Now, this story happened quite some time ago, so the countdown timer itself was a Flash widget. But the widget received a parameter from the HTML DOM, which itself was generated by PHP, and it didn't take long to find the SQL query it was using to find the next event: SELECT next_event FROM event_timer.

Yes, event_timer was a one column, one row table. A quick search through the codebase found the table referenced only one other place: backend_settimer.php.

All the pieces came together: resetting the timer was an entirely manual process. Every Monday, Tina came in, and assuming she remembered, she reset the timer. Some days she did it late, or forgot entirely. Some weeks, she was on vacation. Maybe she remembered to delegate, maybe she didn't.

For roughly 72 weeks, this had been how things had been working.

The good news was that the date was getting parsed with the PHP strtotime function, which meant Alan merely had to go to the backend_settimer.php and set the value to Next Saturday 9pm, and let Tina know she didn't need to do this anymore.

[Advertisement] Continuously monitor your servers for configuration changes, and report when there's configuration drift. Get started with Otter today!