On Thanksgiving Day, Ellis had cuddled up with her sleeping cat on the couch to send holiday greetings to friends. There in her inbox, lurking between several well wishes, was an email from an unrecognized sender with the subject line, Final Account Statement. Upon opening it, she read the following:

Dear Ellis,

Your final account statement dated -1 has been sent to you. Please log into your portal and review your balance due totaling #TOTAL_CHARGES#.

Payment must be received within 30 days of this notice to avoid collection. You may submit payment online via [Payment Portal Link] or by mail to:

Chamberlin Apartments
123 Main Street
Anytown US 12345

If you believe there is an error on your account, please contact us immediately at 212-555-1212.

Thank you for your prompt attention to this matter.

Chamberlin Apartments

Ellis had indeed rented an apartment managed by this company, but had moved out 16 years earlier. She'd never been late with a payment for anything in her life. What a time to receive such a thing, at the start of a long holiday weekend when no one would be able to do anything about it for the next 4 days!

She truly had so much to be grateful for that Thanksgiving, and here was yet more for her list: her broad technical knowledge, her experience working in multiple IT domains, and her many years of writing up just these sorts of stories for The Daily WTF. All of this added up to her laughing instead of panicking. She could just imagine the poor intern who'd hit "Send" by mistake. She also imagined she wasn't the only person who'd received this message. Rightfully scared and angry callers would soon be hammering that phone number, and Ellis was further grateful that she wasn't the one who had to pick up.

"I'll wait for the apology email!" she said out loud with a knowing smile on her face, closing out the browser tab.

Ellis moved on physically and mentally, going forward with her planned Thanksgiving festivities without giving it another thought. The next morning, she checked her inbox with curious anticipation. Had there been a retraction, a please disregard?

No. Instead, there were still more emails from the same sender. The second, sent 7 hours after the first, bore the subject line Second Notice - Outstanding Final Balance:

Dear Ellis,

Our records show that your final balance of #TOTAL_CHARGES# from your residency at your previous residence remains unpaid.

This is your second notice. Please remit payment in full or contact us to discuss the balance to prevent your account from being sent to collections.

Failure to resolve the balance within the next 15 days may result in your account being referred to a third-party collections agency, which could impact your credit rating.

To make payment or discuss your account, please contact us at 212-555-1212 or accounting@chamapts.com.

Sincerely,

Chamberlin Apartments

The third, sent 6 and a half hours later, threatened Final Notice - Account Will Be Sent to Collections.

Dear Ellis,

Despite previous notices, your final account balance remains unpaid.

This email serves as final notice before your account is forwarded to a third-party collections agency for recovery. Once transferred, we will no longer be able to accept payment directly or discuss the account.

To prevent this, payment of #TOTAL_CHARGES# must be paid in full by #CRITICALDATE#.

Please submit payment immediately. Please contact 212-555-1212 to confirm your payment.

Sincerely,

Chamberlin Apartments

It was almost certainly a mistake, but still rather spooky to someone who'd never been in such a situation. There was solace in the thought that, if they really did try to force Ellis to pay #TOTAL_CHARGES# on the basis of these messages, anyone would find it absurd that all 3 notices were sent mere hours apart, on a holiday no less. The first two had also mentioned 30 and 15 days to pay up, respectively.

Suddenly remembering that she probably wasn't the only recipient of these obvious form emails, Ellis thought to check her local subreddit. Sure enough, there was already a post revealing the range of panic and bewilderment they had wrought among hundreds, if not thousands. Current and more recent former tenants had actually seen #TOTAL_CHARGES# populated with the correct amount of monthly rent. People feared everything from phishing attempts to security breaches.

It wasn't until later that afternoon that Ellis finally received the anticipated mea culpa:

We are reaching out to sincerely apologize for the incorrect collection emails you received. These messages were sent in error due to a system malfunction that released draft messages to our entire database.

Please be assured of the following:
The recent emails do not reflect your actual account status.
If your account does have an outstanding balance, that status has not changed, and you would have already received direct and accurate communication from our office.
Please disregard all three messages sent in error. They do not require any action from you.

We understand that receiving these messages, especially over a holiday, was upsetting and confusing, and we are truly sorry for the stress this caused. The issue has now been fully resolved, and our team has worked with our software provider to stop all queued messages and ensure this does not happen again.

If you have any questions or concerns, please feel free to email leasing@chamapts.com. Thank you for your patience and understanding.

All's well that ends well. Ellis thanked the software provider's "system malfunction," whoever or whatever it may've been, that had granted the rest of us a bit of holiday magic to take forward for all time.

It's a holiday in the US today, one where we give thanks. And today, we give thanks to not have this boss. Original. --Remy

Matt works at an accounting firm, as a data engineer. He makes reports for people who don’t read said reports. Accounting firms specialize in different areas of accountancy, and Matt’s firm is a general firm with mid-size clients.

The CEO of the firm is a legacy from the last century. The most advanced technology on his desk is a business calculator and a pencil sharpener. He still doesn’t use a cellphone. But he does have a son, who is “tech savvy”, which gives the CEO a horrible idea of how things work.

Usually, this is pretty light, in that it’s sorting Excel files or sorting the output of an existing report. Sometimes the requests are bizarre or utter nonsense. And, because the boss doesn’t know what the technical folks are doing, some of the IT staff may be a bit lazy about following best practices.

This means that most of Matt’s morning is spent doing what is essentially Tier 1 support before he gets into doing his real job. Recently, there was a worse crunch, as actual support person Lucinda was out for materinity leave, and Jackie, the one other developer, was off on vacation on a foreign island with no Internet. Matt was in the middle of eating a delicious lunch of take-out lo mein when his phone rang. He sighed when he saw the number.

“Matt!” the CEO exclaimed. “Matt! We need to do a build of the flagship app! And a deploy!”

The app was rather large, and a build could take upwards of 45 minutes, depending on the day and how the IT gods were feeling. But the process was automated, the latest changes all got built and deployed each night. Anything approved was released within 24 hours. With everyone out of the office, there hadn’t been any approved changes for a few weeks.

Matt checked the Github to see if something went wrong with the automated build. Everything was fine.

“Okay, so I’m seeing that everything built on GitHub and everything is available in production,” Matt said.

“I want you to do a manual build, like you used to.”

“If I were to compile right now, it could take quite awhile, and redeploying runs the risk of taking our clients offline, and nothing would be any different.”

“Yes, but I want a build that has the changes which Jackie was working on before she left for vacation.”

Matt checked the commit history, and sure enough, Jackie hadn’t committed any changes since two weeks before leaving on vacation. “It doesn’t looked like she pushed those changes to Github.”

“Githoob? I thought everything was automated. You told me the process was automated,” the CEO said.

“It’s kind of like…” Matt paused to think of an analogy that could explain this to a golden retriever. “Your dishwasher, you could put a timer on it to run it every night, but if you don’t load the dishwasher first, nothing gets cleaned.”

There was a long pause as the CEO failed to understand this. “I want Jackie’s front-page changes to be in the demo I’m about to do. This is for Initech, and there’s millions of dollars riding on their account.”

“Well,” Matt said, “Jackie hasn’t pushed- hasn’t loaded her metaphorical dishes into the dishwasher, so I can’t really build them.”

“I don’t understand, it’s on her computer. I thought these computers were on the cloud. Why am I spending all this money on clouds?”

“If Jackie doesn’t put it on the cloud, it’s not there. It’s uh… like a fax machine, and she hasn’t sent us the fax.”

“Can’t you get it off her laptop?”

“I think she took it home with her,” Matt said.

“So?”

“Have you ever seen Star Trek? Unless Scotty can teleport us to Jackie’s laptop, we can’t get at her files.”

The CEO locked up on that metaphor. “Can’t you just hack into it? I thought the NSA could do that.”

“No-” Matt paused. Maybe Matt could try and recreate the changes quickly? “How long before this meeting?” he asked.

“Twenty minutes.”

“Just to be clear, you want me to do a local build with files I don’t have by hacking them from a computer which may or may not be on and connected to the Internet, and then complete a build process which usually takes 45 minutes- at least- deploy to production, so you can do a demo in twenty minutes?”

“Why is that so difficult?” the CEO demanded.

“I can call Jackie, and if she answers, maybe we can figure something out.”

The CEO sighed. “Fine.”

Matt called Jackie. She didn’t answer. Matt left a voicemail and then went back to eating his now-cold lo mein.

One of the things that makes legacy code legacy is that code, over time, rots. Some of that rot comes from the gradual accumulation of fixes, hacks, and kruft. But much of the rot also comes from the tooling going unsupported or entirely out of support.

For example, many years ago, I worked in a Visual Basic 6 shop. The VB6 IDE went out of support in April, 2008, but we continued to use it well into the next decade. This made it challenging to support the existing software, as the IDE frequently broke in response to OS updates. Even when we started running it inside of a VM running an antique version of Windows 2000, we kept running into endless issues getting projects to compile and build.

A fun side effect of that: the VB6 runtime remains supported. So you can run VB6 software on modern Windows. You just can't modify that software.

Greta has inherited an even more antique tech stack. She writes, "I often wonder if I'm the last person on Earth encumbered with this particular stack." She adds, "The IDE is long-deprecated from a vendor that no longer exists- since 2002." Given the project started in the mid 2010s, it may have been a bad choice to use that tech-stack.

It's not as bad as it sounds- while the technology and tooling is crumbling ruins, the team culture is healthy and the C-suite has given Greta wide leeway to solve problems. But that doesn't mean that the tooling isn't a cause of anguish, and even worse than the tooling- the code itself.

"Some things," Greta writes, "are 'typical bad'" and some things "are 'delightfully unique' bad."

For example, the IDE has a concept of "designer" files, for the UI, and "code behind" files, for the logic powering the UI. The IDE frequently corrupts its own internal state, and loses the ability to properly update the designer files. When this happens, if you attempt to open, save, or close a designer file, the IDE pops up a modal dialog box complaining about the corruption, with a "Yes" and "No" option. If you click "No", the modal box goes away- and then reappears because you're seeing this message because you're on a broken designer file. If you click "Yes", the IDE "helpfully" deletes pretty much everything in your designer file.

Nothing about the error message indicates that this might happen.

The language used is a dialect of C++. I say "dialect" because the vendor-supplied compiler implements some cursed feature set between C++98 and C++11 standards, but doesn't fully conform to either. It's only capable of outputting 32-bit x86 code up to a Pentium Pro. Using certain C++ classes, like std::fstream, causes the resulting executable to throw a memory protection fault on exit.

Worse, the vendor supplied class library is C++ wrappers on top of an even more antique Pascal library. The "class" library is less an object-oriented wrapper and more a collection of macros and weird syntax hacks. No source for the Pascal library exists, so forget about ever updating that.

Because the last release of the IDE was circa 2002, running it on any vaguely modern environment is prone to failures, but it also doesn't play nicely inside of a VM. At this point, the IDE works for one session. If you exit it, reboot your computer, or try to close and re-open the project, it breaks. The only fix is to reinstall it. But the reinstall requires you to know which set of magic options actually lets the install proceed. If you make a mistake and accidentally install, say, CORBA support, attempting to open the project in the IDE leads to a cascade of modal error boxes, including one that simply says, "ABSTRACT ERROR" ("My favourite", writes Greta). And these errors don't limit themselves to the IDE; attempting to run the compiler directly also fails.

But, if anything, it's the code that makes the whole thing really challenging to work with. While the UI is made up of many forms, the "main" form is 18,000 lines of code, with absolutely no separation of concerns. Actually, the individual forms don't have a lot of separation of concerns; data is shared between forms via global variables declared in one master file, and then externed into other places. Even better, the various sub-forms are never destroyed, just hidden and shown, which means they remember their state whether you want that or not. And since much of the state is global, you have to be cautious about which parts of the state you reset.

Greta adds:

There are two files called main.cpp, a Station.cpp, and a Station1.cpp. If you were to guess which one owns the software's entry point, you would probably be wrong.

But, as stated, it's not all as bad as it sounds. Greta writes: "I'm genuinely happy to be here, which is perhaps odd given how terrible the software is." It's honestly not that odd; a good culture can go a long way to making wrangling a difficult tech stack happy work.

Finally, Greta has this to say:

We are actively working on a .NET replacement. A nostalgic, perhaps masochistic part of me will miss the old stack and its daily delights.

Our submitter, Gearhead, was embarking on STEM-related research. This required him to pursue funding from a governmental agency that we’ll call the Ministry of Silly Walks. In order to start a grant application and track its status, Gearhead had to create an account on the Ministry website.

The registration page asked for a lot of personal information first. Then Gearhead had to create his own username and password. He used his password generator to create a random string: D\h.|wAi=&:;^t9ZyoO

Upon clicking Save, he received an error.

Your password must be a minimum eight characters long, with no spaces. It must include at least three of the following character types: uppercase letter, lowercase letter, number, special character (e.g., !, $, % , ?).

Perplexed, Gearhead emailed the Ministry’s web support, asking why his registration failed. The reply:

Hello,
The site rejects password generators as hacking attempts. You will need to manually select a password.
Ex. GHott*01

Thank you,

Support

So a long sequence of random characters was an active threat, but a 1990s-era AOL username was just fine. What developer had this insane idea and convinced other people of it? How on earth did they determine what was a "manually selected" string versus a randomly-generated one?

It seems the deciding factor is nothing more than length. If you go to the Ministry’s registration page now, their password guidelines have changed (emphasis theirs):

Must be 8-10 characters long, must contain at least one special character ( ! @ # $ % ^ & * ( ) + = { } | < > \ _ - [ ] / ? ) and no spaces, may contain numbers (0-9), lower and upper case letters (a-z, A-Z). Please note that your password is case sensitive.

Only good can come of forcing tiny passwords.

The more a company or government needs secure practices, the less good they are at secure practices. Is that a law yet? It should be.

Dotan was digging through vendor supplied documentation to understand how to use an API. To his delight, he found a specific function which solved exactly the problem he had, complete with examples of how it was to be used. Fantastic!

He copied one of the examples, and hit compile, and reviewed the list of errors. Mostly, the errors were around "the function you're calling doesn't exist". He went back to the documentation, checked it, went back to the code, didn't find any mistakes, and scratched his head.

Now, it's worth noting the route Dotan took to find the function. He navigated there from a different documentation page, which sent him to an anchor in the middle of a larger documentation page- vendorsite.com/docs/product/specific-api#specific-function.

This meant that as the page loaded, his browser scrolled directly down to the specific-function section of the page. Thus, Dotan missed the gigantic banner at the top of the page for that API, which said this:

/!\ NOTE /!\ NOTE /!\ NOTE /!\ NOTE /!\ NOTE /!\ NOTE /!\ NOTE /!
This doc was written to help flesh out a user API. The features described here are all hypothetical and do not actually exist yet, don't assume anything you see on this page works in any version /!\ NOTE /!\ NOTE /!\ NOTE /!\ NOTE /!\ NOTE /!\ NOTE /!\ NOTE /!\

On one hand, I think providing this kind of documentation is invaluable, both to your end users and for your own development team. It's a great roadmap, a "documentation driven development" process. And I can see that they made an attempt to be extremely clear about it being incomplete and unimplemented- but they didn't think about how people actually used their documentation site. A banner at the top of the page only works if you read the page from top to bottom, but documentation pages you will frequently skip to specific sections of the page.

But there was a deeper issue with the way this particular approach was executed: while the page announced that one shouldn't assume anything works, many of the functions on the page did work. Many did not. There was no rhyme or reason, to version information or other indicators to help a developer understand what was and was not actually implemented.

So while the idea of a documentation-oriented roadmap specifying features that are coming is good, the execution here verged into WTF territory. It was a roadmap, but with all the landmarks erased, so you had no idea where you actually were along the length of that road. And the one warning sign that would help you was hidden behind a bush.

Dotan asks: "WTF is that page doing on the official documentation wiki?"

And I'd say, I understand why it's there, but boy it should have been more clear about what it actually was.

[Advertisement] Keep the plebs out of prod. Restrict NuGet feed privileges with ProGet. Learn more.

Years ago, Brian had a problem: their C# application would crash sometimes. What was difficult to understand was why it was crashing, because it wouldn't crash in response to a user action, or really, any easily observable action.

The basic flow was that the users used a desktop application. Many operations that the users wanted to perform were time consuming, so the application spun up background tasks to do them, thus allowing the user to do other things within the application. And sometimes, the application would just crash, both when the user hadn't done anything, and when all background jobs should have been completed.

The way the background task was launched was this:

seeder.RunSeeder();

It didn't take too much head scratching to realize what was running every time the application crashed: the garbage collector.

RunSeeder returned a Task object, but since Brian's application treated the task as "fire and forget", they didn't worry about the value itself. But C# did- the garbage collector had to clean up that memory.

And this was running under .Net 4.0. This particular version of the .Net framework was a special, quantum realm, at least when it came to tasks. You see, if a Task raises an exception, nothing happens. At least, not right away. No one is notified of the exception unless they inspect the Task object directly. There's a cat in the box, and no one knows the state of the cat unless they open the box.

The application wasn't checking the Task result. The cat remained in a superposition of "exception" and "no exception". But the garbage collector looked at the task. And, in .Net 4.0, Microsoft made a choice about what to do there: when they opened the box and saw an exception (instead of a cat), they chose to crash.

Microsoft's logic here wasn't entirely bad; an uncaught exception means something has gone wrong and hasn't been handled. There's no way to be certain the application is in a safe state to continue. Treating it akin to undefined behavior and letting the application crash was a pretty sane choice.

The fix for Brian's team was simple: observe the exception, and choose not to do anything with it. They truly didn't care- these tasks were fire-and-forget, and failure was acceptable.

seeder.RunSeeder().ContinueWith(t => { var e = t.IsFaulted ? t.Exception : null; }); // Observe exceptions to prevent quantum crashes

This code merely opens the box and sees if there's an exception in there. It does nothing with it.

Now, I'd say as a matter of programming practice, Microsoft was right here. Ignoring exceptions blindly is a definite code smell, even for a fire-and-forget task. Writing the tasks in such a way as they catch and handle any exceptions that bubble up is better, as is checking the results.

But I, and Microsoft, were clearly on the outside in this argument. Starting with .Net 4.5 and moving forward, uncaught exceptions in background tasks were no longer considered show-stoppers. Whether there was a cat or an exception in the box, when the garbage collector observed it, it got thrown away either way.

In the end, this reminds me of my own failing using background tasks in .Net.

[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!

Everyone's got workplace woes. The clueless manager; the disruptive coworker; the cube walls that loom ever higher as the years pass, trapping whatever's left of your soul.

But sometimes, Satan really leaves his mark on a joint. I worked Tech Support there. You may remember The C-Level Ticket. I'm Anonymous. This is my story.

Night after night, my dreams are full of me trying and failing at absolutely everything. Catch a bus? I'm already running late and won't make it. Dial a phone number to get help? I can't recall the memorized sequence, and the keypad's busted anyway. Drive outta danger? The car won't start. Run from a threat? My legs are frozen.

Then I wake up in my bed in total darkness, scared out of my skull, and I can't move for real. Not one muscle works. Even if I could move, I'd stay still because I'm convinced the smallest twitch will give me away to the monster lurking nearby, looking to do me in.

The alarm nags me before the sun's even seen fit to show itself. What day is it? Tuesday? An invisible, overwhelming dread pins me in place under the covers. I can't do it. Not again.

The thing is, hunger, thirst, and cold are even more nagging than the alarm. Dead tired, I force myself up anyway to do the whole thing over.

The office joe that morning was so over-brewed as to be sour. I tossed down the last swig in my mug, checking my computer one more time to make sure no Tech Support fires were raging by instant message or email. Then I threw on my coat and hat and quit my cube, taking the stairs to ground level.

I pushed open a heavy fire-escape door and stepped out into the narrow alley between two massive office buildings. Brisk autumn air and the din of urban motor traffic rushed to greet me. The dull gray sky above threatened rain. Leaning against the far brick wall were Toby and Reynaldo, a couple of network admins, hugging themselves as they nursed smoldering cigarettes. They nodded hello.

I tipped my hat in greeting, slipping toward the usual spot, a patch of asphalt I'd all but worn grooves in by that point. I lit my own cigarette and took in a deep, warming draw.

"Make it last another year," Toby spoke in a mocking tone, tapping ash onto the pavement. "I swear, that jerk can squeeze a nickel until Jefferson poops!"

An ambulance siren blared through the alley for a minute. The rig was no doubt racing toward the hospital down the street.

Reynaldo smirked. "You think Morty finally did it?"

Toby smirked as well.

I raised an eyebrow. "Did what?"

"Morty always says he's gonna run out into traffic one of these days so they can take him to the hospital and he won't have to be here," Reynaldo explained.

I frowned at the morbid suggestion. "Hell of a way to catch a break."

"Well, it's not like we can ask for time off," Toby replied bitterly. "They always find some way to rope us back in."

I nodded in sympathy. "You have it worse than we do. But my sleep's still been jacked plenty of times by 3AM escalated nonsense that shoulda been handled by a different part of the globe."

Reynaldo's eyes lit up fiercely. "They have all the same access and training, but it never falls on them! Yeah, been there."

The door swung open again, admitting a young woman with the weight of the world on her shoulders. This was Megan, a junior developer and recent hire. I tipped my hat while helping myself to another drag.

She hastened my way, pulling a pack of cigarettes from her handbag. With shaking hands, she fumbled to select a single coffin nail. "I quit these things!" she lamented. After returning the pack to her bag, she rummaged through it fruitlessly. "Dammit, where are those matches?!" She glanced up at me with a pleading expression.

I pulled the lighter from my coat pocket. "You sure?"

She nodded like she hadn't been more sure about anything in her entire life.

I lit it for her. She took a lung-filling pull, then exhaled a huge cloud of smoke.

"Goin' that well, huh?" I asked.

Megan also hugged herself, her expression pained. "Every major player in the industry uses our platform, and I have no idea how it hasn't all come crashing down. There are thousands of bugs in the code base. Thousands! It breaks all the time. Most of the senior devs have no clue what they're doing. And now we're about to lose the only guy who understands the scheduling algorithm, the most important thing!"

"That's tough." I had no idea what else to say. Maybe it was enough that I listened.

Megan glanced up nervously at the brewing storm overhead. "I just know that algorithm's gonna get dumped in my lap."

"The curse of competence." I'd seen it plenty of times.

"Ain't that the truth!" She focused on me again with a look of apology. "How've you been?"

I shrugged. "Same old, same old." I figured a fresh war story might help. "Had to image and set up the tech for this new manager's onboarding. Her face is stuck in this permanent glare. Every time she opens her mouth, it's to bawl someone out."

"Ugh."

"The crazy thing is, the walls of her office are completely covered with crucifixes, and all these posters plastered with flowers and hearts and sap like Choose Kindness." I leaned in and lowered my voice. "You know what I think? I think she’s an ancient Roman whose spite has kept her alive for over two thousand years. Those crosses are a threat!"

That teased a small laugh out of Megan. For a moment, the amusement reached her eyes. Then it was gone, overwhelmed by worry. She took to pacing through the narrow alley.

Back at my cube, I found a new urgent ticket at the top of my case load. Patricia Dracora, a senior project manager, had put in a call claiming her computer had been hacked. Her mouse cursor was moving around and clicking things all on its own.

It was too early in the morning for a case like this. That old dread began sneaking up on me again. The name put me on edge as well. Over the years, our paths had never crossed, but her nickname throughout Tech Support, Dracula, betrayed what everyone else made of her.

"Make like a leaf and blow!"

The boss barked his stern command over my shoulder. I stood and turned from my computer to find him at my cubicle threshold with arms folded, blocking my egress.

I couldn't blow, so I shrugged. "Can't be as bad as The Crucifier."

"Dracula's worse than The Crucifier," the boss replied under his breath in a warning tone. "For your own good, don't keep her waiting!" He tossed a thumb over his shoulder for good measure.

When he finally backed out of the way, I made tracks outta there. A few of my peers made eye contact as I passed, looking wary on my behalf.

The ticket pegged Dracora's office in a subfloor I'd never set foot in before. Descending the stairs, I had too much time to think. Of course I didn't expect a real hacking attempt. Peripheral hardware on the fritz, some software glitch: there'd be a simple explanation. What fresh hell would I have to endure to reach that point? That was what my tired brain couldn't let go of. The stimulants hadn't kicked in yet. With the strength of a kitten, I was stepping into a lion's den. A lion who might make me wish for crucifixion by the time it was all over.

From the stairwell, I entered a dank, deserted corridor. Old florescent lighting fixtures hummed and flickered overhead. That, combined with the overwhelming stench of paint fumes, set the stage for a ripping headache. There were no numbers on the walls to lead me to the right place. They must've taken them down to paint and never replaced them. I inched down worn, stained carpeting, peeking into each open gap I found to either side of me. Nothing but darkness, dust, and cobwebs at first. Eventually, I spotted light blaring from one of the open doors ahead of me. I jogged the rest of the way, eager to see any living being by that point.

The room I'd stumbled onto was almost closet-sized. It contained a desk and chair, a laptop docking station, and a stack of cardboard boxes on the floor. Behind the desk was a woman of short stature, a large purse slung over one shoulder. Her arms were folded as she paced back and forth in the space behind her chair. When I appeared, she stopped and looked to me wide-eyed, maybe just as relieved as I was. "Are you Tech Support?"

"Yes, ma'am." I entered the room. "What's—?"

"I don't know how it happened!" Dracora returned to pacing, both hands making tight fists around the straps of the purse she was apparently too wired and distracted to set down. "They made me move here from the fourth floor. I just brought everything down and set up my computer, and now someone has control of the mouse. Look, look!" She stopped and pointed at the monitor.

I rounded the desk. By the time I got there, whatever she'd seen had vanished. Onscreen, the mouse cursor sat still against a backdrop of open browsers and folders. Nothing unusual.

"It was moving, I swear!" Anguished, Dracora pleaded with me to believe her.

It seemed like she wasn't hostile at all, just stressed out and scared. I could handle that. "I'm sure we can figure this out, ma'am. Lemme have a look here."

I sat down at the desk and tried the wireless mouse first. It didn't work at all to move the cursor.

"The hacker's locked us out!" Dracora returned to pacing behind me.

As I sat there, not touching a thing, the mouse cursor shuttled across the screen like it was possessed.

"There! You see?"

Suddenly, somehow, my brain smashed everything together. "Ma'am, I have an idea. Could you please stand still?"

Dracora stopped.

I swiveled around in the chair to face her. "Ma'am, you said you were moving in down here. What's in your purse right now?"

Her visible confusion deepened. "What?"

"The mouse cursor only moves around when you do," I explained.

Her eyes widened. She dug deeply into her purse. A moment later, she pulled out a second wireless mouse. Then she looked to me like she couldn't believe it. "That's it?!"

"That's it!" I replied.

"Oh, lord!" Dracora replaced the dud sitting on her mousepad with the mouse that was actually connected to her machine, wilting over the desk as she did so. "I don't know whether to laugh or cry."

I knew the feeling. But the moment of triumph, I gotta admit, felt pretty swell. "Anything else I can help with, ma'am?"

"No, no! I've wasted enough of your time. Thank you so much!"

I had even more questions on the way back upstairs. With this huge, spacious office building, who was forcing Dracora to be in that pit? How had she garnered such a threatening reputation? Why had my experience been so different from everyone else's? I didn't mention it to the boss or my peers. I broke it all down to Megan in the alley a few days later.

"She even put in a good word for me when she closed the ticket," I told her. "The boss says I'm on the fast track for another promotion." I took a drag from my cigarette, full of bemusement. "I'm already as senior as it gets. The only way up from here is management." I shook my head. "That ain't my thing. Look how well it's gone for Dracora."

Megan lowered her gaze, eyes narrowed. "You said it yourself: the only reward for good work is more work."

And then they buried you ... in a basement, or a box.

I remembered being at the start of my career, like Megan. I remembered feeling horrified by all the decades standing between me and the day when I wouldn't or couldn't ever work again. A couple decades in, some part of me that I'd repressed had resurfaced. What the hell is this? What have I been doing?

Stop caring, a different part replied. Just stop caring. Take things day by day, case by case.

I'd obeyed for so long. Where had it gotten me?

Under my breath, I risked airing my wildest wish for the future. "Someday, I wanna break outta this joint."

Megan blinked up at me. I had her attention. "How?"

"I dunno," I admitted. "I gotta figure it out ... before I go nuts."

[Advertisement] Utilize BuildMaster to release your software with confidence, at the pace your business demands. Download today!

Back in the antediluvian times, when I was in college, people still used floppy disks to work on their papers. This was a pretty untenable arrangement, because floppy disks lost data all the time, and few students had the wherewithal to make multiple copies. Half my time spent working helldesk was breaking out Norton Diskutils to try and rescue people's term papers. To avoid this, the IT department offered network shares where students could store documents. The network share was backed up, tracked versions, and could be accessed from any computer on campus, including the VAX system (in fact, it was stored on the VAX).

I bring this up because we have known for quite some time that companies and governments need to store documents in centrally accessible locations so that you're not reliant on end users correctly managing their files. And if you are a national government, you have to make a choice: either you contract out to a private sector company, or you do it yourself.

South Korea made the choice to do it themselves, with their G-Drive system (short for Government Drive, no relation to Google Drive), a government file store hosted primarily out of a datacenter in Daejeon. Unfortunately, "primarily" is a bit too apropos- last month, a fire in that datacenter destroyed data.

The Interior Ministry explained that while most systems at the Daejeon data center are backed up daily to separate equipment within the same center and to a physically remote backup facility, the G-Drive’s structure did not allow for external backups. This vulnerability ultimately left it unprotected.

Someone, somehow, designed a data storage system that was structurally incapable of doing backups? And then told 750,000 government employees that they should put all their files there?

Even outside of that backup failure, while other services had backups, they did not have a failover site, so when the datacenter went down, the government went down with it.

In total, it looks like about 858TB of data got torched. 647 distinct services were knocked out. At least 90 of them were reported to be unrecoverable (that last link is from a company selling Lithium Ion safety products, but is a good recap). A full recovery was, shortly after the accident, predicted to take a month, but as of October 22, only 60% of services had been restored.

Now, any kind of failure of this scale means heads must roll, and police investigations have gone down the path of illegal subcontracting. The claim is that the government hired a contractor broke the law by subcontracting the work, and that those subcontractors were unqualified for the work they were doing- that while they were qualified to install or remove a li-ion battery, they were not qualified to move one, which is what they were doing and resulted in the fire.

I know too little about Korean laws about government contracting and too little about li-ion battery management to weigh in on this. Certainly, high-storage batteries are basically bombs, and need to be handled with great care and protected well. Though, if one knows how to install and uninstall a battery, moving a battery seems covered in those steps.

But if I were doing a root cause analysis here, while that could be the root cause of the fire, it is not the root cause of the outage. If you build a giant datacenter but can't replicate services to another location, you haven't built a reliable cloud storage system- you've just built an expensive floppy disk that is one trip too close to a fridge magnet away from losing all of your work. In this case, the fridge magnet was made of fire, but the result is the same.

I'm not going to say this problem would have be easy to avoid; actually building resilient infrastructure that fails gracefully under extreme stress is hard. But while it's a hard problem, it's also a well-understood problem. There are best practices, and clearly not one of them was followed.

[Advertisement] Keep all your packages and Docker containers in one place, scan for vulnerabilities, and control who can access different feeds. ProGet installs in minutes and has a powerful free version with a lot of great features that you can upgrade when ready.Learn more.

Chops was a developer for Initrode. Early on a Monday, they were summoned to their manager Gary's office before the caffeine had even hit their brain.

Gary glowered up from his office chair as Chops entered. This wasn't looking good. "We need to talk about the latest commit for Taskmaster."

Taskmaster was a large application that'd been around for decades, far longer than Chops had been an employee. Thousands of internal and external customers relied upon it. Refinements over time had led to remarkable stability, its typical uptime now measured in years. However, just last week, their local installation had unexpectedly suffered a significant crash. Chops had been assigned to troubleshooting and repair.

"What's wrong?" Chops asked.

"Your latest commit decreased the number of unit tests!" Gary replied as if Chops had slashed the tires on his BMW.

Within Taskmaster, some objects that were periodically generated were given a unique ID from a pool. The pool was of limited size and required scanning to find a spare ID. Each time a value was needed, a search began where the last search ended. IDs returned to the pool as objects were destroyed would only be reused when the search wrapped back around to the start.

Chops had discovered a bug in the wrap-around logic that would inevitably produce a crash if Taskmaster ran long enough. They also found that if the number of objects created exceeded the size of the pool, this would trigger an infinite loop.

Rather than attempt to patch any of this, Chops had nuked the whole thing and replaced it with code that assigned each object a universally unique identifier (UUID) from a trusted library UUID generator within its constructor. Gone was the bad code, along with its associated unit tests.

Knowing they would probably only get in a handful of words, Chops wonderered how on earth to explain all this in a way that would appease their manager. "Well—"

"That number must NEVER go down!" Gary snapped.

"But—"

"This is non-negotiable! Roll it back and come up with something better!"

And so Chops had no choice but to remove their solution, put all the janky code back in place, and patch over it with kludge. Every comment left to future engineers contained a tone of apology.

Taskmaster became less stable. Time and expensive developer hours were wasted. Risk to internal and external customers increased. But Gary could rest assured, knowing that his favored metric never faltered on his watch.

[Advertisement] Keep all your packages and Docker containers in one place, scan for vulnerabilities, and control who can access different feeds. ProGet installs in minutes and has a powerful free version with a lot of great features that you can upgrade when ready.Learn more.

Ellis knew she needed a walk after she hurried off of Zoom at the end of the meeting to avoid sobbing in front of the group.

She'd just been attending a free online seminar regarding safe job hunting on the Internet. Having been searching since the end of January, Ellis had already picked up plenty of first-hand experience with the modern job market, one rejection at a time. She thought she'd attend the seminar just to see if there were any additional things she wasn't aware of. The seminar had gone well, good information presented in a clear and engaging way. But by the end of it, Ellis was feeling bleak. Goodness gracious, she'd already been slogging through months of this. Hundreds of job applications with nothing to show for it. All of the scams out there, all of the bad actors preying on people desperate for their and their loved ones' survival!

Ellis' childhood had been plagued with anxiety and depression. It was only as an adult that she'd learned any tricks for coping with them. These tricks had helped her avoid spiraling into full-on depression for the past several years. One such trick was to stop and notice whenever those first feelings hit. Recognize them, feel them, and then respond constructively.

First, a walk. Going out where there were trees and sunshine: Ellis considered this "garbage collection" for her brain. So she stepped out the front door and started down a tree-lined path near her house, holding on to that bleak feeling. She was well aware that if she didn't address it, it would take root and grow into hopelessness, self-loathing, fear of the future. It would paralyze her, leave her curled up on the couch doing nothing. And it would all happen without any words issuing from her inner voice. That was the most insidious thing. It happened way down deep in a place where there were no words at all.

Once she returned home, Ellis forced herself to sit down with a notebook and pencil and think very hard about what was bothering her. She wrote down each sentiment:

This was the first step of a reframing technique she'd just read about in the book Right Kind of Wrong by Amy Edmonson. With the words out, it was possible to look at each statement and determine whether it was rational or irrational, constructive or harmful. Each statement could be replaced with something better.

Ellis proceeded step by step through the list.

• Yes, this will end. Everything ends.
• There's nothing wrong with me. Most businesses are swamped with applications. There's a good chance mine aren't even being looked at before they're being auto-rejected. Remember the growth mindset you learned from Carol Dweck. Each application and interview is giving me experience and making me a better candidate.
• This job market is a novel context that changes every day. That means failure is not only inevitable, it's the only way forward.

Ellis realized that her job hunt was very much like a search algorithm trying to find a path through a maze. When the algorithm encountered a dead end, did it deserve blame? Was it an occasion for shame, embarrassment, and despair? Of course not. Simply backtrack and keep going with the knowledge gained.

Yes, there was truth to the fact that this was the toughest job market Ellis had ever experienced. Therefore, taking a note from Viktor Frankl, she spent a moment reimagining the struggle in a way that made it meaningful to her. Ellis began viewing her job hunt in this dangerous market, her gradual accumulation of survival information, as an act of resistance against it. She now hoped to write all about her experience once she was on the other side, in case her advice might help even one other person in her situation save time and frustration.

While unemployed, she also had the opportunity to employ the search algorithm against entirely new mazes. Could Ellis expand her freelance writing into a sustainable gig, for instance? That would mean exploring all the different ways to be a freelance writer, something Ellis was now curious and excited to explore.

[Advertisement] Keep all your packages and Docker containers in one place, scan for vulnerabilities, and control who can access different feeds. ProGet installs in minutes and has a powerful free version with a lot of great features that you can upgrade when ready.Learn more.

Everyone's got workplace woes. The clueless manager; the disruptive coworker; the cube walls that loom ever higher as the years pass, trapping whatever's left of your soul.

But sometimes, Satan really leaves his mark on a joint. I worked Tech Support there. This is my story. Who am I? Just call me Anonymous.

It starts at the top. A call came in from Lawrence Gibbs, the CEO himself, telling us that a conference room printer was, quote, "leaking." He didn't explain it, he just hung up. The boss ordered me out immediately, told me to step on it. I ignored the elevator, racing up the staircase floor after floor until I reached the dizzying summit of C-Town.

There's less oxygen up there, I'm sure of it. My lungs ached and my head spun as I struggled to catch my breath. The fancy tile and high ceilings made a workaday schmuck like me feel daunted, unwelcome. All the same, I gathered myself and pushed on, if only to learn what on earth "leaking" meant in relation to a printer.

I followed the signs on the wall to the specified conference room. In there, the thermostat had been kicked down into the negatives. The cold cut through every layer of mandated business attire, straight to bone. The scene was thick with milling bystanders who hugged themselves and traded the occasional nervous glance. Gibbs was nowhere to be found.

Remembering my duty, I summoned my nerve. "Tech Support. Where's the printer?" I asked.

Several pointing fingers showed me the way. The large printer/scanner was situated against the far wall, flanking an even more enormous conference table. Upon rounding the table, I was greeted with a grim sight: dozens of sheets of paper strewn about the floor like blood spatter. Everyone was keeping their distance; no one paid me any mind as I knelt to gather the pages. There were 30 in all. Each one was blank on one side, and sported some kind of large, blotchy ring on the other. Lord knew I drank enough java to recognize a coffee mug stain when I saw one, but these weren't actual stains. They were printouts of stains.

The printer was plugged in. No sign of foul play. As I knelt there, unseen and unheeded, I clutched the ruined papers to my chest. Someone had wasted a tree and a good bit of toner, and for what? How'd it go down? Surely Gibbs knew more than he'd let on. The thought of seeking him out, demanding answers, set my heart to pounding. It was no good, I knew. He'd play coy all day and hand me my pink slip if I pushed too hard. As much as I wanted the truth, I had a stack of unpaid bills at home almost as thick as the one in my arms. I had to come up with something else.

There had to be witnesses among the bystanders. I stood up and glanced among them, seeking out any who would return eye contact. There: a woman who looked every bit as polished as everyone else. But for once, I got the feeling that what lay beneath the facade wasn't rotten.

With my eyes, I pleaded for answers.

Not here, her gaze pleaded back.

I was getting somewhere, I just had to arrange for some privacy. I hurried around the table again and weaved through bystanders toward the exit, hoping to beat it out of that icebox unnoticed. When I reached the threshold, I spotted Gibbs charging up the corridor, smoldering with entitlement. "Where the hell is Tech Support?!"

I froze a good distance away from the oncoming executive, whose voice I recognized from a thousand corporate presentations. Instead of putting me to sleep this time, it jolted down my spine like lightning. I had to think fast, or I was gonna lose my lead, if not my life.

"I'm right here, sir!" I said. "Be right back! I, uh, just need to find a folder for these papers."

"I've got one in my office."

A woman's voice issued calmly only a few feet behind me. I spun around, and it was her, all right, her demeanor as cool as our surroundings. She nodded my way. "Follow me."

My spirits soared. At that moment, I would've followed her into hell. Turning around, I had the pleasure of seeing Gibbs stop short with a glare of contempt. Then he waved us out of his sight.

Once we were out in the corridor, she took the lead, guiding me through the halls as I marveled at my luck. Eventually, she used her key card on one of the massive oak doors, and in we went.

You could've fit my entire apartment into that office. The place was spotless. Mini-fridge, espresso machine, even couches: none of it looked used. There were a couple of cardboard boxes piled up near her desk, which sat in front of a massive floor-to-ceiling window admitting ample sunlight.

She motioned toward one of the couches, inviting me to sit. I shook my head in reply. I was dying for a cigarette by that point, but I didn't dare light up within this sanctuary. Not sure what to expect next, I played it cautious, hovering close to the exit. "Thanks for the help back there, ma'am."

"Don't mention it." She walked back to her desk, opened up a drawer, and pulled out a brand-new manila folder. Then she returned to conversational distance and proffered it my way. "You're from Tech Support?"

There was pure curiosity in her voice, no disparagement, which was encouraging. I accepted the folder and stuffed the ruined pages inside. "That's right, ma'am."

She shook her head. "Please call me Leila. I started a few weeks ago. I'm the new head of HR."

Human Resources. That acronym, which usually put me on edge, somehow failed to raise my hackles. I'd have to keep vigilant, of course, but so far she seemed surprisingly OK. "Welcome aboard, Leila. I wish we were meeting in better circumstances." Duty beckoned. I hefted the folder. "Printers don't just leak."

"No." Leila glanced askance, grave.

"Tell me what you saw."

"Well ..." She shrugged helplessly. "Whenever Mr. Gibbs gets excited during a meeting, he tends to lean against the printer and rest his coffee mug on top of it. Today, he must've hit the Scan button with his elbow. I saw the scanner go off. It was so bright ..." She trailed off with a pained glance downward.

"I know this is hard," I told her when the silence stretched too long. "Please, continue."

Leila summoned her mettle. "After he leaned on the controls, those pages spilled out of the printer. And then ... then somehow, I have no idea, I swear! Somehow, all those pages were also emailed to me, Mr. Gibbs' assistant, and the entire board of directors!"

The shock hit me first. My eyes went wide and my jaw fell. But then I reminded myself, I'd seen just as crazy and worse as the result of a cat jumping on a keyboard. A feline doesn't know any better. A top-level executive, on the other hand, should know better.

"Sounds to me like the printer's just fine," I spoke with conviction. "What we have here is a CEO who thinks it's OK to treat an expensive piece of office equipment like his own personal fainting couch."

"It's terrible!" Leila's gaze burned with purpose. "I promise, I'll do everything I possibly can to make sure something like this never happens again!"

I smiled a gallows smile. "Not sure what anyone can do to fix this joint, but the offer's appreciated. Thanks again for your help."

Now that I'd seen this glimpse of better things, I selfishly wanted to linger. But it was high time I got outta there. I didn't wanna make her late for some meeting or waste her time. I backed up toward the door on feet that were reluctant to move.

Leila watched me with a look of concern. "Mr. Gibbs was the one who called Tech Support. I can't close your ticket for you; you'll have to get him to do it. What are you going to do?"

She cared. That made leaving even harder. "I dunno yet. I'll think of something."

I turned around, opened the massive door, and put myself on the other side of it in a hurry, using wall signs to backtrack to the conference room. Would our paths ever cross again? Unlikely. Someone like her was sure to get fired, or quit out of frustration, or get corrupted over time.

It was too painful to think about, so I forced myself to focus on the folder of wasted pages in my arms instead. It felt like a mile-long rap sheet. I was dealing with an alleged leader who went so far as to blame the material world around him rather than accept personal responsibility. I'd have to appeal to one or more of the things he actually cared about: himself, his bottom line, his sense of power.

By the time I returned to the conference room to face the CEO, I knew what to tell him. "You're right, sir, there's something very wrong with this printer. We're gonna take it out here and give it a thorough work-up."

That was how I was able to get the printer out of that conference room for good. Once it underwent "inspection" and "testing," it received a new home in a previously unused closet. Whenever Gibbs got to jawing in future meetings, all he could do was lean against the wall. Ticket closed.

Gibbs remained at the top, doing accursed things that trickled down to the roots of his accursed company. But at least from then on, every onboarding slideshow included a photo of one of the coffee ring printouts, with the title Respect the Equipment.

Thanks, Leila. I can live with that.

Once upon a time, when the Web was young, if you wanted to be a cool kid, you absolutely needed two things on your website: a guestbook for people to sign, and a hit counter showing how many people had visited your Geocities page hosting your Star Trek fan fiction.

These days, we don't see them as often, but companies still like to track the information, especially when it comes to counting downloads. So when Justin started on a new team and saw a download count in their analytics, he didn't think much of it at all. Nor did he think much about it when he saw the download count displayed on the download page.

Another thing that Justin didn't think much about was big piles of commits getting merged in overnight, at least not at first. But each morning, Justin needed to pull in a long litany of changes from a user named "MrStinky". For the first few weeks, Justin was too preoccupied with getting his feet under him, so he didn't think about it too much.

But eventually, he couldn't ignore what he saw in the git logs.

docs: update download count to 51741
docs: update download count to 51740
docs: update download count to 51738

And each commit was exactly what the name implied, a diff like:

- 51740
+ 51741

Each time a user clicked the download link, a ping was sent to their analytics system. Throughout the day, the bot "MrStinky" would query the analytics tool, and create new commits that updated the counter. Overnight, it would bundle those commits into a merge request, approve the request, merge the changes, and then redeploy what was at the tip of main.

"But, WHY?" Justin asked his peers.

One of them just shrugged. "It seemed like the easiest and fastest way at the time?"

"I wanted to wire Mr Stinky up to our content management system's database, but just never got around to it. And this works fine," said another.

Much like the rest of the team, Justin found that there were bigger issues to tackle.

Adam's organization was going through a period of rapid growth. Part of this growth was spinning up new backend services to support new functionality. The growth would have been extremely fast, except for one thing applying back pressure: for some reason, spinning up a new service meant recompiling and redeploying all the other services.

Adam didn't understand why, but it seemed like an obvious place to start poking at something for improvement. All of the services depended on a library called "ServiceLib"- though not all of them actually used the library. The library was a set of utilities for administering, detecting, and interacting with services in their environment- essentially a homegrown fabric/bus architecture.

It didn't take long, looking at the source control history, to understand why there was a rebuild after the release of every service. Each service triggered a one line change in this:

enum class Services
{
    IniTechBase = 103,
    IniTechAdvanced = 99,
    IniTechFooServer = 102,
    …
}

Each service had a unique, numerical identifier, and this mapped them into an enumerated type.

Adam went to the tech lead, Raymond. "Hey, I've got an idea for speeding up our release process- we should stop hard coding the service IDs in ServiceLib."

Raymond looked at Adam like one might examine an over-enthusiastic lemur. "They're not hard-coded. We store them in an enum."

Eventually Raymond got promoted- for all of their heroic work on managing this rapidly expanding library of services. The new tech lead who came on was much more amenable to "not storing rapidly changing service IDs in an enum", and "not making every service depend on a library they often don't need", and "putting admin functionality in every service because they're linked to that library whether they like it or not."

Eventually, ServiceLib became its own service, and actually helped- instead of hindered- delivering new functionality.

Unfortunately, with no more highly visible heroics to deliver functionality, the entire department became a career dead end. Sure, they delivered on time and under budget consistently, but there were no rockstar developers like Raymond on the team anymore, the real up-and-comers who were pushing themselves.

Once upon a time, there was a bank whose business relied on a mainframe. As the decades passed and the 21st century dawned, the bank's bigwigs realized they had to upgrade their frontline systems to applications built in Java and .NET, but—for myriad reasons that boiled down to cost, fear, and stubbornness—they didn't want to migrate away from the mainframe entirely. They also didn't want the new frontline systems to talk directly to the mainframe or vice-versa. So they tasked old-timer Edgar with writing some middleware. Edgar's brainchild was a Windows service that took care of receiving frontline requests, passing them to the mainframe, and sending the responses back.

Edgar's middleware worked well, so well that it was largely forgotten about. It outlasted Edgar himself, who, after another solid decade of service, moved on to another company.

A few years later, our submitter John F. joined the bank's C# team. By this point, the poor middleware seemed to be showing its age. A strange problem had arisen: between 8:00AM and 5:00PM, every 45 minutes or so, it would lock up and have to be restarted. Outside of those hours, there was no issue. The problem was mitigated by automatic restarts, but it continued to inflict pain and aggravation upon internal users and external customers. A true solution had to be found.

Unfortunately, Edgar was long gone. The new "owner" of the middleware was an infrastructure team containing zero developers. Had Edgar left them any documentation? No. Source code? Sort of. Edgar had given a copy of the code to his friend Bob prior to leaving. Unfortunately, Bob's copy was a few point releases behind the version of middleware running in production. It was also in C, and there were no C developers to be found anywhere in the company.

And so, the bank's bigwigs cobbled together a diverse team of experts. There were operating system people, network people, and software people ... including the new guy, John. Poor John had the unenviable task of sifting through Edgar's source code. Just as the C# key sits right next to the C key on a piano, reasoned the bigwigs, C# couldn't be that different from C.

John toiled in an unfamiliar language with no build server or test environment to aid him. It should be no great surprise that he got nowhere. A senior coworker suggested that he check what Windows' Process Monitor registered when the middleware was running. John allowed a full day to pass, then looked at the results: it was now clear that the middleware was constantly creating and destroying threads. John wrote a Python script to analyze the threads, and found that most of them lived for only seconds. However, every 5 minutes, a thread was created but never destroyed.

This only happened during the hours of 8:00AM to 5:00PM.

At the next cross-functional team meeting behind closed doors, John finally had something of substance to report to the large group seated around the conference room table. There was still a huge mystery to solve: where were these middleware-killing threads coming from?

"Wait a minute! Wasn't Frank doing something like that?" one of the other team members piped up.

"Frank!" A department manager with no technical expertise, who insisted on attending every meeting regardless, darted up straight in his chair. For once, he wasn't haranguing them for their lack of progress. He resembled a wolf who'd sniffed blood in the air. "You mean Frank from Accounting?!"

This was the corporate equivalent of an arrest warrant. Frank from Accounting was duly called forth.

"That's my program." Frank stood before the table, laid back and blithe despite the obvious frayed nerves of several individuals within the room. "It queries the middleware every 5 minutes."

They were finally getting somewhere. Galvanized, John's heart pounded. "How?" he asked.

"Well, it could be that the middleware is down, so first, my program opens a connection just to make sure it's working," Frank explained. "If that works, it opens another connection and sends the query."

John's confusion mirrored the multiple frowns that filled the room. He forced himself to carefully parse what he'd just heard. "What happens to the first connection?"

"What do you mean?" Frank asked.

"You said your program opens two connections. What do you do with the first one?"

"Oh! I just use that one to test whether the middleware is up."

"You don't need to do that!" one of the networking experts snarled. "For Pete's sake, take that out of your code! Don't you realize you're tanking this thing for everyone else?"

Frank's expression made clear that he was entirely oblivious to the chaos wrought by his program. Somehow, he survived the collective venting of frustration that followed within that conference room. After one small update to Frank's program, the middleware stabilized—for the time being. And while Frank became a scapegoat and villain to some, he was a hero to many, many more. After all, he single-handedly convinced the bank's bigwigs that the status quo was too precarious. They began to plan out a full migration away from mainframe, a move that would free them from their dependence upon aging, orphaned middleware.

Now that the mystery had been solved, John knew where to look in Edgar's source code. The thread pool had a limit of 10, and every thread began by waiting for input. The middleware could handle bad input well enough, but it hadn't been written to handle the case of no input at all.

[Advertisement] Utilize BuildMaster to release your software with confidence, at the pace your business demands. Download today!

Our anonymous submitter, whom we'll call Craig, worked for GlobalCon. GlobalCon relied on an offshore team on the other side of the world for adding/removing users from the system, support calls, ticket tracking, and other client services. One day at work, an urgent escalated ticket from Martin, the offshore support team lead, fell into Craig's queue. Seated before his cubicle workstation, Craig opened the ticket right away:

The new GlobalCon support website is not working. Appears to have been taken over by ChatGPT. The entire support team is blocked by this.

Instead of feeling any sense of urgency, Craig snorted out loud from perverse amusement.

"What was that now?" The voice of Nellie, his coworker, wafted over the cubicle wall that separated them.

"Urgent ticket from the offshore team," Craig replied.

"What is it this time?" Nellie couldn't suppress her glee.

"They're dead in the water because the new support page was, quote, taken over by ChatGPT."

Nellie laughed out loud.

"Hey! I know humor is important to surviving this job." A level, more mature voice piped up behind Craig from the cube across from his. It belonged to Dana, his manager. "But it really is urgent if they're all blocked. Do your best to help, escalate to me if you get stuck."

"OK, thanks. I got this," Craig assured her.

He was already 99.999% certain that no part of their web domain had gone down or been conquered by a belligerent AI, or else he would've heard of it by now. To make sure, Craig opened support.globalcon.com in a browser tab: sure enough, it worked. Martin had supplied no further detail, no logs or screenshots or videos, and no steps to reproduce, which was sadly typical of most of these escalations. At a loss, Craig took a screenshot of the webpage, opened the ticket, and posted the following: Everything's fine on this end. If it's still not working for you, let's do a screenshare.

Granted, a screensharing session was less than ideal given the 12-hour time difference. Craig hoped that whatever nefarious shenanigans ChatGPT had allegedly committed were resolved by now.

The next day, Craig received an update. Still not working. The entire team is still blocked. We're too busy to do a screenshare, please resolve ASAP.

Craig checked the website again with both laptop and phone. He had other people visit the website for him, trying different operating systems and web browsers. Every combination worked. Two things mystified him: how was the entire offshore team having this issue, and how were they "too busy" for anything if they were all dead in the water? At a loss, Craig attached an updated screenshot to the ticket and typed out the best CYA response he could muster. The new support website is up and has never experienced any issues. With no further proof or steps to reproduce this, I don't know what to tell you. I think a screensharing session would be the best thing at this point.

The next day, Martin parroted his last message almost word for word, except this time he assented to a screensharing session, suggesting the next morning for himself.

It was deep into the evening when Craig set up his work laptop on his kitchen counter and started a call and session for Martin to join. "OK. Can you show me what you guys are trying to do?"

To his surprise, he watched Martin open up Microsoft Teams first thing. From there, Martin accessed a chat to the entire offshore support team from the CPO of GlobalCon. The message proudly introduced the new support website and outlined the steps for accessing it. One of those steps was to visit support.globalcon.com.

The web address was rendered as blue outlined text, a hyperlink. Craig observed Martin clicking the link. A web browser opened up. Lo and behold, the page that finally appeared was www.chatgpt.com.

Craig blinked with surprise. "Hang on! I'm gonna take over for a second."

Upon taking control of the session, Craig switched back to Teams and accessed the link's details. The link text was correct, but the link destination was ChatGPT. It seemed like a copy/paste error that the CPO had tried to fix, not realizing that they'd needed to do more than simply update the link text.

"This looks like a bad link," Craig said. "It got sent to your entire team. And all of you have been trying to access the support site with this link?"

"Correct," Martin replied.

Craig was glad he couldn't be seen frowning and shaking his head. "Lemme show you what I've been doing. Then you can show everyone else, OK?"

After surrendering control of the session, Craig patiently walked Martin through the steps of opening a web browser, typing support.globalcon.com into the header, and hitting Return. The site opened without any issue. From there, Craig taught Martin how to create a bookmark for it.

"Just click on that from now on, and it'll always take you to the right place," Craig said. "In the future, before you click on any hyperlink, make sure you hover your mouse over it to see where it actually goes. Links can be labeled one thing when they actually take you somewhere else. That's how phishing works."

"Oh," Martin said. "Thanks!"

The call ended on a positive note, but left Craig marveling at the irony of lecturing the tech support lead on Internet 101 in the dead of night.

It's a holiday in the US today, so we're taking a long weekend. We flip back to a classic story of a company wanting to fill 15 different positions by hiring only one person. It's okay, Martin handles the database. Original - Remy

A curious email arrived in Phil's Inbox. "Windows Support Engineer required. Must have experience of the following:" and then a long list of Microsoft products.

Phil frowned. The location was convenient; the salary was fine, just the list of software seemed somewhat intimidating. Nevertheless, he replied to the agency saying that he was interested in applying for the position.

A few days later, Phil met Jason, the guy from the recruitment agency, in a hotel foyer. "It's a young, dynamic company", the recruiter explained,"They're growing really fast. They've got tons of funding and their BI Analysis Suite is positioning them to be a leading player in their field."

Phil nodded. "Ummm, I'm a bit worried about this list of products", referring to the job description. "I've never dealt with Microsoft Proxy Server 1.0, and I haven't dealt with Windows 95 OSR2 for a long while."

"Don't worry," Jason assured, "The Director is more an idea man. He just made a list of everything he's ever heard of. You'll just be supporting Windows Server 2003 and their flagship application."

Phil winced. He was a vanilla network administrator – supporting a custom app wasn't quite what he was looking for, but he desperately wanted to get out of his current job.

A few days later, Phil arrived for his interview. The company had rented smart offices on a new business park on the edge of town. He was ushered into the conference room, where he was joined by The Director and The Manager.

"So", said The Manager. "You've seen our brochure?"

"Yeah", said Phil, glancing at the glossy brochure in front of him with bright, Barbie-pink lettering all over it.

"You've seen a demo version of our application – what do you think?"

"Well, I think that it's great!", said Phil. He'd done his research – there were over 115 companies offering something very similar, and theirs wasn't anything special. "I particularly like the icons."

"Wonderful!" The Director cheered while firing up PowerPoint. "These are our servers. We rent some rack space in a data center 100 miles away." Phil looked at the projected picture. It showed a rack of a dozen servers.

"They certainly look nice." said Phil. They did look nice – brand new with green lights.

"Now, we also rent space in another data center on the other side of the country," The Manager added.

"This one is in a former cold-war bunker!" he said proudly. "It's very secure!" Phil looked up at another photo of some more servers.

"What we want the successful applicant to do is to take care of the servers on a day to day basis, but we also need to move those servers to the other data center", said The Director. "Without any interruption of service."

"Also, we need someone to set up the IT for the entire office. You know, email, file & print, internet access – that kind of thing. We've got a dozen salespeople starting next week, they'll all need email."

"And we need it to be secure."

"And we need it to be documented."

Phil was scribbled notes as best he could while the interviewing duo tag teamed him with questions.

"You'll also provide second line support to end users of the application."

"And day-to-day IT support to our own staff. Any questions?"

Phil looked up. "Ah… which back-end database does the application use?" he asked, expecting the answer would be SQL Server or perhaps Oracle, but The Director's reply surprised him.

"Oh, we wrote our own database from scratch. Martin wrote it." Phil realized his mouth was open, and shut it. The Director saw his expression, and explained. "You see, off the shelf databases have several disadvantages – the data gets fragmented, they're not quick enough, and so on. But don't have to worry about that – Martin takes care of the database. Do you have any more questions?"

Phil frowned. "So, to summarize: you want a data center guy to take care of your servers. You want someone to migrate the application from one data center to another, without any outage. You want a network administrator to set up, document and maintain an entire network from scratch. You want someone to provide internal support to the staff. And you want a second line support person to support the our flagship application."

"Exactly", beamed The Director paternally. "We want one person who can do all those things. Can you do that?"

Phil took a deep breath. "I don't know," he replied, and that was the honest answer.

"Right", The Manager said. "Well, if you have any questions, just give either of us a call, okay?"

Moments later, Phil was standing outside, clutching the garish brochure with the pink letters. His head was spinning. Could he do all that stuff? Did he want to? Was Martin a genius or a madman to reinvent the wheel with the celebrated database?

In the end, Phil was not offered the job and decided it might be best to stick it out at his old job for a while longer. After all, compared to Martin, maybe his job wasn't so bad after all.

We talked about singletons a bit last week. That reminded John of a story from the long ago dark ages where we didn't have always accessible mobile Internet access.

At the time, John worked for a bank. The bank, as all banks do, wanted to sell mortgages. This often meant sending an agent out to meet with customers face to face, and those agents needed to show the customer what their future would look like with that mortgage- payment calculations, and pretty little graphs about equity and interest.

Today, this would be a simple website, but again, reliable Internet access wasn't a thing. So they built a client side application. They tested the heck out of it, and it worked well. Sales agents were happy. Customers were happy. The bank itself was happy.

Time passed, as it has a way of doing, and the agents started clamoring for a mobile web version, that they could use on their phones. Now, the first thought was, "Wire it up to the backend!" but the backend they had was a mainframe, and there was a dearth of mainframe developers. And while the mainframe was the source of truth, and the one place where mortgages actually lived, building a mortgage calculator that could do pretty visualizations was far easier- and they already had one.

The client app was in .NET, and it was easy enough to wrap the mortgage calculation objects up in a web service. A quick round of testing of the service proved that it worked just as well as the old client app, and everyone was happy - for awhile.

Sometimes, agents would run a calculation and get absolute absurd results. Developers, putting in exactly the same values into their test environment wouldn't see the bad output. Testing the errors in production didn't help either- it usually worked just fine. There was a Heisenbug, but how could a simple math calculation that had already been tested and used for years have a Heisenbug?

Well, the calculation ran by simulation- it simply iteratively applied payments and interest to generate the entire history of the loan. And as it turns out, because the client application which started this whole thing only ever needed one instance of the calculator, someone had made it a singleton. And in their web environment, this singleton wasn't scoped to a single request, it was a true global object, which meant when simultaneous requests were getting processed, they'd step on each other and throw off the iteration. And testing didn't find it right away, because none of their tests were simulating the effect of multiple simultaneous users.

The fix was simple- stop being a singleton, and ensure every request got its own instance. But it's also a good example of misapplication of patterns- there was no need in the client app to enforce uniqueness via the singleton pattern. A calculator that holds state probably shouldn't be a singleton in the first place.

[Advertisement] Utilize BuildMaster to release your software with confidence, at the pace your business demands. Download today!

Barry rolled into work at 8:30AM to see the project manager waiting at the door, wringing her hands and sweating. She paced a bit while Barry badged in, and then immediately explained the issue:

Today was a major release of their new features. This wasn't just a mere software change; the new release was tied to major changes to a new product line- actual widgets rolling off an assembly line right now. And those changes didn't work.

"I thought we tested this," Barry said.

"We did! And Stu called in sick today!"

Stu was the senior developer on the project, who had written most of the new code.

"I talked to him for a few minutes, and he's convinced it's a data issue. Something in the metadata or something?"

"I'll take a look," Barry said.

He skipped grabbing a coffee from the carafe and dove straight in.

Prior to the recent project, the code had looked something like this:

if (IsProduct1(_productId))
	_programId = 1;
elseif (IsProduct2(_productId))
	_programId = 2;
elseif (IsProduct3(_productId))
	_programId = 3;

Part of the project, however, was about changing the workflow for "Product 3". So Stu had written this code:

if (IsProduct1(_productId))
	_programId = 1;
else if (IsProduct2(_productId))
	_programId = 2;
else if (IsProduct3(_productId))
	_programId = 3;
	DoSomethingProductId3Specific1();
	DoSomethingProductId3Specific2();
	DoSomethingProductId3Specific3();

Since this is C# and not Python, it took Barry all of 5 seconds to spot this and figure out what the problem was and fix it:

if (IsProduct1(_productId))
{
	_programId = 1;
}
else if (IsProduct2(_productId))
{
	_programId = 2;
}
else if (IsProduct3(_productId))
{
	_programId = 3;
	DoSomethingProductId3Specific1();
	DoSomethingProductId3Specific2();
	DoSomethingProductId3Specific3();
}

This brings us to about 8:32. Now, given the problems, Barry wasn't about to just push this change- in addition to running pipeline tests (and writing tests that Stu clearly hadn't), he pinged the head of QA to get a tester on this fix ASAP. Everyone worked quickly, and that meant by 9:30 the fix was considered good and ready to be merged in and pushed to production. Sometime in there, while waiting for a pipeline to complete, Barry managed to grab a cup of coffee to wake himself up.

While Barry was busy with that, Stu had decided that he wasn't feeling that sick after all, and had rolled into the office around 9:00. Which meant that just as Barry was about to push the button to run the release pipeline, an "URGENT" email came in from Stu.

"Hey, everybody, I fixed that bug. Can we get this released ASAP?"

Barry went ahead and released the version that he'd already tested, but out of morbid curiosity, went and checked Stu's fix.

if (IsProduct1(_productId))
	_programId = 1;
else if (IsProduct2(_productId))
	_programId = 2;
else if (IsProduct3(_productId))
{
	_programId = 3;
}

if (IsProduct3(_productId))
{
	DoSomethingProductId3Specific1();
	DoSomethingProductId3Specific2();
	DoSomethingProductId3Specific3();
}

At least this version would have worked, though I'm not sure Stu fully understands what "{}"s mean in C#. Or in most programming languages, if we're being honest.

With Barry's work, the launch went off just a few minutes later than the scheduled time. Since the launch was successful, at the next company "all hands", the leadership team made sure to congratulate the people instrumental in making it happen: that is to say, the lead developer of the project, Stu.

Today is holiday in the US, where we celebrate a cosplay version of history with big meals and getting frustrated with our family. It's also a day where we are thankful- usually to not be at work, but also, thankful to not work with Brad. Original --Remy

Anita parked outside the converted garage, the printed graphic reading Global Entertainment Strategies (GES) above it. When the owner, an old man named Brad, had offered her a position after spotting her in a student computer lab, she thought he was crazy, but a background check confirmed everything he said. Now she wondered if her first intuition was correct.

“Anita, welcome!” Brad seemed to bounce like a toddler as he showed Anita inside. The walls of the converted garage were bare drywall; the wall-mounted AC unit rattled and spat in the corner. In three corners of the office sat discount computer desks. Walls partitioned off Brad’s office in the fourth corner.

He practically shoved Anita into an unoccupied desk. The computer seemed to be running an unlicensed version of Windows 8, with no Office applications of any kind. “Ross can fill you in!” He left the office, slamming the door shut behind him.

“Hi.” Ross rolled in his chair from his desk to Anita’s. “Brad’s a little enthusiastic sometimes.”

“I noticed. Uh, he never told me what game we’re working on, or what platform. Not even a title.”

Ross’s voice lowered to a whisper. “None of us know, either. We’ve been coding in Unity for now. He hired you as a programmer, right? Well, right now we just need someone to manage our documentation. I suggest you prepare yourself.”

Ross led Anita into Brad’s office. Above a cluttered desk hung a sagging whiteboard. Every square inch was covered by one, sometimes several, overlapping sticky notes. Each had a word or two written in Brad’s scrawl.

“We need more than just random post-its with ‘big guns!’ and ‘more action!’” Ross said. “We don’t even know what the title is! We’re going crazy without some kind of direction.”

Anita stared at the wall of sticky notes, feeling her sanity slipping from her mind like a wet noodle. “I’ll try.”

Sticky Escalation

Brad, can we switch to Word for our documentation? It’s getting harder
to read your handwriting, and there’s a lot of post-its that have
nothing to do with the game. This will make it easier to proceed with
development. -Anita

Two minutes after she sent the email, Brad barged out of his office. “Anita, why spend thousands of dollars on software licenses when this works just fine? If you can’t do your job with the tools you have, what kind of a programmer does that make you?”

“Brad, this isn’t going to work forever. Your whiteboard is almost out of room, and you won’t take down any of your non-game stickies!”

“I can’t take any of them down, Anita! Any of them!” He slammed the door to his office behind him.

The next day, Anita was greeted at the door by the enthusiastic Brad she had met before the interview. “I listened to reason, Anita. I hope this is enough for you to finish this documentation and get coding again!”

Brad led Anita into his office. On every wall surface, over the door, even covering part of the floor, were whiteboards. Sticky notes dotted nearly a third of the new whiteboard space.

“Now, Anita, if I don’t see new code from you soon, I may just have to let you go! Now get to work!”

Anita went to sit at her desk, then stopped. Instead, she grabbed a bright red sticky note, wrote the words “I QUIT” with a sharpy, barged into Brad’s office, and stuck it to his monitor. Brad was too stunned to talk as she left the converted garage.

The Avalanche

“Are you doing better?” Jason called Anita a few weeks later. Their short time together at GES has made them comrades-in-arms, and networking was crucial in the business.

“Much,” she said. “I got a real job with an indie developer in Santa Monica. We even have a wiki for our framework!”

“Well, listen to this. The day after you quit, the AC unit in the garage broke. I came into work to see Brad crying in a corner in his office. All of the sticky notes had curled in the humidity and fallen to the floor. The day after he got us all copies of Word.

“Too bad we still don’t know what the title of the game is.”

[Advertisement] ProGet’s got you covered with security and access controls on your NuGet feeds. Learn more.

One of the clients for Rudolf's company was getting furious with them. The dev team was in constant firefighting mode. No new features ever shipped, because the code-base was too fragile to add new features to without breaking something else. What few tests existed were broken. Anyone put on the project burned out and fled in months, sometimes weeks, and rarely after only a few days.

Rudolf wasn't too pleased when management parachuted him into the project to save it. But when he pulled the code and started poking around, it looked bad but not unsalvageable. The first thing he noticed is that, when following the instructions in the README, he couldn't build and run the application. Or maybe he wasn't following the instructions in the README, because the README was a confusing and incoherent mess, which included snippets from unresolved merges. Rudolf's first few days on the project were spent just getting it building and running locally, then updating the README. Once that was done, he started in on fixing the broken tests. There was a lot of work to be done, but it was all doable work. Rudolf could lay out a plan of how to get the project back on track and start delivering new features.

It's about then that Steve, the product owner, called Rudolf in to his office. "What the hell do you think you're doing?"

Rudolf blinked. "Um… what I was asked to do?"

"Three days and you just commit a README update? A couple of unit tests?"

"Well, it was out of date and meant I couldn't-"

"Our client is crazy about their business," Steve said. "Not about READMEs. Not about unit tests. None of that actually helps their business."

Rudolf bit back a "well, actually," while Steve ranted.

"Next thing you're going to tell me is that we should waste time on refactoring, like everybody else did. Time is money, time is new features, and new features are money!"

Suddenly, Rudolf realized that the reason the project had such a high burnout rate had nothing to do with the code itself. And while Rudolf could fix the code, he couldn't fix Steve. So, he did what everyone else had done: kept his head down and struggled through for a few months, and kept poking his manager to get him onto another project. In the meantime, he made this code slightly better for the next person, despite Steve's ranting. Rudolf eventually moved on, and Steve told everyone he was the worst developer that had ever touched the project.

The customer continued to be unhappy.

[Advertisement] Continuously monitor your servers for configuration changes, and report when there's configuration drift. Get started with Otter today!

Back in the heady days of the DotCom Bubble, startups were thick on the ground, and Venture Capital money was a flood- lifting startups atop a tsunami only to crash them back into the ground a short time later. Taliesyn once worked for one such startup.

Taliesyn's manager, Irving, was an expert in AI. In the age of the DotCom Bubble, this meant Irving knew LISP. Knowing LISP was valuable here, because their core product was a database system built on LISP- specifically the Common LISP Object System, an object-oriented bolt-on for LISP.

It was an object-oriented database system, akin to the modern NoSQL databases, but its architecture left a few things to be desired. First, since disk reads and writes were expensive operations, the system avoided them. All updates to data were done in memory, and someday, at some point, when the program felt like it, the changes would be written to disk. This meant that any failures or crashes could lose potentially days of data. Worse, the data was stored in one gigantic text file, which meant corruption could easily take out the entire database.

These were legitimate problems, and due to the architecture, they were going to be challenging to resolve. That was the startup life, however- they had a minimum viable product, and just needed to pour energy into making it something worth using.

Everyone looked to Irving, the AI and LISP expert, to guide them through this.

Irving saw where the real problems lay. "Your database doesn't support SQL," Irving said.

"Well, sure," Taliesyn said. "That's our selling point."

Irving nodded, and then, speaking slowly, as if to a particularly dense child, said, "A database needs to support SQL."

"I mean, a relational database, sure," Taliesyn said, "but we're using an object oriented data model which means we don't need to do joins or-" Taliesyn kept talking, explaining why their database didn't support SQL, why it shouldn't support SQL, and why SQL support was not only off the roadmap, but so far off the roadmap that it was labelled "Here there be dragons."

Irving nodded along, and ended the conversation with a, "Sure, that makes sense."

Everything was fine for a few weeks, until one of Taliesyn's peers on a different team, Angela, shot him an email: "Hey, marketing is getting antsy about a SQL demo, and I've got half a dozen features blocked until you get me a build with that functionality. What's the timeline like?"

Taliesyn was uncertain about what the timeline was like, since he had now clearly slipped into a parallel universe. He politely informed his peer that he had no idea what was going on, but would find out. It didn't take a great sleuth to figure out that Irving had started appending his own features to the roadmap.

Taliesyn tracked Irving down and demanded to know what was going on.

"A customer is already using it!" Irving protested. "They wrote it themselves! So we should be able to do it easily. Frankly, it's embarrassing to say that we can't do something with our own tools that the customer is already doing!"

Taliesyn knew that Irving was either wrong or lying, and asked to talk to the customer. Irving was, in fact, wrong. The customer had used LISP to write an extension to their object database (another one of those selling point features), and this extension used ODBC to open a connection to a relational database. This let them move data between the two different database systems.

"Irving," Taliesyn said, "they're not using SQL on our database, they're connecting to a relational database and using SQL there. SQL doesn't make sense for our data structure! We don't have tables, or keys, or relationships. It's objects! You're promising features that we can't deliver."

Irving was unmoved. "We are making a database. A database must have SQL capability. It's structured query language- it's right there in the name! Our structure should be queried by structured query language."

Taliesyn tried going over Irving's head, but upper management had no interest in actually solving personnel problems. Irving's buzzword laden ideas about why SQL was required seemed to jive with what their customers wanted, and the idea of shipping a non-SQL database was getting lost in the endless quest for the next round of VC funding.

The result was a bolted on monster of broken implementations of SQL that infuriated the few customers who tried it, Irving got promoted, and Taliesyn ran as far from the trainwreck as possible before the company flamed out.

[Advertisement] Continuously monitor your servers for configuration changes, and report when there's configuration drift. Get started with Otter today!

Alan worked on a website which had a weekly event which unlocked at 9PM, Saturday, Eastern Time. The details of the event didn't matter, but this little promotion had been running for about a year and a half, without any issues.

Until one day, someone emailed Alan: "Hey, I checked the site on Sunday, and the countdown timer displays 00:00:00."

Alan didn't check their email on Sunday, and when they checked the site, everything was fine, so he set himself a reminder to check things out next Sunday, and left things alone for a week.

Well, Alan forgot on Sunday. It was his day off after all, but he did remember to check first thing on Monday. When he came in, the timer read 00:00:00. Alan had a twelve o'clock flasher. Oddly, when he checked it after grabbing some coffee, the timer now showed the correct value.

It was time to dig through the code. Now, this story happened quite some time ago, so the countdown timer itself was a Flash widget. But the widget received a parameter from the HTML DOM, which itself was generated by PHP, and it didn't take long to find the SQL query it was using to find the next event: SELECT next_event FROM event_timer.

Yes, event_timer was a one column, one row table. A quick search through the codebase found the table referenced only one other place: backend_settimer.php.

All the pieces came together: resetting the timer was an entirely manual process. Every Monday, Tina came in, and assuming she remembered, she reset the timer. Some days she did it late, or forgot entirely. Some weeks, she was on vacation. Maybe she remembered to delegate, maybe she didn't.

For roughly 72 weeks, this had been how things had been working.

The good news was that the date was getting parsed with the PHP strtotime function, which meant Alan merely had to go to the backend_settimer.php and set the value to Next Saturday 9pm, and let Tina know she didn't need to do this anymore.

[Advertisement] Continuously monitor your servers for configuration changes, and report when there's configuration drift. Get started with Otter today!