Ellis knew she needed a walk after she hurried off of Zoom at the end of the meeting to avoid sobbing in front of the group.

She'd just been attending a free online seminar regarding safe job hunting on the Internet. Having been searching since the end of January, Ellis had already picked up plenty of first-hand experience with the modern job market, one rejection at a time. She thought she'd attend the seminar just to see if there were any additional things she wasn't aware of. The seminar had gone well, good information presented in a clear and engaging way. But by the end of it, Ellis was feeling bleak. Goodness gracious, she'd already been slogging through months of this. Hundreds of job applications with nothing to show for it. All of the scams out there, all of the bad actors preying on people desperate for their and their loved ones' survival!

Ellis' childhood had been plagued with anxiety and depression. It was only as an adult that she'd learned any tricks for coping with them. These tricks had helped her avoid spiraling into full-on depression for the past several years. One such trick was to stop and notice whenever those first feelings hit. Recognize them, feel them, and then respond constructively.

First, a walk. Going out where there were trees and sunshine: Ellis considered this "garbage collection" for her brain. So she stepped out the front door and started down a tree-lined path near her house, holding on to that bleak feeling. She was well aware that if she didn't address it, it would take root and grow into hopelessness, self-loathing, fear of the future. It would paralyze her, leave her curled up on the couch doing nothing. And it would all happen without any words issuing from her inner voice. That was the most insidious thing. It happened way down deep in a place where there were no words at all.

Once she returned home, Ellis forced herself to sit down with a notebook and pencil and think very hard about what was bothering her. She wrote down each sentiment:

This was the first step of a reframing technique she'd just read about in the book Right Kind of Wrong by Amy Edmonson. With the words out, it was possible to look at each statement and determine whether it was rational or irrational, constructive or harmful. Each statement could be replaced with something better.

Ellis proceeded step by step through the list.

• Yes, this will end. Everything ends.
• There's nothing wrong with me. Most businesses are swamped with applications. There's a good chance mine aren't even being looked at before they're being auto-rejected. Remember the growth mindset you learned from Carol Dweck. Each application and interview is giving me experience and making me a better candidate.
• This job market is a novel context that changes every day. That means failure is not only inevitable, it's the only way forward.

Ellis realized that her job hunt was very much like a search algorithm trying to find a path through a maze. When the algorithm encountered a dead end, did it deserve blame? Was it an occasion for shame, embarrassment, and despair? Of course not. Simply backtrack and keep going with the knowledge gained.

Yes, there was truth to the fact that this was the toughest job market Ellis had ever experienced. Therefore, taking a note from Viktor Frankl, she spent a moment reimagining the struggle in a way that made it meaningful to her. Ellis began viewing her job hunt in this dangerous market, her gradual accumulation of survival information, as an act of resistance against it. She now hoped to write all about her experience once she was on the other side, in case her advice might help even one other person in her situation save time and frustration.

While unemployed, she also had the opportunity to employ the search algorithm against entirely new mazes. Could Ellis expand her freelance writing into a sustainable gig, for instance? That would mean exploring all the different ways to be a freelance writer, something Ellis was now curious and excited to explore.

[Advertisement] Keep all your packages and Docker containers in one place, scan for vulnerabilities, and control who can access different feeds. ProGet installs in minutes and has a powerful free version with a lot of great features that you can upgrade when ready.Learn more.

Everyone's got workplace woes. The clueless manager; the disruptive coworker; the cube walls that loom ever higher as the years pass, trapping whatever's left of your soul.

But sometimes, Satan really leaves his mark on a joint. I worked Tech Support there. This is my story. Who am I? Just call me Anonymous.

It starts at the top. A call came in from Lawrence Gibbs, the CEO himself, telling us that a conference room printer was, quote, "leaking." He didn't explain it, he just hung up. The boss ordered me out immediately, told me to step on it. I ignored the elevator, racing up the staircase floor after floor until I reached the dizzying summit of C-Town.

There's less oxygen up there, I'm sure of it. My lungs ached and my head spun as I struggled to catch my breath. The fancy tile and high ceilings made a workaday schmuck like me feel daunted, unwelcome. All the same, I gathered myself and pushed on, if only to learn what on earth "leaking" meant in relation to a printer.

I followed the signs on the wall to the specified conference room. In there, the thermostat had been kicked down into the negatives. The cold cut through every layer of mandated business attire, straight to bone. The scene was thick with milling bystanders who hugged themselves and traded the occasional nervous glance. Gibbs was nowhere to be found.

Remembering my duty, I summoned my nerve. "Tech Support. Where's the printer?" I asked.

Several pointing fingers showed me the way. The large printer/scanner was situated against the far wall, flanking an even more enormous conference table. Upon rounding the table, I was greeted with a grim sight: dozens of sheets of paper strewn about the floor like blood spatter. Everyone was keeping their distance; no one paid me any mind as I knelt to gather the pages. There were 30 in all. Each one was blank on one side, and sported some kind of large, blotchy ring on the other. Lord knew I drank enough java to recognize a coffee mug stain when I saw one, but these weren't actual stains. They were printouts of stains.

The printer was plugged in. No sign of foul play. As I knelt there, unseen and unheeded, I clutched the ruined papers to my chest. Someone had wasted a tree and a good bit of toner, and for what? How'd it go down? Surely Gibbs knew more than he'd let on. The thought of seeking him out, demanding answers, set my heart to pounding. It was no good, I knew. He'd play coy all day and hand me my pink slip if I pushed too hard. As much as I wanted the truth, I had a stack of unpaid bills at home almost as thick as the one in my arms. I had to come up with something else.

There had to be witnesses among the bystanders. I stood up and glanced among them, seeking out any who would return eye contact. There: a woman who looked every bit as polished as everyone else. But for once, I got the feeling that what lay beneath the facade wasn't rotten.

With my eyes, I pleaded for answers.

Not here, her gaze pleaded back.

I was getting somewhere, I just had to arrange for some privacy. I hurried around the table again and weaved through bystanders toward the exit, hoping to beat it out of that icebox unnoticed. When I reached the threshold, I spotted Gibbs charging up the corridor, smoldering with entitlement. "Where the hell is Tech Support?!"

I froze a good distance away from the oncoming executive, whose voice I recognized from a thousand corporate presentations. Instead of putting me to sleep this time, it jolted down my spine like lightning. I had to think fast, or I was gonna lose my lead, if not my life.

"I'm right here, sir!" I said. "Be right back! I, uh, just need to find a folder for these papers."

"I've got one in my office."

A woman's voice issued calmly only a few feet behind me. I spun around, and it was her, all right, her demeanor as cool as our surroundings. She nodded my way. "Follow me."

My spirits soared. At that moment, I would've followed her into hell. Turning around, I had the pleasure of seeing Gibbs stop short with a glare of contempt. Then he waved us out of his sight.

Once we were out in the corridor, she took the lead, guiding me through the halls as I marveled at my luck. Eventually, she used her key card on one of the massive oak doors, and in we went.

You could've fit my entire apartment into that office. The place was spotless. Mini-fridge, espresso machine, even couches: none of it looked used. There were a couple of cardboard boxes piled up near her desk, which sat in front of a massive floor-to-ceiling window admitting ample sunlight.

She motioned toward one of the couches, inviting me to sit. I shook my head in reply. I was dying for a cigarette by that point, but I didn't dare light up within this sanctuary. Not sure what to expect next, I played it cautious, hovering close to the exit. "Thanks for the help back there, ma'am."

"Don't mention it." She walked back to her desk, opened up a drawer, and pulled out a brand-new manila folder. Then she returned to conversational distance and proffered it my way. "You're from Tech Support?"

There was pure curiosity in her voice, no disparagement, which was encouraging. I accepted the folder and stuffed the ruined pages inside. "That's right, ma'am."

She shook her head. "Please call me Leila. I started a few weeks ago. I'm the new head of HR."

Human Resources. That acronym, which usually put me on edge, somehow failed to raise my hackles. I'd have to keep vigilant, of course, but so far she seemed surprisingly OK. "Welcome aboard, Leila. I wish we were meeting in better circumstances." Duty beckoned. I hefted the folder. "Printers don't just leak."

"No." Leila glanced askance, grave.

"Tell me what you saw."

"Well ..." She shrugged helplessly. "Whenever Mr. Gibbs gets excited during a meeting, he tends to lean against the printer and rest his coffee mug on top of it. Today, he must've hit the Scan button with his elbow. I saw the scanner go off. It was so bright ..." She trailed off with a pained glance downward.

"I know this is hard," I told her when the silence stretched too long. "Please, continue."

Leila summoned her mettle. "After he leaned on the controls, those pages spilled out of the printer. And then ... then somehow, I have no idea, I swear! Somehow, all those pages were also emailed to me, Mr. Gibbs' assistant, and the entire board of directors!"

The shock hit me first. My eyes went wide and my jaw fell. But then I reminded myself, I'd seen just as crazy and worse as the result of a cat jumping on a keyboard. A feline doesn't know any better. A top-level executive, on the other hand, should know better.

"Sounds to me like the printer's just fine," I spoke with conviction. "What we have here is a CEO who thinks it's OK to treat an expensive piece of office equipment like his own personal fainting couch."

"It's terrible!" Leila's gaze burned with purpose. "I promise, I'll do everything I possibly can to make sure something like this never happens again!"

I smiled a gallows smile. "Not sure what anyone can do to fix this joint, but the offer's appreciated. Thanks again for your help."

Now that I'd seen this glimpse of better things, I selfishly wanted to linger. But it was high time I got outta there. I didn't wanna make her late for some meeting or waste her time. I backed up toward the door on feet that were reluctant to move.

Leila watched me with a look of concern. "Mr. Gibbs was the one who called Tech Support. I can't close your ticket for you; you'll have to get him to do it. What are you going to do?"

She cared. That made leaving even harder. "I dunno yet. I'll think of something."

I turned around, opened the massive door, and put myself on the other side of it in a hurry, using wall signs to backtrack to the conference room. Would our paths ever cross again? Unlikely. Someone like her was sure to get fired, or quit out of frustration, or get corrupted over time.

It was too painful to think about, so I forced myself to focus on the folder of wasted pages in my arms instead. It felt like a mile-long rap sheet. I was dealing with an alleged leader who went so far as to blame the material world around him rather than accept personal responsibility. I'd have to appeal to one or more of the things he actually cared about: himself, his bottom line, his sense of power.

By the time I returned to the conference room to face the CEO, I knew what to tell him. "You're right, sir, there's something very wrong with this printer. We're gonna take it out here and give it a thorough work-up."

That was how I was able to get the printer out of that conference room for good. Once it underwent "inspection" and "testing," it received a new home in a previously unused closet. Whenever Gibbs got to jawing in future meetings, all he could do was lean against the wall. Ticket closed.

Gibbs remained at the top, doing accursed things that trickled down to the roots of his accursed company. But at least from then on, every onboarding slideshow included a photo of one of the coffee ring printouts, with the title Respect the Equipment.

Thanks, Leila. I can live with that.

Once upon a time, when the Web was young, if you wanted to be a cool kid, you absolutely needed two things on your website: a guestbook for people to sign, and a hit counter showing how many people had visited your Geocities page hosting your Star Trek fan fiction.

These days, we don't see them as often, but companies still like to track the information, especially when it comes to counting downloads. So when Justin started on a new team and saw a download count in their analytics, he didn't think much of it at all. Nor did he think much about it when he saw the download count displayed on the download page.

Another thing that Justin didn't think much about was big piles of commits getting merged in overnight, at least not at first. But each morning, Justin needed to pull in a long litany of changes from a user named "MrStinky". For the first few weeks, Justin was too preoccupied with getting his feet under him, so he didn't think about it too much.

But eventually, he couldn't ignore what he saw in the git logs.

docs: update download count to 51741
docs: update download count to 51740
docs: update download count to 51738

And each commit was exactly what the name implied, a diff like:

- 51740
+ 51741

Each time a user clicked the download link, a ping was sent to their analytics system. Throughout the day, the bot "MrStinky" would query the analytics tool, and create new commits that updated the counter. Overnight, it would bundle those commits into a merge request, approve the request, merge the changes, and then redeploy what was at the tip of main.

"But, WHY?" Justin asked his peers.

One of them just shrugged. "It seemed like the easiest and fastest way at the time?"

"I wanted to wire Mr Stinky up to our content management system's database, but just never got around to it. And this works fine," said another.

Much like the rest of the team, Justin found that there were bigger issues to tackle.

Adam's organization was going through a period of rapid growth. Part of this growth was spinning up new backend services to support new functionality. The growth would have been extremely fast, except for one thing applying back pressure: for some reason, spinning up a new service meant recompiling and redeploying all the other services.

Adam didn't understand why, but it seemed like an obvious place to start poking at something for improvement. All of the services depended on a library called "ServiceLib"- though not all of them actually used the library. The library was a set of utilities for administering, detecting, and interacting with services in their environment- essentially a homegrown fabric/bus architecture.

It didn't take long, looking at the source control history, to understand why there was a rebuild after the release of every service. Each service triggered a one line change in this:

enum class Services
{
    IniTechBase = 103,
    IniTechAdvanced = 99,
    IniTechFooServer = 102,
    …
}

Each service had a unique, numerical identifier, and this mapped them into an enumerated type.

Adam went to the tech lead, Raymond. "Hey, I've got an idea for speeding up our release process- we should stop hard coding the service IDs in ServiceLib."

Raymond looked at Adam like one might examine an over-enthusiastic lemur. "They're not hard-coded. We store them in an enum."

Eventually Raymond got promoted- for all of their heroic work on managing this rapidly expanding library of services. The new tech lead who came on was much more amenable to "not storing rapidly changing service IDs in an enum", and "not making every service depend on a library they often don't need", and "putting admin functionality in every service because they're linked to that library whether they like it or not."

Eventually, ServiceLib became its own service, and actually helped- instead of hindered- delivering new functionality.

Unfortunately, with no more highly visible heroics to deliver functionality, the entire department became a career dead end. Sure, they delivered on time and under budget consistently, but there were no rockstar developers like Raymond on the team anymore, the real up-and-comers who were pushing themselves.

Once upon a time, there was a bank whose business relied on a mainframe. As the decades passed and the 21st century dawned, the bank's bigwigs realized they had to upgrade their frontline systems to applications built in Java and .NET, but—for myriad reasons that boiled down to cost, fear, and stubbornness—they didn't want to migrate away from the mainframe entirely. They also didn't want the new frontline systems to talk directly to the mainframe or vice-versa. So they tasked old-timer Edgar with writing some middleware. Edgar's brainchild was a Windows service that took care of receiving frontline requests, passing them to the mainframe, and sending the responses back.

Edgar's middleware worked well, so well that it was largely forgotten about. It outlasted Edgar himself, who, after another solid decade of service, moved on to another company.

A few years later, our submitter John F. joined the bank's C# team. By this point, the poor middleware seemed to be showing its age. A strange problem had arisen: between 8:00AM and 5:00PM, every 45 minutes or so, it would lock up and have to be restarted. Outside of those hours, there was no issue. The problem was mitigated by automatic restarts, but it continued to inflict pain and aggravation upon internal users and external customers. A true solution had to be found.

Unfortunately, Edgar was long gone. The new "owner" of the middleware was an infrastructure team containing zero developers. Had Edgar left them any documentation? No. Source code? Sort of. Edgar had given a copy of the code to his friend Bob prior to leaving. Unfortunately, Bob's copy was a few point releases behind the version of middleware running in production. It was also in C, and there were no C developers to be found anywhere in the company.

And so, the bank's bigwigs cobbled together a diverse team of experts. There were operating system people, network people, and software people ... including the new guy, John. Poor John had the unenviable task of sifting through Edgar's source code. Just as the C# key sits right next to the C key on a piano, reasoned the bigwigs, C# couldn't be that different from C.

John toiled in an unfamiliar language with no build server or test environment to aid him. It should be no great surprise that he got nowhere. A senior coworker suggested that he check what Windows' Process Monitor registered when the middleware was running. John allowed a full day to pass, then looked at the results: it was now clear that the middleware was constantly creating and destroying threads. John wrote a Python script to analyze the threads, and found that most of them lived for only seconds. However, every 5 minutes, a thread was created but never destroyed.

This only happened during the hours of 8:00AM to 5:00PM.

At the next cross-functional team meeting behind closed doors, John finally had something of substance to report to the large group seated around the conference room table. There was still a huge mystery to solve: where were these middleware-killing threads coming from?

"Wait a minute! Wasn't Frank doing something like that?" one of the other team members piped up.

"Frank!" A department manager with no technical expertise, who insisted on attending every meeting regardless, darted up straight in his chair. For once, he wasn't haranguing them for their lack of progress. He resembled a wolf who'd sniffed blood in the air. "You mean Frank from Accounting?!"

This was the corporate equivalent of an arrest warrant. Frank from Accounting was duly called forth.

"That's my program." Frank stood before the table, laid back and blithe despite the obvious frayed nerves of several individuals within the room. "It queries the middleware every 5 minutes."

They were finally getting somewhere. Galvanized, John's heart pounded. "How?" he asked.

"Well, it could be that the middleware is down, so first, my program opens a connection just to make sure it's working," Frank explained. "If that works, it opens another connection and sends the query."

John's confusion mirrored the multiple frowns that filled the room. He forced himself to carefully parse what he'd just heard. "What happens to the first connection?"

"What do you mean?" Frank asked.

"You said your program opens two connections. What do you do with the first one?"

"Oh! I just use that one to test whether the middleware is up."

"You don't need to do that!" one of the networking experts snarled. "For Pete's sake, take that out of your code! Don't you realize you're tanking this thing for everyone else?"

Frank's expression made clear that he was entirely oblivious to the chaos wrought by his program. Somehow, he survived the collective venting of frustration that followed within that conference room. After one small update to Frank's program, the middleware stabilized—for the time being. And while Frank became a scapegoat and villain to some, he was a hero to many, many more. After all, he single-handedly convinced the bank's bigwigs that the status quo was too precarious. They began to plan out a full migration away from mainframe, a move that would free them from their dependence upon aging, orphaned middleware.

Now that the mystery had been solved, John knew where to look in Edgar's source code. The thread pool had a limit of 10, and every thread began by waiting for input. The middleware could handle bad input well enough, but it hadn't been written to handle the case of no input at all.

[Advertisement] Utilize BuildMaster to release your software with confidence, at the pace your business demands. Download today!

Our anonymous submitter, whom we'll call Craig, worked for GlobalCon. GlobalCon relied on an offshore team on the other side of the world for adding/removing users from the system, support calls, ticket tracking, and other client services. One day at work, an urgent escalated ticket from Martin, the offshore support team lead, fell into Craig's queue. Seated before his cubicle workstation, Craig opened the ticket right away:

The new GlobalCon support website is not working. Appears to have been taken over by ChatGPT. The entire support team is blocked by this.

Instead of feeling any sense of urgency, Craig snorted out loud from perverse amusement.

"What was that now?" The voice of Nellie, his coworker, wafted over the cubicle wall that separated them.

"Urgent ticket from the offshore team," Craig replied.

"What is it this time?" Nellie couldn't suppress her glee.

"They're dead in the water because the new support page was, quote, taken over by ChatGPT."

Nellie laughed out loud.

"Hey! I know humor is important to surviving this job." A level, more mature voice piped up behind Craig from the cube across from his. It belonged to Dana, his manager. "But it really is urgent if they're all blocked. Do your best to help, escalate to me if you get stuck."

"OK, thanks. I got this," Craig assured her.

He was already 99.999% certain that no part of their web domain had gone down or been conquered by a belligerent AI, or else he would've heard of it by now. To make sure, Craig opened support.globalcon.com in a browser tab: sure enough, it worked. Martin had supplied no further detail, no logs or screenshots or videos, and no steps to reproduce, which was sadly typical of most of these escalations. At a loss, Craig took a screenshot of the webpage, opened the ticket, and posted the following: Everything's fine on this end. If it's still not working for you, let's do a screenshare.

Granted, a screensharing session was less than ideal given the 12-hour time difference. Craig hoped that whatever nefarious shenanigans ChatGPT had allegedly committed were resolved by now.

The next day, Craig received an update. Still not working. The entire team is still blocked. We're too busy to do a screenshare, please resolve ASAP.

Craig checked the website again with both laptop and phone. He had other people visit the website for him, trying different operating systems and web browsers. Every combination worked. Two things mystified him: how was the entire offshore team having this issue, and how were they "too busy" for anything if they were all dead in the water? At a loss, Craig attached an updated screenshot to the ticket and typed out the best CYA response he could muster. The new support website is up and has never experienced any issues. With no further proof or steps to reproduce this, I don't know what to tell you. I think a screensharing session would be the best thing at this point.

The next day, Martin parroted his last message almost word for word, except this time he assented to a screensharing session, suggesting the next morning for himself.

It was deep into the evening when Craig set up his work laptop on his kitchen counter and started a call and session for Martin to join. "OK. Can you show me what you guys are trying to do?"

To his surprise, he watched Martin open up Microsoft Teams first thing. From there, Martin accessed a chat to the entire offshore support team from the CPO of GlobalCon. The message proudly introduced the new support website and outlined the steps for accessing it. One of those steps was to visit support.globalcon.com.

The web address was rendered as blue outlined text, a hyperlink. Craig observed Martin clicking the link. A web browser opened up. Lo and behold, the page that finally appeared was www.chatgpt.com.

Craig blinked with surprise. "Hang on! I'm gonna take over for a second."

Upon taking control of the session, Craig switched back to Teams and accessed the link's details. The link text was correct, but the link destination was ChatGPT. It seemed like a copy/paste error that the CPO had tried to fix, not realizing that they'd needed to do more than simply update the link text.

"This looks like a bad link," Craig said. "It got sent to your entire team. And all of you have been trying to access the support site with this link?"

"Correct," Martin replied.

Craig was glad he couldn't be seen frowning and shaking his head. "Lemme show you what I've been doing. Then you can show everyone else, OK?"

After surrendering control of the session, Craig patiently walked Martin through the steps of opening a web browser, typing support.globalcon.com into the header, and hitting Return. The site opened without any issue. From there, Craig taught Martin how to create a bookmark for it.

"Just click on that from now on, and it'll always take you to the right place," Craig said. "In the future, before you click on any hyperlink, make sure you hover your mouse over it to see where it actually goes. Links can be labeled one thing when they actually take you somewhere else. That's how phishing works."

"Oh," Martin said. "Thanks!"

The call ended on a positive note, but left Craig marveling at the irony of lecturing the tech support lead on Internet 101 in the dead of night.

It's a holiday in the US today, so we're taking a long weekend. We flip back to a classic story of a company wanting to fill 15 different positions by hiring only one person. It's okay, Martin handles the database. Original - Remy

A curious email arrived in Phil's Inbox. "Windows Support Engineer required. Must have experience of the following:" and then a long list of Microsoft products.

Phil frowned. The location was convenient; the salary was fine, just the list of software seemed somewhat intimidating. Nevertheless, he replied to the agency saying that he was interested in applying for the position.

A few days later, Phil met Jason, the guy from the recruitment agency, in a hotel foyer. "It's a young, dynamic company", the recruiter explained,"They're growing really fast. They've got tons of funding and their BI Analysis Suite is positioning them to be a leading player in their field."

Phil nodded. "Ummm, I'm a bit worried about this list of products", referring to the job description. "I've never dealt with Microsoft Proxy Server 1.0, and I haven't dealt with Windows 95 OSR2 for a long while."

"Don't worry," Jason assured, "The Director is more an idea man. He just made a list of everything he's ever heard of. You'll just be supporting Windows Server 2003 and their flagship application."

Phil winced. He was a vanilla network administrator – supporting a custom app wasn't quite what he was looking for, but he desperately wanted to get out of his current job.

A few days later, Phil arrived for his interview. The company had rented smart offices on a new business park on the edge of town. He was ushered into the conference room, where he was joined by The Director and The Manager.

"So", said The Manager. "You've seen our brochure?"

"Yeah", said Phil, glancing at the glossy brochure in front of him with bright, Barbie-pink lettering all over it.

"You've seen a demo version of our application – what do you think?"

"Well, I think that it's great!", said Phil. He'd done his research – there were over 115 companies offering something very similar, and theirs wasn't anything special. "I particularly like the icons."

"Wonderful!" The Director cheered while firing up PowerPoint. "These are our servers. We rent some rack space in a data center 100 miles away." Phil looked at the projected picture. It showed a rack of a dozen servers.

"They certainly look nice." said Phil. They did look nice – brand new with green lights.

"Now, we also rent space in another data center on the other side of the country," The Manager added.

"This one is in a former cold-war bunker!" he said proudly. "It's very secure!" Phil looked up at another photo of some more servers.

"What we want the successful applicant to do is to take care of the servers on a day to day basis, but we also need to move those servers to the other data center", said The Director. "Without any interruption of service."

"Also, we need someone to set up the IT for the entire office. You know, email, file & print, internet access – that kind of thing. We've got a dozen salespeople starting next week, they'll all need email."

"And we need it to be secure."

"And we need it to be documented."

Phil was scribbled notes as best he could while the interviewing duo tag teamed him with questions.

"You'll also provide second line support to end users of the application."

"And day-to-day IT support to our own staff. Any questions?"

Phil looked up. "Ah… which back-end database does the application use?" he asked, expecting the answer would be SQL Server or perhaps Oracle, but The Director's reply surprised him.

"Oh, we wrote our own database from scratch. Martin wrote it." Phil realized his mouth was open, and shut it. The Director saw his expression, and explained. "You see, off the shelf databases have several disadvantages – the data gets fragmented, they're not quick enough, and so on. But don't have to worry about that – Martin takes care of the database. Do you have any more questions?"

Phil frowned. "So, to summarize: you want a data center guy to take care of your servers. You want someone to migrate the application from one data center to another, without any outage. You want a network administrator to set up, document and maintain an entire network from scratch. You want someone to provide internal support to the staff. And you want a second line support person to support the our flagship application."

"Exactly", beamed The Director paternally. "We want one person who can do all those things. Can you do that?"

Phil took a deep breath. "I don't know," he replied, and that was the honest answer.

"Right", The Manager said. "Well, if you have any questions, just give either of us a call, okay?"

Moments later, Phil was standing outside, clutching the garish brochure with the pink letters. His head was spinning. Could he do all that stuff? Did he want to? Was Martin a genius or a madman to reinvent the wheel with the celebrated database?

In the end, Phil was not offered the job and decided it might be best to stick it out at his old job for a while longer. After all, compared to Martin, maybe his job wasn't so bad after all.

We talked about singletons a bit last week. That reminded John of a story from the long ago dark ages where we didn't have always accessible mobile Internet access.

At the time, John worked for a bank. The bank, as all banks do, wanted to sell mortgages. This often meant sending an agent out to meet with customers face to face, and those agents needed to show the customer what their future would look like with that mortgage- payment calculations, and pretty little graphs about equity and interest.

Today, this would be a simple website, but again, reliable Internet access wasn't a thing. So they built a client side application. They tested the heck out of it, and it worked well. Sales agents were happy. Customers were happy. The bank itself was happy.

Time passed, as it has a way of doing, and the agents started clamoring for a mobile web version, that they could use on their phones. Now, the first thought was, "Wire it up to the backend!" but the backend they had was a mainframe, and there was a dearth of mainframe developers. And while the mainframe was the source of truth, and the one place where mortgages actually lived, building a mortgage calculator that could do pretty visualizations was far easier- and they already had one.

The client app was in .NET, and it was easy enough to wrap the mortgage calculation objects up in a web service. A quick round of testing of the service proved that it worked just as well as the old client app, and everyone was happy - for awhile.

Sometimes, agents would run a calculation and get absolute absurd results. Developers, putting in exactly the same values into their test environment wouldn't see the bad output. Testing the errors in production didn't help either- it usually worked just fine. There was a Heisenbug, but how could a simple math calculation that had already been tested and used for years have a Heisenbug?

Well, the calculation ran by simulation- it simply iteratively applied payments and interest to generate the entire history of the loan. And as it turns out, because the client application which started this whole thing only ever needed one instance of the calculator, someone had made it a singleton. And in their web environment, this singleton wasn't scoped to a single request, it was a true global object, which meant when simultaneous requests were getting processed, they'd step on each other and throw off the iteration. And testing didn't find it right away, because none of their tests were simulating the effect of multiple simultaneous users.

The fix was simple- stop being a singleton, and ensure every request got its own instance. But it's also a good example of misapplication of patterns- there was no need in the client app to enforce uniqueness via the singleton pattern. A calculator that holds state probably shouldn't be a singleton in the first place.

[Advertisement] Utilize BuildMaster to release your software with confidence, at the pace your business demands. Download today!

Barry rolled into work at 8:30AM to see the project manager waiting at the door, wringing her hands and sweating. She paced a bit while Barry badged in, and then immediately explained the issue:

Today was a major release of their new features. This wasn't just a mere software change; the new release was tied to major changes to a new product line- actual widgets rolling off an assembly line right now. And those changes didn't work.

"I thought we tested this," Barry said.

"We did! And Stu called in sick today!"

Stu was the senior developer on the project, who had written most of the new code.

"I talked to him for a few minutes, and he's convinced it's a data issue. Something in the metadata or something?"

"I'll take a look," Barry said.

He skipped grabbing a coffee from the carafe and dove straight in.

Prior to the recent project, the code had looked something like this:

if (IsProduct1(_productId))
	_programId = 1;
elseif (IsProduct2(_productId))
	_programId = 2;
elseif (IsProduct3(_productId))
	_programId = 3;

Part of the project, however, was about changing the workflow for "Product 3". So Stu had written this code:

if (IsProduct1(_productId))
	_programId = 1;
else if (IsProduct2(_productId))
	_programId = 2;
else if (IsProduct3(_productId))
	_programId = 3;
	DoSomethingProductId3Specific1();
	DoSomethingProductId3Specific2();
	DoSomethingProductId3Specific3();

Since this is C# and not Python, it took Barry all of 5 seconds to spot this and figure out what the problem was and fix it:

if (IsProduct1(_productId))
{
	_programId = 1;
}
else if (IsProduct2(_productId))
{
	_programId = 2;
}
else if (IsProduct3(_productId))
{
	_programId = 3;
	DoSomethingProductId3Specific1();
	DoSomethingProductId3Specific2();
	DoSomethingProductId3Specific3();
}

This brings us to about 8:32. Now, given the problems, Barry wasn't about to just push this change- in addition to running pipeline tests (and writing tests that Stu clearly hadn't), he pinged the head of QA to get a tester on this fix ASAP. Everyone worked quickly, and that meant by 9:30 the fix was considered good and ready to be merged in and pushed to production. Sometime in there, while waiting for a pipeline to complete, Barry managed to grab a cup of coffee to wake himself up.

While Barry was busy with that, Stu had decided that he wasn't feeling that sick after all, and had rolled into the office around 9:00. Which meant that just as Barry was about to push the button to run the release pipeline, an "URGENT" email came in from Stu.

"Hey, everybody, I fixed that bug. Can we get this released ASAP?"

Barry went ahead and released the version that he'd already tested, but out of morbid curiosity, went and checked Stu's fix.

if (IsProduct1(_productId))
	_programId = 1;
else if (IsProduct2(_productId))
	_programId = 2;
else if (IsProduct3(_productId))
{
	_programId = 3;
}

if (IsProduct3(_productId))
{
	DoSomethingProductId3Specific1();
	DoSomethingProductId3Specific2();
	DoSomethingProductId3Specific3();
}

At least this version would have worked, though I'm not sure Stu fully understands what "{}"s mean in C#. Or in most programming languages, if we're being honest.

With Barry's work, the launch went off just a few minutes later than the scheduled time. Since the launch was successful, at the next company "all hands", the leadership team made sure to congratulate the people instrumental in making it happen: that is to say, the lead developer of the project, Stu.

Today is holiday in the US, where we celebrate a cosplay version of history with big meals and getting frustrated with our family. It's also a day where we are thankful- usually to not be at work, but also, thankful to not work with Brad. Original --Remy

Anita parked outside the converted garage, the printed graphic reading Global Entertainment Strategies (GES) above it. When the owner, an old man named Brad, had offered her a position after spotting her in a student computer lab, she thought he was crazy, but a background check confirmed everything he said. Now she wondered if her first intuition was correct.

“Anita, welcome!” Brad seemed to bounce like a toddler as he showed Anita inside. The walls of the converted garage were bare drywall; the wall-mounted AC unit rattled and spat in the corner. In three corners of the office sat discount computer desks. Walls partitioned off Brad’s office in the fourth corner.

He practically shoved Anita into an unoccupied desk. The computer seemed to be running an unlicensed version of Windows 8, with no Office applications of any kind. “Ross can fill you in!” He left the office, slamming the door shut behind him.

“Hi.” Ross rolled in his chair from his desk to Anita’s. “Brad’s a little enthusiastic sometimes.”

“I noticed. Uh, he never told me what game we’re working on, or what platform. Not even a title.”

Ross’s voice lowered to a whisper. “None of us know, either. We’ve been coding in Unity for now. He hired you as a programmer, right? Well, right now we just need someone to manage our documentation. I suggest you prepare yourself.”

Ross led Anita into Brad’s office. Above a cluttered desk hung a sagging whiteboard. Every square inch was covered by one, sometimes several, overlapping sticky notes. Each had a word or two written in Brad’s scrawl.

“We need more than just random post-its with ‘big guns!’ and ‘more action!’” Ross said. “We don’t even know what the title is! We’re going crazy without some kind of direction.”

Anita stared at the wall of sticky notes, feeling her sanity slipping from her mind like a wet noodle. “I’ll try.”

Sticky Escalation

Brad, can we switch to Word for our documentation? It’s getting harder
to read your handwriting, and there’s a lot of post-its that have
nothing to do with the game. This will make it easier to proceed with
development. -Anita

Two minutes after she sent the email, Brad barged out of his office. “Anita, why spend thousands of dollars on software licenses when this works just fine? If you can’t do your job with the tools you have, what kind of a programmer does that make you?”

“Brad, this isn’t going to work forever. Your whiteboard is almost out of room, and you won’t take down any of your non-game stickies!”

“I can’t take any of them down, Anita! Any of them!” He slammed the door to his office behind him.

The next day, Anita was greeted at the door by the enthusiastic Brad she had met before the interview. “I listened to reason, Anita. I hope this is enough for you to finish this documentation and get coding again!”

Brad led Anita into his office. On every wall surface, over the door, even covering part of the floor, were whiteboards. Sticky notes dotted nearly a third of the new whiteboard space.

“Now, Anita, if I don’t see new code from you soon, I may just have to let you go! Now get to work!”

Anita went to sit at her desk, then stopped. Instead, she grabbed a bright red sticky note, wrote the words “I QUIT” with a sharpy, barged into Brad’s office, and stuck it to his monitor. Brad was too stunned to talk as she left the converted garage.

The Avalanche

“Are you doing better?” Jason called Anita a few weeks later. Their short time together at GES has made them comrades-in-arms, and networking was crucial in the business.

“Much,” she said. “I got a real job with an indie developer in Santa Monica. We even have a wiki for our framework!”

“Well, listen to this. The day after you quit, the AC unit in the garage broke. I came into work to see Brad crying in a corner in his office. All of the sticky notes had curled in the humidity and fallen to the floor. The day after he got us all copies of Word.

“Too bad we still don’t know what the title of the game is.”

[Advertisement] ProGet’s got you covered with security and access controls on your NuGet feeds. Learn more.

One of the clients for Rudolf's company was getting furious with them. The dev team was in constant firefighting mode. No new features ever shipped, because the code-base was too fragile to add new features to without breaking something else. What few tests existed were broken. Anyone put on the project burned out and fled in months, sometimes weeks, and rarely after only a few days.

Rudolf wasn't too pleased when management parachuted him into the project to save it. But when he pulled the code and started poking around, it looked bad but not unsalvageable. The first thing he noticed is that, when following the instructions in the README, he couldn't build and run the application. Or maybe he wasn't following the instructions in the README, because the README was a confusing and incoherent mess, which included snippets from unresolved merges. Rudolf's first few days on the project were spent just getting it building and running locally, then updating the README. Once that was done, he started in on fixing the broken tests. There was a lot of work to be done, but it was all doable work. Rudolf could lay out a plan of how to get the project back on track and start delivering new features.

It's about then that Steve, the product owner, called Rudolf in to his office. "What the hell do you think you're doing?"

Rudolf blinked. "Um… what I was asked to do?"

"Three days and you just commit a README update? A couple of unit tests?"

"Well, it was out of date and meant I couldn't-"

"Our client is crazy about their business," Steve said. "Not about READMEs. Not about unit tests. None of that actually helps their business."

Rudolf bit back a "well, actually," while Steve ranted.

"Next thing you're going to tell me is that we should waste time on refactoring, like everybody else did. Time is money, time is new features, and new features are money!"

Suddenly, Rudolf realized that the reason the project had such a high burnout rate had nothing to do with the code itself. And while Rudolf could fix the code, he couldn't fix Steve. So, he did what everyone else had done: kept his head down and struggled through for a few months, and kept poking his manager to get him onto another project. In the meantime, he made this code slightly better for the next person, despite Steve's ranting. Rudolf eventually moved on, and Steve told everyone he was the worst developer that had ever touched the project.

The customer continued to be unhappy.

[Advertisement] Continuously monitor your servers for configuration changes, and report when there's configuration drift. Get started with Otter today!

Back in the heady days of the DotCom Bubble, startups were thick on the ground, and Venture Capital money was a flood- lifting startups atop a tsunami only to crash them back into the ground a short time later. Taliesyn once worked for one such startup.

Taliesyn's manager, Irving, was an expert in AI. In the age of the DotCom Bubble, this meant Irving knew LISP. Knowing LISP was valuable here, because their core product was a database system built on LISP- specifically the Common LISP Object System, an object-oriented bolt-on for LISP.

It was an object-oriented database system, akin to the modern NoSQL databases, but its architecture left a few things to be desired. First, since disk reads and writes were expensive operations, the system avoided them. All updates to data were done in memory, and someday, at some point, when the program felt like it, the changes would be written to disk. This meant that any failures or crashes could lose potentially days of data. Worse, the data was stored in one gigantic text file, which meant corruption could easily take out the entire database.

These were legitimate problems, and due to the architecture, they were going to be challenging to resolve. That was the startup life, however- they had a minimum viable product, and just needed to pour energy into making it something worth using.

Everyone looked to Irving, the AI and LISP expert, to guide them through this.

Irving saw where the real problems lay. "Your database doesn't support SQL," Irving said.

"Well, sure," Taliesyn said. "That's our selling point."

Irving nodded, and then, speaking slowly, as if to a particularly dense child, said, "A database needs to support SQL."

"I mean, a relational database, sure," Taliesyn said, "but we're using an object oriented data model which means we don't need to do joins or-" Taliesyn kept talking, explaining why their database didn't support SQL, why it shouldn't support SQL, and why SQL support was not only off the roadmap, but so far off the roadmap that it was labelled "Here there be dragons."

Irving nodded along, and ended the conversation with a, "Sure, that makes sense."

Everything was fine for a few weeks, until one of Taliesyn's peers on a different team, Angela, shot him an email: "Hey, marketing is getting antsy about a SQL demo, and I've got half a dozen features blocked until you get me a build with that functionality. What's the timeline like?"

Taliesyn was uncertain about what the timeline was like, since he had now clearly slipped into a parallel universe. He politely informed his peer that he had no idea what was going on, but would find out. It didn't take a great sleuth to figure out that Irving had started appending his own features to the roadmap.

Taliesyn tracked Irving down and demanded to know what was going on.

"A customer is already using it!" Irving protested. "They wrote it themselves! So we should be able to do it easily. Frankly, it's embarrassing to say that we can't do something with our own tools that the customer is already doing!"

Taliesyn knew that Irving was either wrong or lying, and asked to talk to the customer. Irving was, in fact, wrong. The customer had used LISP to write an extension to their object database (another one of those selling point features), and this extension used ODBC to open a connection to a relational database. This let them move data between the two different database systems.

"Irving," Taliesyn said, "they're not using SQL on our database, they're connecting to a relational database and using SQL there. SQL doesn't make sense for our data structure! We don't have tables, or keys, or relationships. It's objects! You're promising features that we can't deliver."

Irving was unmoved. "We are making a database. A database must have SQL capability. It's structured query language- it's right there in the name! Our structure should be queried by structured query language."

Taliesyn tried going over Irving's head, but upper management had no interest in actually solving personnel problems. Irving's buzzword laden ideas about why SQL was required seemed to jive with what their customers wanted, and the idea of shipping a non-SQL database was getting lost in the endless quest for the next round of VC funding.

The result was a bolted on monster of broken implementations of SQL that infuriated the few customers who tried it, Irving got promoted, and Taliesyn ran as far from the trainwreck as possible before the company flamed out.

[Advertisement] Continuously monitor your servers for configuration changes, and report when there's configuration drift. Get started with Otter today!

Alan worked on a website which had a weekly event which unlocked at 9PM, Saturday, Eastern Time. The details of the event didn't matter, but this little promotion had been running for about a year and a half, without any issues.

Until one day, someone emailed Alan: "Hey, I checked the site on Sunday, and the countdown timer displays 00:00:00."

Alan didn't check their email on Sunday, and when they checked the site, everything was fine, so he set himself a reminder to check things out next Sunday, and left things alone for a week.

Well, Alan forgot on Sunday. It was his day off after all, but he did remember to check first thing on Monday. When he came in, the timer read 00:00:00. Alan had a twelve o'clock flasher. Oddly, when he checked it after grabbing some coffee, the timer now showed the correct value.

It was time to dig through the code. Now, this story happened quite some time ago, so the countdown timer itself was a Flash widget. But the widget received a parameter from the HTML DOM, which itself was generated by PHP, and it didn't take long to find the SQL query it was using to find the next event: SELECT next_event FROM event_timer.

Yes, event_timer was a one column, one row table. A quick search through the codebase found the table referenced only one other place: backend_settimer.php.

All the pieces came together: resetting the timer was an entirely manual process. Every Monday, Tina came in, and assuming she remembered, she reset the timer. Some days she did it late, or forgot entirely. Some weeks, she was on vacation. Maybe she remembered to delegate, maybe she didn't.

For roughly 72 weeks, this had been how things had been working.

The good news was that the date was getting parsed with the PHP strtotime function, which meant Alan merely had to go to the backend_settimer.php and set the value to Next Saturday 9pm, and let Tina know she didn't need to do this anymore.

[Advertisement] Continuously monitor your servers for configuration changes, and report when there's configuration drift. Get started with Otter today!