From our anonymous submitter:

Having reached the end of the road at a company increasingly swallowed up companies further east which you'd never believe were still afloat, I found myself headhunted for certain specialty software skills. I was reaching the final few years of my expected working span, so I jumped at the chance. The money was (to me at that time) spectacularly good, so I jumped into it.

It started when my first day was spent by me being sent home for the weeks it was still going to take to onboard me. Not bad, engaged to wait, as it were, and the first 6 months was thus and so.

The man who had interviewed me, call him Fred, was intelligent and urbane, and was a joy to meet. He and I clearly hit it off, and lo and behold I was in. It was he who gave me my first assignment, which was mathematical analysis of their core milk-cow program because they needed to find out what it did, and how it did it, so they could perhaps implement it in a more contemporary language.

So I did that, and was just about to publish my findings with him, when Fred inconveniently dropped dead suddenly. In the what-are-we-going-to-do-now-our-key-man-is-no-more confusion, we contractors were forgotten.

For the next 18 months or so (may have been more, may have been less) I was more or less ignored. I spent the time writing a development environment to work on any part of the program conveniently, all the while sitting next to a man who was constantly, forcefully and repetitiously speaking ill of the managers in his line structure. The ridiculously garrulous boss who inherited me thought little of me, and handed me the little work that came my way with active hostility. One or two good guys, but mostly a cabal of elderly men trying to preserve their little money-spinner as long as they could, and a johnny-come-lately trying to increase (and even introduce) automatic processes was less than welcome. During that time I spent quite some time on TDWTF, submitting a gem or two here and there.

No surprise when they finally kicked my arse away. No love lost there. Now working my last couple of years to retirement as a postie.

No punchline here. I want you to know that dropping dead from work is all too real and can happen to anyone.

[Advertisement] ProGet’s got you covered with security and access controls on your NuGet feeds. Learn more.

Rachel joined a new team which was proudly "metrics driven". When she first met with her boss, Zane, he explained his thinking.

"We need to be data-driven to make good decisions, right? We're a manufacturing company. We make widgets. At the end of the day, we need to make the most widgets for the lowest cost of goods sold. So we track that, and that feeds into every decision."

The team oversaw an automated production line, which meant the software was a mix of robotics, embedded firmware, high-level web based monitoring tools, and thickets of dreaded PLC code. And because you can't build an entire factory for test purposes, they only way they could test real-world scales with real-world data was to roll changes out to production. They could simulate, they could run tests on subsets of the system, but a change in the production line software couldn't truly be validated until it rolled out into the real world.

Rachel's first task on the new team involved making some changes to their metrics dashboard. It was viewed as a good way to get her feet wet with the new team. As it turned out, the metrics dashboard was a Google Sheet, with a complex series of formulas that involved multi-level INDEX functions- essentially querying the spreadsheets like they were a database. Why not use an actual database? Oh, they did — six actually — but the company obeyed Remy's Law of Requirements Gathering: "no matter what the requirements the users ask for, what they really wanted was Excel". The database data was pulled into the spreadsheet for reporting.

Now, a complicated sheet pulling in data from not one, but six different databases, they must have a pretty complex model to explain how changes to their software would impact productivity. And since they needed to model the software to make predictions about how it'd behave in production, that model must be extremely useful.

Of course it wasn't. The only metrics they tracked were output metrics, variations on "widgets produced per unit time". There were some performance metrics, so you could maybe potentially identify "oh, our overall throughput dropped because unit 5 became a bottleneck and started taking 1.5 extra seconds per widget", but nothing that actually helped you understand how the complex system made decisions. Or even why unit 5 was taking longer.

For example, there was an automated quality control scanner. It examined widgets as they came off the line, and rejected defective ones based on a computer vision algorithm. Did that subsystem record why it rejected a widget? No, it did not. The CV model was able to tag widgets with a defect category based on what it saw, but that information didn't get recorded anywhere. In fact, it didn't even record how many widgets got rejected. The only way to know was to have an operator on the assembly line count widgets in the bin manually. Since that ate up a bunch of an operator's time, it never happened unless the developers begged for it. And since the operator still couldn't answer the question "why was this widget rejected", it wasn't all that useful anyway.

Every change to the software was scored against the overall output metrics. This meant that when Rachel was ready to push out her first software change, something that would record how many widgets were rejected and why, whether or not it could be deployed was dependent on seeing the change improve, or at least not regress, the widgets-over-time scores. But the widgets-over-time were a noisy metric; it varied based on which operators were working any given shift, or based on supply chain constraints. Or sometimes, based on when one of the machines was last calibrated- theoretically something that happened on a set schedule, but really was up to the operators. This meant the first three times Rachel rolled her code out for a test run, the metrics regressed. Nothing she changed should have impacted the metrics, but the metrics regressed due to environmental issues.

This meant making a simple change could take weeks, because you could only do final validation on the real system, which means you had to mark off a block of time for a test run, you could only run a handful of tests a day, and if metrics regressed you had to account for that before you could release the software for actual production use.

Over the first few months, Rachel added instrumentation to the code. Anything along the way to generating an output widget, she recorded. The hope was that once they had enough data, they could build a useful model of the system. Unfortunately, Zane had other ideas.

"So, you haven't improved our metrics," Zane said. "Which, I remind you, we're a metrics driven organization. Every change needs to improve our metrics."

"Sure, but I'm gathering more data so we have a better idea of what makes our metrics tick. We don't know why our system does some of the things it does, because we don't record any logging about the decisions it makes."

"Right, but we already gather the key metrics."

"But you don't gather the data that tells you why those metrics are what they are!"

"Sure," Zane said. "But those aren't our key metrics."

That, unfortunately for Rachel, was where things landed. Understanding their complex system was a low priority. Pushing top-level metrics without understanding what fed into them, that was the priority. That didn't mean Rachel was powerless: any time she made a change that she thought might help the top level metrics, she also made sure to add instrumentation that explained how that change behaved. It was the compromise that kept Zane happy: she released features that impacted the top-level metrics, but she also made the system more observable.

[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!

All I can say in response to our anonymous submitter's story is, ALMOST?!

With the World Cup being hosted in North America this year, I remembered this story that happened back in 2014. At the time I was working in Brazil, for a company that builds software systems for public services. And, with the World Cup being hosted there, in came the opportunity for local agencies to invest in modernization, with pretty much a blank check to get new services, so long as it was deployed before the end of the World Cup. And so the sales people did what they did best, and went around trying to upsell whoever would be willing to buy — no matter our actual capacity for developing the things.

So it was that I was pulled into this new fancy digital system for the police force of a state capital. However, we had only about 4 engineers available, and what they sold was a project estimated for a team of 20, to be delivered in 3 months, with no room for delay. And it wasn't just our core C&D product, but this massive thing with customized public-facing websites, live tracking of the position of different police cars delivered to a tablet in each car, automated reporting, etc.

First thing: We received a pile of 24 resumes, and were told to choose 16 of those. Maybe 3 were acceptable, but we had to waste 1 month hiring and onboarding 13 other people who were worse than useless. Classic man-month problem. We eventually had to tell management that nothing would be delivered this way, so they did the very best next thing: fly us to this other city, so we could work embedded there, in full crunch mode for the delivery. We pretty much worked 12+ hours a day, 7 days a week, for those next 2 weeks.

Another situation: they wanted this system where people could take a photo of an incident in progress, and submit via this app + website, to be verified by an operator in real-time. We nicknamed it the "dick-pic encyclopedia." Even worse, we only had the budget to run a single server, so this thing receiving public traffic would live in the same system that was tracking police car locations. Luckily they were convinced it was a bad idea so it was only ever online for a short period of time.

Next, was the police car tracking. This was done by a tablet installed in each car, which would be sending and receiving location information. But, 1 week before our deadline, we were hitting a serious bug: everything was working when we ran the tests ourselves, but the cops would report very weird bugs when testing it in the field. So we asked to do some field debugging, and I went on a ride-along. Things were working pretty much fine everywhere, so I asked to be taken to where he remembered seeing the tablets fail—to which the policeman just decides to drive off straight into one of the favelas around the city. I guess I can cross out "doing debugging in a police car passenger seat in a notoriously dangerous neighborhood" off my bucket list. Root cause: turns out cellphone connections would be pretty spotty in those areas, which we weren't handling properly.

Either way, we delivered something on time that was severely below spec, and very much over-budget. Company tried to squirm out of paying overtime (was told that we would gain "prestige" by doing those extra hours), but I put my foot down and left that job shortly after. Last I heard they actually got sued for this and a bunch of similar projects, and almost went under.

[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!

The opposite of meritocracy is kakistocracy: the worst and least-qualified are the ones who rise to the top.

Get real familiar with that word, dear readers. I think you'll need it.

If anyone can back me up on this, it's our submitter, Jared B:

I am a teacher by profession, and worked for a year at an ed-tech company founded by a mechanical engineering professor, Harry. Harry had spent a great deal of time in the 90's developing a C interpreter (yes, you read that right). 30 years later, he remained convinced that his interpreter was the technology of the future, and had founded a company that offered math and computer science curriculum to K-12 students based on C programming.

Originally, he had written a textbook that introduced students to programming using a locally-installed version of his interpreter and custom IDE. A little vain, but no serious problems. As Chromebooks grew popular in schools, he had developed a web IDE where students could write and run C code.

But Harry could never give fully give up on the Windows IDE for his C interpreter. So, he included in the web version a "Run Locally" button for those school computers still running Windows. It worked like so: installing the interpreter and IDE locally would also install a daemon that activated on startup and ran a websocket server. This server had an endpoint which accepted as a parameter a string of C code. It would then pass this C code to the locally-installed interpreter to run.

As you might suspect, there was no authentication whatsoever on this local websocket server. Knowing the form of the protocol, ANY domain could connect to localhost:12345/execute_c_program and send arbitrary code to run (of course, Harry prided himself on the completeness of his C implementation, including execv() and the like). Trick a user into visiting a malicious website, and you automatically had RCE on their computer.

Adding insult to injury, I discovered that the server was bound to 0.0.0.0 so that if you had Harry's software (/malware) installed, any computer on the same network as you could send you arbitrary C code to execute without question.

These vulnerabilities had existed for several years before I joined the company. In all that time, Harry had never hired anybody but his own grad students as software developers, and none of them had noticed the problem. By that time, the software was installed on thousands of school-owned computers throughout the state.

I documented and demonstrated the vulnerabilities to Harry. He did release a new version of the software addressing the issues and citing "security improvements" in the release notes, but there was never a communication to school/district IT leaders to describe the importance of updating. I suspect that Harry should be in serious legal trouble for potentially compromising data related to schools and minors, but I've since moved on and dropped the subject.

During the year I spent at the company (not in any sense as a dev, mind you, but as a lowly curriculum writer), I also discovered and reported a cookie-stealing exploit that would have compromised student and teacher data, as well as a code injection on another of Harry's websites (he decided to demonstrate that his C interpreter could work as a web server via a page where a user could type a math expression, which was then eval()ed server-side without any sanitation). The latter vulnerability gave me remote access, where I discovered thousands of transaction records that included credit card information stored in the clear.

Harry's company is still in business to this day, and has recently been ranked in TIME's list of top American ed-tech companies. Oh, and the office router's admin page still had the default Google-able username and password, but that one's a freebie.

I knew someone like this once, only they were stuck on ColdFusion long after everyone else stopped caring about it. However, I don't think they went on to endanger an entire state's educational system, only to be lauded as a visionary leader. Can't say for sure, though.

[Advertisement] Keep the plebs out of prod. Restrict NuGet feed privileges with ProGet. Learn more.

Today's submitter John F. was migrating data from a Microsoft platform to a Microsoft platform, using Microsoft tools. Absolutely nothing could go wrong, right?

Right?

A few years ago, I was working on a migration. We had sold part of our business, and so we had to extract a whole bunch of customer documents and metadata to provide to the buyer. The documents were stored in SharePoint on-premises, so the first step was extracting the metadata and storing it in a SQL Server database.

A colleague had used Microsoft's ETL tool SSIS to get the process started, and it generated a database schema. But after taking over, I wanted to change to PowerShell for greater control. For speed reasons, I decided to use System.Data.SqlClient.SqlBulkCopy, and getting that going required making sure my PowerShell script had all the correct data types.

One of our fields was the customer number. Customer numbers were up to 10 digits, but the first two were usually 0. Now, I prefer storing customer numbers as text, but someone in the distant past thought, This is a number, and SharePoint has a Number field, so I will use that.

Under the hood, Number fields in SharePoint are actually Doubles. Using a Double to store something exact like a customer number is not really ideal, but double-precision is absolutely enough to represent 10 digit numbers accurately. So what went wrong?

Well, remember we used SSIS to create the original table schema in SQL Server. I then used this table schema to write my script. But it turns out that in SQL Server world, the double-precision type is called float. If you want single-precision, you have to say float(24). I didn't know this, and so when I saw the SQL Server column as a float, I entered float as the corresponding .Net type in my script.

Oops.

So numbers came out of SharePoint as double. They were then converted to float before being inserted into SQL Server. Almost all records were fine, but large customer numbers had their last few digits changed. Testers didn't notice, but fortunately someone picked it up in the full load. We had to generate a list of changed numbers to patch the data after the fact.

[Advertisement] Keep all your packages and Docker containers in one place, scan for vulnerabilities, and control who can access different feeds. ProGet installs in minutes and has a powerful free version with a lot of great features that you can upgrade when ready.Learn more.

Developing software can't simply be done with a text editor and a compiler. There are a variety of other tools we have to bring to bear that support our efforts and keep the team organized, like say, source control.

There are certain tools we all have to use that I would argue, nobody has actually make a version that's any good. Build tooling is one of my go-to examples: there are no good build systems, only build systems that are good enough for this task.

Another is ticket/task management. In fact, I'd go so far as to say, there are no good ticket management tools. Amongst the not good tools, I'd put Jira as one of the not goodest of all.

What makes Jira attractive to companies is the same thing that makes it miserable, and the thing that infects any "enterprise" software platform and turns it into garbage: it has all the features and expect you to build your own workflows with it. You don't merely use Jira, you have to program your own interfaces in Jira to get your workflow into the system. And if you have the misfortune to have a project manager who thinks they're more technical than they are, they'll endlessly spin up new views, new workflows, and rearrange how the work is tracked in lieu of actually working.

I've been on that team.

One of Jira's features is the ability to describe the ticket workflow: the state machine that describes your process from the initial entry of the ticket all the way down to released software or project completion. This includes routing, so that as one team member does their part of the work, it automatically goes to someone else to do the next portion of the work.

Which brings us to Klinsten. They were working on a new team, and wanted to change the ticket status from its current status to whatever came next in the workflow. So they looked at the workflow.

These are two different versions of the same workflow, one with transition labels added, which as you can see, does nothing to clarify the workflow. That it's a mix of Dutch and English doesn't help matters.

The purpose of this workflow is to help the team understand how to sequence and organize their work. But this workflow has so many states and so many transitions, it fails at this goal. Looking at it makes me just want to gesloten my browser tab, because this user isn't accepting any of this.

[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!

"You submitted a pull request."

Indika was, in fact, reviewing the comments she'd gotten on that very same pull request, when her boss, Bill, walked up behind her. What she didn't understand is why Bill said it like it was an accusation.

"Yes?" she replied.

"Okay, well, we don't do that here. You're new, so I'll let it slide, but please review the developer guide."

Well, Indika had reviewed the developer guide, or at least thought she had. As it turned out, there was the official, company wide developer guide. That's the one she'd read. But Bill maintained his own, for his team. He hadn't ever told her about it, but apparently assumed she'd have the oracular blessings of Apollo and find it by herself.

It had this to say:

Branching is prohibited. Merging is a time wasting activity and goes against CI principles. Only use git to commit, push, and pull.

And rebase, presumably, if everyone was just committing on the main branch?

Indika asked one of her co-workers, Elise, over coffee: "Is this real?"

"Yeah," Elise said. "I'm not sure how he found out about your PR, I don't think anybody added him to the review. I mean, why would they?"

"Oh, I sent him the link," Indika said. "Just a whole, 'I'm new here, look at me doing the work!' type heads up."

"Oh yeah, definitely don't do that."

"So we do use PRs?"

Elise nodded. "Of course we do. We're not crazy. We just make sure Bill never finds out."

That seemed like a terrible way to work, but Indika went along with it, at least for a few weeks. Then an opportunity presented itself; she and Bill bumped into each other in the kitchenette grabbing coffee, and nobody else was around. At this point, Indika had already submitted a number of PRs without Bill knowing.

"Bill, I've been meaning to ask, what's your rationale for prohibiting branching?"

Bill loved being asked that question. "Well, well, it comes from twenty years of experience. What exactly does a branch get you?"

"A distinct history of changes that can be maintained and eventually merged in once a large unit of work has been done without disrupting other work that might be in flight?"

"Another point of conflict! A chance for the code you're working on to get stale. A chance to fall behind the rest of the team. Now, for a large open source team, with a lot of collaborators, a branch might make sense. I'm skeptical, but I can at least understand it. But for our internal team? It's just developers seeing a new toy and going, 'oh, shiny!'"

Indika sipped her coffee and went back to her desk. She was fortunate to have a window nearby, and looked at the squirrels playing in the branches of the tree.

I moved some things around on my calendar. 4:00 PM today is open. Please come to the executive floor.

-Leila

For a while I was stunned, staring at the email in front of me. I’d just told my boss I was quitting, refusing a promotion into my recently-deceased mentor Aggie’s shoes. Now, the new head of Human Resources wanted to see me.

Me? A Tech Support drone with one foot out the door? Well, she didn’t know that yet, did she?

Something in me feared where this might lead. But, Leila had stuck her neck out to rescue me from CEO Gibbs. She seemed like she cared about making things better. I decided to hear her out. Figured I owed her that much before I blew outta there for good in two weeks.

I had way too many hours to kill. Between whittling down my overstuffed inbox and resuming casework, it should’ve been easy to distract myself, but I couldn't focus on a single thing. I’d just done what had once seemed impossible. My brain wasn’t letting go of that any time soon.

Megan and Reynaldo also handed in their resignations. I got their messages confirming as much. We met up for lunch at a nearby restaurant and celebrated, but I was distracted. Amid the smiles and positive energy, the meeting with Leila was all I could think about. Should I mention it? I decided not to, not until I knew more.

I dreaded the afternoon slog now more than ever, but somehow, it slogged. When the clock’s hands finally crawled to 3:30, I threw on my coat and hat and darted out for one last smoke break. Then, it was time for C-Town.

Consumed with nervous energy, I shunned the elevator to race up the stairs floor by floor. Figured I’d burn off some stress, which could only help with whatever came next. Also figured I’d have a minute to recover in the vast executive lobby before finding my way to her office. Instead, I found Leila standing right there, every bit as polished as our surroundings. She faced me with surprise. “Hello.”

I tried to speak, laugh, something. Instead, I doubled over, coughing and gasping for air that felt all too thin up in nosebleed territory. While recovering, I couldn’t help but notice the gleaming tile beneath my feet, contrasting against my work shoes encrusted with sidewalk salt. Such details seldom crossed my mind, but the sort of people who worked up there lived and died for such details. Face flush, I cleared my throat one more time and righted myself, looking her way. “’Scuse me.”

“That was a long way up,” Leila remarked, gesturing behind herself. “I thought we could visit the observation deck. Would you like something to eat or drink first?”

Truth be told, I was already dying for another smoke. “No, that’s all right.”

“Follow me.”

She led the way through massive, quiet corridors to a small room with glass walls and ceiling. At this height, all one could see outside was a thick lead wall of fog. Leila strayed up to the far wall, a jewel against the void, and glanced back over her shoulder with chagrin. “I’m sorry, the view’s not very good today.”

“I dunno, I kinda like it.” Something about foggy weather had always intrigued me. With the normal world gone, it seemed like anything could happen.

She beckoned me closer with one hand. I strayed up next to her right side, staring out at the shrouded view.

“I understand you and Agatha Shaw were close,” Leila began quietly. “You have my deepest condolences.”

A two-ton anvil crashed onto my nerves. My fists clenched up at my sides. I worked so hard to hold back the flash-flood of grief that I couldn’t string words together. I only trusted myself to nod.

She glanced my way, hesitating. “Would it be better if we rescheduled?”

“I’m here,” I forced out. “Whatever you have to say, say it.”

She nodded. “First: while you were out of the office, I asked Francis Bronson to hand in his resignation.”

I drew a blank on the name, and blinked her way in confusion.

“The manager who nearly destroyed a printer with his hair dryer,” Leila explained, “after I’d just made a company-wide push for everyone to respect our office equipment.”

“Oh. Hothead!” My nickname for the guy. I’d worked that case a few weeks ago, but it felt more like years. Hothead worked in HR—at least, he had. After disarming him, I’d sent Leila an email, appealing for help against someone who clearly shouldn’t have been managing a supply cabinet, much less human beings. Well, she’d delivered. A warm note of satisfaction offered me a welcome lift out of grief. “Thanks. Really.”

Leila smiled. “We make a good team, I think. Which brings me to the other thing I wanted to discuss. Something new.”

My gaze fastened onto hers with a mix of intrigue and dread.

“You and I both know how badly this place needs to change. Let’s work together and actually fix things. I want to create a Change Management team and make you Team Lead.”

I was speechless, caught completely off-guard.

“The first thing we’d do is attack our company-wide leadership problem. Audits, surveys, hearings … eventually, a reorg. Along with simplifying the corporate structure, we’ll get rid of all the—Hothead, you called him? All the other Hotheads.”

“The biggest hothead is sitting at the top of the whole rotten pyramid,” I blurted. “He ain’t budging. He ain’t signing off on this, either!”

Leila was unfazed. “I think we could frame everything in a way that makes Mr. Gibbs like it. After all, we’re reducing payroll. We’re better positioning ourselves in a tough economic time. Worst-case, we could always use the three magic words: Google Did It.” She smirked.

I couldn’t help smirking back. Then I remembered what I’d done just a few hours earlier. “But I’m outta here. I quit! Put in my notice this morning!”

Leila nodded calmly. “Do you have a new position lined up somewhere else?”

“No. I'm going freelance with friends.”

“Friends who are leaving the company along with you?”

I nodded.

She paused for thought. “If you were to lead my Change Management team instead, you’d be able to recruit internally for your team. Whoever you think would be the most helpful. You’d set the agenda for whatever’s most important to address so that other good people don’t feel like they have to get away from here. All of this would mean a promotion to Director, with a salary and benefits to go with. And if you need more bereavement time, I could arrange an indefinite leave of absence until you feel ready to come back.”

The breath died in my throat. Had I suffered a stroke? I must’ve had a stroke. No, she really said it. She was on the level. I could change things. I could hire my friends to help me do it. A raise, full bennies, working with her every day?

Only a fool would refuse. And yet, my gut ached at the idea.

I stood there, frozen and mute, until I remembered something in my trench coat pocket: RD, the rubber duck I’d miraculously rescued from Aggie’s former office. I reached into my pocket and seized him in my fist.

RD? Aggie? I thought. Whoever’s listening. It all sounds amazing. I know she means it. But … it’s another trap, isn’t it? Staying in this joint for any reason means betraying myself. Betraying everything.

“Listen,” I finally said, “I’m flattered you even offered, but it ain’t right for me. Don’t give up on your idea! I’ve got friends here with ideas of their own for changing things. 32-hour work weeks. The end of free overtime. A union. I’ll send ’em your way. Offer them a spot on your team.”

Leila sighed. “A union would be an especially tough sell to Mr. Gibbs, but that really is a case where Google Did It. We might scare him so much with that idea that everything else would seem harmless in comparison.” She faced me with a sad smile. “A shame that we’re losing you. I was starting to learn some interesting things about printers.”

I’d miss her, too. And yet, I was feeling surprisingly great about my refusal.

“Make sure you file for unemployment,” she said. “We won’t stand in your way.”

I offered my hand. “Thanks for everything, Leila. Whenever it is they kick you outta here for good, come find me.”

She shook, sad smile persisting. “Maybe I will.”

I told Megan and Reynaldo about the new Change Management team. Made sure they knew the offer was on the table in case they preferred that over jumping ship. Both were quick to say no. Like me, they were too excited about our plans to stop now.

For the next couple of weeks, there was still plenty of work to be done: transferring my open tickets to other support reps and all that. But there was barely any time for it. Coworker after coworker stopped by my cube to express their surprise and wish me well in whatever came next.

“You’ll never be problem-free,” one of them advised me. “Go looking for the problems you want to have.”

I felt happier, freer, more determined than I had in ages. It was the conviction of knowing I’d stuck to my guns to do the right thing for myself.

Sanjay also jumped ship to join us. And there was one last surprise that came in the form of a phone call. The name on my work phone’s caller ID was DRACORA, P. So-called “Dracula!” Having a fairer opinion of her than most, I picked up without any sense of dread.

So-called “Dracula!” She hadn’t been so bad at all. Surprised, I picked up in a hurry.

“I’m so sad to hear you’re leaving!” she said. “What kind of freelance work are you doing?”

“All sorts of IT projects,” I replied. “Maybe some consulting on the side.”

“I have a friend who needs help setting up a website for her business. Is that something you could help her with?”

My eyes flew wide open in shock. “Sure!”

“I’ll put you in touch with one another.”

“That’d be swell. Thanks!”

We exchanged contact info. She promised to keep pointing friends our way whenever she could.

On my last day, I sent my personal contact information to my coworkers. I reminded them of Leila’s offer and urged them to keep me posted on their different causes. Then my friends and I walked out of a building that no longer had a hold over any of us.

It felt pretty damn swell.

Our accountant helped us incorporate RD IT Solutions. Only close friends knew it stood for “rubber duck.” The company covered medical and other relevant expenses for everyone. There was no hierarchy. Everyone had equal financial stakes and an equal say in company decisions.

After decades of being an expert at what I did, I was back at square one, learning the ropes. So much of what we had to learn for our business could only be learned through failure. Still, it felt rewarding to challenge my brain in new ways. The new gig let me wear lots of hats, from tech support to coding to business admin.

With no more regular paychecks, we had to tighten down our finances to what was critically important. We worked remotely at whatever times worked best for us, with the occasional meet-up at a public place or someone’s home. We all knew that any one of us having some kind of trouble could count on the rest of the group to help out as best as they could.

Megan quit smoking again. I cut way back myself. Wasn’t trying to, I just haven’t felt the need as much. Also stopped having those nightmares. It no longer feels like I’m living just for time off and weekends. I don’t spend my Sunday nights dreading Monday.

Dracula really did introduce us to our first client. It’s crazy what the universe puts out there when you go looking for it.

We met up remotely for our first client meeting to discuss requirements and expectations. When the topic of deliverables came up, our client scrunched her nose and interrupted Megan mid-sentence. “I don’t trust email! I’d rather you fax me the files.”

We were building a website for her.

“You mean, the source files?” Megan soldiered on bravely.

“Just fax me the codes,” the client said. “My nephew can re-type them into the Internet.”

She provided a fax number, fully expecting us to print out several hundred lines of code to send over. After deploying our first stab at a website that met her requirements, we did just that, mostly out of curiosity.

A few days later, she called us in a huff, saying her website looked like random letters. Her nephew had typed the code into Facebook.

FIN

[Advertisement] ProGet’s got you covered with security and access controls on your NuGet feeds. Learn more.

Greta (previously) sends us more updates from her "Ancient Development Environment".

An important task an IDE must do is report build errors to its users. Arguably, that's one of the most important parts. I wouldn't know, I insist on building from the CLI all the time, because IDEs confuse and frighten me. I recognize I'm the weird one here, who is more comfortable in GDB than in a GUI debugger, but this isn't about me, it's about the IDE Greta is using.

It needs to display an error. Why does it need to display an error? Well, Greta hasn't figured that out yet. The error I'm about to show you doesn't really explain what happened or why or give any hint as to what needs to be done to fix it. To make matters more confusing, it doesn't happen consistently, so simply re-running the build could potentially fix it.

None of that is why we're here, though. What makes this a WTF is how the error is displayed:

Greta shares her bullet points about what she hates about this:

• It's a popup that happens arbitrarily during build, under unknown conditions
• It says nothing about what went wrong, or indeed if anything went wrong
• It's in a non-user-legible XHTML format. It shows the markup of this XHTML rather than it being rendered.
• It arguably shows the XHTML markup in the worst way possible: in a tree view (?!) with one row per source line
• The markup isn't even well-formed. I'll let you count the reasons why.
• The control used is from some sort of legacy windowing toolkit that doesn't support text anti-aliasing.
• The first button on the upper-left, "Get latest C++ Builder Direct headlines from the Internet", does nothing.

I'd honestly forgotten about the era when the Internet was still kinda novel so every application had a "push a button and go to our web page, and this somehow definitely won't just break when we change our URL structure in the future."

Greta adds:

For all of the hatred I harbour for this, the second button ("Information about C++ Builder Direct") brings up the following powerful dose of 90s nostalgia, so I can't stay mad:

Flat-file style databases were designed to fit the constraints of the systems they were running on. You specify your schema in terms of "how many characters in a file we use to store this data", meaning something like this: JOHN    SMITH    12343rd    StAnytown PA12345 is read in my knowing that the first name field is 8 characters wide, the last name field is 8 characters wide, the street number is 4 digits, and so on.

It's also a terrible schema, and woe to anyone with a long name. But many a mainframe had a similar schema.

Now, let's think about maintenance here. What happens when we also want to store a middle initial? We've created for ourselves a problem. Somehow, I have to insert a character into every row, which basically means making a new table with a new schema, copying every record out of it and updating it to use the new schema. I can't just ALTER TABLE like an RDBMS. And worse, every piece of software that touches the table also needs to be updated. On a large legacy system, a simple task like "add a field to our database" could take weeks of developer time, and depending on the software, be a high risk operation.

Which is why the smart developer, when working with flat files, includes padding. Maybe my schema for an address record looks more like this: JOHN    SMITH    12343rd     StAnytown PA12345                . That's 16 characters of padding at the end of the file. Now somebody says that I need to store a middle initial, I can just shrink the padding by one and add a middle initial field, like so: JOHN    SMITH    12343rd     StAnytown PA12345Q               

Is this elegant? No. But it works. I haven't changed the length of the row at all, so I don't need to move data around. Software modules only need to be updated if they care about what's in the middle initial field; if they're out of date, they just think there's a "Q" in the padding, and don't care.

In real-world applications, instead of putting all the padding at the end, you'd usually put the padding in a few spots in the middle of the table. Any time you need a new column, you just steal a few characters from padding. Sure, someday you'll run out of padding, or at least out of padding blocks big enough for your new field, and then you'll have to do the hard work of shuffling data around. But in practice, you can get very far without that happening.

Which brings us to Brenda's adventure. Her team supports an IBM mainframe storing data in VSAM flat files. In other words, they've been doing the sort of thing I just talked about for many, many years.

Of course, in the modern era, you can't just leave your data sitting in an mainframe. Even if the mainframe is the source of truth, you want to be able to report on it and connect it with your other data systems. You need to, somehow, get the data into a modern RDBMS.

So the company hired a bunch of developers to write an extract-transform-load process, which pulls the data out of the mainframe. The mainframe team handed them a "copybook" for the flat file, which described the structure, and the ETL devs went to work.

And maybe those ETL devs didn't understand the importance of padding. Maybe they just missed the padding. Whatever it was, there were several places where the data was structured like SOME_USEFUL_FIELD PADDING PADDING PADDING SOME_OTHER_FIELD, and they opted to split it like so: SOME_USEFUL_FIELD PADDING PAD, DING PADDING SOME_OTHER_FIELD.

When they released this process, it was fine. The padding characters got stripped before displaying, so the users never saw them. They were stored in the database, though, so when someone tried to reconstruct the data in a way that was compatible with the flat files, you could just concatenate the columns together and get a valid result.

It was fine- until it wasn't. The ETL devs, bless their hearts, only tested against the production mainframe. And why not, they were doing read only operations, what's the harm? Had they tested against the development mainframe, they would have seen new features in flight, features which consumed some of that padding, and realized that they should have paid closer attention to the copybook.

But instead, the test cases all passed. The software was, as far as the project managers and ETL developers could tell, working perfectly. So it was accepted, released to production, and running for a few weeks before the mainframe released its features. Those features then ruined all the beautiful reports with extraneous data.

And since the ETL devs were on contract, any request to have them rework it under the original contract was met with a stern "Works as designed". Instead of paying the contractors to come back and rework the system, the mainframe devs instead were tasked with finding different padding fields they could use, padding fields which wouldn't end up ruining any reports management liked to see.

[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!

Honestly, with the wildfire smoke and the oppressive heat, maybe it's time to find a nice quiet cave to hang out in. Something with no natural light and no natural ventilation. I wonder if anybody has a place like that… Original. --Remy

In the long ago, wild-west days of the late 90s, there was an expectation that managers would put up with a certain degree of eccentricity from their software developers. The IT and software boom was still new, people didn't quite know what worked and what didn't, the "nerds had conquered the Earth" and managers just had to roll with this reality. So when Barry D gave the okay to hire Sten, who came with glowing recommendations from his previous employers, Barry and his team were ready to deal with eccentricities.

Of course, on the first day, building services came to Barry with some concerns about Sten's requests for his workspace. No natural light. No ventilation ducts that couldn't be closed. And then the co-workers who had interacted with Sten expressed their concerns.

During the hiring process, Sten had come off as a bit odd, but this seemed unusual. So Barry descended the stairs into the basement, to find Sten's office, hidden between a janitorial closet and the breaker box for the building. Barry knocked on the door.

"Sten awaits you. Enter."

Barry entered, and found Sten precariously perched on an office chair, removing several of the fluorescent bulbs from the ceiling fixture. The already dark space was downright cave-like with Sten's twilight lighting arrangement. "He welcomes you," Sten said.

"Uh, yeah, hi. I'm Barry, I'm working on the Netware 3.x portion of the product, and Carl just wanted me to check in. Everything okay?

"This is acceptable to Sten," Sten said, gesturing at the dim office as he descended from the chair. Sten's watched beeped on the hour, and Sten carefully placed the fluorescent bulb off to the side, in a stack of similarly removed bulbs, and then went to his desk. In rapid succession, he popped open a few pill containers- 5000mg of vitamin C, a handful of herbal and homeopathic pills- and gulped them down. He then washed the pills down with a tea that smelled like a mixture of kombucha and a dead raccoon buried in a dumpster.

"He is pleased to meet you," Sten said, with a friendly nod. Barry blinked, trying to track the conversation. "And he is pleased with it, and has made great progress on building it. You will like his things, yes?"

"Uh… yes?"

"He is pleased, and I hope you can go to him and tell him that he is pleased with this, and set his mind at ease about Sten."

So it went with Sten. He strictly referred to himself in the third person. He frequently spoke in sentences with nothing but pronouns, and frequently reused the same pronoun to refer to different people. The vagueness was confounding, but Sten's skill was in Netware 2.x- a rare and difficult set of skills to find. So long as the code was clear, everything would be fine.

Everything was not fine. While Sten's code didn't have the empty vagueness of unclear pronouns, it also didn't have the clarity of meaningful variable names. Every variable and every method name was given a female first name. "Each of these is named for one of Sten's girlfriends." Given the number of names required, it was improbable that these were real girlfriends, but Sten gave no hint about this being fiction.

There was some consistency about the names. Instead of i, j, and k loop variables, you had Ingrid, Jane, and Katy. Zaria seemed to be only used as a parameter to methods. Karla seemed to be a temporary variable to hold intermediate results. None of these conventions were documented, obviously, and getting Sten to explain them was an exercise in confusion.

It led to some entertaining code reviews. "Michelle here talks to Nancy about Francine, and then Ingrid goes through Francine's purse to find Stacy." This described a method (Michelle) which called another method (Nancy), passing an array (Francine). Nancy iterates across the array (using Ingrid), to find a specific entry in the array (Stacy).

Sten lasted a few weeks at the job. It wasn't a very successful period of time for anyone. Peculiarities aside, the final straw wasn't the odd personal habits or the strange coding conventions- Sten just couldn't produce working code quickly enough to keep up with the rest of the team. Sten had to be let go.

A few weeks later, Barry got a call from a hiring manager at Initrode. Sten had applied, and they were checking the reference. "Yes, Sten worked here," Barry confirmed. After a moment's thought, he added, "I suggest that you bring him in for a second interview, and have him walk you through some code that he's written."

A few weeks after that, Barry got a gift basket from the manager at Initrode.

Thanks for the tip

Sten did not get hired at Initrode.

It's been pretty hot lately. Probably should use a fan to cool off. Mind the trip hazards. Original. --Remy

"It's that time of year again," Robert Rossegger wrote, "you know, when the underpowered air conditioner just can't cope with the non-winter weather? Fortunately, we have a solution for that... and all we need to do is just keep an extra eye on people walking near the (completely ajar) server room door."

"For as long as anyone can remember," Mike E wrote, "the fax machine in one particular office was a bit spotty whenever it was wet out. After having the telco test the lines from the DMARC to the office, I replaced the hardware, looked for water leaks all along the run, and found precisely nothing. The telco disavowed all responsibility, so the best solution I could offer was to tell the users affected by this to look out the window and, if raining, go to another fax machine."

"One day, we had the telco out adding a T1 and they had the cap off of the vault where our cables come in to the building. Being curious by nature, I wandered over when nobody was around and wound up taking this picture. After emailing same to the district manager of the telco, suddenly we had the truck out for an extra day (accompanied by one very sullen technician) and the fax machine worked perfectly from then on."

"I found this when I came back in to work after some time off," writes Sam Nicholson, "that drive is actually earmarked for 'off-site backup'. Also, this is what passes for a server rack at this particular software company. Yes, it's made of wood."

"Some people use 'proper electrical wiring'," writes Mike, "others use 'extension cords'. We, on the other hand, apparently do this."

"I was staying at a hotel in Manhattan and somehow took a wrong turn and wound up in the stairwell," wrote Dan, "not only is all their equipment in a public place (without even a door), it's mostly hanging from cables in several places."

"I spotted this in China," writes Matt, "This poor switch was bolted to a column in the middle of some metal shop about 4m above ground. There were many more curious things, but I decided to keep a low profile and stop taking pictures."

[Advertisement] Keep the plebs out of prod. Restrict NuGet feed privileges with ProGet. Learn more.

There were thousand tickets in the backlog, and I was on the trail of a weird printer issue. I had a suspect, but that wasn't enough to close the ticket. I'm Anonymous. This is my story.

Whenever you hit the Print button, a request launches into the ether. “Print one copy of this file, double-sided.”

But you can’t just chuck it out there aimlessly. You gotta tell it where to go. So that’s why we have Internet Protocol, or IP. Every computer, every printer, every device on every network has one or more IP addresses for different operations. They’re just like the address you scribble on the envelope holding Grandma’s Christmas card. Send your print request to the right IP, and you’re in business. Send it to the wrong IP, and they’ll be shaking their heads on the other end. “Print? But all I know how to do is tell you the weather!”

As for what happens next, you’re at the mercy of the program’s error handling. If you’re lucky, they’ll send a detailed error message. If you’re not, they could ignore you, act up in weird ways, or completely self-destruct in a violent crash.

I was neck-deep in a Tech Support ticket concerning a printer in Human Resources. Not only was Grandma’s Christmas card landing in Pluto, the Plutonians were writing back, in the form of mysterious complaints that were nothing like the documents the printer had been tasked with.

My troubleshooting hadn’t shot much trouble at all, so I’d bugged some friends for ideas. Reynaldo had been reminded of an old, retired HR program for logging anonymous employee complaints. While our developer friend Megan quit our icy outdoor break spot to dig up dirt from the software side of things, Reynaldo and I went to his cubicle to trace IPs. He suspected a conflict … and that’s just what his command-line requests revealed. We leaned over his laptop together, studying a couple of terminal windows and their cryptic output to his even-more-cryptic inputs.

“Whenever that printer got set up in HR, it received an IP that was already in use by this server—” he jabbed an index finger at the relevant part of the screen “—which is apparently still up and running, even though I swear it was decommissioned.” Reynaldo frowned. “Who wants to bet that stupid complaint program has some kind of ability to print cached complaints?”

I frowned, too. “We won’t know for sure unless Megan digs something up.”

Reynaldo flashed me a look of pure cynicism. It wasn’t an indictment of Megan’s skill, more skepticism toward the idea of anyone having properly documented this mess. We had better odds of hitting a billion-dollar jackpot.

I sighed. “It’s an easy workaround, at least: assign the printer a new IP. But we’ve got bigger problems here. Why’s this zombie server still up and kicking? Why’d an IP conflict ever occur in the first place?”

“Don’t get me started on our FUBAR IP allocation and tracking.” Reynaldo’s lowered voice contained plenty of venom. “We’re talking multiple pools to track, multiple spreadsheets that have to be manually edited. Problems that are easy enough to fix! Why spend money on honest-to-goodness network management software?”

Short-term budgeting preventing long-term improvements: a damned shame we’d both seen all too many times. Being a tech support drone, having an innate desire to help that no amount of baloney could stamp out, I once more found myself longing to fix the unfixable.

“In my ticket notes, I’ll make noise about a more permanent solution,” I said. “At the very least, some bigwig oughta leap at the chance to save 3 cents of electricity a year by putting the complaint-spewing server out of its misery.”

I’d already resolved to contact Leila, the new head of HR, about this ticket. She needed to hear about Hothead, the manager-type who, in a fit of frustration, had taken a hair dryer to the same printer and nearly trashed it. I could also tell her about these lingering network vulnerabilities that would continue causing problems for everyone down the line.

I decided to do it without telling Reynaldo. I had the feeling that if I let him in on it, his eyes would roll clear out of his skull.

Working from Reynaldo’s cube, I assigned the HR printer a new IP address and once again cleared its queue. Then I sent Tony, the ticket holder, a private message asking him to give printing another go. I felt pretty good about his chances, but I’d wait for him to give me the all-clear before closing the ticket.

With the messaging app still open on my phone, I saw that Megan had invited me to drop by her cube. From Networking Central, I hoofed it up to Developers’ Row.

Megan’s cube was a familiar spot filled with bright, cartoony posters and figurines, the only splashes of color for miles. The titles and characters drew blanks in my over-the-hill brain. I kept meaning to ask her what they were, why she liked them. Another time, maybe. I found her with her back toward me, leaning intently in her swivel-chair toward the single monitor over her laptop docking station.

“Now a good time?” I asked under my breath.

She whirled around, tense, then relaxed with a smile, her gaze livelier than I’d seen it in a while. She picked up a small external memory stick from her desk to proffer my way. “That’s the HR application’s source code!” she murmured in conspiratorial fashion. “Guess where it came from?”

I smirked. “I know the answer should be ‘a code repository,’ but in this joint, that’s asking too much.”

“You’re right!” she replied. “It was never in a repo, never in source control. It lived and died on one dev’s local machine, a dev who retired way before I got here. His successor hung onto all the old code from that guy’s machine, just in case. But yeah, we don’t officially support the application anymore.”

Megan turned around, using her keyboard to tab over to an open window in a code editor. From over her shoulder, I saw line after line of an archaic programming language that the Egyptians might’ve used to build the pyramids.

“This code’s an undocumented disaster,” she said.

“There was an IP conflict at that,” I told her. “The print requests were going to a server that’s still running this thing. We assigned the printer a new IP, but the mystery server remains a head-scratcher.”

“I’ll trace through and figure out what it does when it gets a print request. Improve its error-handling in general,” Megan promised, more excited than daunted by the prospect. “I mean, if it is still up and running, I could tweak, recompile, and redeploy it so it doesn’t—”

Insistent metallic knocks sounded behind us. “Excuse me!”

It was the sort of pointed voice that somehow ignored your ears and stabbed into your gut instead. Megan froze, tense. I turned to find a woman at the threshold with the bearing of a vindictive hall monitor, rapping a fist against the cube’s bare metal frame.

She leveled a withering frown at Megan. “I was hoping for a status update on the Hewville refresh! Did I hear you say you were planning to work on something else?” The question sounded more like a threat.

This had to be Megan’s boss. Megan remained tense from head to foot. “I—”

“I need you to focus on your assigned projects. The things you can actually bill your time against.” After delivering the condescending reminder, the boss’ glare shifted my way. “And you are?”

I shoved unease aside and put on the game face I’d spent decades perfecting. “Tech Support. I meant no harm, ma’am. I was just asking Megan for help with an open support ticket.”

“She doesn’t have time for that!” the boss scolded. “If you truly need help from this department, then you must escalate your ticket through the proper channels.”

I already knew how that went: pulling together screenshots, logs, and other detailed information, only to receive half of a sentence fragment a few days later, asking for something I’d sent with the first message. In this case, I knew someone would just wag a finger at me about the HR application being out of support. Thanks but no thanks.

Megan struggled to muster one last defense. “This program’s still running on a server somewhere!”

“For changes to existing code, the proper procedure is to file a formal change request through the Project Management Team. The PMT will create a billing code and assign appropriate resources, if they deem it a proper use of development time.” The boss then looked at me like I was a used tissue she was ready to throw in the trash. “If that’s all, I suggest you head back to Tech Support, Mr. … ?”

No way was I handing her my name on a platter. I tried to glance Megan’s way, but she was staring at her lap. I felt bad leaving her in that lurch, but staying would only make things worse for the both of us.

“Later,” I said, both a goodbye and a promise.

I got the hell out of Devsville, slipping down flight after flight of stairs with a lead weight in my chest. For a moment, Megan had brightened in the face of a collaborative challenge. I’d felt a little more alive, too.

Thank God someone had been there to make sure no one helped each other.

The same sicknesses plagued the joint year after year. Almighty budgets. Status quo worship. Hierarchy and miles of red tape. Promising young people like Megan had their spirits crushed, and schmucks like me just put their heads down. Still a huge pile of other support tickets waiting for me, after all.

The frustration and resentment, the desire to do something to fix this, burned in my chest like a bonfire.

Back at my desk, I found a message from Tony confirming the new printer was behaving at last. Bolstered by my friends’ findings, I closed his ticket with notes about how the resolution was only a band-aid on a gaping wound. I messaged Megan, too, thanking her for trying.

And with that emotional bonfire still raging, I settled in and typed out a long email to Leila, detailing in full the most recent shenanigans I’d been a part of.

By the time I finished, I had one bit of good news: Aggie, my old mentor who’d turned manager a few years back, had accepted my meeting request to meet the next afternoon.

She wasn’t my boss, which meant it was still safe to vent her way. I had every intention. My resentment would no longer let itself be buried under this or that technical hiccup. It was insisting upon action.

The next day, I walked to the downtown coffee shop well ahead of the appointed time, glad to have the excuse to be somewhere else. Bought myself a drink and sat down where I could watch the door.

Five minutes late turned into ten … then twenty. No Aggie.

It was totally unlike her. Sure, she’d canceled last-minute before, but she’d always got in touch with me to let me know.

Caffeine jitters fueled a fear I couldn’t shake off. I sent PMs and left voicemails on her cell and work phones, asking her to respond when she could. Back at work, I asked around the department, including my boss.

She’d been AWOL the whole day. No one knew what was up.

Hours of work still stretched in front of me. I could barely sit in my chair, much less look at my monitor.

Just call me, Aggie, I willed, staring at the cell phone clutched in my hand.

She didn’t. Not that afternoon, not that evening. Beneath my fear was this strange gut feeling, this knowing sense that my worries were justified. It was crazy, but there was no talking myself out of it.

Early the next morning, they roped the whole department into the big conference room. Nobody knew what was going on until a small, ashen group of managers, directors, and directors of directors filed to the front and called for attention.

I already knew, deep down, what was coming.

“We’ve we received some very unfortunate news,” one of the senior directors spoke. “Agatha Shaw … passed away in her home after a heart attack.”

A few gasps escaped the assembly. Otherwise, you could’ve heard a pin drop. Wide-eyed looks of shock surged through us like lightning.

Aggie.

My gut had known all along, but my brain still wasn’t having it. Somebody somewhere must have goofed up royal, I thought. Happens all the time in this joint. Aggie had more life and fight in her than I ever did. I—

“Those of you who reported directly to Ms. Shaw will report to Bill Watson for now,” the director continued. “Dismissed.”

The brass began showing themselves out.

The rest of us just sat there, too stunned to move. That’s it? I marveled. No memorial? No counseling? No time off? Not one drip of sympathy?

I could only look on helplessly as Bill Watson, my boss, walked right over to me. He put his hand on my shoulder, leaned over, and muttered into my ear: “My office.”

While my brain reeled, my feet stood me up obediently. They marched me right off to the next chair I dropped into, the one opposite the large desk in Bill’s office.

He settled in on his side. “It’s a real shame about Aggie. We’re screwed without her.”

I just sat there, still reeling.

”My manager is looking for someone to step up in a big way.” Bill nodded in my direction. “You’re ready. You deserve it. Her direct reports are reporting to me for now; over the next few months, I’m transitioning them to you. The promotion will follow, as soon as the next performance review comes around!”

We were just cogs in that joint. Never before had the point been driven home so viscerally. Righteous rage surged up from within, clearing my brain and shooting strength through my limbs. I jumped out of that chair and glared down at him. “Hell no! Find someone else!”

I got the hell out of there and dragged myself all the way home to collapse on my couch.

To be continued ...

[Advertisement] ProGet’s got you covered with security and access controls on your NuGet feeds. Learn more.

While a project manager is frequently called upon for their planning ability, the real skill we want from project managers is their ability to communicate. The job of a project manager is to align the team doing the work, with the organization goals driving the work, with the management and leadership teams trying to understand the work, while juggling all the constraints like budgets, timelines, and the endlessly changing expectations for the project. A good project manager is worth their weight in gold. A bad one will cost their weight in gold.

Mark was hired on as a contractor, reporting to Tegan. Tegan was fresh out of business school, complete with an MBA and a variety of project-management training certifications. Unfortunately for Mark and the rest of the team, and especially unfortunately for Tegan, she had absolutely no real world experience. To make matters worse, this wasn't just a software project: they were working on a system which matched newly developed software with newly designed mechanics and custom build control electronics. A group of experienced software engineers, mechanical engineers, and electrical engineers all found themselves reporting to a bright and shiny MBA. It's a role that she probably could have grown into, but management saw all the acronyms she continuously put after her name, and decided she could just take the whole thing over with no real guidance.

It went badly pretty much from the beginning. Tegan was not a talented communicator. For example, Mark's team needed to know: on what timeline were the electrical engineers going to deliver the first prototypes, so the software team could start running bench tests of their software? Tegan's response was a fortune cookie message about balancing the complicated pipelines and lanes on the Gantt chart and hitting all of their milestones; like a fortune cookie, it was vague, important sounding, but ultimately empty.

Of course, the natural reaction amongst the engineers was to just route around the damage: the various teams could talk to each other just fine without going through Tegan. That, unfortunately, did not go over well with management. Tegan, as the project manager, was their insight into the project. They needed her in the loop on everything. And she couldn't just be informed, she had an MBA. She needed to be making decisions. But she was unqualified to make those decisions, which meant the project gradually ground to a halt. Tegan's emails got more vague, her meetings got longer but accomplished less, and after a certain point, she just stopped replying to key email threads.

The first few days of radio silence seemed like a gift. But as time passed and Tegan seemed uninterested or unable to reply to any of the questions the team had for her, the project started to flounder. The engineering teams escalated this problem to management. Management presumably went back to Tegan. At some point, feeling the weight of everything going wrong around her, Tegan sent out this email, which is definitely the best and clearest communication she managed during the project. It's arguably the clearest, and most accurate communication one could make in this situation:

Team,
I understand all the issues but there are complex interrelations that must be worked out. I am currently constipated on each issue and will let you know when there is movement.

• Tegan
  MBA, CAPM, PMP

Her email cost the project many person-hours as all the engineering teams took a break to have a good laugh about the project manager admitting, in writing, that she was full of crap.

There was, eventually, movement. Tegan moved on to a new position at a different company. Her replacement, Pam, wasn't a new hire, but instead a transfer from another department. She wasn't a great project manager, certainly not worth her weight in gold, but she had enough experience to avoid the worst mistakes, and most important: she was good at regular communication in order to keep things moving.

[Advertisement] Keep the plebs out of prod. Restrict NuGet feed privileges with ProGet. Learn more.

When Gary was called in for a meeting with a few of his managers- because of course he had several- he thought it was going to be for an "attaboy", because things had been going really well for the past few months.

Gary had inherited a mess, and taken over a nightmare application. It was the kind of application that should be a simple CRUD-style data-driven app, but somehow despite only having 20ish entities it managed, someone had generated 500+ controllers for managing them. Most of those controllers were copy/pasted code with minor changes in the WHERE clause of a SQL query.

And that was just the code. The infrastructure was similarly a mess, with duplicate resources provisioned in their cloud host. There was no CI/CD, no unit tests to speak of, no deployment process that wasn't "manually copy these files and pray". And uptime? You've heard about "five nines", but this product was lucky to get even one nine. Especially because the manual deployment process meant a few hours of downtime.

And that was just the infrastructure. The backlog was similarly messy. There were lots of tasks- many thousands- but not a single one had a priority. Most of the tasks were something like, "Fix database timeouts", or "Bug 531" with no description to explain what they were. At best, some of the "new feature" tasks linked to a Google Doc that explained a software roadmap that had been last updated in 2020.

So with no guidance, Gary and the rest of his team got to work. Cloud costs were massive. Just cutting the duplicate resources would help, but with actual planning it wasn't hard to find even bigger wins. In total, Gary got the cloud costs down 60%- essentially saving the company a small multiple of his salary every year.

With that out of the way, getting a CI/CD pipeline running was next. Within a few weeks, manual deployments were gone. Everything was automated. Downtime nearly vanished. And now, with all the cost savings in cloud resources, for a fraction of what they were paying, it was easy to automate provisioning test environments for each new feature.

So Gary was very ready for some congratulations when he sat down with management. He was prepared to discuss all the wins he and the rest of the developers on the project had gotten over the past few months.

"I'm sure you know why we're sitting down," Manager the First said when they settled into the conference room.

"I'm sure," said Manager the Second.

"We have some concerns about your performance," Manager the Third said.

"My performance?" Gary asked.

"Yes," said Manager the First. "Let me pull up the backlog."

"And the roadmap," said Manager the Second.

"Yes, I'm getting that up too, thank you." The trio of managers struggled with pulling up the appropriate pages, and after about 15 minutes, gave up. Instead, they discussed their complaint without visual aids. "You haven't completed any of the tasks on the roadmap. Bug 673 has been open since you started on the team. None of the roadmap milestones have been touched. There's absolutely no progress."

"Okay, but that document was wildly out of date," Gary said. "Instead I put cycles into solving the actual problems we're having. I've saved the company a huge amount of money. I've gotten our development cycle time down to a fraction of what it was. And we have basically no downtime!"

"That's all very nice, I'm sure," said Manager the Third. "But none of that was on our roadmap."

"Well, maybe we should set up a meeting to go over the roadmap," Gary said. "Because a lot of the tasks on there don't make much sense right now-"

"I don't think that's a good use of time," Manager the Second said. "Large meetings are expensive. Just stick to the roadmap, please."

With that, the meeting ended. Gary went back to work…

… updating his resume.

[Advertisement] Keep the plebs out of prod. Restrict NuGet feed privileges with ProGet. Learn more.

The real WTF is that our long-time friend and submitter Argle failed to dissuade all three of his sons from pursuing IT careers of their own:

Back circa 2012, my three sons all got jobs at a company that had a brilliant web project. So brilliant that it had the support of a Disney VP, the mayor of the city, and other VIPs. At one point, my sons asked to borrow money to invest in the project. They are good boys (one is now a senior developer with Proctor & Gamble), so I backed them.

A year later, the project was released late, over budget, and not fully functional.

My boys convinced the CEO to bring me in to fix things. I fixed things. In that time, I found out they had taken bids on the project. Bids were nominally $15,000, some higher, some lower, of course. All but one group that had bid $5,000. Their plan? Hire some programmers in India for $8/hour and pocket the money without having to do work themselves.

Costs had shot well over $35,000 before I was brought in.

After I got the system working, I went to one of the weekly general standups for the company. The CEO walked in and said something like, "I just learned that Facebook was written in PHP. I think we should rewrite the whole project in PHP. That's what we really need to do."

And thus the decision was made.

A meeting was held the next day to discuss how long it would take to remake the project in PHP instead of C#. Bear in mind, a year and a half had been thrown into making the project thus far.

Going around the table, everyone said between 2 and 3 weeks. There was one other programmer in the company who had exactly 2 months of work experience; he simply parroted what the others had said before him. There was also the general contractor who leased the building to the company. He was involved with the project, and was second-to-last to speak. I fully expected this contractor to have more sense. He came in at 3 to 4 weeks.

My mouth dropped open.

It was my turn. You know those psych tests where you get someone who acts sensibly when alone, but conforms with the rest of the crowd when there's more than one? I'm simply not that guy. I said, "Those are absurd estimates! This will take a minimum of 5 months before it's in beta stages and not ready for public consumption for another couple more months."

The next day, I got a call telling me my services were no longer needed because "I wasn't forward-thinking enough for the company."

My boys stayed on another year, so I got regular reports on the "upgrade." Sure enough, just shy of 8 months later, the new system went live.

As they say, the most experienced person will be the one to accurately tell everyone that it will take longer and cost more than everyone else says.

Anyone else have their own intergenerational WTFs? Please share in the comments!

[Advertisement] Keep all your packages and Docker containers in one place, scan for vulnerabilities, and control who can access different feeds. ProGet installs in minutes and has a powerful free version with a lot of great features that you can upgrade when ready.Learn more.

The waters are even more dangerous than we imagined. Have a look at some of the crazed whales our brave submitters and commenters have encountered in the wild.

First comes an Anonymous tale of woe:

Our company makes apps for businesses. We have 1 MAIN client whose CEO can make or break our company, and his wish is our command. He sent a priority email on a Friday night saying the app was slow and needed to be fixed.

The client CEO is so important that he works directly with our CEO, who decided to PM this huge issue.

All weekend, we were trying out tons of different things to optimize this "slow" app that "wasn't loading or refreshing." We deployed the app Monday night after a weekend of unpaid overtime (darn salary). On Tuesday, the account manager made a bug card to officially represent the work we did, and they posted a previously-unseen video of the slowness.

There is a refresh icon that spins when clicked. The video was of the refresh icon, and it was spinning for an extra second after the data loaded (and jumping 2 pixels from padding styling).

That is what was high priority.

I mean, we all hate the system, but sometimes the system is actually there to protect us.

Next, we have Daniel's ongoing peril:

We do digital flyers/circulars/ads. Eight years ago, that meant we got PDFs from retailers and turned them into digital content. One huge retailer (hundreds of stores) wanted a dynamically-created flyer that would have up-to-date pricing twice a day. We didn't have time to build out a full digital solution (which would have made sense), so instead we spent six months banging together a solution with spit and duct tape which baked out hundreds of PDFs every morning and afternoon. This one retailer was responsible for about 40% of our processing power.

We're finally getting somewhat closer to phasing this out, but "it worked" for this long ...

Finally, let's be grateful Brian escaped with his life!

Worked for a company that was building a component of a high-profile weapons platform for one of the major military suppliers. We had taken over the project from another company that was under-performing, so we were already behind schedule from the minute the contract was signed. Of course this company saw fit to treat us more as a subsidiary than a subcontractor. Including, for a time, sending one of their own managers to sit in our lab and observe (read: babysit) us. On Saturdays. Then they demanded we start working shifts to make more use of the lab equipment, and I got the bad draw: 3 AM - noon. Never mind that I had just gotten married (they actually called to tell me this while I was on vacation the week after my wedding) and would like to actually spend some time with my wife ...

That experience soured me on the whole military-industrial complex for a long time. To this day I still get headhunters pinging me to work for that megacorp; I just chuckle and delete their messages.

Have these tales knocked loose any foul memories that your brain tried to repress? Send them to us!

[Advertisement] ProGet’s got you covered with security and access controls on your NuGet feeds. Learn more.

JB has a database table that, at first glance, looks like one of those data warehouse tables that exists to make queries performant. You know the sort, the table that contains every date between 1979 and 2050, or every number out to 1,000,000 or something. It looks dumb, but it helps make certain joins and queries performant.

The database table is called three_alpha_numerics. It has two columns: digit, which contains three characters, and is_numeric, which is a a single character: 'Y' or 'N'. It looks roughly like this:

+-------+------------+
| digit | is_numeric |
+-------+------------+
| 009   | Y          |
+-------+------------+
| 00A   | N          |
+-------+------------+

So, for example, if you wanted all the possible numeric triples, you could SELECT digit FROM three_alpha_numerics WHERE is_numeric = 'Y', which is obviously the easiest thing one can imagine.

So what is this for? Well, it's used by a stored procedure that generates unique IDs. That stored procedure does a left join against another table to find all the unused digits. And here's the real gotcha: that stored procedure only ever uses the rows where is_numeric is Y, meaning the vast majority of the data in this table is never used.

Unique IDs, of course, are an incredibly difficult task for databases to do, so it absolutely makes sense that we create a system that allows us to only have 1,000 unique IDs. That's more than 640, which should be enough for anyone. Having many thousands of unusable alphanumeric triplets is just the cost we have to pay.

[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!

Janči, a master's student of bioinformatics, was seated near the back of a large classroom. This was a simple compulsory elective course geared toward biologists. The professor was currently walking the class through their latest assignment. "We'll need to connect to some Linux servers," he announced.

The other students seated nearby traded blank stares. They were all Mac and Windows users with no IT background. Meanwhile Janči, a veteran Linux user, started feeling a little smug. An easy A was at hand.

"First," the professor continued, "you'll need a private key."

After the professor had explained a few details, the first WTF came in the form of a bulk email sent to the entire class. The private key was attached. The username was the email address it was sent to.

What do you call the exact opposite of a private key? Janči wondered, bemused.

"You'll also need to download an application to help you log in," the professor said. "I recommend MobaXterm."

As he detailed the process of visiting the SSH client website to download the software, Janči tuned out. He didn't need such hand-holding. He accessed OpenSSH, tried connecting ...

... and failed.

Meanwhile, everyone around him was logging in no problem.

Janči's face burned with embarrassment at this second WTF. His first instinct was to blame the deprecated cryptography of the server. He spent most of the remaining lecture time searching for a way to allow his SSH to use SSH-DSS. (It turned out to be supported the whole time, despite the warnings he received.)

Janči then tried to re-download the "private" key and adjust the SSH config file several times. He cycled through different possible usernames associated with his university email account.

No dice.

He was the only person in the class who hadn't yet logged into the server. Not even the professor was able to help him, since he was using Linux.

Embarrassment and frustration mounted. An hour later, out of ideas, Janči fell back to downloading MobaXterm and running it inside Wine.

It didn't work.

The professor offered him a spare Windows box. "Here, try this one."

Janči booted it up, copied the "private" key to the new machine ... and still couldn't sign in.

Now, this was getting suspicious.

The lecture ended. A friend of Janči's hung back while the rest of the students filed out. "Why don't you try logging in with my credentials instead of yours?" she asked.

Janči was up for anything at that point.

It worked. On his own machine, on the Windows box, everywhere.

With that lead in mind, Janči opened the server's /etc/passwd file to look at all the usernames. He noticed that, unlike everyone else, his username and email address didn't match.

His university used Microsoft emails. Everyone had several address aliases, and they could also use whatever email address they liked in the system, even a personal one.

Janči had chosen to use a school email in the form of <number>@uni.uni. Unfortunately, the Ubuntu server didn't like the idea of user being named just <number>, so it had renamed it to user<number>. Some script for generating SSH configuration had probably failed from there, because Janči also discovered that his user home directory was missing a .ssh directory and known_hosts file.

Unfortunately, due to restricted access, he wasn't able to copy them from any of his classmates. In the end, he could connect to the server as any of his classmates, but not as himself.

If you've seen one developer recounting how their AI agent deleted production, you've seen them all. They're mostly not interesting stories. It's like watching someone speeding through traffic on a motorcycle without a helmet: the eventual tragedy is sad, but it's unsurprising and not an interesting story to tell. It's not even interesting as a warning: the kind of person who speeds on a motorcycle without a helmet isn't doing so because they don't understand the danger. They've just decided it doesn't apply to them.

But the founder of PocketOS, Jer, recently shared how- whoopsie!- their AI agent deleted production. There's a lot of ingredients that go into this particular disaster, which I think makes it interesting, because the use of a poorly supervised AI agent is only one ingredient in this absolute trainwreck of a story.

PocketOS is a small company that makes software for rental companies to manage reservations. Car rentals are a big customer, but the tool is more general than that. They manage all of their infrastructure via a service called Railway. Railway is a pretty-looking GUI tool for automating your deployments and the target environments.

PocketOS also is heavily adopting Cursor wrapping around the Claude model. They've paid big bucks for the top-end model offered. Many of their components, like Railway, offer MCP services so that their LLM can do useful things. They're using the Claude LLM to automate as much as they can.

So far, this is all a pretty typical setup. They pointed Claude at their code and gave it a "routine" task, and sent it to work. It toddled through the problem and encountered a credential issue. It "decided" that the fix for this issue was to delete a storage volume and recreate it. It scanned through the code to find a file containing an API key, found it, and then sent a POST request via cURL to delete the volume in question.

Jer writes:

To execute the deletion, the agent went looking for an API token. It found one in a file completely unrelated to the task it was working on. That token had been created for one purpose: to add and remove custom domains via the Railway CLI for our services. We had no idea — and Railway's token-creation flow gave us no warning — that the same token had blanket authority across the entire Railway GraphQL API, including destructive operations like volumeDelete. Had we known a CLI token created for routine domain operations could also delete production volumes, we would never have stored it.

Wait, the tokens you create in Railway all have god-level privileges? That sounds like a terrible idea. And you were storing the token in your code? We'll come back to this in a moment, but sure, this is bad, but you can just restore from backup, right?

The volume was deleted. Because Railway stores volume-level backups in the same volume — a fact buried in their own documentation that says "wiping a volume deletes all backups" — those went with it. Our most recent recoverable backup was three months old.

Oh. Oh no.

Now, I don't think it's literally true that Railway is storing your backups literally in the same volume as the thing they're backing up. I certainly hope not. But they do apparently delete your backups when you delete the volume associated with them. Which is a choice, certainly. A bad one. And one that they documented, according to Jer. It was, in his words, "buried" in the docs.

But let's go back to the tokens for a moment. I am not a Railway user, but I checked out the tool and went through the process of creating a project token. And while no, Railway does not give you big red flags warning you "Hey, this token can do ABSOLUTELY ANYTHING", it also never gives you an opportunity to scope the token. Which, I don't know about you, but the first thing I do when I create an authentication entity is try and figure out how to control its authorizations, because I assume at the start it doesn't have any. That'd be sane.

The scoping happens when you create the token, depending on what context you're in when you do it. It's only a handful of scopes, and no fine grained permissions on API keys at all. The lowest level is "Project" which can do anything to a single environment- which does mean that even if you, like Jer's team, wanted to have a script that changed some DNS settings in production, that same key could be used to delete volumes in production. Which means you really really want to take care of that key, and you certainly don't want to leave it where some junior developer or bumbling AI agent can find it.

Jer also complains that Railway shouldn't allow an API call to take destructive actions without more protections, like forcing someone to type in the name of the thing being deleted or sending a confirmation email, or something. This, I'm more skeptical of. Most cloud providers don't offer anything like this in their APIs, at least that I've seen, because on a certain level, if you're invoking the API with the proper credentials, that's a big enough hill to climb that we can assume you've intended your action. The correct way to protect against this is properly scoped keys and keeping those keys secure and not just lying around in plain text. There's a certain aspect of understanding that you're using a potentially dangerous tool and need to take the responsibility for safety into your own hands; while a table saw can easily take some fingers off, it's perfectly safe when used correctly.

This is all bad, but how can we make it worse? Well, Jer demanded that Claude "explain itself". In a section called "The Agent's Confession", Jer highlights that the agent is able to identify the explict rules that it failed to follow.

Read that again. The agent itself enumerates the safety rules it was given and admits to violating every one. This is not me speculating about agent failure modes. This is the agent on the record, in writing.

No, it is not the agent on record. I see this kind of thing a lot when people talk about LLMs. An LLM cannot explain its reasoning. It cannot go on "the record". It cannot confess to anything. While what it plops out when asked might be interesting, it is not an explanation. The only explanation is that it's a powerful statistical model trying to create a plausible string of tokens! It's simply looking at its context window and your prompt and trying to predict what it should say. It can tell you what rules it violated not because it understands the rules or knows it violated any rules, but because those rules are in its context window. If you ask it right, it'll confess to killing JFK and framing Oswald for the crime.

Jer then tries to ensure that Cursor takes some of the blame, pointing to Cursor's "guardrails" documentation. Except, here, the documentation is actually quite explicit about what those guardrails guarantee. If you're using a first-party tool, it will prohibit unsafe operations. When using 3rd party MCPs, like Railway's, the only guardrail is that it requires human approval for every action- unless you update your allowlist for that MCP. If you put them in your allowlist, the guardrails go away. Jer argues that tools should enforce more protection against LLM behaviors, but the problem with that is people- like the PocketOS team- turn those protections off. And like a lot of safety mistakes, they can get away with it all the way up until the point where they can't.

Jer follows this by listing off a pile of other times using Cursor has caused disasters, which isn't making the argument he thinks it is: yes, Cursor is dangerous, but those dangers are well known. It makes the choice to turn Cursor loose without strict supervision seem even more foolish.

Jer writes:

For now I want this incident understood on its own terms: as a Cursor failure, a Railway failure, and a backup-architecture failure that all happened to one company in one Friday afternoon.

It's also a PocketOS failure. It's a failure to properly assess the tools and environments you chose to use for your product. A failure to read and understand the docs for vital features, like *backups*. A failure to employ even the most basic safeguards. A failure to put a second's thought into key management- even if that key was only for DNS entries, you still shouldn't chuck it in source control. A failure to have a competent backup strategy. It's worth noting that they did restore from a three month old backup, which means they were at one point taking backups outside of Railway's volume setup. That was a wise decision. That they stopped is a failure.

The first rule of disaster retrospectives is that it's never one piece that's the failure. It's never one person's fault, one tool's fault, one vendor's fault. It's a systemic failure. Railway's keys should be finer grained. But also, you shouldn't leave keys lying around. Deleting backups when you delete the volume is a terrible idea, but having only one service for backups (that's also your primary site) is a terrible idea. Claude's ability to enforce its own guardrails should be better, but LLMs are notoriously dangerous about this: you should know better, and by your own words you did.

This is not an anti-AI post, or even a "get a load of this asshole" post. It is a "understand the damn tools you're using" post. Be critical of them. Don't trust them. Ever. Especially LLMs, because the worst part of an LLM is that it takes away the one thing computers used to be good at: predictable, deterministic behavior. But not just LLMs: don't trust your cloud provider, don't trust your infrastructure manager. Dig into them and understand how they work, and if they seem to complicated to understand, than they may be too complicated to trust.

Update: As pointed out in the featured comment below, Railway did finally get a backup restored. So they got their data back. Yay? From the post, Jer remains committed to making this a Railway issue and not a PocketOS issue.

From our Anonymous submitter:

Our company creates graphs to visualize data. We have many small fish customers, but we have one whale who uses our product that is 90% of company revenue. (WTF number 1.)

So if he is not happy, it's all-hands-on deck-mode.

He complained that our APIs and charts are loading slowly for him. For 3 weeks, we've tried a TON of optimizations, including WTF 2: spinning up a special server he alone can hit.

Today, we found out that he's always complaining when he's in his car, driving from home to the office. But since he "totally has the best wifi money can buy," that isn't worth investigating.

WTF 3: thinking wifi and data are always 100% reliable in a car driving around.

Our submitter highlights one of the major pitfalls of the so-called whale client: if they're a bad client, you're in for an extra-bad time.

As I lean harder into freelancing, I'm learning to scan the waters ahead of me for potential whales. My goal is to build up multiple small, diverse income streams, because I've had my own dangerous encounters with whales in the past.

At one employer of mine, there was Facebook, who acted as if they were our new owners rather than a new customer. They'd already produced flashy marketing videos of the sorts of solutions they planned to implement with our software, showing people delighted with the results. In meetings, these things were talked up as amazing game-changers. Meanwhile, I found all the things Facebook wanted to do horribly creepy and invasive.

Even worse, Facebook began dictating how our award-winning technical support should change to accommodate their whims, up to and including having a dedicated toady—er, support rep—who did nothing but field Facebook-related tickets, similar to a technical account manager (TAM).

That was the last straw for me. I left that company before I was forced to deal with any of Facebook's crap.

My second whale sighting occurred at a startup that'd landed Porsche, far and away their biggest client ever. All of a sudden, our timeline for adding new features and fixing bugs became Porsche's honey-do list. All of a sudden, the platform frequently crashed and became unusable for everyone because it couldn't handle the amount of traffic Porsche (and their clients) hurled at it.

On the other hand, there were several times in that startup's existence when a big wad of promised funding failed to materialize. Porsche kept the business afloat and literally kept my lights on.

I find it less than ideal to be at any company's mercy. I want a world that would neither spawn whales nor millions of startups named Sploink, Dink, and Twangle that promise to bring the power of AI to your dinner fork.

Have your own epic whaling adventures? Share with us in the comments!

[Advertisement] Utilize BuildMaster to release your software with confidence, at the pace your business demands. Download today!

Eric O worked for a medical device company. The medical device industry moves slowly, relative to other technical industries. Medical science and safety have their own cadence, and at a certain point, iterating faster doesn't matter much.

Eric was working on a new feature on a system that had been in use for thirteen years. This new feature interacted with a database which stored information about racks of test tubes, and Eric's tests meant creating several entries for racks of test tubes. And that's when Eric discovered that the database only allowed thirty racks. Add any more, it would just roll right back over to one.

This was odd. The database was small- less than 40MB, even in production- and there were automatic tasks to purge old data for compliance purposes. Why a hard limit of thirty?

Eric had only been at the company for a year, so he asked one of the more senior team members, Lester. "Oh yeah, that was before my time. You should probably ask Carl."

Later that day, Eric happened to bump into Carl around the coffee maker, and asked the question. "Oh, yeah, I do vaguely remember something about that. It was in the requirements for the product. I thought it was weird, but didn't think too much about it. You should probably ask Elise, she's been here like twenty years."

Well, now it was getting curious. Eric went over to the "old building", as it was named, the original office for the company on the other side of the parking lot. Most of the offices had moved to the new building a decade earlier, and it mostly served as fabrication and storage, but a few offices remained.

Elise was on the third floor, down a poorly lit hallway, sitting in an office with water-stained acoustical tile in its ceiling. "Oh, yeah, I put that into the requirements document. It's funny, I thought it was weird too, but the system you're working on was a replacement for an older system. Our requirements were derived from those. Let me think… Irving worked on that, but he's dead, god rest him. Penny is retired. Oh, you know, Humbert is still around. He didn't work on that, but he worked on some of the systems that came before that. He's upstairs and on the other side of the building."

Eric went upstairs and to the other side of the building. The fourth floor had been last remodeled circa 1985, and the ugly industrial paint on the wall was made even uglier by the fact that someone had replaced most of the flourescent tubes with LEDs. Most. The mismatched color temperature started Eric down the path of a headache.

Humbert was in an office similar to Elise's. On his desk was a plaque commemerating 40 years of service with the company. Eric asked about the limitation, and Humbert laughed.

"You're working on the latest version of a product that initially started on an old PDP-11 running MUMPS. I mean, the first versions, anyway. We ran to desktop computers as fast as we could. I wrote a version for DOS in… oh… '86? I knew none of the facilities we worked with had more than ten or fifteen racks of tubes, and I needed somehow to limit the size of the database so it all fit on a single 5 1/4" floppy disk. I picked thirty, because it seemed like a good round number. Honestly, I'm shocked that the limit still exists."

So was Eric. There had been several ground-up-rewrites since 1986, before the one Eric maintained had been released thirteen years ago. Each one of them had chosen to maintain the same limitation, without ever considering why it existed. The rule had simply been copied, mindlessly, for 40 years.

"I'm kind of impressed," Eric said to Humbert, "in a horrified way."

"Me too, kid, me too."

[Advertisement] Keep all your packages and Docker containers in one place, scan for vulnerabilities, and control who can access different feeds. ProGet installs in minutes and has a powerful free version with a lot of great features that you can upgrade when ready.Learn more.

Our anonymous submitter relates a tale of simplification gone bad. As this nightmare unfolds, imagine the scenario of a new developer coming aboard at this company. Imagine being the one who has to explain this setup to said newcomer.

Imagine being the newcomer who inherits it.

David's job should have been an easy one. His company's sales data was stored in a database, and every day the reporting system would query a SQL view to get the numbers for the daily key performance indicators (KPIs). Until the company's CTO, who was proudly self-taught, decided that SQL views are hard to maintain, and the system should get the data from one of those new-fangled APIs instead.

But how does one call an API? The reporting system didn't have that option, so the logical choice was Azure Data Factory to call the API, then output the data to a file that the reporting system could read. The only issue was that nobody on the team spoke Azure Data Factory, or for that matter SQL. But no problem, one of David's colleagues assured, they could do all the work in the best and most multifunctional language ever: C#.

But you can't just write C# in a data factory directly, that would be silly. What you can do is have the data factory pipeline call an Azure function, which calls a DLL that contains the bytecode from C#. Oh, and a scheduler outside of the data factory to run the pipeline. To read multiple tables, the pipeline calls a separate function for each table. Each function would be based on a separate source project in C#, with 3 classes each for the HTTP header, content, and response; and a separate factory class for each of the actual classes.

After all, each table had a different set of columns, so you can't just re-use classes for that.

There was one little issue: the reporting system required an XML file, whereas the API would export data in JSON. It would be silly to expect a data factory, of all things, to convert this. So the CTO's solution was to have another C# program (in a DLL called by a function from a pipeline from an external scheduler) that reads the JSON document saved by the earlier program, uses foreach to go over each element, then saves the result as XML. A distinct program for each table, of course, requiring distinct classes for header, content, response, and factories thereof.

Now here's the genius part: to the C# class representing the output data, David's colleague decided to attach one different object for each input table required. The data class would use reflection to iterate over the attached objects, and for each object, use a big switch block to decide which source file to read. This allows the data class to perform joins and calculations before saving to XML.

To make testing easier, each calculation would be a separate function call. For example, calculating a customer's age was a function taking struct CustomerWithBirthDate as input, use a foreach loop to copy all the data except replacing one field, and return a CustomerWithAge struct to pass to the next function. The code performed a bit slowly, but that was an issue for a later year.

So basically, the scheduler calls the data factory, which calls a set of Azure functions, which call a C# function, which calls a set of factory classes to call the API and write the data to a text file. Then, the second scheduler calls a data factory, which calls Azure functions, which call C#, which calls reflection to check attachment classes, which read the text files, then call a series of functions for each join or calculation, then call another set of factory classes to write the data to an XML file, then call the reporting system to update.

Easy as pie, right? So where David's job could have been maintaining a couple hundred lines of SQL views, he instead inherited some 50,000 lines of heavily-duplicated C# code, where adding a new table to the process would easily take a month.

Or as the song goes, Somebody Told Me the User Provider should use an Adaptor to Proxy the Query Factory Builder ...

[Advertisement] ProGet’s got you covered with security and access controls on your NuGet feeds. Learn more.

The utility closet Ellis had inherited and lived with for 17 years had been a cesspool of hazards to life and limb, a collection of tangible WTFs that had everyone asking an uncaring god, "What were they thinking?"

Every contractor who'd ever had to perform any amount of work in there had come away appalled. Many had even called over their buddies to come and see the stunning mess for themselves:

• All of the electrical components, dating from the 1980s, were scarily underpowered for what they were supposed to be powering.
• To get to the circuit breaker box—which was unlabeled, of course—one had to contort themselves around a water heater almost as tall as Ellis herself.
• As the house had no basement, the utility closet was on the first floor in an open house plan. A serious failure with said water heater would've sent 40 gallons (150 liters) of scalding-hot tsunami surging through the living room and kitchen.
• The furnace's return air vent had been screwed into crumbling drywall, and only prayers held it in place. Should it have fallen off, it would never have been replaceable. And Ellis' cat would've darted right in there for the adventure of a lifetime.
• To replace the furnace filter, Ellis had to put on work gloves, unscrew a sharp sheet-metal panel from the side of the furnace, pull the old filter out from behind a brick (the only thing holding it in place), manipulate the filter around a mess of water and natural gas pipes to get it out, thread the new filter in the same way, and then secure it in place with the brick before screwing the panel back on. Ellis always pretended to be an art thief in a museum, slipping priceless paintings around security-system lasers.
• Between the water tank, furnace, water conditioning unit, fiber optical network terminal, and router, there was barely room to breathe, much less enough air to power ignition for the gas appliances. Some genius had solved this by cutting random holes in several walls to admit air from outside. One of these holes was at floor-level. Once, Ellis opened the closet door to find a huge puddle on the floor, making her fear her hot water heater was leaking. As it turned out, a power-washing service had come over earlier that day. When they'd power-washed the exterior of her home, some of that water shot straight through one of those holes she hadn't known about, giving her utility closet a bonus bath.
• If air intake was a problem, venting the appliances' exhaust was an even worse issue. The sheet-metal vents had calcified and rusted over time. If left unaddressed, holes could've formed that would've leaked carbon monoxide into Ellis' house.

Considering all the above, plus the fact that the furnace and air conditioner were coming up on 20 years of service, Ellis couldn't put off corrective action any longer. Last week, over a span of 3 days, contractors came in to exorcise the demons:

• Upgrading electricals that hadn't already been dealt with.
• Replacing the hot water tank with a wall-mounted tankless heater.
• Replacing the furnace and AC with a heat pump and backup furnace, controlled by a new thermostat.
• Creating new pipes for intake and venting (no more reliance on indoor air for ignition).
• Replacing the furnace return air vent with a sturdier one.
• Putting a special hinged door on the side of the furnace, allowing the filter to be replaced in a matter of seconds (RIP furnace brick).

With that much work to be done, there were bound to be hiccups. For instance, when the Internet router was moved, an outage occurred: for no good reason, the optical network terminal refused to talk to Ellis' Wifi router after powering back up. A technician came out a couple days later, reset the Internet router, and everything was fine again.

All in all, it was an amazing and welcome transformation. As each new update came online, Ellis was gratefully satisfied. It seemed as though the demons were finally gone.

Unbeknownst to them all, there was one last vengeful spirit to quell, one final WTF that it was hell-bent on doling out.

It was late Friday afternoon. Despite the installers' best efforts, the new thermostat still wasn't communicating with the new heat pump. Given the timing, they couldn't contact the company rep to troubleshoot. However, the thermostat was properly communicating with the furnace. And so, Ellis was left with the furnace for the weekend. She was told not to mess with the thermostat at all except to adjust the set point as desired. They would follow back up with her on Monday.

For Ellis, that was perfectly fine. With the historically cold winter they'd been enduring in her neck of the woods, heat was all she cared about. She asked whom to contact in case of any issues, and was told to call the main number. With all that squared away, she looked forward to a couple of quiet, stress-free days before diving back into HVAC troubleshooting.

Everything was fine, until it wasn't. Around 11AM on Saturday, Ellis noticed that the thermostat displayed the word "Heating" while the furnace wasn't actually running. Maybe it was about to turn on? 15 minutes went by, then half an hour. Nothing had changed except for the temperature in her house steadily decreasing.

Panic set in at the thought of losing heat in her home indefinitely. That fell on top of a psyche that was already stressed out and emotionally exhausted from the last several days' effort. Struggling for calm, Ellis first tried to call that main number line for help as directed. She noticed right away that it wasn't a real person on the other end asking for her personal information, but an AI agent. The agent informed her that the on-call technician had no availablity that weekend. It would pencil her in for a service appointment on Monday. How did that sound?

"Not good enough!" Ellis cried. "I wanna speak to a representative!"

"I understand!" replied the blithe chatbot. "Hold on, let me transfer you!"

For a moment, Ellis was buoyed with hope. She'd gotten past the automated system. Soon, she'd be talking with a live person who might even be able to walk her through troubleshooting over the phone.

The new agent answered. Ellis began pouring her heart out—then stopped dead when she realized it was another AI agent, this time with a male voice instead of a female one. This one proceeded through nearly the same spiel as the first. It also scheduled her for a Monday service appointment even though the other chatbot had already claimed to have done so.

This was the first time an AI had ever pulled such a trick on Ellis. It was not a good time for it. Ellis hung up and called the only other person she could think to contact: her sales rep. When he didn't answer, she left a voicemail and texts: no heat all weekend was unacceptable. She would really appreciate a call back.

While playing the horrible waiting game, Ellis tried to think about what she could do to fix this. They had told her not to mess with the thermostat. Well, from what she could see, the thermostat was sending a signal to the furnace that the furnace wasn't responding to for whatever reason. It was time to look at the docs. Fortunately, the new furnace's manual was resting right on top of it. She spread it open on her kitchen table.

OK, Ellis thought, this newfangled furnace has an LED display which displays status codes. Her old furnace had lacked such a thing. Lemme find that.

Inside her newly remodeled utility closet, she located the blinking display, knelt, and spied the code: 1dL. Looking that up in the doc's troubleshooting section, she found ... Normal Operation. No action.

The furnace was OK, then? Now what?

Aside from documentation, another thing Ellis knew pretty well was tech support. She decided to break out the ol' turn-it-off-and-on-again. She shut off power to both the furnace and thermostat, waited a few minutes, then switched everything back on, crossing her fingers.

No change. The indoor temperature kept dropping.

Her phone rang: the sales rep. He connected her with the on-call technician for that weekend, who fortunately was able to arrive at her house within the hour.

One tiny thermostat adjustment later, and Ellis was enjoying a warm house once more.

What had happened?

This is where an understanding of heat pumps comes into play. In this configuration, the heat pump is used for cooling and for heating, unless the outside temperature gets very cold. At that point, the furnace kicks in, which is more efficient. (Technology Connections has some cool videos about this if you're curious.)

Everything had been running fine for Ellis while the temperatures had remained below freezing. The problem came when, for the first time in approximately 12 years, the temperature rose above 40F (4C). At that point, the new thermostat decided, without telling Ellis, I'm gonna tell the HEAT PUMP to heat the joint!

... which couldn't do anything just then.

Workaround: the on-call technician switched the thermostat to an emergency heat mode that used the furnace no matter what.

Ellis had been told not to goof around with the thermostat. Even if she had, as a heat pump neophyte, she wouldn't have known to go looking for such a setting. She might've dug it up in a manual. Someone could've walked her through it over the phone. Oh, well. There is heat again, which is all that matters.

They will attempt to bring the heat pump online soon. We shall see if the story ends here, or if this becomes The WTF That Wouldn't Die.

P.S. When Ellis explained the AI answering service's deceptive behavior, she was told that the agent had been universally complained about ever since they switched to it. Fed up, they told Ellis they're getting rid of it. She feels pretty chuffed about more people seeing the light concerning garbage AI that creates far more problems than it solves.

[Advertisement] ProGet’s got you covered with security and access controls on your NuGet feeds. Learn more.

On Thanksgiving Day, Ellis had cuddled up with her sleeping cat on the couch to send holiday greetings to friends. There in her inbox, lurking between several well wishes, was an email from an unrecognized sender with the subject line, Final Account Statement. Upon opening it, she read the following:

Dear Ellis,

Your final account statement dated -1 has been sent to you. Please log into your portal and review your balance due totaling #TOTAL_CHARGES#.

Payment must be received within 30 days of this notice to avoid collection. You may submit payment online via [Payment Portal Link] or by mail to:

Chamberlin Apartments
123 Main Street
Anytown US 12345

If you believe there is an error on your account, please contact us immediately at 212-555-1212.

Thank you for your prompt attention to this matter.

Chamberlin Apartments

Ellis had indeed rented an apartment managed by this company, but had moved out 16 years earlier. She'd never been late with a payment for anything in her life. What a time to receive such a thing, at the start of a long holiday weekend when no one would be able to do anything about it for the next 4 days!

She truly had so much to be grateful for that Thanksgiving, and here was yet more for her list: her broad technical knowledge, her experience working in multiple IT domains, and her many years of writing up just these sorts of stories for The Daily WTF. All of this added up to her laughing instead of panicking. She could just imagine the poor intern who'd hit "Send" by mistake. She also imagined she wasn't the only person who'd received this message. Rightfully scared and angry callers would soon be hammering that phone number, and Ellis was further grateful that she wasn't the one who had to pick up.

"I'll wait for the apology email!" she said out loud with a knowing smile on her face, closing out the browser tab.

Ellis moved on physically and mentally, going forward with her planned Thanksgiving festivities without giving it another thought. The next morning, she checked her inbox with curious anticipation. Had there been a retraction, a please disregard?

No. Instead, there were still more emails from the same sender. The second, sent 7 hours after the first, bore the subject line Second Notice - Outstanding Final Balance:

Dear Ellis,

Our records show that your final balance of #TOTAL_CHARGES# from your residency at your previous residence remains unpaid.

This is your second notice. Please remit payment in full or contact us to discuss the balance to prevent your account from being sent to collections.

Failure to resolve the balance within the next 15 days may result in your account being referred to a third-party collections agency, which could impact your credit rating.

To make payment or discuss your account, please contact us at 212-555-1212 or accounting@chamapts.com.

Sincerely,

Chamberlin Apartments

The third, sent 6 and a half hours later, threatened Final Notice - Account Will Be Sent to Collections.

Dear Ellis,

Despite previous notices, your final account balance remains unpaid.

This email serves as final notice before your account is forwarded to a third-party collections agency for recovery. Once transferred, we will no longer be able to accept payment directly or discuss the account.

To prevent this, payment of #TOTAL_CHARGES# must be paid in full by #CRITICALDATE#.

Please submit payment immediately. Please contact 212-555-1212 to confirm your payment.

Sincerely,

Chamberlin Apartments

It was almost certainly a mistake, but still rather spooky to someone who'd never been in such a situation. There was solace in the thought that, if they really did try to force Ellis to pay #TOTAL_CHARGES# on the basis of these messages, anyone would find it absurd that all 3 notices were sent mere hours apart, on a holiday no less. The first two had also mentioned 30 and 15 days to pay up, respectively.

Suddenly remembering that she probably wasn't the only recipient of these obvious form emails, Ellis thought to check her local subreddit. Sure enough, there was already a post revealing the range of panic and bewilderment they had wrought among hundreds, if not thousands. Current and more recent former tenants had actually seen #TOTAL_CHARGES# populated with the correct amount of monthly rent. People feared everything from phishing attempts to security breaches.

It wasn't until later that afternoon that Ellis finally received the anticipated mea culpa:

We are reaching out to sincerely apologize for the incorrect collection emails you received. These messages were sent in error due to a system malfunction that released draft messages to our entire database.

Please be assured of the following:
The recent emails do not reflect your actual account status.
If your account does have an outstanding balance, that status has not changed, and you would have already received direct and accurate communication from our office.
Please disregard all three messages sent in error. They do not require any action from you.

We understand that receiving these messages, especially over a holiday, was upsetting and confusing, and we are truly sorry for the stress this caused. The issue has now been fully resolved, and our team has worked with our software provider to stop all queued messages and ensure this does not happen again.

If you have any questions or concerns, please feel free to email leasing@chamapts.com. Thank you for your patience and understanding.

All's well that ends well. Ellis thanked the software provider's "system malfunction," whoever or whatever it may've been, that had granted the rest of us a bit of holiday magic to take forward for all time.

It's a holiday in the US today, one where we give thanks. And today, we give thanks to not have this boss. Original. --Remy

Matt works at an accounting firm, as a data engineer. He makes reports for people who don’t read said reports. Accounting firms specialize in different areas of accountancy, and Matt’s firm is a general firm with mid-size clients.

The CEO of the firm is a legacy from the last century. The most advanced technology on his desk is a business calculator and a pencil sharpener. He still doesn’t use a cellphone. But he does have a son, who is “tech savvy”, which gives the CEO a horrible idea of how things work.

Usually, this is pretty light, in that it’s sorting Excel files or sorting the output of an existing report. Sometimes the requests are bizarre or utter nonsense. And, because the boss doesn’t know what the technical folks are doing, some of the IT staff may be a bit lazy about following best practices.

This means that most of Matt’s morning is spent doing what is essentially Tier 1 support before he gets into doing his real job. Recently, there was a worse crunch, as actual support person Lucinda was out for materinity leave, and Jackie, the one other developer, was off on vacation on a foreign island with no Internet. Matt was in the middle of eating a delicious lunch of take-out lo mein when his phone rang. He sighed when he saw the number.

“Matt!” the CEO exclaimed. “Matt! We need to do a build of the flagship app! And a deploy!”

The app was rather large, and a build could take upwards of 45 minutes, depending on the day and how the IT gods were feeling. But the process was automated, the latest changes all got built and deployed each night. Anything approved was released within 24 hours. With everyone out of the office, there hadn’t been any approved changes for a few weeks.

Matt checked the Github to see if something went wrong with the automated build. Everything was fine.

“Okay, so I’m seeing that everything built on GitHub and everything is available in production,” Matt said.

“I want you to do a manual build, like you used to.”

“If I were to compile right now, it could take quite awhile, and redeploying runs the risk of taking our clients offline, and nothing would be any different.”

“Yes, but I want a build that has the changes which Jackie was working on before she left for vacation.”

Matt checked the commit history, and sure enough, Jackie hadn’t committed any changes since two weeks before leaving on vacation. “It doesn’t looked like she pushed those changes to Github.”

“Githoob? I thought everything was automated. You told me the process was automated,” the CEO said.

“It’s kind of like…” Matt paused to think of an analogy that could explain this to a golden retriever. “Your dishwasher, you could put a timer on it to run it every night, but if you don’t load the dishwasher first, nothing gets cleaned.”

There was a long pause as the CEO failed to understand this. “I want Jackie’s front-page changes to be in the demo I’m about to do. This is for Initech, and there’s millions of dollars riding on their account.”

“Well,” Matt said, “Jackie hasn’t pushed- hasn’t loaded her metaphorical dishes into the dishwasher, so I can’t really build them.”

“I don’t understand, it’s on her computer. I thought these computers were on the cloud. Why am I spending all this money on clouds?”

“If Jackie doesn’t put it on the cloud, it’s not there. It’s uh… like a fax machine, and she hasn’t sent us the fax.”

“Can’t you get it off her laptop?”

“I think she took it home with her,” Matt said.

“So?”

“Have you ever seen Star Trek? Unless Scotty can teleport us to Jackie’s laptop, we can’t get at her files.”

The CEO locked up on that metaphor. “Can’t you just hack into it? I thought the NSA could do that.”

“No-” Matt paused. Maybe Matt could try and recreate the changes quickly? “How long before this meeting?” he asked.

“Twenty minutes.”

“Just to be clear, you want me to do a local build with files I don’t have by hacking them from a computer which may or may not be on and connected to the Internet, and then complete a build process which usually takes 45 minutes- at least- deploy to production, so you can do a demo in twenty minutes?”

“Why is that so difficult?” the CEO demanded.

“I can call Jackie, and if she answers, maybe we can figure something out.”

The CEO sighed. “Fine.”

Matt called Jackie. She didn’t answer. Matt left a voicemail and then went back to eating his now-cold lo mein.

One of the things that makes legacy code legacy is that code, over time, rots. Some of that rot comes from the gradual accumulation of fixes, hacks, and kruft. But much of the rot also comes from the tooling going unsupported or entirely out of support.

For example, many years ago, I worked in a Visual Basic 6 shop. The VB6 IDE went out of support in April, 2008, but we continued to use it well into the next decade. This made it challenging to support the existing software, as the IDE frequently broke in response to OS updates. Even when we started running it inside of a VM running an antique version of Windows 2000, we kept running into endless issues getting projects to compile and build.

A fun side effect of that: the VB6 runtime remains supported. So you can run VB6 software on modern Windows. You just can't modify that software.

Greta has inherited an even more antique tech stack. She writes, "I often wonder if I'm the last person on Earth encumbered with this particular stack." She adds, "The IDE is long-deprecated from a vendor that no longer exists- since 2002." Given the project started in the mid 2010s, it may have been a bad choice to use that tech-stack.

It's not as bad as it sounds- while the technology and tooling is crumbling ruins, the team culture is healthy and the C-suite has given Greta wide leeway to solve problems. But that doesn't mean that the tooling isn't a cause of anguish, and even worse than the tooling- the code itself.

"Some things," Greta writes, "are 'typical bad'" and some things "are 'delightfully unique' bad."

For example, the IDE has a concept of "designer" files, for the UI, and "code behind" files, for the logic powering the UI. The IDE frequently corrupts its own internal state, and loses the ability to properly update the designer files. When this happens, if you attempt to open, save, or close a designer file, the IDE pops up a modal dialog box complaining about the corruption, with a "Yes" and "No" option. If you click "No", the modal box goes away- and then reappears because you're seeing this message because you're on a broken designer file. If you click "Yes", the IDE "helpfully" deletes pretty much everything in your designer file.

Nothing about the error message indicates that this might happen.

The language used is a dialect of C++. I say "dialect" because the vendor-supplied compiler implements some cursed feature set between C++98 and C++11 standards, but doesn't fully conform to either. It's only capable of outputting 32-bit x86 code up to a Pentium Pro. Using certain C++ classes, like std::fstream, causes the resulting executable to throw a memory protection fault on exit.

Worse, the vendor supplied class library is C++ wrappers on top of an even more antique Pascal library. The "class" library is less an object-oriented wrapper and more a collection of macros and weird syntax hacks. No source for the Pascal library exists, so forget about ever updating that.

Because the last release of the IDE was circa 2002, running it on any vaguely modern environment is prone to failures, but it also doesn't play nicely inside of a VM. At this point, the IDE works for one session. If you exit it, reboot your computer, or try to close and re-open the project, it breaks. The only fix is to reinstall it. But the reinstall requires you to know which set of magic options actually lets the install proceed. If you make a mistake and accidentally install, say, CORBA support, attempting to open the project in the IDE leads to a cascade of modal error boxes, including one that simply says, "ABSTRACT ERROR" ("My favourite", writes Greta). And these errors don't limit themselves to the IDE; attempting to run the compiler directly also fails.

But, if anything, it's the code that makes the whole thing really challenging to work with. While the UI is made up of many forms, the "main" form is 18,000 lines of code, with absolutely no separation of concerns. Actually, the individual forms don't have a lot of separation of concerns; data is shared between forms via global variables declared in one master file, and then externed into other places. Even better, the various sub-forms are never destroyed, just hidden and shown, which means they remember their state whether you want that or not. And since much of the state is global, you have to be cautious about which parts of the state you reset.

Greta adds:

There are two files called main.cpp, a Station.cpp, and a Station1.cpp. If you were to guess which one owns the software's entry point, you would probably be wrong.

But, as stated, it's not all as bad as it sounds. Greta writes: "I'm genuinely happy to be here, which is perhaps odd given how terrible the software is." It's honestly not that odd; a good culture can go a long way to making wrangling a difficult tech stack happy work.

Finally, Greta has this to say:

We are actively working on a .NET replacement. A nostalgic, perhaps masochistic part of me will miss the old stack and its daily delights.

Our submitter, Gearhead, was embarking on STEM-related research. This required him to pursue funding from a governmental agency that we’ll call the Ministry of Silly Walks. In order to start a grant application and track its status, Gearhead had to create an account on the Ministry website.

The registration page asked for a lot of personal information first. Then Gearhead had to create his own username and password. He used his password generator to create a random string: D\h.|wAi=&:;^t9ZyoO

Upon clicking Save, he received an error.

Your password must be a minimum eight characters long, with no spaces. It must include at least three of the following character types: uppercase letter, lowercase letter, number, special character (e.g., !, $, % , ?).

Perplexed, Gearhead emailed the Ministry’s web support, asking why his registration failed. The reply:

Hello,
The site rejects password generators as hacking attempts. You will need to manually select a password.
Ex. GHott*01

Thank you,

Support

So a long sequence of random characters was an active threat, but a 1990s-era AOL username was just fine. What developer had this insane idea and convinced other people of it? How on earth did they determine what was a "manually selected" string versus a randomly-generated one?

It seems the deciding factor is nothing more than length. If you go to the Ministry’s registration page now, their password guidelines have changed (emphasis theirs):

Must be 8-10 characters long, must contain at least one special character ( ! @ # $ % ^ & * ( ) + = { } | < > \ _ - [ ] / ? ) and no spaces, may contain numbers (0-9), lower and upper case letters (a-z, A-Z). Please note that your password is case sensitive.

Only good can come of forcing tiny passwords.

The more a company or government needs secure practices, the less good they are at secure practices. Is that a law yet? It should be.

Dotan was digging through vendor supplied documentation to understand how to use an API. To his delight, he found a specific function which solved exactly the problem he had, complete with examples of how it was to be used. Fantastic!

He copied one of the examples, and hit compile, and reviewed the list of errors. Mostly, the errors were around "the function you're calling doesn't exist". He went back to the documentation, checked it, went back to the code, didn't find any mistakes, and scratched his head.

Now, it's worth noting the route Dotan took to find the function. He navigated there from a different documentation page, which sent him to an anchor in the middle of a larger documentation page- vendorsite.com/docs/product/specific-api#specific-function.

This meant that as the page loaded, his browser scrolled directly down to the specific-function section of the page. Thus, Dotan missed the gigantic banner at the top of the page for that API, which said this:

/!\ NOTE /!\ NOTE /!\ NOTE /!\ NOTE /!\ NOTE /!\ NOTE /!\ NOTE /!
This doc was written to help flesh out a user API. The features described here are all hypothetical and do not actually exist yet, don't assume anything you see on this page works in any version /!\ NOTE /!\ NOTE /!\ NOTE /!\ NOTE /!\ NOTE /!\ NOTE /!\ NOTE /!\

On one hand, I think providing this kind of documentation is invaluable, both to your end users and for your own development team. It's a great roadmap, a "documentation driven development" process. And I can see that they made an attempt to be extremely clear about it being incomplete and unimplemented- but they didn't think about how people actually used their documentation site. A banner at the top of the page only works if you read the page from top to bottom, but documentation pages you will frequently skip to specific sections of the page.

But there was a deeper issue with the way this particular approach was executed: while the page announced that one shouldn't assume anything works, many of the functions on the page did work. Many did not. There was no rhyme or reason, to version information or other indicators to help a developer understand what was and was not actually implemented.

So while the idea of a documentation-oriented roadmap specifying features that are coming is good, the execution here verged into WTF territory. It was a roadmap, but with all the landmarks erased, so you had no idea where you actually were along the length of that road. And the one warning sign that would help you was hidden behind a bush.

Dotan asks: "WTF is that page doing on the official documentation wiki?"

And I'd say, I understand why it's there, but boy it should have been more clear about what it actually was.

[Advertisement] Keep the plebs out of prod. Restrict NuGet feed privileges with ProGet. Learn more.

Years ago, Brian had a problem: their C# application would crash sometimes. What was difficult to understand was why it was crashing, because it wouldn't crash in response to a user action, or really, any easily observable action.

The basic flow was that the users used a desktop application. Many operations that the users wanted to perform were time consuming, so the application spun up background tasks to do them, thus allowing the user to do other things within the application. And sometimes, the application would just crash, both when the user hadn't done anything, and when all background jobs should have been completed.

The way the background task was launched was this:

seeder.RunSeeder();

It didn't take too much head scratching to realize what was running every time the application crashed: the garbage collector.

RunSeeder returned a Task object, but since Brian's application treated the task as "fire and forget", they didn't worry about the value itself. But C# did- the garbage collector had to clean up that memory.

And this was running under .Net 4.0. This particular version of the .Net framework was a special, quantum realm, at least when it came to tasks. You see, if a Task raises an exception, nothing happens. At least, not right away. No one is notified of the exception unless they inspect the Task object directly. There's a cat in the box, and no one knows the state of the cat unless they open the box.

The application wasn't checking the Task result. The cat remained in a superposition of "exception" and "no exception". But the garbage collector looked at the task. And, in .Net 4.0, Microsoft made a choice about what to do there: when they opened the box and saw an exception (instead of a cat), they chose to crash.

Microsoft's logic here wasn't entirely bad; an uncaught exception means something has gone wrong and hasn't been handled. There's no way to be certain the application is in a safe state to continue. Treating it akin to undefined behavior and letting the application crash was a pretty sane choice.

The fix for Brian's team was simple: observe the exception, and choose not to do anything with it. They truly didn't care- these tasks were fire-and-forget, and failure was acceptable.

seeder.RunSeeder().ContinueWith(t => { var e = t.IsFaulted ? t.Exception : null; }); // Observe exceptions to prevent quantum crashes

This code merely opens the box and sees if there's an exception in there. It does nothing with it.

Now, I'd say as a matter of programming practice, Microsoft was right here. Ignoring exceptions blindly is a definite code smell, even for a fire-and-forget task. Writing the tasks in such a way as they catch and handle any exceptions that bubble up is better, as is checking the results.

But I, and Microsoft, were clearly on the outside in this argument. Starting with .Net 4.5 and moving forward, uncaught exceptions in background tasks were no longer considered show-stoppers. Whether there was a cat or an exception in the box, when the garbage collector observed it, it got thrown away either way.

In the end, this reminds me of my own failing using background tasks in .Net.

[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!

Everyone's got workplace woes. The clueless manager; the disruptive coworker; the cube walls that loom ever higher as the years pass, trapping whatever's left of your soul.

But sometimes, Satan really leaves his mark on a joint. I worked Tech Support there. You may remember The C-Level Ticket. I'm Anonymous. This is my story.

Night after night, my dreams are full of me trying and failing at absolutely everything. Catch a bus? I'm already running late and won't make it. Dial a phone number to get help? I can't recall the memorized sequence, and the keypad's busted anyway. Drive outta danger? The car won't start. Run from a threat? My legs are frozen.

Then I wake up in my bed in total darkness, scared out of my skull, and I can't move for real. Not one muscle works. Even if I could move, I'd stay still because I'm convinced the smallest twitch will give me away to the monster lurking nearby, looking to do me in.

The alarm nags me before the sun's even seen fit to show itself. What day is it? Tuesday? An invisible, overwhelming dread pins me in place under the covers. I can't do it. Not again.

The thing is, hunger, thirst, and cold are even more nagging than the alarm. Dead tired, I force myself up anyway to do the whole thing over.

The office joe that morning was so over-brewed as to be sour. I tossed down the last swig in my mug, checking my computer one more time to make sure no Tech Support fires were raging by instant message or email. Then I threw on my coat and hat and quit my cube, taking the stairs to ground level.

I pushed open a heavy fire-escape door and stepped out into the narrow alley between two massive office buildings. Brisk autumn air and the din of urban motor traffic rushed to greet me. The dull gray sky above threatened rain. Leaning against the far brick wall were Toby and Reynaldo, a couple of network admins, hugging themselves as they nursed smoldering cigarettes. They nodded hello.

I tipped my hat in greeting, slipping toward the usual spot, a patch of asphalt I'd all but worn grooves in by that point. I lit my own cigarette and took in a deep, warming draw.

"Make it last another year," Toby spoke in a mocking tone, tapping ash onto the pavement. "I swear, that jerk can squeeze a nickel until Jefferson poops!"

An ambulance siren blared through the alley for a minute. The rig was no doubt racing toward the hospital down the street.

Reynaldo smirked. "You think Morty finally did it?"

Toby smirked as well.

I raised an eyebrow. "Did what?"

"Morty always says he's gonna run out into traffic one of these days so they can take him to the hospital and he won't have to be here," Reynaldo explained.

I frowned at the morbid suggestion. "Hell of a way to catch a break."

"Well, it's not like we can ask for time off," Toby replied bitterly. "They always find some way to rope us back in."

I nodded in sympathy. "You have it worse than we do. But my sleep's still been jacked plenty of times by 3AM escalated nonsense that shoulda been handled by a different part of the globe."

Reynaldo's eyes lit up fiercely. "They have all the same access and training, but it never falls on them! Yeah, been there."

The door swung open again, admitting a young woman with the weight of the world on her shoulders. This was Megan, a junior developer and recent hire. I tipped my hat while helping myself to another drag.

She hastened my way, pulling a pack of cigarettes from her handbag. With shaking hands, she fumbled to select a single coffin nail. "I quit these things!" she lamented. After returning the pack to her bag, she rummaged through it fruitlessly. "Dammit, where are those matches?!" She glanced up at me with a pleading expression.

I pulled the lighter from my coat pocket. "You sure?"

She nodded like she hadn't been more sure about anything in her entire life.

I lit it for her. She took a lung-filling pull, then exhaled a huge cloud of smoke.

"Goin' that well, huh?" I asked.

Megan also hugged herself, her expression pained. "Every major player in the industry uses our platform, and I have no idea how it hasn't all come crashing down. There are thousands of bugs in the code base. Thousands! It breaks all the time. Most of the senior devs have no clue what they're doing. And now we're about to lose the only guy who understands the scheduling algorithm, the most important thing!"

"That's tough." I had no idea what else to say. Maybe it was enough that I listened.

Megan glanced up nervously at the brewing storm overhead. "I just know that algorithm's gonna get dumped in my lap."

"The curse of competence." I'd seen it plenty of times.

"Ain't that the truth!" She focused on me again with a look of apology. "How've you been?"

I shrugged. "Same old, same old." I figured a fresh war story might help. "Had to image and set up the tech for this new manager's onboarding. Her face is stuck in this permanent glare. Every time she opens her mouth, it's to bawl someone out."

"Ugh."

"The crazy thing is, the walls of her office are completely covered with crucifixes, and all these posters plastered with flowers and hearts and sap like Choose Kindness." I leaned in and lowered my voice. "You know what I think? I think she’s an ancient Roman whose spite has kept her alive for over two thousand years. Those crosses are a threat!"

That teased a small laugh out of Megan. For a moment, the amusement reached her eyes. Then it was gone, overwhelmed by worry. She took to pacing through the narrow alley.

Back at my cube, I found a new urgent ticket at the top of my case load. Patricia Dracora, a senior project manager, had put in a call claiming her computer had been hacked. Her mouse cursor was moving around and clicking things all on its own.

It was too early in the morning for a case like this. That old dread began sneaking up on me again. The name put me on edge as well. Over the years, our paths had never crossed, but her nickname throughout Tech Support, Dracula, betrayed what everyone else made of her.

"Make like a leaf and blow!"

The boss barked his stern command over my shoulder. I stood and turned from my computer to find him at my cubicle threshold with arms folded, blocking my egress.

I couldn't blow, so I shrugged. "Can't be as bad as The Crucifier."

"Dracula's worse than The Crucifier," the boss replied under his breath in a warning tone. "For your own good, don't keep her waiting!" He tossed a thumb over his shoulder for good measure.

When he finally backed out of the way, I made tracks outta there. A few of my peers made eye contact as I passed, looking wary on my behalf.

The ticket pegged Dracora's office in a subfloor I'd never set foot in before. Descending the stairs, I had too much time to think. Of course I didn't expect a real hacking attempt. Peripheral hardware on the fritz, some software glitch: there'd be a simple explanation. What fresh hell would I have to endure to reach that point? That was what my tired brain couldn't let go of. The stimulants hadn't kicked in yet. With the strength of a kitten, I was stepping into a lion's den. A lion who might make me wish for crucifixion by the time it was all over.

From the stairwell, I entered a dank, deserted corridor. Old florescent lighting fixtures hummed and flickered overhead. That, combined with the overwhelming stench of paint fumes, set the stage for a ripping headache. There were no numbers on the walls to lead me to the right place. They must've taken them down to paint and never replaced them. I inched down worn, stained carpeting, peeking into each open gap I found to either side of me. Nothing but darkness, dust, and cobwebs at first. Eventually, I spotted light blaring from one of the open doors ahead of me. I jogged the rest of the way, eager to see any living being by that point.

The room I'd stumbled onto was almost closet-sized. It contained a desk and chair, a laptop docking station, and a stack of cardboard boxes on the floor. Behind the desk was a woman of short stature, a large purse slung over one shoulder. Her arms were folded as she paced back and forth in the space behind her chair. When I appeared, she stopped and looked to me wide-eyed, maybe just as relieved as I was. "Are you Tech Support?"

"Yes, ma'am." I entered the room. "What's—?"

"I don't know how it happened!" Dracora returned to pacing, both hands making tight fists around the straps of the purse she was apparently too wired and distracted to set down. "They made me move here from the fourth floor. I just brought everything down and set up my computer, and now someone has control of the mouse. Look, look!" She stopped and pointed at the monitor.

I rounded the desk. By the time I got there, whatever she'd seen had vanished. Onscreen, the mouse cursor sat still against a backdrop of open browsers and folders. Nothing unusual.

"It was moving, I swear!" Anguished, Dracora pleaded with me to believe her.

It seemed like she wasn't hostile at all, just stressed out and scared. I could handle that. "I'm sure we can figure this out, ma'am. Lemme have a look here."

I sat down at the desk and tried the wireless mouse first. It didn't work at all to move the cursor.

"The hacker's locked us out!" Dracora returned to pacing behind me.

As I sat there, not touching a thing, the mouse cursor shuttled across the screen like it was possessed.

"There! You see?"

Suddenly, somehow, my brain smashed everything together. "Ma'am, I have an idea. Could you please stand still?"

Dracora stopped.

I swiveled around in the chair to face her. "Ma'am, you said you were moving in down here. What's in your purse right now?"

Her visible confusion deepened. "What?"

"The mouse cursor only moves around when you do," I explained.

Her eyes widened. She dug deeply into her purse. A moment later, she pulled out a second wireless mouse. Then she looked to me like she couldn't believe it. "That's it?!"

"That's it!" I replied.

"Oh, lord!" Dracora replaced the dud sitting on her mousepad with the mouse that was actually connected to her machine, wilting over the desk as she did so. "I don't know whether to laugh or cry."

I knew the feeling. But the moment of triumph, I gotta admit, felt pretty swell. "Anything else I can help with, ma'am?"

"No, no! I've wasted enough of your time. Thank you so much!"

I had even more questions on the way back upstairs. With this huge, spacious office building, who was forcing Dracora to be in that pit? How had she garnered such a threatening reputation? Why had my experience been so different from everyone else's? I didn't mention it to the boss or my peers. I broke it all down to Megan in the alley a few days later.

"She even put in a good word for me when she closed the ticket," I told her. "The boss says I'm on the fast track for another promotion." I took a drag from my cigarette, full of bemusement. "I'm already as senior as it gets. The only way up from here is management." I shook my head. "That ain't my thing. Look how well it's gone for Dracora."

Megan lowered her gaze, eyes narrowed. "You said it yourself: the only reward for good work is more work."

And then they buried you ... in a basement, or a box.

I remembered being at the start of my career, like Megan. I remembered feeling horrified by all the decades standing between me and the day when I wouldn't or couldn't ever work again. A couple decades in, some part of me that I'd repressed had resurfaced. What the hell is this? What have I been doing?

Stop caring, a different part replied. Just stop caring. Take things day by day, case by case.

I'd obeyed for so long. Where had it gotten me?

Under my breath, I risked airing my wildest wish for the future. "Someday, I wanna break outta this joint."

Megan blinked up at me. I had her attention. "How?"

"I dunno," I admitted. "I gotta figure it out ... before I go nuts."

[Advertisement] Utilize BuildMaster to release your software with confidence, at the pace your business demands. Download today!

Back in the antediluvian times, when I was in college, people still used floppy disks to work on their papers. This was a pretty untenable arrangement, because floppy disks lost data all the time, and few students had the wherewithal to make multiple copies. Half my time spent working helldesk was breaking out Norton Diskutils to try and rescue people's term papers. To avoid this, the IT department offered network shares where students could store documents. The network share was backed up, tracked versions, and could be accessed from any computer on campus, including the VAX system (in fact, it was stored on the VAX).

I bring this up because we have known for quite some time that companies and governments need to store documents in centrally accessible locations so that you're not reliant on end users correctly managing their files. And if you are a national government, you have to make a choice: either you contract out to a private sector company, or you do it yourself.

South Korea made the choice to do it themselves, with their G-Drive system (short for Government Drive, no relation to Google Drive), a government file store hosted primarily out of a datacenter in Daejeon. Unfortunately, "primarily" is a bit too apropos- last month, a fire in that datacenter destroyed data.

The Interior Ministry explained that while most systems at the Daejeon data center are backed up daily to separate equipment within the same center and to a physically remote backup facility, the G-Drive’s structure did not allow for external backups. This vulnerability ultimately left it unprotected.

Someone, somehow, designed a data storage system that was structurally incapable of doing backups? And then told 750,000 government employees that they should put all their files there?

Even outside of that backup failure, while other services had backups, they did not have a failover site, so when the datacenter went down, the government went down with it.

In total, it looks like about 858TB of data got torched. 647 distinct services were knocked out. At least 90 of them were reported to be unrecoverable (that last link is from a company selling Lithium Ion safety products, but is a good recap). A full recovery was, shortly after the accident, predicted to take a month, but as of October 22, only 60% of services had been restored.

Now, any kind of failure of this scale means heads must roll, and police investigations have gone down the path of illegal subcontracting. The claim is that the government hired a contractor broke the law by subcontracting the work, and that those subcontractors were unqualified for the work they were doing- that while they were qualified to install or remove a li-ion battery, they were not qualified to move one, which is what they were doing and resulted in the fire.

I know too little about Korean laws about government contracting and too little about li-ion battery management to weigh in on this. Certainly, high-storage batteries are basically bombs, and need to be handled with great care and protected well. Though, if one knows how to install and uninstall a battery, moving a battery seems covered in those steps.

But if I were doing a root cause analysis here, while that could be the root cause of the fire, it is not the root cause of the outage. If you build a giant datacenter but can't replicate services to another location, you haven't built a reliable cloud storage system- you've just built an expensive floppy disk that is one trip too close to a fridge magnet away from losing all of your work. In this case, the fridge magnet was made of fire, but the result is the same.

I'm not going to say this problem would have be easy to avoid; actually building resilient infrastructure that fails gracefully under extreme stress is hard. But while it's a hard problem, it's also a well-understood problem. There are best practices, and clearly not one of them was followed.

[Advertisement] Keep all your packages and Docker containers in one place, scan for vulnerabilities, and control who can access different feeds. ProGet installs in minutes and has a powerful free version with a lot of great features that you can upgrade when ready.Learn more.

Chops was a developer for Initrode. Early on a Monday, they were summoned to their manager Gary's office before the caffeine had even hit their brain.

Gary glowered up from his office chair as Chops entered. This wasn't looking good. "We need to talk about the latest commit for Taskmaster."

Taskmaster was a large application that'd been around for decades, far longer than Chops had been an employee. Thousands of internal and external customers relied upon it. Refinements over time had led to remarkable stability, its typical uptime now measured in years. However, just last week, their local installation had unexpectedly suffered a significant crash. Chops had been assigned to troubleshooting and repair.

"What's wrong?" Chops asked.

"Your latest commit decreased the number of unit tests!" Gary replied as if Chops had slashed the tires on his BMW.

Within Taskmaster, some objects that were periodically generated were given a unique ID from a pool. The pool was of limited size and required scanning to find a spare ID. Each time a value was needed, a search began where the last search ended. IDs returned to the pool as objects were destroyed would only be reused when the search wrapped back around to the start.

Chops had discovered a bug in the wrap-around logic that would inevitably produce a crash if Taskmaster ran long enough. They also found that if the number of objects created exceeded the size of the pool, this would trigger an infinite loop.

Rather than attempt to patch any of this, Chops had nuked the whole thing and replaced it with code that assigned each object a universally unique identifier (UUID) from a trusted library UUID generator within its constructor. Gone was the bad code, along with its associated unit tests.

Knowing they would probably only get in a handful of words, Chops wonderered how on earth to explain all this in a way that would appease their manager. "Well—"

"That number must NEVER go down!" Gary snapped.

"But—"

"This is non-negotiable! Roll it back and come up with something better!"

And so Chops had no choice but to remove their solution, put all the janky code back in place, and patch over it with kludge. Every comment left to future engineers contained a tone of apology.

Taskmaster became less stable. Time and expensive developer hours were wasted. Risk to internal and external customers increased. But Gary could rest assured, knowing that his favored metric never faltered on his watch.

[Advertisement] Keep all your packages and Docker containers in one place, scan for vulnerabilities, and control who can access different feeds. ProGet installs in minutes and has a powerful free version with a lot of great features that you can upgrade when ready.Learn more.

Ellis knew she needed a walk after she hurried off of Zoom at the end of the meeting to avoid sobbing in front of the group.

She'd just been attending a free online seminar regarding safe job hunting on the Internet. Having been searching since the end of January, Ellis had already picked up plenty of first-hand experience with the modern job market, one rejection at a time. She thought she'd attend the seminar just to see if there were any additional things she wasn't aware of. The seminar had gone well, good information presented in a clear and engaging way. But by the end of it, Ellis was feeling bleak. Goodness gracious, she'd already been slogging through months of this. Hundreds of job applications with nothing to show for it. All of the scams out there, all of the bad actors preying on people desperate for their and their loved ones' survival!

Ellis' childhood had been plagued with anxiety and depression. It was only as an adult that she'd learned any tricks for coping with them. These tricks had helped her avoid spiraling into full-on depression for the past several years. One such trick was to stop and notice whenever those first feelings hit. Recognize them, feel them, and then respond constructively.

First, a walk. Going out where there were trees and sunshine: Ellis considered this "garbage collection" for her brain. So she stepped out the front door and started down a tree-lined path near her house, holding on to that bleak feeling. She was well aware that if she didn't address it, it would take root and grow into hopelessness, self-loathing, fear of the future. It would paralyze her, leave her curled up on the couch doing nothing. And it would all happen without any words issuing from her inner voice. That was the most insidious thing. It happened way down deep in a place where there were no words at all.

Once she returned home, Ellis forced herself to sit down with a notebook and pencil and think very hard about what was bothering her. She wrote down each sentiment:

This was the first step of a reframing technique she'd just read about in the book Right Kind of Wrong by Amy Edmonson. With the words out, it was possible to look at each statement and determine whether it was rational or irrational, constructive or harmful. Each statement could be replaced with something better.

Ellis proceeded step by step through the list.

• Yes, this will end. Everything ends.
• There's nothing wrong with me. Most businesses are swamped with applications. There's a good chance mine aren't even being looked at before they're being auto-rejected. Remember the growth mindset you learned from Carol Dweck. Each application and interview is giving me experience and making me a better candidate.
• This job market is a novel context that changes every day. That means failure is not only inevitable, it's the only way forward.

Ellis realized that her job hunt was very much like a search algorithm trying to find a path through a maze. When the algorithm encountered a dead end, did it deserve blame? Was it an occasion for shame, embarrassment, and despair? Of course not. Simply backtrack and keep going with the knowledge gained.

Yes, there was truth to the fact that this was the toughest job market Ellis had ever experienced. Therefore, taking a note from Viktor Frankl, she spent a moment reimagining the struggle in a way that made it meaningful to her. Ellis began viewing her job hunt in this dangerous market, her gradual accumulation of survival information, as an act of resistance against it. She now hoped to write all about her experience once she was on the other side, in case her advice might help even one other person in her situation save time and frustration.

While unemployed, she also had the opportunity to employ the search algorithm against entirely new mazes. Could Ellis expand her freelance writing into a sustainable gig, for instance? That would mean exploring all the different ways to be a freelance writer, something Ellis was now curious and excited to explore.

[Advertisement] Keep all your packages and Docker containers in one place, scan for vulnerabilities, and control who can access different feeds. ProGet installs in minutes and has a powerful free version with a lot of great features that you can upgrade when ready.Learn more.

Everyone's got workplace woes. The clueless manager; the disruptive coworker; the cube walls that loom ever higher as the years pass, trapping whatever's left of your soul.

But sometimes, Satan really leaves his mark on a joint. I worked Tech Support there. This is my story. Who am I? Just call me Anonymous.

It starts at the top. A call came in from Lawrence Gibbs, the CEO himself, telling us that a conference room printer was, quote, "leaking." He didn't explain it, he just hung up. The boss ordered me out immediately, told me to step on it. I ignored the elevator, racing up the staircase floor after floor until I reached the dizzying summit of C-Town.

There's less oxygen up there, I'm sure of it. My lungs ached and my head spun as I struggled to catch my breath. The fancy tile and high ceilings made a workaday schmuck like me feel daunted, unwelcome. All the same, I gathered myself and pushed on, if only to learn what on earth "leaking" meant in relation to a printer.

I followed the signs on the wall to the specified conference room. In there, the thermostat had been kicked down into the negatives. The cold cut through every layer of mandated business attire, straight to bone. The scene was thick with milling bystanders who hugged themselves and traded the occasional nervous glance. Gibbs was nowhere to be found.

Remembering my duty, I summoned my nerve. "Tech Support. Where's the printer?" I asked.

Several pointing fingers showed me the way. The large printer/scanner was situated against the far wall, flanking an even more enormous conference table. Upon rounding the table, I was greeted with a grim sight: dozens of sheets of paper strewn about the floor like blood spatter. Everyone was keeping their distance; no one paid me any mind as I knelt to gather the pages. There were 30 in all. Each one was blank on one side, and sported some kind of large, blotchy ring on the other. Lord knew I drank enough java to recognize a coffee mug stain when I saw one, but these weren't actual stains. They were printouts of stains.

The printer was plugged in. No sign of foul play. As I knelt there, unseen and unheeded, I clutched the ruined papers to my chest. Someone had wasted a tree and a good bit of toner, and for what? How'd it go down? Surely Gibbs knew more than he'd let on. The thought of seeking him out, demanding answers, set my heart to pounding. It was no good, I knew. He'd play coy all day and hand me my pink slip if I pushed too hard. As much as I wanted the truth, I had a stack of unpaid bills at home almost as thick as the one in my arms. I had to come up with something else.

There had to be witnesses among the bystanders. I stood up and glanced among them, seeking out any who would return eye contact. There: a woman who looked every bit as polished as everyone else. But for once, I got the feeling that what lay beneath the facade wasn't rotten.

With my eyes, I pleaded for answers.

Not here, her gaze pleaded back.

I was getting somewhere, I just had to arrange for some privacy. I hurried around the table again and weaved through bystanders toward the exit, hoping to beat it out of that icebox unnoticed. When I reached the threshold, I spotted Gibbs charging up the corridor, smoldering with entitlement. "Where the hell is Tech Support?!"

I froze a good distance away from the oncoming executive, whose voice I recognized from a thousand corporate presentations. Instead of putting me to sleep this time, it jolted down my spine like lightning. I had to think fast, or I was gonna lose my lead, if not my life.

"I'm right here, sir!" I said. "Be right back! I, uh, just need to find a folder for these papers."

"I've got one in my office."

A woman's voice issued calmly only a few feet behind me. I spun around, and it was her, all right, her demeanor as cool as our surroundings. She nodded my way. "Follow me."

My spirits soared. At that moment, I would've followed her into hell. Turning around, I had the pleasure of seeing Gibbs stop short with a glare of contempt. Then he waved us out of his sight.

Once we were out in the corridor, she took the lead, guiding me through the halls as I marveled at my luck. Eventually, she used her key card on one of the massive oak doors, and in we went.

You could've fit my entire apartment into that office. The place was spotless. Mini-fridge, espresso machine, even couches: none of it looked used. There were a couple of cardboard boxes piled up near her desk, which sat in front of a massive floor-to-ceiling window admitting ample sunlight.

She motioned toward one of the couches, inviting me to sit. I shook my head in reply. I was dying for a cigarette by that point, but I didn't dare light up within this sanctuary. Not sure what to expect next, I played it cautious, hovering close to the exit. "Thanks for the help back there, ma'am."

"Don't mention it." She walked back to her desk, opened up a drawer, and pulled out a brand-new manila folder. Then she returned to conversational distance and proffered it my way. "You're from Tech Support?"

There was pure curiosity in her voice, no disparagement, which was encouraging. I accepted the folder and stuffed the ruined pages inside. "That's right, ma'am."

She shook her head. "Please call me Leila. I started a few weeks ago. I'm the new head of HR."

Human Resources. That acronym, which usually put me on edge, somehow failed to raise my hackles. I'd have to keep vigilant, of course, but so far she seemed surprisingly OK. "Welcome aboard, Leila. I wish we were meeting in better circumstances." Duty beckoned. I hefted the folder. "Printers don't just leak."

"No." Leila glanced askance, grave.

"Tell me what you saw."

"Well ..." She shrugged helplessly. "Whenever Mr. Gibbs gets excited during a meeting, he tends to lean against the printer and rest his coffee mug on top of it. Today, he must've hit the Scan button with his elbow. I saw the scanner go off. It was so bright ..." She trailed off with a pained glance downward.

"I know this is hard," I told her when the silence stretched too long. "Please, continue."

Leila summoned her mettle. "After he leaned on the controls, those pages spilled out of the printer. And then ... then somehow, I have no idea, I swear! Somehow, all those pages were also emailed to me, Mr. Gibbs' assistant, and the entire board of directors!"

The shock hit me first. My eyes went wide and my jaw fell. But then I reminded myself, I'd seen just as crazy and worse as the result of a cat jumping on a keyboard. A feline doesn't know any better. A top-level executive, on the other hand, should know better.

"Sounds to me like the printer's just fine," I spoke with conviction. "What we have here is a CEO who thinks it's OK to treat an expensive piece of office equipment like his own personal fainting couch."

"It's terrible!" Leila's gaze burned with purpose. "I promise, I'll do everything I possibly can to make sure something like this never happens again!"

I smiled a gallows smile. "Not sure what anyone can do to fix this joint, but the offer's appreciated. Thanks again for your help."

Now that I'd seen this glimpse of better things, I selfishly wanted to linger. But it was high time I got outta there. I didn't wanna make her late for some meeting or waste her time. I backed up toward the door on feet that were reluctant to move.

Leila watched me with a look of concern. "Mr. Gibbs was the one who called Tech Support. I can't close your ticket for you; you'll have to get him to do it. What are you going to do?"

She cared. That made leaving even harder. "I dunno yet. I'll think of something."

I turned around, opened the massive door, and put myself on the other side of it in a hurry, using wall signs to backtrack to the conference room. Would our paths ever cross again? Unlikely. Someone like her was sure to get fired, or quit out of frustration, or get corrupted over time.

It was too painful to think about, so I forced myself to focus on the folder of wasted pages in my arms instead. It felt like a mile-long rap sheet. I was dealing with an alleged leader who went so far as to blame the material world around him rather than accept personal responsibility. I'd have to appeal to one or more of the things he actually cared about: himself, his bottom line, his sense of power.

By the time I returned to the conference room to face the CEO, I knew what to tell him. "You're right, sir, there's something very wrong with this printer. We're gonna take it out here and give it a thorough work-up."

That was how I was able to get the printer out of that conference room for good. Once it underwent "inspection" and "testing," it received a new home in a previously unused closet. Whenever Gibbs got to jawing in future meetings, all he could do was lean against the wall. Ticket closed.

Gibbs remained at the top, doing accursed things that trickled down to the roots of his accursed company. But at least from then on, every onboarding slideshow included a photo of one of the coffee ring printouts, with the title Respect the Equipment.

Thanks, Leila. I can live with that.

Once upon a time, when the Web was young, if you wanted to be a cool kid, you absolutely needed two things on your website: a guestbook for people to sign, and a hit counter showing how many people had visited your Geocities page hosting your Star Trek fan fiction.

These days, we don't see them as often, but companies still like to track the information, especially when it comes to counting downloads. So when Justin started on a new team and saw a download count in their analytics, he didn't think much of it at all. Nor did he think much about it when he saw the download count displayed on the download page.

Another thing that Justin didn't think much about was big piles of commits getting merged in overnight, at least not at first. But each morning, Justin needed to pull in a long litany of changes from a user named "MrStinky". For the first few weeks, Justin was too preoccupied with getting his feet under him, so he didn't think about it too much.

But eventually, he couldn't ignore what he saw in the git logs.

docs: update download count to 51741
docs: update download count to 51740
docs: update download count to 51738

And each commit was exactly what the name implied, a diff like:

- 51740
+ 51741

Each time a user clicked the download link, a ping was sent to their analytics system. Throughout the day, the bot "MrStinky" would query the analytics tool, and create new commits that updated the counter. Overnight, it would bundle those commits into a merge request, approve the request, merge the changes, and then redeploy what was at the tip of main.

"But, WHY?" Justin asked his peers.

One of them just shrugged. "It seemed like the easiest and fastest way at the time?"

"I wanted to wire Mr Stinky up to our content management system's database, but just never got around to it. And this works fine," said another.

Much like the rest of the team, Justin found that there were bigger issues to tackle.

Adam's organization was going through a period of rapid growth. Part of this growth was spinning up new backend services to support new functionality. The growth would have been extremely fast, except for one thing applying back pressure: for some reason, spinning up a new service meant recompiling and redeploying all the other services.

Adam didn't understand why, but it seemed like an obvious place to start poking at something for improvement. All of the services depended on a library called "ServiceLib"- though not all of them actually used the library. The library was a set of utilities for administering, detecting, and interacting with services in their environment- essentially a homegrown fabric/bus architecture.

It didn't take long, looking at the source control history, to understand why there was a rebuild after the release of every service. Each service triggered a one line change in this:

enum class Services
{
    IniTechBase = 103,
    IniTechAdvanced = 99,
    IniTechFooServer = 102,
    …
}

Each service had a unique, numerical identifier, and this mapped them into an enumerated type.

Adam went to the tech lead, Raymond. "Hey, I've got an idea for speeding up our release process- we should stop hard coding the service IDs in ServiceLib."

Raymond looked at Adam like one might examine an over-enthusiastic lemur. "They're not hard-coded. We store them in an enum."

Eventually Raymond got promoted- for all of their heroic work on managing this rapidly expanding library of services. The new tech lead who came on was much more amenable to "not storing rapidly changing service IDs in an enum", and "not making every service depend on a library they often don't need", and "putting admin functionality in every service because they're linked to that library whether they like it or not."

Eventually, ServiceLib became its own service, and actually helped- instead of hindered- delivering new functionality.

Unfortunately, with no more highly visible heroics to deliver functionality, the entire department became a career dead end. Sure, they delivered on time and under budget consistently, but there were no rockstar developers like Raymond on the team anymore, the real up-and-comers who were pushing themselves.

Once upon a time, there was a bank whose business relied on a mainframe. As the decades passed and the 21st century dawned, the bank's bigwigs realized they had to upgrade their frontline systems to applications built in Java and .NET, but—for myriad reasons that boiled down to cost, fear, and stubbornness—they didn't want to migrate away from the mainframe entirely. They also didn't want the new frontline systems to talk directly to the mainframe or vice-versa. So they tasked old-timer Edgar with writing some middleware. Edgar's brainchild was a Windows service that took care of receiving frontline requests, passing them to the mainframe, and sending the responses back.

Edgar's middleware worked well, so well that it was largely forgotten about. It outlasted Edgar himself, who, after another solid decade of service, moved on to another company.

A few years later, our submitter John F. joined the bank's C# team. By this point, the poor middleware seemed to be showing its age. A strange problem had arisen: between 8:00AM and 5:00PM, every 45 minutes or so, it would lock up and have to be restarted. Outside of those hours, there was no issue. The problem was mitigated by automatic restarts, but it continued to inflict pain and aggravation upon internal users and external customers. A true solution had to be found.

Unfortunately, Edgar was long gone. The new "owner" of the middleware was an infrastructure team containing zero developers. Had Edgar left them any documentation? No. Source code? Sort of. Edgar had given a copy of the code to his friend Bob prior to leaving. Unfortunately, Bob's copy was a few point releases behind the version of middleware running in production. It was also in C, and there were no C developers to be found anywhere in the company.

And so, the bank's bigwigs cobbled together a diverse team of experts. There were operating system people, network people, and software people ... including the new guy, John. Poor John had the unenviable task of sifting through Edgar's source code. Just as the C# key sits right next to the C key on a piano, reasoned the bigwigs, C# couldn't be that different from C.

John toiled in an unfamiliar language with no build server or test environment to aid him. It should be no great surprise that he got nowhere. A senior coworker suggested that he check what Windows' Process Monitor registered when the middleware was running. John allowed a full day to pass, then looked at the results: it was now clear that the middleware was constantly creating and destroying threads. John wrote a Python script to analyze the threads, and found that most of them lived for only seconds. However, every 5 minutes, a thread was created but never destroyed.

This only happened during the hours of 8:00AM to 5:00PM.

At the next cross-functional team meeting behind closed doors, John finally had something of substance to report to the large group seated around the conference room table. There was still a huge mystery to solve: where were these middleware-killing threads coming from?

"Wait a minute! Wasn't Frank doing something like that?" one of the other team members piped up.

"Frank!" A department manager with no technical expertise, who insisted on attending every meeting regardless, darted up straight in his chair. For once, he wasn't haranguing them for their lack of progress. He resembled a wolf who'd sniffed blood in the air. "You mean Frank from Accounting?!"

This was the corporate equivalent of an arrest warrant. Frank from Accounting was duly called forth.

"That's my program." Frank stood before the table, laid back and blithe despite the obvious frayed nerves of several individuals within the room. "It queries the middleware every 5 minutes."

They were finally getting somewhere. Galvanized, John's heart pounded. "How?" he asked.

"Well, it could be that the middleware is down, so first, my program opens a connection just to make sure it's working," Frank explained. "If that works, it opens another connection and sends the query."

John's confusion mirrored the multiple frowns that filled the room. He forced himself to carefully parse what he'd just heard. "What happens to the first connection?"

"What do you mean?" Frank asked.

"You said your program opens two connections. What do you do with the first one?"

"Oh! I just use that one to test whether the middleware is up."

"You don't need to do that!" one of the networking experts snarled. "For Pete's sake, take that out of your code! Don't you realize you're tanking this thing for everyone else?"

Frank's expression made clear that he was entirely oblivious to the chaos wrought by his program. Somehow, he survived the collective venting of frustration that followed within that conference room. After one small update to Frank's program, the middleware stabilized—for the time being. And while Frank became a scapegoat and villain to some, he was a hero to many, many more. After all, he single-handedly convinced the bank's bigwigs that the status quo was too precarious. They began to plan out a full migration away from mainframe, a move that would free them from their dependence upon aging, orphaned middleware.

Now that the mystery had been solved, John knew where to look in Edgar's source code. The thread pool had a limit of 10, and every thread began by waiting for input. The middleware could handle bad input well enough, but it hadn't been written to handle the case of no input at all.

[Advertisement] Utilize BuildMaster to release your software with confidence, at the pace your business demands. Download today!

Our anonymous submitter, whom we'll call Craig, worked for GlobalCon. GlobalCon relied on an offshore team on the other side of the world for adding/removing users from the system, support calls, ticket tracking, and other client services. One day at work, an urgent escalated ticket from Martin, the offshore support team lead, fell into Craig's queue. Seated before his cubicle workstation, Craig opened the ticket right away:

The new GlobalCon support website is not working. Appears to have been taken over by ChatGPT. The entire support team is blocked by this.

Instead of feeling any sense of urgency, Craig snorted out loud from perverse amusement.

"What was that now?" The voice of Nellie, his coworker, wafted over the cubicle wall that separated them.

"Urgent ticket from the offshore team," Craig replied.

"What is it this time?" Nellie couldn't suppress her glee.

"They're dead in the water because the new support page was, quote, taken over by ChatGPT."

Nellie laughed out loud.

"Hey! I know humor is important to surviving this job." A level, more mature voice piped up behind Craig from the cube across from his. It belonged to Dana, his manager. "But it really is urgent if they're all blocked. Do your best to help, escalate to me if you get stuck."

"OK, thanks. I got this," Craig assured her.

He was already 99.999% certain that no part of their web domain had gone down or been conquered by a belligerent AI, or else he would've heard of it by now. To make sure, Craig opened support.globalcon.com in a browser tab: sure enough, it worked. Martin had supplied no further detail, no logs or screenshots or videos, and no steps to reproduce, which was sadly typical of most of these escalations. At a loss, Craig took a screenshot of the webpage, opened the ticket, and posted the following: Everything's fine on this end. If it's still not working for you, let's do a screenshare.

Granted, a screensharing session was less than ideal given the 12-hour time difference. Craig hoped that whatever nefarious shenanigans ChatGPT had allegedly committed were resolved by now.

The next day, Craig received an update. Still not working. The entire team is still blocked. We're too busy to do a screenshare, please resolve ASAP.

Craig checked the website again with both laptop and phone. He had other people visit the website for him, trying different operating systems and web browsers. Every combination worked. Two things mystified him: how was the entire offshore team having this issue, and how were they "too busy" for anything if they were all dead in the water? At a loss, Craig attached an updated screenshot to the ticket and typed out the best CYA response he could muster. The new support website is up and has never experienced any issues. With no further proof or steps to reproduce this, I don't know what to tell you. I think a screensharing session would be the best thing at this point.

The next day, Martin parroted his last message almost word for word, except this time he assented to a screensharing session, suggesting the next morning for himself.

It was deep into the evening when Craig set up his work laptop on his kitchen counter and started a call and session for Martin to join. "OK. Can you show me what you guys are trying to do?"

To his surprise, he watched Martin open up Microsoft Teams first thing. From there, Martin accessed a chat to the entire offshore support team from the CPO of GlobalCon. The message proudly introduced the new support website and outlined the steps for accessing it. One of those steps was to visit support.globalcon.com.

The web address was rendered as blue outlined text, a hyperlink. Craig observed Martin clicking the link. A web browser opened up. Lo and behold, the page that finally appeared was www.chatgpt.com.

Craig blinked with surprise. "Hang on! I'm gonna take over for a second."

Upon taking control of the session, Craig switched back to Teams and accessed the link's details. The link text was correct, but the link destination was ChatGPT. It seemed like a copy/paste error that the CPO had tried to fix, not realizing that they'd needed to do more than simply update the link text.

"This looks like a bad link," Craig said. "It got sent to your entire team. And all of you have been trying to access the support site with this link?"

"Correct," Martin replied.

Craig was glad he couldn't be seen frowning and shaking his head. "Lemme show you what I've been doing. Then you can show everyone else, OK?"

After surrendering control of the session, Craig patiently walked Martin through the steps of opening a web browser, typing support.globalcon.com into the header, and hitting Return. The site opened without any issue. From there, Craig taught Martin how to create a bookmark for it.

"Just click on that from now on, and it'll always take you to the right place," Craig said. "In the future, before you click on any hyperlink, make sure you hover your mouse over it to see where it actually goes. Links can be labeled one thing when they actually take you somewhere else. That's how phishing works."

"Oh," Martin said. "Thanks!"

The call ended on a positive note, but left Craig marveling at the irony of lecturing the tech support lead on Internet 101 in the dead of night.

It's a holiday in the US today, so we're taking a long weekend. We flip back to a classic story of a company wanting to fill 15 different positions by hiring only one person. It's okay, Martin handles the database. Original - Remy

A curious email arrived in Phil's Inbox. "Windows Support Engineer required. Must have experience of the following:" and then a long list of Microsoft products.

Phil frowned. The location was convenient; the salary was fine, just the list of software seemed somewhat intimidating. Nevertheless, he replied to the agency saying that he was interested in applying for the position.

A few days later, Phil met Jason, the guy from the recruitment agency, in a hotel foyer. "It's a young, dynamic company", the recruiter explained,"They're growing really fast. They've got tons of funding and their BI Analysis Suite is positioning them to be a leading player in their field."

Phil nodded. "Ummm, I'm a bit worried about this list of products", referring to the job description. "I've never dealt with Microsoft Proxy Server 1.0, and I haven't dealt with Windows 95 OSR2 for a long while."

"Don't worry," Jason assured, "The Director is more an idea man. He just made a list of everything he's ever heard of. You'll just be supporting Windows Server 2003 and their flagship application."

Phil winced. He was a vanilla network administrator – supporting a custom app wasn't quite what he was looking for, but he desperately wanted to get out of his current job.

A few days later, Phil arrived for his interview. The company had rented smart offices on a new business park on the edge of town. He was ushered into the conference room, where he was joined by The Director and The Manager.

"So", said The Manager. "You've seen our brochure?"

"Yeah", said Phil, glancing at the glossy brochure in front of him with bright, Barbie-pink lettering all over it.

"You've seen a demo version of our application – what do you think?"

"Well, I think that it's great!", said Phil. He'd done his research – there were over 115 companies offering something very similar, and theirs wasn't anything special. "I particularly like the icons."

"Wonderful!" The Director cheered while firing up PowerPoint. "These are our servers. We rent some rack space in a data center 100 miles away." Phil looked at the projected picture. It showed a rack of a dozen servers.

"They certainly look nice." said Phil. They did look nice – brand new with green lights.

"Now, we also rent space in another data center on the other side of the country," The Manager added.

"This one is in a former cold-war bunker!" he said proudly. "It's very secure!" Phil looked up at another photo of some more servers.

"What we want the successful applicant to do is to take care of the servers on a day to day basis, but we also need to move those servers to the other data center", said The Director. "Without any interruption of service."

"Also, we need someone to set up the IT for the entire office. You know, email, file & print, internet access – that kind of thing. We've got a dozen salespeople starting next week, they'll all need email."

"And we need it to be secure."

"And we need it to be documented."

Phil was scribbled notes as best he could while the interviewing duo tag teamed him with questions.

"You'll also provide second line support to end users of the application."

"And day-to-day IT support to our own staff. Any questions?"

Phil looked up. "Ah… which back-end database does the application use?" he asked, expecting the answer would be SQL Server or perhaps Oracle, but The Director's reply surprised him.

"Oh, we wrote our own database from scratch. Martin wrote it." Phil realized his mouth was open, and shut it. The Director saw his expression, and explained. "You see, off the shelf databases have several disadvantages – the data gets fragmented, they're not quick enough, and so on. But don't have to worry about that – Martin takes care of the database. Do you have any more questions?"

Phil frowned. "So, to summarize: you want a data center guy to take care of your servers. You want someone to migrate the application from one data center to another, without any outage. You want a network administrator to set up, document and maintain an entire network from scratch. You want someone to provide internal support to the staff. And you want a second line support person to support the our flagship application."

"Exactly", beamed The Director paternally. "We want one person who can do all those things. Can you do that?"

Phil took a deep breath. "I don't know," he replied, and that was the honest answer.

"Right", The Manager said. "Well, if you have any questions, just give either of us a call, okay?"

Moments later, Phil was standing outside, clutching the garish brochure with the pink letters. His head was spinning. Could he do all that stuff? Did he want to? Was Martin a genius or a madman to reinvent the wheel with the celebrated database?

In the end, Phil was not offered the job and decided it might be best to stick it out at his old job for a while longer. After all, compared to Martin, maybe his job wasn't so bad after all.

We talked about singletons a bit last week. That reminded John of a story from the long ago dark ages where we didn't have always accessible mobile Internet access.

At the time, John worked for a bank. The bank, as all banks do, wanted to sell mortgages. This often meant sending an agent out to meet with customers face to face, and those agents needed to show the customer what their future would look like with that mortgage- payment calculations, and pretty little graphs about equity and interest.

Today, this would be a simple website, but again, reliable Internet access wasn't a thing. So they built a client side application. They tested the heck out of it, and it worked well. Sales agents were happy. Customers were happy. The bank itself was happy.

Time passed, as it has a way of doing, and the agents started clamoring for a mobile web version, that they could use on their phones. Now, the first thought was, "Wire it up to the backend!" but the backend they had was a mainframe, and there was a dearth of mainframe developers. And while the mainframe was the source of truth, and the one place where mortgages actually lived, building a mortgage calculator that could do pretty visualizations was far easier- and they already had one.

The client app was in .NET, and it was easy enough to wrap the mortgage calculation objects up in a web service. A quick round of testing of the service proved that it worked just as well as the old client app, and everyone was happy - for awhile.

Sometimes, agents would run a calculation and get absolute absurd results. Developers, putting in exactly the same values into their test environment wouldn't see the bad output. Testing the errors in production didn't help either- it usually worked just fine. There was a Heisenbug, but how could a simple math calculation that had already been tested and used for years have a Heisenbug?

Well, the calculation ran by simulation- it simply iteratively applied payments and interest to generate the entire history of the loan. And as it turns out, because the client application which started this whole thing only ever needed one instance of the calculator, someone had made it a singleton. And in their web environment, this singleton wasn't scoped to a single request, it was a true global object, which meant when simultaneous requests were getting processed, they'd step on each other and throw off the iteration. And testing didn't find it right away, because none of their tests were simulating the effect of multiple simultaneous users.

The fix was simple- stop being a singleton, and ensure every request got its own instance. But it's also a good example of misapplication of patterns- there was no need in the client app to enforce uniqueness via the singleton pattern. A calculator that holds state probably shouldn't be a singleton in the first place.

[Advertisement] Utilize BuildMaster to release your software with confidence, at the pace your business demands. Download today!

Barry rolled into work at 8:30AM to see the project manager waiting at the door, wringing her hands and sweating. She paced a bit while Barry badged in, and then immediately explained the issue:

Today was a major release of their new features. This wasn't just a mere software change; the new release was tied to major changes to a new product line- actual widgets rolling off an assembly line right now. And those changes didn't work.

"I thought we tested this," Barry said.

"We did! And Stu called in sick today!"

Stu was the senior developer on the project, who had written most of the new code.

"I talked to him for a few minutes, and he's convinced it's a data issue. Something in the metadata or something?"

"I'll take a look," Barry said.

He skipped grabbing a coffee from the carafe and dove straight in.

Prior to the recent project, the code had looked something like this:

if (IsProduct1(_productId))
	_programId = 1;
elseif (IsProduct2(_productId))
	_programId = 2;
elseif (IsProduct3(_productId))
	_programId = 3;

Part of the project, however, was about changing the workflow for "Product 3". So Stu had written this code:

if (IsProduct1(_productId))
	_programId = 1;
else if (IsProduct2(_productId))
	_programId = 2;
else if (IsProduct3(_productId))
	_programId = 3;
	DoSomethingProductId3Specific1();
	DoSomethingProductId3Specific2();
	DoSomethingProductId3Specific3();

Since this is C# and not Python, it took Barry all of 5 seconds to spot this and figure out what the problem was and fix it:

if (IsProduct1(_productId))
{
	_programId = 1;
}
else if (IsProduct2(_productId))
{
	_programId = 2;
}
else if (IsProduct3(_productId))
{
	_programId = 3;
	DoSomethingProductId3Specific1();
	DoSomethingProductId3Specific2();
	DoSomethingProductId3Specific3();
}

This brings us to about 8:32. Now, given the problems, Barry wasn't about to just push this change- in addition to running pipeline tests (and writing tests that Stu clearly hadn't), he pinged the head of QA to get a tester on this fix ASAP. Everyone worked quickly, and that meant by 9:30 the fix was considered good and ready to be merged in and pushed to production. Sometime in there, while waiting for a pipeline to complete, Barry managed to grab a cup of coffee to wake himself up.

While Barry was busy with that, Stu had decided that he wasn't feeling that sick after all, and had rolled into the office around 9:00. Which meant that just as Barry was about to push the button to run the release pipeline, an "URGENT" email came in from Stu.

"Hey, everybody, I fixed that bug. Can we get this released ASAP?"

Barry went ahead and released the version that he'd already tested, but out of morbid curiosity, went and checked Stu's fix.

if (IsProduct1(_productId))
	_programId = 1;
else if (IsProduct2(_productId))
	_programId = 2;
else if (IsProduct3(_productId))
{
	_programId = 3;
}

if (IsProduct3(_productId))
{
	DoSomethingProductId3Specific1();
	DoSomethingProductId3Specific2();
	DoSomethingProductId3Specific3();
}

At least this version would have worked, though I'm not sure Stu fully understands what "{}"s mean in C#. Or in most programming languages, if we're being honest.

With Barry's work, the launch went off just a few minutes later than the scheduled time. Since the launch was successful, at the next company "all hands", the leadership team made sure to congratulate the people instrumental in making it happen: that is to say, the lead developer of the project, Stu.

Today is holiday in the US, where we celebrate a cosplay version of history with big meals and getting frustrated with our family. It's also a day where we are thankful- usually to not be at work, but also, thankful to not work with Brad. Original --Remy

Anita parked outside the converted garage, the printed graphic reading Global Entertainment Strategies (GES) above it. When the owner, an old man named Brad, had offered her a position after spotting her in a student computer lab, she thought he was crazy, but a background check confirmed everything he said. Now she wondered if her first intuition was correct.

“Anita, welcome!” Brad seemed to bounce like a toddler as he showed Anita inside. The walls of the converted garage were bare drywall; the wall-mounted AC unit rattled and spat in the corner. In three corners of the office sat discount computer desks. Walls partitioned off Brad’s office in the fourth corner.

He practically shoved Anita into an unoccupied desk. The computer seemed to be running an unlicensed version of Windows 8, with no Office applications of any kind. “Ross can fill you in!” He left the office, slamming the door shut behind him.

“Hi.” Ross rolled in his chair from his desk to Anita’s. “Brad’s a little enthusiastic sometimes.”

“I noticed. Uh, he never told me what game we’re working on, or what platform. Not even a title.”

Ross’s voice lowered to a whisper. “None of us know, either. We’ve been coding in Unity for now. He hired you as a programmer, right? Well, right now we just need someone to manage our documentation. I suggest you prepare yourself.”

Ross led Anita into Brad’s office. Above a cluttered desk hung a sagging whiteboard. Every square inch was covered by one, sometimes several, overlapping sticky notes. Each had a word or two written in Brad’s scrawl.

“We need more than just random post-its with ‘big guns!’ and ‘more action!’” Ross said. “We don’t even know what the title is! We’re going crazy without some kind of direction.”

Anita stared at the wall of sticky notes, feeling her sanity slipping from her mind like a wet noodle. “I’ll try.”

Sticky Escalation

Brad, can we switch to Word for our documentation? It’s getting harder
to read your handwriting, and there’s a lot of post-its that have
nothing to do with the game. This will make it easier to proceed with
development. -Anita

Two minutes after she sent the email, Brad barged out of his office. “Anita, why spend thousands of dollars on software licenses when this works just fine? If you can’t do your job with the tools you have, what kind of a programmer does that make you?”

“Brad, this isn’t going to work forever. Your whiteboard is almost out of room, and you won’t take down any of your non-game stickies!”

“I can’t take any of them down, Anita! Any of them!” He slammed the door to his office behind him.

The next day, Anita was greeted at the door by the enthusiastic Brad she had met before the interview. “I listened to reason, Anita. I hope this is enough for you to finish this documentation and get coding again!”

Brad led Anita into his office. On every wall surface, over the door, even covering part of the floor, were whiteboards. Sticky notes dotted nearly a third of the new whiteboard space.

“Now, Anita, if I don’t see new code from you soon, I may just have to let you go! Now get to work!”

Anita went to sit at her desk, then stopped. Instead, she grabbed a bright red sticky note, wrote the words “I QUIT” with a sharpy, barged into Brad’s office, and stuck it to his monitor. Brad was too stunned to talk as she left the converted garage.

The Avalanche

“Are you doing better?” Jason called Anita a few weeks later. Their short time together at GES has made them comrades-in-arms, and networking was crucial in the business.

“Much,” she said. “I got a real job with an indie developer in Santa Monica. We even have a wiki for our framework!”

“Well, listen to this. The day after you quit, the AC unit in the garage broke. I came into work to see Brad crying in a corner in his office. All of the sticky notes had curled in the humidity and fallen to the floor. The day after he got us all copies of Word.

“Too bad we still don’t know what the title of the game is.”

[Advertisement] ProGet’s got you covered with security and access controls on your NuGet feeds. Learn more.

One of the clients for Rudolf's company was getting furious with them. The dev team was in constant firefighting mode. No new features ever shipped, because the code-base was too fragile to add new features to without breaking something else. What few tests existed were broken. Anyone put on the project burned out and fled in months, sometimes weeks, and rarely after only a few days.

Rudolf wasn't too pleased when management parachuted him into the project to save it. But when he pulled the code and started poking around, it looked bad but not unsalvageable. The first thing he noticed is that, when following the instructions in the README, he couldn't build and run the application. Or maybe he wasn't following the instructions in the README, because the README was a confusing and incoherent mess, which included snippets from unresolved merges. Rudolf's first few days on the project were spent just getting it building and running locally, then updating the README. Once that was done, he started in on fixing the broken tests. There was a lot of work to be done, but it was all doable work. Rudolf could lay out a plan of how to get the project back on track and start delivering new features.

It's about then that Steve, the product owner, called Rudolf in to his office. "What the hell do you think you're doing?"

Rudolf blinked. "Um… what I was asked to do?"

"Three days and you just commit a README update? A couple of unit tests?"

"Well, it was out of date and meant I couldn't-"

"Our client is crazy about their business," Steve said. "Not about READMEs. Not about unit tests. None of that actually helps their business."

Rudolf bit back a "well, actually," while Steve ranted.

"Next thing you're going to tell me is that we should waste time on refactoring, like everybody else did. Time is money, time is new features, and new features are money!"

Suddenly, Rudolf realized that the reason the project had such a high burnout rate had nothing to do with the code itself. And while Rudolf could fix the code, he couldn't fix Steve. So, he did what everyone else had done: kept his head down and struggled through for a few months, and kept poking his manager to get him onto another project. In the meantime, he made this code slightly better for the next person, despite Steve's ranting. Rudolf eventually moved on, and Steve told everyone he was the worst developer that had ever touched the project.

The customer continued to be unhappy.

[Advertisement] Continuously monitor your servers for configuration changes, and report when there's configuration drift. Get started with Otter today!

Back in the heady days of the DotCom Bubble, startups were thick on the ground, and Venture Capital money was a flood- lifting startups atop a tsunami only to crash them back into the ground a short time later. Taliesyn once worked for one such startup.

Taliesyn's manager, Irving, was an expert in AI. In the age of the DotCom Bubble, this meant Irving knew LISP. Knowing LISP was valuable here, because their core product was a database system built on LISP- specifically the Common LISP Object System, an object-oriented bolt-on for LISP.

It was an object-oriented database system, akin to the modern NoSQL databases, but its architecture left a few things to be desired. First, since disk reads and writes were expensive operations, the system avoided them. All updates to data were done in memory, and someday, at some point, when the program felt like it, the changes would be written to disk. This meant that any failures or crashes could lose potentially days of data. Worse, the data was stored in one gigantic text file, which meant corruption could easily take out the entire database.

These were legitimate problems, and due to the architecture, they were going to be challenging to resolve. That was the startup life, however- they had a minimum viable product, and just needed to pour energy into making it something worth using.

Everyone looked to Irving, the AI and LISP expert, to guide them through this.

Irving saw where the real problems lay. "Your database doesn't support SQL," Irving said.

"Well, sure," Taliesyn said. "That's our selling point."

Irving nodded, and then, speaking slowly, as if to a particularly dense child, said, "A database needs to support SQL."

"I mean, a relational database, sure," Taliesyn said, "but we're using an object oriented data model which means we don't need to do joins or-" Taliesyn kept talking, explaining why their database didn't support SQL, why it shouldn't support SQL, and why SQL support was not only off the roadmap, but so far off the roadmap that it was labelled "Here there be dragons."

Irving nodded along, and ended the conversation with a, "Sure, that makes sense."

Everything was fine for a few weeks, until one of Taliesyn's peers on a different team, Angela, shot him an email: "Hey, marketing is getting antsy about a SQL demo, and I've got half a dozen features blocked until you get me a build with that functionality. What's the timeline like?"

Taliesyn was uncertain about what the timeline was like, since he had now clearly slipped into a parallel universe. He politely informed his peer that he had no idea what was going on, but would find out. It didn't take a great sleuth to figure out that Irving had started appending his own features to the roadmap.

Taliesyn tracked Irving down and demanded to know what was going on.

"A customer is already using it!" Irving protested. "They wrote it themselves! So we should be able to do it easily. Frankly, it's embarrassing to say that we can't do something with our own tools that the customer is already doing!"

Taliesyn knew that Irving was either wrong or lying, and asked to talk to the customer. Irving was, in fact, wrong. The customer had used LISP to write an extension to their object database (another one of those selling point features), and this extension used ODBC to open a connection to a relational database. This let them move data between the two different database systems.

"Irving," Taliesyn said, "they're not using SQL on our database, they're connecting to a relational database and using SQL there. SQL doesn't make sense for our data structure! We don't have tables, or keys, or relationships. It's objects! You're promising features that we can't deliver."

Irving was unmoved. "We are making a database. A database must have SQL capability. It's structured query language- it's right there in the name! Our structure should be queried by structured query language."

Taliesyn tried going over Irving's head, but upper management had no interest in actually solving personnel problems. Irving's buzzword laden ideas about why SQL was required seemed to jive with what their customers wanted, and the idea of shipping a non-SQL database was getting lost in the endless quest for the next round of VC funding.

The result was a bolted on monster of broken implementations of SQL that infuriated the few customers who tried it, Irving got promoted, and Taliesyn ran as far from the trainwreck as possible before the company flamed out.

[Advertisement] Continuously monitor your servers for configuration changes, and report when there's configuration drift. Get started with Otter today!

Alan worked on a website which had a weekly event which unlocked at 9PM, Saturday, Eastern Time. The details of the event didn't matter, but this little promotion had been running for about a year and a half, without any issues.

Until one day, someone emailed Alan: "Hey, I checked the site on Sunday, and the countdown timer displays 00:00:00."

Alan didn't check their email on Sunday, and when they checked the site, everything was fine, so he set himself a reminder to check things out next Sunday, and left things alone for a week.

Well, Alan forgot on Sunday. It was his day off after all, but he did remember to check first thing on Monday. When he came in, the timer read 00:00:00. Alan had a twelve o'clock flasher. Oddly, when he checked it after grabbing some coffee, the timer now showed the correct value.

It was time to dig through the code. Now, this story happened quite some time ago, so the countdown timer itself was a Flash widget. But the widget received a parameter from the HTML DOM, which itself was generated by PHP, and it didn't take long to find the SQL query it was using to find the next event: SELECT next_event FROM event_timer.

Yes, event_timer was a one column, one row table. A quick search through the codebase found the table referenced only one other place: backend_settimer.php.

All the pieces came together: resetting the timer was an entirely manual process. Every Monday, Tina came in, and assuming she remembered, she reset the timer. Some days she did it late, or forgot entirely. Some weeks, she was on vacation. Maybe she remembered to delegate, maybe she didn't.

For roughly 72 weeks, this had been how things had been working.

The good news was that the date was getting parsed with the PHP strtotime function, which meant Alan merely had to go to the backend_settimer.php and set the value to Next Saturday 9pm, and let Tina know she didn't need to do this anymore.

[Advertisement] Continuously monitor your servers for configuration changes, and report when there's configuration drift. Get started with Otter today!