CSE 370: Software Engineering Principles

Adaora Chukwu

Introduction

Adaora Chukwu found her calling in the bustling world of e-commerce. Globemart.com, a titan of online retailing, was on the brink of another groundbreaking technological shift, and Adaora was at the heart of it. Unbeknownst to her, however, an ominous shadow lingered. Amidst the confluence of modern and legacy systems lay the most critical software vulnerability ever discovered, one that threatened to unleash chaos and destroy everything she had worked for.

Determination and Resilience

Adaora was born in Nigeria, a country with a long and rich history. Adaora, however, was fascinated by the future. She was particularly interested in computers, even though her family couldn't afford one. Her dreams of exploring the digital world seemed distant but her passion was unwavering. Adaora's father encouraged her ambitions, instilling the belief that hard work and resilience could bridge the gap between dreams and reality. Adaora knew she would be a programmer someday, and, if she didn't give up, a leader.

Adaora's professional journey began in Lagos. Even with over 80% of the country's startups, finding a job in technology was difficult. She worked as a receptionist for three years before landing an internship with GreenGrocers.com. Once she was in, however, she never looked back. Adaora immersed herself in the world of digital fulfillment and the code that drove it. Five months later she was offered a full time software engineering position. Her dream had come true.

An active contributor to sites like Stack Overflow and Reddit, Adaora was noticed by other people too. After a few years of work in Lagos, a recruiter from the United States reached out with a possible job opportunity at Globemart. Adaora's technological skills and familiarity with African markets really stood out. Several interviews later, Globemart offered her the job. She was moving to the United States!

Rise to Leadership

Adaora's new job started in a maze of cubicles. The area hummed with the sound of caffeine-fueled brainstorming sessions and clicking keyboards. Even though her first tasks were making minor bug fixes and writing technical documentation, Adaora felt like she was in heaven. Globalmart's technological labyrinth of homegrown systems was fascinating.

"I see potential here, Adaora. Not just in our systems, but in you," her manager, Elena, remarked one day.

Adaora had a real knack for understanding complex software and Elena knew it. It was impossible for anyone to completely understand the hundreds of systems at Globemart but it seemed like Adaora was going to try. Elena was never surprised when she received a text late at night, with another question about their systems, from Adaora.

The questions soon turned into suggestions. Adaora envisioned a future where agility and innovation coexisted with the robust legacy systems that were the company's backbone. Not one to hide her thoughts, Adaora submitted several proposals for system improvements that she believed would revolutionize the way Globemart functioned.

No one was surprised when Adaora's unique blend of humility and relentless pursuit of improvement finally caught the eye of senior leadership. She was not just solving problems, she was reimagining solutions. Her eventual rise to a directorial position was a testament to her impact at the company.

Tasked with overseeing a new government procurement (GP) program, and the European, Middle-eastern, and African (EMEA) program, a group comprising over six dozen systems, Adaora eventually found herself at the crossroads of Globemart's past and its future.

Embracing CI/CD

The adoption of Continuous Integration/Continuous Delivery (CI/CD) marked a pivotal chapter in Adaora's career. She understood early-on that Globemart needed to maintain its competitive edge by evolving beyond the confines of manual deployments and embracing the future.

"Imagine a world where changes go live in minutes, not days," Adaora mused to her team.

It was an enticing vision. Many of Globemart's configurations and deployments were managed by a system called PAC2000. Designed in the late 1990's, PAC2000 was a fortress of forms, approvals and manual processes that seemed impregnable in its resistance to change.

Deploying a single line of code to production through PAC2000 was a daunting task. The process started by filling out a Deployment Request Form (DRF). The DRF required detailed descriptions of code changes, potential impacts, rollback plans, and several other items besides. Even though it was submitted electronically, it had to be printed, signed, and physically delivered to the PAC2000 administration team's office as well. Then the real battle began.

Each application affected by the deployment required a separate approval from its respective manager, who were almost always busy with other more important tasks. It took days, sometimes weeks, to obtain all the approvals. Once the application manager approvals were obtained, the DRF had to be signed-off by the quality assurance team. The quality assurance team conducted their own review to ensure that all the necessary testing had been completed and documented.

The next step was the Change Advisory Board (CAB) meeting, a weekly congregation of senior IT management and stakeholders from various departments. The CAB's role was to assess the risks and business impacts of all proposed deployments, prioritizing them according to the company's objectives and requirements. Getting approval from the CAB required meticulous preparation of the DRF. Unanswered questions or gaps in planning could result in a deferral to the next week's meeting, or worse, outright rejection.

The final step was obtaining clearance from the Security Compliance team. With Globemart's strict adherence to security protocols and commitment to protecting customer data, this was no small feat. The Security Compliance team would rigorously evaluate the deployment's adherence to security best practices, often requiring additional documentation or modifications to the code to address potential vulnerabilities.

Adaora marveled at the size and complexity of the process. It was a stark contrast to the agile, innovative environment she envisioned. Determined to move beyond PAC2000, she began the GP program by establishing a fully automated verification and validation process. Her goal was not just to employ CI/CD but to fundamentally transform how her team thought about software maintenance.

The CI/CD pipeline they built was integrated with a version control system called Git. When a developer checked in new code, the pipeline would automatically assemble it, build it, test it, deploy it and run it in a staging environment. If anything failed, the process was automatically halted, the code rolled back, and the developer notified.

Once the changes were in the staging environment, a human tester was notified to review them. If anything seemed wrong, the tester rolled the changes back. If everything seemed right, they promoted the changes to production—all with the single click of a button. The pipeline was so good they were able to promote changes several times a day if they wanted.

The EMEA program, a brittle tapestry of intricately woven, custom-built legacy systems, spanning multiple countries, languages and regulatory environments, wasn't going to be as easy. Transitioning it to the new CI/CD framework required more than technical skill. It required a deep understanding of the business and cultural nuances across the entire geographical region as well. It was terrifying and exciting all at the same time; just the kind of challenge Adaora was looking for.

A Tempest Arises

The news broke like a sudden squall. It was Thursday, December 9th. Everyone, including Adaora, was caught off guard by the news. A critical vulnerability in Log4J was exposed. The confidentiality, integrity, and availability of over 93% of the world's enterprise systems was at risk.

Apache's Log4J was an enterprise-grade software library that added logging capability to Java applications. Released in 2001, it became one of the most deployed pieces of open source software ever published. As a supporting utility, many organizations weren't even aware they were using Log4J as an embedded component in their systems.

The Log4J vulnerability, dubbed Log4Shell, was discovered by Alibaba, one of Globemart's major competitors. Anyone with basic hacking skills could exploit it to corrupt data, leak sensitive information, or perform a number of other malicious activities. Security researchers gave it the highest possible severity rating.

Adaora assembled her team for the company-wide emergency meeting. She knew the situation was serious but couldn't help feeling a little annoyed. Adaora knew their systems were built to the highest security standards. She just didn't believe the company was in quite as much trouble as everyone seemed to think.

"We have a critical situation on our hands," John Hansen, the Chief Technology Officer, began. "There are over 600 applications at Globemart that use Log4J and every one of them is at risk."

The strategy was simple: triage and patch. Every system would be designated as tier one, two or three. Tier one systems would be secured first, followed by tier two, and finally tier three.

“What about our other projects?” someone asked.

"Keep working on them,” John replied. “I know it seems impossible but we don't have a choice. You'll have to find a way to do both.”

A wave of indistinct murmuring rose through the room. Adaora didn't need to hear what they were saying to know what it was about. Even if she was right, and Log4Shell wasn't as bad as everyone thought, the timing couldn't have been worse.

It was the height of e-commerce season. Most of Globemart's money came during the period between Black Friday and the ground shipping cutoff date—the last day they could ship and guarantee delivery before Christmas. That money was sacred. There were all kinds of restrictions on system changes and deployments.

In addition, Adaora's team was working under an extremely tight schedule. Ever since they finished the CI/CD pipeline, the GP program seemed to have taken on a life of its own. Sales needed several critical updates by January of the coming year and Adaora's team needed every minute they had to stay on track. They'd been working flat out for several weeks already.

More than that, Adaora was under constant pressure to keep the EMEA program going. Finance wanted several enhancements to revitalize international business and, as they regularly pointed out, Globemart needed the money. Adaora had just agreed on a schedule with them when Log4Shell came to light. A new round of negotiating wouldn't go over well at all. The timing was terrible indeed.

The Eye of the Storm

Adaora arrived at work early Friday morning. She hadn't even managed to hang up her jacket when the phone rang sharply. It was Alex from the Security Compliance team. He wanted to know about her plan to deploy the patch that Apache just released. A second light on her phone started blinking before she could answer him. It was Darren, a senior figure in the finance department. He was probably calling to remind her about the GP development schedule. Adaora merged the calls before continuing.

Alex insisted that Adaora patch her systems immediately. When she questioned him further, he replied honestly. He didn't know how bad Log4Shell really was. That was why patching was so urgent. It would be foolish to find out the hard way. Darren's stance was equally clear and just as uncompromising. The patching process must not derail their other work. From his point of view, the risk to the company's bottom line was greater.

Adaora appreciated both arguments. Studying it in her mind, she decided Alex's carried more weight. They'd patch the systems now. Darren wasn't happy and let them both know. He respected the decision though. Adaora had an excellent reputation.

Adaora finished the call and set about gathering her team. The task was enormous; six tier one systems needed immediate attention. The remaining tier two systems would be completed the following week. The tier three systems after that. Thanks to the CI/CD pipeline, the GP program would be smooth. The EMEA program, as usual, presented a more significant challenge.

Jordan, a key member of Adaora's team, volunteered to spearhead the planning and execution of the EMEA patches. Everyone else quickly rallied behind him, managing the DRF and following up on the intricate web of approvals needed from other managers. Fortunately, the CAB convened for an extended session to expedite all the patch approvals. Pushing through their exhaustion, Adaora's team worked late into the night and throughout the next day. The last patch for the tier one systems was deployed at 8pm on Saturday evening. It was a grueling two day marathon but they made it.

Adaora was proud of her team's efforts. Patching the tier one systems was a significant achievement. They weren't out of the storm yet however. They couldn't keep working at this pace and not pay for it, and the tier two and three systems still needed to be patched. They were definitely going to lose a week, maybe even more, no matter how hard they pushed themselves. Adaora was determined. They'd made it this far together. She knew they'd find their way through to the end.

The Second Wave

Adaora came to work on Monday with a new sense of hope. She hung up her jacket, sat down at her computer and signed in, smiling with determination at the challenges ahead. Her smile faded almost instantly. Apache had released another announcement earlier that morning. The initial Log4Shell patch was insufficient. A new vulnerability had been discovered, one that required an immediate follow up patch.

A sense of worried anticipation hung in the air as Adaora gathered her team for yet another emergency meeting. She could see the fatigue in their faces and hear the strain in their faces as she looked around. The weekend had taken a terrible toll and now they were back to square one. The low buzz of tired conversations died to complete silence when Adaora broke the news.

“Can't you just tell them no?” asked a member of the team, real desperation in her voice.

Adaora found herself at a crossroads. Safeguarding their systems, reputation and trust was important. Strengthening Globemart's financial situation and ensuring their future success was just as important. The well-being of her team, their morale, and capacity to endure the incredible demands being placed upon them was also important. Was the new vulnerability so severe that it warranted back-to-back emergency patches? Could there be more undisclosed vulnerabilities lurking, waiting to surface? Would they be in the same situation tomorrow, the day after that, or the day after that? How was she going to find her way through this second wave without compromising them on any front?

There were no easy answers but one thing was for certain: the decision she made would define her career and have a real impact on Globemart itself. Adaora continued to ponder. What was the right course of action? What should she do?