ESHARP.NET

Technology and life with Eyvonne Sharp

Escalate This

By 2 Comments

A few weeks ago, this tweet by David Peñaloza riled me up a bit.


So much so that I tweeted this in response:


Whenever I respond with intense emotion, I try to unpack my thoughts before diving into a rant. I’ve had some time to think.

Anyone who’s been a key technical contributor during a large scale enterprise outage knows the routine. An outage occurs. On-call teams across the organization jump on a call. Each group opens tickets with every vendor in the environment. Leadership demands 15-minute updates. Engineers troubleshoot and gather data in real-time, which is heard, misunderstood, miscommunicated, and relayed out of context. If the outage is severe enough, vendor account teams are engaged. Someone demands escalation.

The details may vary depending on the scale of the outage, the IT team’s health, and the level of vendor investment within the organization. In my experience, this scenario plays out with shocking regularity.

First, I want to acknowledge that many times escalation is necessary. If a team is stuck and isn’t making progress, they need to get unstuck. Sometimes, engineers are too close to the problem and cannot see it clearly. Other times, a support engineer or team lead does not have the skills to deal with the issue at hand. If a team repeatedly needs fifteen more minutes with no substantive change or new information, intervention may be necessary. These issues may require a nudge, push, or demand from someone in authority.

More commonly, the demand for escalation arises from a fundamentally flawed assumption — the belief that those with executive power and positional authority can fix a problem merely by applying increasing pressure. Many have operated under this assumption for so long, it has become a culturally ingrained self-fulfilling prophecy.
Several incorrect beliefs feed this false assumption.

False Assumption 1: The people working on the issue are incompetent.

In many cases, the immediate demand for escalation comes from believing that the individuals working on the issue are incompetent, over their head, or uninformed. In many instances, IT leaders trust vendor representatives more than they will trust their people — the people who build, maintain, and support systems day-to-day. This should not be. The primary source for documentation, in-depth knowledge, and operational guidance must come from the team responsible for the system’s ultimate operation. Vendors can and should help. They have deep expertise in their particular product but often lack foundational organizational details, knowledge of the interaction surfaces between solutions, and the business drivers for the systems in question.

This lack of trust is toxic to an IT organization. It exposes the environment to risk, devalues the highest contributors, and leaves you vulnerable to unscrupulous vendors. If a vendor representative is unhelpful, by all means, escalate. However, escalation within minutes of the report of an outage signals larger organizational problems.

False Assumption 2: More pressure will fix the issue faster.

No one wants to sit on a 24-hour escalation call. No. One. Yelling, threats of penalty, and blame-shifting do nothing to resolve an issue. In my experience, these behaviors delay resolution, destroy culture, and demoralize the very people required to solve the problem. Escalation often brings more non-productive eyes to the issue, requires more communication, provides more avenues for misinformation, and piles pressure onto the small troupe of people who can resolve the issue. Adding stress, requiring excessive updates, and threatening people with their jobs will not improve their clarity of thought. Also, pressure may rush changes before full consideration and create additional problems.

I once participated in a 200 member outage call in which the most senior leader demanded all troubleshooting happen on that call. Eventually, a brave group of engineers created a secondary call where they talked openly about the data they were gathering and resolved the issue while staying engaged on the “leadership” call. This level of dysfunction increases cognitive load to an unmanageable level and dramatically increases MTTR.

False Assumption 3: When systems come back, everything’s fine.

After a severe outage, teams breathe a sigh of relief, congratulate one another on their hard work, and continue with their daily lives unchanged. Many leaders require an RCA to explain the outage. In the absence of a healthy culture, this process is worthless. Without two-way-trust, the people writing the RCA will withhold information, include vague explanations, and lay blame in the most convenient place — often a recently departed employee or a vendor failure. Based on an incomplete RCA, management will demand changes to ensure this never happens again. Ill-informed solutions will be implemented that increase technical debt, may not mitigate the risk, and may add risk in another area of the IT stack.

When an organization insists on escalation at the first sign of a problem, they’re signaling mistrust, an unhealthy culture, and organizational dysfunction. Pricey support contracts ensure vendors will cope with these challenges. IT teams inside these organizations suffer most as they’re disincentivized, demoralized, and discouraged. Over time, the attrition of great talent reinforces wrongheadedness and furthers the negative spiral.

If your organization has developed a knee-jerk habit of escalation at the first sign of a problem, consider a healthy dose of self-reflection. What harmful attitudes and behaviors drive your escalation habit? What can you change today, before your next outage, to make life better for everyone?

Network Field Day 17 Signals Big Changes Ahead for Networking

By 1 Comment

Rarely in our careers do we get the opportunity to participate in magic. When I say magic, I’m not referring to mystical unicorns or the vendor-promised pixie dust. I’m talking about those fleeting times when we connect with a group of people with shared interests, keen abilities, and an energy surrounding their craft. When we do, we usually can’t plan, prepare for, or prompt these moments. More than once in my career, these experiences have passed by before I’ve fully appreciated their significance.

Last week, I flew to Silicon Valley along with eleven talented, interesting, and challenging delegates for my second Network Field Day. For three days, we soaked in presentations from networking vendors who shared their vision for the future of networking. We asked questions, compared notes, and challenged the presenters. We ate and laughed and shared battle scars and compared experiences in real time. In short, it was magic.

Over the next few weeks, I’ll share more with you about the specific presentations we saw and what we learned from several vendors. But for now, I want to share some themes I observed and the trends you can expect to see coming to a network near you.

Open Source is a Thing

Before you roll your eyes and tell me I’m twenty years too late in this declaration, hear me out. For years, open source tools and solutions have been invading IT. Open source has fueled the Internet since its inception and we’ve seen the open source marketplace mature for our friends in the server space. However, until very recently, the haughty network engineer has declared his (or her) domain too mission critical for open source software. New offerings, even from incumbents, will no longer allow a network engineer to make such an argument with credibility. Nearly every presentation included a link to a GitHub repository where tools and demos can be downloaded.

Containers and Kubernetes are Coming to a Datacenter Near You

Historically, network engineers have focused their attention from the IP address down the stack. They’ve paid little attention to the platforms which host applications. This trend is changing. Containers and Kubernetes made a significant appearance in several presentations. As networking vendors move toward open source tools and continue to virtualize their solutions, our networking gear will begin to look a lot more like servers. A forward thinking network engineer will not be able to ignore containers, Kubernetes, or Linux and maintain relevance.

The Ball is Rolling

Organizations, and entire industries, have inertia. For decades, the networking industry has grown rapidly while only making small incremental technology changes. The larger the industry has grown, the more slowly it has evolved. The interconnectedness of our networks, the protocols on which our industry standardized, and the foundational nature of networking as it relates to the IT stack have contributed to this slowness. Networking is a large industry with lots of inertia which has been sitting still.

Several factors have converged and are pushing the networking industry with enough force to make it move. The most significant force is the cloud which is proving that infrastructure can be provisioned with the click of a button. Although the cloud brings with it new challenges that we will have to solve, it is forcing all of IT infrastructure, and most notably networking, to change how we operate. Other factors which deserve mention are the countless developers, network engineers, hobbyists, and entrepreneurs who’ve seen a vision for a better network and have been working, in small ways, to bring that vision about.

In Closing

Although the industry has been slow to move toward this new automated future, the incumbents are beginning to see that we cannot continue on as we have. They are trying to change. I expect we will see a couple more years of reorganization, bumbling, and lots of intent-based marketing spin but in the long run, our networks will be more automated, we will think more about systems than devices, and there will be a lot of cool new tech to learn.

Why your network should look more like your servers

By 1 Comment

Most of us in the networking community are aware of a groundswell of change coming in the networking industry. Vendors struggle to hang on to market share. The cloud is shifting workloads from our datacenters to cloud providers. The community is talking about software-defined and intent-based solutions while many of our cli-slinging colleagues are blissfully unaware of the technological tsunami building off the shores of our enterprise networks.  
Underneath all the buzzwords exists a fundamental concept that needs to disrupt our industry.

Enter Disaggregation

The best way to describe network disaggregation is to travel back in time 20 years and look at the server/compute space. Back in the day, applications ran on a monolithic stack. Consider the AS 400. IBM would provided the hardware, software, application, and support for the entire system. Customers bought the entire stack, expecting a turn-key solution.

Over time, companies like Microsoft began to sell a desktop operating system which ran on Intel-based hardware provided by any number of hardware vendors. Over time, the industry realized we could do that same thing with servers. Companies began to buy server hardware from one vendor, an operating system from another vendor, and an application from yet another vendor. Microsoft, in part, propelled a disaggregation movement which depleted the monolithic AS 400 stack. Disaggregating operating systems from server hardware plowed the ground for virtualization which spawned a new industry and enabled huge reliability and productivity gains. 

Unfortunately, the network world did not follow the same path. In large part, enterprises still buy their networks as a monolith. They purchase networking hardware, software, and applications from a vendor like Cisco and expect that vendor to support the entire stack. As problems arise, customers have looked to vendors to solve problems with a new monolith, forcing a forklift of the old one.

Finally, a ground swell of change is beginning. Companies like Cumulus Networks, Pluribus Networks, IP Infusion and others are building network operating systems that run on “whitebox” switches. Hardware vendors like Dell, Edgecore, and HP are building switches that run these operating systems.  

The fundamental missing piece (Hint: It’s not technology)

The biggest challenge to move to a disaggregated world is not technology, it’s the people who support the technology. Instead of understanding how to implement the newest vendor products, network architects and engineers need to understand the functionality of their networks. They need to understand what networking protocols do and what their organizations need. They need to have a passion for making their networks more efficient, more flexible, and more in line with business needs.  

In addition, a vendor ecosystem needs to grow to support disaggregated network technology. Enterprises will need help moving to a disaggregated network infrastructure. They will need reliable partners to lean on to architect and design their solutions and to provide support as problems arise.  

For the eager engineer

What can you do if you’re interested in learning more about disaggregation? First, download a copy of a network operating system and fire it up in your favorite emulator (GNS 3 or EVE-NG). Second, begin to talk to your colleagues about disaggregation. Third, begin to build your own justification for why disaggregation is a good idea. Fourth, get more involved in the community!

Network disaggregation is an emerging technology, but it’s not vaporware. Companies with high incentive to cut costs and who have strong engineering talent are implementing disaggregated solutions today. As leading edge companies prove out disaggregated solutions, more enterprises will implement. It may take a few years, but disaggregation is coming to a network near you.  

Can Cisco Restore Confidence with SD-Access?

By Leave a Comment

At Cisco Live 2017, the world’s leading networking vendor shone a spotlight on their new line of campus switches and their emerging software platform coined SD-Access. Some of the buzz was familiar like a new programmable ASIC with more power to process packets. We heard the expected refrain of speeds and feeds, port density, redundancy, and PoE. But we heard new things as well.

After years of industry buzz surrounding Software Defined Networking, Cisco announced their first serious foray into the software-defined space. Software-defined Access combines a suite of Cisco software products to improve usability, build automation, and aid in troubleshooting. Combining ISE and two new software tools, DNA Center and Network Data Platform, Cisco promises a fully automated, secure campus fabric.  On it’s face, it’s difficult to not get excited about SD-Access. Many of us have been begging for more robust, software-driven solutions from Cisco. In many ways, SD-Access is the first coherent unified strategy in that direction.

But…

Those of us who’ve been around networking for any length of time have history. We remember the Cisco Live where OnePK was all the rage. Sessions were packed to the gills with eager networkers learning a new API that was going to be integrated into every Cisco platform. OnePK didn’t survive a year. We remember hyped, but incomplete, products that never delivered on their promises. Beyond the failed promises, we’ve given countless sleep-hours to unexpected production outages caused by software bugs. Many of us can’t remember the last TAC case we opened whose root cause wasn’t a software bug.

At the same time, we freely acknowledge that we’ve built our careers around Cisco solutions and products. We want to believe that Cisco is entering a new era, that the leadership of Chuck Robbins is bringing about transformation inside of an organization that’s grown bloated with success.

With all that in mind, what can Cisco do to restore our confidence in the next era of software-defined solutions?

  1. Ship working products. More than anything, we must be able to trust that the products we buy will work as promised.
  2. Stay focused on delivering real solutions to real problems. Stop creating solutions that look for a problem. We have plenty of problems to solve.
  3. Integrate. Cisco’s integration of newly acquired Viptela will speak volumes.
  4. Unify the organization. Customers have grown weary of inter-BU competition and inconsistent product messaging. We expect all of Cisco to be on one team.

Even with my cautious optimism following Cisco Live, it will take a few years of successful product deployments to renew my confidence. As a follower of SD-WAN in general, and a fan of Viptela in particular, I cannot overstate the importance of the successful integration of Viptela’s full SD-WAN platform.

By next Cisco Live we should know if SD-Access is just another marketing campaign or if it’s becoming a new way of life for Cisco. We will know if SD-Access works. We will see how Cisco rolls Viptela into the fold and if they make the difficult decisions required to do so.

Until then, many of us will continue to watch and wait.

Are we really headed for a post SD-WAN World?

By Leave a Comment

Last week, in our Network Collective show about the Cisco’s purchase of Viptela, I suggested that many vendors will soon be headed toward a post-SD-WAN view of the world. Jordan laughed and several people have mentioned it to me since. Let me explain what I meant.

In the past two years, the SD-WAN space has exploded in both the number of vendors and in industry hype. Anyone who has had a product that could be related to the WAN has begun to market an SD-WAN solution. And because Cisco, the leading networking vendor, did not have an offering that could compete on its merits, vendors clambered to fill the space championed by Viptela, VeloCloud, and other leading SD-WAN companies.

Cisco’s acquisition changes this entire dynamic. The 800 pound gorilla has entered the marketplace and the shift will be seismic. Now, SD-WAN vendors are not merely competing on merits (or more accurately marketing buzzwords), they’re also competing with the considerable name recognition and industry inertia of Cisco.

Companies who saw opportunity to scoop up customers in the absence of a strong Cisco presence will begin to reconsider their marketing strategy. Sales and marketing teams will gather to determine the next set of industry buzzwords and SD-WAN will become a phase.

What about the technology of SD-WAN? Over time, we will simply call it the WAN. The features of built-in encryption, intelligent traffic steering, and carrier agnosticism will become part and parcel to every serious branch connectivity solution. Eventually, we will talk about the old days of the WAN with nostalgia, just like 9600 baud modems.

Are we there yet? Not hardly. But mark my words, the post-SD-WAN days will be here before we know it.