ESHARP.NET

Technology and life with Eyvonne Sharp

Cisco Buys Viptela: What’s next for SD-WAN?

By Leave a Comment

Join me tonight (5/2) over at #NetworkCollective as we discuss Cisco’s purchase of Viptela live at 7PM. We’ll opine on this industry development and what it means for Cisco, Viptela, SD-WAN, and you!

 

It’s no secret that I’m a big fan of Viptela’s SD-WAN technology.  I presented at their FutureWan SD-WAN Virtual Summit. I talked about my experience with their solution in my video series on SD-WAN. I’ve been an SD-WAN evangelist for some time.

At the same time, I’ve not been a big fan of Cisco’s competing iWan offering — a cobbled-together collection of technologies which have been marketed as SD-WAN.  Two months ago, I wrote about Cisco’s growing identity crisis as it relates to SD-WAN with Viptela’s solution in mind.

Now that Cisco has announced the purchase of Viptela for $610M in cash, what does this mean for the future of Cisco, Viptela, and SD-WAN?

First, the SD-WAN space is crowded.  With more than two dozen vendors, every one that touches the WAN markets an “SD-WAN” solution.  Now that Cisco has a viable SD-WAN offering, I expect the market to thin and for definitions to become more clear.

There are a few things Cisco must do.  First, they must tightly integrate Viptela’s software into their WAN offerings.  They cannot treat Viptela as a different business unit that runs independently like they did with Meraki.  Cisco’s entire WAN portfolio must run Viptela’s SD-WAN software and it must happen fast.

Second, they must embrace the culture of customer focus that has been so attractive at Viptela.  In recent years, Cisco has adopted a posture among their technical teams in which they tell their customers what they need instead of listening to customers. For example, in a meeting with high-level Cisco engineers, we had to justify running BGP on a WAN router.  We needed to dynamically distribute routing information from our sites — not an unlikely use case.

Most importantly, Cisco must continue the innovation that Viptela began.  Although the technology a huge step forward, there is much work yet to do.  The user interface of Viptela’s management console could use some improvement.  Better visibility into traffic flows and path selection would be helpful.  Viptela needs to refine their cloud deployment models and make it easier for customers to extend their infrastructure into the cloud.

I’ve been critical of Cisco in recent months, yet I choose to remain hopeful.  I’m hopeful that the meteoric rise of SD-WAN has shaken Cisco out of their complacency.  I’m hopeful that they will fully integrate Viptela technology into their WAN routing platforms.  And I’m hopeful that customers will see not only technology benefits, increased operational efficiency, and security, but also overall cost savings, from implementing Viptela’s (now Cisco’s) SD-WAN technologies for years into the future.

A girl can dream, can’t she?

6 Troubleshooting Lessons I Learned from Economic Psychology

By 1 Comment

My latest read,  The Undoing Project: A Friendship that Changed our Minds, explores the friendship of Daniel Kahneman and Amos Tversky — two psychologists whose study of irrational human economic choices led to a Nobel Prize in Economics. While the book discusses their friendship in detail, it also highlights their work which has influenced a wide array of disciplines including medicine, professional athletics, and and military recruitment.

I was particularly struck by their work on decision making. Through several studies, Kahneman and Tversky realized that human judgement is skewed by how questions are asked, how information is presented, and by how closely a scenario aligns with their existing biases. For example, they discovered when given 5 seconds to determine the product of 1 x 2 x 3 x 4 x 5 x 6 x 7 x 8, subjects consistently made a much smaller guess than when asked to determine the product of 8 x 7 x 6 x 5 x 4 x 3 x 2 x 1, although the actual product is exactly the same.

The implications of their work stuck a chord.

For those of us who work in technology, we pride ourselves in our rationalism. We deal in the territory of binary math, algorithms, and logic. Yet we often fail to step back and see ourselves as mere mortals. We acknowledge some disciplines suffer from irrationalism, but not ours. We are thinkers! Kahneman and Tversky, who studied doctors, graduate students, and economists, document the irrationality of human decision making. Even as technologists, we’re not immune from the human condition.

***

It doesn’t take long for an IT professional to find themselves on a bridge-line with several dozen people working to recover from a production outage. People ask questions, present theories, and there are usually more questioners than problem-solvers on the call. What can we learn from the discoveries of Tversky and Kahneman in this situations? What are strategies we can follow to solve the problem and get off that bridge line faster? Here are a few of my take-aways after reading about their work.

  1. Understand the problem. Skip this step at your peril. Ask: What is the problem? When did it start? What systems are affected? Has a trustworthy IT professional verified the problem? Ask questions to help determine scope: Is the problem local, regional, national, global? What systems are affected? Can the problem be replicated? Ask questions that have measurable answers.
  2. Gather real data. Don’t delve hours into an outage without doing your research. We don’t always have perfect metrics but there are several places we can look to help us determine what’s wrong:
    • Log data: Gather log data, but unless the error messages explicitly point to the issue at hand consider these a data point.
    • Testing tools: For network engineers these tools may be as simple as ping and trace route. Record your results.
    • Monitoring systems: Netflow data, snmp data, and alerting systems can be extremely helpful in deconstructing the issue.
    • Metrics and CLI output: Take a look at interface counters, show commands, and other device level data.
  3. Know your normal. Often, troubleshooting can be derailed for hours chasing log messages or incidents that are completely normal. For example, know how many sites typically experience an outage in a normal day. Review historical data to see if network utilization is particularly high, low, or normal.  Understand your environment well enough to know when something is normal and when it’s out of place.  In Kahneman and Tversky’s vernacular, this relates to the base-rate.
  4. Trust but verify. (The Dr. House corollary would be “Everybody lies.”) If someone on the bridge line suggests a course of action, ask questions — especially if they’re very confident. Why do you think that? What led you to that conclusion? Does the data we’re gathering support your theory? How can we test your theory? Don’t tweak configurations based on guesses that aren’t supported by data. You may introduce more problems that will muddy the data you’re collecting.
  5. Use existing models to conceptualize the problem. OSI is particularly helpful here. For example, if traceroute consistently works from end-to-end, you can be fairly confident layer 3 connectivity is solid and the problem is higher up in the stack. You can direct application support to investigate the application and you can check appliances like WAN-OP, Proxy, or SSL-Decryption that primarily work at the application layer.
  6. Avoid blame and emotionalism. I was on a call recently where the first words I heard a contributor say were, “We haven’t changed anything in months so it can’t possibly be <our problem>!” It’s fair and reasonable to mention that you have not made any changes as a data point for discovery. But it’s not helpful to deflect questions to avoid blame. The problem is what the problem is. Deflecting blame only slows troubleshooting. Eventually, the team will determine the problem and restore service (unless the issue spontaneously resolves leaving everyone scratching their heads). If the problem is not in your area of responsibility, the facts will prove it. If the problem is within your area of responsibility, you will only delay efforts to restore service by deflecting and covering your tracks. You cannot will the problem to be outside of your area of responsibility. Don’t hide information. Don’t try to shape the narrative. You’ll increase resolution time, make the call drag on, and gather a few enemies in the process.

A few additional thoughts:

  • Ask what changed, but don’t stay there too long. Outages are often caused by a change in state. But typically, if a change in state is known, the change will be reversed and operations will return to normal quickly. I’ve seen troubleshooting devolve into a drawn-out who-done-it before the problem is solved. If you use rational and systematic troubleshooting techniques you will find the problem. Don’t get marred in trying to solve the wrong problem.
  • Be wary of your initial judgement. Just because you had an outage last month that looked similar doesn’t mean this outage is the same scenario. Often, we will form an initial judgement of the situation and cling to it too long — even when data contradicts that judgement.

Tomes can be written on the troubleshooting process. For me, The Undoing Project was a reminder that human reasoning has its limits and that we need tools and processes to keep our brains on track.

Cisco’s Identity Crisis: Complexity, Pride, and SD-WAN

By 11 Comments

Our Cisco team has been reaching out to get feedback on our relationship with Cisco and its products — a healthy practice for any vendor. I’ve tried to be open, honest, and consistent in all our talks.

As I mentally review our conversations, I conclude I’ve been contradictory. On one hand, I’ve talked about how the industry is changing and Cisco’s products need to evolve in a software-defined marketplace. At the same time, I’ve decried their decision to move last-generation data center products to the campus portfolio to make way for newer technology.

My contradictions reveal that I haven’t articulated my true concerns. There’s a problem underneath these problems.

I’ve been watching presentations by Russ White on network architecture and complexity. He makes the point, and I’m paraphrasing, that many of our technological advances don’t solve complexity, they move complexity to a different place in the stack. Engineers and architects must determine if the complexity changes are worth the trade-offs. We must ask if added complexity solves the problem at hand without creating undo stress on the system.

With that in mind consider Cisco, a company in love with complexity. They’ve built their business making complex systems. Their culture breeds nerd knobs. They’ve built certification tracks — through which many network engineers have built their careers — to develop expert level understanding of their products.

At the same time, engineers operate in a culture where we believe configuration and operational complexity have inherent value. We unconsciously embrace the following logic: Networks are complex. One must be smart to understand networks. I understand networks. Therefore, I’m smart.

We extrapolate this logic and believe that complexity, for complexity’s sake, makes us superior. In truth, our pride has tied gordian knot with complexity and we don’t know how to unravel it.

Cisco has fallen into this trap. They don’t have a technology problem, they’re suffering an identity crisis.

Enter SD-WAN

SD-WAN is unravelling the knot. Cisco has insisted that the level of complexity we experience in managing our networks is inherent. If you want multi-path selection, prioritized traffic by application, and quality of service you have to make sacrifices. It’s hard of course, and barely possible. After all, we’re solving difficult problems. There are caveats, bugs, and boundary cases but there is no other way. It’s a pipe dream to expect simplicity in management and operation of a system so complex.

The best SD-WAN vendors are proving these assertions wrong. You can have multi-path selection, prioritized traffic by application, and quality of service with an operational efficiency previously unimagined.

Is there complexity in an SD-WAN enabled network? Sure! But strong centralized management tools significantly reduce configuration and operational complexity.

I’ve heard people say, “SD-WAN technologies are not new.”

Using this logic, you could argue that the iPhone wasn’t really something new. When the iPhone was first announced, we already had mobile phones, mp3 players, web browsers, digital cameras, and touch screens. Apple simply created a management interface and software platform to make all those technologies work well together in one small form factor. You could perform the same functions without an iPhone but you had to use 5 separate devices that weren’t designed to work as a unit. The iPhone married several technologies and sparked a movement, reimagined the internet, and enabled an entire generation to communicate in ways they couldn’t before.

Will SD-WAN have the same mass-market consumer enablement as the iPhone? No. But within the microcosm if network engineering, we may soon discover that SD-WAN has sparked its own movement. At the very least, SD-WAN vendors prove the challenges we face can be met in new ways. They’re forcing the stalwarts to sit up and take notice. They bring a promise that we no longer have to choose between unmanageable complexity and non-functional simplicity. In my book, that’s a win regardless of who wins the WAN.

Want more to think about?

Watch Engineer vs. Complexity, Russ White at NANOG

Network Field Day 14 and the Best Community Ever!

By 1 Comment

This week, I will participate as a first-time delegate for Networking Field Day 14. I’m excited, honored, and a bit intimidated by this great opportunity.

At Tech Field Day, industry vendors present intensely technical product information to network practitioners. The presentations are live-streamed, complete with unscripted question and answer sessions, and later archived over at the Tech Field Day web site. Delegates ask probing questions in a public forum and are often able to separate marketing from reality in ways that were impossible before social media.

Many of the #NFD14 delegates I’ve known through Twitter and have met at Cisco Live. I’ve listened to, and benefited from, Greg Ferro’s Packet Pushers podcast for years. Others will be new faces for me.

The Best Community Ever

Tech Field Day represents the best of the IT community. But I must say the networking community rises above any other, professional or non-professional, community which I’ve endeavored to be a part. They’re welcoming, inclusive, and downright helpful.

In many instances, I’ve reached out to a subject matter expert on Twitter to discuss a particular challenge I’ve faced. In one instance, I exchanged several emails about the benefits and downsides of ASA clustering – when to use it and when to implement a standard HA pair instead. In other cases, I’ve used the networking community to fact-check vendors. For example, Vendor A says Vendor B’s hardware falls down in high-load scenarios. Is that really true? The community has helped clarify.

Beyond these great traits, members of the networking community fulfill and break the stereotype of the “IT Guy” at the same time. Most of them fly their geek fly high — without apology. But at the same time, they’re witty, snarky, funny and more diverse than any stereotype would indicate. In my experience, their snark is lighthearted and rarely directed at one another. Vendors, executive leadership, corporate processes, and horrible applications bear the brunt of community criticism –- and in many instances rightfully so. Anyone, at any skill level, with a legitimate desire to learn their craft and grow as a network practitioner, will be welcomed.

So, if you tune into the live stream of Networking Field Day 14 and have a question, reach out! Use the #NFD14 hash tag or mention one of us in your tweets. I’ve done it in the past. Through the delegates, you’ll have direct access to vendors in ways you may not otherwise enjoy. Take advantage of it. We’re one big community with the same problems and challenges. If you have a question, I’m sure others do too. I hope you’ll join us for the live stream sessions next week. I’ll blog as I’m able but for instant (and often stream-of-consciousness) comments, watch my Twitter stream. We’ll enjoy experiencing networking awesomeness together!

This post wouldn’t be complete without a shout-out to those in the network community who have reached out to me personally, helped me be a part, and encouraged my participation even when I felt clueless and bumbling. Follow these folks on Twitter, you won’t regret it.

Tom Hollingsworth (@networkingnerd)
Amy Renee (@amyengineer)
Chris Church (@layer_3)
Ethan Banks (@ecbanks)
Greg Ferro (@etherealmind)
Scott McDermitt (@ScottM32768)

 

And, check out the Network Field Day 14 delegate page to follow all the #NFD14 delegates.

Identity Matters, ISE and the Future of Networking

By 5 Comments

The more I work with Cisco ISE (Identity Services Engine), the more possibilities I see. In my opinion, it is the most exciting Cisco product since UCS. It’s the only product I’ve seen that provides such a high level of flexibility, control, and centralized configuration for network edge access.

With ISE, you can authenticate, profile, and posture any wired or wireless device that connects to your network. Policy is configured in a centralized controller and pushed to clients when they connect to the network. Based on a myriad of identity and profiling criteria, you can apply a vlan, push a DACL, or inject a Security Group Tag for each client. Today, all of that information is used only for security purposes, but think about the possibilities!

What if every packet on your network is tagged with an identifier based on an amalgam of criteria including: user identity, device type, AD group, application flow, etc? Consider the opportunities if each packet is proactively encoded with a handle that distinguishes it based on complex criteria. What if this criteria is centrally managed and abstracted into a structure that allows you to make quick decisions in hardware? It’s reasonable to conclude that not only security decisions, but routing, QOS, and optimization could be configured based on this identity tag in the packet. And, all of this policy can be pushed from a centralized controller into a data plane of your network.

Granted, ISE doesn’t do this today. It provides authentication, authorization, profiling, and posture services and is solely a security tool. However, the potential power of the platform is limitless.

Of course, ISE is a proprietary Cisco solution that only works well in an all Cisco environment. Aside from standard radius authentication, all of the great ISE features are Cisco only. However, if the solution were more open and interoperable with other networking vendors, it could become a huge platform to improve the entire networking industry.

For Cisco, ISE should be a huge component to their long-term strategy for centralized network control, automation, and security. For a vendor that receives a lot of flack that they’re not a software company, ISE is a great software product.