kurtcms.org

VMware VeloCloud SD-WAN Orchestrator API: Detect and Alert of WAN Anomaly

Posted on November 26, 2021 — 8 Minutes Read

Traditional IP network built on top of purpose-built equipment and point-to-point circuits is tried and trusted no less, yet being hardware-defined, the rigidity of its architecture disallows for agile and iterative development that taps into the programmability of software to rapidly adapt a product to the changing business needs of modem enterprises. As software is eating the world, and is disrupting nearly every industry and company, a Software Defined Network (SDN) was as such conceived, and a framework that decouples the control function of a network from the forwarding one, that would allow for a software-defined overlay network on top of the hardware-defined underlay infrastructure, with a management plane that responds to the real-time network status and the type of applications traversing in it, and a data plane that forwards frames and routes packets accordingly, was then proposed. Data centres and large campuses were first to adapt SDN in their backhaul networks, for the cost saving and the performance and resiliency benefits that come with its programmability. The same principle of separating the management and data planes was quickly applied to the Wide Area Network (WAN) as well for the same cost and performance benefits, and thus gave rise to Software-Defined WAN (SD-WAN) that is transport-agnostic, application-centric and that routes network traffic in accordance to the real-time network status and the service levels required by the respective applications.

For the lack of an open standard by the Internet Engineering Task Force (IETF) or other bodies, technology vendors took the opportunity to advance their own implementations with often proprietary software that are not interoperable with one another, and as these implementations were quickly adopted by enterprises for their cost savings and their performance and resiliency benefits over traditional hardware-defined networks, in a lucrative and rapidly growing industry, initiatives for an open standard were futile and were at most restricted to a collection of abstract service attributes that observes instead of leads the technological development. While details of these various vendor implementations vary, often they build from the ground up or upon other implementations of User Datagram Protocol (UDP) -based tunnelling protocol with business-grade encryption such as Advanced Encryption Standard (AES), to overcome some of the well-known design shortcomings with Transmission Control Protocol (TCP) i.e. the stringent congestion avoidance algorithm and the limited two-byte header value for a window size of up to 64 KB that is unfit for the high-bandwidth networks of today. These various vendor implementations often provide a set of Application Programmable Interface (API) as well for interacting with the existing network monitoring and security management systems in the enterprise network, that is compulsory for any brownfield deployment. Tapping into the programmability of the API, what follows will be a Python app that calls the VMware VeloCloud SD-WAN Orchestrator (VCO) API for an automated detection for WAN anomaly in the underlay infrastructure upon which the SD-WAN is built. The rest of the code is containerised with Docker Compose for a modular and cloud-native deployment that fits in any microservice architecture, and is shared on Github for reference and further development. With the Python app containerised with Docker Compose, deployment is as simple as:

Download a copy of the app;
Create the environment variables for the VCO authentication and for email notification, and modify the sampling durations and interval and the crontab if needed; and
Docker Compose or build and run the image manually to start the app, or alternatively run the Python script as a standalone service.

Git Clone

Download a copy of the app with git clone. Be sure to pass the --recurse-submodules argument to initialise and update each submodule in the repository.

$ git clone --recurse-submodules https://github.com/kurtcms/vco-api-wan-anomaly-alert /app/vco-api-wan-anomaly-alert/

Environment Variables

The app expects the hostname, the API token or the username and password for the VCO; as well as the SMTPS port number, SMTP server address, the alert receiver email address, the alert sender email address and password; as environment variables in a .env file in the same directory.

Should both the API token, and the username and password, for the VCO be present, the app will always use the API token.

Be sure to create the .env file.

$ nano /app/vco-api-wan-anomaly-alert/.env

And define the variables accordingly.

VCO_HOSTNAME = 'vco.managed-sdwan.com/'

# Either the API token
VCO_TOKEN = '(redacted)'

# Or the username and password
VCO_USERNAME = 'kurtcms'
VCO_PASSWORD = '(redacted)'

# For email notification
EMAIL_SSL_PORT = 465
EMAIL_SMTP_SERVER = 'smtp.kurtcms.org'
EMAIL_SENDER = '[email protected]'
EMAIL_RECEIVER = '[email protected]'
EMAIL_SENDER_PASSWORD = '(redacted)'

Sampling Durations and Interval

The intervals for the WAN quality metrics are 300 seconds i.e. 5 minutes and 3,600 seconds i.e. 60 minutes, for the present and historical baseline respectively, with a sampling interval of 300 seconds i.e. 5 minutes. All of these are passed to the respective function as argument at runtime and may be adjusted if needed.

$ nano /app/vco-api-wan-anomaly-alert/vco_api_wan_anomaly_alert.py

Modify the values as appropriate.

conn.detect_wan_anomaly(min_per_sample = 5,
    interval_sec_present = 300,
    interval_sec_hist = 3600)
'''
min_per_sample of 5 i.e. one sample every 5 minutes
interval_sec_present of 300 i.e. 5 minutes
interval_sec_hist of 3600 i.e. 60 minutes
'''

Crontab

By default the app is scheduled with cron to retrieve the WAN quality metrics every 5 minutes, with stdout and stderr redirected to the main process for Docker logs.

Modify the crontab if a different schedule is required.

$ nano /app/vco-api-wan-anomaly-alert/crontab

Docker Container

Packaged as a container, the app is a standalone, executable package that may be run on Docker Engine. Be sure to have Docker installed.

Docker Compose

With Docker Compose, the app may be provisioned with a single command.

Install Docker and Docker Compose with the Bash script that comes with app.

$ chmod +x /app/vco-api-wan-anomaly-alert/docker-compose/docker-compose.sh \
    && /app/vco-api-wan-anomaly-alert/docker-compose/docker-compose.sh

Start the containers with Docker Compose.

$ docker-compose -f /app/vco-api-wan-anomaly-alert/docker-compose.yml up -d

Stopping the container is as simple as a single command.

$ docker-compose -f /app/vco-api-wan-anomaly-alert/docker-compose.yml down

Build and Run

Otherwise the Docker image can also be built manually.

$ docker build -t vco_api_wan_anomaly_alert /app/vco-api-wan-anomaly-alert/

Run the image with Docker once it is ready.

$ docker run -it --rm --name vco_api_wan_anomaly_alert vco_api_wan_anomaly_alert

Standalone Python Script

Alternatively the vco_api_wan_anomaly_alert.py script may be deployed as a standalone service.

Dependencies

In which case be sure to install the following required libraries for the vco_api_main.py:

Install them with pip3:

$ pip3 install requests python-dotenv numpy pandas

Cron

The script may then be executed with a task scheduler such as cron that runs it once every 5 minutes for example.

$ (crontab -l; echo "*/5 * * * * /usr/bin/python3 /app/vco-api-wan-anomaly-alert/vco_api_wan_anomaly_alert.py") | crontab -

Email Alert

Email alert will be sent from EMAIL_SENDER to EMAIL_RECEIVER should an anomaly be found. The subject of the email will be WAN Anomaly Alert with the details of the anomaly in the email body.

Latency (download, ms) of WAN BT Business Broadband between Edge LDN-vVCE and its associated Gateway is found to be 100.0 and is 2 standard deviation(s) away from the mean of 75.0 and standard deviation of 10.0 of the 60.0 minute(s) before.
Jitter (download, ms) of WAN BT Business Broadband between Edge LDN-vVCE and its associated Gateway is found to be 5.0 and is 2 standard deviation(s) away from the mean of 2.0 and standard deviation of 1.0 of the 60.0 minute(s) before.
Packet Loss (download, %) of WAN BT Business Broadband between Edge LDN-vVCE and its associated Gateway is found to be 5.0 and is 2 standard deviation(s) away from the mean of 1.0 and standard deviation of 1.0 of the 60.0 minute(s) before.

Thoughts

Open source software on a vendor-neutral standard is the foundation for a collaborative development of a pace and scale beyond the confine of any individual organisation. It is a misunderstanding that open sourcing a software means surrendering the secret recipe of a product to the competitors and the rest of world, for there are countless examples of companies, such as Red Hat with its commercial and professionally-supported Red Hat Enterprise Linux and its free and community-maintained CentOS Linux, that are both derived from the same Fedora Linux source code that Red Hat sponsors, yet each with a different customer segment and a commercial and support model, that allows Red Hat to earn a fair profit while allowing it to contribute to the community, and vice versa. For historical reasons however, from the start SD-WAN did not embrace open source development nor a vendor-neutral standard, and it is likely the API by these various vendor implementations is the last step towards this direction than the first, especially considering the convergence of SD-WAN with cloud-native security into an integrated product that is delivered as a service from the cloud, in a service architecture connotated by Gartner as Secure Access Service Edge (SASE), that will be increasingly proprietary as vendors race ahead with their own and non-interoperable implementations in attempt to capture this lucrative enterprise networking and security market.