Mission Impossible Code Part 2: Extreme Multilingual IaC (via Preflight TCP Connect Testing a List of Endpoints in Both Bash and PowerShell)

It is not possible for me to count the number of times this code has saved me support calls because I never get those calls ;) A huge part of my work is to build DevOps IaC automation code as tools in a company that runs around 50% Windows and %50% Linux across their many SaaS software stacks.

One of the main types of IaC my team builds is deployment automation for DevOps agents that are designed to run on any of the 10’s of thousands of instances at the company - agents for things like vulnerability scanning, malware scanning, log aggregation and monitoring. Generally these agents are wiring up to an internal or external cloud tenant environment for reporting and/or administration.

Everyday at my job I learn of a new environment I’ve never heard of before that someone is trying to run my team’s code in. Frequently the environment setup is at fault when these DevOps agents error out on their tenant registration calls. After way too many escalations that resulted in the discovery that the environment is at fault - I decided we need to preflight check the tenant URLs we would need to connect to and report failures in logging so that tooling users could easily distinguish when their environment was not allowing endpoint access.

Another common case for endpoint verification is when code depends on public or external package management endpoints for things like yum or chocolatey packages. However, the approach is solid for endpoints of all type whether public or private, local or remote.

If you take a look at a lot of your automation code it may make fundamental assumptions about available endpoints and if it will run in environments that are out of your control, endpoint connectivity validation will save you boatloads of support time :)

Mission Impossible Priority - while Mission Impossible Coding applies to anything that requires ruggedness, in toolmaking the return on investment is much higher since, by definition, tools are run in a huge variety of environments they were not tested for in their development cycle.

Constructed Simplicity (Conciseness) is The Tip of An Iceberg

While epitomized by the Blaise Pascal quote “I have made this longer than usual because I have not had the time to make it shorter.”, I find that the process of creating concise, simple observations and solutions is complicated and sometimes complex.

Creating concise designs and solutions is a deep passion of mine. However, there is a frustration that is quick on the heals of creating something that is concise. In my line of work, at some point, you usually have to justify your final work. That is the point at which the rest of the iceberg of thinking that backs the concise tip comes to the fore. Frequently the reaction is that it can’t possibly be that involved or that you are spinning up reasons on the fly to simply bolster an idea or solution that was arrived at haphazardly.

Why bother addressing this sentiment? Because concise, mission impossible style, solutions can easily be criticized as lacking sophistication in their implementation. But much like a Mark Twain quote - that ruddy external appearance belies a hard wrought balance of many interrelated tradeoffs to get to a simple solution.

Said another way, solutions that are earnestly designed for conciseness can be rewarded by a perception of being the opposite of what they are - simplistic, backed with little or no thought.

This Mission Impossible Coding series exposes the submerged methodological icebergs below the waterline of the visible and concise solutions it attempts to arrive at.

Architecture Heuristics: Requirements, Constraints, Desirements, Serendipities, Applicability, Limitations and Alternatives

The following list demonstrates the Architectural thrust of the solution. This approach is intended to be pure to simplicity of operation and maintenance, rather than purity of a language or framework or development methodology. It is also intended to have the least possible dependencies. It’s an approach I call “Mission Impossible Coding” because it enables the code to get it’s job done no matter what.

• Requirement: (Satisfied) Leverage TCP connect for end point validation, not ping, because:
  • It should always be possible while ping and/or UDP are frequently blocked.
  • It verifies end-to-end service connectivity right to an individual port.
• Requirement: (Satisfied) Allow input parameters to be passed as a simple string - this avoid problems when the parameter must be passed through a complex stack of languages because we aren’t trying to pass a complex data type.
• Requirement: (Satisfied) Allow multiple endpoints to be checked with one parameter.
• Requirement: (Satisfied) Keep the same data type and formatting for the parameter so that it can be passed to either Windows or Linux without transformations.
• Requirement: (Satisfied) Always informationally log what parameters are being processed before attempting to use them - this helps debugging when parameters are unwittingly being overridden or if they are malformed by a handoff somewhere above in the stack.
• Desirement: (Satisfied) Use similar code structure for Bash and PowerShell to facilitate using the scripts as a cross-language learning aid.
• Requirement: (Satisfied) Use methods that have the broadest shell version and os version support.
  • Requirement: (Satisfied) Use methods that do not require code or utilities to be installed - more important in the face of container-minimalized OS configurations.
  • Desirement: (Satisfied) Use an argument format that does not require a ton of additional code to parse in Bash (hence this solution uses a space for key value pair delimiter).
  • Serendipity: (Discovered) PowerShell version can run on PowerShell Core in native Linux, Bash version can run in Windows Services for Linux (WSL) on Windows.
• Constraint: (Satisfied) Avoid using language native function definitions to allow implementers to use their own coding styles or standards for where the loop should be and whether a function should emit informational messaging and other details.

Expansion on Architectural Heuristics

Mission Impossible Pattern Philosophy

Mission Impossible Code samples are intended to be both 1) usable directly for production use and 2) a top notch pattern to use for your own innovation. More details on why are in the post Back to Basics: Testable Reference Pattern Manifesto (With Testable Sample Code)

No Ping Because You Know Ping

The need to avoid reliance on ping as a connectivity verification tool is relatively obvious as it (and UDP traffic) is understandably blocked in so many circumstances. However, doing a TCP connect attempt also verifies the entire end-to-end conversation right down the specific required ports and includes verification that a return conversation is possible.

Reduced or Eliminated Runtime Requirements

Common approaches to the problem of verifing connectivity on Linux include using nc or telnet - both of which are not present on many minimalistic distros. On Windows PowerShell 4.x and later Test-NetConnection provides TCP connectivity testing - but is not available on PowerShell Core nor older versions of PowerShell.

All of these are avoided by the solution code in this article so that it can run in the broadest number of contexts. As an unintended positive consequent, it turns out that the PowerShell code works on Windows and Linux. The Bash version also works on Windows Services for Linux running under Windows.

Mission Impossible Priority - The unintended positive consequences of reducing or eliminating dependencies are frequent and delightful, like biting into a jelly donut for the first time ;) Dependency reduction / elimination is usually worth it.

First Class Design Priority: Parameters Get Passed Through Complex, Multi-layer Stacks With Value Overrides Allowed At Many Levels

It is important to consider that shell code must be called by something else to run - sometimes by a huge, multi-layered stack of “something else”. In complex automation stacks, each hand-off between layers represents risks for data types that are more complex than a string.

This code could have been built to take a complex data type like a hashtable or array. In my experience this generally leads to massive rabbit holes of effort to pass the data between layers of automation infrastructure. Since each language handles the syntax of complex data types differently, such data may need to be escaped or reformatted between layers due to parsing constraints. Many times specific layers don’t have complex data types or it takes special transformations to pass into the next layer.

To further frustrate the efforts, it can be very difficult to get debugging visibility to the hand-off between these processing environments. On top of this lack of visibility, most parsers don’t give direct errors on the problem. In fact, many times your parameter, now malformed, is not detected until you try to use it.

The string data type is the most elemental and universal. By using a simple, structured string - the pain of complex type passing can be avoided and parsing is handled by the shell code once the data is received into the shell environment. Not only does this allow passing through many layers of automation with ease, it allows Windows and Linux to share the exact same format. Which in my case, allows the same argument to be seamlessly passed to the Windows or Linux branch of the code at the appropriate point in the many layered automation stack.

Mission Impossible Priority - how many times have you seen yourself or others stay idealistically fixated on using a data type to the point of wasting many hours. Here is a key place where a dedication to “Mission Impossible Coding” dictates a pragmatic override of idealism to what will consistently work in the broadest number of scenarios. Idealism is a secondary to pragmatism.

By the way, the format I have chosen here may not be friendly to your specific stack of parameter handoffs - but you can see that it uses a space to delimit key value pairs and equals sign to delimit key from value - feel free to adjust those in a way that is compatible with your stack. In fact, if anyone has devised universal delimiters that pass through most known layers without incident - I’d be interested on a comment on this article.

Logging From A Toolmakers Perspective

A healthy orientation to verbose informational logging is a toolmakers friend as it encourages development users to self-diagnose and self-resolve problems they would otherwise reach out for support on.

A final point to consider with parameters is to always 1) informationally log them 2) before attempting to use them. Informationally means don’t only log errors - this handles use cases where the bottom level automation is receiving working parameters, but they are not the correct.

The complex stacks our parameters pass through frequently allow parameter overrides at many of these levels - informationally logging them allows a super quick find of an unwitting parameter override. Once again, more critical if your automation is tooling used by others since you don’t have control over the parameter stack. The reason for logging before using them is so that any error handling that might happen does not negate your ability to report the parameters in use at the time.

Mission Impossible Priority informational logging of parameters before API calls: 1. Always communicate verbosely with your support team, 2. When you are going to die, write a note who dun it to you.

Built-in List Processing From The Start

This code was built from day 1 with processing a list in mind. This design heuristic is stolen directly from PowerShell where CMDLet design makes it exceedingly simple to process lists of things that are pipelined in - so simple, there isn’t much reason not to do it. No matter what language you prefer, how many times have you had to refactor a solution to process a list when you originally built it for a single value. Hard to list them all isn’t it?

Timeout is Required

Initially I didn’t bother with timeout functionality in the PowerShell version because the default was bearable. However, when it was run on Azure Cloud Shell - it was unbearable. A strong commitment to pre-mission testing was the only reason this came to the fore.

Mission Impossible Priority - relentless pre-mission testing of all alternatives and contingencies is standard operating behavior.

Mission Impossible Code Bonus: Minimal, Universal Logging

Logging is fundamental to Mission Impossible Coding. I also have Bash and PowerShell code designed with the same eye to minimal, universal reuse. You can implement it with this code by retrieving it from here: MICode-MinimalUniversal-Logging

The Code

Here is the Bash - last url will purposely fail.

urlportpairlist="outlook.com=80 google.com=80 test.com=442"
failurecount=0
for urlportpair in $urlportpairlist; do
  set -- $(echo $urlportpair | tr '=' ' ') ; url=$1 ; port=$2
  echo "TCP Test of $url on $port"
  timeout 3 bash -c "cat < /dev/null > /dev/tcp/$url/$port"
  if [ "$?" -ne 0 ]; then
    echo "  Connection to $url on port $port failed"
    ((failurecount++))
  else
    echo "  Connection to $url on port $port succeeded"
  fi
done

if [ $failurecount -gt 0 ]; then
 echo "$failurecount tcp connect tests failed."
 exit $failurecount
fi

Here is the PowerShell - last url will purposely fail.

$UrlPortPairList="outlook.com=80 google.com=80 test.com=442"
$FailureCount=0 ; $ConnectTimeoutMS = '3000'
foreach ($UrlPortPair in $UrlPortPairList.split(' '))
{
  $array=$UrlPortPair.split('='); $url=$array[0]; $port=$array[1]
  write-host "TCP Test of $url on $port"
  $ErrorActionPreference = 'SilentlyContinue'
  $conntest = (new-object net.sockets.tcpclient).BeginConnect($url,$port,$null,$null)
  $conntestwait = $conntest.AsyncWaitHandle.WaitOne($ConnectTimeoutMS,$False)
  if (!$conntestwait)
  { write-host "  Connection to $url on port $port failed"
    $conntest.close()
    $FailureCount++
  }
  else
  { write-host "  Connection to $url on port $port succeeded" }
}
If ($FailureCount -gt 0)
{ write-host "$FailureCount tcp connect tests failed."
  Exit $FailureCount
}

Tested On

PowerShell Version (Windows and Linux)

• PowerShell 5.1 on Windows 10
• PowerShell Core 7.0.0.2 on Windows 10
• PowerShell Core 6.2.1 on Amazon Linux 2 in Docker on Windows 10
• PowerShell Core 6.2.2 on LinuxMint 19.1 On A Laptop
• PowerShell on Azure CloudShell

Bash Version (Windows and Linux)

• Ubuntu 18.04.1 LTS in Windows Services for Linux (WSL) on Windows 10
• Amazon Linux 2 in Docker on Windows 10
• Bash on LinuxMint 19.1 On A Laptop
• Bash on Azure CloudShell

Code For This Article

MICode-MinimalUniversal-TCPConnectPreflightCheck