Pearl Diving - Just In Time Learning of Mature Coding Habits For a New Stack

I mused how we sometimes do the equivalent of Pearl Diving when taking on new skills. Its the idea of deep learning a stack of one or more new things while under the pressure of needing the end state code to reflect a maturity level significantly higher than your beginner expertise in that stack. It’s a variation on deep dive, but with the treasure of learning multiple new technologies. I have captured the details about Pearl Diving - and when you should use it (because you usually should not) - in this post.

During the effort described in Building a Best Practice CloudFormation Custom Resource Pattern, it was necessary to learn and create mature-ish code in the following stack: CloudFormation Custom Resources, Lambda (to back the custom resource), Python and Boto3.

Pearl Diving is also one of the Mission Impossible Code Diametric Spectrums: Have Seasoned Skills Through Experience and Practice <=AND=> Do Pearl Diving To Learn Skills Deeply and Quickly.

Tim Poffenbarger and I were discussing what more universal human learning experience is similar to Pearl Diving that might help with understanding the idea. We came up with learning to drive a car. You have to simultaneously become fairly well skilled in: 1) operating the vehicle, 2) following unstated rules of the road, as well as 3) posted signs and 4) watch for the dynamic hazards posed by other vehicles. If you learned on a stick shift you also had a mini-stack inside of operating the vehicle as you got used 1a) easing the clutch, 1b) while easing the throttle and 1c) knowing when to shift. Should I even mention adding the emergency / parking break in order to do a uphill start in a standard? In any case, learning to drive seems to be another required Pearl Diving learning expedition that many of us go through - and then get to go through again when teaching our young ones to drive ;)

Pearl Diving is, in my opinion, usually a learning antipattern because it combines learning new languages and their best practices right away. So it is a place that I feel reluctance to go. To that end, I have documented some of the inherent problem / solution heuristics that forced the Pearl Diving path on this project.

The pressure of needing it to be fairly mature from the start included:

• Small Code Required: A prime objective of this solution is to avoid breaking any size barriers that would force it to be more than one CloudFormation template. Those limits are 4KB for a Lambda function (when inline in a CloudFormation template) and the overall CloudFormation template size limit of 460KB (when sourced from S3). To leave room for future enhancements to the overall solution, a persistent “size minimizing” rule must be applied. To this end, some finer best practices for Python coding were regressed to keep the code small.
• Deep Stack Debugging: Another reason for Pearl Diving mode on this project was the call stack complexity required for final testing. It happens within Python, which is within Lambda, which is within CloudFormation. So this forced the issue of needing a mature implementation of exception handling and logging - because it is the only way to debug that deep. This investment paid off massively as soon as it was made. A previous post discussing this common problem described it as “5 miles out in a tunnel at the bottom of a mine shaft”: At the Coal Face: Code for Debugging Deep PowerShell Execution
• IaC Test Cycles Are Long: Personally I feel that Infrastructure as Code iteration cycles can be quite long compared to other types of testing and prototyping. Combined with how deep CloudFormation Custom Resources are in the call stack, this increases the desire to invest in a working pattern for future functions - especially with exception handling and logging for debugging.
• High Reuse Potential: The specific need for a CloudFormation template consumer to select their networking location is very recurrent and many solutions are overly complex in what they require the end user to provide for parameters. This sub-solution will likely be reused many times.
• Near-Future Feature Request Probability: The current solution allows one to pick the VPC, but then it uses all subnets of that VPC. I am pretty sure I will receive a feature request for either a) selecting specific subnets or b) selecting less than all the subnets of the target VPC. To these ends the implementation returns ordered lists of subnets and AZs so they can be predictably correlated and it returns the total subnets (index) - both of these are unused for this implementation at the current time. Some would argue this is an overly early optimization - I would argue it is a good medium term bet based on how many other of these types of templates I’ve worked on in the past, combined with the vast audience I hope can benefit from this automation.
• Crossing a Threshold of Implementation Complexity: Another, but important reason for Pearl Diving is to have a working example best practice template for new solutions. For a long time I have avoided the overhead of creating Custom Resources in CloudFormation - but the solution needs are only getting more complex and so having a pattern for creating all kinds of new Custom Resources is extremely valuable as it will accelerate future efforts substantially.

It is important to know that in past Pearl Diving expeditions I have frequently done a strategic quit on servicing one or more of the Pearl Diving pressures similar to those listed above for this solution. As with Mission Impossible Code heuristics, it seems to be the constraints of the heuristics during experimentation that leads to innovating a solution that fits more of them than expected - but it is hard to predict a head of time what ones will be significantly deoptimized or abandoned.

Pearl Diving Is Better And Faster With an Expert Coach Along

Pearl Diving can be done solo - but it takes much, much longer to work through unfamiliar syntax, object structures and handoffs than if you have a expert coach along! An individuals that help a Pearl Diver must have a coaching perspective because by definition the Pearl Diver will constantly ask questions that are out of their depth. Non-coaching experts will be frustrated with the lack of fundamental knowledge of the technologies in view. Coaching experts - especially with regard to learning code - tend to recognize the desire of someone who is proficient in high level best practices in other languages, will have a natural desire to acquire best practice knowledge quickly - especially the kind that is not easily revealed by search engines. For instance, in this Pearl Dive, it was very challenging to find examples that used boto3 “resources” rather than the more verbose “client” examples. It was hard to even come to an understanding that resources were more appropriate for my implementation needs than clients.

My expert coach in this effort was the Python Wizard Tim Poffenbarger. He patiently offered wisdom and examples to the long sets of questions I had. Huge thanks to Tim!