Configure Over Code & Declarative Programming

Hello friends

I have some good news to share. We crossed 600 members in Godspeed's Discord channel recently! :D Also, I have an important and cool news coming soon. Stay tuned for that. In meantime, let us more forward with our learning path towards 10X engineering.

Before we proceed, please ask yourself a question.

Who are you writing code for?

The codebase you author is not for you, but for the successors who work on the project after you are gone, and the organisation you work for.

In the previous post we looked into the 8 important guardrails, and the first one called Schema Driven Development (SDD) and Single Source of Truth (SST). In this post we will look into Configure over Code (CoC) and Declarative Programming (DP). Example of CoC and DP are actually reflected in the examples shared in the previous SDD and SST blog as well. SDD and SST are also examples of configure over code and declarative development. To get most out of this post, make sure to also read the previous entry on SST and SDD too!

Benefits of Configure over code and Declarative programming

By following CoC and DP you are giving your future maintainers and organisation more efficiency & effectiveness because they get

• More maintainability & prevention of technical debt

• More flexibility and agility (Via separation of concerns (aka decoupling), and single source of truth principle)

• More reusability

• More readability

• Less developer switching cost and experience requirement

• Less probability of errors (BIR)

• Less time to find root causes of errors and and apply fixes (MTTR)

How do these benefits happen?

By following some best practices like SDD, SST, CoC, DP etc, every org developer focuses on the tip of the iceberg for creation and maintenance of projects, and not the whole iceberg.

In the picture shown below, the part of iceberg which is below water (i.e. 90% of iceberg) is typically the not-actual-business-logic. Your actual business logic is 10% only! Whether you are a student (learning concepts or developing hobby projects), or a startup (with one or many services) or an enterprise (with many teams and many services), and you develop your applications using the traditional approach with bare Nodejs and Express server, trust me on this -

In traditional approach you will be working on the whole iceberg and adding 10X extra effort in developing, debugging, altering or maintaining the software.

Watch a video: There is a 5 minute intro by me on Configure over Code & Declarative Programming as a first class citizen and guardrail of Godspeed's meta-framework

Note: I am using Godspeed, Springboot etc as just examples. You can apply, develop and use these concepts anywhere.

Related and important concepts

Before we dive deep into CoC and DP, lets understand other related concepts along with these two

Convention over code

"Convention over code" prioritizes adhering to established conventions and patterns rather than writing explicit configuration or logic, promoting simplicity and consistency in software development.

Example in the Java world: In frameworks like Hibernate, by following naming conventions for entities and their fields (e.g., class names match table names, field names match column names), developers can avoid specifying explicit mappings, reducing boilerplate code and configuration.

// In the Java Hibernate world, the name of the class corresponds 
// to the table names and the properties to the field name
@Entity
public class Product { //Name of the table in database
    @Id 
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    private Long id; //Convention over code: the id column
    private String name; //Convention over code: the name column
    private double price; //Convention over code: the price column

    // Constructors, getters, and setters
    // Omitted for brevity
}

Note: Every language ecosystem behaves differently on this principle. While Java ecosystem generally follows convention over code approach, in the Javascript world, there is barely any consensus on conventions. For ex. location of transpiled code could be in dist, build or server directory. Or the main entrypoint of the program could lie in app.js, main.js or index.js` . The database schemas when defined using Mongoose JS follow similar approach to Hibernate, while Prisma JS follows its own syntax for model declaration.

Code over configuration

"Code over configuration" emphasises writing explicit code to configure and customise the behaviour of a system, favouring flexibility and fine-grained control over relying on predefined conventions or configuration files.

An example about setting up database services

public class AppConfig {
    public static void main(String[] args) {
        //Initialize the DB connection from within the code
        DatabaseConnection connection = 
            new DatabaseConnection("jdbc:mysql://localhost:3306/mydb", "username", "password");
        // Explicitly initialise different (table related)  
        // services with the database connection instance
        UserService userService = new UserService(connection);
        ProductService userService = new ProductService(connection);
        // Configure additional services, repositories, and components as needed
        // Explicitly set up dependencies and configurations in code
    }
}

An example from ExpressJS setup

Can you check this example and figure out which concerns could have been separated as configurations or declarations? Hint: Check the JWT setup, Schema validations (input and output).

// app.js
const express = require('express');
const jwt = require('jsonwebtoken');
const bodyParser = require('body-parser');
const { validateInput, validateOutput } = require('./middleware/validationMiddleware');

const app = express();

// Secret key for JWT token
const secretKey = 'your_secret_key';

// Middleware for parsing JSON bodies
app.use(bodyParser.json());

// Middleware for JWT authentication
function authenticateJWT(req, res, next) {
    const token = req.headers.authorization;

    if (token) {
        jwt.verify(token, secretKey, (err, user) => {
            if (err) {
                return res.status(403).json({ message: 'Invalid token' });
            }
            req.user = user;
            next();
        });
    } else {
        res.status(401).json({ message: 'Token is required' });
    }
}

// Sample route with JWT authentication middleware and input/output validation middleware
app.post('/api/users', authenticateJWT, validateInput, (req, res) => {
    // Logic to handle user creation
    // Assuming req.body contains user data
    const user = req.body;
    // Validate user data against hard-coded schema for input validation
    if (!user.name || !user.email || !user.password) {
        return res.status(400).json({ message: 'Invalid user data' });
    }
    // Perform database operations to create user
    // Return response
    res.status(201).json({ message: 'User created successfully' });
});

// Sample middleware for output validation
app.use((req, res, next) => {
    // Validate response data against hard-coded schema for output validation
    if (res.statusCode === 200 && res.body && res.body.message) {
        // Assuming the expected response format is { message: 'Some message' }
        if (typeof res.body.message !== 'string') {
            return res.status(500).json({ message: 'Internal server error' });
        }
    }
    next();
});

// Start server
const PORT = process.env.PORT || 3000;
app.listen(PORT, () => {
    console.log(`Server is running on port ${PORT}`);
});

Configuration over Code

"Configure over code" emphasizes storing configuration settings externally rather than hard-coding them into the source code, enhancing flexibility and maintainability.

Example in the Java world: In a Spring Boot application, database connection properties are stored in an external application.properties or application.yml file rather than being hardcoded in the Java code.

In the previous example of ExpressJS setup, could you figure out the concerns you could have separated? Check how the same is done in Godspeed.

ExpressJS Setup

Configuring the middleware

# Configuration of http service using Express
type: express
port: 3000

# limits of sile size and body
request_body_limit: 20000 #bytes
file_size_limit: 200000 #bytes

#jwt or oauth2 settings to run by default on every event
authn:
  jwt: # best practice is to store secrets in environment variables and not hardcode here.
    secretOrKey: <%config.jwt.secretOrKey%>
    audience: <%config.jwt.audience%>
    issuer: <%config.jwt.issuer%>

# Further configurations ommitted for brevity

Endpoint setup

You will see here separation of concern: Express setup is decoupled with endpoint setup including urls, validations, authn and authz.

http.post./health:
  summary: Health check of the service 
  description: Check whether the service running
  fn: health.check #event handler
  authn: false #disable JWT auth on this endpoint
  authz: auth.custom_authz #apply custom authorization on this endpoint
  body: #requestBody as per swagger spec (automatic validation)
    content:
      application/json:
        schema:
          type: object
          required: ['name']
          properties:
            name:
              type: string
              example: Godspeed

  responses: #swagger spec for automatic validation of response
    200:
      content:
        application/text:
          schema:
            type: string

Read the previous blog on schema driven development or check these detailed explainer videos of working with eventsources and events in Godspeed.

Declarative Programming

Declarative programming is a programming paradigm that emphasizes expressing the desired outcome or behavior of a program without specifying the detailed control flow or algorithmic steps.

Imperative approach

const numbers = [1, 2, 3, 4, 5, 6];
const evenNumbers = [];
for (let i = 0; i < numbers.length; i++) {
    if (numbers[i] % 2 === 0) {
        evenNumbers.push(numbers[i]);
    }
}
console.log(evenNumbers); // Output: [2, 4, 6]

Declarative approach

const numbers = [1, 2, 3, 4, 5, 6];
const evenNumbers = numbers.filter(num => num % 2 === 0);
console.log(evenNumbers); // Output: [2, 4, 6]

Even more declarative approach

import {even} from './utils';
const numbers = [1, 2, 3, 4, 5, 6];
const evenNumbers = even(numbers);
console.log(evenNumbers); // Output: [2, 4, 6]

A complex use case with a datastore

An example from Elasticgraph setup. Elasticgraph is a highly declarative ORM over Elasticsearch developed by yours truly (and his apprentices) when working for the office of His Holiness the 14th Dalai Lama :-)

# Automatically calculate data of some 
# relationships from other relationships
[conference]
speakers = '+talks.speaker' #As soon as a talk is linked to a conferece, or an already linked talk gets linked to a speaker, *the talk’s speaker is also linked to the conference as one of its speakers, if not already linked before*. Vice versa happens if the talk is unlinked to its speaker, or the talk is removed from the conference
topics = '+talks.topics' #As soon as a talk is linked to an conference, or a topic is set to an already linked talk, the talk’s topic is also added to the conference as one of its topics, if not already there. Vice versa happens if the talk is unliked to the conference, or the topic is removed from the talk.
[‘person’]
grandChildren = +‘children.children’ #Whenever a person’s child gets a new child, the new child gets added to the person’s grandchildren
[‘folder’]
fileTypes = ‘+childFolders.fileTypes +childFiles.type’ #Calculate union of all file types existing in the entire folder tree (recursively). Anytime, any file gets added to any child folder in this tree, the type of that file gets unioned with the list of fileTypes of that child folder, and all its parent folders up in the hierarchy.

Abstractions

Abstractions can be called the overarching principle or higher principle or superset of declarative programming and configure over code approach. Abstraction allows us to work with objects at a higher level of conceptualisation while encapsulating the specific details of each (lower level) implementation's behaviour. For ex. Inheritance in Object Oriented Programming or the function even() shown in the configure over code example above.

Abstractions are about A. Finding repeating patterns B. Expressing the repeatable patterns in one place (as single source of truth) and C. Reusing (or implementing) them in multiple places.

Abstraction of Eventsource in Godspeed

// Express, Fastify, Apollo Graphql, Cron, Kafka etc
// In Godspeed, all event sources implement GSEventSource abstraction
export abstract class GSEventSource {
  config: PlainObject;

  client: false | PlainObject;

  datasources: PlainObject;

  constructor(config: PlainObject, datasources: PlainObject) {
    this.config = config;
    this.client = false;
    this.datasources = datasources;
  };

  public async init() { 
    //Initialise the client for ex. Express service
    this.client = await this.initClient();
  }
  // Initialise the event source service or client with middleware
  protected abstract initClient(): Promise<PlainObject>;
  // Register listening and processing of events based on 
  // the event shcemas shown above
  abstract subscribeToEvent(
    eventKey: string,
    eventConfig: PlainObject,
    eventHandler: (event: GSCloudEvent, eventConfig: PlainObject) => Promise<GSStatus>,
    event?: PlainObject
  ): Promise<void>
}

Example EventSource Implementation

Here is implement of Cron eventsource in Godspeed

export default class EventSource extends GSEventSource {
protected initClient(): Promise<PlainObject> {
    return Promise.resolve(cron);
}
subscribeToEvent(
    eventKey: string,
    eventConfig: PlainObject,
    processEvent: (
    event: GSCloudEvent,
    eventConfig: PlainObject
    ) => Promise<GSStatus>
): Promise<void> {
    let [,schedule, timezone] = eventKey.split(".");
    let client = this.client;
    client.schedule(
          schedule,
          async () => {
            const event = new GSCloudEvent...;
            await processEvent(event, eventConfig);
            return Promise.resolve()
          },
          {
            timezone,
          }
        );
    return Promise.resolve();
}

Abstraction of event definition independent of the cron library used

Notice how cron events' setup is decoupled from the use of node-cron library

# event for Shedule a task for every minute.
cron.* * * * *.Asia/Kolkata: //event key
  fn: every_minute

Abstraction of event handler business logic from the event source libraries

In the example shown below, notice below how the business logic of a REST or Graphql API endpoint or CRON event or Kafka event is decoupled from

1. The actual event source mechanism (whether sync - REST, graphql, gRpc. Or async - Cron, message Bus or websocket)

2. The eventsource libraries (Express, Fastify, Apollo, Kafka, Salesforce, socket library etc)

import { GSContext, PlainObject } from "@godspeedsystems/core";

export default function (ctx: GSContext, args: PlainObject) {
    //GSContext contains inputs of the event as a pure JSON object, 
    // irrespective of the eventsource.
    // Inputs has body, headers, params, query, user JSON objects
    // As output (response), you return the code and headers 
    // with the data. 
    // The respective eventsource plugin does the job of 
    // deserializing the input to JSON and serializing the JSON 
    // response as per its SDK and protocol. 
    // This means the event handler code is independent of 
    // the event source used. (Separation of concern)
    return {
        data: 'Its working! ' + ctx.inputs?.data?.body.name,
        code: 200,
        success: true,
        headers: {
            custom_response_header: 'something'
        }
    }
}

The above example shows decoupling of functions with eventsource related code. Event ElysiaJS a Bunjs based web framework handles pure functions similarly to Godspeed's approach. (FYI You could add ElysisJS as an eventsource in Godspeed's meta-framework too.)

When to apply CoC or DP approaches?

When writing a new application, feature or service think about

1. Which configuration options are not going to change or change extremely rarely in this project during its lifetime? (For ex. port of your REST service or your JWT setup)

2. Which lines of code or implementation patterns are repeating or may repeat (even two times)? (Remember the CoC examples above where validation boilerplate code was repeated)

3. Is there some logic which could be simply declared instead of coded? (Remember Java Hibernate or Elasticgraph configurations shown above, or the automatic validations of the event schemas)

4. Are there any eventsource or datasource libraries merging with your business logic? Can you decouple your business logic from the libraries you use? (Remember eventsource, event and event handler examples from above)

5. Can you use single source of truth to generate remaining things like documentation (Postman, Swagger), CRUD APIs, integration tests etc from the source, instead of handwriting them or using AI to generate them? Even AI would mostly go wrong or be inefficient without using SST.

6. How can you make it easier for new maintainers to onboard this project? It is easier and faster for anyone (even AI) to fix things through configurations than finding a fix in between thousands of lines of code.

7. Which interfaces or declarations can you give to your future maintainers

Conclusion

Repeating again - 10X engineers don't write code for themselves. They write for the future maintainers of their projects.

Whether an AI or a human engineer is at work, by following we make their work easier & more efficient, from both project creation and maintenance perspective. Stay tuned for more on best practices on 10X engineering. If you like this blog, do share and recommend to your peers!

References

1. Blog: Schema Driven Development and Single Source of Truth

2. Explainer video: Eventsources in Godspeed with CoC approach

3. Explainer video: Events in Godspeed with SDD, SST and CoC approach