Refactoring Compas code-gen: part 3

Recently we started working on refactoring the code generators of Compas.js. Compas.js is a framework based on a structure providing generated routers, validators, typed api clients and more.

Code-gen processs

The code-gen package is built like any piece of software in three main steps:

• Acquiring input
• Transforming the input
• Output the result

@compas/code-gen is no different. The input is defined with the TypeCreator in forms like T.uuid() or T.array() and passed to Generator#add. The generator then does validation, processes the input and builds in memory strings representing source files. And those files are then written out to disk, ready to be used in your project. In this post we will check out how Compas.js has created those source files and what is awaiting in this refactor.

Status quo

Through the almost three years that Compas has code-generation, I have thought many times about what would be the optimal way of creating these source files. As a result of that process we now have a code-generator with various different ways of, what boils down to, string concatenation.

There are a lot of 3's in these posts about refactoring Compas code-gen. I did not expect that, and just now linked it to the Rule of three.

Templating, a form of reusable string concatenation (at least that's the goal), should be a solved problem. Take the Mustache package for your favorite language and of you go! However, the complexity that would then end up in those templates, just does not feel good. Which then is potentially repeated for all targets that we have currently and then again for each new target we add in the future.

Where we should go

All that experimentation boils down to a few requirements or rules in the to be designed new way of templating. Logic-less templates is probably the most important part that we should start doing. Inline conditionals, loops or even functions inside templates shift a lot of responsibilities from the caller to the template. Inside these templates you don't have the full power of the language where the code-generator is written in, like in our case JavaScript. This results in the following scenario cherry-picked from the existing code-base.

// Inside the CRUD generator

export function someCrudGeneratorFunction(data) {
  return `
    ${/* more logic before */}

    ${data.inlineRelations.length > 0 ? "await Promise.all([" : ""}
    ${partialAsString(data.inlineRelations.map(
       (it) =>
         `queries.${it.entityName}Delete(sql),`,
     ))}
    ${data.inlineRelations.length > 0 ? "])" : ""}

    ${/* more logic before */}
  `;
}

This could more easily be written as:

export function someCrudGeneratorFunction(data) {
  let result = "";

  if (data.inlineRelations.length > 0) {
    // We can even add a comment here why we wrap this in a `Promise.all`
    result += `await Promise.all([`;
  }

  for (const inlineRelation of data.inlineRelations) {
     result += `queries.${inlineRelation.entityName}Delete(sql),`
  }

  if (data.inlineRelations.length > 0) {
    result += `]);`;
  }
}

Sure it is a few lines longer, but now we have all the flexibility to add comments and easily extract variables for repeated conditions in if-statements through the power of an IDE.

Output consistency is also an important item on my wishlist. The formatting of generated doc-blocks and the style of generated code in general differ a lot between the different existing generators. This will add up when we are going to add support for more targets or add new generators for existing language targets. The CRUD generator, for example, outputs doc-blocks, functions and just better reusable code compared to router generator.

Existing options

For completeness, below are the template options we use currently in @compas/code-gen:

String concatenation

Nothing more, nothing less. Mostly done with template literals like in the above example. Contains a lot of inline ternary operators (true ? "foo" :"bar") and inline .map().join() calls.

Tagged templates

Tagged templates in JavaScript are a form of template literals, but the values are passed to a function allowing processing of the arguments. This is often used by database client libraries to prevent SQL injection attacks. The @compas/store package also exposes query to do just that.

function myLiteral(strings, ...args) {
  console.log({ strings, args });
}

myLiteral`My ${"tagged"} template ${"function"}`;
// -> { strings: ["My ", " template ", ""], args: ["tagged", "function"] }

The code-generators also have a tagged template function called js. It allows inline functions to share state, return other functions and eventually return a value.

/**
 *.  The first inline function sets `C` on the state.value
 *  `A` is printed as is
 *  `B` is returned immediately when the function is called
 *  `C` is not returned immediately, but only in the second invocation, based on `state.value`
 */
const myComplexABC = js`
  ${(state) => { state.value = "C"; return ""; }}
  A
  ${(state) => "B"}
  ${(state) => state.phase === "COLLECT" ? state.value : undefined}
`;

.tmpl files

The last major option that we have currently is .tmpl files. These are used in the apiClient and reactQuery generator. They allow full inline JavaScript, and are transformed to functions. These functions are then called with some state, which are exposed as global variables via with-statements.

An outputted string
{{= "An evaluated " + "and printed expression" }}
{{ const foo = "use inline JS"; }}
{{= foo }}

{{ if ("conditions and loops" === "are supported as well") { }}
  {{= "A bit unreadable tho" }}
{{ } }}

Different targets

As previously mentioned in Part 1 and Part 2 of this series, we think that Compas.js should not be limited to just JavaScript, TypeScript, Koa for routing or Axios as the library of choice for api clients. The new template system should invite contributors to add their target of choice, without having to learn various different templating systems, and duplicating all the logic that is in these templates.

We are still working on redesigning the generators with logic-less, reusable and target specific templating. The end result probably warrants its own post as well, so expect that in the coming months. In the mean time, feel free to share any ideas or reference material you have around these topics!