Mountains & Software: Similar Peaks

We have climbed similar mountains many times over across our usual projects. Going through these repetitions, I too have put together my toolkit for abstracting over implementation details each time, so as to scale the usual hurdles faster. This series explores such meta-programming, special trees, and the resulting code-generators I have built for my enterprise cloud automation and CI/CD integration use-cases so far.

Builders & Generators

In the world of rapid application development and 'no-code'/'low-code' app builders, code generators are a necessarily evil in the tool-chain. Code should be as DRY or repetition free as possible for maintainability. But that applies only for 'same code'.

When it comes to 'similar code', a lot of our code can be seen as blatant repetitions of "the same kinds of things" - if not "the same things". If your framework needs a new file for a new controller, your code-generator schematics can and should be generating that controller and its test files. This is possible and quite routine with nx, ng and similar cli tools. What if you know that you are going to need to run that generator for each of your hundred controllers?

A build system that integrates code-generators can add, update or remove parts of the code-base itself. Seen this way, code-generators are the 'actors' that generate code while builders are the 'directors' that supervise the action of code-generation itself. As for knowing which controllers to generate, there usually is a signal or annotation to find out. For instance, if you are writing a new controller layer for each of your user facing entities, you can mark them as such in your database schema itself or a configuration file somewhere. Same for per-folder view files in folder-based routers used by frameworks and meta-frameworks like SvelteKit, Remix, Next, AnalogJS and Astro.

Prisma ORM provides a way to annotate any part of your database schema (entity model, enum, attribute) with an active comment. Any comment marked with three slashes /// are picked up by cli-tools and libraries that tap into the capabilities of the ORM. In the case of a code-generator, this can be your own ts-morph scripts based on Prisma's database meta model format or DMMF.

Prisma & The Tree

Prisma's DMMF is an abstract syntax tree (AST) and an AST is one your best friends as a meta-programmer. Builders and generators can be guided by the AST in their internal logic. To work with your database's Prisma AST/DMMF, this is one way to expose it for programmatic use:

import { Prisma } from "@prisma/client";
import type { BaseDMMF } from "@prisma/client/runtime/library";

function getDMMF(): BaseDMMF {
    return Prisma.dmmf;
}

It's a one-liner to abstract us from how Prisma decides to expose the DMMF in a future version as these parts of the library are not public or expected to stay the same. This utility can now be used every where else in our code, say, to introspect which tables are present in the database of the currently running system:

export function getModelNames(): string[] {
    return getDMMF().datamodel.models
        .map(m => m.dbName ?? m.name);
}

Or to find metadata about the fields or columns in our table/model of interest:

export function getFieldMetadata(model: string) {
    const allModels = getDMMF().datamodel.models;
    const selectedModel = allModels.find(mod => mod.dbName === model);
    if (selectedModel) {
        return selectedModel.fields;
    }

    throw new Error("Model not found");
}

This isn't a particularly type-safe code code as the input is not all possible strings and the output is quite a complex type. Let's constrain the types as an exercise to get to the know the AST better and to make the above function safer to use. Let's click through the definitions in @prisma/client/runtime/library.d.ts as a starter:

// not exported, but you are advised to inline this in your code
type ReadonlyDeep_2<O> = {
    +readonly [K in keyof O]: ReadonlyDeep_2<O[K]>;
}

// can be used as a reference for how to deal with tables (models) and columns (fields)
export type Model = ReadonlyDeep_2<{
        name: string;
        dbName: string | null;
        fields: Field[];
        uniqueFields: string[][];
        uniqueIndexes: uniqueIndex[];
        documentation?: string;
        primaryKey: PrimaryKey | null;
        isGenerated?: boolean;
}>;

At this point we can type the getFieldMetadata function as follows:

export function getFieldMetaData(model: string): ReadonlyDeep_2<DMMF.Field[]> {..}

The string input type is still too wide but the output is well-typed for advanced use-cases. Wouldn't it be better if we knew our model names at the type-level, ahead of time? Thanks to the trivial getModelNames function, we can dump this information in a typescript file itself. That file would look something like this:

// zen/entities.const.ts
export const Principal = "Principal" as const;
export const Account = "Account" as const;
export const Session = "Session" as const;
export const VerificationToken = "VerificationToken" as const;
export const Authenticator = "Authenticator" as const;
export const Payment = "Payment" as const;
export const Appointment = "Appointment" as const;
export const AppointmentType = "AppointmentType" as const;
export const Location = "Location" as const;
export const Patient = "Patient" as const;
export const Equipment = "Equipment" as const;
export const Service = "Service" as const;
export const Provider = "Provider" as const;
export const Form = "Form" as const;

An exhaustive list of all models that need to be represented in our user interfaces can be generated like this into a file and can then be used as a type-level information in our future functions.

// zen/entities.type.ts
import { Principal, Account, Session, VerificationToken, Authenticator, Payment, Appointment, AppointmentType, Location, Patient, Equipment, Service, Provider, Form } from "./entities";
const Entities = {
    Principal, 
    Account, 
    Session, 
    VerificationToken, 
    Authenticator, 
    Payment, 
    Appointment, 
    AppointmentType, 
    Location, 
    Patient, 
    Equipment, 
    Service, 
    Provider, 
    Form
    // 100+ similar table names
} as const;

export type EntitiesType = typeof Entities;
export type NTTKey = keyof EntitiesType;

Now we can have a much more type-safe signature for our getFieldMetadata function:

export function getFieldMetaData(model: NTTKey): ReadonlyDeep_2<DMMF.Field[]> {..}

A big part of code-generation - or any advanced library code - is making illegal states impossible. In this case we have ensured that we only allow valid string names for our input and a read-only Field[] type as our output.

ts-morph & Morphisms

In this series, ts-morph will be our library of choice for demonstrating the file generation part of code-gen. The above file with table names as const can be generated as follows with ts-morph:

export const ModelNames: string[] = Prisma.dmmf.datamodel.models.map(
    (m: DMMF.Model) => m.name,
);

function statements(lCase = false) {
    return ModelNames.map((ModelName) => {
        const name = lCase ? firstCharLower(ModelName) : ModelName;
        return {
            declarationKind: VariableDeclarationKind.Const,
            declarations: [
                {
                    name,
                    initializer: `"${name}" as const`,
                },
            ],
            isExported: true,
        };
    });
}


function createEntities(project: Project): void {
    const entitiesTsFile = project.createSourceFile(`${zenPath}/entities.ts`, '', {
        overwrite: true,
    });

    entitiesTsFile.addVariableStatements(statements());
    entitiesTsFile.saveSync();
}

export function initiateQb() {
    const project = new Project({
        tsConfigFilePath: './tsconfig.json',
        skipAddingFilesFromTsConfig: true,
    });

    createEntities(project);
    createEntitiesType(project);
    // ... other createXXX functions
    project.saveSync();
}
initiateQb();

Generating the file zen/entities.type.ts from above works similarly, except for we have to import the constants from the earlier const file:

function createEntitiesType(project: Project): void {
    const entitiesTypeTsFile = project.createSourceFile(
        `${zenPath}/entities-type.ts`,
        '',
        {
            overwrite: true,
        },
    );

    entitiesTypeTsFile.addImportDeclarations([
        {
            moduleSpecifier: './entities',
            namedImports: [...ModelNames],
        },
    ]);

    entitiesTypeTsFile.addVariableStatement({
        declarationKind: VariableDeclarationKind.Const,
        declarations: [
            {
                name: 'Entities',
                initializer: `{${ModelNames.join(', \n')}} as const`,
            },
        ],
    });

    entitiesTypeTsFile.addTypeAlias({
        name: 'EntitiesType',
        type: 'typeof Entities',
        isExported: true,
    });
    entitiesTypeTsFile.addTypeAlias({
        name: 'NTTKey',
        type: 'keyof EntitiesType',
        isExported: true,
    });

    entitiesTypeTsFile.saveSync();
}

By this point, we have an example of generated code and a solid foundation for further generating controllers, test data, CRUD UIs and even basic visualizations for each of our models.

ts-morph is all it takes once we have a decent AST, like that of Prisma's DMMF. And we will be using a similar code-setup for generating most parts of a CRM that we would have manually (slowly) coded otherwise. Morphism is an abstraction over functions themselves. In our case, our basic morphisms are effects: code being written to a file. And the overall work is about composition of such morphisms.