How I log and alert

Logs should be long enough to have all the needed information and short enough not to cause any effect on the system (large log files, etc.). They should also be in a good format for easy traceability and alertness.
Below is my current strategy for log and alerting.

My current project use Splunk for logs, it supports alert to Slack!

How I log

NOTE:

• If you want to log the output/result of a function, log it inside the function, so you don't have to duplicate that log, also you don't forgot to add it outside whenever the function is called/used.

This is one of my current log:

        console.log(
          `[ModuleName_Alert]`,
          JSON.stringify({
            title: 'TOO_MUCH_TOKEN',
            curToken,
            maxToken: this.MAX_TOKEN,
            prompt: request.prompt,
            first100charOfQuestion: request.question.substring(0, 100),
          }),
        );

This is what I follow:

• It should have 2 params:

  • 1s one is the name of the log: use it to find the log or to trigger the alert

  • 2nd one is to provide extra information that needs to be checked:

    • This one should be an object/json so if you can easy to parse the data with splunk query. By doing that you can do so much more, like filter, add condition...

    • it should have a title to be able to do another filter or a sub-filter

    • it can have a friendly message

    • long content that no need to get it all should substring them

How I use Splunk

By log as above procedure, I'm able to filter in Splunk like below:

index=sleek-eks-sg-logs "ZENDESK_API"
| rex "ZENDESK_API (?<json>.*)"
| spath input=json
| table function message zendeskTicketId

How I alert

Send message to Slack with a certain condition is a built-in feature of Splunk.

With the more importance alert I use realtime alert, but it seems it not working right now. So the more importance the alert is the the short the I set the time for sending message to Slack.

And this is the alert on the Slack