Custom Log4j Appenders for Java Logging

Introduction

Logging is a crucial part of software development, giving important insights into how applications work. While frameworks like Log4j and Logback offer strong logging features, there are times when developers need custom logging solutions to meet specific needs or to fit smoothly with existing systems.

In this article, we delve into the realm of custom Log4j appenders, exploring how they can be leveraged to enhance logging capabilities in Java or Spring Boot applications. Whether it's uniquely formatting logs, sending logs to specialized destinations like Splunk, Elasticsearch, Kafka, etc., or integrating with external systems, custom appenders empower developers to fine-tune logging behaviors according to their needs.

Pre-requistes

JDK 8+ (We are using JDK17)
Maven Build Tool (Should work with Gradle too)
Any java/spring boot application (We will bootstrap one for demo purpose)

Bootstrap Spring Boot Application

We will quickly use Spring IO Initialzr to get Spring Boot app bootstrapped as below

We will add quick Rest Controller backed by simple Calculator Service offering basic Math operations as below

CalculatorController.java

package in.virendraoswal.controller;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;

import in.virendraoswal.service.CalculatorService;

/**
 * 
 * @author Virendra
 *
 */
@RestController
public class CalculatorController {

    @Autowired
    CalculatorService calculatorService;

    @GetMapping("/add")
    public int add(@RequestParam int num1, @RequestParam int num2) {
        return calculatorService.add(num1, num2);
    }

    @GetMapping("/subtract")
    public int subtract(@RequestParam int num1, @RequestParam int num2) {
        return calculatorService.subtract(num1, num2);
    }

    @GetMapping("/multiply")
    public int multiply(@RequestParam int num1, @RequestParam int num2) {
        return calculatorService.multiply(num1, num2);
    }

    @GetMapping("/divide")
    public int divide(@RequestParam int num1, @RequestParam int num2) {
        return calculatorService.divide(num1, num2);
    }

}

CalculatorService.java

package in.virendraoswal.service;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.stereotype.Service;

/**
 * 
 * @author Virendra
 *
 */
@Service
public class CalculatorService {

    private static Logger LOG = LoggerFactory.getLogger(CalculatorService.class);

    public int add(int num1, int num2) {
        LOG.info("Adding 2 numbers ...");
        return num1 + num2;
    }

    public int subtract(int num1, int num2) {
        LOG.info("Substracting 2 numbers ...");
        return num1 - num2;
    }

    public int multiply(int num1, int num2) {
        LOG.info("Multiplying 2 numbers ...");
        return num1 * num2;
    }

    public int divide(int num1, int num2) {
        LOG.info("Dividing 2 numbers ...");
        try {
            return num1 / num2;
        } catch (Exception e) {
            LOG.error("Error occured during divide", e);
        }

        return 0;
    }
}

Inject Log4j2 Starter

We will add Log4j2 dependencies via Starter to avoid bringing in individual libraries associated with it, while also excluding the Logback library, which comes by default with Spring Boot.

<dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
            <exclusions>
                <exclusion>
                    <groupId>org.springframework.boot</groupId>
                    <artifactId>spring-boot-starter-logging</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>
        <!-- https://mvnrepository.com/artifact/org.springframework.boot/spring-boot-starter-log4j2 -->
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-log4j2</artifactId>
        </dependency>
    </dependencies>

We will Custom Console Appender log4j.xml to print those to the System Console

log4.xml

<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="INFO">
    <Appenders>
        <Console name="Console" target="SYSTEM_OUT">
            <PatternLayout pattern="%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n" />
        </Console>
    </Appenders>
    <Loggers>
        <Root level="INFO">
            <AppenderRef ref="Console" />
        </Root>
    </Loggers>
</Configuration>

We have already added some Log statements for INFO and one for ERROR in the divide method to catch exceptions when we try to put divisor by Zero(0) for the future part of an article on how we use this log event.

Now that Lo4J2 is integrated with the Spring Boot application with a few log events in place, we will quickly test to see if all is fine.

We will test successfully divide test by hitting the URL

11:46:48.990 [http-nio-8080-exec-4] INFO  in.virendraoswal.service.CalculatorService - Dividing 2 numbers ...

We will also test add by hitting the URL

11:47:39.641 [http-nio-8080-exec-5] INFO in.virendraoswal.service.CalculatorService - Adding 2 numbers ...

We will now test the exception-based divide test by hitting the URL

In this case we get exception, which will be logged at ERROR log event as below

11:49:28.303 [http-nio-8080-exec-1] ERROR in.virendraoswal.service.CalculatorService - Error occured during divide
java.lang.ArithmeticException: / by zero
    at in.virendraoswal.service.CalculatorService.divide(CalculatorService.java:35) [classes/:?]
    at in.virendraoswal.controller.CalculatorController.divide(CalculatorController.java:38) [classes/:?]
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]

Custom Appender to process ERROR-based Log Events

We will now add Custom Appender to capture ERROR log events and process them to perform some activity.

We will create an Exception/Error Appender class which will extend org.apache.logging.log4j.core.appender.AbstractAppender

We define a Plugin which helps to extend the functionality of Log4j.

ExceptionAppender.java

package in.virendraoswal.logging.appender;

import org.apache.logging.log4j.core.Appender;
import org.apache.logging.log4j.core.Core;
import org.apache.logging.log4j.core.Filter;
import org.apache.logging.log4j.core.LogEvent;
import org.apache.logging.log4j.core.appender.AbstractAppender;
import org.apache.logging.log4j.core.config.Property;
import org.apache.logging.log4j.core.config.plugins.Plugin;
import org.apache.logging.log4j.core.config.plugins.PluginAttribute;
import org.apache.logging.log4j.core.config.plugins.PluginElement;
import org.apache.logging.log4j.core.config.plugins.PluginFactory;
import org.apache.logging.log4j.core.filter.ThresholdFilter;

@Plugin(name = "ExceptionAppender", category = Core.CATEGORY_NAME, elementType = Appender.ELEMENT_TYPE, printObject = true)
public class ExceptionAppender extends AbstractAppender {

    public ExceptionAppender(String name, Filter filter) {
        super(name, filter, null, true, Property.EMPTY_ARRAY);
    }

    @PluginFactory
    public static ExceptionAppender createAppender(@PluginAttribute("name") String name,
            @PluginElement("ThresholdFilter") ThresholdFilter filter) {
        return new ExceptionAppender(name, filter);
    }

    public void append(LogEvent event) {
        // Implement your custom logging logic here
        System.out.println("Exception Appender invoked... ");
        if (event.getLevel().equals(org.apache.logging.log4j.Level.ERROR)) {
            System.out.println(event.getThrown());
            System.out.println(event.getSource());
            // Write log message to custom destination (e.g., file, database, etc.)
            System.out.println("Writing log message to external destination ...");
        }
    }
}

Here we override append(LogEvent) and put custom logic on what to do when the Log Event is captured. We can get a lot of info about exceptions and it's detailed information.

@Plugin defines Appender, the name over here should match the tag in log4.xml

Now that class is defined we need to add that as an Appender in the log4j.xml file as below.

updated log4.xml

<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="INFO">
    <Appenders>
        <Console name="Console" target="SYSTEM_OUT">
            <PatternLayout pattern="%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n" />
        </Console>
        <ExceptionAppender name="exceptionAppender" />
    </Appenders>
    <Loggers>
        <Root level="INFO">
            <AppenderRef ref="Console" />
            <AppenderRef ref="exceptionAppender" />
        </Root>
    </Loggers>
</Configuration>

Here we defined Custom Appender <ExceptionAppender> which matches the plugin name and is also referred within <Loggers> to activate appended.

Scanning Custom Appender

Just adding the Custom Appender won't work; we need to tell Log4j where to find the custom appender and register it as a plugin for use during application log events.

There are a couple of ways to scan Custom Appender

Package Scan

One via package scan where we define log4.xml as below

  <?xml version="1.0" encoding="UTF-8"?>
  <Configuration status="INFO" packages="in.virendraoswal.logging.appender">
      <Appenders>
          <Console name="Console" target="SYSTEM_OUT">
              <PatternLayout pattern="%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n" />
          </Console>
          <ExceptionAppender name="exceptionAppender" />
      </Appenders>
      <Loggers>
          <Root level="INFO">
              <AppenderRef ref="Console" />
              <AppenderRef ref="exceptionAppender" />
          </Root>
      </Loggers>
  </Configuration>

Here we tell which package to scan to find custom plugins/appenders for creation.

However, when you run an application you might get a warning as below suggesting this has been deprecated

  WARN StatusConsoleListener The use of package scanning to locate plugins is deprecated and will be removed in a future release

Annotation Processor via Maven Plugin

In this approach, we define a maven plugin for the Log4j Plugin processor to scan plugins at compile time and be available for use.

Add the below maven plugin to your project

  <plugin>
      <groupId>org.apache.maven.plugins</groupId>
      <artifactId>maven-compiler-plugin</artifactId>
      <executions>
          <execution>
              <id>log4j-plugin-processor</id>
              <goals>
                  <goal>compile</goal>
              </goals>
              <phase>process-classes</phase>
              <configuration>
                  <proc>only</proc>
                  <annotationProcessors>
                      <annotationProcessor>org.apache.logging.log4j.core.config.plugins.processor.PluginProcessor</annotationProcessor>
                  </annotationProcessors>
              </configuration>
          </execution>
      </executions>
  </plugin>

That is for integrating Custom Appender into your application to extend Log4j.

Now that integration is done, we will hit the negative divide scenario to generate an exception which will be logged as an ERROR log event.

On hitting we get an exception and now if we see, we get an additional console output from the appender as below suggesting working of the same

Exception Appender invoked... 
java.lang.ArithmeticException: / by zero
in.virendraoswal.service.CalculatorService.divide(CalculatorService.java:37)
Writing log message to external destination ...

Voila! We have implemented Custom Log4j Appender to perform some additional processing in case of ERROR Log Events. Now we can write only ERROR messages to say, Kafka, Elastic Search, and Splunk and build a custom dashboard on top of it directly rather than ingest every log and filter based on log patterns to do some analysis.

However, if you look closely, our Custom Appender gets invoked on every log event captured though within our Custom Appender, we process only if it's ERROR.

To further avoid, invoking Custom Appender only when an ERROR event is logged, we can put the Threshold filter below

<ExceptionAppender name="exceptionAppender">
    <ThresholdFilter level="ERROR" onMatch="ACCEPT" onMismatch="DENY" />
</ExceptionAppender>

Filters allow Log Events to be evaluated to determine if or how they should be published, there are different types of filters we can use.

Resources

Thank you for reading, If you have reached it so far, please like the article, It will encourage me to write more such articles. Do share your valuable suggestions, I appreciate your honest feedback and suggestions!

I would love to connect with you on Twitter | LinkedIn

Enhance Java Application Logging with Custom Log4j Appenders

Table of contents