Data Engineering: Getting Started with Apache NiFi(Part 1)

Apache NiFi is a powerful low-code dataflow platform based on flow-based programming principles. With its user-friendly interface, extensive configurability, and guaranteed delivery features, NiFi has become a reliable tool in the world of data engineering.

In this Data Engineering blog series, we’ll explore NiFi by using it as a data ingestion tool. We’ll walk through its installation, initial setup, and demonstrate how to move data with it.

---

What is Apache NiFi?

Apache NiFi is open-source software designed to automate data flow between systems. Originally developed by the NSA under the "Niagara Files" project, it was open-sourced in 2014 and donated to the Apache Software Foundation.

Key Features:

• Visual web-based UI for designing and monitoring data flows

• Data routing and transformation capabilities

• Built-in data provenance for end-to-end flow tracking

• Reliable delivery - even in unreliable network conditions

• Built in real-time streaming capabilitiy

I use NiFi primarily for data ingestion and have found it especially reliable in cases of intermittent connectivity. It’s also a lightweight solution for many streaming use cases.

---

Installing Apache NiFi

First, ensure your system meets the requirements listed in the NiFi Administration Guide.

NiFi comes in several flavors:

• Standard: Full NiFi server

• Stateless: Executes flows as jobs without state/queues

• Toolkit: CLI utilities for managing NiFi environments

• MiNiFi: Lightweight version for IoT/edge devices

The link below will take you to the official page to download NiFi:

Download Apache NiFi

---

Unpack and Navigate to the NiFi Directory

After downloading the NiFi binary (usually a .tar.gz or .zip file), unpack it and navigate into the directory:

# For tar.gz (Linux/macOS)
tar -xvzf nifi-<version>-bin.tar.gz
cd nifi-<version>

# For .zip (Linux/macOS)
unzip nifi-<version>.zip #(You need to have the unzip utility installed)
cd nifi-<version>

In the next steps we’ll move into the conf directory and generate our keystore and truststore there:

cd conf

---

Setting Up Apache NiFi

Note: NiFi 2.x.x enforces HTTPS. You’ll need certificates set up before starting the server. Self Signed Certs work.

Prerequisite

• Java 21 must be installed and available in your system's PATH.

---

Generating Certificates for HTTPS

1. Generate a Self-Signed Certificate and Keystore

keytool -genkeypair \
  -alias nifi-cert \
  -keyalg RSA \
  -keysize 2048 \
  -validity 365 \
  -storetype PKCS12 \
  -keystore keystore.p12 \
  -storepass yourpassword \
  -keypass yourpassword \
  -dname "CN=localhost, OU=Dev, O=YourOrg, L=City, S=State, C=US"

Explanation:

• -alias nifi-cert: Friendly name for the cert file

• -keyalg RSA: Key algorithm

• -keysize 2048: RSA key length

• -validity 365: Valid for 1 year

• -storetype PKCS12: Format required by NiFi.

• -dname: Subject details

2. Create a Truststore

Export the Public Certificate:

keytool -exportcert \
  -keystore keystore.p12 \
  -storetype PKCS12 \
  -alias nifi-cert \
  -file nifi-cert.cer \
  -storepass yourpassword

Import it into a Truststore:

keytool -importcert \
  -alias nifi-cert \
  -file nifi-cert.cer \
  -keystore truststore.p12 \
  -storetype PKCS12 \
  -storepass yourpassword \
  -noprompt

Configuring nifi.properties

Edit conf/nifi.properties to include the SSL configs we generated. Be mindful of where the keystore and truststore you generated were stored, you’ll need to update the nifi.properties with those paths:

Keystore Settings:

nifi.security.keystore=./conf/keystore.p12
nifi.security.keystoreType=PKCS12
nifi.security.keystorePasswd=yourpassword
nifi.security.keyPasswd=yourpassword

Truststore Settings:

nifi.security.truststore=./conf/truststore.p12
nifi.security.truststoreType=PKCS12
nifi.security.truststorePasswd=yourpassword

Enable HTTPS:

nifi.web.https.port=8443
nifi.web.https.host=localhost

Starting Apache NiFi

Navigate to the bin directory and run:

./nifi.sh start

Then open your browser and go to:

https://localhost:8443/nifi/

You should see the NiFi login screen and UI.