EFK Stack: Elasticsearch, Fluentd and Kibana on Docker

EFK stack visualized

Why ditch Logstash?

Memory Memory Memory

Why Fluentd?

  1. Memory issues? No more: Fluentd is written in a combination of C language and Ruby, and requires very little system resource. The vanilla instance runs on 30–40MB of memory and can process 13,000 events/second/core. Fluentd also has a lighter version literally made available to run on embedded devices, Fluent Bit promises to run on memory consumption of about 450KB!!
  2. 500+ Plugins: Fluentd has a flexible plugin system that allows the community to extend its functionality. Our 500+ community-contributed plugins connect dozens of data sources and data outputs. By leveraging the plugins, you can start making better use of your logs right away.
  3. Built-in Reliability: Fluentd supports memory- and file-based buffering to prevent inter-node data loss. Fluentd also supports robust failover and can be set up for high availability. 2,000+ data-driven companies rely on Fluentd to differentiate their products and services through better use and understanding of their log data.
  4. Open Source, Open Source, Open Source: Fluentd is an open-source data collector, which lets you unify the data collection and consumption for better use and understanding of data.

Getting your hands dirty

Prerequisites

  1. Docker
  2. Node.js

Step 1: Create the docker-compose file

docker-compose.yml

Step 2: Writing the Fluentd Dockerfile

The alpine build of Fluentd with elastic search plugin added

Step 3: Fluentd Configuration File

Fluentd Conf file. Place it in ./fluentd/conf/fluentd.conf

Step 4: Run them all

Run docker-compose upand wait for the output. Then please visit http://localhost:5601/ in your browser. Then, you need to set up the index name pattern for Kibana. Please specify fluentd-* to Index name or pattern and press Create button. Then, go to Discover the tab to seek for the logs. Now we’ll be sending in the logs with node.

Step 5: Minimal Node setup

Logs showing up in Kibana

--

--

Sometimes it is the people no one can imagine anything of, do the things no one can imagine.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
aesher9o1

Sometimes it is the people no one can imagine anything of, do the things no one can imagine.