Elasticsearch, Kibana, and NestJS Dockerized application

Hello everyone! I am going to describe a simple tutorial with an example of a TODO application where under the hood will be NestJS, Elasticsearch, Kibana, and Swagger (For API documentation). It would be good if you have some knowledge about elasticsearch. But no problem! follow along with the contents, you will get what is elasticsearch & why should we use this amazing engine. So let’s jump in.

N.B. The introduction of the tutorial is here TODO-APPLICATION.

What is elasticsearch?
Elasticsearch is a distributed, free and open search and analytics engine for all types of data, including textual, numerical, geospatial, structured, and unstructured which is built on Apache Lucene.

What is kibana?
Kibana is an free and open frontend application that sits on top of the Elastic Stack, providing search and data visualization capabilities for data indexed in Elasticsearch which is Commonly known as the charting tool for the Elastic Stack.

Step-1: Install a few npm modules

docker-compose run nestjs npm install @elastic/elasticsearch @nestjs/elasticsearch @nestjs/config elasticdump

Step-2: Update .env file (N.B esOne is the container name of elasticsearch)

ELASTICSEARCH_NODE=http://esOne:9200
ELASTICSEARCH_USERNAME=elastic
ELASTICSEARCH_PASSWORD=changeme

Step-3: Update docker-compose.yml file

N.B. We are going to use single node in elasticsearch but it is good to have multiple elasticsearch node.

Step-4: Create src/todo/service/todo-search.service.ts, src/todo/types/todoSearchBody.interface.ts & src/todo/types/todoSearchResult.interface.ts

This file will be used for the CRUD operation as well as for search operation in elasticsearch.
Step-5: Update src/todo/service/todo.service.ts file. In the constructor add TodoSearchService which will use for the sending value of one service to another.

constructor(
   ...
   private todoSearchService: TodoSearchService
) {}

Update createTodo(), updateTodoById(), deleteTodoById() and add searchForTodos() function for the following reasons:
1. Whenever we create/update/delete a todo we should also keep update to the elasticsearch.
2. And the searchForTodos() function will use for the search of todo list from elasticsearch.

async createTodo(todoDto: TodoDto, user: User): Promise<TodoPayload> {
   const newTodo = await this.todoRepository.createTodo(todoDto, user);
   this.todoSearchService.indexTodo(newTodo, user.username);
   return newTodo;
}
...
async updateTodoById(id: number, todoDto: TodoDto, user: User): Promise<TodoPayload> {
   const todo = await this.getTodoById(id, user);
   todo.title = todoDto.title
   todo.description = todoDto.description
   todo.updatedDate = new Date();
const updated = await todo.save();
   if (updated) {
       await this.todoSearchService.update(todo, user.username);
       return {
          id: todo.id,
          title: todo.title,
          description: todo.description,
          createdDate: todo.createdDate,
          updatedDate: todo.updatedDate
       } 
   }
   throw new NotFoundException(todo.id);   
}
async deleteTodoById(id: number, user: User): Promise<{ message: string }> {
   const todo = await this.todoRepository.delete({ id, userId: user.id })
if (todo.affected === 0) {
       throw new NotFoundException(`This ${id} is not found`)
   }
   await this.todoSearchService.remove(id);
   return { message: 'Deleted successfully !' }
}
async searchForTodos(text: string, user: User) {
   const results = await this.todoSearchService.search(text, user.username);
   if (results) {
       return results;
   }
   return [];
}

Update src/todo/todo.controller.ts file

@Get('/search-todo')
searchTodo(@GetUser() user: User, @Query('search') search: string) {
  if (search) {
     return this.todoService.searchForTodos(search, user);
  }
  return [];
}

Step-6: Update src/todo/todo.module.ts file

Step-7: Let’s start the application

docker-compose build // If you add/upgrade anything better to build
docker-compose up // Will start all the services
/* Or you could use */
docker-compose up --build

After successfully start the application, If you insert a new todo for the first time then the index of elasticsearch will create while inserting the data. But it is better to create it manually. So in your browser type http://localhost:5601. You will see that the kibana will start. Go to dev-tools and paste the below query.

PUT todos
{
  "mappings": {
      "properties": {
        "id" : {
          "type" : "long"
        },
        "todo" : {
          "type": "object", 
          "properties" : {
            "createdDate" : {
              "type" : "date"
            },
            "description" : {
              "type" : "text"
            },
            "title" : {
              "type" : "text"
            },
            "updatedDate" : {
              "type" : "date"
            }
          }
        },
        "userEmail" : {
          "type" : "text",
          "fields": {
            "keyword": {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        }
      } 
  }
}

You see the userEmail keyword has fields keyword. Because we want to search todo list according to the respective users where email should unique. The exact match for the text is not possible with “type”: ”text” and that’s why we are using keyword. Link of term-query documentation.

So let’s do a search of todo within the swagger.
Now, what if we want to add our existing data to the elasticsearch. YES, we should add our existing data. So follow the reset of the steps.

Step-8: Fetch all the data from Postgres to a JSON file called tododb.json.

\copy (SELECT array_to_json(array_agg(r)) FROM (select "todo"."id", "user"."username" "userEmail", title, description, "todo"."createdDate", "todo"."updatedDate" from "user" inner join "todo" on "todo"."userId" = "user"."id") r) TO '/tmp/tododb.json';

we also created a volume which is pgfile wherein in the pgfile folder you can see a file is created with JSON data. Now let’s arrange it as elasticsearch’s mapping pattern. For that reason, we have to create a dynamic way.
Create a file named streamData.ts

execute the following command which will organize your fetched data.

docker-compose run nestjs npx ts-node streamData.ts

The formatted.json (line-3) file is the output that will be used as an input file for indexing data in elasticsearch with the following command.

docker-compose run nestjs node_modules/elasticdump/bin/elasticdump --input=formatted.json --output=http://esOne:9200/

It will index all of your data to elasticsearch. Now, search todo list again you will get your expected results.
Extra command to do certain things-

curl -XDELETE http://localhost:9200/todos // To delete a index
/* To see the mapping */
curl http://localhost:9200/todos/_mapping?pretty
/* To see the results */
curl http://localhost:9200/todos/_search?pretty

GitHub repo. So if you think it is worthy then please do share & clap this tutorial.
THANK YOU!!!