In a Hadoop Cluster, you are dealing with very large amount of data being written to or read from the disk using the network traffic. In order to handle the Hadoop data operation in a better way, you need to tell the Namenode that where and how the slave machines(Data nodes) are distributed. To achieve…
Details
File compression is always better to store large amount of data in HDFS, it reduces significant amount of space, when a raw data is compressed, in Hadoop we use various IO Compression codecs depends on the scenario and data access frequency etc. There are various compression types available and below are the default codecs supported…
Details
Tune you Linux OS to maximize the Hadoop Cluster performance by disabling “Tuned Daemon” in RHEL 7 OS. If you are setting up Hadoop Cluster in RHEL 7 servers, then you should disable the Tuned daemon Service which comes as default daemon in RHEL 7 version of Linux. Tuned daemon is introduced to boots the…
Details
Lets look into migrating the whole ElasticSearch index from one ElasticSearch to another one. To achieve this you need few prerequisite like NodeJS, npm and elasticdump plugin. Install elasticdump plugin using the below command npm install -g elasticdump Once the elastic dump plugin is installed, use the elasticdump utility to migrate the data…
Details
Fix under replicated blocks in HDFS In this tutorial, let’s look into, How to Fix under replicated blocks in HDFS? How to check any under replicated, missing or corrupted blocks in the cluster? You can check all the blocks information from Namenode UI or with the help of fsck command. Namnode UI – http://<namenode-IP>:50070/dfshealth.html#tab-overview 2. Using…
Details
Let’s discuss on enabling Hive Metastore NULL result for an empty string. In Hive, while inserting values, if some columns have empty strings and you want to display it as NULL when queried the table. From the below image, you can see the empty string are displayed just like that. And this property applies to only…
Details
Lets us look into S3 bucket Hive sentry permission to create external table. If you are planning to create an external table and to have the data in S3. We must give the hive sentry permission on S3 bucket URI. S3 Bucket Hive Sentry Permission: – Below are the sentry rules for providing permission to create external…
Details
We will look in to Hive Sentry Roles commands. Here in this post you will get all sentry role creation and grant permission commands. 1. Create hive sentry role :- CREATE ROLE `role_name`; # Here were are using group name as hadoop-developer CREATE ROLE `hadoop-developer_role`; It is always a good practice to create a role…
Details
InvalidSnapshotNameException If you are getting the InvalidSnapshotNameException exception in elasticsearch while creating the snapshot for the indices. { “error”:”InvalidSnapshotNameException[[snapshot_name_goes_here:Logstash] Invalid snapshot name [Logstash], must be lowercase]”, “status”:400 } From the exception itself, It is clearly shown that, Elasticsearch accepts only lower-case name for snapshot, This is applicable for both Snapshot and Indices as well. If you…
Details
Elasticsearch Cluster setup Installing and configuring Elasticsearch cluster for production purpose. Elastic search is one of the popular full-text search engine, used for real time distributed search and analysis of data. In my previous post, we have seen “How to install and configure Elasticsearch in single node” . Single node will be used for development…
Details