RegionServer crashing with error "Direct buffer memory"

Region server might crash with below error

INFO  [main] zookeeper.ZooKeeper: Initiating client connection, connectString=ip-100-122-218-159.us-east-1.ec2.aws.symcpe.net:2181,ip-******.us-east-1.ec2.aws.net:2181,ip-*****.us-east-1.ec2.aws.net:2181 sessionTimeout=180000 watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@253c1256
ERROR [main] regionserver.HRegionServerCommandLine: Region server exiting
java.lang.RuntimeException: Failed construction of Regionserver: class org.apache.hadoop.hbase.regionserver.HRegionServer
 at org.apache.hadoop.hbase.regionserver.HRegionServer.constructRegionServer(HRegionServer.java:2666)
 at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.start(HRegionServerCommandLine.java:64)
 at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.run(HRegionServerCommandLine.java:87)
 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
 at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
 at org.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:2681)
Caused by: java.lang.reflect.InvocationTargetException
 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
 at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
 at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
 at org.apache.hadoop.hbase.regionserver.HRegionServer.constructRegionServer(HRegionServer.java:2664)
 ... 5 more
Caused by: java.lang.OutOfMemoryError: Direct buffer memory
 at java.nio.Bits.reserveMemory(Bits.java:658)
 at java.nio.DirectByteBuffer.(DirectByteBuffer.java:123)
 at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
 at org.apache.zookeeper.ClientCnxnSocket.(ClientCnxnSocket.java:51)
 at org.apache.zookeeper.ClientCnxnSocketNIO.(ClientCnxnSocketNIO.java:48)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
 at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
 at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
 at java.lang.Class.newInstance(Class.java:442)
 at org.apache.zookeeper.ZooKeeper.getClientCnxnSocket(ZooKeeper.java:1779)
 at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:447)
 at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:380)
 at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.checkZk(RecoverableZooKeeper.java:141)
 at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.(RecoverableZooKeeper.java:128)
 at org.apache.hadoop.hbase.zookeeper.ZKUtil.connect(ZKUtil.java:136)
 at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.(ZooKeeperWatcher.java:171)
 at org.apache.hadoop.hbase.regionserver.HRegionServer.(HRegionServer.java:593)
 ... 10 more

This might happen because the Direct Memory size is not large enough to hold the offheap block cache (if configured) and some offset from HDFS client offheap buffers (usually small).

In order to recover the RegionServer change the MaxDirectMemorySize param.

You can use Ambari to update the "HBase off-heap MaxDirectMemorySize"  ro you can also update the "-XX: MaxDirectMemorySize=" param value in "hbase-env" file.

HMaster error during startup

HMaster throws error as below during startup

FATAL [ip-:16000.activeMasterManager] master.HMaster: The coprocessor org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor threw java.lang.RuntimeException: java.io.FileNotFoundException: /etc/hbase/2.5.0.55-1/0/xasecure-audit.xml (No such file or directory)

This is happening because the ranger co-processor is defined in xml config to load but the requested plugin is not installed. Remove below attributes from hbase-site.xml

  • hbase.coprocessor.master.classes
  • hbase.coprocessor.region.classes
  • hbase.coprocessor.regionserver.classes
Now restart the HMaster process. 

Unable to start namenode. Error "couldn't find resource file location"

Sometimes starting the namenode might throw below error.
  
ERROR config.RangerConfiguration (RangerConfiguration.java:addResourceIfReadable(110)) - addResourceIfReadable(ranger-hdfs-security.xml): couldn't find resource file location
INFO  provider.AuditProviderFactory (AuditProviderFactory.java:(77)) - AuditProviderFactory: creating..
FATAL conf.Configuration (Configuration.java:loadResource(2672)) - error parsing conf file:/etc/hadoop/2.5.0.55-1/0/xasecure-audit.xml
java.io.FileNotFoundException: /etc/hadoop/2.5.0.55-1/0/xasecure-audit.xml (No such file or directory)

This is causing because, ranger plugin is not installed and authorization provider class is registered for ranger. Your config (hdfs-site.xml) will have below xml attribute with the value
 
ATTRIBUTE : dfs.namenode.inode.attributes.provider.class
VALUE: org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer
    

Now search for above attribute in hdfs-site.xml and remove attribute.Now you are good to start the service.

Install PIP on CentOS 7

Python is widely used programing language and pip is the tool need to install different python packages.

Before you go ahead check if you have pip installed by using command below.
$ pip --version pip 9.0.1 from /Library/Python/2.7/site-packages (python 2.7)

You can install pip in multiple ways on CentOS. I will discuss here two ways to install pip. 

Install PIP using yup

Update the packages first, just to be updated.
$ yum -y update
Once update is complete, you can run below command to install the pip. 

$ yum -y install python-pip
If you get error, did not found anything to install python-pip then you need to enable Extra Packages for Enterprise Linux (EPEL) repository, as python-pip is part of EPEL repo. If you don't want to enable EPEL repo then you can use other installation method.

Install PIP using curl

First step is to download the python script to install pip. 
$ curl "https://bootstrap.pypa.io/get-pip.py" -o "get-pip.py"
Once the script is downloaded you can run below command to install pip.
$ python get-pip.py
If you get permission error or write error run above command using sudo

Setup python 2.7 on CentOS 7

Python is most commonly used development and scripting tool. You can setup Python on CentOS by multiple ways.

Setup Python using Yum

This is simplest method to install development tools on CentOS. Run below commands to setup the python using yum.
[vagrant@localhost ~]$ sudo yum -y update [vagrant@localhost ~]$ sudo yum groupinstall -y development
Or
[vagrant@localhost ~]$ sudo yum groupinstall -y 'development tools'

Setup Python using source code

To setup python using source code, you would first download the source code for the version you want to install. Then build the source code and configure.

You can use "wget" command to download the compressed source code. If you don't have wget installed, use yum to install wget as below.
[vagrant@localhost ~]$ sudo yum install wget
You would need "xz" tool to decode the source code.
[vagrant@localhost ~]$ sudo yum install xz-libs
Now you are ready to download the source code. 
Download the source code for python 2.7.6 using below command. You can choose which version you want to install and download sources from here.
[vagrant@localhost ~]$ wget http://www.python.org/ftp/python/2.7.6/Python-2.7.6.tar.xz
Decode the archive then extract using below command.
[vagrant@localhost ~]$ xz -d Python-2.7.6.tar.xz [vagrant@localhost ~]$ tar -xvf Python-2.7.6.tar
Traverse to extracted python source code directory then run configure.
[vagrant@localhost ~]$ cd Python-2.7.6 [vagrant@localhost ~]$ ./configure --prefix=/usr/local
Run 'make install' build the source and install the binaries. You can also use the 'altinstall' to replace older installation.
[vagrant@localhost ~]$ make altinstall
Once the installation complete, you can verify installation by running the python command. If you get the command not found then update the path variable as below.
[vagrant@localhost ~]$ export PATH="/usr/local/bin:$PATH"
Now your python is setup for use.

How to Install Java 8 on CentOS 7

Java is most common requirement for many of the Open Source softwares. You can follow below steps to install Java on CentOS vm.

You can first check the list of available packages for Java using below command.

$ yum search java | grep 'java-'


You will get the list like above. Now depending on what is your requirement you can choose the package. If you don't see any list as above, you will need to update the REPO for java.

To install JRE you can run below command.


$ sudo yum install java-1.8.0-openjdk.x86_64

To install JDK you can use below command.

$ sudo yum install java-1.8.0-openjdk*

The above commands will ask for downloading the required dependencies, press Y and enter to continue with installation. Once installation completes it will display the dependencies installed and complete message.

Now you can verify the installation by running below command.

$ java -version
openjdk version "1.8.0_144"
OpenJDK Runtime Environment (build 1.8.0_144-b01)
OpenJDK 64-Bit Server VM (build 25.144-b01, mixed mode) 

You should see the java version as shown in above output.

Setup CentOS 7 VM on VirtualBox

This is step by step guide which will let you run the CentOS 7 VM on your virtual box installation. In this tutorials are illustrated using Mac.

Installing VirtualBox

VirtualBox is a powerful x86 and AMD64/Intel64 virtualization product for enterprise as well as home use




Installing Vagrant

Vagrant manages virtual machines hosted in Oracle VirtualBox. A virtual machine is a software implementation of a computer, running a complete operating system stack on a virtualizer.

  • Download Vagrant from here. Install the vagrant.
  • Check the installation by running below command.

Setup CentOS VM

Once you are done with vagrant, now you are ready to setup your CentOS VM. 
Run below command, which will create "vagrantfile" under the current working directory. The vagrantfile can be used to configure and provision the VM.


Once you see above message you can run "vagrant up" command, which will pull the CentOS image and create a VM under virtual box, as shown in below image.


Connecting to VM

Once the CentOS is up successfully, you can check in the VirtualBox its setting as well as you can change those.

If CentOS is running then you can connect VM via ssh using below command.


Now your CentOS VM is ready to use.

You can take a snapshot of VM before use, so it can be discarded anytime and clean VM will be ready to use. You can also delete the VM using Virtualbox as well. 

Nice little changes in iOS 11

Yesterday Sept 19 2017 Apple released iOS 11. iOS 11 is bundled with lots of features, which you might be aware right now, but there are more little changes. 

  • First striking change you will notice once you get started after upgrading to iOS 11 is the icons. The icons of iOS pre-loaded apps would feel different, and would give you feel of having a new device in hand. 
  • The calculator app UI is redesigned as shown below 

  • If you take a screenshot, iOS 11 will let you edit the screenshot before saving.


  • One thing I noticed that the margin left for the header part is more in majority of the apps like settings, photos. I am not sure this thing is with iPhone 5s only or with others as well, but certainly it eats up screen real estate.



I would keep updating this list as and when I encounter new ones.

Golang: Http POST Request with JSON Body example

Go standard library comes with "net/http" package which has excellent support for HTTP Client and Server.   In order to post JSON ...