Secondary NameNode in hadoop is a specially dedicated node in HDFS cluster whose main function is to take checkpoints of the file system metadata present on namenode. It is not a backup namenode. It just checkpoints namenode’s file system namespace. The Secondary NameNode is a helper to the primary NameNode but not replace for primary namenode.
As the NameNode is the single point of failure in HDFS, if NameNode fails entire HDFS file system is lost. So in order to overcome this, Hadoop implemented Secondary NameNode whose main function is to store a copy of FsImage file and edits log file.
FsImage is a snapshot of the HDFS file system metadata at a certain point of time and EditLog is a transaction log which contains records for every change that occurs to file system metadata. So, at any point of time, applying edits log records to FsImage (recently saved copy) will give the current status of FsImage, i.e. file system metadata.
Whenever a NameNode is restarted, the latest status of FsImage is built by applying edits records on last saved copy of FsImage. Since, NameNode merges FsImage and EditLog files only at start up, the edits log file might get very large over time on a busy cluster. That means, if the EditLog is very large, NameNode restart process result in some considerable delay in the availability of file system. So, it is important keep the edits log as small as possible which is one of the main functions of Secondary NameNode.
Secondary NameNode is not a true backup Namenode and cann’t serve primary NameNode’s operations.
It usually runs on a different machine than the primary NameNode since its memory requirements are same as the primary NameNode.
Secondary NameNode Functions:
1. Stores a copy of FsImage file and edits log.2. Periodically applies edits log records to FsImage file and refreshes the edits log. And sends this updated FsImage file to NameNode so that NameNode doesn’t need to re-apply the EditLog records during its start up process. Thus Secondary NameNode makes NameNode start up process fast.
3. If NameNode is failed, File System metadata can be recovered from the last saved FsImage on the Secondary NameNode but Secondary NameNode can’t take the primary NameNode’s functionality.
4. Check pointing of File system metadata is performed.
The checkpoint process is controlled by two configuration parameters.
dfs.namenode.checkpoint.period - 1 hour by default, which is the maximum delay between two consecutive checkpoints dfs.namenode.checkpoint.txns - 1 million by default, which is the number of uncheckpointed transactions on the NameNode which will force an urgent checkpoint, even if the checkpoint period has not been reached. |
Below is the screen shot of NameNode’s current directory:
Note that fsimage version number is 165 and edits_inprogress log which is in progress from last checkpoint fsimage165 is numbered as 166 and this will be merged with fsimage165 during next restart of namenode and fsimage166 will be created.
Secondary NameNode’s current directory looks like shown below. note that there is no corresponding version of edits_inprogress_166 which are real time edits in namenode.
Below two options can be used with secondarynamenode command.
1. -geteditsize: This option is to get the size of edits_ingress file on Name Node.
2. -checkpoint [force]: Checkpoints the Secondary namenode if EditLog size >= fs.checkpoint.size. If force is used, checkpoint is created irrespective of EditLog size.
Note: Though Secondary NameNode in hadoop maintains merged copy of the last saved FsImage and EditLog, it lags the latest state of primary NameNode because this merging of FsImage and EditLog happens only at certain intervals of time, by default for every one hour.
So, If NameNode fails completely, then there will be definitely loss to at least a small amount of data. To Overcome this, Hadoop Implemented Backup Node and Checkpoint Node.
Refer the posts Backup Node and Checkpoint Node for further details on this.
No comments:
Post a Comment