12 November 2013

FSImage analysis using Pig -- Scalable NameDistribution

HDFS-1110 introduces an interesting feature to reduce memory footprint of the NameNode. In a nutshell you'll probably end up having duplicates in name of files (for instance part-00000). This feature allows same filenames to share the same byte array storing the name in the FSDirectory structure in order to save memory.

The presented solution is not prefect as the name duplicates are found at NameNode startup time when the FSImage is loaded from the disk. This means that if you never stops your NameNode you'll never take advantage of names de-duplication. Ok in the real world it's not really the case :). Well, let's move forward.

The patch adds an additional processor in the OIV, the Offline Image Viewer, in order to compute the name distribution (or said differently count the number of filenames and sort them by descending order to display the most popular filenames).

The problem with OIV's NameDistribution processor is that it keeps all the names in memory and increment counters for each filename. This can result in OutOfMemory exception if your FSImage is too big.

So the idea here is to redo the NameDistribution but using Apache Pig.

Detailed Steps

The FSImage, once dumped out of the NameNode, needs to be converted into a textual format to enable Pig to read it (this step is not explicitly specified in the Offline Image Viewer documentation, which is somehow misleading to new users). After being uploaded into HDFS, a Pig script is run

1. Retrieve the latest FSImage from the (active) NameNode
curl --silent "http://${activeNamenode}:50070/getimage?getimage=1&txid=latest" -o fsimage.bin

2. Convert the fsimage to textual "delimited" representation
hdfs oiv -i fsimage.bin -o fsimage.delim -p Delimited

3. Upload the fsimage.delim to HDFS
hdfs dfs -copyFromLocal fsimage.delim /tmp

4. Run the pig script given below
pig -p fsimage=/tmp/fsimage.delim -f nameDistribution.pig

5. Retrieve the results
hdfs dfs -cat /tmp/nameDistribution/part*

--- nameDistribution.pig ---

set job.name 'FSImage NameDistribution'

files = LOAD '$fsimage' USING PigStorage('\t') AS (path:chararray, replication:int, modTime:chararray, accessTime:chararray, blockSize:long, numBlocks:int, fileSize:long, NamespaceQuota:int, DiskspaceQuota:int, perms:chararray, username:chararray, groupname:chararray);

filenames = FOREACH files GENERATE
    SUBSTRING(path, (int)LAST_INDEX_OF(path, '/'),
    (int)SIZE(path)) as filename;

filenamesGroup = GROUP filenames BY filename;

filenamesCount = FOREACH filenamesGroup GENERATE 
    group as filename, 
    COUNT(filenames) as n;

filenamesCount = ORDER filenamesCount BY n DESC;

STORE filenamesCount INTO '/tmp/nameDistribution';

No comments:

Post a Comment