Corrupt blocks in HDFS?
07 Jan 2016Few weeks ago we saw corrupt blocks in HDFS in jmx output (like http://namenode:50070/jmx). So we didn't start panic and run fsck. Hadoop fsck output said status HEALTHY and this was true.
...
"PendingReplicationBlocks" : 0,
"UnderReplicatedBlocks" : 0,
"CorruptBlocks" : 5,
"ScheduledReplicationBlocks" : 0,
"PendingDeletionBlocks" : 0,
...
Result: Dont trust jmx everytime!
UPDATE: same situation is in HBase jmx and Regions In Transition (ritcount). We had some RIT, but jmx sayed "ritCount":0
. JMX is bigger and biiger liars every day.