One more step to autodeploy


For maintenance of our servers we use Satl Stack. Few days ago we installed packages from states and define versions in pillars. So we had for every cluster some states (e.g. workers.sls, launchers.sls, masters.sls). When we wanted add some package we had to edit state and pillar. Additionally, if we had to adjust config for this package we had edit more states. One day we said we have to find better way for install package. The way where deployers can edit (via merge request to our gitlab - where we have our salt states with pillars) configs and versions that are in productions servers.

Read more


Corrupt blocks in HDFS?


Few weeks ago we saw corrupt blocks in HDFS in jmx output (like http://namenode:50070/jmx). So we didn't start panic and run fsck. Hadoop fsck output said status HEALTHY and this was true.

    "PendingReplicationBlocks" : 0,
    "UnderReplicatedBlocks" : 0,
    "CorruptBlocks" : 5,
    "ScheduledReplicationBlocks" : 0,
    "PendingDeletionBlocks" : 0,

Result: Dont trust jmx everytime!

UPDATE: same situation is in HBase jmx and Regions In Transition (ritcount). We had some RIT, but jmx sayed "ritCount":0. JMX is bigger and biiger liars every day.