Hadoop dashboard

Now we have dashboard for out hadoop clusters. On github you can see sourcecode (link is bottom).

Read more

Better CLI status for Apache Solr

Our team have some Apache Solr servers in two clusters. But WebUI is for us realy cluttering, when you have more collections and shards.

As you can see, this is not realy useful.

Read more

One more step to autodeploy

For maintenance of our servers we use Satl Stack. Few days ago we installed packages from states and define versions in pillars. So we had for every cluster some states (e.g. workers.sls, launchers.sls, masters.sls). When we wanted add some package we had to edit state and pillar. Additionally, if we had to adjust config for this package we had edit more states. One day we said we have to find better way for install package. The way where deployers can edit (via merge request to our gitlab - where we have our salt states with pillars) configs and versions that are in productions servers.

Read more

Azkaban workflow manager + eboshi

Some days ago we looked for some scheduler for execute our MapReduce jobs. We chose Azkaban workflow manager and eboshi (because we wanted autodeploy via Salt Stack). Now I want to write about our experiences with this project.

Read more

Corrupt blocks in HDFS?

Few weeks ago we saw corrupt blocks in HDFS in jmx output (like http://namenode:50070/jmx). So we didn't start panic and run fsck. Hadoop fsck output said status HEALTHY and this was true.

...
    "PendingReplicationBlocks" : 0,
    "UnderReplicatedBlocks" : 0,
    "CorruptBlocks" : 5,
    "ScheduledReplicationBlocks" : 0,
    "PendingDeletionBlocks" : 0,
...

Result: Dont trust jmx everytime!

UPDATE: same situation is in HBase jmx and Regions In Transition (ritcount). We had some RIT, but jmx sayed "ritCount":0. JMX is bigger and biiger liars every day.