Hadoop dashboard
Now we have dashboard for out hadoop clusters. On github you can see sourcecode (link is bottom).
#hadoopNow we have dashboard for out hadoop clusters. On github you can see sourcecode (link is bottom).
#hadoopOur team have some Apache Solr servers in two clusters. But WebUI is for us realy cluttering, when you have more collections and shards.
As you can see, this is not realy useful.
#solrFor maintenance of our servers we use Satl Stack. Few days ago we installed packages from states and define versions in pillars. So we had for every cluster some states (e.g. workers.sls, launchers.sls, masters.sls). When we wanted add some package we had to edit state and pillar. Additionally, if we had to adjust config for this package we had edit more states. One day we said we have to find better way for install package. The way where deployers can edit (via merge request to our gitlab - where we have our salt states with pillars) configs and versions that are in productions servers.
#saltSome days ago we looked for some scheduler for execute our MapReduce jobs. We chose Azkaban workflow manager and eboshi (because we wanted autodeploy via Salt Stack). Now I want to write about our experiences with this project.
#yarnFew weeks ago we saw corrupt blocks in HDFS in jmx output (like http://namenode:50070/jmx). So we didn't start panic and run fsck. Hadoop fsck output said status HEALTHY and this was true.
...
"PendingReplicationBlocks" : 0,
"UnderReplicatedBlocks" : 0,
"CorruptBlocks" : 5,
"ScheduledReplicationBlocks" : 0,
"PendingDeletionBlocks" : 0,
...
Result: Dont trust jmx everytime!
UPDATE: same situation is in HBase jmx and Regions In Transition (ritcount). We had some RIT, but jmx sayed "ritCount":0
. JMX is bigger and biiger liars every day.