sysadm.cz

Welcome to Jekyll!

2022-09-28T13:40:51-05:00

You’ll find this post in your _posts directory. Go ahead and edit it and re-build the site to see your changes. You can rebuild the site in many different ways, but the most common way is to run jekyll serve, which launches a web server and auto-regenerates your site when a file is updated.

To add new posts, simply add a file in the _posts directory that follows the convention YYYY-MM-DD-name-of-post.ext and includes the necessary front matter. Take a look at the source for this post to get an idea about how it works.

Jekyll also offers powerful support for code snippets:

def print_hi(name)
  puts "Hi, #{name}"
end
print_hi('Tom')
#=> prints 'Hi, Tom' to STDOUT.

Check out the Jekyll docs for more info on how to get the most out of Jekyll. File all bugs/feature requests at Jekyll’s GitHub repo. If you have questions, you can ask them on Jekyll Talk.

Better CLI status for Apache Solr

2016-06-21T00:00:00-05:00

Our team have some Apache Solr servers in two clusters. But WebUI is for us realy cluttering, when you have more collections and shards.

As you can see, this is not realy useful.

So I wrote script for monitor state of solr cluster. This script get information from API and show it in table (script use tabulate). You can see it on my github

PS: I dont know yet if state recovery failed is in /solr/admin/collections?action=clusterstatus (but gone state is in API as down :-( ). So we will see...

Enjoy. Next i will write about bad default balance policy in Solr.

solr-status

sksolr1.ng(sksolr1.ng):~ # solr-status -H sksolr1.ko.seznam.cz
+---------+-----------+-------+----------+--------------+--------+-------------------+
| server  |   leaders |   all |   active |   recovering |   down |   recovery failed |
+=========+===========+=======+==========+==============+========+===================+
| sksolr1 |       110 |   219 |      219 |            0 |      0 |                 0 |
+---------+-----------+-------+----------+--------------+--------+-------------------+
| sksolr2 |       111 |   219 |      219 |            0 |      0 |                 0 |
+---------+-----------+-------+----------+--------------+--------+-------------------+
| sksolr3 |       110 |   222 |      222 |            0 |      0 |                 0 |
+---------+-----------+-------+----------+--------------+--------+-------------------+
| sksolr4 |       110 |   222 |      222 |            0 |      0 |                 0 |
+---------+-----------+-------+----------+--------------+--------+-------------------+
| sksolr5 |       109 |   221 |      221 |            0 |      0 |                 0 |
+---------+-----------+-------+----------+--------------+--------+-------------------+
| sksolr6 |       110 |   220 |      220 |            0 |      0 |                 0 |
+---------+-----------+-------+----------+--------------+--------+-------------------+
| sksolr7 |       110 |   219 |      219 |            0 |      0 |                 0 |
+---------+-----------+-------+----------+--------------+--------+-------------------+
| sksolr8 |       110 |   218 |      218 |            0 |      0 |                 0 |
+---------+-----------+-------+----------+--------------+--------+-------------------+

countshards  = 880
countleaders = 880

One more step to autodeploy

2016-04-08T00:00:00-05:00

For maintenance of our servers we use Satl Stack. Few days ago we installed packages from states and define versions in pillars. So we had for every cluster some states (e.g. workers.sls, launchers.sls, masters.sls). When we wanted add some package we had to edit state and pillar. Additionally, if we had to adjust config for this package we had edit more states. One day we said we have to find better way for install package. The way where deployers can edit (via merge request to our gitlab - where we have our salt states with pillars) configs and versions that are in productions servers.

We had states and pillars in this format.

state-launcher.sls

clusterlauncher:
  pkg.installed:
    - name:
      - szn-email-job: {{ pillar['szn-email-job'] }}
      - szn-email-activity: {{ pillar['szn-email-activity'] }}
      - szn-fulltext-generator: {{ pillar['szn-fulltext-generator'] }}
      - szn-fulltext-dump: {{ pillar['szn-fulltext-dump'] }}
    - refresh: True

pillar-cluster.sls

szn-email-job: 0.1.0
szn-email-activity: 0.0.17
szn-fulltext-generator: 0.1.21
szn-fulltext-dump: 0.0.3

We made adjustments to our states and pillars. What we want know about package? We want to know where to install package, his version and config file. So we chose this format for our new pillars.

new-pillar-cluster.sls

#there is two ways for define package for install
# 1)
#   package-name: 0.0.1
#
# 2)
#   package-name:
#     version: 0.0.1
#    [conf-src: salt://path/to/conf]
#    [conf-dst: /path/to/conf/on/server]
pkgs:
  - clusterlauncher\d+\.ng\.seznam\.cz:
    szn-email-job: 0.1.0
    szn-email-activity:
      version: 0.0.17
      conf-src: salt://conf/szn-email-activity/
      conf-dst: /www/szn-email-activity/conf/
    szn-fulltext-generator: 0.1.21
    szn-fulltext-dump:
      version: 0.0.3
      conf-src: salt://conf/szn-fulltext-dump/config.xml
      conf-dst: /www/szn-fulltext-dump/conf/config.xml

You will se that is optionally if we use conf or not. Plus we can copy one file or directory (if conf-dst last char is "/"). After this we must deploy state for installation packages and copy configs. But there was problem. Jinja can`t work with regex. Really! So we had to add workaround via salt module.

my_re.py

import re

def match(pattern, st, flags=0):
  return re.match(pattern, st, flags)

Ok. After this we can finaly deploy states. We wrote two states:

pkg-regex-install.sls

{% if pillar['pkgs'] is defined %}

regex-install:
  pkg.installed:
    - pkgs:
#### we can`t have empty list of pkgs for install (when for loop is empty) ####
      - apt
{% for regex, pkgs in salt['pillar.get']('pkgs', {}).iteritems() %}
  {% if salt['my_re.match'](regex, grains['fqdn']) != None %}
    {% for pkg, metadata in pkgs.iteritems() %}
      {% if 'version' in metadata %}
      - {{ pkg }}: {{ metadata['version'] }}
      {% else %}
      - {{ pkg }}: {{ metadata }}
      {% endif %}
    {% endfor %}
  {% endif %}
{% endfor %}
    - refresh: True

{% endif %}

config-regex.sls

{% if pillar['pkgs'] is defined %}

{% for regex, pkgs in salt['pillar.get']('pkgs', {}).iteritems() %}
  {% if salt['my_re.match'](regex, grains['fqdn']) != None %}
    {% for pkg, metadata in pkgs.iteritems() %}
      {% if 'conf-dst' in metadata %} 
{{ metadata['conf-dst'] }}:
        {% if metadata['conf-dst'][-1] == "/" %}
  file.recurse:
        {% else %}
  file.managed:
        {% endif %}
    - source: {{ metadata['conf-src'] }}
    - template: jinja
      {% endif %}
    {% endfor %}
  {% endif %}
{% endfor %}

{% endif %}

Now when we want to add some package we edit only one file. Plus deployers can themself send merge request with his changes in config or version. Then sysadmin only check changes and run state.highstate on salt master to server.

Azkaban workflow manager + eboshi

2016-02-16T00:00:00-06:00

Some days ago we looked for some scheduler for execute our MapReduce jobs. We chose Azkaban workflow manager and eboshi (because we wanted autodeploy via Salt Stack). Now I want to write about our experiences with this project.

What we installed:

mysql
azkaban-webserver
azkaban-executor
eboshi

We edited azkaban-webserver and azkaban-executor start scripts for Daemontools.

azkaban-web-start.sh
@@ -45,7 +45,5 @@
 fi
 AZKABAN_OPTS="$AZKABAN_OPTS -server -Dcom.sun.management.jmxremote -Djava.io.tmpdir=$tmpdir -Dexecutorport=$executorport -Dserverpath=$serverpath"
 
-java $AZKABAN_OPTS $JAVA_LIB_PATH -cp $CLASSPATH azkaban.webapp.AzkabanWebServer -conf $azkaban_dir/conf $@ &
-
-echo $! > $azkaban_dir/currentpid
+exec java $AZKABAN_OPTS $JAVA_LIB_PATH -cp $CLASSPATH azkaban.webapp.AzkabanWebServer -conf $azkaban_dir/conf $@ >> $azkaban_dir/log/azkaban-web-`date +%F`.log

When we had azkaban webserver and executor online, we created some flows for our job. Like this budgetupdatercpi-r.job

# budgetupdater cpi daily reaggregation job
type=command
dependencies=impressfraud-r
command=/etc/init.d/szn-sklik-budgetupdater daily-azkaban cpi ${date}

With help of these defined files we can create extensive flow like this

History graph of some flow executions look like this. This is a really good helper when your jobs execution time incomprehensible rises.

eboshi and autodeploy

Because we have self signed certificate for azkaban webserver we had to edit every requests call in eboshi.

-r = requests.post(url + "/schedule", data=params)
+r = requests.post(url + "/schedule", verify=False, data=params)

The schedules are created automatically from this pillar

scheduler:
  clicklinkerspark:
    time: 00,02,AM,CET
    period: 1d
  clicklinkerspark-inc:
    time: 00,02,AM,CET
    period: 30m
  coeccompressor-daily:
    time: 00,57,AM,CET
    period: 1d
  patternwebsaggregator:
    time: 06,02,AM,CET
    period: 1d

This salt state calls eboshi and creates all scheludes defined in pillar in this for loop.

{% for flow, metadata in salt['pillar.get']('scheduler', {}).iteritems() %}
{{ flow }}-create-schedule:
  cmd.run:
    - name: /usr/local/bin/eboshi addSchedule --url https://{{ grains['fqdn'] }}:8443 --username admins6 --password {{ pillar['pass.azkaban.admins6'] }} --project sklik --flow {{ flow }} --date 01/01/2016 --time {{ metadata['time'] }} --period {{ metadata['period'] }}
{% endfor %}

At pillar are defined versions of packages our jobs. Nowadays when we want edit/add some schedule we only edit pillar file and call salt "HOSTNAME" state.highstate. This call update our package and set all schedules.

Corrupt blocks in HDFS?

2016-01-07T00:00:00-06:00

Few weeks ago we saw corrupt blocks in HDFS in jmx output (like http://namenode:50070/jmx). So we didn't start panic and run fsck. Hadoop fsck output said status HEALTHY and this was true.

...
    "PendingReplicationBlocks" : 0,
    "UnderReplicatedBlocks" : 0,
    "CorruptBlocks" : 5,
    "ScheduledReplicationBlocks" : 0,
    "PendingDeletionBlocks" : 0,
...

Result: Dont trust jmx everytime!

UPDATE: same situation is in HBase jmx and Regions In Transition (ritcount). We had some RIT, but jmx sayed "ritCount":0. JMX is bigger and biiger liars every day.

Upgrade OpsCenter to version 5.2.2

2015-12-05T00:00:00-06:00

Today I decided how I upgraded OpsCenter on one of us cluster. One of reasons was that box "Cluster Health" still loading only and I want try the newest version.

This bring me some throuble (like every upgrade :-)). First I had to upgrade all datastax-agents on each node. After that on WebUI OpsCenter I saw "No Data" in some boxes (specifically a "write requests", "disk utilization", "write request latency" and "load").

Solution for this "No Data" trouble was some alters in cassandra keyspace for OpsCenter.

ALTER TABLE "OpsCenter"."events" ADD message text;
ALTER TABLE "OpsCenter"."events" ADD column_family text;
ALTER TABLE "OpsCenter"."events" ADD target_node text;
ALTER TABLE "OpsCenter"."events" ADD event_source text;
ALTER TABLE "OpsCenter"."events" ADD "keyspace" text;
ALTER TABLE "OpsCenter"."events" ADD api_source_ip text;
ALTER TABLE "OpsCenter"."events" ADD user text;
ALTER TABLE "OpsCenter"."events" ADD source_node text;
ALTER TABLE "OpsCenter"."events" with comment = '{"info": "OpsCenter management data.", "version": [5, 2, 2]}';
ALTER TABLE "OpsCenter"."rollups60" RENAME column1 to timestamp;
ALTER TABLE "OpsCenter"."rollups60" with comment = '{"info": "OpsCenter management data.", "version": [5, 2, 2]}';
ALTER TABLE "OpsCenter"."rollups300" RENAME column1 to timestamp;
ALTER TABLE "OpsCenter"."rollups300" with comment = '{"info": "OpsCenter management data.", "version": [5, 2, 2]}';
ALTER TABLE "OpsCenter"."rollups7200" RENAME column1 to timestamp;
ALTER TABLE "OpsCenter"."rollups7200" with comment = '{"info": "OpsCenter management data.", "version": [5, 2, 2]}';
ALTER TABLE "OpsCenter"."rollups86400" RENAME column1 to timestamp;
ALTER TABLE "OpsCenter"."rollups86400" with comment = '{"info": "OpsCenter management data.", "version": [5, 2, 2]}';

After this i saw data in all graph. Next step was increase replication factor on keyspace "Opscenter" and run "nodetool repair". This was rather tuning that upgrade step.

cqlsh> ALTER KEYSPACE "OpsCenter" WITH REPLICATION = { 'class' : 'NetworkTopologyStrategy', 'KO' : 3, 'NG' : 3 };

After all this steps OpsCenter was still unstable. OpsCenter and datastax-agents had connections trouble so big that the simplest way was drop keyspace for OpsCenter purge opscenter and datastax-agents and install it all again.

server:~ # apt-get purge opscenter datastax-agent
server:~ # cqlsh
cqlsh > DROP KEYSPACE "OpsCenter";
cqlsh >
server:~ # apt-get install opscenter datastax-agent
server:~ # service opscenter start
server:~ # service datastax-agent

After this Opscenter looks fine and we haven't one problem with that.

Welcome to my new page

2015-11-24T00:00:00-06:00

Hi, welcome on my new page, which is running on my Raspberry Pi under my TV :-)

I am system administrator from Czech Republic and I would like to write about my job. I am team leader BigData admins in Seznam.cz . Seznam.cz is the most visited web portal in Czech Republic. Homepage has more than 2 million visits per day.

Our team administers more than 900 physical servers with more than 10PB data. We administer services like Hadoop, Hbase, MapReduce, Yarn, Kafka, Cassandra etc. We use SaltStack, cssh and internal software for installation servers called Puzzle, which makes out job easier.

So I will write mostly about that and I hope I will write periodically.

Enjoy your visit.