Pipelines and workers in Go

I have some lists of users that I get and, for each user, I need to apply some rules (text formatting, max length, and who knows what other business rules can come up in the future), then send it further to another service. If I get the user again, I have to ignore them from the entire process. If one of the rules tells the user is not eligible, I have to stop the entire process, no need to go the next rules.

If you read the previous paragraph again, you can see some if statements that you should avoid from the technical implementation, but of course, not from the business rules:

  • If user was already processed, continue
  • If max length is exceeded, truncate
  • If a rule tells user is not OK, stop

I look at the rules as being some workers in a pipeline. Every worker does its job and sends its work to the next worker. Here’s how I’ve handled this. Continue reading Pipelines and workers in Go

Migrating an infrastructure

At the end of last year I was given the task to move an infrastructure spread on various providers (cloud and physical servers) to a new cloud. My role was to discover the entire infrastructure and services communication, come up with a strategy of how things are going to take place, then manage and supervise the sysadmin in charge of effectively moving machines.  This was a production environment, so minimum down time was desired. Most of the work was during night time.

It was a challenge from day zero, being a legacy multi component system with no specific flow. I had high level knowledge of the project, but subtle info came up every day. And I learned and used some things that I’d like to mention.

Planning everything ahead

I took each component (or group of components where required) individually and wrote down on a whiteboard their role, technologies, firewalls, running processes, storage info, communication flow with the other components, and a step-by-step process of how the migration should go. Continue reading Migrating an infrastructure

Start now

If you have some technical debt, an architecture issue, an old bug, or just some poorly developed features, now is the right moment to start changing. And I’m all pragmatic about it. No motivational stuff.

I’m talking about real problems which hold you back, waste time and money. Not about small optimizations which don’t bring you value, not every problem needs a solution.

To start doesn’t necessarily mean to actually do or to rush, you can start analyzing the need and the effort, think of required steps (technical analysis, people analysis, business requirements and so on).

Then start refactoring that old class or make a new one, maybe extract that feature into a microservice, upgrade that server, change that old technology, rethink that architecture so you can change easier in the future.

You know your business, but get fresh ideas from other people, maybe you’re too stiffed after many years on the project.

Just start the process now, because later everything will require more and more resources.

 

Why I chose to build a framework at a 3-hour interview

I encourage the use of known frameworks over custom ones, but I decided not to use one when, at an interview, I had 3 hours to prove the technical aspects I’ve been talking about at the first encounter with the employer.

The task was simple: build a REST web service (using PHP and MySQL) and make some operations on a resource, and a small JavaScript client to interact with the service.

I felt that using a known framework for a basic task like this I couldn’t show too many principles I use when developing. It wasn’t the moment to build a production ready app, it was just to show I can DO what I KNOW.

So my 3-hour journey building a framework from scratch started. It had basic components like: Continue reading Why I chose to build a framework at a 3-hour interview

Docker container with internet connection but no working DNS server

I’ve met a network setup where common DNS servers like 8.8.8.8 were not working and domain names could not be resolved inside Docker containers.

Find out the DNS servers on your system (I was using Ubuntu 16.04):

# Get the name of the interface you're using to connect to your network
ifconfig

# Then get DNS servers associated to it (I've used the first one in this list)
nmcli device show <interfacename> | grep IP4.DNS | awk '{print $2}'

 

Now you have two options (“x.x.x.x” will be the DNS you chose from above):

  • You can run containers with the dns flag: 
    docker run -tid --dns x.x.x.x ubuntu:16.04 bash
  • Or, as I did, you can configure the DNS to be used automatically by all future containers:
    • open /etc/network/interfaces
    • add “dns-nameservers x.x.x.x” after “iface lo inet loopback”
    • restart interfaces: sudo ifdown -a && sudo ifup -a 
      sudo sed -i '/iface lo inet loopback/a dns-nameservers x.x.x.x' /etc/network/interfaces
sudo ifdown -a && sudo ifup -a

Loading time in development

I’ve seen web applications taking about 1 minute to load in the browser while developing. After you hit refresh, you just wait… 40, 50, 60 seconds.

Same for background services: you write code, then “sudo service name restart” and wait 20, 30 seconds to have the service up again.

It’s very sad there are developers who are OK with that, they don’t complain, they just do their work and… wait. And some are doing this for years.

I try to understand each unpleasant situation I meet, some of them having deep roots that are difficult to break (not impossible). Although, the case of such poor loading time in development overwhelms my patience. I find no reason this should be accepted for. It hits the product, the business itself.

Avoiding IF statements for business rules

One moment a project can get on an unwanted road is when new business rules pop in. Even more if this happens on a tight deadline. You just throw some innocent if statements, and that’s how hell can unleash…

Later, other rules will be requested by business. Each will have at least one if statement and maybe interactions with some services. Then one day you notice yourself tangled in the code you’d prefer to never see.

There are ways to implement these rules fast and efficient, avoiding most of those undesired if statements. This is my situation:

  • I have a Song resource and a User resource
  • A Song can be restricted to a User based on multiple conditions
  • If a Song is restricted to a User, we have to let them know why, and this can be for reasons like:
    • Song is not released yet
    • Song was retired by the record company
    • User is from a country where we don’t have rights to offer the Song
    • User has a plan which doesn’t allow him to play the Song
    • and so on

The “what” doesn’t matter, the “how” is the key. And business folks can change their mind any time. They want a new rule fast, or cancel one, or change it. Continue reading Avoiding IF statements for business rules

Types comparison and functions

PHP will let you change a variable’s type at any moment, and there’s nothing you can do about it. While you can use data structures, argument type declaration, and return type declaration to save a lot of damage, you can still change types.

<?php

declare(strict_types=1);

function showTypes(int $number) {
    echo gettype($number) . "\n"; //integer

    $number = "string";
    echo gettype($number) . "\n"; //string
}

showTypes(2);

In order to help this situation a little bit, I’m using the appropriate type comparison operators and functions:

<?php

$number = 1;

// OK
if ($number > 0) {}

// Not OK
if (!$number) {}
if ($number === '') {}
if (strlen($number) > 0) {}


$string = 'Lorem ipsum';

// OK
if ($string === '') {}
if (strlen($string) > 0) {}

// Not OK
if (!$string) {}


$list = [];

// OK
if (count($list) > 0) {}

// Not OK
if (!$list) {}


$bool = false;

// OK
if (true === $bool) {}
if (!$bool) {}

// Not OK
if ($bool == '') {}


$var = null; // if I know a var can be null

// OK
if (null === $var) {}
if (is_null($var)) {}

// Not OK
if (!$var) {}

As such, I’m being more specific about the type variables are holding. In clean code it’s faster to see what a variable holds, but in legacy systems you can see a variable hundreds of lines later after declaration, or you can get a value from a function with no return type or DocBlock.

Treat each variable as close as possible to its type.

Using data structures for types

Data types should be very specific. Anyone using a variable should know exactly what type it is, how it looks like (if it’s a structure). While for some languages it’s common sense and really enforced, others will let you mess with a variable’s type, no matter it’s a primitive, an object, an array.

I’m going to talk about data coming from JSON, databases or other sources, which can be represented into data structures.

In a language like Go, you map data into well defined structures.

package main

import (
    "encoding/json"
    "fmt"
)

type Person struct {
    ID int
    Age int
    Name string
}

func main() {
    string := `{
        "id":453,
        "age":26,
        "name":"John Doe"
    }`

    input := []byte(string)

    person := Person{}
    err := json.Unmarshal(input, &person)
    if err != nil {
        panic(err)
    }

    fmt.Println(person) // Person struct
    fmt.Println(person.ID) // integer
    fmt.Println(person.Age) // integer
    fmt.Println(person.Name) // string
}

You know exactly that you have an object of type Person, with integer ID, integer Age, and string Name. No need to check anything anywhere. If you mess up, you’ll know at compile time.

A dynamic typed language like PHP has a different approach. Continue reading Using data structures for types

Compile Libvirt for Python on Ubuntu

Installing Libvirt on Ubuntu for Python is as easy as:

sudo apt install -y python python-pip libvirt-dev
pip install libvirt-python

 

With the default installation, you could miss some of the Libvirt API bindings exposed to the Python package, although you have the latest version.  So if you ever need to compile the library yourself, here you go (my setup was Libvirt 4.0.0 on Ubuntu 16.04 with Python 2.7):

#!/usr/bin/env bash

WORK_DIR="/tmp/libvirt"

sudo apt update
sudo apt install -y git

# LIBVIRT

WORK_DIR_LIBVIRT="$WORK_DIR/libvirt"
mkdir -p $WORK_DIR_LIBVIRT
cd $WORK_DIR_LIBVIRT

LIBVIRT_VERSION="v4.0.0"
git clone -b $LIBVIRT_VERSION --single-branch --depth 1 https://github.com/libvirt/libvirt.git .
git checkout $LIBVIRT_VERSION

sudo apt install -y \
	gettext \
	libtool \
	autoconf \
	autopoint \
	pkg-config \
	xsltproc \
	libxml2-utils
./bootstrap

sudo apt install -y \
	libnl-3-dev \
	libnl-route-3-dev \
	libxml2-dev \
	libdevmapper-dev \
	libpciaccess-dev \
	python
./configure

sudo apt install -y intltool
aclocal

make
sudo make install


# LIBVIRT PYTHON

WORK_DIR_LIBVIRT_PYTHON="$WORK_DIR/python"
mkdir -p $WORK_DIR_LIBVIRT_PYTHON
cd $WORK_DIR_LIBVIRT_PYTHON

LIBVIRT_PYTHON_VERSION="v4.0.0"
git clone -b $LIBVIRT_PYTHON_VERSION --single-branch --depth 1 https://github.com/libvirt/libvirt-python .
git checkout $LIBVIRT_PYTHON_VERSION

sudo apt install -y python-dev

python setup.py build
python setup.py install


# CLEANUP

rm -r $WORK_DIR
sudo apt purge -y \
	gettext \
	libtool \
	autoconf \
	autopoint \
	pkg-config \
	xsltproc \
	libxml2-utils \
	libnl-3-dev \
	libnl-route-3-dev \
	libxml2-dev \
	libdevmapper-dev \
	libpciaccess-dev \
	intltool \
	python-dev

The above may vary depending on your Ubuntu setup. Just pay attention to errors regarding missing tools.

Now check for the symbols you needed:

sudo apt install -y binutils
nm -g /usr/local/lib/libvirt.so

 

I recommend compilation attempts inside an isolated environment.