The mid-language crisis

A while after I wrote the Apixu libraries for Go and PHP, they asked me if I could help them with some issue a client had when trying to install the Python library. I quickly tested and sent them steps on how to install all requirements. And I noticed the library could use some additions.

My experience with Python was of a few lines and I felt like it would be a good context to write a few more, to understand the language a little better, to learn about its requirements, ways to set up a library and write some tests to validate JSON schemas. A clean language with a simple package manager and clear error handling. Basic integrations with Flask and Django were pretty smooth.

One thing I like about Apixu is they have multiple libraries and after freshening the Python one I was inspired to do more. And there was where to choose from.

JavaScript is an old friend but we met only in the browser some years ago. It was time to face its server side of the moon and add some touches to the NodeJS library. I’m not a huge fan of callbacks, but Promises are indeed a nice way to handle responses. Express was easy to start with, I like its micro framework feeling. The package manager, npm, doesn’t seem too far from PHP’s composer, we got along well. Continue reading The mid-language crisis

Encrypt column in PostgreSQL

PostgreSQL has a nice encryption (and hashing) module called pgcrypto which is easy to use. Using a provided key, you can quickly encrypt a column which contains sensitive information.

While the setup is fast and the usage is simple, there could be some disadvantages in some contexts:

  • be careful how you send the encryption key to the database server (if public, use SSL for transport, else keep it in a private network at least)
  • data is encrypted/decrypted in the database, so the transport is in plain (watch out for memory dump attacks)
  • some queries are very slow, as the decrypt operation is performed on the entire table if you want to sort or filter by encrypted columns

Continue reading Encrypt column in PostgreSQL

PostgreSQL NOTNULL vs IS NOT NULL vs NOT ISNULL

Today I’ve found out a flavor of PostgreSQL when checking if a composite type column is not null. There are different ways of checking for not null columns:

  • NOTNULL
  • IS NOT NULL
  • NOT ISNULL

For primitive values any of the above works, while for the composite types there is a particularity.

Given a table of users with a deleted column of a composite type action, and some users marked as deleted, I need all users that are deleted. Continue reading PostgreSQL NOTNULL vs IS NOT NULL vs NOT ISNULL

Bitbucket to GitHub import tool

GitHub now offers free private repositories. For those wanting to transfer their private repositories from Bitbucket to GitHub, I’ve written a very basic tool to help you with this.

I’ve tested it only with some git repositories, most probably there are cases I didn’t reach, so adjustments could be needed.

Check it on Bit… GitHub.

You can also import repositories manually if you wish.

Quickly upgrade Go to latest stable version

#!/usr/bin/env bash

installpath="/usr/local"

if [[ `whoami` != "root" ]]; then
    echo run as root
    exit 1
fi

if [[ `which jq` == "" ]]; then
    apt update && apt install -y jq
fi

if [[ `which curl` == "" ]]; then
    apt update && apt install -y curl
fi

check=`curl https://golang.org/dl/?mode=json`

stable=`echo $check | jq -r '.[0].stable'`
if [[ "$stable" != "true" ]]; then
    exit 0
fi

newversion=`echo $check | jq -r '.[0].version'`
currentversion=`$installpath/go/bin/go version 2> /dev/null`

if [[ "$currentversion" == *"$newversion"* ]]; then
    exit 0
fi

cd /tmp
file=$newversion.linux-amd64.tar.gz
`curl https://dl.google.com/go/$file > $file`
`rm -rf $installpath/go/`
`tar -C $installpath -xzf $file`
`rm $file`

Trim string starting suffix

Today I wanted to remove, from a string, the substring starting with a specified string.

For a line of code with a comment started with #, I wanted to remove everything starting at #, so the line “line with #comm#ent” should become “line with “.

The test for this is:

func TestRemoveFromStringAfter(t *testing.T) {
   tests := []struct {
      input,
      after,
      expected string
   }{
      {
         input:    "line with #comm#ent",
         after:    "#",
         expected: "line with ",
      },
      {
         input:    "line to clean",
         after:    "abc",
         expected: "line to clean",
      },
      {
         input:    "line to clean",
         after:    "l",
         expected: "",
      },
      {
         input:    "",
         after:    "",
         expected: "",
      },
      {
         input:    " ",
         after:    "",
         expected: " ",
      },
   }

   for i, test := range tests {
      result := RemoveFromStringAfter(test.input, test.after)
      if result != test.expected {
         t.Fatalf("Failed at test: %d", i)
      }
   }
}

I tried to use TrimSuffix and TrimFunc from the strings package, but they weren’t getting me where I wanted. Then, all of a sudden, it stroke me: a string can be treated as a slice and a subslice is what I need. A subslice which ends right before the position of the suffix I give.

So I take the position of the suffix and extract a substring of the input string:

func RemoveFromStringAfter(input, after string) string {
   if after == "" {
      return input
   }

   if index := strings.Index(input, after); index > -1 {
      input = input[:index]
   }

   return input
}

Test multiple PHP exceptions

There are cases when you have a base exception class which is extended by other exceptions. For a package called User, you could have a UserException which is inherited by UserNotFoundException, UserPasswordException and so on.

The base exception of your package allows you to catch any thrown exception from inside the package.

So there’s a base exception, a specific one and the service using them:

class BaseException extends Exception
{
}

class NoItemsException extends BaseException
{
}

class Service
{
    /**
     * @param array $items
     * @throws NoItemsException
     */
    public function save(array $items)
    {
        if (count($items) === 0) {
            throw new NoItemsException('Nothing to save');
        }
    }
}

And you want to catch all package exceptions:

$service = new Service();
try {
    $service->save([]);
} catch (BaseException $e) {
    die($e->getMessage());
}

And a test for the case:

class ServiceTest extends TestCase
{
    /**
     * @expectedException NoItemsException
     */
    public function testSaveWithNoItems()
    {
        $service = new Service();
        $service->save([]);
    }
}

How do you make sure NoItemsException extends BaseException?
With the above test, if you change NoItemsException to extend Exception instead of BaseException, the test will still pass, but the behaviour won’t be the expected one, because you won’t be catching the exception anymore:

Fatal error: Uncaught NoItemsException: Nothing to save... in /src/Service.php on line 20

NoItemsException: Nothing to save... in /src/Service.php on line 20

Your test must explicitly test both exceptions:

class ServiceTest extends TestCase
{
    public function testSaveWithNoItems()
    {
        $service = new Service();

        try {
            $service->save([]);
            $this->fail('No exception thrown');
        } catch (\Exception $e) {
            $this->assertInstanceOf(BaseException::class, $e);
            $this->assertInstanceOf(NoItemsException::class, $e);
        }
    }
}

QA tools for Go apps in CI/CD

Go Meta Linter is a great tool to run code quality checks including vet, static analysis, security, linting and others. I’ve used it a few times, enjoyed it, and I’ve built a basic setup to include it into CI/CD, along with the unit tests execution.

All you need is Docker and a Docker images repository like Docker Hub. You’ll build an image to run the tools in, push it to your repository, then pull it on your CI/CD machine and run a container from it, as simply as:

docker run -ti --rm \
    -e PKG=github.com/andreiavrammsd/dotenv-editor \
    -e CONFIG=dev/.gometalinter.json \
    -v $PWD:/app \
    yourdockerusername/go-qa-tools \
    make

(later edit: I archived the example package)

Of course, it can be integrated into a service like Travis:

sudo: required

language: minimal

install:
- docker pull andreiavrammsd/go-qa-tools

script:
- docker run -ti -e PKG=github.com/andreiavrammsd/dotenv-editor -e CONFIG=dev/.gometalinter.json -v $PWD:/app andreiavrammsd/go-qa-tools make

See the full Go QA tools setup on GitHub.

Database migrations

Database migrations can be easily integrated into your deploy system, running as a decoupled process, so it can be replaced anytime by other tools if needed, and working with it without interfering with the project itself.

The entire process can be isolated into a Docker container or the tools can be all installed directly on your machine. Presented setup is for CentOS.

Let’s assume the following context:

– Machine to run the migrations from (with Docker installed)
– MySQL database with access from the machine mentioned above
– A secrets manager to keep the database access credentials safe
– Git repository holding the migration files included (there will be a directory with all the migration files in the proper format).
– Private SSH key to access the above mentioned repository

Every time you deploy your app, you could run all the migrations you committed to your repository. Your deploy system should trigger the migration tool at the proper moment.

The key in this setup is migrate, a flexible tool which I had no problems with.
As presented in this Dockerfile, there are different tools used to perform each required step:
– Get migration files from the repository
– Get a secret string with database credentials from the secrets manager
– Extract the database credentials from the secret string
– Execute the migrations

Take a look at the full setup on GitHub.

PHP unit testing with real coverage

If you really need to cover all your code by tests, watch out for your short if statements.

Given the following class:

<?php

class Person
{
    /**
     * @var string
     */
    private $gender;

    /**
     * @param string $gender
     */
    public function setGender(string $gender)
    {
        $this->gender = $gender;
    }

    /**
     * @return string
     */
    public function getTitle() : string
    {
        return $this->gender === 'f' ? 'Mrs.' : 'Mr.';
    }
}

And a PHP Unit test:

<?php

use PHPUnit\Framework\TestCase;

class PersonTest extends TestCase
{
    /**
     * @dataProvider gendersAndTitle
     * @param $gender
     * @param $expectedTitle
     */
    public function testTitle($gender, $expectedTitle)
    {
        $person = new Person();
        $person->setGender($gender);

        $title = $person->getTitle();
        $this->assertEquals($expectedTitle, $title);
    }

    public function gendersAndTitle() : array
    {
        return [
            ['f', 'Mrs.'],
        ];
    }
}

If you run the test with coverage, you get a 100% coverage. But the data provider has only data for the “f/Mrs.” case, so the else branch of the short if is not actually tested, though the tested code reached the line while running the test.

Update the getTitle method from Person class using the normal if statement:

public function getTitle() : string
{
    if ($this->gender === 'f') {
        return 'Mrs.';
    }

    return 'Mr.';
}

Execute the test again and you get 80% coverage.

Here’s a Dockerfile to quickly test it yourself:

FROM php:7.2-cli-alpine3.8

RUN apk add --update --no-cache make alpine-sdk autoconf && \
    pecl install xdebug && \
    docker-php-ext-enable xdebug && \
    apk del alpine-sdk autoconf && \
    wget -O phpunit https://phar.phpunit.de/phpunit-6.phar && chmod +x phpunit

WORKDIR /src

Save the Person class to Person.php file and the test to PersonTest.php.

docker build -t phpunit-coverage .
docker run --rm -ti -v $PWD:/src phpunit-coverage sh

./phpunit --bootstrap Person.php --coverage-html coverage --whitelist Person.php .

See the coverage directory (index.html) created after running the test.

Clean up when you’re done:

docker rmi phpunit-coverage