Quickly upgrade Go to latest stable version

#!/usr/bin/env bash

installpath="/usr/local"

if [[ `whoami` != "root" ]]; then
    echo run as root
    exit 1
fi

if [[ `which jq` == "" ]]; then
    apt update && apt install -y jq
fi

if [[ `which curl` == "" ]]; then
    apt update && apt install -y curl
fi

check=`curl https://golang.org/dl/?mode=json`

stable=`echo $check | jq -r '.[0].stable'`
if [[ "$stable" != "true" ]]; then
    exit 0
fi

newversion=`echo $check | jq -r '.[0].version'`
currentversion=`$installpath/go/bin/go version 2> /dev/null`

if [[ "$currentversion" == *"$newversion"* ]]; then
    exit 0
fi

cd /tmp
file=$newversion.linux-amd64.tar.gz
`curl https://dl.google.com/go/$file > $file`
`rm -rf $installpath/go/`
`tar -C $installpath -xzf $file`
`rm $file`

Trim string starting suffix

Today I wanted to remove, from a string, the substring starting with a specified string.

For a line of code with a comment started with #, I wanted to remove everything starting at #, so the line “line with #comm#ent” should become “line with “.

The test for this is:

func TestRemoveFromStringAfter(t *testing.T) {
   tests := []struct {
      input,
      after,
      expected string
   }{
      {
         input:    "line with #comm#ent",
         after:    "#",
         expected: "line with ",
      },
      {
         input:    "line to clean",
         after:    "abc",
         expected: "line to clean",
      },
      {
         input:    "line to clean",
         after:    "l",
         expected: "",
      },
      {
         input:    "",
         after:    "",
         expected: "",
      },
      {
         input:    " ",
         after:    "",
         expected: " ",
      },
   }

   for i, test := range tests {
      result := RemoveFromStringAfter(test.input, test.after)
      if result != test.expected {
         t.Fatalf("Failed at test: %d", i)
      }
   }
}

I tried to use TrimSuffix and TrimFunc from the strings package, but they weren’t getting me where I wanted. Then, all of a sudden, it stroke me: a string can be treated as a slice and a subslice is what I need. A subslice which ends right before the position of the suffix I give.

So I take the position of the suffix and extract a substring of the input string:

func RemoveFromStringAfter(input, after string) string {
   if after == "" {
      return input
   }

   if index := strings.Index(input, after); index > -1 {
      input = input[:index]
   }

   return input
}

QA tools for Go apps in CI/CD

Go Meta Linter is a great tool to run code quality checks including vet, static analysis, security, linting and others. I’ve used it a few times, enjoyed it, and I’ve built a basic setup to include it into CI/CD, along with the unit tests execution.

All you need is Docker and a Docker images repository like Docker Hub. You’ll build an image to run the tools in, push it to your repository, then pull it on your CI/CD machine and run a container from it, as simply as:

docker run -ti --rm \
    -e PKG=github.com/andreiavrammsd/dotenv-editor \
    -e CONFIG=dev/.gometalinter.json \
    -v $PWD:/app \
    yourdockerusername/go-qa-tools \
    make

Of course, it can be integrated into a service like Travis:

sudo: required

language: minimal

install:
- docker pull andreiavrammsd/go-qa-tools

script:
- docker run -ti -e PKG=github.com/andreiavrammsd/dotenv-editor -e CONFIG=dev/.gometalinter.json -v $PWD:/app andreiavrammsd/go-qa-tools make

See the full Go QA tools setup on Github.

Match sorted and unsorted integers

I was wondering if there’s a performance difference between matching the integers from two slices, once if the numbers are sorted and once if they’re not. I didn’t stress the hell out of the situation, I went up to 10k numbers.

For small sets, of course, the difference is not worth mentioning. For large slices, if you really, really focus on performance, you could be better with sorted values, if the values are already sorted; if you sort them each time, the loss will be there.

var a = []int{ ... }
var b = []int{ ... }

func IterateNotSorted() int {
   count := 0
   for _, i := range a {
      for _, j := range b {
         if i == j {
            count++
            break
         }
      }
   }

   return count
}

var c = []int{ ... }
var d = []int{ ... }

func IterateSorted() int {
   count := 0
   for _, i := range c {
      for _, j := range d {
         if i == j {
            count++
            break
         }
      }
   }

   return count
}

Fill in the slices with some numbers and test it yourself.

func BenchmarkIterateNotSorted(b *testing.B) {
   for n := 0; n < b.N; n++ {
      IterateNotSorted()
   }
}

func BenchmarkIterateSorted(b *testing.B) {
   for n := 0; n < b.N; n++ {
      IterateSorted()
   }
}

 

Apixu Go: A Golang package for Apixu weather service

Not long ago I’ve mentioned Apixu in a post about handling errors. I’ve find out about this service on DevForum, a development discussions platform I visit daily. What I like the most about Apixu is that they have various languages libraries for consuming their API. Not great libraries and not all of them are complete, but they try to offer as much variations as they can for their service.

I noticed they were missing a Go library and I was missing an idea to learn new things on. And I just started writing the code until it got to a full package which covers all API methods with error handling, both JSON and XML formats, unit tested, versioned.

It has a simple interface which clearly defines the API methods with their input parameters and responses. And it can be extended for custom needs.

Some  important things I learned from the process are simplicity, segregation and isolation, specific errors, memory management, and creating custom marshalers.

Check it out on Github. See documentation for the package and for the API.

Report for github.com/andreiavrammsd/apixu-go GoDoc for github.com/andreiavrammsd/apixu-go

In the end, they adopted my package among their official ones.

Unit testing and interfaces

  • Good code needs tests
  • Tests require good design
  • Good design implies decoupling
  • Interfaces help decouple
  • Decoupling lets you write tests
  • Tests help having good code

Good code and unit testing come hand in hand, and sometimes the bridge between them are interfaces. When you have an interface, you can easily “hide” any implementation behind it, even a mock for a unit test.

An important subject of unit testing is managing external dependencies. The tests should directly cover the unit while using fake replacements (mocks) for the dependencies.

I was given the following code and asked to write tests for it:

package mail

import (
   "fmt"
   "net"
   "net/smtp"
   "strings"
)

func ValidateHost(email string) (err error) {
   mx, err := net.LookupMX(host(email))
   if err != nil {
      return err
   }

   client, err := smtp.Dial(fmt.Sprintf("%s:%d", mx[0].Host, 25))
   if err != nil {
      return err
   }

   defer func() {
      if er := client.Close(); er != nil {
         err = er
      }
   }()

   if err = client.Hello("checkmail.me"); err != nil {
      return err
   }
   if err = client.Mail("testing-email-host@gmail.com"); err != nil {
      return err
   }
   return client.Rcpt(email)
}

func host(email string) (host string) {
   i := strings.LastIndexByte(email, '@')
   return email[i+1:]
}

The first steps were to identify test cases and dependencies: Continue reading Unit testing and interfaces

PostgreSQL batch operations in Go

Consider the following case: When creating a user (database insert) with their profile (another insert), other users must be updated (database update) with a new score value. Score is just a float for which a dummy formula will be used. And then an action record is needed (insert), which marks the fact that a user was created.

The tech context is PostgreSQL in Go with pgx as database driver and Echo framework for the HTTP server. The database setup is straight forward using Docker; it also includes a database management interface which will be available at http://localhost:54321. If you clone the sample repository, and start the setup with Docker Compose (docker compose up -d), when the PostgreSQL Docker container is built, a database is created with the schema used in this post.

CREATE TABLE "users" (
  "id" serial NOT NULL,
  "username" CHARACTER VARYING (100) NOT NULL,
  "score" DECIMAL NOT NULL DEFAULT 0,
  "created" TIMESTAMP(0) WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
  "updated" TIMESTAMP(0) WITH TIME ZONE
);

CREATE TABLE "user_profile" (
  "user_id" INTEGER NOT NULL,
  "firstname" CHARACTER VARYING (100) NOT NULL,
  "lastname" CHARACTER VARYING (100) NOT NULL
);

CREATE TABLE "actions" (
  "id" serial NOT NULL,
  "description" text NOT NULL,
  "created" TIMESTAMP(0) WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP
);

Data integrity is of interest, so all the queries will be sent on a database transaction. And because there are multiple user update queries, they will be sent all at the same time in a batch of operations. Continue reading PostgreSQL batch operations in Go

Unmarshal JSON and XML into the same Go structure slice with proper data type

I’m consuming a REST service which gives various lists in both JSON and XML formats, something similar to these ones:

[  
   {  
      "id":803267,
      "name":"Paris, Ile-de-France, France",
      "region":"Ile-de-France",
      "country":"France",
      "is_day":1,
      "localtime":"2018-05-12 12:53"
   },
   {  
      "id":760995,
      "name":"Batignolles, Ile-de-France, France",
      "region":"Ile-de-France",
      "country":"France",
      "is_day":0,
      "localtime":"2018-05-12"
   }
]
<?xml version="1.0" encoding="UTF-8"?>
<root>
   <geo>
      <id>803267</id>
      <name>Paris, Ile-de-France, France</name>
      <region>Ile-de-France</region>
      <country>France</country>
      <is_day>1</is_day>
     <localtime>2018-05-12 12:53</localtime>
   </geo>
   <geo>
      <id>760995</id>
      <name>Batignolles, Ile-de-France, France</name>
      <region>Ile-de-France</region>
      <country>France</country>
      <is_day>0</is_day>
      <localtime>2018-05-12</localtime>
   </geo>
</root>

And I wanted to get them into a slice of this type of structures:

type Locations []Location

type Location struct {
   ID        int      `json:"id" xml:"id"`
   Name      string   `json:"name" xml:"name"`
   Region    string   `json:"region" xml:"region"`
   Country   string   `json:"country" xml:"country"`
   IsDay     int      `json:"is_day" xml:"is_day"`
   LocalTime string   `json:"localtime" xml:"localtime"`
}

It’s straight forward for JSON: Continue reading Unmarshal JSON and XML into the same Go structure slice with proper data type

API authorization through middlewares

When dealing with API authorization based on access tokens, permissions (user can see a list, can create an item, can delete one etc) and/or account types (administrator, moderator, normal user etc), I’ve seen the approach of checking requirements inside the HTTP handlers functions.

This post and attached source code do not handle security, API design, data storage patterns or any other best practices that do not aim directly at the main subject: Authorization through middlewares. All other code is just for illustrating the idea as a whole.

A classical way of dealing various authorization checks is to verify everything inside the handler function.

type User struct {
   Username    string
   Type        string
   Permissions uint8
}

var CanDoAction uint8 = 1

func tokenIsValid(token string) bool {
   // ...
   return true
}

func getUserHandler(c echo.Context) error {
   // Check authorization token
   token := c.Get("token").(string)
   if !tokenIsValid(token) {
      return c.NoContent(http.StatusUnauthorized)
   }

   user := c.Get("user").(User)

   // Check account type
   if user.Type != "admin" {
      return c.NoContent(http.StatusForbidden)
   }

   // Check permission for handler
   if user.Permissions&CanDoAction == 0 {
      return c.NoContent(http.StatusForbidden)
   }

   // Get data and send it as response
   data := struct {
      Username string `json:"username"`
   }{
      Username: user.Username,
   }

   return c.JSON(http.StatusOK, data)
}

The handler is doing more than its purpose. Continue reading API authorization through middlewares