Go Future with Reflection

While exercising Go with various Future implementations, at one moment reflection hit me. Using reflection to inspect types beats the point of strong types, but it never hurts to play.

I want to use any function within a Future and then call it asynchronous and wait for results. I could have a function to create the Future which holds the user function. Not having generics yet, any function could be passed by interface{}, then called by reflection.

So the user function is passed to a function which returns another function that can be called with the user function arguments. The user function will be executed on a new routine.

// Callable is the function to call in order to start running the passed function.
type Callable func(args ...interface{}) *Future

// New returns a Callable.
func New(function interface{}) Callable {

For a sum function, the usage is:

sum := func(a, b int) (int, error) {
   return a + b, nil
}
future := futurereflect.New(sum)(1, 2)

result, err := future.Result()

The Future should allow waiting for the user function to be executed and get its result.

type Future struct {
   functionType  reflect.Type
   functionValue reflect.Value
   args          []interface{}
   wait          chan struct{}
   result        interface{}
   err           error
}

// Wait blocks until function is done.
func (f *Future) Wait() {
   <-f.wait
}

// Result retrieves result and error. It blocks if function is not done.
func (f *Future) Result() (interface{}, error) {
   <-f.wait
   return f.result, f.err
}

The rest is just implementation, but it feels like pure madness. I did it for fun, to experiment, there are cases missing. And a language should not be pushed where it doesn’t belong. Use each language in its style.

Run the code on Go Playground.

Continue reading Go Future with Reflection

CGO examples

Nothing new or special regarding CGO. I have been exploring it lately and it took me some time to understand its deep aspects, mostly on passing Go callbacks to C. And I wanted to keep close some full examples.

I created a small repository on GitHub which shows interactions of Go code with a C library (included) by passing and/or receiving numbers, strings, byte arrays, structs, enums, and calling Go functions from the C library.

Sources of inspiration and learning:

See the GitHub repository with the CGO examples and compile the code yourself on your machine or inside the Docker environment you’ll find there.

Custom Go package manager

I was working on a Go project with no modules support and introducing the modules was not an option at the time. All dependencies were installed by go get. When breaking changes were introduced in some packages, the builds started failing.

A simple way to quickly fix the issue was to write a very basic package manager which could get dependencies by the specified version (which go get does not support in GOPATH mode).

A package list can be a simple file which declares packages line by line. If no version specified, go get will be used. If a release version, commit hash or branch is specified, a git clone and checkout on the specified version will be invoked.

github.com/gorilla/mux@v1.7.2
github.com/go-playground/validator
github.com/apixu/apixu-go@0fe1e52

I wrote a Bash script which parses the file and handles the process for each of the two cases.

#!/usr/bin/env bash

FILE=$PWD/packages

[[ -z "${GOPATH}" ]] && echo "GOPATH not set" && exit 1;

GOSRC=${GOPATH}/src
[[ ! -d "${GOSRC}" ]] && echo "${GOSRC} directory does not exist" && exit 1;

while IFS= read -r line
do
    IFS=\@ read -a fields <<< "$line"
    pkg=${fields[0]}
    v=${fields[1]}

    echo ${GOSRC}/${pkg}

    if [[ -z ${v} ]]
    then
        go get -u ${pkg}
    else
        PKGSRC=${GOSRC}/${pkg}
        rm -rf ${PKGSRC}
        mkdir -p ${PKGSRC}
        cd ${PKGSRC}
        git clone https://${pkg} .
        git checkout ${v}
        cd - > /dev/null
    fi

    echo
done < "${FILE}"

Abstract dependencies

Projects depend on packages, internal ones or from 3rd parties. In all cases, your architecture should dictate what it needs and not evolve around a package, otherwise changing the dependency with another package and mocking it in tests could take unnecessary effort and time.

While it’s very easy to include a package in any file you need and start using it, time will show it’s painful to spread dependencies all over. Someday you’ll want to change a package because it’s not maintained anymore or you discover security issues which lead to loss of trust, or maybe you just want to experiment with another package.

This is where abstraction comes in, helping to decouple an implementation in your project, to set the rules which a dependency must follow. Each package has its own API, thus you need to wrap it by your API.

To show an example, I’ve chosen to integrate a validation package into Echo framework. The validator package requires you to tag your structs with its validation rules. Continue reading Abstract dependencies

Error handling in Echo framework

If misused, error handling and logging in Go can become too verbose and tangled. If the two are decoupled, you can just pass the error forward (maybe add some context to it if necessary, or have specific error structs) and have it logged or sent to various systems (like New Relic) somewhere else in the code, not where you receive it from a function call.

One of the features I appreciate the most in Echo framework is the HTTPErrorHandler which can be customized to any needs. And combined with the recover middleware, error management becomes an easy task.

package main

import (
   "errors"
   "fmt"
   "net/http"

   "github.com/labstack/echo/v4"
   "github.com/labstack/echo/v4/middleware"
   "github.com/labstack/gommon/log"
)

func main() {
   server := echo.New()
   server.Use(
      middleware.Recover(),   // Recover from all panics to always have your server up
      middleware.Logger(),    // Log everything to stdout
      middleware.RequestID(), // Generate a request id on the HTTP response headers for identification
   )
   server.Debug = false
   server.HideBanner = true
   server.HTTPErrorHandler = func(err error, c echo.Context) {
      // Take required information from error and context and send it to a service like New Relic
      fmt.Println(c.Path(), c.QueryParams(), err.Error())

      // Call the default handler to return the HTTP response
      server.DefaultHTTPErrorHandler(err, c)
   }

   server.GET("/users", func(c echo.Context) error {
      users, err := dbGetUsers()
      if err != nil {
         return err
      }

      return c.JSON(http.StatusOK, users)
   })

   server.GET("/posts", func(c echo.Context) error {
      posts, err := dbGetPosts()
      if err != nil {
         return err
      }

      return c.JSON(http.StatusOK, posts)
   })

   log.Fatal(server.Start(":8088"))
}

func dbGetUsers() ([]string, error) {
   return nil, errors.New("database error")
}

func dbGetPosts() ([]string, error) {
   panic("an unhandled error occurred")
   return nil, nil
}

Avoid logging in functions:

server.GET("/users", func(c echo.Context) error {
   users, err := dbGetUsers()
   if err != nil {
      log.Errorf("cannot get users: %s", err)
      return err
   }

   return c.JSON(http.StatusOK, users)
})

Keep the handler functions clean, check your errors and pass them forward.

Automated Jenkins CI setup for Go projects

I was helping a friend on a project with some tasks amongst which code quality. I’ve set up some tools (gometalinter first, then golangci-lint) and integrated them with Travis CI.

Then GitHub decided to give free private repositories and my friend made the project private. Travis and other CI systems require paid plans for private repositories so I decided to set up a Jenkins environment. I’ve used Jenkins before, but never configured it myself. I thought it’s going to be a quick click-click install process, but I’ve found myself in front of various plugins, configurations, credentials and all sort of requirements.

One of the first things that come in my mind when I have to do something is if I’ll have to do it again later and if I should automate the process. And that’s how a pet project was born. My purpose was to write a configuration file, install the environment, then configure jobs for the repositories I need.

I’ve created two job templates, one for triggering a build on branch push, one for pull requests (required webhooks are created automatically). Based on the templates, I just wanted to create a new job, insert the GitHub repository url and start using the job.

The job templates are aiming at Go projects to run tests and code quality tools on (all required tools are installed automatically). Other configured actions are automatic backups and cleanups. And you’ll also find some setup scripts for basic server security and requirements.

There are things which could be improved, but now I have a click-click CI system setup called Go Jenkins CI.

 

 

Quickly upgrade Go to latest stable version

#!/usr/bin/env bash

installpath="/usr/local"

if [[ `whoami` != "root" ]]; then
    echo run as root
    exit 1
fi

if [[ `which jq` == "" ]]; then
    apt update && apt install -y jq
fi

if [[ `which curl` == "" ]]; then
    apt update && apt install -y curl
fi

check=`curl https://golang.org/dl/?mode=json`

stable=`echo $check | jq -r '.[0].stable'`
if [[ "$stable" != "true" ]]; then
    exit 0
fi

newversion=`echo $check | jq -r '.[0].version'`
currentversion=`$installpath/go/bin/go version 2> /dev/null`

if [[ "$currentversion" == *"$newversion"* ]]; then
    exit 0
fi

cd /tmp
file=$newversion.linux-amd64.tar.gz
`curl https://dl.google.com/go/$file > $file`
`rm -rf $installpath/go/`
`tar -C $installpath -xzf $file`
`rm $file`

Trim string starting suffix

Today I wanted to remove, from a string, the substring starting with a specified string.

For a line of code with a comment started with #, I wanted to remove everything starting at #, so the line “line with #comm#ent” should become “line with “.

The test for this is:

func TestRemoveFromStringAfter(t *testing.T) {
   tests := []struct {
      input,
      after,
      expected string
   }{
      {
         input:    "line with #comm#ent",
         after:    "#",
         expected: "line with ",
      },
      {
         input:    "line to clean",
         after:    "abc",
         expected: "line to clean",
      },
      {
         input:    "line to clean",
         after:    "l",
         expected: "",
      },
      {
         input:    "",
         after:    "",
         expected: "",
      },
      {
         input:    " ",
         after:    "",
         expected: " ",
      },
   }

   for i, test := range tests {
      result := RemoveFromStringAfter(test.input, test.after)
      if result != test.expected {
         t.Fatalf("Failed at test: %d", i)
      }
   }
}

I tried to use TrimSuffix and TrimFunc from the strings package, but they weren’t getting me where I wanted. Then, all of a sudden, it stroke me: a string can be treated as a slice and a subslice is what I need. A subslice which ends right before the position of the suffix I give.

So I take the position of the suffix and extract a substring of the input string:

func RemoveFromStringAfter(input, after string) string {
   if after == "" {
      return input
   }

   if index := strings.Index(input, after); index > -1 {
      input = input[:index]
   }

   return input
}

QA tools for Go apps in CI/CD

Go Meta Linter is a great tool to run code quality checks including vet, static analysis, security, linting and others. I’ve used it a few times, enjoyed it, and I’ve built a basic setup to include it into CI/CD, along with the unit tests execution.

All you need is Docker and a Docker images repository like Docker Hub. You’ll build an image to run the tools in, push it to your repository, then pull it on your CI/CD machine and run a container from it, as simply as:

docker run -ti --rm \
    -e PKG=github.com/andreiavrammsd/dotenv-editor \
    -e CONFIG=dev/.gometalinter.json \
    -v $PWD:/app \
    yourdockerusername/go-qa-tools \
    make

Of course, it can be integrated into a service like Travis:

sudo: required

language: minimal

install:
- docker pull andreiavrammsd/go-qa-tools

script:
- docker run -ti -e PKG=github.com/andreiavrammsd/dotenv-editor -e CONFIG=dev/.gometalinter.json -v $PWD:/app andreiavrammsd/go-qa-tools make

See the full Go QA tools setup on Github.

Match sorted and unsorted integers

I was wondering if there’s a performance difference between matching the integers from two slices, once if the numbers are sorted and once if they’re not. I didn’t stress the hell out of the situation, I went up to 10k numbers.

For small sets, of course, the difference is not worth mentioning. For large slices, if you really, really focus on performance, you could be better with sorted values, if the values are already sorted; if you sort them each time, the loss will be there.

var a = []int{ ... }
var b = []int{ ... }

func IterateNotSorted() int {
   count := 0
   for _, i := range a {
      for _, j := range b {
         if i == j {
            count++
            break
         }
      }
   }

   return count
}

var c = []int{ ... }
var d = []int{ ... }

func IterateSorted() int {
   count := 0
   for _, i := range c {
      for _, j := range d {
         if i == j {
            count++
            break
         }
      }
   }

   return count
}

Fill in the slices with some numbers and test it yourself.

func BenchmarkIterateNotSorted(b *testing.B) {
   for n := 0; n < b.N; n++ {
      IterateNotSorted()
   }
}

func BenchmarkIterateSorted(b *testing.B) {
   for n := 0; n < b.N; n++ {
      IterateSorted()
   }
}