PostgreSQL batch operations in Go

Consider the following case: When creating a user (database insert) with their profile (another insert), other users must be updated (database update) with a new score value. Score is just a float for which a dummy formula will be used. And then an action record is needed (insert), which marks the fact that a user was created.

The tech context is PostgreSQL in Go with pgx as database driver and Echo framework for the HTTP server. The database setup is straight forward using Docker; it also includes a database management interface which will be available at http://localhost:54321. If you clone the sample repository, and start the setup with Docker Compose (docker compose up -d), when the PostgreSQL Docker container is built, a database is created with the schema used in this post.

CREATE TABLE "users" (
  "id" serial NOT NULL,
  "username" CHARACTER VARYING (100) NOT NULL,
  "score" DECIMAL NOT NULL DEFAULT 0,
  "created" TIMESTAMP(0) WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
  "updated" TIMESTAMP(0) WITH TIME ZONE
);

CREATE TABLE "user_profile" (
  "user_id" INTEGER NOT NULL,
  "firstname" CHARACTER VARYING (100) NOT NULL,
  "lastname" CHARACTER VARYING (100) NOT NULL
);

CREATE TABLE "actions" (
  "id" serial NOT NULL,
  "description" text NOT NULL,
  "created" TIMESTAMP(0) WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP
);

Data integrity is of interest, so all the queries will be sent on a database transaction. And because there are multiple user update queries, they will be sent all at the same time in a batch of operations. Continue reading PostgreSQL batch operations in Go

Unmarshal JSON and XML into the same Go structure slice with proper data type

I’m consuming a REST service which gives various lists in both JSON and XML formats, something similar to these ones:

[  
   {  
      "id":803267,
      "name":"Paris, Ile-de-France, France",
      "region":"Ile-de-France",
      "country":"France",
      "is_day":1,
      "localtime":"2018-05-12 12:53"
   },
   {  
      "id":760995,
      "name":"Batignolles, Ile-de-France, France",
      "region":"Ile-de-France",
      "country":"France",
      "is_day":0,
      "localtime":"2018-05-12"
   }
]
<?xml version="1.0" encoding="UTF-8"?>
<root>
   <geo>
      <id>803267</id>
      <name>Paris, Ile-de-France, France</name>
      <region>Ile-de-France</region>
      <country>France</country>
      <is_day>1</is_day>
     <localtime>2018-05-12 12:53</localtime>
   </geo>
   <geo>
      <id>760995</id>
      <name>Batignolles, Ile-de-France, France</name>
      <region>Ile-de-France</region>
      <country>France</country>
      <is_day>0</is_day>
      <localtime>2018-05-12</localtime>
   </geo>
</root>

And I wanted to get them into a slice of this type of structures:

type Locations []Location

type Location struct {
   ID        int      `json:"id" xml:"id"`
   Name      string   `json:"name" xml:"name"`
   Region    string   `json:"region" xml:"region"`
   Country   string   `json:"country" xml:"country"`
   IsDay     int      `json:"is_day" xml:"is_day"`
   LocalTime string   `json:"localtime" xml:"localtime"`
}

It’s straight forward for JSON: Continue reading Unmarshal JSON and XML into the same Go structure slice with proper data type

API authorization through middlewares

When dealing with API authorization based on access tokens, permissions (user can see a list, can create an item, can delete one etc) and/or account types (administrator, moderator, normal user etc), I’ve seen the approach of checking requirements inside the HTTP handlers functions.

This post and attached source code do not handle security, API design, data storage patterns or any other best practices that do not aim directly at the main subject: Authorization through middlewares. All other code is just for illustrating the idea as a whole.

A classical way of dealing various authorization checks is to verify everything inside the handler function.

type User struct {
   Username    string
   Type        string
   Permissions uint8
}

var CanDoAction uint8 = 1

func tokenIsValid(token string) bool {
   // ...
   return true
}

func getUserHandler(c echo.Context) error {
   // Check authorization token
   token := c.Get("token").(string)
   if !tokenIsValid(token) {
      return c.NoContent(http.StatusUnauthorized)
   }

   user := c.Get("user").(User)

   // Check account type
   if user.Type != "admin" {
      return c.NoContent(http.StatusForbidden)
   }

   // Check permission for handler
   if user.Permissions&CanDoAction == 0 {
      return c.NoContent(http.StatusForbidden)
   }

   // Get data and send it as response
   data := struct {
      Username string `json:"username"`
   }{
      Username: user.Username,
   }

   return c.JSON(http.StatusOK, data)
}

The handler is doing more than its purpose. Continue reading API authorization through middlewares

Join domain on Ubuntu

It took me a while to find out how to join a domain (Microsoft Active Directory) on Ubuntu (16.04). I’ve adapted this guide to the following:

sudo apt update
sudo apt install ssh -y
wget https://github.com/BeyondTrust/pbis-open/releases/download/8.5.2/pbis-open-8.5.2.265.linux.x86_64.deb.sh
sudo chmod +x pbis-open-8.5.2.265.linux.x86_64.deb.sh
sudo ./pbis-open-8.5.2.265.linux.x86_64.deb.sh
sudo domainjoin-cli join <domain> <username>
sudo reboot #if necessary

This is not an universal solution. There are chances you’ll need extra steps depending on your network’s configuration.

Sending body on HTTP GET request

I’ve heard a few times “you can’t, it’s not possible to send a body on a request made with GET method”. Well… You can. You can send a body on a GET request and you can read the body. It’s against the standards and you should not do it, as it will mess your design. But technically, it can be done.

package main

import (
   "encoding/json"
   "io/ioutil"
   "log"
   "net/http"
)

func main() {
   http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
      body, err := ioutil.ReadAll(r.Body)
      if err != nil {
         log.Println(err)
         return
      }

      input := &struct {
         A string `json:"a"`
         B string `json:"b"`
      }{}
      if err := json.Unmarshal(body, input); err != nil {
         log.Println(err)
         return
      }

      data, err := json.Marshal(input)
      if err != nil {
         log.Println(err)
         return
      }
      w.Write(data)
   })

   log.Fatal(http.ListenAndServe(":8118", nil))
}
curl -X GET \
  http://localhost:8118 \
  -H 'content-type: application/json' \
  -d '{"a": "val1", "b":"val2"}'
{"a":"val1","b":"val2"}

Playing with PHP extensions: building a map

I’ve ran into PHP-CPP, a C++ library which helps you build your own PHP extensions in an easier way than writing them from scratch. And I decided to play a little bit. It has good documentation and examples.

Installing is not always out of the box depending on the environment. I like to keep things clean on my working machines so I wanted to have everything in a Docker container, but I quickly ran into an issue installing the library. Fortunately, I’ve found a solution pretty fast.

The first idea that came into my mind was a simple map. Nothing special, I’m not bringing any improvements, I just wanted to convert the following into an extension. Continue reading Playing with PHP extensions: building a map

Race condition on Echo context with GraphQL in Go

Given an API setup with GraphQL and Echo, a colleague ran into a race condition situation. There was a concurrent read/write issue on Echo’s context. GraphQL runs its resolvers in parallel if set so, and when context is shared between resolvers, things can go wrong.

I took a look into Echo’s context implementation and I saw a simple map is used for Get/Set.

For every API call, a handle functions is given an Echo context and executes the GraphQL schema with the specified context.

func handle(c echo.Context) error {
 schema, err := gqlgo.ParseSchema(
  ...
  gqlgo.MaxParallelism(10),
 )

 schema.Exec(
  c,
  ...
 )
}

My solution was to use a custom context which embeds the original one and uses a concurrent map instead of Echo’s. Continue reading Race condition on Echo context with GraphQL in Go

Concurrency in PHP with Swoole

Swoole is a CLI server written in C as a PHP extension. It handles HTTP, TCP and socket servers, offering some really interesting features. The ones I’ve tested and I’m pretty excited about are related to concurrency.

The authors have implemented a concurrency model similar to Go‘s, using coroutines and channels for inter-process communication. It also has a memory map, atomic objects, locks, buffers, timers and others.

I’ve tested their 2.2 version found in PECL, but very recently they’ve release the 4.0.0 version. My enthusiasm suffered a little bit when they’ve introduced a pretty nasty bug with 4.0.0, but they were fast about fixing it, although a new version is not released at the time of writing (but you can compile from master branch).
It’s funny that one day everything was working as expected, and the other day I spent some time until I’ve figured they had just released a new version, and that was causing my issue. I had installed the extension without specifying the version, so a new install got me the latest one.

Below is a very small example of concurrency using routines and a channel. Continue reading Concurrency in PHP with Swoole

Read Go documentation

A few weeks ago I’ve finished reading the Go documentation. The Go 1.10 documentation Android app is very helpful! It’s very easy to read and it has auto bookmarks; whenever you get back into the app, you can return to the chapter you were reading, at the same line you were before closing it.

It was very nice to find out a lot of details. Of course, I don’t remember all of them after just one reading, but when I bump into some situations (performance, how slices really work, libraries, the memory model etc) it’s easier to start researching further if I don’t remember the point exactly.

I don’t want to write Go without understanding as best as I can its way of getting things done.