How to build up-to-date VIPS packages for Debian

Posted on by Renaud Chaput 7 mins read

Some context

The excellent image_processing Ruby gem recently hit 1.0 and added support for a libvips based backend, in addition to the original ImageMagick one. I read a lot of good things about libvips and its architecture had always seemed much better and modern than ImageMagick, so I decided to spend some time optimizing the image generation code on Talegraph and experiment with the two backends and various settings. I settled on libvips as it generated smaller files and was much faster than shelling out to ImageMagick’s convert utility (see the official benchmark to get an idea of how fast and efficient libvips is).

I work on MacOS and installing libvips was easy using brew: brew install vips installs the latest libvips release (8.6.3 when I am writing this). Unfortunately, when pushing the changes to my staging environment (based on Debian Linux), things became a bit more complex.

First, ruby-vips (used behind the hood by image_processing) required libvips 8.2 or above. This forces you to use the most recent Debian version, Stretch, as it provides a 8.4 libvips package, where the previous version (Jessie) only provides libvips 7.40. But libvips is quickly improving and image_processing requires some functions only available in libvips 8.6. I did not want to switch to a new distribution with more up-to-date packages, so I decided to build my own libvips.

Building custom libvips packages

My goal here is to build proper .deb packages that I can easily install on any system. I do not want to locally build the library on every system, as it would install a lot of additional packages and takes much more time than just installing the package. This is especially important on CI where every run starts from a base image.

As Debian already provides a libvips package (but for an older version), it is easy to start from it. The following steps are run on a base Debian Stretch VM, but it should also work on Jessie.

First we need to install the build-essential package to have the needed build tools (compiler, headers, …), as well as the devscripts packages containing a few useful commands to work with Debian packages:

$ apt install build-essential devscripts
[...]

Now, we need to download the latest libvips source code from the Github releases page:

$ curl -LO https://github.com/jcupitt/libvips/releases/download/v8.6.3/vips-8.6.3.tar.gz
$ tar xvf vips-8.6.3.tar.gz
$ cd vips-8.6.3

Then we download the build directory from the Debian Package page:

curl -LO http://http.debian.net/debian/pool/main/v/vips/vips_8.4.5-1.debian.tar.xz
tar xvf vips_8.4.5-1.debian.tar.xz

We now have a debian directory which contains all the files needed to build the package. But those are for libvips 8.4.5 and we need to make a few changes before building it.

Let’s update the Debian changelog with our new version. For this we can use the dch CLI, available in the devscripts package. Let’s add a changelog entry to debian/changelog, using our name and email:

$ EMAIL=renchap@gmail.com NAME="Renaud Chaput" dch -v 8.6.3-1 "Update to version 8.6.3"

Now, before building the package we need to install all the libvips dependencies. The easiest way is to run mk-build-deps (included in the devscripts package we installed earlier). This will create an empty vips-build_deps package that does not install any files but depends on the needed packages. Thus installing this package will pull all dependencies in one go!

$ mk-build-deps
[...]
dpkg-deb: building package 'vips-build-deps' in '../vips-build-deps_8.6.3-1_all.deb'.

The package has been created.
Attention, the package has been created in the current directory,
not in ".." as indicated by the message above!
$ sudo apt install ./vips-build-deps_8.6.3-1_all.deb
[...]
$

Whew, that was a lot of deps :) Now that they are installed, we can (finally!) proceed to build the package:

$ debuild -i -us -uc -b
[...]
dh_install -ppython-vipscc
dh_install: Cannot find (any matches for) "debian/tmp/usr/lib/python*.*" (tried in "." and "debian/tmp")
dh_install: python-vipscc missing files: debian/tmp/usr/lib/python*.*
dh_install: missing files, aborting
/usr/share/cdbs/1/rules/debhelper.mk:233: recipe for target 'binary-install/python-vipscc' failed
make: *** [binary-install/python-vipscc] Error 255
dpkg-buildpackage: error: fakeroot debian/rules binary gave error exit status 2
debuild: fatal error at line 1116:
dpkg-buildpackage -rfakeroot -us -uc -i -b failed
$

Unfortunately, the build should not succeed and you should have got an error. This is because we are trying to package the VIPS python bindings, which have been deprecated in libvips 8.6 and are no longer built by default, so the required files can not be found and dpkg can’t package them. As you should no longer use these binding (and move to pyvips), let’s disable the python-vipscc package:

Open debian/control in your favorite text editor and comment out (by adding #) the block beginning with Package: python-vipscc. This will prevent this package to be built.

Finally launch debuild again, and this time it should work:

$ debuild -i -us -uc -b
[...]
Now running lintian...
Finished running lintian.
$

And we are done! The generated packages are in the parent directory:

$ ls ../*.deb
../gir1.2-vips-8.0_8.6.3-1_amd64.deb
../libvips42-dbgsym_8.6.3-1_amd64.deb
../libvips-doc_8.6.3-1_all.deb
../libvips-tools-dbgsym_8.6.3-1_amd64.deb
../libvips42_8.6.3-1_amd64.deb
../libvips-dev_8.6.3-1_amd64.deb
../libvips-tools_8.6.3-1_amd64.deb

We can now install the package locally and test that everything works:

$ sudo apt install ../libvips42_8.6.3-1_amd64.deb ../libvips-dev_8.6.3-1_amd64.deb
$ sudo apt install ruby ruby-dev
$ sudo gem install ruby-vips
$ irb(main):001:0> require "vips"
=> true
irb(main):002:0> l = Vips::Image.black 10, 10
=> #<Image 10x10 uchar, 1 bands, b-w>
irb(main):003:0>

Everything is working 🎉 If it does not on your system, you might get an error like this one:

LoadError: Could not open library 'glib-2.0': /usr/lib/x86_64-linux-gnu/glib-2.0: cannot read file data: Is a directory.
Could not open library 'libglib-2.0.so': libglib-2.0.so: cannot open shared object file: No such file or directory
from /var/lib/gems/2.3.0/gems/ffi-1.9.23/lib/ffi/library.rb:147:in `block in ffi_lib'
from /var/lib/gems/2.3.0/gems/ffi-1.9.23/lib/ffi/library.rb:100:in `map'
[...]

This is because the ffi gem is loading shared libraries naively, using their name (here it tries to load a libglib-2.0.so library). This works on most systems as there are symlinks from those simple names to the real .so, but Debian packages do not include those links. You need the -dev packages to get them. In our case, we solve this with by installing our libvips-dev package as well (and it will pull libglib2.0-dev). Once those packages are installed, we can check that the symlinks exists, and can require "vips" without error.

$ ls -l /usr/lib/x86_64-linux-gnu/libglib-2.0.so
lrwxrwxrwx 1 root root 38 Mar 19  2017 /usr/lib/x86_64-linux-gnu/libglib-2.0.so -> /lib/x86_64-linux-gnu/libglib-2.0.so.0

Distribution and installation

Now that the packages are built, just copy the files to the target system and install them. If you already have a custom package repository, you can use it. I did not want to setup one, so I did it manually.

I copied all the generated .deb to a Google Cloud Storage bucket to make them available from anywhere. You can obviously choose another way of distributing the files, it was the easiest one for me.

Installing libvips on CI

On CI, I am using the following script to download, check and install the packages before installing the gems:

#!/bin/sh
mkdir /tmp/install_vips
cd /tmp/install_vips

curl -O https://storage.googleapis.com/my-packages-bucket/debian/stretch/libvips-dev_8.6.3-1_amd64.deb
curl -O https://storage.googleapis.com/my-packages-bucket/debian/stretch/libvips42_8.6.3-1_amd64.deb

echo "13de4ec10cc5474bba219d1f68522074c69966d2d809a54e90a4ad5d4d28b081  libvips42_8.6.3-1_amd64.deb
7495c3ab4b5a26a7ef4ec046970871e1855e9e533fe4b5cdc457b469b00738f1  libvips-dev_8.6.3-1_amd64.deb" | shasum -c -a 256

if [ $? != 0 ]; then echo "Bad checksum" && exit 2; fi

sudo apt install ./libvips-dev_8.6.3-1_amd64.deb ./libvips42_8.6.3-1_amd64.deb

It takes less than 30 seconds on CircleCI, much better than building from source at each run!

Installing libvips on production servers

My servers are managed using Chef. Unfortunately the apt_package resource does not accept local packages (feature request here), so I am using this recipe instead:

# attributes/vips.rb
default["vips"].tap do |vips|
  vips["version"] = "8.6.3-1"
  vips["repo_url"] = "https://storage.googleapis.com/my-packages-bucket/debian"
  vips["checksums"]["libvips42"] = "13de4ec10cc5474bba219d1f68522074c69966d2d809a54e90a4ad5d4d28b081"
  vips["checksums"]["libvips-dev"] = "7495c3ab4b5a26a7ef4ec046970871e1855e9e533fe4b5cdc457b469b00738f1"
end

# recipes/vips.rb
vips_attr = node["vips"]

%w[libvips42 libvips-dev].each do |package|
  deb_name = "#{package}_#{vips_attr["version"]}_amd64.deb"
  remote_file "/var/chef/cache/#{deb_name}" do
    source "#{vips_attr["repo_url"]}/#{node["lsb"]["codename"]}/#{deb_name}"
    backup false
    mode "644"
    checksum vips_attr["checksums"][package]
  end

  # Can not use dpkg_package as we need to install deps
  bash "install #{package}" do
    code "apt-get install -q -y /var/chef/cache/#{deb_name}"
    not_if "dpkg-query -W #{package}"
  end
end