Ett öppet brev till LRF om grisars situation

(A note to my english readers: This text is a part of an ongoing debate about animal welfare in Sweden. For general information in English, please checkout peta.org)

Jag skickade nyss följande brev till Lantbrukarnas Riksförbund. Om jag får något svar återkommer jag med ett nytt inlägg.

Jag har under de senaste dagarna följt diskussionen om djurhållning i Sverige utifrån djurrättsalliansens uppmärksammade filmklipp från svenska grisfarmar. Efter att ha läst ert korta pressmeddelande har jag några funderingar och ett konkret förslag.460x345-images-stories-grisfarmer-skane-angamollan-grisfarm-skane-00058

Om det stämmer at grisfarmarna kontrolleras på ett oberoende sätt och anmärkningarna är få så kanske det är så att det finns något systematiskt fel i hur kontrollerna utförs, eller hur reglerna är utformade? De lagstadgade minimireglerna för hur stora grupper och hur mycket utrymme grisarna behöver kanske inte räcker för att göra grisarna lugna nog att inte bita öronen av varandra?

En annan möjlighet är att kontrollerna sker på ett sätt som gör det lätt för djurfarmare att vara ögontjänare. Bokar kontrollanterna tid en vecka innan? Sker kontrollerna bara på tisdagar klockan fyra?

Om ni menar allvar när ni skriver att svensk grisproduktion inte har något att dölja har jag ett konkret förslag: Bjud in djurrättsalliansen och andra intresserade organisationer att göra oanmälda kontroller. Publicera protokoll och uppföljningar av alla missförhållanden på er hemsida. Erkänn att era medlemmar ibland har problem med att leva upp till en nivå som gör att det ser okej ut när någon filmar och berätta hur ni arbetar för att göra det bättre.

Om det nu är så att de filmer som publicerats inte är representativa så borde det inte vara så svårt att få bukt med några få problematiska situationer.

Med vänliga hälsningar
Noa Resare, Umeå

easter break hack project: runasdaemon

People running unix like systems expect daemon processes behave in certain ways. A proper daemon is expected to detach from the terminal or script from which it was started. It is expected to write it’s output to syslog and it is expected to shut down gracefully when receiving the TERM signal.

Software developers, on the other hand, sometimes write software that fails to meet those expectations. Unfortunately some software environments, such as java, is missing critical functionality to make daemons behave the unix way.

To bridge this gap I spent some time this weekend to write the last parts of a small program that attempts to solve this problem. The program, which I have unimaginatively named runasdaemon will spawn a small child process which is responsible for making some other program behave as a daemon. It will detach properly from the calling process and it will write any output from the program to syslog. It will attempt to shut down properly if it’s sent the SIGTERM signal and if that takes too long (5 seconds by default) it will kill the child process and exit.

There are some other nice features as well. It will in fact not detach directly from the controlling program. If the child process exits right away it will propagate any error messages to the calling user. It also creates a unix domain socket that can be used to communicate with the child.

The code is available here.

Spotify kernel patching: finding a needle in the haystack

Spotify makes a significant investment in writing code to automate the provisioning of servers in our data centers. Our goal is to automate hardware configuration and operating system installation so that we can mark a system for installation, flip the power switch and have it installed and added to the pool of available hardware used to build the Spotify backend service.

One important link in this chain is to have the ability to automatically configure the RAID hardware to provide the proper logical drives for our various use cases. This is typically done using a binary only tool provided by the hardware vendor: 3Ware uses tw_cli, Adaptec uses arcconf and some LSI cards uses the sas2ircu tool which will be at the center of the rest of this post. We have been using that card in quite a few servers lately, and it has worked flawlessly. However, when we got a new type of hardware from Dell, the PowerEdge C5220 suddenly our RAID volume configuration code broke. More specifically the sas2icru tool died triggering a segfault on querying the current state of the raid card by issuing “sas2ircu 0 display”

We got in touch with Dell support, but since Debian GNU/Linux is not officially supported on their hardware they could not really help out. So, we tried installing Red Hat Enterprise Linux 6 on a server and that one turned out not to exhibit the problem. The binary tool worked when running the Red Hat kernel. In an ideal world, hardware vendors (Are you listening, LSI?) would release their tools and specifications as free software projects, and it would probably be fairly straight forward to fix the tool. In this case, our options were limited to switching Linux distribution, a monumentally large undertaking, or modifying our kernel to match the behavior of the RHEL kernel.

So, a few days back I started looking into fixing the kernel, more specifically the mpt2sas driver that was used by the LSI Logic / Symbios Logic SAS2008 PCI Express Fusion-MPT card in our machines. It turns out that the driver in our kernel tree, which is pretty close to the standard Debian Squeeze kernel, based on upstream Linux 2.6.32 is a fairly large and complex piece of code. The first glance at the RHEL6 kernel, more specifically the 2.6.32-279.14.1.el6 version, looks promising. It’s based on the same upstream version, so the mpt2sas driver can’t be that different from the Debian one, right? Well, that assumption turned out to be spectacularly wrong. In fact, the difference is vast. Looking at the source code of the RHEL6 kernel reveals that the mpt2sas driver is backported from somewhere around the upstream Linux release 3.4. The Debian version contains select few patches on top of the driver that was committed to the mainline kernel in September 2009. In contrast, the RHEL version contains the result of development from LSI as well as hundreds of kernel changes merged into the mainline kernel as late as March 2012. The full drivers is around 20 000 lines of code, and the (non-unified) diff between the RHEL and Debian version of the driver is well north of 10 000 lines. Looking at the diff, there seemed to be a high number of significant changes to the code, as would be expected.

Fortunately, since we had lots of hardware exhibiting the problem laying around, and I had some time to play around I decided to attempt to track down the change between the driver shipped in Linux version 2.6.32 and 3.4 that fixed the segfault problem. Before I describe my methodology in more detail, a word of warning: If you attempt this, please note that modifying the drivers for the RAID controller you use to access and store data with will probably silently corrupt your data in very unpleasant ways. Be careful.

What I did was an attempt to get a late driver to compile in our kernel source tree. If got a driver module built, I could copy that to a test machine running a ramdisk based system, load it and see if it resolves our problem. A module that did not exhibit the problem would, in itself, not be that valuable, but it would make it possible to binary search to find the smallest change that would solve our problem. If such a code change was found, it is entirely possible that the change could be applied on top of the Debian kernel and we had a fix for our problem. Given that the change was simple enough, it might even be that such a change could be considered safe enough to put into our production kernel.

So, only a couple of roadblocks in the way. First off, compile a good version. Fortunately we already have a version controlled set of scripts that is used to modify the standard Debian kernel building setup and make it build with our patches on our TeamCity build cluster. To speed up things a bit further, I borrowed a fast machine (16 core, 48G RAM, SSD), made sure that the build ran with the magical DEB_BUILD_OPTIONS=parallel=8 environment variable set.

When I attempted to copy the 3.4 driver straight into the tree, it turned out that compilation failed somewhat with a missing symbol DEF_SCSI_QCMD. Reading the comments in scsi_host.h about this new macro it seemed like this was part of some pretty major changes to the SCSI subsystem relating to how locking is done. Without delving too deeply in this, I tracked down the introduction of these changes to 2.6.37. So, i decided to try and get 2.6.36 up and running instead. The compilation failures here were more straightforward. Adding a few defines, and removing some code made it build. To my great joy, it also turned out that when loading the updated kernel module, the segfault problem was gone.

With one known good version and one known bad version, it was a simple matter of binary searching for the change that makes the difference. I made a silly error when executing this plan, and ended up building 13 different modules before I found the patch ebda4d38df542e1ff4747c4daadfc7da250b4fa6. A seven line change to the _ctrl_do_mpt_command() function. Wonderfully enough, this change applies cleanly the Debian version of the driver, and solves our problem.

We have now been running the patched kernel in a few weeks on a wide variety of hardware and we have not discovered any problems with it so far.

Apple ate my java web start functionality

Yesterday I installed an update to OSX Mountain Lion called Java for OS X 2012-006. I assumed it was a minor update, but it turned out to be more than that. It turns out that Apple removed some functionality I rely upon, namely the ability to run java web start applications. When I run javaws in a terminal after the upgrade, I get the following popup

Well, that was annoying, but there seems to be hope. I followed the More info… link and installed the referenced package from Oracle. Still no luck. The javaws tool still didn’t work.

After learning a bit more than I wanted to know how OSX pkg files work, I found out that the downloaded software installs a new javaws binary in a pretty nonobvious location. It turns out to be

/Library/Internet Plug-Ins/JavaAppletPlugin.plugin/Contents/Home/bin/javaws

(note that since there is a space in one of the dir names, you need to put the path in quotes in Terminal, or escape the space with backslash)

Thanks for nothing, Apple.

Recording audio conversations in OSX

Since I work remotely I use various technologies for audio and video conferencing such as Skype, Google Hangout and Jabber Video. Sometimes it is useful to be able to record conversations for various reasons. This is something that is a bit trickier than one would have expected to set up in an OSX machine. So, this is what I did to be able to record both myself and others. The examples are using Skype, but the technics works just as well with other programs.

  1. Install Soundflower, a free extension to OSX that adds the ability to route sound between applications. I have used version 1.6.6b without problems on my mac running Mountain Lion.
  2. Open System Preferences and select the Sound preference pane. If everything worked as expected with the Soundflower installation, two new Input and two new Output devices should be visible now. Ignore the 64 channel versions, and focus on the 2 channel versions for now
  3. Start Skype and select Audio/Video under Preferences.
  4. Select “Soundflower (2ch)” for speakers. This will redirect the output to soundflower and make it available to other programs.
  5. Start the Soundflowerbed application, which is located in /Applications/Soundflower This brings up a new flower icon to the right on your screen. Click on it and and select the output device that you want to use to make your skype call. I selectCreative USB Headset, since that’s what I will be using. If you skip this step, recording will work fine but you will not hear any audio from Skype in your headphones.
  6. Start the Audio MIDI Setup application located in /Applications/Utilities.
  7. Create an Aggregate Device by clicking the plus sign down to the left and add Soundflower 2ch and Creative USB Headset (or whatever your headphones are called) to it. This will create something from which you can record four channels of sound simultaneously. Two channels from whatever is playing sound into the Soundflower – in this case Skype – and two channels from the headset microphone.
  8. Download and install Audacity, a free software program for audio recording and editing. I used version 2.0.2 successfully.
  9. At the drop down menu by the lower microphone icon, select “Aggregate device” and select 3 in the drop down directly to the left of it, which configures the number of channels to record. 
  10. When you press record, three new channels will show up in the main area of the Audacity window, one channel should contain the sound from your mic, and two channels should contain the remote end of your Skype conversation
  11. Switch back to Skype and call the test service echo123 to verify that everything works.

back

So, this is a bit embarrassing. People working in small internet oriented companies write blogs, engage in hacker culture, contribute to free software projects. Then they get recruited, sucked into some larger organization and go mostly silent on the net. So, in retrospect that is pretty much exactly what I did. I started working for Spotify in early 2010, and since then I stopped blogging.

I don’t want to be that person, and I think I have some interesting things to write about. Since I last wrote, I have grown quite an interest in Coffee, for example. And roller coasters. And I’ve switched Linux distribution of choice. It feels good to be back.

My password generator

How do you create a really good password that you don’t need to remember but that you might occasionally need to write on paper or type into a keyboard? These days modern operating system provide really good sources of randomness, and one method that is often used is to read some randomness from the operating system PRNG located at /dev/random and run the data through the base64 encoding to get letters, numbers, + (plus) and / (slash). However, those passwords are not that conveinent and sometimes when I write them down people mistake my zeroes for capital o and things like that.

What I wanted was a password generator that could output a configurable length password using only easily distinguishable letters and numbers, so I wrote one. As usual I place this code in the public domain, feel free to use it any way you want.

Features:

  1. The entropy of the password is as good as the underlying operating system. If you use a recent Linux or OSX version, the data returned from /dev/random is quite good.
  2. The code is simple and it is easy to verify that the program actually uses the entropy that it reads.
  3. The resulting passwords are easy to type on keyboards and write on paper without confusing the reader with similar characters such as 1 (one) and l (lower case l).
  4. The length of the password is configurable.
#!/usr/bin/python

import sys

# alphanumeric chars minus l, I, O, 0, 1
alphabet = "abcdefghijkmnopqrstuvwxyzABCDEFGHJKLMNPQRSTUVWXYZ23456789"
# Some expeimentation told me that 2 ** 5.8 = 55.7
BITS_PER_CHAR = 5.8

# The default password length has the capacity of a bit more
# than 64 bits of entropy.
DEFAULT_LEN = 12

def main(args):
	count = DEFAULT_LEN
	if len(args) > 1:
		if args[1] == '-h':
			usage()
			return
		elif args[1] == '-c' and len(args) > 2:
			count = int(sys.argv[2])
		else:
			usage()
			return
	print(create_password(count))

def string_to_bignum(s):
	num = 0
	for c in s:
		num = ord(c) + (num << 8);
	return num

def create_password(length):
	random = open("/dev/random", "r")
	needed_bytes = (int)(length * BITS_PER_CHAR) / 8 + 1
	n = string_to_bignum(random.read(needed_bytes))
	random.close()
	s = ""
	for i in xrange(length):
		s = s + alphabet[n % len(alphabet)]
		n = n / len(alphabet)
	return s

def usage():
	print """mkpasswd [-h] [-c COUNT]
Create a random password using the operating system's entropy pool
using a 57 character alphabet of letters and numbers. The characters
in the alphabet excludes characters and letters easily confusable such
as I and 1. 

Each password character holds about 5.8 bits of entropy, so the
standard 12 character password can hold a theroretical maximum of
69 bits of entropy.

The actual entropy present in any generated password is a function
of the entropy gathering algortihm present in the kernel of your
operating system.

  -h        display this help text
  -c COUNT  create a password with COUNT characters."""  

if __name__ == '__main__':
	main(sys.argv)

D-Link DWL-G510 in CentOS 5

Some days ago I spent some time getting a PCI WiFi 802.11G network card to work in CentOS 5.4. My card identifies itself as a “D-Link System Inc AirPlus G DWL-G510″ (PCI id 1186:3c09) and uses the “RaLink RT2561/RT61 rev B 802.11g” chipset (PCI id 1814:0302)DWL-G510

A driver for this card is included in the standard CentOS 5 kernel (tested with kernel-2.6.18-164.9.1.el5) under the name rt61pci, however to be able to function a binary firmware is needed. Some searching revealed that the firmware is available in Fedora’s rt61pci-firmware package. I rebuilt the package and put it in my CentOS playground repository.

So, if you have such a card and want to make it work in CentOS5 you might want to try running rpm -ivh http://rpm.resare.com/centos5-playground/i386/rt61pci-firmware-1.2-6.el5.noarch.rpm as root. Once that is done, running modprobe rt61pci should do the trick.