blog

ubuntu and btrfs backups using send receive

Since I’ve moved to Ubuntu Linux I’ve been mostly using btrfs for all my filesystems. btrfs makes it super easy to create snapshots of subvolumes and to go back in time to a previous snapshot (just use mv). I always create a snapshot before updating packages so that I can easily go back if something breaks.

I’ve been using Back In Time for my backups but this app takes more then 30 minutes for  every single backup, no matter how small, and does a lot of disk I/O in the process, even just to transfer 2 or 3 files (I think that happens because it’s creating  hard links).  I’ve recently discovered that btrfs has a send / receive functionality which allows sending a snapshot to another btrfs partition and it also allows to send just the diff between snapshots. Using btrfs send and btrfs receive it’s possible to setup a backup process entirely based on btrfs.

I found that btrfs send / receive is broken  in the current ubuntu version 13.04, so I had to uninstall the btrfs-tools package and compile from the source in git of btrfs-progs. After that btrfs send and btrfs receive seem to work fine. I’m now doing all my backups with btrfs which on average now take just one or two minutes, much much faster then with Back In Time.

Finally I created a Haskell Shelly script to automate my backup procedure. It’s still very incomplete, but I put here in case it’s useful to others. Be careful with this script, it might delete snapshots that it shouldn’t so use at your own risk !

Tags | haskell

Man and dog

Monad Transformers Hello World

I’ve been searching the webs for a simple example of monad transformers in action in Haskell. Don’t remember if I actually did find one or not, but in any case I’m posting my own Monad Transformers hello world. I really don’t understand why texts explaining Haskell concepts such as these can’t have a simple hello world example to just to see the concept in use in it’s most simple form.

OSX Lion – script for automatic ad-hoc network creation

I have to create an ad-hoc network everytime I need to play with supercollider from a tablet or phone, and this is very annoying, so I automated the process using two apple scripts which I bundled as applications and are now sitting on my desktop. The script was copied from somewhere on the net and altered to work on my OSX Lion configuration, so it would have to be adapted if the one has different items order on the menu bar. Also my static IP configuration is named “osc-iphone”, this should be renamed to whatever network configuration one uses for the ad-hoc network.

Go back to default configuration:

Portballintrae

New site

New website featuring my projects in a more organized manner. I’ve copied over the content from the soundofsyntax.wordpress.com blog.

A look back on my work with ScalaCollider

I have just recently completed my first project using ScalaCollider. ScalaCollider is a front-end written in Scala, a hybrid object-oriented/functional language that runs on the Java virtual machine. I would like to share my impressions of using ScalaCollider now that I have come to use it for quite some time. I will mostly compare it to SuperCollider (using SCApp on the mac).

Positive:
– I can use all the infrastructure around the JVM and Scala. It’s easy to manage dependencies and to bootstrap a project using Simple Build Tool (SBT). There are several extremely competent IDE’s available that are used by developers at companies like Twitter. There are performance analysis tools available that are used by professionals. It’s possible to build an executable that can be distributed and that runs with a simple doble-click easily. This was possible in SuperCollider using standalones but it’s a complicated process, which meant I didn’t do it often enough. With scala it’s so easy that I keep binary copies of my project during development at several stages and after each concert (it’s one command in sbt). I also keep a copy of scsynth and the plugins which means that I have an executable that is exactly the same setup I had at that particular concert.
– I personally use the Intellij IDEA IDE, which is amazing. Coming from SuperCollider.app, IDEA just blew me away. It’s easy to edit code, it can reformat code easily, refactor code (change name of variables, classes, etc across multiple files), there’s no more tabbing and hitting space to format text, just place the cursor anywhere on screen and start writing. Because scala is a static language, the autocompletion is mind-boggling at first (in a good way !). Also help is available just by hovering the cursor over class name and hitting a key combination. Help is also available during auto-completion, so it’s easy to check what each method available does (at least after the help system is configured correctly, which might take some time).
– Scala is a new language that is still evolving. It has many new features that are ground-breaking such as implicit conversions, which allows having type classes like in Haskell and extending libraries that we cannot change. It’s a cutting-edge language, full of new ideas and possibilities. SuperCollider in comparison feels old. For instance, the scalaz library is incredibly powerful and advanced. It’s basically pure math. It’s quite a challenge to use it, but what you get in return is a new way of looking at things. Through scala I came in contact with a lot of new concepts (realized in libraries that I can use) such as Option/Maybe, Algebraic Data Types, Futures/Promises, Monads, Applicative Functors, Functors, Foldables, Functional Reactive Programming (Events and EventStreams), etc.
– Even if scala is a language that compiles down to byte-code, it’s still possible to use it as if it was interpreted using a REPL (Read Evaluate Print Loop), which is an interactive prompt where we can evaluates snippets of code. It’s therefore possible to write small scripts without declaring any classes. The environment to do this is not as nice as in SuperCollider, for instance, it’s not currently possible to select a portion of code and evaluate that, although that is just because no one implemented that yet (although sciss has tried already to make a plugin for IDEA to do this). I wouldn’t use ScalaCollider for live coding at this point, since the type system can become cumbersome when trying to code things quickly, although most of the times it’s quite invisible.
– It’s possible to use any Java or Scala library together with ScalaCollider that means that there is already code out there to do almost anything that we can think off, and it’s easy to interface with those libraries because they are all written either in Java or Scala (no need to go down to c or c++). I was astonished when I implemented the functionality of discovering my iphone when using TouchOSC with Bonjour/Zeroconf with just a few lines of code by importing a java library. The same was true of json parsing. It’s possible for instance to do graphical stuff using processing (Java) or SPDE (a Scala version of processing). The possibilities are endless. And most of the times these libraries are actually written by computer scientists, or people that work in companies or institutions, so the quality standards are quite high.

Negative:
– Some things are missing, such as Routines (there are some implementations available, but not final yet) and Patterns and others are still work in progress. Most of what is implemented is already working quite well though, I detected few bugs.
– It takes a bit of time to setup the development environment. Things are quite smoother now then some months ago though, and there is more documentation available to guide us through the process. It took me quite some time to understand how to get the help system going. Now the latest version of the SBT plugin automatically pulls the source jars when generating an IDEA project, which means I get the help there automatically. It took me some time to figure out how to use the continuations plugins in IDEA, which is necessary to use Routines. Hopefully all of this will be ironed out with time.
– It’s not as easy to just evaluate pieces of code, and check what is going on in the code by checking the value of variables. This is in part because the coding style used (which I copied from a sample application) has more classes inside classes, which makes the code more organized, but also more difficult to check things just by evaluating expressions. On the other hand, there is a debugger available, something which is quite handy to check things. It’s magic stepping through the execution process and actually looking at the lines. I confess I still don’t fully understand how to properly use a debugger and many times end up putting a lot of println’s all over the place, but that’s more a limitation that I have.
– The Scala language is more complex then SCLang, there are more things to grasp in order to use ScalaCollider. For instance, scala is statically typed, which means that I got a lot of errors about something being of the wrong type (usually collections) and I didn’t always know how to solve this issues. It gets better with time, but still it might take more time to code something (unless off course we are using an advanced feature of scala that actually makes it less code anyway). I don’t think the average SuperCollider user would have an easy time in scala.
– I found the GUI code a LOT more complex than in SuperCollider. The place where I lost the most time was with scala swing, which is a GUI toolkit. It’s badly documented, and it’s just more complex, although also more powerful. Scala swing has very powerful layout managers, so when I started using it I was delighted that I would never type pixel locations in Scala. But after some time it turned out that it actually takes more time to make the layout system do the right thing then actually typing the hardcoded pixel values by hand (provided I would never change those values, which is often not true).
– ScalaCollider runs in an environment (the Scala development environment) which is not specifically made for music. That means that many things are more complex then they need to be for audio work (things like the GUIs, selecting files from disk, serialization of objects, etc).

All in all, using ScalaCollider has been a quite positive experience. If not for anything else, I learned a lot along the way. I also lost a lot (a LOT !) of time, so ScalaCollider is not for those that are in a hurry to get something done. I had to figure out a couple of things on my own which took extra time. For instance to save settings to disk I had to learn on how to do serialization using the JSON serialization provided by liftweb, and to write some custom code for my classes (in SuperCollider all I needed to do was, .writeArchive or .asCompileString).

ScalaColliderZ

ScalaColliderZ is a library that provides some added features for ScalaCollider based on the scalaz library. Scalaz is a library for pure functional programming in Scala, which takes a lot of inspiration from Haskell. It has several type classes such as Monad, Monoid, Functor, etc.

ScalaColliderZ provides some methods (many more could be implemented) to deal with asynchronous tasks. Things such as sending synthdefs to the server or loading buffers are done in ScalaCollider using responder functions: we pass a function to the method that gets executed whenever the task has completed. The problem with this is that it’s not easy to sequence several tasks. For example it’s convenient to be able to load a buffer and send a synthdef and then play a synth when both tasks have completed. Promises are perfect for this. A promise is a kind of container that will have a value of certain type at some moment in the future. This is similar to Lists that are containers that can have multiple items of the same type or Option which is a container that might or might not have an item. It’s therefore also possible to use map and flatMap to get a new Promise from an existing one and foreach to execute an action when a promise get it’s value. Example:

promise{ /* do something that takes some time*/; 4 }.map(_+1).foreach(println)

Currently there are methods to load Buffers (BufferPromise.load) and to send synthDefs (mySynthDef.loadP) that return a a promise:

And an example of sequencing:

ScalaColliderZ also adds type classes instances for GE for Monoid:

I hope to add more stuff in the future.

How to configure Ubuntu 11.10 for using 2 RME PCI Hammerfall DSP MADI cards together

The objective is to be able to use the 2 RME cards as one single 128 channel sound card. Ubuntu 11.10 does not detect the RME HDSPM out of the box, so we need to compile a new vanilla linux kernel obtained from git. It is also a good idea to make a small detour from the instructions below and configure the kernel for low-latency, but I will leave that for the brave souls (although it’s easy).

git clone https://github.com/tiwai/sound
cd sound
cp /boot/config-`uname -r` .config
yes '' | make oldconfig
sed -rie 's/echo "+"/#echo "+"/' scripts/setlocalversion
make-kpkg clean
CONCURRENCY_LEVEL=`getconf _NPROCESSORS_ONLN` fakeroot make-kpkg --initrd --append-to-version=-custom kernel_image kernel_headers
sudo dpkg -i linux-image-3.3.0-rc1-custom_3.3.0-rc1-custom-10.00.Custom_amd64.deb 
sudo dpkg -i linux-headers-3.3.0-rc1-custom_3.3.0-rc1-custom-10.00.Custom_amd64.deb
sudo update-initramfs -ck 3.3.0-rc1-custom
sudo update-grub

Reboot and you should be running the new kernel which should detect the cards. Doing

cat /proc/asound/cards

should list the cards.

Next we need to make sure we can name the cards individually since hw:0 will possible refer a different card after every reboot. To do this we place the following inside /etc/modprobe.d/hdspm.conf:

options snd_hdspm id=foo,bar

To merge the two sound cards into one for ALSA we create a ~/.asoundrc.

In our case we have the hdspms connected to RME’s ADI-648 units via optical madi cables. One of the ADI’s is the wordclock master for the the other ADI-648 and the pci cards, so we need to set the clock source and the inputs source:

amixer -c foo set 'System Clock Mode' 'AutoSync' && amixer -c bar set 'System Clock Mode' 'AutoSync'
amixer -c foo set 'Input Select' 'optical' && amixer -c bar set 'Input Select' 'optical'

Now we unmute the cards:

for i in `seq 1 64`; do amixer -c foo set 'Chn',$i 64 && amixer -c bar set 'Chn',$i 64; done

To get rid of xruns jack should run with realtime priorities (change johndoe to your user name):

dpkg-reconfigure -p high jackd
usermod -a -G audio johndoe

You can check your privileges with:

ulimit -r -l

You can now start jackd for the 128 channels sound card

jackd -R -P 70 -d alsa -r44100 -dmergedcards -p128 -o128 -i128

and have fun !

Thanks to Adrian Knoth and Florian Faber for all the help with the hdspm driver !

It took me about 2 months to compile this information, so I hope this is useful to someone.

OSX internet sharing under the hood

I was curious this week to know what exactly is OSX’s internet sharing doing. It starts 3 processes as follows:

bootpd -d -P
natpmpd -d -y en0 en1
named -c /etc/com.apple.named.proxy.conf -f

From the man pages:

bootpd:  DHCP/BOOTP/NetBoot server
natpmpd: natpmpd is a daemon implementing NAT-PMP. NAT-PMP is a protocol for handling port forwarding requests from clients behind a NAT. It is normally
invoked by InternetSharing(8).
named: named is a Domain Name System (DNS) server.

The internetSharing process is responsible for starting these processes:

InternetSharing:

InternetSharing is the back-end for the Internet Sharing feature. It is
responsible for configuring the network interfaces, the DHCP server
bootpd(8), the network address translation daemon natd(8), and the Inter-
net domain name server named(8). named(8) is run in caching-only mode
and allows the DHCP server to always offer the same DNS server address to
the DHCP clients, regardless of the value of the actual DNS server
addresses.

more /etc/com.apple.named.proxy.conf
// Declares control channels to be used by the rndc utility.
// It is recommended that 127.0.0.1 be the only address used.
// This also allows non-privileged users on the local host to manage
// your name server. An empty control set means the utility is disabled.
//
controls {
};

options {
directory "/var/named";
/*
* If there is a firewall between you and nameservers you want
* to talk to, you might need to uncomment the query-source
* directive below. Previous versions of BIND always asked
* questions using port 53, but BIND 8.1 uses an unprivileged
* port by default.
*/
// query-source address * port 53;

listen-on { 192.168.2.1; };
forward first;
forwarders { 10.0.2.1; };
};

//
// a caching only nameserver config
//
zone "." IN {
type hint;
file "named.ca";
};

zone "localhost" IN {
type master;
file "localhost.zone";
allow-update { none; };
};

zone "0.0.127.in-addr.arpa" IN {
type master;
file "named.local";
allow-update { none; };
};

acl can_query {
any;
};

edit1:
The reason I made this investigation was that I was curious about what in internet sharing causes disruption of networks. Turns out that the DHCP server is configured to through out  ip’s in any of the mac’s network interfaces. If instead it was doing

bootpd -d -P -i en1

then it would give out ip addresses only to clients on the wireless interface. I’m I missing something or why didn’t apple think of this ?