code

ubuntu and btrfs backups using send receive

Since I’ve moved to Ubuntu Linux I’ve been mostly using btrfs for all my filesystems. btrfs makes it super easy to create snapshots of subvolumes and to go back in time to a previous snapshot (just use mv). I always create a snapshot before updating packages so that I can easily go back if something breaks.

I’ve been using Back In Time for my backups but this app takes more then 30 minutes for  every single backup, no matter how small, and does a lot of disk I/O in the process, even just to transfer 2 or 3 files (I think that happens because it’s creating  hard links).  I’ve recently discovered that btrfs has a send / receive functionality which allows sending a snapshot to another btrfs partition and it also allows to send just the diff between snapshots. Using btrfs send and btrfs receive it’s possible to setup a backup process entirely based on btrfs.

I found that btrfs send / receive is broken  in the current ubuntu version 13.04, so I had to uninstall the btrfs-tools package and compile from the source in git of btrfs-progs. After that btrfs send and btrfs receive seem to work fine. I’m now doing all my backups with btrfs which on average now take just one or two minutes, much much faster then with Back In Time.

Finally I created a Haskell Shelly script to automate my backup procedure. It’s still very incomplete, but I put here in case it’s useful to others. Be careful with this script, it might delete snapshots that it shouldn’t so use at your own risk !

Tags | haskell

Monad Transformers Hello World

I’ve been searching the webs for a simple example of monad transformers in action in Haskell. Don’t remember if I actually did find one or not, but in any case I’m posting my own Monad Transformers hello world. I really don’t understand why texts explaining Haskell concepts such as these can’t have a simple hello world example to just to see the concept in use in it’s most simple form.

OSX Lion – script for automatic ad-hoc network creation

I have to create an ad-hoc network everytime I need to play with supercollider from a tablet or phone, and this is very annoying, so I automated the process using two apple scripts which I bundled as applications and are now sitting on my desktop. The script was copied from somewhere on the net and altered to work on my OSX Lion configuration, so it would have to be adapted if the one has different items order on the menu bar. Also my static IP configuration is named “osc-iphone”, this should be renamed to whatever network configuration one uses for the ad-hoc network.

Go back to default configuration:

A look back on my work with ScalaCollider

I have just recently completed my first project using ScalaCollider. ScalaCollider is a front-end written in Scala, a hybrid object-oriented/functional language that runs on the Java virtual machine. I would like to share my impressions of using ScalaCollider now that I have come to use it for quite some time. I will mostly compare it to SuperCollider (using SCApp on the mac).

Positive:
– I can use all the infrastructure around the JVM and Scala. It’s easy to manage dependencies and to bootstrap a project using Simple Build Tool (SBT). There are several extremely competent IDE’s available that are used by developers at companies like Twitter. There are performance analysis tools available that are used by professionals. It’s possible to build an executable that can be distributed and that runs with a simple doble-click easily. This was possible in SuperCollider using standalones but it’s a complicated process, which meant I didn’t do it often enough. With scala it’s so easy that I keep binary copies of my project during development at several stages and after each concert (it’s one command in sbt). I also keep a copy of scsynth and the plugins which means that I have an executable that is exactly the same setup I had at that particular concert.
– I personally use the Intellij IDEA IDE, which is amazing. Coming from SuperCollider.app, IDEA just blew me away. It’s easy to edit code, it can reformat code easily, refactor code (change name of variables, classes, etc across multiple files), there’s no more tabbing and hitting space to format text, just place the cursor anywhere on screen and start writing. Because scala is a static language, the autocompletion is mind-boggling at first (in a good way !). Also help is available just by hovering the cursor over class name and hitting a key combination. Help is also available during auto-completion, so it’s easy to check what each method available does (at least after the help system is configured correctly, which might take some time).
– Scala is a new language that is still evolving. It has many new features that are ground-breaking such as implicit conversions, which allows having type classes like in Haskell and extending libraries that we cannot change. It’s a cutting-edge language, full of new ideas and possibilities. SuperCollider in comparison feels old. For instance, the scalaz library is incredibly powerful and advanced. It’s basically pure math. It’s quite a challenge to use it, but what you get in return is a new way of looking at things. Through scala I came in contact with a lot of new concepts (realized in libraries that I can use) such as Option/Maybe, Algebraic Data Types, Futures/Promises, Monads, Applicative Functors, Functors, Foldables, Functional Reactive Programming (Events and EventStreams), etc.
– Even if scala is a language that compiles down to byte-code, it’s still possible to use it as if it was interpreted using a REPL (Read Evaluate Print Loop), which is an interactive prompt where we can evaluates snippets of code. It’s therefore possible to write small scripts without declaring any classes. The environment to do this is not as nice as in SuperCollider, for instance, it’s not currently possible to select a portion of code and evaluate that, although that is just because no one implemented that yet (although sciss has tried already to make a plugin for IDEA to do this). I wouldn’t use ScalaCollider for live coding at this point, since the type system can become cumbersome when trying to code things quickly, although most of the times it’s quite invisible.
– It’s possible to use any Java or Scala library together with ScalaCollider that means that there is already code out there to do almost anything that we can think off, and it’s easy to interface with those libraries because they are all written either in Java or Scala (no need to go down to c or c++). I was astonished when I implemented the functionality of discovering my iphone when using TouchOSC with Bonjour/Zeroconf with just a few lines of code by importing a java library. The same was true of json parsing. It’s possible for instance to do graphical stuff using processing (Java) or SPDE (a Scala version of processing). The possibilities are endless. And most of the times these libraries are actually written by computer scientists, or people that work in companies or institutions, so the quality standards are quite high.

Negative:
– Some things are missing, such as Routines (there are some implementations available, but not final yet) and Patterns and others are still work in progress. Most of what is implemented is already working quite well though, I detected few bugs.
– It takes a bit of time to setup the development environment. Things are quite smoother now then some months ago though, and there is more documentation available to guide us through the process. It took me quite some time to understand how to get the help system going. Now the latest version of the SBT plugin automatically pulls the source jars when generating an IDEA project, which means I get the help there automatically. It took me some time to figure out how to use the continuations plugins in IDEA, which is necessary to use Routines. Hopefully all of this will be ironed out with time.
– It’s not as easy to just evaluate pieces of code, and check what is going on in the code by checking the value of variables. This is in part because the coding style used (which I copied from a sample application) has more classes inside classes, which makes the code more organized, but also more difficult to check things just by evaluating expressions. On the other hand, there is a debugger available, something which is quite handy to check things. It’s magic stepping through the execution process and actually looking at the lines. I confess I still don’t fully understand how to properly use a debugger and many times end up putting a lot of println’s all over the place, but that’s more a limitation that I have.
– The Scala language is more complex then SCLang, there are more things to grasp in order to use ScalaCollider. For instance, scala is statically typed, which means that I got a lot of errors about something being of the wrong type (usually collections) and I didn’t always know how to solve this issues. It gets better with time, but still it might take more time to code something (unless off course we are using an advanced feature of scala that actually makes it less code anyway). I don’t think the average SuperCollider user would have an easy time in scala.
– I found the GUI code a LOT more complex than in SuperCollider. The place where I lost the most time was with scala swing, which is a GUI toolkit. It’s badly documented, and it’s just more complex, although also more powerful. Scala swing has very powerful layout managers, so when I started using it I was delighted that I would never type pixel locations in Scala. But after some time it turned out that it actually takes more time to make the layout system do the right thing then actually typing the hardcoded pixel values by hand (provided I would never change those values, which is often not true).
– ScalaCollider runs in an environment (the Scala development environment) which is not specifically made for music. That means that many things are more complex then they need to be for audio work (things like the GUIs, selecting files from disk, serialization of objects, etc).

All in all, using ScalaCollider has been a quite positive experience. If not for anything else, I learned a lot along the way. I also lost a lot (a LOT !) of time, so ScalaCollider is not for those that are in a hurry to get something done. I had to figure out a couple of things on my own which took extra time. For instance to save settings to disk I had to learn on how to do serialization using the JSON serialization provided by liftweb, and to write some custom code for my classes (in SuperCollider all I needed to do was, .writeArchive or .asCompileString).

ScalaColliderZ

ScalaColliderZ is a library that provides some added features for ScalaCollider based on the scalaz library. Scalaz is a library for pure functional programming in Scala, which takes a lot of inspiration from Haskell. It has several type classes such as Monad, Monoid, Functor, etc.

ScalaColliderZ provides some methods (many more could be implemented) to deal with asynchronous tasks. Things such as sending synthdefs to the server or loading buffers are done in ScalaCollider using responder functions: we pass a function to the method that gets executed whenever the task has completed. The problem with this is that it’s not easy to sequence several tasks. For example it’s convenient to be able to load a buffer and send a synthdef and then play a synth when both tasks have completed. Promises are perfect for this. A promise is a kind of container that will have a value of certain type at some moment in the future. This is similar to Lists that are containers that can have multiple items of the same type or Option which is a container that might or might not have an item. It’s therefore also possible to use map and flatMap to get a new Promise from an existing one and foreach to execute an action when a promise get it’s value. Example:

promise{ /* do something that takes some time*/; 4 }.map(_+1).foreach(println)

Currently there are methods to load Buffers (BufferPromise.load) and to send synthDefs (mySynthDef.loadP) that return a a promise:

And an example of sequencing:

ScalaColliderZ also adds type classes instances for GE for Monoid:

I hope to add more stuff in the future.

Functional fibonacci numbers

There are many ways to calculate the fibonacci numbers sequence in scala, but I really like this one:

def fibs(n:Int)= {
    lazy val fibSeq:Stream[Int] = 0 #:: 1 #:: (fibSeq zip fibSeq.tail).map{ case (a,b) => a+b }
    fibSeq.take(n).toList
}

#:: is the lazy cons method, and fibs zip fibs.tail creates a stream of pairs with the latest two numbers in the sequence.

In haskell it can be even more compact:

fib :: Int -> Integer
fib n = fibs !! n
  where
    fibs = 0 : 1 : zipWith (+) fibs (tail fibs)

Using streams in scala for functional style programming

The paper “why functional programming matters” by John Hughes is a very nice example of using streams to calculate an approximation of the square root of a number using the Newton method.

The #:: method is a lazy cons, that only calculates the element to be added to the head of the list when needed. The repeat function creates a stream by repeatedly applying f to the last element of the  stream starting from the value a. The within function keeps looking for better matches in the stream until the successive values are close enough to each other. By chaining these functions you get the result:

/**
 *
 * Newton-Raphson Square Roots
 * as is shown in "why functional programming matters"
 * http://www.cs.utexas.edu/~shmat/courses/cs345/whyfp.pdf
 */

object NewtonRoots {

  //algorithm to approach the square root of n
  def next(n:Double)(x:Double) = (x + n/x)/2

  //create a stream composed of repeatedly applying f
  def repeat(f:Double=>Double, a:Double):Stream[Double] = a #:: repeat(f,f(a))

  //look for a good enough match in the stream
  def within(eps:Double, s:Stream[Double]):Double = s match {
    case a #:: b #:: rest => if( (a-b).abs < eps) b else within(eps, b#::rest)
  }

   //look for a good enough match in the stream.
   // A better way is to see how close a/b is to 1,
   // to avoid the rounding errors
  def relative(eps:Double, s:Stream[Double]):Double = s match {
    case a #:: b #:: rest => if( (a/b -1).abs < eps) b else within(eps, b#::rest)
  }

  //find the square root of 3 starting from the approximation
  // value 1, until approximations are within 0.01
  def main(args: Array[String]) {
    println( within(0.01, repeat( next(3), 1 ) ) )
  }

}

scala 2.9.1 + sbt 0.11 + Intellij IDEA (10/11) + scala continuations plugin

It took me some time to figure out how to compile a project in Intellij IDEA with the scala continuations plugin turned on. I was getting this error when I ran the project:

bad option: -P:continuations:enable

Here is what you need to do to make it work:

1. In your build.sbt file add:

   autoCompilerPlugins := true
   
   addCompilerPlugin("org.scala-lang.plugins" % "continuations" % "2.9.1")
   
   scalacOptions += "-P:continuations:enable"

2. In sbt run

   reload

   and then

   update

3. Generate the Intellij IDEA project using the sbt command

   gen-idea

4. Open the Intellij IDEA project and go to “Project Structure” -> “Modules” ->”Scala” and add the continuations plugin jar file in the “Compiler Plugins” pane. The jar should be in ~/.ivy2/cache/org.scala-lang.plugins/continuations/jars/continuations-2.9.1.jar .
5. Tick the “Enable Continuations” option in the “Compiler Options” pane.

Some examples of pure functional programs that actually do something.

I’ve been struggling with functional programming lately. The part about using pure functions is very easy for me to get, what was/Is difficult is to understand how does a pure program contact with the outside world and manages to keep state.
I then studied a bit about the IO monad, and I managed to understand how to contact with the outside world. The final piece of the puzzle that I’m still missing is out to piece all this together and how to keep the state in a functional way. I’ve coded some very simple examples to teach myself how these things are done. The examples are scala code and use the scalaz library.

In this first example I just read one line at a time from the prompt and then write it back. The app will do this forever until it is killed. The part that causes it looping forever is done by the recursive function forever, which just repeats the same action over and over:

 def forever(theIO:IO[Unit]):IO[Unit] = theIO >>=| forever(theIO)

I find it quite magical way the way this compact syntax encapsulates the action of just doing something over and over again.

import scalaz._
import Scalaz._
import effects._

/*

This program just echo the input

 */

object Test1 {

  def forever(theIO:IO[Unit]):IO[Unit] = theIO >>=| forever(theIO)

  def functionalMain = {
   forever(for(
      line

On the second example I read one line from the prompt and then add it to a list. I was looking for ways to do this without storing this list anywhere. The method that causes the looping in this case is slightly more complicated, because it has to keep propagating an argument forever, by just passing it to the next action. This way we manage to keep state, a list with all the previous strings, without actually storing this list anywhere. Again, I find this quite magical.

import scalaz._
import Scalaz._
import effects._

object Test2 {

  def getStringAndAddToList(list:List[String]):IO[List[String]] =
    for(
      line (A=>IO[A])):A=>IO[A] = makeIO(_) >>= forever(makeIO)

  def functionalMain = {
    val action = forever(getStringAndAddToList _)
    action(Nil)
  }

  def main(args: Array[String]) {
    functionalMain.unsafePerformIO
  }

}

Tags | pure functional programming, scala, scalaz

scala – iterate through elements of collection with their indexes

I finally discovered a nice syntax for iterating through a collection with the indices of each element provided:

List(9,9,9,9,9,9).zipWithIndex

res0: List[(Int, Int)] = List((9,0), (9,1), (9,2), (9,3), (9,4), (9,5))