• MikeMandel

I'm a first year grad student in the EE department working with Dan on computer systems that understand audio and music, problems like source separation, modelling sound sources, music information retrieval, music recommendation, etc. My background is CS stuff like machine vision, machine learning, and software engineering with a bunch of signal processing thrown in, so I'm not so good at soldering or PIC programming. I also play the saxophone(s), didgeridoo, etc.

As far as projects I'd like to work on, I'd be interested in some sort of interactive composition project (because I wouldn't be doing the composition), although building instruments or a larger, more physical installation might be cool too. One idea I had that might be cool is a dance party in a box: the users supply the dancing and the system watches them with a camera and comes up with music to go along with it.

mim ee columbia edu

Java Masher

1/30/06

I wrote up some documentation on my web page, which includes a link to download the current incarnation of the masher. If you'd like a copy of the code, send me an email and I'll email you back a zip or tarball of it, the arch/tla/version control stuff isn't quite set up for public access. You will also need java.sound packages, which are included in any jdk you download from sun from http://java.sun.com. There's also a more complete implementation for linux of the java.sound API provided by http://tritonus.org.

To compile the code:

$ mkdir jardir
$ jikes -d jardir/ -classpath $CLASSPATH:getopt-1.0.12.jar *.jar
$ cd jardir; jar cmf META-INF/MANIFEST.MF ../meap.jar *.class META-INF/ gnu/ javazoom/ org/ ; cd ..

Finding New Sounds

Chunks

I have chopped up the Beatles' Got To Get You Into My Life and Paperback Writer and saved the chunks along with their features in a matlab file.

To use them, load the matlab file with the command "load('chunks.mat')". Once you load the file, you should have two matrices (t_feat and s_feat) and two cell arrays (t_chunks and s_chunks). Each element in the cell arrays is a chunk of the waveform from one of the songs. They go in order, but I'm not sure which song is which without listening to them. Each row of the feature matrices corresponds to one chunk, so that t_feat(i,:) describes t_chunks{i}.

02/28/05

New version of onset detector. Better feature detector including temporal envelope. New version of Come Closer

02/17/05

In the vein of the mashup Doug forwarded (http://djzebra.free.fr/DJZebra_ComeCloser.mp3) I have made my own. It's a wav file because I couldn't compress it to mp3. All I did, was use my code for finding onsets, extracting features at each onset, and reassembling pieces of one song so that they match up with those of another song (modulo some randomization). All of the scripts are currently written in matlab, but it wouldn't be hard at all to port them to some realtime setup. I was also thinking that it wouldn't be that hard to write a matlab MEX wrapper around portaudio or some such to get realtime sound working in matlab. It took around 30 seconds to make "Come Closer" in matlab using this code.

Currently I'm only using the simplest of features (avg STFT frame), but we could go crazy and put in whatever features we wanted. There are also obviously much more sophisticated ways of determining when a sound ends, currently I take the minimum of the next onset and half a second after the current onset. The difficult part of making a mashup or whatever you want to call it with the current features is finding songs that will be compatible with each other. Compatibility might include being in the same key, having the same chords, having a similar dynamic range, etc. In general, the songs should have chunks that sound similar in order for this to work the best. Attempting to mix one song up and lay it out according to itself, with the included randomness, leads to something less than recognizeable, although not bad. It should be noted that even if everything else is missing from such a mix, the beat seems to be pretty well preserved.

Tristan Jehan has apparently not released his program for doing similar things, but he does have some examples of what he calls Musical Cross Synthesis on his website.

02/07/05

I made a recording of our office here for about an hour and tried to get some interesting sounds out of it. Unfortunately, there weren't very many interesting sounds to be had except for my testing at the beginning. Next week I'll head over the Lerner and make a better recording over therer, perhaps one with a real mic (as real as it can be and still plug in to an 1/8" jack). Check out my code and clips.