onTheCode

The SHARC Timbre Dataset v. 2.0: XML Format

2007-10-19T19:23:30Z

SHARC is a dataset of musical timbre information that I collected by analyzing over 1300 orchestral musical instrument notes. Specifically, the information is amplitude and phase data from a selected steady-state portion of each note. The dataset is now available in XML format. publications). One of the projects I undertook was to compile a collection of information on musical instrument tones, which I called SHARC ("Sandell Harmonic Archive"). I've described SHARC in a few places before: in an article from 1991, and in the release notes from the original distribution. Briefly, though, what I did was this. I had a collection of CDs consisting of individually performed notes of all the standard instruments of the orchestra, one recording for each note in the respective instrument's playable range. For each note, I chose a middle portion of the recording, during the note's steady state, and performed a spectrum analysis. I saved the amplitudes and phases of all the harmonics of the pitch's fundamental frequency up to a ceiling of 10kHz. In my first version of the distribution (which you can still download in compressed tar format), SHARC consisted of a series of files, one for each note that was analyzed, organized into directories by instrument. That was 1994; since then, XML has come into being and I've now released SHARC in an XML format. I'm calling this SHARC's "2.0" release, and back-versioning the original distribution to "1.0" (even though I timidly referred to it at as version 0.921 at the time). In this blog article, I'll describe the design of this 2.0 version, for the convenience of anyone who would like to work with it. Let's consider the XML that specifies a single instrument and all of its notes, and their harmonics. The rough outline of the XML is:

  
       
         
         
      ...etc...
    
       
      
      
      ...etc...
    
...etc...

The element has the following attributes:

id: the instrument's short name, containing no spaces, suitable for variable names and querystring parameters
name: the instrument's longer, more descriptive name
source: the cd from which the tone originated
cd: the volume of cd
track: the track on the cd
numNotes: the number of notes for this instrument

Here is a sample element:

The element has the following attributes:

pitch: the notes pitch and octave number, e.g. c4 = middle C. Sharps are specified with the letter s, e.g. 'fs4' rather than 'f#4'.

seq: the sequential order number of the note in the series (i.e. starting at 1 with the first note)

keyNum: numerical location of the pitch on a piano keyboard, where middle C = 48

fundHz: the frequency of the note's fundamental (e.g. a4 = 440)

numHarms: the number of harmonics (i.e. the number of elements to follow)

Here is a sample note element:

Finally we have the harmonic data itself, contained in the element. The harmonic amplitude value is the text node of the element, expressed as a linear value (i.e. not in dB). The attributes for the element are:

n: the sequential order number of the harmonic in the series (i.e. starting at 1 with the first harmonic)
p: phase, expressed in the range between negative and positive pi

Here is a sample sequence of a few elements:

    32.91
    2131.69
    5878.0

Using the brief names 'n' and 'p' keeps the size of the XML document lower. For similar reasons, the frequency of each harmonic is not given. To obtain the frequency of the harmonic, you simply multiply the value of the "n" attribute by the value of the "fundHz' attribute of the 'note' element. As I said, that is a rough sketch of the XML; to simplify the explanation I left out some of the detail. In addition to what I have discussed so far, each instrument element, and each note element, has a sibling element which contains useful metadata. Here is a sample element for an element:


    
        32.7
        
        c1
        0.0
    
    
        f4
        10463.69
        15389.0
    
    c1 cs1 d1 ds1 e1 f1 fs1 gs1 a1 as1 
       b1 c2 cs2 d2 ds2 e2 f2 fs2 g2 gs2 a2 as2 
       b2 c3 cs3 d3 ds3 e3 f3 fs3 g3 gs3 a3 as3 
       b3 c4 cs4 d4 ds4 e4 f4

The logic behind the element is mostly convenience for applications that will be constructing graphic plots from the data. For example, having the highest and lowest frequency specified here, rather than making it necessary to traverse through the data to find it, makes it easier for a program to set up the minimum and maximum for a graphic plot. The element is another convenience that keeps the user from having to issue a thorny xpath query just to get a list of all the instrument's pitches. Let's drill down into the details of this element. The text node of the ranges/lowest/harmonicFreq element is the lowest frequency of any harmonic in the entire instrument's collection. Obviously, this is always harmonic 1 of the instrument's lowest note. The attributes for harmonicFreq convey this, as well as the pitch (c1) and keyNum (12). The element ranges/lowest/pitch contains the same information, but described in terms of the lowest pitch and its fundamental frequency. This redundancy has little impact since it is occurs just once for the instrument. Information about the lowest amplitude harmonic to be found in the instrument is given in the ranges/lowest/amplitude element. For the instrument in question, this honor goes to a#1 (keyNum of 22, fundamental frequency of 58.27 Hz), 10 semitones above the instrument's lowest note, the 145th harmonic (frequency of 8449.15 Hz). The ranges/highest element provides equivalent data for the highest harmonic frequency, highest pitch and highest amplitude. Here is a sample element for an element:

This element provides data similar to instrument/ranges, but in terms of the highest/lowest frequency and amplitude harmonics for the note in question. The XML was designed in a way that the entire SHARC dataset could be combined into a single XML file (i.e. as a series of instrument elements), and this file is in fact available for download in zip format. However, this file is quite large (nearly 3 meg), which will put quite a burden on parsers, and especially DOM parsers. For more efficient processing, I have placed each instrument into its own dataset file. For a summary, here is a shorthand showing the overall design of the xml, with attributes shown in red and text nodes in blue:

tree

    instrument (id, name, source, cd, track, numNotes)
        ranges
            lowest
                harmonicFreq (harmNum, keyNum, pitch) [frequency]
                pitch (fundHz, keyNum) [pitch]
                amplitude (freqHz, keyNum, pitch, fundHz, harmNum) [amplitude]
            highest 
                (all same as lowest)
        note (pitch, seq, keyNum, fundHz, numHarms)
            ranges
                lowest
                    amplitude (freqHz, harmNum) [amplitude]
                    harmonicFreq (harmNum) [frequency]
                highest
                    all same as lowest)
            a (n, p) [amplitude]

I'm not attached to this particular XML design, and I may come out with a 3.0 version some day. One change I expect to make in a future version is to move a lot of information that is in attributes to elements, which means that more queries would return element nodes that could be further processed. Another idea I have is to make a secondary, "bare bones" release, that would have no metadata, for quicker processing. Enjoy playing with the data!]]>

Maven2 Introduction part 1: the Coordinate System

2007-07-13T17:15:00Z

Maven is gaining traction as the premiere form of Java code organization and managing builds. The entire world won't convert overnight, but adoption is likely to steadily increase once more people get over the learning curve and conceptual difference with ant scripts, the previous prevailing model of build management. In this article I'm going to try to take a bit of the steepness off of the learning curve for you.

Avoiding confusion between different versions of the same jars

Maintaining only one copy of the same jar on your computer (instead of having one copy for each of your projects that use it)

Having a mechanism that retrieves jars (and making sure it is the right version as well) from the internet for you without you having to think about where it should be stored

Maven does this my having a highly structured approach to dependencies, and importantly, the adherence to this framework by the community who uses Maven. In this part of the article I'll start by covering the cornerstone of the Maven appeoach, the "Coordinate System," then we'll move on to Maven repositories.

The Coordinate System: groupId, artifactId and version

At the core of Maven 2 is its method of identifying resources (mostly jar files) by a strictly followed practice of file and directory naming. The goal is similar to that of XML namespaces and the java packaging conventions: to define items as distinct points in space according to a unversally followed set of conventions. It's simply these four identifiers:

groupId: usually a reversed domain name such as com.lowagie.
artifactId: a common name for the resource, such as itext.
version: a version indicator such as 1.4. Numbers and decimals are typical, but not required, values for the version.
packaging: the type of end product which could be ear or war, but is most often jar (and therefore the default, so packaging need not be specified)

(A side note about the groupId: there are many jars out there that do not use their organization's reverse domain name. In fact, they comprise some of the most widely used jars out there: log4j, jdom, ant, and xalan to name a few. All they use for their groupId is their simple well-known name (log4j, jdom, ant, and xalan for the examples just mentioned), and their artifactId is the same. These famous jars just happen to have been on the scene during an earlier version of Maven before the convention of using reversed domain names took hold; they held on to their old coordinate locations instead of updating.)

Here is the location of a jar file named itest-1.4.jar in a proper Maven repository. The initial part is chosen by the individual user (c:/.m2/repository) but everything following that is dictated by the coordinate system:
c:/.m2/repository/com/lowagie/itext/1.4/itext-1.4.jar
In the same directory as itest-1.4.jar, you will find the file itest-1.4.pom. This is an XML file containing the following: 4.0.0 com.lowagie itext jar 1.4
In the project's root directory, you will find a file called pom.xml, and that file will contain the same lines as above, but wrapped inside a element: 4.0.0 com.lowagie itext jar 1.4
Retrieval of resources over the internet is integral to maven. A jar can be referred to by its URL on a known repository. The first part of the URL is specific to the repository, whereas the rest follows the file structure of the coordinate system. Here is a URL for the location of a jar at the well-known repository ibiblio: http://mirrors.ibiblio.org/pub/mirrors/maven2/com/lowagie/itext/1.4/itext-1.4.jar
Now here is a Maven command line statement. It's purpose is to install a jar in a repository, but don't worry about that right now; just notice how the coordinate system manifests itself on a typical command line statement. mvn install:install-file -DgroupId=com.lowagie \ -DartifactId=itext \ -Dversion=1.4 \ -Dpackaging=jar \ -Dfile=itext-1.4.jar
Occasionally a point in space is referenced with a single line of text, using the format
groupId:artifactId:packaging:version, as in:
com.lowagie:itext:jar:1.4

These 5 situations show you most of the ways in which jars are referenced in the Maven world. There is a maddening consistency and pervasiveness to the Coordinate System throughout Maven. The more you learn about Maven, the more you discover you've already learned it. ]]>

S-Corps for Software Contracters

2006-10-23T02:20:40Z