-

Just to show that it's not limited to Spotify, here is an example of the watermark artifacts through another distributor.

What the watermark sounds like

Spectrogram of the difference between a watermarked and unwatermarked UMG track. The energy is concentrated in two bands between about 1 khz and 3.5 khz - where the human ear is most sensitive.

The watermark scheme modulates the total energy in two different bands, 1khz to 2.3 khz and 2.3 to 3.6 khz. This is known as a spread spectrum watermark. The energy is concentrated in the most perceptually sensitive frequencies because that makes it more difficult to attack or remove without (further) audible distortion.

The energy is increased or reduced in 0.04 second blocks. It can be characterized as a fluttering or tremolo sound. Listen closely to the original vs. watermarked audio samples and try to focus on the 1 khz to 3.6 khz noise range. It helps to wear headphones in a quiet environment.

Audio samples

Here is a short sample (excerpt: Three Doors Down - When You're Young). These are lossless original and watermarked files; what you hear is not a result of compression.
Original:

Watermarked:

If the difference between the two isn't clear, here it is by itself:
Difference:

Technical details

The watermark does not start until 1 second into the audio. After this the signal is divided into 0.08 second blocks. Each block is divided in two: some amount of energy is added to the first half and the same amount is subtracted from the second half. This coding scheme allows blind detection (without access to the original file). The actual information in the watermark is not easily recovered because it is modulated by a pseudo random sequence, which is generated by a secret key.

Removing the watermark

Since the watermark creates audible distortion, it's worthwhile to try to reduce it. I wrote a script that analyzes the block energy and applies some smoothing. This is the result so far.
Watermarked:

Restored:

More discussion

Hydrogenaudio forums on watermarking

UMG Watermarks audiophile files, pisses off paying customers

Why do labels watermark tracks? Watermarking simplifies copyright enforcement by letting a company track music on peer-to-peer networks. "It gives them the ability to put pressure on policy makers and ISPs to do filtering," says Fred Von Lohmann, an Electronic Frontier Foundation attorney. That may be about the best explanation you will find. See DRM Is Dead, But Watermarks Rise From Its Ashes

I don't have anything against watermarking, but I have a problem as a consumer when it is poorly implemented and destroys music I've downloaded legally.

Why isn't UMG's watermark talked about more? Maybe people think it's a compression issue, as I did, and blame the streaming service/distributor. Or maybe people just aren't noticing the artifacts. That is fine, but really, if you've spent more than $39 on your speakers, you will probably care. Especially if you are paying the full retail price for something advertised as lossless audio.

Abstract:

Wave field synthesis (WFS) is a spatial audio rendering technique that produces a physical approximation of wavefronts for virtual sources. Large loudspeaker arrays can simulate a virtual source that exists outside of the listening room. The technique is traditionally limited to the horizontal plane due to the prohibitive cost of planar loudspeaker arrays. Multiple-line-array wave field synthesis is proposed as an extension to linear WFS. This method extends the virtual source space in the vertical direction using a fraction of the number of loudspeakers required for plane arrays. This paper describes a listening test and software environment capable of driving a loudspeaker array according to the proposed extension, as well as the construction of a modular loudspeaker array that can be adapted to multiple-line configurations.

Itsy Bitsy Spider by TextAssist

The built-in text-to-speech on Mac OS also has the ability to sing your melodies through phonemic modifiers and TUNE syntax. (The full specification is available at the link.) TUNE embedded speech commands alter intonation by controlling pitch, word emphasis, pause length, etc. This stuff is baked in, so if you're on a Mac, try highlighting the following code block, control-click, and go Speech > Start Speaking:

[[inpt TUNE]]
_
1OW {D 1066; P 109.0:0 119.8:12 153.0:30 164.6:39 154.0:50 131.7:62 120.7:69 111.2:73 97.1:89 103.1:93}
_
h {D 194; P 123.0:0 130.0:12}
1EY {D 652; P 147.0:0 159.0:14 161.0:32 147.3:63 112.0:91}
_
w {D 161; P 109.0:0}
1AW {D 1171; P 116.0:0 153.3:17 148.2:33 151.0:51 133.0:55 112.4:71 99.5:74 79.0:89 65.7:98 119.0:100}
% {D 293}
_
D {D 132; P 127.0:0}
IH {D 318; P 160.0:0 136.5:43 114.6:63 112.0:88}
s {D 145; P 112.0:0 119.0:95}
_
IH {D 189; P 119.0:0 119.0:13 119.0:27 120.0:87 119.0:93}
z {D 180; P 117.0:0 117.0:64 117.0:86 117.0:93}
_
k {D 80; P 117.0:0 113.0:81 164.0:95}
1UW {D 820; P 164.0:0 168.0:9 172.0:18 169.5:46 139.0:74 116.8:86}
l {D 260; P 104.0:0 92.0:64 104.0:100}
. {D 212}
[[inpt TEXT]]

There's also a simpler mode for phonemic modifiers. The first line here is spoken with default interpretation, and the second line uses the modifiers.  Try it:


You talkin' to me
[[inpt PHON]] [[slnc 500]] [[rate -30]]
+yUW _1tAOl=kIHn ~AX [[pbas +3]]+mIY?

XCode also ships with a hidden gem called Repeat After Me that helps you build this funky syntax from your own spoken phrase. It extracts pitch contour and fits phonetic onsets of a typed phrase to your spoken phrase. 

Tones that fall in the HDR range of the histogram are tones that are impossible to display on your monitor and therefore appear as blown out or completely stopped up spots in an image.

In video games with an HDR graphics pipeline, the exposure is controlled automatically. http://www.youtube.com/watch?v=-spSQYtVfuk The game acts like a digital video camera, truncating the histogram based on what's in the center of the frame.

However, with static HDR photographs or panoramas on a computer screen, there is no such region-of-interest-based automatic exposure. The entire picture has to be tone mapped, flattening out the image's dynamic range. If you ask me, that defeats the purpose of retaining HDR images in the first place.

One way around this is to create an interactive HDR viewer that would tune the exposure for, say, the region near the mouse cursor. But an even better option would be to track the user's gaze. Now we're in academic territory.

Interactive viewing could be implemented in browser with these cool HDR Javascript tools: http://pfstools.sourceforge.net/hdrhtml.html. Actual gaze tracking in the browser, well, that I'm not so sure about.

Another cool site that demonstrates true HDR: http://www.hdrlabs.com/gallery/realhdr/

Camera Capture in OpenCV 2.x

#include "opencv\cv.h"
#include "opencv\highgui.h"
using namespace cv; ...
VideoCapture cap(0); // open the default camera
if(!cap.isOpened()) // check if we succeeded
return -1;
for(;;) {
Mat frame;
cap >> frame; // get a new frame from camera
imshow("Camera Preview", frame);
if(waitKey(30) >= 0) break;
}

OpenCV 2.1 Visual Studio Project Starter

• C/C++ > Include Directories:
• X:\OpenCV2.1\include
• Linker > Additional Library Directories:
• X:\OpenCV2.1\lib
• Linker > Input > Additional Dependencies: (this will vary from one project to the next, but you probably want the first three)
• cv210.lib
• cxcore210.lib
• highgui210.lib
• cvaux210.lib
• cxts210.lib
• ml210.lib
• opencv_ffmpeg210.lib
• Add to PATH environment variable (or copy DLLs into output directory):
• X:\OpenCV2.1\bin

OpenCV 2.3.1 Visual Studio Project Starter

OpenCV 2.3.1 directory structure is different from other versions. I downloaded from OpenCV-2.3.1-win-superpack.exe. This assumes x86, 32-bit architecture, so change accordingly if you are on 64-bit Windows. Note this also includes libs for static linking (instead of requiring DLLs) in OpenCV2.3\build\[architecture]\[compiler]\staticlib. Project settings (this is kind of a deluxe set that was necessary to build the opencv_stitching example):

• C/C++ > Include Directories:
• X:\OpenCV2.3\build\include
• X:\OpenCV2.3\modules\imgproc\src
• Linker > Additional Library Directories:
• X:\OpenCV2.3\build\x86\vc10\lib
• Linker > Input > Additional Dependencies: (this will vary from one project to the next, but you probably want the first three)
• opencv_core231.lib
• opencv_highgui231.lib
• opencv_imgproc231.lib
• opencv_features2d231.lib
• opencv_flann231.lib
• opencv_gpu231.lib
• opencv_haartraining_engine.lib
• opencv_legacy231.lib
• opencv_ml231.lib
• opencv_objdetect231.lib
• opencv_ts231.lib
• opencv_video231.lib
• opencv_calib3d231.lib
• opencv_contrib231.lib

• Add to PATH environment variable (or copy DLLs into output directory):
• X:\OpenCV2.3\build\x86\vc10\bin
• X:\OpenCV2.3\build\common\tbb\ia32\vc10 (optional, for Intel Threading Building Blocks parallel processing support)

And you'll need to include the right headers at the top of your source. You have the option of including the entire OpenCV 1 and OpenCV 2 C++-style interface, or just the C++-style interface. This gives you everything, since cv.h and highgui.h include the C++ interface:

#include "opencv/cv.h"
#include "opencv/highgui.h"
...
using namespace cv;

If you are feeling dangerous, you can use the C++ interface exclusively with .hpp:

#include "opencv2/core/core.hpp"
#include "opencv2/highgui/highgui.hpp"
...
using namespace cv;

Integrating OpenCV in larger programs
with rich interfaces

OpenCV highgui is great for experimenting.  But eventually, you might want to develop a useful stand-alone program.  So for this, cv::imshow and cvWaitKey are not going to cut it. See discussion at StackOverflow, Integrating OpenCV with larger programs.

In most cases, there is no magic to handing off image data to other libraries. The standard container is an unsigned char array. If your image is 320x240 RGB, then your array size is 3 x 320 x 240 x sizeof(unsigned char) bytes. Since this is an unlabeled container, the consumer of this data will also need to be supplied with the image height, width, and color mode. You can initialize a cv::Mat with raw data by supplying a pointer (this assumes RGB color mode): Mat img(rawDataWidth, rawDataHeight, CV_8UC3, rawData); and later access the raw data with img.data. The only snags you are likely to come across are in dealing with row order and RGB vs. BGR format. OpenCV provides cv::flip() and cv::cvtColor() to deal with this.

I have had success using videoInput library to access DirectShow devices. The library enumerates and lists friendly names of connected devices, and generally gives you more information when something goes wrong. It's much better than guessing capture device IDs, and essential if you are going to provide some camera selection to the user.

Qt + OpenCV

Key points:

• Qt is a good option because Qt is great and has a big community.
• Open a new thread to run cv::VideoCapture in a loop and emit signal after frame capture. Use Qt'smsleep mechanism, not OpenCV. So, we are still using OpenCV highgui for capture.
• Convert cv::Mat to QtImage:
  QImage qtFrame(cvFrame.data, cvFrame.size().width, cvFrame.size().height, cvFrame.step, QImage::Format_RGB888);qtFrame = qtFrame.rgbSwapped();
• Optional: Render with GLWidget. Convert QtImage to GLFormat with Qt built-in method:
  m_GLFrame = QGLWidget::convertToGLFormat(frame);this->updateGL();

Juce + OpenCV

I have also had success integrating OpenCV with Juce/OpenGL, rendering a capture frame on rotating surface in OpenGL. Converting a cv::Mat to an OpenGL texture goes like this:

declarations:

cv::Mat frame; // object to receive a captured frame.
cv::VideoCapture cap;  // video capture object
CvMat *_img;
CvMat *arrMat, *cvimage, stub;

in camera capture loop (usually a separate thread):

cap >> frame;
/* following will convert cv::Mat to CvMat data format, which can be
	used as a texture in OpenGL */
if (_img == 0)
	_img = new CvMat(frame);
CvArr *arr = _img;
arrMat = cvGetMat(arr, &stub);
cvimage = cvCreateMat(arrMat->rows, arrMat->cols, CV_8UC3);
cvConvertImage(arrMat, cvimage, 0);
//sleep function here:
  waitKey(33); //OpenCV
//Sleep(33);   //windows.h
//msleep(33);  //Qt
//wait(33);    //Juce

in the OpenGL render function:

glTexImage2D(GL_TEXTURE_2D, 0, GL_RGB, cvimage->cols, cvimage->rows, 0,
	GL_BGR_EXT /* GL_BGR */, GL_UNSIGNED_BYTE, cvimage->data.ptr);

I have a feeling this sample code, which I borrowed from this OpenGL+OpenCV tutorial, could be even simpler; I also don't like that it's using OpenCV 1.0 style structures. More on Juce/OpenGL later.

Google Latitude:

Stractor:

Okay, that's nice. Here are some other Stractor concepts...maybe we'll see some of these soon:

http://www.butterscotch.com/news/168/Google-Continues-To-Improve-And-Streamline-Maps-In-New-Update

Music Smasher is currently hosted at mattmontag.com/smasher.

Note that due to differences in the APIs, the result counts aren't telling the whole truth (especially for Grooveshark ). I hope I can fix that.
Please post any problems or suggestions in the comments.
Technical notes coming soon.

Update 9/9/2011:

Music Smasher has been pretty useful during regular Spotify browsing. Today I learned Spotify doesn't have any Antiloop, Mark Ronson's Bike Song, or any of the Fabriclive series.

Update 12/24/2011:

Results are now linked to for streaming. Rdio tracks that are unavailable for streaming or download will no longer show up in results.

I love Spotify. But I've noticed some weird artifacts on a few albums. It sounds like a fluttery warble noise in the midrange. It's most noticeable during big string or choir sections with broad spectral content.

The problem seems limited to certain albums. The worst album I've come across is this Pascal Roge Poulenc album.

Spotify uses Ogg Vorbis encoding. Free streaming accounts get Ogg Vorbis q5, 160kbps. I wonder if this is a normal Ogg Vorbis artifact, or something more complicated. Here are some audio samples. Can you hear it?

Michael Jackson - Ben / on Spotify
Poulenc - Piano Concerto Movement 2 (linked above)
Poulenc - Piano Concerto Movement 3
Poulenc - Piano Sonata Movement 1

And the worst example - yikes, is this intentional?
Three Doors Down - When You're Young / on Spotify

Update 1:

I just upgraded to Spotify Premium and switched to 320kbps high quality streaming. I'm sad to say the nasty artifacts are still audible on these songs.

Update 2:

Due to investigative effort by Ulysses at spotifyclassical.com, this is no longer a mystery but indeed just a case of bad compression; it turns out maybe 50 percent of the Spotify library is actually available at 320kbps.

If this is true then I don't like Ogg Vorbis, since their 160kbps rate seems to have far more annoying artifacts than any 160kbps MP3 I've ever heard. Ogg is supposed to be better. I have not been able to reproduce the artifacts with my own Ogg compressor.

Update 3:

I compared the latest Three Doors Down album on Spotify, Rdio, and FLAC original audio. It's pretty clear that sometimes, the labels just supply bad tracks to the streaming services. The Spotify and Rdio tracks were identical.  I heard the same artifacts at the same points in the song.  The FLAC was very different.  This changes some of my earlier conclusions. The artifacts are not Ogg 160kbps artifacts.

To answer my question above, the warbling noise is definitely not part of the original Three Doors Down - When You're Young  track.  Now the question is, exactly what format are these tracks provided in, and why is there such variance even among a single label's catalog? Are these differences intentional?

Is it possible to "reverse engineer" the PCM data and recover the original MP3 frames, instead of reencoding and incurring more compression artifacts?

This is analagous to the question of whether you can revert a decompressed bitmap to a JPEG source, covered by this paper on Exact JPEG recompression. The authors are able to recover the compression parameters (quantized DCT coefficients) for 96% of the blocks** in images compressed at a JPEG quality of 75%. Cool!

So Spotify provides exactly the aforementioned API, with opaque methods for getting raw audio streams. Other services, whether they offer an API or not, provide the user with an uncompressed audio stream one way or another. So it would be a "big deal" to be able to recover the original compressed data.

Google Scholar queries for audio and mp3 recompression give lots of results related to audio watermarking, but not much in the way of optimal audio recompression. Seems to be a very interesting and potentially valuable research opportunity.

*Or any perceptual audio codec (AAC, Ogg Vorbis, etc.)

**Prior to JPEG compression, images are chopped into 8x8 pixel squares. Each block is compressed independently.