Friday, January 27, 2012

beaTunes 3.5 Early Access 6: Close to Release

beaTunes2 logoMostly, to bridge the gap until 3.5 is actually released, here's EA6. Hardly any changes - so not much to write about.

As always, there is some more info in the NOTES.txt file.

Alright, here's the obligatory warning: Before downloading and installing this, please make sure you understand what Early Access means:

  • Absolutely no warranty for whatever
  • Features may or may not work, appear, and disappear
  • It may not be possible to migrate data to future or previous versions (even though I make a reasonable effort)
  • This version will cease to function one week after its release
  • You cannot buy this version

And here are the download links:

Important

For those still using beaTunes 2: beaTunes 3.x changes beaTunes' internal database layout quite a bit, which takes a while. Do not interrupt this process, even if it takes very long! beaTunes 2.x will not be able to use the database anymore, after you ran beaTunes 3.x. If you intend to try 3.x and then go back to 2.x, make sure you back up the database before you try 3.x (depending on your OS it's in ~/Library/Application Support/beaTunes/Database, C:\Users\<username>\AppData\Local\tagtraum industries\beaTunes\database or C:\Documents and Settings\<username>\Local Settings\Application Data\tagtraum industries\beaTunes\database)! You also might want to check out this article about exporting beaTunes data.

Feedback

Please provide feedback! Let me know what does not work. Especially audio decoding problems, OutOfMemoryErrors and UI hangs and hiccups.

Thanks.

PS: The update from 3.0 to 3.5 will be free.

Labels:

Thursday, January 19, 2012

beaTunes 3.5 Early Access 5: small and BIG Fixes

beaTunes2 logoTesting pays off. Always. Latest proof of this truism is a subtle but nasty bug I found in one of beaTunes' BPM algorithms. It's fixed as of EA5, but earlier 3.5-EA releases definitely produced bad results. Other changes in this release are some small inspection improvements, but not much else on the beaTunes side of things.

That said, the Windows version now comes with a bundled Java 7 runtime.

As always, there is some more info in the NOTES.txt file.

Alright, here's the obligatory warning: Before downloading and installing this, please make sure you understand what Early Access means:

  • Absolutely no warranty for whatever
  • Features may or may not work, appear, and disappear
  • It may not be possible to migrate data to future or previous versions (even though I make a reasonable effort)
  • This version will cease to function one week after its release
  • You cannot buy this version

And here are the download links:

Important

For those still using beaTunes 2: beaTunes 3.x changes beaTunes' internal database layout quite a bit, which takes a while. Do not interrupt this process, even if it takes very long! beaTunes 2.x will not be able to use the database anymore, after you ran beaTunes 3.x. If you intend to try 3.x and then go back to 2.x, make sure you back up the database before you try 3.x (depending on your OS it's in ~/Library/Application Support/beaTunes/Database, C:\Users\<username>\AppData\Local\tagtraum industries\beaTunes\database or C:\Documents and Settings\<username>\Local Settings\Application Data\tagtraum industries\beaTunes\database)! You also might want to check out this article about exporting beaTunes data.

Feedback

Please provide feedback! Let me know what does not work. Especially audio decoding problems, OutOfMemoryErrors and UI hangs and hiccups.

Thanks.

PS: The update from 3.0 to 3.5 will be free.

Labels: , ,

Thursday, January 12, 2012

beaTunes 3.5 Early Access 4: Improved Inspections, Faster Contextshop

beaTunes2 logoPretty much all prio1 3.5 todos are addressed. That means, we are getting closer to the final 3.5 release. Today's EA4 release features a couple of bug fixes as well as some performance improvements for the context shop code. The Order by action should work again, (Album Version) issues can now be ignored, and the different rating inspection is somewhat improved to take track ids into account.

As always, there is some more info in the NOTES.txt file.

Alright, here's the obligatory warning: Before downloading and installing this, please make sure you understand what Early Access means:

  • Absolutely no warranty for whatever
  • Features may or may not work, appear, and disappear
  • It may not be possible to migrate data to future or previous versions (even though I make a reasonable effort)
  • This version will cease to function one week after its release
  • You cannot buy this version

And here are the download links:

Important

For those still using beaTunes 2: beaTunes 3.x changes beaTunes' internal database layout quite a bit, which takes a while. Do not interrupt this process, even if it takes very long! beaTunes 2.x will not be able to use the database anymore, after you ran beaTunes 3.x. If you intend to try 3.x and then go back to 2.x, make sure you back up the database before you try 3.x (depending on your OS it's in ~/Library/Application Support/beaTunes/Database, C:\Users\<username>\AppData\Local\tagtraum industries\beaTunes\database or C:\Documents and Settings\<username>\Local Settings\Application Data\tagtraum industries\beaTunes\database)! You also might want to check out this article about exporting beaTunes data.

Feedback

Please provide feedback! Let me know what does not work. Especially audio decoding problems, OutOfMemoryErrors and UI hangs and hiccups.

Thanks.

PS: The update from 3.0 to 3.5 will be free.

Labels: ,

Tuesday, January 10, 2012

Increasing General Lookup Performance

beaTunes2 logoYou might remember my post back from October, in which I promised to keep working on server-side performance. Well, as you might have noticed, upgrading the master server didn't happen in November... However, I didn't drop the ball and in the past couple of weeks finally found the time to actually get this done. Today the new master processing server went live. And so far the numbers look promising.

For those of you who care about technical details: The load on the server is disk-bound. That means, that neither the main memory, nor the CPU are the bottleneck, but rather the speed of input/output operations to disk (as opposed to the network). How do I know this? Simple. Just run the Unix tool top and check the I/O wait load (wa), i.e. the amount of time the CPU waits for I/O operations to complete. If that number is substantially higher than user CPU time (us) and system CPU time (sy), you just know that disk I/O is the limiting factor. Of course there are multiple approaches one can take to increase throughput. E.g. split up the load onto multiple servers, don't write as much or defer writing, attempt to cache smarter, decrease I/O wait, ... For now, I chose the last option, i.e. decrease I/O wait by adding more hardware. What used to be a RAID1 with two disks, is now a RAID0 with seven disks. You might correctly observe that this kind of system is prone to disk failures, as the failure rate for RAID0 with seven disks is something to take seriously... And yes. It's true. But for now I'm willing to take the risk, because the MySQL replication slave I'm running, is hosted on a RAID1 system and its backup is kept somewhere completely different. So even when the master system goes belly up, I can restore the system quickly based on the slower but way more reliable backup server. Plus, there is always the option to create a hot standby for the master, running the same setup.

I'll know in a couple of days, if the system is up for the load. But as I said, things look very promising so far.

Labels:

Thursday, December 22, 2011

beaTunes 3.5 Early Access 3: Faster Startup, faster Data Fetching

beaTunes2 logoThis week's release is focusing on a number of issues with the internal database. As some of you may know, beaTunes uses a little embedded database called H2. It's open source, written in Java, and has served us well so far. In the previous 3.5EA versions I had switched the database engine to MVCC (Multi-Version Concurrency Control). As it turns out, this is not the best setting for the kind of workload beaTunes generates, as often times the entire database gets locked. Experiments have shown, that the MULTI_THREADED setting leads to far better results, as it allows to execute multiple SQL statements in parallel. This is particularly useful when executing a long running statement in the background, while still keeping the UI responsive. EA3 is now the first release with MULTI_THREADED enabled. As a result, I hope you see a more responsive, more fun application. For those of you with small libraries (less than 10,000 songs), the difference may not be noticeable at all.

While investigating optimal database settings, I also spent some quality time with certain key SQL statements in the application and was able to tune some of them, essentially removing a bunch of SQL joins. As a result, beaTunes should start up much faster, especially for those people with ten thousands of songs. Additionally, switching between different playlists should be a whole lot faster.

On the algorithm front not a whole lot has changed. Memory consumption for color calculation should be a little lower though, as I have rewritten some classes to have fewer or smaller members.

As always, there is some more info in the NOTES.txt file.

Alright, here's the obligatory warning: Before downloading and installing this, please make sure you understand what Early Access means:

  • Absolutely no warranty for whatever
  • Features may or may not work, appear, and disappear
  • It may not be possible to migrate data to future or previous versions (even though I make a reasonable effort)
  • This version will cease to function three weeks after its release
  • You cannot buy this version

And here are the download links:

Important

For those still using beaTunes 2: beaTunes 3.x changes beaTunes' internal database layout quite a bit, which takes a while. Do not interrupt this process, even if it takes very long! beaTunes 2.x will not be able to use the database anymore, after you ran beaTunes 3.x. If you intend to try 3.x and then go back to 2.x, make sure you back up the database before you try 3.x (depending on your OS it's in ~/Library/Application Support/beaTunes/Database, C:\Users\<username>\AppData\Local\tagtraum industries\beaTunes\database or C:\Documents and Settings\<username>\Local Settings\Application Data\tagtraum industries\beaTunes\database)! You also might want to check out this article about exporting beaTunes data.

Feedback

Please provide feedback! Let me know what does not work. Especially audio decoding problems, OutOfMemoryErrors and UI hangs and hiccups.

Thanks.

PS: The update from 3.0 to 3.5 will be free.

Labels:

Thursday, December 15, 2011

beaTunes 3.5 Early Access 2: Lowering Memory Consumption

beaTunes2 logoSince it's only been a week, there aren't too many changes. However, some of them are crucial, especially for those of you having huge collections. To make it short: EA2 contains a couple of fixes and a number of performance improvements.

As always, there is some more info in the NOTES.txt file.

Alright, here's the obligatory warning: Before downloading and installing this, please make sure you understand what Early Access means:

  • Absolutely no warranty for whatever
  • Features may or may not work, appear, and disappear
  • It may not be possible to migrate data to future or previous versions (even though I make a reasonable effort)
  • This version will cease to function one week after its release
  • You cannot buy this version

And here are the download links:

Important

For those still using beaTunes 2: beaTunes 3.x changes beaTunes' internal database layout quite a bit, which takes a while. Do not interrupt this process, even if it takes very long! beaTunes 2.x will not be able to use the database anymore, after you ran beaTunes 3.x. If you intend to try 3.x and then go back to 2.x, make sure you back up the database before you try 3.x (depending on your OS it's in ~/Library/Application Support/beaTunes/Database, C:\Users\<username>\AppData\Local\tagtraum industries\beaTunes\database or C:\Documents and Settings\<username>\Local Settings\Application Data\tagtraum industries\beaTunes\database)! You also might want to check out this article about exporting beaTunes data.

Feedback

Please provide feedback! Let me know what does not work. Especially audio decoding problems, OutOfMemoryErrors and UI hangs and hiccups.

Thanks.

PS: The update from 3.0 to 3.5 will be free.

Labels:

Friday, December 9, 2011

beaTunes 3.5 Early Access 1: Faster, more responsive

beaTunes2 logoSurprising to some, I have not been idle in the past couple of months since the first release of beaTunes 3. In preparation of beaTunes 3.5, I have spent a lot of time on improving Jipes and the SampledSP collection. Both are open source libraries and central to beaTunes 3.x. Let me explain why..

Traditionally, Java offers very poor multimedia support. To illustrate this: In Java you can't just load and play an AAC file. Instead you have to find or write some code that does it for you. Support out of the box is not available. The same is true for all but the most simple (PCM) or obscure (μ-law) encodings.

The SampledSP collection remedies this situation by providing a bridge to native audio libraries like Apple QuickTime (32bit), Apple CoreAudio (OS X) and Microsoft MediaFoundation (>= Windows 7). Furthermore, it provides basic resampling capabilities.

beaTunes 3.5 will take full advantage of these libraries. It will also better take advantage of Jipes. Here's how: To analyze, all versions of beaTunes prior 3.5 first decoded the audio file to memory. Then, one after another, each requested algorithm would operate on the decoded audio data. While this approach is very efficient for multipass algorithms (decode once, process many times), it is flawed, when you want to process many files in parallel, as decoded audio can easily be 50MB and bigger. And once you have used lots of memory, the system becomes less responsive, because the OS is busy with memory management... not a pleasant situation. Starting with version 3.5, beaTunes processes small chunks of audio at a time, drastically reducing peak memory consumption, thus easing the strain on the OS. At the same time, processing steps that many algorithms have in common are only executed once, thus saving some work. The magical framework that makes this possible is Jipes.

To make this all work in beaTunes 3.5, I had to rewrite some of the existing algorithms and the task handling infrastructure. Most plugin APIs stayed unchanged, though. Additionally, the bundled JRuby and Groovy scripting engines were updated to current versions.

Better Responsiveness

While the audio processing stuff is probably the biggest change under the hood, I have also spent some time on other, more immediate performance aspects. One of them being UI responsiveness. When playing with 3.5, you will notice that scrolling around in playlists is a lot smoother than it used to be. There may be a slight flicker when beaTunes has to load data from the database, but typically values should appear in less than 0.5s. In fact, I have not seen this take more than 0.25s. Another area of work has been general responsiveness during standard tasks, like adding analysis tasks to the queue. Here, you should also notice an improvement.

Last but not least, I rewrote much of the online resources interaction code. Before 3.5, beaTunes first sent all data to the server, processed all the results and then added the unsuccessful tasks to the queue for local processing. Considering, that server processing takes a long time, but not a whole lot of resources, while local processing takes a lot of resources and a lot of time, this is a poor approach. In version 3.5 beaTunes sends data to the server (indicated by a message in the status bar of the task queue). In parallel, all tasks are added to the queue. But unlike before, beaTunes does not wait for all results from the online database to be processed before processing the remaining tasks locally. Instead, a task in the queue is processed locally right after we received data from the server for the task at hand (if at that point it is still necessary at all). In essence, both ways of processing (with online resources and locally) are more or less parallelized, thus significantly reducing the total processing time.

One last but important remark about colors: beaTunes 3.5 calculates colors slightly differently. If you have a big collection (i.e. more than 10,000 songs), this can slow down analysis significantly, as many colors (not just for the song you added to the queue) have to be changed in the background. Also, colors calculated with prior versions will have a different meaning, thus similarity can only be computed correctly, after all colors have been re-calculated.

As always, there is some more info in the NOTES.txt file.

Alright, here's the obligatory warning: Before downloading and installing this, please make sure you understand what Early Access means:

  • Absolutely no warranty for whatever
  • Features may or may not work, appear, and disappear
  • It may not be possible to migrate data to future or previous versions (even though I make a reasonable effort)
  • This version will cease to function one week after its release
  • You cannot buy this version

And here are the download links:

Important

For those still using beaTunes 2: beaTunes 3.x changes beaTunes' internal database layout quite a bit, which takes a while. Do not interrupt this process, even if it takes very long! beaTunes 2.x will not be able to use the database anymore, after you ran beaTunes 3.x. If you intend to try 3.x and then go back to 2.x, make sure you back up the database before you try 3.x (depending on your OS it's in ~/Library/Application Support/beaTunes/Database, C:\Users\<username>\AppData\Local\tagtraum industries\beaTunes\database or C:\Documents and Settings\<username>\Local Settings\Application Data\tagtraum industries\beaTunes\database)! You also might want to check out this article about exporting beaTunes data.

Feedback

Please provide feedback! Let me know what does not work. Especially audio decoding problems, OutOfMemoryErrors and UI hangs and hiccups.

Thanks.

PS: The update from 3.0 to 3.5 will be free.

Labels: , , , ,