Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[pjcast]

So, I thought I'd play around with creating a pitch detection / feedback application. I tried a couple audio APIs today.. Notably, FmodEx and Portaudio.

Of course, Fmod had ease of use going for it. However, I was never able to do what I wanted (record audio from Mic and also loop it back to the Mic). I know (or assume) it can do this.. But was not able to get it working. Though, the pitch detection was sweet in that there was already a demo with just a few lines of code can do that (with its built in FFT engine). But, the inability to playback the audio was a problem.

Hence, I tried Portaudio (I've used in the past for some Voip related apps). I really like the license, and it works on multiple platforms and requires no license per system (like fmod does). The downside, is that it is raw - meaning, there is no real support for games/fft/3d positioning, etc. But, with one of their test examples, I was able to get what I wanted (minus the pitch detection) - audio in and out from the Mic. At first I was a bit disappointed because the latency (between 100ms - 200ms) - and this is with no processing (using WinMME and DSound backends). Until I downloaded the ASIO4ALL driver for my PC and the ASIO SDK to build ASIO support into Portaudio... And, what I saw after that was a miracle! The extremely noticeable latency of 1/4 or so of a second went down to just about nil (meaning, I heard no latency, but there was probably 10-20ms or so).

Hence, I think portaudio is a good thing to dedicate support to. The fact, though, that it has no 3d position may be a problem. However, implementing such a feature into a software mixer would not be so trivial (and I believe it was the way Doom3 went before creative sued them relating to a shadow technique and forced them to use their API or so I recall). Seems to me that sound doesn't needed to be that advanced (and certainly not as expensive as Fmod makes it) for a game. But, this is coming from someone who mostly uses a laptop with laptop speakers or headphones (so most I ever hear is stereo or pseudo 3d anyway) - and when I do go on my desktop with 5.1 speakers, not many games I have make use of that or do I even notice.

Oh, and I also tried using the ASIO driver with the fmod demos... and, wow, I got some sound playback (not sure if that was intentional on the demo or a bug), but the delay was anywhere from 100ms to 15 seconds! Not sure where that came from, I didn't mess with anything, and it just worked with portaudio.

Anyway, portaudio FTW!

[mysterycoder]

Heh looks very cool! Would you mind posting some of your experimentations?

It really seems like OpenAl is in a funk, with the last release in 2007, but I think the new, open-source OpenAlSoft is promising.
It does the 3d positional effects you mentioned in a software mixer:

Distance attenuation, doppler shift, and directional sound emitters are among the features handled by the API.
More advanced effects, including air absorption, low-pass filters, and reverb, are available through the EFX extension.
It also facilitates streaming audio, multi-channel buffers, and audio capture.

Perhaps it needs a Portaudio backend.

Edit:
I suppose you could give Portaudio a "OpenAl" back-end by utilising some of the already written 3D Positioning code in OpenAlSoft.

To me, sound is a big deal. I really enjoy it when everything else is sealed out and all I can hear is the sound. A lot of games don't make good use of it, but it really enhances the experience if it is beautifully orchestrated and not so generic.

When it's that quiet, lossy compression at low bitrates can get noticeable. I know lossy compression is really good at the higher bit rates, but I really would like to get OggFLAC(.oga) working. I had tried just a while back, but couldn't get it to work.

[pjcast]

Perhaps they could somehow be combined. Portaudio is a very thin wrapper around other APIs (ASIO, DirectSound, ALSA - think, etc). One thing about ASIO, is that while it reduces latency of audio mixing down to almost nothing - no other app can use the sound card at the same time (and, there is a chance ASIO4All driver may fail to work with certain sounds cards - sould work everywhere W2k+ but you never know). The reason why no other app can use the sound card, is that ASIO4ALL driver is bypassing the internal windows software mixer (when enabled by an application) - if other apps were to use the sound card, they need the Windows mixer running - so you have to stop your ASIO app first. A pretty high limitation, but trade off is worth it for near latency free audio (such as real time pitch detection and mic playback to sound card out).

As for using Flac, I'm sure it would be cool for voices, but I would still recommend simply using wav for smaller sound effects and mostly vorbis for streaming audio (though, Flac may be great, you have to show me to make me a believer .

I think via CPU/software, it would be trivial to be just about as good as something as Fmod (of course, not as polished with all the features / tools /platforms) with regards to 5.1+ audio, 3D positioning, et al. That OpenALSoft might be a good thing, I don't know anything about it - hopefully, it is not tied up with the original OpenAL and Creative's stranglehold.

When I have something with pitch detection, I may very well post it. Still playing around. The almost-no latency Mic loopback is accomplished simply by downloading the ASIO SDK + portaudio (and copying over ASIOSDK folder to build location) and building portaudio. That is the only dependency it has. Then, I installed ASIO4ALL Drivers for my laptop. I then took the test/Pa_test_wire.c and made a project from it (they have no projects for their demos)... To forcefully select ASIO device (has to be some way at runtime I'm sure, but I couldn't quite figure it out quickly) I disabled all devices except ASIO when building Portaudio (by setting PA_NO_WMME & PA_NO_DS preprocessor flags).

[pjcast]

Using some examples here and there I've been able to do a bit of pitch detection. Not quite 100% accurate, can fluctuate a bit at certain frequencies (mostly the top end octave). Seems perfectly fine, though, for the normal human vocal range of freqs (80 Hz to 1100 Hz - Wikipedia). Though, I've only mainly tested/verified correct frequency detection against generated sin waves in code. Though, using a guitar against a mic (I can't find my proper jack connectors for PC connection) it gets the pitch pretty much right. There are some little hiccups/fluctuations, but I suppose some of it might just be normal frequency shift/harmonic detections. Also could be clipping.

Anyway, I've been thinking of what framework I might use to try and visualize what I'm looking at better (anything is better than Console out and generating CSV files ). My previous thought was WPF + .Net 3.5 (since I've been using it at work lately and love it). But, that severely limits me to Windows... and greatly increases the size/time to install the app. Though, it is so quick to develop nice GUI/multimedia apps in.

Another choice is .Net + Winforms (and possibly Mogre).. Which, opens up the platforms a little bit more (but, still limited). Install size/time is not significant like 3.5 is. However, then I have to maintain an interop layer to interface with the native code. While, not a difficult task, certainly can consume some time, increase API complexity (maintaining a C interface to C++ backend) - and can introduce some subtle, and yet difficult to find, interop issues.

So, I may go back into the world of just straight C++ That would give me the easiest to maintain API, at the expense of a bit longer of implementation time (especially for GUI related tasks). But at least, I can have an excuse to work with Ogre again. In fact, I may look at using the WGE platform! Though, I have no idea what its current state is.

But, these are things I would want/need to implement:
* Portaudio for input/output
* FFT / Enhanced Auto correlation pitch detection (mostly have this going via some examples) - though, my placement of this directly in the audio callback function is not ideal Would have to queue up bytes to do this in another thread.. Also have to reduce memory allocations/deallocations done for this process.
* A decent/Basic/fast GUI... Last I looked, CEGUI was too much (will have to look at what GUI engine we last talked about for WGE)
* Probably use ffmpeg for all audio/video decoders instead of trying to use pieces of different decoder libs directly (I think ffmpeg supports, Flac, Vorbis, Mp3, et al) - ffmpeg is fast and easily enough to use API (libavcodec). Can be a pain to compile under windows (MinGW), but once you have it compiled you can use it from MSVC. Previously, I've been concerned about ffmpeg because of patented codecs contained within. however, I think as long as I am not encoding, and not distributing thousands of products it will be good enough to disable the most infringing of codecs in the library.


What I'm sort of shooting for is a GHWT / Singstar experience. OIS can handle all the instruments (not mic)... slash vocal trainer application. The one limiting thing about those games is while they give you a slight idea to sing up or down (pitch), they don't tell you what exact note you should be reaching - so, makes it hard to be consistent or really even help as a vocal trainer. Possibly, with a decent backend in place, something else could be made from it as well - I don't know, like some kind of RPG that requires to rock out on guitar, or vocals to destroy a monster .

I'm not ready to post any code for this regards, as I am not certain what I want to do for sure. I also haven't created much, nor is it any kind of usable state.

[mysterycoder]

Just a quick note, the latest version of OpenAL-Soft added a portaudio backend.

[pjcast]

Wow, step away for a bit and someone goes and does the work for you Nice.

[mysterycoder]

Do you still want to add the 3D positioning effects and such and use portaudio. Or, is this just too good to pass up? 

[pjcast]

I think it is better to go with OpenAL Soft... I think I have reconsidered making a voice analysis game. I came across http://littlebigstar.net/main/ which I think is pretty cool (and perhaps one day open). Also, I've come to think that while ASIO is cool for low latency audio (especially for capture and subsequent playback), it is more of a headache than it is worth (doesn't always work with every device, and having users download ASIO drivers, etc).

Though, I would like to work on some kind of small game. I wonder what ideas you might have for a sample game (which could help drive engine development)

[mysterycoder]

I'm not really sure about what kind of game might be best.

I really do like RPGs, and as stated in that PM from a while back, think it would be doable if we split it up into independent chapters in the style of Shigesato Itoi (creator of the mother series).

I think that it would be helpful for what we choose to include an environment with outdoor terrain (to motivate the development of both terrain and paging). Also, though this is probably characteristic of most games, it would be cool to have scripts setup to react to triggers in the environment. For example, if I get within 10 units of a lamp-post, have it turn on.

I don't really know what to do, as all game-types have their drawbacks, and they can't cover all the features, so I'm a little lost direction-wise.
I'm hoping you have some ideas kicking around.