How to get a low latency audio in Android4.4 ?

[e***@gmail.com]

I saw Glenn Kasten's video in YouTube about android5.0 audio latency
improvement, that’s an amazing short latency!

Recently, I met an issue like this: Audio latency is very serious at
our device. Use openSLES, there’s about 200+ms delay.

So I wonder how to gain a low latency audio in Android4.4 ?

----------------------------------------------------------
My idea as below:

There're three buffers: 1. share buffer used by client(AudioTrack) and
server(AudioFlinger); 2. mainBuffer for audio mix; 3. buffer in alsa
driver

So I think low latency can be get by reducing size of these three buffers.
XRUN would happen if buffer size is too small, so there must be a balance
value.


-----------------------------------------------------------
Now my method as below:

1. Set period size and count in HAL;

Output: 256 * 8

Input256 * 6

2. Set kMinNormalMixBufferSizeMs and kMaxNormalMixBufferSizeMs to 5
and 6 respectively in Threads.cpp;

3. Set minBufCount = 1 in both PlaybackThread::createTrack_l and
RecordThread::createRecordTrack_l;

4. Don't double input buffer in AudioRecord::getMinFrameCount;


-----------------------------------------------------------
My local test result:

Local loopback test: direct HAL read and write.

stream read/write, I open a direct API in AudioFlinger to invoke
PlaybackThread->mOutputSink->write; and
RecordThread->getInputStream()->stream->read.

In theory, HAL read/write should not be longer than pcm read write, I
don’t know why it cost 30ms more. Maybe by Vendor’s resample in HAL
and IPC, need more investigation.

57 ~ 60ms delay.


Local loopback test: direct pcm device read and write. DO NOT use
android API, operate on alsa pcm device directly.

20 ~ 30ms delay.


Test method:

Use laptop computer to record sound before enter Mic and out from
speaker, then analysis pcm wave by CoolEditor.



Thank you so much for your help!


Evan.Xu

[Glenn Kasten]

Thanks for watching the videos.

To get lower latency, it takes a partnership of the app developer, Android
framework, OEM and SoC vendor (including audio HAL),
kernel, and testing. Here are a few of the key things for each part.

app:
Use Android native audio based on OpenSL ES.
Check for feature android.hardware.audio.low_latency
see
http://developer.android.com/guide/topics/manifest/uses-feature-element.html
Use the recommended native buffer size and sample rate;
see http://developer.android.com/reference/android/media/AudioManager.html#getProperty(java.lang.String)
[the same buffer size and sample rate should be used for input also]
Usually an OpenSL ES buffer count of 1 is sufficient.
Keep your callback handlers short, without bursty CPU usage,
and no unbounded blocking. Avoid priority inversion.
See https://source.android.com/devices/audio/avoiding_pi.html
You may want to consider using a non-blocking library
to communicate between input and output callback handlers.
See https://source.android.com/devices/audio/avoiding_pi.html#nonBlockingAlgorithms

framework:
We've been gradually improving the framework over successive releases.
This article describes the initial design:
https://source.android.com/devices/audio/latency_design.html
The article needs to be updated to describe more recent improvements
(especially on input side). I'm making a note to self to update that.
Generally, newer releases
can offer lower latency than older releases on the same device.
We expect to continue to make more improvements in future,
but I can't give a schedule.

OEM and SoC vendor, including audio HAL:
You mentioned 6 or 8 buffers in HAL. I find that usually 2 buffers is
enough.
Avoid priority inversion in the HAL. Pay attention especially to mutexes.
Avoid or reduce extra audio processing in HAL such as signal "enhancements"
for lower latency use cases.
If there is a DSP after the app processor, try to keep any processing
by the DSP minimal.
I noticed you made some changes to the portable audio framework.
Those changes should not be necessary if you do the other suggestions
here.

Kernel:
Reduce worst-case interrupt disable and preemption disable times.
Power management can sometimes interfere with scheduling.
Check if your DRM security kernel has any long dark times
during which the regular kernel is unable to run.

Testing:
Here are links to the test techniques and tools that I've used:
https://source.android.com/devices/audio/latency_measure.html
https://source.android.com/devices/audio/loopback.html
https://source.android.com/devices/audio/testing_circuit.html

For an overview of latency issues, see
https://source.android.com/devices/audio/latency.html

Post by e***@gmail.com
I saw Glenn Kasten's video in YouTube about android5.0 audio latency
improvement, that’s an amazing short latency!
Recently, I met an issue like this: Audio latency is very serious at
our device. Use openSLES, there’s about 200+ms delay.
So I wonder how to gain a low latency audio in Android4.4 ?
----------------------------------------------------------
There're three buffers: 1. share buffer used by client(AudioTrack) and
server(AudioFlinger); 2. mainBuffer for audio mix; 3. buffer in alsa
driver
So I think low latency can be get by reducing size of these three buffers.
XRUN would happen if buffer size is too small, so there must be a balance
value.
-----------------------------------------------------------
1. Set period size and count in HAL;
Output: 256 * 8
Input256 * 6
2. Set kMinNormalMixBufferSizeMs and kMaxNormalMixBufferSizeMs to 5
and 6 respectively in Threads.cpp;
3. Set minBufCount = 1 in both PlaybackThread::createTrack_l and
RecordThread::createRecordTrack_l;
4. Don't double input buffer in AudioRecord::getMinFrameCount;
-----------------------------------------------------------
Local loopback test: direct HAL read and write.
stream read/write, I open a direct API in AudioFlinger to invoke
PlaybackThread->mOutputSink->write; and
RecordThread->getInputStream()->stream->read.
In theory, HAL read/write should not be longer than pcm read write, I
don’t know why it cost 30ms more. Maybe by Vendor’s resample in HAL
and IPC, need more investigation.
57 ~ 60ms delay.
Local loopback test: direct pcm device read and write. DO NOT use
android API, operate on alsa pcm device directly.
20 ~ 30ms delay.
Use laptop computer to record sound before enter Mic and out from
speaker, then analysis pcm wave by CoolEditor.
Thank you so much for your help!
Evan.Xu

[e***@gmail.com]

Hi Glenn

Thank your so much for your reply!

圚 2015幎3月2日星期䞀 UTC+8䞊午11:23:34Glenn Kasten写道

Post by Glenn Kasten
Thanks for watching the videos.
To get lower latency, it takes a partnership of the app developer, Android
framework, OEM and SoC vendor (including audio HAL),
kernel, and testing. Here are a few of the key things for each part.
Use Android native audio based on OpenSL ES.
Check for feature android.hardware.audio.low_latency
see
http://developer.android.com/guide/topics/manifest/uses-feature-element.html
Use the recommended native buffer size and sample rate;
see
http://developer.android.com/reference/android/media/AudioManager.html#getProperty(java.lang.String)
[the same buffer size and sample rate should be used for input also]
Usually an OpenSL ES buffer count of 1 is sufficient.
Keep your callback handlers short, without bursty CPU usage,
and no unbounded blocking. Avoid priority inversion.
See https://source.android.com/devices/audio/avoiding_pi.html
You may want to consider using a non-blocking library
to communicate between input and output callback handlers. See
https://source.android.com/devices/audio/avoiding_pi.html#nonBlockingAlgorithms
We've been gradually improving the framework over successive releases.
https://source.android.com/devices/audio/latency_design.html
The article needs to be updated to describe more recent improvements
(especially on input side). I'm making a note to self to update that.
Generally, newer releases
can offer lower latency than older releases on the same device.
We expect to continue to make more improvements in future,
but I can't give a schedule.
You mentioned 6 or 8 buffers in HAL. I find that usually 2 buffers is
enough.
Avoid priority inversion in the HAL. Pay attention especially to mutexes.
Avoid or reduce extra audio processing in HAL such as signal "enhancements"
for lower latency use cases.
If there is a DSP after the app processor, try to keep any processing
by the DSP minimal.
I noticed you made some changes to the portable audio framework.
Those changes should not be necessary if you do the other suggestions
here.
Reduce worst-case interrupt disable and preemption disable times.
Power management can sometimes interfere with scheduling.
Check if your DRM security kernel has any long dark times
during which the regular kernel is unable to run.
https://source.android.com/devices/audio/latency_measure.html
https://source.android.com/devices/audio/loopback.html
https://source.android.com/devices/audio/testing_circuit.html
For an overview of latency issues, see
https://source.android.com/devices/audio/latency.html

Post by e***@gmail.com
I saw Glenn Kasten's video in YouTube about android5.0 audio latency
improvement, that’s an amazing short latency!
Recently, I met an issue like this: Audio latency is very serious at
our device. Use openSLES, there’s about 200+ms delay.
So I wonder how to gain a low latency audio in Android4.4 ?
----------------------------------------------------------
There're three buffers: 1. share buffer used by client(AudioTrack) and
server(AudioFlinger); 2. mainBuffer for audio mix; 3. buffer in alsa
driver
So I think low latency can be get by reducing size of these three
buffers. XRUN would happen if buffer size is too small, so there must be a
balance value.
-----------------------------------------------------------
1. Set period size and count in HAL;
Output: 256 * 8
Input256 * 6
2. Set kMinNormalMixBufferSizeMs and kMaxNormalMixBufferSizeMs to 5
and 6 respectively in Threads.cpp;
3. Set minBufCount = 1 in both PlaybackThread::createTrack_l and
RecordThread::createRecordTrack_l;
4. Don't double input buffer in AudioRecord::getMinFrameCount;
-----------------------------------------------------------
Local loopback test: direct HAL read and write.
stream read/write, I open a direct API in AudioFlinger to invoke
PlaybackThread->mOutputSink->write; and
RecordThread->getInputStream()->stream->read.
In theory, HAL read/write should not be longer than pcm read write, I
don’t know why it cost 30ms more. Maybe by Vendor’s resample in HAL
and IPC, need more investigation.
57 ~ 60ms delay.
Local loopback test: direct pcm device read and write. DO NOT use
android API, operate on alsa pcm device directly.
20 ~ 30ms delay.
Use laptop computer to record sound before enter Mic and out from
speaker, then analysis pcm wave by CoolEditor.
Thank you so much for your help!
Evan.Xu

[Glenn Kasten]

In my March 1 post, I neglected to provide a link to one important
resource, the NDK documentation:
after installing the NDK, see
<NDKroot>/docs/Additional_library_docs/opensles/index.html
especiall section "Performance". This section has some useful information
about latency reduction techniques.

Post by e***@gmail.com
Hi Glenn
Thank your so much for your reply!
圚 2015幎3月2日星期䞀 UTC+8䞊午11:23:34Glenn Kasten写道

Post by Glenn Kasten
Thanks for watching the videos.
To get lower latency, it takes a partnership of the app developer,
Android framework, OEM and SoC vendor (including audio HAL),
kernel, and testing. Here are a few of the key things for each part.
Use Android native audio based on OpenSL ES.
Check for feature android.hardware.audio.low_latency
see
http://developer.android.com/guide/topics/manifest/uses-feature-element.html
Use the recommended native buffer size and sample rate;
see
http://developer.android.com/reference/android/media/AudioManager.html#getProperty(java.lang.String)
[the same buffer size and sample rate should be used for input also]
Usually an OpenSL ES buffer count of 1 is sufficient.
Keep your callback handlers short, without bursty CPU usage,
and no unbounded blocking. Avoid priority inversion.
See https://source.android.com/devices/audio/avoiding_pi.html
You may want to consider using a non-blocking library
to communicate between input and output callback handlers. See
https://source.android.com/devices/audio/avoiding_pi.html#nonBlockingAlgorithms
We've been gradually improving the framework over successive releases.
https://source.android.com/devices/audio/latency_design.html
The article needs to be updated to describe more recent improvements
(especially on input side). I'm making a note to self to update that.
Generally, newer releases
can offer lower latency than older releases on the same device.
We expect to continue to make more improvements in future,
but I can't give a schedule.
You mentioned 6 or 8 buffers in HAL. I find that usually 2 buffers is
enough.
Avoid priority inversion in the HAL. Pay attention especially to mutexes.
Avoid or reduce extra audio processing in HAL such as signal
"enhancements"
for lower latency use cases.
If there is a DSP after the app processor, try to keep any processing
by the DSP minimal.
I noticed you made some changes to the portable audio framework.
Those changes should not be necessary if you do the other suggestions
here.
Reduce worst-case interrupt disable and preemption disable times.
Power management can sometimes interfere with scheduling.
Check if your DRM security kernel has any long dark times
during which the regular kernel is unable to run.
https://source.android.com/devices/audio/latency_measure.html
https://source.android.com/devices/audio/loopback.html
https://source.android.com/devices/audio/testing_circuit.html
For an overview of latency issues, see
https://source.android.com/devices/audio/latency.html

Post by e***@gmail.com
I saw Glenn Kasten's video in YouTube about android5.0 audio latency
improvement, that’s an amazing short latency!
Recently, I met an issue like this: Audio latency is very serious at
our device. Use openSLES, there’s about 200+ms delay.
So I wonder how to gain a low latency audio in Android4.4 ?
----------------------------------------------------------
There're three buffers: 1. share buffer used by client(AudioTrack) and
server(AudioFlinger); 2. mainBuffer for audio mix; 3. buffer in alsa
driver
So I think low latency can be get by reducing size of these three
buffers. XRUN would happen if buffer size is too small, so there must be a
balance value.
-----------------------------------------------------------
1. Set period size and count in HAL;
Output: 256 * 8
Input256 * 6
2. Set kMinNormalMixBufferSizeMs and kMaxNormalMixBufferSizeMs to 5
and 6 respectively in Threads.cpp;
3. Set minBufCount = 1 in both PlaybackThread::createTrack_l and
RecordThread::createRecordTrack_l;
4. Don't double input buffer in AudioRecord::getMinFrameCount;
-----------------------------------------------------------
Local loopback test: direct HAL read and write.
stream read/write, I open a direct API in AudioFlinger to invoke
PlaybackThread->mOutputSink->write; and
RecordThread->getInputStream()->stream->read.
In theory, HAL read/write should not be longer than pcm read write, I
don’t know why it cost 30ms more. Maybe by Vendor’s resample in HAL
and IPC, need more investigation.
57 ~ 60ms delay.
Local loopback test: direct pcm device read and write. DO NOT use
android API, operate on alsa pcm device directly.
20 ~ 30ms delay.
Use laptop computer to record sound before enter Mic and out from
speaker, then analysis pcm wave by CoolEditor.
Thank you so much for your help!
Evan.Xu

[Glenn Kasten]

More resources on audio latency:

- video "audio latency: buffer sizes"
at

- article "Audio Latency for App Developers"
at https://source.android.com/devices/audio/latency_app.html

Post by Glenn Kasten
In my March 1 post, I neglected to provide a link to one important
after installing the NDK, see
<NDKroot>/docs/Additional_library_docs/opensles/index.html
especiall section "Performance". This section has some useful information
about latency reduction techniques.

Post by e***@gmail.com
Hi Glenn
Thank your so much for your reply!
圚 2015幎3月2日星期䞀 UTC+8䞊午11:23:34Glenn Kasten写道

Post by Glenn Kasten
Thanks for watching the videos.
To get lower latency, it takes a partnership of the app developer,
Android framework, OEM and SoC vendor (including audio HAL),
kernel, and testing. Here are a few of the key things for each part.
Use Android native audio based on OpenSL ES.
Check for feature android.hardware.audio.low_latency
see
http://developer.android.com/guide/topics/manifest/uses-feature-element.html
Use the recommended native buffer size and sample rate;
see
http://developer.android.com/reference/android/media/AudioManager.html#getProperty(java.lang.String)
[the same buffer size and sample rate should be used for input also]
Usually an OpenSL ES buffer count of 1 is sufficient.
Keep your callback handlers short, without bursty CPU usage,
and no unbounded blocking. Avoid priority inversion.
See https://source.android.com/devices/audio/avoiding_pi.html
You may want to consider using a non-blocking library
to communicate between input and output callback handlers. See
https://source.android.com/devices/audio/avoiding_pi.html#nonBlockingAlgorithms
We've been gradually improving the framework over successive releases.
https://source.android.com/devices/audio/latency_design.html
The article needs to be updated to describe more recent improvements
(especially on input side). I'm making a note to self to update that.
Generally, newer releases
can offer lower latency than older releases on the same device.
We expect to continue to make more improvements in future,
but I can't give a schedule.
You mentioned 6 or 8 buffers in HAL. I find that usually 2 buffers is
enough.
Avoid priority inversion in the HAL. Pay attention especially to mutexes.
Avoid or reduce extra audio processing in HAL such as signal
"enhancements"
for lower latency use cases.
If there is a DSP after the app processor, try to keep any processing
by the DSP minimal.
I noticed you made some changes to the portable audio framework.
Those changes should not be necessary if you do the other suggestions
here.
Reduce worst-case interrupt disable and preemption disable times.
Power management can sometimes interfere with scheduling.
Check if your DRM security kernel has any long dark times
during which the regular kernel is unable to run.
https://source.android.com/devices/audio/latency_measure.html
https://source.android.com/devices/audio/loopback.html
https://source.android.com/devices/audio/testing_circuit.html
For an overview of latency issues, see
https://source.android.com/devices/audio/latency.html

Post by e***@gmail.com
I saw Glenn Kasten's video in YouTube about android5.0 audio latency
improvement, that’s an amazing short latency!
Recently, I met an issue like this: Audio latency is very serious at
our device. Use openSLES, there’s about 200+ms delay.
So I wonder how to gain a low latency audio in Android4.4 ?
----------------------------------------------------------
There're three buffers: 1. share buffer used by client(AudioTrack) and
server(AudioFlinger); 2. mainBuffer for audio mix; 3. buffer in alsa
driver
So I think low latency can be get by reducing size of these three
buffers. XRUN would happen if buffer size is too small, so there must be a
balance value.
-----------------------------------------------------------
1. Set period size and count in HAL;
Output: 256 * 8
Input256 * 6
2. Set kMinNormalMixBufferSizeMs and kMaxNormalMixBufferSizeMs to 5
and 6 respectively in Threads.cpp;
3. Set minBufCount = 1 in both PlaybackThread::createTrack_l and
RecordThread::createRecordTrack_l;
4. Don't double input buffer in AudioRecord::getMinFrameCount;
-----------------------------------------------------------
Local loopback test: direct HAL read and write.
stream read/write, I open a direct API in AudioFlinger to invoke
PlaybackThread->mOutputSink->write; and
RecordThread->getInputStream()->stream->read.
In theory, HAL read/write should not be longer than pcm read write, I
don’t know why it cost 30ms more. Maybe by Vendor’s resample in HAL
and IPC, need more investigation.
57 ~ 60ms delay.
Local loopback test: direct pcm device read and write. DO NOT use
android API, operate on alsa pcm device directly.
20 ~ 30ms delay.
Use laptop computer to record sound before enter Mic and out from
speaker, then analysis pcm wave by CoolEditor.
Thank you so much for your help!
Evan.Xu

[Anton Persson]

Hi Glenn,

I know this isn't probably on your table, so to speak, but wouldn't it be
good if Google put a bit more pressure on companies like - hmm - Samsung,
to support this better? I just implemented support for Samsung's
"Professional Audio SDK" in my app (vuKNOB on play store) and noticed that
the latency using that is much better on a Galaxy S6 than what I get with
the OpenSL ES APIs on the same device. So the platform on that device
clearly can achieve fair latency but it's not available using standard
Android APIs.

Best Regards
Anton

Post by Glenn Kasten
- video "audio latency: buffer sizes"
at http://youtu.be/PnDK17zP9BI
- article "Audio Latency for App Developers"
at https://source.android.com/devices/audio/latency_app.html

Post by Glenn Kasten
In my March 1 post, I neglected to provide a link to one important
after installing the NDK, see
<NDKroot>/docs/Additional_library_docs/opensles/index.html
especiall section "Performance". This section has some useful
information about latency reduction techniques.

Post by e***@gmail.com
Hi Glenn
Thank your so much for your reply!
圚 2015幎3月2日星期䞀 UTC+8䞊午11:23:34Glenn Kasten写道

Post by Glenn Kasten
Thanks for watching the videos.
To get lower latency, it takes a partnership of the app developer,
Android framework, OEM and SoC vendor (including audio HAL),
kernel, and testing. Here are a few of the key things for each part.
Use Android native audio based on OpenSL ES.
Check for feature android.hardware.audio.low_latency
see
http://developer.android.com/guide/topics/manifest/uses-feature-element.html
Use the recommended native buffer size and sample rate;
see
http://developer.android.com/reference/android/media/AudioManager.html#getProperty(java.lang.String)
[the same buffer size and sample rate should be used for input also]
Usually an OpenSL ES buffer count of 1 is sufficient.
Keep your callback handlers short, without bursty CPU usage,
and no unbounded blocking. Avoid priority inversion.
See https://source.android.com/devices/audio/avoiding_pi.html
You may want to consider using a non-blocking library
to communicate between input and output callback handlers. See
https://source.android.com/devices/audio/avoiding_pi.html#nonBlockingAlgorithms
We've been gradually improving the framework over successive releases.
https://source.android.com/devices/audio/latency_design.html
The article needs to be updated to describe more recent improvements
(especially on input side). I'm making a note to self to update that.
Generally, newer releases
can offer lower latency than older releases on the same device.
We expect to continue to make more improvements in future,
but I can't give a schedule.
You mentioned 6 or 8 buffers in HAL. I find that usually 2 buffers is
enough.
Avoid priority inversion in the HAL. Pay attention especially to mutexes.
Avoid or reduce extra audio processing in HAL such as signal "enhancements"
for lower latency use cases.
If there is a DSP after the app processor, try to keep any processing
by the DSP minimal.
I noticed you made some changes to the portable audio framework.
Those changes should not be necessary if you do the other suggestions
here.
Reduce worst-case interrupt disable and preemption disable times.
Power management can sometimes interfere with scheduling.
Check if your DRM security kernel has any long dark times
during which the regular kernel is unable to run.
https://source.android.com/devices/audio/latency_measure.html
https://source.android.com/devices/audio/loopback.html
https://source.android.com/devices/audio/testing_circuit.html
For an overview of latency issues, see
https://source.android.com/devices/audio/latency.html

Post by e***@gmail.com
I saw Glenn Kasten's video in YouTube about android5.0 audio latency
improvement, that’s an amazing short latency!
Recently, I met an issue like this: Audio latency is very serious at
our device. Use openSLES, there’s about 200+ms delay.
So I wonder how to gain a low latency audio in Android4.4 ?
----------------------------------------------------------
There're three buffers: 1. share buffer used by client(AudioTrack) and
server(AudioFlinger); 2. mainBuffer for audio mix; 3. buffer in alsa
driver
So I think low latency can be get by reducing size of these three
buffers. XRUN would happen if buffer size is too small, so there must be a
balance value.
-----------------------------------------------------------
1. Set period size and count in HAL;
Output: 256 * 8
Input256 * 6
2. Set kMinNormalMixBufferSizeMs and kMaxNormalMixBufferSizeMs to 5
and 6 respectively in Threads.cpp;
3. Set minBufCount = 1 in both PlaybackThread::createTrack_l and
RecordThread::createRecordTrack_l;
4. Don't double input buffer in AudioRecord::getMinFrameCount;
-----------------------------------------------------------
Local loopback test: direct HAL read and write.
stream read/write, I open a direct API in AudioFlinger to invoke
PlaybackThread->mOutputSink->write; and
RecordThread->getInputStream()->stream->read.
In theory, HAL read/write should not be longer than pcm read write, I
don’t know why it cost 30ms more. Maybe by Vendor’s resample in HAL
and IPC, need more investigation.
57 ~ 60ms delay.
Local loopback test: direct pcm device read and write. DO NOT use
android API, operate on alsa pcm device directly.
20 ~ 30ms delay.
Use laptop computer to record sound before enter Mic and out from
speaker, then analysis pcm wave by CoolEditor.
Thank you so much for your help!
Evan.Xu

--

You received this message because you are subscribed to the Google Groups
"android-ndk" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at http://groups.google.com/group/android-ndk.
For more options, visit https://groups.google.com/d/optout.

[Felix Homann]

Hi Anton,

but wouldn't it be good if Google put a bit more pressure on companies
like - hmm - Samsung, to support this better? I [...] noticed that the
latency using that [i.e. Samsung's "Professional Audio SDK"] is much better
on a Galaxy S6 than what I get with the OpenSL ES APIs on the same device.

How should they put pressure on Samsung? Samsung's "Professional Audio SDK"

- *does* provide low latency audio (at least better latency, haven't
tested; no Samsung devices here except for Galaxy Nexus and Nexus 10),
- it *does* provide familiar and easy(!) to use APIs (JACK)
- it *does* provide routing between different apps through JACK APIs
(See the sticky announcement of Peter Brinkmann's Patchfield on top of this
group? It never gained any momentum. I'm quite sure this is partially
because it doesn't make any sense without low latency and multi channel
audio.)

In all of these areas stock Android is miles behind!

Google itself has to change audio on Android. Not by putting pressure on
others but by just delivering! And I don't think a little tweaking can do.
Android needs a totally different, maybe additional, set of APIs for pro
audio. JACK would be the first choice as Samsung have correctly realized.
(I guess Google decline to use it because it's LGPL and not BSD or Apache).

OpenSL ES on the other hand is a major PITA. So much boiler plate code for
even the simplest things! So limited in what it can actually do! See e.g.
the upcoming multichannel audio support in Android M - it is only available
in the JAVA APIs while it is most urgently needed for *professional* audio
that which have a need for low latency audio. That's insane!

Than there's there's still this limited and unpredictable support fur USB
audio interfaces.

1. Since every Android device uses a different kernel version, support
for USB audio is different on all of them: Some audio interfaces work on
some Android devices but not on others.
2. If your device is supported by the kernel but has some USB mixer
controls that device is generally unusable since those controls are not
exposed in any Android API.

IMO, Android versions should specify the kernel version they use, too, so
that device support - not only for audio devices - is unified!

I personally have given up on low latency audio on Android. I still do
audio on Android a lot but nothing that requires multi channel or low
latency audio anymore. Those applications that need low latency have been
moved to a CubieTruck running Debian that I can control via OSC from my
Android devices. That's much simpler and much better *for me*.

BTW, have you tried to use a USB audio interface on an Android device? In
most cases you have to choose between charging the Android device or using
the audio interface. That's totally unsuitable for *professional* audio!
(*Some* Android devices can be charged via USB-OTG cables with power
supply while using *some* audio interfaces but again it's totally
unpredictable which combinations work and which do not.) This might be
solved with USB 3.1 in future devices though.

Glenn is a very gentle, friendly and helpful person. I'm pretty sure he
feels my (our?) pain and would surely want things to change rather sooner
than later. I am very thankful for all he has already done! Unfortunately
though, Google still does not seem to see the need for professional audio
apps on Android. That's a pity!

[Anton Persson]

Hi Felix,

I fully agree with you. I just don't see why Samsung, who offers both APIs,
can't deliver small buffers both through JACK and OpenSL.

They are both there already...

/Anton

Post by Felix Homann
Hi Anton,

but wouldn't it be good if Google put a bit more pressure on companies
like - hmm - Samsung, to support this better? I [...] noticed that the
latency using that [i.e. Samsung's "Professional Audio SDK"] is much better
on a Galaxy S6 than what I get with the OpenSL ES APIs on the same device.

How should they put pressure on Samsung? Samsung's "Professional Audio SDK"
- *does* provide low latency audio (at least better latency, haven't
tested; no Samsung devices here except for Galaxy Nexus and Nexus 10),
- it *does* provide familiar and easy(!) to use APIs (JACK)
- it *does* provide routing between different apps through JACK APIs
(See the sticky announcement of Peter Brinkmann's Patchfield on top of this
group? It never gained any momentum. I'm quite sure this is partially
because it doesn't make any sense without low latency and multi channel
audio.)
In all of these areas stock Android is miles behind!
Google itself has to change audio on Android. Not by putting pressure on
others but by just delivering! And I don't think a little tweaking can do.
Android needs a totally different, maybe additional, set of APIs for pro
audio. JACK would be the first choice as Samsung have correctly realized.
(I guess Google decline to use it because it's LGPL and not BSD or Apache).
OpenSL ES on the other hand is a major PITA. So much boiler plate code for
even the simplest things! So limited in what it can actually do! See e.g.
the upcoming multichannel audio support in Android M - it is only available
in the JAVA APIs while it is most urgently needed for *professional* audio
that which have a need for low latency audio. That's insane!
Than there's there's still this limited and unpredictable support fur USB
audio interfaces.
1. Since every Android device uses a different kernel version, support
for USB audio is different on all of them: Some audio interfaces work on
some Android devices but not on others.
2. If your device is supported by the kernel but has some USB mixer
controls that device is generally unusable since those controls are not
exposed in any Android API.
IMO, Android versions should specify the kernel version they use, too, so
that device support - not only for audio devices - is unified!
I personally have given up on low latency audio on Android. I still do
audio on Android a lot but nothing that requires multi channel or low
latency audio anymore. Those applications that need low latency have been
moved to a CubieTruck running Debian that I can control via OSC from my
Android devices. That's much simpler and much better *for me*.
BTW, have you tried to use a USB audio interface on an Android device? In
most cases you have to choose between charging the Android device or using
the audio interface. That's totally unsuitable for *professional* audio!
(*Some* Android devices can be charged via USB-OTG cables with power
supply while using *some* audio interfaces but again it's totally
unpredictable which combinations work and which do not.) This might be
solved with USB 3.1 in future devices though.
Glenn is a very gentle, friendly and helpful person. I'm pretty sure he
feels my (our?) pain and would surely want things to change rather sooner
than later. I am very thankful for all he has already done! Unfortunately
though, Google still does not seem to see the need for professional audio
apps on Android. That's a pity!
--
You received this message because you are subscribed to the Google Groups
"android-ndk" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at http://groups.google.com/group/android-ndk.
For more options, visit https://groups.google.com/d/optout.

[Brad Justice]

I would not expect audio latency suitable for pro audio to come directly
from Google. Their goal is to provide a generic solution suitable for a
wide variety of OEMs. For this market the definition "low latency" is an
order of magnitude greater than that desired for the pro audio market.

Even if you look at successful pro audio platforms such as the Mac, they do
not support this sort of latency out of the box. They work because they are
highly configurable in the field in both hardware and software.

In order to support very low audio latency you need to look beyond the
audio stack. Any piece of hardware, any driver, any OS feature integrated
into the device can cause delays that result in dropouts. In the absence of
field configuration, this needs to be addressed by an OEM who cares about a
niche market, as it is very device specific.

Unfortunately it does not seem to me that Samsung is that OEM. Per
statements by Samsung at their San Francisco developers conferences, they
are running Jack with two 512 sample buffers. At their 48K sample rate this
pushes buffer latency above the 20 ms line. Even at that, my testing on a
Note 3 indicated a significant underrun issue.

Brad Justice
http://proaudioandroid.blogspot.com/

Post by e***@gmail.com
I saw Glenn Kasten's video in YouTube about android5.0 audio latency
improvement, that’s an amazing short latency!
Recently, I met an issue like this: Audio latency is very serious at
our device. Use openSLES, there’s about 200+ms delay.
So I wonder how to gain a low latency audio in Android4.4 ?
----------------------------------------------------------
There're three buffers: 1. share buffer used by client(AudioTrack) and
server(AudioFlinger); 2. mainBuffer for audio mix; 3. buffer in alsa
driver
So I think low latency can be get by reducing size of these three buffers.
XRUN would happen if buffer size is too small, so there must be a balance
value.
-----------------------------------------------------------
1. Set period size and count in HAL;
Output: 256 * 8
Input256 * 6
2. Set kMinNormalMixBufferSizeMs and kMaxNormalMixBufferSizeMs to 5
and 6 respectively in Threads.cpp;
3. Set minBufCount = 1 in both PlaybackThread::createTrack_l and
RecordThread::createRecordTrack_l;
4. Don't double input buffer in AudioRecord::getMinFrameCount;
-----------------------------------------------------------
Local loopback test: direct HAL read and write.
stream read/write, I open a direct API in AudioFlinger to invoke
PlaybackThread->mOutputSink->write; and
RecordThread->getInputStream()->stream->read.
In theory, HAL read/write should not be longer than pcm read write, I
don’t know why it cost 30ms more. Maybe by Vendor’s resample in HAL
and IPC, need more investigation.
57 ~ 60ms delay.
Local loopback test: direct pcm device read and write. DO NOT use
android API, operate on alsa pcm device directly.
20 ~ 30ms delay.
Use laptop computer to record sound before enter Mic and out from
speaker, then analysis pcm wave by CoolEditor.
Thank you so much for your help!
Evan.Xu