Code coverage like BullseyeCoverage for NDK

[Adrian Vintu]

Hi all,

Is anyone using some kind of code coverage tools for the NDK C++
builds? We were looking for something like BullseyeCoverage but
unfortunately it does not support Android http://www.bullseye.com/platform.html

Has anyone experimented with this?

Thank you,
Best regards,
Adrian

[Nikolai Kondrashov]

Post by Adrian Vintu
Is anyone using some kind of code coverage tools for the NDK C++
builds? We were looking for something like BullseyeCoverage but
unfortunately it does not support Android http://www.bullseye.com/platform.html
Has anyone experimented with this?

We're using gcov/lcov quite successfully. It should support C++, but
we're using it with C only at the moment.

There are a few hacks required, though.

Sincerely,
Nick

-----------------------------------------------------------------------------------
This email message is for the sole use of the intended recipient(s) and may contain
confidential information. Any unauthorized review, use, disclosure or distribution
is prohibited. If you are not the intended recipient, please contact the sender by
reply email and destroy all copies of the original message.
-----------------------------------------------------------------------------------

[Nikolai Kondrashov]

Are the tests running on device?
Can I please get a little bit more info on the hacking required? Maybe
you have some links, articles?

Here is our version of lcov, supporting Android toolchain:
https://github.com/spbnick/lcov-android/ - use "android" branch.

Sincerely,
Nick

-----------------------------------------------------------------------------------
This email message is for the sole use of the intended recipient(s) and may contain
confidential information. Any unauthorized review, use, disclosure or distribution
is prohibited. If you are not the intended recipient, please contact the sender by
reply email and destroy all copies of the original message.
-----------------------------------------------------------------------------------

[Anu Varshini]

Could someone please tell me the steps to implement code coverage (gcov) in
android O.

Post by Adrian Vintu

Post by Adrian Vintu
Is anyone using some kind of code coverage tools for the NDK C++
builds? We were looking for something like BullseyeCoverage but
unfortunately it does not support Android

http://www.bullseye.com/platform.html

Post by Adrian Vintu
Has anyone experimented with this?

We're using gcov/lcov quite successfully. It should support C++, but
we're using it with C only at the moment.
There are a few hacks required, though.
Sincerely,
Nick
-----------------------------------------------------------------------------------
This email message is for the sole use of the intended recipient(s) and may contain
confidential information. Any unauthorized review, use, disclosure or distribution
is prohibited. If you are not the intended recipient, please contact the sender by
reply email and destroy all copies of the original message.
-----------------------------------------------------------------------------------

[Nikolai Kondrashov]

Adrian,

let's share our discussion with the list.

BTW, I just remembered that you need the atexit trick only if you use
dlopen/dlclose.

I think I will push our version of lcov to github soon.

Sincerely,
Nick

-------- Original Message --------
Subject: Re: Code coverage like BullseyeCoverage for NDK
Date: Fri, 8 Jul 2011 13:42:02 +0200

Are the tests running on device?

Sure.

Can I please get a little bit more info on the hacking required? Maybe
you have some links, articles?

There are many articles regarding general gcov and lcov usage. However, I
don't know of any Android-specific articles.

The problem is that Android uses its own toolchain and it has modifications
WRT code coverage instrumentation.

If you plan to get coverage data only for userspace code, then you will only
need to modify lcov a bit.

If you need kernel code coverage data, then you will need kernel patches as
well.

We haven't yet released neither of these, and I don't know if it's possible,
but I'll check. However, lcov fixes are pretty easy to do.

Then there are bugs in gcov instrumentation libraries in some toolchain
versions. And you will have trouble with getting coverage out of shared
libraries, because of bionic handling of __cxa_atexit and __dso_handle, but
this one is solvable by force-linking your own atexit implementation.

If you are still interested, I think I could provide you with at least some
details.

Sincerely,
Nick

-----------------------------------------------------------------------------------
This email message is for the sole use of the intended recipient(s) and may contain
confidential information. Any unauthorized review, use, disclosure or distribution
is prohibited. If you are not the intended recipient, please contact the sender by
reply email and destroy all copies of the original message.
-----------------------------------------------------------------------------------

[Nikolai Kondrashov]

Adrian,

let's share our discussion with the list.

BTW, I just remembered that you need the atexit trick only if you use
dlopen/dlclose.

I think I will push our version of lcov to github soon.

Sincerely,
Nick

-------- Original Message --------
Subject: Re: Code coverage like BullseyeCoverage for NDK
Date: Fri, 8 Jul 2011 13:42:02 +0200

Are the tests running on device?

Sure.

Can I please get a little bit more info on the hacking required? Maybe
you have some links, articles?

There are many articles regarding general gcov and lcov usage. However, I
don't know of any Android-specific articles.

The problem is that Android uses its own toolchain and it has modifications
WRT code coverage instrumentation.

If you plan to get coverage data only for userspace code, then you will only
need to modify lcov a bit.

If you need kernel code coverage data, then you will need kernel patches as
well.

We haven't yet released neither of these, and I don't know if it's possible,
but I'll check. However, lcov fixes are pretty easy to do.

Then there are bugs in gcov instrumentation libraries in some toolchain
versions. And you will have trouble with getting coverage out of shared
libraries, because of bionic handling of __cxa_atexit and __dso_handle, but
this one is solvable by force-linking your own atexit implementation.

If you are still interested, I think I could provide you with at least some
details.

Sincerely,
Nick

-----------------------------------------------------------------------------------
This email message is for the sole use of the intended recipient(s) and may contain
confidential information. Any unauthorized review, use, disclosure or distribution
is prohibited. If you are not the intended recipient, please contact the sender by
reply email and destroy all copies of the original message.
-----------------------------------------------------------------------------------

[Nikolai Kondrashov]

Hi Adrian,

Please, keep the discussion on the list. Use "Reply to all".

I think I need a rooted phone, in order to "sudo make install" the
packages - is this correct?

I'm sorry, I don't know. I work on development boards - they don't need
rooting. And I'm not sure what you mean by "sudo make install".

I mean, I am installing lcov to the phone, right?

You don't need it there, you don't need any tool there. You only need to
build your programs with -fprofile-arcs -ftest-coverage and link with -lgcc
-lgcov.

Where does the compilation take place? On the device or off target? If
it's off target, how do I hook the gcc flags into the Android toolchain?

You can modify your Android.mk's. For example, add this:
LOCAL_CFLAGS += -fprofile-arcs -ftest-coverage
LOCAL_LDFLAGS += -lgcc -lgcov

However, you will need to understand how it works in the end in order to
maintain it.

Sincerely,
Nick

-----------------------------------------------------------------------------------
This email message is for the sole use of the intended recipient(s) and may contain
confidential information. Any unauthorized review, use, disclosure or distribution
is prohibited. If you are not the intended recipient, please contact the sender by
reply email and destroy all copies of the original message.
-----------------------------------------------------------------------------------

[Adrian Vintu]

Hi Nick,

Sorry about not replying to all, I think gmail changed something in their
interface/default behavior.

Good news, I managed to get coverage and it looks splendid :)

One tip:
when setting GCOV_PREFIX_STRIP=1 I got no files.
setting GCOV_PREFIX_STRIP=4 writes the files to
GCOV_PREFIX/trunk/projectA/app/obj/src/...xyz.gcda
setting GCOV_PREFIX_STRIP=123 writes the files to GCOV_PREFIX/xyz.gcda -
i.e. flattens the directory hierarchy

I found the values 4 and 123 by luck...

I have one question for you: I am trying to deploy a shared lib in an apk
and cover it. I do not get any coverage files - also, no exceptions when
building/running.

You mentioned there is some trouble when covering a shared lib. Can you
please detail a little bit?

Thank you for all your help,
Having coverage is just the cherry on top of the cake :)

Best regards,
Adrian



On Tue, Jul 12, 2011 at 2:11 PM, Nikolai Kondrashov

Post by Nikolai Kondrashov
Hi Adrian,
Please, keep the discussion on the list. Use "Reply to all".

I think I need a rooted phone, in order to "sudo make install" the
packages - is this correct?

I'm sorry, I don't know. I work on development boards - they don't need
rooting. And I'm not sure what you mean by "sudo make install".
I mean, I am installing lcov to the phone, right?
You don't need it there, you don't need any tool there. You only need to
build your programs with -fprofile-arcs -ftest-coverage and link with -lgcc
-lgcov.
Where does the compilation take place? On the device or off target? If

it's off target, how do I hook the gcc flags into the Android toolchain?

LOCAL_CFLAGS += -fprofile-arcs -ftest-coverage
LOCAL_LDFLAGS += -lgcc -lgcov
However, you will need to understand how it works in the end in order to
maintain it.
Sincerely,
Nick
------------------------------**------------------------------**
-----------------------
This email message is for the sole use of the intended recipient(s) and may contain
confidential information. Any unauthorized review, use, disclosure or distribution
is prohibited. If you are not the intended recipient, please contact the sender by
reply email and destroy all copies of the original message.
------------------------------**------------------------------**
-----------------------

[Nikolai Kondrashov]

Hi Adrian,

Post by Adrian Vintu
Good news, I managed to get coverage and it looks splendid :)

Great :)!

Post by Adrian Vintu
when setting GCOV_PREFIX_STRIP=1 I got no files.
setting GCOV_PREFIX_STRIP=4 writes the files to
GCOV_PREFIX/trunk/projectA/app/obj/src/...xyz.gcda
setting GCOV_PREFIX_STRIP=123 writes the files to GCOV_PREFIX/xyz.gcda -
i.e. flattens the directory hierarchy
I found the values 4 and 123 by luck...

I'd say you should have just read the documentation instead of relying on
luck :)

Post by Adrian Vintu
I have one question for you: I am trying to deploy a shared lib in an
apk and cover it. I do not get any coverage files - also, no exceptions
when building/running.

Check that the library source code gets compiled with the correct options.
Check that you have gcov symbols in your library by running
"nm library.so | grep gcov".

Post by Adrian Vintu
You mentioned there is some trouble when covering a shared lib. Can you
please detail a little bit?

Unless you use dlopen/dlclose to link it at runtime, while using atexit in
the library at the same time, there should be no problem.

Post by Adrian Vintu
Thank you for all your help,
Having coverage is just the cherry on top of the cake :)

You're welcome :)

Sincerely,
Nick

-----------------------------------------------------------------------------------
This email message is for the sole use of the intended recipient(s) and may contain
confidential information. Any unauthorized review, use, disclosure or distribution
is prohibited. If you are not the intended recipient, please contact the sender by
reply email and destroy all copies of the original message.
-----------------------------------------------------------------------------------

[Adrian Vintu]

Hi Nick,

See my comments in blue below.

As always, many many tanks on your help :)

Best regards,
Adrian

Post by Nikolai Kondrashov
Hi Adrian,

Post by Adrian Vintu
Good news, I managed to get coverage and it looks splendid :)

Great :)!

Post by Adrian Vintu
when setting GCOV_PREFIX_STRIP=1 I got no files.
setting GCOV_PREFIX_STRIP=4 writes the files to
GCOV_PREFIX/trunk/projectA/**app/obj/src/...xyz.gcda
setting GCOV_PREFIX_STRIP=123 writes the files to GCOV_PREFIX/xyz.gcda -
i.e. flattens the directory hierarchy
I found the values 4 and 123 by luck...

I'd say you should have just read the documentation instead of relying on
luck :)

well, the documentation suggests values 1 and 4. 1 did not work for me - I
have no idea why. number 123 is just perfect - it's not in the
documentation. could be that my google skills are failing me, if there is a
good explanation of this flag I would love to have it.

Post by Nikolai Kondrashov
I have one question for you: I am trying to deploy a shared lib in an

Post by Adrian Vintu
apk and cover it. I do not get any coverage files - also, no exceptions
when building/running.

Check that the library source code gets compiled with the correct options.
Check that you have gcov symbols in your library by running
"nm library.so | grep gcov".

I see what you mean. I will give it a go.

Post by Nikolai Kondrashov
You mentioned there is some trouble when covering a shared lib. Can you

Post by Adrian Vintu
please detail a little bit?

Unless you use dlopen/dlclose to link it at runtime, while using atexit in
the library at the same time, there should be no problem.

One lib seems to use dlopen/dlclose and another lib atexit - should not be a
problem I guess.

Post by Nikolai Kondrashov
Thank you for all your help,

Post by Adrian Vintu
Having coverage is just the cherry on top of the cake :)

You're welcome :)
Sincerely,
Nick
------------------------------**------------------------------**
-----------------------
This email message is for the sole use of the intended recipient(s) and may contain
confidential information. Any unauthorized review, use, disclosure or distribution
is prohibited. If you are not the intended recipient, please contact the sender by
reply email and destroy all copies of the original message.
------------------------------**------------------------------**
-----------------------

[Nikolai Kondrashov]

Post by Adrian Vintu

Post by Nikolai Kondrashov
I'd say you should have just read the documentation instead of relying on
luck :)

well, the documentation suggests values 1 and 4. 1 did not work for me -
I have no idea why. number 123 is just perfect - it's not in the
documentation. could be that my google skills are failing me, if there
is a good explanation of this flag I would love to have it.

GCOV_PREFIX and GCOV_PREFIX_STRIP is a typical pair of variables used to
relocate files. GCOV_PREFIX_STRIP is the number of directory components to
be removed from the start of the source file paths and GCOV_PREFIX is a
prefix to be added to the paths.

So, if your DIR_OUT was /home/adrian/android/out, for example and you wish
to put your files under /data/coverage on the target device, you could
define these variables as such:

# Remove four components, i.e. /home/adrian/android/out
export GCOV_PREFIX_STRIP=4
# Prepend /data/coverage
export GCOV_PREFIX=/data/coverage

Thus an object file on host named
/home/adrian/android/out/debug/target/product/blah/obj/EXECUTABLES/gzip_intermediates/minigzip.o
will get the coverage data output to
/data/coverage/debug/target/product/blah/obj/EXECUTABLES/gzip_intermediates/minigzip.gcda
on the device.

Sincerely,
Nick

-----------------------------------------------------------------------------------
This email message is for the sole use of the intended recipient(s) and may contain
confidential information. Any unauthorized review, use, disclosure or distribution
is prohibited. If you are not the intended recipient, please contact the sender by
reply email and destroy all copies of the original message.
-----------------------------------------------------------------------------------

[Adrian Vintu]

Hi Nick,

I am having troubles getting coverage from an apk. I have checked that the
debug/coverage .so files are included, but I cannot get any feedback.

In logcat I have seen that java!? uses dlopen to load the libs. You
mentioned that some tweaking needs to be done in this case. Can you please
detail on that?

Thank you,
Best regards,
Adrian


On Tue, Jul 19, 2011 at 2:44 PM, Nikolai Kondrashov

Post by Nikolai Kondrashov

Post by Nikolai Kondrashov
I'd say you should have just read the documentation instead of

Post by Nikolai Kondrashov
relying on
luck :)

well, the documentation suggests values 1 and 4. 1 did not work for me -
I have no idea why. number 123 is just perfect - it's not in the
documentation. could be that my google skills are failing me, if there
is a good explanation of this flag I would love to have it.

GCOV_PREFIX and GCOV_PREFIX_STRIP is a typical pair of variables used to
relocate files. GCOV_PREFIX_STRIP is the number of directory components to
be removed from the start of the source file paths and GCOV_PREFIX is a
prefix to be added to the paths.
So, if your DIR_OUT was /home/adrian/android/out, for example and you wish
to put your files under /data/coverage on the target device, you could
# Remove four components, i.e. /home/adrian/android/out
export GCOV_PREFIX_STRIP=4
# Prepend /data/coverage
export GCOV_PREFIX=/data/coverage
Thus an object file on host named
/home/adrian/android/out/**debug/target/product/blah/obj/**
EXECUTABLES/gzip_**intermediates/minigzip.o
will get the coverage data output to
/data/coverage/debug/target/**product/blah/obj/EXECUTABLES/**
gzip_intermediates/minigzip.**gcda
on the device.
Sincerely,
Nick
------------------------------**------------------------------**
-----------------------
This email message is for the sole use of the intended recipient(s) and may contain
confidential information. Any unauthorized review, use, disclosure or distribution
is prohibited. If you are not the intended recipient, please contact the sender by
reply email and destroy all copies of the original message.
------------------------------**------------------------------**
-----------------------

[Nikolai Kondrashov]

Post by Adrian Vintu
I am having troubles getting coverage from an apk. I have checked that
the debug/coverage .so files are included, but I cannot get any feedback.
In logcat I have seen that java!? uses dlopen to load the libs. You
mentioned that some tweaking needs to be done in this case. Can you
please detail on that?

The problem is that glibc remembers which library registered which atexit
handler and calls them on appropriate dlclose calls. Libgcov relies on this.
Bionic doesn't register this relation (although it is perfectly capable of
storing and checking it). As a result, gcov atexit handler gets called only
when the main program exits, i.e. when the handler is gone already.

There is a way to workaround the segfault by force-linking atexit
implementation, which behaves similarly to glibc, to all of your shared
libraries.

Like this one:

#include <stddef.h>

extern int __cxa_atexit(void (*func)(void *), void *arg, void *dso);

extern void *__dso_handle;

int
atexit(void (*func)(void))
{
return __cxa_atexit((void (*)(void *))func, NULL, &__dso_handle);
}

Make a library of this and then force-link it to your shared libraries with:

LOCAL_WHOLE_STATIC_LIBRARIES += libatexit

Sincerely,
Nick

-----------------------------------------------------------------------------------
This email message is for the sole use of the intended recipient(s) and may contain
confidential information. Any unauthorized review, use, disclosure or distribution
is prohibited. If you are not the intended recipient, please contact the sender by
reply email and destroy all copies of the original message.
-----------------------------------------------------------------------------------

[Adrian Vintu]

Many thanks Nick, I will try this and let you know on the result.

Best regards,
Adrian


On Mon, Aug 8, 2011 at 3:06 PM, Nikolai Kondrashov

Post by Nikolai Kondrashov

Post by Adrian Vintu
I am having troubles getting coverage from an apk. I have checked that
the debug/coverage .so files are included, but I cannot get any feedback.
In logcat I have seen that java!? uses dlopen to load the libs. You
mentioned that some tweaking needs to be done in this case. Can you
please detail on that?

The problem is that glibc remembers which library registered which atexit
handler and calls them on appropriate dlclose calls. Libgcov relies on this.
Bionic doesn't register this relation (although it is perfectly capable of
storing and checking it). As a result, gcov atexit handler gets called only
when the main program exits, i.e. when the handler is gone already.
There is a way to workaround the segfault by force-linking atexit
implementation, which behaves similarly to glibc, to all of your shared
libraries.
#include <stddef.h>
extern int __cxa_atexit(void (*func)(void *), void *arg, void *dso);
extern void *__dso_handle;
int
atexit(void (*func)(void))
{
return __cxa_atexit((void (*)(void *))func, NULL, &__dso_handle);
}
LOCAL_WHOLE_STATIC_LIBRARIES += libatexit
Sincerely,
Nick
------------------------------**------------------------------**
-----------------------
This email message is for the sole use of the intended recipient(s) and may contain
confidential information. Any unauthorized review, use, disclosure or distribution
is prohibited. If you are not the intended recipient, please contact the sender by
reply email and destroy all copies of the original message.
------------------------------**------------------------------**
-----------------------

[Praveen Jain]

Hi

I was looking for a code coverage tool for Android apps and I foud this
group.
I know it is not for app but atleast if you can point me to right direction.

Please help me with suggestions / tool.
My app has native code also.

Regards
-Praveen

Post by Adrian Vintu
Many thanks Nick, I will try this and let you know on the result.
Best regards,
Adrian

Post by Nikolai Kondrashov

Post by Adrian Vintu
I am having troubles getting coverage from an apk. I have checked that
the debug/coverage .so files are included, but I cannot get any feedback.
In logcat I have seen that java!? uses dlopen to load the libs. You
mentioned that some tweaking needs to be done in this case. Can you
please detail on that?

The problem is that glibc remembers which library registered which atexit
handler and calls them on appropriate dlclose calls. Libgcov relies on this.
Bionic doesn't register this relation (although it is perfectly capable of
storing and checking it). As a result, gcov atexit handler gets called only
when the main program exits, i.e. when the handler is gone already.
There is a way to workaround the segfault by force-linking atexit
implementation, which behaves similarly to glibc, to all of your shared
libraries.
#include <stddef.h>
extern int __cxa_atexit(void (*func)(void *), void *arg, void *dso);
extern void *__dso_handle;
int
atexit(void (*func)(void))
{
return __cxa_atexit((void (*)(void *))func, NULL, &__dso_handle);
}
LOCAL_WHOLE_STATIC_LIBRARIES += libatexit
Sincerely,
Nick
------------------------------**------------------------------**
-----------------------
This email message is for the sole use of the intended recipient(s) and may contain
confidential information. Any unauthorized review, use, disclosure or distribution
is prohibited. If you are not the intended recipient, please contact the sender by
reply email and destroy all copies of the original message.
------------------------------**------------------------------**
-----------------------

[Chris Stratton]

Post by Nikolai Kondrashov
So, if your DIR_OUT was /home/adrian/android/out, for example and you wish
to put your files under /data/coverage on the target device, you could
# Remove four components, i.e. /home/adrian/android/out
export GCOV_PREFIX_STRIP=4
# Prepend /data/coverage
export GCOV_PREFIX=/data/coverage

Jumping in rather late, but thought I'd point out that /data/coverage is not
a path prefix which unprivileged (ie, apk-delivered) code is likely to be
allowed to write to on a real device. Allowable paths would be something
under the app's files directory or the external storage (with appropriate
manifest permission). Since these are slightly device-dependent, it would
probably be necessary to build with approximate knowledge of the specific
target device.

[Nikolai Kondrashov]

Post by Chris Stratton
Jumping in rather late, but thought I'd point out that /data/coverage is
not a path prefix which unprivileged (ie, apk-delivered) code is likely
to be allowed to write to on a real device.

Thanks, Chris. I didn't have the chance to develop apps for Android yet.
Sorry if there is any confusion.

Sincerely,
Nick

-----------------------------------------------------------------------------------
This email message is for the sole use of the intended recipient(s) and may contain
confidential information. Any unauthorized review, use, disclosure or distribution
is prohibited. If you are not the intended recipient, please contact the sender by
reply email and destroy all copies of the original message.
-----------------------------------------------------------------------------------

[Evgeny]

Hi Everyone,

I'm trying to get coverage for native Android code as follows:

1. Build app with '-fprofile-arcs', '-ftest-coverage' LOCAL_CFLAGS flags
and 'lgcov', 'lgcc' LOCAL_LDFLAGS flag (gcno files are created during this
process)
2. Run app on Android API16 emulator (need exacltly this API version) (gcda
files are created on the emulator)
3. pull files using adb such a way gcda and corresponding gcno are in the
same directory
4. run customised lcov mentioned there (android branch)

As a result I get for every coverage-related file:

*.gcno:version '404*', prefer '408*'
*.gcda:version '404*', prefer '408*'
geninfo: WARNING: gcov did not create any files for ... *.gcda

gcov and gcc versions are the same.

Are my steps correct? If steps are correct, have you any idea of how to fix
these issues?

Post by Nikolai Kondrashov

Post by Adrian Vintu

Post by Nikolai Kondrashov
I'd say you should have just read the documentation instead of relying on
luck :)

well, the documentation suggests values 1 and 4. 1 did not work for me -
I have no idea why. number 123 is just perfect - it's not in the
documentation. could be that my google skills are failing me, if there
is a good explanation of this flag I would love to have it.

GCOV_PREFIX and GCOV_PREFIX_STRIP is a typical pair of variables used to
relocate files. GCOV_PREFIX_STRIP is the number of directory components to
be removed from the start of the source file paths and GCOV_PREFIX is a
prefix to be added to the paths.
So, if your DIR_OUT was /home/adrian/android/out, for example and you wish
to put your files under /data/coverage on the target device, you could
# Remove four components, i.e. /home/adrian/android/out
export GCOV_PREFIX_STRIP=4
# Prepend /data/coverage
export GCOV_PREFIX=/data/coverage
Thus an object file on host named
/home/adrian/android/out/debug/target/product/blah/obj/EXECUTABLES/gzip_intermediates/minigzip.o
will get the coverage data output to
/data/coverage/debug/target/product/blah/obj/EXECUTABLES/gzip_intermediates/minigzip.gcda
on the device.
Sincerely,
Nick
-----------------------------------------------------------------------------------
This email message is for the sole use of the intended recipient(s) and may contain
confidential information. Any unauthorized review, use, disclosure or distribution
is prohibited. If you are not the intended recipient, please contact the sender by
reply email and destroy all copies of the original message.
-----------------------------------------------------------------------------------