Camera2 API development

Jolla is literally in administration with probably over half of their staff taking redundancy or effectively massive pay cuts.

I would be surprised if Jolla deliberately made system development difficult unless it’s part of their proprietary code or makes significant demands on their time. Especially since they’ll be deprived of Camera2 goodness as well on their 10 iiis

For me, Snap Camera works very well on Dalvik though I haven’t used it with a >2 camera phone yet.

How far has anyone gotten with native Camera2 API?

well, one could certainly argue about the level of intent, deliberate/careless/whatever, but also bear in mind that i was saying what someone ELSE found to be the case. in any case, this is what i was referring to:

at the very least, this is not the minimum level of community support that i think would be required to justify statements to the effect of ‘the community can implement it, we are too busy’, even though they really ARE just actually too busy.

3 Likes

During the summer I gave a look to the Camera 2 API, and here are some complements to the information @vlagged provided.

The main purpose of the task is to replace the underlying implementation of droidmedia with a Camera 2 API. But as a simpler target, I just tried to take a picture from Sailfish OS side, using a Camera 2 API function in droidmedia.

The first step is to compile the Android side of droidmedia. For this, I followed part of this tutorial: Sailfish X Xperia Android 10 Build and Flash | Sailfish OS Documentation namely

repo init -u https://github.com/mer-hybris/android.git -b $HAVERSION -m tagged-localbuild.xml
repo sync -jX --fetch-submodules
git clone --recurse-submodules https://github.com/mer-hybris/droid-src-sony droid-src -b "hybris-"$HAVERSION
ln -s droid-src/patches
droid-src/apply-patches.sh --mb
./setup-sources.sh --mb

source build/envsetup.sh
lunch aosp_$DEVICE-user
git clone https://github.com/sailfishos/droidmedia external/droidmedia
make libdroidmedia

This creates a library libdroidmedia.so under out/target/product/$HABUILD_DEVICE/system/lib64/libdroidmedia.so. I copy this library to a user directory on the phone (let’s call it devel).

The second step is to compile the libhybris glue of droidmedia for Sailfish OS side of things. I followed these steps in the SDK docker:

cd external/droimedia
mb2 -t SailfishOS-4.5-aarch64 -s rpm/droidmedia-devel.spec build
sb2 -t SailfishOS-4.5-aarch64.default -R rpm -U --force RPMS/droidmedia-devel-0.20170214.0-1.aarch64.rpm

This compile and install the libhybris glue in the SDK. I’ll use it later to compile a simple executable that is supposed to tkae a picture.

This small executable is made of the following main:

#include <QtQuick>
#include <QDebug>

#include <droidmediacamera2.h>
#include <sailfishapp.h>

void plop()
{
    qDebug() << droid_media_camera2_get_number_of_cameras();
}

int main(int argc, char *argv[])
{
    QScopedPointer<QGuiApplication> app(SailfishApp::application(argc, argv));
    app->setOrganizationName("Demo");
    app->setApplicationName("Sextant");

    plop();

    DroidMediaCamera2 *camera = droid_media_camera2_connect(1);
    if (camera) {
        qDebug() << droid_media_camera2_take_picture(camera);
        //droid_media_camera2_disconnect(camera);
    }

    QScopedPointer<QQuickView> view(SailfishApp::createView());
    view->setSource(SailfishApp::pathToMainQml());
    view->show();
    return app->exec();
}

As you see, it is calling a new droidmedia function droid_media_camera2_get_number_of_cameras() testing the listing of cameras and then calling droid_media_camera2_connect(1) to connect to the second listed camera and finally call droid_media_camera2_take_picture(camera) to shot a picture. These functions are basically the same than the currently existing ones, except that they are supposed to use a Camera 2 API.

I implemented these functions in droidmedia, see my branch GitHub - dcaliste/droidmedia at camera2, following various sources like android.hardware.camera2  |  Android Developers or https://github.com/justinjoy/native-camera2/blob/master/app/src/main/jni/native-camera2-jni.cpp

To run the executable (called sextant here), I compile it within the SDK, using my new libhybris glue from my droidmedia branch and send it to the phone under the same devel directory. To run it, one needs to export the devel directory as a source for hybris libraries:

cd devel
HYBRIS_LD_LIBRARY_PATH=$PWD:/usr/libexec/droid-hybris/system/lib64 ./sextant

What is working: the listing of cameras and their characteristics. What is not working: shooting a picture :(( The Camera2 functions are not reporting any error as far as I know. It is reporting that the session is ready. But nothing happens.

So I’m stuck there. Since the Camera 2 implementation is open source in Android, I added some debug messages here and there, but was not able to understand what was wrong.

17 Likes

Interesting. I have some ideas.

2 Likes

Looking at a kotlin example they seem to initialize a few things before. Maybe some of this is helpful? https://github.com/android/camera-samples/blob/main/Camera2Basic/app/src/main/java/com/example/android/camera2/basic/fragments/CameraFragment.kt#L213

1 Like

I think that just writing some wrappers to the Camera2 api would be easier and more useful. Camera2 is supported in the ndk, so in theory it should (almost) work without any Java stuff. Just adding a few functions to take a picture isn’t very useful even if it works, because Camera 1 API does that already. The nice thing about Camera2 api is the control it gives you. You can set the preview and capture parameters which gives you a lot of flexibility.

just writing some wrappers to the Camera2 api would be easier and more useful.

Hopefully, that’s what I’m doing.

Camera2 is supported in the ndk, so in theory it should (almost) work without any Java stuff.

There’s no Java stuff in anything I’ve done so far.

Just adding a few functions to take a picture isn’t very useful even if it works, because Camera 1 API does that already.

Indeed, but if the testing run is not even able to take a picture, well, I can set all the fine tuned parameters I want, I’ll be disappointed with the output… Or the absence of.

So far, I’ve tried to assemblate the various calls of Camera2 API to take a picture, but I failed to get it working. As @thigg suggested, I may have missed some initialisation. I don’t know. Or I’m not letting an Android event loop run to process the events. No idea.

15 Likes

Did you try to see if you get a valid preview first?

I have now pushed some work-in-progress code to a camera2 branch GitHub - sailfishos/droidmedia at camera2. Currently viewfinder and still image capture works as do most of the camera controls. Various issues still exist in the code such as in correct colors in viewfinder (appears to be BGR vs RGB issue) and flash not always working reliably. Video recording is completely unimplemented at the moment and I will look into that after I try to fix the current issues in the code.

42 Likes

This is so empowering I am dancing in my room as we speak.

First, I’d like to confirm that I can reproduce the results: capture works on my Zenfone, but also the BGR vs RGB viewfinder and flash issues.

Second, I feel as this is such a great example, like a tutorial that has an “exercise left to the reader” in the end. Now we know how it should be done. And I’d like the challenge the community to organize around how to proceed with this proof of concept that is so hopeful.

(This is all assuming that mal is overloaded with other porting/volte/appsupport/ffmpeg wherever his wizardry is required and would not have time to continue. If you @mal have input on this plan shoot!)

Preamble to the proposal:

We need to understand that the droidmediacamera2.cpp is effectively linking other implementations for the functions and methods present in droidmediacamera.cpp. That is, one is compiled and linked in, or the other, never both. Right now, the criteria is “Android 7 or newer”.
We need to somehow understand where/when these methods/functions are called, what they did in v1, and what shall they do in v2, and be smart about overlapping efforts & discussions around this. Like, learn together, somehow.

The needs I think need to be covered:

  • Having a clear diff between the new v2 and old v1 API implementation - so that getting around is easy. (also I notice everything involving v2 api is more verbose).
  • Having a “masterV2” branch that can absorb the different contributions for the “camera2” branch that are community reviewed
  • Having a clear set of tasks and comments/pointers contributed to them, in addition to code. We need to be more of us “fresh” people to focus on different things to be able to fill the gap of one experienced would contain in its entirety.

So my proposal (and please chime in to disagree or improve) of another guy with little time on his hands:

  • we could set up a fork into an organization (e.g. sailfishos-open but any)
  • have a copy of the original droidmedia master and a working branch with droidmediacamera.cpp implemented as V2 that would clearly diff (reordered methods etc) against the V1 API) - this is all temporarily to get a sense of progress
  • have as many branches as needed having requests against the above working branch
  • have some plan board or issue tracker that can host discussions around the current and future issues, quirks, or video implementation tasks and even stubbing unfinished work etc.
  • Some docs around how to contribute/build if you’re not a porter (I think @dcaliste started some with his bold foray into this, I am guilty to having already access to hadk env…)

All this could technically be in the sailfishos/droidmedia repo, if we could get Boards/Issues/Wiki access I guess.

Again, this is first stab, top of my head, way to organize some “swarming” around this already great work, If you feel I overreact (I may) or have other ideas, shoot.

(I’m gonna add a quirk to remove edge detection on zenfone regardless so there’s that :supervillain:)

13 Likes

I tried to set this up for the xa2, but there was no branch for android8 in Branches · mer-hybris/droid-src-sony · GitHub

I guess I actually need to read the documentation, right?
But I got stuck here on

Please download the latest Sailfish OS HADK (Hardware Adaptation Development Kit) from within this link.

but there I cant figure out where to download the HADK yet. I guess I a missing some elementary knowledge here and the documentation is outdated…

1 Like

I’m no porter or HADK regular user, but maybe you need for an XA2 this repo instead : GitHub - mer-hybris/droid-src-sony-nile

1 Like

I tried that one, but then I need lunch and make is not finding a droidmedia target. I think the problem is that i am not in the HABUILD_SDK. It looks like installation of it is explained in the HADK pdf and this doesn’t look like it is done in 20 minutes, so I must leave this for later…

Just to get the libdroidmedia.so, I didn’t follow the HADK pdf. Maybe it’s wrong by the way. But just having the Android sources, and compile there was enough.

For lunch, I just type the TAB key for autocompletion, and took the value that corresponds to my device (here a xqau52, its written on the back of the phone, for a 10ii device at least).

Then, the make target is libdroidmedia, not droidmedia.

1 Like

hm where does lunch come from? is it python-lunch?

[thilo@waschmaschine git]$ lunch
-bash: lunch: command not found
[thilo@waschmaschine git]$ whereis lunch
lunch:
[thilo@waschmaschine git]$ make libdroidmedia
make: *** No rule to make target 'libdroidmedia'.  Stop.
[thilo@waschmaschine git]$

ah… i might be failing here…

[thilo@waschmaschine git]$ source build/envsetup.sh
-bash: build/envsetup.sh: No such file or directory

Did you run source build/envsetup.sh in the Android source tree before issuing lunch ? It should create some shell functions like lunch to handle Android build routines.

1 Like

hmm…

[thilo@waschmaschine git]$ ls -lah build/envsetup.sh
lrwxrwxrwx 1 thilo thilo 16 27. Sep  11:39 build/envsetup.sh -> make/envsetup.sh
[thilo@waschmaschine git]$ ls make
ls: cannot access 'make': No such file or directory
[thilo@waschmaschine git]$ ls build/make/

EDIT:
looking at this my initial repo sync probably failed…

I’ll try again from the start… (with repo sync

Did you fix your build setup?

Just as I said before, I am lucky to already have a such build environment, since I already built the HADK bits.

I imagine that to only make droidmedia one would “just” need to:

  • have the manifests for the specific device. I don’t think vendor or kernel matter, so it might just work with the AOSP manifests of the specific adaptation, say, Android 10. Find out by devel-su and grepping /system/build.prop for ro.system.build.version.release
  • use repo command, which needs to be installed separately, to sync the sources. Might require skipping submodules.
  • use source build/envsetup.sh or just the . for source
  • checkout droidmedia using git clone in external/droidmedia and make sure you’re on camera2 branch
  • breakfast $yourdevice or, in the case of AOSP, find a codename that is not emulated (maybe aosp_arm64-eng will work). Otherwise, $yourdevice is the codename you fetched the manifests for.
  • export PORT_ARCH="aarch64"
  • make droidmedia. That should find the makefile wherever so even in external/droidmedia. And should output some libdroidmedia.so, figure it out where from the output. (it should be out/target/product/$yourdevice/system/lib64/libdroidmedia.so)
  • scp the libdroidmedia.so to the device. Use devel-su to overwrite it in /usr/libexec/droid-hybris/system/lib64/libdroidmedia.so.
  • If the above fails zypper in -f droidmedia to have back your system
  • If you want to see logging, use /usr/libexec/droid-hybris/system/bin/logcat while operating the camera. ALOGI used today may not output, you can tryi # setprop log.tag INFO or just change the source to ALOGE

Warning: this is written away from my build system, take it with a grain of salt and duckduckgo it.

4 Likes

I actually fixed it today. I had to delete the folder and refetch the repositories. Make ran and then i ran out of time, but I built the .so :slight_smile:

5 Likes

I tried compiling for Xperia 10 III following the instructions above and on docs.sailfishos.org

make droidmedia
20:48:01 Build sandboxing disabled due to nsjail error.
============================================
PLATFORM_VERSION_CODENAME=REL
PLATFORM_VERSION=11
TARGET_PRODUCT=aosp_xqbt52
TARGET_BUILD_VARIANT=user
TARGET_BUILD_TYPE=release
TARGET_ARCH=arm64
TARGET_ARCH_VARIANT=armv8-2a
TARGET_CPU_VARIANT=generic
TARGET_2ND_ARCH=arm
TARGET_2ND_ARCH_VARIANT=armv8-2a
TARGET_2ND_CPU_VARIANT=generic
HOST_ARCH=x86_64
HOST_2ND_ARCH=x86
HOST_OS=linux
HOST_OS_EXTRA=Linux-6.2.0-34-generic-x86_64-Ubuntu-20.04.2-LTS
HOST_CROSS_OS=windows
HOST_CROSS_ARCH=x86
HOST_CROSS_2ND_ARCH=x86_64
HOST_BUILD_TYPE=release
BUILD_ID=RQ3A.211001.001
OUT_DIR=out
PRODUCT_SOONG_NAMESPACES=device/sony/common device/sony/lena vendor/qcom/opensource/core-utils vendor/qcom/opensource/display/sm8250 vendor/qcom/opensource/display-commonsys-intf/sm8250
============================================
external/droidmedia/Android.mk was modified, regenerating...
[ 23% 56/239] including external/droidmedia/Android.mk ...
external/droidmedia/Android.mk:48: warning: *** ANDROID_MICRO undefined. Assuming 0
20:48:23 Disallowed PATH tool "xmllint" used: []string{"xmllint", "--xpath", "string(/device[@name=\"Android\"]/item[@name=\"battery.capacity\"])", "device/sony/pdx213/overlay/frameworks/base/core/res/res/xml/power_profile.xml"}
20:48:23 See https://android.googlesource.com/platform/build/+/master/Changes.md#PATH_Tools for more information.
[ 50% 121/239] including hybris/hybris-boot/Android.mk ...
hybris/hybris-boot/Android.mk:71: warning: ********************* /boot appears to live on /dev/block/bootdevice/by-name/boot
hybris/hybris-boot/Android.mk:72: warning: ********************* /data appears to live on /dev/block/bootdevice/by-name/userdata
"xmllint" is not allowed to be used. See https://android.googlesource.com/platform/build/+/master/Changes.md#PATH_Tools for more information.
Could not determine android architecture. Please pass it as 2nd argument using gettargetarch.
/bin/bash: external/audioflingerglue/detect_build_targets.sh: No such file or directory
[ 51% 123/239] including hybris/mw/libhybris/libhybris/compat/hwc2/Android.mk ...
hybris/mw/libhybris/libhybris/compat/hwc2/Android.mk:8: warning: *** ANDROID_VERSION_MINOR undefined. Assuming 0
hybris/mw/libhybris/libhybris/compat/hwc2/Android.mk:13: warning: *** ANDROID_VERSION_MICRO undefined. Assuming 0
[ 51% 124/239] including hybris/mw/libhybris/libhybris/compat/input/Android.mk ...
/bin/bash: line 0: test: too many arguments
ninja: no work to do.

#### build completed successfully (38 seconds) ####

but I cannot find the droidmedia.so file:

ls out/target/product/pdx213/system/lib64/
android.frameworks.bufferhub@1.0.so                  android.hardware.graphics.common@1.1.so    android.hidl.token@1.0-utils.so  libbufferhubqueue.so   libfmq.so                liblogwrap.so                libsurfaceflinger.so
android.frameworks.vr.composer@1.0.so                android.hardware.graphics.common@1.2.so    android.hidl.token@1.0.so        libc++.so              libfs_mgr.so             liblp.so                     libsync.so
android.frameworks.vr.composer@2.0.so                android.hardware.graphics.composer@2.1.so  ld-android.so                    libcgrouprc.so         libgralloctypes.so       liblzma.so                   libtimestats.so
android.hardware.boot@1.0.so                         android.hardware.graphics.composer@2.2.so  libEGL.so                        libcrypto.so           libgraphicsenv.so        libnativebridge_lazy.so      libtimestats_proto.so
android.hardware.boot@1.1.so                         android.hardware.graphics.composer@2.3.so  libGLESv1_CM.so                  libcrypto_utils.so     libgsi.so                libnativeloader_lazy.so      libtinyxml2.so
android.hardware.common-V1-ndk_platform.so           android.hardware.graphics.composer@2.4.so  libGLESv2.so                     libcutils.so           libgui.so                libnativewindow.so           libui.so
android.hardware.configstore-utils.so                android.hardware.graphics.mapper@2.0.so    libSurfaceFlingerProp.so         libdexfile_support.so  libhardware.so           libpackagelistparser.so      libunwindstack.so
android.hardware.configstore@1.0.so                  android.hardware.graphics.mapper@2.1.so    libadbd_auth.so                  libext2_blkid.so       libhidl-gen-utils.so     libpcre2.so                  libutils.so
android.hardware.configstore@1.1.so                  android.hardware.graphics.mapper@3.0.so    libadbd_fs.so                    libext2_com_err.so     libhidlbase.so           libpdx_default_transport.so  libvintf.so
android.hardware.graphics.allocator@2.0.so           android.hardware.graphics.mapper@4.0.so    libandroid_runtime_lazy.so       libext2_e2p.so         libhidltransport.so      libprocessgroup.so           libvndksupport.so
android.hardware.graphics.allocator@3.0.so           android.hardware.media@1.0.so              libbacktrace.so                  libext2_misc.so        libhwbinder.so           libprotobuf-cpp-lite.so      libz.so
android.hardware.graphics.allocator@4.0.so           android.hardware.power-V1-cpp.so           libbase.so                       libext2_quota.so       libhwc2_compat_layer.so  libprotoutil.so
android.hardware.graphics.bufferqueue@1.0.so         android.hardware.power@1.0.so              libbinder.so                     libext2_uuid.so        libinput.so              libselinux.so
android.hardware.graphics.bufferqueue@2.0.so         android.hardware.power@1.1.so              libbinder_ndk.so                 libext2fs.so           libkeyutils.so           libsparse.so
android.hardware.graphics.common-V1-ndk_platform.so  android.hardware.power@1.2.so              libbootloader_message.so         libext4_utils.so       liblayers_proto.so       libsquashfs_utils.so
android.hardware.graphics.common@1.0.so              android.hardware.power@1.3.so              libbufferhub.so                  libfec.so              liblog.so                libstatslog.so

Any ideas?

(If I need to clean and rerun droidmedia, how would I do this? I have been switching between master and camera2 to trigger partial rebuilds)