跳到主要内容

4. Qualcomm IM SDK

QIM（Qualcomm's Intelligent Multimedia）可实现多媒体和 AI/ML 应用的部署。利用开源的多媒体框架 GStreamer，在多媒体和机器学习领域提供简单的 API 和插件。这些插件使应用程序开发人员能够在各个领域开发多种多媒体和 AI/ML 应用程序。

使用 Raspberry Pi High Quality Camera（IMX477），进行演示：

信息

下面演示IMX477摄像头使用了 WS1053516 镜头。

  • 访问 https://github.com/rubikpi-ai/demo 下载 RUBIKPi_QIM_models.zip 文件。

  • 将RUBIKPi_QIM_models.zip解压，在PC中执行下面命令将文件夹中的全部内容传输到 RUBIK Pi 3 中。

adb root
adb shell mount -o rw,remount /
cd RUBIKPi_QIM_models
adb push ./* /opt
  • 在RUBIK Pi上执行下面命令设置摄像头参数。
echo multiCameraLogicalXMLFile=kodiak_dc.xml > /var/cache/camera/camxoverridesettings.txt
echo enableNCSService=FALSE >> /var/cache/camera/camxoverridesettings.txt
export XDG_RUNTIME_DIR=/dev/socket/weston
export WAYLAND_DISPLAY=wayland-1
setprop persist.overlay.use_c2d_blit 2
  • 在RUBIK Pi 3上执行下面命令运行demo。
gst-ai-parallel-inference

或

gst-launch-1.0 -e --gst-debug=1 qtiqmmfsrc name=camsrc camera=0 ! video/x-raw\(memory:GBM\),format=NV12,width=1280,height=720,framerate=30/1 ! queue ! qtivsplit name=split split. ! queue ! qtivcomposer name=mixer1 sink_0::position="<0, 0>" sink_0::dimensions="<960, 540>" sink_1::position="<0, 0>" sink_1::dimensions="<960,540>" ! queue ! waylandsink sync=false x=0 y=0 width=960 height=540 \
split. ! queue ! qtimlvconverter ! queue ! qtimltflite delegate=external external-delegate-path=libQnnTFLiteDelegate.so \
external-delegate-options="QNNExternalDelegate,backend_type=htp;" model=/opt/inceptionv3.tflite ! queue ! qtimlvclassification threshold=40.0 results=2 module=mobilenet labels=/opt/classification.labels ! video/x-raw,format=BGRA,width=960,height=540 ! queue ! mixer1. split. ! queue ! qtivcomposer name=mixer2 sink_0::dimensions="<960,540>"sink_0::position="<0,0>" sink_1::dimensions="<960,540>" sink_1::position="<0,0>" ! \
queue ! waylandsink sync=false x=960 y=0 width=960 height=540 split. ! queue ! qtimlvconverter ! queue ! qtimlsnpe delegate=dsp model=/opt/yolonas.dlc layers="</heads/Mul, /heads/Sigmoid>" ! queue ! qtimlvdetection threshold=40.0 results=10 module=yolo-nas labels=/opt/yolonas.labels ! video/x-raw,format=BGRA,width=960,height=540 ! queue ! mixer2.  \
split. ! queue ! qtivcomposer name=mixer3 sink_0::position="<0, 0>" sink_0::dimensions="<960, 540>" sink_1::position="<0, 0>" sink_1::dimensions="<960,540>" ! queue ! waylandsink sync=false x=0 y=540 width=960 height=540 \
split. ! queue ! qtimlvconverter ! queue ! qtimltflite delegate=external external-delegate-path=libQnnTFLiteDelegate.so \
external-delegate-options="QNNExternalDelegate,backend_type=htp;" model=/opt/posenet_mobilenet_v1.tflite ! queue ! qtimlvpose threshold=51.0 results=2 module=posenet labels=/opt/posenet_mobilenet_v1.labels constants="Posenet,q-offsets=<128.0,128.0,117.0>,q-scales=<0.0784313753247261,0.0784313753247261,1.3875764608383179>;" ! queue ! mixer3. split. ! queue ! qtivcomposer name=mixer4 sink_0::dimensions="<960,540>"sink_0::position="<0,0>" sink_1::dimensions="<960,540>" sink_1::position="<0,0>" sink_1::alpha=0.5 ! \
queue ! waylandsink sync=false x=960 y=540 width=960 height=540 split. ! queue ! qtimlvconverter ! queue ! qtimlsnpe delegate=dsp model=/opt/deeplabv3_resnet50.dlc ! queue ! qtimlvsegmentation module=deeplab-argmax labels=/opt/deeplabv3_resnet50.labels ! queue ! mixer4.
  • demo运行结果：