Intelligent Camera Stream Integration and Warning System (ICSW)

The Project Summary provides an overview about Intelligent Camera Stream Integration and Warning System ICSW project, which was part of the Google Summer of Code (GSoC) 2025 program. The ICSW project was mentored by The Linux Foundation (TLF) through its Automotive Grade Linux (AGL) open source initiative. The purpose of the ICSW project was to develop a modular system that servers as a warning system for automotive application.

As part of this work, the project focused on:

  1. Researching and benchmarking the capabilities of different hardware backends for automotive applications
  2. Detecting and classifying environmental objects that directly affects the driving behavior in real time
  3. Visualizing relevant detected objects and conveying appropriate warning messages.

The progress achieved so far lays the foundations for the planned ICSW system. The work completed during the Google Summer of Code 2025 program managed to address the following components of the system:

  1. Inference models cam_infer_models
    • for detecting surrounding objects and traffic signs.
  2. PipeWire Multimedia Layer (co-hosted in cam_infer_models)
    • for managing and controlling the flow of camera streams.
  3. Flutter Application camera_streams_app
    • for visual demonstration of retrieved and processed camera streams, and user visual warnings.
  4. gRPC Communication Layer (protos in camera_idl)
    • for intercommunication between system components, and for activating or switching detection modes.

Project Highlights

The planned timeline for the GSoC 2025 program for large projects spans 22 weeks. The following table provides references for each week’s progress regarding the ICSW system development:

Week 1 Week 2 Week 3 Week 4* Week 5 AMM
Week 7 Week 8 Week 9* Week 10 Week 11* Midterm Evaluation
Week 12 Week 13 Break Break Week 16* Break
Week 18 Week 19 Week 20 Week 21 Week 22* Final Evaluation

The figure below illustrates my dedicated working hours throughout the GSoC 2025 program timeline:

GSoC25 work hours Figure 1. Weekly Working Hours During GSoC 2025

During the program, the following milestones contributed most significantly to the final outcome of the ICSW application:

  1. Week 4: Three-node PipeWire graph that produces two video streams.
  2. Week 9: PipeWire graph that processes live camera input and hosts objects detector on the detection playback stream
  3. Week 11: Flutter application capable of communicating with inference model(s) and retrieving object detections
  4. Week 15: First PR #1 to be merged for the environment CI job
  5. Week 22: Submitted Gerrit change #31262 containing the finalized Yocto recipes, and built the AGL Image

The following table describes the submitted Pull Requests along with their status throughout the project:

Repository PR Status Description
cam_infer_models PR #1 ✅ merged YOLO CI Job
cam_infer_models PR #2 ✅ merged TFLite, CI Job
cam_infer_models PR #3 🚧 ongoing Refactoring
camera_idl PR #1 ✅ merged Refactoring
meta-agl-demo Change #31262 ✅ merged ICSW Integration

Key Takeaways

  1. Learned about the Yocto Project, its components (recipes and layers), and how to use Gerrit effectively.
  2. Gained some experience developing Flutter applications and organizing PipeWire graphs.
  3. Implemented the communication layer using gRPC.
  4. Understood how to use GitHub actions workflow with CI jobs
  5. Realized the importance of not getting stuck in planning or reading documentation, but to start creating tangible code early instead. Special thanks to Joel for emphasizing this point.
  6. Learned that regaining momentum quickly after breaks is vital to achieving goals. This challenge is clearly observed in Figure 1, which shows reduced hours after the midterm and vacation weeks.

Future Work

  1. Enhancement and continued contribution to the ICSW system, including completion of pending tasks:
    • Improve SSD-Mobilenet TFLite model
    • Enhance the Flutter Application lifecycle management
    • Merge the PR #3
    • Documentation
  2. Hardware implementation on the NanoPC-T6.
  3. Continuous contribution to the AGL project.

Acknowledgments

I am deeply grateful for the opportunity to participate in the GSoC program and contribute to AGL this year. I would like to highlight the invaluable guidance and continuous support provided by my mentors, throughout the project, as well as the great role of the AGL community. Special thanks go to my mentors Joel, Jan-Simon, George, Walt, and Scott from whom I have learned a lot, both technically and personally.