How Gen AI will transform in-car user experiences

Self-driving cars are one of the earliest and most exciting AI prophecies for humanity. With the help of artificial intelligence, we’ve realized this fantasy over the years. But today, AI can transform the in-car experience beyond just the autonomous driving functions.

While the innovation leaders in the automotive industry are already offering an AI-powered driving experience, traditional OEMs are still in the transition phase. In meanwhile, automotive customers are still waiting for the automotive industry’s iPhone moment, where technology transforms the user experience and eliminates the following pain points: 

• Rule-based interactions, not intelligent
• Virtual assistants with limited conversational capabilities, not natural
• Irrelevant infotainment apps, not useful
• Virtual assistants with insufficient learning capabilities, not personalized

Gen AI opportunities for the OEM

Gen AI can revolutionize the driving experience through realistic and engaging content, making it more fun and less of a chore. 

For the scope of this article, let’s call this Gen AI-enabled in-car intelligent assistant a car-pilot.

The following diagram shows how the car-pilot can enhance customer experience in everyday driving.

Fig 1: How a Gen AI-enabled car-pilot can enhance the customer experience in everyday driving

Intelligent, proactive assistance: A car-pilot can be the driver's left hand for all in-car services. It will integrate with in-car sensor data and machine learning models to predict the driver's intent or requirement under different contexts. It will enable the driver to access any car settings, service, or information through a single conversational interface that can generate natural and coherent responses.

Users will receive proactive services without having to manually select individual applications. The car-pilot will understand user feedback on the proactive services and continuously improve quality and performance throughout a product's lifecycle.

Fig 2: Continuous quality and performance improvement

Enabling in-car Human-Machine Interaction (HMI):  The car-pilot will power more natural and multimodal in-car Human Machine Interaction (HMI), such as images, videos, and sounds, to enhance the user experience and provide rich feedback on proactive services.

• Natural language communication: LLM-enabled natural language capability enables engaging user interactions. OEMs can use prompt engineering and fine-tuning methods to enable the car-pilot to interact with users in a way that resonates with their brand culture.
• Visual content generation: It can generate visual content based on its understanding of the context and user preferences. And generate metaverse (AR/VR) content for entertainment under safe conditions.
• Car control interaction: The car-pilot predicts preferred in-car climate settings, proactively suggests safe driving operations under extreme weather conditions, informs users on underutilized functions, and collects user feedback on several features. Through in-car cameras and sensors, the car-pilot can detect the user's mood and mental and physical status and cheer them up through peppy music, a conversation, or a message. 

Situational awareness and active assistance: The intelligent car-pilot will better understand the users, predict their needs and provide proactive assistance. For example, the car-pilot could greet the user with daily content to add to the driving experience. 

If the car-pilot notices that the user is anxious in traffic, it could proactively play a piece of their favorite music to help them relax. By combining knowledge graphs and machine learning, OEMs can use Gen AI to create more powerful, comprehensive, and contextually accurate AI solutions, enabling car-pilots to provide proactive assistance to users.

Next-level personalization: The intelligent car-pilot learns from the driver's behavior, feedback, and preferences to create a personalized profile. This profile reflects the driver's identity and values, further helping customize the in-car HMI.

For example, the car-pilot can change the HMI's appearance to match the driver's preferences or its voice and tone to match the driver's personality. 

Fig 3: Offering personalized experience by customizing the in-car HMI

Considerations while building a car-pilot 

Data privacy and security: When building a solution like car-pilot, OEMs must keep as much data as possible on the vehicle side and desensitize and encrypt it when updating it to the cloud.  

OEMs must strictly control and protect the access to user privacy data. They must consider the regulations on personal data and driving safety during the design and implementation phases. The content and service provided to users must also comply with relevant laws and regulations.

Data quality and availability: Gen AI relies on high-quality data to train models and generate realistic and responsible content. OEMs should set up well-defined data collection and governance policies.

User experience: Only when the car pilot brings users a truly wonderful experience will users seek out the product. OEMs must work on strong product strategies to manage user expectations. They can leverage design and AI to understand evolving user preferences and make updates when feasible continually.

The product design should balance performance and resource consumption, optimizing the generative model architecture, parameters, and inference methods to reduce the computational cost and latency of content generation.

Integration with the ecosystem: The design and development of the car-pilot require multiple technologies and interdisciplinary R&D empowerment. A dedicated product and technical team that manages the top-level design, development, and cross-departmental collaboration is necessary. Traditional OEMs must change their organizational structure to be successful in the competition.Technical design considerations 

The intelligent car-pilot will be built on a new AI ecosystem that connects the foundation model with the vehicle-side models and application systems through APIs. The foundation model recognizes multimodal input and context (text, image, video, audio, and code). It then generates executable code based on APIs to complete specific tasks.

The foundation model can extract specific tasks from user instructions and help select the most relevant API for code generation. Additionally, it can absorb information from the existing API knowledge base, quickly learn how to use APIs and match them to specific tasks per the API usage history. Explicit code verification mechanisms help confirm the reliability and trustworthiness of the generated code.

The following architecture shows a reference design of the intelligent car pilot.
Fig 4: Reference design of the intelligent car-pilot

The foundation model is at the centre of the architecture and is connected to the vehicle-side models and application systems through APIs. This architecture provides a flexible and scalable framework for the intelligent car-pilot. It can be adapted to different vehicles and applications and easily updated with new features and capabilities.

Putting AI in the car-pilot seat

The automobile industry is undergoing disruptive changes. OEMs may face various challenges, but one thing is certain: Gen AI will significantly alter the automotive landscape. While innovators like Tesla have led the autonomous driving movement, customers are hungry for tech-driven in-car experiences. Gen AI can create an intelligent car-pilot to make the driving experience more personalized, engaging, and enjoyable.

Getting to innovation faster: collaboration is the key

The race for in-vehicle AI dominance is heating up, with established car manufacturers and new players scrambling to develop intelligent assistants. However, building this complex ecosystem requires massive R&D investments and development and carries significant risks. Car manufacturers are struggling to cope with the different technological requirements and fierce competition and need the large talent pool of the tech giants.

This is where technology companies come in. Their expertise in software and AI can enable automakers to develop new features faster, integrate data seamlessly, utilize advanced AI capabilities, and improve information security. Both sides benefit from this strategic collaboration: Automakers gain the critical AI edge to stay competitive while technology companies expand their reach in the lucrative automotive market. Ultimately, this powerful synergy will shape the future of AI in vehicles.