Speaking the Same Language: Message Types

In MentraOS Cloud, everything happens through messages. Think of them as the conversations between glasses, cloud, and apps. Let’s learn the language!

The Four Conversations

There are four main conversations happening:

Glasses → Cloud (What the glasses tell us)
Cloud → Glasses (What we tell the glasses)
Apps → Cloud (What apps ask us to do)
Cloud → Apps (What we tell the apps)

All message types are defined in packages/sdk/src/types/message-types.ts.

Conversation 1: Glasses → Cloud

These messages come from the mobile app (remember, it’s the bridge between glasses and cloud). Types are defined in packages/sdk/src/types/messages/glasses-to-cloud.ts:

Connection Messages

// "Hello, I'm here!"
// From ConnectionInit interface (glasses-to-cloud.ts:15-19)
{
  type: "CONNECTION_INIT",
  userId?: "user@example.com",   // Optional user ID
  coreToken?: "eyJhbGci..."      // Optional auth token
}

// "The user pressed a button!"
// From ButtonPress interface
{
  type: "BUTTON_PRESS",
  buttonId: "main",  // The button identifier
  pressType: "short" // or "long"
}

// "The user moved their head!"
// From HeadPosition interface
{
  type: "HEAD_POSITION",
  position: "up"  // or "down" - Simple up/down detection only!
}

App Control

// "Start this app!"
{
  type: "START_APP",
  packageName: "com.translator.app"
}

// "Stop this app!"
{
  type: "STOP_APP",
  packageName: "com.translator.app"
}

Media Events

// "Here's a photo!"
// From PhotoTaken interface
{
  type: "PHOTO_TAKEN",
  photoData: ArrayBuffer,  // Binary photo data
  mimeType: "image/jpeg",
  timestamp: Date
}

// "I played that audio!"
{
  type: "AUDIO_PLAY_RESPONSE",
  requestId: "audio-456",
  success: true
}

System Updates

// "Here's my current status!"
{
  type: "CORE_STATUS_UPDATE",
  batteryLevel: 85,
  wifiConnected: true,
  bluetoothConnected: true,
  activeApps: ["com.translator.app"]
}

// "Phone got a notification!"
// From PhoneNotification interface
{
  type: "PHONE_NOTIFICATION",
  notificationId: "notif-123",
  app: "com.whatsapp",
  title: "Sarah",
  content: "Hey, are you free for lunch?",
  priority: "normal"  // "low" | "normal" | "high"
}

Location Updates

// From LocationUpdate interface
{
  type: "LOCATION_UPDATE",
  lat: 37.7749,
  lng: -122.4194,
  accuracy: 5.0,  // meters
  correlationId: "loc-123"
}

Conversation 2: Cloud → Glasses

These messages tell the glasses what to do:

Connection Management

// "Welcome! You're connected!"
// From ConnectionAck interface (cloud-to-glasses.ts:19-23)
{
  type: "CONNECTION_ACK",
  sessionId: "session-789",
  userSession: {  // Partial<UserSession>
    userId: "user@example.com",
    startTime: "2024-01-01T00:00:00Z",
    activeAppSessions: [],
    appSubscriptions: {},
    requiresAudio: false,
    isTranscribing: false
    // ...other session data
  },
  timestamp: "2024-01-01T00:00:00Z"
}

// "Something went wrong!"
{
  type: "CONNECTION_ERROR",
  error: "Invalid authentication token"
}

Display Updates

// "Show this on screen!"
// Display messages use specific layout types from packages/sdk/src/types/layouts.ts
{
  type: "DISPLAY_EVENT",
  layout: {
    type: "TEXT_WALL",  // Or other LayoutType enum values
    text: "72°F and sunny"
  }
}

// Reference card layout
{
  type: "DISPLAY_EVENT",
  layout: {
    type: "REFERENCE_CARD",
    title: "Weather",
    text: "72°F and sunny\nHigh: 78°F Low: 62°F"
  }
}

// "Change dashboard mode!"
{
  type: "DASHBOARD_MODE_CHANGE",
  enabled: true,
  alwaysOn: true
}

Media Requests

// "Take a photo!"
{
  type: "PHOTO_REQUEST",
  requestId: "photo-123",
  settings: {
    resolution: "HIGH"
  }
}

// "Play this audio!"
{
  type: "AUDIO_PLAY_REQUEST",
  requestId: "audio-456",
  audioData: "base64encoded...",
  format: "mp3"
}

Settings Updates

// "Update your settings!"
{
  type: "SETTINGS_UPDATE",
  settings: {
    microphoneEnabled: true,
    displayBrightness: 75,
    language: "en-US"
  }
}

Conversation 3: Apps → Cloud

Apps use these messages to interact with the system:

Connection Setup

// "I want to connect!"
// From AppConnectionInit interface (app-to-cloud.ts:20-25)
{
  type: "CONNECTION_INIT",
  packageName: "com.translator.app",
  sessionId: "session-789",
  apiKey: "app_api_key_here"
}

// "I want these data streams!"
{
  type: "SUBSCRIPTION_UPDATE",
  subscriptions: [
    {
      type: "TRANSCRIPTION",
      config: {
        languages: ["en-US", "es-ES"],
        interimResults: true
      }
    },
    {
      type: "BUTTON_PRESS",
      config: {
        buttons: ["MAIN"]
      }
    }
  ]
}

Display Requests

// "Show this layout!"
// From DisplayRequest interface in packages/sdk/src/types/messages/app-to-cloud.ts
{
  type: "DISPLAY_REQUEST",
  packageName: "com.translator.app",
  view: "MAIN",  // ViewType: "MAIN" | "DASHBOARD" | "ALWAYS_ON"
  layout: {
    type: "TEXT_WALL",
    text: "Hola, ¿cómo estás?"
  },
  durationMs: 5000,  // Optional: Show for 5 seconds
  forceDisplay: false
}

// Reference card display
{
  type: "DISPLAY_REQUEST",
  packageName: "com.translator.app",
  view: "MAIN",
  layout: {
    type: "REFERENCE_CARD",
    title: "Translation",
    text: "Spanish: Hola\nEnglish: Hello"
  }
}

// "Update the dashboard!"
{
  type: "DASHBOARD_CONTENT_UPDATE",
  content: {
    text: "Translator Active"
    // Note: No icon field in actual implementation
  }
}

Media Requests

// "Take a photo!"
{
  type: "PHOTO_REQUEST",
  metadata: {
    purpose: "document-scan"
  }
}

// "Play this sound!"
{
  type: "AUDIO_PLAY_REQUEST",
  audioUrl: "https://example.com/sound.mp3",
  volume: 0.8
}

Conversation 4: Cloud → Apps

The cloud sends these to apps:

Connection Management

// "You're connected!"
// From AppConnectionAck interface (cloud-to-app.ts:27-33)
{
  type: "CONNECTION_ACK",
  settings: [/* app-specific settings */],
  mentraosSettings: { /* system settings */ },
  config: { /* app config */ },
  capabilities: {
    hasDisplay: true,
    hasCamera: true,
    // ... device capabilities
  }
}

Data Streams

This is the main way apps receive data. Stream types are defined in packages/sdk/src/types/streams.ts:

// "Here's transcription data you subscribed to!"
// From TranscriptionData interface
{
  type: "DATA_STREAM",
  streamType: "transcription",  // lowercase!
  data: {
    type: "transcription",
    text: "Hello world",
    isFinal: true,
    transcribeLanguage: "en-US",
    startTime: 1234567890,
    endTime: 1234567892,
    speakerId: "speaker-1",
    duration: 2.0,
    provider: "deepgram",
    confidence: 0.95
  }
}

// Button press event
{
  type: "DATA_STREAM",
  streamType: "button_press",  // lowercase with underscore!
  data: {
    type: "button_press",
    buttonId: "main",
    pressType: "short"  // or "long"
  }
}

// Photo data
{
  type: "DATA_STREAM",
  streamType: "photo_taken",  // lowercase with underscore!
  data: {
    type: "photo_taken",
    photoData: ArrayBuffer,
    mimeType: "image/jpeg",
    timestamp: Date
  }
}

// Head position event
{
  type: "DATA_STREAM",
  streamType: "head_position",
  data: {
    type: "head_position",
    position: "up"  // or "down"
  }
}

System Updates

// "Settings changed!"
{
  type: "SETTINGS_UPDATE",
  settings: {
    theme: "dark",
    notifications: true
  }
}

// "Your app is being stopped!"
{
  type: "APP_STOPPED",
  reason: "USER_REQUEST" // or "SYSTEM_SHUTDOWN", "ERROR"
}

Message Flow Example

Let’s trace a message through the system when Alex says “What’s the weather?”:

Glasses capture audio → Send audio data to cloud
Cloud transcribes → “What’s the weather?”
Cloud finds subscribers → Weather app is subscribed to transcriptions

Cloud sends to app:

{
  type: "DATA_STREAM",
  streamType: "transcription",
  data: {
    type: "transcription",
    text: "What's the weather?",
    isFinal: true,
    transcribeLanguage: "en-US",
    startTime: 1234567890,
    endTime: 1234567892
  }
}

App processes → Fetches weather data

App responds:

{
  type: "DISPLAY_REQUEST",
  packageName: "com.weather.app",
  view: "MAIN",
  layout: {
    type: "REFERENCE_CARD",
    title: "San Francisco Weather",
    text: "72°F and sunny\nHigh: 78°F Low: 62°F"
  }
}

Cloud validates → Checks rate limits, formats display

Cloud sends to glasses:

{
  type: "DISPLAY_EVENT",
  layout: {
    type: "REFERENCE_CARD",
    title: "San Francisco Weather",
    text: "72°F and sunny\nHigh: 78°F Low: 62°F"
  }
}

Alex sees the weather on their glasses display!

Important Patterns

Request-Response Pattern

Some messages have a request ID for tracking:

App sends PHOTO_REQUEST with requestId: "photo-123"
Cloud forwards to glasses
Glasses respond with PHOTO_TAKEN with same requestId
Cloud matches response to request and sends to app

Subscription Pattern

Apps don’t get all data, only what they subscribe to. Available stream types (from packages/sdk/src/types/streams.ts): Hardware Streams:

button_press - Button press events
head_position - Head up/down position
location_update - GPS location updates
vps_coordinates - Visual positioning system data

Audio Streams:

transcription - Speech-to-text results
translation - Real-time translation
VAD - Voice activity detection
audio_chunk - Raw audio data

Phone Streams:

phone_notification - Incoming notifications
phone_notification_dismissed - Notification dismissals
calendar_event - Calendar updates

System Streams:

start_app - App start requests
stop_app - App stop requests
core_status_update - System status

Media Streams:

photo_taken - Photo capture events
rtmp_stream_status - RTMP streaming status
managed_stream_status - Managed streaming status

Fire-and-Forget Pattern

Some messages don’t expect responses:

Display updates just happen
Settings updates are applied silently
Status updates are informational

What’s Next?

Now you understand the language of MentraOS! Next, let’s see how apps connect and use these messages to create amazing experiences.

Getting Started

Cloud Overview

Cloud Architecture

SDK Reference

Development

Message types

Speaking the Same Language: Message Types

The Four Conversations

Conversation 1: Glasses → Cloud

Connection Messages

User Actions

App Control

Media Events

System Updates

Location Updates

Conversation 2: Cloud → Glasses

Connection Management

Display Updates

Media Requests

Settings Updates

Conversation 3: Apps → Cloud

Connection Setup

Display Requests

Media Requests

Conversation 4: Cloud → Apps

Connection Management

Data Streams

System Updates

Message Flow Example

Important Patterns

Request-Response Pattern

Subscription Pattern

Fire-and-Forget Pattern

What’s Next?

Getting Started

Cloud Overview

Cloud Architecture

SDK Reference

Development

​Speaking the Same Language: Message Types

​The Four Conversations

​Conversation 1: Glasses → Cloud

​Connection Messages

​User Actions

​App Control

​Media Events

​System Updates

​Location Updates

​Conversation 2: Cloud → Glasses

​Connection Management

​Display Updates

​Media Requests

​Settings Updates

​Conversation 3: Apps → Cloud

​Connection Setup

​Display Requests

​Media Requests

​Conversation 4: Cloud → Apps

​Connection Management

​Data Streams

​System Updates

​Message Flow Example

​Important Patterns

​Request-Response Pattern

​Subscription Pattern

​Fire-and-Forget Pattern

​What’s Next?

Speaking the Same Language: Message Types

The Four Conversations

Conversation 1: Glasses → Cloud

Connection Messages

User Actions

App Control

Media Events

System Updates

Location Updates

Conversation 2: Cloud → Glasses

Connection Management

Display Updates

Media Requests

Settings Updates

Conversation 3: Apps → Cloud

Connection Setup

Display Requests

Media Requests

Conversation 4: Cloud → Apps

Connection Management

Data Streams

System Updates

Message Flow Example

Important Patterns

Request-Response Pattern

Subscription Pattern

Fire-and-Forget Pattern

What’s Next?