> ## Documentation Index
> Fetch the complete documentation index at: https://cloud-docs.mentra.glass/llms.txt
> Use this file to discover all available pages before exploring further.

# Message types

# Speaking the Same Language: Message Types

In MentraOS Cloud, everything happens through messages. Think of them as the conversations between glasses, cloud, and apps. Let's learn the language!

## The Four Conversations

There are four main conversations happening:

1. **Glasses → Cloud** (What the glasses tell us)
2. **Cloud → Glasses** (What we tell the glasses)
3. **Apps → Cloud** (What apps ask us to do)
4. **Cloud → Apps** (What we tell the apps)

All message types are defined in `packages/sdk/src/types/message-types.ts`.

## Conversation 1: Glasses → Cloud

These messages come from the mobile app (remember, it's the bridge between glasses and cloud). Types are defined in `packages/sdk/src/types/messages/glasses-to-cloud.ts`:

### Connection Messages

```typescript theme={null}
// "Hello, I'm here!"
// From ConnectionInit interface (glasses-to-cloud.ts:15-19)
{
  type: "CONNECTION_INIT",
  userId?: "user@example.com",   // Optional user ID
  coreToken?: "eyJhbGci..."      // Optional auth token
}
```

### User Actions

```typescript theme={null}
// "The user pressed a button!"
// From ButtonPress interface
{
  type: "BUTTON_PRESS",
  buttonId: "main",  // The button identifier
  pressType: "short" // or "long"
}

// "The user moved their head!"
// From HeadPosition interface
{
  type: "HEAD_POSITION",
  position: "up"  // or "down" - Simple up/down detection only!
}
```

### App Control

```typescript theme={null}
// "Start this app!"
{
  type: "START_APP",
  packageName: "com.translator.app"
}

// "Stop this app!"
{
  type: "STOP_APP",
  packageName: "com.translator.app"
}
```

### Media Events

```typescript theme={null}
// "Here's a photo!"
// From PhotoTaken interface
{
  type: "PHOTO_TAKEN",
  photoData: ArrayBuffer,  // Binary photo data
  mimeType: "image/jpeg",
  timestamp: Date
}

// "I played that audio!"
{
  type: "AUDIO_PLAY_RESPONSE",
  requestId: "audio-456",
  success: true
}
```

### System Updates

```typescript theme={null}
// "Here's my current status!"
{
  type: "CORE_STATUS_UPDATE",
  batteryLevel: 85,
  wifiConnected: true,
  bluetoothConnected: true,
  activeApps: ["com.translator.app"]
}

// "Phone got a notification!"
// From PhoneNotification interface
{
  type: "PHONE_NOTIFICATION",
  notificationId: "notif-123",
  app: "com.whatsapp",
  title: "Sarah",
  content: "Hey, are you free for lunch?",
  priority: "normal"  // "low" | "normal" | "high"
}
```

### Location Updates

```typescript theme={null}
// From LocationUpdate interface
{
  type: "LOCATION_UPDATE",
  lat: 37.7749,
  lng: -122.4194,
  accuracy: 5.0,  // meters
  correlationId: "loc-123"
}
```

## Conversation 2: Cloud → Glasses

These messages tell the glasses what to do:

### Connection Management

```typescript theme={null}
// "Welcome! You're connected!"
// From ConnectionAck interface (cloud-to-glasses.ts:19-23)
{
  type: "CONNECTION_ACK",
  sessionId: "session-789",
  userSession: {  // Partial<UserSession>
    userId: "user@example.com",
    startTime: "2024-01-01T00:00:00Z",
    activeAppSessions: [],
    appSubscriptions: {},
    requiresAudio: false,
    isTranscribing: false
    // ...other session data
  },
  timestamp: "2024-01-01T00:00:00Z"
}

// "Something went wrong!"
{
  type: "CONNECTION_ERROR",
  error: "Invalid authentication token"
}
```

### Display Updates

```typescript theme={null}
// "Show this on screen!"
// Display messages use specific layout types from packages/sdk/src/types/layouts.ts
{
  type: "DISPLAY_EVENT",
  layout: {
    type: "TEXT_WALL",  // Or other LayoutType enum values
    text: "72°F and sunny"
  }
}

// Reference card layout
{
  type: "DISPLAY_EVENT",
  layout: {
    type: "REFERENCE_CARD",
    title: "Weather",
    text: "72°F and sunny\nHigh: 78°F Low: 62°F"
  }
}

// "Change dashboard mode!"
{
  type: "DASHBOARD_MODE_CHANGE",
  enabled: true,
  alwaysOn: true
}
```

### Media Requests

```typescript theme={null}
// "Take a photo!"
{
  type: "PHOTO_REQUEST",
  requestId: "photo-123",
  settings: {
    resolution: "HIGH"
  }
}

// "Play this audio!"
{
  type: "AUDIO_PLAY_REQUEST",
  requestId: "audio-456",
  audioData: "base64encoded...",
  format: "mp3"
}
```

### Settings Updates

```typescript theme={null}
// "Update your settings!"
{
  type: "SETTINGS_UPDATE",
  settings: {
    microphoneEnabled: true,
    displayBrightness: 75,
    language: "en-US"
  }
}
```

## Conversation 3: Apps → Cloud

Apps use these messages to interact with the system:

### Connection Setup

```typescript theme={null}
// "I want to connect!"
// From AppConnectionInit interface (app-to-cloud.ts:20-25)
{
  type: "CONNECTION_INIT",
  packageName: "com.translator.app",
  sessionId: "session-789",
  apiKey: "app_api_key_here"
}

// "I want these data streams!"
{
  type: "SUBSCRIPTION_UPDATE",
  subscriptions: [
    {
      type: "TRANSCRIPTION",
      config: {
        languages: ["en-US", "es-ES"],
        interimResults: true
      }
    },
    {
      type: "BUTTON_PRESS",
      config: {
        buttons: ["MAIN"]
      }
    }
  ]
}
```

### Display Requests

```typescript theme={null}
// "Show this layout!"
// From DisplayRequest interface in packages/sdk/src/types/messages/app-to-cloud.ts
{
  type: "DISPLAY_REQUEST",
  packageName: "com.translator.app",
  view: "MAIN",  // ViewType: "MAIN" | "DASHBOARD" | "ALWAYS_ON"
  layout: {
    type: "TEXT_WALL",
    text: "Hola, ¿cómo estás?"
  },
  durationMs: 5000,  // Optional: Show for 5 seconds
  forceDisplay: false
}

// Reference card display
{
  type: "DISPLAY_REQUEST",
  packageName: "com.translator.app",
  view: "MAIN",
  layout: {
    type: "REFERENCE_CARD",
    title: "Translation",
    text: "Spanish: Hola\nEnglish: Hello"
  }
}

// "Update the dashboard!"
{
  type: "DASHBOARD_CONTENT_UPDATE",
  content: {
    text: "Translator Active"
    // Note: No icon field in actual implementation
  }
}
```

### Media Requests

```typescript theme={null}
// "Take a photo!"
{
  type: "PHOTO_REQUEST",
  metadata: {
    purpose: "document-scan"
  }
}

// "Play this sound!"
{
  type: "AUDIO_PLAY_REQUEST",
  audioUrl: "https://example.com/sound.mp3",
  volume: 0.8
}
```

## Conversation 4: Cloud → Apps

The cloud sends these to apps:

### Connection Management

```typescript theme={null}
// "You're connected!"
// From AppConnectionAck interface (cloud-to-app.ts:27-33)
{
  type: "CONNECTION_ACK",
  settings: [/* app-specific settings */],
  mentraosSettings: { /* system settings */ },
  config: { /* app config */ },
  capabilities: {
    hasDisplay: true,
    hasCamera: true,
    // ... device capabilities
  }
}
```

### Data Streams

This is the main way apps receive data. Stream types are defined in `packages/sdk/src/types/streams.ts`:

```typescript theme={null}
// "Here's transcription data you subscribed to!"
// From TranscriptionData interface
{
  type: "DATA_STREAM",
  streamType: "transcription",  // lowercase!
  data: {
    type: "transcription",
    text: "Hello world",
    isFinal: true,
    transcribeLanguage: "en-US",
    startTime: 1234567890,
    endTime: 1234567892,
    speakerId: "speaker-1",
    duration: 2.0,
    provider: "deepgram",
    confidence: 0.95
  }
}

// Button press event
{
  type: "DATA_STREAM",
  streamType: "button_press",  // lowercase with underscore!
  data: {
    type: "button_press",
    buttonId: "main",
    pressType: "short"  // or "long"
  }
}

// Photo data
{
  type: "DATA_STREAM",
  streamType: "photo_taken",  // lowercase with underscore!
  data: {
    type: "photo_taken",
    photoData: ArrayBuffer,
    mimeType: "image/jpeg",
    timestamp: Date
  }
}

// Head position event
{
  type: "DATA_STREAM",
  streamType: "head_position",
  data: {
    type: "head_position",
    position: "up"  // or "down"
  }
}
```

### System Updates

```typescript theme={null}
// "Settings changed!"
{
  type: "SETTINGS_UPDATE",
  settings: {
    theme: "dark",
    notifications: true
  }
}

// "Your app is being stopped!"
{
  type: "APP_STOPPED",
  reason: "USER_REQUEST" // or "SYSTEM_SHUTDOWN", "ERROR"
}
```

## Message Flow Example

Let's trace a message through the system when Alex says "What's the weather?":

1. **Glasses capture audio** → Send audio data to cloud

2. **Cloud transcribes** → "What's the weather?"

3. **Cloud finds subscribers** → Weather app is subscribed to transcriptions

4. **Cloud sends to app**:
   ```typescript theme={null}
   {
     type: "DATA_STREAM",
     streamType: "transcription",
     data: {
       type: "transcription",
       text: "What's the weather?",
       isFinal: true,
       transcribeLanguage: "en-US",
       startTime: 1234567890,
       endTime: 1234567892
     }
   }
   ```

5. **App processes** → Fetches weather data

6. **App responds**:
   ```typescript theme={null}
   {
     type: "DISPLAY_REQUEST",
     packageName: "com.weather.app",
     view: "MAIN",
     layout: {
       type: "REFERENCE_CARD",
       title: "San Francisco Weather",
       text: "72°F and sunny\nHigh: 78°F Low: 62°F"
     }
   }
   ```

7. **Cloud validates** → Checks rate limits, formats display

8. **Cloud sends to glasses**:
   ```typescript theme={null}
   {
     type: "DISPLAY_EVENT",
     layout: {
       type: "REFERENCE_CARD",
       title: "San Francisco Weather",
       text: "72°F and sunny\nHigh: 78°F Low: 62°F"
     }
   }
   ```

9. **Alex sees the weather** on their glasses display!

## Important Patterns

### Request-Response Pattern

Some messages have a request ID for tracking:

* App sends `PHOTO_REQUEST` with `requestId: "photo-123"`
* Cloud forwards to glasses
* Glasses respond with `PHOTO_TAKEN` with same `requestId`
* Cloud matches response to request and sends to app

### Subscription Pattern

Apps don't get all data, only what they subscribe to. Available stream types (from `packages/sdk/src/types/streams.ts`):

**Hardware Streams:**

* `button_press` - Button press events
* `head_position` - Head up/down position
* `location_update` - GPS location updates
* `vps_coordinates` - Visual positioning system data

**Audio Streams:**

* `transcription` - Speech-to-text results
* `translation` - Real-time translation
* `VAD` - Voice activity detection
* `audio_chunk` - Raw audio data

**Phone Streams:**

* `phone_notification` - Incoming notifications
* `phone_notification_dismissed` - Notification dismissals
* `calendar_event` - Calendar updates

**System Streams:**

* `start_app` - App start requests
* `stop_app` - App stop requests
* `core_status_update` - System status

**Media Streams:**

* `photo_taken` - Photo capture events
* `rtmp_stream_status` - RTMP streaming status
* `managed_stream_status` - Managed streaming status

### Fire-and-Forget Pattern

Some messages don't expect responses:

* Display updates just happen
* Settings updates are applied silently
* Status updates are informational

## What's Next?

Now you understand the language of MentraOS! Next, let's see [how apps connect](./5-sdk-and-apps.md) and use these messages to create amazing experiences.