> ## Documentation Index
> Fetch the complete documentation index at: https://cloud-docs.mentra.glass/llms.txt
> Use this file to discover all available pages before exploring further.

# App WebSocket

> WebSocket endpoint for third-party app connections

## Endpoint

<CodeGroup>
  ```bash Production theme={null}
  wss://api.mentra.glass/app-ws
  ```

  ```bash Development theme={null}
  wss://devapi.mentra.glass/app-ws
  ```

  ```bash Local theme={null}
  ws://localhost:8002/app-ws
  ```
</CodeGroup>

## Authentication

Apps authenticate using a JWT token in the Authorization header, along with session information in custom headers.

### Headers

```typescript theme={null}
{
  "Authorization": "Bearer <appJwt>",
  "x-user-id": "user@example.com",
  "x-session-id": "session-123",
  "Host": "cloud.mentraos.com",
  "Upgrade": "websocket",
  "Connection": "Upgrade"
}
```

### JWT Token Structure

The appJwt must contain:

* `packageName`: The app's package identifier
* `apiKey`: The app's API key

## Connection Flow

<Steps>
  <Step title="Webhook Notification">
    App receives webhook when user starts the app
  </Step>

  <Step title="WebSocket Connection">
    App connects to WebSocket with JWT and session headers
  </Step>

  <Step title="Authentication">
    Server validates JWT token and verifies API key
  </Step>

  <Step title="Session Binding">
    App is bound to the user's active session
  </Step>

  <Step title="Connection Init">
    App sends CONNECTION\_INIT to complete handshake
  </Step>
</Steps>

## Message Types

### App → Cloud Messages

All message types are defined in `packages/sdk/src/types/messages/app-to-cloud.ts`:

#### CONNECTION\_INIT

```json theme={null}
{
  "type": "CONNECTION_INIT",
  "packageName": "com.example.app",
  "sessionId": "session-123",
  "apiKey": "app_api_key_here"
}
```

#### SUBSCRIPTION\_UPDATE

```json theme={null}
{
  "type": "SUBSCRIPTION_UPDATE",
  "subscriptions": [
    {
      "type": "transcription",
      "config": {
        "languages": ["en-US", "es-ES"],
        "interimResults": true
      }
    },
    {
      "type": "button_press",
      "config": {
        "buttons": ["main"]
      }
    }
  ]
}
```

#### DISPLAY\_REQUEST

```json theme={null}
{
  "type": "DISPLAY_REQUEST",
  "packageName": "com.example.app",
  "view": "MAIN",
  "layout": {
    "type": "TEXT_WALL",
    "text": "Hello from app!"
  },
  "durationMs": 5000,
  "forceDisplay": false
}
```

#### PHOTO\_REQUEST

```json theme={null}
{
  "type": "PHOTO_REQUEST",
  "requestId": "photo-456",
  "saveToGallery": false,
  "customWebhookUrl": "https://app.example.com/photo-webhook"
}
```

#### AUDIO\_PLAY\_REQUEST

```json theme={null}
{
  "type": "AUDIO_PLAY_REQUEST",
  "requestId": "audio-789",
  "audioUrl": "https://example.com/sound.mp3",
  "volume": 0.8
}
```

### Cloud → App Messages

All message types are defined in `packages/sdk/src/types/messages/cloud-to-app.ts`:

#### CONNECTION\_ACK

```json theme={null}
{
  "type": "CONNECTION_ACK",
  "settings": [],           // App-specific settings
  "mentraosSettings": {},   // System settings
  "config": {},             // App config
  "capabilities": {
    "hasDisplay": true,
    "hasCamera": true,
    "hasMicrophone": true,
    "hasSpeaker": true
  }
}
```

#### DATA\_STREAM

```json theme={null}
{
  "type": "DATA_STREAM",
  "streamType": "transcription",
  "data": {
    "type": "transcription",
    "text": "Hello world",
    "isFinal": true,
    "transcribeLanguage": "en-US",
    "startTime": 1234567890,
    "endTime": 1234567892,
    "confidence": 0.95
  }
}
```

#### SETTINGS\_UPDATE

```json theme={null}
{
  "type": "SETTINGS_UPDATE",
  "settings": {
    "theme": "dark",
    "language": "en-US"
  }
}
```

#### APP\_STOPPED

```json theme={null}
{
  "type": "APP_STOPPED",
  "reason": "USER_REQUEST"
}
```

## Subscription System

Apps must subscribe to the data streams they want to receive. Available stream types are defined in `packages/sdk/src/types/streams.ts`:

### Hardware Streams

* `button_press` - Button press events
* `head_position` - Head position changes
* `location_update` - GPS location updates
* `vps_coordinates` - Visual positioning system

### Audio Streams

* `transcription` - Speech-to-text results
* `translation` - Real-time translation
* `VAD` - Voice activity detection
* `audio_chunk` - Raw audio data

### Phone Streams

* `phone_notification` - Incoming notifications
* `phone_notification_dismissed` - Dismissal events
* `calendar_event` - Calendar updates

### System Streams

* `start_app` - App start requests
* `stop_app` - App stop requests
* `core_status_update` - System status

### Media Streams

* `photo_taken` - Photo capture events
* `rtmp_stream_status` - RTMP streaming status
* `managed_stream_status` - Managed streaming

## Error Handling

### Authentication Errors

```json theme={null}
{
  "type": "CONNECTION_ERROR",
  "error": "Invalid API key"
}
```

### Subscription Errors

```json theme={null}
{
  "type": "SUBSCRIPTION_ERROR",
  "error": "Invalid stream type: unknown_stream"
}
```

## Heartbeat

Apps must send heartbeat messages to maintain their connection:

```json theme={null}
{
  "type": "HEARTBEAT",
  "timestamp": 1234567890
}
```

If no heartbeat is received for 20 seconds, the app enters a grace period before being disconnected.

## Implementation Details

* **Service**: `AppWebSocketService` (packages/cloud/src/services/websocket/websocket-app.service.ts)
* **Manager**: `AppManager` handles app lifecycle and state
* **Authentication**: API key validation against app database
* **Subscriptions**: `SubscriptionService` manages data stream routing

## SDK Usage

The MentraOS SDK handles all WebSocket communication automatically:

```typescript theme={null}
import { AppSession } from '@mentraos/sdk';

const session = new AppSession({
  packageName: 'com.example.app',
  apiKey: process.env.MENTRAOS_API_KEY
});

// Connect to session
await session.connect(sessionId, userId);

// Subscribe to events
session.events.onTranscription((data) => {
  console.log('User said:', data.text);
});

// Send display request
await session.layouts.showTextWall('Hello World!');
```