Skip to main content

Documentation Index

Fetch the complete documentation index at: https://cloud-docs.mentra.glass/llms.txt

Use this file to discover all available pages before exploring further.

​
Endpoint

wss://api.mentra.glass/app-ws

​
Authentication

Apps authenticate using a JWT token in the Authorization header, along with session information in custom headers.

​
Headers

{
  "Authorization": "Bearer <appJwt>",
  "x-user-id": "user@example.com",
  "x-session-id": "session-123",
  "Host": "cloud.mentraos.com",
  "Upgrade": "websocket",
  "Connection": "Upgrade"
}

​
JWT Token Structure

The appJwt must contain:
  • packageName: The app’s package identifier
  • apiKey: The app’s API key

​
Connection Flow

1

Webhook Notification

App receives webhook when user starts the app
2

WebSocket Connection

App connects to WebSocket with JWT and session headers
3

Authentication

Server validates JWT token and verifies API key
4

Session Binding

App is bound to the user’s active session
5

Connection Init

App sends CONNECTION_INIT to complete handshake

​
Message Types

​
App → Cloud Messages

All message types are defined in packages/sdk/src/types/messages/app-to-cloud.ts:

​
CONNECTION_INIT

{
  "type": "CONNECTION_INIT",
  "packageName": "com.example.app",
  "sessionId": "session-123",
  "apiKey": "app_api_key_here"
}

​
SUBSCRIPTION_UPDATE

{
  "type": "SUBSCRIPTION_UPDATE",
  "subscriptions": [
    {
      "type": "transcription",
      "config": {
        "languages": ["en-US", "es-ES"],
        "interimResults": true
      }
    },
    {
      "type": "button_press",
      "config": {
        "buttons": ["main"]
      }
    }
  ]
}

​
DISPLAY_REQUEST

{
  "type": "DISPLAY_REQUEST",
  "packageName": "com.example.app",
  "view": "MAIN",
  "layout": {
    "type": "TEXT_WALL",
    "text": "Hello from app!"
  },
  "durationMs": 5000,
  "forceDisplay": false
}

​
PHOTO_REQUEST

{
  "type": "PHOTO_REQUEST",
  "requestId": "photo-456",
  "saveToGallery": false,
  "customWebhookUrl": "https://app.example.com/photo-webhook"
}

​
AUDIO_PLAY_REQUEST

{
  "type": "AUDIO_PLAY_REQUEST",
  "requestId": "audio-789",
  "audioUrl": "https://example.com/sound.mp3",
  "volume": 0.8
}

​
Cloud → App Messages

All message types are defined in packages/sdk/src/types/messages/cloud-to-app.ts:

​
CONNECTION_ACK

{
  "type": "CONNECTION_ACK",
  "settings": [],           // App-specific settings
  "mentraosSettings": {},   // System settings
  "config": {},             // App config
  "capabilities": {
    "hasDisplay": true,
    "hasCamera": true,
    "hasMicrophone": true,
    "hasSpeaker": true
  }
}

​
DATA_STREAM

{
  "type": "DATA_STREAM",
  "streamType": "transcription",
  "data": {
    "type": "transcription",
    "text": "Hello world",
    "isFinal": true,
    "transcribeLanguage": "en-US",
    "startTime": 1234567890,
    "endTime": 1234567892,
    "confidence": 0.95
  }
}

​
SETTINGS_UPDATE

{
  "type": "SETTINGS_UPDATE",
  "settings": {
    "theme": "dark",
    "language": "en-US"
  }
}

​
APP_STOPPED

{
  "type": "APP_STOPPED",
  "reason": "USER_REQUEST"
}

​
Subscription System

Apps must subscribe to the data streams they want to receive. Available stream types are defined in packages/sdk/src/types/streams.ts:

​
Hardware Streams

  • button_press - Button press events
  • head_position - Head position changes
  • location_update - GPS location updates
  • vps_coordinates - Visual positioning system

​
Audio Streams

  • transcription - Speech-to-text results
  • translation - Real-time translation
  • VAD - Voice activity detection
  • audio_chunk - Raw audio data

​
Phone Streams

  • phone_notification - Incoming notifications
  • phone_notification_dismissed - Dismissal events
  • calendar_event - Calendar updates

​
System Streams

  • start_app - App start requests
  • stop_app - App stop requests
  • core_status_update - System status

​
Media Streams

  • photo_taken - Photo capture events
  • rtmp_stream_status - RTMP streaming status
  • managed_stream_status - Managed streaming

​
Error Handling

​
Authentication Errors

{
  "type": "CONNECTION_ERROR",
  "error": "Invalid API key"
}

​
Subscription Errors

{
  "type": "SUBSCRIPTION_ERROR",
  "error": "Invalid stream type: unknown_stream"
}

​
Heartbeat

Apps must send heartbeat messages to maintain their connection: 
{
  "type": "HEARTBEAT",
  "timestamp": 1234567890
}
If no heartbeat is received for 20 seconds, the app enters a grace period before being disconnected.

​
Implementation Details

  • Service: AppWebSocketService (packages/cloud/src/services/websocket/websocket-app.service.ts)
  • Manager: AppManager handles app lifecycle and state
  • Authentication: API key validation against app database
  • Subscriptions: SubscriptionService manages data stream routing

​
SDK Usage

The MentraOS SDK handles all WebSocket communication automatically: 
import { AppSession } from '@mentraos/sdk';

const session = new AppSession({
  packageName: 'com.example.app',
  apiKey: process.env.MENTRAOS_API_KEY
});

// Connect to session
await session.connect(sessionId, userId);

// Subscribe to events
session.events.onTranscription((data) => {
  console.log('User said:', data.text);
});

// Send display request
await session.layouts.showTextWall('Hello World!');