App WebSocket

Endpoint

wss://api.mentra.glass/app-ws

wss://devapi.mentra.glass/app-ws

ws://localhost:8002/app-ws

Authentication

Apps authenticate using a JWT token in the Authorization header, along with session information in custom headers.

Headers

{
  "Authorization": "Bearer <appJwt>",
  "x-user-id": "user@example.com",
  "x-session-id": "session-123",
  "Host": "cloud.mentraos.com",
  "Upgrade": "websocket",
  "Connection": "Upgrade"
}

JWT Token Structure

The appJwt must contain:

packageName: The app’s package identifier
apiKey: The app’s API key

Connection Flow

Webhook Notification

App receives webhook when user starts the app

WebSocket Connection

App connects to WebSocket with JWT and session headers

Authentication

Server validates JWT token and verifies API key

Session Binding

App is bound to the user’s active session

Connection Init

App sends CONNECTION_INIT to complete handshake

Message Types

App → Cloud Messages

All message types are defined in packages/sdk/src/types/messages/app-to-cloud.ts:

CONNECTION_INIT

{
  "type": "CONNECTION_INIT",
  "packageName": "com.example.app",
  "sessionId": "session-123",
  "apiKey": "app_api_key_here"
}

SUBSCRIPTION_UPDATE

{
  "type": "SUBSCRIPTION_UPDATE",
  "subscriptions": [
    {
      "type": "transcription",
      "config": {
        "languages": ["en-US", "es-ES"],
        "interimResults": true
      }
    },
    {
      "type": "button_press",
      "config": {
        "buttons": ["main"]
      }
    }
  ]
}

DISPLAY_REQUEST

{
  "type": "DISPLAY_REQUEST",
  "packageName": "com.example.app",
  "view": "MAIN",
  "layout": {
    "type": "TEXT_WALL",
    "text": "Hello from app!"
  },
  "durationMs": 5000,
  "forceDisplay": false
}

PHOTO_REQUEST

{
  "type": "PHOTO_REQUEST",
  "requestId": "photo-456",
  "saveToGallery": false,
  "customWebhookUrl": "https://app.example.com/photo-webhook"
}

AUDIO_PLAY_REQUEST

{
  "type": "AUDIO_PLAY_REQUEST",
  "requestId": "audio-789",
  "audioUrl": "https://example.com/sound.mp3",
  "volume": 0.8
}

Cloud → App Messages

All message types are defined in packages/sdk/src/types/messages/cloud-to-app.ts:

CONNECTION_ACK

{
  "type": "CONNECTION_ACK",
  "settings": [],           // App-specific settings
  "mentraosSettings": {},   // System settings
  "config": {},             // App config
  "capabilities": {
    "hasDisplay": true,
    "hasCamera": true,
    "hasMicrophone": true,
    "hasSpeaker": true
  }
}

DATA_STREAM

{
  "type": "DATA_STREAM",
  "streamType": "transcription",
  "data": {
    "type": "transcription",
    "text": "Hello world",
    "isFinal": true,
    "transcribeLanguage": "en-US",
    "startTime": 1234567890,
    "endTime": 1234567892,
    "confidence": 0.95
  }
}

SETTINGS_UPDATE

{
  "type": "SETTINGS_UPDATE",
  "settings": {
    "theme": "dark",
    "language": "en-US"
  }
}

APP_STOPPED

{
  "type": "APP_STOPPED",
  "reason": "USER_REQUEST"
}

Subscription System

Apps must subscribe to the data streams they want to receive. Available stream types are defined in packages/sdk/src/types/streams.ts:

Hardware Streams

button_press - Button press events
head_position - Head position changes
location_update - GPS location updates
vps_coordinates - Visual positioning system

Audio Streams

transcription - Speech-to-text results
translation - Real-time translation
VAD - Voice activity detection
audio_chunk - Raw audio data

Phone Streams

phone_notification - Incoming notifications
phone_notification_dismissed - Dismissal events
calendar_event - Calendar updates

System Streams

start_app - App start requests
stop_app - App stop requests
core_status_update - System status

Media Streams

photo_taken - Photo capture events
rtmp_stream_status - RTMP streaming status
managed_stream_status - Managed streaming

Error Handling

Authentication Errors

{
  "type": "CONNECTION_ERROR",
  "error": "Invalid API key"
}

Subscription Errors

{
  "type": "SUBSCRIPTION_ERROR",
  "error": "Invalid stream type: unknown_stream"
}

Heartbeat

Apps must send heartbeat messages to maintain their connection:

{
  "type": "HEARTBEAT",
  "timestamp": 1234567890
}

If no heartbeat is received for 20 seconds, the app enters a grace period before being disconnected.

Implementation Details

Service: AppWebSocketService (packages/cloud/src/services/websocket/websocket-app.service.ts)
Manager: AppManager handles app lifecycle and state
Authentication: API key validation against app database
Subscriptions: SubscriptionService manages data stream routing

SDK Usage

The MentraOS SDK handles all WebSocket communication automatically:

import { AppSession } from '@mentraos/sdk';

const session = new AppSession({
  packageName: 'com.example.app',
  apiKey: process.env.MENTRAOS_API_KEY
});

// Connect to session
await session.connect(sessionId, userId);

// Subscribe to events
session.events.onTranscription((data) => {
  console.log('User said:', data.text);
});

// Send display request
await session.layouts.showTextWall('Hello World!');

Overview

WebSocket Endpoints

REST Endpoints

Endpoint

Authentication

Headers

JWT Token Structure

Connection Flow

Message Types

App → Cloud Messages

CONNECTION_INIT

SUBSCRIPTION_UPDATE

DISPLAY_REQUEST

PHOTO_REQUEST

AUDIO_PLAY_REQUEST

Cloud → App Messages

CONNECTION_ACK

DATA_STREAM

SETTINGS_UPDATE

APP_STOPPED

Subscription System

Hardware Streams

Audio Streams

Phone Streams

System Streams

Media Streams

Error Handling

Authentication Errors

Subscription Errors

Heartbeat

Implementation Details

SDK Usage

​Endpoint

​Authentication

​Headers

​JWT Token Structure

​Connection Flow

​Message Types

​App → Cloud Messages

​CONNECTION_INIT

​SUBSCRIPTION_UPDATE

​DISPLAY_REQUEST

​PHOTO_REQUEST

​AUDIO_PLAY_REQUEST

​Cloud → App Messages

​CONNECTION_ACK

​DATA_STREAM

​SETTINGS_UPDATE

​APP_STOPPED

​Subscription System

​Hardware Streams

​Audio Streams

​Phone Streams

​System Streams

​Media Streams

​Error Handling

​Authentication Errors

​Subscription Errors

​Heartbeat

​Implementation Details

​SDK Usage

Endpoint

Authentication

Headers

JWT Token Structure

Connection Flow

Message Types

App → Cloud Messages

CONNECTION_INIT

SUBSCRIPTION_UPDATE

DISPLAY_REQUEST

PHOTO_REQUEST

AUDIO_PLAY_REQUEST

Cloud → App Messages

CONNECTION_ACK

DATA_STREAM

SETTINGS_UPDATE

APP_STOPPED

Subscription System

Hardware Streams

Audio Streams

Phone Streams

System Streams

Media Streams

Error Handling

Authentication Errors

Subscription Errors

Heartbeat

Implementation Details

SDK Usage