Glasses WebSocket

Endpoint

wss://api.mentra.glass/glasses-ws

wss://devapi.mentra.glass/glasses-ws

ws://localhost:8002/glasses-ws

Authentication

The glasses WebSocket requires a JWT token in the Authorization header. This token is obtained from the mobile app after user authentication.

Headers

{
  "Authorization": "Bearer <coreToken>",
  "Host": "cloud.mentraos.com",
  "Upgrade": "websocket",
  "Connection": "Upgrade"
}

JWT Token Structure

The coreToken must contain:

sub: User identifier
email: User’s email address
organizations: Array of user’s organizations (optional)
defaultOrg: Default organization (optional)

Connection Flow

Authentication

Mobile app sends JWT token in Authorization header

Validation

Server validates JWT token using AUGMENTOS_AUTH_JWT_SECRET

Session Creation

If valid, creates or retrieves UserSession for the user

Connection Init

Client must send CONNECTION_INIT message within 30 seconds

Message Types

Client → Server Messages

All message types are defined in packages/sdk/src/types/messages/glasses-to-cloud.ts:

CONNECTION_INIT

{
  "type": "CONNECTION_INIT",
  "userId": "user@example.com",    // Optional
  "coreToken": "eyJhbGci..."       // Optional
}

BUTTON_PRESS

{
  "type": "BUTTON_PRESS",
  "buttonId": "main",
  "pressType": "short" // or "long"
}

HEAD_POSITION

{
  "type": "HEAD_POSITION",
  "position": "up" // or "down"
}

LOCATION_UPDATE

{
  "type": "LOCATION_UPDATE",
  "lat": 37.7749,
  "lng": -122.4194,
  "accuracy": 5.0,
  "correlationId": "loc-123"
}

PHOTO_TAKEN

{
  "type": "PHOTO_TAKEN",
  "photoData": ArrayBuffer,
  "mimeType": "image/jpeg",
  "timestamp": "2024-01-20T10:30:00Z"
}

START_APP / STOP_APP

{
  "type": "START_APP",
  "packageName": "com.example.app"
}

Server → Client Messages

All message types are defined in packages/sdk/src/types/messages/cloud-to-glasses.ts:

CONNECTION_ACK

{
  "type": "CONNECTION_ACK",
  "sessionId": "session-123",
  "userSession": {  // Partial<UserSession> from transformUserSessionForClient
    "userId": "user@example.com",
    "startTime": "2024-01-01T00:00:00Z",
    "activeAppSessions": ["com.app1", "com.app2"],
    "loadingApps": [],
    "appSubscriptions": {
      "com.app1": ["audio", "transcription"],
      "com.app2": ["notification"]
    },
    "requiresAudio": true,
    "minimumTranscriptionLanguages": ["en-US"],
    "isTranscribing": true
  },
  "timestamp": "2024-01-01T00:00:00Z"
}

CONNECTION_ERROR

{
  "type": "CONNECTION_ERROR",
  "error": "Invalid authentication token"
}

DISPLAY_EVENT

{
  "type": "DISPLAY_EVENT",
  "layout": {
    "type": "TEXT_WALL",
    "text": "Hello World"
  }
}

PHOTO_REQUEST

{
  "type": "PHOTO_REQUEST",
  "requestId": "photo-123",
  "settings": {
    "resolution": "HIGH"
  }
}

Error Handling

Authentication Errors

If authentication fails, the server responds with HTTP 401:

HTTP/1.1 401 Unauthorized
Content-Type: application/json

{
  "type": "CONNECTION_ERROR",
  "error": "Invalid or expired token"
}

Connection Timeout

If CONNECTION_INIT is not received within 30 seconds:

{
  "type": "CONNECTION_ERROR",
  "error": "Connection initialization timeout"
}

Heartbeat

The server sends WebSocket ping frames every 10 seconds to check connection health. The client must respond with pong frames or the connection will be closed.

Implementation Details

Service: GlassesWebSocketService (packages/cloud/src/services/websocket/websocket-glasses.service.ts)
Authentication: JWT validation in setupWebSocketServers method
Session Management: UserSession class manages the connection lifecycle
Message Routing: Messages are routed to appropriate managers in the UserSession

Example Connection

const ws = new WebSocket('wss://cloud.mentraos.com/glasses-ws', {
  headers: {
    'Authorization': `Bearer ${coreToken}`
  }
});

ws.on('open', () => {
  // Send CONNECTION_INIT immediately
  ws.send(JSON.stringify({
    type: 'CONNECTION_INIT',
    userId: 'user@example.com',      // Optional
    coreToken: 'eyJhbGci...'         // Optional
  }));
});

ws.on('message', (data) => {
  const message = JSON.parse(data);
  
  switch (message.type) {
    case 'CONNECTION_ACK':
      console.log('Connected!', message.sessionId);
      break;
    case 'DISPLAY_EVENT':
      // Show content on glasses
      showOnDisplay(message.layout);
      break;
  }
});

Overview

WebSocket Endpoints

REST Endpoints

Endpoint

Authentication

Headers

JWT Token Structure

Connection Flow

Message Types

Client → Server Messages

CONNECTION_INIT

BUTTON_PRESS

HEAD_POSITION

LOCATION_UPDATE

PHOTO_TAKEN

START_APP / STOP_APP

Server → Client Messages

CONNECTION_ACK

CONNECTION_ERROR

DISPLAY_EVENT

PHOTO_REQUEST

Error Handling

Authentication Errors

Connection Timeout

Heartbeat

Implementation Details

Example Connection

​Endpoint

​Authentication

​Headers

​JWT Token Structure

​Connection Flow

​Message Types

​Client → Server Messages

​CONNECTION_INIT

​BUTTON_PRESS

​HEAD_POSITION

​LOCATION_UPDATE

​PHOTO_TAKEN

​START_APP / STOP_APP

​Server → Client Messages

​CONNECTION_ACK

​CONNECTION_ERROR

​DISPLAY_EVENT

​PHOTO_REQUEST

​Error Handling

​Authentication Errors

​Connection Timeout

​Heartbeat

​Implementation Details

​Example Connection

Endpoint

Authentication

Headers

JWT Token Structure

Connection Flow

Message Types

Client → Server Messages

CONNECTION_INIT

BUTTON_PRESS

HEAD_POSITION

LOCATION_UPDATE

PHOTO_TAKEN

START_APP / STOP_APP

Server → Client Messages

CONNECTION_ACK

CONNECTION_ERROR

DISPLAY_EVENT

PHOTO_REQUEST

Error Handling

Authentication Errors

Connection Timeout

Heartbeat

Implementation Details

Example Connection