Skip to main content

Documentation Index

Fetch the complete documentation index at: https://cloud-docs.mentra.glass/llms.txt

Use this file to discover all available pages before exploring further.

​
Endpoint

wss://api.mentra.glass/glasses-ws

​
Authentication

The glasses WebSocket requires a JWT token in the Authorization header. This token is obtained from the mobile app after user authentication.

​
Headers

{
  "Authorization": "Bearer <coreToken>",
  "Host": "cloud.mentraos.com",
  "Upgrade": "websocket",
  "Connection": "Upgrade"
}

​
JWT Token Structure

The coreToken must contain:
  • sub: User identifier
  • email: User’s email address
  • organizations: Array of user’s organizations (optional)
  • defaultOrg: Default organization (optional)

​
Connection Flow

1

Authentication

Mobile app sends JWT token in Authorization header
2

Validation

Server validates JWT token using AUGMENTOS_AUTH_JWT_SECRET
3

Session Creation

If valid, creates or retrieves UserSession for the user
4

Connection Init

Client must send CONNECTION_INIT message within 30 seconds

​
Message Types

​
Client → Server Messages

All message types are defined in packages/sdk/src/types/messages/glasses-to-cloud.ts:

​
CONNECTION_INIT

{
  "type": "CONNECTION_INIT",
  "userId": "user@example.com",    // Optional
  "coreToken": "eyJhbGci..."       // Optional
}

​
BUTTON_PRESS

{
  "type": "BUTTON_PRESS",
  "buttonId": "main",
  "pressType": "short" // or "long"
}

​
HEAD_POSITION

{
  "type": "HEAD_POSITION",
  "position": "up" // or "down"
}

​
LOCATION_UPDATE

{
  "type": "LOCATION_UPDATE",
  "lat": 37.7749,
  "lng": -122.4194,
  "accuracy": 5.0,
  "correlationId": "loc-123"
}

​
PHOTO_TAKEN

{
  "type": "PHOTO_TAKEN",
  "photoData": ArrayBuffer,
  "mimeType": "image/jpeg",
  "timestamp": "2024-01-20T10:30:00Z"
}

​
START_APP / STOP_APP

{
  "type": "START_APP",
  "packageName": "com.example.app"
}

​
Server → Client Messages

All message types are defined in packages/sdk/src/types/messages/cloud-to-glasses.ts:

​
CONNECTION_ACK

{
  "type": "CONNECTION_ACK",
  "sessionId": "session-123",
  "userSession": {  // Partial<UserSession> from transformUserSessionForClient
    "userId": "user@example.com",
    "startTime": "2024-01-01T00:00:00Z",
    "activeAppSessions": ["com.app1", "com.app2"],
    "loadingApps": [],
    "appSubscriptions": {
      "com.app1": ["audio", "transcription"],
      "com.app2": ["notification"]
    },
    "requiresAudio": true,
    "minimumTranscriptionLanguages": ["en-US"],
    "isTranscribing": true
  },
  "timestamp": "2024-01-01T00:00:00Z"
}

​
CONNECTION_ERROR

{
  "type": "CONNECTION_ERROR",
  "error": "Invalid authentication token"
}

​
DISPLAY_EVENT

{
  "type": "DISPLAY_EVENT",
  "layout": {
    "type": "TEXT_WALL",
    "text": "Hello World"
  }
}

​
PHOTO_REQUEST

{
  "type": "PHOTO_REQUEST",
  "requestId": "photo-123",
  "settings": {
    "resolution": "HIGH"
  }
}

​
Error Handling

​
Authentication Errors

If authentication fails, the server responds with HTTP 401: 
HTTP/1.1 401 Unauthorized
Content-Type: application/json

{
  "type": "CONNECTION_ERROR",
  "error": "Invalid or expired token"
}

​
Connection Timeout

If CONNECTION_INIT is not received within 30 seconds: 
{
  "type": "CONNECTION_ERROR",
  "error": "Connection initialization timeout"
}

​
Heartbeat

The server sends WebSocket ping frames every 10 seconds to check connection health. The client must respond with pong frames or the connection will be closed.

​
Implementation Details

  • Service: GlassesWebSocketService (packages/cloud/src/services/websocket/websocket-glasses.service.ts)
  • Authentication: JWT validation in setupWebSocketServers method
  • Session Management: UserSession class manages the connection lifecycle
  • Message Routing: Messages are routed to appropriate managers in the UserSession

​
Example Connection

const ws = new WebSocket('wss://cloud.mentraos.com/glasses-ws', {
  headers: {
    'Authorization': `Bearer ${coreToken}`
  }
});

ws.on('open', () => {
  // Send CONNECTION_INIT immediately
  ws.send(JSON.stringify({
    type: 'CONNECTION_INIT',
    userId: 'user@example.com',      // Optional
    coreToken: 'eyJhbGci...'         // Optional
  }));
});

ws.on('message', (data) => {
  const message = JSON.parse(data);
  
  switch (message.type) {
    case 'CONNECTION_ACK':
      console.log('Connected!', message.sessionId);
      break;
    case 'DISPLAY_EVENT':
      // Show content on glasses
      showOnDisplay(message.layout);
      break;
  }
});