> ## Documentation Index
> Fetch the complete documentation index at: https://cloud-docs.mentra.glass/llms.txt
> Use this file to discover all available pages before exploring further.

# Glasses WebSocket

> WebSocket endpoint for smart glasses connections via mobile app

## Endpoint

<CodeGroup>
  ```bash Production theme={null}
  wss://api.mentra.glass/glasses-ws
  ```

  ```bash Development theme={null}
  wss://devapi.mentra.glass/glasses-ws
  ```

  ```bash Local theme={null}
  ws://localhost:8002/glasses-ws
  ```
</CodeGroup>

## Authentication

The glasses WebSocket requires a JWT token in the Authorization header. This token is obtained from the mobile app after user authentication.

### Headers

```typescript theme={null}
{
  "Authorization": "Bearer <coreToken>",
  "Host": "cloud.mentraos.com",
  "Upgrade": "websocket",
  "Connection": "Upgrade"
}
```

### JWT Token Structure

The coreToken must contain:

* `sub`: User identifier
* `email`: User's email address
* `organizations`: Array of user's organizations (optional)
* `defaultOrg`: Default organization (optional)

## Connection Flow

<Steps>
  <Step title="Authentication">
    Mobile app sends JWT token in Authorization header
  </Step>

  <Step title="Validation">
    Server validates JWT token using `AUGMENTOS_AUTH_JWT_SECRET`
  </Step>

  <Step title="Session Creation">
    If valid, creates or retrieves UserSession for the user
  </Step>

  <Step title="Connection Init">
    Client must send CONNECTION\_INIT message within 30 seconds
  </Step>
</Steps>

## Message Types

### Client → Server Messages

All message types are defined in `packages/sdk/src/types/messages/glasses-to-cloud.ts`:

#### CONNECTION\_INIT

```json theme={null}
{
  "type": "CONNECTION_INIT",
  "userId": "user@example.com",    // Optional
  "coreToken": "eyJhbGci..."       // Optional
}
```

#### BUTTON\_PRESS

```json theme={null}
{
  "type": "BUTTON_PRESS",
  "buttonId": "main",
  "pressType": "short" // or "long"
}
```

#### HEAD\_POSITION

```json theme={null}
{
  "type": "HEAD_POSITION",
  "position": "up" // or "down"
}
```

#### LOCATION\_UPDATE

```json theme={null}
{
  "type": "LOCATION_UPDATE",
  "lat": 37.7749,
  "lng": -122.4194,
  "accuracy": 5.0,
  "correlationId": "loc-123"
}
```

#### PHOTO\_TAKEN

```json theme={null}
{
  "type": "PHOTO_TAKEN",
  "photoData": ArrayBuffer,
  "mimeType": "image/jpeg",
  "timestamp": "2024-01-20T10:30:00Z"
}
```

#### START\_APP / STOP\_APP

```json theme={null}
{
  "type": "START_APP",
  "packageName": "com.example.app"
}
```

### Server → Client Messages

All message types are defined in `packages/sdk/src/types/messages/cloud-to-glasses.ts`:

#### CONNECTION\_ACK

```json theme={null}
{
  "type": "CONNECTION_ACK",
  "sessionId": "session-123",
  "userSession": {  // Partial<UserSession> from transformUserSessionForClient
    "userId": "user@example.com",
    "startTime": "2024-01-01T00:00:00Z",
    "activeAppSessions": ["com.app1", "com.app2"],
    "loadingApps": [],
    "appSubscriptions": {
      "com.app1": ["audio", "transcription"],
      "com.app2": ["notification"]
    },
    "requiresAudio": true,
    "minimumTranscriptionLanguages": ["en-US"],
    "isTranscribing": true
  },
  "timestamp": "2024-01-01T00:00:00Z"
}
```

#### CONNECTION\_ERROR

```json theme={null}
{
  "type": "CONNECTION_ERROR",
  "error": "Invalid authentication token"
}
```

#### DISPLAY\_EVENT

```json theme={null}
{
  "type": "DISPLAY_EVENT",
  "layout": {
    "type": "TEXT_WALL",
    "text": "Hello World"
  }
}
```

#### PHOTO\_REQUEST

```json theme={null}
{
  "type": "PHOTO_REQUEST",
  "requestId": "photo-123",
  "settings": {
    "resolution": "HIGH"
  }
}
```

## Error Handling

### Authentication Errors

If authentication fails, the server responds with HTTP 401:

```http theme={null}
HTTP/1.1 401 Unauthorized
Content-Type: application/json

{
  "type": "CONNECTION_ERROR",
  "error": "Invalid or expired token"
}
```

### Connection Timeout

If CONNECTION\_INIT is not received within 30 seconds:

```json theme={null}
{
  "type": "CONNECTION_ERROR",
  "error": "Connection initialization timeout"
}
```

## Heartbeat

The server sends WebSocket ping frames every 10 seconds to check connection health. The client must respond with pong frames or the connection will be closed.

## Implementation Details

* **Service**: `GlassesWebSocketService` (packages/cloud/src/services/websocket/websocket-glasses.service.ts)
* **Authentication**: JWT validation in `setupWebSocketServers` method
* **Session Management**: `UserSession` class manages the connection lifecycle
* **Message Routing**: Messages are routed to appropriate managers in the UserSession

## Example Connection

```typescript theme={null}
const ws = new WebSocket('wss://cloud.mentraos.com/glasses-ws', {
  headers: {
    'Authorization': `Bearer ${coreToken}`
  }
});

ws.on('open', () => {
  // Send CONNECTION_INIT immediately
  ws.send(JSON.stringify({
    type: 'CONNECTION_INIT',
    userId: 'user@example.com',      // Optional
    coreToken: 'eyJhbGci...'         // Optional
  }));
});

ws.on('message', (data) => {
  const message = JSON.parse(data);
  
  switch (message.type) {
    case 'CONNECTION_ACK':
      console.log('Connected!', message.sessionId);
      break;
    case 'DISPLAY_EVENT':
      // Show content on glasses
      showOnDisplay(message.layout);
      break;
  }
});
```