HL Collector API

Complete reference for the Hyperliquid market data collection API. Base URL: https://smartsui.io

Overview

HL Collector is a real-time data collection service for Hyperliquid perpetual futures. It captures full L2 order book snapshots (20 levels each side) and all trades via WebSocket, stores them in DuckDB, and exposes them through this REST API.

Base URL

https://smartsui.io

All endpoints are relative to this base. HTTPS is enforced via Nginx + Let's Encrypt.

Authentication

All endpoints except /api/health require authentication. Two methods are supported:

1. API Key (recommended for scripts/AI)

Pass via X-API-Key header. Create keys from the admin dashboard or POST /api/keys/create.

httpGET /api/status
X-API-Key: hlc_your_key_here

2. JWT Bearer Token (for browser sessions)

Obtain via POST /auth/login. Tokens expire after 24 hours. Refresh via POST /auth/refresh.

httpGET /api/status
Authorization: Bearer eyJhbGciOiJIUzI1NiIs...

Rate Limiting

100 requests per minute per IP address. Returns 429 Too Many Requests when exceeded.

Error Handling

All errors return JSON with a detail field:

json{
  "detail": "Authentication required"
}

Auth Endpoints

Authenticate and receive a JWT token.

json{
  "username": "your_username",
  "password": "your_password"
}

json{
  "token": "eyJhbGciOiJIUzI1NiIs...",
  "expires_in": 86400
}

Refresh a JWT token. Cannot be used with API keys.

json{
  "token": "eyJhbGciOiJIUzI1NiIs...",
  "expires_in": 86400
}

Data Endpoints

Health check. No auth required. Use for monitoring/uptime checks.

json{
  "status": "ok",              // "ok" or "degraded" (ws disconnected)
  "ws_connected": true,
  "last_message_age_s": 0.8,   // seconds since last WS message flushed
  "db_writable": true,
  "disk_pct": 6,               // disk usage percentage
  "assets": ["BTC", "ETH", "SOL"],
  "uptime_h": 12.5
}

Detailed server status with message counts and buffer state.

json{
  "uptime_s": 45000.3,
  "uptime_human": "12h 30m 0s",
  "ws_connected": true,
  "total_msgs": 285000,
  "msgs_per_sec": 6.3,
  "db_size_mb": 142.5,
  "db_path": "/data/hl_market.duckdb",
  "counts": {
    "book_received": 85000,
    "book_inserted": 85000,
    "trades_received": 200000,
    "trades_inserted": 198500
  },
  "flush_errors": 0,
  "buffers": {
    "book": 12,        // rows buffered, not yet flushed
    "trades": 34
  },
  "assets": ["BTC", "ETH", "SOL"]
}

Per-asset data integrity diagnostics. Shows row counts, intervals, completeness, and gap count over the last 24 hours.

json{
  "data_integrity": "OK",      // "OK" or "ERRORS"
  "per_asset": {
    "BTC": {
      "l2book": {
        "total_rows": 85000,
        "avg_interval_s": 0.5,
        "rows_last_hour": 7100,
        "expected_rows_last_hour": 7200,
        "completeness_last_hour_pct": 98.6,
        "last_insert": "2026-03-08T13:22:50.478532+00:00"
      },
      "trades": {
        "total_rows": 198000,
        "avg_interval_s": 0.22,
        "rows_last_hour": 16200,
        "expected_rows_last_hour": 25200,
        "completeness_last_hour_pct": 64.3,
        "last_insert": "2026-03-08T13:22:49.476686+00:00"
      },
      "gaps_24h": 2
    }
  }
}

Detailed stats for a single asset including live market data from the ring buffer cache.

json{
  "asset": "BTC",
  "l2book": {
    "total_rows": 85000,
    "date_range": {
      "start": "2026-03-08T13:15:00+00:00",
      "end": "2026-03-08T13:22:50+00:00"
    },
    "rows_today": 85000,
    "current_rate_per_sec": 2.0
  },
  "trades": {
    "total_rows": 198000,
    "date_range": { "start": "...", "end": "..." },
    "rows_today": 198000,
    "current_rate_per_sec": 4.5
  },
  "live": {
    "mid": 87250.5,
    "bid": 87250.0,
    "ask": 87251.0,
    "spread_bps": 0.11,
    "obi_20": 0.12,
    "last_trade": {
      "price": 87250.0,
      "side": "B",
      "size": 0.015
    }
  },
  "gaps_24h": [
    {"asset": "BTC", "type": "l2book", "gap_ms": 3200, "at": "2026-03-08T14:00:01+00:00"}
  ]
}

Last N order book snapshots from the in-memory ring buffer (up to 300). Full 20-level depth each side.

json[
  {
    "ts_ns": 1772976170000000000,
    "exchange_ts_ms": 1772976170000,
    "coin": "BTC",
    "best_bid": 87250.0,
    "best_ask": 87251.0,
    "mid": 87250.5,
    "microprice": 87250.3,
    "spread_bps": 0.11,
    "obi_5": 0.08,
    "obi_10": 0.12,
    "obi_20": 0.15,
    "total_bid_qty": 45.2,
    "total_ask_qty": 38.7,
    "bid_px_01": 87250.0, "bid_sz_01": 1.2,
    "ask_px_01": 87251.0, "ask_sz_01": 0.8,
    "bid_px_02": 87249.0, "bid_sz_02": 2.5,
    "ask_px_02": 87252.0, "ask_sz_02": 1.1,
    ...
  }
]

Last N trades from the in-memory ring buffer (up to 500).

json[
  {
    "ts_ns": 1772976176427660693,
    "exchange_ts_ms": 1772976176089,
    "coin": "BTC",
    "price": 87071.0,
    "size": 0.00018,
    "side": "A",
    "tid": 795991757588405,
    "buyer": "0x2cb5...fbe86",
    "seller": "0x98be...b941e4"
  }
]

Order Book Imbalance time series from the ring buffer. Lightweight subset of book data for charting OBI trends.

json[
  {
    "ts_ns": 1772976170000000000,
    "mid": 87250.5,
    "spread_bps": 0.11,
    "obi_5": 0.08,
    "obi_10": 0.12,
    "obi_20": 0.15,
    "microprice": 87250.3
  }
]

Bulk Download

Download historical data from DuckDB. Supports Parquet, CSV, and JSON output. Use this for backtesting and analysis.

bash# Last 2 hours of BTC book data as parquet
curl -H "X-API-Key: hlc_..." \
  "https://smartsui.io/api/bulk/BTC/l2book?last_hours=2" -o btc_book.parquet

# Last 1000 ETH trades as JSON
curl -H "X-API-Key: hlc_..." \
  "https://smartsui.io/api/bulk/ETH/trades?last_rows=1000&format=json"

# BTC trades between two times, only price+size+side columns, as CSV
curl -H "X-API-Key: hlc_..." \
  "https://smartsui.io/api/bulk/BTC/trades?after=2026-03-08T12:00:00Z&before=2026-03-08T13:00:00Z&columns=ts_ns,price,size,side&format=csv"

# Downsampled book data (every 10th row)
curl -H "X-API-Key: hlc_..." \
  "https://smartsui.io/api/bulk/BTC/l2book?last_hours=24&downsample=10" -o btc_ds.parquet

Virtual file listing. Returns data chunked into 15-minute windows with row counts and download URLs. Compatible with the legacy parquet-file interface.

json[
  {
    "path": "trades/BTC/BTC_2026-03-08_13-15.parquet",
    "name": "BTC_2026-03-08_13-15.parquet",
    "asset": "BTC",
    "dtype": "trades",
    "rows": 1884,
    "download_url": "/api/bulk/BTC/trades?after=2026-03-08T13:15:00+00:00&before=2026-03-08T13:30:00+00:00"
  }
]

Legacy download endpoint. Parses a filename like BTC_2026-03-08_12-00.parquet to extract the 15-minute time window and returns parquet data from DuckDB.

bashcurl -H "X-API-Key: hlc_..." \
  "https://smartsui.io/api/download/l2book/BTC/BTC_2026-03-08_13-15.parquet" -o book.parquet

Streaming

Server-Sent Events stream for real-time data. Each event is a JSON object. Sends keepalive comments every 15 seconds.

bash# Stream BTC book + trades
curl -H "X-API-Key: hlc_..." \
  "https://smartsui.io/api/stream/BTC?types=book,trade"

# Output:
# data: {"type":"book","coin":"BTC","mid":87250.5,...}
# data: {"type":"trade","coin":"BTC","price":87250.0,...}
# : keepalive

WebSocket feed for browser clients. Requires JWT token as query parameter (API keys not supported). Sends all assets in a single connection.

json// Init (first message)
{
  "type": "init",
  "data": {
    "assets": ["BTC", "ETH", "SOL"],
    "books": {
      "BTC": {
        "mid": 87250.5, "best_bid": 87250.0, "best_ask": 87251.0,
        "spread_bps": 0.11, "obi_5": 0.08, "obi_10": 0.12, "obi_20": 0.15,
        "microprice": 87250.3, "total_bid": 45.2, "total_ask": 38.7,
        "bids": [[87250.0, 1.2], [87249.0, 2.5], ...],  // top 10
        "asks": [[87251.0, 0.8], [87252.0, 1.1], ...]
      }
    },
    "trades": {
      "BTC": [
        {"coin": "BTC", "ts_ms": 1772976176089, "price": 87071.0,
         "size": 0.00018, "side": "A", "tid": 795991757588405,
         "buyer": "0x...", "seller": "0x..."}
      ]
    },
    "status": {
      "uptime_s": 45000.3, "ws_connected": true,
      "total_msgs": 285000, "msgs_per_sec": 6.3
    }
  }
}

// Book update
{"type": "book", "coin": "BTC", "mid": 87250.5, "spread_bps": 0.11, ...}

// Trade
{"type": "trade", "coin": "ETH", "price": 3420.5, "size": 1.2, "side": "B", ...}

// Status (every 5 seconds)
{
  "type": "status", "uptime_s": 45005.3, "uptime_human": "12h 30m 5s",
  "ws_connected": true, "total_msgs": 285032, "msgs_per_sec": 6.3,
  "db_size_mb": 142.5, "assets": ["BTC", "ETH", "SOL"]
}

// Ping (keepalive)
{"type": "ping"}

Management Endpoints

List active assets.

json{"assets": ["BTC", "ETH", "SOL"]}

Add a new asset to collect. Validates against Hyperliquid's available coins.

json{"coin": "DOGE"}

json{"success": true, "coin": "DOGE", "assets": ["BTC", "ETH", "SOL", "DOGE"]}

Remove an asset from collection. Historical data is retained in DuckDB.

json{"success": true, "assets": ["BTC", "ETH"]}

List API keys (prefix only, not full key).

json[
  {
    "id": "hlc_0eded9c82989",
    "name": "my-bot",
    "prefix": "hlc_0eded9c8...",
    "created": "2026-03-08T12:00:00+00:00",
    "last_used": "2026-03-08T13:22:00+00:00"
  }
]

Create a new API key. The full key is only shown once in the response.

json// Request
{"name": "my-trading-bot"}

// Response
{"key": "hlc_a1b2c3d4e5f6...", "name": "my-trading-bot"}

Revoke an API key. Use the id from GET /api/keys (first 16 chars).

json{"success": true}

Delete ALL data from both tables. Requires the reset password (separate from login password). Irreversible.

json// Request
{"password": "reset_password_here"}

// Response
{"success": true}

Data Schema: Book Snapshot

Each book snapshot contains computed metrics plus 20 levels of bid/ask depth. The DuckDB table name is book.

Full column list in order:

columnsts_ns, exchange_ts_ms, coin, best_bid, best_ask, mid, microprice, spread_bps,
obi_5, obi_10, obi_20, total_bid_qty, total_ask_qty,
bid_px_01, bid_sz_01, ask_px_01, ask_sz_01,
bid_px_02, bid_sz_02, ask_px_02, ask_sz_02,
... (repeating for levels 03-20)
bid_px_20, bid_sz_20, ask_px_20, ask_sz_20

Data Schema: Trade

Each trade is a single fill on Hyperliquid. The DuckDB table name is trades.

WebSocket Message Types

Python Examples

pythonimport requests
import pandas as pd
import io

BASE = "https://smartsui.io"
HEADERS = {"X-API-Key": "hlc_your_key_here"}

# ── Health check (no auth needed) ──
health = requests.get(f"{BASE}/api/health").json()
print(f"Status: {health['status']}, Assets: {health['assets']}")

# ── Server status ──
status = requests.get(f"{BASE}/api/status", headers=HEADERS).json()
print(f"Uptime: {status['uptime_human']}, Msgs/sec: {status['msgs_per_sec']}")
print(f"Book rows: {status['counts']['book_inserted']:,}")
print(f"Trade rows: {status['counts']['trades_inserted']:,}")

# ── Live book snapshots (last 60) ──
book = requests.get(f"{BASE}/api/book/BTC?n=60", headers=HEADERS).json()
df_book = pd.DataFrame(book)
print(df_book[["mid", "spread_bps", "obi_20"]].tail())

# ── Live trades (last 100) ──
trades = requests.get(f"{BASE}/api/trades/ETH?n=100", headers=HEADERS).json()
df_trades = pd.DataFrame(trades)
print(df_trades[["price", "size", "side"]].tail())

# ── OBI time series ──
obi = requests.get(f"{BASE}/api/obi/BTC", headers=HEADERS).json()
df_obi = pd.DataFrame(obi)

# ── Bulk download: last 4 hours of BTC book as parquet ──
r = requests.get(
    f"{BASE}/api/bulk/BTC/l2book?last_hours=4",
    headers=HEADERS
)
df = pd.read_parquet(io.BytesIO(r.content))
print(f"Downloaded {len(df):,} book rows, {len(df.columns)} columns")

# ── Bulk download: last 1000 trades as JSON ──
trades_json = requests.get(
    f"{BASE}/api/bulk/BTC/trades?last_rows=1000&format=json",
    headers=HEADERS
).json()
df_trades = pd.DataFrame(trades_json)

# ── Bulk download: specific time range, selected columns, as CSV ──
r = requests.get(
    f"{BASE}/api/bulk/ETH/trades"
    f"?after=2026-03-08T12:00:00Z"
    f"&before=2026-03-08T13:00:00Z"
    f"&columns=ts_ns,price,size,side"
    f"&format=csv",
    headers=HEADERS
)
df = pd.read_csv(io.StringIO(r.text))

# ── Per-asset stats ──
stats = requests.get(f"{BASE}/api/stats/BTC", headers=HEADERS).json()
print(f"BTC mid: {stats['live']['mid']}, spread: {stats['live']['spread_bps']}bps")

# ── Diagnostic (data integrity) ──
diag = requests.get(f"{BASE}/api/diagnostic", headers=HEADERS).json()
for asset, data in diag["per_asset"].items():
    book_pct = data["l2book"]["completeness_last_hour_pct"]
    print(f"{asset} book completeness: {book_pct}%")

# ── SSE streaming ──
import sseclient  # pip install sseclient-py
r = requests.get(
    f"{BASE}/api/stream/BTC?types=book,trade",
    headers=HEADERS,
    stream=True
)
client = sseclient.SSEClient(r)
for event in client.events():
    data = json.loads(event.data)
    if data.get("type") == "trade":
        print(f"Trade: {data['side']} {data['size']}@{data['price']}")

JavaScript Examples

javascriptconst BASE = "https://smartsui.io";
const HEADERS = { "X-API-Key": "hlc_your_key_here" };

// ── Fetch status ──
const status = await fetch(`${BASE}/api/status`, { headers: HEADERS })
  .then(r => r.json());
console.log(`Uptime: ${status.uptime_human}, ${status.msgs_per_sec} msg/s`);

// ── Live book ──
const book = await fetch(`${BASE}/api/book/BTC?n=60`, { headers: HEADERS })
  .then(r => r.json());
const latest = book[book.length - 1];
console.log(`BTC mid: ${latest.mid}, spread: ${latest.spread_bps}bps`);

// ── Bulk download as JSON ──
const trades = await fetch(
  `${BASE}/api/bulk/BTC/trades?last_rows=500&format=json`,
  { headers: HEADERS }
).then(r => r.json());

// ── SSE stream ──
const evtSource = new EventSource(
  `${BASE}/api/stream/BTC?types=book,trade`,
  // Note: EventSource doesn't support custom headers.
  // Use JWT token via query param workaround or fetch with ReadableStream.
);
evtSource.onmessage = (e) => {
  const data = JSON.parse(e.data);
  if (data.type === "book") {
    console.log(`BTC mid=${data.mid} spread=${data.spread_bps}bps`);
  }
};

// ── WebSocket feed ──
// First get JWT
const loginResp = await fetch(`${BASE}/auth/login`, {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({ username: "user", password: "pass" })
}).then(r => r.json());

const ws = new WebSocket(`wss://smartsui.io/ws/feed?token=${loginResp.token}`);
ws.onmessage = (e) => {
  const msg = JSON.parse(e.data);
  switch (msg.type) {
    case "init":
      console.log("Assets:", msg.data.assets);
      break;
    case "book":
      console.log(`${msg.coin} mid=${msg.mid}`);
      break;
    case "trade":
      console.log(`${msg.coin} ${msg.side} ${msg.size}@${msg.price}`);
      break;
  }
};

cURL Examples

bashKEY="hlc_your_key_here"

# Health (no auth)
curl https://smartsui.io/api/health

# Status
curl -H "X-API-Key: $KEY" https://smartsui.io/api/status

# Diagnostic
curl -H "X-API-Key: $KEY" https://smartsui.io/api/diagnostic

# Per-asset stats
curl -H "X-API-Key: $KEY" https://smartsui.io/api/stats/BTC

# Live book (last 60 snapshots)
curl -H "X-API-Key: $KEY" https://smartsui.io/api/book/BTC?n=60

# Live trades
curl -H "X-API-Key: $KEY" https://smartsui.io/api/trades/ETH?n=100

# OBI series
curl -H "X-API-Key: $KEY" https://smartsui.io/api/obi/SOL

# Bulk download as parquet
curl -H "X-API-Key: $KEY" \
  "https://smartsui.io/api/bulk/BTC/l2book?last_hours=2" -o btc.parquet

# Bulk download as JSON
curl -H "X-API-Key: $KEY" \
  "https://smartsui.io/api/bulk/BTC/trades?last_rows=100&format=json"

# List virtual files
curl -H "X-API-Key: $KEY" "https://smartsui.io/api/files?asset=BTC&dtype=trades"

# SSE stream
curl -N -H "X-API-Key: $KEY" "https://smartsui.io/api/stream/BTC?types=book,trade"

# Login (get JWT)
curl -X POST https://smartsui.io/auth/login \
  -H "Content-Type: application/json" \
  -d '{"username":"user","password":"pass"}'

# List assets
curl -H "X-API-Key: $KEY" https://smartsui.io/api/assets

# Add asset
curl -X POST -H "X-API-Key: $KEY" -H "Content-Type: application/json" \
  -d '{"coin":"DOGE"}' https://smartsui.io/api/assets/add

# Remove asset
curl -X DELETE -H "X-API-Key: $KEY" https://smartsui.io/api/assets/SOL

# Create API key
curl -X POST -H "X-API-Key: $KEY" -H "Content-Type: application/json" \
  -d '{"name":"my-bot"}' https://smartsui.io/api/keys/create

# List API keys
curl -H "X-API-Key: $KEY" https://smartsui.io/api/keys

# Delete API key
curl -X DELETE -H "X-API-Key: $KEY" https://smartsui.io/api/keys/hlc_0eded9c82989