---
sidebar_position: 2
---

# Data Monitoring

## Overview

The data monitoring module is responsible for acquiring, normalizing, and storing market data that feeds into the analysis engine.

## Requirements

- **Latency**: Near real-time (< 1 minute delay)
- **Coverage**: Forex major pairs + selected stocks/indices
- **Granularity**: 1-minute candles, tick data where available
- **History**: Minimum 30 days rolling window for backtesting
- **Reliability**: Automatic reconnection, data gap detection

## Watched Instruments

### Forex Pairs (Initial)
| Pair | Session | Notes |
|---|---|---|
| EUR/USD | London + NY | Most liquid, tight spreads |
| GBP/USD | London + NY | High volatility windows |
| USD/JPY | Tokyo + London | Trend-following opportunities |
| AUD/USD | Sydney + London | Commodity correlation |

### Indices (Future)
| Index | Session | Notes |
|---|---|---|
| S&P 500 (SPX) | NY | Broad market sentiment |
| NASDAQ 100 (NDX) | NY | Tech sector proxy |

## Data Pipeline

```
┌──────────────┐     ┌──────────────┐     ┌──────────────┐
│  API Source  │────▶│  Normalizer  │────▶│   Storage    │
│              │     │              │     │              │
│ • REST poll  │     │ • OHLCV fmt  │     │ • SQLite/TS  │
│ • WebSocket  │     │ • Timestamps │     │ • Partitioned│
│ • Fallback   │     │ • Validation │     │ • Indexed    │
└──────────────┘     └──────────────┘     └──────────────┘
```

## Data Schema

```sql
CREATE TABLE candles (
    id INTEGER PRIMARY KEY,
    symbol TEXT NOT NULL,
    timeframe TEXT NOT NULL,  -- '1m', '5m', '15m', '1h'
    timestamp DATETIME NOT NULL,
    open REAL NOT NULL,
    high REAL NOT NULL,
    low REAL NOT NULL,
    close REAL NOT NULL,
    volume REAL,
    created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
    UNIQUE(symbol, timeframe, timestamp)
);

CREATE INDEX idx_candles_symbol_time ON candles(symbol, timeframe, timestamp DESC);
```

## Health Checks

- Data freshness monitoring (alert if > 5 min stale)
- Gap detection and backfill
- Source failover (primary → secondary API)
- Rate limit tracking per API key