# Sirius Scrape API

HTTP API micro-SaaS untuk **scraping & data-extraction** tanpa mengelola browser/proxy sendiri.
Satu endpoint untuk metadata halaman, ekstraksi via CSS-selector, dan fetch JSON API dengan JSONPath.
Dibangun dengan mitigasi anti-SSRF (manual DNS + IP pinning + revalidasi tiap redirect hop).

- Base URL (contoh): `https://api.siriusx.id`
- Format: JSON in, JSON out
- Auth: header `x-api-key`
- Versi: `v1`

---

## 1. Autentikasi

Setiap request ke endpoint `/v1/*` wajib menyertakan API key pada header:

```
x-api-key: sk_live_xxxxxxxxxxxx
```

Jika key tidak ada atau tidak valid, respons `401`:

```json
{ "ok": false, "error": "unauthorized: valid x-api-key required" }
```

> Catatan: bila server dijalankan tanpa `API_KEYS` (mode lokal/dev), auth dilewati dan
> semua request dianggap tenant `anonymous`. Untuk produksi selalu set `API_KEYS`.

---

## 2. Rate limit

Rate limit dihitung per API key memakai sliding window.

| Header respons          | Arti                                  |
|-------------------------|---------------------------------------|
| `x-ratelimit-remaining` | sisa kuota pada window saat ini        |
| `retry-after`           | detik menunggu (hanya saat `429`)      |

Default: `60 request / 60 detik`. Bila terlampaui, respons `429`:

```json
{ "ok": false, "error": "rate limit exceeded", "retryAfterSec": 42 }
```

---

## 3. Bentuk respons umum

Sukses:

```json
{ "ok": true, "data": { /* ... */ } }
```

Gagal:

```json
{ "ok": false, "error": "pesan error" }
```

---

## 4. Endpoints

### GET /health  (publik, tanpa auth)

```bash
curl https://api.siriusx.id/health
```

```json
{
  "ok": true,
  "service": "api-server",
  "endpoints": ["/health", "POST /v1/scrape", "POST /v1/extract", "POST /v1/json"]
}
```

---

### POST /v1/scrape - metadata halaman

Ambil metadata terstruktur dari sebuah URL: `title`, `description`, `canonical`,
Open Graph (`og:*`), dan heading `h1/h2/h3`.

Body:

| field | tipe   | wajib | keterangan          |
|-------|--------|-------|---------------------|
| `url` | string | ya    | URL halaman HTML     |

```bash
curl -X POST https://api.siriusx.id/v1/scrape \
  -H "x-api-key: sk_live_xxx" \
  -H "content-type: application/json" \
  -d '{"url":"https://example.com"}'
```

```json
{
  "ok": true,
  "data": {
    "status": 200,
    "metadata": {
      "title": "Example Domain",
      "description": null,
      "canonical": null,
      "og": {},
      "h1": ["Example Domain"],
      "h2": [],
      "h3": []
    }
  }
}
```

---

### POST /v1/extract - ekstraksi via CSS selector

Ekstrak elemen dari HTML memakai CSS-selector ringan. Bisa memberi `html` langsung
atau `url` (server akan fetch). Selector yang didukung: `tag`, `.class`, `#id`,
`[attr]`, `[attr=val]`, dan descendant (dipisah spasi, dievaluasi pada simple-selector terakhir).

Body:

| field      | tipe   | wajib | keterangan                                    |
|------------|--------|-------|-----------------------------------------------|
| `selector` | string | ya    | CSS-selector ringan                            |
| `url`      | string | *     | sumber HTML (bila `html` kosong)               |
| `html`     | string | *     | HTML mentah (alternatif dari `url`)            |
| `limit`    | number | tidak | batasi jumlah item yang dikembalikan           |

`*` = salah satu dari `html` atau `url` wajib ada.

```bash
curl -X POST https://api.siriusx.id/v1/extract \
  -H "x-api-key: sk_live_xxx" \
  -H "content-type: application/json" \
  -d '{"url":"https://news.ycombinator.com","selector":"a.titlelink","limit":5}'
```

```json
{
  "ok": true,
  "data": {
    "count": 5,
    "items": [
      { "tag": "a", "text": "Judul...", "attrs": { "href": "https://..." }, "html": "Judul..." }
    ]
  }
}
```

---

### POST /v1/json - fetch JSON API + JSONPath-lite

Fetch endpoint JSON eksternal (GET/POST/dll) lalu ambil bagian tertentu via JSONPath-lite.
Dukungan path: dot (`a.b`), index (`[0]`, negatif `[-1]`), wildcard (`*`).

Body:

| field     | tipe          | wajib | keterangan                              |
|-----------|---------------|-------|-----------------------------------------|
| `url`     | string        | ya    | endpoint JSON                            |
| `method`  | string        | tidak | default `GET`                            |
| `headers` | object        | tidak | header tambahan                          |
| `body`    | string/object | tidak | body request (object diserialisasi JSON) |
| `path`    | string        | tidak | JSONPath-lite untuk memilih bagian hasil |

```bash
curl -X POST https://api.siriusx.id/v1/json \
  -H "x-api-key: sk_live_xxx" \
  -H "content-type: application/json" \
  -d '{"url":"https://api.github.com/repos/nodejs/node","path":"stargazers_count"}'
```

```json
{ "ok": true, "data": { "status": 200, "result": 108000 } }
```

---

## 5. Kode error

| HTTP | error                                         | penyebab                                   |
|------|-----------------------------------------------|--------------------------------------------|
| 400  | `url required` / `selector required`          | field wajib tidak ada                      |
| 400  | `html or url required`                        | `/v1/extract` tanpa sumber HTML            |
| 400  | `invalid JSON body`                           | body bukan JSON valid                      |
| 401  | `unauthorized: valid x-api-key required`      | API key hilang/salah                       |
| 405  | `method not allowed`                          | endpoint `/v1/*` harus POST                |
| 413  | `body too large`                              | request body > `MAX_BODY`                  |
| 429  | `rate limit exceeded`                         | kuota rate-limit habis                     |
| 502  | `upstream did not return valid JSON`          | `/v1/json` upstream bukan JSON             |
| 502  | `blocked IP literal` / `resolved to blocked IP` | target SSRF (private/reserved IP) ditolak |
| 502  | `timeout` / `too many redirects`              | masalah jaringan upstream                  |

---

## 6. Keamanan (SSRF mitigation)

Semua fetch keluar melewati `safeFetch`:

- Resolve DNS manual, tolak IP privat/reserved (IPv4 & IPv6, termasuk IPv4-mapped & CGNAT).
- Pin koneksi ke IP tervalidasi (`host = IP`, `servername = hostname` untuk SNI benar).
- Redirect ditangani manual, DNS/IP **direvalidasi tiap hop** (cegah DNS-rebinding & redirect-to-internal).
- Hanya `http:`/`https:`, timeout, dan cap ukuran body respons.

> Ini mitigasi defense-in-depth, bukan jaminan absolut. Deploy tetap di belakang egress firewall bila memungkinkan.

---

## 7. Konfigurasi (ENV)

| ENV              | default   | keterangan                                   |
|------------------|-----------|----------------------------------------------|
| `PORT`           | `8787`    | port listen                                  |
| `API_KEYS`       | -         | `key1:free,key2:pro` (tier opsional)         |
| `RATE_LIMIT`     | `60`      | request per window per key                   |
| `RATE_WINDOW_MS` | `60000`   | ukuran window (ms)                           |
| `MAX_BODY`       | `262144`  | max request body (byte)                      |
| `FETCH_TIMEOUT`  | `10000`   | timeout fetch upstream (ms)                  |
| `FETCH_MAX_BODY` | `2097152` | cap ukuran body respons upstream (byte)      |

Menjalankan:

```bash
API_KEYS="sk_live_abc:pro,sk_test_xyz:free" RATE_LIMIT=120 node server.cjs
```