Merge branch 'beta'

2025-06-16 16:28:50 +00:00 · 2023-02-24 23:56:03 -05:00
parent 0e6ff2b0de 30b8c387a2
commit 7ff40c52c1
49 changed files with 2043 additions and 386 deletions
--- a/7
+++ b/7
@ -4,7 +4,8 @@ WORKDIR /build
 RUN apt update && apt install -y --no-install-recommends \
    git g++ make pkg-config libtool ca-certificates \
    libyaml-perl libtemplate-perl libssl-dev zlib1g-dev \
-    liblmdb-dev libflatbuffers-dev libsecp256k1-dev libb2-dev
+    liblmdb-dev libflatbuffers-dev libsecp256k1-dev libb2-dev \
+    libzstd-dev

 COPY . .
 RUN git submodule update --init
@ -15,9 +16,9 @@ FROM ubuntu:jammy as runner
 WORKDIR /app

 RUN apt update && apt install -y --no-install-recommends \
-    liblmdb0 libflatbuffers1 libsecp256k1-0 libb2-1 \
+    liblmdb0 libflatbuffers1 libsecp256k1-0 libb2-1 libzstd1 \
    && rm -rf /var/lib/apt/lists/*

 COPY --from=build /build/strfry strfry
 ENTRYPOINT ["/app/strfry"]
-CMD ["relay"]
+CMD ["relay"]
--- a/2
+++ b/2
@ -3,4 +3,4 @@ OPT = -O3 -g

 include golpe/rules.mk

-LDLIBS += -lsecp256k1 -lb2
+LDLIBS += -lsecp256k1 -lb2 -lzstd
--- a/README.md
+++ b/README.md
@ -1,5 +1,7 @@
 # strfry - a nostr relay

+![strfry logo](docs/strfry.svg)
+
 strfry is a relay for the [nostr protocol](https://github.com/nostr-protocol/nostr)

 * Supports most applicable NIPs: 1, 9, 11, 12, 15, 16, 20, 22
@ -28,7 +30,7 @@ Either the full set of messages in the DB can be synced, or the results of one o

 A C++20 compiler is required, along with a few other common dependencies. On Debian/Ubuntu use these commands:

-    sudo apt install -y git build-essential libyaml-perl libtemplate-perl libssl-dev zlib1g-dev liblmdb-dev libflatbuffers-dev libsecp256k1-dev libb2-dev
+    sudo apt install -y git build-essential libyaml-perl libtemplate-perl libssl-dev zlib1g-dev liblmdb-dev libflatbuffers-dev libsecp256k1-dev libb2-dev libzstd-dev
    git submodule update --init
    make setup-golpe
    make -j4
--- a/32
+++ b/32
@ -1,5 +1,10 @@
-features
-  finish syncing
+0.1 release
+  when disk is full it should log warning but not crash
+  disable sync
+
+0.2 release
+  ? why isn't the LMDB mapping CLOEXEC
+  fix sync
    * logging of bytes up/down
    * up/both directions
    * error handling and reporting
@ -7,24 +12,21 @@ features
    * limit on number of concurrent sync requests
    * full-db scan limited by since/until
    * `strfry sync` command always takes at least 1 second due to batching delay. figure out better way to flush
-  bool values in config
-    config for compression
-    config for TCP keepalive
-  db versioning
-  document config options, detailed default config file
+
+features
  less verbose default logging
-  nice new config "units" feature, ie 1d instead of 86400
-  make it easier for a thread to setup a quadrable env
-  opt: PubkeyKind scans could be done index-only
+  multiple sync connections in one process/config
+  NIP-42 AUTH
+  slow-reader detection and back-pressure
+  ? relay block-list events
+  ? if a client disconnects, delete all its pending write messages

 rate limits
-  slow-reader detection and back-pressure
+  ! event writes per second per ip
  max connections per ip (nginx?)
  max bandwidth up/down (nginx?)
-  event writes per second per ip
-  max number of concurrent REQs per connection/ip
  ? limit on total number of events from a DBScan, not just per filter
+  ? time limit on DBScan

 misc
-  periodic reaping of disconnected sockets
-  ? websocket-level pings
+  ? periodic reaping of disconnected sockets (maybe autoping is doing this already)
--- a/docs/plugins.md
+++ b/docs/plugins.md
@ -0,0 +1,86 @@
+# Write policy plugins
+
+In order to reduce complexity, strfry's design attempts to keep policy logic out of its core relay functionality. Instead, this logic can be implemented by operators by installing a write policy plugin. Among other things, plugins can be used for the following:
+
+* White/black-lists (particular pubkeys can/can't post events)
+* Rate-limits
+* Spam filtering
+
+A plugin can be implemented in any programming language that supports reading lines from stdin, decoding JSON, and printing JSON to stdout. If a plugin is installed, strfry will send the event (along with some other information like IP address) to the plugin over stdin. The plugin should then decide what to do with it and print out a JSON object containing this decision.
+
+Whenever the script's modification-time changes, or the plugin settings in `strfry.conf` change, the plugin will be reloaded upon the next write attempt.
+
+If configured, When a plugin is loaded some number of recently stored events will be sent to it as a "lookback". This is useful for populating the initial rate-limiting state. Plugins should print nothing in response to a lookback message.
+
+
+## Input messages
+
+Input messages contain the following keys:
+
+* `type`: Either `new` or `lookback`
+* `event`: The event posted by the client, with all the required fields such as `id`, `pubkey`, etc
+* `receivedAt`: Unix timestamp of when this event was received by the relay
+* `sourceType`: Where this event came from. Typically will be `IP4` or `IP6`, but in lookback can also be `Import`, `Stream`, or `Sync`.
+* `sourceInfo`: Specifics of the event's source. Either an IP address or a relay URL (for stream/sync)
+
+
+## Output messages
+
+In response to `new` events, the plugin should print a JSONL message (minified JSON followed by a newline). It should contain the following keys:
+
+* `id`: The event ID taken from the `event.id` field of the input message
+* `action`: Either `accept`, `reject`, or `shadowReject`
+* `msg`: The NIP-20 response message to be sent to the client. Only used for `reject`
+
+
+## Example: Whitelist
+
+Here is a simple example `whitelist.js` plugin that will reject all events except for those in a whitelist:
+
+    #!/usr/bin/env node
+
+    const whiteList = {
+        '003ba9b2c5bd8afeed41a4ce362a8b7fc3ab59c25b6a1359cae9093f296dac01': true,
+    };
+
+    const rl = require('readline').createInterface({
+      input: process.stdin,
+      output: process.stdout,
+      terminal: false
+    });
+
+    rl.on('line', (line) => {
+        let req = JSON.parse(line);
+
+        if (req.type === 'lookback') { 
+            return; // do nothing
+        } 
+
+        if (req.type !== 'new') {
+            console.error("unexpected request type"); // will appear in strfry logs
+            return;
+        }
+
+        let res = { id: req.event.id }; // must echo the event's id
+
+        if (whiteList[req.event.pubkey]) {
+            res.action = 'accept';
+        } else {
+            res.action = 'reject';
+            res.msg = 'blocked: not on white-list';
+        }
+
+        console.log(JSON.stringify(res));
+    });
+
+To install:
+
+* Make the script executable: `chmod a+x whitelist.js`
+* In `strfry.conf`, configure `relay.writePolicy.plugin` to `./whitelist.js`
+
+
+## Notes
+
+* If applicable, you should ensure stdout is *line buffered* (for example, in perl use `$|++`).
+* If events are being rejected with `error: internal error`, then check the strfry logs. The plugin is misconfigured or failing.
+* When returning an action of `accept`, it doesn't necessarily guarantee that the event will be accepted. The regular strfry checks are still subsequently applied, such as expiration, deletion, etc.
--- a/docs/strfry.svg
+++ b/docs/strfry.svg
@ -0,0 +1,135 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<svg
+   xmlns:dc="http://purl.org/dc/elements/1.1/"
+   xmlns:cc="http://creativecommons.org/ns#"
+   xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
+   xmlns:svg="http://www.w3.org/2000/svg"
+   xmlns="http://www.w3.org/2000/svg"
+   xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
+   xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
+   width="273.42755"
+   height="164.88681"
+   viewBox="0 0 72.344371 43.626303"
+   version="1.1"
+   id="svg8"
+   inkscape:version="1.0.2 (e86c870879, 2021-01-15)"
+   sodipodi:docname="strfry.svg">
+  <defs
+     id="defs2" />
+  <sodipodi:namedview
+     id="base"
+     pagecolor="#ffffff"
+     bordercolor="#666666"
+     borderopacity="1.0"
+     inkscape:pageopacity="0.0"
+     inkscape:pageshadow="2"
+     inkscape:zoom="1.4"
+     inkscape:cx="425.62493"
+     inkscape:cy="332.10665"
+     inkscape:document-units="mm"
+     inkscape:current-layer="layer1"
+     inkscape:document-rotation="0"
+     showgrid="false"
+     inkscape:window-width="2256"
+     inkscape:window-height="1449"
+     inkscape:window-x="0"
+     inkscape:window-y="55"
+     inkscape:window-maximized="0"
+     inkscape:snap-global="false"
+     units="px"
+     fit-margin-top="0"
+     fit-margin-left="0"
+     fit-margin-right="0"
+     fit-margin-bottom="0" />
+  <metadata
+     id="metadata5">
+    <rdf:RDF>
+      <cc:Work
+         rdf:about="">
+        <dc:format>image/svg+xml</dc:format>
+        <dc:type
+           rdf:resource="http://purl.org/dc/dcmitype/StillImage" />
+        <dc:title></dc:title>
+      </cc:Work>
+    </rdf:RDF>
+  </metadata>
+  <g
+     inkscape:label="Layer 1"
+     inkscape:groupmode="layer"
+     id="layer1"
+     transform="translate(-0.47515038,-0.38460946)">
+    <path
+       style="fill:#000000;fill-rule:evenodd;stroke:#000000;stroke-width:1.05483;stroke-miterlimit:4;stroke-dasharray:none"
+       id="path10"
+       sodipodi:type="arc"
+       sodipodi:cx="20.230062"
+       sodipodi:cy="35.06189"
+       sodipodi:rx="23.492462"
+       sodipodi:ry="14.519141"
+       sodipodi:start="0"
+       sodipodi:end="3.1415927"
+       sodipodi:open="true"
+       sodipodi:arc-type="arc"
+       d="m 43.722525,35.06189 a 23.492462,14.519141 0 0 1 -11.746232,12.573945 23.492462,14.519141 0 0 1 -23.4924622,0 A 23.492462,14.519141 0 0 1 -3.2623997,35.06189"
+       transform="rotate(-16.074499)" />
+    <path
+       style="fill:none;stroke:#000000;stroke-width:0;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
+       d="M 6.7581523,34.98122 51.82955,21.99376"
+       id="path839" />
+    <path
+       style="fill:none;fill-rule:evenodd;stroke:#cccccc;stroke-width:1.05483;stroke-miterlimit:4;stroke-dasharray:none"
+       id="path10-3"
+       sodipodi:type="arc"
+       sodipodi:cx="21.828932"
+       sodipodi:cy="33.4571"
+       sodipodi:rx="23.492462"
+       sodipodi:ry="14.519141"
+       sodipodi:start="1.9010787"
+       sodipodi:end="2.4139426"
+       sodipodi:open="true"
+       sodipodi:arc-type="arc"
+       d="M 14.210088,47.191493 A 23.492462,14.519141 0 0 1 4.2861835,43.114023"
+       transform="rotate(-16.074499)" />
+    <path
+       style="fill:none;stroke:#000000;stroke-width:1.05483;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
+       d="m 7.6868303,36.80407 -5.145448,0.089"
+       id="path869" />
+    <circle
+       style="fill:none;fill-rule:evenodd;stroke:#a05a2c;stroke-width:1.05483;stroke-miterlimit:4;stroke-dasharray:none"
+       id="path840"
+       sodipodi:type="arc"
+       sodipodi:cx="-7.4551802"
+       sodipodi:cy="36.181274"
+       sodipodi:rx="1.8518577"
+       sodipodi:ry="1.8518577"
+       sodipodi:start="0.80077333"
+       sodipodi:end="6.0732171"
+       sodipodi:arc-type="arc"
+       d="m -6.1660062,37.510713 a 1.8518577,1.8518577 0 0 1 -2.2540959,0.251163 1.8518577,1.8518577 0 0 1 -0.8067226,-2.119724 1.8518577,1.8518577 0 0 1 1.8507266,-1.311046 1.8518577,1.8518577 0 0 1 1.7321042,1.464188"
+       transform="rotate(-16.074499)"
+       sodipodi:open="true" />
+    <path
+       style="fill:none;stroke:#000000;stroke-width:1.05483;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
+       d="m 52.164781,24.68443 3.867228,-2.36129"
+       id="path844" />
+    <path
+       style="fill:#ff0000;stroke:#ff0000;stroke-width:1.05483;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
+       d="m 10.948045,33.32733 c -1.1750857,-5.24274 -1.6744347,-11.1915 1.472996,-15.569619 1.423012,2.379429 2.09828,4.544259 4.995851,5.547079 -0.103954,-5.894677 0.193467,-15.3508774 5.291818,-21.7412674 1.919625,8.205413 3.528507,7.003438 8.107224,13.5523884 C 31.60789,12.559069 36.805134,5.6860276 39.28265,2.4919416 38.345155,14.634616 43.270588,17.29133 45.462955,23.53662"
+       id="path888"
+       sodipodi:nodetypes="ccccccc" />
+    <path
+       style="fill:#ffcc00;stroke:#ffcc00;stroke-width:0.918782;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
+       d="m 13.837936,32.51027 c -1.023526,-4.56654 -2.071127,-4.4679 0.670352,-8.28133 1.239474,2.07253 2.571378,4.66998 5.531855,4.56272 4.15165,-7.40717 0.247498,-12.827476 4.113983,-17.838764 1.626937,6.41989 3.327101,1.106217 7.556866,10.706084 0.698817,-2.871689 3.314685,-8.129053 5.043041,-10.432865 1.72178,6.610351 2.016157,4.825611 5.337656,13.224545"
+       id="path888-6"
+       sodipodi:nodetypes="ccccccc" />
+    <rect
+       style="fill:#c87137;stroke:#a05a2c;stroke-width:0.139765;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:4;stroke-dasharray:none;paint-order:markers stroke fill"
+       id="rect914"
+       width="20.739071"
+       height="2.2097089"
+       x="34.946976"
+       y="47.017033"
+       ry="1.0685775"
+       transform="rotate(-31.269169)" />
+  </g>
+</svg>
--- a/fbs/nostr-index.fbs
+++ b/fbs/nostr-index.fbs
@ -1,16 +1,27 @@
 namespace NostrIndex;

-table Tag {
+struct Fixed32Bytes {
+    val: [ubyte:32];
+}
+
+table TagGeneral {
    key: uint8;
    val: [ubyte];
 }

+table TagFixed32 {
+    key: uint8;
+    val: Fixed32Bytes;
+}
+
 table Event {
-    id: [ubyte];
-    pubkey: [ubyte];
+    id: Fixed32Bytes;
+    pubkey: Fixed32Bytes;
    created_at: uint64;
    kind: uint64;
-    tags: [Tag];
+    tagsGeneral: [TagGeneral];
+    tagsFixed32: [TagFixed32];
+    expiration: uint64;
 }

 table Empty {}
--- a/2
+++ b/2
--- a/golpe.yaml
+++ b/golpe.yaml
@ -1,26 +1,40 @@
 appName: strfry
-
-quadrable: true
+onAppStartup: true
+useGlobalH: true
+customLMDBSetup: true

 flatBuffers: |
    include "../fbs/nostr-index.fbs";

+includes: |
+    inline std::string_view sv(const NostrIndex::Fixed32Bytes *f) {
+        return std::string_view((const char *)f->val()->data(), 32);
+    }
+
 tables:
-  Event:
-    tableId: 1
-
-    primaryKey: quadId
-
+  ## DB meta-data. Single entry, with id = 1
+  Meta:
+    fields:
+      - name: dbVersion
+      - name: endianness
+
+  ## Meta-info of nostr events, suitable for indexing
+  ## Primary key is auto-incremented, called "levId" for Local EVent ID
+  Event:
    fields:
-      - name: quadId
      - name: receivedAt # microseconds
      - name: flat
        type: ubytes
        nestedFlat: NostrIndex.Event
+      - name: sourceType
+      - name: sourceInfo
+        type: ubytes

    indices:
      created_at:
        integer: true
+      receivedAt:
+        integer: true
      id:
        comparator: StringUint64
      pubkey:
@ -34,30 +48,74 @@ tables:
        multi: true
      deletion: # eventId, pubkey
        multi: true
+      expiration:
+        integer: true
+        multi: true
+      replace: # pubkey, d-tag, kind
+        multi: true

    indexPrelude: |
        auto *flat = v.flat_nested();
        created_at = flat->created_at();
        uint64_t indexTime = *created_at;
+        receivedAt = v.receivedAt();

        id = makeKey_StringUint64(sv(flat->id()), indexTime);
        pubkey = makeKey_StringUint64(sv(flat->pubkey()), indexTime);
        kind = makeKey_Uint64Uint64(flat->kind(), indexTime);
        pubkeyKind = makeKey_StringUint64Uint64(sv(flat->pubkey()), flat->kind(), indexTime);

-        for (const auto &tagPair : *(flat->tags())) {
+        for (const auto &tagPair : *(flat->tagsGeneral())) {
+            auto tagName = (char)tagPair->key();
+            auto tagVal = sv(tagPair->val());
+
+            tag.push_back(makeKey_StringUint64(std::string(1, tagName) + std::string(tagVal), indexTime));
+
+            if (tagName == 'd' && replace.size() == 0) {
+                replace.push_back(makeKey_StringUint64(std::string(sv(flat->pubkey())) + std::string(tagVal), flat->kind()));
+            }
+        }
+
+        for (const auto &tagPair : *(flat->tagsFixed32())) {
            auto tagName = (char)tagPair->key();
            auto tagVal = sv(tagPair->val());
            tag.push_back(makeKey_StringUint64(std::string(1, tagName) + std::string(tagVal), indexTime));
            if (flat->kind() == 5 && tagName == 'e') deletion.push_back(std::string(tagVal) + std::string(sv(flat->pubkey())));
        }

+        if (flat->expiration() != 0) {
+            expiration.push_back(flat->expiration());
+        }
+
+  CompressionDictionary:
+    fields:
+      - name: dict
+        type: ubytes
+
+tablesRaw:
+  ## Raw nostr event JSON, possibly compressed
+  ## keys are levIds
+  ## vals are prefixed with a type byte:
+  ##   0: no compression, payload follows
+  ##   1: zstd compression. Followed by Dictionary ID (native endian uint32) then compressed payload
+  EventPayload:
+    flags: 'MDB_INTEGERKEY'
+
 config:
  - name: db
-    desc: "Directory that contains strfry database"
+    desc: "Directory that contains the strfry LMDB database"
    default: "./strfry-db/"
    noReload: true

+  - name: dbParams__maxreaders
+    desc: "Maximum number of threads/processes that can simultaneously have LMDB transactions open"
+    default: 256
+    noReload: true
+  - name: dbParams__mapsize
+    desc: "Size of mmap() to use when loading LMDB (default is 10TB, does *not* correspond to disk-space used)"
+    default: 10995116277760
+    noReload: true
+
  - name: relay__bind
    desc: "Interface to listen on. Use 0.0.0.0 to listen on all interfaces"
    default: "127.0.0.1"
@ -66,6 +124,13 @@ config:
    desc: "Port to open for the nostr websocket protocol"
    default: 7777
    noReload: true
+  - name: relay__nofiles
+    desc: "Set OS-limit on maximum number of open files/sockets (if 0, don't attempt to set)"
+    default: 1000000
+    noReload: true
+  - name: relay__realIpHeader
+    desc: "HTTP header that contains the client's real IP, before reverse proxying (ie x-real-ip) (MUST be all lower-case)"
+    default: ""

  - name: relay__info__name
    desc: "NIP-11: Name of this server. Short/descriptive (< 30 characters)"
@ -97,6 +162,25 @@ config:
  - name: relay__maxFilterLimit
    desc: "Maximum records that can be returned per filter"
    default: 500
+  - name: relay__maxSubsPerConnection
+    desc: "Maximum number of subscriptions (concurrent REQs) a connection can have open at any time"
+    default: 20
+
+  - name: relay__writePolicy__plugin
+    desc: "If non-empty, path to an executable script that implements the writePolicy plugin logic"
+    default: ""
+  - name: relay__writePolicy__lookbackSeconds
+    desc: "Number of seconds to search backwards for lookback events when starting the writePolicy plugin (0 for no lookback)"
+    default: 0
+
+  - name: relay__compression__enabled
+    desc: "Use permessage-deflate compression if supported by client. Reduces bandwidth, but slight increase in CPU"
+    default: true
+    noReload: true
+  - name: relay__compression__slidingWindow
+    desc: "Maintain a sliding window buffer for each connection. Improves compression, but uses more memory"
+    default: true
+    noReload: true

  - name: relay__logging__dumpInAll
    desc: "Dump all incoming messages"
@ -112,15 +196,19 @@ config:
    default: false

  - name: relay__numThreads__ingester
+    desc: Ingester threads: route incoming requests, validate events/sigs
    default: 3
    noReload: true
  - name: relay__numThreads__reqWorker
+    desc: reqWorker threads: Handle initial DB scan for events
    default: 3
    noReload: true
  - name: relay__numThreads__reqMonitor
+    desc: reqMonitor threads: Handle filtering of new events
    default: 3
    noReload: true
  - name: relay__numThreads__yesstr
+    desc: yesstr threads: Experimental yesstr protocol
    default: 1
    noReload: true

@ -141,7 +229,7 @@ config:
    default: 300
  - name: events__maxNumTags
    desc: "Maximum number of tags allowed"
-    default: 250
+    default: 2000
  - name: events__maxTagValSize
    desc: "Maximum size for tag values, in bytes"
-    default: 255
+    default: 1024
--- a/src/ActiveMonitors.h
+++ b/src/ActiveMonitors.h
@ -1,5 +1,7 @@
 #pragma once

+#include <unordered_map>
+
 #include "golpe.h"

 #include "Subscription.h"
@ -13,26 +15,35 @@ struct ActiveMonitors : NonCopyable {
        Subscription sub;

        Monitor(Subscription &sub_) : sub(std::move(sub_)) {}
+        Monitor(const Monitor&) = delete; // pointers to filters inside sub must be stable because they are stored in MonitorSets
    };

-    using ConnMonitor = std::map<SubId, Monitor>;
-    std::map<uint64_t, ConnMonitor> conns; // connId -> subId -> Monitor
+    using ConnMonitor = std::unordered_map<SubId, Monitor>;
+    flat_hash_map<uint64_t, ConnMonitor> conns; // connId -> subId -> Monitor

    struct MonitorItem {
        Monitor *mon;
        uint64_t latestEventId;
    };

-    using MonitorSet = std::map<NostrFilter*, MonitorItem>; // FIXME: flat_map here
-    std::map<std::string, MonitorSet> allIds;
-    std::map<std::string, MonitorSet> allAuthors;
-    std::map<std::string, MonitorSet> allTags;
-    std::map<uint64_t, MonitorSet> allKinds;
+    using MonitorSet = flat_hash_map<NostrFilter*, MonitorItem>;
+    btree_map<std::string, MonitorSet> allIds;
+    btree_map<std::string, MonitorSet> allAuthors;
+    btree_map<std::string, MonitorSet> allTags;
+    btree_map<uint64_t, MonitorSet> allKinds;
    MonitorSet allOthers;

+    std::string tagSpecBuf = std::string(256, '\0');
+    const std::string &getTagSpec(uint8_t k, std::string_view val) {
+        tagSpecBuf.clear();
+        tagSpecBuf += (char)k;
+        tagSpecBuf += val;
+        return tagSpecBuf;
+    }
+

  public:
-    void addSub(lmdb::txn &txn, Subscription &&sub, uint64_t currEventId) {
+    bool addSub(lmdb::txn &txn, Subscription &&sub, uint64_t currEventId) {
        if (sub.latestEventId != currEventId) throw herr("sub not up to date");

        {
@ -43,10 +54,15 @@ struct ActiveMonitors : NonCopyable {
        auto res = conns.try_emplace(sub.connId);
        auto &connMonitors = res.first->second;

+        if (connMonitors.size() >= cfg().relay__maxSubsPerConnection) {
+            return false;
+        }
+
        auto subId = sub.subId;
        auto *m = &connMonitors.try_emplace(subId, sub).first->second;

        installLookups(m, currEventId);
+        return true;
    }

    void removeSub(uint64_t connId, const SubId &subId) {
@ -84,7 +100,7 @@ struct ActiveMonitors : NonCopyable {
            }
        };

-        auto processMonitorsPrefix = [&](std::map<std::string, MonitorSet> &m, const std::string &key, std::function<bool(const std::string&)> matches){
+        auto processMonitorsPrefix = [&](btree_map<std::string, MonitorSet> &m, const std::string &key, std::function<bool(const std::string&)> matches){
            auto it = m.lower_bound(key.substr(0, 1));

            if (it == m.end()) return;
@ -95,7 +111,7 @@ struct ActiveMonitors : NonCopyable {
            }
        };

-        auto processMonitorsExact = [&]<typename T>(std::map<T, MonitorSet> &m, const T &key, std::function<bool(const T &)> matches){
+        auto processMonitorsExact = [&]<typename T>(btree_map<T, MonitorSet> &m, const T &key, std::function<bool(const T &)> matches){
            auto it = m.upper_bound(key);

            if (it == m.begin()) return;
@ -124,10 +140,15 @@ struct ActiveMonitors : NonCopyable {
            }));
        }

-        for (const auto &tag : *flat->tags()) {
-            // FIXME: can avoid this allocation:
-            auto tagSpec = std::string(1, (char)tag->key()) + std::string(sv(tag->val()));
+        for (const auto &tag : *flat->tagsFixed32()) {
+            auto &tagSpec = getTagSpec(tag->key(), sv(tag->val()));
+            processMonitorsExact(allTags, tagSpec, static_cast<std::function<bool(const std::string&)>>([&](const std::string &val){
+                return tagSpec == val;
+            }));
+        }

+        for (const auto &tag : *flat->tagsGeneral()) {
+            auto &tagSpec = getTagSpec(tag->key(), sv(tag->val()));
            processMonitorsExact(allTags, tagSpec, static_cast<std::function<bool(const std::string&)>>([&](const std::string &val){
                return tagSpec == val;
            }));
@ -174,7 +195,7 @@ struct ActiveMonitors : NonCopyable {
            } else if (f.tags.size()) {
                for (const auto &[tagName, filterSet] : f.tags) {
                    for (size_t i = 0; i < filterSet.size(); i++) {
-                        std::string tagSpec = std::string(1, tagName) + filterSet.at(i);
+                        auto &tagSpec = getTagSpec(tagName, filterSet.at(i));
                        auto res = allTags.try_emplace(tagSpec);
                        res.first->second.try_emplace(&f, MonitorItem{m, currEventId});
                    }
@ -207,7 +228,7 @@ struct ActiveMonitors : NonCopyable {
            } else if (f.tags.size()) {
                for (const auto &[tagName, filterSet] : f.tags) {
                    for (size_t i = 0; i < filterSet.size(); i++) {
-                        std::string tagSpec = std::string(1, tagName) + filterSet.at(i);
+                        auto &tagSpec = getTagSpec(tagName, filterSet.at(i));
                        auto &monSet = allTags.at(tagSpec);
                        monSet.erase(&f);
                        if (monSet.empty()) allTags.erase(tagSpec);
--- a/src/DBScan.h
+++ b/src/DBScan.h
@ -30,7 +30,7 @@ struct DBScan {
    };

    struct TagScan {
-        std::map<char, FilterSetBytes>::const_iterator indexTagName;
+        flat_hash_map<char, FilterSetBytes>::const_iterator indexTagName;
        size_t indexTagVal = 0;
        std::string search;
    };
@ -49,10 +49,16 @@ struct DBScan {
    std::string resumeKey;
    uint64_t resumeVal;

+    enum class KeyMatchResult {
+        Yes,
+        No,
+        NoButContinue,
+    };
+
    std::function<bool()> isComplete;
    std::function<void()> nextFilterItem;
    std::function<void()> resetResume;
-    std::function<bool(std::string_view, bool&)> keyMatch;
+    std::function<KeyMatchResult(std::string_view, bool&)> keyMatch;

    DBScan(const NostrFilter &f_) : f(f_) {
        remainingLimit = f.limit;
@ -74,7 +80,7 @@ struct DBScan {
                resumeVal = MAX_U64;
            };
            keyMatch = [&, state](std::string_view k, bool&){
-                return k.starts_with(state->prefix);
+                return k.starts_with(state->prefix) ? KeyMatchResult::Yes : KeyMatchResult::No;
            };
        } else if (f.authors && f.kinds) {
            scanState = PubkeyKindScan{};
@ -98,16 +104,22 @@ struct DBScan {
                resumeVal = MAX_U64;
            };
            keyMatch = [&, state](std::string_view k, bool &skipBack){
-                if (!k.starts_with(state->prefix)) return false;
-                if (state->prefix.size() == 32 + 8) return true;
+                if (!k.starts_with(state->prefix)) return KeyMatchResult::No;
+                if (state->prefix.size() == 32 + 8) return KeyMatchResult::Yes;

                ParsedKey_StringUint64Uint64 parsedKey(k);
-                if (parsedKey.n1 <= f.kinds->at(state->indexKind)) return true;
+                if (parsedKey.n1 == f.kinds->at(state->indexKind)) {
+                    return KeyMatchResult::Yes;
+                } else if (parsedKey.n1 < f.kinds->at(state->indexKind)) {
+                    // With a prefix pubkey, continue scanning (pubkey,kind) backwards because with this index
+                    // we don't know the next pubkey to jump back to
+                    return KeyMatchResult::NoButContinue;
+                }

                resumeKey = makeKey_StringUint64Uint64(parsedKey.s, f.kinds->at(state->indexKind), MAX_U64);
                resumeVal = MAX_U64;
                skipBack = true;
-                return false;
+                return KeyMatchResult::No;
            };
        } else if (f.authors) {
            scanState = PubkeyScan{};
@ -126,7 +138,7 @@ struct DBScan {
                resumeVal = MAX_U64;
            };
            keyMatch = [&, state](std::string_view k, bool&){
-                return k.starts_with(state->prefix);
+                return k.starts_with(state->prefix) ? KeyMatchResult::Yes : KeyMatchResult::No;
            };
        } else if (f.tags.size()) {
            scanState = TagScan{f.tags.begin()};
@ -150,7 +162,7 @@ struct DBScan {
                resumeVal = MAX_U64;
            };
            keyMatch = [&, state](std::string_view k, bool&){
-                return k.substr(0, state->search.size()) == state->search;
+                return k.substr(0, state->search.size()) == state->search ? KeyMatchResult::Yes : KeyMatchResult::No;
            };
        } else if (f.kinds) {
            scanState = KindScan{};
@ -170,7 +182,7 @@ struct DBScan {
            };
            keyMatch = [&, state](std::string_view k, bool&){
                ParsedKey_Uint64Uint64 parsedKey(k);
-                return parsedKey.n1 == state->kind;
+                return parsedKey.n1 == state->kind ? KeyMatchResult::Yes : KeyMatchResult::No;
            };
        } else {
            scanState = CreatedAtScan{};
@ -188,7 +200,7 @@ struct DBScan {
                resumeVal = MAX_U64;
            };
            keyMatch = [&, state](std::string_view k, bool&){
-                return true;
+                return KeyMatchResult::Yes;
            };
        }
    }
@ -208,7 +220,8 @@ struct DBScan {
                    return false;
                }

-                if (!keyMatch(k, skipBack)) return false;
+                auto matched = keyMatch(k, skipBack);
+                if (matched == KeyMatchResult::No) return false;

                uint64_t created;

@ -232,18 +245,20 @@ struct DBScan {
                }

                bool sent = false;
-                uint64_t quadId = lmdb::from_sv<uint64_t>(v);
+                uint64_t levId = lmdb::from_sv<uint64_t>(v);

-                if (f.indexOnlyScans) {
+                if (matched == KeyMatchResult::NoButContinue) {
+                    // Don't attempt to match filter
+                } else if (f.indexOnlyScans) {
                    if (f.doesMatchTimes(created)) {
-                        handleEvent(quadId);
+                        handleEvent(levId);
                        sent = true;
                    }
                } else {
-                    auto view = env.lookup_Event(txn, quadId);
+                    auto view = env.lookup_Event(txn, levId);
                    if (!view) throw herr("missing event from index, corrupt DB?");
                    if (f.doesMatch(view->flat_nested())) {
-                        handleEvent(quadId);
+                        handleEvent(levId);
                        sent = true;
                    }
                }
@ -280,7 +295,7 @@ struct DBScanQuery : NonCopyable {

    size_t filterGroupIndex = 0;
    bool dead = false;
-    std::unordered_set<uint64_t> alreadySentEvents; // FIXME: flat_set here, or roaring bitmap/judy/whatever
+    flat_hash_set<uint64_t> alreadySentEvents;

    uint64_t currScanTime = 0;
    uint64_t currScanSaveRestores = 0;
@ -298,16 +313,16 @@ struct DBScanQuery : NonCopyable {
        while (filterGroupIndex < sub.filterGroup.size()) {
            if (!scanner) scanner = std::make_unique<DBScan>(sub.filterGroup.filters[filterGroupIndex]);

-            bool complete = scanner->scan(txn, [&](uint64_t quadId){
+            bool complete = scanner->scan(txn, [&](uint64_t levId){
                // If this event came in after our query began, don't send it. It will be sent after the EOSE.
-                if (quadId > sub.latestEventId) return;
+                if (levId > sub.latestEventId) return;

                // We already sent this event
-                if (alreadySentEvents.find(quadId) != alreadySentEvents.end()) return;
-                alreadySentEvents.insert(quadId);
+                if (alreadySentEvents.find(levId) != alreadySentEvents.end()) return;
+                alreadySentEvents.insert(levId);

                currScanRecordsFound++;
-                cb(sub, quadId);
+                cb(sub, levId);
            }, [&]{
                currScanRecordsTraversed++;
                return hoytech::curr_time_us() - startTime > timeBudgetMicroseconds;
--- a/src/Decompressor.cpp
+++ b/src/Decompressor.cpp
@ -0,0 +1,5 @@
+#include "golpe.h"
+
+#include "Decompressor.h"
+
+DictionaryBroker globalDictionaryBroker;
--- a/src/Decompressor.h
+++ b/src/Decompressor.h
@ -0,0 +1,68 @@
+#pragma once
+
+#include <zstd.h>
+#include <zdict.h>
+
+#include <mutex>
+
+#include "golpe.h"
+
+
+struct DictionaryBroker {
+    std::mutex mutex;
+    flat_hash_map<uint32_t, ZSTD_DDict*> dicts;
+
+    ZSTD_DDict *getDict(lmdb::txn &txn, uint32_t dictId) {
+        std::lock_guard<std::mutex> guard(mutex);
+
+        auto it = dicts.find(dictId);
+        if (it != dicts.end()) return it->second;
+
+        auto view = env.lookup_CompressionDictionary(txn, dictId);
+        if (!view) throw herr("couldn't find dictId ", dictId);
+        auto dictBuffer = view->dict();
+
+        auto *dict = dicts[dictId] = ZSTD_createDDict(dictBuffer.data(), dictBuffer.size());
+
+        return dict;
+    }
+};
+
+extern DictionaryBroker globalDictionaryBroker;
+
+
+struct Decompressor {
+    ZSTD_DCtx *dctx;
+    flat_hash_map<uint32_t, ZSTD_DDict*> dicts;
+    std::string buffer;
+
+    Decompressor() {
+        dctx = ZSTD_createDCtx();
+    }
+
+    ~Decompressor() {
+        ZSTD_freeDCtx(dctx);
+    }
+
+    void reserve(size_t n) {
+        buffer.resize(n);
+    }
+
+    // Return result only valid until one of: a) next call to decompress()/reserve(), or Decompressor destroyed
+
+    std::string_view decompress(lmdb::txn &txn, uint32_t dictId, std::string_view src) {
+        auto it = dicts.find(dictId);
+        ZSTD_DDict *dict;
+
+        if (it == dicts.end()) {
+            dict = dicts[dictId] = globalDictionaryBroker.getDict(txn, dictId);
+        } else {
+            dict = it->second;
+        }
+
+        auto ret = ZSTD_decompress_usingDDict(dctx, buffer.data(), buffer.size(), src.data(), src.size(), dict);
+        if (ZDICT_isError(ret)) throw herr("zstd decompression failed: ", ZSTD_getErrorName(ret));
+
+        return std::string_view(buffer.data(), ret);
+    }
+};
--- a/src/PluginWritePolicy.h
+++ b/src/PluginWritePolicy.h
@ -0,0 +1,217 @@
+#pragma once
+
+#include <string.h>
+#include <errno.h>
+#include <spawn.h>
+#include <unistd.h>
+#include <stdio.h>
+#include <sys/types.h>
+#include <sys/wait.h>
+#include <sys/types.h>
+#include <sys/stat.h>
+#include <signal.h>
+
+#include <memory>
+
+#include "golpe.h"
+
+
+enum class WritePolicyResult {
+    Accept,
+    Reject,
+    ShadowReject,
+};
+
+
+struct PluginWritePolicy {
+    struct RunningPlugin {
+        pid_t pid;
+        std::string currPluginPath;
+        uint64_t lookbackSeconds;
+        struct timespec lastModTime;
+        FILE *r;
+        FILE *w;
+
+        RunningPlugin(pid_t pid, int rfd, int wfd, std::string currPluginPath, uint64_t lookbackSeconds) : pid(pid), currPluginPath(currPluginPath), lookbackSeconds(lookbackSeconds) {
+            r = fdopen(rfd, "r");
+            w = fdopen(wfd, "w");
+            setlinebuf(w);
+            {
+                struct stat statbuf;
+                if (stat(currPluginPath.c_str(), &statbuf)) throw herr("couldn't stat plugin: ", currPluginPath);
+                lastModTime = statbuf.st_mtim;
+            }
+        }
+
+        ~RunningPlugin() {
+            fclose(r);
+            fclose(w);
+            kill(pid, SIGTERM);
+            waitpid(pid, nullptr, 0);
+        }
+    };
+
+    std::unique_ptr<RunningPlugin> running; 
+
+    WritePolicyResult acceptEvent(std::string_view jsonStr, uint64_t receivedAt, EventSourceType sourceType, std::string_view sourceInfo, std::string &okMsg) {
+        const auto &pluginPath = cfg().relay__writePolicy__plugin;
+
+        if (pluginPath.size() == 0) {
+            running.reset();
+            return WritePolicyResult::Accept;
+        }
+
+        try {
+            if (running) {
+                if (pluginPath != running->currPluginPath || cfg().relay__writePolicy__lookbackSeconds != running->lookbackSeconds) {
+                    running.reset();
+                } else {
+                    struct stat statbuf;
+                    if (stat(pluginPath.c_str(), &statbuf)) throw herr("couldn't stat plugin: ", pluginPath);
+                    if (statbuf.st_mtim.tv_sec != running->lastModTime.tv_sec || statbuf.st_mtim.tv_nsec != running->lastModTime.tv_nsec) {
+                        running.reset();
+                    }
+                }
+            }
+
+            if (!running) {
+                setupPlugin();
+                sendLookbackEvents();
+            }
+
+            auto request = tao::json::value({
+                { "type", "new" },
+                { "event", tao::json::from_string(jsonStr) },
+                { "receivedAt", receivedAt / 1000000 },
+                { "sourceType", eventSourceTypeToStr(sourceType) },
+                { "sourceInfo", sourceType == EventSourceType::IP4 || sourceType == EventSourceType::IP6 ? renderIP(sourceInfo) : sourceInfo },
+            });
+
+            std::string output = tao::json::to_string(request);
+            output += "\n";
+
+            if (::fwrite(output.data(), 1, output.size(), running->w) != output.size()) throw herr("error writing to plugin");
+
+            tao::json::value response;
+
+            while (1) {
+                char buf[8192];
+                if (!fgets(buf, sizeof(buf), running->r)) throw herr("pipe to plugin was closed (plugin crashed?)");
+
+                try {
+                    response = tao::json::from_string(buf);
+                } catch (std::exception &e) {
+                    LW << "Got unparseable line from write policy plugin: " << buf;
+                    continue;
+                }
+
+                if (response.at("id").get_string() != request.at("event").at("id").get_string()) throw herr("id mismatch");
+
+                break;
+            }
+
+            okMsg = response.optional<std::string>("msg").value_or("");
+
+            auto action = response.at("action").get_string();
+            if (action == "accept") return WritePolicyResult::Accept;
+            else if (action == "reject") return WritePolicyResult::Reject;
+            else if (action == "shadowReject") return WritePolicyResult::ShadowReject;
+            else throw herr("unknown action: ", action);
+        } catch (std::exception &e) {
+            LE << "Couldn't setup PluginWritePolicy: " << e.what();
+            running.reset();
+            okMsg = "error: internal error";
+            return WritePolicyResult::Reject;
+        }
+    }
+
+
+
+    struct Pipe : NonCopyable {
+        int fds[2] = { -1, -1 };
+
+        Pipe() {
+            if (::pipe(fds)) throw herr("pipe failed: ", strerror(errno));
+        }
+
+        Pipe(int fd0, int fd1) {
+            fds[0] = fd0;
+            fds[1] = fd1;
+        }
+
+        ~Pipe() {
+            if (fds[0] != -1) ::close(fds[0]);
+            if (fds[1] != -1) ::close(fds[1]);
+        }
+
+        int saveFd(int offset) {
+            int fd = fds[offset];
+            fds[offset] = -1;
+            return fd;
+        }
+    };
+
+    void setupPlugin() {
+        auto path = cfg().relay__writePolicy__plugin;
+        LI << "Setting up write policy plugin: " << path;
+
+        Pipe outPipe;
+        Pipe inPipe;
+
+        pid_t pid;
+        char *argv[] = { nullptr, };
+
+        posix_spawn_file_actions_t file_actions;
+
+        if (
+            posix_spawn_file_actions_init(&file_actions) ||
+            posix_spawn_file_actions_adddup2(&file_actions, outPipe.fds[0], 0) ||
+            posix_spawn_file_actions_adddup2(&file_actions, inPipe.fds[1], 1) ||
+            posix_spawn_file_actions_addclose(&file_actions, outPipe.fds[0]) ||
+            posix_spawn_file_actions_addclose(&file_actions, outPipe.fds[1]) ||
+            posix_spawn_file_actions_addclose(&file_actions, inPipe.fds[0]) ||
+            posix_spawn_file_actions_addclose(&file_actions, inPipe.fds[1])
+        ) throw herr("posix_span_file_actions failed: ", strerror(errno));
+
+        auto ret = posix_spawn(&pid, path.c_str(), &file_actions, nullptr, argv, nullptr);
+        if (ret) throw herr("posix_spawn failed to invoke '", path, "': ", strerror(errno));
+
+        running = make_unique<RunningPlugin>(pid, inPipe.saveFd(0), outPipe.saveFd(1), path, cfg().relay__writePolicy__lookbackSeconds);
+    }
+
+    void sendLookbackEvents() {
+        if (running->lookbackSeconds == 0) return;
+
+        Decompressor decomp;
+        auto now = hoytech::curr_time_us();
+
+        uint64_t start = now - (running->lookbackSeconds * 1'000'000);
+
+        auto txn = env.txn_ro();
+
+        env.generic_foreachFull(txn, env.dbi_Event__receivedAt, lmdb::to_sv<uint64_t>(start), lmdb::to_sv<uint64_t>(0), [&](auto k, auto v) {
+            if (lmdb::from_sv<uint64_t>(k) > now) return false;
+
+            auto ev = env.lookup_Event(txn, lmdb::from_sv<uint64_t>(v));
+            if (!ev) throw herr("unable to look up event, corrupt DB?");
+
+            auto sourceType = (EventSourceType)ev->sourceType();
+            std::string_view sourceInfo = ev->sourceInfo();
+
+            auto request = tao::json::value({
+                { "type", "lookback" },
+                { "event", tao::json::from_string(getEventJson(txn, decomp, ev->primaryKeyId)) },
+                { "receivedAt", ev->receivedAt() / 1000000 },
+                { "sourceType", eventSourceTypeToStr(sourceType) },
+                { "sourceInfo", sourceType == EventSourceType::IP4 || sourceType == EventSourceType::IP6 ? renderIP(sourceInfo) : sourceInfo },
+            });
+
+            std::string output = tao::json::to_string(request);
+            output += "\n";
+
+            if (::fwrite(output.data(), 1, output.size(), running->w) != output.size()) throw herr("error writing to plugin");
+
+            return true;
+        });
+    }
+};
--- a/src/RelayCron.cpp
+++ b/src/RelayCron.cpp
@ -1,18 +1,15 @@
 #include "RelayServer.h"

+#include "gc.h"
+

 void RelayServer::cleanupOldEvents() {
-    struct EventDel {
-        uint64_t nodeId;
-        uint64_t deletedNodeId;
-    };
-
-    std::vector<EventDel> expiredEvents;
+    std::vector<uint64_t> expiredLevIds;

    {
        auto txn = env.txn_ro();

-        auto mostRecent = getMostRecentEventId(txn);
+        auto mostRecent = getMostRecentLevId(txn);
        uint64_t cutoff = hoytech::curr_time_s() - cfg().events__ephemeralEventsLifetimeSeconds;
        uint64_t currKind = 20'000;

@ -31,10 +28,10 @@ void RelayServer::cleanupOldEvents() {
                    return false;
                }

-                uint64_t nodeId = lmdb::from_sv<uint64_t>(v);
+                uint64_t levId = lmdb::from_sv<uint64_t>(v);

-                if (nodeId != mostRecent) { // prevent nodeId re-use
-                    expiredEvents.emplace_back(nodeId, 0);
+                if (levId != mostRecent) { // prevent levId re-use
+                    expiredLevIds.emplace_back(levId);
                }

                return true;
@ -44,29 +41,31 @@ void RelayServer::cleanupOldEvents() {
        }
    }

-    if (expiredEvents.size() > 0) {
-        LI << "Deleting " << expiredEvents.size() << " ephemeral events";
+    if (expiredLevIds.size() > 0) {
+        auto qdb = getQdbInstance();

        auto txn = env.txn_rw();

-        quadrable::Quadrable qdb;
-        qdb.init(txn);
-        qdb.checkout("events");
-
+        uint64_t numDeleted = 0;
        auto changes = qdb.change();

-        for (auto &e : expiredEvents) {
-            auto view = env.lookup_Event(txn, e.nodeId);
-            if (!view) throw herr("missing event from index, corrupt DB?");
-            changes.del(flatEventToQuadrableKey(view->flat_nested()), &e.deletedNodeId);
+        for (auto levId : expiredLevIds) {
+            auto view = env.lookup_Event(txn, levId);
+            if (!view) continue; // Deleted in between transactions
+            deleteEvent(txn, changes, *view);
+            numDeleted++;
        }

        changes.apply(txn);

-        for (auto &e : expiredEvents) {
-            if (e.deletedNodeId) env.delete_Event(txn, e.nodeId);
-        }
-
        txn.commit();
+
+        if (numDeleted) LI << "Deleted " << numDeleted << " ephemeral events";
    }
 }
+
+void RelayServer::garbageCollect() {
+    auto qdb = getQdbInstance();
+
+    quadrableGarbageCollect(qdb, 1);
+}
--- a/src/RelayIngester.cpp
+++ b/src/RelayIngester.cpp
@ -29,7 +29,7 @@ void RelayServer::runIngester(ThreadPool<MsgIngester>::Thread &thr) {
                            if (cfg().relay__logging__dumpInEvents) LI << "[" << msg->connId << "] dumpInEvent: " << msg->payload; 

                            try {
-                                ingesterProcessEvent(txn, msg->connId, secpCtx, arr[1], writerMsgs);
+                                ingesterProcessEvent(txn, msg->connId, msg->ipAddr, secpCtx, arr[1], writerMsgs);
                            } catch (std::exception &e) {
                                sendOKResponse(msg->connId, arr[1].at("id").get_string(), false, std::string("invalid: ") + e.what());
                                LI << "Rejected invalid event: " << e.what();
@ -82,7 +82,7 @@ void RelayServer::runIngester(ThreadPool<MsgIngester>::Thread &thr) {
    }
 }

-void RelayServer::ingesterProcessEvent(lmdb::txn &txn, uint64_t connId, secp256k1_context *secpCtx, const tao::json::value &origJson, std::vector<MsgWriter> &output) {
+void RelayServer::ingesterProcessEvent(lmdb::txn &txn, uint64_t connId, std::string ipAddr, secp256k1_context *secpCtx, const tao::json::value &origJson, std::vector<MsgWriter> &output) {
    std::string flatStr, jsonStr;

    parseAndVerifyEvent(origJson, secpCtx, true, true, flatStr, jsonStr);
@ -98,7 +98,7 @@ void RelayServer::ingesterProcessEvent(lmdb::txn &txn, uint64_t connId, secp256k
        }
    }

-    output.emplace_back(MsgWriter{MsgWriter::AddEvent{connId, hoytech::curr_time_us(), std::move(flatStr), std::move(jsonStr)}});
+    output.emplace_back(MsgWriter{MsgWriter::AddEvent{connId, std::move(ipAddr), hoytech::curr_time_us(), std::move(flatStr), std::move(jsonStr)}});
 }

 void RelayServer::ingesterProcessReq(lmdb::txn &txn, uint64_t connId, const tao::json::value &arr) {
--- a/src/RelayReqMonitor.cpp
+++ b/src/RelayReqMonitor.cpp
@ -14,6 +14,7 @@ void RelayServer::runReqMonitor(ThreadPool<MsgReqMonitor>::Thread &thr) {
    });


+    Decompressor decomp;
    ActiveMonitors monitors;
    uint64_t currEventId = MAX_U64;

@ -22,14 +23,16 @@ void RelayServer::runReqMonitor(ThreadPool<MsgReqMonitor>::Thread &thr) {

        auto txn = env.txn_ro();

-        uint64_t latestEventId = getMostRecentEventId(txn);
+        uint64_t latestEventId = getMostRecentLevId(txn);
        if (currEventId > latestEventId) currEventId = latestEventId;

        for (auto &newMsg : newMsgs) {
            if (auto msg = std::get_if<MsgReqMonitor::NewSub>(&newMsg.msg)) {
+                auto connId = msg->sub.connId;
+
                env.foreach_Event(txn, [&](auto &ev){
                    if (msg->sub.filterGroup.doesMatch(ev.flat_nested())) {
-                        sendEvent(msg->sub.connId, msg->sub.subId, getEventJson(txn, ev.primaryKeyId));
+                        sendEvent(connId, msg->sub.subId, getEventJson(txn, decomp, ev.primaryKeyId));
                    }

                    return true;
@ -37,15 +40,17 @@ void RelayServer::runReqMonitor(ThreadPool<MsgReqMonitor>::Thread &thr) {

                msg->sub.latestEventId = latestEventId;

-                monitors.addSub(txn, std::move(msg->sub), latestEventId);
+                if (!monitors.addSub(txn, std::move(msg->sub), latestEventId)) {
+                    sendNoticeError(connId, std::string("too many concurrent REQs"));
+                }
            } else if (auto msg = std::get_if<MsgReqMonitor::RemoveSub>(&newMsg.msg)) {
                monitors.removeSub(msg->connId, msg->subId);
            } else if (auto msg = std::get_if<MsgReqMonitor::CloseConn>(&newMsg.msg)) {
                monitors.closeConn(msg->connId);
            } else if (std::get_if<MsgReqMonitor::DBChange>(&newMsg.msg)) {
                env.foreach_Event(txn, [&](auto &ev){
-                    monitors.process(txn, ev, [&](RecipientList &&recipients, uint64_t quadId){
-                        sendEventToBatch(std::move(recipients), std::string(getEventJson(txn, quadId)));
+                    monitors.process(txn, ev, [&](RecipientList &&recipients, uint64_t levId){
+                        sendEventToBatch(std::move(recipients), std::string(getEventJson(txn, decomp, levId)));
                    });
                    return true;
                }, false, currEventId + 1);
--- a/src/RelayReqWorker.cpp
+++ b/src/RelayReqWorker.cpp
@ -4,12 +4,13 @@


 struct ActiveQueries : NonCopyable {
-    using ConnQueries = std::map<SubId, DBScanQuery*>;
-    std::map<uint64_t, ConnQueries> conns; // connId -> subId -> DBScanQuery*
+    Decompressor decomp;
+    using ConnQueries = flat_hash_map<SubId, DBScanQuery*>;
+    flat_hash_map<uint64_t, ConnQueries> conns; // connId -> subId -> DBScanQuery*
    std::deque<DBScanQuery*> running;

-    void addSub(lmdb::txn &txn, Subscription &&sub) {
-        sub.latestEventId = getMostRecentEventId(txn);
+    bool addSub(lmdb::txn &txn, Subscription &&sub) {
+        sub.latestEventId = getMostRecentLevId(txn);

        {
            auto *existing = findQuery(sub.connId, sub.subId);
@ -19,10 +20,16 @@ struct ActiveQueries : NonCopyable {
        auto res = conns.try_emplace(sub.connId);
        auto &connQueries = res.first->second;

+        if (connQueries.size() >= cfg().relay__maxSubsPerConnection) {
+            return false;
+        }
+
        DBScanQuery *q = new DBScanQuery(sub);

        connQueries.try_emplace(q->sub.subId, q);
        running.push_front(q);
+
+        return true;
    }

    DBScanQuery *findQuery(uint64_t connId, const SubId &subId) {
@ -63,8 +70,12 @@ struct ActiveQueries : NonCopyable {
            return;
        }

-        bool complete = q->process(txn, cfg().relay__queryTimesliceBudgetMicroseconds, cfg().relay__logging__dbScanPerf, [&](const auto &sub, uint64_t quadId){
-            server->sendEvent(sub.connId, sub.subId, getEventJson(txn, quadId));
+        auto cursor = lmdb::cursor::open(txn, env.dbi_EventPayload);
+
+        bool complete = q->process(txn, cfg().relay__queryTimesliceBudgetMicroseconds, cfg().relay__logging__dbScanPerf, [&](const auto &sub, uint64_t levId){
+            std::string_view key = lmdb::to_sv<uint64_t>(levId), val;
+            if (!cursor.get(key, val, MDB_SET_KEY)) throw herr("couldn't find event in EventPayload, corrupted DB?");
+            server->sendEvent(sub.connId, sub.subId, decodeEventPayload(txn, decomp, val, nullptr, nullptr));
        });

        if (complete) {
@ -93,7 +104,12 @@ void RelayServer::runReqWorker(ThreadPool<MsgReqWorker>::Thread &thr) {

        for (auto &newMsg : newMsgs) {
            if (auto msg = std::get_if<MsgReqWorker::NewSub>(&newMsg.msg)) {
-                queries.addSub(txn, std::move(msg->sub));
+                auto connId = msg->sub.connId;
+
+                if (!queries.addSub(txn, std::move(msg->sub))) {
+                    sendNoticeError(connId, std::string("too many concurrent REQs"));
+                }
+
                queries.process(this, txn);
            } else if (auto msg = std::get_if<MsgReqWorker::RemoveSub>(&newMsg.msg)) {
                queries.removeSub(msg->connId, msg->subId);
--- a/src/RelayServer.h
+++ b/src/RelayServer.h
@ -47,6 +47,7 @@ struct MsgWebsocket : NonCopyable {
 struct MsgIngester : NonCopyable {
    struct ClientMessage {
        uint64_t connId;
+        std::string ipAddr;
        std::string payload;
    };

@ -62,6 +63,7 @@ struct MsgIngester : NonCopyable {
 struct MsgWriter : NonCopyable {
    struct AddEvent {
        uint64_t connId;
+        std::string ipAddr;
        uint64_t receivedAt;
        std::string flatStr;
        std::string jsonStr;
@ -147,7 +149,7 @@ struct RelayServer {
    void runWebsocket(ThreadPool<MsgWebsocket>::Thread &thr);

    void runIngester(ThreadPool<MsgIngester>::Thread &thr);
-    void ingesterProcessEvent(lmdb::txn &txn, uint64_t connId, secp256k1_context *secpCtx, const tao::json::value &origJson, std::vector<MsgWriter> &output);
+    void ingesterProcessEvent(lmdb::txn &txn, uint64_t connId, std::string ipAddr, secp256k1_context *secpCtx, const tao::json::value &origJson, std::vector<MsgWriter> &output);
    void ingesterProcessReq(lmdb::txn &txn, uint64_t connId, const tao::json::value &origJson);
    void ingesterProcessClose(lmdb::txn &txn, uint64_t connId, const tao::json::value &origJson);

@ -160,6 +162,7 @@ struct RelayServer {
    void runYesstr(ThreadPool<MsgYesstr>::Thread &thr);

    void cleanupOldEvents();
+    void garbageCollect();

    // Utils (can be called by any thread)

@ -168,23 +171,24 @@ struct RelayServer {
        hubTrigger->send();
    }

-    void sendToConn(uint64_t connId, std::string &payload) {
-        tpWebsocket.dispatch(0, MsgWebsocket{MsgWebsocket::Send{connId, std::move(payload)}});
-        hubTrigger->send();
-    }
-
    void sendToConnBinary(uint64_t connId, std::string &&payload) {
        tpWebsocket.dispatch(0, MsgWebsocket{MsgWebsocket::SendBinary{connId, std::move(payload)}});
        hubTrigger->send();
    }

    void sendEvent(uint64_t connId, const SubId &subId, std::string_view evJson) {
-        std::string reply = std::string("[\"EVENT\",\"");
-        reply += subId.sv();
+        auto subIdSv = subId.sv();
+
+        std::string reply;
+        reply.reserve(13 + subIdSv.size() + evJson.size());
+
+        reply += "[\"EVENT\",\"";
+        reply += subIdSv;
        reply += "\",";
        reply += evJson;
        reply += "]";
-        sendToConn(connId, reply);
+
+        sendToConn(connId, std::move(reply));
    }

    void sendEventToBatch(RecipientList &&list, std::string &&evJson) {
--- a/src/RelayWebsocket.cpp
+++ b/src/RelayWebsocket.cpp
@ -1,5 +1,3 @@
-#include <stdio.h>
-
 #include "RelayServer.h"

 #include "app_git_version.h"
@ -19,46 +17,6 @@ static std::string preGenerateHttpResponse(const std::string &contentType, const
 };


-static std::string renderSize(uint64_t si) {
-    if (si < 1024) return std::to_string(si) + "b";
-
-    double s = si;
-    char buf[128];
-    char unit;
-
-    do {
-        s /= 1024;
-        if (s < 1024) {
-            unit = 'K';
-            break;
-        }
-
-        s /= 1024;
-        if (s < 1024) {
-            unit = 'M';
-            break;
-        }
-
-        s /= 1024;
-        if (s < 1024) {
-            unit = 'G';
-            break;
-        }
-
-        s /= 1024;
-        unit = 'T';
-    } while(0);
-
-    ::snprintf(buf, sizeof(buf), "%.2f%c", s, unit);
-    return std::string(buf);
-}
-
-static std::string renderPercent(double p) {
-    char buf[128];
-    ::snprintf(buf, sizeof(buf), "%.1f%%", p * 100);
-    return std::string(buf);
-}
-

 void RelayServer::runWebsocket(ThreadPool<MsgWebsocket>::Thread &thr) {
    struct Connection {
@ -81,7 +39,7 @@ void RelayServer::runWebsocket(ThreadPool<MsgWebsocket>::Thread &thr) {

    uWS::Hub hub;
    uWS::Group<uWS::SERVER> *hubGroup;
-    std::map<uint64_t, Connection*> connIdToConnection;
+    flat_hash_map<uint64_t, Connection*> connIdToConnection;
    uint64_t nextConnectionId = 1;

    std::string tempBuf;
@ -110,7 +68,14 @@ void RelayServer::runWebsocket(ThreadPool<MsgWebsocket>::Thread &thr) {



-    hubGroup = hub.createGroup<uWS::SERVER>(uWS::PERMESSAGE_DEFLATE | uWS::SLIDING_DEFLATE_WINDOW, cfg().relay__maxWebsocketPayloadSize);
+    {
+        int extensionOptions = 0;
+
+        if (cfg().relay__compression__enabled) extensionOptions |= uWS::PERMESSAGE_DEFLATE;
+        if (cfg().relay__compression__slidingWindow) extensionOptions |= uWS::SLIDING_DEFLATE_WINDOW;
+
+        hubGroup = hub.createGroup<uWS::SERVER>(extensionOptions, cfg().relay__maxWebsocketPayloadSize);
+    }

    if (cfg().relay__autoPingSeconds) hubGroup->startAutoPing(cfg().relay__autoPingSeconds * 1'000);

@ -126,21 +91,28 @@ void RelayServer::runWebsocket(ThreadPool<MsgWebsocket>::Thread &thr) {
    });

    hubGroup->onConnection([&](uWS::WebSocket<uWS::SERVER> *ws, uWS::HttpRequest req) {
-        std::string addr = ws->getAddress().address;
        uint64_t connId = nextConnectionId++;

+        Connection *c = new Connection(ws, connId);
+
+        if (cfg().relay__realIpHeader.size()) {
+            auto header = req.getHeader(cfg().relay__realIpHeader.c_str()).toString();
+            c->ipAddr = parseIP(header);
+            if (c->ipAddr.size() == 0) LW << "Couldn't parse IP from header " << cfg().relay__realIpHeader << ": " << header;
+        }
+
+        if (c->ipAddr.size() == 0) c->ipAddr = ws->getAddressBytes();
+
+        ws->setUserData((void*)c);
+        connIdToConnection.emplace(connId, c);
+
        bool compEnabled, compSlidingWindow;
        ws->getCompressionState(compEnabled, compSlidingWindow);
-        LI << "[" << connId << "] Connect from " << addr
+        LI << "[" << connId << "] Connect from " << renderIP(c->ipAddr)
           << " compression=" << (compEnabled ? 'Y' : 'N')
           << " sliding=" << (compSlidingWindow ? 'Y' : 'N')
        ;

-        Connection *c = new Connection(ws, connId);
-        c->ipAddr = addr;
-        ws->setUserData((void*)c);
-        connIdToConnection.emplace(connId, c);
-
        if (cfg().relay__enableTcpKeepalive) {
            int optval = 1;
            if (setsockopt(ws->getFd(), SOL_SOCKET, SO_KEEPALIVE, &optval, sizeof(optval))) {
@ -156,7 +128,7 @@ void RelayServer::runWebsocket(ThreadPool<MsgWebsocket>::Thread &thr) {
        auto upComp = renderPercent(1.0 - (double)c->stats.bytesUpCompressed / c->stats.bytesUp);
        auto downComp = renderPercent(1.0 - (double)c->stats.bytesDownCompressed / c->stats.bytesDown);

-        LI << "[" << connId << "] Disconnect from " << c->ipAddr
+        LI << "[" << connId << "] Disconnect from " << renderIP(c->ipAddr)
           << " UP: " << renderSize(c->stats.bytesUp) << " (" << upComp << " compressed)"
           << " DN: " << renderSize(c->stats.bytesDown) << " (" << downComp << " compressed)"
        ;
@ -173,7 +145,7 @@ void RelayServer::runWebsocket(ThreadPool<MsgWebsocket>::Thread &thr) {
        c.stats.bytesDown += length;
        c.stats.bytesDownCompressed += compressedSize;

-        tpIngester.dispatch(c.connId, MsgIngester{MsgIngester::ClientMessage{c.connId, std::string(message, length)}});
+        tpIngester.dispatch(c.connId, MsgIngester{MsgIngester::ClientMessage{c.connId, c.ipAddr, std::string(message, length)}});
    });


@ -198,15 +170,18 @@ void RelayServer::runWebsocket(ThreadPool<MsgWebsocket>::Thread &thr) {
            } else if (auto msg = std::get_if<MsgWebsocket::SendBinary>(&newMsg.msg)) {
                doSend(msg->connId, msg->payload, uWS::OpCode::BINARY);
            } else if (auto msg = std::get_if<MsgWebsocket::SendEventToBatch>(&newMsg.msg)) {
-                for (auto &item : msg->list) {
-                    tempBuf.clear();
-                    tempBuf += "[\"EVENT\",\"";
-                    tempBuf += item.subId.sv();
-                    tempBuf += "\",";
-                    tempBuf += msg->evJson;
-                    tempBuf += "]";
+                tempBuf.reserve(13 + MAX_SUBID_SIZE + msg->evJson.size());
+                tempBuf.resize(10 + MAX_SUBID_SIZE);
+                tempBuf += "\",";
+                tempBuf += msg->evJson;
+                tempBuf += "]";

-                    doSend(item.connId, tempBuf, uWS::OpCode::TEXT);
+                for (auto &item : msg->list) {
+                    auto subIdSv = item.subId.sv();
+                    auto *p = tempBuf.data() + MAX_SUBID_SIZE - subIdSv.size();
+                    memcpy(p, "[\"EVENT\",\"", 10);
+                    memcpy(p + 10, subIdSv.data(), subIdSv.size());
+                    doSend(item.connId, std::string_view(p, 13 + subIdSv.size() + msg->evJson.size()), uWS::OpCode::TEXT);
                }
            }
        }
--- a/src/RelayWriter.cpp
+++ b/src/RelayWriter.cpp
@ -1,13 +1,12 @@
 #include "RelayServer.h"

+#include "PluginWritePolicy.h"
+

 void RelayServer::runWriter(ThreadPool<MsgWriter>::Thread &thr) {
-    quadrable::Quadrable qdb;
-    {
-        auto txn = env.txn_ro();
-        qdb.init(txn);
-    }
-    qdb.checkout("events");
+    auto qdb = getQdbInstance();
+
+    PluginWritePolicy writePolicy;

    while(1) {
        auto newMsgs = thr.inbox.pop_all();
@ -18,7 +17,20 @@ void RelayServer::runWriter(ThreadPool<MsgWriter>::Thread &thr) {

        for (auto &newMsg : newMsgs) {
            if (auto msg = std::get_if<MsgWriter::AddEvent>(&newMsg.msg)) {
-                newEvents.emplace_back(std::move(msg->flatStr), std::move(msg->jsonStr), msg->receivedAt, msg);
+                EventSourceType sourceType = msg->ipAddr.size() == 4 ? EventSourceType::IP4 : EventSourceType::IP6;
+                std::string okMsg;
+                auto res = writePolicy.acceptEvent(msg->jsonStr, msg->receivedAt, sourceType, msg->ipAddr, okMsg);
+
+                if (res == WritePolicyResult::Accept) {
+                    newEvents.emplace_back(std::move(msg->flatStr), std::move(msg->jsonStr), msg->receivedAt, sourceType, std::move(msg->ipAddr), msg);
+                } else {
+                    auto *flat = flatbuffers::GetRoot<NostrIndex::Event>(msg->flatStr.data());
+                    auto eventIdHex = to_hex(sv(flat->id()));
+
+                    LI << "[" << msg->connId << "] write policy blocked event " << eventIdHex << ": " << okMsg;
+
+                    sendOKResponse(msg->connId, eventIdHex, res == WritePolicyResult::ShadowReject, okMsg);
+                }
            }
        }

@ -37,7 +49,7 @@ void RelayServer::runWriter(ThreadPool<MsgWriter>::Thread &thr) {
            bool written = false;

            if (newEvent.status == EventWriteStatus::Written) {
-                LI << "Inserted event. id=" << eventIdHex << " qdbNodeId=" << newEvent.nodeId;
+                LI << "Inserted event. id=" << eventIdHex << " levId=" << newEvent.levId;
                written = true;
            } else if (newEvent.status == EventWriteStatus::Duplicate) {
                message = "duplicate: have this event";
--- a/src/RelayYesstr.cpp
+++ b/src/RelayYesstr.cpp
@ -6,12 +6,7 @@


 void RelayServer::runYesstr(ThreadPool<MsgYesstr>::Thread &thr) {
-    quadrable::Quadrable qdb;
-    {
-        auto txn = env.txn_ro();
-        qdb.init(txn);
-    }
-
+    auto qdb = getQdbInstance();

    struct SyncState {
        quadrable::MemStore m;
@ -20,7 +15,7 @@ void RelayServer::runYesstr(ThreadPool<MsgYesstr>::Thread &thr) {
    struct SyncStateCollection {
        RelayServer *server;
        quadrable::Quadrable *qdb;
-        std::map<uint64_t, std::map<uint64_t, SyncState>> conns; // connId -> reqId -> SyncState
+        flat_hash_map<uint64_t, flat_hash_map<uint64_t, SyncState>> conns; // connId -> reqId -> SyncState

        SyncStateCollection(RelayServer *server_, quadrable::Quadrable *qdb_) : server(server_), qdb(qdb_) {}

@ -53,20 +48,20 @@ void RelayServer::runYesstr(ThreadPool<MsgYesstr>::Thread &thr) {
                // FIXME: The following blocks the whole thread for the query duration. Should interleave it
                // with other requests like RelayReqWorker does.

-                std::vector<uint64_t> quadEventIds;
+                std::vector<uint64_t> levIds;
                auto filterGroup = NostrFilterGroup::unwrapped(tao::json::from_string(filterStr));
                Subscription sub(1, "junkSub", filterGroup);
                DBScanQuery query(sub);

                while (1) {
-                    bool complete = query.process(txn, MAX_U64, cfg().relay__logging__dbScanPerf, [&](const auto &sub, uint64_t quadId){
-                        quadEventIds.push_back(quadId);
+                    bool complete = query.process(txn, MAX_U64, cfg().relay__logging__dbScanPerf, [&](const auto &sub, uint64_t levId){
+                        levIds.push_back(levId);
                    });

                    if (complete) break;
                }

-                LI << "Filter matched " << quadEventIds.size() << " local events";
+                LI << "Filter matched " << levIds.size() << " local events";

                qdb->withMemStore(s.m, [&]{
                    qdb->writeToMemStore = true;
@ -74,8 +69,8 @@ void RelayServer::runYesstr(ThreadPool<MsgYesstr>::Thread &thr) {

                    auto changes = qdb->change();

-                    for (auto id : quadEventIds) {
-                        changes.putReuse(txn, id);
+                    for (auto levId : levIds) {
+                        changes.putReuse(txn, levId);
                    }

                    changes.apply(txn);
--- a/src/Subscription.h
+++ b/src/Subscription.h
@ -1,14 +1,16 @@
 #pragma once

+#include <parallel_hashmap/phmap_utils.h>
+
 #include "filters.h"


 struct SubId {
-    char buf[64];
+    char buf[72];

    SubId(std::string_view val) {
-        static_assert(MAX_SUBID_SIZE == 63, "MAX_SUBID_SIZE mismatch");
-        if (val.size() > 63) throw herr("subscription id too long");
+        static_assert(MAX_SUBID_SIZE == 71, "MAX_SUBID_SIZE mismatch");
+        if (val.size() > 71) throw herr("subscription id too long");
        if (val.size() == 0) throw herr("subscription id too short");

        auto badChar = [](char c){
@ -28,10 +30,19 @@ struct SubId {
    std::string str() const {
        return std::string(sv());
    }
+
+    bool operator==(const SubId &o) const {
+        return o.sv() == sv();
+    }
 };

-inline bool operator <(const SubId &s1, const SubId &s2) {
-    return s1.sv() < s2.sv();
+namespace std {
+    // inject specialization of std::hash
+    template<> struct hash<SubId> {
+        std::size_t operator()(SubId const &p) const {
+            return phmap::HashState().combine(0, p.sv());
+        }
+    };
 }


--- a/src/WriterPipeline.h
+++ b/src/WriterPipeline.h
@ -7,9 +7,16 @@
 #include "events.h"


+struct WriterPipelineInput {
+    tao::json::value eventJson;
+    EventSourceType sourceType;
+    std::string sourceInfo;
+};
+
+
 struct WriterPipeline {
  public:
-    hoytech::protected_queue<tao::json::value> inbox;
+    hoytech::protected_queue<WriterPipelineInput> inbox;
    hoytech::protected_queue<bool> flushInbox;

  private:
@ -28,7 +35,7 @@ struct WriterPipeline {
                auto msgs = inbox.pop_all();

                for (auto &m : msgs) {
-                    if (m.is_null()) {
+                    if (m.eventJson.is_null()) {
                        writerInbox.push_move({});
                        break;
                    }
@ -37,13 +44,13 @@ struct WriterPipeline {
                    std::string jsonStr;

                    try {
-                        parseAndVerifyEvent(m, secpCtx, true, true, flatStr, jsonStr);
+                        parseAndVerifyEvent(m.eventJson, secpCtx, true, true, flatStr, jsonStr);
                    } catch (std::exception &e) {
-                        LW << "Rejected event: " << m << " reason: " << e.what();
+                        LW << "Rejected event: " << m.eventJson << " reason: " << e.what();
                        continue;
                    }

-                    writerInbox.push_move({ std::move(flatStr), std::move(jsonStr), hoytech::curr_time_us() });
+                    writerInbox.push_move({ std::move(flatStr), std::move(jsonStr), hoytech::curr_time_us(), m.sourceType, std::move(m.sourceInfo) });
                }
            }
        });
@ -51,12 +58,7 @@ struct WriterPipeline {
        writerThread = std::thread([&]() {
            setThreadName("Writer");

-            quadrable::Quadrable qdb;
-            {
-                auto txn = env.txn_ro();
-                qdb.init(txn);
-            }
-            qdb.checkout("events");
+            auto qdb = getQdbInstance();

            while (1) {
                // Debounce
@ -120,7 +122,7 @@ struct WriterPipeline {
    }

    void flush() {
-        inbox.push_move(tao::json::null);
+        inbox.push_move({ tao::json::null, EventSourceType::None, "" });
        flushInbox.wait();
    }
 };
--- a/src/cmd_compact.cpp
+++ b/src/cmd_compact.cpp
@ -4,6 +4,8 @@
 #include <docopt.h>
 #include "golpe.h"

+#include "gc.h"
+

 static const char USAGE[] =
 R"(
--- a/src/cmd_delete.cpp
+++ b/src/cmd_delete.cpp
@ -0,0 +1,93 @@
+#include <iostream>
+
+#include <docopt.h>
+#include "golpe.h"
+
+#include "DBScan.h"
+#include "events.h"
+#include "gc.h"
+
+
+static const char USAGE[] =
+R"(
+    Usage:
+      delete [--age=<age>] [--filter=<filter>] [--dry-run] [--no-gc]
+)";
+
+
+void cmd_delete(const std::vector<std::string> &subArgs) {
+    std::map<std::string, docopt::value> args = docopt::docopt(USAGE, subArgs, true, "");
+
+    uint64_t age = MAX_U64;
+    if (args["--age"]) age = args["--age"].asLong();
+
+    std::string filterStr;
+    if (args["--filter"]) filterStr = args["--filter"].asString();
+
+    bool dryRun = args["--dry-run"].asBool();
+    bool noGc = args["--no-gc"].asBool();
+
+
+
+    if (filterStr.size() == 0 && age == MAX_U64) throw herr("must specify --age and/or --filter");
+    if (filterStr.size() == 0) filterStr = "{}";
+
+
+    auto filter = tao::json::from_string(filterStr);
+    auto now = hoytech::curr_time_s();
+
+    if (age != MAX_U64) {
+        if (age > now) age = now;
+        if (filter.optional<uint64_t>("until")) throw herr("--age is not compatible with filter containing 'until'");
+
+        filter["until"] = now - age;
+    }
+
+
+    auto filterGroup = NostrFilterGroup::unwrapped(filter, MAX_U64);
+    Subscription sub(1, "junkSub", filterGroup);
+    DBScanQuery query(sub);
+
+
+    btree_set<uint64_t> levIds;
+
+    {
+        auto txn = env.txn_ro();
+
+        while (1) {
+            bool complete = query.process(txn, MAX_U64, false, [&](const auto &sub, uint64_t levId){
+                levIds.insert(levId);
+            });
+
+            if (complete) break;
+        }
+    }
+
+    if (dryRun) {
+        LI << "Would delete " << levIds.size() << " events";
+        return;
+    }
+
+
+    auto qdb = getQdbInstance();
+
+    LI << "Deleting " << levIds.size() << " events";
+
+    {
+        auto txn = env.txn_rw();
+
+        auto changes = qdb.change();
+
+        for (auto levId : levIds) {
+            auto view = env.lookup_Event(txn, levId);
+            if (!view) continue; // Deleted in between transactions
+            deleteEvent(txn, changes, *view);
+        }
+
+        changes.apply(txn);
+
+        txn.commit();
+    }
+
+    if (!noGc) quadrableGarbageCollect(qdb, 2);
+}
--- a/src/cmd_dict.cpp
+++ b/src/cmd_dict.cpp
@ -0,0 +1,242 @@
+#include <zstd.h>
+#include <zdict.h>
+
+#include <iostream>
+#include <random>
+
+#include <docopt.h>
+#include "golpe.h"
+
+#include "DBScan.h"
+#include "events.h"
+
+
+static const char USAGE[] =
+R"(
+    Usage:
+      dict stats [--filter=<filter>]
+      dict train [--filter=<filter>] [--limit=<limit>] [--dictSize=<dictSize>]
+      dict compress [--filter=<filter>] [--dictId=<dictId>] [--level=<level>]
+      dict decompress [--filter=<filter>]
+)";
+
+
+void cmd_dict(const std::vector<std::string> &subArgs) {
+    std::map<std::string, docopt::value> args = docopt::docopt(USAGE, subArgs, true, "");
+
+    std::string filterStr;
+    if (args["--filter"]) filterStr = args["--filter"].asString();
+    else filterStr = "{}";
+
+    uint64_t limit = MAX_U64;
+    if (args["--limit"]) limit = args["--limit"].asLong();
+
+    uint64_t dictSize = 100'000;
+    if (args["--dictSize"]) dictSize = args["--dictSize"].asLong();
+
+    uint64_t dictId = 0;
+    if (args["--dictId"]) dictId = args["--dictId"].asLong();
+
+    int level = 3;
+    if (args["--level"]) level = args["--level"].asLong();
+
+
+    Decompressor decomp;
+    std::vector<uint64_t> levIds;
+
+
+    auto txn = env.txn_ro();
+
+    auto filterGroup = NostrFilterGroup::unwrapped(tao::json::from_string(filterStr), MAX_U64);
+    Subscription sub(1, "junkSub", filterGroup);
+    DBScanQuery query(sub);
+
+    while (1) {
+        bool complete = query.process(txn, MAX_U64, false, [&](const auto &sub, uint64_t levId){
+            levIds.push_back(levId);
+        });
+
+        if (complete) break;
+    }
+
+    LI << "Filter matched " << levIds.size() << " records";
+
+
+    if (args["stats"].asBool()) {
+        uint64_t totalSize = 0;
+        uint64_t totalCompressedSize = 0;
+        uint64_t numCompressed = 0;
+
+        btree_map<uint32_t, uint64_t> dicts;
+
+        env.foreach_CompressionDictionary(txn, [&](auto &view){
+            auto dictId = view.primaryKeyId;
+            if (!dicts.contains(dictId)) dicts[dictId] = 0;
+            return true;
+        });
+
+        for (auto levId : levIds) {
+            std::string_view raw;
+
+            bool found = env.dbi_EventPayload.get(txn, lmdb::to_sv<uint64_t>(levId), raw);
+            if (!found) throw herr("couldn't find event in EventPayload, corrupted DB?");
+
+            uint32_t dictId;
+            size_t outCompressedSize;
+
+            auto json = decodeEventPayload(txn, decomp, raw, &dictId, &outCompressedSize);
+
+            totalSize += json.size();
+            totalCompressedSize += dictId ? outCompressedSize : json.size();
+
+            if (dictId) {
+                numCompressed++;
+                dicts[dictId]++;
+            }
+        }
+
+        auto ratio = renderPercent(1.0 - (double)totalCompressedSize / totalSize);
+
+        std::cout << "Num compressed: " << numCompressed << " / " << levIds.size() << "\n";
+        std::cout << "Uncompressed size: " << renderSize(totalSize) << "\n";
+        std::cout << "Compressed size:   " << renderSize(totalCompressedSize) << " (" << ratio << ")" << "\n";
+        std::cout << "\ndictId : events\n";
+
+        for (auto &[dictId, n] : dicts) {
+            std::cout << "  " << dictId << " : " << n << "\n";
+        }
+    } else if (args["train"].asBool()) {
+        std::string trainingBuf;
+        std::vector<size_t> trainingSizes;
+
+        if (levIds.size() > limit) {
+            LI << "Randomly selecting " << limit << " records";
+            std::random_device rd;
+            std::mt19937 g(rd());
+            std::shuffle(levIds.begin(), levIds.end(), g);
+            levIds.resize(limit);
+        }
+
+        for (auto levId : levIds) {
+            std::string json = std::string(getEventJson(txn, decomp, levId));
+            trainingBuf += json;
+            trainingSizes.emplace_back(json.size());
+        }
+
+        std::string dict(dictSize, '\0');
+
+        LI << "Performing zstd training...";
+
+        auto ret = ZDICT_trainFromBuffer(dict.data(), dict.size(), trainingBuf.data(), trainingSizes.data(), trainingSizes.size());
+        if (ZDICT_isError(ret)) throw herr("zstd training failed: ", ZSTD_getErrorName(ret));
+
+        txn.abort();
+        txn = env.txn_rw();
+
+        uint64_t newDictId = env.insert_CompressionDictionary(txn, dict);
+
+        std::cout << "Saved new dictionary, dictId = " << newDictId << std::endl;
+
+        txn.commit();
+    } else if (args["compress"].asBool()) {
+        if (dictId == 0) throw herr("specify --dictId or --decompress");
+
+        txn.abort();
+        txn = env.txn_rw();
+
+        auto view = env.lookup_CompressionDictionary(txn, dictId);
+        if (!view) throw herr("couldn't find dictId ", dictId);
+        auto dict = view->dict();
+
+        auto *cctx = ZSTD_createCCtx();
+        auto *cdict = ZSTD_createCDict(dict.data(), dict.size(), level);
+
+        uint64_t origSizes = 0;
+        uint64_t compressedSizes = 0;
+        uint64_t pendingFlush = 0;
+        uint64_t processed = 0;
+
+        std::string compressedData(500'000, '\0');
+
+        for (auto levId : levIds) {
+            std::string_view orig;
+
+            try {
+                orig = getEventJson(txn, decomp, levId);
+            } catch (std::exception &e) {
+                continue;
+            }
+
+            auto ret = ZSTD_compress_usingCDict(cctx, compressedData.data(), compressedData.size(), orig.data(), orig.size(), cdict);
+            if (ZDICT_isError(ret)) throw herr("zstd compression failed: ", ZSTD_getErrorName(ret));
+
+            origSizes += orig.size();
+            compressedSizes += ret;
+
+            std::string newVal;
+
+            if (ret + 4 < orig.size()) {
+                newVal += '\x01';
+                newVal += lmdb::to_sv<uint32_t>(dictId);
+                newVal += std::string_view(compressedData.data(), ret);
+            } else {
+                newVal += '\x00';
+                newVal += orig;
+            }
+
+            env.dbi_EventPayload.put(txn, lmdb::to_sv<uint64_t>(levId), newVal);
+
+            pendingFlush++;
+            processed++;
+            if (pendingFlush > 10'000) {
+                txn.commit();
+
+                LI << "Progress: " << processed << "/" << levIds.size();
+                pendingFlush = 0;
+
+                txn = env.txn_rw();
+            }
+        }
+
+        txn.commit();
+
+        LI << "Original event sizes: " << origSizes;
+        LI << "New event sizes:      " << compressedSizes;
+    } else if (args["decompress"].asBool()) {
+        txn.abort();
+        txn = env.txn_rw();
+
+        uint64_t pendingFlush = 0;
+        uint64_t processed = 0;
+
+        for (auto levId : levIds) {
+            std::string_view orig;
+
+            try {
+                orig = getEventJson(txn, decomp, levId);
+            } catch (std::exception &e) {
+                continue;
+            }
+
+            std::string newVal;
+
+            newVal += '\x00';
+            newVal += orig;
+
+            env.dbi_EventPayload.put(txn, lmdb::to_sv<uint64_t>(levId), newVal);
+
+            pendingFlush++;
+            processed++;
+            if (pendingFlush > 10'000) {
+                txn.commit();
+
+                LI << "Progress: " << processed << "/" << levIds.size();
+                pendingFlush = 0;
+
+                txn = env.txn_rw();
+            }
+        }
+
+        txn.commit();
+    }
+}
--- a/src/cmd_export.cpp
+++ b/src/cmd_export.cpp
@ -20,19 +20,32 @@ void cmd_export(const std::vector<std::string> &subArgs) {
    if (args["--since"]) since = args["--since"].asLong();
    if (args["--until"]) until = args["--until"].asLong();

+    Decompressor decomp;
+
    auto txn = env.txn_ro();

+    auto dbVersion = getDBVersion(txn);
+    auto qdb = getQdbInstance(txn);
+
    env.generic_foreachFull(txn, env.dbi_Event__created_at, lmdb::to_sv<uint64_t>(since), lmdb::to_sv<uint64_t>(0), [&](auto k, auto v) {
        if (lmdb::from_sv<uint64_t>(k) > until) return false;

        auto view = env.lookup_Event(txn, lmdb::from_sv<uint64_t>(v));
        if (!view) throw herr("missing event from index, corrupt DB?");

+        if (dbVersion == 0) {
+            std::string_view raw;
+            bool found = qdb.dbi_nodesLeaf.get(txn, lmdb::to_sv<uint64_t>(view->primaryKeyId), raw);
+            if (!found) throw herr("couldn't find leaf node in quadrable, corrupted DB?");
+            std::cout << raw.substr(8 + 32 + 32) << "\n";
+            return true;
+        }
+
        if (!args["--include-ephemeral"].asBool()) {
            if (isEphemeralEvent(view->flat_nested()->kind())) return true;
        }

-        std::cout << getEventJson(txn, view->primaryKeyId) << "\n";
+        std::cout << getEventJson(txn, decomp, view->primaryKeyId) << "\n";

        return true;
    });
--- a/src/cmd_gc.cpp
+++ b/src/cmd_gc.cpp
@ -0,0 +1,22 @@
+#include <unistd.h>
+#include <stdio.h>
+
+#include <docopt.h>
+#include "golpe.h"
+
+#include "gc.h"
+
+
+static const char USAGE[] =
+R"(
+    Usage:
+      gc
+)";
+
+
+void cmd_gc(const std::vector<std::string> &subArgs) {
+    std::map<std::string, docopt::value> args = docopt::docopt(USAGE, subArgs, true, "");
+
+    auto qdb = getQdbInstance();
+    quadrableGarbageCollect(qdb, 2);
+}
--- a/src/cmd_import.cpp
+++ b/src/cmd_import.cpp
@ -5,12 +5,13 @@

 #include "events.h"
 #include "filters.h"
+#include "gc.h"


 static const char USAGE[] =
 R"(
    Usage:
-      import [--show-rejected] [--no-verify]
+      import [--show-rejected] [--no-verify] [--no-gc]
 )";


@ -19,15 +20,11 @@ void cmd_import(const std::vector<std::string> &subArgs) {

    bool showRejected = args["--show-rejected"].asBool();
    bool noVerify = args["--no-verify"].asBool();
+    bool noGc = args["--no-gc"].asBool();

    if (noVerify) LW << "not verifying event IDs or signatures!";

-    quadrable::Quadrable qdb;
-    {
-        auto txn = env.txn_ro();
-        qdb.init(txn);
-    }
-    qdb.checkout("events");
+    auto qdb = getQdbInstance();

    auto txn = env.txn_rw();

@ -42,7 +39,7 @@ void cmd_import(const std::vector<std::string> &subArgs) {
    };

    auto flushChanges = [&]{
-        writeEvents(txn, qdb, newEvents);
+        writeEvents(txn, qdb, newEvents, 0);

        uint64_t numCommits = 0;

@ -59,6 +56,7 @@ void cmd_import(const std::vector<std::string> &subArgs) {

        logStatus();
        LI << "Committing " << numCommits << " records";
+
        txn.commit();

        txn = env.txn_rw();
@ -84,7 +82,7 @@ void cmd_import(const std::vector<std::string> &subArgs) {
            continue;
        }

-        newEvents.emplace_back(std::move(flatStr), std::move(jsonStr), hoytech::curr_time_us());
+        newEvents.emplace_back(std::move(flatStr), std::move(jsonStr), hoytech::curr_time_us(), EventSourceType::Import, "");

        if (newEvents.size() >= 10'000) flushChanges();
    }
@ -92,4 +90,6 @@ void cmd_import(const std::vector<std::string> &subArgs) {
    flushChanges();

    txn.commit();
+
+    if (!noGc) quadrableGarbageCollect(qdb, 2);
 }
--- a/src/cmd_info.cpp
+++ b/src/cmd_info.cpp
@ -14,14 +14,10 @@ R"(
 void cmd_info(const std::vector<std::string> &subArgs) {
    std::map<std::string, docopt::value> args = docopt::docopt(USAGE, subArgs, true, "");

-    quadrable::Quadrable qdb;
-    {
-        auto txn = env.txn_ro();
-        qdb.init(txn);
-    }
-    qdb.checkout("events");
+    auto qdb = getQdbInstance();

    auto txn = env.txn_ro();

+    std::cout << "DB version: " << getDBVersion(txn) << "\n";
    std::cout << "merkle root: " << to_hex(qdb.root(txn)) << "\n";
 }
--- a/src/cmd_monitor.cpp
+++ b/src/cmd_monitor.cpp
@ -21,6 +21,7 @@ void cmd_monitor(const std::vector<std::string> &subArgs) {

    auto txn = env.txn_ro();

+    Decompressor decomp;
    ActiveMonitors monitors;

    std::string line;
@ -54,10 +55,10 @@ void cmd_monitor(const std::vector<std::string> &subArgs) {
    }

    env.foreach_Event(txn, [&](auto &ev){
-        monitors.process(txn, ev, [&](RecipientList &&recipients, uint64_t quadId){
+        monitors.process(txn, ev, [&](RecipientList &&recipients, uint64_t levId){
            for (auto &r : recipients) {
                if (r.connId == interestConnId && r.subId.str() == interestSubId) {
-                    std::cout << getEventJson(txn, quadId) << "\n";
+                    std::cout << getEventJson(txn, decomp, levId) << "\n";
                }
            }
        });
--- a/src/cmd_relay.cpp
+++ b/src/cmd_relay.cpp
@ -48,6 +48,10 @@ void RelayServer::run() {
        cleanupOldEvents();
    });

+    cron.repeat(60 * 60 * 1'000'000UL, [&]{
+        garbageCollect();
+    });
+
    cron.setupCb = []{ setThreadName("cron"); };

    cron.run();
--- a/src/cmd_scan.cpp
+++ b/src/cmd_scan.cpp
@ -20,23 +20,23 @@ void cmd_scan(const std::vector<std::string> &subArgs) {
    uint64_t pause = 0;
    if (args["--pause"]) pause = args["--pause"].asLong();

-    bool metrics = false;
-    if (args["--metrics"]) metrics = true;
-
+    bool metrics = args["--metrics"].asBool();

    std::string filterStr = args["<filter>"].asString();
-    auto filterGroup = NostrFilterGroup::unwrapped(tao::json::from_string(filterStr));

+
+    auto filterGroup = NostrFilterGroup::unwrapped(tao::json::from_string(filterStr), MAX_U64);
    Subscription sub(1, "junkSub", filterGroup);
-
    DBScanQuery query(sub);


+    Decompressor decomp;
+
    auto txn = env.txn_ro();

    while (1) {
-        bool complete = query.process(txn, pause ? pause : MAX_U64, metrics, [&](const auto &sub, uint64_t quadId){
-            std::cout << getEventJson(txn, quadId) << "\n";
+        bool complete = query.process(txn, pause ? pause : MAX_U64, metrics, [&](const auto &sub, uint64_t levId){
+            std::cout << getEventJson(txn, decomp, levId) << "\n";
        });

        if (complete) break;
--- a/src/cmd_stream.cpp
+++ b/src/cmd_stream.cpp
@ -31,9 +31,10 @@ void cmd_stream(const std::vector<std::string> &subArgs) {
    if (dir != "up" && dir != "down" && dir != "both") throw herr("invalid direction: ", dir, ". Should be one of up/down/both");


-    std::unordered_set<std::string> downloadedIds;
+    flat_hash_set<std::string> downloadedIds;
    WriterPipeline writer;
    WSConnection ws(url);
+    Decompressor decomp;

    ws.onConnect = [&]{
        if (dir == "down" || dir == "both") {
@ -63,7 +64,7 @@ void cmd_stream(const std::vector<std::string> &subArgs) {
                    if (origJson.get_array().size() < 3) throw herr("array too short");
                    auto &evJson = origJson.at(2);
                    downloadedIds.emplace(from_hex(evJson.at("id").get_string()));
-                    writer.inbox.push_move(std::move(evJson));
+                    writer.inbox.push_move({ std::move(evJson), EventSourceType::Stream, url });
                } else {
                    LW << "Unexpected EVENT";
                }
@ -80,7 +81,7 @@ void cmd_stream(const std::vector<std::string> &subArgs) {

    {
        auto txn = env.txn_ro();
-        currEventId = getMostRecentEventId(txn);
+        currEventId = getMostRecentLevId(txn);
    }

    ws.onTrigger = [&]{
@ -98,7 +99,7 @@ void cmd_stream(const std::vector<std::string> &subArgs) {
            }

            std::string msg = std::string("[\"EVENT\",");
-            msg += getEventJson(txn, ev.primaryKeyId);
+            msg += getEventJson(txn, decomp, ev.primaryKeyId);
            msg += "]";

            ws.send(msg);
--- a/src/cmd_sync.cpp
+++ b/src/cmd_sync.cpp
@ -133,22 +133,14 @@ void cmd_sync(const std::vector<std::string> &subArgs) {
    std::unique_ptr<SyncController> controller;
    WriterPipeline writer;
    WSConnection ws(url);
-
-    quadrable::Quadrable qdb;
-    {
-        auto txn = env.txn_ro();
-        qdb.init(txn);
-    }
-    qdb.checkout("events");
-
+    auto qdb = getQdbInstance();


    ws.reconnect = false;


-
    if (filterStr.size()) {
-        std::vector<uint64_t> quadEventIds;
+        std::vector<uint64_t> levIds;

        tao::json::value filterJson;

@ -167,14 +159,14 @@ void cmd_sync(const std::vector<std::string> &subArgs) {
        auto txn = env.txn_ro();

        while (1) {
-            bool complete = query.process(txn, MAX_U64, false, [&](const auto &sub, uint64_t quadId){
-                quadEventIds.push_back(quadId);
+            bool complete = query.process(txn, MAX_U64, false, [&](const auto &sub, uint64_t levId){
+                levIds.push_back(levId);
            });

            if (complete) break;
        }

-        LI << "Filter matched " << quadEventIds.size() << " local events";
+        LI << "Filter matched " << levIds.size() << " local events";

        controller = std::make_unique<SyncController>(&qdb, &ws);

@ -184,8 +176,8 @@ void cmd_sync(const std::vector<std::string> &subArgs) {

            auto changes = qdb.change();

-            for (auto id : quadEventIds) {
-                changes.putReuse(txn, id);
+            for (auto levId : levIds) {
+                changes.putReuse(txn, levId);
            }

            changes.apply(txn);
@ -228,7 +220,7 @@ void cmd_sync(const std::vector<std::string> &subArgs) {
            controller->finish(txn,
                [&](std::string_view newLeaf){
                    // FIXME: relay could crash client here by sending invalid JSON
-                    writer.inbox.push_move(tao::json::from_string(std::string(newLeaf)));
+                    writer.inbox.push_move(WriterPipelineInput{ tao::json::from_string(std::string(newLeaf)), EventSourceType::Sync, url });
                },
                [&](std::string_view){
                }
--- a/src/constants.h
+++ b/src/constants.h
@ -1,3 +1,6 @@
 #pragma once

-const size_t MAX_SUBID_SIZE = 63;
+const uint64_t CURR_DB_VERSION = 1;
+const size_t MAX_SUBID_SIZE = 71; // Statically allocated size in SubId
+const uint64_t MAX_TIMESTAMP = 17179869184; // Safety limit to ensure it can fit in quadrable key. Good until year 2514.
+const size_t MAX_INDEXED_TAG_VAL_SIZE = 255;
--- a/src/events.cpp
+++ b/src/events.cpp
@ -8,41 +8,65 @@ std::string nostrJsonToFlat(const tao::json::value &v) {

    // Extract values from JSON, add strings to builder

-    auto loadHexStr = [&](std::string_view k, uint64_t size){
-        auto s = from_hex(v.at(k).get_string(), false);
-        if (s.size() != size) throw herr("unexpected size of hex data");
-        return builder.CreateVector((uint8_t*)s.data(), s.size());
-    };
-
-    auto idPtr = loadHexStr("id", 32);
-    auto pubkeyPtr = loadHexStr("pubkey", 32);
+    auto id = from_hex(v.at("id").get_string(), false);
+    auto pubkey = from_hex(v.at("pubkey").get_string(), false);
    uint64_t created_at = v.at("created_at").get_unsigned();
    uint64_t kind = v.at("kind").get_unsigned();

-    std::vector<flatbuffers::Offset<NostrIndex::Tag>> tagPtrs;
+    if (id.size() != 32) throw herr("unexpected id size");
+    if (pubkey.size() != 32) throw herr("unexpected pubkey size");
+
+    std::vector<flatbuffers::Offset<NostrIndex::TagGeneral>> tagsGeneral;
+    std::vector<flatbuffers::Offset<NostrIndex::TagFixed32>> tagsFixed32;
+
+    uint64_t expiration = 0;
+
    if (v.at("tags").get_array().size() > cfg().events__maxNumTags) throw herr("too many tags: ", v.at("tags").get_array().size());
    for (auto &tagArr : v.at("tags").get_array()) {
        auto &tag = tagArr.get_array();
        if (tag.size() < 2) throw herr("too few fields in tag");

        auto tagName = tag.at(0).get_string();
-        if (tagName.size() != 1) continue; // only single-char tags need indexing

        auto tagVal = tag.at(1).get_string();
-        if (tagVal.size() < 1 || tagVal.size() > cfg().events__maxTagValSize) throw herr("tag val too small/large: ", tagVal.size());
+
        if (tagName == "e" || tagName == "p") {
            tagVal = from_hex(tagVal, false);
-            if (tagVal.size() != 32) throw herr("unexpected size for e/p tag");
-        }
-        auto tagValPtr = builder.CreateVector((uint8_t*)tagVal.data(), tagVal.size());
+            if (tagVal.size() != 32) throw herr("unexpected size for fixed-size tag");

-        tagPtrs.push_back(NostrIndex::CreateTag(builder, (uint8_t)tagName[0], tagValPtr));
+            tagsFixed32.emplace_back(NostrIndex::CreateTagFixed32(builder,
+                (uint8_t)tagName[0],
+                (NostrIndex::Fixed32Bytes*)tagVal.data()
+            ));
+        } else if (tagName == "expiration") {
+            if (expiration == 0) {
+                expiration = parseUint64(tagVal);
+                if (expiration == 0) expiration = 1; // special value to indicate expiration of 0 was set
+            }
+        } else if (tagName.size() == 1) {
+            if (tagVal.size() == 0) throw herr("tag val empty");
+            if (tagVal.size() > cfg().events__maxTagValSize) throw herr("tag val too large: ", tagVal.size());
+
+            if (tagVal.size() <= MAX_INDEXED_TAG_VAL_SIZE) {
+                tagsGeneral.emplace_back(NostrIndex::CreateTagGeneral(builder,
+                    (uint8_t)tagName[0],
+                    builder.CreateVector((uint8_t*)tagVal.data(), tagVal.size())
+                ));
+            }
+        }
    }
-    auto tagsPtr = builder.CreateVector<flatbuffers::Offset<NostrIndex::Tag>>(tagPtrs);

    // Create flatbuffer

-    auto eventPtr = NostrIndex::CreateEvent(builder, idPtr, pubkeyPtr, created_at, kind, tagsPtr);
+    auto eventPtr = NostrIndex::CreateEvent(builder,
+        (NostrIndex::Fixed32Bytes*)id.data(),
+        (NostrIndex::Fixed32Bytes*)pubkey.data(),
+        created_at,
+        kind,
+        builder.CreateVector<flatbuffers::Offset<NostrIndex::TagGeneral>>(tagsGeneral),
+        builder.CreateVector<flatbuffers::Offset<NostrIndex::TagFixed32>>(tagsFixed32),
+        expiration
+    );

    builder.Finish(eventPtr);

@ -105,7 +129,9 @@ void verifyEventTimestamp(const NostrIndex::Event *flat) {
    uint64_t latest = now + cfg().events__rejectEventsNewerThanSeconds;

    if (ts < earliest) throw herr("created_at too early");
-    if (ts > latest) throw herr("created_at too late");
+    if (ts > latest || ts > MAX_TIMESTAMP) throw herr("created_at too late");
+
+    if (flat->expiration() != 0 && flat->expiration() <= now) throw herr("event expired");
 }

 void parseAndVerifyEvent(const tao::json::value &origJson, secp256k1_context *secpCtx, bool verifyMsg, bool verifyTime, std::string &flatStr, std::string &jsonStr) {
@ -144,32 +170,71 @@ std::optional<defaultDb::environment::View_Event> lookupEventById(lmdb::txn &txn
    return output;
 }

-uint64_t getMostRecentEventId(lmdb::txn &txn) {
-    uint64_t output = 0;
+uint64_t getMostRecentLevId(lmdb::txn &txn) {
+    uint64_t levId = 0;

    env.foreach_Event(txn, [&](auto &ev){
-        output = ev.primaryKeyId;
+        levId = ev.primaryKeyId;
        return false;
    }, true);

-    return output;
+    return levId;
 }

-std::string_view getEventJson(lmdb::txn &txn, uint64_t quadId) {
+
+// Return result validity same as getEventJson(), see below
+
+std::string_view decodeEventPayload(lmdb::txn &txn, Decompressor &decomp, std::string_view raw, uint32_t *outDictId, size_t *outCompressedSize) {
+    if (raw.size() == 0) throw herr("empty event in EventPayload");
+
+    if (raw[0] == '\x00') {
+        if (outDictId) *outDictId = 0;
+        return raw.substr(1);
+    } else if (raw[0] == '\x01') {
+        raw = raw.substr(1);
+        if (raw.size() < 4) throw herr("EventPayload record too short to read dictId");
+        uint32_t dictId = lmdb::from_sv<uint32_t>(raw.substr(0, 4));
+        raw = raw.substr(4);
+
+        decomp.reserve(cfg().events__maxEventSize);
+        std::string_view buf = decomp.decompress(txn, dictId, raw);
+
+        if (outDictId) *outDictId = dictId;
+        if (outCompressedSize) *outCompressedSize = raw.size();
+        return buf;
+    } else {
+        throw("Unexpected first byte in EventPayload");
+    }
+}
+
+// Return result only valid until one of: next call to getEventJson/decodeEventPayload, write to/closing of txn, or any action on decomp object
+
+std::string_view getEventJson(lmdb::txn &txn, Decompressor &decomp, uint64_t levId) {
    std::string_view raw;
-    bool found = env.dbiQuadrable_nodesLeaf.get(txn, lmdb::to_sv<uint64_t>(quadId), raw);
-    if (!found) throw herr("couldn't find leaf node in quadrable, corrupted DB?");
-    return raw.substr(8 + 32 + 32);
+
+    bool found = env.dbi_EventPayload.get(txn, lmdb::to_sv<uint64_t>(levId), raw);
+    if (!found) throw herr("couldn't find event in EventPayload, corrupted DB?");
+
+    return decodeEventPayload(txn, decomp, raw, nullptr, nullptr);
 }



-void writeEvents(lmdb::txn &txn, quadrable::Quadrable &qdb, std::vector<EventToWrite> &evs) {
+
+
+void deleteEvent(lmdb::txn &txn, quadrable::Quadrable::UpdateSet &changes, defaultDb::environment::View_Event &ev) {
+    changes.del(flatEventToQuadrableKey(ev.flat_nested()));
+    env.dbi_EventPayload.del(txn, lmdb::to_sv<uint64_t>(ev.primaryKeyId));
+    env.delete_Event(txn, ev.primaryKeyId);
+}
+
+
+
+void writeEvents(lmdb::txn &txn, quadrable::Quadrable &qdb, std::vector<EventToWrite> &evs, uint64_t logLevel) {
    std::sort(evs.begin(), evs.end(), [](auto &a, auto &b) { return a.quadKey < b.quadKey; });

    auto changes = qdb.change();
-
-    std::vector<uint64_t> eventIdsToDelete;
+    std::string tmpBuf;

    for (size_t i = 0; i < evs.size(); i++) {
        auto &ev = evs[i];
@ -188,57 +253,80 @@ void writeEvents(lmdb::txn &txn, quadrable::Quadrable &qdb, std::vector<EventToW

        if (isReplaceableEvent(flat->kind())) {
            auto searchKey = makeKey_StringUint64Uint64(sv(flat->pubkey()), flat->kind(), MAX_U64);
-            uint64_t otherEventId = 0;

            env.generic_foreachFull(txn, env.dbi_Event__pubkeyKind, searchKey, lmdb::to_sv<uint64_t>(MAX_U64), [&](auto k, auto v) {
                ParsedKey_StringUint64Uint64 parsedKey(k);
                if (parsedKey.s == sv(flat->pubkey()) && parsedKey.n1 == flat->kind()) {
                    if (parsedKey.n2 < flat->created_at()) {
-                        otherEventId = lmdb::from_sv<uint64_t>(v);
+                        auto otherEv = env.lookup_Event(txn, lmdb::from_sv<uint64_t>(v));
+                        if (!otherEv) throw herr("missing event from index, corrupt DB?");
+                        if (logLevel >= 1) LI << "Deleting event (replaceable). id=" << to_hex(sv(otherEv->flat_nested()->id()));
+                        deleteEvent(txn, changes, *otherEv);
                    } else {
                        ev.status = EventWriteStatus::Replaced;
                    }
                }
                return false;
            }, true);
+        } else {
+            std::string replace;

-            if (otherEventId) {
-                auto otherEv = env.lookup_Event(txn, otherEventId);
-                if (!otherEv) throw herr("missing event from index, corrupt DB?");
-                changes.del(flatEventToQuadrableKey(otherEv->flat_nested()));
-                eventIdsToDelete.push_back(otherEventId);
+            for (const auto &tagPair : *(flat->tagsGeneral())) {
+                auto tagName = (char)tagPair->key();
+                if (tagName != 'd') continue;
+                replace = std::string(sv(tagPair->val()));
+                break;
+            }
+
+            if (replace.size()) {
+                auto searchStr = std::string(sv(flat->pubkey())) + replace;
+                auto searchKey = makeKey_StringUint64(searchStr, flat->kind());
+
+                env.generic_foreachFull(txn, env.dbi_Event__replace, searchKey, lmdb::to_sv<uint64_t>(MAX_U64), [&](auto k, auto v) {
+                    ParsedKey_StringUint64 parsedKey(k);
+                    if (parsedKey.s == searchStr && parsedKey.n == flat->kind()) {
+                        auto otherEv = env.lookup_Event(txn, lmdb::from_sv<uint64_t>(v));
+                        if (!otherEv) throw herr("missing event from index, corrupt DB?");
+
+                        if (otherEv->flat_nested()->created_at() < flat->created_at()) {
+                            if (logLevel >= 1) LI << "Deleting event (d-tag). id=" << to_hex(sv(otherEv->flat_nested()->id()));
+                            deleteEvent(txn, changes, *otherEv);
+                        } else {
+                            ev.status = EventWriteStatus::Replaced;
+                        }
+                    }
+
+                    return false;
+                }, true);
            }
        }

        if (flat->kind() == 5) {
            // Deletion event, delete all referenced events
-            for (const auto &tagPair : *(flat->tags())) {
+            for (const auto &tagPair : *(flat->tagsFixed32())) {
                if (tagPair->key() == 'e') {
                    auto otherEv = lookupEventById(txn, sv(tagPair->val()));
                    if (otherEv && sv(otherEv->flat_nested()->pubkey()) == sv(flat->pubkey())) {
-                        LI << "Deleting event. id=" << to_hex(sv(tagPair->val()));
-                        changes.del(flatEventToQuadrableKey(otherEv->flat_nested()));
-                        eventIdsToDelete.push_back(otherEv->primaryKeyId);
+                        if (logLevel >= 1) LI << "Deleting event (kind 5). id=" << to_hex(sv(tagPair->val()));
+                        deleteEvent(txn, changes, *otherEv);
                    }
                }
            }
        }

        if (ev.status == EventWriteStatus::Pending) {
-            changes.put(ev.quadKey, ev.jsonStr, &ev.nodeId);
+            ev.levId = env.insert_Event(txn, ev.receivedAt, ev.flatStr, (uint64_t)ev.sourceType, ev.sourceInfo);
+
+            tmpBuf.clear();
+            tmpBuf += '\x00';
+            tmpBuf += ev.jsonStr;
+            env.dbi_EventPayload.put(txn, lmdb::to_sv<uint64_t>(ev.levId), tmpBuf);
+
+            changes.put(ev.quadKey, "");
+
+            ev.status = EventWriteStatus::Written;
        }
    }

    changes.apply(txn);
-
-    for (auto eventId : eventIdsToDelete) {
-        env.delete_Event(txn, eventId);
-    }
-
-    for (auto &ev : evs) {
-        if (ev.status == EventWriteStatus::Pending) {
-            env.insert_Event(txn, ev.nodeId, ev.receivedAt, ev.flatStr);
-            ev.status = EventWriteStatus::Written;
-        }
-    }
 }
--- a/src/events.h
+++ b/src/events.h
@ -4,7 +4,7 @@

 #include "golpe.h"

-#include "constants.h"
+#include "Decompressor.h"



@ -45,16 +45,37 @@ inline const NostrIndex::Event *flatStrToFlatEvent(std::string_view flatStr) {


 std::optional<defaultDb::environment::View_Event> lookupEventById(lmdb::txn &txn, std::string_view id);
-uint64_t getMostRecentEventId(lmdb::txn &txn);
-std::string_view getEventJson(lmdb::txn &txn, uint64_t quadId);
+uint64_t getMostRecentLevId(lmdb::txn &txn);
+std::string_view decodeEventPayload(lmdb::txn &txn, Decompressor &decomp, std::string_view raw, uint32_t *outDictId, size_t *outCompressedSize);
+std::string_view getEventJson(lmdb::txn &txn, Decompressor &decomp, uint64_t levId);

 inline quadrable::Key flatEventToQuadrableKey(const NostrIndex::Event *flat) {
-    return quadrable::Key::fromIntegerAndHash(flat->created_at(), sv(flat->id()).substr(0, 23));
+    uint64_t timestamp = flat->created_at();
+    if (timestamp > MAX_TIMESTAMP) throw herr("timestamp is too large to encode in quadrable key");
+    return quadrable::Key::fromIntegerAndHash(timestamp, sv(flat->id()).substr(0, 27));
 }




+enum class EventSourceType {
+    None = 0,
+    IP4 = 1,
+    IP6 = 2,
+    Import = 3,
+    Stream = 4,
+    Sync = 5,
+};
+
+inline std::string eventSourceTypeToStr(EventSourceType t) {
+    if (t == EventSourceType::IP4) return "IP4";
+    else if (t == EventSourceType::IP6) return "IP6";
+    else if (t == EventSourceType::Import) return "Import";
+    else if (t == EventSourceType::Stream) return "Stream";
+    else if (t == EventSourceType::Sync) return "Sync";
+    else return "?";
+}
+


 enum class EventWriteStatus {
@ -70,18 +91,21 @@ struct EventToWrite {
    std::string flatStr;
    std::string jsonStr;
    uint64_t receivedAt;
+    EventSourceType sourceType;
+    std::string sourceInfo;
    void *userData = nullptr;
    quadrable::Key quadKey;
-    uint64_t nodeId = 0;
    EventWriteStatus status = EventWriteStatus::Pending;
+    uint64_t levId = 0;

    EventToWrite() {}

-    EventToWrite(std::string flatStr, std::string jsonStr, uint64_t receivedAt, void *userData = nullptr) : flatStr(flatStr), jsonStr(jsonStr), receivedAt(receivedAt), userData(userData) {
+    EventToWrite(std::string flatStr, std::string jsonStr, uint64_t receivedAt, EventSourceType sourceType, std::string sourceInfo, void *userData = nullptr) : flatStr(flatStr), jsonStr(jsonStr), receivedAt(receivedAt), sourceType(sourceType), sourceInfo(sourceInfo), userData(userData) {
        const NostrIndex::Event *flat = flatbuffers::GetRoot<NostrIndex::Event>(flatStr.data());
        quadKey = flatEventToQuadrableKey(flat);
    }
 };


-void writeEvents(lmdb::txn &txn, quadrable::Quadrable &qdb, std::vector<EventToWrite> &evs);
+void writeEvents(lmdb::txn &txn, quadrable::Quadrable &qdb, std::vector<EventToWrite> &evs, uint64_t logLevel = 1);
+void deleteEvent(lmdb::txn &txn, quadrable::Quadrable::UpdateSet &changes, defaultDb::environment::View_Event &ev);
--- a/src/filters.h
+++ b/src/filters.h
@ -2,8 +2,6 @@

 #include "golpe.h"

-#include "constants.h"
-

 struct FilterSetBytes {
    struct Item {
@ -18,16 +16,15 @@ struct FilterSetBytes {
    // Sizes are post-hex decode 

    FilterSetBytes(const tao::json::value &arrHex, bool hexDecode, size_t minSize, size_t maxSize) {
-        std::vector<std::string> arr;
+        if (maxSize > MAX_INDEXED_TAG_VAL_SIZE) throw herr("maxSize bigger than max indexed tag size");

-        uint64_t totalSize = 0;
+        std::vector<std::string> arr;

        for (const auto &i : arrHex.get_array()) {
            arr.emplace_back(hexDecode ? from_hex(i.get_string(), false) : i.get_string());
            size_t itemSize = arr.back().size();
            if (itemSize < minSize) throw herr("filter item too small");
            if (itemSize > maxSize) throw herr("filter item too large");
-            totalSize += itemSize;
        }

        std::sort(arr.begin(), arr.end());
@ -114,7 +111,7 @@ struct NostrFilter {
    std::optional<FilterSetBytes> ids;
    std::optional<FilterSetBytes> authors;
    std::optional<FilterSetUint> kinds;
-    std::map<char, FilterSetBytes> tags;
+    flat_hash_map<char, FilterSetBytes> tags;

    uint64_t since = 0;
    uint64_t until = MAX_U64;
@ -122,7 +119,7 @@ struct NostrFilter {
    bool neverMatch = false;
    bool indexOnlyScans = false;

-    explicit NostrFilter(const tao::json::value &filterObj) {
+    explicit NostrFilter(const tao::json::value &filterObj, uint64_t maxFilterLimit) {
        uint64_t numMajorFields = 0;

        for (const auto &[k, v] : filterObj.get_object()) {
@ -148,7 +145,7 @@ struct NostrFilter {
                    if (tag == 'p' || tag == 'e') {
                        tags.emplace(tag, FilterSetBytes(v, true, 32, 32));
                    } else {
-                        tags.emplace(tag, FilterSetBytes(v, false, 1, cfg().events__maxTagValSize));
+                        tags.emplace(tag, FilterSetBytes(v, false, 1, MAX_INDEXED_TAG_VAL_SIZE));
                    }
                } else {
                    throw herr("unindexed tag filter");
@ -166,10 +163,9 @@ struct NostrFilter {

        if (tags.size() > 2) throw herr("too many tags in filter"); // O(N^2) in matching, just prohibit it

-        if (limit > cfg().relay__maxFilterLimit) limit = cfg().relay__maxFilterLimit;
+        if (limit > maxFilterLimit) limit = maxFilterLimit;

-        indexOnlyScans = numMajorFields <= 1;
-        // FIXME: pubkeyKind scan could be serviced index-only too
+        indexOnlyScans = (numMajorFields <= 1) || (numMajorFields == 2 && authors && kinds);
    }

    bool doesMatchTimes(uint64_t created) const {
@ -190,7 +186,7 @@ struct NostrFilter {
        for (const auto &[tag, filt] : tags) {
            bool foundMatch = false;

-            for (const auto &tagPair : *(ev->tags())) {
+            for (const auto &tagPair : *(ev->tagsFixed32())) {
                auto eventTag = tagPair->key();
                if (eventTag == tag && filt.doesMatch(sv(tagPair->val()))) {
                    foundMatch = true;
@ -198,6 +194,16 @@ struct NostrFilter {
                }
            }

+            if (!foundMatch) {
+                for (const auto &tagPair : *(ev->tagsGeneral())) {
+                    auto eventTag = tagPair->key();
+                    if (eventTag == tag && filt.doesMatch(sv(tagPair->val()))) {
+                        foundMatch = true;
+                        break;
+                    }
+                }
+            }
+
            if (!foundMatch) return false;
        }

@ -209,18 +215,18 @@ struct NostrFilterGroup {
    std::vector<NostrFilter> filters;

    // Note that this expects the full array, so the first two items are "REQ" and the subId
-    NostrFilterGroup(const tao::json::value &req) {
+    NostrFilterGroup(const tao::json::value &req, uint64_t maxFilterLimit = cfg().relay__maxFilterLimit) {
        const auto &arr = req.get_array();
        if (arr.size() < 3) throw herr("too small");

        for (size_t i = 2; i < arr.size(); i++) {
-            filters.emplace_back(arr[i]);
+            filters.emplace_back(arr[i], maxFilterLimit);
            if (filters.back().neverMatch) filters.pop_back();
        }
    }

    // Hacky! Deserves a refactor
-    static NostrFilterGroup unwrapped(tao::json::value filter) {
+    static NostrFilterGroup unwrapped(tao::json::value filter, uint64_t maxFilterLimit = cfg().relay__maxFilterLimit) {
        if (!filter.is_array()) {
            filter = tao::json::value::array({ filter });
        }
@ -231,7 +237,7 @@ struct NostrFilterGroup {
            pretendReqQuery.push_back(e);
        }

-        return NostrFilterGroup(pretendReqQuery);
+        return NostrFilterGroup(pretendReqQuery, maxFilterLimit);
    }

    bool doesMatch(const NostrIndex::Event *ev) const {
--- a/src/gc.h
+++ b/src/gc.h
@ -0,0 +1,33 @@
+#pragma once
+
+#include "golpe.h"
+
+
+inline void quadrableGarbageCollect(quadrable::Quadrable &qdb, int logLevel = 0) {
+    quadrable::Quadrable::GarbageCollector<flat_hash_set<uint64_t>> gc(qdb);
+    quadrable::Quadrable::GCStats stats;
+
+    if (logLevel >= 2) LI << "Running garbage collection";
+
+    {
+        auto txn = env.txn_ro();
+
+        if (logLevel >= 2) LI << "GC: mark phase";
+        gc.markAllHeads(txn);
+        if (logLevel >= 2) LI << "GC: sweep phase";
+        stats = gc.sweep(txn);
+    }
+
+    if (logLevel >= 2) {
+        LI << "GC: Total nodes:   " << stats.total;
+        LI << "GC: Garbage nodes: " << stats.garbage << " (" << renderPercent((double)stats.garbage / stats.total) << ")";
+    }
+
+    if (stats.garbage) {
+        auto txn = env.txn_rw();
+        if (logLevel >= 1) LI << "GC: deleting " << stats.garbage << " garbage nodes";
+        gc.deleteNodes(txn);
+        txn.commit();
+    }
+
+}
--- a/src/global.h
+++ b/src/global.h
@ -0,0 +1,23 @@
+#pragma once
+
+#include <parallel_hashmap/phmap.h>
+#include <parallel_hashmap/btree.h>
+
+using namespace phmap;
+
+
+#include <quadrable.h>
+
+quadrable::Quadrable getQdbInstance(lmdb::txn &txn);
+quadrable::Quadrable getQdbInstance();
+
+
+#include "constants.h"
+
+
+std::string renderIP(std::string_view ipBytes);
+std::string renderSize(uint64_t si);
+std::string renderPercent(double p);
+uint64_t parseUint64(const std::string &s);
+std::string parseIP(const std::string &ip);
+uint64_t getDBVersion(lmdb::txn &txn);
--- a/src/misc.cpp
+++ b/src/misc.cpp
@ -0,0 +1,105 @@
+#include <arpa/inet.h>
+#include <stdio.h>
+
+#include <algorithm>
+#include <string>
+
+#include "golpe.h"
+
+
+std::string renderIP(std::string_view ipBytes) {
+    char buf[128];
+
+    if (ipBytes.size() == 4) {
+        inet_ntop(AF_INET, ipBytes.data(), buf, sizeof(buf));
+    } else if (ipBytes.size() == 16) {
+        inet_ntop(AF_INET6, ipBytes.data(), buf, sizeof(buf));
+    } else {
+        throw herr("invalid size of ipBytes, unable to render IP");
+    }
+
+    return std::string(buf);
+}
+
+std::string parseIP(const std::string &ip) {
+    int af = ip.find(':') != std::string::npos ? AF_INET6 : AF_INET;
+    unsigned char buf[16];
+
+    int ret = inet_pton(af, ip.c_str(), &buf[0]);
+    if (ret == 0) return "";
+
+    return std::string((const char*)&buf[0], af == AF_INET6 ? 16 : 4);
+}
+
+
+std::string renderSize(uint64_t si) {
+    if (si < 1024) return std::to_string(si) + "b";
+
+    double s = si;
+    char buf[128];
+    char unit;
+
+    do {
+        s /= 1024;
+        if (s < 1024) {
+            unit = 'K';
+            break;
+        }
+
+        s /= 1024;
+        if (s < 1024) {
+            unit = 'M';
+            break;
+        }
+
+        s /= 1024;
+        if (s < 1024) {
+            unit = 'G';
+            break;
+        }
+
+        s /= 1024;
+        unit = 'T';
+    } while(0);
+
+    ::snprintf(buf, sizeof(buf), "%.2f%c", s, unit);
+    return std::string(buf);
+}
+
+
+
+std::string renderPercent(double p) {
+    char buf[128];
+    ::snprintf(buf, sizeof(buf), "%.1f%%", p * 100);
+    return std::string(buf);
+}
+
+
+
+uint64_t parseUint64(const std::string &s) {
+    auto digitChar = [](char c){
+        return c >= '0' && c <= '9';
+    };
+
+    if (!std::all_of(s.begin(), s.end(), digitChar)) throw herr("non-digit character");
+
+    return std::stoull(s);
+}
+
+
+
+uint64_t getDBVersion(lmdb::txn &txn) {
+    uint64_t dbVersion;
+
+    {
+        auto s = env.lookup_Meta(txn, 1);
+
+        if (s) {
+            dbVersion = s->dbVersion();
+        } else {
+            dbVersion = 0;
+        }
+    }
+
+    return dbVersion;
+}
--- a/src/onAppStartup.cpp
+++ b/src/onAppStartup.cpp
@ -0,0 +1,88 @@
+#include <sys/time.h>
+#include <sys/resource.h>
+#include <string.h>
+#include <errno.h>
+
+#include "golpe.h"
+
+
+static void dbCheck(lmdb::txn &txn, const std::string &cmd) {
+    auto dbTooOld = [&](uint64_t ver) {
+        LE << "Database version too old: " << ver << ". Expected version " << CURR_DB_VERSION;
+        LE << "You should 'strfry export' your events, delete (or move) the DB files, and 'strfry import' them";
+        throw herr("aborting: DB too old");
+    };
+
+    auto dbTooNew = [&](uint64_t ver) {
+        LE << "Database version too new: " << ver << ". Expected version " << CURR_DB_VERSION;
+        LE << "You should upgrade your version of 'strfry'";
+        throw herr("aborting: DB too new");
+    };
+
+    auto s = env.lookup_Meta(txn, 1);
+
+    if (!s) {
+        {
+            // The first version of the DB didn't use a Meta entry -- we consider this version 0
+
+            bool eventFound = false;
+
+            env.foreach_Event(txn, [&](auto &ev){
+                eventFound = true;
+                return false;
+            });
+
+            if (cmd == "export" || cmd == "info") return;
+            if (eventFound) dbTooOld(0);
+        }
+
+        env.insert_Meta(txn, CURR_DB_VERSION, 1);
+        return;
+    }
+
+    if (s->endianness() != 1) throw herr("DB was created on a machine with different endianness");
+
+    if (s->dbVersion() < CURR_DB_VERSION) {
+        if (cmd == "export" || cmd == "info") return;
+        dbTooOld(s->dbVersion());
+    }
+
+    if (s->dbVersion() > CURR_DB_VERSION) {
+        dbTooNew(s->dbVersion());
+    }
+}
+
+static void setRLimits() {
+    if (!cfg().relay__nofiles) return;
+    struct rlimit curr;
+
+    if (getrlimit(RLIMIT_NOFILE, &curr)) throw herr("couldn't call getrlimit: ", strerror(errno));
+
+    if (cfg().relay__nofiles > curr.rlim_max) throw herr("Unable to set NOFILES limit to ", cfg().relay__nofiles, ", exceeds max of ", curr.rlim_max);
+
+    curr.rlim_cur = cfg().relay__nofiles;
+
+    if (setrlimit(RLIMIT_NOFILE, &curr)) throw herr("Failed setting NOFILES limit to ", cfg().relay__nofiles, ": ", strerror(errno));
+}
+
+
+
+quadrable::Quadrable getQdbInstance(lmdb::txn &txn) {
+    quadrable::Quadrable qdb;
+    qdb.init(txn);
+    qdb.checkout("events");
+    return qdb;
+}
+
+quadrable::Quadrable getQdbInstance() {
+    auto txn = env.txn_ro();
+    return getQdbInstance(txn);
+}
+
+void onAppStartup(lmdb::txn &txn, const std::string &cmd) {
+    dbCheck(txn, cmd);
+
+    setRLimits();
+
+    (void)getQdbInstance(txn);
+}
--- a/strfry.conf
+++ b/strfry.conf
@ -2,16 +2,30 @@
 ## Default strfry config
 ##

-# Directory that contains strfry database
+# Directory that contains the strfry LMDB database (restart required)
 db = "./strfry-db/"

+dbParams {
+    # Maximum number of threads/processes that can simultaneously have LMDB transactions open (restart required)
+    maxreaders = 256
+
+    # Size of mmap() to use when loading LMDB (default is 10TB, does *not* correspond to disk-space used) (restart required)
+    mapsize = 10995116277760
+}
+
 relay {
-    # Interface to listen on. Use 0.0.0.0 to listen on all interfaces
+    # Interface to listen on. Use 0.0.0.0 to listen on all interfaces (restart required)
    bind = "127.0.0.1"

-    # Port to open for the nostr websocket protocol
+    # Port to open for the nostr websocket protocol (restart required)
    port = 7777

+    # Set OS-limit on maximum number of open files/sockets (if 0, don't attempt to set) (restart required)
+    nofiles = 1000000
+
+    # HTTP header that contains the client's real IP, before reverse proxying (ie x-real-ip) (MUST be all lower-case)
+    realIpHeader = ""
+
    info {
        # NIP-11: Name of this server. Short/descriptive (< 30 characters)
        name = "strfry default"
@ -26,10 +40,10 @@ relay {
        contact = "unset"
    }

-    # Maximum accepted incoming websocket frame size (should be larger than max event and yesstr msg)
+    # Maximum accepted incoming websocket frame size (should be larger than max event and yesstr msg) (restart required)
    maxWebsocketPayloadSize = 131072

-    # Websocket-level PING message frequency (should be less than any reverse proxy idle timeouts)
+    # Websocket-level PING message frequency (should be less than any reverse proxy idle timeouts) (restart required)
    autoPingSeconds = 55

    # If TCP keep-alive should be enabled (detect dropped connections to upstream reverse proxy)
@ -41,6 +55,25 @@ relay {
    # Maximum records that can be returned per filter
    maxFilterLimit = 500

+    # Maximum number of subscriptions (concurrent REQs) a connection can have open at any time
+    maxSubsPerConnection = 20
+
+    writePolicy {
+        # If non-empty, path to an executable script that implements the writePolicy plugin logic
+        plugin = ""
+
+        # Number of seconds to search backwards for lookback events when starting the writePolicy plugin (0 for no lookback)
+        lookbackSeconds = 0
+    }
+
+    compression {
+        # Use permessage-deflate compression if supported by client. Reduces bandwidth, but slight increase in CPU (restart required)
+        enabled = true
+
+        # Maintain a sliding window buffer for each connection. Improves compression, but uses more memory (restart required)
+        slidingWindow = true
+    }
+
    logging {
        # Dump all incoming messages
        dumpInAll = false
@ -56,12 +89,16 @@ relay {
    }

    numThreads {
+        # Ingester threads: route incoming requests, validate events/sigs (restart required)
        ingester = 3

+        # reqWorker threads: Handle initial DB scan for events (restart required)
        reqWorker = 3

+        # reqMonitor threads: Handle filtering of new events (restart required)
        reqMonitor = 3

+        # yesstr threads: Experimental yesstr protocol (restart required)
        yesstr = 1
    }
 }
@ -83,8 +120,8 @@ events {
    ephemeralEventsLifetimeSeconds = 300

    # Maximum number of tags allowed
-    maxNumTags = 250
+    maxNumTags = 2000

    # Maximum size for tag values, in bytes
-    maxTagValSize = 255
+    maxTagValSize = 1024
 }
--- a/test/dumbFilter.pl
+++ b/test/dumbFilter.pl
@ -1,9 +1,9 @@
 #!/usr/bin/env perl

-use JSON::XS;
-
 use strict;

+use JSON::XS;
+
 binmode(STDOUT, ":utf8");

 my $filterJson = shift || die "need filter";
--- a/test/filterFuzzTest.pl
+++ b/test/filterFuzzTest.pl
@ -1,3 +1,5 @@
+#!/usr/bin/env perl
+
 use strict;
 use Data::Dumper;
 use JSON::XS;
@ -188,8 +190,8 @@ sub testScan {
    #print JSON::XS->new->pretty(1)->encode($fg);
    print "$fge\n";

-    my $resA = `./strfry --config test/strfry.conf export 2>/dev/null | perl test/dumbFilter.pl '$fge' | jq -r .pubkey | sort | sha256sum`;
-    my $resB = `./strfry --config test/strfry.conf scan '$fge' | jq -r .pubkey | sort | sha256sum`;
+    my $resA = `./strfry export 2>/dev/null | perl test/dumbFilter.pl '$fge' | jq -r .pubkey | sort | sha256sum`;
+    my $resB = `./strfry scan '$fge' | jq -r .pubkey | sort | sha256sum`;

    print "$resA\n$resB\n";

@ -202,6 +204,38 @@ sub testScan {
 }


+sub testMonitor {
+    my $monCmds = shift;
+    my $interestFg = shift;
+
+    my $fge = encode_json($interestFg);
+    print "filt: $fge\n\n";
+
+    print "DOING MONS\n";
+    my $pid = open2(my $outfile, my $infile, './strfry monitor | jq -r .pubkey | sort | sha256sum');
+    for my $c (@$monCmds) { print $infile encode_json($c), "\n"; }
+    close($infile);
+
+    my $resA = <$outfile>;
+
+    waitpid($pid, 0);
+    my $child_exit_status = $? >> 8;
+    die "monitor cmd died" if $child_exit_status;
+
+    print "DOING SCAN\n";
+    my $resB = `./strfry scan '$fge' 2>/dev/null | jq -r .pubkey | sort | sha256sum`;
+
+    print "$resA\n$resB\n";
+
+    if ($resA eq $resB) {
+        print "-----------MATCH OK-------------\n\n\n";
+    } else {
+        print STDERR "$fge\n";
+        die "MISMATCH";
+    }
+}
+
+

 srand($ENV{SEED} || 0);

@ -215,32 +249,7 @@ if ($cmd eq 'scan') {
 } elsif ($cmd eq 'monitor') {
    while (1) {
        my ($monCmds, $interestFg) = genRandomMonitorCmds();
-
-        my $fge = encode_json($interestFg);
-        print "filt: $fge\n\n";
-
-        print "DOING MONS\n";
-        my $pid = open2(my $outfile, my $infile, './strfry --config test/strfry.conf monitor | jq -r .pubkey | sort | sha256sum');
-        for my $c (@$monCmds) { print $infile encode_json($c), "\n"; }
-        close($infile);
-
-        my $resA = <$outfile>;
-
-        waitpid($pid, 0);
-        my $child_exit_status = $? >> 8;
-        die "monitor cmd died" if $child_exit_status;
-
-        print "DOING SCAN\n";
-        my $resB = `./strfry --config test/strfry.conf scan '$fge' 2>/dev/null | jq -r .pubkey | sort | sha256sum`;
-
-        print "$resA\n$resB\n";
-
-        if ($resA eq $resB) {
-            print "-----------MATCH OK-------------\n\n\n";
-        } else {
-            print STDERR "$fge\n";
-            die "MISMATCH";
-        }
+        testMonitor($monCmds, $interestFg);
    }
 } else {
    die "unknown cmd: $cmd";
--- a/test/strfry.conf
+++ b/test/strfry.conf
@ -1,6 +1 @@
-db = "./strfry-db/"
-
-relay {
-    port = 7777
-    maxFilterLimit = 1000000000000
-}
+db = "./strfry-db-test/"
--- a/test/writeTest.pl
+++ b/test/writeTest.pl
@ -0,0 +1,210 @@
+#!/usr/bin/env perl
+
+use strict;
+
+use Data::Dumper;
+use JSON::XS;
+
+
+my $ids = [
+    {
+        sec => 'c1eee22f68dc218d98263cfecb350db6fc6b3e836b47423b66c62af7ae3e32bb',
+        pub => '003ba9b2c5bd8afeed41a4ce362a8b7fc3ab59c25b6a1359cae9093f296dac01',
+    },
+    {
+        sec => 'a0b459d9ff90e30dc9d1749b34c4401dfe80ac2617c7732925ff994e8d5203ff',
+        pub => 'cc49e2a58373abc226eee84bee9ba954615aa2ef1563c4f955a74c4606a3b1fa',
+    },
+];
+
+
+
+## Basic insert
+
+doTest({
+    events => [
+        qq{--sec $ids->[0]->{sec} --content "hi" --kind 1 },
+        qq{--sec $ids->[0]->{sec} --content "hi 2" --kind 1 },
+    ],
+    verify => [ 0, 1, ],
+});
+
+## Replacement, newer timestamp
+
+doTest({
+    events => [
+        qq{--sec $ids->[0]->{sec} --content "hi" --kind 10000 --created-at 5000 },
+        qq{--sec $ids->[0]->{sec} --content "hi 2" --kind 10000 --created-at 5001 },
+        qq{--sec $ids->[0]->{sec} --content "hi" --kind 10000 --created-at 5000 },
+    ],
+    verify => [ 1, ],
+});
+
+## Replacement is dropped
+
+doTest({
+    events => [
+        qq{--sec $ids->[0]->{sec} --content "hi" --kind 10000 --created-at 5001 },
+        qq{--sec $ids->[0]->{sec} --content "hi 2" --kind 10000 --created-at 5000 },
+    ],
+    verify => [ 0, ],
+});
+
+## Doesn't replace some else's event
+
+doTest({
+    events => [
+        qq{--sec $ids->[0]->{sec} --content "hi" --kind 10000 --created-at 5000 },
+        qq{--sec $ids->[1]->{sec} --content "hi 2" --kind 10000 --created-at 5001 },
+    ],
+    verify => [ 0, 1, ],
+});
+
+## Doesn't replace different kind
+
+doTest({
+    events => [
+        qq{--sec $ids->[0]->{sec} --content "hi" --kind 10001 --created-at 5000 },
+        qq{--sec $ids->[1]->{sec} --content "hi 2" --kind 10000 --created-at 5001 },
+    ],
+    verify => [ 0, 1, ],
+});
+
+
+## Deletion
+
+doTest({
+    events => [
+        qq{--sec $ids->[0]->{sec} --content "hi" --kind 1 --created-at 5000 },
+        qq{--sec $ids->[0]->{sec} --content "hi" --kind 1 --created-at 5001 },
+        qq{--sec $ids->[0]->{sec} --content "hi" --kind 1 --created-at 5002 },
+        qq{--sec $ids->[0]->{sec} --content "blah" --kind 5 --created-at 6000 -e EV_2 -e EV_0 },
+    ],
+    verify => [ 1, 3, ],
+});
+
+## Can't delete someone else's event
+
+doTest({
+    events => [
+        qq{--sec $ids->[0]->{sec} --content "hi" --kind 1 --created-at 5000 },
+        qq{--sec $ids->[1]->{sec} --content "blah" --kind 5 --created-at 6000 -e EV_0 },
+    ],
+    verify => [ 0, 1, ],
+});
+
+## Deletion prevents re-adding same event
+
+doTest({
+    events => [
+        qq{--sec $ids->[0]->{sec} --content "hi" --kind 1 --created-at 5000 },
+        qq{--sec $ids->[0]->{sec} --content "blah" --kind 5 --created-at 6000 -e EV_0 },
+        qq{--sec $ids->[0]->{sec} --content "hi" --kind 1 --created-at 5000 },
+    ],
+    verify => [ 1, ],
+});
+
+
+
+## Parameterized Replaceable Events
+
+doTest({
+    events => [
+        qq{--sec $ids->[0]->{sec} --content "hi1" --kind 1 --created-at 5000 --tag d myrepl },
+        qq{--sec $ids->[0]->{sec} --content "hi2" --kind 1 --created-at 5001 --tag d myrepl },
+    ],
+    verify => [ 1, ],
+});
+
+## d tags have to match
+
+doTest({
+    events => [
+        qq{--sec $ids->[0]->{sec} --content "hi1" --kind 1 --created-at 5000 --tag d myrepl },
+        qq{--sec $ids->[0]->{sec} --content "hi2" --kind 1 --created-at 5001 --tag d myrepl2 },
+        qq{--sec $ids->[0]->{sec} --content "hi3" --kind 1 --created-at 5002 --tag d myrepl },
+    ],
+    verify => [ 1, 2, ],
+});
+
+## Kinds have to match
+
+doTest({
+    events => [
+        qq{--sec $ids->[0]->{sec} --content "hi1" --kind 1 --created-at 5000 --tag d myrepl },
+        qq{--sec $ids->[0]->{sec} --content "hi2" --kind 2 --created-at 5001 --tag d myrepl },
+    ],
+    verify => [ 0, 1, ],
+});
+
+## Pubkeys have to match
+
+doTest({
+    events => [
+        qq{--sec $ids->[0]->{sec} --content "hi1" --kind 1 --created-at 5000 --tag d myrepl },
+        qq{--sec $ids->[1]->{sec} --content "hi2" --kind 1 --created-at 5001 --tag d myrepl },
+    ],
+    verify => [ 0, 1, ],
+});
+
+## Timestamp
+
+doTest({
+    events => [
+        qq{--sec $ids->[0]->{sec} --content "hi1" --kind 1 --created-at 5001 --tag d myrepl },
+        qq{--sec $ids->[0]->{sec} --content "hi2" --kind 1 --created-at 5000 --tag d myrepl },
+    ],
+    verify => [ 0, ],
+});
+
+
+
+sub doTest {
+    my $spec = shift;
+
+    cleanDb();
+
+    my $eventIds = [];
+
+    for my $ev (@{ $spec->{events} }) {
+        $ev =~ s{EV_(\d+)}{$eventIds->[$1]}eg;
+        push @$eventIds, addEvent($ev);
+    }
+
+    my $finalEventIds = [];
+
+    {
+        open(my $fh, '-|', './strfry --config test/strfry.conf export 2>/dev/null') || die "$!";
+        while(<$fh>) {
+            push @$finalEventIds, decode_json($_)->{id};
+        }
+    }
+
+    die "incorrect eventIds lengths" if @{$spec->{verify}} != @$finalEventIds;
+
+    for (my $i = 0; $i < @$finalEventIds; $i++) {
+        die "id mismatch" if $eventIds->[$spec->{verify}->[$i]] ne $finalEventIds->[$i];
+    }
+}
+
+
+sub cleanDb {
+    system("mkdir -p strfry-db-test");
+    system("rm -f strfry-db-test/data.mdb");
+}
+
+sub addEvent {
+    my $ev = shift;
+
+    system(qq{ nostril $ev >test-eventXYZ.json });
+
+    my $eventJson = `cat test-eventXYZ.json`;
+
+    system(qq{ <test-eventXYZ.json ./strfry --config test/strfry.conf import --no-gc 2>/dev/null });
+
+    system(qq{ rm test-eventXYZ.json });
+
+    my $event = decode_json($eventJson);
+
+    return $event->{id};
+}