Negentropy Syncing

1 Overview

Two parties often hold overlapping sets of events. A client re-opens after a week offline. A relay mirrors another relay. Fetching everything again with REQ wastes bandwidth. Even fetching all IDs wastes bandwidth when most IDs match.

This NIP defines a sync extension for that case. It works for client-to-relay sync and for relay-to-relay sync. When the two sides share events, it uses less bandwidth than transferring the full event set, or even just the full ID list.

The extension is a Nostr-friendly wrapper around the Negentropy protocol. Negentropy applies a technique called Range-Based Set Reconciliation: split the ID space into ranges, compare a short fingerprint per range, and recurse only into ranges that differ. Negentropy is a binary protocol, so this wrapper hex-encodes its messages inside normal Nostr JSON arrays. The full Negentropy Protocol V1 specification is part of this NIP (§4 to §6 below).

The sync itself moves no events. It only tells the client which IDs it has that the other side needs, and which IDs it needs that the other side has. The client then uploads with EVENT and downloads with REQ, as defined in NIP-01.

2 Sync Messages

We call the two sides the client (the initiator) and the relay, even when the initiator is another relay. The wrapper adds four message types on top of the NIP-01 WebSocket connection:

NEG-OPEN

[
  "NEG-OPEN",
  <subscription ID string>,
  <filter>,
  <initialMessage, hex-encoded>
]

• The subscription ID identifies which sync a later message refers to. It only needs to be unique among the concurrent sync subscriptions on this WebSocket connection. An integer that increments once per NEG-OPEN is fine.
• Sync subscription IDs live in a separate namespace from REQ subscription IDs. The two never collide.
• A NEG-OPEN that reuses a currently open sync subscription ID first closes the existing subscription.
• The filter is a normal NIP-01 filter. It defines the event set on both sides.
• initialMessage is the client's first Negentropy binary message, hex-encoded. See §4.

NEG-ERR

If the relay cannot service a request, it returns an error. After a NEG-ERR, the subscription is closed.

[
  "NEG-ERR",
  <subscription ID string>,
  <reason code string>
]

Error reasons use the NIP-01 format: a machine-readable single-word prefix, then :, then a human-readable message. The suggested reasons are:

NEG-MSG

After the open, the relay and the client alternate. Each side parses the message it received, builds a reply, and sends it back:

[
  "NEG-MSG",
  <subscription ID string>,
  <message, hex-encoded>
]

message is one Negentropy binary message, hex-encoded. Both directions use the same binary format (§4).

NEG-CLOSE

When the client is done, it tells the relay to release its resources:

[
  "NEG-CLOSE",
  <subscription ID string>
]

Disconnecting the WebSocket has the same effect.

3 The Sync Loop

The high-level procedure has five steps:

1. The client chooses a filter. It selects its local events that match the filter, or uses a cache. It builds the initial Negentropy message from that set.
2. The client sends NEG-OPEN with the filter and the initial message.
3. The relay selects its local events that match the filter, or uses a cache.
4. The relay builds a response and returns it in a NEG-MSG.
5. The client parses the message. It learns IDs it has that the relay needs, and IDs it needs that the relay has. To continue, it sends a new NEG-MSG and waits for step 4 again. To stop, it sends NEG-CLOSE or disconnects.

The loop only exchanges IDs. To move the events, the client uploads with EVENT and downloads with REQ. It may do this on the same WebSocket connection, in parallel with further NEG-MSG rounds.

4 Negentropy Wire Format

This section and the next two restate the Negentropy Protocol V1 appendix. The appendix calls the two sides client and server; in Nostr terms the server is the relay. The client sends the first message. The server replies. The client keeps querying until it is satisfied, then terminates. Messages in both directions have the same format.

Preparation: records and ordering

Each side has a collection of records. A record is a 64-bit numeric timestamp plus a 256-bit ID. For Nostr events, these are created_at and the event id. Before syncing, each side sorts its records by timestamp, ascending. When two timestamps are equal, sort lexically by ID, ascending by first differing byte.

The goal: the client learns the set of IDs it has that the server does not, and the set of IDs the server has that it does not.

Varint

A varint is a variable-sized unsigned integer. It is written as base-128 digits, most significant digit first, with as few digits as possible. Bit eight (the high bit) is set on every byte except the last:

Varint := <Digit+128>* <Digit>

Id

An ID is a fixed byte-string of length 32:

Id := Byte{32}

Message

A reconciliation message is one protocol version byte, then an ordered list of ranges:

Message := <protocolVersion (Byte)> <Range>*

• The current protocol version is 1, encoded as the byte 0x61. Version 2 will be 0x62, and so forth.
• If a server receives a version it cannot handle, it replies with a single byte: the highest version it supports. The client can then downgrade and retry its message.
• Each range covers a contiguous section of the timestamp/ID space. The first range starts at timestamp 0 and an all-zero ID.
• Ranges are always adjacent. There are no gaps.
• If the last range does not end at infinity, an implicit Skip-to-infinity range is appended. The range list therefore always covers the full timestamp/ID space.

Range

A range is an upper bound, a mode, and a payload. The mode selects the payload type:

Range       := <upperBound (Bound)> <mode (Varint)> <payload (Skip | Fingerprint | IdList)>

Skip        :=                                          (mode 0 - empty)
Fingerprint := Byte{16}                                 (mode 1)
IdList      := <length (Varint)> <ids (Id)>*          (mode 2)

The message grammar fixes the format, not the strategy. A typical implementation compares fingerprints per range. Equal fingerprints mean the range matches; answer it with Skip. Different fingerprints mean a difference hides inside; split the range into sub-ranges and recurse. Once a range is small, send an IdList so the other side can diff it directly and reply with its own IdList of that range. The IDs each side sees in received lists, minus the IDs it already has, are the IDs it needs; and the reverse gives the IDs the other side needs.

5 Bounds and Timestamp Encoding

Every range has an inclusive lower bound and an exclusive upper bound. Only the upper bound is encoded. The lower bound of a range is the upper bound of the previous range, or timestamp 0 with an all-zero ID for the first range.

A bound is an encoded timestamp plus a variable-length ID prefix. The prefix disambiguates records that share a timestamp:

Bound := <encodedTimestamp (Varint)> <length (Varint)> <idPrefix (Byte)>*

The timestamp is delta-encoded:

• The infinity timestamp is encoded as 0.
• Every other timestamp is encoded as 1 + offset. The offset is the difference between this timestamp and the previously encoded timestamp in the same message.
• The initial offset base is 0. It resets at the beginning of each message.
• Offsets are never negative. Ranges in a message are encoded in ascending order, they never overlap, and an upper bound's timestamp is greater than or equal to its lower bound's timestamp.

The idPrefix rules:

• Its size is given by length: between 0 and 32 bytes, inclusive.
• Use the shortest prefix that separates the first record of this range from the last record of the previous range.
• If those two records have different timestamps, use length 0. Otherwise use the byte-length of their common ID prefix, plus 1.
• When the prefix is shorter than 32 bytes, the omitted trailing bytes are implicitly zero.

6 Fingerprint Algorithm

The 16-byte fingerprint of a range is computed in four steps:

1. Add the element IDs mod 2²⁵⁶. Interpret each ID as a 32-byte little-endian unsigned integer.
2. Concatenate the sum with the number of elements in the range, encoded as a varint (§4).
3. Hash the result with SHA-256.
4. Take the first 16 bytes.

7 Worked Example

First, the bytes. The client opens with the laziest possible message: one range from zero to infinity, answered with a fingerprint of everything it has. The relay's fingerprint differs, so it splits. Its reply names its two newest event IDs explicitly and fingerprints the rest:

61              -- protocol version 1 (0x61)
-- Range 1: [ts 0 … ts 1749700000) --
86 c2 a9 9b 21  -- bound timestamp: varint(1 + 1749700000), first offset base is 0
00              -- bound idPrefix length: 0 (timestamps differ)
01              -- mode 1: Fingerprint
3c9d…77a2       -- 16-byte digest of all relay IDs below ts 1749700000
-- Range 2: [ts 1749700000 … infinity) --
00              -- bound timestamp: 0 = infinity
00              -- bound idPrefix length: 0
02              -- mode 2: IdList
02              -- 2 IDs follow
b1a649eb…       -- Id (32 bytes)
d2b699fa…       -- Id (32 bytes)

Now the full session. The client is missing d2b699fa… and holds one event the relay lacks. Orange lines go to the relay. Blue lines come from the relay:

-- open: filter + initial message (one fingerprint range over everything) --
→ ["NEG-OPEN", "N1", {"kinds":[1], "authors":["22a12a12…"]}, "610000015fe2…9b31"]
← ["NEG-MSG", "N1", "6186c2a99b2100013c9d…77a200000202b1a649eb…d2b699fa…"]

-- client answers the fingerprint range, replies with its own IdList for range 2 --
→ ["NEG-MSG", "N1", "6186c2a99b21000000000202b1a649eb…e4c02231…"]
← ["NEG-MSG", "N1", "61000000"]   -- one Skip range to infinity: reconciled

-- client learned: it needs d2b699fa…, the relay needs e4c02231… --
→ ["EVENT", {"id":"e4c02231…", "kind":1, …}]
← ["OK", "e4c02231…", true, ""]
→ ["REQ", "dl", {"ids":["d2b699fa…"]}]
← ["EVENT", "dl", {"id":"d2b699fa…", …}]
← ["EOSE", "dl"]

→ ["NEG-CLOSE", "N1"]

-- a rejected sync on another subscription looks like this --
← ["NEG-ERR", "N2", "blocked: this query is too big"]

8 Version History

NIP-77 is young and has stayed close to its original text. Notable changes, newest first:

• 20262026-06Code-fence cleanup across the repository; message examples relabeled from jsonc to yaml.
• 20252025-12Added the relay tag to the status line, marking this NIP as a relay-side feature.
• 20252025-08Typo fixes.
• Origin2025-05fiatjaf_ merged the spec, wrapping hoytech's Negentropy protocol for Nostr and inlining the full V1 wire-format appendix.

9 See Also

• NIP-01 — defines the filter object, the EVENT and REQ messages used for the actual transfer, and the error-prefix format.
• NIP-45 — COUNT: a simpler way to count matching events when you trust the relay's number.
• NIP-42 — client authentication; relays may restrict expensive sync queries to authenticated clients.
• NIP-11 — the relay information document, where a relay can advertise supported NIPs.