Documentation
¶
Overview ¶
Package jsonldb provides a generic, concurrent-safe, JSONL-backed data store.
Overview ¶
The package centers around Table, a generic container that stores rows in a JSONL (JSON Lines) file with full in-memory caching for fast reads. Tables are safe for concurrent use by multiple goroutines.
Concurrency: Pessimistic Locking ¶
Table uses pessimistic locking: Table.Modify holds the write lock for the entire read-modify-write operation. This guarantees success without retries, unlike optimistic CAS which requires retry loops when concurrent writes collide. The tradeoff is lower throughput under high contention, but this is acceptable for local file storage with low concurrency.
Secondary Indexes ¶
UniqueIndex and Index provide O(1) lookups by arbitrary keys, staying synchronized with table mutations via TableObserver.
File Format ¶
JSONL files with line 1 as schema header, subsequent lines as JSON rows. Rows are sorted by ID on load if out of order (handles clock drift, manual edits).
Index ¶
- func InitIDSlice(instance, totalInstances int) error
- type ID
- type Index
- type Row
- type Table
- func (t *Table[T]) AddObserver(obs TableObserver[T])
- func (t *Table[T]) Append(row T) (err error)
- func (t *Table[T]) Delete(id ID) (T, error)
- func (t *Table[T]) Get(id ID) T
- func (t *Table[T]) Iter(startID ID) iter.Seq[T]
- func (t *Table[T]) Len() int
- func (t *Table[T]) Modify(id ID, fn func(row T) error) (T, error)
- func (t *Table[T]) Update(row T) (T, error)
- type TableObserver
- type UniqueIndex
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func InitIDSlice ¶
InitIDSlice configures ID generation for multi-instance deployments.
When running multiple parallel processes that generate IDs, each process must call InitIDSlice with a unique instance number to prevent ID collisions. The slice field's low-order bits are partitioned among instances.
Parameters:
- instance: this process's instance number (0 to totalInstances-1)
- totalInstances: total number of parallel processes
Example: with 3 servers, call InitIDSlice(0, 3), InitIDSlice(1, 3), and InitIDSlice(2, 3) on each respective server.
Must be called before any NewID calls. Returns an error if parameters are invalid or if totalInstances exceeds the available slice bits.
Types ¶
type ID ¶
type ID uint64
ID is a time-sortable 64-bit identifier inspired by LUCI IDs.
IDs encode a 10µs timestamp and a monotonically increasing slice for collision avoidance. They are lexicographically sortable when encoded as strings, making them suitable for use as table keys and filenames. The zero value (0) represents an invalid/unset ID.
func DecodeID ¶
DecodeID parses an encoded string back to an ID.
Empty string or "0" decode to zero ID. Returns an error for invalid input.
func NewID ¶
func NewID() ID
NewID generates a new time-based ID.
IDs are guaranteed to be unique and monotonically increasing within a process. Multiple calls in the same 10µs interval use an incrementing slice counter. When multiple instances are configured via InitIDSlice, the slice counter is partitioned so each instance uses non-overlapping values. If the slice overflows, it spins until the next interval to maintain uniqueness.
func (ID) MarshalJSON ¶
MarshalJSON implements json.Marshaler.
func (ID) String ¶
String encodes the ID as a compact, sortable string.
Uses a custom 64-character alphabet ordered by ASCII value (not base64), encoding 6 bits per character in big-endian order. This ensures lexicographic string comparison matches numeric comparison, making IDs sortable as strings. Leading zero-characters are stripped for compactness. Zero IDs return "0".
func (*ID) UnmarshalJSON ¶
UnmarshalJSON implements json.Unmarshaler.
type Index ¶
type Index[K comparable, T Row[T]] struct { // contains filtered or unexported fields }
Index provides O(1) lookup by a non-unique secondary key.
The index is built from existing table data when created and kept synchronized via the TableObserver interface. All operations are concurrent-safe.
func NewIndex ¶
func NewIndex[K comparable, T Row[T]](table *Table[T], keyFunc func(T) K) *Index[K, T]
NewIndex creates a non-unique index on the given table.
The keyFunc extracts the index key from each row. Multiple rows may share the same key.
func (*Index[K, T]) OnAppend ¶
func (idx *Index[K, T]) OnAppend(row T)
OnAppend implements TableObserver.
func (*Index[K, T]) OnDelete ¶
func (idx *Index[K, T]) OnDelete(row T)
OnDelete implements TableObserver.
func (*Index[K, T]) OnUpdate ¶
func (idx *Index[K, T]) OnUpdate(prev, curr T)
OnUpdate implements TableObserver.
type Row ¶
type Row[T any] interface { // Clone returns a deep copy of the row. // // Used to prevent mutations to cached data. Clone() T // GetID returns the unique identifier for this row. // // Must be non-zero. GetID() ID // Validate checks data integrity. // // Called on load and before every write. Return an error to reject invalid data. Validate() error }
Row is implemented by types that can be stored in a Table.
type Table ¶
type Table[T Row[T]] struct { // contains filtered or unexported fields }
Table is a concurrent-safe, generic JSONL-backed data store with in-memory caching.
All read and write operations are protected by a read-write mutex, making Table safe for concurrent use by multiple goroutines. Write operations (Append, Update, Delete) are atomic and immediately persisted to disk.
The JSONL file format uses the first line as a schema header containing version and column definitions. Subsequent lines are JSON-encoded rows.
Rows are stored in insertion order and indexed by ID for O(1) lookups. All returned rows are clones to prevent accidental mutation of cached data.
func NewTable ¶
NewTable creates a Table and loads existing data from the JSONL file at path.
If the file doesn't exist, an empty table is created and the schema is auto-discovered from type T via reflection. Returns an error if the file exists but cannot be read or contains invalid data.
func (*Table[T]) AddObserver ¶
func (t *Table[T]) AddObserver(obs TableObserver[T])
AddObserver registers an observer to receive mutation notifications.
The observer is immediately called with OnAppend for each existing row, allowing indexes to be built from current table state. Observers are called while the table lock is held; see TableObserver.
func (*Table[T]) Append ¶
Append adds a new row to the table and persists it.
Returns an error if the row fails validation, has a zero ID, or has a duplicate ID. If the new row's ID is less than the last row's ID (e.g., clock drift), the row is inserted at the correct position and the entire file is rewritten to maintain sorted order.
func (*Table[T]) Delete ¶
Delete removes a row by ID and persists the change.
Returns the deleted row, or nil if no row with that ID exists. The entire table is rewritten to disk on success.
func (*Table[T]) Iter ¶
Iter returns an iterator over clones of rows with ID strictly greater than startID.
Pass 0 to iterate over all rows from the beginning. The reader lock is held for the duration of iteration; avoid long-running operations inside the loop to prevent blocking writers.
func (*Table[T]) Modify ¶
Modify atomically reads, modifies, and writes a row.
The callback fn receives a clone of the current row and should modify it. Returns the modified row on success, or an error if the row doesn't exist or validation fails.
Modify uses pessimistic locking: the write lock is held for the entire operation (read, callback, validate, write). This guarantees the operation succeeds on the first attempt without retry loops, unlike optimistic CAS which may require retries under contention. The tradeoff is that fn should complete quickly to avoid blocking other operations.
If fn returns an error, the row is not modified. If validation fails after fn returns, the row is not modified. If the disk write fails, the in-memory state is rolled back.
type TableObserver ¶
type TableObserver[T Row[T]] interface { OnAppend(row T) OnUpdate(prev, curr T) OnDelete(row T) }
TableObserver receives notifications about table mutations.
Observers are called synchronously while the table lock is held. Implementations must not call back into the table or acquire locks that could cause deadlock.
type UniqueIndex ¶
type UniqueIndex[K comparable, T Row[T]] struct { // contains filtered or unexported fields }
UniqueIndex provides O(1) lookup by a unique secondary key.
The index is built from existing table data when created and kept synchronized via the TableObserver interface. All operations are concurrent-safe.
func NewUniqueIndex ¶
func NewUniqueIndex[K comparable, T Row[T]](table *Table[T], keyFunc func(T) K) *UniqueIndex[K, T]
NewUniqueIndex creates a unique index on the given table.
The keyFunc extracts the index key from each row. Keys must be unique; if duplicates exist in the table, the last row with each key wins.
func (*UniqueIndex[K, T]) Get ¶
func (idx *UniqueIndex[K, T]) Get(key K) T
Get returns the row with the given key, or nil if not found.
func (*UniqueIndex[K, T]) OnAppend ¶
func (idx *UniqueIndex[K, T]) OnAppend(row T)
OnAppend implements TableObserver.
func (*UniqueIndex[K, T]) OnDelete ¶
func (idx *UniqueIndex[K, T]) OnDelete(row T)
OnDelete implements TableObserver.
func (*UniqueIndex[K, T]) OnUpdate ¶
func (idx *UniqueIndex[K, T]) OnUpdate(prev, curr T)
OnUpdate implements TableObserver.
Source Files