"What started as a single, humble function evolved into a complete, coherent ecosystem for manipulating data structures, a journey in API design guided by the principles of purity, pedagogy, and the principle of least power."

The dot ecosystem is a suite of composable tools for working with nested data structures like JSON, YAML, and Python dictionaries. Each tool follows the Unix philosophy: it does one thing exceptionally well, and they're designed to work together seamlessly.

# Install from PyPI
pip install dotsuite

# Clone and install
git clone https://github.com/queelius/dotsuite.git
cd dotsuite
pip install -e .

# For development with testing tools
make install-dev

# Build and publish (for maintainers)
make build         # Build distribution packages
make publish-test  # Publish to TestPyPI
make publish       # Publish to PyPI

It always starts with a simple problem. You have a nested dictionary or JSON payload, and you need to get a value buried deep inside. You write data['user']['contacts'][0]['email'] and pray that no key or index is missing along the way, lest your program crash with a KeyError. This leads to brittle, defensive code full of try/except blocks.

What began as a simple helper function, dotget, evolved through questions and insights into a complete ecosystem. The result is a mathematically grounded, pedagogically structured collection of tools that makes data manipulation predictable, safe, and expressive.

The ecosystem is built on four fundamental pillars, each answering a core question about data:

Tools for finding and extracting values from within documents.

Tools for asking boolean questions and validating data.

Tools for reshaping and modifying data structures.

Tools for lifting single-document operations to collections via boolean and relational algebra.

import sys
sys.path.insert(0, 'src')  # If running from repo root

# Import from the four pillars
from depth.dotget.core import get
from depth.dotstar.core import search
from truth.dotquery.core import Q
from shape.dotmod.core import set_

# Simple exact addressing
data = {"users": [{"name": "Alice", "role": "admin"}]}
name = get(data, "users.0.name")  # "Alice"

# Pattern matching with wildcards
all_names = search(data, "users.*.name")  # ["Alice"]

# Boolean logic queries (fluent factory)
is_admin = Q("users.0.role").equals("admin").check(data)  # True

# Immutable modifications
new_data = set_(data, "users.0.status", "active")

Philosophy: Start simple with dotget for known paths, add dotstar for patterns, use dotselect for complex queries. The dotpath engine underpins them all with extensible, Turing-complete addressing.

Philosophy: Boolean questions should be separate from data extraction. Start with dotexists or dotequals for simple checks, lift to collections with dotany/dotall, and compose complex logic with dotquery.

Philosophy: Immutable by default. dotmod for precise edits, dotbatch for transactional safety, dotpipe for creating new data shapes, dotpluck for projection.

Philosophy: Lift single-document operations to collections. dotfilter provides set operations with boolean logic, while dotrelate enables database-style joins and projections.

• Compositionality: Every tool composes cleanly with others
• Immutability: Original data is never modified
• Pedagogical: Simple tools graduate to powerful ones
• Single Purpose: Each tool has one clear responsibility
• Interoperability: Common patterns work across all tools
• Performance: Lazy evaluation and efficient algorithms
• Safety: Graceful handling of missing paths and malformed data

Many tools are intentionally simple enough that you can copy their core logic rather than add a dependency:

# The essence of dotget
def get(data, path, default=None):
    try:
        for segment in path.split('.'):
            data = data[int(segment)] if segment.isdigit() and isinstance(data, list) else data[segment]
        return data
    except (KeyError, IndexError, TypeError):
        return default

Every tool works from the command line, making them perfect for shell scripts and data pipelines:

# Check if any user is an admin
cat users.json | dotany users.*.role --equals admin && echo "Admin found"

# Extract all email addresses
cat contacts.json | dotstar "contacts.*.email" > emails.txt

# Join users with their orders
dotrelate join --left-on user_id --right-on user_id users.jsonl orders.jsonl

dotquery offers both a fluent Python builder and a declarative DSL:

from truth.dotquery.core import Q, Query

# Programmatic (fluent factory with operator overloading)
query = Q("role").equals("admin") & Q("login_count").greater(10)

# Declarative (DSL string)
query = Query("equals role admin and greater login_count 10")

# Both produce equivalent ASTs

The ecosystem is designed as a learning journey:

1. Hello World: dotget, dotexists: O(1) mental load
2. Patterns: Add dotstar, dotmod, dotequals: wildcards and basic changes
3. Quantifiers: dotany, dotall, dotpluck, dotbatch: lift to collections, batch edits
4. Power User: dotselect, dotquery, dotpipe: complex selection and reshaping
5. Expert: dotpath, dotfilter, dotrelate: extensible engine and relational algebra

Each stage builds on the previous, with no tool becoming obsolete. A dotget call is still the right choice when you know the exact path.

The ecosystem is built on solid mathematical principles:

• Addressing forms a free algebra on selectors (Turing-complete via user-defined reducers)
• Logic implements Boolean algebra with homomorphic lifting to set operations
• Transformations are endofunctors on document spaces with monoid composition
• Collections lift via functorial map/filter operations preserving algebraic structure

This ensures predictable composition, parallelizability, and mathematical correctness.

Each tool has comprehensive documentation under docs/tools/:

• dotget: Simple exact addressing
• dotstar: Wildcard pattern matching
• dotselect: Advanced selection with predicates
• dotpath: Extensible path engine
• dotexists: Path existence checking
• dotequals: Path value equality
• dotany: Existential quantifier
• dotall: Universal quantifier
• dotquery: Compositional logic engine
• dotmod: Immutable modifications
• dotbatch: Atomic transactions
• dotpipe: Data transformation pipelines
• dotpluck: Value extraction and reshaping
• dotfilter: Boolean algebra on collections
• dotrelate: Relational operations

While dotsuite focuses on pedagogy and simplicity, for production use cases requiring advanced features like streaming, complex path operations, and S-expression queries, consider JAF (Just Another Flow). JAF implements similar concepts to dotfilter and dotpipe in a feature-complete, production-ready package with:

• Lazy streaming evaluation for large datasets
• Advanced path system with regex, fuzzy matching, and wildcards
• S-expression query language
• Index-preserving result sets for powerful set operations
• Support for multiple data sources (files, directories, stdin, compressed)

Think of dotsuite as the "learn by building" approach and JAF as the "battle-tested solution", both valuable for different purposes.

The dot ecosystem welcomes contributions. Each tool lives in its own directory with its own tests and documentation. See CONTRIBUTING.md for guidelines.

MIT License. Use freely, modify as needed, and contribute back when you can.

The dot ecosystem: from simple paths to sophisticated data algebras, one tool at a time.