Structured Context Language (SCL)

A standardized agent loop runtime for context engineering.
SCL is a middleware layer for LLM-powered agents — analogous to what SQL is for databases and Hibernate is for Java.

---

Overview

Context engineering decomposes LLM interaction into three independent dimensions, unified into one event-driven runtime:

Dimension	Description	Nature
Business Content	Instructions tailored to specific prompts and scenarios	The "query"
Tool Calling	Functions, MCP, and skills that fetch or mutate external data	Spatial expansion
Memory Management	Selecting relevant history from multi-turn conversations	Temporal expansion

SCL provides a pluggable, observable agent loop that handles all three dimensions through a standardized interface — so you focus on your business logic, not the plumbing.

---

Features

• Unified Provider Interface — Anthropic, OpenAI, Google, xAI, Groq, OpenRouter, and any OpenAI-compatible endpoint via a single API
• RAG-based Tool Selection — Progressively loads relevant tool definitions using BM25 + embedding hybrid search, avoiding context bloat
• File-first Persistence — File system as the backbone; REST API validates and delegates to file watcher for decoupling
• Pluggable Storage — File system, OceanBase, and PostgreSQL/pgvector backends via a common StoreBase interface
• Composite Embedding — Cache → local (SentenceTransformer) → web API (OpenAI-compatible) with automatic fallback
• Observability — Full OpenTelemetry instrumentation (traces, metrics, structured logs) across all components
• Built-in Toolset — File read/write, grep, bash, git, cron — extensible via capability registration
• Multiple Runtime Forms — RESTful service, interactive TUI, or direct library import
• Minimalist Core — No built-in planners, sub-agents, or background processes; you control the orchestration

---

Quick Start

# Install
pip install -e .

# Start the service
scl

# Submit a task via REST API
curl -X POST http://localhost:8080/tasks \
  -H "Content-Type: application/json" \
  -d '{"system_prompt": "You are a helpful assistant.", "capacity": ["bash", "file_read"]}'

# Or drop a file into the watch directory
echo '{"system_prompt": "Hello"}' > ./todo_folder/task.json

See the Getting Started Guide for full instructions.

---

Use as a Library

from scl.meta.task import Task
from scl.queue.task_queue import TaskQueue
from scl.processor.task_processor import TaskProcessor

queue = TaskQueue()
processor = TaskProcessor(queue)
processor.start()

task = Task(system_prompt="You are a helpful assistant.")
queue.add(task)  # auto-notifies the processor

---

Documentation

Document	Description
Overview & Philosophy	Vision, design principles, and philosophy
Architecture	Component breakdown, data flow, and system design
Core Concepts	Task, Capability, CapRegistry, Embedding, Storage, Queues, Processors
Getting Started	Installation, configuration, and quick start
SDK Reference	API reference and common usage patterns
Development Roadmap	Status, roadmap, and contributing

Research

• A Way to Auto-Scaling Capabilities for Agents — RAG-based tool selection evaluated against BFCL, MCPToolBench++, and ToolE benchmarks

---

Architecture in Brief

Listeners (REST / File Watch / Internal)
        │
        ▼  (write files)
todo_folder/  ─── file-based persistence layer
        │
        ▼
  Queue System  ─── TaskQueue, CapabilityTaskQueues, Awaiting Queues
        │
        ▼
 Processors  ─── TaskProcessor, CapTaskProcessor, Awaiting Processors
        │
        ▼
 Core Services  ─── CapRegistry (RAG), Embedding, Storage, LLM Chat
        │
        ▼
 Observability  ─── OpenTelemetry (traces, metrics, logs)

---

Project Status

Current version: 0.1.0 — Active development.

Area	Status
Core agent loop	✅ Stable
RAG tool selection	✅ Stable (BM25 + Embedding hybrid)
REST & file watch	✅ Operational
Storage backends	✅ fsstore, ⏳ OceanBase, ⏳ pgvector
Docker deployment	✅ Ready
WebSocket hooks	📋 Planned
Debug framework	📋 Planned

See Development Roadmap.

---

Benchmarks

SCL's RAG-based capability selection has been evaluated against industry benchmarks:

Benchmark	Type	Top-5 Recall
BFCL (multiple)	Function call selection	95.2%
BFCL (parallel)	Parallel function calls	93.8%
MCPToolBench++ (single)	MCP tool selection	99.6%
ToolE (Qwen3-Embedding)	Tool selection	83.7%

Details in the research blog.

---

Contributing

# Setup
pip install -e ".[dev]"

# Run checks
make lint        # ruff linter
make typecheck   # mypy
make test        # pytest with coverage
make check       # all of the above

# See all targets
make help

Pull requests are welcome. Please maintain OpenTelemetry instrumentation and structured logging for new components.

---

License

Apache License 2.0