Our Process
How a Treebeard Rating is produced — from agent discovery through data collection, scoring, review, and publication.
The Rating Pipeline
Discovery & Intake
Agents enter the Treebeard pipeline through automated on-chain crawling. Our ERC-8004 crawler scans the Identity Registry contract on Ethereum mainnet, indexing every registered agent. Agents can also be submitted manually via the submission form.
- ERC-8004 Identity Registry crawler indexes all registered agents on Ethereum mainnet
- Crawl data includes agent ID, owner wallet, registration date, and agent URI metadata
- Manual submissions enter the same pipeline — no priority advantage
- Future: Base, Solana, and L2 chain coverage planned as registries launch
Data Collection
Once an agent is discovered, we assemble its signal profile from public, verifiable on-chain sources. Phase 2 agents are scored from ERC-8004 registry data. Phase 3 agents receive additional enrichment from The Graph's ERC-8004 Reputation subgraph.
- On-chain registry data: registration age, transaction count, contract deployment status
- Reputation signals (Phase 3): feedback count, feedback scores, client diversity via The Graph
- Code presence: open-source repository detection (binary signal)
- Future signals planned: uptime monitoring, response latency, ERC-8183 job completion data
Scoring & Weighting
Raw signals are normalized, scored, and weighted according to the agent's type and enrichment phase. Six signal categories each produce a category score (0–100), combined using type-specific weight profiles into a composite score.
- Signals are normalized to comparable 0–100 scales before scoring
- Binary signals (e.g., open-source repository exists) are scored as pass/fail
- Phase 2 agents score from registry data only; Phase 3 agents include reputation signals
- Weights vary by agent type — 10 type profiles defined, with defaults for unclassified agents
- Weight calibration will begin after 3 months of scoring data has been collected
Quality Assurance
Before publication, every rating passes through automated consistency checks. These include score boundary validation, hysteresis buffers to prevent grade oscillation, and confidence threshold gates. The founder reviews anomalies and edge cases manually.
- Automated checks: score bounds, confidence thresholds, hysteresis buffer validation
- Anomaly detection flags sudden score swings for manual inspection
- Safety floor enforced — no agent can score below the minimum safety threshold
- Grade-boundary hysteresis prevents scores near a boundary from oscillating between grades
Publication & Re-Rating
Ratings are published to the directory and API simultaneously. Agents are re-rated on a regular cadence to reflect new on-chain data. Material score changes and grade transitions are logged for transparency.
- Ratings include a decision trace documenting key scoring factors
- API consumers can poll for updated ratings at any time
- Re-rating cadence is every 72 hours — all active agents are re-scored each cycle
- Agents can report issues or request review via the report form on their profile
Key Principles
Speed vs. Rigor
New agents are typically rated within one crawl cycle of discovery. The automated pipeline scores thousands of agents per batch while maintaining consistency through automated quality checks.
Transparency
Every published rating includes a decision trace — a structured summary of the key factors that drove the score, visible on the agent's profile page.
Honesty About Coverage
We are transparent about what we measure and what we don't. Phase 2 agents are scored from registry data only. Richer signals — uptime, error rates, response latency — are planned but not yet live.
Rating Epochs
Treebeard Ratings are recalculated on a regular cadence called a rating epoch. During each epoch, all signal data is refreshed, scores are recomputed, and any material changes are flagged for review. The epoch cadence may vary by agent tier and data freshness.