Search (Web Search/Scrape) — foundations RFC

Status

*ccepted*— ratificada 20260509 (mesmo dia da abertura) como parte da onda piloto de bootstrap servicesai. Implementação iniciada em `servicesaisearch; tickets em servicesaisearchbacklogpending/{001..005}`.

Summary

Web searchscraping pra agents — análogo Perplexity APITavilyFirecrawlExa.

Motivation

Foi referenciado no commit recente (/k-evolve koder-ai vs firecrawl+wispr+testsprite em 20260509)! Agents sem web search = dump completo do mundo na sessão. Foundation crítica pra agentes úteis.

Scope

In

• Web search (proxy + fallback engine)
• Scrape (Firecrawl-like)
• Citations
• Dedup

Out (yet)

• Deep research multi-step (escopo agents)

Initial design

Surfaces

• backend/ — Go API + worker scrape
• app/ — não aplicável v1

Key APIs

• POST /v1/search/web — search
• POST /v1/search/scrape — fetch + clean
• POST /v1/search/news — news vertical

Dependencies

• services/ai/gateway — LLM rerank
• services/ai/cache — search results
• infra/data/kdb-blob — scrape cache

Relation to existing sectors

• Pré-requisito de agents úteis em produção
• Consome cache pra reduzir custo

Selfhostedfirst analysis (5 gates)

• *1 Feature parity* zero
• *2 Performance* N/A
• *3 Stability* N/A
• *4 Capability* SearxNG self-hosted + Firecrawl FOSS viáveis
• *5 Critical-path readiness* destrava agents de produção

Open questions

• Q1: Default engine — SearxNG self-hosted ou proxy comercial?
• Q2: Robots.txt compliance enforce no service?

Next steps

1. Ratificar esta RFC (1 round de comments).
2. Criar sector dir services/ai/search/ com koder.toml, README.md, skeleton.
3. Abrir tickets de implementação em services/ai/search/backlog/pending/.
4. Registrar em meta/docs/stack/registries/self-hosted-pairs.md se substituir externo.