Vision Agents

Use it when

• You need an AI agent that watches a live video feed and gives spoken real-time feedback, such as sports coaching, workout form correction, or drone monitoring
• You are hand-wiring WebRTC, STT, TTS, turn detection, and a YOLO or Roboflow model around a realtime LLM and the plumbing keeps breaking

What it solves

Not the fit when

• Sensitive video or audio unless the privacy, retention, region, and training terms of every selected edge, model, STT, TTS, and phone provider are acceptable
• Assuming the free Stream allowance covers production volume or that model, speech, and phone providers are free
• Medical, safety, fraud, or security decisions without independent validation and human oversight
• Reading small text in video; the README warns models may hallucinate scores and signs
• Continuous video understanding over long sessions; the README says context degrades after roughly 30 seconds
• Video-only triggering; realtime models require audio or text input to respond
• batch offline video file analysis
• document ocr
• static image classification training
• web page scraping

Install

uv add vision-agents (optional extras: uv add "vision-agents[getstream, openai, elevenlabs, deepgram]"); requires a free Stream API key plus model provider keys

Invoke

Python: agent = Agent(edge=getstream.Edge(), agent_user=agent_user, instructions="...", llm=gemini.Realtime(fps=10), processors=[ultralytics.YOLOPoseProcessor(model_path="yolo11n-pose.pt")])

Alternatives

No reviewed alternatives recorded yet.