下载地址
https://www.zhengdianxue.com/wp-content/uploads/skills/org-02qudk26--computer-use-agents.zip将提示词发送给你的 AI 安装该 skills
请根据 https://skillhub.cn/install/skillhub.md,安装 @org-02qudk26/computer-use-agents。构建能像人类一样与计算机交互的 AI 代理——查看屏幕、移动光标、点击按钮、输入文字。涵盖 Anthropic 的 Computer Use、OpenAI 的
阅读全文 (展开查看完整说明)
概述
计算机操作代理(Computer Use Agents)
模式
感知-推理-行动循环(Perception-Reasoning-Action Loop)
计算机操作代理的基础架构:观察屏幕、推理下一步动作、执行动作、重复。该循环通过迭代管线将视觉模型与动作执行整合在一起。
关键组件:
- 感知(PERCEPTION):截图捕获当前屏幕状态
- 推理(REASONING):视觉-语言模型分析并规划
- 行动(ACTION):执行鼠标/键盘操作
- 反馈(FEEDBACK):观察结果,继续或纠正
关键洞察:视觉代理在“思考”阶段(1-5 秒)完全静止,形成可检测到的停顿模式。
何时使用:['从零构建任何计算机操作代理', '将视觉模型与桌面控制集成', '理解代理行为模式']
from anthropic import Anthropic
from PIL import Image
import base64
import pyautogui
import time
class ComputerUseAgent:
"""
Perception-Reasoning-Action loop implementation.
Based on Anthropic Computer Use patterns.
"""
def __init__(self, client: Anthropic, model: str = "claude-sonnet-4-20250514"):
self.client = client
self.model = model
self.max_steps = 50 # Prevent runaway loops
self.action_delay = 0.5 # Seconds between actions
def capture_screenshot(self) -> str:
"""Capture screen and return base64 encoded image."""
screenshot = pyautogui.screenshot()
# Resize for token efficiency (1280x800 is good balance)
screenshot = screenshot.resize((1280, 800), Image.LANCZOS)
import io
buffer = io.BytesIO()
screenshot.save(buffer, format="PNG")
return base64.b64encode(buffer.getvalue()).decode()
def execute_action(self, action: dict) -> dict:
"""Execute mouse/keyboard action on the computer."""
action_type = action.get("type")
if action_type == "click":
x, y = action["x"], action["y"]
button = action.get("button", "left")
pyautogui.click(x, y, button=button)
return {"success": True, "action": f"clicked at ({x}, {y})"}
elif action_type == "type":
text = action["text"]
pyautogui.typewrite(text, interval=0.02)
return {"success": True, "action": f"typed {len(text)} chars"}
elif action_type == "key":
key = action["key"]
pyautogui.press(key)
return {"success": True, "action": f"pressed {key}"}
elif action_type == "scroll":
direction = action.get("direction", "down")
amount = action.get("amount", 3)
scroll = -amount if direction == "down" else amount
pyautogui.scroll(scroll)
return {"success": True, "action": f"scrolled {dir沙箱化环境模式(Sandboxed Environment Pattern)
计算机操作代理必须在隔离、沙箱化的环境中运行。绝不要让代理直接访问你的主系统——安全风险太高。请使用带有虚拟桌面的 Docker 容器。
关键隔离要求:
- 网络(NETWORK):仅限制为必要端点
- 文件系统(FILESYSTEM):只读或限定在临时目录
- 凭据(CREDENTIALS):无主机凭据访问权限
- 系统调用(SYSCALLS):过滤危险的系统调用
- 资源(RESOURCES):限制 CPU、内存、时间
目标是“最小化爆炸半径”——即使代理出错,损害也被限制在沙箱之内。
何时使用:['部署任何计算机操作代理', '安全地测试代理行为', '运行不可信的自动化任务']
# Dockerfile for sandboxed computer use environment
# Based on Anthropic's reference implementation pattern
FROM ubuntu:22.04
# Install desktop environment
RUN apt-get update && apt-get install -y
xvfb
x11vnc
fluxbox
xterm
firefox
python3
python3-pip
supervisor
# Security: Create non-root user
RUN useradd -m -s /bin/bash agent &&
mkdir -p /home/agent/.vnc
# Install Python dependencies
COPY requirements.txt /tmp/
RUN pip3 install -r /tmp/requirements.txt
# Security: Drop capabilities
RUN apt-get install -y --no-install-recommends libcap2-bin &&
setcap -r /usr/bin/python3 || true
# Copy agent code
COPY --chown=agent:agent . /app
WORKDIR /app
# Supervisor config for virtual display + VNC
COPY supervisord.conf /etc/supervisor/conf.d/
# Expose VNC port only (not desktop directly)
EXPOSE 5900
# Run as non-root
USER agent
CMD ["/usr/bin/supervisord", "-c", "/etc/supervisor/conf.d/supervisord.conf"]
---
# docker-compose.yml with security constraints
version: '3.8'
services:
computer-use-agent:
build: .
ports:
- "5900:5900" # VNC for observation
- "8080:8080" # API for control
# Security constraints
security_opt:
- no-new-privileges:true
- seccomp:seccomp-profile.json
# Resource limits
deploy:
resources:
limits:
cpus: '2'
memory: 4G
reservations:
cpus: '0.5'
memory: 1G
# Network isolation
networks:
- agent-network
# No access to host filesystem
volumes:
- agent-tmp:/tmp
# Read-only root filesystem
read_only: true
tmpfs:
- /run
- /var/run
# Environment
environment:
- DISPLAY=:99
- NO_PROXY=localhost
networks:
agent-network:
driver: bridge
internal: true # No internet by default
volumes:
agent-tmp:
---
# Python wrapper with additional runtime sandboxing
import subprocess
import os
from dataclasses imAnthropic Computer Use 实现
使用 Claude 的 computer use 能力的官方实现模式。Claude 3.5 Sonnet 是首个提供 computer use 的前沿模型。Claude Opus 4.5 现被称为“全球最强的 computer use 模型”。
关键能力:
- screenshot:捕获当前屏幕状态
- mouse:点击、移动、拖拽操作
- keyboard:输入文字、按键
- bash:运行 shell 命令
- text_editor:查看与编辑文件
工具版本:
- computer_20251124(Opus 4.5):新增 zoom 动作用于细节查看
- computer_20250124(所有其他模型):标准能力
关键限制:“某些 UI 元素(如下拉框和滚动条)Claude 可能难以操作”——Anthropic 文档
何时使用:['构建生产级计算机操作代理', '需要最高质量的视觉理解', '完整的桌面控制(不仅是浏览器)']
from anthropic import Anthropic
from anthropic.types.beta import (
BetaToolComputerUse20241022,
BetaToolBash20241022,
BetaToolTextEditor20241022,
)
import subprocess
import base64
from PIL import Image
import io
class AnthropicComputerUse:
"""
Official Anthropic Computer Use implementation.
Requires:
- Docker container with virtual display
- VNC for viewing agent actions
- Proper tool implementations
"""
def __init__(self):
self.client = Anthropic()
self.model = "claude-sonnet-4-20250514" # Best for computer use
self.screen_size = (1280, 800)
def get_tools(self) -> list:
"""Define computer use tools."""
return [
BetaToolComputerUse20241022(
type="computer_20241022",
name="computer",
display_width_px=self.screen_size[0],
display_height_px=self.screen_size[1],
),
BetaToolBash20241022(
type="bash_20241022",
name="bash",
),
BetaToolTextEditor20241022(
type="text_editor_20241022",
name="str_replace_editor",
),
]
def execute_tool(self, name: str, input: dict) -> dict:
"""Execute a tool and return result."""
if name == "computer":
return self._handle_computer_action(input)
elif name == "bash":
return self._handle_bash(input)
elif name == "str_replace_editor":
return self._handle_editor(input)
else:
return {"error": f"Unknown tool: {name}"}
def _handle_computer_action(self, input: dict) -> dict:
"""Handle computer control actions."""
action = input.get("action")
if action == "screenshot":
# Capture via xdotool/scrot
subprocess.run(["scrot", "/tmp/screenshot.png"])
with open("/tmp/screenshot.png", "rb") as f:
⚠️ 棘手之处(Sharp Edges)
| Issue | Severity | Solution |
|---|---|---|
| Issue | critical | ## 纵深防御——没有单一方案能奏效 |
| Issue | medium | ## 为动作加入拟人化随机性 |
| Issue | high | ## 尽可能使用键盘替代方案 |
| Issue | medium | ## 接受取舍 |
| Issue | high | ## 实现上下文管理 |
| Issue | high | ## 监控并限制成本 |
| Issue | critical | ## 始终使用沙箱化 |