Implement Borealis Agents #2

New Issue

nicole · 2025-04-03T20:21:50-06:00

nicole commented

2025-04-03 20:21:50 -06:00

A "lightweight reverse provisioning" model, where agents are dumb(ish), and Borealis acts as a brain that tells them what to do and where to send the data. Essentially building a decentralized input mesh with a central control UI. Let's break it down:

🔄 TL;DR: Your Proposed Design Is Solid and Scalable

✅ Centralized control (Borealis server)
✅ Decentralized data capture (Collectors)
✅ Node-based provisioning
✅ Live feedback/UI orchestration
✅ Cleaner separation of UI vs. OS-level interaction

This gives you scalability, real-time control, and beautiful UX for managing agents from within the visual graph.

🧩 Architecture Overview (Based on Your Vision)

⚙️ 1. Agent Startup

Collector Agent is a standalone app or script
User launches it and enters the API endpoint (URL + optional token)
It performs a POST /api/agent/register or POST /api/agent/checkin every X seconds with a UUID (Agent-123), hostname, etc.

📟 2. Borealis Backend State Management

Server keeps a dict or in-memory store of known agents:

agents = {
  "Agent-123": {
    "status": "orphaned",
    "last_seen": timestamp,
    "config": None,
    "ip": "x.x.x.x"
  }
}

🎛️ 3. Frontend - API Collector Node

React Flow node shows a dropdown of orphaned agents
Once one is selected, the node sends a POST /api/agent/provision with:
- region dimensions
- collection type (e.g. screenshot only, OCR, pixel match, etc.)
Server updates the agent’s config and sets status = provisioned

📡 4. Agent Polls for Tasks

Agent periodically does a GET /api/agent/config?agent_id=Agent-123
If provisioned:
- Agent starts screen region capture (QT overlay)
- Sends image (base64 or multipart) to /api/agent/data
- Borealis server OCRs and stores the result

🧠 5. Node Receives Data

The corresponding API Collector node fetches or subscribes to updates from Borealis server
Displays status, most recent values, or relays to downstream nodes

✨ Benefits of This Design

Feature	Description
Agent Flexibility	Agents stay simple. They don't need to know anything about OCR, logic, or backend—they just collect and ship.
Dynamic Control	Users can reconfigure agents mid-flight—change what they collect without restarting them.
ReactFlow Integration	Visual, user-friendly provisioning of agents through node UI is powerful and intuitive.
Security Options	You could expand this to include tokens, authentication, or ACLs per agent or user.
Horizontal Scalability	Works locally or with agents across different machines or networks—just use a public Borealis server URL.

🔐 Bonus Considerations

Agent ID: Use persistent UUIDs stored on the agent side, or generate one on first boot and save it.
Secure Registration: Optionally require a token or auth header to prevent rogue agents.
Agent Heartbeats: Use timestamps to detect offline/disconnected agents and display status in UI (green = connected, red = offline).
WebSocket or Polling: You can start with polling (simpler) and move to WebSocket push-based updates later if needed.

🛠 Suggested Endpoints (Phase 1)

# Agent-Initiated
POST /api/agent/checkin     -> { agent_id, hostname }
GET /api/agent/config       -> agent_id=Agent-123
POST /api/agent/data        -> { agent_id, payload }

# UI-Initiated
GET /api/agents             -> returns all agents + status (for dropdown)
POST /api/agent/provision   -> { agent_id, config }
GET /api/agent/:id/status   -> useful for frontend polling

✅ Create an agents registry in your Flask backend (server.py)
✅ Add a POST /api/agent/checkin that creates/updates agent state
✅ Add GET /api/agents that the frontend calls to populate the dropdown in the API Collector node
✅ Add POST /api/agent/provision that stores the region/task config
✅ Have the agent poll GET /api/agent/config?agent_id=... every 2–5s
✅ Build React node logic to interact with these endpoints and mark agents as "associated"
⬜ (Optional): Add WebSocket-based push for real-time updates later

Would you like me to draft the full backend Flask endpoints and a minimal React Flow node template for the "API Collector" node to bring this to life?

A **"lightweight reverse provisioning" model**, where agents are dumb(ish), and Borealis acts as a brain that tells them what to do and where to send the data. Essentially building a **decentralized input mesh with a central control UI**. Let's break it down: --- ## 🔄 TL;DR: Your Proposed Design Is Solid and Scalable ✅ **Centralized control (Borealis server)** ✅ **Decentralized data capture (Collectors)** ✅ **Node-based provisioning** ✅ **Live feedback/UI orchestration** ✅ **Cleaner separation of UI vs. OS-level interaction** This gives you scalability, real-time control, and beautiful UX for managing agents from within the visual graph. --- ## 🧩 Architecture Overview (Based on Your Vision) ### ⚙️ 1. **Agent Startup** - Collector Agent is a standalone app or script - User launches it and enters the API endpoint (URL + optional token) - It performs a `POST /api/agent/register` or `POST /api/agent/checkin` every X seconds with a UUID (Agent-123), hostname, etc. ### 📟 2. **Borealis Backend State Management** - Server keeps a dict or in-memory store of known agents: ```python agents = { "Agent-123": { "status": "orphaned", "last_seen": timestamp, "config": None, "ip": "x.x.x.x" } } ``` ### 🎛️ 3. **Frontend - API Collector Node** - React Flow node shows a dropdown of orphaned agents - Once one is selected, the node sends a `POST /api/agent/provision` with: - region dimensions - collection type (e.g. screenshot only, OCR, pixel match, etc.) - Server updates the agent’s config and sets `status = provisioned` ### 📡 4. **Agent Polls for Tasks** - Agent periodically does a `GET /api/agent/config?agent_id=Agent-123` - If provisioned: - Agent starts screen region capture (QT overlay) - Sends image (base64 or multipart) to `/api/agent/data` - Borealis server OCRs and stores the result ### 🧠 5. **Node Receives Data** - The corresponding API Collector node fetches or subscribes to updates from Borealis server - Displays status, most recent values, or relays to downstream nodes --- ## ✨ Benefits of This Design | Feature | Description | |--------|-------------| | **Agent Flexibility** | Agents stay simple. They don't need to know anything about OCR, logic, or backend—they just collect and ship. | | **Dynamic Control** | Users can reconfigure agents mid-flight—change what they collect without restarting them. | | **ReactFlow Integration** | Visual, user-friendly provisioning of agents through node UI is powerful and intuitive. | | **Security Options** | You could expand this to include tokens, authentication, or ACLs per agent or user. | | **Horizontal Scalability** | Works locally or with agents across different machines or networks—just use a public Borealis server URL. | --- ## 🔐 Bonus Considerations - **Agent ID**: Use persistent UUIDs stored on the agent side, or generate one on first boot and save it. - **Secure Registration**: Optionally require a token or auth header to prevent rogue agents. - **Agent Heartbeats**: Use timestamps to detect offline/disconnected agents and display status in UI (green = connected, red = offline). - **WebSocket or Polling**: You can start with polling (simpler) and move to WebSocket push-based updates later if needed. --- ## 🛠 Suggested Endpoints (Phase 1) ```http # Agent-Initiated POST /api/agent/checkin -> { agent_id, hostname } GET /api/agent/config -> agent_id=Agent-123 POST /api/agent/data -> { agent_id, payload } # UI-Initiated GET /api/agents -> returns all agents + status (for dropdown) POST /api/agent/provision -> { agent_id, config } GET /api/agent/:id/status -> useful for frontend polling ``` --- ## 🚀 Next Steps I Recommend 1. ✅ Create an `agents` registry in your Flask backend (`server.py`) 2. ✅ Add a `POST /api/agent/checkin` that creates/updates agent state 3. ✅ Add `GET /api/agents` that the frontend calls to populate the dropdown in the API Collector node 4. ✅ Add `POST /api/agent/provision` that stores the region/task config 5. ✅ Have the agent poll `GET /api/agent/config?agent_id=...` every 2–5s 6. ✅ Build React node logic to interact with these endpoints and mark agents as "associated" 7. ⬜ (Optional): Add WebSocket-based push for real-time updates later --- Would you like me to draft the full backend Flask endpoints and a minimal React Flow node template for the "API Collector" node to bring this to life?

nicole added the

Kind/Enhancement

Kind/Feature

labels 2025-04-03 20:21:50 -06:00

nicole self-assigned this 2025-04-03 20:24:15 -06:00

nicole added the

Priority

Medium

label 2025-04-03 20:44:03 -06:00

nicole commented

2025-04-04 03:21:22 -06:00

Agent Based communication implemented successfully. Some things still need some refinement, and the agents will be expanded to collect more types of data in the future, currently only collects base64-encoded screenshots of a specific region of the screen on either a local or remote computer and displays them in the WebUI of Borealis if you attach an Image Viewer node to the output of the API Data Collector nodes

nicole commented

2025-04-09 02:07:09 -06:00

Need to incorporate the ability to have multiple nodes that connect to the same agent datastream. Disabling / greying out already-adopted nodes prevents us from being able to have different workflows process data from the same initial source, which could be seen as a functional limitation.

Remove the automated agent dropdown menu grey-out code
Add a "View Raw Data" link as well as quick-copy materialUI icon for sharing raw screensharing datastreams from screenshot-provisioned agents

Need to incorporate the ability to have multiple nodes that connect to the same agent datastream. Disabling / greying out already-adopted nodes prevents us from being able to have different workflows process data from the same initial source, which could be seen as a functional limitation. - Remove the automated agent dropdown menu grey-out code - Add a "View Raw Data" link as well as quick-copy materialUI icon for sharing raw screensharing datastreams from screenshot-provisioned agents

nicole changed title from ~~Implement Agent-Based API Data Collectors~~ to Implement Borealis Agents

2025-04-11 16:12:35 -06:00

nicole commented

2025-04-11 16:13:40 -06:00

Multiple instances of the same agent node successfully implemented. More Node UI design changes will be coming before this core issue is completed.

nicole closed this issue

2025-04-13 06:47:16 -06:00

nicole added this to the Borealis - Feature Roadmap project 2025-05-16 14:42:05 -06:00

nicole moved this to Done in Borealis - Feature Roadmap on 2025-05-16 14:43:13 -06:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: bunny-lab/Borealis#2