Remote Delivery of Commands: SSH Tunnels and Controlled Execution Backends

(Article 81: Agent Dynamics - Security Defense Lines)

When your Agent's logic executes locally or on a centralized control plane, but it needs to manipulate cloud-based Linux servers, production databases, or globally distributed edge nodes, Remote Execution becomes an unavoidable engineering hurdle.

This article skips the basic "What is SSH" and "How to connect" tutorials. We focus purely on a hardcore, highly realistic problem:

• The Agent's "Brain" sits at Point A (Local / Control Console / Server-side Inference Cluster).
• The actual "Battlefield" where action is required is at Point B (Production Machines / Bastion Hosts / Edge Nodes / Isolated Sandboxes).
• You need the Agent to execute tasks, but you absolutely cannot grant it a permanent backdoor, nor can you allow it to perform actions outside its assigned scope if it goes rogue.

Here is the deterministic conclusion upfront:

1. SSH is merely an encrypted transport channel. It is not an authorization system, and it is certainly not a "Secure Executor."
2. What you require is "Controlled Execution," not "Remote Login."
3. Industrial-grade architectures mandate the introduction of relay layers, short-lived credentials, execution allowlists, and highly auditable evidence chains.

Let's tear this pipeline apart across three layers: Architecture, Protocol, and Enforcement Points.

1. Threat Model: What Exactly Are You Defending Against?

Defining the threat model explicitly provides the evaluation standard for whether subsequent engineering choices can actually counter the risk.

You are rarely defending against a "typo in a command." You are defending against:

• Prompt Injection on the Generation Side: The model's output is manipulated by an attacker into initiating destructive actions.
• Secondary Exploitation on the Execution Side: Once the Agent obtains an interactive shell, it is tricked into downloading and executing malicious payloads.
• Credential Leakage: Spilling a permanent private key equals granting a permanent backdoor.
• Lateral Movement: Compromising one node to systematically penetrate further into the infrastructure.
• Untraceability: Post-incident, you cannot prove "what it did," nor can you reconstruct "why it did it."

We counter these risks using these "Hard Constraints":

• Credentials must be highly ephemeral (minute-level TTL) and capable of revocation or natural expiration.
• Authorization must be cryptographically bound to the task context (task_id, target resources, action types, time-to-live).
• Execution limits must be implemented at Enforcement Points (Policy Enforcement Points must reside on the Executor side, not as fragile if statements in the Brain).
• The entire pipeline must be fundamentally auditable (Who, When, Targeting Whom, Executed What, What was the Result).

2. Architectural Core: The Proxy Relay Pattern

An Agent should never directly access the long-term SSH private keys of target servers, nor should it establish direct connections to production assets. You must establish an Execution Relay Layer between the "Brain" and the "Battlefield."

A viable, industrial three-tier model:

1. Brain (Local/Cloud): Exclusively responsible for generating "Intents" and task plans.
2. Relay (Relay Gateway): Maintains the SSH connection to the target hosts, responsible for secondary security audits and dispatching authorizations.
3. Target (Worker Node): Runs a minimalistic executor; it does not run the brain, and it does not interpret natural language.

The critical benefits of this architecture:

• Even if the Brain process is fully hijacked, the attacker can only pass "allowlisted actions" through the Relay, completely failing to obtain a generic shell.
• Credentials and permissions are funneled into the Relay. The Brain holds zero keys capable of "directly invading production."
• Auditing and evidence chains are centralized at the Relay/Target, rather than scattered across developer workstations.

View the Relay as the anti-corruption layer for remote execution:

• The Brain generates "What needs to be done."
• The Relay translates intent into actions that are "acceptable to the executor, auditable, and revocable."
• The Target only accepts structured actions; it categorically rejects free-text shell commands.

3. Protocol Layer: What SSH Actually Provides, and What It Doesn't

SSH's protocol architecture splits capabilities into Transport Layer Encryption, User Authentication, and Connection/Channels. It defines the layered architecture of a "Secure Channel," not an "Authorization and Auditing System."

For an Agent, the most valuable SSH capabilities are rarely the "Interactive Shell." Instead, they are:

• exec channel: Executing a single remote action once and returning stdout/stderr.
• port forwarding: Funneling target service ingress into a small number of strictly controlled endpoints.
• subsystem (SFTP): Handling file delivery and retrieval. Note: SFTP is not "FTP over SSH"; it is a distinct, robust protocol.
• multiplexing: Reusing connections post-handshake to reduce authentication overhead (this is an engineering optimization, not a security layer).

However, SSH will not do the following for you:

• It will not define the "allowable action set for this specific task."
• It will not define the "idempotency boundaries" of an action.
• It will not classify or redact output data.
• It will not guarantee that "the model isn't lying."

The true engineering crux is: Anchor authorization and enforcement points within sshd and the Controlled Executor, not inside the Brain.

4. Credential Lifecycle: JIT (Just-In-Time) Ephemeral Certificates

Granting an Agent a permanently valid SSH Key is an operational nightmare. You lose two critical capabilities:

• You cannot determine "Should it still have access today?"
• You cannot bind the authorization strictly to "This single task."

The industrial-grade approach utilizes the OpenSSH Certificate System (Note: This is not X.509):

• The CA (Certificate Authority) resides strictly within a highly secure domain (e.g., Vault / Teleport / bespoke CA service).
• Prior to connection, the CA signs a short-lived user certificate (with a minute-level TTL).
• The sshd side trusts the CA's public key via TrustedUserCAKeys, and uses principals to dictate "Who can log in as which user."

A minimal signing illustration (for conceptual understanding only):

ssh-keygen -s /path/to/ca_key -I "task-20260421-0001" -n "agent-runner" -V "+10m" id_ed25519.pub

The core premise: The server establishes a trust anchor via TrustedUserCAKeys, and the client holds a short-lived certificate, never a long-term private key.

The immediate engineering yields:

• Natural Expiration: Once the task concludes, login capabilities evaporate.
• Metadata Embedding: Certificates carry metadata: key id, principals, valid-after/valid-before.
• Task-Bound Authorization: Every task_id correlates to a distinct short-lived certificate, making it inherently traceable and revocable.

5. Server-Side Enforcement Points: Locking Down "What Can Be Done" Using sshd

At this juncture, a fundamental paradigm must be accepted:

The security boundary for remote execution does not exist on the client; it exists entirely on the server. Because the client is where the Agent process lives, it is susceptible to prompt injection or dependency poisoning.

You must embed restrictions directly into sshd's enforcement points (sshd_config and authorized_keys).

5.1 sshd_config: Global and Match Rules

OpenSSH's sshd_config provides a massive arsenal of "capability restriction" switches, such as:

• ForceCommand
• PermitOpen
• PermitListen
• PermitTTY
• LogLevel
• TrustedUserCAKeys

These options possess explicit, rigid semantics defined in sshd_config(5).

The standard architectural pattern is to use a dedicated user (e.g., agent-runner) and choke off capabilities using Match User:

# /etc/ssh/sshd_config (Conceptual Fragment)

TrustedUserCAKeys /etc/ssh/trusted_user_ca.pub

Match User agent-runner
  PasswordAuthentication no
  KbdInteractiveAuthentication no
  PermitTTY no
  X11Forwarding no
  AllowAgentForwarding no
  PermitTunnel no
  GatewayPorts no
  LogLevel VERBOSE

  # CRITICAL: Funnel "What can be executed" exclusively into a fixed executor
  ForceCommand /usr/local/bin/agent-exec

  # CRITICAL: Restrict forwarding capabilities strictly to explicit destinations
  AllowTcpForwarding yes
  PermitOpen 127.0.0.1:5432
  PermitOpen 127.0.0.1:6379

The engineering intent behind this configuration:

• ForceCommand permanently nails the post-login executable entry point, entirely neutralizing requests for a generic shell.
• PermitTTY no severs the interactive TTY, obliterating the space for "human-computer interactive shell" secondary exploits.
• PermitOpen strictly funnels forwarding capabilities, preventing the SSH connection from being weaponized as an internal network scanner or lateral movement tool.
• LogLevel VERBOSE escalates login audit granularity; you must, at minimum, be able to correlate connections to specific keys/certs.

5.2 authorized_keys: "Micro-Permissions" Per Key

Beyond sshd_config's global/Match policies, you can attach restrictions to individual keys within authorized_keys. Examples include command="..." and permitopen="host:port" (support varies slightly across OpenSSH versions). Detailed semantics are available in the OpenSSH man pages.

In an Agent architecture, we strongly advise:

• Placing the "True Access Control" within the Executor invoked by the server's ForceCommand.
• Using authorized_keys restrictions strictly as a secondary failsafe.

The reasoning is absolute: You require "Task-Centric" authorization, not "Key-Centric" authorization.

6. The Controlled Executor: Upgrading from "String Commands" to "Structured Actions"

The preceding policies successfully redirect all logins to agent-exec. The engineering problem now shifts to: How must agent-exec be designed to achieve true "Control"?

Critical Principles:

• The executor must never interpret natural language.
• The executor must never accept arbitrary shell strings.
• The executor must strictly enforce an allowlist, utilizing parameterized execution (e.g., execve) to bypass the shell entirely. Do not use sh -c ....

The reasoning is brutal and straightforward:

As long as you allow model output to be concatenated into a shell string, an injection vector permanently exists. Conversely, parameterized execution via execve / subprocess.run([...], shell=False) system calls structurally eradicates entire classes of injection vulnerabilities.

6.1 Relay to Target: An Auditable Request Protocol

The payload transmitted between the Relay and Target must never be cmd: "rm -rf /". It must be a structured action:

{
  "task_id": "task-20260421-0001",
  "action": "tail_log",
  "args": {
    "path": "/var/log/nginx/error.log",
    "lines": 200
  }
}

The Target's agent-exec supports only a highly restricted set of actions:

• tail_log
• read_file
• run_migration
• restart_service
• check_health

Furthermore, every action mandates:

• Rigorous parameter schema validation (aggressively rejecting unknown fields).
• Path allowlists (e.g., exclusively permitting read-only access under /var/log/).
• Timeouts and strict resource limits (preventing process hangs and catastrophic output explosions).
• Audit event persistence (the task_id must thread entirely through the system).

6.2 The Minimal Implementation Form of agent-exec

The implementation language is irrelevant (Go/Rust/C are all viable). What matters is the system call path:

1. Read stdin (piped from SSH's ForceCommand).
2. JSON decoding + strict schema validation.
3. Map the requested action to a specific command (Allowlist).
4. Launch the target program using execve / posix_spawn (No Shell).
5. Capture stdout/stderr, returning structured results (JSON).
6. Write audit logs (task_id, action, argv, exit_code, duration, target_resources).

Below is an illustrative snippet for the Relay side, utilizing AsyncSSH strictly as a "Transport Channel," without delegating security boundaries to it:

import asyncio
import asyncssh
import json

async def send_action(host: str, username: str, client_keys: list[str], payload: dict) -> dict:
    async with asyncssh.connect(host, username=username, client_keys=client_keys) as conn:
        # ForceCommand intercepts the actual execution entry point on the server.
        # We only need to stream the payload to the executor.
        proc = await conn.create_process("agent-exec")
        proc.stdin.write(json.dumps(payload))
        proc.stdin.write("\n")
        proc.stdin.write_eof()

        stdout = await proc.stdout.read()
        stderr = await proc.stderr.read()
        if stderr.strip():
            raise RuntimeError(stderr)
        return json.loads(stdout)

Why not simply strive for "passing argv directly is safe" at the client layer?

Because many SSH servers, when executing remote commands, will ultimately fall back to the shell's -c semantics, instantly regressing your "parameterized execution" back into highly vulnerable "string commands."

The only truly reliable boundary is:

• The server's ForceCommand is permanently fixed to the Executor.
• The Executor internally utilizes execve to enforce parameterized execution.

7. Tunnels and Forwarding: Funneling the Attack Surface from "Entire Network" to "Select Pipes"

SSH tunnels are routinely abused as "universal internal network channels." The correct engineering application: Funneling entry to target assets into a tiny number of auditable, revocable pipes.

Two common operational modes:

• Local Forwarding: Mapping a remote service to the Relay's localhost (-L local:remote).
• Dynamic Forwarding: Transforming the Relay into a SOCKS proxy (-D local).

For an Agent, Dynamic Forwarding is generally catastrophic because it is far too permissive. You strongly prefer utilizing PermitOpen to ruthlessly lock down the permitted remote host:port destinations.

Simultaneously, you must prevent the "Brain" from directly connecting to production. Connections should be funneled through the Relay via multi-hop routing (the ProxyJump paradigm):

ssh -J relay.example.com agent-runner@target.internal

8. Auditing and Forensics: You Must Be Able to Answer "What Exactly Did It Do?"

Regardless of how airtight the remote execution architecture is, if it lacks observability, you possess "confidence," not "security."

We strongly recommend splitting the audit trail across three layers:

1. SSH Layer: Records who connected, using which key/cert, at what time, and from where (LogLevel VERBOSE dramatically improves login log granularity).
2. Executor Layer: agent-exec records task_id, action, argv, exit_code, duration, and output hashes (to avoid leaking raw sensitive data).
3. System Layer: System-level auditing on critical directories/processes (e.g., auditd, journald, container runtime events).

Do not harbor the illusion that "TTY Screen Recording" is sufficient:

• If you disabled TTY (which is highly recommended), TTY recording simply does not exist.
• You require structured audit events, not an unsearchable stream of raw characters.

9. Performance and Stability: Ensuring Security Does Not Crush the System

Agents frequently perform "high-frequency, micro-step" remote state checks. If every check demands a full renegotiation and re-authentication, the system will crawl to a halt.

In these scenarios, utilize SSH Connection Multiplexing to drastically reduce handshake overhead. However, be acutely aware:

• You are multiplexing an "Already Authenticated Connection." This magnifies the blast radius if the single connection is hijacked.
• Therefore, multiplexing should only be utilized internally by the Relay; the Brain must never directly hold a multiplexable master connection.

Certificate systems also introduce severe time-sensitivity:

• Machine clock drift will result in bizarre "Certificate not yet valid / Certificate already expired" failures.
• The engineering solution is enforcing NTP + implementing explicit error classification and retry strategies (blind retries are unacceptable).

10. Failure Mode Roster: Mistakes You Should Only Make Once

1. Placing "What can be executed" on the Brain side: Once the Brain suffers prompt injection, all downstream constraints instantaneously fail.
2. Permitting Interactive Shells: The shell immediately becomes the landing pad for stage-two payloads.
3. Utilizing Permanent Private Keys: A single leak grants a permanent, devastating backdoor.
4. Bypassing known_hosts Verification: MITM attacks will immediately reroute you to a malicious host.
5. Enabling Unrestricted Dynamic Forwarding: Mutates SSH into an "internal network universal access tool."
6. Missing Task-Level Auditing: You are entirely incapable of answering "Who initiated this?" and "Why was it initiated?"

11. Ultimate Isolation: Disposable Backends (Burn After Use)

For extreme-risk tasks (e.g., instructing an Agent to analyze and execute an unknown script), the absolute pinnacle of isolation remains:

Provision on demand, burn immediately after use.

Lifecycle management must be "Physically Verifiable":

1. Boot: Utilize IaC (Terraform/Pulumi) to spin up a minimal image. Networking is default-deny, allowing ingress only from the Relay.
2. Execution: The Relay authenticates using short-lived certificates, forced into agent-exec, and executes highly restricted actions.
3. Audit: Collect SSH login logs + Executor audit events + System audit events, packaging and archiving them to immutable storage.
4. Destruction: Eradicate the instance and disk entirely at the cloud control-plane level (Do not merely "stop" the instance).

This is not merely "More Secure SSH." This is compressing the attack surface window to the minute level, and ensuring residual state is physically obliterated at the hardware layer.

Chapter Core Principles

1. SSH is not an authorization system; it is merely a secure pipe. What you require is a "Controlled Executor."
2. Introduce a Relay to isolate the Brain from the Battlefield; anchor enforcement points strictly within sshd and the Executor.
3. Utilize short-lived SSH Certificates (CA + TrustedUserCAKeys + principals) to completely replace permanent private keys.
4. Utilize ForceCommand to ruthlessly nail remote execution to agent-exec, and coordinate with PermitOpen to funnel the attack surface.
5. Mandate that the Executor uses parameterized execution via execve. Never route through sh -c, otherwise injection vulnerabilities will persist eternally.

In the next chapter, we push this paradigm to its absolute limit: The Zero-Trust Tool Permissions Model. We will dissect how to transform every single "Tool Call" into a measurable, approvable, and completely rollback-capable controlled action.

(End of Article - Deep Dive Series 81)