Episode #15
Don’t Let AI Go Rogue: Human Governance for Agentic Systems
As AI systems become more capable of reasoning, using tools, accessing data, and taking action on their own, a new question is becoming impossible to ignore: how much autonomy should we actually give them? In this episode of Ship Happens , host Per Krogslund sits down with Justin Kuiper, Lead Architect at Future Tech, to explore the security and governance challenges of agentic AI—and what engineers can do to keep increasingly autonomous systems within safe and understandable boundaries. With a background in Air Force space operations and cybersecurity, Justin brings lessons from mission-critical systems to the rapidly evolving world of AI. He explains how different environments prioritize availability, confidentiality, and mission assurance, and why those tradeoffs become even more important when software can make decisions and take actions with real-world consequences. Per and Justin dig into the importance of human governance, least privilege, agent identity, observability, break-glass controls, and specification engineering. They explore the risk of “confident misalignment”—when an AI system confidently pursues an outcome that isn't actually what its human operators intended—and why giving agents more capability doesn't mean giving them unlimited authority. The conversation also looks at AI in space, data sovereignty, model selection, token economics, and the emerging challenge of securing entire agent ecosystems rather than individual applications. The practical message for builders is clear: define what your agents are allowed to do, know who is responsible for their actions, constrain their access, observe what they're doing, document the system, and keep humans in control when the stakes are high. Because autonomous software can move fast. Your governance needs to move with it. What You’ll Learn Why agentic AI introduces a different class of security and governance challenges What space systems can teach us about designing software for high-consequence environments Why human governance becomes more important as AI systems become more autonomous What “confident misalignment” means and why it can be dangerous How least privilege can be applied to AI agents and the tools they use Why every agent needs a clear identity and an accountable owner How observability and break-glass controls can help keep autonomous workflows under control Why the workflow—not just the individual application—can become the security problem How specification engineering can create stronger guardrails around AI behavior What data sovereignty means in an increasingly agent-driven ecosystem How model selection and token economics factor into AI architecture Why organizations need to decide on an agentic operating model before deploying autonomous systems Why “fail closed” can be an important principle for high-consequence AI systems Episode Chapters 00:00 — Satan in the Workflow 01:25 — Meet Justin Kuiper 03:22 — Securing Software in Space 06:06 — AI Security and Black Boxes 11:33 — Human Governance for Agents 13:51 — Least Privilege for Tools 16:03 — Agent Identity and Accountability 17:27 — Agents in Production Workflows 19:55 — Confident Misalignment Risks 20:52 — Accountability for AI Weapons 22:12 — Technology That Explores 22:38 — Balancing Human Control 24:05 — Losing Institutional Know-How 26:19 — Specification Engineering Guardrails 26:52 — AI in Space: The Timeline 27:44 — Model Choices and Sovereignty 28:44 — Tokenomics and the Data Mesh 30:31 — Securing Agent Ecosystems 34:31 — The Monday Morning Playbook 37:56 — Avoiding Overengineering 40:03 — Trust in Autonomous Software 41:10 — Closing Thanks Key Takeaways Autonomy Needs Boundaries Giving an AI agent the ability to act independently doesn't mean giving it unlimited authority. Engineers need to define the systems, data, tools, and decisions an agent can access—and where human approval is required. Humans Still Own the Governance The more autonomous software becomes, the more important it is to establish clear responsibility. Someone needs to understand what the system is designed to do, what constraints are in place, and who is accountable when it behaves unexpectedly. Least Privilege Should Apply to AI Agents shouldn't automatically receive broad access simply because they might need it someday. Limiting an agent's permissions to the minimum required for its task can reduce the potential impact of a compromised or misbehaving system. Agent Identity Matters If an agent can take actions independently, those actions need to be attributable. Identity allows organizations to understand which agent acted, what it was authorized to do, and where responsibility ultimately sits. Watch the Workflow Individual components can appear secure while the interactions between them introduce unexpected risks. As agents move between applications, APIs, data sources, and tools, understanding the entire workflow becomes critical. Watch for Confident Misalignment One of the biggest risks isn't necessarily an AI that refuses to work. It's an AI that confidently does the wrong thing because its interpretation of the objective differs from what its human operator actually intended. Build for Failure In high-consequence environments, systems need a safe way to stop. Break-glass controls, escalation paths, observability, and fail-closed behavior can provide critical safeguards when an autonomous system reaches the limits of its authority. About Justin Kuiper Justin Kuiper is a Lead Architect at Future Tech with a background in Air Force space operations and cybersecurity. His experience working with complex, mission-critical systems gives him a unique perspective on the challenges of securing increasingly autonomous technology. In this episode, Justin connects lessons from space systems and cybersecurity with the emerging world of agentic AI, exploring how engineers can build systems that are not only capable, but also constrained, observable, and accountable. Resources & Links Future Tech — Justin Kuiper’s organization MITRE ATT&CK — Framework for understanding adversary tactics and techniques Docker — The open platform for building, shipping, and running applications Hosted by Simplecast, an AdsWizz company. See pcm.adswizz.com for information about our collection and use of personal data for advertising.
