Image generated using DALL·E (OpenAI).
What Happens When Systems Engineers Cannot Fully Explain Their Own Systems
For more than 2,500 years, the ancient game of Go has challenged the world’s greatest strategic thinkers, its possible board positions so vast that they exceed the number of atoms in the observable universe. Mastery requires decades of study.
By 2016, Lee Sedol was a Go world champion. Over a two‑decade professional career, he had won 18 international titles and developed a deep understanding of the game’s strategic subtleties, making him one of the strongest players in the world.
In March of that year, he faced an artificial intelligence system called AlphaGo. During game 2 of the five‑game match, the AI made a move that stunned the professional Go community. On move 37, AlphaGo placed a stone in a position so unconventional that commentators believed the system had made a mistake. Lee Sedol was so shocked and confused that he stood up from the board and left the room to try to understand what had just happened. He came back to resume play, but the strange move slowly began to reveal its true purpose. What first appeared to be an error turned out to be a brilliant strategic decision. In retrospect, Move 37 became a striking example of emergence: a powerful system‑level behavior arising from interactions among millions of internal parameters rather than from a move explicitly designed by human engineers. AlphaGo eventually won the match, and the moment became famous in AI history as “Move 37.” [1]
But the most remarkable part of this story is not simply that AlphaGo played a brilliant move. It is that even the engineers and programmers who built the system could not fully explain why it chose that move. As AI researcher Dario Amodei has noted, many modern machine learning systems are “opaque, making them effectively black boxes even to their designers.” [2] In other words, the reasoning that produced Move 37 existed somewhere inside the system, but it was not expressed in a form that engineers could easily inspect or explain.
Lee Sedol during the AlphaGo match in Seoul, 2016. Photo courtesy of DeepMind.
The Role of the Black Box in System Design
For systems engineers, the phrase “black box” has a very specific meaning. The INCOSE Systems Engineering Handbook defines a black box as “a representative of an external view of the system (attributes).” [3] It represents an early abstraction used during system development. Engineers first define what the system must accomplish before determining how the system will accomplish it.
In this stage, the system is treated as a functional entity with defined inputs, outputs, and behaviors. Stakeholder needs are translated into expected system responses. The focus is on defining system functionality without prematurely constraining the internal design.
This abstraction is intentional. It allows engineers to explore different architectural approaches that could satisfy the required functionality. However, Systems Engineering does not stop at the black box. The discipline requires engineers to eventually open the box and understand how the system achieves its behavior.
Why Systems Engineers Open the Box
Once system behavior is defined, engineers must transition from the black box view to a deeper understanding of how the system achieves that behavior. This is often described as moving from a black box to a “white box” understanding.
In this phase, engineers decompose the system into components, define interfaces, and analyze the interactions that produce system‑level behavior. This decomposition creates traceability between stakeholder requirements and the physical or logical structures that implement them.
Understanding the internal structure of a system enables engineers to perform verification, validation, and risk analysis. Without this visibility, it becomes difficult to predict how the system will behave under changing conditions, unexpected inputs, or operational stress.
Complex systems frequently exhibit emergent behavior. System-level capabilities may arise from the interaction of components rather than from any single element. Systems Engineering acknowledges this reality. What matters is whether the engineers responsible for the system understand the interactions that produce these behaviors.
AI and the Return of the Black Box
What makes modern AI unusual is that engineers may never fully open and understand what is inside the box. In effect, the system can remain in the black‑box stage of understanding far longer than traditional Systems Engineering practice would expect. In many AI systems, the internal logic emerges during training rather than from explicit design, leaving engineers with a system that behaves like a permanent black box.
The INCOSE Systems Engineering Handbook reminds us that this challenge is not entirely new. Section 1.3.7 introduces the concept of complexity and notes that some systems exhibit behaviors that emerge from interactions among their components rather than from explicit design [3]. The INCOSE publication A Complexity Primer for Systems Engineers describes characteristics such as emergence, feedback, and self‑organization, where structure and behavior arise from within the system itself [4]. Modern AI systems exhibit many of these traits. During training, neural networks adjust billions of internal parameters and develop patterns that were never directly programmed. In this sense, AlphaGo’s surprising move may not simply be an AI curiosity. It may be a glimpse of how complex, adaptive systems behave when their internal logic evolves beyond straightforward human design.
As AI researcher Chris Olah has observed, modern neural networks are “not programmed in the traditional sense. They are trained, and their internal representations emerge during optimization.” [5] Engineers define the architecture, training data, and learning algorithms, but the internal logic that ultimately governs the system emerges during the training process. Patterns, strategies, and representations develop through billions of iterative adjustments inside the neural network. From a systems perspective, this is a classic example of emergence and self‑organization, two defining characteristics of complex systems. The behavior of the overall system arises from interactions among its components rather than from explicit human design.
The Systems Engineering Perspective
Systems Engineering has long emphasized the importance of understanding interactions, interfaces, and emergent behavior within complex systems. Emergence is not unusual. Many large systems exhibit capabilities that arise from component interactions rather than from explicit design decisions.
The concern arises when emergence occurs within structures that engineers cannot clearly observe or explain. In those situations, predicting system behavior becomes more difficult, and engineering confidence decreases.
For systems engineers responsible for safety, reliability, and mission success, opacity introduces risk. Systems that remain poorly understood become harder to verify, validate, and safely operate over time. This does not mean complex systems should be avoided. It means engineers must remain aware of the limits of their understanding and treat those limits as engineering risks to be managed.
A Question Worth Asking
The rapid advancement of AI technologies makes this conversation particularly relevant. As organizations deploy increasingly complex models, systems engineers must ask an important question.
Are we building systems that we understand well enough to responsibly operate?
If a system remains a black box even to its designers, it raises difficult questions about verification, validation, and long‑term operational trust. Complex systems will always surprise their designers. The goal of Systems Engineering is not to eliminate complexity, but to understand it well enough to design, verify, and operate systems responsibly.
As systems engineers, we should ask ourselves a difficult question:
Are the systems being deployed today understood well enough to trust?
Optional Reader Resource
References
Silver, David et al. Mastering the Game of Go with Deep Neural Networks and Tree Search. Nature, vol. 529, 2016, pp. 484–489. https://www.nature.com/articles/nature16961
Amodei, Dario et al. “Concrete Problems in AI Safety.” arXiv, 2016. https://arxiv.org/abs/1606.06565
International Council on Systems Engineering (INCOSE). INCOSE Systems Engineering Handbook: A Guide for System Life Cycle Processes and Activities, Version 5. Hoboken, NJ: Wiley, 2023.
International Council on Systems Engineering (INCOSE). A Complexity Primer for Systems Engineers. INCOSE Technical Publication, 2021.
Olah, Chris. “The Building Blocks of Interpretability.” Distill, 2018. https://distill.pub/2018/building-blocks/
