Responsible Use of AI When Developing Critical Systems

The most useful question about AI in software development is what happens when it gets something wrong.

A mistake in a report layout might inconvenience a user until a patch arrives. A mistake in a component that handles device data could lose information that cannot be recovered. A mistake in shared infrastructure could affect the entire product.

Those consequences deserve different development rules.

At Tartabit, we use AI to help develop and improve IoT Bridge. It gives our engineers useful ways to investigate problems, work through ideas, implement bounded features, and keep supporting material current. The opportunity is substantial. So is our responsibility to protect the systems customers depend on.

Our approach starts with a simple principle: the greater the potential impact of a mistake, the less autonomy we give AI.

We think about that principle as three concentric rings. Informally, you could call them the “concentric rings of doom”: move toward the center, and the consequences of getting something wrong become broader and harder to repair.

The name is deliberately memorable. The engineering decisions behind it are practical: what can this change affect, who must control its implementation, and what would recovery involve?

AI autonomy increases toward the outside. Potential impact and review rigor increase toward the center. Human accountability applies to every ring.

The inner ring: protect the core

At the center is the common code and shared infrastructure that powers the rest of IoT Bridge. Changes here can reach well beyond the feature that prompted them.

A shared function may look small when viewed in isolation. Its importance comes from the components that depend on it, the assumptions they make, and the behavior they have come to rely on. A local improvement can become a product-wide regression if those relationships are misunderstood.

This is not an area for autonomous, agentic development. By agentic development, we mean giving an AI tool a goal and allowing it to work through implementation steps with substantial independence.

In the core, engineers lead the work. If AI assists with analysis, explanation, or a proposed change, everything it contributes must be reviewed and understood. A plausible implementation is only a starting point. The developer needs to know why it is correct, how it fits the existing architecture, and what its consequences are for the rest of the product.

The review extends beyond whether a particular function works. It must consider the shared behavior other components expect and the failure conditions that may not appear in the simplest test case.

The recovery burden explains the caution. Problems in this ring typically require rebuilding the product and delivering a full point release. An apparently modest change can carry a large validation and deployment responsibility.

AI can help an engineer reason about that work. The engineer remains responsible for the reasoning and the implementation.

The middle ring: preserve the data

The middle ring contains components whose failures have a more limited reach, but can still have serious consequences. They may not affect every part of IoT Bridge. They can, however, put customer data at risk.

For this ring, we have zero tolerance for “vibe coding”: accepting generated functionality because it looks convincing or appears to work, without understanding how it behaves.

AI's role here is that of a pair programmer. It can support investigation, compare an implementation with an established pattern, help analyze an unexpected result, or suggest cases the developer should examine. The human developer owns the design and directs the implementation.

That distinction matters. Asking AI to help understand a failure is a bounded task. Asking it to take over a data-handling feature and make whatever changes seem necessary gives it a much broader set of decisions. In this ring, those decisions stay with our engineers.

Existing paradigms are especially valuable. If IoT Bridge already has a trusted way to perform an operation, a new implementation should follow that approach. A feature request should not become an opportunity for an AI tool to invent a different architecture or expand the scope of the work.

Everything in this ring is reviewed. Validation must address the behavior that matters to data integrity, including what happens when an operation fails or is interrupted. A successful demonstration of the normal path is insufficient evidence for a component whose mistakes could lose data.

Recovery is generally more contained than in the core. A problem may be addressed by hot fixing and deploying one or two components, without rebuilding the entire product. But a smaller deployment does not make data loss acceptable. Replacing the faulty code and recovering the information it mishandled are very different problems.

This is why the middle ring still demands close human control.

The outer ring: give AI room to work

The outer ring is where agentic AI development can take a larger role. It includes isolated presentation work and ancillary features whose failure cannot affect the running data path or compromise data integrity.

Parts of the user interface and noncritical reporting are useful examples. Within a clearly defined task, an agent can take on more of the implementation, helping us deliver improvements while freeing developers to concentrate on work that needs their deeper attention.

There are still boundaries. The task needs a defined scope, an expected result, and a clear understanding of the components the agent is allowed to change. Generated code is reviewed and tested independently before it is accepted. The agent's own statement that the work is finished is not the acceptance decision.

The depth of review and validation is proportionate to the impact. An isolated display change does not need the same breadth of investigation as a change to a shared runtime dependency. It still needs to work correctly for the people who use it.

The defining condition is isolation: a regression in this ring must be contained so that runtime continues unaffected. That separation makes it possible to correct these issues quickly with a zero-downtime patch.

This creates a productive place to use AI's implementation capabilities. We can allow more autonomy because the architectural boundary limits the consequences and makes recovery straightforward.

Classify the consequence, not the folder

The rings are a way to judge changes, not permanent labels attached to directories.

A UI task is not automatically low risk. A screen that changes operational settings can influence the running system. A reporting change that modifies shared data access or writes to operational records may reach far beyond presentation. Such work needs the controls of the more critical ring it touches.

The same applies when a task grows. An agent working on a display improvement might conclude that it needs to change a shared library. At that point, the original outer-ring classification no longer covers the work. The change needs to be reassessed before proceeding.

Three questions help make the boundary concrete:

• How far could a mistake spread? Consider every component that depends on the behavior being changed.
• Could it lose data or disrupt runtime? A limited set of affected components can still carry serious operational consequences.
• What would recovery require? A full point release, a component hotfix, and an isolated patch represent different levels of exposure.

When a change crosses rings, the more critical part determines the level of control. The convenience of using an agent should never decide the risk classification.

Use the codebase as the guide

One of our most useful assets for AI-assisted development is the depth of our human-written codebase.

IoT Bridge already contains implementations shaped by experience: established conventions, familiar structures, and patterns that have been exercised over time. We guide our AI tools with those conventions and examples from existing features so new work follows the approaches our engineers already understand and trust.

That means asking the tool to find and follow the relevant implementation before creating something new. It means providing enough context to understand the surrounding code. It also means reviewing for consistency, not just checking whether the new feature appears to function.

This helps limit implementation drift. We do not want a growing collection of individually plausible features that each use a different approach to the same problem. Consistency makes the product easier to review, support, and extend.

Existing code is context, not automatic proof of correctness. Engineers still need to decide whether a pattern fits the new situation. AI makes that context easier to use; it does not remove the need for judgment.

Documentation is a particularly useful place to start

Some of the most valuable improvements have come from work around the product code.

Our product documentation is written in Markdown and served from a Git repository. That gives documentation changes a familiar, reviewable form: a set of edits that can be examined alongside the feature they describe.

Using AI to automatically prepare documentation updates as features are created has been a significant improvement in how we use developer time. The implementation provides the context, the existing documentation provides the structure, and the tool helps produce the corresponding customer-facing explanation.

The developer can then spend attention on accuracy: does the text describe what the feature actually does, explain its limits, and give customers the information they need?

Documentation still needs review. Incorrect instructions can confuse users or lead them to configure a feature incorrectly. The advantage is that much of the repeatable writing and maintenance can be handled without changing the product's runtime code.

It also helps close a familiar gap in software development. A feature is more useful when its documentation arrives with it, while the implementation details are still fresh in the developer's mind.

Spend developer attention where it matters

The same principle applies to maintaining design files, preparing regression tests, and other repetitive work that supports engineering.

These tasks are necessary, but much of the effort involves gathering context, following an existing structure, or repeating a process. AI can reduce that effort and give developers more time for difficult design decisions, failure analysis, and careful review.

Regression testing illustrates both the opportunity and the responsibility. AI can help draft tests and identify additional scenarios. Engineers still need to decide what correct behavior means and whether the tests would detect a meaningful failure. A test that simply repeats an implementation's assumptions provides little independent assurance.

The value comes from improving the amount of useful work a team can complete while preserving accountability for the result. Lines of generated code are a poor measure of that value. Clearer documentation, more relevant test coverage, faster investigation, and consistent implementations are much more useful outcomes.

Manage the risk. Deliver a better product.

Responsible use of AI starts with understanding the consequences of a mistake and setting the level of autonomy accordingly. Our three rings make that principle practical: human-led development at the core, closely directed AI assistance where data is at risk, and bounded agentic development where failures are isolated and quickly recoverable. Engineers remain accountable in every ring.

Within those boundaries, AI gives us more capacity to improve IoT Bridge. Offloading repeatable work creates time to build useful functionality, investigate difficult problems, strengthen regression coverage, and examine reliability and security more closely. Following established code patterns and keeping documentation current helps turn that additional capacity into improvements customers can use and trust.

The goal is better functionality, greater reliability, and stronger security for our customers. AI contributes to that goal when its output is supported by sound architecture, independent validation, and human judgment. Faster development creates an opportunity; disciplined engineering determines whether customers benefit from it.

That is how we intend to keep moving IoT Bridge forward: take advantage of new tools, manage their risks deliberately, and keep the quality of the customer’s running system at the center of every decision.

Talk to Tartabit about IoT Bridge and the engineering behind your IoT data infrastructure.