By Miroslav Jeras, CTO, Pebble
A lot of software development is about things users will never see, and in many ways that is exactly how it should be.
Pebble Remote provides a web-based interface for monitoring, managing and controlling playout environments, often across multiple channels and sites. For the people using it, the important thing is that it is there when they need it and continues to behave as expected, however long it has been running.
That makes maintaining it rather different from working on a typical web application. A browser application used occasionally can usually be restarted without much consequence. Software sitting in a broadcast control environment may be open continuously for days or weeks, receiving live information throughout that time. Small problems that would barely register elsewhere can therefore become significant.
Designing for continuous operation
Much of our recent engineering work on Remote has focused on exactly those kinds of issues. One example is memory management. Over time, we found several cases where subscriptions to real-time data, or the event listeners associated with them, could remain active after the part of the application that created them was no longer needed.
The individual issues could be fixed, of course, but seeing the same sort of behaviour in more than one area told us that there was a broader question to address. Rather than continue dealing with each instance separately, we looked again at how subscriptions are created, tracked and closed across the application.
That is an important distinction when developing software for continuous operation. Sometimes fixing the fault in front of you is enough; at other times, the fault is an indication that the underlying approach needs attention.
Remote depends heavily on live data, since operators need to see what is happening across their channels as it happens. The application therefore maintains ongoing subscriptions to automation and channel events instead of simply checking for new information every so often.
In practice, that brings its own complications. A connection can still appear to be active even though updates have stopped arriving; a component can subscribe repeatedly when it should not; or a subscription that is no longer required may continue consuming resources in the background.
These are not always dramatic failures, and that is part of the difficulty. If an application stops completely, it is obvious that something has gone wrong. If it remains open but is no longer showing current information, the problem can be much harder to spot.
As a result, we have spent time improving not only the way these connections are managed, but also the feedback provided when something does go wrong. If a live feed is no longer behaving as expected, the operator needs to know about it rather than being left with an interface that appears to be working normally. The result is not necessarily a change an operator will immediately notice, but a more consistent approach to managing the live connections that Remote relies on, and clearer feedback when those connections are interrupted.
Keeping security current
Security work follows much the same principle. It is part of maintaining software, not something that can be completed once and then left alone.
The technologies underneath an application continue to change. Supported versions move on, libraries are updated and new vulnerabilities are identified. Keeping software current therefore means reviewing those dependencies regularly rather than waiting until they become a larger problem.
Over the past year, that has meant upgrading core components used by Remote, including Node.js, database libraries and container components. We have also reviewed HTTPS and certificate handling, the management of sensitive configuration information, WebSocket communications and installer permissions.
There is always a balance to be struck here, because engineering time spent updating a library or changing the way a background process works is time that cannot be spent adding something new to the interface. Much of that work may never be particularly visible to customers either.
For software used in a live broadcast environment, though, keeping the underlying technology current is part of maintaining the product. Addressing these areas continuously gives us a more secure and maintainable foundation, while allowing that work to build up simply creates a much bigger job later.
Looking beyond the individual fix
One of the more useful lessons from this work has been the importance of looking beyond individual faults. If the same type of issue appears in several places, it is worth asking whether there is a better way of handling that area consistently across the software rather than fixing each case as it appears.
That thinking has influenced the way we now approach subscriptions, permissions, credentials and application behaviour when data is interrupted or unavailable. It is less visible than adding a new feature, but it makes the software easier to maintain and gives us a stronger base for future development.
For the operator, none of these internal details should require much thought. They have channels to run and decisions to make, and the software needs to support them without becoming another thing they have to worry about.
A significant amount of development work will always happen below the surface as a result. It may not produce the most obvious changes from one release to the next, but for software designed to sit at the heart of a 24/7 broadcast operation, it is part of what keeps the system dependable over time.
### ENDS ###
