In order to allow external access to our immersive video streams it is necessary to provide a web proxy to them that can manage discovery and security issues as well as allowing access through firewalls. The internal C++ applications that are doing the reconstruction work use a network protocol that can also operate over Web-Sockets to connect with, for example, Node.js. The protocol is based around MsgPack and uses an RPC style approach with messages.
The web-service would provide a way of listing what streams are available as well as allowing the creation of new virtual cameras, or the sharing of existing virtual cameras, that can be controlled by the user. Eventually it is expected that scenes from multiple locations can be merged together, producing a combined virtual view from multiple source views. Such mixing would likely require specific computing resources, all of which would need to be managed by this web-service or at least can be controlled via the web-service.
The service would need to provide at least the following:
- Login and security
- Snapshot thumbnails of available streams
- Video or jpeg feeds of streams
- Control of camera positions by clients
- Configuration of new virtual cameras in specific scenes
- Listing and search of cameras and scenes
- Groups of cameras, public and private
- Statistical information about scenes + cameras
- Access to overlay or additional data associated with feeds
- Automatic camera control options
- Sharing of cameras and scene configurations by users with other groups of users
More advanced functionality to add eventually: Composite / mix multiple scenes together. To achieve this it has to be possible to perform filtering/clipping as well as merge operations on the different sources. This requires a sophisticated interface and configuration system that is comprehensible to users, meaning that a great deal of automation is required. Exactly what is exposed via the web-service for this functionality is unclear currently. Dependency / data-flow / graph-based approaches are common in top end video processing and modelling tools, we would need something similar.
Another advanced feature is the ability to add virtual content. Similar to above, this has to be mixed into a scene but it must also be simple to create such content via the web interface.