... | ... | @@ -63,16 +63,16 @@ The prototype is to focus on tele-immersive video and interfaces required to ope |
|
|
# 2. OVERALL DESCRIPTION
|
|
|
The prototype will be a new platform that enables the capture, mixing, filtering, analysis, recording and streaming of teleimmersive 4D audio-visual data in a real-time, modular and flexible manner so that individual data sources or algorithms can be replaced or evolved in the future. The key features are therefore:
|
|
|
|
|
|
* **F1:** *Generation* - Scene reconstruction from fixed lab spaces as a 4D-AV data source
|
|
|
* **F2:** *Transformation* - 4D-AV mixing and filtering system, from any number of arbitrary data sources
|
|
|
* **F3:** *Analysis* - 4D-AV Analysis system to extract and embed higher-level data about a scene
|
|
|
* **F4:** *Recording* - 4D-AV recording and playback as a generic 4D-AV data source
|
|
|
* **F5:** *Streaming* - Lossless and lossy compression of 4D-AV for storage and streaming
|
|
|
* **F6:** *Presentation* - Interactive visualisation front-end(s) from a 4D-AV stream
|
|
|
* **F7:** *Control* - User configuration and alteration of the pipelines and components in C1-C6
|
|
|
* **F8:** *Representation* - Common 4D-AV data format for all components for flexibility
|
|
|
* **F1:** *Generation* - Scene reconstruction from fixed lab spaces as a volumetric data source
|
|
|
* **F2:** *Transformation* - Volumetric mixing and filtering system, from any number of arbitrary data sources
|
|
|
* **F3:** *Analysis* - Volumetric analysis system to extract and embed higher-level data about a scene
|
|
|
* **F4:** *Recording* - Volumetric recording and playback as a generic volumetric data source
|
|
|
* **F5:** *Streaming* - Lossless and lossy compression of volumetric data for storage and streaming
|
|
|
* **F6:** *Presentation* - Interactive visualisation front-end(s) from a volumetric stream
|
|
|
* **F7:** *Control* - User configuration and alteration of the pipelines and components in F1-F6
|
|
|
* **F8:** *Representation* - Common volumetric data format for all components for flexibility
|
|
|
|
|
|
Additional virtual data sources may also be considered to allow mixing of real and virtual content into 4D-AV scenes, therefore suggesting a requirement that the representation used be independent of, or flexible with respect to, source. The analysis outputs are also to be attached or embedded in some manner into the 4D-AV stream, perhaps as an addition data channel. For example, semantic information about objects such as classification or motion information. This analysis information may also be used for predictive purpose to resolve latency issues and help with both compression and initial source capture quality. To be used in this way the data must flow not only forward with the stream but also back to previous processing steps.
|
|
|
Additional virtual data sources may also be considered to allow mixing of real and virtual content into volumetric scenes, therefore suggesting a requirement that the representation used be independent of, or flexible with respect to, source. The analysis outputs are also to be attached or embedded in some manner into the volumetric stream, perhaps as an addition data channel. For example, semantic information about objects such as classification or motion information. This analysis information may also be used for predictive purpose to resolve latency issues and help with both compression and initial source capture quality. To be used in this way the data must flow not only forward with the stream but also back to previous processing steps.
|
|
|
|
|
|
Overall there must be a great deal of user control over the configuration of the platform to enable custom mixing, filtering, recording and presentation. However, the use cases presented in the next section can be used to limit the scope of flexibility.
|
|
|
|
... | ... | @@ -132,12 +132,12 @@ We will rely, at least initially, on OpenCV and the Point Cloud Library. It is a |
|
|
## 3.1 FUNCTIONAL REQUIREMENTS
|
|
|
|
|
|
### 3.1.1 Generation (F1)
|
|
|
* **3.1.1.1** Capture a 4D-AV scene in real-time from a fixed laboratory with multiple people
|
|
|
* **3.1.1.1** Capture a volumetric scene in real-time from a fixed laboratory with multiple people
|
|
|
* **3.1.1.2** Allow 4D scenes to be generated virtually from some scene description and animation
|
|
|
* **3.1.1.3** Support the translation of more traditional formats into our 4D-AV representation as a source
|
|
|
* **3.1.1.3** Support the translation of more traditional formats into our volumetric representation as a source
|
|
|
|
|
|
### 3.1.2 Transformation (F2)
|
|
|
* **3.1.2.1** Merge multiple 4D-AV scenes into a single scene, with no restriction on the origin of the scene
|
|
|
* **3.1.2.1** Merge multiple volumetric scenes into a single scene, with no restriction on the origin of the scene
|
|
|
* **3.1.2.2** Filter a scene to exclude all but a specified collection of entities
|
|
|
* **3.1.2.3** Filter a scene to exclude specific entities
|
|
|
* **3.1.2.4** Filter a scene by bounded region
|
... | ... | @@ -154,12 +154,12 @@ We will rely, at least initially, on OpenCV and the Point Cloud Library. It is a |
|
|
* **3.1.3.4** Support motion analysis for gestures and body language especially
|
|
|
|
|
|
### 3.1.4 Recording (F4)
|
|
|
* **3.1.4.1** Allow a 4D-AV scene to be saved to disk
|
|
|
* **3.1.4.2** Allow a 4D-AV scene to be loaded and replayed from disk
|
|
|
* **3.1.4.1** Allow a volumetric scene to be saved to disk
|
|
|
* **3.1.4.2** Allow a volumetric scene to be loaded and replayed from disk
|
|
|
|
|
|
### 3.1.5 Streaming (F5)
|
|
|
* **3.1.5.1** Support compressed adaptive bitrate streaming of 4D-AV scenes
|
|
|
* **3.1.5.2** Support lossless compression of 4D-AV scenes
|
|
|
* **3.1.5.1** Support compressed adaptive bitrate streaming of volumetric scenes
|
|
|
* **3.1.5.2** Support lossless compression of volumetric scenes
|
|
|
* **3.1.5.3** Streams can contain non-AV data streams for additional forms of basic or higher-level data
|
|
|
* **3.1.5.4** Allow for channel selection or exclusion from the data stream
|
|
|
* **3.1.5.5** Stream decoding must function within a web-browser
|
... | ... | @@ -177,6 +177,8 @@ We will rely, at least initially, on OpenCV and the Point Cloud Library. It is a |
|
|
* **3.1.8.4** Must allow a wide range of properties to be kept about spatio-temporal locations
|
|
|
* **3.1.8.5** Must allow for higher-level representations of properties in addition to raw data
|
|
|
* **3.1.8.6** Follow a relational paradigm, not an object-oriented one
|
|
|
* **3.1.8.7** Should be some derivative of a point cloud and not a mesh or image-based
|
|
|
* **3.1.8.8** Consider the use of or support for Function-Representation (F-Rep)
|
|
|
|
|
|
### 3.1.9 Miscellaneous
|
|
|
|
... | ... | |