PROVISION ITN

The PROVISION ITN targeted the important area of perceptually-oriented video coding and particularly focused on how to deliver higher volumes of more immersive content over congested band-limited channels. PROVISION delivered world-leading technical innovation through a unique highly integrated training programme, producing skilled staff, efficient and validated solutions and impact through standardisation. Only through interdisciplinary collaboration across academia and industry, could the required transformational research be carried out. The projects objectives were to:

1. provide the highest levels of technical and transferable skills training to the researcher team;
2. conduct training in the context of an international research collaboration between some of the leading universities, research institutes, companies and end-user enterprises in Europe;
3. complement the PROVISION consortium with international experts as associated partners, exposing the young scientists to cultural and organisational variations in research;
4. enhance the PROVISION consortium by internationally respected visiting scientists from the public and the private sector to guarantee optimal conditions for all its researchers (ESR and ER). In the project, five Full Partners were involved: Fraunhofer HHI (DE), University of Bristol (UK), BBC (UK), RWTH (DE) and University of Nantes (FR). Also, 6 Associated Partners – YouTube (US), Netflix (US), Technicolor (FR), VITEC (FR), Queen Marry University (UK) and TU Berlin (DE) participated in PROVISION, which offered secondments and technical guidance to the project fellows.

To achieve the project goals and objectives, 18 fellows participated in the project. The fellows achieved significant technological progress within their research teams, as well as during the individual secondments. Here, scientific advancement in the area of perceptual video coding were achieved through the fellows’ individual work, which was structured along 5 scientific work packages.

The technological highlights in each work package included:

• identification of shortcoming in current video codecs and measurement tool contribution to international standardisation in WP1,
• optical-flow-based motion modelling for irregular textures and 3D structural reconstruction for prediction in WP2,
• three-cluster texture model with adapted coding, new content synthesis methods through template matching and subsampling in WP3,
• visual attention models for 2D and 360° video, development of perceptually optimized intra coding, new visual disruption quality metric and non-reference VQA models in WP4,
• successful core validation through oral/poster presentations in international events, demonstrations, subjective assessment, feedback by international experts and joint topic-based working groups in WP5.

The novelty of the scientific work has been proven through a number of scientific publications, including more than 50 papers at high-level international conferences, such as ICIP, ICME, ICASSP, MMSP, PCS, EUSIPCO, QoMEX and international journals, such as Electronic Imaging. Several publications were honoured with Best Paper Awards. Furthermore, the fellows have provided more than 15 contributions to standard and are co-inventor of 4 patents. In addition to these activities, a novel type of activity, Grand Challenges, also took place. Here, PROVISION submitted two proposals in 2017 and PROVISION was offered to organize both. Each fellow significantly contributed to the scientific community in image processing and video coding, with the most advanced research topic within PROVISION. In detail, this included the creation and definition of test sets, frameworks and core experiments, research on rate-distortion-curve prediction from uncompressed content and improved reference picture list sorting and signalling. Furthermore, methods for fast template matching for intra prediction, multiple transform residual coding and variable bitrate rate control provided higher compression efficiency in comparison to previous state of the art video codecs. Special emphasis on perception in video coding was given through the research topics on Low-complexity spatio-temporal adaptation for video compression, video texture analysis and classification, perceptual RDO augmentation and just-noticeable-difference optimization and intra prediction. Detailed work on optimization for motion information in video coding has been done through fellows by fine detail reconstruction for B pictures, motion-based texture synthesis for 2D video content, as well as for 3D content by motion compensation using estimated 3D-Models. One main topic was the inclusion of perceptual metrics and attention modelling in video coding. Here, research topics on short-term perceptual fidelity metric for dynamic texture coding, perceptual modelling of temporal structural artefacts, image feature characterization of lightfield content and video chain optimization using content characteristics were carried out. In addition to video coding, the last work item on video chain optimization also targeted transmission optimization. Each of these individual fellow’s project advanced the previous state of the art. This could be achieved through PROVISION training activities as a central component in the career development of its researchers, enabling them to enhance existing and develop new skills, both related to technical and complementary topics. This included training on different video coding architectures, challenges of commercializing video technology, overview, perspectives and principles in standardisation, as well as on vitae career development, pitching ideas and presentation skills, video making sessions, technical project management, patents and IPR in video compression, writing technical research papers and PhD planning and organization, start-ups and business plans, CV production and interview techniques. All Fellows gained experience in presenting their work during work package updates and engaged in discussions on progress and planning. Fellows finally gained significant presentation experience through the PROVISION Public Workshop (Berlin) and Summer School on Video Compression and Processing (SVCP) (Kerkrade).

Through their scientific achievements and presentations to the scientific community at respective conferences and standardization meetings, as well as the wider audience at more general events and exhibitions, the fellows highly successfully promoted PROVISION technology beyond the international community in image processing. After the end of their PROVISION engagement, all fellows successfully secured their future technical careers with continuing employments at the full partner’s premises, at new employments at technical companies in the area of video compression or at an own start-up.