Joint Source/Channel Coding

Goal: Improved video transmission for very low delay applications (e.g. video telephony) in error prone environments.

A video signal that is compressed by a hybrid video coder like H.264/MPEG4-AVC is extremely vulnerable to transmission errors. Transmission errors can be reduced by appropriate channel coding techniques. For channels without memory, such as the AWGN channel, channel coding techniques provide very significant reductions of transmission errors at a comparably moderate bit rate overhead. For the mobile fading channel or the Internet, however, the effective use of forward error correction is limited when assuming a small end-to-end delay. Here the use of error resilience techniques in the source codec becomes important.

In Inter mode, i.e., when motion-compensated prediction is utilized, the loss of information in one frame has a considerable impact on the quality of the following frames. As a result, spatio-temporal error propagation is a typical transmission error effect for predictive coding. Because errors remain visible for a longer period of time, the resulting artifacts are particularly annoying to end users. To some extent the impairment caused by transmission errors decays over time due to leakage in the prediction loop. However, the leakage in standardized video decoders like H.264/MPEG4-AVC is not very strong, and quick recovery can only be achieved when image regions are encoded in Intra mode, i.e., without reference to a previous frame. The Intra mode, however, is not selected very frequently during normal encoding and completely Intra coded frames are not usually inserted in real-time encoded video as is done for storage or broadcast applications. Instead, only single macroblocks, i.e., parts of the picture, are encoded in Intra mode for regions that cannot be predicted efficiently.

The work on this subject started with motion estimation and mode decision with multiple reference pictures in error-prone environments [3], [12], [16]. The idea of estimating the distortion introduced by packet losses at the encoder and utilizing this estimate in the encoder control has been driven further and been proposed and accepted by VCEG during the development of H.26L. This estimate is obtained by running multiple versions of the decoder at the encoder side with each version simulating a realization of the random decoding process [1], [5]-[7], [13]-[15].

Impact of Transmission Errors

If just one part or an entire frame is missing => concealment at decoder side => reference pictures at coder and decoder differ => motion compensation => spatio-temporal error propagation
Spatio-temporally propagated errors decay slowly => mitigate error propagation
Mitigating error propagation
- Use of multiple reference pictures
- INTRA picture and INTRA macroblock refresh

Assignment of Intra Macroblock Coding

Intra coding provides lower coding efficiency than Inter coding => Trade-off between error resilience and coding efficiency
Random transmission errors cause that decoding result becomes random variable
Model decoding random variable using N sample functions
Optimize encoding operation for average decoding result for a given packet-loss rate p

Modified Coder Control

Given the N decoded versions with random packet losses of probabilityp at the encoder
Lagrangian Mode Decision

$$D_{2}(M|Q)+\lambda_{M}\cdot R(M|Q)$$

With modified distortion measure

Approach adaptively increases costs for Inter coding and therefore increases Intra coding rate.

Publications

Publications in Journals

2003

Thomas Stockhammer, Miska M. Hannuksela, and Thomas Wiegand:
H.264/AVC in Wireless Environments,
IEEE Transactions on Circuits and Systems for Video Technology, vol. 13, no. 7, pp. 657-673, July 2003.

Publications in Conference Proceedings

2003

Thomas Stockhammer, Tobias Oelbaum, Florian Obermeier, and Thomas Wiegand:
Video Coding and Transport Layer Techniques for H.264-based Transmission over Packet-Lossy Networks,
IEEE International Conference on Image Processing (ICIP'03), Barcelona, Spain, September 2003.

2002

Thomas Stockhammer and Thomas Wiegand:
Optimized Transmission of H.26L Video over Packet-Lossy Networks,
IEEE International Conference on Image Processing (ICIP'02), Rochester, NY, USA, September 2002.

2001

Gary J. Sullivan, Thomas Wiegand, and Thomas Stockhammer:
Using the Draft H.26L Video Coding Standard for Mobile Applications,
IEEE International Conference on Image Processing (ICIP'01), Thessaloniki, Greece, September 2001, Invited Paper.

Contributions to Standardization