New perspectives in video compression are reinforced by recent advances in scal- able video coding. MCTF-based coders provide high flexibility in bit stream scal- ability across different temporal, spatial, and quality resolutions. In addition, they provide better error resilience than conventional (prediction based) coders. In fact, MCTF-based coders are better able to separate relevant from irrelevant informa- tion. The temporal low-pass bands highlight information that is consistent over a large number of frames, establishing a powerful means for exploiting multiple frame redundancies not achievable by conventional frame-to-frame or multiframe
prediction methods. Moreover, noise and quickly changing information that can- not be handled by motion compensation appear in the temporal, high-pass bands, which can supplement the low-pass bands for more accurate signal reproduction whenever desirable, provided that a sufficient data rate is available. Hence, the de- noising process that is often applied as a preprocessing step before conventional video compression is an integral part of scalable MCTF-based coders.
Due to the nonrecursive structure, higher degrees of freedom are possible for both encoder and decoder optimization. In principle, a decoder could integrate additional signal synthesis elements whenever the received information is incom- plete, such as frame-rate up-conversion, film grain noise overlay, or other ele- ments of texture and motion synthesis, which could easily be integrated as a part of the MCTF synthesis process without losing any synchronization between en- coder and decoder. From this point of view, even though many elements of MCTF in the lifting interpretation can be regarded as extensions of proven techniques from MC prediction-based coders, this framework exhibits and enables a number of radically new options in video encoding. However, when a wavelet transform is applied for encoding of the low-pass and high-pass frames resulting from the MCTF process, the commonalities with 2D wavelet coding methods are obvi- ous. If the sequence of spatial and temporal filtering is exchanged (2D+t in- stead oft+2D wavelet transform), MCTF can be interpreted as a framework for further interframe compression of (intra frame restricted) 2D wavelet representa- tions such as JPEG 2000. From this point of view, a link between the previously separate worlds of 2D wavelet coding with their excellent scalability properties and compression-efficient motion-compensated video coding schemes is estab- lished by MCTF. This shows the high potential for future developments in the area of motion picture compression, even allowing seamless transition between intra frame and inter frame coding methods, depending on the application re- quirements for flexible random access, scalability, high compression, and error resilience. Furthermore, scalable protection of content, allowing access manage- ment for different resolution qualities of video signals, is a natural companion of scalable compression methods.
Nevertheless, a number of topics can be identified that still require further re- search, but may also lead to even higher compression performance of this new class of video coding algorithms. These include
• Strategies for motion estimation and motion vector encoding, including consideration of prediction and update steps, bidirectional prediction, and update filtering, as well as combined estimation over different levels of the temporal wavelet tree.
• Application and optimization of nonblock-based motion compensation, which is more natural used in combination with spatial wavelet decom- position.
• Scalability of motion information.
• Optimum adaptation of the spatial/temporal decomposition trees, including consideration of integrated solutions of spatial/temporal filtering.
• Optimization of spatial/temporal encoding, including psychovisual proper- ties.
• Rate–distortion optimum truncation of scalable streams, including the trade-offs at various rates.
• Creation of complexity-scalable video coding bit streams.
BIBLIOGRAPHY
[1] G. Karlsson and M. Vetterli. “Subband coding of video signals for packet switched networks,” Proc. Visual Commun. Image Process., SPIE vol. 845, pp. 446–456, 1987.
[2] T. Kronander. “Some aspects of perception based image coding,” Ph.D. Thesis, Linkửping University, 1989.
[3] J.-R. Ohm. “Three-dimensional subband coding with motion compensation,” IEEE Trans. Image Processing, no. 3, pp. 559–571, 1994.
[4] S.-J. Choi and J. W. Woods. “Motion compensated 3D subband coding of video,”
IEEE Trans. Image Processing, vol. 8, pp. 155–167, 1999.
[5] B. Pesquet-Popescu and V. Bottreau. “Three-dimensional lifting schemes for motion- compensated video compression,” Proc. IEEE ICASSP, pp. 1793–1796, Salt Lake City, USA, May 2001.
[6] H. Radha, M. van der Schaar, and Y. Chen. “The MPEG-4 Fine-Grained Scalable Video Coding Method for Multimedia Streaming over IP,” IEEE Trans. on Multime- dia, vol. 3, Issue 1, pp. 53–68, March 2001.
[7] W. Li. “Overview of Fine Granularity Scalability in MPEG-4 video standard,” IEEE Trans. Circuits and Systems for Video Technology, vol. 11, no. 3, pp. 301–317, 2001.
[8] U. Horn, K. W. Stuhlmüller, M. Link, and B. Girod. “Robust internet video trans- mission based on scalable coding and unequal error protection,” Signal Processing:
Image Commun., September 1999.
[9] K. Rose and S. L. Regunathan. “Toward optimality in scalable predictive coding,”
IEEE Trans. Image Processing, vol. 10, pp. 965–976, July 2001.
[10] M. van der Schaar and H. Radha. “Adaptive Motion-Compensation Fine-Granularity- Scalability (AMC-FGS) for Wireless Video,” IEEE Trans. on Circuits and Systems for Video Technology, June 2002.
[11] M. van der Schaar and D. S. Turaga. “Unconstrained motion compensated temporal filtering framework for wavelet video coding,” Proc. IEEE ICASSP, May 2003.
[12] F. Wu, S. Li, and Y.-Q. Zhang. “DCT-prediction based progressive fine granularity scalability coding,” Proc. IEEE ICIP 2000, Vancouver, Canada, vol. 3, pp. 556–559, September 10–13, 2000.
[13] F. Wu, S. Li, and Y.-Q. Zhang. “A framework for efficient progressive fine granular- ity scalable video coding,” IEEE Trans. Circuits and Systems for Video Technology, special issue on streaming video, vol. 11, no. 3, pp. 332–344, 2001.
[14] H. Huang, C. Wang, and T. Chiang. “A robust fine granularity scalability using trellis- based predictive leak,” IEEE Trans. Circuits and Systems for Video Technology, vol.
12, no. 6, pp. 372–385, 2002.
[15] A. Fuldseth and T. A. Ramstad. “Robust subband video coding with leaky predic- tion,” in Proc. DSP Workshop, Loen, Norway, pp. 57–60, September 1996.
[16] Y. Gao and L.-H. Chau. “An Efficient Fine Granularity Scalable Coding Scheme Us- ing Adaptive Leaky Prediction,” Fourth International Conference on Information, Communications & Signal Processing and Fourth Pacific-Rim Conference on Multi- media (ICICS-PCM 2003), Singapore, December 2003.
[17] X. Sun, F. Wu, S. Li, W. Gao, and Y.-Q. Zhang. “Macroblock-based progressive fine granularity scalable video coding,” Proc. IEEE ICME, Japan, August 2001.
[18] Feng Wu, Shipeng Li, Xiaoyan Sun, Bing Zeng, and Ya-Qin Zhang. “Macroblock- based progressive fine granularity scalable video coding,” International Journal of Imaging Systems and Technology, vol. 13, Issue 6, pp. 297–307, 2003.
[19] W. H. Peng and Y.-K. Chen. “Mode-adaptive fine granularity scalability,” Proc. IEEE ICIP, pp. 993–996, Greece, 2001.
[20] W.-H. Peng and Y.-K. Chen. “Enhanced Mode-Adaptive Fine Granularity Scala- bility,” International Journal of Imaging Systems and Technology, vol. 13, no. 6, pp. 308–321, March 2004.
[21] L. Luo, J. Li, S. Li, Z. Zhuang, and Y.-Q. Zhang. “Motion compensated lifting wavelet and its application in video coding,” Proc. IEEE ICME, Tokyo, Japan, August 2001.
[22] D. Turaga, M. van der Schaar, and B. Pesquet. “Complexity Scalable Motion Com- pensated Wavelet Video Encoding,” IEEE Trans. on Circuits and Systems for Video Technology, vol. 15, no. 8, pp. 982–993, August 2005.
[23] J. Barbarien, Y. Andreopoulos, A. Munteanu, P. Schelkens, and J. Cornelis. “Coding of motion vectors produced by wavelet-domain motion estimation,” Proc. PCS 2003, pp. 193–198, April 2003.
[24] C. Tillier and B. Pesquet-Popescu. “3D, 3-Band, 3-Tap Temporal Lifting for Scal- able Video Coding,” Proc. IEEE ICIP 2003, pp. 14–17, Barcelona, Spain, September 2003.
[25] C. Bergeron, C. Lamy-Bergot, G. Pau, and B. Pesquet-Popescu. “Temporal scalability through adaptive M-band filterbanks for robust H264/MPEG-4 AVC coding,” Special issue on Video Analysis and Coding for Robust Transmission. JASP, 2006.
[26] C. Tillier, B. Pesquet-Popescu, and M. van der Schaar. “3-Band Temporal Struc- tures for Scalable Video Coding,” IEEE Trans. on Image Processing, vol. 15, Issue 9, pp. 2545–2557, September 2006.
[27] G. Pau and B. Pesquet-Popescu. “Comparison of Spatial M-Band Filter Banks for t+2D Video Coding,” Proc. of SPIE/IEEE VCIP 2005, Beijing, China, July 2005.
[28] G. Pau, B. Pesquet-Popescu, and G. Piella. “Modified M-Band Synthesis Filter Bank for Fractional Scalability of Images,” IEEE Signal Processing Letters, vol. 13, Issue 6, pp. 345–348, June 2006.
[29] M. Trocan, C. Tillier, and B. Pesquet-Popescu. “Joint wavelet packets for group of frames in MCTF,” Proc. SPIE, San Diego, USA, July 2005.
[30] A. Secker and D. Taubman. “Highly Scalable Video Compression with Scalable Mo- tion Coding,” IEEE Trans. on Image Processing, vol. 13, no. 8, pp. 1029–1041, Au- gust 2004.
[31] R. Xiong, F. Wu, J. Xu, S. Li, and Y.-Q. Zhang. “Barbell lifting wavelet transform for highly scalable video coding,” Proc. Picture Coding Symposium 2004, San Francisco, CA, USA, December 2004.
[32] Bo Feng, Jizheng Xu, Feng Wu, Shiqiang Yang, and Shipeng Li. “Energy distributed update steps (EDU) in lifting based motion compensated video coding,” Proc. IEEE ICIP 2004, Singapore, October 2004.
[33] A. Said and W. A. Pearlman. “A new fast and efficient image codec based on set partitioning in hierarchical trees,” IEEE Trans. on Circ. and Systems for Video Techn., vol. 6, pp. 243–250, June 1996.
[34] D. Taubman. “High performance scalable image compression with EBCOT,” IEEE Trans. on Image Processing, no. 9, pp. 1158–1170, 2000.
[35] S.-T. Hsiang and J. W. Woods. “Embedded image coding using zeroblocks of sub- band/wavelet coefficients and context modeling,” MPEG-4 Workshop and Exhibition at ISCAS 2000, Geneva, Switzerland, May 2000.
[36] J. Xu, S. Li, and Y.-Q. Zhang. “A wavelet video coder using three dimensional em- bedded subband coding with optimized truncation (3D ESCOT),” IEEE-PCM 2000, Sydney, December 2000.
[37] J. Xu, Z. Xiong, S. Li, and Y.-Q. Zhang. “3D embedded subband coding with optimal truncation (3D-ESCOT),” Applied and Computational Harmonic Analysis 10, p. 589, May 2001.
[38] S.-T. Hsiang and J. W. Woods. “Embedded video coding using invertible motion compensated 3D subband/wavelet filter bank,” Signal Processing: Image Commun., vol. 16, pp. 705–724, May 2001.
[39] S.-T. Hsiang, J. Woods, and J.-R. Ohm. “Invertible temporal subband/wavelet fil- ter banks with half-pixel accurate motion compensation,” IEEE Trans. on Image Processing 13, pp. 1018–1028, August 2004.
[40] M. van der Schaar and Y. Andreopoulos. “Rate-distortion-complexity modeling for network and receiver aware adaptation,” IEEE Trans. on Multimedia, June 2005.
[41] Y. Wang, M. van der Schaar, S. F. Chang, and A. Loui. “Content-Based Optimal MDA Operation Prediction for Scalable Video Coding Systems Using Subjective Quality Evaluation,” IEEE Trans. on Circuits and Systems for Video Technology—Special issue on Analysis and Understanding for Video Adaptation, October 2005.
[42] J. R. Ohm, M. van der Schaar, and J. Woods. “Interframe wavelet coding: Motion Picture Representation for Universal Scalability,” EURASIP Signal Processing: Im- age Communication, Special issue on Digital Cinema, 2004.
[43] D. Turaga, M. van der Schaar, Y. Andreopoulos, A. Munteanu, and P. Schelkens.
“Unconstrained motion compensated temporal filtering (UMCTF) for efficient and flexible interframe wavelet video coding,” EURASIP Signal Processing: Image Com- munication, 2004.
[44] M. van der Schaar and H. Radha. “A hybrid temporal-SNR Fine-Granular Scalabil- ity for Internet Video,” IEEE Trans. on Circuits and Systems for Video Technology, March 2001.
[45] H. Radha, M. van der Schaar, and S. Karande. “Scalable Video TranScaling for the Wireless Internet,” EURASIP Journal of Applied Signal Processing (JASP)—Special issue on Multimedia over IP and Wireless Networks, No. 2, pp. 265–279, 2004.
[46] Y. Andreopoulos, M. van der Schaar, A. Munteanu, J. Barbarien, P. Schelkens, and J. Cornelis. “Fully-Scalable Wavelet Video Coding using In-Band Motion- Compensated Temporal Filtering,” Proc. on IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2003.
[47] J. Ye and M. van der Schaar. “Fully Scalable 3-D Overcomplete Wavelet Video Cod- ing Using Adaptive Motion Compensated Temporal Filtering,” Proc. SPIE Video Communications and Image Processing (VCIP), 2003.
[48] Y. Andreopoulos, A. Munteanu, J. Barbarien, M. van der Schaar, J. Cornelis, and P. Schelkens. “In-band motion compensated temporal filtering,” EURASIP Signal Processing: Image Communication (special issue on “Subband/Wavelet Interframe Video Coding”), vol. 19, no. 7, pp. 653–673, August 2004.
6 Scalable Audio Coding
Jin Li