dk1874_05.pdf

Parallel Architectures for

Programmable Video Signal

Processing

Zhao Wu and Wayne Wolf

Princeton University, Princeton, New Jersey

1 INTRODUCTION

Modern digital video applications, ranging from video compression to content

analysis, require both high computation rates and the ability to run a variety of

complex algorithms. As a result, many groups have developed programmable

architectures tuned for video applications. There have been four solutions to this

problem so far: modiﬁcations of existing microprocessor architectures, applica-

tion-speciﬁc architectures, fully programmable video signal processors (VSPs),

and hybrid systems with reconﬁgurable hardware. Each approach has both advan-

tages and disadvantages. They target the market from different perspectives. In-

struction set extensions are motivated by the desire to speed up video signal

processing (and other multimedia applications) by software solely rather than

by special-purpose hardware. Application-speciﬁc architectures are designed to

implement one or a few applications (e.g., MPEG-2 decoding). Programmable

VSPs are architectures designed from the ground up for multiple video applica-

tions and may not perform well on traditional computer applications. Finally,

reconﬁgurable systems intend to achieve high performance while maintaining

ﬂexibility.

Generally speaking, video signal processing covers a wide range of applica-

tions from simple digital ﬁltering through complex algorithms such as object

recognition. In this survey, we focus on advanced digital architectures, which are

intended for higher-end video applications. Although we cannot address every

possible video-related design, we cover major examples of video architectures

that illustrate the major axes of the design space. We try to enumerate all the

cutting-edge companies and their products, but some companies did not provide

much detail (e.g., chip architecture, performance, etc.) about their products,so

we do not have complete knowledge about some Integrated circuits (ICs) and

systems. Originally, we intended to study only the IC chips for video signal pro-

cessing, but reconﬁgurable systems also emerge as a unique solution, so we think

it is worth mentioning these systems as well.

The next section introduces some basic concepts in video processing algo-

rithms, followed by an early history of VSPs in Section 3. This is just to serve

as a brief introduction of the rapidly evolving industry. Beginning in Section 6,

we discuss instruction set extensions of modern microprocessors. In Section 5,

we compare the existing architectures of some dedicated video codecs. Then, in

Section 6, we contrast in detail and analyze the pros and cons of several program-

mable VSPs. In Section 7, we introduce systems based on reconﬁgurable comput-

ing, which is another interesting approach for video signal processing. Finally,

conclusions are drawn in Section 8.

2 BACKGROUND

Although we cannot provide a comprehensive introduction to video processing

algorithms here, we can introduce a few terms and concepts to motivate the archi-

tectural features found in video processing chips. Video compression was an early

motivating application for video processing; today, there is increased interest in

video analysis.

The Motion Pictures Experts Group (MPEG) ( www.cselt.it) has been con-

tinuously developing standards for video compression. MPEG-1, -2, and -4 are

complete, and at this writing, work on MPEG-7 is underway. We refer the reader

to the MPEG website for details on MPEG-1 and -2 and to a special issue of

IEEE Transactions on Circuits and Systems for Video Technology for a special

issue on MPEG-4. The MPEG standards apply several different techniques for

video compression. One technique, which was also used for image compression

in the JPEG standard (JPEG book) is coding using the discrete cosine transform

(DCT). The DCT is a frequency transform which is used to transform an array

of pixels (an 8

8 array in MPEG and JPEG) into a spatial frequency spectrum;

the two-dimensional DCT for the 2D array can be found by computing two 1D

DCTs on the blocks. Specialized algorithms have been developed for computing

the DCT efﬁciently. Once the DCT is computed, lossy compression algorithms

will throw away coefﬁcients which represent high-spatial frequencies, because

those represent ﬁne details which are harder to resolve by the human eye, particu-

larly in moving objects. DCT is one of the two most computation-intensive opera-

tions in MPEG.

The other expensive operation in MPEG-style compression is block motion

estimation. Motion estimation is used to encode one frame in terms of another

(DCT is used to compress data within a single frame). As shown in Figure 1 , in

MPEG-1 and -2, a macroblock (a 16

1). The

reference macroblock is compared to the selected macroblock by two-dimen-

sional correlation: Corresponding pixels are compared and the sum of the magni-

tudes of the differences is computed. If the selected macroblock can be matched

within a given tolerance, in the other frame, then the macroblock need be sent

only once for both frames. A region around the macroblock’s original position

is chosen as the search area in the other frame; several algorithms exist which

avoid performing the correlation at every offset within the search region. The

macroblock is given a motion vector that describes its position in the new frame

relative to its original position. Because matches are not, in general, exact, a

difference pattern is sent to describe the corrections made after applying the mac-

roblock in the new context.

MPEG-1 and -2 provide three major types of frames. The I-frame is coded

without motion estimation. DCT is used to compress blocks, but a lossily com-

pressed version of the entire frame is encoded in the MPEG bit stream. A P-

2 p

Figure 1 Block motion estimation.

16 array of pixels composed of four

blocks) taken from one frame is correlated within a distance p of the macroblock’s

current position (giving a total search window of size 2 p

frame is predicted using motion estimation. A P-frame is encoded relative to an

earlier I-frame. If a sufﬁciently good macroblock can be found from the I-frame,

then a motion vector is sent rather than the macroblock itself; if no match is

found, the DCT-compressed macroblock is sent. A B-frame is bidirectionally

encoded using motion estimation from frames both before and after the frame

in time (frames are buffered in memory to allow bidirectional motion prediction).

MPEG-4 introduces methods for describing and working with objects in the video

stream. Other detailed information about the compression algorithm can be found

in the MPEG standard [1].

Wavelet-based algorithms have been advocated as an alternative to block-

based motion estimation. Wavelet analysis uses ﬁlter banks to perform a hierar-

chical frequency decomposition of the entire image. As a result, wavelet-based

programs have somewhat different characteristics than block-based algorithms.

Content analysis of video tries to extract useful information from video

frames. The results of content analysis can be used either to search a video data-

base or to provide summaries that can be viewed by humans. Applications include

video libraries and surveillance. For example, algorithms may be used to extract

key frames from videos. The May and June 1998 issues of the Proceedings of

the IEEE and the March 1998 issue of IEEE Signal Processing Magazine survey

multimedia computing and signal processing algorithms.

3 EARLY HISTORY OF VLSI VIDEO PROCESSING

An early programmable VSP was the Texas Instruments TMS34010 graphics

system processor (GSP) [2]. This chip was released in 1986. It is a 32-bit micro-

processor optimized for graphics display systems. It supports various pixel for-

mats (1-, 2-, 4-, 8-, and 16-bit) and operations and can accelerate graphics inter-

face efﬁciently. The processor operates at a clock speed from 40 to 60 MHz,

achieving a peak performance of 7.6 million instructions per second (MIPS).

Philips Semiconductors developed early dedicated video chips for special-

ized video processors. Philips announced two digital multistandard color de-

coders at almost the same time. Both the SAA9051 [3] and the SAA7151 [4]

integrate a luminance processor and chrominance processor on-chip and are able

to separate 8-bit luminance and 8-bit chrominance from digitized S-Video or

composite video sources as well as generate all the synchronization and control

signals. Both VSPs support PAL, NTSC, and SECAM standards.

In the early days of JPEG development, its computational kernels could

not be implemented in real time on typical CPUs, so dedicated DCT/IDCT (dis-

crete cosine transform–inverse DCT) units, Huffman encoder/decoder, were built

to form a multichip JPEG codec [another solution was multiple digital signal

processors (DSPs)]. Soon, the multiple modules could be integrated onto a single

chip. Then, people began to think about real-time MPEG. Although MPEG-1

decoders were only a little more complicated than JPEG decoders, MPEG-1 en-

coders were much more difﬁcult. At the beginning, encoders that are fully compli-

ant to MPEG-1 standards could not be built. Instead, people had to come up

with some compromise solutions. First, motion-JPEG or I-frame-only (where the

motion estimation part of the standard is completely dropped) encoders were

designed. Later, forward prediction frames were added in IP-frame encoders.

Finally, bidirectional prediction frames were implemented. The development also

went through a whole procedure from multichip to singlechip. Meanwhile, the

microprocessors became so powerful that some software MPEG-1 players could

support real-time playback of small images. The story of MPEG-2 was very simi-

lar to MPEG-1 and began as soon as the ﬁrst single-chip MPEG-1 decoder was

born. Like MPEG-1, it also experienced asymptotic approaches from simpliﬁed

standards to fully compliant versions, and from multichip solutions to single chip

solutions.

The late 1980s and early 1990s saw the announcement of several complex,

programmable VSPs. Important examples include chips from Matsushita [5],

NTT [6], Philips [7], and NEC [8]. All of these processors were high-performance

parallel processors architected from the ground up for real-time video signal pro-

cessing. In some cases, these chips were designed as showcase chips to display

the capabilities of submicron very-large-scale integration (VLSI) fabrication pro-

cesses. As a result, their architectural features were, in some cases, chosen for

their ability to demonstrate a high clock rate rather than their effectiveness for

video processing. The Philips VSP-1 and NEC processor were probably the most

heavily used of these chips.

The software (compression standards, algorithms, etc.) and hardware (in-

struction set extensions, dedicated codecs, programmable VSPs) developments

of video signal processing are in parallel and rely heavily on each other. On one

hand, no algorithms could be realized without hardware support; on the other

hand, it is the software that makes a processor useful. Modern VLSI technology

not only makes possible but also encourages the development of coding algo-

rithms—had developers not been able to implement MPEG-1 in hardware, it may

not have become popular enough to inspire the creation of MPEG-2.

4 INSTRUCTION SET EXTENSIONS FOR VIDEO

SIGNAL PROCESSING

The idea of providing special instructions for graphics rendering in a general-

purpose processor is not new; it appeared as early as 1989 when Intel introduced

i860, which has instructions for Z-buffer checks [9]. Motorola’s 88110 is another

example of using special parallel instructions to handle multiple pixel data simul-

Plik z chomika:

Inne pliki z tego folderu:

Inne foldery tego chomika: