Introduction to Imaging and Multimedia 9/5/13

Introduction to Imaging and Multimedia

Fall 2013
Ahmed Elgammal
Rutgers University

  Introduction to Imaging and Multimedia

  This course has two integrated components:
  Digital Multimedia
  Image Processing and Computer Vision
  You can think of it as an introduction to
multimedia with a special focus on digital imaging
and video

A. Elgammal, Rutgers 1
Introduction to Imaging and Multimedia 9/5/13

  What is Multimedia ?

What is Multimedia
  When different people mention the term multimedia, they often
have quite different, or even opposing, viewpoints.
  A PC vendor: a PC that has sound capability, a DVD-ROM
drive, and perhaps the superiority of multimedia-enabled
microprocessors that understand additional multimedia
instructions.
  A consumer entertainment vendor: interactive cable TV with
hundreds of digital channels available, or a cable TV-like
service delivered over a high-speed Internet connection.
  A Computer Science (CS) student: applications that use
multiple modalities, including text, images, drawings
(graphics), animation, video, sound including speech, and
interactivity.
  Evolving definition…

A. Elgammal, Rutgers 2
Introduction to Imaging and Multimedia 9/5/13

Multimedia in Computer Science

  Digital Multimedia - Computational Multimedia

Digital Multimedia is the field concerned with

computer-controlled integration of text, graphics,
images, videos, audio, and any other medium where
every type of information can be represented,
transmitted and processed digitally.

Digital Multimedia: Historical Perspective

  The word multimedia was coined in the beginning of
the 1990s
  Mainly after the success of digital audio recording on
CDs
  The next anticipated step was to create digital content
involving image, text, video, along with the audio
  Multimedia CD-ROMs: ex, Encyclopedia Britannica
  Experience only limited to a single user interacting
with a PC
  Things have changed dramatically since then.

A. Elgammal, Rutgers 3
Introduction to Imaging and Multimedia 9/5/13

Example MM Applications
  Multimedia involves multiple modalities of text, audio, images,
drawings, animation, and video. Examples of how these
modalities are put to use:
  Video teleconferencing.
  Distributed lectures for higher education.
  Tele-medicine.
  Co-operative work environments.
  Searching in (very) large video and image databases for target visual
objects.
  “Augmented” reality: placing real-appearing computer graphics and
video objects into scenes.
  Including audio cues for where video-conference participants are
located. Taking into account gaze direction and attention of participants
as well

Example MM Applications
  Building searchable features into new video, and enabling
very high to very low-bit-rate use of new, scalable
multimedia products.
  Making multimedia components editable. allow the user side
to decide what components, video, graphics, etc., are
actually viewed; allow the client to move components
around or delete them. Making components distributed.
  Building “inverse-Hollywood” applications that can recreate
the process by which a video was made. This then allows
storyboard pruning and concise video summarization.
  Using voice-recognition to build an interactive environment,
say a kitchen-wall web browser

A. Elgammal, Rutgers 4
Introduction to Imaging and Multimedia 9/5/13

Multimedia life cycle

  Four main phases in multimedia life cycle
  Multimedia Generation, Authoring, Capturing…
  Multimedia Representation, and processing
  Multimedia Retrieval: answering user queries.
  Multimedia Delivery

MM Generation MM Archival MM Retrieval

User ?
Sensors / input

Capturing
(digitization) Retrieval
User ?
Indexing
Archival
Authoring MM Delivery
Editing
User ?

User ? Client

A. Elgammal, Rutgers 5
Introduction to Imaging and Multimedia 9/5/13

Inherent Quality of Multimedia Data

  Digital: it is always bits and bytes
  Voluminous: the storage and transmission bandwidth
limitations require that the data be compressed
  Interactive
  Real-time and synchronization: very small and
bounded delay while transmitting information. Intra-
media and inter-media synchronization

Multimedia Research Topics and Projects

To the computer science researcher, multimedia consists of a wide
variety of topics:
  Multimedia processing and coding: multimedia content
analysis, content-based multimedia retrieval, multimedia
security, audio/image/video processing, compression, etc.
  Multimedia system support and networking: network protocols,
Internet, operating systems, servers and clients, quality of
service (QoS), and databases.
  Multimedia tools, end-systems and applications: hypermedia
systems, user interfaces, authoring systems. Multi-modal
interaction and integration: “ubiquity” web-everywhere devices,
multimedia education including Computer Supported
Collaborative Learning and design, and applications of virtual
environments.

A. Elgammal, Rutgers 6
Introduction to Imaging and Multimedia 9/5/13

  Multimedia and Computer Science: Multimedia is in the

intersection among different areas
  Graphics,
  HCI,
  Visualization,
  Computer vision,
  Data compression,
  Graph theory,
  Networking,
  Database systems,
  Data mining
  Architecture and operating systems,
  ….

Multimedia Systems
  A Multimedia System is a system capable of
processing multimedia data and applications.
  A Multimedia System is characterized by the
processing, storage, generation, manipulation and
rendition of Multimedia information.

A. Elgammal, Rutgers 7
Introduction to Imaging and Multimedia 9/5/13

A. Elgammal, Rutgers 8
Introduction to Imaging and Multimedia 9/5/13

Components of a Multimedia System

  Capture devices
  e.g. Video Camera, Microphone, Digitising/Sampling Hardware, etc.
  Storage Devices
  e.g. Hard disks, CD-ROMs, DVD, Blu-ray, etc
  Communication Networks
  Internet, wireless internet, etc.
  Computer Systems
  e.g. Multimedia Desktop machines, Workstations, smart phones, iPads
  Rendering Devices
  e.g. CD-quality speakers, HDTV, Hi-Res monitors, Color printers etc.

Challenges for Multimedia Systems

  Distributed Networks
  Temporal relationship between data
  Render different data at same time continuously.
  Sequencing within the media (e.g. playing frames in correct
order/time frame in video)
  Synchronization — inter-media scheduling
  E.g. Video and Audio — Lip synchronization is clearly important for
humans to watch playback of video and audio and even animation
and audio.

A. Elgammal, Rutgers 9
Introduction to Imaging and Multimedia 9/5/13

Hardware
  All contemporary personal computers are quite
capable of displaying MM content
  Even Almost all cell phones
  For authoring content, a more powerful machine is
needed
  Bandwidth is still and always will be an issue

Multimedia System Desirable Features

  Very High Processing Power

  Special Hardware/Software needed (e.g. GPUs)
  Efficient I/O
  Large Storage and Memory
  High Speed Network Support

A. Elgammal, Rutgers 10
Introduction to Imaging and Multimedia 9/5/13

Multimedia History
  Before the digital age…

A. Elgammal, Rutgers 11
Introduction to Imaging and Multimedia 9/5/13

Multimedia: Text

Papyrus Parchment
Stone

Paper
  Han Dynasty of China (202 BC).Raw material such as tree bark was finely chopped, mixed with
water, spread onto screens, and dried. Well guarded secret
  Introduced in Europe in 600AD through the Middle East
  First paper mill in Europe was in Spain, in 1120. More mills appeared in Italy in
about the 13th century. They used hemp and linen rags as a source of fiber. Paper is
recorded as being manufactured in both Italy and Germany by 1400.

A. Elgammal, Rutgers 12
Introduction to Imaging and Multimedia 9/5/13

Printing Press

  In 1451, Johannes Gutenberg and Johannes Fust went into

partnership and produced a forty-two line Bible and a
thirty-two line Latin Grammar

  By 1465, printing presses based on Gutenberg's moveable

type could be found in Italy, by 1470 in Paris. London
followed in 1480. By 1499, there were presses in
Stockholm, Constantinople and Lisbon.

  By 1500, Europe contained in excess of nine million

volumes, of thirty thousand titles all of which came off the
presses of more than one thousand printers.

Multimedia Data: Text

  Text
  Input: keyboard, touch pad
  Stored and input character by character.
  Storage of text is 1 or 2 bytes per character.
  Other forms of data (e.g. Spreadsheet files, XML) may store format as
text (with formatting).
  Format: Raw text or formatted text
e.g HTML, Rich Text Format (RTF), Word or a programming language
source
  Not temporal — BUT may have natural implied sequence e.g. HTML
format sequence, Sequence of Java program statements.
  Size Not significant compared with other multimedia data.

A. Elgammal, Rutgers 13
Introduction to Imaging and Multimedia 9/5/13

Images Painting on some

media: From
Prehistoric times till
today

Camera obscura Niepce 1825 Daguerre 1838 Maxwell 1861

Mozi 400BC (8 hours) (10 mn) (color)
Aristotle 350BC
Ibn al-Haitham, 1000AD

1888 Eastman's Kodak camera on the market with

the slogan "You press the button, we do the rest"

Now: digital CCD 1969 at AT&T

Multimedia Data: Images

  Images
  Still pictures which (uncompressed) are represented as a bitmap (a grid
of pixels).
  Input: Digital camera, scanner or generated by graphics editor programs
(e.g. Paint)
  Analog sources will require digitizing.
  Stored at 1 bit per pixel (Black and White), 8 Bits per pixel (Grey Scale,
Color Map) or 24 Bits per pixel (True Color)
  Size: a 512x512 Grey scale image takes up 1/4 Mb, a 512x512 24 bit
image takes 3/4 Mb with no compression.
  Storage increases with image size
  Compression is commonly applied

A. Elgammal, Rutgers 14
Introduction to Imaging and Multimedia 9/5/13

Stored music

1598

15th
century
1870 Player
piano

Stored audio

1860:phonotaugraph

A. Elgammal, Rutgers 15
Introduction to Imaging and Multimedia 9/5/13

Modern period: music

1877 1904

Read-
Write !
Then digital

Multimedia Data: Audio

  Audio
  Audio signals are continuous analog signals.
  Input: microphones and then digitized and stored
  usually compressed.
  CD Quality Audio requires 16-bit sampling at 44.1 KHz
  1 Minute of Mono CD quality audio requires 5 Mb.

A. Elgammal, Rutgers 16
Introduction to Imaging and Multimedia 9/5/13

Multimedia Data: Video

  Video
  Input: usually captured by a video camera.
  There are a variety of video (analog and digital) formats
  Raw video can be regarded as being a series of single images.
  There are typically 25, 30 or 50 frames per second.
  a 512x512 size monochrome video images take 25*0.25 = 6.25Mb for 1
second to store uncompressed.
  Digital video clearly needs to be compressed.

Environment

user

Traditional Multimedia Interface vs. Ubiquitous Interface

A. Elgammal, Rutgers 17
Introduction to Imaging and Multimedia 9/5/13

Multimedia in a Ubiquitous Computing World

  Multimedia is becoming available all the time around us.
  Ubiquitous computing focus on integrating computation into the
environment, rather than having computers as distinct objects.
  Ubiquitous computing covers wide range of research topics,
such as distributed computing, mobile computing, and sensor
networks.
  Mobile phones now are the most commonly used multimedia
devices.

What this course is not about

  This course is not about
  Web design
  Multimedia tools
  Computer graphics and animation
  Internet technology

A. Elgammal, Rutgers 18
Introduction to Imaging and Multimedia 9/5/13

Resources
  Multimedia Systems Ch 1
  Some Slides by Prof G. Medioni @ USC

A. Elgammal, Rutgers 19