Abstract image of a woman sitting on books and studying on a laptop

Next-Generation User Interface Technologies for Mobile and Consumer Devices (2nd Edition)

Duff & Phelps, profile picture
Duff & Phelps

The document discusses next-generation user interface technologies for mobile and consumer devices, including gesture recognition, eye tracking, speech recognition, and biometric sensors. It highlights the significant impact of such technologies on market share and notes a growing trend in mergers and acquisitions among major tech companies in this sector. The report also provides an overview of various user interface technologies, recent M&A activities, and profiles key private companies involved in these advancements.

Economy & Finance
1 of 104
Downloaded 17 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
Next-Generation User Interface Technologies for Mobile and Consumer Devices Topics Include: • Gesture Recognition • Eye Tracking • Touch Screens, Haptics and Display Technologies • Speech Recognition / Language Processing / Mics • Security Biometrics • Sensor Fusion and Middleware • Brain, Heart and Other Biometric Sensors • Image and Vision Processing • Virtual and Augmented Reality March 2016
Page 2 Next-Generation User Interface Technologies for Mobile and Consumer Devices (2nd Edition) Report Overview Increasingly, the most differentiating features of consumer electronics devices are the user interfaces. Touch-screen displays have had a substantial impact on the smartphone market, and gesture recognition has played a major role in the gaming industry. Emerging user interface technologies have the potential to significantly affect market share in smartphones, tablets, computers, televisions and a variety of other electronic devices. As a result, there has been significant user interface technology M&A activity, including acquisitions by many of the largest technology companies (e.g., Apple, Google, Intel, Microsoft, Qualcomm, Facebook, Amazon, etc.). Given the growing importance of these technologies, there may be a robust M&A market in this sector over the next several years. This report provides a brief overview of a variety of mobile device user interface technologies. In addition to discussing the technologies themselves, the report also summarizes some M&A transactions in this sector and provides highlights of a variety of relevant private technology companies.
Page 3 About the Author …………………………………………..….………………...… 4 About Pagemill Partners and Duff & Phelps ………………………....……… 5 Select Pagemill Partners’ Transactions ………………………....…………..… 6 Preface to the 2nd Edition ………………………………………....…………..… 8 Select Pagemill / Duff & Phelps Contacts …………………….……………… 10 Chapter 1 – Introduction …………………………………………....…………..… 11 Chapter 2 – Gesture Recognition ……………………………………………….. 13 Chapter 3 – Eye Tracking ………………………….…………………………..… 20 Chapter 4 – Touch Screens, Haptics and Display Technologies ………..… 24 Chapter 5 – Speech Recognition / Language Processing / Mics .……….. 38 Chapter 6 – Security Biometrics …………………………………………...….… 48 Chapter 7 – Sensor Fusion and Middleware ………………………………….. 58 Chapter 8 – Brain, Heart and Other Biometric Sensors …….………..…..… 61 Chapter 9 – Image and Vision Processing …………………………...……..… 68 Chapter 10 – Virtual and Augmented Reality ………………...…………....… 80 Chapter 11 – Some Final Thoughts ……………………………………………….. 96 Appendix – M&A Transaction Summary ……………………………………….. 97 Table of Contents
Page 4 About the Author “Good bankers, like good tea, can only be appreciated when they are in hot water.” – Jaffar Hussein (Governor, Malaysian Central Bank) Mark Grossman, Managing Director Mark Grossman is a managing director at Pagemill Partners. At Pagemill, he has successfully completed more than 40 transactions (primarily M&A) around the world across a variety of technology sectors. Previously, Mark worked as an investment banker at SVB Alliant, where he was involved in a variety of technology M&A transactions. Prior to becoming an investment banker, Mark was a managing director and senior technology research analyst at several Wall Street firms. As an analyst, he wrote research on technology industry trends and covered dozens of public technology companies, and he received a number of accolades including the Wall Street Journal "Best on the Street" award. Mark began his financial career as an Associate at Goldman Sachs. Prior to graduate school, Mark was a Project Engineer at General Electric, where he managed the development of a variety of complex electronics systems. Mark has a bachelor's degree in electrical engineering, with a computer science minor, from Polytechnic University (now part of New York University), and both a master’s degree (Electrical Engineering) and an MBA from the Massachusetts Institute of Technology, where he completed a two-year dual- degree program. About the Author, Pagemill Partners, and Duff & Phelps
Page 5 About Pagemill Partners, a Division of Duff & Phelps Securities, LLC Pagemill Partners is a premier investment bank focused on successful transactions for middle-market technology companies. Pagemill Partners specializes in merger and acquisition advisory services and has successfully completed more than 225 transactions since the beginning of 2005. These include transactions with IBM, Microsoft, Intel, Oracle, Computer Associates, Qualcomm, Salesforce, SAP, and many other industry leaders. Pagemill Partners works in close partnership with its clients to help achieve extraordinary outcomes, and is able to repeatedly do so through the team's combination of transaction experience, domain expertise, and personal commitment to the success and objectives of its clients and their shareholders. Pagemill's team has collectively completed hundreds of transactions across a wide range of industries, geographies, and strategic circumstances. Pagemill is a Division of Duff & Phelps Securities, LLC. About Duff & Phelps Duff & Phelps is the premier global valuation and corporate finance advisor, with expertise in complex valuation, dispute consulting, M&A and restructuring. The firm’s more than 2,000 employees serve a diverse range of clients from more than 70 offices in North America, Europe and Asia. To learn more, visit www.duffandphelps.com. M&A advisory, capital raising and restructuring services in the United States are provided by Duff & Phelps Securities, LLC, Member FINRA/SIPC. Pagemill Partners is a Division of Duff & Phelps Securities, LLC. M&A advisory and capital raising services in the United Kingdom and Germany are provided by Duff & Phelps Securities Ltd., which is authorized and regulated by the Financial Conduct Authority.
Page 6 Select Pagemill Partners’ Transactions has been acquired by has been acquired by has been acquired by has been acquired by has been acquired by has been acquired by has been acquired by has been acquired by has been acquired by has been acquired by has been acquired by has been acquired by has been acquired by has been acquired by has been acquired by has been acquired by has acquired has been acquired by has been acquired by has been acquired by
Page 7 Select Pagemill Partners’ Transactions (cont.) has been acquired by has been acquired by has been acquired by has been acquired by has been acquired by has been acquired by has been acquired by has been acquired by has been acquired by has been acquired by has been acquired by has been acquired by has been acquired by has been acquired by has been acquired by has been acquired by has been acquired by has been acquired byhas acquired has been acquired by
Page 8 Update to the First Edition of the Report Introduction In the first edition of this report (issued in April 2014) we noted that, given the growing importance of human interface technologies for mobile devices, there would likely be a number of acquisitions in the sector. This has proven to be the case. In addition to a few high-profile M&A transactions (e.g., Facebook acquiring Oculus), a variety of companies in the sector, including several we highlighted in the first edition, were acquired. Some examples include the following (the appendix contains a more detailed list):  FlyBy Media (vision processing software) – Acquired by Apple  Faceshift (facial analysis software) – Acquired by Apple  SoftKinetic (3-D image sensor/gesture recognition) – Acquired by Sony  Vocal IQ (natural language technology) – Acquired by Apple  Saffron (cognitive computing) – Acquired by Intel  Qualcomm’s Vuforia Unit (augmented reality) – Acquired by PTC  CogniVue (vision processing) – Acquired by Freescale/NXP  Looksery (augmented reality “lenses”) – Acquired by Snapchat  Pebbles Interface (hand/gesture recognition) – Acquired by Facebook  Cypress’ Touch Unit (touch controllers) – Acquired by Parade  Metaio (augmented reality) – Acquired by Apple  N-Trig (stylus for tablets) – Acquired by Microsoft  Kolor (virtual reality software) – Acquired by GoPro  Lemoptix (MEMS pico-projection) – Acquired by Intel  Nimble VR (hand tracking) – Acquired by Facebook  13th Lab (augmented reality tracking) – Acquired by Facebook  Analog Devices’ MEMS Mic Unit (MEMS microphones) – Acquired by InvenSense.  Vision Factory (image and object recognition) – Acquired by Google  Movea (sensor fusion software) – Acquired by InvenSense  Sensor Platforms (sensor fusion software) – Acquired by Audience (which was later acquired by Knowles) Preface to the 2nd Edition
Page 9 In addition to M&A activity, several relevant companies have completed significant capital raises since the first edition. A few examples include:  Magic Leap (augmented reality) – Raised $542 million in late 2014 and just recently announced a new $793.5 million funding  Movidius (vision processing) – Completed a $40 million private capital raise  Tobii (eye tracking) – Had an IPO on the Nasdaq Stockholm exchange There has also been a substantial amount of new product activity. For example, Microsoft announced HoloLens; Samsung introduced Gear VR; and Sony and HTC have demonstrated virtual reality solutions. Given all of the recent activity, we thought the timing was right for a second edition. Beyond updating M&A transaction activity, the second edition also discusses recent sector and technology developments.
Author Mark Grossman Managing Director Silicon Valley / Boston mgrossman@pmib.com U.S. M&A Contacts Christian Bennett Managing Director Silicon Valley cbennett@pmib.com Gaurav Bhasin Managing Director Silicon Valley gbhasin@pmib.com Milledge Hart Managing Director Silicon Valley mhart@pmib.com Kevin Iudicello Managing Director Silicon Valley kiudicello@pmib.com James Kochman Managing Director Silicon Valley jkochman@pmib.com Patrick Manion Managing Director Silicon Valley pmanion@pmib.com Nels Nelsen Managing Director Silicon Valley nnelsen@pmib.com Matt Sachse Managing Director Silicon Valley msachse@pmib.com Charles Welch Managing Director Silicon Valley cwelch@pmib.com Canada M&A Contacts Scott Munro Managing Director Toronto / Silicon Valley smunro@pmib.com Ross Fletcher Head of Canada M&A Toronto ross.fletcher@duffandphelps.com China M&A Contacts David Lu Head of China M&A Shanghai / Hong Kong david.lu@duffandphelps.com Germany M&A Contacts Andreas Stoecklin Country Head for Germany Munich / Frankfurt andreas.stoecklin@duffandphelps.com Klaus Pflum Senior Advisor Munich klaus.pflum@duffandphelps.com Ireland M&A Contact Aidan Flynn Director Dublin aidan.flynn@duffandphelps.com UK M&A Contact Henry Wells Head of UK M&A London henry.wells@duffandphelps.com Contact Us
Page 11 “The last thing one knows – is what to put first.” – Blaise Pascal Introduction In 2007, Apple revolutionized the technology industry with the introduction of the iPhone. While the iPhone had many impressive characteristics, the most differentiating feature was its touch screen user interface, which helped catapult Apple into a leadership position within the smartphone market. Apple had previously transformed the computer industry when it introduced the Macintosh in 1984. The most significant feature of the Macintosh was its innovative computer mouse interface, which enabled intuitive “point and click” use rather than having to type in long obscure DOS commands. Nintendo dominated the gaming market in 2007 and 2008 as a result of the Nintendo Wii. The key feature of the Wii was its Wii Remote controller, which could detect 3-D motions by the user and made games much more interactive. Microsoft subsequently gained significant share in gaming owing in large part to its Kinect gesture recognition technology. As all of these examples indicate, the features that most differentiate electronics products are commonly human–machine interface (HMI) technologies. As form factors continue to shrink in size and become more pervasive (e.g., wearable devices, automobile electronics, smart home), HMI technologies will become increasingly critical. Not surprisingly, there has been growing interest from a variety of major technology companies. For example, Intel Capital created a $100 million fund a few years ago to invest in “perceptual computing” technologies such as 3-D image sensors, touch apps, imaging, gesture, voice, emotion sensing and biometrics. Many of the largest technology companies have already made one or more acquisitions in the HMI space (e.g., Microsoft, Intel, Qualcomm, Amazon, Apple, Google, Facebook). While there have already been a number of important acquisitions in this sector, there may continue to be a robust M&A market in HMI technologies over the next several years. This report provides a brief overview of next-generation user interface technologies that either are currently used in mobile electronics or may be used in the future. These include gesture recognition, eye tracking, speech recognition, touch screen haptics, vision/image processing, virtual reality, augmented reality, and biometric sensor technologies. The report also includes a brief overview of human interface technologies used for security (e.g., fingerprint sensing, voice authentication) and sensor fusion. In addition to providing overviews of each technology, the report also discusses M&A activity in each sector and provides descriptions of a variety of innovative private companies addressing next-generation user interface technologies. Chapter 1 – Introduction
Page 12 A Note on Highlighted Companies and M&A Deals The highlighted companies in this report (most of which are private companies or small non-U.S.-based public companies) are not meant to be an all-inclusive list., There are hundreds of start-ups addressing next-generation HMI technology; however, we believe it provides a good sampling of what is out there. We included companies based on the strength of their technology, degree of innovation, and level of public information, but also tried to include a mix of early stage start-ups and more well-known later-stage companies. Importantly, while we may have had calls or meetings with some of the highlighted companies, we have included only publicly available information—Meaning, in many cases, these companies may have extremely interesting next-generation solutions or unannounced design wins, but we included only information that is either on their Web sites or in published articles. Similarly, Pagemill was an advisor for a number of the transactions listed in this report, but we included deal information only if available from public sources. Report Overview The chapters in this document include:  Chapter 2 provides an overview of gesture technologies, in which users can control applications through the movement of their hands, fingers, head or body.  Chapter 3 discusses eye tracking, in which a person’s gaze can be used to control what is displayed on the screen.  Chapter 4 describes touch screens, touch screen haptic technologies, and other emerging display technologies.  Chapter 5 covers speech recognition, voice assistant technologies, MEMS microphones and natural language processing.  Chapter 6 describes security biometric technologies such as fingerprint sensors, voice authentication, facial recognition and eye recognition.  Chapter 7 contains an overview of sensor fusion and middleware solutions that utilize the data from a variety of sensors to enhance the user experience.  Chapter 8 provides an overview of biometric sensors (heart rate, brain wave, pulse, blood pressure, air quality, etc.) for health/wellness applications.  Chapter 9 discusses a variety of image and vision technologies such as image sensors, image recognition, vision processing and other related topics.  Chapter 10 describes virtual and augmented reality technologies and also briefly discusses holographic imaging.  Chapter 11 provides some final thoughts.  The Appendix summarizes M&A transactions mentioned throughout the document.
Page 13 “I have a big problem with conductors who gesture a lot.” – James Levine Introduction Gesture recognition refers to technologies in which the movement of a person’s hands, fingers, head or body can be used to control electronics. In one common implementation, one or more optical sensors capture images of the user and advanced software interprets the images to determine what action (if any) the user is requesting. For example, a TV viewer might be able to bring a finger to his lips to mute sound or a PC user could scroll by moving her hands up or down. This is generally much more intuitive for users than typing on a keyboard, and can significantly enhance usability. The highest-profile implementations of gesture recognition have been in gaming. As noted in the Introduction chapter, Nintendo’s Wii changed the gaming market with its Wii Remote controllers. Microsoft subsequently rolled out its Kinect technology, which enables users to play games and make selections using only hand and body gestures, without holding any type of controller. Gesture recognition has also been implemented in a number of high-end televisions (e.g., LG and Samsung remote controls) and computers (e.g., Intel’s RealSense). There is also significant interest in implementing gesture recognition in cars, to enable drivers to control certain car features (make selections from the entertainment system, answer a call, etc.) with just a wave of the hand, as that is viewed as much safer than having the driver look down to press buttons on the dashboard. There are a number of methods for implementing gesture recognition. The Wii Remote, for example, included an MEMS accelerometer that could detect user movements. The Wii MotionPlus further improved motion detection by adding a MEMS gyro. However, there is growing interest in implementing gesture recognition vision systems in which optical sensors repeatedly capture images of the user, and complex software algorithms are then used to determine what the user’s gestures are, without the user having to hold any device. As with most of the technologies described in this report, it is relatively easy to develop a simple gesture recognition system that works in ideal conditions, but the challenge is to make it robust under a broad range of circumstances. For example, with vision-based gesture systems, users can be at different distances and angles from the image sensors and have a broad range of different physical characteristics and movements. Additionally, there can be many different types of lighting conditions. Consumers can quickly become frustrated with new technologies when they work only intermittently or when there are latencies, so making these solutions robust is critical but can take many years of development and optimization. Chapter 2 – Gesture Recognition
Page 14 As many devices now incorporate conventional CMOS image sensors, optical gesture recognition can potentially be implemented with no additional hardware costs. However, to improve the performance of optical gesture recognition systems, some solutions utilize more specialized optical imaging sensors. Specifically, optical gesture recognition can generally be made more robust when images are represented in 3-D (with a “z” dimension), rather than with conventional 2-D representations. For example, with a 2-D representation it may be difficult to determine if a person’s hand is moving toward or away from the camera. As a result, a number of solutions have been developed that can capture “3-D” images (stereoscopic, time-of-flight, etc.). With stereoscopic solutions, two separate image sensors are used to capture the same image but from slightly different locations (a “right” and “left” image). By comparing the two images, software can estimate the distance of objects in the image. Another approach is referred to as “time-of-flight” (“ToF”). ToF is a type of light detection and ranging (LIDAR) technology. Light (typically invisible infrared light) is emitted and the reflected light from each object in the image is captured by an array of sensors. By calculating the amount of time it takes for the infrared light to complete a round trip, the distance of each object in the image can be determined, enabling a 3-D representation of the image. This greatly enhances the ability for accurate gesture recognition, but also adds some additional cost (e.g., infrared LEDs, image sensors that capture infrared light) relative to 2-D solutions. ToF was part of the gesture recognition technology developed by Canesta (acquired by Microsoft). While vision-based gesture recognition is most common, a number of other gesture recognition technologies have been developed, based on technologies such as ultrasound and electrical impulses from muscles. Gesture recognition has been implemented by a variety of major technology companies in a broad range of devices. Some examples include:  Microsoft accelerated the market for gesture recognition with the original Kinect accessory product and Kinect is a key part of the Xbox One gaming system.  Intel has announced its RealSense 3-D cameras which add depth-sensing to help improve the accuracy of gesture recognition and facial movements.  A variety of companies have introduced gesture-related remote controls for TVs.  Gesture recognition is an increasingly important part of virtual and augmented reality solutions. For example, if a user tries to grab something in a virtual reality game, but the system cannot accurately track his hand, it will result in a poor user experience.
Page 15 Select Gesture Recognition M&A Transactions Some notable acquisitions in the gesture recognition space have included the following:  Atheer/ONtheGO Platforms – In December 2015, Atheer (an augmented reality smart glasses company) announced an agreement to acquire ONtheGO Platforms. ONtheGo created gesture interfaces using standard cameras in mobile devices. Atheer indicated that the technology will be used to give customers an enhanced ability to use natural gestures with its smart glasses platform.  Sony/SoftKinetic – In October 2015, Sony announced that it completed the acquisition of Belgium-based SoftKinetic Systems. SoftKinetic was highlighted in the first edition of this report. It had developed a suite of gesture recognition solutions including 3-D time-of-flight CMOS image sensor chips, module level cameras, and complete stand-alone cameras. In addition, the company developed a substantial amount of middleware to enable easy implementation of gesture recognition. Sony indicated that it will focus on combining SoftKinetic's ToF range image sensor technology expertise with its own technologies with the aim of developing a next generation of range image sensors and solutions.  Facebook(Oculus)/Pebbles Interface – In July 2015, Oculus announced that they had acquired Israeli-based Pebbles Interface. Pebbles had spent five years developing optics, sensor systems and algorithms to detect and track hand movement. Oculus is expected to utilize Pebbles’ hand tracking technologies for the Oculus virtual reality solutions (e.g., enabling the system to more accurately track a user’s hand movements, so when a user moves his hands in the real world it can accurately control actions in the virtual world). The value of the transaction was not announced, but the Wall Street Journal reported that one person close to the transaction indicated a value around “$60 million.”  Facebook(Oculus)/Nimble VR – In December 2014, Nimble VR reported that it was joining Oculus. Nimble had developed its own 3-D camera (Nimble Sense) primarily for skeletal hand tracking in order to improve the virtual reality experience (tracking the user’s hand movements to improve virtual reality play). Nimble’s demo of Nimble Sense had already used the Oculus virtual reality headset.  Fairchild/Xsens – In January 2014, Fairchild Semiconductor completed the acquisition of Xsens for approximately $57.8 million in cash (per Fairchild’s 10-K). Xsens developed 3-D motion tracking products for professional applications based upon miniature MEMS inertial sensor technology. Xsens targeted markets such as robotics, marine industry, unmanned aerial vehicles, 3-D animation, virtual reality, and sports/health science. Xsens was based in The Netherlands.
Page 16  Apple/PrimeSense – In November 2013, Apple acquired Israeli-based PrimeSense for reportedly about $345 million (per the 451 Group). PrimeSense initially developed 3-D sensing software and camera systems utilizing off-the-shelf components, and later developed its own 3-D imaging SoC chip as well. The Company was best known for licensing the hardware design and chip used in the original Xbox 360 Kinect motion sensing system. Its “light coding” technology uses near-IR light and off-the-shelf image sensors. When reflected off images, the light is distorted based on the distance and this information is used to determine distance.  Google/Flutter – In October 2013, Google acquired Flutter, a private start-up which developed a gesture recognition app that worked with conventional Web cameras (requires only downloading the software). The transaction size was not announced but the 451 Group estimated the valuation at approximately $40 million.  Intel/Omek Interactive – In July 2013, Intel acquired Omek Interactive. Omek was an Israeli company – in Hebrew, Omek means “depth” – focused on the gesture recognition sector. The company developed software for creating a gesture recognition interface based on information provided by a 3-D camera. Intel Capital had been an investor. The transaction valuation was estimated to be approximately $50 million (per the 451 Group).  Qualcomm/GestureTek and EPOS – In July 2011, Qualcomm announced that it acquired certain assets from GestureTek. Qualcomm noted that the technology would be integrated into future versions of its SnapDragon solutions. Qualcomm also acquired ultrasound technologies for stylus and gesture recognition from Israeli- based EPOS Development in September 2012.  Microsoft/Canesta and 3DV – Canesta was a private company that developed CMOS 3-D image sensors. Unlike conventional image sensors, Canesta’s “time of flight” technology also measured the arrival time of collected photons, enabling the distance of each pixel of the image to be calculated which could then be used to determine the distance of objects from the sensor (essentially LIDAR on a chip). Honda had been an investor in Canesta, as Canesta had been working on using its technology to determine the distance of nearby cars (per a New York Times article) but later focused on video game applications. In October 2010, it was announced that Microsoft was acquiring Canesta. Separately, in 2009, Microsoft reportedly acquired 3DV Systems for about $35 million (per the 451 Group). 3DV had developed 3-D motion detection cameras.
Page 17 Select Private Gesture Recognition Company Overviews There are a variety of private companies focused on gesture recognition. This section highlights a few of them. We included a depth sensing ToF sensor company focused primarily on gesture recognition (pmd), but note that some depth sensing technology companies that primarily address other markets are included in Chapter 9. Gestigon Gestigon has developed advanced hand and finger gesture recognition software for use in a variety of applications. The company believes its technology is much more accurate than other approaches. Its initial target market has been automotive (using gestures to control infotainment, navigation, and climate features within a car) and the company has indicated that it is actively working with several major automotive companies. However, it has more recently begun addressing consumer electronics and other applications. In September 2015, Gestigon announced Carnival, an extension of its product line to address virtual reality and augmented reality markets. Its technology can track hands and provide a life-like representation of a user’s hands within a virtual environment making the virtual reality experience more realistic. At CES 2016, Gestigon announced a project with pmd (discussed later) that combines Samsung’s GearVR, pmd’s depth sensor, and Gestigon’s Carnival AR/VR Interaction Suite to showcase virtual reality. It also collaborated with Inuitive (a vision processor company discussed in Chapter 10) for virtual reality applications. Gestigon completed a Series A capital raise in September 2015. The company was founded in 2011 and is based in Germany with an office in Silicon Valley. MUV Interactive MUV Interactive has developed “The Bird,” which is a small device that a user wears on the finger. Using a combination of inertial sensing and optical sensing, Bird can accurately track finger movement (gesture recognition) which can then be used for a broad range of control applications. For example, Bird works with a TV or projector to transform any surface or image into a “touchscreen” without requiring any actual touching. That is, if an image of a Web site is projected on a wall from an overhead projector connected to a notebook PC, a user with the Bird can scroll up and down, “click” on a link, zoom in and out, move a pointer around on the image, or perform a variety of other tasks by pointing the finger with the Bird on it towards the projected image and moving the finger around. Bird can operate even if the user is across the room from the TV or projected image. Bird can also be used to accurately control drones. It integrates a microphone (to enable voice control), a small capacitive touch surface (for taps and swipes), and a Bluetooth interface. MUV believes the solution has many applications ranging from business presentations, classrooms, drones, smart televisions, and home appliances. The company’s Web site allows pre-orders of Bird indicating that it will begin shipping in Spring 2016. MUV is based in Israel.
Page 18 Pmd Technologies Pmd Technologies focuses on 3-D time of flight depth sensing solutions, with gesture recognition being one of the major applications. The company has developed its own 3-D ToF imaging sensing chip (PhotonICs) but has more recently been collaborating with Infineon on the jointly developed Real3 family which combines pmd's ToF distance- measurement technology and Infineon's manufacturing capabilities (including a specialized CMOS process optimized for ToF using micro-lens technology). The two companies indicate that the Real3 sensors are used as part of Google’s Project Tango (discussed later in the report). Pmd has also developed a number of complete 3-D depth cameras that integrate either the PhotonICs or Real3 chip and provide a complete reference design for OEMs. Its upcoming CamBoard pico flex solution is targeted at integration into smartphones. Pmd addresses consumer, industrial, and automotive markets and is based in Germany. Chirp Microsystems Chirp has developed advanced gesture recognition solutions. However, in contrast to most gesture solutions that rely on image sensors and video, Chirp’s solutions utilize ultrasonic audio. That is, a transducer creates ultrasonic waves and those waves are reflected by objects. A sensor measures the echoes and based on the ToF of the ultrasonic waves, the Chirp solution can determine the location of nearby objects. By repeatedly taking measurements, gestures can be recognized. Analyzing audio is orders of magnitude easier than analyzing video, thus Chirp believes its solution has many compelling advantages, including dramatically lower computational requirements, significantly less power consumption, and smaller size. For devices that don’t already have embedded image sensors, it is also significantly less expensive. Chirp is targeting consumer electronics, smart homes, industrial automation, and IoT. The technology was originally developed at UC Berkeley and Chirp is located in Berkeley, California. Leap Motion Leap Motion has developed a peripheral device (Leap Motion Controller) that connects to a PC or Mac computer or notebook via a USB cable and enables hand gesture recognition. The device is relatively small (only about 3” long) and sits on the table in front of a computer. The Controller tracks hands movements at up to 200 frames per second using infrared cameras and enables a broad range of gesture related controls. A variety of games and apps have been developed that utilize the Leap Motion Controller and the company has an app page on its Web site. The Leap Motion Controller sells for $79.99 on the company’s Web site. Leap Motion announced a beta version of Leap Motion for virtual reality (e.g., enabling a user to see their hands move in virtual reality). It recently announced that it signed a Memorandum of Understanding with SK Telecom for 5G “AR/VR Service Development.” One of its major investors, Highland Capital, has established a $25 million fund for companies developing technologies based on the Leap Controller. Investors in Leap Motion include Highland Capital, Andreessen Horowitz, Founders Fund, and SoSventures International.
Page 19 PointGrab PointGrab was highlighted in the 1st edition of this report as an early pioneer in gesture recognition for consumer devices, but has shifted its focus since then to home automation and IoT applications. Its PointSwitch solution is an optical module that utilizes machine learning algorithms to analyze visual input. Its technology can sense occupancy, ambient light, and occupants’ behavior, which can be used to reduce energy consumption, improve comfort, and lower operational costs. The company indicates that its solutions are being developed in collaboration with leading providers in lighting and building control systems. PointGrab is based in Israel. Extreme Reality Extreme Reality (EXT3-D) has developed 3-D motion analysis software solutions that work with conventional CMOS image sensors. The company’s Extreme Motion technology captures images of people and then creates skeletal models of the people, which are then used to analyze motion. The company believes this core “skeleton positioning technology” enables much more accurate body motion analysis than other conventional body motion tracking solutions and it can be implemented on any system with a basic Webcam. The company’s technology has been incorporated into a number of games. It also announced the technology is incorporated into Warchief and that it has a partnership with TVisual, a leader in China‘s digital signage market for interactive signage. Extreme Reality was founded in 2005 and is based in Israel. Thalmic Labs The bulk of gesture recognition technologies are based on optical systems, but Thalmic Labs has developed a nine-axis motion sensor armband (Myo) that can read the electrical signals from muscle movements in the forearm. The armband incorporates a gyroscope, an accelerometer, and a magnetometer. As a result, the armband can identify a broad range of bodily gestures as well as the movement parameters of the arm (direction, speed, and angle). The output of these sensors is converted to digital and processed and the resulting commands are sent via Bluetooth to a smartphone or computer. The company supports a variety of games (e.g., Fruit Ninja) and applications (e.g., controlling a Parrot drone) and opened an online marketplace for the more than 100 existing Myo-based apps. Thalmic is based in Canada and its investors include Salesforce.com CEO Marc Benioff and ATI co-founder Lee Lau.
Page 20 “Beauty is in the eye of the beer holder.” - Kinky Friedman Introduction Eye tracking involves determining where a person is looking at or the motion of the eye. This can then be used to control what is displayed on the screen or to help make selections. Eye tracking is not a new concept. For example, in the early 1900s a non- intrusive eye tracker was developed that reflected light off of the subject’s eyes and recorded the results on film. During the 1950s and 1960s, Alfred Yarbus performed a variety of eye tracking experiments and published the first major book on the subject. Most of the early eye tracking solutions were developed for psychological testing, but during the 1980s and 1990s eye tracking had some limited commercial use for marketing studies and to assist handicapped people with computers. More recently, however, there has been growing interest in utilizing eye tracking technology as an interface for electronic devices, including smart phones, computers, and automotive applications. Although there are a variety of different eye tracking technologies, most of the solutions addressing consumer applications utilize the optical method in which light (in many cases infrared light) is reflected by the eye and then captured by an image sensor. The images are then analyzed to determine where the user is looking. Because different parts of the eye (cornea, pupil, sclera, etc.) reflect and absorb light differently, the direction of a person’s gaze can be determined from the images. This can then be used to control the location of the cursor/pointer on a display or to control the display in other ways. For example, a user looking at a map on a display could potentially adjust the map in any direction just by looking at the edge of the map, or zoom in just by staring at a certain spot for a period of time, making it easier to navigate. While many eye tracking solutions have been developed, the challenge in commercializing the technology is making it work consistently across a broad range of users (e.g., different eye types, some with contact lens, some with glasses, etc.) in many different environments (e.g., dark rooms, light rooms, etc.) under a variety of circumstances (users in different positions, viewing angles and distances from the screen). In addition, penetrating the consumer market requires a low cost, small form factor with low power consumption. Historically, many of the eye tracking solutions worked well in ideal conditions but had problems in real world situations. Chapter 3 – Eye Tracking
Page 21 While eye tracking solutions vary, they often include: a light source (it is possible to rely on conventional lighting, but many solutions use one or more LED lights, and often infrared LED sources are used as infrared light is not visible to the human eye); an image sensor (in some cases a traditional CMOS image sensor and in other cases an infrared image sensor to improve performance); a processor (in some cases the main processor is relied on to handle the analytical processing, but some solutions have a dedicated chip for this); and software (which analyzes the captured images to determine where the user is looking and provides an interface to various applications). There are a number of applications for which simply gazing at a particular point can cause the application to take action. For example, a mapping program could scroll up, down, left or right if the user stares at certain spots on the edges of the map for a period of time. A game could adjust the scene based on which direction the user is looking. However, one general limitation of eye tracking is that in many cases it still requires using some other method to “click”/select the item that the user is gazing at (i.e., using gaze, a user can move the screen pointer to a specific menu item on the screen, but then must hold his gaze while he presses a key on the keypad or uses his voice to select it). Making selections based solely on gaze is challenging since it can lead to inadvertent selections as the user examines his choices. Blinking has been explored as a way to make selections (stare at a point and then blink for a period of time to select) but that is also challenging to implement since people naturally blink every few seconds. However, there are applications in which gaze can be used to make choices (staring at a spot for a certain length of time), especially if the app is specifically optimized for eye tracking. Eye tracking is still early stage with respect to commercial adoption in high volume consumer devices, but there appears to be growing interest in it for a variety of applications (e.g., games) and one eye tracking company, Tobii, had an IPO in 2015. In addition, it could become more important for virtual/augmented reality applications which historically tracked head movement (i.e., change the view as the user turns his head), but could provide a more realistic experience by also adjusting based on eye movement. There is also interest in optimizing AR and VR displays by providing higher resolution to the area that the user is looking at versus other areas (foveated rendering), which can provide a variety of advantages. Eye tracking can also be used for reducing power consumption (e.g., the screen dims when the user isn’t looking at it). Eye Tracking M&A Below are a few eye tracking acquisitions. Note that there have been a number of eye- related security biometric deals, but these are covered in the chapter on biometric security. There have been relatively few eye tracking M&A transactions, likely because eye tracking has not yet gained significant traction in high volume products yet. This could change if the technology becomes more commonplace in consumer and virtual reality solutions.
Page 22  Alcon/SMI’s Ophthalmic Unit – In November 2012, Alcon announced that it acquired the Ophthalmic division of SMI, a private company based in Berlin (discussed below). SMI addresses a variety of eye tracking markets, but sold off its Ophthalmic unit (which specializes in ocular surgery guidance technology) to Alcon.  Facebook/SiteHawk – In March 2012, SiteHawk announced that its team was “joining the Facebook team.” SiteHawk developed eye tracking technology that used conventional Web cameras (rather than any special infrared or other type of specialized hardware). SiteHawk noted that the deal was only for the team and that its existing products were not part of the deal and remain independent of Facebook.  TandemLaunch/Mirametrix – In 2011, TandemLaunch (an accelerator/incubator for tech companies) acquired Mirametrix, a private company that developed eye tracking technologies and solutions. Select Eye Tracking Companies The following are brief overviews of some eye tracking companies: Tobii Tobii has developed a number of eye tracking solutions. Its products incorporate near-IR micro-projectors, optical sensors, and image processing software to determine the location a person is looking at. The Company has several business units: Dynovox (which is focused on eye tracking for users that have ALS, cerebral palsy, Parkinson’s or other medical issues that prevent natural hand movement); Pro (eye tracking for market research); Tech (for OEMs looking to integrate eye tracking), and EyeX (an eye tracking controller that Tobii sells to consumers and developers). In a September 2015 investor presentation, Tobii indicated that Dynovox and Pro accounted for about 73% and 21% of revenue, respectively. Although consumer applications historically represented only a relatively small portion of sales, the company has announced a number of video games that utilize its eye tracking (Assassin’s Creed Rouge, Flight Simulator, etc.) and that it is targeting a variety of high volume end markets. Tobii also supports power consumption reducing features such as dimming the screen if the user is not looking at it and keeping the screen awake if the user is looking (even if they hadn’t touched a key for several minutes). Tobii also powers biometric facial recognition for Windows 10 (Windows Hello). Tobii has announced its own ASIC (EyeChip) for eye tracking which can make implementation easier for some applications and further optimizes performance. Tobii is based in Norway with offices in the U.S., Germany, Japan and China. The company went public on the NASDAQ Stockholm market in April 2015.
Page 23 SensoMotoric Instruments (SMI) SensoMotoric Instruments (SMI) was founded in 1991 and has developed a broad range of computer vision products including eye tracking peripherals and glasses. Its Eye Gaze and Tracking Systems unit sells a variety of advanced eye tracking solutions (systems, glasses, software, etc.) for professional and research applications (e.g., sports training, market research, psychology studies, etc.). The company indicates that more than 6,000 of its eye tracking systems are in operation and some of its listed customers include HP, Rite-Aid, and Volkswagen. It recently announced an automated analysis solution that eliminates the need for manual coding and greatly simplifies eye tracking analysis. During the past couple of years, SMI has introduced solutions for virtual reality and augmented reality applications (e.g., integrating its eye tracking with Epson’s Moverio BT- 200 head mounted display, an eye tracking upgrade package for the Oculus Rift, a Google Glass prototype with eye tracking, etc.). To address high volume markets, SMI has developed an OEM Solutions business focused on integration into consumer products and the company has introduced general reference platforms that OEMs can implement for mobile/desktop, augmented, and virtual reality applications. Earlier this year, SMI and OmniVision announced an alliance in which OmniVision's global shutter CameraCubeChip will be integrated into SMI's virtual reality and augmented reality eye tracking platforms. At CES 2016, SMI announced it supports foveated rendering, which enables VR headsets to provide higher resolutions to areas the user is looking at (and less resolution for other areas). SMI historically also had a business unit focused on eye care but, as previously mentioned, SMI sold its Ophthalmic Division to Alcon in 2012. SMI is based in Germany. The Eye Tribe The Eye Tribe is an eye tracking company focused on mobile devices. The Eye Tribe software enables eye control on mobile devices, allowing hands-free navigation of websites and apps, including eye activated login, enhanced gaming experiences, and cloud-based user engagement analytics. The company sells a $99 USB 3.0 eye tracking device on its Web site. During CES 2016, it demonstrated its new eye tracking solution that is optimized for virtual reality applications. It enables foveated rendering (in which higher resolution is provided for critical areas of an image – such as the area in which the user is looking at) which can make a VR headset much more efficient and can also enable better imaging for smartphone-based VR devices. Much less processing is required since high resolution isn’t used for areas where the user isn’t looking. Eye Tribe recently announced a new Tracker Pro professional eye tracking solution for $199, with shipments targeted for June 2016, as well as an eye biometric authentication method. The Eye Tribe was founded by four PhDs from the University of Copenhagen and is based in Denmark.
Page 24 “Touch is the mother of the senses.” - Helen Fisher Introduction Although the original iPhone had many innovations, the touch screen was one of the most differentiating features when it first launched. While basic touch screens had been used for decades, Apple was first to optimize the touch screen interface to make it easy to use and cost effective. As a result, virtually all smart phones and tablets now incorporate a touch screen, and an increasing number of notebook PCs and other devices have touch displays. This chapter discusses touch screens, haptics, and other emerging display related technologies. Touch Screen Technologies There are a variety of different types of touch screen technologies. A brief overview of the general mainstream touch displays include:  Capacitive – Capacitive touchscreens take advantage of the fact that the human body is an electrical conductor. A capacitive touchscreen typically includes an insulator (e.g., glass) and a transparent conductive layer such as indium tin oxide (ITO). When a finger touches the screen it distorts the screen’s electrostatic field and causes a change in capacitance which can be measured to determine the finger’s location. There are actually a variety of different types of capacitive touch technologies. The most common general type for high-end portable devices (smart phones, tablets, etc.) is mutual projective capacitance in which there is a matrix of rows and columns with a capacitor at each intersection. A voltage is applied to the rows and columns and when a finger touches the screen, the local electrostatic field changes, reducing the mutual capacitance, which can be measured to determine the location at which the screen was touched. A major advantage of this approach is that it enables multi- touch (it can recognize multiple fingers) which is critical for many features that consumers are now used to (zooming in and out using two fingers, etc.). Many other capacitive technologies and a variety of non-capacitive touch technologies can recognize only one touch at a time. One disadvantage of capacitive is that it doesn’t work if a person is wearing gloves or with an ordinary stylus/pen, although special pens have been developed that can be used (for example, in 2015 Microsoft acquired N-Trig, an Israeli company that developed the stylus for the Surface Pro 3). Most of the major high-profile touch devices (iPhone, iPad, Samsung Galaxy, etc.) utilize capacitive touch technology. Chapter 4 – Touch Screens, Haptics and Display Technologies
Page 25  Resistive – The general principle behind resistive touch screens is that there are two thin transparent layers coated with transparent conductive material (the outer layer covered on the back, the inner layer coated on the front) separated by a small gap. When the outer screen is pressed, the two layers touch, completing an electrical circuit and creating a voltage divider. Based on the resistance of the materials and measured voltages, the location of a touch can be determined. Resistive touchscreens are relatively low cost and work well with a stylus (especially good for handwriting recognition) or when a user is wearing gloves. However, it generally doesn’t support multi-touch well and isn’t as sensitive as capacitive touch since the user has to press down on the screen. As a result, while there have been a number of phones with resistive touchscreens, most high-end mobile devices utilize capacitive. Resistive is common in other markets such as ATM machines.  Other (Infrared, Ultrasound, etc.) – Over the years, many different types of touch screen technologies have been developed. Examples are listed below. o Infrared (IR LEDs and photodetectors are placed around the edge of the display and can detect if a finger or object touches the screen) o Ultrasound/surface acoustic wave (ultrasonic waves pass over the surface of the display, and when the display is touched some of the wave is absorbed) o Infrared projection (a transparent sheet is illuminated with infrared light and “leaks” light when the user touches the screen) o Optical imaging (image sensors around the edge of the screen capture shadows to pinpoint the location of each touch) o Dispersive (detects the piezoelectricity in the glass due to touch, which can be used to pinpoint the touch location) o Acoustic pulse (a touch generates a sound wave and sound transducers are used to determine the position of the touch) Although a variety of new touch technologies have been developed as potential alternatives to capacitive touch, capacitive works very well, is robust, can implement multi-touch, and costs have been driven down with volume. As a result, displacing this technology in devices such as smart phones and tablets is challenging. However, as touch technology becomes increasingly pervasive in a variety of products, there are a growing number of different applications for touch screens, providing a variety of opportunities for other touch technologies. For example, the capacitive technology used in smart phones generally doesn’t scale well to large size screens and many applications don’t require multi-touch. One recent innovation in touch has been Apple’s introduction of “3D Touch” in the iPhone 6s which not only recognizes the location of where a user is touching but can differentiate between “hard” and “soft” touches, creating a much wider range of user inputs. For example, a soft press can allow the user to preview an email or website while a harder press can open the message or Web page. Apple indicates that when a user presses on the display, capacitive sensors measure the microscopic changes in the distance between
Page 26 the cover glass and the backlight, and the measurements from these sensors can be used to determine the level of force. Some of the major suppliers of capacitive touch screen controller chips include Atmel, Parade (which acquired Cypress’ mobile touch screen business), and Synaptics, as well as a variety of companies in Korea and Taiwan. A recent trend has been integration of touch controllers and display drivers. Touch Screen Technology M&A The following are a few touch screen related M&A transactions:  Parade/Cypress Mobile TrueTouch unit – In June 2015, Cypress announced that it was divesting its TrueTouch Mobile touchscreen business to Parade Technologies for $100 million. This unit provided touch screen controllers for a variety of consumer markets (smartphones, tablets, notebooks/PCs, etc.). Cypress indicated that it will continue to provide TrueTouch solutions to its automotive and home appliance customers.  Synaptics/Renesas SP Drivers – In June 2014, Synaptics announced an agreement to acquire Renesas SP Drivers for about $515 million in enterprise value. The acquired business focused on small and medium-sized display driver ICs for smartphones and tablets and Synaptics indicated that the deal would accelerate its product roadmap for integrating touch controllers and display drivers.  FocalTech/Orise – In April 2014, FocalTech and Orise announced that the two companies would merge in a stock swap (each share of FocalTech swapping for 4.8 newly issued shares of Orise). FocalTech was focused on touch controllers and Orise was a major supplier of display drivers. As with the Synaptics deal noted above, the primary reason given for the deal was the interest in integrating touch controllers with drivers into single chip solutions.  Microsoft/Perceptive Pixel – In July 2012, Microsoft acquired Perceptive Pixel. Perceptive had developed very large screen multi-touch displays.  TPK/MasTouch and Cando – TPK has been a major touch screen module supplier and has made a number of acquisitions. It acquired a majority of the shares of MasTouch Optoelectronics (a controller chip company). In 2011, it acquired a just- under 20% stake in Cando for about $190 million (per Capital IQ). Cando is one of TPK’s largest touch sensor suppliers and is a subsidiary of AU Optronics.  Gores/TE Connectivity EloTouch Products – In 2012, private equity firm Gores Group acquired the touch screen solutions business from TE Connectivity for $380 million, or slightly less than 1X revenue (per Capital IQ). The business provided a range of different types of touch screen technologies.
Page 27 Select Touch Screen Technology Companies There are a number of start-up companies that have developed alternative technologies to the capacitive touch sensing technologies commonly used in current smartphones. In some cases, their focus is either large screen markets such as television or large PC displays (as capacitive doesn’t scale to larger sizes well) or on smaller screens such as wearables (in which multi-touch isn’t critical). In other cases, the technology provides certain features or advantages that conventional capacitive screens do not provide. Some examples of touch screen technology companies are below. NextInput NextInput has developed a force sensitive touch surface technology (ForceTouch). In addition to being able to determine the location of a touch, its technology also can determine the amount of force applied by a finger or stylus. This enables a variety of new ways to control applications. That is, a light touch can be interpreted differently from a stronger touch, which can greatly expand the number of potential control gestures. Adding force makes interaction much more intuitive in many cases. It can also enhance many applications (e.g., gaming). Unlike Apple’s 3D Touch technology which utilizes traditional capacitive sensing, the NextInput solution (ForceTouch Array) is a MEMS sensor layer below the surface of the display. NextInput indicates that its solution has many significant competitive advantages relative to Apple’s 3D Touch and other types of force touch technologies. As the NextInput solution measures force directly, the company believes that ForceTouch has 10x better accuracy and sensitivity relative to alternative methods. As a result, it can not only more accurately determine if a user is pressing hard or soft and where the user is pressing, but can identify a broader range of force levels (not just hard and soft). Specifically, the company indicates that its technology enables touch event location with better than 1mm resolution and measures applied force in the range of a few grams to greater than one kilogram. NextInput has also indicated that its technology is substantially less costly (in volume) and consumes about half the power of capacitive touch, and that it can work with a stylus, and in virtually any environment (high EMI location, underwater, etc.). In July 2015, NextInput formally introduced its MEMS-based FT-4010F solution (1.3mm x 1.3mm x 0.63mm). In December 2015, it announced an $8 million funding round led by Sierra Ventures and two strategic corporate investors and that Steve Nasiri (founder of InvenSense) joined the Board. In January 2016, it announced that its technology was being incorporated into a smartwatch developed by GoerTek. NextInput was originally based in Atlanta, but relocated its headquarters to Silicon Valley.
Page 28 FlatFrog FlatFrog has developed a touch display technology (InGlass) in which infrared light is injected into a transparent screen (plastic, glass, etc.) and is detected at the other end. The light travels through the material via total internal reflection, but when a finger touches the surface, it causes a disturbance to the light which is detected. Using advanced signal processing algorithms, the location of the touch can be determined. FlatFrog indicates that its technology supports multi-touch and that it is less costly than capacitive, especially for larger displays since their sensors are around the edge of the display whereas the number of capacitive sensors increases with display area. Its technology can also be used to detect pressure since the amount of pressure impacts how much light escapes from the medium. The company offers touch module solutions up to 110” and indicates that it can measure up to 1,000 levels of pressure and can support up to 80 multi-touches. In 2013, FlatFrog announced a partnership with Dialog Semiconductor, in which Dialog developed a Smartwave multi-touch ASIC specifically optimized for FlatFrog’s technology. The company is initially focused on niche markets such as education (smart board displays), but anticipates moving into high volume consumer applications over the next couple of years. FlatFrog is privately held with headquarters in Lund, Sweden (near Copenhagen, Denmark) and an office in Silicon Valley. Its investors include Intel Capital as well as Invus, Sunstone Capital, and Fårö Capital. Neonode A small public company that has been focused primarily on alternative touch screen technologies is Neonode. Neonode’s zForce is based on infrared technology in which infrared light is projected over the surface of the screen, and photodetectors measure the light patterns. When a finger or stylus touches the screen, it interferes with the infrared light and based on the measurement of the light and advanced algorithms, the location of the object or finger can be determined. The technology doesn’t need additional screen layers, works with a stylus or gloved fingers, and doesn’t require force on the display. The company has leveraged its core technology to also implement proximity sensing, such that the device can detect finger or object movements that are around the outside of the device rather than just on the screen. It recently announced its AirBar solution (a plug and play gesture recognition solution for a broad range of devices) and a number of automotive demos (with Volvo, Autoliv, etc.). It also recently announced its technology is integrated into a family of Lexmark printers. Neonode had an IPO in September 2013.
Page 29 Touch Screen Haptics One limitation of traditional touch screens is the lack of any tactile sensation when touching the screen. For example, many people find it difficult to type more than short messages on a touch screen keyboard since they can’t feel the buttons, which is why there is still demand for keyboards that connect to tablets and why many people still use notebook PCs rather than tablets. However, there is growing interest in technologies that will change that. Haptics technology involves providing tactile feedback to users to improve the user’s experience. Haptic technology is already used in a variety of applications. For example, there have been a number of game controllers/joysticks that include haptics, such that the controller vibrates in certain gaming situations to make the game seem more realistic. The most basic use of haptics in handsets is for “vibration mode” in which the phone vibrates when a call is received. Haptic technologies, however, are increasingly being incorporated into touch screens in order to improve the user experience and provide feedback when the display is touched. The most common method for implementing haptics on a display is through the use of actuators (motors that control movement). One or more small actuators are incorporated under or on the side of the screen. When the user touches the screen, the actuator can be turned on by the app or operating system, which causes the display to vibrate and provides tactile feedback. Depending on where the actuators are placed, the types of actuators used, the frequency that they operate at, and the number of actuators used, a variety of different effects can be created. For example, when a user touches a specific spot on the screen (e.g., a letter on a keyboard), it can be made to feel like that particular spot is providing tactile feedback. The goal with high quality haptics is to give the impression to the user that she is touching actual physical keys, for example, even though the display remains flat. In addition to typing, haptics can greatly improve the user experience in a variety of other applications such as gaming. In general, the greater the number of actuators used and the more complex the actuators (wider frequency range, ability to implement complex waveforms, etc.), the more realistic the haptic experience can be, but that also results in higher cost and greater space requirements. Haptics is also important in automotive applications as it reduces the amount of time the driver needs to look at touch controls. The largest nearly pure play haptics technology company is Immersion Corporation, which is a publicly listed company. Immersion has developed a variety of haptics technologies that it licenses to other companies. According to Immersion’s December 2015 Investor Presentation, 63% of its revenue in Q3:15 came from mobile devices, but it also addresses gaming (23%) automotive (7%), and medical (7%) markets. Immersion has over 2,000 patents, most of which are related to haptics. It offers a variety of different haptic solutions, including single actuator, multi-actuator, and high-fidelity actuator solutions. It also provides software and effect libraries to help developers create desired haptic effects.
Page 30 According to a 2015 press release by research firm MarketsandMarkets, the overall haptics technology market is expected to reach nearly $30 billion by 2020 at an estimated CAGR of more than 25% from 2014 to 2020. In 2013, Apple was awarded a patent for a haptic feedback system that is optimized for multi-touch surfaces ("Method and apparatus for localization of haptic feedback"). The patent involves using two actuators in which one actuator provides feedback to the user and the other creates a vibration to suppress the first to prevent it from propagating to the rest of the screen, resulting in localized haptic feedback. Select Haptics M&A Transaction A notable haptics acquisition was the following:  Parker Hannifin/Bayer MaterialScience/Artificial Muscle – Artificial Muscle was spun out of SRI International in 2004 to commercialize its Electroactive Polymer Artificial Muscle technology. Artificial Muscle was focused on haptics for mobile devices. EAP technology consists of a thin layer of dielectric polymer film sandwiched between compliant electrodes. When voltage is applied across the electrodes, the electrodes attract each other. As a result, the film contracts in thickness and expands in area, which can be used to create haptic effects. In 2010, Artificial Muscle was acquired by Bayer MaterialScience. In 2014, Parker Hannifin announced that it acquired intellectual property and licenses for the Artificial Muscle technology from Bayer MaterialScience. Select Private Haptics Related Companies A number of start-ups have addressed the haptics market, including the following: Redux Labs Redux Labs has developed differentiated haptics (Surface Sensation) and speakerless audio (Surface Sound) technologies for smartphones, tablets, PCs, automotive, and other applications. Its solutions are based on its patented “bending wave” technology that controls sub-sonic and sonic waves across a flat or curved surface. Its Surface Sensation haptic solutions deliver a customizable haptic effect to a specific coordinate on the screen and include transducers (piezoelectric transducers for small form factor devices and voice coil exciters for larger form factors), control modules or chips, and algorithms for implementing its bending wave technology. Its haptic technology can deliver feedback at very specific points anywhere on the display and the company believes it provides substantial advantages relative to traditional actuator-based haptics.
Page 31 Because its transducers can manipulate waves, the glass of a smartphone, tablet or TV screen can also be transformed into a surround sound loudspeaker (its speakerless “Surface Sound” audio technology), eliminating the need for separate speakers or sound bars and potentially reducing the size and cost of products. Redux indicates that it is addressing a variety of end markets including consumer electronics devices, automotive applications (providing tactile feedback enables drivers to better focus on driving), PC/mobile devices, industrial, and augmented/virtual reality. The Redux Web site indicates that it has exclusive rights to exploit over 300 granted patents in bending wave technology with over 100 additional patent applications filed. The company is based in Cambridgeshire, England (a few miles from Cambridge). Novasentis Novasentis (previously Strategic Polymer Sciences) has developed a unique and innovative actuator/sensor based on electrical-mechanical polymers (EMP) related to a new class of ferroelectric materials. The EMP material is ultra-thin, light and flexible and elongates when an electric field is applied and produces electric charge (piezo-electric) when it is deformed due to pressure (from a finger) and can, therefore, act as both an actuator and a sensor. With digital control, the actuators can produce precise localized vibrations, sounds, or deformation. It can also be used as a sensor to recognize touch/pressure. Leveraging this core technology, Novasentis has developed its Clic family of ultrathin (<200 microns) haptic actuators. The company indicates that its technology is much thinner and requires relatively low operating voltages when compared to traditional vibrational motors or piezoelectric benders. Novasentis was founded in 2006 by former Apple executive Ralph Russo and the inventor of the EMP technology, Dr. Qiming Zhang. In November 2013, Novasentis announced it raised an $8 million Series B round that included new investor Samsung Ventures and previous investor Chengwei Capital. The company is based in Burlingame, California and has offices in Pennsylvania, Korea, and Japan.
Page 32 Tactus Technology Tactus has developed a unique technology based on microfluidics that can create transparent physical buttons on a touch screen surface on demand. For example, when a user wants to type on a keyboard, physical keyboard buttons rise from the surface enabling users to feel the keyboard buttons as they type (eliminating one of the main complaints regarding touch screen typing). Unlike traditional haptics solutions that try to give the illusion that there are physical buttons, the Tactus solution actually creates temporary physical buttons on the screen. In addition to the advantages of “tactile feel,” the Tactus solution also enables finger resting (i.e., some pressure on the buttons is required to type a key). When the user doesn’t want to use the keyboard, the physical buttons recede into the screen leaving a flat smooth screen. While keyboards are the most obvious use for the Tactus solution, the technology can create buttons (or other shapes) anywhere on the screen which can be used for a variety of other applications. The underlying Tactus technology is based on microfluidics. Small transparent fluid channels are routed throughout the “Tactile Layer” and the fluid can expand the polymer layer above it to create physical buttons. A small controller chip is used to control the fluid channels and create or recede buttons. An API is provided to enable applications to control where and when buttons are created. The Tactus solution is designed to integrate with capacitive touch screens, with the Tactile layer replacing the cover lens (top-most layer). While its long term goal is to have its technology directly integrated into devices, Tactus introduced a product called Phorm in early 2015. Phorm is an add-on case for an iPad-Mini that includes a transparent front piece and a back piece with a switch on it. When the switch is turned on, tiny buttons rise from the transparent cover over the displayed keypad, and then recede when the switch is turned off. The Web site indicates the price is $99 but that it is sold out. The company announced a Series B capital raise in early 2015 that included IPV Capital. Tactus is based in Fremont, CA. Ultrahaptics Ultrahaptics has developed haptics technology that enables users to receive tactile feedback without actually touching anything. The technology uses ultrasound which is projected through the air onto the user’s skin in such a way that it creates a sensation in which the user “feels” like he is touching something. For example, users can ‘feel’ touch- less buttons, get feedback for mid-air gestures, or interact with virtual objects. This could obviously have a broad range of applications. In 2015, it announced that it is supporting Jaguar Land Rover in the investigation of a mid-air touch system for a Predictive Infotainment Screen. The company was founded in 2013 based on technology originally developed at the University of Bristol, UK. In November, 2015, Ultrahaptics announced that it completed a Series A funding round of £10.1 million led by Woodford Investment Management alongside existing shareholder IP Group plc.
Page 33 Aito Aito has developed a software enhanced piezo technology that can both recognize touch and provide haptic feedback. Its technology utilizes low cost single layer piezoelectric discs that can both sense and actuate. Piezoelectricity can also be utilized to wake up the user interface to achieve maximal energy efficiency. The solutions are extremely thin (0.3mm) and work through a broad range of surfaces (e.g., under aluminum, carbon, etc.) and the company believes it is significantly more robust than other haptic solutions and has numerous other advantages. Its product families include Touch (LED/buzzer feedback), HapticTouch (adds haptic feedback), and LE (<1uA, low power). Aito is initially targeting automotive and appliance applications, but believes its solutions can be used in many markets. At CES 2016, it displayed its technology in the Texas Instruments booth (software on a TI microcontroller) and announced that LG is using Aito’s technology in a line of kitchen appliances. Aito is based in Amsterdam with R&D in Helsinki. Investors include INKEF Capital and Sirius Venture Capital. Quantum Dots Quantum dots are ultra-small (a few nanometers) nanocrystals of semiconducting material. They have a number of interesting properties including florescence, in which the color emitted is determined by the size of the particles. Specifically, smaller quantum dots typically have shorter wavelengths (colors on the blue end of the spectrum) while larger quantum dots have longer wavelengths (colors on the red end of the spectrum), although the exact emission depends on the quantum dot material. The most common material for quantum dots is cadmium selenide (CdSe). Because quantum dots can be tuned to produce a very precise wavelength (color), they have a variety of applications including displays. The properties of “quantum dots” were first discovered in the early 1980s, although the name “quantum dot” was initially coined in the late 1980s. Quantum dot technology is initially being used primarily for improving the backlighting of conventional LCD displays. Typically, LCD displays use white LED backlighting. The white light from the LEDs is then color filtered to produce specific colored pixels (red, blue, green). However, this has a number of issues. LEDs don’t naturally produce white light and so “white LEDs” are created by coating blue LEDs with a phosphor material. The resulting “white” light is poor quality. In addition, the subsequent filtering process results in reduced power efficiency. Quantum dots can be used to resolve some of these problems. For example, one approach is to use conventional blue LEDs (as the main light source) along with red and green quantum dots to provide RGB colors. This can produce a much broader color gamut and, therefore, much better images than white LED backlit displays (most color displays produce only about two thirds of the NTSC standards for color quality whereas 100% is possible with quantum dots).
Page 34 Quantum dots also reduce power consumption (less filtering) which is important as backlighting is one of the most significant power drains in mobile devices. In high volume, it is also believed that quantum dots can be much lower cost. Many of the initial implementations of quantum dots are in a film over the backlight, but it could also be directly integrated into LEDs. A number of major OEMs have recently begun selling quantum dot TVs and monitors and it is expected to become more commonplace over time. At CES 2016, for example, a substantial portion of Samsung’s booth was dedicated to quantum dot TVs. Longer term, there is the potential for quantum dots to have an even more significant impact on display architectures, as the technology improves and designs are further optimized around the technology. According to a press release by Touch Display Research, the quantum dot display and lighting component market is projected to reach $10.6 billion by 2025. Select Quantum Dot Companies There are several companies developing quantum dot solutions for display applications. One public company is Nanoco which is listed on AIM and is developing cadmium free quantum dots (which it indicates is more environment friendly). Two major privately-held quantum dot companies are listed below. In addition to being used for displaying images, quantum dot technology can also be utilized for other applications including image sensing (e.g., InVisage is highlighted in a later chapter). QD Vision QD Vision has developed advanced quantum dot technology (Color IQ) for displays. The company was founded in 2004 by researchers from MIT including MIT professors Vladimir Bulcovic and Moungi Bawendi (two pioneers in quantum dot research). After more than a decade of development, QD Vision announced a number of commercial partnerships and customers in 2015. Some examples include: a partnership with Konka for TVs in the Chinese market; commercial shipments for a TCL 65” 4K UHD TV; the launch of the first quantum dot monitor (with Philips Monitors); and the launch of a curved quantum dot TV from HiSense. At CES 2016, the company announced a strategic partnership with Sigma Designs to develop a cost-effective UHD TV platform with High Dynamic Range (HDR) and Rec. 2020 performance. It also demonstrated a monitor that covers 87 percent of the Rec. 2020 color gamut, which it indicated is the widest coverage commercially available now. QD Vision indicates that it has shipped more than one million Color IQ optics products and that it has more than 250 patents and patents pending. QD Vision has raised more than $75 million in financing from venture capital firms, including North Bridge, Highland, Passport Capital, Novus Energy Partners, Capricorn Investment Group, BASF Venture Capital, and Tsing Capital. It is headquartered in Lexington, Massachusetts.
Page 35 Nanosys Nanosys is a major supplier of quantum dot solutions. The company sells an advanced Quantum Dot Concentrate material and licenses its technology and designs, including its Quantum Dot Enhancement Film (QDEF), to industrial supply chain partners. The Quantum Dot Enhancement Film is designed to replace an existing film (diffuser) in a display’s backlight unit. The quantum dot film combines trillions of quantum dots in a thin sheet that emits white light when stimulated by blue light. Each QDEF sheet contains two plastic barrier films sandwiching a layer of quantum dots. Nanosys has announced a number of major commercial partnerships. It has partnered with 3M on Quantum Dot Enhancement Film. A couple of recent announcements include a partnership for integrating 3M QDEF film (using Nanosys’ quantum dots) with AU Optronics’ backlighting, and the launch of a Hisense 4K 65" ULED Curved Smart TV using Nanosys technology. In January 2016, TCL announced that it is using Nanosys Quantum Dots for a new line of Premium UltraHD HDR Televisions. Nanosys has a significant partnership with Samsung, a major proponent of quantum dot technology, including patent licenses, a strategic investment, and a follow on investment. Nanosys announced in March 2015 that it doubled its manufacturing capacity for quantum dot material to 25 tons per year at its Silicon Valley plant (enough for 6 million 60” UHD TVs per year). Nanosys has a number of investors including Samsung Ventures, ARCH Venture Partners, Venrock, El Dorado Ventures, Polaris Partners, Harris & Harris, and Lux Capital. New Touch Display Materials The vast majority of touch screens (including both capacitive and resistive) utilize indium tin oxide (ITO) as a transparent conductor. Indium is a relatively expensive material so there have been a number of companies that have tried to develop lower cost alternatives. Another reason there is interest in alternative transparent conductive materials is to improve the performance of touch displays and to help make them less fragile and more flexible. An example of a company that has developed an alternative to ITO is discussed briefly below. Select Touch Screen Material Company Cambrios Technologies Cambrios Technologies has invented a transparent coating material (ClearOhm), based on silver nanowires, that has a number of advantages relative to ITO. Because silver conducts very well, ClearOhm has very low resistance/high conductivity which helps improve electrical performance. In addition, the material is highly flexible and less prone to breaking than ITO. It can be used on glass (e.g., Gorilla glass) as well as plastic or OLED displays, and can scale to large size displays. Lastly, Cambrios indicates that in high volumes ClearOhm can be substantially less expensive to produce than ITO.
Page 36 As a result, Cambrios believes its solution will be an important part of next-generation touch technologies across a variety of displays. In addition to mobile device touch screens, Cambrios’ technology has applications for solar, medical, and OLED lighting. Over the past few years, Cambrios has announced partnerships with a broad range of companies in the touch screen supply chain (e.g., Nissha Printing, Sumitomo, Chisso, TPK, Okura, Shin-Etsu, etc.) as well as a variety of other types of companies (e.g., Crystalsol GmbH, Claymount, Heraeus, etc.). In January 2016, Cambrios announced, along with its customer LG Electronics Chemical & Electronic Material division, that its technology is now being used by OEMs such as Acer (AIOs and monitors), Cisco, HP (AIOs), Lenovo (AIOs), LG, MSI (AIOs and monitors), and Toshiba (Satellite desktops). Cambrios is based in Silicon Valley and its investors include several strategic parties (including Samsung) as well as a number of VCs including ARCH Venture Partners, Alloy Ventures, and others. Pico-Projection and HUD Displays Pico-projectors are another type of displaying technology although it does not use an actual display. That is, pico-projectors enable users to display images or videos on a screen, wall, or other surface. Integrating a pico-projector into a smartphone or tablet would make it easier for other people to see images or videos and could be used for business purposes (e.g., showing PowerPoint slides without needing a large traditional projector) or for entertaining friends (e.g., showing videos). Unlike traditional large bulb-based projectors, pico-projectors generally use LEDs or laser diodes for illumination. While integrating some type of projection into a mobile device is relatively easy, the challenge is making it small (ideally so it doesn’t increase the size of the overall device) and effective (powerful enough for the projected image to be clear even from a distance) with low power consumption. A variety of relatively small pico-projector add-on products are available, although the long- term goal has been to incorporate the technology directly into notebook PCs, tablets and smartphones. There have been a few smartphones that have integrated pico-projectors (e.g., the Samsung Galaxy Beam), but they typically are much thicker than conventional smartphones in order to accommodate the projection technology and generally the illumination power has been relatively weak compared to conventional projectors. Several public chip companies have developed pico-projector chipsets (e.g., Texas Instruments, STMicroelectronics). A pure-play public company in the pico-projector sector is MicroVision. MicroVision’s PicoP technology utilizes a MEMS scanner which is coupled with blue, red, and green laser lights to create a projected image.
Page 37 A somewhat related market to pico-projection is heads-up displays (HUD) for applications such as automobiles. Increasingly consumers are making decisions about which car to purchase based not on the engine features, but on the car’s infotainment/display systems. As the amount of electronics in cars increases, the number of displays a driver may want to look at also increases. However, the more a driver looks down at the dashboard, the less focused he is on the road. With HUD technology, images can be projected right on the car’s windshield, making it easier for a driver to see without looking down. For example, road images for navigation can be projected on the windshield, making it easier for drivers to follow directions. Warning signals can also be displayed on the windshield so the driver can immediately see when there is an issue, rather than waiting to notice that a warning light on the dashboard is on. As the technologies required for pico-projectors and HUDs are similar, some of the pico-projector companies are also addressing the HUD market. Pico-projection can also be relevant for augmented and virtual reality (projecting images onto the user’s glasses or headsets). Select Pico-Projection M&A Transactions Examples of pico-projection M&A transactions are:  Intel/Lemoptix – In March 2015, Lemoptix announced that it had been acquired by Intel. Lemoptix was highlighted in the first edition of this report and had developed advanced pico-projection solutions based on MEMS laser scanning. The company was leveraging this core technology to address a variety of end markets including pico-projection for mobile devices, automotive heads up displays, gesture recognition, and wearable devices. Lemoptix was based in Switzerland and was a spin-off from the Swiss Federal Institute of Technology.  STMicroelectronics/bTendo – In August 2012, STMicroelectronics acquired smartphone projection technology company bTendo (per the 451 Group and company press releases). bTendo was based in Israel and had already been working closely with STMicro for a couple of years prior to the acquisition. The bTendo scanning laser projection engine produces a focus-free high-resolution output that can display images or video on surfaces.
Page 38 “She had lost the art of conversation but not, unfortunately, the power of speech.” - George Bernard Shaw Introduction Voice is a natural interface technology that has become increasingly common in electronics. Speech recognition has been around for many years. At the 1962 World’s Fair, for example, IBM demonstrated a system that could understand 16 words, and in the 1970s a system developed by Carnegie Mellon University could recognize more than 1,000 words. However, most of the early speech recognition systems required saying each word slowly one at a time. By the 1990s, consumer speech recognition programs became available (e.g., Dragon), although they were fairly expensive and required a significant amount of set-up time as users had to repeat words multiple times to “train” the software. With significant improvements in processing power and algorithms, speech recognition has dramatically improved during the past decade and has been incorporated into a variety of devices. For many devices, speech recognition is used for simple commands (e.g., providing an address to a navigation system, calling a cell phone number by saying the number or the name of the person, changing the station on a TV, etc.). However, with Apple’s introduction of the Siri personal assistant, speech has become a full-fledged interface technology that allows users to control and interact with mobile devices. Accomplishing this requires not only translating speech into words (basic speech-to-text technology), but also advanced natural language query technologies that are capable of piecing the words together into sentences/commands and then “understanding” what the user is asking for. This is much more complex as languages and grammar rules are complicated, words can have a variety of meanings depending on the context, and users can ask questions or give commands in a variety of different ways. As mobile products shrink in size (such as wearable devices), speech recognition is expected to become an even more common interface, as it doesn’t require a keyboard or large screen. For example, speech recognition is included in Google Glass, the Samsung Galaxy Gear watch, and the Apple watch. Chapter 5 – Speech Recognition / Language Processing / Mics
Page 39 A number of the major technology companies have developed voice assistant technologies (Apple Siri, Google Now, Samsung S Voice, Microsoft's Cortana, Amazon's Alexa, etc.) and many mobile devices incorporate some type of speech technology to varying degrees. Amazon’s Echo solution (which integrates 7 microphones and connects to its Alexa cloud- based voice service) was specifically developed to enable voice control in home environments. Facebook has also launched a personal assistant technology (M). A public company that is primarily focused on speech recognition related solutions is Nuance Communications, which sells a broad range of different types of speech recognition products for a wide range of different end markets. One area of focus for improving speech recognition is that many of the existing speech recognition systems compress the user’s speech and send it over the network to dedicated servers that do the analysis needed to translate and understand the speech and analyze the words, and the results are then sent back over the network to the device. This, however, causes some latency issues and doesn’t work if the device does not have network access. As a result, one approach is to perform the analysis in the device, but this requires efficient algorithms as processing power in a mobile device is limited and complex processing can consume significant power and drain the battery. There is also interest in combining speech recognition with noise reduction/voice enhancement technologies to further optimize performance. That is, users increasingly speak at a distance from devices and often in noisy environments. For the speech recognition to work, the device must be able to extract the voice commands from the background noise. By combining speech recognition with voice enhancement technologies, the overall performance of voice-based assistants can be improved. Microphone technology has become increasingly important, especially as mobile devices must increasingly capture voices from users that are several feet away. The mobile device industry has largely transitioned from conventional microphones to MEMS microphones and the number of MEMS microphones per device has been increasing (typically 2 to 4 MEMS microphones per smartphone). There is interest in further improving these microphones. Artificial Intelligence/Natural Language Technology It is one thing for an electronic device to translate simple pre-arranged voice commands (“On”, “Off”, “Louder”, “Softer”, etc.) into words (basic text-to-speech) and actions, but it is another, much more complex task to analyze a complex set of unstructured words and sentences and determine what a user is looking for. While Siri, Google Now, and other speech-related solutions have become much better at understanding and responding to simple commands and questions, they are still a long way from being able to have a real conversation and understand complex questions or complicated commands. As a result, there has been significant interest in artificial intelligence and deep learning solutions that can be used for enhancing speech interfaces as well as for other types of user interactions. These technologies often have a broad range of applications in addition to user interaction but could become increasingly important over time.
Page 40 Select Speech Recognition and Natural Language M&A Transactions Unlike many of the other technologies discussed in this document in which there have been only a handful of M&A transactions, there have been a large number of speech technology related transactions during the past couple of decades. For example, Nuance Communications alone has acquired more than 40 companies since 2005 when it was formed by the merger of Nuance and Scan Soft (and Scan Soft had acquired many companies before that), and a large portion of those acquisitions were speech-related. (Pagemill alone has completed 9 transactions with Nuance). Below are a small handful of the many speech and natural language related M&A transactions in recent years. We mention only those related to the consumer markets, as there have also been many acquisitions of companies focused on speech recognition for business, medical, and call center type applications. We also included some high profile artificial intelligence/cognitive computing acquisitions (e.g., Google/DeepMind) in which it is believed that the technology could be used for natural language analysis.  Apple/Vocal IQ – In October 2015, it was reported by the Financial Times that Apple acquired Vocal IQ, a natural language technology company based in the UK. Vocal IQ developed a self-learning dialog API, which can learn from interactions with people, improving a device’s ability to interact with a person over time.  Intel/Saffron – In October 2015, Intel announced on its blog that it had acquired Saffron, a cognitive computing platform provider. Intel stated that Saffron ingests data from disparate sources and automatically connects the dots to help improve decision-making. Intel indicated that in addition to enterprise analytics, Saffron’s technology can be used in consumer devices that need to see, sense and interpret complex information in real time.  Facebook/Wit.ai – In January 2015, Facebook acquired Wit.ai, a speech recognition company that had been founded only about 18 months earlier. Wit.ai had developed an API for developing voice-activated interfaces, making it easier to create speech recognition and voice activation solutions. Investors included Andreessen Horowitz and Y Combinator.  Google (DeepMind)/Dark Blue Labs – In October 2014, following up on its DeepMind acquisition noted below, Google acquired UK-based Dark Blue Labs. Dark Blue Labs specialized in deep learning technologies for understanding natural language. Their research focused on enabling computers, robots or other digital devices to better understand what users are asking for.  Google/DeepMind – In January 2014, Google announced that it was acquiring DeepMind. The terms of the deal were not disclosed, but a number of media reports (Forbes and others) noted that sources indicated the price was over $400 million. DeepMind was a UK-based artificial intelligence company that developed advanced learning algorithm solutions that could learn from experience. It could, for example,
Page 41 figure out how to score highly on video games without any pre-programming related to the games. While DeepMind can be used for a broad range of applications, interfacing with humans is viewed to be one potential application.  Apple/Novauris – In January 2014, TechCrunch reported that Apple acquired Novauris Technologies in 2013. Novauris was an automatic speech recognition technology company, which was founded in 2002 and grew out of Dragon Systems (later acquired by Nuance). Novauris developed server-based speech recognition systems that could process several simultaneous voice access requests.  Yahoo/SkyPhrase – In December 2013, Yahoo acquired SkyPhrase (per the 451 Group), a start-up that had been focused on building solutions that better understand natural language. This core technology can improve speech recognition systems by providing a deeper understanding of what the user is asking for.  Intel/Indisys – In September 2013, Intel acquired Indisys for reportedly more than $26 million (per the 451 Group). Indisys was a Spanish company focused on natural language recognition technologies.  Facebook/Mobile Technologies (Jibbigo) – In August 2013, Facebook acquired Mobile Technologies (doing business as Jibbigo), a speech recognition and machine translation app company. The app enables users to text or record voice content in over 25 languages that is then translated on screen or read in another language (per the 451 Group).  Google/SR Tech Group Patents, SayNow, Phonetic Arts – In July 2013, Google acquired a speech-related patent portfolio from SR Tech Group LLC (including a patent for “Speech interface for search engines”). In January 2011, Google acquired SayNow which had been viewed as a Google Voice competitor. In December 2010, Google acquired Phonetic Arts which had developed advanced speech synthesis (text-to-speech) technologies (per the 451 Group).  Google/Wavii – In April 2013, Google acquired Wavii, (a natural language technology processing start-up focused on news summaries), for reportedly about $30 million (per the 451Group).  Amazon/Evi, Ivona, and Yap – Amazon has made several acquisitions in the speech technology area. In April 2013, Amazon reportedly acquired Evi Technologies for $26 million (per the 451 Group). Evi’s True Knowledge Answer Engine attempts to understand questions by analyzing its database of facts, and it can be used for general search as well as voice interactions. Amazon also reportedly acquired Ivona Software (multi-lingual speech synthesis system) in early 2013, and Yap (voice to text) in November 2011.
Page 42  Apple/Siri – In April 2010, Apple acquired Siri. Siri was initially available as an app in the App Store, but Apple later incorporated the Siri Assistant into the iPhone 4s in October 2011. The transaction price was not announced, but reportedly the valuation was about $200 million (per the 451 Group).  Nuance Communications/SVOX, Vlingo, Loquendo, and Many, Many Others – As previously noted, Nuance has acquired more than 40 companies since 2005, many of which had speech recognition technologies. Some examples include: SVOX for about $123 million ($81 million upfront, $42 million deferred), Vlingo for $196 million (excluding the equity interest that Nuance already owned), Loquendo for $75 million, and Ditech Networks for $23 million (per 10K reports, Capital IQ and the 451 Group). Select Private Speech Recognition/Natural Language Companies During the past decade, a large number of speech recognition companies have been acquired but there are still a variety of private companies in the sector. There are also a variety of natural language processing companies. Sensory Sensory is a major supplier of speech recognition solutions, although it has also recently diversified into other related markets including voice and vision authentication. Sensory provides a broad range of speech recognition software solutions including:  TrulyHandsFree – Recognizes keywords and enables consumers to control devices using voice commands, even in noisy environments. It incorporates voice activation, so power consumption is extremely low. It is designed to accurately recognize phrases even when embedded in sentences and surrounded by noise. The latest version incorporates deep learning technologies to greatly improve accuracy.  TrulySecure – A combined voice and facial recognition authentication solution for mobile devices and PCs. Authentication is on-device, and almost instantaneous. TrulySecure is the first face and voice biometrics solution to be FIDO UAF Certified.  TrulyNatural – An embedded large vocabulary speech recognition system that can operate on-device, rather than requiring cloud-based servers. This reduces latency and works even if the device cannot connect to the cloud. In addition to a broad range of software solutions, Sensory also offers speech processing chips for customers looking for a complete chip/software solution. The NLP-5x, for example, is optimized for speech processing, including speech recognition, text-to-speech (TTS), and high quality music and speech synthesis. The chip integrates a 16-bit DSP and a variety of digital and analog processing blocks and communication interfaces, and works with Sensory’s FluentChip firmware.
Page 43 Sensory indicates that its speech recognition solutions have been incorporated into close to a billion products from hundreds of customers including AT&T, Hasbro, JVC, Kenwood, LG, Mattel, Motorola, Plantronics, Samsung, Sony, SEGA, Uniden, and V-Tech. In January, Sensory announced that its TrulyHandsFree technology will be offered on Samsung’s ARTIK IoT module platform to enable hands-free IoT products. It also announced a partnership with Philips, combining Philips’ voice enhancement technology with Sensory’s solutions. Sensory was founded in 1994 and its headquarters is in Silicon Valley. Basis Technology One issue with natural language processing is that there are many difference languages spoken around the world and the natural language analysis differs for each language. Basis is a leading provider of software solutions for extracting meaningful intelligence from multilingual text and digital devices. Its Rosette linguistics platform uses advanced natural language processing techniques to improve information retrieval, text mining, machine learning, statistics, and computational linguistics. Rosette provides capabilities such as: identifying the language of incoming text; providing a normalized representation in Unicode; locating names, places, and other key concepts from a body of unstructured text; and name matching and name translation for names in foreign languages and scripts. Rosette can, for example, help search across multiple languages for names or places. Basis also has a digital forensics group for assistance with analyzing multi-lingual text. Basis indicates that its solutions have been used by over 200 major firms (e.g., Amazon, EMC, Exalead/Dassault, Fujitsu, Google, Hewlett-Packard, Microsoft, Oracle) and governments around the world. The company also indicates that it is the top provider of Asian linguistic technology to web search engines, including Ask.com, Google, Microsoft Bing, and Yahoo!. Basis is based in Cambridge, MA. VoiceBox VoiceBox Technologies has developed a variety of solutions for speech recognition and natural language understanding. Its solutions focus on home, automotive, and mobile applications. The company indicates that it developed the first in-car natural language application and that its solutions have been used by Toyota, Chrysler/Fiat, Renault, Mazda, Subaru, TomTom and AT&T (for their DriveStudio initiative). The company also notes that its solutions are deployed in a variety of mobile phones and wearable devices. VoiceBox indicates that one of its advantages is that its technology tracks context throughout a conversation (Contextual Natural Language Understanding), making it easier for it to understand what the user is really looking for. VoiceBox is based in Bellevue, Washington.
Page 44 Semantic Machines Semantic Machines is focused on using artificial intelligence to enable natural conversation between people and computers. The company indicates that its technology goes beyond understanding commands to understanding conversations. This allows computers to better communicate, understand goals, and accomplish tasks for users. Semantic Machines believes that its technology will have a major impact on a variety of sectors including search, e-commerce, social media, and productivity software. The company indicates that a number of its team members previously worked on Siri and Google Now. In December 2015, Semantic announced that it completed a $12.3 million capital raise. Its investors include Bain Capital Ventures, General Catalyst Partners, Justin Kan, and Ray Stata. It has offices in Berkeley and Boston. MindMeld MindMeld (previously known as Expect Labs) has developed artificial intelligence technologies for voice driven applications. The company was founded in 2011 and launched its MindMeld platform in 2014. The platform enables customers to create intelligent voice driven interfaces for a broad range of apps and devices. Partnerships have been announced for several markets including smartwatches and healthcare. In early 2014, it announced the release of the MindMeld API, which it indicated was the first context-driven developer platform and cloud-based service, and enables developers to deliver better search and content discovery using contextual signals instead of keywords. Applications include intelligent assistants for mobile devices, improved website search, and voice and context driven dashboards. In December 2015, the company announced the launch of MindMeld 2.0 for enabling enterprises to create advanced voice interfaces that are intelligent and accurate for search and navigation. The company indicated that the new version enables state-of-the-art natural language understanding with human-like accuracy, and that it is in trials with leading online services including Spotify. Investors include Google, Samsung, Intel, Telefónica, Liberty Global, Greylock Partners, Bessemer Venture Partners, IDG Ventures, KPG Ventures, In-Q-Tel, and Quest Venture Partners. MEMS Microphone Technology Historically, users held mobile phones next to their mouths when speaking. However, increasingly people speak to devices that are much further away. For example, with video conferencing applications (Facetime, etc.), consumers will typically hold a smart phone or tablet at arm’s length from their face. When using Skype with a notebook PC or speaking to Siri on a tablet or using a smartphone for video recording, as just a few examples, the person speaking is often several feet from the device. This makes it much more difficult for the device to obtain a high quality voice/audio signal.
Page 45 An important part of any voice related system is the microphone, which converts audio signals into electrical signals. A broad range of different types of microphones have been invented going all the way back to Alexander Graham Bell (“liquid transmitter” microphone) and Thomas Edison (carbon microphone). Each of the various microphones is designed such that incoming acoustic signals ultimately cause a change in an electrical signal. For example, a basic traditional (DC biased capacitor) condenser microphone has a thin diaphragm that acts as one plate of a capacitor and a separate fixed plate. Acoustic waves cause the diaphragm to vibrate and, as a result, the distance between the plates changes, which causes the capacitance to change. The circuit is designed such that the change in capacitance results in a change in voltage across the capacitor. As a result, the voltage changes based on incoming sound. This basic capacitive microphone was invented at Bell Labs in 1916. The electret microphone (also invented at Bell Labs), a type of capacitive microphone which uses a permanently charged material (eliminating the need for a polarizing power supply), had been commonly used in mobile phones for many years. More recently, however, the technology industry has migrated to MEMS-based microphones for many products. MEMS technology involves the creation of tiny mechanical structures using semiconductor manufacturing process technologies. This enables microphones that are significantly smaller than traditional microphones, which also makes it easier to incorporate multiple microphones into the same device, providing much better sound quality. Once MEMS microphones became commonplace, OEMs quickly moved to using two microphones to enhance audio quality. The iPhone 5 incorporated a third MEMS microphone and the iPhone 6s has four MEMS microphones. Relative to conventional microphones, MEMS microphones are also typically more robust, less temperature sensitive, and consume less power. Because they are semiconductor devices, MEMS microphones can be more easily integrated into electronic circuits. Although MEMS microphones are manufactured using a significantly different process than conventional microphones, most use the same basic principles of operation (only at a much smaller size). That is, most MEMS microphones use capacitive technology in which there is a fixed plate (back-plate) and a movable plate (membrane). Incoming audio signals move the membrane, which changes the capacitance between the plates and creates an electrical signal representing the incoming sound. While microphones are inherently analog, MEMS microphones are often classified as digital or analog based on the output of the component. With a digital MEMS microphone, the electrical signal is converted to digital bits before external transmission, whereas analog microphones have an analog output (and the signal is converted to digital at some point later in the signal chain). Some of the key characteristics of microphones are: sensitivity (how much the electrical signal changes for a given change in pressure from the acoustic waves); directionality (how much the sensitivity changes based on the speaker’s location relative to the mic); and signal to noise ratio (how much noise is produced by the mic relative to the actual voice signal).
Page 46 Major suppliers of MEMS microphones include Knowles, AAC (China), Goertek (China), BSE (South Korea), STMicroelectronics, InvenSense, Bosch, Hosiden (Japan), NeoMEMS (China), and Cirrus Logic. Infineon is not a supplier of end MEMS microphone solutions but develops and sells MEMS microphone die to a variety of companies that then package and complete the solutions. For example, AAC and Goertek had been suppliers of conventional electret microphones but became major suppliers of MEMS microphones primarily by leveraging Infineon’s core MEMS solutions. Select MEMS Microphone M&A Transactions Some major MEMS microphone transactions include the following:  InvenSense/Analog Devices’ MEMS Microphone Unit – In October 2014, InvenSense announced that it was acquiring Analog Devices’ MEMS microphone business for $100 million, plus an earn-out based on results from the unit over a 12 month period. InvenSense had been a major supplier of MEMS gyroscopes and was already in the process of developing its own MEMS microphone solutions.  Cirrus Logic/Wolfson – In April 2014, Cirrus Logic announced the acquisition of Wolfson Microelectronics for an enterprise value of £278 million (approximately $467 million at the time). Wolfson was a Scotland-based public company that was primarily focused on analog audio chips such as audio DACs, but also had a small MEMS microphone business. Wolfson entered the MEMS microphone market through the acquisition of a small start-up named Oligon in 2007.  Bosch/Akustica – In 2009, Bosch acquired Akustica. Akustica was an early pioneer in MEMS microphones and was based in Pittsburgh. Akustica became part of Bosch’s MEMS unit (Bosch Sensortec GmbH) which had already been a major supplier of MEMS accelerometers.  Dover/Knowles – In 2005, Dover acquired Knowles for $750 million. Knowles was a major supplier of MEMS microphones. At the time of the acquisition, Dover indicated that it anticipated revenue of approximately $210 million from Knowles. Subsequently, in February 2014, Dover spun off Knowles as a separate company to its existing shareholders and Knowles began trading as a public company in March 2014. Dover included some product lines in Knowles that were not part of the original Knowles, but their main product remains MEMS microphones.
Page 47 Select MEMS Microphone Company Below is an overview of one of the start-ups focused on innovative MEMS microphones. Vesper As previously noted, the vast majority of MEMS microphones utilize capacitive sensing technology in which a membrane vibrates based on incoming sound waves which changes the distance between the membrane (which acts as one capacitor plate) and a fixed capacitor plate. Vesper (previously known as Baker Calling) is a privately-held company based in Boston that has developed a differentiated MEMS microphone family that utilizes piezoelectric materials, rather than traditional capacitive sensing. Piezoelectric materials produce a change in voltage based on how much pressure is applied. Thus, incoming acoustic waves cause a change of pressure, which results in a change in voltage that can be measured and processed. Vesper believes that its single layer piezoelectric cantilever structure has many advantages relative to the dual or triple membrane structures used for capacitive devices. Specifically, it has indicated that its microphones have significantly higher signal-to-noise ratios enabling clearer sound capture at much greater distances. Vesper also indicates that its microphones have very little variation from mic to mic, making them well suited for multi- microphone arrays, which are increasingly common. They also indicate that it is very robust and naturally waterproof. Vesper has stated that it plans to begin commercial shipments in the first quarter of 2016. Its investors include Accomplice and unannounced strategic partners.
Page 48 “Happiness has many roots, but none more important than security.” - Edward Stettinius, Jr. Introduction The previously discussed HMI technologies were primarily for enhancing the user experience by making it easier for a user to control an application or device without using a conventional keyboard/mouse. Another aspect of HMI technologies, however, is security. Remembering and having to manually type in passwords is cumbersome and has security risks. In addition, it is impractical for many small form factor devices. As a result, there is growing interest in utilizing biometric interface technologies for security purposes. Some examples include fingerprint sensors, voice authentication technology, and facial/eye recognition systems. This chapter briefly discusses some of these technologies. Fingerprint Recognition Fingerprint readers enable users to log onto a device or application by swiping or placing a finger on a sensor. The user will have her fingerprint read a few times during set-up and can then subsequently gain access by placing her finger on (or swiping across) the fingerprint sensor. Historically, the two general types of fingerprint sensors were area sensors (in which the user places a finger motionless on a large sensor array) and swipe sensors (in which the user swipes a finger across the sensor and the images of each portion of the finger are pieced together by the computer). The sensor array in a swipe sensor is generally significantly smaller than that of a traditional area sensor, resulting in a much smaller footprint as well as lower cost for a swipe sensor, since the sensor array’s cost is typically proportional to its size. The disadvantage of a swipe sensor is that it generally is not as robust as an area sensor since multiple images of the finger have to be pieced together and users swipe their fingers at a variety of different speeds, which can make it more difficult to capture fingerprints, resulting in more false negatives and false positives. For portable devices, however, it had generally been viewed that the size and cost advantages of swipe sensors outweighed other factors and most of the fingerprint sensors that had been implemented in notebook PCs and mobile phones for several years were swipe sensors. Chapter 6 – Security Biometrics
Page 49 However, the iPhone 5s introduced Apple’s Touch ID sensor which is small enough to fit within the Home button. Unlike traditional area sensors that are relatively large so the user’s entire fingertip can fit, Apple’s “touch sensor” solution reads only a portion of the finger. That is, during set-up the user’s fingerprint is read in a number of different positions and subsequently the system can identify the user from just a small portion of the fingerprint. The steel ring around the home button detects a finger and activates the sensor. The iPhone 6s incorporates a similar sensor but it operates slightly faster. With the Galaxy 6, Samsung introduced a similar home button sensor into its smartphone. A variety of different technologies have been used to capture fingerprint images. Some examples include: thermal (the ridges of a finger contact the sensor and have a higher measured temperature than the valleys of the finger which can be used to create an image), optical imaging (the finger is illuminated and an image sensor captures fingerprint images), ultrasound (very high frequency sound waves penetrate the epidermal layer of skin and the reflected signal is captured), pressure (pressure from ridges of a finger is used to create an image), passive capacitive (the dermal layer of the finger and the sensor array act as parallel plate capacitors separated by the epidermal layer; the measured capacitance varies between the ridges and valleys of the finger), active capacitance (similar to passive capacitance but a small voltage is applied to the finger creating an electric field related to the pattern of ridges and valleys in the dermal layer), and RF (an RF signal is applied to the user’s finger and the resulting signal which is changed based on the fingerprint patterns is then read by a detector). Although each type of fingerprint sensor technology has been commercialized, many of them are not well suited for the consumer markets (e.g., not robust enough, too large, too expensive, etc.). Optical fingerprint sensors, for example, are commonplace for conventional stand-alone fingerprint readers, but have not historically gained traction in mobile devices due to size and power requirements. In general, one of the biggest challenges is the significant variations in types of fingers and environmental factors (e.g., finger may be dirty, clammy, dry, etc.), and with swipe sensors there is significant variation in how users swipe their fingers. Most of the fingerprint sensors implemented in mobile consumer devices utilize some type of capacitive or RF technology, which can be implemented at relatively low cost in small form factors. In addition, both of these technologies make measurements based on the dermal layer of skin, rather than just the external epidermal layer (which is more prone to changes based on external factors such as whether the finger is dry, wet, or has lotion or other materials on it). For example, Apple’s Touch ID and sensors from Fingerprints utilize capacitive technologies and sensors from Validity (now part of Synaptics) utilize RF technologies. AuthenTec had historically used RF but UPEK (which it acquired) utilized capacitive technologies.
Page 50 With consumers increasingly using smartphones for purchases and financial transactions, fingerprint sensing is increasingly used not just for accessing the phone but to validate identity for certain types of transactions and mobile payment systems. There has been significant consolidation of fingerprint sensor companies in recent years (Synaptics acquiring Validity, AuthenTec acquiring UPEK, Apple acquiring AuthenTec). These acquisitions are discussed in more detail below. Select Security Fingerprint Sensor M&A Transactions There have been a number of fingerprint/biometric acquisitions in the general security market. For example, Francisco Partners acquired Cross Match Technology, which later merged with DigitalPersona (which had acquired Identity Stream), and Assa Abloy (HID) acquired Lumidigm. However, those companies tend to focus on markets such as government, financial, retail, defense, and law enforcement. Below are a few specifically related to the consumer markets.  Apple/Privaris’ Patents – It was reported by CNN that Apple acquired three patents from Privaris in 2012 and more than twenty additional patents in 2014. Privaris developed a variety of portable fingerprint sensor solutions but its patents covered a variety of fingerprint sensor and touch screen technologies.  Synaptics/Validity – In October 2013, Synaptics announced that it agreed to acquire Validity for approximately $92.5 million in stock and cash plus an earn-out of $162.5 million for a total potential consideration of up to $255 million (per Capital IQ). Validity was a privately-held fingerprint sensor solutions company. Its solutions were incorporated into a number of notebooks, tablets, and phones.  IDEX/PicoField – In September 2013, IDEX announced an agreement to acquire the assets and intellectual property of PicoField Technologies, which was focused on touch fingerprint sensor design and development tailored for consumer devices.  Apple/AuthenTec (which acquired UPEK) – In July 2012, Apple announced that it was acquiring AuthenTec, a small cap public company focused primarily on fingerprint sensor solutions, for $356 million, which was about a 58% premium to AuthenTec’s stock price at the time of the announcement (per the 451 Group). AuthenTec had about $70 million of revenue in 2011. In 2010, AuthenTec had acquired UPEK, another major fingerprint sensor company which was originally spun out of STMicroelectronics.
Page 51 Facial Recognition With facial recognition, images of a person’s face are taken via an image sensor. When the user wants to log on later, the image of the person’s face is compared against the set-up pictures using a variety of complex algorithms to determine if it is the same person, in which case access is granted. Typically, there are certain key parameters the system focuses on to help match images (e.g., distance between the eyes, width of the nose, shape of the cheekbone, etc.). The incremental cost of this technology can be relatively low assuming a device already has a standard image sensor. However, accurate facial recognition is very challenging, since a person’s face can change quite a bit from day or day (e.g., shaved/not shaved, before or after a haircut, bad hair day, smiling/frowning, tan/not tan, etc.). In addition, room lighting can have a significant impact on facial images so features will vary based on ambient lighting. Also, the technology has to adjust for the fact that a person’s face may be at varying distances and angles from the camera. All of this makes facial recognition challenging, especially for mobile devices with limited processing power. In many mobile device implementations, facial data is transmitted over the network to servers that perform the actual face matching and the results are sent back to the device. While facial recognition has been used extensively in law enforcement agencies, it has had only moderate traction in consumer devices. Google introduced Face Unlock in Android 4.0 (Ice Cream Sandwich) in 2011. Face Unlock can be used as an alternative to a screen lock PIN. A couple of initial criticisms of Face Unlock were that there was a high level of false positives (i.e., someone with a face that was somewhat similar to the phone’s owner could get past security) and that photos of a person could be used to gain access. In subsequent Android updates, changes were made to improve the technology and to require the user to blink to gain access (so that static photos won’t work). However, the percentage of users actively utilizing facial recognition to unlock their phones is still thought to be relatively low. Facial recognition in general was expected to get a boost when Google first introduced Google Glass a few years ago as there had been reports that it would have a feature enabling Glass wearers to identify people (and bring up information about the person) just by looking at them. However, a congressional committee wrote an open letter to Google expressing privacy concerns if Glass included facial recognition capabilities. Subsequently, Google prohibited the use of facial recognition (for Glass) for identifying anyone other than the user.
Page 52 Although we included facial recognition in the chapter on security, there is also significant interest in facial recognition for other applications. For example, facial recognition can be used to identify people to provide user preferences. A smart TV could identify which person is watching and change settings based on that (e.g., bring up that person’s favorite types of shows). An automobile could determine who is driving and make seat, mirror, and entertainment adjustments based on that. Another major area of interest for facial recognition is social media, such that images or videos of a person can be identified and tagged. For example, facial recognition technologies have been used by both Facebook and Google for searching to find photos and videos of a person. As a specific example, when a Google+ user turns on the “Find my Face” feature, photos and videos that the user tags are utilized to develop a model of the user’s face. When someone else looks at a photo or video with a face that appears to match the user’s face, they receive a suggestion to tag the user. As a result, there have been a number of facial recognition acquisitions related to photo tagging. Select Facial Recognition M&A Acquisitions There have been a number of consumer-related facial recognition acquisitions, although several have been more focused on facial recognition for social media type applications rather than security. Some examples include:  Dropbox/KBVT – In September 2014, Dropbox acquired Kriegman-Belhumeur Vision Technology (KBVT), which focused on advanced facial recognition software and had two professors considered experts in the field (Peter Belhumeur and David Kriegman).  Google/Viewdle and PittPatt – In October 2012, Google (through Motorola Mobility) acquired Viewdle, a Ukrainian company focused on facial recognition technologies for applications such as photo tagging. In addition to facial recognition, Viewdle was also diversifying into pattern recognition (gestures, etc.). The deal was reportedly in the $40 to $45 million range (per the 451 Group). In 2011, Google acquired Pittsburgh Pattern Recognition (PittPatt). PittPatt was a start-up that was formed from a project at Carnegie Mellon University and developed facial recognition technology to match people’s faces, including face detection, face tracking, and face recognition.  Facebook/Face.com – In June 2012, Facebook acquired Face.com, an Israeli start- up focused on facial recognition technologies for photo-tagging and related applications. Its solutions helped developers build applications that could identify people from photos, although it stopped providing its facial recognition APIs after the Facebook acquisition. The deal value was reportedly approximately $60 million (per Capital IQ).
Page 53  Apple/Polar Rose – In 2010, Apple reportedly acquired Polar Rose for approximately $29 million (per TechCrunch). Polar Rose used facial recognition for auto-tagging of photos for photo sharing sites. Eye Recognition Another type of security technology is eye recognition, in which images of the user’s eye are captured during set-up and subsequently compared to images of the eye of the person trying to log in. If the images match, the user is granted access. There are two general types of eye recognition technologies – iris recognition and retina recognition. With iris recognition, images of a person’s eye are captured (typically near- infrared light is used to illuminate the eye) and the unique patterns of the iris are used for matching purposes. Retina scans involve shining a bright light into the pupil and capturing images of blood vessels in the back of the eye. The term retina scan is sometimes incorrectly used to refer to all eye recognition technologies, but iris recognition is by far the most commonly used and less invasive technology. An advantage of iris recognition compared to other security technologies is that unlike a person’s facial features, for example, which can change from day to day, the patterns in a person’s eye vary little over time and so matching is generally easier. A negative is that many people don’t like the idea of having their eyes scanned, so making it as unobtrusive as possible is critical (i.e., having to move an eye close to the camera is generally not desirable). Too strong a flash of infrared light can damage the retina so weaker flashes are used but that typically requires the eye to be relatively close to the camera. Iris recognition has been used for many years in a variety of general security applications (high security building access, etc.) but there has been interest in incorporating it into mobile devices. The goal is to make it effective at a distance, such that the user doesn’t have to move closer to the device than during normal usage. Select Eye Recognition M&A Transactions There have been a number of eye recognition acquisitions for government and enterprise applications but only a few related to consumer markets.  Voxx/EyeLock – In August 2015, Voxx International announced that it entered into an agreement to acquire the assets and IP and a controlling interest in EyeLock, which had developed iris recognition security technology for government, business, and consumer markets. EyeLock had more than 70 patents granted and pending. The myris iris authentication device is available at more than 1,000 retail locations across the U.S. The price wasn’t announced but financial documents later filed by Voxx indicate the deal was about $20 million with a $15.5 million outlay at closing.
Page 54  Tessera (FotoNation)/Smart Sensors – In February 2015, Tessera’s FotoNation unit announced the acquisition of Smart Sensors and indicated the transaction closed in late 2014. Smart Sensors provided advanced iris recognition technology and was based in Bath, England. Voice Authentication As many devices now include a microphone for voice communications, there is interest in utilizing voice authentication technology in which a device can recognize a person’s voice for security access. With voice authentication, typically a person will pick a specific pass phrase and repeat the phrase a few times in set-up mode. Subsequently, the security system then requires the user to say the specific phrase and can recognize if it is the user’s voice (another person saying the phrase would not gain access). An advantage of voice authentication is that the incremental cost can be minimal; assuming the device already has a microphone, as it doesn’t require additional hardware such as a fingerprint sensor. It is also well suited for wearable devices that are relatively small. One of the issues that some voice recognition solutions have is that it requires significant processing power which can’t be supported by the device, so the voice information is sent over the network for analysis and a server on the other end performs the computations and lets the device know if access should be granted or not. This can result in latency issues and also does not work if there is no network access (zero bars). Some historical solutions also had issues if there was background noise and had too many false negatives. Select Voice Authentication M&A Transactions There have been a number of voice authentication technology acquisitions, although most of the targets were focused on voice authentication for institutions, rather than for consumer devices. Some voice authentication M&A transactions have included:  Cirrus Logic/KIVOX – In the third quarter of 2015, AGNITO announced that it divested its KIVOX mobile assets to Cirrus Logic and that the deal closed in July 2015. The sale included the KIVOX voice biometrics solutions for mobile applications including an SDK and IP. The technology allows for the voice biometrics engine to be deployed on mobile devices for security and authentication purposes and provides for on-device voice matching.  Verint/Victrio – In October 2013, Verint announced the acquisition of Victrio which had developed “passive” voice biometrics solution with predictive analysis that can accurately detect fraudsters and authenticate customers without caller interruption.
Page 55  VoiceTrust/Perceive Solutions – In February 2012, VoiceTrust acquired the Montreal based voice technology firm Perceive Solutions. Perceive had developed voice biometrics solutions that can identify and authenticate speakers in any language with free form speech.  Nuance/PerSay – In 2010, Nuance acquired PerSay. PerSay was an Israeli start-up focused on the voice biometrics markets, for systems used in the banking, insurance, governments, and telecommunications industries. Many Other Biometric Security Technologies There are a variety of other biometric security technologies that have been developed. As an example, we have seen solutions in which the user can gain access through handwriting recognition (the user writes a word on the touch screen using a finger), gesture movements, and heart/pulse measurements. There is also growing interest in multi-factor authentication, in which two or more security factors are used (e.g., voice and facial recognition) to provide an even higher level of security. Select Biometric Security Technology Companies The following are a few examples of biometric security companies. There are a variety of companies that have developed biometric solutions for government, military, corporate, and telephony applications, but we included companies specifically focused on mobile device applications. Although we primarily highlight private companies in this report, we included a couple of small cap European public companies. Porticus Technology Porticus is a small privately-held company that has developed unique voice authentication and multi-factor solutions. Based on more than a decade of research in voice forensics and biometrics, Porticus has developed its VoiceKeyID voice authentication solution, which can be used for unlocking a device or for enabling access to certain applications (e.g., financial transactions, etc.). A user decides on a pass phrase and repeats it three times at set up, and then subsequently speaks the phrase to gain security access. The Porticus solution is based on voice phonemes, so it is language independent (it doesn’t try to translate the spoken pass phrase but just confirms that the spoken phrase sounds the same). The Porticus solution was specifically developed for mobile handsets and operates entirely in the device without requiring any network access (eliminating latency problems and issues related to not having network access). It was also designed to have low power consumption and requires relatively little processing power. The software can be incorporated into a chip or implemented through the operating system or an app.
Page 56 Although the basic solution provides two factor authentication (user’s voice, pass phrase), it supports four factor authentication (including the device’s ID and the device’s location). Thus, security could potentially be granted only if the user has the correct mobile phone, is in certain locations, knows the pass phrase, and has a voice match. This provides significantly more security than simply typing in a password. Porticus has a licensing business model. The company has a number of patents around its voice authentication technology, as well as multi-factor security incorporating voice authentication in general. VoiceKeyID is available for iOS and Android OS platforms, and runs on a variety of chips including Qualcomm’s SnapDragon. The Company is headquartered in Wellesley, Massachusetts and maintains research and development facilities in Vilnius, Lithuania. Vkansee Vkansee has developed optical-based fingerprint sensor solutions for mobile devices. Optical-based fingerprint sensors are commonplace for stationary applications (e.g., building access) as it can be highly accurate, but it was not historically usable in mobile applications (handsets, notebook PCs, etc.) as optical readers were large and consumed significant power. However, Vkansee has developed an optical-based solution that they believe resolves these issues and provides many advantages relative to the fingerprint sensors currently used. Specifically, they indicate that their fingerprint sensors have higher resolution (2000 dpi, more than 4X most alternatives), better accuracy, smaller size (1.5mm thin), and incorporate a variety of differentiated anti-spoofing technologies. Their sensors also consume less than 0.1W and have a detection feature so it is asleep when not in use. In addition, the solution operates under-glass, which can provide many advantages relative to other technologies that require the glass to be etched to implement the fingerprint sensor. Vkansee is based in New York with an office in China. Fingerprint Cards AB Fingerprint Cards (FPC) sells a variety of area, swipe and touch fingerprint sensors which all utilize capacitive technology. FPC’s sensors contain tens of thousands of small capacitive plates, each with their own electrical circuit embedded in the chip. When a finger is placed on the sensor electrical charges are generated, creating a pattern between the finger’s ridges or valleys and the sensor’s plates. Using these charges, the sensor measures the capacitance pattern across the surface. The measured values are digitized and processed. FPC’s high sensitive pixel amplifier (HSPA) technology allows each pixel element in the sensor to detect very weak signals, improving image quality. FPC has announced a variety of design wins over the past year including several smartphones from Huawei and ZTE. It recently announced solutions targeting the automotive market. FPC has a variety of patents related to its own fingerprint technologies, but also acquired more than 100 patents in 2013 related to wireless technology. The company is headquartered in Sweden. Its stock is listed on the Stockholm exchange.
Page 57 Next Biometrics Next Biometrics provides fingerprint sensor solutions. Its solutions utilize “active thermal” sensing (rather than capacitive or RF) that can be implemented on a low temperature polysilicon process which is also commonly used to produce displays. The active thermal technology applies a short heat pulse to each sensor pixel and the ridges of the finger produce a different temperature differential from the valleys. By measuring the heat response of the user’s finger and analyzing the data, Next’s solution can create a fingerprint image. Because its solution uses a low cost polysilicon process rather than a silicon fab, the company believes that its product can be produced at a significantly lower cost relative to traditional fingerprint sensors. Next has focused on area sensors which generally have the best performance but historically were not cost effective for consumer devices. Target markets include smartphones, tablets, notebooks, traditional security, and other segments that are not currently addressed (“Next-enabled”). The company has indicated that its sensors are designed into a variety of 2016 Dell notebooks and tablets, and it has announced a number of other customers and distributors. Next had an IPO in mid-2014 (Oslo Axess) and is based in Norway. Animetrics Animetrics is a developer of 2-D-to-3-D face recognition and face creation solutions. The Animetrics product family delivers facial recognition identity solutions using technologies that are capable of addressing high levels of variation in pose and lighting conditions. The Animetrics technology takes a 2-D image of a face and generates a 3-D model for accurate verification and identification. This 3-D model can also be rendered as a lifelike avatar. Animetrics has historically targeted military and government type applications (identifying suspects or missing people) running on computer systems, and has achieved traction in these markets. However, more recently it has introduced solutions for smartphones. Specifically, the FaceR MobileID solution is designed to help recognize and match faces using a mobile phone, and the FaceR CredentialME is an advanced facial recognition security application. These solutions can run on iPhone or Android smartphones and leverage Animetrics’ core FACEngine facial recognition technology that converts 2-D to 3-D images. Animetrics has several patents and is based in New Hampshire.
Page 58 “I believe in a long, prolonged, derangement of the senses … in order to obtain the unknown.” - Jim Morrison Introduction The number of sensors in mobile devices continues to increase. Many devices now have a variety of positioning sensors (accelerometers, gyroscopes, magnetometers, etc.) as well as a number of interface sensors (CMOS image sensors, microphones, etc.). Adding sensors to a device is relatively easy, but effectively utilizing all of the data to greatly enhance the performance or functionality of the device is more complex. In addition, many sensors require a significant amount of “behind the scenes” management (e.g., calibration). Sensor fusion refers to the idea of combining the data from a variety of different sensors and sources to enhance the overall performance of the product. Beyond basic sensor fusion, there is growing interest in making smartphones and devices “contextually aware” such that they provide users with relevant information based on all available data sources and this requires utilizing data from many sensors and a variety of other data sources, and interpreting the data to provide higher level information. A number of companies have developed sensor fusion related technologies including semiconductor chips and software/middleware solutions. Select Sensor Fusion Company M&A Transactions Two of the four sensor fusion companies we highlighted in the first edition of this report were acquired. These included:  InvenSense/Movea – In July 2014, InvenSense announced that it agreed to acquire Movea for $60.9 million, plus an additional $13 million of potential contingent consideration. Movea was highlighted in the first edition of this report and had developed a broad range of middleware solutions addressing the sensor fusion/motion processing markets. Its technology bridges the gap between a broad range of sensors and applications, and includes technologies related to gesture- based control, activity monitoring, and indoor navigation. The company licensed its technology for products such as smart phones/mobile devices, smart TVs/set-top boxes, gesture-enabled remote controls, and sports/wellness products. Chapter 7 – Sensor Fusion and Middleware
Page 59  Audience (Knowles)/Sensor Platforms – In June 2014, Audience announced that it acquired Sensor Platforms for $61 million (Audience was later acquired by Knowles). Sensor Platforms’ FreeMotion Library targeted sensor fusion and user context awareness for mobile applications. The Library provides the intelligence to combine and process data from sensors (e.g., accelerometers, barometers, gyroscopes, magnetometers) on mobile devices, and interpret that data into contextual information meaningful to application developers. The FreeMotion Library works with Android and Windows 8 operating systems, and can also be embedded in microcontrollers and/or smart sensors. Prior to the acquisition, Sensor Platforms announced collaborations with several major companies (Murata, Nvidia, etc.). Select Sensor Fusion Company Examples Apple has developed its own sensor fusion coprocessors for the iPhone and iPad. Its M7 coprocessor was included in the iPhone 5s. The M7 (based on ARM Cortex-M3) collects and processes data from a variety of sensors (accelerometers, gyroscopes and compasses), offloading this from the main A7 processor. The latest iPhone 6s includes an updated coprocessor (A9). As discussed in a later chapter, Microsoft’s HoloLens augmented reality product incorporates a “Holographic Processing Unit” which is a dedicated chip that performs a variety of tasks including many related to sensor processing and analysis. Examples of private companies addressing sensor fusion are given below. Hillcrest Labs Hillcrest Labs has developed a broad range of software and hardware solutions for motion- enabled products. The company’s Freespace MotionEngine software products include: Smart TV (for smart TVs, set-top boxes, and other connected TV devices), Mobile (for smartphones and tablets), Lite (in-air pointing for remote controls and mice), and Wear (low power activity tracking, sleep monitoring, context awareness, gesture control, and navigation activity tracking for wearable devices). These solutions provide a variety of sensor fusion and calibration algorithms, and provide application ready data. In addition, Hillcrest has developed a sensor hub software stack (SH-1 Sensor Hub) that is optimized for and can be pre-integrated into the Atmel ARM-based SAMD 20 microcontroller. This provides a complete “always on” chip level solution for implementing sensor fusion. The company also has modules and white-label products (pointer, remote). Hillcrest has an IP portfolio that includes over 130 granted and 235 filed patents exclusively owned by Hillcrest Labs. In addition it has a one-way license of a portfolio of motion patents from Technicolor. Some recent design wins announced by Hillcrest have included Bosch (head mounted displays), Lenovo (smart TVs), Coolpad (smartphone), Skyworth (smart TVs), Alcatel (smartwatch), and ZTE (smartphones). Hillcrest is based in Rockville, Maryland and is funded by NEA, AllianceBernstein, Columbia Capital, and Grotech Ventures.
Page 60 PNI Sensor Corporation PNI has been an innovative sensor company since the 1980s. For many years, PNI was primarily known for its magneto-inductive sensors and has generally been recognized for being a technology leader in magnetometer/compass chips. The company continues to sell a variety of geomagnetic compass chips and compass modules. For example, PNI indicated that its RM3100 solution delivers more than 20x higher performance than existing MEMS or Hall Effect geomagnetic sensors. PNI’s digital compass solutions have been used in a variety of products and markets. While the company was primarily known for geomagnetic products, more recently PNI introduced Sensor Fusion chips and algorithms. Most smart mobile devices now incorporate a variety of sensors including an accelerometer (to detect the orientation of the phone), gyroscope (to measure tilt/rotation), a digital compass (to determine absolute direction as it can identify which way is North), and sometimes a barometer (to determine altitude which can improve GPS tracking). However, managing all of these sensors is complicated and includes a variety of “behind the scenes” activities such as calibrating the sensors and filtering the data. This often eats up a significant portion of the main processor’s cycles, resulting in substantial power consumption and reduced device performance. The PNI SENtral Fusion Sensor chip is designed from scratch to manage other sensors including gyros, accelerometers, and magnetometers. This provides a variety of advantages. For example, PNI indicates that its SENtral chip can perform sensor fusion/management at less than 1% of the power consumption of a general microprocessor, which can help significantly extend battery life. Using a proprietary “constant calibration” technology and filter algorithms, PNI indicates that Sentral can also significantly improve accuracy and reliability and reduce the need for complex algorithm development, enabling easy implementation for motion processing. Although Sentral is a chip-based solution, importantly PNI can also implement many of its sensor fusion algorithms in software. PNI has also introduced Sentral-A sensor fusion solution for Android-based products. Alternative approaches require either using a general MCU running a variety of 3rd party software or running a variety of software on the applications processor. PNI indicates that its approach enables dramatically less power consumption (typically <10% of alternatives). The new Sentral-A2 supports the latest versions of Android (Lollipop and Marshmallow). PNI recently announced SENtrace, a coprocessor for wearables providing highly accurate, ultra-low power pedestrian tracking for indoor locations and dense urban canyons. The solution enables devices to track location within one meter per 100 meters traveled while cutting power by 10-fold (utilizing dead reckoning and other advanced algorithms, it greatly reduces the number of times GPS readings need to be taken). Applications range from fitness to tracking children. PNI’s products are used in a variety of consumer devices, including the Wii U Game Pad, the Sony smart watch, as well as in a variety of robotic, navigation and automotive applications. The company is based in Santa Rosa, CA.
Page 61 “The brain is a wonderful organ; it starts working the moment you get up in the morning and does not stop until you get into the office.” - Robert Frost Introduction With growing interest in wearable devices and in utilizing smartphones for body/health monitoring, dozens of companies have developed solutions that can monitor signals from the human body such as heart rate, temperature, and pulse rate for health/wellness/medical type applications. There have even been interesting developments in using brainwaves for controlling applications. Over time, the number of biometric sensors in smartphones and wearable devices is anticipated to significantly increase. This section briefly discusses some of these biometric sensor technologies. Heart Sensors (ECG Sensors) Electrocardiogram (ECG) sensors measure electrical activity of the heart. Heart muscles have negative charges (membrane potential) at rest and during each heartbeat the charge shifts towards zero (depolarization) and then back again. By measuring and analyzing the resulting electrical signals over a period of time, an ECG sensor can determine a variety of information about the heart (heart rate, abnormalities, etc.). ECG sensors are commonly used by hospitals, but there is growing interest in utilizing ECG technology for consumer applications, such as tracking health/wellness from a smartphone. Neuro-technology (EEG Sensors) There is interest in measuring and using brain wave activity. This typically involves wearing a headset that includes an electroencephalogram (EEG) sensor. The EEG signals from the brain are converted to digital and analyzed and can be used to determine if a person is focused/concentrating or not (for research/education applications) and if there may be wellness issues (for health reasons). It can also be used to sense a person’s mood. In some cases, EEG sensors can even be used to make simple control selections. For example, by measuring a user’s EEG signals when he is thinking about “jumping” and measuring it when he is thinking about “running”, an application can potentially allow the user to control a game character that has to run and periodically jump just by thinking, without using any other type of control or body movement. Typically, this requires some amount of set-up time for each user, as brain wave patterns vary from person to person in different circumstances. However, this offers the potential for some “science fiction like” control by thought alone. Chapter 8 – Brain, Heart and Other Biometric Sensors
Page 62 Pulse Rate, Blood Pressure, and Many Other Biometric Sensors There is growing interest in incorporating a variety of biometric sensors into smartphones and wearable devices for health/wellness applications. Some vital signs that can now be measured include heart rate, pulse rate, temperature, respiration rate, and blood-alcohol level (although in some cases the accuracy of these mobile sensor products has not been quite as good as desired). Beyond these, there are sensors that can analyze sweat and skin for health reasons. Although more difficult to accurately measure than pulse or heart rate, there is strong interest in being able to measure blood pressure, as high blood pressure is a key early warning sign for many potential health issues (heart disease, stroke, kidney problems), but relatively few people measure their blood pressure on a regular basis. This would likely change if the technology were available in smartphones. The ability to monitor these vital signs also provides more data for doctors to diagnose patients (as opposed to just measuring once while in their office) and key data could easily be forwarded to physicians, enabling easier remote monitoring. Analyzing this type of data across large numbers of users could greatly improve the ability to predict health issues. There is also growing interest in sensors that can measure parameters such as pollution and pollen levels to alert the user so he or she can make choices that take those things into account. In general, it is believed that being able to easily measure key vital signs and parameters could help catch potential health issues early and improve overall health levels across the world. Growing Number of Biometric Sensors in Smartphones While all of the previously mentioned vital signs can be measured using bulky medical equipment in hospitals and doctors’ offices, the challenge is to implement the solutions in mobile devices or wearables, which are relatively small and require very low power consumption. Some companies have developed external add-on sensor solutions that can attach to a tablet or smartphone, but the general target is to make a sensor small enough so that it can eventually be integrated into a mobile phone. With MEMS and other advanced sensor technologies, this is becoming increasingly achievable and the number of sensors in portable devices is expected to continue to sharply increase over time. Select Biometric Solution Companies A variety of companies have announced sensors, software or wearable devices to measure health-related parameters and many of the new smart watches incorporate some vital sign sensors. In December 2015, Samsung indicated that it began selling a new processor specifically targeting health-focused wearable products and that it can take multiple measurements (including body fat, skin temperature and heart rate). The following are some companies focused on biometric solutions:
Page 63 Leman Micro Devices Leman Micro Devices has developed advanced sensor solutions that can enable a smartphone to measure a variety of health parameters. Specifically, the company indicates that its “Smartphone Blood Pressure System (SBPS),” along with its Elemdy App and device driver software, enables smartphones to measure blood pressure with medical accuracy. It also enables accurate measurement of blood oxygen, heart rate, respiration rate, and body temperature, and includes a one-lead electrocardiogram (ECG). As high blood pressure is one of the most significant tell-tale signs of strokes and heart disease, Leman believes this could have a substantial impact on improving health and wellness around the world. While its SBPS module could be an add-on to a smart phone, it is designed to be small enough such that it could be directly integrated into smartphones, transforming phones into wellness devices. The company believes that just as cameras and GPS were initially a novelty in smartphones and later became a standard feature, health care functions such as blood pressure monitoring will also become commonplace. The company anticipates commercial shipments in 2016. Leman was founded in 2010 and is based in Lausanne, Switzerland. It has been funded by angel investors, VCs, and two unannounced major players in the smartphone sector. NeuroSky NeuroSky has introduced a number of solutions targeting both EEG sensors (brainwaves) and ECG sensors (heart rate). It has developed integrated chips, modules, and software/algorithms addressing these markets, and also offers a complete headset. The NeuroSky CardioChip is a system-on-a-chip (SoC) solution addressing the health and wellness market. It can be used to monitor ECG, heart rate, and heart rate variability. The chip integrates a low noise amplifier, a 16 bit ADC, sensor-off detection, hardwired DSP blocks, and a systems management unit. It targets bio-signals ranging from microvolts to millivolts. NeuroSky also provides a variety of complex algorithms to help determine heart rate, respiration rate, stress level, and other parameters based on the readings. NeuroSky has been a pioneer in developing solutions for the EEG/brainwave monitoring market. Its ThinkGear AM (TGAM) EEG sensor module senses the signals from the brain, filters out extraneous signals, and converts the results to digital for a variety of applications. The module includes NeuroSky’s own semiconductor chip which is optimized for this application. The chip includes A/D conversion, amplification off head detection, noise filtering for both EMG and 50/60Hz AC powerline interference, and NeuroSky’s eSense brainwave pattern technology. NeuroSky also provides advanced algorithms that can be used to determine mental activity/meditation, familiarity, and other parameters. In addition, NeuroSky offers a complete headset solution. NeuroSky indicates that its TGAM solution powers over 1 million consumer EEG devices around the globe. It recently announced that it is working with ARM on secure remote patient monitoring solutions.
Page 64 In November 2013, NeuroSky announced that it received an investment from SoftBank to accelerate the development of mobile health/wellness solutions. NeuroSky was founded in 2006, and is headquartered in San Jose, California with offices in Asia and Europe. MC10 MC10 is a pioneer in flexible electronics, with a focus on biosensor applications. One of the major issues with bio-sensing is that traditional electronics are relatively bulky. Although a silicon die is very thin, traditional chip packaging (to protect the silicon chips) and printed circuit boards (to interconnect different chips) are thick and rigid. Ideally biosensors could be applied directly onto the skin without a person even noticing it while it is on. MC10 has developed technologies that allow extremely thin, flexible, conformal, transparent devices that can stretch, bend, and twist while on a person’s skin. This has the potential to greatly accelerate the penetration of biosensors into several markets. The MC10 solutions generally incorporate sensors to measure important parameters (temperature, heart rate, hydration level, UV levels, etc.) and then wirelessly transmit the data to a smartphone or other device. Because of the thinness of the MC10 solution, it can easily be incorporated into wrist bands or other accessory products, but even more significantly can be directly stamped onto the skin itself. MC10 developed a prototype called BioStamp, which can be stamped directly onto the skin and is then sealed with a spray-on bandage. The sensors can detect parameters such as body temperature, hydration levels, and UV exposure. This information can then be uploaded to a smartphone or sent to a doctor. The patch uses a thin film battery. The imprint can last for several days, withstanding stretching, flexing and even showering. In January 2016, the company unveiled its BioStamp Research Connect System, a system that provides researchers with the ability to gather physiological data through the use of its BioStamp sensors and a cloud-based software system. MC10 has also utilized its technology for a variety of other markets (e.g., it partnered with Reebok to develop CheckLight, a head impact indicator for contact sports). At CES, L'Oreal unveiled “My UV Patch” (using MC10’s technology), the first-ever stretchable skin sensor designed to monitor UV exposure and help consumers educate themselves about sun protection. MC10 believes there are a large number of additional applications for its flexible electronics, well beyond bio-sensing. MC10’s investors include North Bridge Venture Partners, Braemar Energy Ventures, Windham Venture Partners, Aberdare Ventures, Osage University Partners, Terawatt Ventures, Medtronic, Mitsui Ventures, and several undisclosed strategic investors. MC10 is headquartered in Lexington, MA.
Page 65 Cambridge CMOS Sensors Cambridge CMOS Sensors (CCMOSS) has developed advanced gas sensor solutions for consumer applications. The company’s MEMS micro-hotplate technology enables metal oxide (MOX) gas sensors that are very small with very low power consumption and fast response times. CCMOSS indicates that its solution is about 97% lower in power consumption relative to conventional sensors and many of its sensors come in small 2x3mm packages. The company also indicates its solutions are more stable with less drift compared to traditional sensors. Applications include measuring carbon monoxide, ethanol (to determine blood alcohol level of a person), and general indoor air quality (volatile organic compounds, particulates, microbial contaminants, etc.). The company plans to introduce a variety of additional sensors over time using its core technologies. Its solutions can be used with smartphones, tablets, and IoT devices. The resistive micro-hotplates are fabricated using a silicon dioxide membrane and include an embedded tungsten heater acting as a heating element for the MOX based sensing material. The micro-hotplate can be used to heat the MOX material to up to 450°C and the electrical resistance of the MOX sensing material can be monitored to detect the target gas (e.g., if the sensing material reacts with carbon monoxide, the resistance decreases; the change in resistance can then be determined by the circuitry enabling the detection of carbon monoxide). The company was founded in 2008 and is based in Cambridge, UK. Xsensio Xsensio is an early stage Switzerland-based company that is developing an ultra-small sensing solution that is designed to sense electrolytes, metabolites, small molecules and proteins on the surface on a person’s skin. The company indicates that compared to existing solutions which rely on electrochemistry, its sensing technology platform is 10,000x smaller in size, and can achieve attomolar concentrations. Xsensio is a spinoff of the Nanolab at the Swiss Institute of Technology (EPFL) in Lausanne, Switzerland. Si-Ware Si-Ware is a mixed-signal ASIC company but it has developed its own family of infrared (FT-IR) spectrometers using MEMS technology. Si-Ware believes its solution provides the same functionality as standard bench top FT-IR spectrometer instruments, but with dramatically smaller size and cost. Spectrometers can quantify and identify materials based on their spectral response. There are many applications for the sensor ranging from helping farmers analyze soil to numerous industrial uses, but as cost is reduced with higher volumes, it could have a number of consumer applications. Si-Ware is based in Egypt.
Page 66 HMicro Monitoring patient vital signs today often requires connecting numerous cables and wires to patients in a hospital which makes it difficult for them to move or sleep and hinders recovery. As such, there is growing interest in battery powered sensors that attach to the body, measure key vital signs, and then wirelessly transmit the data for further processing, monitoring, and analysis. This has many benefits but one of the historical challenges has been the wireless technology, which must be extremely robust (even with hundreds of patients and other wireless devices around) and must be highly power efficient (since the sensors often operate on a tiny battery). Traditional WiFi is generally viewed as too power hungry and other conventional wireless solutions have issues that make them ill-suited for this type of application. To address this, HMicro has developed a wireless solution that is well-suited for medical and bio-sensing applications. Its REACH SoC is a highly integrated chip that incorporates three different radios (its own ultra-low power Ultra-Wideband solution, a medical band radio, and WiFi) as well as a smart power management unit, a sensor subsystem with digital interfaces, and an application processor that customers can program. The triple radio provides significant redundancy and the chip intelligently switches among the radios to optimize efficiency while ensuring link integrity. Leveraging its SoC, the company has also developed module level solutions to enable easier customer integration. HMicro believes its technology will enable many “wired” medical applications to go “wireless” over the next several years and can transform the industry. While initially targeting medical, HMicro believes it can address many applications (auto, audio streaming, consumer, etc.). HMicro announced that it completed a $5.5 million capital raise in October 2015. In November 2015, it announced a partnership with Flextronics and that it became a member of the Flextronics Lab IX Start-Up Program. Flextronics and HMicro are jointly working together on a variety of projects including small sensor modules and an ECG patch and adaptor. Investors include Seraph Group, Reddy Capital Partners, and X/Seed Capital Partners. HMicro is based in Fremont, CA. Toumaz Group Toumaz Group has two divisions. The Frontier Silicon portion of the business focuses on digital radio chips and connected audio and was formed when Toumaz acquired Frontier Silicon in August 2012. The Toumaz Sensium Healthcare division has developed ultra-low power wireless solutions for monitoring patient vital signs, addressing the previously noted need for advanced wireless solutions for medical sensor applications. Sensium Vitals is an ultra-low power wireless system that tracks vital signs of general care patients using light weight wearable single-use patches that monitor heart-rate, respiration rate and temperature, and communicate the data wirelessly to any Web enabled device or smartphone. Sensium Vitals is the first ultra-low power, wearable, wireless patient monitoring product to have received FDA (510k) clearance. Toumaz also indicates that it developed the first radio chip to enable the wireless connection of medical devices in compliance to the latest IEEE 802.15.6 Medical Body Area Network (MBAN) standard. Toumaz is a public company listed on the AIM exchange and is based in London.
Page 67 Linear Dimensions Linear Dimensions is a semiconductor and algorithm company that has developed a variety of solutions for bio-sensing related applications, including low power sensors. The company historically focused on developed analog and mixed-signal ASICs for customers but has increasingly developed a variety of its own innovative technologies. Its solutions can measure a broad range of vital signs (ECG, EKG, heart rate, pulse rate, etc.). The company indicates that its expertise in algorithms (including its advanced PEAL algorithm technology) provides significant advantages relative to other sensor solutions, which often work well in the lab but not when in motion (or require additional chips and sensors to compensate for motion). Linear indicates that its solutions remove noise and motion artifacts before they enter the system, rather than relying on additional power hungry devices to clean the signal up later. Some of its offerings include advanced photodetectors (which combined with its algorithms and VCSELs can measure a variety of vital signs); bio-impedance sensing solutions that measure ECG, IPG and EEG; bio-sensing reference designs; “lab on a chip” solutions; and a variety of precision analog and power management solutions. Linear Dimensions is based in Silicon Valley. Affective Computing/Emotion Analytics Affective computing is the concept of computing devices being able to recognize (and even simulate) human emotions. This is a type of biometric sensing, but typically, sensory input is used to determine the user’s emotional state. This could include optical imaging, voice input, or even brainwave patterns. By analyzing sensory data (the user’s facial expressions, body language and gestures, speech, vital signs, etc.), computing devices can determine a user’s mood and can use this information to better meet the user’s needs. For example, if a user is getting frustrated with a game, the device can adjust the difficulty level or provide some hints. If the user appears to be in a bad mood, it can play music to cheer the person up. Longer term, this should improve the ability of computers to interact with people, including in either virtual reality or augmented reality settings. Select Emotion Analytics M&A Transaction A recent example of an emotion analytics acquisition is given below:  Apple/Emotient – In January 2016, the Wall Street Journal reported that Apple acquired Emotient, an artificial intelligence company focused on analyzing faces to determine the user’s emotional state. According to the Journal, Emotient was unable to raise additional capital and its investors included Intel Capital.
Page 68 “Every man takes the limits of his own field of vision for the limits of the world.” - Arthur Schopenhauer Introduction Image sensors are increasingly pervasive in a broad range of mobile devices. These sensors are now being used for more than just taking conventional photos. Earlier chapters discussed some potential applications for image sensors such as gesture recognition, eye tracking, and facial/eye recognition, which typically rely on optical image sensors. However, there are a variety of others. For example, there is growing interest in image recognition in which devices can recognize objects within captured images. Beyond basic photographs, there is growing demand for higher quality images and video as well as 360 panoramic images. Analyzing incoming video data to extract key information is extremely computationally intensive requiring advancements in both hardware and software. To address this, some companies have developed chips optimized for these applications. This chapter discusses a variety of technologies related to image/vision processing, many of which are also relevant for virtual and augmented reality (discussed in the next chapter). Image Capture Sensors A critical part of any digital camera or video camera is the sensor that captures the incoming light and converts it to electrical signals that can be stored or processed. Historically the two main types of image sensors were CCDs (Charge Couple Devices) and CMOS image sensors. For a number of years, CCDs were considered to have much better image quality and CMOS image sensors were used only for lower end applications. However, during the past couple of decades CMOS image sensor quality has significantly improved while costs have been sharply reduced (as CMOS is also used for the vast majority of semiconductor chips). As a result, CMOS image sensors have become pervasive in a broad range of consumer applications. CMOS image sensors are semiconductor components that incorporate tiny photodetectors which convert incoming photons into electrical charge and ultimately into an electrical signal that can be digitized and processed to represent the captured images. Color filters over the photodetectors enable the capture of color and tiny micro-lenses improve the image quality. The number of pixels that a CMOS image sensor can capture has continued to increase as process geometries shrink (e.g., from 1 megapixel a number of years ago to 16 megapixels or more). However, as the number of pixels increase the size of each pixel (photodetector) tends to decrease, which limits the amount of light that is captured by each pixel which can reduce image quality, creating engineering challenges. Chapter 9 – Image and Vision Processing
Page 69 As image sensors are increasingly used for video, they must be able to capture many images per second (e.g., at least 30 frames per second, with growing interest in 45 and 60). To enable high quality images, there is demand for better quality sensors with high resolutions that can adjust to various lighting conditions and enable auto-focus and high dynamic range and other advanced features. Some of the major suppliers of CMOS image sensors include Sony, OmniVision, Samsung, ON Semiconductor, and Canon. An emerging area of interest for image sensing is light field cameras (plenoptic camera). These cameras capture a variety of information about a scene beyond what a normal RGB camera obtains (e.g., intensity of light, the direction the light rays are moving, etc.). This can enable functions such as refocusing an image after it has been captured. It also has advantages for creating 3-D images. A variety of light field cameras have been developed using different technologies (e.g., arrays of lenses, arrays of cameras, etc.), but they historically have had limitations and were very expensive. In November 2015, LYTRO (a pioneer in light field cameras) announced its Immerge family of light field cameras specifically targeting the high end professional market for capturing video for virtual reality applications. Over time, these types of cameras could become more commonplace if the technology improves and cost is reduced. Another area of interest is capturing 3-D images (rather than traditional 2-D images). This requires obtaining depth information about objects in the image. Technologies like time of flight (ToF) image sensors (in which the distance of an object can be determined by the time it takes light to travel from the camera to the object and back) were previously discussed in the chapter on gesture recognition. While gesture recognition is one important application for depth sensing, there are many others. For example, capturing 3-D information about an object or a room makes it significantly easier to implement a model of that image in virtual or augmented reality applications. A related topic to depth sensing is range detecting in which distances to objects are determined, often using image sensors (in some cases the application may not need to actually capture an image but just needs to obtain the distance). Applications for range detection include automotive (for collision avoidance), drones (collision avoidance), industrial (ensuring items are correctly positioned during manufacturing), law enforcement (speed detection), security, robotic guidance, and occupancy sensing. Select Image Capture / Sensor M&A Transactions Some recent image sensor acquisitions include:  AMS/CMOSIS – In November 2015, Austria-based AMS announced that it was acquiring Belgium-based CMOSIS from private equity firm TA Associates for 220 million euros. Revenue for CMOSIS for 2015 was projected to be approximately 60 million euros, per the announcement. CMOSIS sold a variety of high end CMOS image sensors, although its focus was mainly on non-consumer applications such as
Page 70 medical, scientific, and machine vision. TA Associates had acquired CMOSIS less than two years earlier (TA announced that it was acquiring CMOSIS in December 2013). CMOSIS acquired AWAIBA (another image sensor company) in 2014.  Sony/Toshiba Image Sensor Unit – In October 2015, Sony announced that it would acquire Toshiba’s image sensor business for approximately 19 billion yen (about $155 million). Toshiba had a relatively small market share in image sensors but the transaction also included manufacturing facilities.  Hua Capital/OmniVision – OmniVision is a major supplier of CMOS image sensors. In April 2015, OmniVision agreed to be acquired by a consortium of Chinese private equity firms for $29.75 per share or about $1.9 billion. The consortium included Hua Capital Management, Citi Capital Holdings and GoldStone Investment Co. OmniVision had announced several months before that it received an unsolicited offer from the investor group.  Apple/LinX – In April 2015, the Wall Street Journal reported that Apple acquired LinX for $20 million. LinX was an Israeli company that specialized in small camera modules for smartphone cameras. Limited information was publicly available about the LinX solutions, but its website showed multi-lens modules that provided many competitive advantages (ability to refocus an image after it has been captured, 3-D object modeling, high image quality in low light environments, etc.).  ON Semiconductor/Aptina – In June 2014, ON Semiconductor announced that it was acquiring Aptina Imaging for approximately $400 million in cash. Aptina was a major supplier of CMOS image sensors for cameras, mobile devices, and gaming platforms. At the time of the announcement, ON indicated that Aptina had approximately $532 million in trailing revenue with gross and operating margins of approximately 29 percent and 3 percent, respectively. Aptina had previously been the image sensor business of Micron, which sold a majority of Aptina to private equity investors (Riverwood and TPG Capital) in 2009.  ON Semiconductor/Truesense Imaging – In April 2014, ON Semiconductor announced that it was acquiring Truesense Imaging for approximately $92 million in cash. Truesense sold image sensors primarily for industrial markets (machine vision, surveillance, traffic monitoring, medical and scientific). Truesense Imaging’s revenue for 2013 was approximately $79 million with gross and operating margins of approximately 44 percent and 23 percent, respectively. In addition to Aptina and Truesense, ON previously acquired a CMOS image sensor business from Cypress Semiconductor in 2011 for $31.4 million.
Page 71 Select Image Sensor/Capture and Range Detection Companies Examples of image sensor, image capture, and range detection companies are below. InVisage InVisage has developed an innovative image capture technology called QuantumFilm which it believes has substantial advantages relative to conventional CMOS digital image sensor technology and utilizes quantum dot technology (quantum dot technology was previously discussed in the chapter on touch screens and displays). Specifically, the company indicates that silicon image sensors sense light linearly and saturate when the number of electrons a pixel can store exceeds a pixel’s fixed full well capacity (FWC), which is related to the size of a pixel. As pixel size continues to shrink, it results in poor dynamic range and image quality. In contrast, the QuantumFilm sensor provides a much higher FWC for a given pixel size and also has a non-linear response to light, similar to film. In addition, its QuantumCinema technology optimizes the response (linear in the darker regions, more film like in the brighter regions) to greatly improve image quality in a broad range of lighting environments, including those with a mixture of dark and bright regions. It also enables slow motion video and incorporates an electronic global shutter (full frame capture). In November 2015, InVisage introduced its initial product, the 13 megapixel Quantum13 camera sensor, which it indicated is the first image sensor that uses quantum dot film. It includes a global shutter and supports 4K video. InVisage stated that initial shipments to leading smartphone vendors would commence by the end of 2015. In 2014, InVisage announced an $18 million funding which brought its total funding to more than $100 million. Investors include GGV Capital, Nokia Growth Partners, RockPort Capital, InterWest, Intel Capital and OnPoint Technologies. Later that year, InVisage raised additional funding from China Oceanside. The company is based in Silicon Valley. Occipital Occipital originally developed a barcode scanning app and a 360 Panorama app, but more recently has shifted its focus to advanced 3-D scanning. Its Structure Sensor is an add-on to an iPad that enables a user to rapidly capture 3-D images/depth maps of objects or entire rooms. This enables easy creation of 3-D models. For example, a user could capture a 3-D image of a couch and then use that model in an application to see how the couch would look in a particular room. Rapidly creating 3-D images and models of objects greatly simplifies the creation of 3-D applications, including virtual and augmented reality. Its Structure Core solution is optimized for AR/VR, SLAM, and robot vision. While the company sells hardware scanners, it indicates that most of the core technology is in its software and SDK. Last year, Occipital acquired Lynx Laboratories, which had its own 3-D scanner technology but shifted to a software focus with its software running on Occipital hardware. Occipital indicated Lynx’s IP can reduce the error of 3-D sensors by up to 90%. In July 2015, the company raised $13 million from Intel Capital, Shea Ventures, Grishin Robotics and Foundry Group.
Page 72 Sensl Sensl has developed a unique silicon photomultiplier (SiPM) sensor technology. Unlike traditional CMOS image sensors, photomultipliers are extremely sensitive to light (can multiply the current produced by incident light by more than 10 million times). There are a number of applications for which this type of sensitivity is required (e.g., medical imaging). Historically, this was accomplished using vacuum tube technology (one of the few remaining markets for vacuum tubes) for high end applications in which performance was important and by avalanche photodiodes for more cost-sensitive applications. However, Sensl believes its silicon sensor technology has enormous advantages relative to those traditional products. As would be expected, its solution is dramatically smaller, more robust, and less costly relative to vacuum tube-based photomultipliers and it doesn’t require the high voltages that traditional photomultipliers do. Prior to Sensl, however, silicon-based sensors could not achieve the performance of tube-based solutions. On the lower end, its solution provides substantially better performance and other technical advantages compared to avalanche photodiodes (high end performance at lower end cost). Sensl initially targeted medical applications (as well as military and scientific applications) and is in discussions with several major companies in those markets. However, more recently it has begun to leverage its technology for range detection/LiDAR type applications in which it believes its technology can provide compelling advantages due to its high level of sensitivity and performance advantages. Some of these applications include: automotive range detection, aerial surveying, robotics, and autonomous vehicles. Longer term, as Sensl drives the cost of its SiPM solutions down through higher volumes, it believes that it can penetrate high volume consumer markets such as gaming and augmented/virtual reality products. Sensl is based in Ireland. Leddartech Leddartech has developed advanced ranging and detection solutions that utilize image sensors and time of flight technology. Its “light-based radar” sends very short light pulses of (invisible) light about 100,000 times per second to actively illuminate an area of interest. The sensor captures the light backscattered from objects (either fixed or moving) over its detection area, and processes the signals to precisely map their location and other attributes. The company indicates it has the only light-based sensor technology with the capability to read all elements of the returned signal at once and to analyze them simultaneously. Its products include the LeddarCore chip and its family of LeddarOne complete modules. The company has indicated that it working with a variety of major partners in automotive and other markets. Leddartech was founded in 2007 and is based in Quebec City.
Page 73 Image Recognition One area of growing importance for mobile devices is image recognition. Using image sensors a device might not only capture images but also recognize what those images are and take appropriate actions. So if a person, for example, captures an image of a certain product on a smartphone in the store, the smartphone might be able to recognize what the product is and bring up relevant information (reviews, coupons, etc.) about that product. However, recognizing images is no simple task. For example, Coke products can come in a variety of shapes and sizes and the user could be capturing images of the product from a number of different distances and angles with a variety of other objects around it. Image recognition is used in a variety of applications. Amazon’s iOS app, for example, incorporates its “Flow” technology, which enables users to capture images of products which are then recognized through visual search technologies. As a result, a user can simply point the iPhone camera at products the customer wants to buy (books, DVDs, grocery items, etc.) and the items register in the Amazon app and can then be purchased, if available on Amazon. Facial recognition (discussed in a previous chapter) is a type of image recognition. Image recognition is an important part of augmented reality solutions as the systems must be able to recognize and track objects from many positions. A variety of image recognition software solutions have been developed. Select Image Recognition M&A Transactions Some examples of recent image recognition M&A transactions are noted below. Although we are generally including only transactions from the past few years in this report, it is interesting to note that Google acquired an image recognition technology company named Neven Vision back in 2006. Neven’s technology could identify objects and people in images. We also note that some relevant transactions are included in the following chapter on virtual/augmented reality.  Google (DeepMind)/Vision Factory – In October 2014, Google indicated that it had acquired Vision Factory, a UK-based company focused on applying artificial intelligence techniques to enhance the accuracy and speed of object recognition and improving vision-based computer systems.  Qualcomm/Euvision – In September 2014, Qualcomm acquired Euvision, as reported by TechCrunch. Euvision was spun out of the University of Amsterdam and was focused on using artificial intelligence techniques for image recognition applications. Its initial product was an app that could analyze and organize photos on a smartphone into different categories.
Page 74  Twitter/Madbits – In September 2014, Madbits announced that it had been acquired by Twitter. Madbits developed visual intelligence solutions that could analyze raw media and extract and understand relevant information. The company used deep learning to understand the content of an image.  Google/Jetpac – In August 2014, Google acquired Jetpac. Jetpac developed technology that could analyze images to obtain contextual information. For example, it could analyze the details in a photo to determine what location it was taken, or analyze photos of a restaurant to determine who it might appeal to. Beyond photos, the technology was also capable of examining and analyzing video.  Qualcomm/Kooaba – In January 2014, it was reported that Qualcomm Connected Experiences acquired Kooaba (per the 451 Group). Kooaba was a Switzerland- based company that developed a variety of image recognition technologies. That is, its technology could recognize objects within captured images and then bring up relevant information about the objects.  Pinterest/VisualGraph – In January 2014, Pinterest acquired VisualGraph, a small start-up focused on image recognition and visual search.  Google/Industrial Perception – In December 2013, a New York Times article reported that Google had acquired seven robotic related acquisitions, including Industrial Perception, which was focused on computer vision systems for controlling robotic arms and other applications.  Yahoo/IQ Engines and LookFlow –Yahoo acquired LookFlow (image recognition technology company) in October 2013 and IQ Engines (which had an image recognition platform that could help identify objects, landmarks, or people in an image for mobile and Web applications) in August 2013 (per the 451 Group).  Dropbox/Anchovi Labs – In September 2012, Dropbox acquired Anchovi Labs, which was focused on vision processing and image classification using learning and artificial intelligence technologies (per the 451 Group). Select Image Recognition Companies Below is an example of an image recognition company. Some image recognition related companies are included in the next chapter on virtual and augmented reality. LogoGrab LogoGrab has developed a browser-based image recognition solution focused on logo recognition. Its WebScanner solution eliminates the need for users to download a specific app, making it easier for marketing campaigns to achieve greater reach. Its LogoGrab technology can be easily integrated into mobile Web sites or customers’ apps. LogoGrab also has a Visual Listener solution that can accurately detect logos in social media utilizing LogoGrab’s large database of corporate logos.
Page 75 LogoGrab has a variety of major customers. For example, in September 2015 it announced a digital campaign with Heineken in 85 countries related to the James Bond movie Spectre (by “grabbing” the logo on limited edition Heineken packaging, fans could unlock behind the scenes footage from the movie). LogoGrab has also done work with Coke and others. The company is based in Dublin, Ireland. Vision Processing Mobile devices are incorporating a growing number of image sensors. In addition, the complexity of image sensors is increasing (e.g., high pixel count sensors, 3-D/depth sensors, light field sensors, etc.). Augmented reality technologies are becoming more commonplace and there are many applications in which there is interest in not just capturing images but in being able to analyze and post-process images and determine what objects are in the image. Emerging technologies like simultaneous localization and mapping (SLAM), which enable devices to build internal maps of local environments without having prior knowledge of the local environment, require substantial processing power. In many cases, the processing required for these types of applications greatly exceeds the capabilities of mobile devices, which have limited processing power and cannot consume too much power. Traditional microprocessors are not specifically designed to handle vision processing. Just as digital signal processors (DSPs) and graphic processing units (GPUs) were developed to better handle tasks that were not well suited for ordinary processors (e.g., baseband cell phone processing, PC graphics), some companies have developed vision processing chips (VPUs) that are specifically optimized for the types of parallel processing required for vision/image applications for mobile devices. This can help enable a variety of advanced features and functions while also freeing up the main/application processor to better handle other tasks and greatly improve the user experience. One development in vision processing is Google’s Project Tango. Project Tango was announced by Google a few years ago and is focused on giving mobile devices a human- like understanding of space and objects by using computer vision and sensors. As an example, a device may be able to develop a complete internal map of the inside of a user’s house and then use that for a variety of applications such as interior design or even having game characters play, for example, “hide and seek” in images of the user’s house rather than in a generic house. An initial prototype looked like a traditional Android phone (although thicker) and incorporated a regular image sensor (4MP camera), a depth sensing sensor/receiver, and a lower resolution motion tracking camera, as well as dedicated vision processing chips. Subsequently, Google released a Project Tango development kit in mid- 2014, which was a 7 inch tablet with a motion tracking camera, depth sensing, and many other functions. A variety of major tech companies have announced support for Project Tango.
Page 76 Select Vision Processing M&A Transactions Vision processors are a relatively new segment, but there has already been some M&A activity in the sector.  Apple/FlyBy – In January 2016, it was reported by the Financial Times that Apple acquired FlyBy Media, which developed vision-based software for smartphones and other devices. FlyBy was a vision-based software partner for Project Tango. Its focus was connecting the real world with digital content.  Freescale/CogniVue – In September 2015, Freescale announced that it had acquired CogniVue. CogniVue was highlighted in the first edition of this report and had developed software, IP, and an Image Cognition Processor (ICP) that were optimized for vision type applications. Its CV2201 APEX chip was designed specifically for vision processing and incorporates a RISC processor and local dedicated memory. CogniVue indicated it could achieve a 100X advantage in performance per power relative to conventional approaches. The APEX could either be a stand-alone chip or its IP core could be integrated into a larger SoC. CogniVue had previously announced a partnership with Freescale addressing vision processing in automotive applications. Select Vision Processing Companies The following describes a couple of companies that have developed vision processor chip solutions, as well as a company that provides middleware specifically targeting vision processing applications. Movidius Movidius is a leader in the emerging vision processing chip market. The company has developed advanced imaging chipsets and algorithms that enable always-on smart capture and continuous vision processing applications for mobile devices. As traditional processors and graphics processors are not specifically optimized for vision-type applications, Movidius believes that those types of chips are not well-suited for vision processing, especially for mobile devices which require not only extreme levels of computation but have significant power consumption limitations. Movidius’ second-generation chip, the Myriad 2 Mobile Vision Processor, is a Vision Processor SoC that incorporates a variety of differentiated features. Myriad 2 integrates 12 proprietary SHAVE (Streaming Hybrid Architecture Vector Engine) processors, 2 RISC processors, an optimized intelligence memory fabric, a variety of advanced imaging/vision hardware accelerators, and a number of I/O interfaces. The SHAVE processors incorporate a hybrid stream processor architecture combining the best features of GPUs, DSPs and RISC as well as unique features such as hardware support for sparse data structures. The architecture is designed to maximize performance-per-watt while
Page 77 maintaining ease of programmability. The chip is produced on 28nm process technology. The result is extremely high performance for vision/imaging applications with ultra-low power consumption (e.g., two trillion 16-bit operations per second while consuming an average of less than 500 milliwatts). In addition to the Myriad 2 chip itself, Movidius has developed a broad range of vision-related algorithms (machine learning, 3-D depth, tracking/navigation, etc.) and an SDK to enable easy implementation for customers. In January 2016, Movidius announced that it is working with Google to accelerate the adoption of deep learning in mobile devices. As part of the agreement, Google will source Movidius processors and the entire Movidius software development environment, and Google will contribute to Movidius’ neural network technology roadmap. The target is to enable future products that have the ability to understand images and audio with incredible speed and accuracy, offering a more personal and contextualized computing experience. Movidius is headquartered in Silicon Valley with a major office in Ireland and a recently opened office in China. In April 2015, Movidius announced that it had raised $40 million in new funding. The investment was led by Summit Bridge Capital and included new contributions from ARCH Venture Partners and Sunny Optical, as well as early investors including Atlantic Bridge Capital, AIB Seed Capital Fund, Capital-E, DFJ Esprit, and Robert Bosch Venture Capital. Inuitive Inuitive has developed a 3-D imaging and computer vision multi-core processor (NU3000) and NU3000-based camera solutions. It indicates that its solution is specifically optimized for applications such as augmented reality as it incorporates not only 3-D image processing and computer vision, but also depth sensing processing. The NU3000 integrates a computer vision processor, an image processor, and a general purpose processor on one chip (two Ceva DSPs and an ARM Cortex A5 and ARM Neon Vector Processor), as well as a variety of proprietary hardware accelerators. The solution incorporates a 3-D Depth Engine based on a proprietary “Assisted Stereoscopy” technology and can provide a real-time localization and mapping solution. It can process 3 video streams in real time (2 depth cameras and 1 RGB image sensor). Its open architecture allows integration of a customer’s software into the solution and offloads the computationally intensive processing from a main CPU. Inuitive recently announced a reference design dedicated for Google’s Project Tango. The reference module incorporates the NU3000 with two cameras arranged in a stereoscopic setting, a fish-eye camera for 2-D wide-angle imaging, and an inertial motion unit. The company was founded in 2012 and is based in Israel. Its investors include T-Venture (the VC Company of Deutsche Telekom) and 7-Main.
Page 78 Van Gogh Imaging VanGogh Imaging develops object recognition analysis and vision processing middleware for mobile and embedded devices. There is growing interest in implementing advanced 3-D vision-based solutions, but many companies lack the expertise to create some of the key functions. By using Van Gogh’s Starry Night software, customers can more rapidly develop solutions for a variety of applications. Starry Night is a computer vision Unity1 plugin that utilizes a patented shape-based registration technique which uses priori information about a scene or object. VanGogh indicates that it is highly tolerant to noise and can be fully automated, eliminating the need for manual post processing to create accurate 3-D models. It can examine a scene, recognize objects, and create 3-D models of those objects for applications ranging from 3-D printing, augmented reality, automotive, and robotics. In January, the company announced its Starry Night-MR computer vision software which is specifically optimized for augmented and virtual reality applications. Unlike many solutions, Starry Night can handle dynamic real world environments with moving objects and changing lighting conditions. VanGogh is based in McLean, Virginia. Video/Image Processing Another important aspect of video and image capture is the subsequent processing required to optimize and store the video. In many cases, the format of the video or images that are captured does not match with the format of how the video/image is displayed, and must be transformed. In some cases, the incoming images need to be “cleaned up” to improve quality and make adjustments based on lighting. There may be interest in zooming in or out, adding special effects, correcting geometrical distortions caused by the motion of the camera in 3-D space, or enhancing the captured images due to poor lighting conditions. Given the enormous amount of data required for video, there is often a need to compress the video before storing or transmitting it, and then subsequently decompress it before playing the video. All of these types of functions require significant processing power and advanced software, and in some applications this needs to be done in real time. As a result, there are a number of companies that have developed advanced software and/or chips to perform some of these tasks. Select Video/Image Processing Company Below are two companies that have solutions related to video and image processing. GEO Semiconductor GEO Semiconductor provides a variety of solutions related to image and video processing. GEO was formed in 2009 and acquired a number of key products and patents from Silicon Optix. In January 2013, GEO completed the acquisition of Maxim’s video compression business unit (much of which came from Mobilygen, a mobile compression start-up that Maxim acquired in 2008).
Page 79 GEO has four core technologies: 1) its eWARP geometric processing engine (which can scale, transform, and rotate images to resolve a variety of alignment, calibration, and other issues associated with advanced camera and projection systems); 2) advanced video compression (virtualized H.264 compression solutions scalable to 4K resolutions); 3) Image Signal Processing (ISP) which provides a variety of advanced camera control features and algorithms (dynamic color and exposure control, advanced adaptive noise reduction, algorithms to optimize the representation of the scene including both dark and bright areas); and 4) massively parallel video processing architectures to optimize performance. GEO has developed a number of chips (video camera processors, geometric processors) that integrate one or more of these technologies. For example, its GW4300 camera processor integrates a high dynamic range ISP processor, an eWarp processor, an on- screen display controller and a Tensilica CPU running a variety of video processing algorithms. GEO also sells Realta, a highly integrated highly parallel SoC that was the first fully software programmable video array processor capable of performing over 1 trillion operations per second. GEO’s solutions are used in a broad range of markets including consumer, automotive, security, and IoT. In mid-2015, the company announced a line of video processors specifically optimized for automotive applications, such as a digital rear view mirror. GEO has operations in San Jose, Toronto, Orlando, and Bangalore. Irida Labs Irida Labs develops software and silicon IP for embedded video and vision processing applications such as video stabilization, face detection/recognition, low-light video enhancement, and pedestrian detection. Its target markets include mobile, wearables, surveillance, automotive and consumer electronics markets. The company has announced partnerships with CEVA, ARM, and Cadence/Tensilica. Irida was founded in 2007 and is based in Greece.
Page 80 “Reality leaves a lot to the imagination.” - John Lennon Introduction A number of visionary HMI technologies appeared on Star Trek. In addition to the main Enterprise computer and Data (a lifelike android), one of the most interesting was the holodeck, a simulated reality facility in which people could interact with computer generated people and objects which appeared, communicated, and felt completely realistic to anyone inside the holodeck. While anything approaching the holodeck is still many decades away, there are a number of technologies under development that will move us in that direction. During the past couple of years, there have been a number of high profile virtual reality and augmented reality product announcements from some of the largest technology companies in the world. Virtual reality and augmented reality are sometimes used interchangeably but actually refer to somewhat different (although related) types of technologies. Specifically:  Virtual Reality – With virtual reality, the user is immersed in a computer simulated reality that is separate from the real world (other than the user’s movements). For an immersive virtual reality experience, the user typically wears some type of opaque eyewear/helmet such that he cannot see the real world and sees and hears only computer generated images and audio (although the images/audio could be recorded real world images or videos).  Augmented Reality – With augmented reality (also referred to as mixed reality), the user sees the real world (what is in the room around him) but computer generated images and other sensory input is added. For example, a user might see an actual table in the room he is standing in, but also see a computer generated alien sitting under that table. For an immersive augmented reality experience, real world images have to be combined with computer simulated images. Both of these technologies are expected to become increasingly commonplace over the next few years and could have an impact on a variety of markets (beyond the obvious gaming markets). Implementing immersive virtual or augmented reality simulations requires combining many of the technologies previously discussed in this report, in addition to highly sophisticated software and significant computational power. This chapter discusses both virtual and augmented reality, including a section at the end on holographic imaging which is still in its infancy but could become significant over time. Chapter 10 – Virtual and Augmented Reality
Page 81 Virtual Reality As previously noted, virtual reality (VR) creates an entirely computer-simulated environment. At its simplest, a conventional video game can be considered virtual reality since it simulates another world that is completely computer generated. However, the term virtual reality is more commonly used to refer to immersive experiences in which the users have the sense (at least to some extent) that they are in a different virtual world. Historically, this was most commonly implemented through helmet mounted displays that provide 3-D video imaging and audio. Basic virtual reality products were developed as far back as the 1960s (although some of the early helmet models were so heavy they had to be suspended from the ceiling to wear). In the early 1990s, Sega introduced the Sega VR headset which incorporated LCD screens in the visor and sensors that enabled the system to track the user’s head movements (it was initially anticipated to be used for both arcade games and home consoles but was deployed only in arcades). A variety of basic virtual reality arcade games were subsequently developed but until recently the lack of processing power prevented systems from creating truly immersive virtual reality experiences. The virtual reality industry received a boost a couple of years ago when Facebook announced the acquisition of Oculus for over $2 billion. Given the large amount paid for the acquisition and that Facebook has historically focused on the mass market, the acquisition accelerated interest in virtual reality. For an interactive virtual reality experience to feel as real as possible, the images must adjust based on the user’s movements. For example, if the user turns her head or moves, the scene must rapidly adjust to show that 3-D view. To make the experience interactive, the system would ideally track the user’s gestures (e.g., if the user lifts his hands, his avatar in the virtual world also lifts his hands). One issue with VR is because the user may be moving around but cannot see the room he is actually physically in, there is a need to warn him if he will run into an actual object (e.g., a wall). All of these things require a significant amount of sensors, sensor processing, 3-D image processing and computational power. Oculus Rift Rift is the high profile virtual reality headset developed by Oculus, which is now part of Facebook. The Rift is an advanced virtual reality headset that connects to a PC via cable (the PC runs the virtual reality software so Rift is not a stand-alone device). The Rift headset incorporates a variety of technologies including high resolution OLED displays and some additional features to improve image quality. The headset also incorporates high quality lenses, headphones (for 3-D audio), and has full 6 degree of freedom rotational and positional tracking. The positional tracking is accomplished through an external infrared tracking sensor system called Constellation. The Rift comes with an Xbox One wireless gamepad which can be used as a controller with Rift-based games. Oculus has also developed its own Oculus Touch controller which is optimized for virtual reality applications. A variety of software and apps have been developed for Rift.
Page 82 Several preliminary versions of Rift have been introduced, but Oculus has indicated that the commercial version of Rift for consumers will be released in 2016. Oculus recently began accepting pre-orders for Rift with a June 2016 estimated delivery date. The price is $599 and includes two games. In addition, the Oculus software has been used as part of the Samsung Gear VR solution which was already commercially released in late 2015 and is discussed below. Oculus has also indicated that it is already working on a next-generation VR solution beyond the Rift. Other Virtual Reality Products Besides the Oculus Rift, a number of other major technology companies have announced virtual reality solutions. Below are some of the high profile products. However, many other companies have also introduced innovative VR solutions (many shown at CES 2016).  Samsung Gear VR – The Gear VR is a head-mounted virtual reality accessory device developed in conjunction with Oculus (it uses software from Oculus). It is designed to work with Samsung’s smartphones (e.g., Galaxy Note 5, S6, etc.). The headset incorporates a number of sensors (accelerometer, gyro, proximity, etc.) and the Samsung smartphone directly snaps into the Gear headset (the smartphone produces the images and the headset has an advanced optical lens system to create the immersive experience). Samsung introduced a couple of preliminary development versions but formally began selling the consumer version in November 2015. The Gear VR enables 360 degree virtual reality viewing. A number of virtual reality games, videos, shows, and apps have been developed for Gear.  Sony PlayStation VR Headset – PlayStation VR (previously referred to as Project Morpheus) is a virtual reality headset developed by Sony that is expected to work with the PlayStation 4 game system. A prototype shown in 2015 included an OLED display capable of displaying images at 120 frames per second. It incorporates advanced head tracking and an output that enables others to watch what the user is seeing on a separate TV display. While the final product has not yet been released, Sony has indicated that it will likely be introduced during the first half of 2016.  HTC Vive – HTC announced the Vive in early 2015 with plans for commercial shipments in 2016. The Vive was developed in conjunction with Valve Corporation, a video game developer. It is a head-mounted display solution that incorporates two small high resolution screens (one for each eye) with a 90Hz refresh rate (90 frames per second) as well as over 70 sensors. Although it is a VR solution, it includes front facing cameras that can identify objects in a room to, at a minimum, prevent users from hitting a wall or an obstacle (its Chaperone technology which can alert users if they are getting too close to real world objects). A number of game developers have announced support.
Page 83  Google Cardboard – Google Cardboard is an ultra-low cost virtual reality solution in which the headset is made from cardboard and relatively inexpensive lenses. Google doesn’t actually sell the product but provides the specs for free. The solution works with a smartphone which when inserted into the Cardboard headset provides the actual display. Google software divides the smartphone display into two sets of images (one for each eye), and creates a 3-D stereoscopic image. A number of major companies (Volvo, LG, Mattel, etc.) have released Cardboard-based solutions as part of promotional campaigns. Augmented Reality Augmented reality (AR) refers to combining real world images with digitally generated data and/or images. That is, in contrast to virtual reality in which all of the imagery is computer generated, augmented reality augments the real world images a user can see. A simple example of augmented reality is the yellow “first down line” that is added to the field during NFL TV broadcasts (i.e., you see the real time images of the field along with a computer generated yellow line that appears to be a real line on the field). A goal with AR is to integrate the digitally created content as seamlessly as possible such that it appears as part of the real world image. Augmented technologies can be implemented in a variety of ways. For example, real world images can be captured via image sensors and displayed on a screen while a variety of 3-D graphics are added to enhance the image. Alternatively, for more immersive experiences, augmented reality can be implemented through glasses or head mounted devices in which the user sees the real world but computer generated graphics are added to enhance the images that a person sees. Although Google Glass was not specifically developed as an augmented reality device, several developers have created apps to implement AR on Google Glass. The number of potential applications for augmented reality is enormous. For example, the real world image of a car engine can be augmented to show the user how to perform certain types of car maintenance (graphics and instructions overlaid on the actual engine image). AR can overlay a real time image of an object or place that a person is looking at with relevant information (what the object is, information from online resources about the object or place, etc.). It has many potential uses for field service (an “expert” can guide a non-expert in how to repair something without physically being there). AR is commonly used for advertising purposes (adding 3-D images to real world objects), and can be used for engineering design, navigation, and a variety of other applications. Automotive heads- up displays (HUDs) in which navigation or warning images are displayed on the driver’s windshield is a type of augmented reality. AR is also expected to become increasingly common for gaming applications in which gamers can play in their own house (e.g., shoot at images of characters moving around in the actual room they are in).
Page 84 Implementing immersive augmented reality is very complex. For example, if an augmented reality game has an image of an alien sitting on a couch, the system has to be able to track where the user is in the room and where the couch is in order to ensure the computer simulated alien remains in the right spot regardless of where the user moves to in the room. Several major technology companies have developed AR-related software solutions and augmented reality has been incorporated into a variety of advertising campaigns. For example, these solutions might combine image recognition with AR such that when a camera from a smart phone recognizes a certain product, it creates computer simulated images or games around the product. A number of stores have used augmented reality to assist customers in deciding on different colors or what they would look like with different clothing/makeup or what furniture to buy (enabling users to see what a room might look like with different colors or different pieces of furniture). There have also been some augmented reality applications for fashion (e.g., allowing people to “try on” different clothing or make-up without physically putting anything on). As augmented reality improves, the list of potential applications is expected to significantly increase. Microsoft HoloLens A major augmented reality development was Microsoft’s HoloLens which the company announced in early 2015. HoloLens is a glasses-type headset that incorporates a variety of processors, sensors and imaging technology and enables users to see both the real world and 3-D computer generated images, which Microsoft refers to as “holograms” (technically they aren’t true holograms in that you need to be wearing the HoloLens glasses to see the images, but Microsoft believes the imaging goes beyond traditional augmented reality images). In one demo of HoloLens, a user sees an image of a large laser-type weapon appear on his hand, and it remains on his hand regardless of where his hand moves. Subsequently a computer generated alien ship appears to crash through the wall of the room the user is standing in and aliens begin to jump out and attack him, but he can shoot them with his laser weapon. In another demo, a 3-D Minecraft city appears on top of a real table in the room in which the user is standing in, and the user can then interact with the city. HoloLens is part of Microsoft’s Windows Holographic computing platform initiative. The HoloLens is still in development, but reportedly incorporates a high end CPU, a graphics processor (GPU) and a new holographic processing unit (HPU), which is a custom sensor fusion type coprocessor developed by Microsoft. The HPU processes sensor data (gesture recognition, mapping, speech processing, etc.) and according to Microsoft can handle terabytes of information in real time. HoloLens also integrates many different types of sensors (accelerometer, gyro, magnetometer), as well as depth cameras, video camera, and microphones. In addition, it contains an innovative “light engine” and multi-layer combiner lenses that enable users to see the computer generated images. As the HoloLens is still in development, the product is still subject to modifications. Microsoft has not set a date for the release of a commercial HoloLens solution, but indicated that it plans to introduce a development version in the first quarter of 2016.
Page 85 Select Virtual Reality/Augmented Reality M&A Transactions Examples of virtual reality and augmented reality M&A transactions include:  Apple/Faceshift – In November 2015, TechCrunch reported that Apple acquired Faceshift, a facial analysis start-up. The Faceshift demos show how a user can move their facial features (smile, frown, open/close eyes, move head) and an avatar on a computer display performs the same movements. Faceshift reportedly developed technology used in the latest Star Wars film, and is based in Switzerland. This technology could potentially be used for virtual/augmented reality applications.  PTC/Vuforia(Qualcomm) – In October 2015, PTC announced that it was acquiring Qualcomm’s Vuforia business for $65 million. Vuforia was a leader in augmented reality solutions and software. Qualcomm indicated that Vuforia was used by over 175,000 registered developers and powered more than 20,000 apps.  Blippar/Binocular and Layar – Blippar is a private augmented reality company (highlighted later) but has already made acquisitions. In October 2015, it acquired Binocular, which developed augmented reality applications (virtual “try on”) for companies. For example, it created an app for Luxotica that allows users to try on glasses using a smartphone, without physically touching glasses. Blippar confirmed in June 2014 that it had acquired Layar, which developed a popular augmented reality platform. The Layar SDK enables easy mobile AR development.  Snapchat/Looksery – In September 2015, TechCrunch reported that Snapchat acquired Looksery. Looksery was a private company that developed animated “video lenses” which enable users to modify the look of their faces during video calls. For example, the lenses could make a person appear as a monster or a cat or a variety of other options. Alternatively it could also improve the image (eliminate blemishes, etc.). Ukrainian news agency Ain.ua indicated that although there was no confirmation, its sources indicated that the price was approximately $150 million.  Facebook/Surreal Vision – In May 2015, Oculus announced that it had acquired Surreal Vision. Surreal Vision was focused on real-time 3-D scene reconstruction (generating an accurate representation of the real world in the virtual world).  Apple/Metaio – In May 2015, it was reported (by Forbes, TechInsider, and several others) that Apple acquired Metaio. Metaio was highlighted in the first edition of this report and had been a major supplier of augmented reality solutions including Creator (an AR publishing tool that enables users to create a broad range of AR materials using drag and drop), SDK (a software development kit for AR), Cloud (lets users manage AR and content in a single place), Visual Search, Junaio (advanced AR Browser for iPhone and Android), and Engineer (hardware IP). Metaio reportedly had well over a thousand customers including Volkswagen, Mitsubishi, Audi, McDonalds, Macy’s, Adidas, Red Bull, and Lego. Metaio was based in Germany.
Page 86  Daqri/ARToolsworks – In May 2015, private company Daqri acquired ARToolworks. ARToolworks was an early player in augmented reality and had been founded in 2001. It developed an open source AR toolkit and had a variety of early AR patents. Daqri is developing an AR headset mostly for industrial applications.  Google/Tilt Brush – In April 2015, Google acquired Tilt Brush, which enables users to paint in 3-D in virtual reality and then share the images with others. The Tilt Brush web site indicates that it works with the HTC Vive.  GoPro/Kolor – In April 2015, GoPro acquired Kolor, a French virtual reality software company that lets users to combine multiple photographs or videos to create 360- degree panoramas and videos ("spherical content"). GoPro indicated that its spherical videos can be viewed on Google Cardboard and would soon be available for Oculus VR and the Samsung Gear VR.  Facebook(Oculus)/13th Lab – In December 2014, Oculus announced that it had acquired 13th Lab, which developed advanced augmented reality tracking and 3-D construction framework technology. Its PointCloud SDK was a development tool focused on Simultaneous Localization and Mapping (SLAM), image detection, and tracking. The company was based in Sweden.  Google/Quest Visual – In May 2014, Google acquired Quest Visual. Quest Visual developed the Word Lens augmented reality translation application. Word Lens could capture images that contain foreign words (menus, signs, etc.), translate the words, and display the image with the translated words on it.  Facebook/Oculus – In March 2014, Facebook announced that it was acquiring Oculus Rift for more than $2 billion ($400 million in cash, about $1.6 billion in stock, and up to an additional $300 million if certain milestones are met). Oculus had developed an advanced virtual reality head-mounted display headset which was not yet shipping commercially. Oculus initially raised $2.5 million in Kickstarter campaign. Select Virtual Reality/Augmented Reality Private Companies There have been a large number of recently formed virtual and augmented reality companies. Below are just a few examples. In addition to companies that have created AR and VR products, we included some companies that have developed solutions for capturing 3-D video and images for virtual reality applications.
Page 87 Magic Leap Magic Leap is high profile but stealth mode private technology company that has developed advanced augmented reality solutions. The company indicates that it has combined the physics of the visual world with how humans perceive things to create a Dynamic Digitized Lightfield Signal. Based on that core technology, the company has developed a variety of solutions, but has not yet released details to the general public. Some preliminary videos show impressive augmented reality scenarios (e.g., an animated robot hiding under a real world chair, a computer generated model of the solar system floating in an office). Magic Leap has provided only limited information about its solution, but has indicated that it actually projects images onto a user’s retina, as opposed to having the user look at a screen. It also indicated that it uses light field technology and that the user can move their eyes around to focus on different objects. In October 2014, it was reported by Reuters that Magic Leap had raised $542 million from investors that included Google, Qualcomm, Kleiner Perkins, Andreessen Horowitz, and Obvious Ventures. In February 2016, the company announced that it completed an additional $793.5 million funding round, led by Alibaba Group, with continued investment from Google and Qualcomm, and a variety of new investors including Warner Brothers and several major financial institutions. Magic Leap is headquartered in Dania Beach, Florida. Wikitude Wikitude is a major supplier of augmented reality software platform solutions. The company’s all-in-one AR solution includes image recognition and tracking, 3-D model rendering, video overlay, location based AR, and other features. It allows customers to easily develop apps and solutions that incorporate augmented reality. Its SDK supports a variety of products (Android, iOS, etc.). The latest version of the Wikitude SDK includes geo-based AR, image recognition and tracking services, extended image tracking, as well as 3-D modeling and presentation layers. The company recently implemented Cloud Recognition for massive image libraries, released the next-generation of its Wikitude SDK (v5), and has developed SLAM 3-D Tracking for tracking real world objects for augmented reality applications. Wikitude’s Cloud Recognition service allows customers to work with thousands of target images hosted in the cloud. Its Studio product enables customers to create their own AR solutions without requiring any programming. Wikitude indicates that its SDK is powering over 10,000 working apps, built by a community of over 50,000 active developers. Wikitude is based in Austria and was founded in 2009. Investors include Gamma Capital Partners and tecnet Capital.
Page 88 Marxent Marxent is a virtual/augmented reality technology company focused on visual commerce applications. The company’s VisualCommerce SaaS platform solution powers apps from many major companies such as Lowes, ToysRUs, GE Healthcare, Simmons Bedding, Toyota, American Woodmark, Novartis, AZEK Building Products, Dell, NetApp, and many other major companies. For example, Lowe’s advanced Holoroom kiosk (which allows customers to select a variety of furniture or appliances, choose/modify colors, and then see the room in virtual reality using Oculus and/or Google Cardboard headsets) is based on Marxent’s VisualCommerce technology. Lowe’s recently indicated that it was rolling out the Holoroom to its stores in the U.S. after some initial testing. A variety of companies utilize Marxent’s technology to augment their catalogs to enable augmented/virtual reality experiences. These types of applications can significantly improve sales but require building a 3-D database of products and Marxent has expertise in building 3-D product databases for retail, manufacturing, and e-commerce. Marxent is based in Ohio with an office in Florida. Blippar Blippar is an image recognition and augmented reality solutions company. The Blippar app allows a user’s smartphone (or other mobile device) to recognize, through the smartphone’s camera, products or objects. The act of scanning an item with a device is called "blipping." When a user “blipps” something with the Blippar app, the app determines what the object is and then checks to see if there is content associated with the image. Utilizing augmented reality, the app displays this additional content on the display on top of the actual product image. For example, if you point a smartphone with the Blippar app at a box of Lucky Charms, it recognizes the product and then a 3-D AR game appears on the smartphone. Blippar has partnerships with a broad range of products including Heinz ketchup, Maybelline makeup, Jaguar cars, Perrier water, Star Wars Rebels, Heineken beer, Coke, Pepsi, and many others. The Blippar app is available on iOS, Android, Windows and wearables. In addition to the Blippar app, Blippar provides a variety of other solutions including Blippbuilder (self-service tool), Layar SDK (enabling others to include Blippar AR into their apps), and also offers strategy and customization services. Although private, Blippar has made several acquisitions (Layar, Binocular, etc.). Blippar is based in San Francisco and has several offices around the world. FaceCake FaceCake Marketing Technologies is an augmented reality company focused on retail applications that allow users to “try on” clothing or makeup without actually physically putting the products on. FaceCake’s Virtual Try-On solutions (for apparel, cosmetics, accessories and more) enables consumers to compare looks, get style advice, and make informed choices, in real time. It also can provide users with a variety of product recommendations based on the images. FaceCake’s solutions can work in-store or at- home on many types of devices (in-store kiosks, PCs, tablets, mobile phones).
Page 89 FaceCake’s offerings include Swivel (which allows users to “virtually” try on a variety of different types of clothing and accessories), Swivel Close-Up (which enables users to try on different types of cosmetics and makeup colors), Swivel Smart Design Signage (augmented reality advertising for stores), and ShadeScout (a smartphone app that enables users to find makeup that matches a specific color they find). The privately held company is based in Calabasas, CA. uSens uSens has developed wireless headsets that support both virtual and augmented reality. The uSens Impression Pi headsets incorporate its internally developed gesture recognition, head tracking, and position tracking technologies. The company’s “Super Reality” technology enables a transition between virtual and augmented reality. The device includes an IR tracker and a sensor unit and works with a variety of smartphones that snap into the headset. In addition to gaming, the company believes its headset could be utilized in education, healthcare, entertainment, and training. In 2015, the company announced that it completed a $5.5 million pre-Series A round co-led by IDG Ventures, Lebox Capital, and Maison Capital, with Stone Capital, Fulcrum Capital, ChuanCheng Fund, and Chalor Capital also participating. ModiFace ModifFace has developed a variety of advanced facial visualization augmented reality solutions. The ModiFace Mirror is a photo-realistic real-time makeup and skin simulation that enables users to see what they would look like with makeup or other changes (e.g., a woman could “try on” many different types and colors of lipstick, eye shadow and other cosmetic products without having to actually physically put anything on her face). The technology can make recommendations based on the person’s face and can also simulate a variety of other changes (teeth whitening, anti-aging treatments, etc.). ModiFace recently introduced an HD version with a variety of advanced features. In addition, ModiFace provides a solution that enables users to upload a photo of their face to a Website and then modify how their face looks based on selecting various types of makeup. ModiFace also provides a broad range of apps that allow the user to see what they would look like with different hair color, hair styles (including celebrity hair styles), eye color (colored contacts), anti-aging treatments, and a variety of other changes. It can also help photo edit to implement a variety of changes. ModiFace’s technologies make it significantly easier for consumers to decide which product best suits them without having to try things out, and they can do it from home. Its technology is used by over 150 web and mobile apps, which have been downloaded nearly 60 million times. Some of the companies that have introduced ModiFace-based augmented reality apps include Allergan, Amore Pacific, L'Oreal, Yves Rocher, Jane Iredale, Matrix, Unilever, and Honest Beauty. ModiFace was founded in 2006.
Page 90 Jaunt Jaunt provides an end-to-end solution for creating cinematic virtual reality experiences. The company has developed a suite of hardware and software tools to enable the creation of high quality immersive content that can be viewed on a variety of virtual reality platforms. Its professional grade stereographic NEO camera was specifically designed to create immersive video and includes 360 degree capture, 3-D sound-field microphones, and advanced optics. The company’s advanced rendering software and photography algorithms transform the recorded video data into an immersive viewing experience. The content can be viewed on a variety of VR headsets (Google Cardboard, Samsung Gear, Oculus Rift, etc.). Jaunt has created and distributed a broad range of VR content and has announced a long list of partners it is working with on virtual reality content. In September 2015, it announced a $65 million capital raise that included investors such as Walt Disney Company; Evolution Media Partners (CAA, TPG Growth, Participant Media), China Media Capital, Madison Square Garden Company, Google Ventures, Highland Capital, Redpoint Ventures, Sky and SV Angel. Matterport Matterport has developed an advanced professional camera that enables users to capture accurate 3-D representations of rooms and houses. The camera incorporates a variety of 2-D and 3-D sensors and can scan a room (spinning around) to capture a panoramic view. In addition, Matterport’s software can piece together the various room images to create a complete 3-D model of a house. So, for example, a real estate agent could use the Matterport camera and software and easily develop a comprehensive 3-D image of a house, enabling potential buyers to do a virtual “walkthrough” of the house (moving from room to room, looking in any direction) before visiting. Matterport provides a complete end- to-end 3-D media platform, enabling users to upload the data from the cameras to the cloud, after which Matterport processes the data to create and host immersive 3-D models, which can then be viewed anywhere with a browser. In addition to real estate, other initial applications include hospitality, insurance, education, and retailing (e.g., capture images of a home and try out different furniture, etc.). The company is also working with Google's Project Tango and Intel's RealSense and hopes to eventually bring its technology into a broad range of consumer mobile applications, especially as 3-D imaging becomes more commonplace in mobile devices. Matterport completed a $30 million Series C round in 2015 which was led by Qualcomm Ventures. Other investors include GIC (Singapore’s sovereign wealth fund), Lux Capital, DCM Ventures, Felicis Ventures, Greylock, Navitas Capital, AMD Ventures, AME Cloud Ventures, iGlobe Partners, Rothenberg Ventures, Blake Krikorian (Sling Media founder), and Gordon Segal (Crate & Barrel founder). It is based in Silicon Valley.
Page 91 VideoStitch VideoStitch has developed technology that enables 360 degree video to be captured and then viewed, including viewing using virtual reality headsets. That is, by using multiple video cameras with overlapping views, a complete 360 degree view of an area can be captured. VideoStitch’s advanced software then rapidly stitches together the individual videos into a single, high resolution, seamless panoramic video, that can be streamed to the cloud or elsewhere. The software provides a regular video file or stream that can be watched, but it enables the user to modify the video view based on which direction the user wants to see. The technology can also be used with virtual reality headsets in which the user can turn to see different views just as in real life. Its products include Studio (360 video post-production) and Vahana VR (for virtual reality applications using Oculus Rift). VideoStitch also offers GoPro-based multi-camera rigs to help customers more easily capture video. VideoStitch indicates that it has over 1000 customers in 45 countries including entertainment, media and Fortune 500 companies (e.g., Facebook, Red Bull, Sony Entertainment). At CES 2016, VideoStitch and Intel announced that they are collaborating on high quality VR Video live streaming. The companies showcased 4K Live 360 video streaming using Intel hardware. The Live stream was taken from 6 action sports cameras that were stitched together live on a notebook PC. VideoStitch also announced that it is developing live stitching software that incorporates Intel’s Quick Sync technology, with an expected release mid-2016. VideoStitch is headquartered in Paris with a major office in San Francisco. Holographic Imaging There are a variety of technologies that provide advanced 3-D imaging without the aid of helmets or special eyewear. Magic Leap’s solution can apparently provide 3-D images without the aid of any type of glasses. Although Microsoft refers to the 3-D images generated by HoloLens as holograms, a true holographic image can be seen without the aid of specialized eyewear or headsets (and the viewer can walk around and see the image from different angles as if it were a real object). A traditional hologram records information about the light that comes from an object in all directions (rather than in just one direction like a photo). Typically two lasers are used – one is directed at the image itself (object beam) and the reflected light hits the recording medium, the other (reference beam) is directed at the recording medium. The interference pattern between the two lasers is recorded on the medium. The physical patterns on the recording medium appear to be random marks but when it is lit up in an appropriate manner, the light is diffracted in such a way that it creates a floating holographic image. The “father” of holography, Dennis Gabor, won the Nobel Prize in Physics in 1971.
Page 92 As noted, holograms were historically created by capturing the reflected light from actual physical objects. However, there has been growing interest in digital holograms, in which holograms are created from 3-D computer images. There are currently several companies that can transform digital images to holograms and some even sell “holoprinters” which enable users to create their own holograms. Specifically, the holoprinter analyzes a digital image and prints out holographic film which when lit up produces a holographic image. While the ability to view a static holographic image has many applications (e.g., an architect can show a 3-D image of what a building complex might look like before it is built, a doctor can view a 3-D image of a patient’s organs, etc.), the longer term goal is to create solutions that enable computers, mobile devices and gaming systems to project real-time dynamic holographic images (i.e., holographic 3-D images of anything that would normally be displayed on a flat 2-D screen). The classic example is the Princess Leia hologram projected by R2-D2 in the original Star Wars movie (“Help me, Obi-Wan Kenobi!”). Although this level of sophistication is still years away, there are a number of companies working towards developing advanced dynamic holographic solutions. This requires implementing some type of technology to rapidly change the way light is diffracted (simulate the holographic film interference patterns, but change them in real-time to make the images dynamic). A variety of approaches have been used (digital light mirrors, addressable liquid crystal, advanced light guides in the backlight, etc.) to implement the required patterns. However, there are still numerous engineering challenges as the amount of data and computations required for holograms is extremely large, the optics are complex, the images need to rapidly change, and relevant costs have to be driven down. For many of the initial prototypes, image quality is mediocre, size is small and there are other issues, but the technology is expected to greatly improve over the next few years. Another application for holography is telepresence. Telepresence includes a number of technologies that simulate a person being present or allowing a person to feel as if he is present at a location other than the one he is at. This goes beyond simple video conferencing and may include, for example, creating holographic/3-D images of the people on the other end of the line. While there have been some examples of commercial holographic image deployments and several announced “breakthroughs,” the technology is still early stage with respect to consumer applications. However, over the next several years we anticipate that there will be significant interest in holograms as users would prefer seeing and interacting with 3-D images without having to wear headsets or special glasses. Longer term, it could become a critical mainstream technology for a variety of products.
Page 93 Select Holography Companies A few examples of companies focused on holographic technology include: Zebra Imaging Zebra Imaging is a digital hologram company. Customers can upload computer images to the company and it converts the images into color or monochrome hologram prints that the customer can then use to create holographic images. Its holograms can be viewed using simple halogen or LED light sources. The company also sells its own line of holographic printers, enabling customers to create their own holographic prints and images. Zebra’s solutions are used across a variety of industries (construction, architecture, retail, etc.). Zebra recently announced the availability of a 3-D Hologram Creator plugin for Autodesk Revit software, enabling Revit users to quickly create designs that can be printed as 3-D holographic prints. Zebra also announced it signed a contract with the U.S. Department of Homeland Security to deliver high-resolution 3-D maps to Customs and Border Protection. Importantly, Zebra is currently focused on optimizing its ZScape holographic motion display solution, which is a self-contained system that links to a computer workstation via an Ethernet connection, and enables images displayed on a computer monitor to appear as 3- D holograms. It was initially developed with help from DARPA, but the technology has the potential for a broad range of markets, including high volume mainstream consumer applications. In November 2015, Zebra announced that it acquired Rattan Software which provided expertise to a variety of customers (Samsung, Virtuix Omni, etc.) on utilizing light field technology. Zebra is based in Austin, Texas. Holoxica Holoxica provides digital hologram prints to customers (i.e., customers provide digital image files and receive hologram prints that can produce holographic images when light is applied). It addresses a variety of markets including medical (holograms using data from CT and MRI scans), engineering design (holograms from CAD and other graphic models), and construction/architecture (holograms to show designs before physical construction). More significantly, the company is also developing advanced real-time interactive holographic displays. Its first generation solution enables simple dynamic holograms such as numbers, which can be used to create, for example, a holographic digital clock. Its second generation solution provides a variety of more complex images (but still flattish – similar to a heads-up display on a car windshield) and also enables the user to draw lines and figures in mid-air. Importantly, Holoxica is currently working on its third generation solution which would provide full 3-D volumetric holographic images and could be used for a variety of real commercial applications. The company is developing its solution such that the technology could be relatively easily commercialized within the current LCD supply chain.
Page 94 While Holoxica will initially focus on certain niche target markets, the company believes that longer term its holographic displays will have many high volume consumer applications. Holoxica is located at the Scottish Microelectronics Centre at Edinburgh University. RealView Imaging RealView Imaging is an Israeli holographic imaging company, with an initial focus on medical applications. Their system is able to transform 3-D digital data from medical imaging equipment and project 3-D holographic still images or video streams that float in mid-air. The projected image is in high resolution color and can be viewed from a variety of angles. The company has also developed a number of technologies to enable interaction with the images. Its “image intimacy” technology allows users to, for example, mark up the holographic images, modify them, or add other digital images to them. The company is working on a variety of other medical applications (fetal imaging, etc.). Nvidia has indicated that the RealView technology uses a cluster of Nvidia Tesla GPUs and Quadro cards. RealView completed a clinical study with Philips demonstrating the feasibility of using live 3- D holographic visualization to guide minimally-invasive heart disease procedures. While RealView has primarily focused on medical, longer term it believes its technology has a variety of applications in other markets including consumer electronics. LEIA 3D LEIA 3D has developed holographic technology that enables 3-D without eyewear. The company replaces the light guide that is standard in the backlight of virtually all LCD displays with a more complex light guide with nanoscale gratings. This diffractive backlight allows the projection of different images in various directions of space. This enables content that looks 3-D from any viewpoint and that can be updated at video rates. The company sells a development kit (including a holographic display module) for $999. Although the development kit is monochrome, the company indicates it already has color technology available. LEIA has also developed a technology called Hover Touch, which can sense the presence of a finger at some distance from the screen, enabling interaction with 3-D images. While it provides a development kit, it is primarily focused on working with OEMs interested in integrating its technology. The company indicates that its technology is designed such that it can be implemented with relatively little incremental cost relative to current displays and is targeting a variety of high volume mainstream applications. The company was founded by David Fattal, who invented the technology at HP Labs. LEIA is headquartered in Silicon Valley.
Page 95 Kino-Mo While true immersive hologram technology is still many years away, Kino-Mo has developed a solution that creates what appear to be holographic images. Specifically, it has developed a device with a rotating four arm propeller covered in LED lights and sensors. As the propeller rotates, microprocessors with advanced software algorithms track the LEDs and rapidly turn the LEDs on and off in such a way that it creates images that appear as true 3-D holographic images (the rotating blades themselves are not visible). At CES 2016, Kino-Mo displayed a broad range of complex 3-D images that appeared to be true (futuristic movie-like) holograms floating in mid-air and viewable from a variety of different angles (as long as you were in front of it). The company is initially targeting the technology for advertising but it could potentially have a variety of uses, including consumer applications. Kino-Mo’s initial product was a video bike in which advertisers can transfer images to the bike and the images are displayed on the bike’s wheels as the cyclist rides the bike (customers have included Intel, Samsung, GE, and BNP Paribas). The company is based in London.
Page 96 “And in the end….the love you take is equal to the love you make.” - Paul McCartney The technology landscape can change very quickly. A decade ago (when the leading mobile phone OEMs included Nokia, Motorola, and Sony Ericsson), who would have predicted that the handset market would soon be dominated by companies such as Apple, Samsung, Huawei, Xiaomi, LG, and Lenovo (with Google providing a significant portion of the operating systems)? Over the next decade, it is certainly possible that we will see continued market share leadership changes, and this may be based on new products incorporating innovative human interface technologies that greatly improve the user’s experience. There is growing interest in HMI technologies from a broad range of different types of companies (e.g., smartphone OEMs, consumer electronics companies, software companies, chipmakers, Web companies, social media companies, etc.). It is interesting to note that some of the companies listed in this report that have made user interface related acquisitions and/or investments include Apple, Google, Yahoo, Intel, Qualcomm, Microsoft, Amazon, Samsung, and Facebook. With HMI technologies becoming increasingly important, a variety of start-ups focused on innovative user interface solutions, and strong interest from many of the largest technology companies, there may continue to be a robust M&A market in this sector over the next several years. The technologies discussed in this report only scratch the surface of the types of human interface technologies that are being developed. It will be very interesting to see the number of new and innovative technologies that are created over the next decade! We hope this report has been useful. We tried to highlight a variety of technologies and companies, but obviously had to limit the number of topics covered, the amount of space devoted to each topic, and the number of companies covered. Our hope is to periodically issue future editions with updated data and additional topics. As a result, if there are any comments or recommendations, let us know. We will attempt to address any comments we receive in future editions. Finally, if you have, or know anyone that has, M&A interest (sell side or buy side) in this sector, or any area within technology for that matter, please give us a call. We look forward to speaking with you! Chapter 11 – Some Final Thoughts
Page 97 “I am a brain, Watson. The rest of me is a mere appendix.” - Sherlock Holmes The following table is a summary of many of the M&A transactions noted throughout this report. In many cases, the transaction itself and/or the valuation of the deal were not publically announced by the buyers or seller, and we relied on third party data sources for this information when available (which may not necessarily be completely accurate). In some cases, the Enterprise Value listed does not include earn-outs. Pagemill was an advisor on a number of the M&A transactions listed below, but we include data only if the data is publicly available. Sources: Capital IQ, 451 Group, company reports, and media reports. Date Acquirer Target Target Description Enterprise  Value ($M) Jan‐2016 Apple FlyBy Media Vision processing software for smartphones.     Jan‐2016 Apple Emotient Artifical intelligence technology to analyze user faces to determine  emotional state.   Dec‐2015 Atheer OntheGO  Platforms Gesture interfaces using standard cameras in mobile devices.   Nov‐2015 AMS CMOSIS CMOS image sensors. $235 Nov‐2015 Zebra Imaging Rattan Software Expertise in light field technology for a variety of applications,  including holographic imaging.   Nov‐2015 Apple Faceshift Facial analysis software that can enable an avatar to have the  same expressions as the user.   Oct‐2015 Sony SoftKinetic  Systems Gesture recognition solutions including ToF CMOS image sensors,  camera modules, middleware, and cameras.   Oct‐2015 Apple Vocal IQ Natural language self‐learning technology which can learn from  interactions with people over time.   Oct‐2015 Intel Saffron Cognitive computing platforms for analytics and decision making,  and potentially for consumer interaction. Oct‐2015 Sony Toshiba Image  Sensor unit Toshiba's image sensor unit and manufacturing facilities. $155 Oct‐2015 PTC Qualcomm  Vuforia unit Major provider of augmented reality technology solutions. $65 Oct‐2015 Blippar Binocular Augmented reality applications for customers.   Sep‐2015 Freescale CogniVue Vision processor chips and IP.     Appendix: M&A Transaction Summary
Page 98 Sources: Capital IQ, 451 Group, company reports, and media reports. Date Acquirer Target Target Description Enterprise  Value ($M) Sep‐2015 Snapchat Looksery Augmented reality video "lenses" that enable users to change their  appearance during video calls. $150 Aug‐2015 Voxx EyeLock Iris recognition technology including more than 70 patents. $20 Jul‐2015 Facebook Pebbles Interface Optics, sensor systems, and algorithms to detect and track hand  movement. $60 Jul‐2015 Cirrus Logic AGNITO's KIVOX  unit KIVOX voice authentication technologies for mobile devices for on  device voice matching.   Jun‐2015 Parade Cypress Mobile  Touch Unit Cypress' touch controller business unit for mobile devices  (Cypress continues to supply auto and appliances). $100 May‐2015 Facebook Surreal Vision Real‐time 3D scene reconstruction (accurate representation of the  real world in the virtual world). May‐2015 Apple Metaio Major supplier of augmented reality solutions, including an AR  publishing tool, an SDK, and other AR solutions.   May‐2015 Daqri ARToolworks Open source AR toolkit and a variety of AR patents.   Apr‐2015 Microsoft N‐Trig Stylus used for writing on tablets, including the Surface Pro 3. $30 Apr‐2015 Hua Capital  and others OmniVision Major supplier of CMOS image sensors. $1,900 Apr‐2015 Apple LinX Small camera modules for smartphones. $20 Apr‐2015 Google Tilt Brush Enables users to paint in 3D in virtual reality environments.   Apr‐2015 GoPro Kolor Virtual reality software that enables users to combine multiple  images or videos to create 360 degree videos. Mar‐2015 Intel Lemoptix MEMS scanning based pico‐projection solutions for handsets,  auto HUDs, gesture recognition, and wearables.   Jan‐2015 Facebook Wit.ai API for developing voice activated interfaces making it easier to  implement speech recognition/voice activation.    Dec‐2014 Facebook Nimble VR 3D camera for skeletal hand tracking for virtual reality type  applications.   Dec‐2014 Apple Privaris Patents A variety of patents related to fingerprint sensing technology.   Dec‐2014 Tessera Smart Sensors Iris recognition technology.   Dec‐2014 Facebook 13th Lab Augmented reality tracking and 3D construction framework  technology for SLAM and image detection.   Oct‐2014 Google Dark Blue Labs Deep learning technologies for understanding natural language  and what users are asking for.  
Page 99 Sources: Capital IQ, 451 Group, company reports, and media reports. Date Acquirer Target Target Description Enterprise  Value ($M) Oct‐2014 InvenSense Analog Devices'  MEMS Mic unit Analog Devices' MEMS microphone business unit. $100 Oct‐2014 Google Vision Factory Artifical intelligence technology to enhance the speed of object  recognition and improve vision based systems.     Sep‐2014 Dropbox KBVT Facial recognition software.   Sep‐2014 Qualcomm Euvision Artificial intelligence for improving image recognition including  an app that could analyze photos on a smartphone.   Sep‐2014 Twitter Madbits Deep learning technologies to understand the content of an image.   Aug‐2014 Google Jetpac Technology to analyze images and video and extract contextual  data.   Jul‐2014 InvenSense Movea Sensor fusion middleware solutions including technologies for  gesture control and activity monitoring. $61 Jun‐2014 Synaptics Renesas SP  Drivers Display drivers for small and medium size displays. $515 Jun‐2014 Parker Hannifin Bayer/Artifical  Muscle Unit Bayer MaterialScience's Artificial Muscle haptics business  (electroactive polymer based).    Jun‐2014 Audience Sensor Platforms Sensor fusion solutions for mobile devices.   Software to combine  and process sensor data. $41 Jun‐2014 ON Semi Aptina Major supplier of CMOS image sensors. $400 Jun‐2014 Blippar Layar Augmented reality platform.   May‐2014 Google Quest Visual Augmented reality solution that can capture images of words and  translate them and then display the translated words.   Apr‐2014 FocalTech Orise Merger of FocalTech (touch controllers) and Orise (display  drivers). $288 Apr‐2014 Cirrus Logic Wolfson Primarily audio analog chips but included a small MEMS  microphone unit. $467 Apr‐2014 ON Semi Truesense  Imaging Image sensors primarily for industrial markets. $92 Mar‐2014 Facebook Oculus Major supplier of virtual reality headset (Rift) and related  technologies. $2,000 Jan‐2014 Google DeepMind  Technologies General‐purpose, artificial learning, algorithm‐based simulation  software. $400 Jan‐2014 Qualcomm Kooaba  Mobile image recognition software application for software  developers and businesses to integrate into mobile applications. Jan‐2014 Fairchild Xsens 3D motion tracking using MEMS inertial sensing solutions. $58
Page 100 Sources: Capital IQ, 451 Group, company reports, and media reports. Date Acquirer Target Target Description Enterprise  Value ($M) Jan‐2014 Apple Novauris Automatic speech recognition technology that could process  several simultaneous voice access requests.   Jan‐2014 Pinterest VisualGraph Image recognition and visual search.   Dec‐2013 Google Industrial  Perception 3D vision systems for robotics and other applications.   Dec‐2013 Yahoo SkyPhrase Speech recognition software that enables users to more easily  communicate with computers by using natural language. Dec‐2013 TA Associates CMOSIS CMOS image sensors.   Nov‐2013 Apple PrimeSense 3D gesture recognition/machine vision technologies for digital  devices; provides SoC solutions and software. $345 Oct‐2013 Verint Systems Victrio Biometric voice recognition and authentication SaaS that enables  secure access to accounts and passive background monitoring. Oct‐2013 Yahoo LookFlow Image recognition and machine learning technology.   Oct‐2013 Synaptics Validity Sensors Fingerprint biometric sensors for use in identity authentication. $93 Oct‐2013 Google Flutter Gesture recognition software application that enables PC users to  control a range of applications via a built‐in webcam. $40 Sep‐2013 IDEX PicoField  Technologies Biometric consumer device fingerprint scanning and sensor  systems for mobile device manufacturers. Sep‐2013 Intel Indisys Natural language recognition software and Web and mobile  artificial intelligence interactive customer service software. $26 Aug‐2013 Facebook Jibbigo (Mobile  Technologies) Speech‐to‐speech mobile device based translation software that  translates spoken words for use with iOS and Android devices. Aug‐2013 Yahoo IQ Engines Image recognition technology that can determine which objects  are in a scene for photo tagging and other applications. Jul‐2013 Google SR Tech Group,  LLC ‐ Patents Portfolio of speech‐related patents including "Speech interface for  search engines".   Jul‐2013 Intel Omek Interactive Gesture recognition software and hardware for use in consumer  electronics, auto, gaming and healthcare. $50 Apr‐2013 Google wavii Smart phone news aggregation and reader application that  delivers customized news feeds . $30 Apr‐2013 Amazon Evi Technologies  Mobile speech‐recognition application that enables mobile device  users to receive answers to spoken questions and instructions. $26 Jan‐2013 Amazon Ivona Multi‐lingual speech synthesis systems. Jan‐2013 GEO  Semiconductor Maxim's  Compress. Unit Video compression technologies business unit.
Page 101 Sources: Capital IQ, 451 Group, company reports, and media reports. Date Acquirer Target Target Description Enterprise  Value ($M) Jan‐2013 TPK MasTouch  Optoelectronics Projected capacitive mode touch‐and‐control panel boards. Jan‐2013 Nuance VirtuOz Web‐based virtual assistant customer service software. Nov‐2012 Alcon SMI's Opthalmic  Unit Eye tracking solutions for opthalmic use (ocular surgery  guidance).   Oct‐2012 Google Viewdle Facial recognition software for mobile devices.  Also developing  image recogntion solutions (smiles, gestures, etc.). $40 Sep‐2012 Dropbox Anchovi Labs Computer vision and articifial intelligence technologies for  classifying objects.   Sep‐2012 Nuance Ditech Networks Voice‐to‐text conversion and voice quality assurance software. $23 Sep‐2012 Qualcomm EPOS  Development Embedded semiconductor software that uses digital positioning  technology to control user input movements. Aug‐2012 STMicro‐ electronics bTendo Embedded MEMS for use in video‐projection applications for  mobile device and digital camera manufacturers. Jul‐2012 Apple AuthenTec Biometric fingerprint authentication sensor systems and software  for computer and electronic device manufacturers. $356 Jul‐2012 Microsoft Perceptive Pixel Multi‐touch display technologies and large screen displays.   Jun‐2012 Facebook Face.Com Facial recogntion (primarily for recognition of faces in uploaded  photos). $60 Apr‐2012 Gores Group TE's Touch Screen  Unit Touch screen solutions business (sensor‐based touch screen  displays and computers sold under EloTouch brand). $380 Mar‐2012 Facebook SiteHawk Eye tracking solutions using conventional Webcams (acquisition  of the team).   Feb‐2012 VoiceTrust Perceive  Solutions Biometric speech‐recognition software for the purpose of  authenticating users. Dec‐2011 Nuance Vlingo Mobile speech‐recognition voice command application (Nuance  paid $196.3M, but already had an equity stake in Vlingo). $225 Nov‐2011 Amazon Yap White‐label voice‐to‐speech recognition SaaS. Aug‐2011 Nuance Loquendo Multi‐language recognition and text‐to‐speech conversion  software. $75 Jul‐2011 Qualcomm GestureTek Camera‐enabled gesture recognition software. Jul‐2011 Google PittPatt Facial recognition software that identifies human faces in  photographs and tracks the motion of human faces in video. $38 Jun‐2011 Nuance SVOX Desktop and mobile voice automation and authentication, speech  recognition and text‐to‐speech conversion software. $123
Page 102 Sources: Capital IQ, 451 Group, company reports, and media reports. Date Acquirer Target Target Description Enterprise  Value ($M) May‐2011 TPK Cando  Multi‐touch capacitive touch panel products (TPK acquired only  about 20% of the equity for about $190 million). $955 Feb‐2011 TandemLaunch  Technologies Mirametrix Gaze tracking systems for use in vision research and content  analytics applications. Jan‐2011 Google SayNow Voice/telephony related solutions including conference‐call type  voice meetings. Jan‐2011 Nuance PerSay Biometric voice authentication software. Dec‐2010 Google Phonetic Arts Speech synthesis (speech to text) technologies and solutions. Oct‐2010 Microsoft Canesta 3D image sensor semiconductors and software for consumer  electronics and videogames and auto. Sep‐2010 Apple Polar Rose Facial recognition for photo sharing sites. $29 May‐2010 Tessera Siimpel MEMS auto‐focus chips for smartphone cameras. $15 Apr‐2010 Apple Siri Mobile application that enables iPhone and iPod Touch users to  ask questions and receive answers and recommendations. $200 Mar‐2010 Bayer Material  Science Artificial Muscle User touch‐based interface devices and haptic actuators for use in  consumer electronics and medical devices. Jan‐2010 AuthenTec UPEK Biometric security and access control systems for computer and  electronics manufacturers. $32 Aug‐2009 Bosch Akustica Major supplier of MEMS microphones.   Mar‐2009 Microsoft 3DV Motion‐detection cameras used in gaming and web‐conferencing  by both consumers and enterprises. $35 Aug‐2005 Dover Knowles Major supplier of MEMS microphones. $750
Page 103 Notes
Page 104 Pagemill Partners Pagemill Partners A Division of Duff & Phelps Securities, LLC 1950 University Circle, Suite 400, East Palo Alto, CA 94303 Phone: +1-650-354-4000 Fax: +1-650-319-8791 This document is published solely for informational purposes, and is not to be construed as legal advice or tax advice. In addition, nothing contained herein is to be construed as an offer or recommendation, or the solicitation of an offer, to sell or buy a specific security, nor is any of the information contained herein intended to constitute an analysis of any company or security reasonably sufficient to form the basis for any investment decision. You are strongly encouraged to seek advice from qualified legal counsel for specific questions you may have regarding M&A transactions. The information herein is based in part on third party sources that we believe to be reliable, but have not been verified by us to be accurate. As such, we do not guarantee that the information is accurate or complete. Pagemill Partners is engaged in the business of providing M&A, private placement, and valuation advisory services to technology companies. Pagemill Partners may have provided in the past, may currently be providing, or may in the future provide advisory services to one or more of the companies mentioned in this document. The contents of this document are under copyright to Pagemill Partners. The material should not be copied nor distributed to other parties without the prior express written permission of Pagemill Partners. The Pagemill Partners logo is a trademark of Pagemill Partners. Copyright 2016 Pagemill Partners, a Division of Duff & Phelps Securities, LLC. All rights reserved. Member FINRA, SIPC.

More Related Content

PDF
Q1 2017-european-chemicals-update
Duff & Phelps
 
PDF
Industrial Distribution Industry Insights - January 2015
Duff & Phelps
 
PDF
TMUS Q3 20215 Earnings - Investor Factbook
TMUS_IR
 
PDF
Korean Film & Broadcasting Content Industry
Jeehyun Moon
 
PDF
[2016 Outlook] Media Advertisement
Jeehyun Moon
 
PDF
The World of Pay TV
Jeehyun Moon
 
PDF
Ernst&Young Luxury factbook-2015
Umberto Tessitore
 
DOCX
Hawthorne Distribution Global Written Report
Ryan Spetnagel
 
Q1 2017-european-chemicals-update
Duff & Phelps
 
Industrial Distribution Industry Insights - January 2015
Duff & Phelps
 
TMUS Q3 20215 Earnings - Investor Factbook
TMUS_IR
 
Korean Film & Broadcasting Content Industry
Jeehyun Moon
 
[2016 Outlook] Media Advertisement
Jeehyun Moon
 
The World of Pay TV
Jeehyun Moon
 
Ernst&Young Luxury factbook-2015
Umberto Tessitore
 
Hawthorne Distribution Global Written Report
Ryan Spetnagel
 

What's hot (20)

PDF
4Thought
Petra Smith
 
PDF
The auto industry's real challenge
Strategy&, a member of the PwC network
 
PDF
Retail sector update_2013
Nihat Kerim Bildik
 
PDF
Mobile content market_in_finland_2012-2016
Kalle Snellman
 
PDF
Q3 2015 Investor Call Slides
broadwind
 
PDF
TMUS Q4 2014 Investor Factbook
TMUS_IR
 
PDF
Aranca views - Consumer Staples − Losing Steam?
Aranca
 
PDF
Strategic sourcing outlook 2015
Philippe Pelland
 
PDF
EIB 2014
Richard Campbell
 
PDF
Future of manufacturing in China -2016 version
Fred Lemoine
 
PDF
Mercer Capital's Value Focus: Transportation & Logistics | Q1 2021 | Feature...
Mercer Capital
 
PDF
Annual manufacturing-report-2015
Rick Bouter
 
PDF
Posterscope Media Digest - April 2014
Posterscope
 
PDF
VTech Press Release Financial Report FY 2014 (See especially the CMS Sections)
Simpson & Simpson, PLLC
 
PDF
jabil circuit Annual Report 2002
finance22
 
PDF
Paradigm shift in supply chain management for chemical operating models
EY
 
PDF
Why the Banking Industry Needs $600 Billion Worth of M&A
Strategy&, a member of the PwC network
 
PDF
Responding to a crisis
accenture
 
DOCX
The Amr Research Supply Chain Top 25 For 2009
MarjoleinLondon
 
PDF
firm analysis
Oritsematosan Dudu
 
4Thought
Petra Smith
 
The auto industry's real challenge
Strategy&, a member of the PwC network
 
Retail sector update_2013
Nihat Kerim Bildik
 
Mobile content market_in_finland_2012-2016
Kalle Snellman
 
Q3 2015 Investor Call Slides
broadwind
 
TMUS Q4 2014 Investor Factbook
TMUS_IR
 
Aranca views - Consumer Staples − Losing Steam?
Aranca
 
Strategic sourcing outlook 2015
Philippe Pelland
 
EIB 2014
Richard Campbell
 
Future of manufacturing in China -2016 version
Fred Lemoine
 
Mercer Capital's Value Focus: Transportation & Logistics | Q1 2021 | Feature...
Mercer Capital
 
Annual manufacturing-report-2015
Rick Bouter
 
Posterscope Media Digest - April 2014
Posterscope
 
VTech Press Release Financial Report FY 2014 (See especially the CMS Sections)
Simpson & Simpson, PLLC
 
jabil circuit Annual Report 2002
finance22
 
Paradigm shift in supply chain management for chemical operating models
EY
 
Why the Banking Industry Needs $600 Billion Worth of M&A
Strategy&, a member of the PwC network
 
Responding to a crisis
accenture
 
The Amr Research Supply Chain Top 25 For 2009
MarjoleinLondon
 
firm analysis
Oritsematosan Dudu
 
Ad

Similar to Next-Generation User Interface Technologies for Mobile and Consumer Devices (2nd Edition) (20)

PDF
A Call to Action for Generative AI in 2024
Results
 
PDF
4400_F14_LIU_REPORT
Eric Liu
 
PDF
The Robos Are Coming - How AI will revolutionize Insurance 0117
Graham Clark
 
PDF
Mobile exalead-whitepaper-a4-8-lw
CarolineFlamand
 
PPT
Presentation To Seda Technology Programme
Elton050505
 
PDF
Romanko chasopys july2015_post
ProstirChasopys
 
PDF
mobility-new-enterprise-platform-2033534 (1)
Debora Cintron
 
PDF
Example of Analyst Coverage
Tim Peters, MBA
 
PDF
Vision mobile cross-platform_developer_tools_2012
SlashData
 
PDF
FinalProjectReport
Shane Stiller
 
PPTX
Webinar for Apr 2019 - AI Powered Insurer
The Digital Insurer
 
PDF
Product Development Project
Julie May
 
PDF
The 20 most admired companies to watch 2018
Insights success media and technology pvt ltd
 
DOCX
List of top 5 emerging technologies for business
SameerShaik43
 
PDF
Kindle Fire Hdx Essay
Renee Jones
 
PDF
Bringing AI into the Enterprise: A Machine Learning Primer
mercatoradvisory
 
PDF
Scaling Up IoT Business in International Market Using Golden Circle and Marke...
Yatindra shashi
 
PPTX
Micosoft PEST, SWOT
mortress
 
PDF
Various Types of Vendors that Exist in the Software Ecosystem
Pallavi Srivastava
 
PDF
The 10 most innovative embedded system solution providers 2018
Insights success media and technology pvt ltd
 
A Call to Action for Generative AI in 2024
Results
 
4400_F14_LIU_REPORT
Eric Liu
 
The Robos Are Coming - How AI will revolutionize Insurance 0117
Graham Clark
 
Mobile exalead-whitepaper-a4-8-lw
CarolineFlamand
 
Presentation To Seda Technology Programme
Elton050505
 
Romanko chasopys july2015_post
ProstirChasopys
 
mobility-new-enterprise-platform-2033534 (1)
Debora Cintron
 
Example of Analyst Coverage
Tim Peters, MBA
 
Vision mobile cross-platform_developer_tools_2012
SlashData
 
FinalProjectReport
Shane Stiller
 
Webinar for Apr 2019 - AI Powered Insurer
The Digital Insurer
 
Product Development Project
Julie May
 
The 20 most admired companies to watch 2018
Insights success media and technology pvt ltd
 
List of top 5 emerging technologies for business
SameerShaik43
 
Kindle Fire Hdx Essay
Renee Jones
 
Bringing AI into the Enterprise: A Machine Learning Primer
mercatoradvisory
 
Scaling Up IoT Business in International Market Using Golden Circle and Marke...
Yatindra shashi
 
Micosoft PEST, SWOT
mortress
 
Various Types of Vendors that Exist in the Software Ecosystem
Pallavi Srivastava
 
The 10 most innovative embedded system solution providers 2018
Insights success media and technology pvt ltd
 
Ad

More from Duff & Phelps (20)

PDF
Hospitalist M&A Landscape – Winter 2018 – 2019
Duff & Phelps
 
PDF
Food and Beverage M&A Landscape Late Fall 2018
Duff & Phelps
 
PDF
Healthcare Services Sector Update – November 2018
Duff & Phelps
 
PDF
Capital Markets Insights – Late Fall 2018
Duff & Phelps
 
PDF
Regulatory Watch: Asset Management Q3
Duff & Phelps
 
PDF
Industry Multiples India Report: Fifth Edition
Duff & Phelps
 
PDF
Restaurant Quarterly Update - Fall 2018
Duff & Phelps
 
PDF
LEED Market Study - Nov 2018
Duff & Phelps
 
PDF
Healthcare Services Sector Update – October 2018
Duff & Phelps
 
PDF
IP Value Summit 2018 - Agenda
Duff & Phelps
 
PDF
Regulatory Focus October 2018
Duff & Phelps
 
PPTX
Staffing Industry Insights - Fall 2018
Duff & Phelps
 
PDF
Medical Device Contract Manufacturing Update – Fall 2018
Duff & Phelps
 
PDF
Duff & Phelps Valuation Insights Greater China Edition October 2018
Duff & Phelps
 
PDF
Canadian M&A Insights Fall 2018
Duff & Phelps
 
PDF
Healthcare Services Sector Update September 2018
Duff & Phelps
 
PDF
Cost Trend Update Bulletin July-2018
Duff & Phelps
 
PDF
Regulatory Focus September 2018
Duff & Phelps
 
PDF
Hedge Fund and Private Equity Fund - Structures, Regulation and Criminal Risks
Duff & Phelps
 
PDF
Food and Beverage M&A Landscape - Summer 2018
Duff & Phelps
 
Hospitalist M&A Landscape – Winter 2018 – 2019
Duff & Phelps
 
Food and Beverage M&A Landscape Late Fall 2018
Duff & Phelps
 
Healthcare Services Sector Update – November 2018
Duff & Phelps
 
Capital Markets Insights – Late Fall 2018
Duff & Phelps
 
Regulatory Watch: Asset Management Q3
Duff & Phelps
 
Industry Multiples India Report: Fifth Edition
Duff & Phelps
 
Restaurant Quarterly Update - Fall 2018
Duff & Phelps
 
LEED Market Study - Nov 2018
Duff & Phelps
 
Healthcare Services Sector Update – October 2018
Duff & Phelps
 
IP Value Summit 2018 - Agenda
Duff & Phelps
 
Regulatory Focus October 2018
Duff & Phelps
 
Staffing Industry Insights - Fall 2018
Duff & Phelps
 
Medical Device Contract Manufacturing Update – Fall 2018
Duff & Phelps
 
Duff & Phelps Valuation Insights Greater China Edition October 2018
Duff & Phelps
 
Canadian M&A Insights Fall 2018
Duff & Phelps
 
Healthcare Services Sector Update September 2018
Duff & Phelps
 
Cost Trend Update Bulletin July-2018
Duff & Phelps
 
Regulatory Focus September 2018
Duff & Phelps
 
Hedge Fund and Private Equity Fund - Structures, Regulation and Criminal Risks
Duff & Phelps
 
Food and Beverage M&A Landscape - Summer 2018
Duff & Phelps
 

Recently uploaded (20)

PPTX
Financial literacy among Collage students.pptx
veonicapayton
 
PDF
Private Equity in Action: Sector-Specific Investments for High Growth”
SumitsouravMahapatra
 
PDF
01 KEY PROVISIONS on NGPA and PROFESSIONALIZATION.pdf
marnellielangit
 
PPT
Business Process Analysis and Quality Management (PMgt 771) with 2 Credit Housr
abenezargetachew1
 
PDF
Best Accounting Outsourcing Companies in The USA
Fino Patry
 
PPTX
Chapter 2 - Essentials of Economics.pptx
loubna26
 
PPTX
Case study for Financial statements for Accounts
AGHILA6
 
PPT
Relevant Information & Alternative Choice Decisions
Adugna37
 
PDF
MPEDA Export License Apply Online for Seafood Export License in India.pdf
Ykg global
 
PPT
Project_finance_introduction in finance.ppt
PrashantSharma109724
 
PPTX
Lecture on Perfect Competition of the course introduction to microeconomics
TaasinBinHossainAlvi1
 
PDF
2012_The dark side of valuation a jedi guide to valuing difficult to value co...
mkgyver
 
PPTX
Risk Based Audit - Key to managhe the bussines & Creating the value
AangSanjaya2
 
PPT
Joints.ppts qxqwcesceecsxwdsxwcdewd2wsew
d2kjwp2qgd
 
PPTX
DOC-20250604-WA0001.pbbgjjghhyt gg fromptx
Pankaj901069
 
PPTX
ratio analysis presentation for graduate
NaziaNaeemVisitingTe1
 
PPT
CompanionAsset_9780128146378_Chapter04.ppt
PENGFEIGAO7
 
PDF
Very useful ppt for your banking assignments BANKING.pptx.pdf
smitraiyni07
 
PDF
Chapterrrrrrrrrrrrrrrrrrrrrrrrr 2_AP.pdf
tkhanhha06
 
PDF
Pension Trustee Training (1).pdf From Salih Shah
Henry Tapper
 
Financial literacy among Collage students.pptx
veonicapayton
 
Private Equity in Action: Sector-Specific Investments for High Growth”
SumitsouravMahapatra
 
01 KEY PROVISIONS on NGPA and PROFESSIONALIZATION.pdf
marnellielangit
 
Business Process Analysis and Quality Management (PMgt 771) with 2 Credit Housr
abenezargetachew1
 
Best Accounting Outsourcing Companies in The USA
Fino Patry
 
Chapter 2 - Essentials of Economics.pptx
loubna26
 
Case study for Financial statements for Accounts
AGHILA6
 
Relevant Information & Alternative Choice Decisions
Adugna37
 
MPEDA Export License Apply Online for Seafood Export License in India.pdf
Ykg global
 
Project_finance_introduction in finance.ppt
PrashantSharma109724
 
Lecture on Perfect Competition of the course introduction to microeconomics
TaasinBinHossainAlvi1
 
2012_The dark side of valuation a jedi guide to valuing difficult to value co...
mkgyver
 
Risk Based Audit - Key to managhe the bussines & Creating the value
AangSanjaya2
 
Joints.ppts qxqwcesceecsxwdsxwcdewd2wsew
d2kjwp2qgd
 
DOC-20250604-WA0001.pbbgjjghhyt gg fromptx
Pankaj901069
 
ratio analysis presentation for graduate
NaziaNaeemVisitingTe1
 
CompanionAsset_9780128146378_Chapter04.ppt
PENGFEIGAO7
 
Very useful ppt for your banking assignments BANKING.pptx.pdf
smitraiyni07
 
Chapterrrrrrrrrrrrrrrrrrrrrrrrr 2_AP.pdf
tkhanhha06
 
Pension Trustee Training (1).pdf From Salih Shah
Henry Tapper
 

Next-Generation User Interface Technologies for Mobile and Consumer Devices (2nd Edition)

  • 1. Next-Generation User Interface Technologies for Mobile and Consumer Devices Topics Include: • Gesture Recognition • Eye Tracking • Touch Screens, Haptics and Display Technologies • Speech Recognition / Language Processing / Mics • Security Biometrics • Sensor Fusion and Middleware • Brain, Heart and Other Biometric Sensors • Image and Vision Processing • Virtual and Augmented Reality March 2016
  • 2. Page 2 Next-Generation User Interface Technologies for Mobile and Consumer Devices (2nd Edition) Report Overview Increasingly, the most differentiating features of consumer electronics devices are the user interfaces. Touch-screen displays have had a substantial impact on the smartphone market, and gesture recognition has played a major role in the gaming industry. Emerging user interface technologies have the potential to significantly affect market share in smartphones, tablets, computers, televisions and a variety of other electronic devices. As a result, there has been significant user interface technology M&A activity, including acquisitions by many of the largest technology companies (e.g., Apple, Google, Intel, Microsoft, Qualcomm, Facebook, Amazon, etc.). Given the growing importance of these technologies, there may be a robust M&A market in this sector over the next several years. This report provides a brief overview of a variety of mobile device user interface technologies. In addition to discussing the technologies themselves, the report also summarizes some M&A transactions in this sector and provides highlights of a variety of relevant private technology companies.
  • 3. Page 3 About the Author …………………………………………..….………………...… 4 About Pagemill Partners and Duff & Phelps ………………………....……… 5 Select Pagemill Partners’ Transactions ………………………....…………..… 6 Preface to the 2nd Edition ………………………………………....…………..… 8 Select Pagemill / Duff & Phelps Contacts …………………….……………… 10 Chapter 1 – Introduction …………………………………………....…………..… 11 Chapter 2 – Gesture Recognition ……………………………………………….. 13 Chapter 3 – Eye Tracking ………………………….…………………………..… 20 Chapter 4 – Touch Screens, Haptics and Display Technologies ………..… 24 Chapter 5 – Speech Recognition / Language Processing / Mics .……….. 38 Chapter 6 – Security Biometrics …………………………………………...….… 48 Chapter 7 – Sensor Fusion and Middleware ………………………………….. 58 Chapter 8 – Brain, Heart and Other Biometric Sensors …….………..…..… 61 Chapter 9 – Image and Vision Processing …………………………...……..… 68 Chapter 10 – Virtual and Augmented Reality ………………...…………....… 80 Chapter 11 – Some Final Thoughts ……………………………………………….. 96 Appendix – M&A Transaction Summary ……………………………………….. 97 Table of Contents
  • 4. Page 4 About the Author “Good bankers, like good tea, can only be appreciated when they are in hot water.” – Jaffar Hussein (Governor, Malaysian Central Bank) Mark Grossman, Managing Director Mark Grossman is a managing director at Pagemill Partners. At Pagemill, he has successfully completed more than 40 transactions (primarily M&A) around the world across a variety of technology sectors. Previously, Mark worked as an investment banker at SVB Alliant, where he was involved in a variety of technology M&A transactions. Prior to becoming an investment banker, Mark was a managing director and senior technology research analyst at several Wall Street firms. As an analyst, he wrote research on technology industry trends and covered dozens of public technology companies, and he received a number of accolades including the Wall Street Journal "Best on the Street" award. Mark began his financial career as an Associate at Goldman Sachs. Prior to graduate school, Mark was a Project Engineer at General Electric, where he managed the development of a variety of complex electronics systems. Mark has a bachelor's degree in electrical engineering, with a computer science minor, from Polytechnic University (now part of New York University), and both a master’s degree (Electrical Engineering) and an MBA from the Massachusetts Institute of Technology, where he completed a two-year dual- degree program. About the Author, Pagemill Partners, and Duff & Phelps
  • 5. Page 5 About Pagemill Partners, a Division of Duff & Phelps Securities, LLC Pagemill Partners is a premier investment bank focused on successful transactions for middle-market technology companies. Pagemill Partners specializes in merger and acquisition advisory services and has successfully completed more than 225 transactions since the beginning of 2005. These include transactions with IBM, Microsoft, Intel, Oracle, Computer Associates, Qualcomm, Salesforce, SAP, and many other industry leaders. Pagemill Partners works in close partnership with its clients to help achieve extraordinary outcomes, and is able to repeatedly do so through the team's combination of transaction experience, domain expertise, and personal commitment to the success and objectives of its clients and their shareholders. Pagemill's team has collectively completed hundreds of transactions across a wide range of industries, geographies, and strategic circumstances. Pagemill is a Division of Duff & Phelps Securities, LLC. About Duff & Phelps Duff & Phelps is the premier global valuation and corporate finance advisor, with expertise in complex valuation, dispute consulting, M&A and restructuring. The firm’s more than 2,000 employees serve a diverse range of clients from more than 70 offices in North America, Europe and Asia. To learn more, visit www.duffandphelps.com. M&A advisory, capital raising and restructuring services in the United States are provided by Duff & Phelps Securities, LLC, Member FINRA/SIPC. Pagemill Partners is a Division of Duff & Phelps Securities, LLC. M&A advisory and capital raising services in the United Kingdom and Germany are provided by Duff & Phelps Securities Ltd., which is authorized and regulated by the Financial Conduct Authority.
  • 6. Page 6 Select Pagemill Partners’ Transactions has been acquired by has been acquired by has been acquired by has been acquired by has been acquired by has been acquired by has been acquired by has been acquired by has been acquired by has been acquired by has been acquired by has been acquired by has been acquired by has been acquired by has been acquired by has been acquired by has acquired has been acquired by has been acquired by has been acquired by
  • 7. Page 7 Select Pagemill Partners’ Transactions (cont.) has been acquired by has been acquired by has been acquired by has been acquired by has been acquired by has been acquired by has been acquired by has been acquired by has been acquired by has been acquired by has been acquired by has been acquired by has been acquired by has been acquired by has been acquired by has been acquired by has been acquired by has been acquired byhas acquired has been acquired by
  • 8. Page 8 Update to the First Edition of the Report Introduction In the first edition of this report (issued in April 2014) we noted that, given the growing importance of human interface technologies for mobile devices, there would likely be a number of acquisitions in the sector. This has proven to be the case. In addition to a few high-profile M&A transactions (e.g., Facebook acquiring Oculus), a variety of companies in the sector, including several we highlighted in the first edition, were acquired. Some examples include the following (the appendix contains a more detailed list):  FlyBy Media (vision processing software) – Acquired by Apple  Faceshift (facial analysis software) – Acquired by Apple  SoftKinetic (3-D image sensor/gesture recognition) – Acquired by Sony  Vocal IQ (natural language technology) – Acquired by Apple  Saffron (cognitive computing) – Acquired by Intel  Qualcomm’s Vuforia Unit (augmented reality) – Acquired by PTC  CogniVue (vision processing) – Acquired by Freescale/NXP  Looksery (augmented reality “lenses”) – Acquired by Snapchat  Pebbles Interface (hand/gesture recognition) – Acquired by Facebook  Cypress’ Touch Unit (touch controllers) – Acquired by Parade  Metaio (augmented reality) – Acquired by Apple  N-Trig (stylus for tablets) – Acquired by Microsoft  Kolor (virtual reality software) – Acquired by GoPro  Lemoptix (MEMS pico-projection) – Acquired by Intel  Nimble VR (hand tracking) – Acquired by Facebook  13th Lab (augmented reality tracking) – Acquired by Facebook  Analog Devices’ MEMS Mic Unit (MEMS microphones) – Acquired by InvenSense.  Vision Factory (image and object recognition) – Acquired by Google  Movea (sensor fusion software) – Acquired by InvenSense  Sensor Platforms (sensor fusion software) – Acquired by Audience (which was later acquired by Knowles) Preface to the 2nd Edition
  • 9. Page 9 In addition to M&A activity, several relevant companies have completed significant capital raises since the first edition. A few examples include:  Magic Leap (augmented reality) – Raised $542 million in late 2014 and just recently announced a new $793.5 million funding  Movidius (vision processing) – Completed a $40 million private capital raise  Tobii (eye tracking) – Had an IPO on the Nasdaq Stockholm exchange There has also been a substantial amount of new product activity. For example, Microsoft announced HoloLens; Samsung introduced Gear VR; and Sony and HTC have demonstrated virtual reality solutions. Given all of the recent activity, we thought the timing was right for a second edition. Beyond updating M&A transaction activity, the second edition also discusses recent sector and technology developments.
  • 10. Author Mark Grossman Managing Director Silicon Valley / Boston [email protected] U.S. M&A Contacts Christian Bennett Managing Director Silicon Valley [email protected] Gaurav Bhasin Managing Director Silicon Valley [email protected] Milledge Hart Managing Director Silicon Valley [email protected] Kevin Iudicello Managing Director Silicon Valley [email protected] James Kochman Managing Director Silicon Valley [email protected] Patrick Manion Managing Director Silicon Valley [email protected] Nels Nelsen Managing Director Silicon Valley [email protected] Matt Sachse Managing Director Silicon Valley [email protected] Charles Welch Managing Director Silicon Valley [email protected] Canada M&A Contacts Scott Munro Managing Director Toronto / Silicon Valley [email protected] Ross Fletcher Head of Canada M&A Toronto [email protected] China M&A Contacts David Lu Head of China M&A Shanghai / Hong Kong [email protected] Germany M&A Contacts Andreas Stoecklin Country Head for Germany Munich / Frankfurt [email protected] Klaus Pflum Senior Advisor Munich [email protected] Ireland M&A Contact Aidan Flynn Director Dublin [email protected] UK M&A Contact Henry Wells Head of UK M&A London [email protected] Contact Us
  • 11. Page 11 “The last thing one knows – is what to put first.” – Blaise Pascal Introduction In 2007, Apple revolutionized the technology industry with the introduction of the iPhone. While the iPhone had many impressive characteristics, the most differentiating feature was its touch screen user interface, which helped catapult Apple into a leadership position within the smartphone market. Apple had previously transformed the computer industry when it introduced the Macintosh in 1984. The most significant feature of the Macintosh was its innovative computer mouse interface, which enabled intuitive “point and click” use rather than having to type in long obscure DOS commands. Nintendo dominated the gaming market in 2007 and 2008 as a result of the Nintendo Wii. The key feature of the Wii was its Wii Remote controller, which could detect 3-D motions by the user and made games much more interactive. Microsoft subsequently gained significant share in gaming owing in large part to its Kinect gesture recognition technology. As all of these examples indicate, the features that most differentiate electronics products are commonly human–machine interface (HMI) technologies. As form factors continue to shrink in size and become more pervasive (e.g., wearable devices, automobile electronics, smart home), HMI technologies will become increasingly critical. Not surprisingly, there has been growing interest from a variety of major technology companies. For example, Intel Capital created a $100 million fund a few years ago to invest in “perceptual computing” technologies such as 3-D image sensors, touch apps, imaging, gesture, voice, emotion sensing and biometrics. Many of the largest technology companies have already made one or more acquisitions in the HMI space (e.g., Microsoft, Intel, Qualcomm, Amazon, Apple, Google, Facebook). While there have already been a number of important acquisitions in this sector, there may continue to be a robust M&A market in HMI technologies over the next several years. This report provides a brief overview of next-generation user interface technologies that either are currently used in mobile electronics or may be used in the future. These include gesture recognition, eye tracking, speech recognition, touch screen haptics, vision/image processing, virtual reality, augmented reality, and biometric sensor technologies. The report also includes a brief overview of human interface technologies used for security (e.g., fingerprint sensing, voice authentication) and sensor fusion. In addition to providing overviews of each technology, the report also discusses M&A activity in each sector and provides descriptions of a variety of innovative private companies addressing next-generation user interface technologies. Chapter 1 – Introduction
  • 12. Page 12 A Note on Highlighted Companies and M&A Deals The highlighted companies in this report (most of which are private companies or small non-U.S.-based public companies) are not meant to be an all-inclusive list., There are hundreds of start-ups addressing next-generation HMI technology; however, we believe it provides a good sampling of what is out there. We included companies based on the strength of their technology, degree of innovation, and level of public information, but also tried to include a mix of early stage start-ups and more well-known later-stage companies. Importantly, while we may have had calls or meetings with some of the highlighted companies, we have included only publicly available information—Meaning, in many cases, these companies may have extremely interesting next-generation solutions or unannounced design wins, but we included only information that is either on their Web sites or in published articles. Similarly, Pagemill was an advisor for a number of the transactions listed in this report, but we included deal information only if available from public sources. Report Overview The chapters in this document include:  Chapter 2 provides an overview of gesture technologies, in which users can control applications through the movement of their hands, fingers, head or body.  Chapter 3 discusses eye tracking, in which a person’s gaze can be used to control what is displayed on the screen.  Chapter 4 describes touch screens, touch screen haptic technologies, and other emerging display technologies.  Chapter 5 covers speech recognition, voice assistant technologies, MEMS microphones and natural language processing.  Chapter 6 describes security biometric technologies such as fingerprint sensors, voice authentication, facial recognition and eye recognition.  Chapter 7 contains an overview of sensor fusion and middleware solutions that utilize the data from a variety of sensors to enhance the user experience.  Chapter 8 provides an overview of biometric sensors (heart rate, brain wave, pulse, blood pressure, air quality, etc.) for health/wellness applications.  Chapter 9 discusses a variety of image and vision technologies such as image sensors, image recognition, vision processing and other related topics.  Chapter 10 describes virtual and augmented reality technologies and also briefly discusses holographic imaging.  Chapter 11 provides some final thoughts.  The Appendix summarizes M&A transactions mentioned throughout the document.
  • 13. Page 13 “I have a big problem with conductors who gesture a lot.” – James Levine Introduction Gesture recognition refers to technologies in which the movement of a person’s hands, fingers, head or body can be used to control electronics. In one common implementation, one or more optical sensors capture images of the user and advanced software interprets the images to determine what action (if any) the user is requesting. For example, a TV viewer might be able to bring a finger to his lips to mute sound or a PC user could scroll by moving her hands up or down. This is generally much more intuitive for users than typing on a keyboard, and can significantly enhance usability. The highest-profile implementations of gesture recognition have been in gaming. As noted in the Introduction chapter, Nintendo’s Wii changed the gaming market with its Wii Remote controllers. Microsoft subsequently rolled out its Kinect technology, which enables users to play games and make selections using only hand and body gestures, without holding any type of controller. Gesture recognition has also been implemented in a number of high-end televisions (e.g., LG and Samsung remote controls) and computers (e.g., Intel’s RealSense). There is also significant interest in implementing gesture recognition in cars, to enable drivers to control certain car features (make selections from the entertainment system, answer a call, etc.) with just a wave of the hand, as that is viewed as much safer than having the driver look down to press buttons on the dashboard. There are a number of methods for implementing gesture recognition. The Wii Remote, for example, included an MEMS accelerometer that could detect user movements. The Wii MotionPlus further improved motion detection by adding a MEMS gyro. However, there is growing interest in implementing gesture recognition vision systems in which optical sensors repeatedly capture images of the user, and complex software algorithms are then used to determine what the user’s gestures are, without the user having to hold any device. As with most of the technologies described in this report, it is relatively easy to develop a simple gesture recognition system that works in ideal conditions, but the challenge is to make it robust under a broad range of circumstances. For example, with vision-based gesture systems, users can be at different distances and angles from the image sensors and have a broad range of different physical characteristics and movements. Additionally, there can be many different types of lighting conditions. Consumers can quickly become frustrated with new technologies when they work only intermittently or when there are latencies, so making these solutions robust is critical but can take many years of development and optimization. Chapter 2 – Gesture Recognition
  • 14. Page 14 As many devices now incorporate conventional CMOS image sensors, optical gesture recognition can potentially be implemented with no additional hardware costs. However, to improve the performance of optical gesture recognition systems, some solutions utilize more specialized optical imaging sensors. Specifically, optical gesture recognition can generally be made more robust when images are represented in 3-D (with a “z” dimension), rather than with conventional 2-D representations. For example, with a 2-D representation it may be difficult to determine if a person’s hand is moving toward or away from the camera. As a result, a number of solutions have been developed that can capture “3-D” images (stereoscopic, time-of-flight, etc.). With stereoscopic solutions, two separate image sensors are used to capture the same image but from slightly different locations (a “right” and “left” image). By comparing the two images, software can estimate the distance of objects in the image. Another approach is referred to as “time-of-flight” (“ToF”). ToF is a type of light detection and ranging (LIDAR) technology. Light (typically invisible infrared light) is emitted and the reflected light from each object in the image is captured by an array of sensors. By calculating the amount of time it takes for the infrared light to complete a round trip, the distance of each object in the image can be determined, enabling a 3-D representation of the image. This greatly enhances the ability for accurate gesture recognition, but also adds some additional cost (e.g., infrared LEDs, image sensors that capture infrared light) relative to 2-D solutions. ToF was part of the gesture recognition technology developed by Canesta (acquired by Microsoft). While vision-based gesture recognition is most common, a number of other gesture recognition technologies have been developed, based on technologies such as ultrasound and electrical impulses from muscles. Gesture recognition has been implemented by a variety of major technology companies in a broad range of devices. Some examples include:  Microsoft accelerated the market for gesture recognition with the original Kinect accessory product and Kinect is a key part of the Xbox One gaming system.  Intel has announced its RealSense 3-D cameras which add depth-sensing to help improve the accuracy of gesture recognition and facial movements.  A variety of companies have introduced gesture-related remote controls for TVs.  Gesture recognition is an increasingly important part of virtual and augmented reality solutions. For example, if a user tries to grab something in a virtual reality game, but the system cannot accurately track his hand, it will result in a poor user experience.
  • 15. Page 15 Select Gesture Recognition M&A Transactions Some notable acquisitions in the gesture recognition space have included the following:  Atheer/ONtheGO Platforms – In December 2015, Atheer (an augmented reality smart glasses company) announced an agreement to acquire ONtheGO Platforms. ONtheGo created gesture interfaces using standard cameras in mobile devices. Atheer indicated that the technology will be used to give customers an enhanced ability to use natural gestures with its smart glasses platform.  Sony/SoftKinetic – In October 2015, Sony announced that it completed the acquisition of Belgium-based SoftKinetic Systems. SoftKinetic was highlighted in the first edition of this report. It had developed a suite of gesture recognition solutions including 3-D time-of-flight CMOS image sensor chips, module level cameras, and complete stand-alone cameras. In addition, the company developed a substantial amount of middleware to enable easy implementation of gesture recognition. Sony indicated that it will focus on combining SoftKinetic's ToF range image sensor technology expertise with its own technologies with the aim of developing a next generation of range image sensors and solutions.  Facebook(Oculus)/Pebbles Interface – In July 2015, Oculus announced that they had acquired Israeli-based Pebbles Interface. Pebbles had spent five years developing optics, sensor systems and algorithms to detect and track hand movement. Oculus is expected to utilize Pebbles’ hand tracking technologies for the Oculus virtual reality solutions (e.g., enabling the system to more accurately track a user’s hand movements, so when a user moves his hands in the real world it can accurately control actions in the virtual world). The value of the transaction was not announced, but the Wall Street Journal reported that one person close to the transaction indicated a value around “$60 million.”  Facebook(Oculus)/Nimble VR – In December 2014, Nimble VR reported that it was joining Oculus. Nimble had developed its own 3-D camera (Nimble Sense) primarily for skeletal hand tracking in order to improve the virtual reality experience (tracking the user’s hand movements to improve virtual reality play). Nimble’s demo of Nimble Sense had already used the Oculus virtual reality headset.  Fairchild/Xsens – In January 2014, Fairchild Semiconductor completed the acquisition of Xsens for approximately $57.8 million in cash (per Fairchild’s 10-K). Xsens developed 3-D motion tracking products for professional applications based upon miniature MEMS inertial sensor technology. Xsens targeted markets such as robotics, marine industry, unmanned aerial vehicles, 3-D animation, virtual reality, and sports/health science. Xsens was based in The Netherlands.
  • 16. Page 16  Apple/PrimeSense – In November 2013, Apple acquired Israeli-based PrimeSense for reportedly about $345 million (per the 451 Group). PrimeSense initially developed 3-D sensing software and camera systems utilizing off-the-shelf components, and later developed its own 3-D imaging SoC chip as well. The Company was best known for licensing the hardware design and chip used in the original Xbox 360 Kinect motion sensing system. Its “light coding” technology uses near-IR light and off-the-shelf image sensors. When reflected off images, the light is distorted based on the distance and this information is used to determine distance.  Google/Flutter – In October 2013, Google acquired Flutter, a private start-up which developed a gesture recognition app that worked with conventional Web cameras (requires only downloading the software). The transaction size was not announced but the 451 Group estimated the valuation at approximately $40 million.  Intel/Omek Interactive – In July 2013, Intel acquired Omek Interactive. Omek was an Israeli company – in Hebrew, Omek means “depth” – focused on the gesture recognition sector. The company developed software for creating a gesture recognition interface based on information provided by a 3-D camera. Intel Capital had been an investor. The transaction valuation was estimated to be approximately $50 million (per the 451 Group).  Qualcomm/GestureTek and EPOS – In July 2011, Qualcomm announced that it acquired certain assets from GestureTek. Qualcomm noted that the technology would be integrated into future versions of its SnapDragon solutions. Qualcomm also acquired ultrasound technologies for stylus and gesture recognition from Israeli- based EPOS Development in September 2012.  Microsoft/Canesta and 3DV – Canesta was a private company that developed CMOS 3-D image sensors. Unlike conventional image sensors, Canesta’s “time of flight” technology also measured the arrival time of collected photons, enabling the distance of each pixel of the image to be calculated which could then be used to determine the distance of objects from the sensor (essentially LIDAR on a chip). Honda had been an investor in Canesta, as Canesta had been working on using its technology to determine the distance of nearby cars (per a New York Times article) but later focused on video game applications. In October 2010, it was announced that Microsoft was acquiring Canesta. Separately, in 2009, Microsoft reportedly acquired 3DV Systems for about $35 million (per the 451 Group). 3DV had developed 3-D motion detection cameras.
  • 17. Page 17 Select Private Gesture Recognition Company Overviews There are a variety of private companies focused on gesture recognition. This section highlights a few of them. We included a depth sensing ToF sensor company focused primarily on gesture recognition (pmd), but note that some depth sensing technology companies that primarily address other markets are included in Chapter 9. Gestigon Gestigon has developed advanced hand and finger gesture recognition software for use in a variety of applications. The company believes its technology is much more accurate than other approaches. Its initial target market has been automotive (using gestures to control infotainment, navigation, and climate features within a car) and the company has indicated that it is actively working with several major automotive companies. However, it has more recently begun addressing consumer electronics and other applications. In September 2015, Gestigon announced Carnival, an extension of its product line to address virtual reality and augmented reality markets. Its technology can track hands and provide a life-like representation of a user’s hands within a virtual environment making the virtual reality experience more realistic. At CES 2016, Gestigon announced a project with pmd (discussed later) that combines Samsung’s GearVR, pmd’s depth sensor, and Gestigon’s Carnival AR/VR Interaction Suite to showcase virtual reality. It also collaborated with Inuitive (a vision processor company discussed in Chapter 10) for virtual reality applications. Gestigon completed a Series A capital raise in September 2015. The company was founded in 2011 and is based in Germany with an office in Silicon Valley. MUV Interactive MUV Interactive has developed “The Bird,” which is a small device that a user wears on the finger. Using a combination of inertial sensing and optical sensing, Bird can accurately track finger movement (gesture recognition) which can then be used for a broad range of control applications. For example, Bird works with a TV or projector to transform any surface or image into a “touchscreen” without requiring any actual touching. That is, if an image of a Web site is projected on a wall from an overhead projector connected to a notebook PC, a user with the Bird can scroll up and down, “click” on a link, zoom in and out, move a pointer around on the image, or perform a variety of other tasks by pointing the finger with the Bird on it towards the projected image and moving the finger around. Bird can operate even if the user is across the room from the TV or projected image. Bird can also be used to accurately control drones. It integrates a microphone (to enable voice control), a small capacitive touch surface (for taps and swipes), and a Bluetooth interface. MUV believes the solution has many applications ranging from business presentations, classrooms, drones, smart televisions, and home appliances. The company’s Web site allows pre-orders of Bird indicating that it will begin shipping in Spring 2016. MUV is based in Israel.
  • 18. Page 18 Pmd Technologies Pmd Technologies focuses on 3-D time of flight depth sensing solutions, with gesture recognition being one of the major applications. The company has developed its own 3-D ToF imaging sensing chip (PhotonICs) but has more recently been collaborating with Infineon on the jointly developed Real3 family which combines pmd's ToF distance- measurement technology and Infineon's manufacturing capabilities (including a specialized CMOS process optimized for ToF using micro-lens technology). The two companies indicate that the Real3 sensors are used as part of Google’s Project Tango (discussed later in the report). Pmd has also developed a number of complete 3-D depth cameras that integrate either the PhotonICs or Real3 chip and provide a complete reference design for OEMs. Its upcoming CamBoard pico flex solution is targeted at integration into smartphones. Pmd addresses consumer, industrial, and automotive markets and is based in Germany. Chirp Microsystems Chirp has developed advanced gesture recognition solutions. However, in contrast to most gesture solutions that rely on image sensors and video, Chirp’s solutions utilize ultrasonic audio. That is, a transducer creates ultrasonic waves and those waves are reflected by objects. A sensor measures the echoes and based on the ToF of the ultrasonic waves, the Chirp solution can determine the location of nearby objects. By repeatedly taking measurements, gestures can be recognized. Analyzing audio is orders of magnitude easier than analyzing video, thus Chirp believes its solution has many compelling advantages, including dramatically lower computational requirements, significantly less power consumption, and smaller size. For devices that don’t already have embedded image sensors, it is also significantly less expensive. Chirp is targeting consumer electronics, smart homes, industrial automation, and IoT. The technology was originally developed at UC Berkeley and Chirp is located in Berkeley, California. Leap Motion Leap Motion has developed a peripheral device (Leap Motion Controller) that connects to a PC or Mac computer or notebook via a USB cable and enables hand gesture recognition. The device is relatively small (only about 3” long) and sits on the table in front of a computer. The Controller tracks hands movements at up to 200 frames per second using infrared cameras and enables a broad range of gesture related controls. A variety of games and apps have been developed that utilize the Leap Motion Controller and the company has an app page on its Web site. The Leap Motion Controller sells for $79.99 on the company’s Web site. Leap Motion announced a beta version of Leap Motion for virtual reality (e.g., enabling a user to see their hands move in virtual reality). It recently announced that it signed a Memorandum of Understanding with SK Telecom for 5G “AR/VR Service Development.” One of its major investors, Highland Capital, has established a $25 million fund for companies developing technologies based on the Leap Controller. Investors in Leap Motion include Highland Capital, Andreessen Horowitz, Founders Fund, and SoSventures International.
  • 19. Page 19 PointGrab PointGrab was highlighted in the 1st edition of this report as an early pioneer in gesture recognition for consumer devices, but has shifted its focus since then to home automation and IoT applications. Its PointSwitch solution is an optical module that utilizes machine learning algorithms to analyze visual input. Its technology can sense occupancy, ambient light, and occupants’ behavior, which can be used to reduce energy consumption, improve comfort, and lower operational costs. The company indicates that its solutions are being developed in collaboration with leading providers in lighting and building control systems. PointGrab is based in Israel. Extreme Reality Extreme Reality (EXT3-D) has developed 3-D motion analysis software solutions that work with conventional CMOS image sensors. The company’s Extreme Motion technology captures images of people and then creates skeletal models of the people, which are then used to analyze motion. The company believes this core “skeleton positioning technology” enables much more accurate body motion analysis than other conventional body motion tracking solutions and it can be implemented on any system with a basic Webcam. The company’s technology has been incorporated into a number of games. It also announced the technology is incorporated into Warchief and that it has a partnership with TVisual, a leader in China‘s digital signage market for interactive signage. Extreme Reality was founded in 2005 and is based in Israel. Thalmic Labs The bulk of gesture recognition technologies are based on optical systems, but Thalmic Labs has developed a nine-axis motion sensor armband (Myo) that can read the electrical signals from muscle movements in the forearm. The armband incorporates a gyroscope, an accelerometer, and a magnetometer. As a result, the armband can identify a broad range of bodily gestures as well as the movement parameters of the arm (direction, speed, and angle). The output of these sensors is converted to digital and processed and the resulting commands are sent via Bluetooth to a smartphone or computer. The company supports a variety of games (e.g., Fruit Ninja) and applications (e.g., controlling a Parrot drone) and opened an online marketplace for the more than 100 existing Myo-based apps. Thalmic is based in Canada and its investors include Salesforce.com CEO Marc Benioff and ATI co-founder Lee Lau.
  • 20. Page 20 “Beauty is in the eye of the beer holder.” - Kinky Friedman Introduction Eye tracking involves determining where a person is looking at or the motion of the eye. This can then be used to control what is displayed on the screen or to help make selections. Eye tracking is not a new concept. For example, in the early 1900s a non- intrusive eye tracker was developed that reflected light off of the subject’s eyes and recorded the results on film. During the 1950s and 1960s, Alfred Yarbus performed a variety of eye tracking experiments and published the first major book on the subject. Most of the early eye tracking solutions were developed for psychological testing, but during the 1980s and 1990s eye tracking had some limited commercial use for marketing studies and to assist handicapped people with computers. More recently, however, there has been growing interest in utilizing eye tracking technology as an interface for electronic devices, including smart phones, computers, and automotive applications. Although there are a variety of different eye tracking technologies, most of the solutions addressing consumer applications utilize the optical method in which light (in many cases infrared light) is reflected by the eye and then captured by an image sensor. The images are then analyzed to determine where the user is looking. Because different parts of the eye (cornea, pupil, sclera, etc.) reflect and absorb light differently, the direction of a person’s gaze can be determined from the images. This can then be used to control the location of the cursor/pointer on a display or to control the display in other ways. For example, a user looking at a map on a display could potentially adjust the map in any direction just by looking at the edge of the map, or zoom in just by staring at a certain spot for a period of time, making it easier to navigate. While many eye tracking solutions have been developed, the challenge in commercializing the technology is making it work consistently across a broad range of users (e.g., different eye types, some with contact lens, some with glasses, etc.) in many different environments (e.g., dark rooms, light rooms, etc.) under a variety of circumstances (users in different positions, viewing angles and distances from the screen). In addition, penetrating the consumer market requires a low cost, small form factor with low power consumption. Historically, many of the eye tracking solutions worked well in ideal conditions but had problems in real world situations. Chapter 3 – Eye Tracking
  • 21. Page 21 While eye tracking solutions vary, they often include: a light source (it is possible to rely on conventional lighting, but many solutions use one or more LED lights, and often infrared LED sources are used as infrared light is not visible to the human eye); an image sensor (in some cases a traditional CMOS image sensor and in other cases an infrared image sensor to improve performance); a processor (in some cases the main processor is relied on to handle the analytical processing, but some solutions have a dedicated chip for this); and software (which analyzes the captured images to determine where the user is looking and provides an interface to various applications). There are a number of applications for which simply gazing at a particular point can cause the application to take action. For example, a mapping program could scroll up, down, left or right if the user stares at certain spots on the edges of the map for a period of time. A game could adjust the scene based on which direction the user is looking. However, one general limitation of eye tracking is that in many cases it still requires using some other method to “click”/select the item that the user is gazing at (i.e., using gaze, a user can move the screen pointer to a specific menu item on the screen, but then must hold his gaze while he presses a key on the keypad or uses his voice to select it). Making selections based solely on gaze is challenging since it can lead to inadvertent selections as the user examines his choices. Blinking has been explored as a way to make selections (stare at a point and then blink for a period of time to select) but that is also challenging to implement since people naturally blink every few seconds. However, there are applications in which gaze can be used to make choices (staring at a spot for a certain length of time), especially if the app is specifically optimized for eye tracking. Eye tracking is still early stage with respect to commercial adoption in high volume consumer devices, but there appears to be growing interest in it for a variety of applications (e.g., games) and one eye tracking company, Tobii, had an IPO in 2015. In addition, it could become more important for virtual/augmented reality applications which historically tracked head movement (i.e., change the view as the user turns his head), but could provide a more realistic experience by also adjusting based on eye movement. There is also interest in optimizing AR and VR displays by providing higher resolution to the area that the user is looking at versus other areas (foveated rendering), which can provide a variety of advantages. Eye tracking can also be used for reducing power consumption (e.g., the screen dims when the user isn’t looking at it). Eye Tracking M&A Below are a few eye tracking acquisitions. Note that there have been a number of eye- related security biometric deals, but these are covered in the chapter on biometric security. There have been relatively few eye tracking M&A transactions, likely because eye tracking has not yet gained significant traction in high volume products yet. This could change if the technology becomes more commonplace in consumer and virtual reality solutions.
  • 22. Page 22  Alcon/SMI’s Ophthalmic Unit – In November 2012, Alcon announced that it acquired the Ophthalmic division of SMI, a private company based in Berlin (discussed below). SMI addresses a variety of eye tracking markets, but sold off its Ophthalmic unit (which specializes in ocular surgery guidance technology) to Alcon.  Facebook/SiteHawk – In March 2012, SiteHawk announced that its team was “joining the Facebook team.” SiteHawk developed eye tracking technology that used conventional Web cameras (rather than any special infrared or other type of specialized hardware). SiteHawk noted that the deal was only for the team and that its existing products were not part of the deal and remain independent of Facebook.  TandemLaunch/Mirametrix – In 2011, TandemLaunch (an accelerator/incubator for tech companies) acquired Mirametrix, a private company that developed eye tracking technologies and solutions. Select Eye Tracking Companies The following are brief overviews of some eye tracking companies: Tobii Tobii has developed a number of eye tracking solutions. Its products incorporate near-IR micro-projectors, optical sensors, and image processing software to determine the location a person is looking at. The Company has several business units: Dynovox (which is focused on eye tracking for users that have ALS, cerebral palsy, Parkinson’s or other medical issues that prevent natural hand movement); Pro (eye tracking for market research); Tech (for OEMs looking to integrate eye tracking), and EyeX (an eye tracking controller that Tobii sells to consumers and developers). In a September 2015 investor presentation, Tobii indicated that Dynovox and Pro accounted for about 73% and 21% of revenue, respectively. Although consumer applications historically represented only a relatively small portion of sales, the company has announced a number of video games that utilize its eye tracking (Assassin’s Creed Rouge, Flight Simulator, etc.) and that it is targeting a variety of high volume end markets. Tobii also supports power consumption reducing features such as dimming the screen if the user is not looking at it and keeping the screen awake if the user is looking (even if they hadn’t touched a key for several minutes). Tobii also powers biometric facial recognition for Windows 10 (Windows Hello). Tobii has announced its own ASIC (EyeChip) for eye tracking which can make implementation easier for some applications and further optimizes performance. Tobii is based in Norway with offices in the U.S., Germany, Japan and China. The company went public on the NASDAQ Stockholm market in April 2015.
  • 23. Page 23 SensoMotoric Instruments (SMI) SensoMotoric Instruments (SMI) was founded in 1991 and has developed a broad range of computer vision products including eye tracking peripherals and glasses. Its Eye Gaze and Tracking Systems unit sells a variety of advanced eye tracking solutions (systems, glasses, software, etc.) for professional and research applications (e.g., sports training, market research, psychology studies, etc.). The company indicates that more than 6,000 of its eye tracking systems are in operation and some of its listed customers include HP, Rite-Aid, and Volkswagen. It recently announced an automated analysis solution that eliminates the need for manual coding and greatly simplifies eye tracking analysis. During the past couple of years, SMI has introduced solutions for virtual reality and augmented reality applications (e.g., integrating its eye tracking with Epson’s Moverio BT- 200 head mounted display, an eye tracking upgrade package for the Oculus Rift, a Google Glass prototype with eye tracking, etc.). To address high volume markets, SMI has developed an OEM Solutions business focused on integration into consumer products and the company has introduced general reference platforms that OEMs can implement for mobile/desktop, augmented, and virtual reality applications. Earlier this year, SMI and OmniVision announced an alliance in which OmniVision's global shutter CameraCubeChip will be integrated into SMI's virtual reality and augmented reality eye tracking platforms. At CES 2016, SMI announced it supports foveated rendering, which enables VR headsets to provide higher resolutions to areas the user is looking at (and less resolution for other areas). SMI historically also had a business unit focused on eye care but, as previously mentioned, SMI sold its Ophthalmic Division to Alcon in 2012. SMI is based in Germany. The Eye Tribe The Eye Tribe is an eye tracking company focused on mobile devices. The Eye Tribe software enables eye control on mobile devices, allowing hands-free navigation of websites and apps, including eye activated login, enhanced gaming experiences, and cloud-based user engagement analytics. The company sells a $99 USB 3.0 eye tracking device on its Web site. During CES 2016, it demonstrated its new eye tracking solution that is optimized for virtual reality applications. It enables foveated rendering (in which higher resolution is provided for critical areas of an image – such as the area in which the user is looking at) which can make a VR headset much more efficient and can also enable better imaging for smartphone-based VR devices. Much less processing is required since high resolution isn’t used for areas where the user isn’t looking. Eye Tribe recently announced a new Tracker Pro professional eye tracking solution for $199, with shipments targeted for June 2016, as well as an eye biometric authentication method. The Eye Tribe was founded by four PhDs from the University of Copenhagen and is based in Denmark.
  • 24. Page 24 “Touch is the mother of the senses.” - Helen Fisher Introduction Although the original iPhone had many innovations, the touch screen was one of the most differentiating features when it first launched. While basic touch screens had been used for decades, Apple was first to optimize the touch screen interface to make it easy to use and cost effective. As a result, virtually all smart phones and tablets now incorporate a touch screen, and an increasing number of notebook PCs and other devices have touch displays. This chapter discusses touch screens, haptics, and other emerging display related technologies. Touch Screen Technologies There are a variety of different types of touch screen technologies. A brief overview of the general mainstream touch displays include:  Capacitive – Capacitive touchscreens take advantage of the fact that the human body is an electrical conductor. A capacitive touchscreen typically includes an insulator (e.g., glass) and a transparent conductive layer such as indium tin oxide (ITO). When a finger touches the screen it distorts the screen’s electrostatic field and causes a change in capacitance which can be measured to determine the finger’s location. There are actually a variety of different types of capacitive touch technologies. The most common general type for high-end portable devices (smart phones, tablets, etc.) is mutual projective capacitance in which there is a matrix of rows and columns with a capacitor at each intersection. A voltage is applied to the rows and columns and when a finger touches the screen, the local electrostatic field changes, reducing the mutual capacitance, which can be measured to determine the location at which the screen was touched. A major advantage of this approach is that it enables multi- touch (it can recognize multiple fingers) which is critical for many features that consumers are now used to (zooming in and out using two fingers, etc.). Many other capacitive technologies and a variety of non-capacitive touch technologies can recognize only one touch at a time. One disadvantage of capacitive is that it doesn’t work if a person is wearing gloves or with an ordinary stylus/pen, although special pens have been developed that can be used (for example, in 2015 Microsoft acquired N-Trig, an Israeli company that developed the stylus for the Surface Pro 3). Most of the major high-profile touch devices (iPhone, iPad, Samsung Galaxy, etc.) utilize capacitive touch technology. Chapter 4 – Touch Screens, Haptics and Display Technologies
  • 25. Page 25  Resistive – The general principle behind resistive touch screens is that there are two thin transparent layers coated with transparent conductive material (the outer layer covered on the back, the inner layer coated on the front) separated by a small gap. When the outer screen is pressed, the two layers touch, completing an electrical circuit and creating a voltage divider. Based on the resistance of the materials and measured voltages, the location of a touch can be determined. Resistive touchscreens are relatively low cost and work well with a stylus (especially good for handwriting recognition) or when a user is wearing gloves. However, it generally doesn’t support multi-touch well and isn’t as sensitive as capacitive touch since the user has to press down on the screen. As a result, while there have been a number of phones with resistive touchscreens, most high-end mobile devices utilize capacitive. Resistive is common in other markets such as ATM machines.  Other (Infrared, Ultrasound, etc.) – Over the years, many different types of touch screen technologies have been developed. Examples are listed below. o Infrared (IR LEDs and photodetectors are placed around the edge of the display and can detect if a finger or object touches the screen) o Ultrasound/surface acoustic wave (ultrasonic waves pass over the surface of the display, and when the display is touched some of the wave is absorbed) o Infrared projection (a transparent sheet is illuminated with infrared light and “leaks” light when the user touches the screen) o Optical imaging (image sensors around the edge of the screen capture shadows to pinpoint the location of each touch) o Dispersive (detects the piezoelectricity in the glass due to touch, which can be used to pinpoint the touch location) o Acoustic pulse (a touch generates a sound wave and sound transducers are used to determine the position of the touch) Although a variety of new touch technologies have been developed as potential alternatives to capacitive touch, capacitive works very well, is robust, can implement multi-touch, and costs have been driven down with volume. As a result, displacing this technology in devices such as smart phones and tablets is challenging. However, as touch technology becomes increasingly pervasive in a variety of products, there are a growing number of different applications for touch screens, providing a variety of opportunities for other touch technologies. For example, the capacitive technology used in smart phones generally doesn’t scale well to large size screens and many applications don’t require multi-touch. One recent innovation in touch has been Apple’s introduction of “3D Touch” in the iPhone 6s which not only recognizes the location of where a user is touching but can differentiate between “hard” and “soft” touches, creating a much wider range of user inputs. For example, a soft press can allow the user to preview an email or website while a harder press can open the message or Web page. Apple indicates that when a user presses on the display, capacitive sensors measure the microscopic changes in the distance between
  • 26. Page 26 the cover glass and the backlight, and the measurements from these sensors can be used to determine the level of force. Some of the major suppliers of capacitive touch screen controller chips include Atmel, Parade (which acquired Cypress’ mobile touch screen business), and Synaptics, as well as a variety of companies in Korea and Taiwan. A recent trend has been integration of touch controllers and display drivers. Touch Screen Technology M&A The following are a few touch screen related M&A transactions:  Parade/Cypress Mobile TrueTouch unit – In June 2015, Cypress announced that it was divesting its TrueTouch Mobile touchscreen business to Parade Technologies for $100 million. This unit provided touch screen controllers for a variety of consumer markets (smartphones, tablets, notebooks/PCs, etc.). Cypress indicated that it will continue to provide TrueTouch solutions to its automotive and home appliance customers.  Synaptics/Renesas SP Drivers – In June 2014, Synaptics announced an agreement to acquire Renesas SP Drivers for about $515 million in enterprise value. The acquired business focused on small and medium-sized display driver ICs for smartphones and tablets and Synaptics indicated that the deal would accelerate its product roadmap for integrating touch controllers and display drivers.  FocalTech/Orise – In April 2014, FocalTech and Orise announced that the two companies would merge in a stock swap (each share of FocalTech swapping for 4.8 newly issued shares of Orise). FocalTech was focused on touch controllers and Orise was a major supplier of display drivers. As with the Synaptics deal noted above, the primary reason given for the deal was the interest in integrating touch controllers with drivers into single chip solutions.  Microsoft/Perceptive Pixel – In July 2012, Microsoft acquired Perceptive Pixel. Perceptive had developed very large screen multi-touch displays.  TPK/MasTouch and Cando – TPK has been a major touch screen module supplier and has made a number of acquisitions. It acquired a majority of the shares of MasTouch Optoelectronics (a controller chip company). In 2011, it acquired a just- under 20% stake in Cando for about $190 million (per Capital IQ). Cando is one of TPK’s largest touch sensor suppliers and is a subsidiary of AU Optronics.  Gores/TE Connectivity EloTouch Products – In 2012, private equity firm Gores Group acquired the touch screen solutions business from TE Connectivity for $380 million, or slightly less than 1X revenue (per Capital IQ). The business provided a range of different types of touch screen technologies.
  • 27. Page 27 Select Touch Screen Technology Companies There are a number of start-up companies that have developed alternative technologies to the capacitive touch sensing technologies commonly used in current smartphones. In some cases, their focus is either large screen markets such as television or large PC displays (as capacitive doesn’t scale to larger sizes well) or on smaller screens such as wearables (in which multi-touch isn’t critical). In other cases, the technology provides certain features or advantages that conventional capacitive screens do not provide. Some examples of touch screen technology companies are below. NextInput NextInput has developed a force sensitive touch surface technology (ForceTouch). In addition to being able to determine the location of a touch, its technology also can determine the amount of force applied by a finger or stylus. This enables a variety of new ways to control applications. That is, a light touch can be interpreted differently from a stronger touch, which can greatly expand the number of potential control gestures. Adding force makes interaction much more intuitive in many cases. It can also enhance many applications (e.g., gaming). Unlike Apple’s 3D Touch technology which utilizes traditional capacitive sensing, the NextInput solution (ForceTouch Array) is a MEMS sensor layer below the surface of the display. NextInput indicates that its solution has many significant competitive advantages relative to Apple’s 3D Touch and other types of force touch technologies. As the NextInput solution measures force directly, the company believes that ForceTouch has 10x better accuracy and sensitivity relative to alternative methods. As a result, it can not only more accurately determine if a user is pressing hard or soft and where the user is pressing, but can identify a broader range of force levels (not just hard and soft). Specifically, the company indicates that its technology enables touch event location with better than 1mm resolution and measures applied force in the range of a few grams to greater than one kilogram. NextInput has also indicated that its technology is substantially less costly (in volume) and consumes about half the power of capacitive touch, and that it can work with a stylus, and in virtually any environment (high EMI location, underwater, etc.). In July 2015, NextInput formally introduced its MEMS-based FT-4010F solution (1.3mm x 1.3mm x 0.63mm). In December 2015, it announced an $8 million funding round led by Sierra Ventures and two strategic corporate investors and that Steve Nasiri (founder of InvenSense) joined the Board. In January 2016, it announced that its technology was being incorporated into a smartwatch developed by GoerTek. NextInput was originally based in Atlanta, but relocated its headquarters to Silicon Valley.
  • 28. Page 28 FlatFrog FlatFrog has developed a touch display technology (InGlass) in which infrared light is injected into a transparent screen (plastic, glass, etc.) and is detected at the other end. The light travels through the material via total internal reflection, but when a finger touches the surface, it causes a disturbance to the light which is detected. Using advanced signal processing algorithms, the location of the touch can be determined. FlatFrog indicates that its technology supports multi-touch and that it is less costly than capacitive, especially for larger displays since their sensors are around the edge of the display whereas the number of capacitive sensors increases with display area. Its technology can also be used to detect pressure since the amount of pressure impacts how much light escapes from the medium. The company offers touch module solutions up to 110” and indicates that it can measure up to 1,000 levels of pressure and can support up to 80 multi-touches. In 2013, FlatFrog announced a partnership with Dialog Semiconductor, in which Dialog developed a Smartwave multi-touch ASIC specifically optimized for FlatFrog’s technology. The company is initially focused on niche markets such as education (smart board displays), but anticipates moving into high volume consumer applications over the next couple of years. FlatFrog is privately held with headquarters in Lund, Sweden (near Copenhagen, Denmark) and an office in Silicon Valley. Its investors include Intel Capital as well as Invus, Sunstone Capital, and Fårö Capital. Neonode A small public company that has been focused primarily on alternative touch screen technologies is Neonode. Neonode’s zForce is based on infrared technology in which infrared light is projected over the surface of the screen, and photodetectors measure the light patterns. When a finger or stylus touches the screen, it interferes with the infrared light and based on the measurement of the light and advanced algorithms, the location of the object or finger can be determined. The technology doesn’t need additional screen layers, works with a stylus or gloved fingers, and doesn’t require force on the display. The company has leveraged its core technology to also implement proximity sensing, such that the device can detect finger or object movements that are around the outside of the device rather than just on the screen. It recently announced its AirBar solution (a plug and play gesture recognition solution for a broad range of devices) and a number of automotive demos (with Volvo, Autoliv, etc.). It also recently announced its technology is integrated into a family of Lexmark printers. Neonode had an IPO in September 2013.
  • 29. Page 29 Touch Screen Haptics One limitation of traditional touch screens is the lack of any tactile sensation when touching the screen. For example, many people find it difficult to type more than short messages on a touch screen keyboard since they can’t feel the buttons, which is why there is still demand for keyboards that connect to tablets and why many people still use notebook PCs rather than tablets. However, there is growing interest in technologies that will change that. Haptics technology involves providing tactile feedback to users to improve the user’s experience. Haptic technology is already used in a variety of applications. For example, there have been a number of game controllers/joysticks that include haptics, such that the controller vibrates in certain gaming situations to make the game seem more realistic. The most basic use of haptics in handsets is for “vibration mode” in which the phone vibrates when a call is received. Haptic technologies, however, are increasingly being incorporated into touch screens in order to improve the user experience and provide feedback when the display is touched. The most common method for implementing haptics on a display is through the use of actuators (motors that control movement). One or more small actuators are incorporated under or on the side of the screen. When the user touches the screen, the actuator can be turned on by the app or operating system, which causes the display to vibrate and provides tactile feedback. Depending on where the actuators are placed, the types of actuators used, the frequency that they operate at, and the number of actuators used, a variety of different effects can be created. For example, when a user touches a specific spot on the screen (e.g., a letter on a keyboard), it can be made to feel like that particular spot is providing tactile feedback. The goal with high quality haptics is to give the impression to the user that she is touching actual physical keys, for example, even though the display remains flat. In addition to typing, haptics can greatly improve the user experience in a variety of other applications such as gaming. In general, the greater the number of actuators used and the more complex the actuators (wider frequency range, ability to implement complex waveforms, etc.), the more realistic the haptic experience can be, but that also results in higher cost and greater space requirements. Haptics is also important in automotive applications as it reduces the amount of time the driver needs to look at touch controls. The largest nearly pure play haptics technology company is Immersion Corporation, which is a publicly listed company. Immersion has developed a variety of haptics technologies that it licenses to other companies. According to Immersion’s December 2015 Investor Presentation, 63% of its revenue in Q3:15 came from mobile devices, but it also addresses gaming (23%) automotive (7%), and medical (7%) markets. Immersion has over 2,000 patents, most of which are related to haptics. It offers a variety of different haptic solutions, including single actuator, multi-actuator, and high-fidelity actuator solutions. It also provides software and effect libraries to help developers create desired haptic effects.
  • 30. Page 30 According to a 2015 press release by research firm MarketsandMarkets, the overall haptics technology market is expected to reach nearly $30 billion by 2020 at an estimated CAGR of more than 25% from 2014 to 2020. In 2013, Apple was awarded a patent for a haptic feedback system that is optimized for multi-touch surfaces ("Method and apparatus for localization of haptic feedback"). The patent involves using two actuators in which one actuator provides feedback to the user and the other creates a vibration to suppress the first to prevent it from propagating to the rest of the screen, resulting in localized haptic feedback. Select Haptics M&A Transaction A notable haptics acquisition was the following:  Parker Hannifin/Bayer MaterialScience/Artificial Muscle – Artificial Muscle was spun out of SRI International in 2004 to commercialize its Electroactive Polymer Artificial Muscle technology. Artificial Muscle was focused on haptics for mobile devices. EAP technology consists of a thin layer of dielectric polymer film sandwiched between compliant electrodes. When voltage is applied across the electrodes, the electrodes attract each other. As a result, the film contracts in thickness and expands in area, which can be used to create haptic effects. In 2010, Artificial Muscle was acquired by Bayer MaterialScience. In 2014, Parker Hannifin announced that it acquired intellectual property and licenses for the Artificial Muscle technology from Bayer MaterialScience. Select Private Haptics Related Companies A number of start-ups have addressed the haptics market, including the following: Redux Labs Redux Labs has developed differentiated haptics (Surface Sensation) and speakerless audio (Surface Sound) technologies for smartphones, tablets, PCs, automotive, and other applications. Its solutions are based on its patented “bending wave” technology that controls sub-sonic and sonic waves across a flat or curved surface. Its Surface Sensation haptic solutions deliver a customizable haptic effect to a specific coordinate on the screen and include transducers (piezoelectric transducers for small form factor devices and voice coil exciters for larger form factors), control modules or chips, and algorithms for implementing its bending wave technology. Its haptic technology can deliver feedback at very specific points anywhere on the display and the company believes it provides substantial advantages relative to traditional actuator-based haptics.
  • 31. Page 31 Because its transducers can manipulate waves, the glass of a smartphone, tablet or TV screen can also be transformed into a surround sound loudspeaker (its speakerless “Surface Sound” audio technology), eliminating the need for separate speakers or sound bars and potentially reducing the size and cost of products. Redux indicates that it is addressing a variety of end markets including consumer electronics devices, automotive applications (providing tactile feedback enables drivers to better focus on driving), PC/mobile devices, industrial, and augmented/virtual reality. The Redux Web site indicates that it has exclusive rights to exploit over 300 granted patents in bending wave technology with over 100 additional patent applications filed. The company is based in Cambridgeshire, England (a few miles from Cambridge). Novasentis Novasentis (previously Strategic Polymer Sciences) has developed a unique and innovative actuator/sensor based on electrical-mechanical polymers (EMP) related to a new class of ferroelectric materials. The EMP material is ultra-thin, light and flexible and elongates when an electric field is applied and produces electric charge (piezo-electric) when it is deformed due to pressure (from a finger) and can, therefore, act as both an actuator and a sensor. With digital control, the actuators can produce precise localized vibrations, sounds, or deformation. It can also be used as a sensor to recognize touch/pressure. Leveraging this core technology, Novasentis has developed its Clic family of ultrathin (<200 microns) haptic actuators. The company indicates that its technology is much thinner and requires relatively low operating voltages when compared to traditional vibrational motors or piezoelectric benders. Novasentis was founded in 2006 by former Apple executive Ralph Russo and the inventor of the EMP technology, Dr. Qiming Zhang. In November 2013, Novasentis announced it raised an $8 million Series B round that included new investor Samsung Ventures and previous investor Chengwei Capital. The company is based in Burlingame, California and has offices in Pennsylvania, Korea, and Japan.
  • 32. Page 32 Tactus Technology Tactus has developed a unique technology based on microfluidics that can create transparent physical buttons on a touch screen surface on demand. For example, when a user wants to type on a keyboard, physical keyboard buttons rise from the surface enabling users to feel the keyboard buttons as they type (eliminating one of the main complaints regarding touch screen typing). Unlike traditional haptics solutions that try to give the illusion that there are physical buttons, the Tactus solution actually creates temporary physical buttons on the screen. In addition to the advantages of “tactile feel,” the Tactus solution also enables finger resting (i.e., some pressure on the buttons is required to type a key). When the user doesn’t want to use the keyboard, the physical buttons recede into the screen leaving a flat smooth screen. While keyboards are the most obvious use for the Tactus solution, the technology can create buttons (or other shapes) anywhere on the screen which can be used for a variety of other applications. The underlying Tactus technology is based on microfluidics. Small transparent fluid channels are routed throughout the “Tactile Layer” and the fluid can expand the polymer layer above it to create physical buttons. A small controller chip is used to control the fluid channels and create or recede buttons. An API is provided to enable applications to control where and when buttons are created. The Tactus solution is designed to integrate with capacitive touch screens, with the Tactile layer replacing the cover lens (top-most layer). While its long term goal is to have its technology directly integrated into devices, Tactus introduced a product called Phorm in early 2015. Phorm is an add-on case for an iPad-Mini that includes a transparent front piece and a back piece with a switch on it. When the switch is turned on, tiny buttons rise from the transparent cover over the displayed keypad, and then recede when the switch is turned off. The Web site indicates the price is $99 but that it is sold out. The company announced a Series B capital raise in early 2015 that included IPV Capital. Tactus is based in Fremont, CA. Ultrahaptics Ultrahaptics has developed haptics technology that enables users to receive tactile feedback without actually touching anything. The technology uses ultrasound which is projected through the air onto the user’s skin in such a way that it creates a sensation in which the user “feels” like he is touching something. For example, users can ‘feel’ touch- less buttons, get feedback for mid-air gestures, or interact with virtual objects. This could obviously have a broad range of applications. In 2015, it announced that it is supporting Jaguar Land Rover in the investigation of a mid-air touch system for a Predictive Infotainment Screen. The company was founded in 2013 based on technology originally developed at the University of Bristol, UK. In November, 2015, Ultrahaptics announced that it completed a Series A funding round of £10.1 million led by Woodford Investment Management alongside existing shareholder IP Group plc.
  • 33. Page 33 Aito Aito has developed a software enhanced piezo technology that can both recognize touch and provide haptic feedback. Its technology utilizes low cost single layer piezoelectric discs that can both sense and actuate. Piezoelectricity can also be utilized to wake up the user interface to achieve maximal energy efficiency. The solutions are extremely thin (0.3mm) and work through a broad range of surfaces (e.g., under aluminum, carbon, etc.) and the company believes it is significantly more robust than other haptic solutions and has numerous other advantages. Its product families include Touch (LED/buzzer feedback), HapticTouch (adds haptic feedback), and LE (<1uA, low power). Aito is initially targeting automotive and appliance applications, but believes its solutions can be used in many markets. At CES 2016, it displayed its technology in the Texas Instruments booth (software on a TI microcontroller) and announced that LG is using Aito’s technology in a line of kitchen appliances. Aito is based in Amsterdam with R&D in Helsinki. Investors include INKEF Capital and Sirius Venture Capital. Quantum Dots Quantum dots are ultra-small (a few nanometers) nanocrystals of semiconducting material. They have a number of interesting properties including florescence, in which the color emitted is determined by the size of the particles. Specifically, smaller quantum dots typically have shorter wavelengths (colors on the blue end of the spectrum) while larger quantum dots have longer wavelengths (colors on the red end of the spectrum), although the exact emission depends on the quantum dot material. The most common material for quantum dots is cadmium selenide (CdSe). Because quantum dots can be tuned to produce a very precise wavelength (color), they have a variety of applications including displays. The properties of “quantum dots” were first discovered in the early 1980s, although the name “quantum dot” was initially coined in the late 1980s. Quantum dot technology is initially being used primarily for improving the backlighting of conventional LCD displays. Typically, LCD displays use white LED backlighting. The white light from the LEDs is then color filtered to produce specific colored pixels (red, blue, green). However, this has a number of issues. LEDs don’t naturally produce white light and so “white LEDs” are created by coating blue LEDs with a phosphor material. The resulting “white” light is poor quality. In addition, the subsequent filtering process results in reduced power efficiency. Quantum dots can be used to resolve some of these problems. For example, one approach is to use conventional blue LEDs (as the main light source) along with red and green quantum dots to provide RGB colors. This can produce a much broader color gamut and, therefore, much better images than white LED backlit displays (most color displays produce only about two thirds of the NTSC standards for color quality whereas 100% is possible with quantum dots).
  • 34. Page 34 Quantum dots also reduce power consumption (less filtering) which is important as backlighting is one of the most significant power drains in mobile devices. In high volume, it is also believed that quantum dots can be much lower cost. Many of the initial implementations of quantum dots are in a film over the backlight, but it could also be directly integrated into LEDs. A number of major OEMs have recently begun selling quantum dot TVs and monitors and it is expected to become more commonplace over time. At CES 2016, for example, a substantial portion of Samsung’s booth was dedicated to quantum dot TVs. Longer term, there is the potential for quantum dots to have an even more significant impact on display architectures, as the technology improves and designs are further optimized around the technology. According to a press release by Touch Display Research, the quantum dot display and lighting component market is projected to reach $10.6 billion by 2025. Select Quantum Dot Companies There are several companies developing quantum dot solutions for display applications. One public company is Nanoco which is listed on AIM and is developing cadmium free quantum dots (which it indicates is more environment friendly). Two major privately-held quantum dot companies are listed below. In addition to being used for displaying images, quantum dot technology can also be utilized for other applications including image sensing (e.g., InVisage is highlighted in a later chapter). QD Vision QD Vision has developed advanced quantum dot technology (Color IQ) for displays. The company was founded in 2004 by researchers from MIT including MIT professors Vladimir Bulcovic and Moungi Bawendi (two pioneers in quantum dot research). After more than a decade of development, QD Vision announced a number of commercial partnerships and customers in 2015. Some examples include: a partnership with Konka for TVs in the Chinese market; commercial shipments for a TCL 65” 4K UHD TV; the launch of the first quantum dot monitor (with Philips Monitors); and the launch of a curved quantum dot TV from HiSense. At CES 2016, the company announced a strategic partnership with Sigma Designs to develop a cost-effective UHD TV platform with High Dynamic Range (HDR) and Rec. 2020 performance. It also demonstrated a monitor that covers 87 percent of the Rec. 2020 color gamut, which it indicated is the widest coverage commercially available now. QD Vision indicates that it has shipped more than one million Color IQ optics products and that it has more than 250 patents and patents pending. QD Vision has raised more than $75 million in financing from venture capital firms, including North Bridge, Highland, Passport Capital, Novus Energy Partners, Capricorn Investment Group, BASF Venture Capital, and Tsing Capital. It is headquartered in Lexington, Massachusetts.
  • 35. Page 35 Nanosys Nanosys is a major supplier of quantum dot solutions. The company sells an advanced Quantum Dot Concentrate material and licenses its technology and designs, including its Quantum Dot Enhancement Film (QDEF), to industrial supply chain partners. The Quantum Dot Enhancement Film is designed to replace an existing film (diffuser) in a display’s backlight unit. The quantum dot film combines trillions of quantum dots in a thin sheet that emits white light when stimulated by blue light. Each QDEF sheet contains two plastic barrier films sandwiching a layer of quantum dots. Nanosys has announced a number of major commercial partnerships. It has partnered with 3M on Quantum Dot Enhancement Film. A couple of recent announcements include a partnership for integrating 3M QDEF film (using Nanosys’ quantum dots) with AU Optronics’ backlighting, and the launch of a Hisense 4K 65" ULED Curved Smart TV using Nanosys technology. In January 2016, TCL announced that it is using Nanosys Quantum Dots for a new line of Premium UltraHD HDR Televisions. Nanosys has a significant partnership with Samsung, a major proponent of quantum dot technology, including patent licenses, a strategic investment, and a follow on investment. Nanosys announced in March 2015 that it doubled its manufacturing capacity for quantum dot material to 25 tons per year at its Silicon Valley plant (enough for 6 million 60” UHD TVs per year). Nanosys has a number of investors including Samsung Ventures, ARCH Venture Partners, Venrock, El Dorado Ventures, Polaris Partners, Harris & Harris, and Lux Capital. New Touch Display Materials The vast majority of touch screens (including both capacitive and resistive) utilize indium tin oxide (ITO) as a transparent conductor. Indium is a relatively expensive material so there have been a number of companies that have tried to develop lower cost alternatives. Another reason there is interest in alternative transparent conductive materials is to improve the performance of touch displays and to help make them less fragile and more flexible. An example of a company that has developed an alternative to ITO is discussed briefly below. Select Touch Screen Material Company Cambrios Technologies Cambrios Technologies has invented a transparent coating material (ClearOhm), based on silver nanowires, that has a number of advantages relative to ITO. Because silver conducts very well, ClearOhm has very low resistance/high conductivity which helps improve electrical performance. In addition, the material is highly flexible and less prone to breaking than ITO. It can be used on glass (e.g., Gorilla glass) as well as plastic or OLED displays, and can scale to large size displays. Lastly, Cambrios indicates that in high volumes ClearOhm can be substantially less expensive to produce than ITO.
  • 36. Page 36 As a result, Cambrios believes its solution will be an important part of next-generation touch technologies across a variety of displays. In addition to mobile device touch screens, Cambrios’ technology has applications for solar, medical, and OLED lighting. Over the past few years, Cambrios has announced partnerships with a broad range of companies in the touch screen supply chain (e.g., Nissha Printing, Sumitomo, Chisso, TPK, Okura, Shin-Etsu, etc.) as well as a variety of other types of companies (e.g., Crystalsol GmbH, Claymount, Heraeus, etc.). In January 2016, Cambrios announced, along with its customer LG Electronics Chemical & Electronic Material division, that its technology is now being used by OEMs such as Acer (AIOs and monitors), Cisco, HP (AIOs), Lenovo (AIOs), LG, MSI (AIOs and monitors), and Toshiba (Satellite desktops). Cambrios is based in Silicon Valley and its investors include several strategic parties (including Samsung) as well as a number of VCs including ARCH Venture Partners, Alloy Ventures, and others. Pico-Projection and HUD Displays Pico-projectors are another type of displaying technology although it does not use an actual display. That is, pico-projectors enable users to display images or videos on a screen, wall, or other surface. Integrating a pico-projector into a smartphone or tablet would make it easier for other people to see images or videos and could be used for business purposes (e.g., showing PowerPoint slides without needing a large traditional projector) or for entertaining friends (e.g., showing videos). Unlike traditional large bulb-based projectors, pico-projectors generally use LEDs or laser diodes for illumination. While integrating some type of projection into a mobile device is relatively easy, the challenge is making it small (ideally so it doesn’t increase the size of the overall device) and effective (powerful enough for the projected image to be clear even from a distance) with low power consumption. A variety of relatively small pico-projector add-on products are available, although the long- term goal has been to incorporate the technology directly into notebook PCs, tablets and smartphones. There have been a few smartphones that have integrated pico-projectors (e.g., the Samsung Galaxy Beam), but they typically are much thicker than conventional smartphones in order to accommodate the projection technology and generally the illumination power has been relatively weak compared to conventional projectors. Several public chip companies have developed pico-projector chipsets (e.g., Texas Instruments, STMicroelectronics). A pure-play public company in the pico-projector sector is MicroVision. MicroVision’s PicoP technology utilizes a MEMS scanner which is coupled with blue, red, and green laser lights to create a projected image.
  • 37. Page 37 A somewhat related market to pico-projection is heads-up displays (HUD) for applications such as automobiles. Increasingly consumers are making decisions about which car to purchase based not on the engine features, but on the car’s infotainment/display systems. As the amount of electronics in cars increases, the number of displays a driver may want to look at also increases. However, the more a driver looks down at the dashboard, the less focused he is on the road. With HUD technology, images can be projected right on the car’s windshield, making it easier for a driver to see without looking down. For example, road images for navigation can be projected on the windshield, making it easier for drivers to follow directions. Warning signals can also be displayed on the windshield so the driver can immediately see when there is an issue, rather than waiting to notice that a warning light on the dashboard is on. As the technologies required for pico-projectors and HUDs are similar, some of the pico-projector companies are also addressing the HUD market. Pico-projection can also be relevant for augmented and virtual reality (projecting images onto the user’s glasses or headsets). Select Pico-Projection M&A Transactions Examples of pico-projection M&A transactions are:  Intel/Lemoptix – In March 2015, Lemoptix announced that it had been acquired by Intel. Lemoptix was highlighted in the first edition of this report and had developed advanced pico-projection solutions based on MEMS laser scanning. The company was leveraging this core technology to address a variety of end markets including pico-projection for mobile devices, automotive heads up displays, gesture recognition, and wearable devices. Lemoptix was based in Switzerland and was a spin-off from the Swiss Federal Institute of Technology.  STMicroelectronics/bTendo – In August 2012, STMicroelectronics acquired smartphone projection technology company bTendo (per the 451 Group and company press releases). bTendo was based in Israel and had already been working closely with STMicro for a couple of years prior to the acquisition. The bTendo scanning laser projection engine produces a focus-free high-resolution output that can display images or video on surfaces.
  • 38. Page 38 “She had lost the art of conversation but not, unfortunately, the power of speech.” - George Bernard Shaw Introduction Voice is a natural interface technology that has become increasingly common in electronics. Speech recognition has been around for many years. At the 1962 World’s Fair, for example, IBM demonstrated a system that could understand 16 words, and in the 1970s a system developed by Carnegie Mellon University could recognize more than 1,000 words. However, most of the early speech recognition systems required saying each word slowly one at a time. By the 1990s, consumer speech recognition programs became available (e.g., Dragon), although they were fairly expensive and required a significant amount of set-up time as users had to repeat words multiple times to “train” the software. With significant improvements in processing power and algorithms, speech recognition has dramatically improved during the past decade and has been incorporated into a variety of devices. For many devices, speech recognition is used for simple commands (e.g., providing an address to a navigation system, calling a cell phone number by saying the number or the name of the person, changing the station on a TV, etc.). However, with Apple’s introduction of the Siri personal assistant, speech has become a full-fledged interface technology that allows users to control and interact with mobile devices. Accomplishing this requires not only translating speech into words (basic speech-to-text technology), but also advanced natural language query technologies that are capable of piecing the words together into sentences/commands and then “understanding” what the user is asking for. This is much more complex as languages and grammar rules are complicated, words can have a variety of meanings depending on the context, and users can ask questions or give commands in a variety of different ways. As mobile products shrink in size (such as wearable devices), speech recognition is expected to become an even more common interface, as it doesn’t require a keyboard or large screen. For example, speech recognition is included in Google Glass, the Samsung Galaxy Gear watch, and the Apple watch. Chapter 5 – Speech Recognition / Language Processing / Mics
  • 39. Page 39 A number of the major technology companies have developed voice assistant technologies (Apple Siri, Google Now, Samsung S Voice, Microsoft's Cortana, Amazon's Alexa, etc.) and many mobile devices incorporate some type of speech technology to varying degrees. Amazon’s Echo solution (which integrates 7 microphones and connects to its Alexa cloud- based voice service) was specifically developed to enable voice control in home environments. Facebook has also launched a personal assistant technology (M). A public company that is primarily focused on speech recognition related solutions is Nuance Communications, which sells a broad range of different types of speech recognition products for a wide range of different end markets. One area of focus for improving speech recognition is that many of the existing speech recognition systems compress the user’s speech and send it over the network to dedicated servers that do the analysis needed to translate and understand the speech and analyze the words, and the results are then sent back over the network to the device. This, however, causes some latency issues and doesn’t work if the device does not have network access. As a result, one approach is to perform the analysis in the device, but this requires efficient algorithms as processing power in a mobile device is limited and complex processing can consume significant power and drain the battery. There is also interest in combining speech recognition with noise reduction/voice enhancement technologies to further optimize performance. That is, users increasingly speak at a distance from devices and often in noisy environments. For the speech recognition to work, the device must be able to extract the voice commands from the background noise. By combining speech recognition with voice enhancement technologies, the overall performance of voice-based assistants can be improved. Microphone technology has become increasingly important, especially as mobile devices must increasingly capture voices from users that are several feet away. The mobile device industry has largely transitioned from conventional microphones to MEMS microphones and the number of MEMS microphones per device has been increasing (typically 2 to 4 MEMS microphones per smartphone). There is interest in further improving these microphones. Artificial Intelligence/Natural Language Technology It is one thing for an electronic device to translate simple pre-arranged voice commands (“On”, “Off”, “Louder”, “Softer”, etc.) into words (basic text-to-speech) and actions, but it is another, much more complex task to analyze a complex set of unstructured words and sentences and determine what a user is looking for. While Siri, Google Now, and other speech-related solutions have become much better at understanding and responding to simple commands and questions, they are still a long way from being able to have a real conversation and understand complex questions or complicated commands. As a result, there has been significant interest in artificial intelligence and deep learning solutions that can be used for enhancing speech interfaces as well as for other types of user interactions. These technologies often have a broad range of applications in addition to user interaction but could become increasingly important over time.
  • 40. Page 40 Select Speech Recognition and Natural Language M&A Transactions Unlike many of the other technologies discussed in this document in which there have been only a handful of M&A transactions, there have been a large number of speech technology related transactions during the past couple of decades. For example, Nuance Communications alone has acquired more than 40 companies since 2005 when it was formed by the merger of Nuance and Scan Soft (and Scan Soft had acquired many companies before that), and a large portion of those acquisitions were speech-related. (Pagemill alone has completed 9 transactions with Nuance). Below are a small handful of the many speech and natural language related M&A transactions in recent years. We mention only those related to the consumer markets, as there have also been many acquisitions of companies focused on speech recognition for business, medical, and call center type applications. We also included some high profile artificial intelligence/cognitive computing acquisitions (e.g., Google/DeepMind) in which it is believed that the technology could be used for natural language analysis.  Apple/Vocal IQ – In October 2015, it was reported by the Financial Times that Apple acquired Vocal IQ, a natural language technology company based in the UK. Vocal IQ developed a self-learning dialog API, which can learn from interactions with people, improving a device’s ability to interact with a person over time.  Intel/Saffron – In October 2015, Intel announced on its blog that it had acquired Saffron, a cognitive computing platform provider. Intel stated that Saffron ingests data from disparate sources and automatically connects the dots to help improve decision-making. Intel indicated that in addition to enterprise analytics, Saffron’s technology can be used in consumer devices that need to see, sense and interpret complex information in real time.  Facebook/Wit.ai – In January 2015, Facebook acquired Wit.ai, a speech recognition company that had been founded only about 18 months earlier. Wit.ai had developed an API for developing voice-activated interfaces, making it easier to create speech recognition and voice activation solutions. Investors included Andreessen Horowitz and Y Combinator.  Google (DeepMind)/Dark Blue Labs – In October 2014, following up on its DeepMind acquisition noted below, Google acquired UK-based Dark Blue Labs. Dark Blue Labs specialized in deep learning technologies for understanding natural language. Their research focused on enabling computers, robots or other digital devices to better understand what users are asking for.  Google/DeepMind – In January 2014, Google announced that it was acquiring DeepMind. The terms of the deal were not disclosed, but a number of media reports (Forbes and others) noted that sources indicated the price was over $400 million. DeepMind was a UK-based artificial intelligence company that developed advanced learning algorithm solutions that could learn from experience. It could, for example,
  • 41. Page 41 figure out how to score highly on video games without any pre-programming related to the games. While DeepMind can be used for a broad range of applications, interfacing with humans is viewed to be one potential application.  Apple/Novauris – In January 2014, TechCrunch reported that Apple acquired Novauris Technologies in 2013. Novauris was an automatic speech recognition technology company, which was founded in 2002 and grew out of Dragon Systems (later acquired by Nuance). Novauris developed server-based speech recognition systems that could process several simultaneous voice access requests.  Yahoo/SkyPhrase – In December 2013, Yahoo acquired SkyPhrase (per the 451 Group), a start-up that had been focused on building solutions that better understand natural language. This core technology can improve speech recognition systems by providing a deeper understanding of what the user is asking for.  Intel/Indisys – In September 2013, Intel acquired Indisys for reportedly more than $26 million (per the 451 Group). Indisys was a Spanish company focused on natural language recognition technologies.  Facebook/Mobile Technologies (Jibbigo) – In August 2013, Facebook acquired Mobile Technologies (doing business as Jibbigo), a speech recognition and machine translation app company. The app enables users to text or record voice content in over 25 languages that is then translated on screen or read in another language (per the 451 Group).  Google/SR Tech Group Patents, SayNow, Phonetic Arts – In July 2013, Google acquired a speech-related patent portfolio from SR Tech Group LLC (including a patent for “Speech interface for search engines”). In January 2011, Google acquired SayNow which had been viewed as a Google Voice competitor. In December 2010, Google acquired Phonetic Arts which had developed advanced speech synthesis (text-to-speech) technologies (per the 451 Group).  Google/Wavii – In April 2013, Google acquired Wavii, (a natural language technology processing start-up focused on news summaries), for reportedly about $30 million (per the 451Group).  Amazon/Evi, Ivona, and Yap – Amazon has made several acquisitions in the speech technology area. In April 2013, Amazon reportedly acquired Evi Technologies for $26 million (per the 451 Group). Evi’s True Knowledge Answer Engine attempts to understand questions by analyzing its database of facts, and it can be used for general search as well as voice interactions. Amazon also reportedly acquired Ivona Software (multi-lingual speech synthesis system) in early 2013, and Yap (voice to text) in November 2011.
  • 42. Page 42  Apple/Siri – In April 2010, Apple acquired Siri. Siri was initially available as an app in the App Store, but Apple later incorporated the Siri Assistant into the iPhone 4s in October 2011. The transaction price was not announced, but reportedly the valuation was about $200 million (per the 451 Group).  Nuance Communications/SVOX, Vlingo, Loquendo, and Many, Many Others – As previously noted, Nuance has acquired more than 40 companies since 2005, many of which had speech recognition technologies. Some examples include: SVOX for about $123 million ($81 million upfront, $42 million deferred), Vlingo for $196 million (excluding the equity interest that Nuance already owned), Loquendo for $75 million, and Ditech Networks for $23 million (per 10K reports, Capital IQ and the 451 Group). Select Private Speech Recognition/Natural Language Companies During the past decade, a large number of speech recognition companies have been acquired but there are still a variety of private companies in the sector. There are also a variety of natural language processing companies. Sensory Sensory is a major supplier of speech recognition solutions, although it has also recently diversified into other related markets including voice and vision authentication. Sensory provides a broad range of speech recognition software solutions including:  TrulyHandsFree – Recognizes keywords and enables consumers to control devices using voice commands, even in noisy environments. It incorporates voice activation, so power consumption is extremely low. It is designed to accurately recognize phrases even when embedded in sentences and surrounded by noise. The latest version incorporates deep learning technologies to greatly improve accuracy.  TrulySecure – A combined voice and facial recognition authentication solution for mobile devices and PCs. Authentication is on-device, and almost instantaneous. TrulySecure is the first face and voice biometrics solution to be FIDO UAF Certified.  TrulyNatural – An embedded large vocabulary speech recognition system that can operate on-device, rather than requiring cloud-based servers. This reduces latency and works even if the device cannot connect to the cloud. In addition to a broad range of software solutions, Sensory also offers speech processing chips for customers looking for a complete chip/software solution. The NLP-5x, for example, is optimized for speech processing, including speech recognition, text-to-speech (TTS), and high quality music and speech synthesis. The chip integrates a 16-bit DSP and a variety of digital and analog processing blocks and communication interfaces, and works with Sensory’s FluentChip firmware.
  • 43. Page 43 Sensory indicates that its speech recognition solutions have been incorporated into close to a billion products from hundreds of customers including AT&T, Hasbro, JVC, Kenwood, LG, Mattel, Motorola, Plantronics, Samsung, Sony, SEGA, Uniden, and V-Tech. In January, Sensory announced that its TrulyHandsFree technology will be offered on Samsung’s ARTIK IoT module platform to enable hands-free IoT products. It also announced a partnership with Philips, combining Philips’ voice enhancement technology with Sensory’s solutions. Sensory was founded in 1994 and its headquarters is in Silicon Valley. Basis Technology One issue with natural language processing is that there are many difference languages spoken around the world and the natural language analysis differs for each language. Basis is a leading provider of software solutions for extracting meaningful intelligence from multilingual text and digital devices. Its Rosette linguistics platform uses advanced natural language processing techniques to improve information retrieval, text mining, machine learning, statistics, and computational linguistics. Rosette provides capabilities such as: identifying the language of incoming text; providing a normalized representation in Unicode; locating names, places, and other key concepts from a body of unstructured text; and name matching and name translation for names in foreign languages and scripts. Rosette can, for example, help search across multiple languages for names or places. Basis also has a digital forensics group for assistance with analyzing multi-lingual text. Basis indicates that its solutions have been used by over 200 major firms (e.g., Amazon, EMC, Exalead/Dassault, Fujitsu, Google, Hewlett-Packard, Microsoft, Oracle) and governments around the world. The company also indicates that it is the top provider of Asian linguistic technology to web search engines, including Ask.com, Google, Microsoft Bing, and Yahoo!. Basis is based in Cambridge, MA. VoiceBox VoiceBox Technologies has developed a variety of solutions for speech recognition and natural language understanding. Its solutions focus on home, automotive, and mobile applications. The company indicates that it developed the first in-car natural language application and that its solutions have been used by Toyota, Chrysler/Fiat, Renault, Mazda, Subaru, TomTom and AT&T (for their DriveStudio initiative). The company also notes that its solutions are deployed in a variety of mobile phones and wearable devices. VoiceBox indicates that one of its advantages is that its technology tracks context throughout a conversation (Contextual Natural Language Understanding), making it easier for it to understand what the user is really looking for. VoiceBox is based in Bellevue, Washington.
  • 44. Page 44 Semantic Machines Semantic Machines is focused on using artificial intelligence to enable natural conversation between people and computers. The company indicates that its technology goes beyond understanding commands to understanding conversations. This allows computers to better communicate, understand goals, and accomplish tasks for users. Semantic Machines believes that its technology will have a major impact on a variety of sectors including search, e-commerce, social media, and productivity software. The company indicates that a number of its team members previously worked on Siri and Google Now. In December 2015, Semantic announced that it completed a $12.3 million capital raise. Its investors include Bain Capital Ventures, General Catalyst Partners, Justin Kan, and Ray Stata. It has offices in Berkeley and Boston. MindMeld MindMeld (previously known as Expect Labs) has developed artificial intelligence technologies for voice driven applications. The company was founded in 2011 and launched its MindMeld platform in 2014. The platform enables customers to create intelligent voice driven interfaces for a broad range of apps and devices. Partnerships have been announced for several markets including smartwatches and healthcare. In early 2014, it announced the release of the MindMeld API, which it indicated was the first context-driven developer platform and cloud-based service, and enables developers to deliver better search and content discovery using contextual signals instead of keywords. Applications include intelligent assistants for mobile devices, improved website search, and voice and context driven dashboards. In December 2015, the company announced the launch of MindMeld 2.0 for enabling enterprises to create advanced voice interfaces that are intelligent and accurate for search and navigation. The company indicated that the new version enables state-of-the-art natural language understanding with human-like accuracy, and that it is in trials with leading online services including Spotify. Investors include Google, Samsung, Intel, Telefónica, Liberty Global, Greylock Partners, Bessemer Venture Partners, IDG Ventures, KPG Ventures, In-Q-Tel, and Quest Venture Partners. MEMS Microphone Technology Historically, users held mobile phones next to their mouths when speaking. However, increasingly people speak to devices that are much further away. For example, with video conferencing applications (Facetime, etc.), consumers will typically hold a smart phone or tablet at arm’s length from their face. When using Skype with a notebook PC or speaking to Siri on a tablet or using a smartphone for video recording, as just a few examples, the person speaking is often several feet from the device. This makes it much more difficult for the device to obtain a high quality voice/audio signal.
  • 45. Page 45 An important part of any voice related system is the microphone, which converts audio signals into electrical signals. A broad range of different types of microphones have been invented going all the way back to Alexander Graham Bell (“liquid transmitter” microphone) and Thomas Edison (carbon microphone). Each of the various microphones is designed such that incoming acoustic signals ultimately cause a change in an electrical signal. For example, a basic traditional (DC biased capacitor) condenser microphone has a thin diaphragm that acts as one plate of a capacitor and a separate fixed plate. Acoustic waves cause the diaphragm to vibrate and, as a result, the distance between the plates changes, which causes the capacitance to change. The circuit is designed such that the change in capacitance results in a change in voltage across the capacitor. As a result, the voltage changes based on incoming sound. This basic capacitive microphone was invented at Bell Labs in 1916. The electret microphone (also invented at Bell Labs), a type of capacitive microphone which uses a permanently charged material (eliminating the need for a polarizing power supply), had been commonly used in mobile phones for many years. More recently, however, the technology industry has migrated to MEMS-based microphones for many products. MEMS technology involves the creation of tiny mechanical structures using semiconductor manufacturing process technologies. This enables microphones that are significantly smaller than traditional microphones, which also makes it easier to incorporate multiple microphones into the same device, providing much better sound quality. Once MEMS microphones became commonplace, OEMs quickly moved to using two microphones to enhance audio quality. The iPhone 5 incorporated a third MEMS microphone and the iPhone 6s has four MEMS microphones. Relative to conventional microphones, MEMS microphones are also typically more robust, less temperature sensitive, and consume less power. Because they are semiconductor devices, MEMS microphones can be more easily integrated into electronic circuits. Although MEMS microphones are manufactured using a significantly different process than conventional microphones, most use the same basic principles of operation (only at a much smaller size). That is, most MEMS microphones use capacitive technology in which there is a fixed plate (back-plate) and a movable plate (membrane). Incoming audio signals move the membrane, which changes the capacitance between the plates and creates an electrical signal representing the incoming sound. While microphones are inherently analog, MEMS microphones are often classified as digital or analog based on the output of the component. With a digital MEMS microphone, the electrical signal is converted to digital bits before external transmission, whereas analog microphones have an analog output (and the signal is converted to digital at some point later in the signal chain). Some of the key characteristics of microphones are: sensitivity (how much the electrical signal changes for a given change in pressure from the acoustic waves); directionality (how much the sensitivity changes based on the speaker’s location relative to the mic); and signal to noise ratio (how much noise is produced by the mic relative to the actual voice signal).
  • 46. Page 46 Major suppliers of MEMS microphones include Knowles, AAC (China), Goertek (China), BSE (South Korea), STMicroelectronics, InvenSense, Bosch, Hosiden (Japan), NeoMEMS (China), and Cirrus Logic. Infineon is not a supplier of end MEMS microphone solutions but develops and sells MEMS microphone die to a variety of companies that then package and complete the solutions. For example, AAC and Goertek had been suppliers of conventional electret microphones but became major suppliers of MEMS microphones primarily by leveraging Infineon’s core MEMS solutions. Select MEMS Microphone M&A Transactions Some major MEMS microphone transactions include the following:  InvenSense/Analog Devices’ MEMS Microphone Unit – In October 2014, InvenSense announced that it was acquiring Analog Devices’ MEMS microphone business for $100 million, plus an earn-out based on results from the unit over a 12 month period. InvenSense had been a major supplier of MEMS gyroscopes and was already in the process of developing its own MEMS microphone solutions.  Cirrus Logic/Wolfson – In April 2014, Cirrus Logic announced the acquisition of Wolfson Microelectronics for an enterprise value of £278 million (approximately $467 million at the time). Wolfson was a Scotland-based public company that was primarily focused on analog audio chips such as audio DACs, but also had a small MEMS microphone business. Wolfson entered the MEMS microphone market through the acquisition of a small start-up named Oligon in 2007.  Bosch/Akustica – In 2009, Bosch acquired Akustica. Akustica was an early pioneer in MEMS microphones and was based in Pittsburgh. Akustica became part of Bosch’s MEMS unit (Bosch Sensortec GmbH) which had already been a major supplier of MEMS accelerometers.  Dover/Knowles – In 2005, Dover acquired Knowles for $750 million. Knowles was a major supplier of MEMS microphones. At the time of the acquisition, Dover indicated that it anticipated revenue of approximately $210 million from Knowles. Subsequently, in February 2014, Dover spun off Knowles as a separate company to its existing shareholders and Knowles began trading as a public company in March 2014. Dover included some product lines in Knowles that were not part of the original Knowles, but their main product remains MEMS microphones.
  • 47. Page 47 Select MEMS Microphone Company Below is an overview of one of the start-ups focused on innovative MEMS microphones. Vesper As previously noted, the vast majority of MEMS microphones utilize capacitive sensing technology in which a membrane vibrates based on incoming sound waves which changes the distance between the membrane (which acts as one capacitor plate) and a fixed capacitor plate. Vesper (previously known as Baker Calling) is a privately-held company based in Boston that has developed a differentiated MEMS microphone family that utilizes piezoelectric materials, rather than traditional capacitive sensing. Piezoelectric materials produce a change in voltage based on how much pressure is applied. Thus, incoming acoustic waves cause a change of pressure, which results in a change in voltage that can be measured and processed. Vesper believes that its single layer piezoelectric cantilever structure has many advantages relative to the dual or triple membrane structures used for capacitive devices. Specifically, it has indicated that its microphones have significantly higher signal-to-noise ratios enabling clearer sound capture at much greater distances. Vesper also indicates that its microphones have very little variation from mic to mic, making them well suited for multi- microphone arrays, which are increasingly common. They also indicate that it is very robust and naturally waterproof. Vesper has stated that it plans to begin commercial shipments in the first quarter of 2016. Its investors include Accomplice and unannounced strategic partners.
  • 48. Page 48 “Happiness has many roots, but none more important than security.” - Edward Stettinius, Jr. Introduction The previously discussed HMI technologies were primarily for enhancing the user experience by making it easier for a user to control an application or device without using a conventional keyboard/mouse. Another aspect of HMI technologies, however, is security. Remembering and having to manually type in passwords is cumbersome and has security risks. In addition, it is impractical for many small form factor devices. As a result, there is growing interest in utilizing biometric interface technologies for security purposes. Some examples include fingerprint sensors, voice authentication technology, and facial/eye recognition systems. This chapter briefly discusses some of these technologies. Fingerprint Recognition Fingerprint readers enable users to log onto a device or application by swiping or placing a finger on a sensor. The user will have her fingerprint read a few times during set-up and can then subsequently gain access by placing her finger on (or swiping across) the fingerprint sensor. Historically, the two general types of fingerprint sensors were area sensors (in which the user places a finger motionless on a large sensor array) and swipe sensors (in which the user swipes a finger across the sensor and the images of each portion of the finger are pieced together by the computer). The sensor array in a swipe sensor is generally significantly smaller than that of a traditional area sensor, resulting in a much smaller footprint as well as lower cost for a swipe sensor, since the sensor array’s cost is typically proportional to its size. The disadvantage of a swipe sensor is that it generally is not as robust as an area sensor since multiple images of the finger have to be pieced together and users swipe their fingers at a variety of different speeds, which can make it more difficult to capture fingerprints, resulting in more false negatives and false positives. For portable devices, however, it had generally been viewed that the size and cost advantages of swipe sensors outweighed other factors and most of the fingerprint sensors that had been implemented in notebook PCs and mobile phones for several years were swipe sensors. Chapter 6 – Security Biometrics
  • 49. Page 49 However, the iPhone 5s introduced Apple’s Touch ID sensor which is small enough to fit within the Home button. Unlike traditional area sensors that are relatively large so the user’s entire fingertip can fit, Apple’s “touch sensor” solution reads only a portion of the finger. That is, during set-up the user’s fingerprint is read in a number of different positions and subsequently the system can identify the user from just a small portion of the fingerprint. The steel ring around the home button detects a finger and activates the sensor. The iPhone 6s incorporates a similar sensor but it operates slightly faster. With the Galaxy 6, Samsung introduced a similar home button sensor into its smartphone. A variety of different technologies have been used to capture fingerprint images. Some examples include: thermal (the ridges of a finger contact the sensor and have a higher measured temperature than the valleys of the finger which can be used to create an image), optical imaging (the finger is illuminated and an image sensor captures fingerprint images), ultrasound (very high frequency sound waves penetrate the epidermal layer of skin and the reflected signal is captured), pressure (pressure from ridges of a finger is used to create an image), passive capacitive (the dermal layer of the finger and the sensor array act as parallel plate capacitors separated by the epidermal layer; the measured capacitance varies between the ridges and valleys of the finger), active capacitance (similar to passive capacitance but a small voltage is applied to the finger creating an electric field related to the pattern of ridges and valleys in the dermal layer), and RF (an RF signal is applied to the user’s finger and the resulting signal which is changed based on the fingerprint patterns is then read by a detector). Although each type of fingerprint sensor technology has been commercialized, many of them are not well suited for the consumer markets (e.g., not robust enough, too large, too expensive, etc.). Optical fingerprint sensors, for example, are commonplace for conventional stand-alone fingerprint readers, but have not historically gained traction in mobile devices due to size and power requirements. In general, one of the biggest challenges is the significant variations in types of fingers and environmental factors (e.g., finger may be dirty, clammy, dry, etc.), and with swipe sensors there is significant variation in how users swipe their fingers. Most of the fingerprint sensors implemented in mobile consumer devices utilize some type of capacitive or RF technology, which can be implemented at relatively low cost in small form factors. In addition, both of these technologies make measurements based on the dermal layer of skin, rather than just the external epidermal layer (which is more prone to changes based on external factors such as whether the finger is dry, wet, or has lotion or other materials on it). For example, Apple’s Touch ID and sensors from Fingerprints utilize capacitive technologies and sensors from Validity (now part of Synaptics) utilize RF technologies. AuthenTec had historically used RF but UPEK (which it acquired) utilized capacitive technologies.
  • 50. Page 50 With consumers increasingly using smartphones for purchases and financial transactions, fingerprint sensing is increasingly used not just for accessing the phone but to validate identity for certain types of transactions and mobile payment systems. There has been significant consolidation of fingerprint sensor companies in recent years (Synaptics acquiring Validity, AuthenTec acquiring UPEK, Apple acquiring AuthenTec). These acquisitions are discussed in more detail below. Select Security Fingerprint Sensor M&A Transactions There have been a number of fingerprint/biometric acquisitions in the general security market. For example, Francisco Partners acquired Cross Match Technology, which later merged with DigitalPersona (which had acquired Identity Stream), and Assa Abloy (HID) acquired Lumidigm. However, those companies tend to focus on markets such as government, financial, retail, defense, and law enforcement. Below are a few specifically related to the consumer markets.  Apple/Privaris’ Patents – It was reported by CNN that Apple acquired three patents from Privaris in 2012 and more than twenty additional patents in 2014. Privaris developed a variety of portable fingerprint sensor solutions but its patents covered a variety of fingerprint sensor and touch screen technologies.  Synaptics/Validity – In October 2013, Synaptics announced that it agreed to acquire Validity for approximately $92.5 million in stock and cash plus an earn-out of $162.5 million for a total potential consideration of up to $255 million (per Capital IQ). Validity was a privately-held fingerprint sensor solutions company. Its solutions were incorporated into a number of notebooks, tablets, and phones.  IDEX/PicoField – In September 2013, IDEX announced an agreement to acquire the assets and intellectual property of PicoField Technologies, which was focused on touch fingerprint sensor design and development tailored for consumer devices.  Apple/AuthenTec (which acquired UPEK) – In July 2012, Apple announced that it was acquiring AuthenTec, a small cap public company focused primarily on fingerprint sensor solutions, for $356 million, which was about a 58% premium to AuthenTec’s stock price at the time of the announcement (per the 451 Group). AuthenTec had about $70 million of revenue in 2011. In 2010, AuthenTec had acquired UPEK, another major fingerprint sensor company which was originally spun out of STMicroelectronics.
  • 51. Page 51 Facial Recognition With facial recognition, images of a person’s face are taken via an image sensor. When the user wants to log on later, the image of the person’s face is compared against the set-up pictures using a variety of complex algorithms to determine if it is the same person, in which case access is granted. Typically, there are certain key parameters the system focuses on to help match images (e.g., distance between the eyes, width of the nose, shape of the cheekbone, etc.). The incremental cost of this technology can be relatively low assuming a device already has a standard image sensor. However, accurate facial recognition is very challenging, since a person’s face can change quite a bit from day or day (e.g., shaved/not shaved, before or after a haircut, bad hair day, smiling/frowning, tan/not tan, etc.). In addition, room lighting can have a significant impact on facial images so features will vary based on ambient lighting. Also, the technology has to adjust for the fact that a person’s face may be at varying distances and angles from the camera. All of this makes facial recognition challenging, especially for mobile devices with limited processing power. In many mobile device implementations, facial data is transmitted over the network to servers that perform the actual face matching and the results are sent back to the device. While facial recognition has been used extensively in law enforcement agencies, it has had only moderate traction in consumer devices. Google introduced Face Unlock in Android 4.0 (Ice Cream Sandwich) in 2011. Face Unlock can be used as an alternative to a screen lock PIN. A couple of initial criticisms of Face Unlock were that there was a high level of false positives (i.e., someone with a face that was somewhat similar to the phone’s owner could get past security) and that photos of a person could be used to gain access. In subsequent Android updates, changes were made to improve the technology and to require the user to blink to gain access (so that static photos won’t work). However, the percentage of users actively utilizing facial recognition to unlock their phones is still thought to be relatively low. Facial recognition in general was expected to get a boost when Google first introduced Google Glass a few years ago as there had been reports that it would have a feature enabling Glass wearers to identify people (and bring up information about the person) just by looking at them. However, a congressional committee wrote an open letter to Google expressing privacy concerns if Glass included facial recognition capabilities. Subsequently, Google prohibited the use of facial recognition (for Glass) for identifying anyone other than the user.
  • 52. Page 52 Although we included facial recognition in the chapter on security, there is also significant interest in facial recognition for other applications. For example, facial recognition can be used to identify people to provide user preferences. A smart TV could identify which person is watching and change settings based on that (e.g., bring up that person’s favorite types of shows). An automobile could determine who is driving and make seat, mirror, and entertainment adjustments based on that. Another major area of interest for facial recognition is social media, such that images or videos of a person can be identified and tagged. For example, facial recognition technologies have been used by both Facebook and Google for searching to find photos and videos of a person. As a specific example, when a Google+ user turns on the “Find my Face” feature, photos and videos that the user tags are utilized to develop a model of the user’s face. When someone else looks at a photo or video with a face that appears to match the user’s face, they receive a suggestion to tag the user. As a result, there have been a number of facial recognition acquisitions related to photo tagging. Select Facial Recognition M&A Acquisitions There have been a number of consumer-related facial recognition acquisitions, although several have been more focused on facial recognition for social media type applications rather than security. Some examples include:  Dropbox/KBVT – In September 2014, Dropbox acquired Kriegman-Belhumeur Vision Technology (KBVT), which focused on advanced facial recognition software and had two professors considered experts in the field (Peter Belhumeur and David Kriegman).  Google/Viewdle and PittPatt – In October 2012, Google (through Motorola Mobility) acquired Viewdle, a Ukrainian company focused on facial recognition technologies for applications such as photo tagging. In addition to facial recognition, Viewdle was also diversifying into pattern recognition (gestures, etc.). The deal was reportedly in the $40 to $45 million range (per the 451 Group). In 2011, Google acquired Pittsburgh Pattern Recognition (PittPatt). PittPatt was a start-up that was formed from a project at Carnegie Mellon University and developed facial recognition technology to match people’s faces, including face detection, face tracking, and face recognition.  Facebook/Face.com – In June 2012, Facebook acquired Face.com, an Israeli start- up focused on facial recognition technologies for photo-tagging and related applications. Its solutions helped developers build applications that could identify people from photos, although it stopped providing its facial recognition APIs after the Facebook acquisition. The deal value was reportedly approximately $60 million (per Capital IQ).
  • 53. Page 53  Apple/Polar Rose – In 2010, Apple reportedly acquired Polar Rose for approximately $29 million (per TechCrunch). Polar Rose used facial recognition for auto-tagging of photos for photo sharing sites. Eye Recognition Another type of security technology is eye recognition, in which images of the user’s eye are captured during set-up and subsequently compared to images of the eye of the person trying to log in. If the images match, the user is granted access. There are two general types of eye recognition technologies – iris recognition and retina recognition. With iris recognition, images of a person’s eye are captured (typically near- infrared light is used to illuminate the eye) and the unique patterns of the iris are used for matching purposes. Retina scans involve shining a bright light into the pupil and capturing images of blood vessels in the back of the eye. The term retina scan is sometimes incorrectly used to refer to all eye recognition technologies, but iris recognition is by far the most commonly used and less invasive technology. An advantage of iris recognition compared to other security technologies is that unlike a person’s facial features, for example, which can change from day to day, the patterns in a person’s eye vary little over time and so matching is generally easier. A negative is that many people don’t like the idea of having their eyes scanned, so making it as unobtrusive as possible is critical (i.e., having to move an eye close to the camera is generally not desirable). Too strong a flash of infrared light can damage the retina so weaker flashes are used but that typically requires the eye to be relatively close to the camera. Iris recognition has been used for many years in a variety of general security applications (high security building access, etc.) but there has been interest in incorporating it into mobile devices. The goal is to make it effective at a distance, such that the user doesn’t have to move closer to the device than during normal usage. Select Eye Recognition M&A Transactions There have been a number of eye recognition acquisitions for government and enterprise applications but only a few related to consumer markets.  Voxx/EyeLock – In August 2015, Voxx International announced that it entered into an agreement to acquire the assets and IP and a controlling interest in EyeLock, which had developed iris recognition security technology for government, business, and consumer markets. EyeLock had more than 70 patents granted and pending. The myris iris authentication device is available at more than 1,000 retail locations across the U.S. The price wasn’t announced but financial documents later filed by Voxx indicate the deal was about $20 million with a $15.5 million outlay at closing.
  • 54. Page 54  Tessera (FotoNation)/Smart Sensors – In February 2015, Tessera’s FotoNation unit announced the acquisition of Smart Sensors and indicated the transaction closed in late 2014. Smart Sensors provided advanced iris recognition technology and was based in Bath, England. Voice Authentication As many devices now include a microphone for voice communications, there is interest in utilizing voice authentication technology in which a device can recognize a person’s voice for security access. With voice authentication, typically a person will pick a specific pass phrase and repeat the phrase a few times in set-up mode. Subsequently, the security system then requires the user to say the specific phrase and can recognize if it is the user’s voice (another person saying the phrase would not gain access). An advantage of voice authentication is that the incremental cost can be minimal; assuming the device already has a microphone, as it doesn’t require additional hardware such as a fingerprint sensor. It is also well suited for wearable devices that are relatively small. One of the issues that some voice recognition solutions have is that it requires significant processing power which can’t be supported by the device, so the voice information is sent over the network for analysis and a server on the other end performs the computations and lets the device know if access should be granted or not. This can result in latency issues and also does not work if there is no network access (zero bars). Some historical solutions also had issues if there was background noise and had too many false negatives. Select Voice Authentication M&A Transactions There have been a number of voice authentication technology acquisitions, although most of the targets were focused on voice authentication for institutions, rather than for consumer devices. Some voice authentication M&A transactions have included:  Cirrus Logic/KIVOX – In the third quarter of 2015, AGNITO announced that it divested its KIVOX mobile assets to Cirrus Logic and that the deal closed in July 2015. The sale included the KIVOX voice biometrics solutions for mobile applications including an SDK and IP. The technology allows for the voice biometrics engine to be deployed on mobile devices for security and authentication purposes and provides for on-device voice matching.  Verint/Victrio – In October 2013, Verint announced the acquisition of Victrio which had developed “passive” voice biometrics solution with predictive analysis that can accurately detect fraudsters and authenticate customers without caller interruption.
  • 55. Page 55  VoiceTrust/Perceive Solutions – In February 2012, VoiceTrust acquired the Montreal based voice technology firm Perceive Solutions. Perceive had developed voice biometrics solutions that can identify and authenticate speakers in any language with free form speech.  Nuance/PerSay – In 2010, Nuance acquired PerSay. PerSay was an Israeli start-up focused on the voice biometrics markets, for systems used in the banking, insurance, governments, and telecommunications industries. Many Other Biometric Security Technologies There are a variety of other biometric security technologies that have been developed. As an example, we have seen solutions in which the user can gain access through handwriting recognition (the user writes a word on the touch screen using a finger), gesture movements, and heart/pulse measurements. There is also growing interest in multi-factor authentication, in which two or more security factors are used (e.g., voice and facial recognition) to provide an even higher level of security. Select Biometric Security Technology Companies The following are a few examples of biometric security companies. There are a variety of companies that have developed biometric solutions for government, military, corporate, and telephony applications, but we included companies specifically focused on mobile device applications. Although we primarily highlight private companies in this report, we included a couple of small cap European public companies. Porticus Technology Porticus is a small privately-held company that has developed unique voice authentication and multi-factor solutions. Based on more than a decade of research in voice forensics and biometrics, Porticus has developed its VoiceKeyID voice authentication solution, which can be used for unlocking a device or for enabling access to certain applications (e.g., financial transactions, etc.). A user decides on a pass phrase and repeats it three times at set up, and then subsequently speaks the phrase to gain security access. The Porticus solution is based on voice phonemes, so it is language independent (it doesn’t try to translate the spoken pass phrase but just confirms that the spoken phrase sounds the same). The Porticus solution was specifically developed for mobile handsets and operates entirely in the device without requiring any network access (eliminating latency problems and issues related to not having network access). It was also designed to have low power consumption and requires relatively little processing power. The software can be incorporated into a chip or implemented through the operating system or an app.
  • 56. Page 56 Although the basic solution provides two factor authentication (user’s voice, pass phrase), it supports four factor authentication (including the device’s ID and the device’s location). Thus, security could potentially be granted only if the user has the correct mobile phone, is in certain locations, knows the pass phrase, and has a voice match. This provides significantly more security than simply typing in a password. Porticus has a licensing business model. The company has a number of patents around its voice authentication technology, as well as multi-factor security incorporating voice authentication in general. VoiceKeyID is available for iOS and Android OS platforms, and runs on a variety of chips including Qualcomm’s SnapDragon. The Company is headquartered in Wellesley, Massachusetts and maintains research and development facilities in Vilnius, Lithuania. Vkansee Vkansee has developed optical-based fingerprint sensor solutions for mobile devices. Optical-based fingerprint sensors are commonplace for stationary applications (e.g., building access) as it can be highly accurate, but it was not historically usable in mobile applications (handsets, notebook PCs, etc.) as optical readers were large and consumed significant power. However, Vkansee has developed an optical-based solution that they believe resolves these issues and provides many advantages relative to the fingerprint sensors currently used. Specifically, they indicate that their fingerprint sensors have higher resolution (2000 dpi, more than 4X most alternatives), better accuracy, smaller size (1.5mm thin), and incorporate a variety of differentiated anti-spoofing technologies. Their sensors also consume less than 0.1W and have a detection feature so it is asleep when not in use. In addition, the solution operates under-glass, which can provide many advantages relative to other technologies that require the glass to be etched to implement the fingerprint sensor. Vkansee is based in New York with an office in China. Fingerprint Cards AB Fingerprint Cards (FPC) sells a variety of area, swipe and touch fingerprint sensors which all utilize capacitive technology. FPC’s sensors contain tens of thousands of small capacitive plates, each with their own electrical circuit embedded in the chip. When a finger is placed on the sensor electrical charges are generated, creating a pattern between the finger’s ridges or valleys and the sensor’s plates. Using these charges, the sensor measures the capacitance pattern across the surface. The measured values are digitized and processed. FPC’s high sensitive pixel amplifier (HSPA) technology allows each pixel element in the sensor to detect very weak signals, improving image quality. FPC has announced a variety of design wins over the past year including several smartphones from Huawei and ZTE. It recently announced solutions targeting the automotive market. FPC has a variety of patents related to its own fingerprint technologies, but also acquired more than 100 patents in 2013 related to wireless technology. The company is headquartered in Sweden. Its stock is listed on the Stockholm exchange.
  • 57. Page 57 Next Biometrics Next Biometrics provides fingerprint sensor solutions. Its solutions utilize “active thermal” sensing (rather than capacitive or RF) that can be implemented on a low temperature polysilicon process which is also commonly used to produce displays. The active thermal technology applies a short heat pulse to each sensor pixel and the ridges of the finger produce a different temperature differential from the valleys. By measuring the heat response of the user’s finger and analyzing the data, Next’s solution can create a fingerprint image. Because its solution uses a low cost polysilicon process rather than a silicon fab, the company believes that its product can be produced at a significantly lower cost relative to traditional fingerprint sensors. Next has focused on area sensors which generally have the best performance but historically were not cost effective for consumer devices. Target markets include smartphones, tablets, notebooks, traditional security, and other segments that are not currently addressed (“Next-enabled”). The company has indicated that its sensors are designed into a variety of 2016 Dell notebooks and tablets, and it has announced a number of other customers and distributors. Next had an IPO in mid-2014 (Oslo Axess) and is based in Norway. Animetrics Animetrics is a developer of 2-D-to-3-D face recognition and face creation solutions. The Animetrics product family delivers facial recognition identity solutions using technologies that are capable of addressing high levels of variation in pose and lighting conditions. The Animetrics technology takes a 2-D image of a face and generates a 3-D model for accurate verification and identification. This 3-D model can also be rendered as a lifelike avatar. Animetrics has historically targeted military and government type applications (identifying suspects or missing people) running on computer systems, and has achieved traction in these markets. However, more recently it has introduced solutions for smartphones. Specifically, the FaceR MobileID solution is designed to help recognize and match faces using a mobile phone, and the FaceR CredentialME is an advanced facial recognition security application. These solutions can run on iPhone or Android smartphones and leverage Animetrics’ core FACEngine facial recognition technology that converts 2-D to 3-D images. Animetrics has several patents and is based in New Hampshire.
  • 58. Page 58 “I believe in a long, prolonged, derangement of the senses … in order to obtain the unknown.” - Jim Morrison Introduction The number of sensors in mobile devices continues to increase. Many devices now have a variety of positioning sensors (accelerometers, gyroscopes, magnetometers, etc.) as well as a number of interface sensors (CMOS image sensors, microphones, etc.). Adding sensors to a device is relatively easy, but effectively utilizing all of the data to greatly enhance the performance or functionality of the device is more complex. In addition, many sensors require a significant amount of “behind the scenes” management (e.g., calibration). Sensor fusion refers to the idea of combining the data from a variety of different sensors and sources to enhance the overall performance of the product. Beyond basic sensor fusion, there is growing interest in making smartphones and devices “contextually aware” such that they provide users with relevant information based on all available data sources and this requires utilizing data from many sensors and a variety of other data sources, and interpreting the data to provide higher level information. A number of companies have developed sensor fusion related technologies including semiconductor chips and software/middleware solutions. Select Sensor Fusion Company M&A Transactions Two of the four sensor fusion companies we highlighted in the first edition of this report were acquired. These included:  InvenSense/Movea – In July 2014, InvenSense announced that it agreed to acquire Movea for $60.9 million, plus an additional $13 million of potential contingent consideration. Movea was highlighted in the first edition of this report and had developed a broad range of middleware solutions addressing the sensor fusion/motion processing markets. Its technology bridges the gap between a broad range of sensors and applications, and includes technologies related to gesture- based control, activity monitoring, and indoor navigation. The company licensed its technology for products such as smart phones/mobile devices, smart TVs/set-top boxes, gesture-enabled remote controls, and sports/wellness products. Chapter 7 – Sensor Fusion and Middleware
  • 59. Page 59  Audience (Knowles)/Sensor Platforms – In June 2014, Audience announced that it acquired Sensor Platforms for $61 million (Audience was later acquired by Knowles). Sensor Platforms’ FreeMotion Library targeted sensor fusion and user context awareness for mobile applications. The Library provides the intelligence to combine and process data from sensors (e.g., accelerometers, barometers, gyroscopes, magnetometers) on mobile devices, and interpret that data into contextual information meaningful to application developers. The FreeMotion Library works with Android and Windows 8 operating systems, and can also be embedded in microcontrollers and/or smart sensors. Prior to the acquisition, Sensor Platforms announced collaborations with several major companies (Murata, Nvidia, etc.). Select Sensor Fusion Company Examples Apple has developed its own sensor fusion coprocessors for the iPhone and iPad. Its M7 coprocessor was included in the iPhone 5s. The M7 (based on ARM Cortex-M3) collects and processes data from a variety of sensors (accelerometers, gyroscopes and compasses), offloading this from the main A7 processor. The latest iPhone 6s includes an updated coprocessor (A9). As discussed in a later chapter, Microsoft’s HoloLens augmented reality product incorporates a “Holographic Processing Unit” which is a dedicated chip that performs a variety of tasks including many related to sensor processing and analysis. Examples of private companies addressing sensor fusion are given below. Hillcrest Labs Hillcrest Labs has developed a broad range of software and hardware solutions for motion- enabled products. The company’s Freespace MotionEngine software products include: Smart TV (for smart TVs, set-top boxes, and other connected TV devices), Mobile (for smartphones and tablets), Lite (in-air pointing for remote controls and mice), and Wear (low power activity tracking, sleep monitoring, context awareness, gesture control, and navigation activity tracking for wearable devices). These solutions provide a variety of sensor fusion and calibration algorithms, and provide application ready data. In addition, Hillcrest has developed a sensor hub software stack (SH-1 Sensor Hub) that is optimized for and can be pre-integrated into the Atmel ARM-based SAMD 20 microcontroller. This provides a complete “always on” chip level solution for implementing sensor fusion. The company also has modules and white-label products (pointer, remote). Hillcrest has an IP portfolio that includes over 130 granted and 235 filed patents exclusively owned by Hillcrest Labs. In addition it has a one-way license of a portfolio of motion patents from Technicolor. Some recent design wins announced by Hillcrest have included Bosch (head mounted displays), Lenovo (smart TVs), Coolpad (smartphone), Skyworth (smart TVs), Alcatel (smartwatch), and ZTE (smartphones). Hillcrest is based in Rockville, Maryland and is funded by NEA, AllianceBernstein, Columbia Capital, and Grotech Ventures.
  • 60. Page 60 PNI Sensor Corporation PNI has been an innovative sensor company since the 1980s. For many years, PNI was primarily known for its magneto-inductive sensors and has generally been recognized for being a technology leader in magnetometer/compass chips. The company continues to sell a variety of geomagnetic compass chips and compass modules. For example, PNI indicated that its RM3100 solution delivers more than 20x higher performance than existing MEMS or Hall Effect geomagnetic sensors. PNI’s digital compass solutions have been used in a variety of products and markets. While the company was primarily known for geomagnetic products, more recently PNI introduced Sensor Fusion chips and algorithms. Most smart mobile devices now incorporate a variety of sensors including an accelerometer (to detect the orientation of the phone), gyroscope (to measure tilt/rotation), a digital compass (to determine absolute direction as it can identify which way is North), and sometimes a barometer (to determine altitude which can improve GPS tracking). However, managing all of these sensors is complicated and includes a variety of “behind the scenes” activities such as calibrating the sensors and filtering the data. This often eats up a significant portion of the main processor’s cycles, resulting in substantial power consumption and reduced device performance. The PNI SENtral Fusion Sensor chip is designed from scratch to manage other sensors including gyros, accelerometers, and magnetometers. This provides a variety of advantages. For example, PNI indicates that its SENtral chip can perform sensor fusion/management at less than 1% of the power consumption of a general microprocessor, which can help significantly extend battery life. Using a proprietary “constant calibration” technology and filter algorithms, PNI indicates that Sentral can also significantly improve accuracy and reliability and reduce the need for complex algorithm development, enabling easy implementation for motion processing. Although Sentral is a chip-based solution, importantly PNI can also implement many of its sensor fusion algorithms in software. PNI has also introduced Sentral-A sensor fusion solution for Android-based products. Alternative approaches require either using a general MCU running a variety of 3rd party software or running a variety of software on the applications processor. PNI indicates that its approach enables dramatically less power consumption (typically <10% of alternatives). The new Sentral-A2 supports the latest versions of Android (Lollipop and Marshmallow). PNI recently announced SENtrace, a coprocessor for wearables providing highly accurate, ultra-low power pedestrian tracking for indoor locations and dense urban canyons. The solution enables devices to track location within one meter per 100 meters traveled while cutting power by 10-fold (utilizing dead reckoning and other advanced algorithms, it greatly reduces the number of times GPS readings need to be taken). Applications range from fitness to tracking children. PNI’s products are used in a variety of consumer devices, including the Wii U Game Pad, the Sony smart watch, as well as in a variety of robotic, navigation and automotive applications. The company is based in Santa Rosa, CA.
  • 61. Page 61 “The brain is a wonderful organ; it starts working the moment you get up in the morning and does not stop until you get into the office.” - Robert Frost Introduction With growing interest in wearable devices and in utilizing smartphones for body/health monitoring, dozens of companies have developed solutions that can monitor signals from the human body such as heart rate, temperature, and pulse rate for health/wellness/medical type applications. There have even been interesting developments in using brainwaves for controlling applications. Over time, the number of biometric sensors in smartphones and wearable devices is anticipated to significantly increase. This section briefly discusses some of these biometric sensor technologies. Heart Sensors (ECG Sensors) Electrocardiogram (ECG) sensors measure electrical activity of the heart. Heart muscles have negative charges (membrane potential) at rest and during each heartbeat the charge shifts towards zero (depolarization) and then back again. By measuring and analyzing the resulting electrical signals over a period of time, an ECG sensor can determine a variety of information about the heart (heart rate, abnormalities, etc.). ECG sensors are commonly used by hospitals, but there is growing interest in utilizing ECG technology for consumer applications, such as tracking health/wellness from a smartphone. Neuro-technology (EEG Sensors) There is interest in measuring and using brain wave activity. This typically involves wearing a headset that includes an electroencephalogram (EEG) sensor. The EEG signals from the brain are converted to digital and analyzed and can be used to determine if a person is focused/concentrating or not (for research/education applications) and if there may be wellness issues (for health reasons). It can also be used to sense a person’s mood. In some cases, EEG sensors can even be used to make simple control selections. For example, by measuring a user’s EEG signals when he is thinking about “jumping” and measuring it when he is thinking about “running”, an application can potentially allow the user to control a game character that has to run and periodically jump just by thinking, without using any other type of control or body movement. Typically, this requires some amount of set-up time for each user, as brain wave patterns vary from person to person in different circumstances. However, this offers the potential for some “science fiction like” control by thought alone. Chapter 8 – Brain, Heart and Other Biometric Sensors
  • 62. Page 62 Pulse Rate, Blood Pressure, and Many Other Biometric Sensors There is growing interest in incorporating a variety of biometric sensors into smartphones and wearable devices for health/wellness applications. Some vital signs that can now be measured include heart rate, pulse rate, temperature, respiration rate, and blood-alcohol level (although in some cases the accuracy of these mobile sensor products has not been quite as good as desired). Beyond these, there are sensors that can analyze sweat and skin for health reasons. Although more difficult to accurately measure than pulse or heart rate, there is strong interest in being able to measure blood pressure, as high blood pressure is a key early warning sign for many potential health issues (heart disease, stroke, kidney problems), but relatively few people measure their blood pressure on a regular basis. This would likely change if the technology were available in smartphones. The ability to monitor these vital signs also provides more data for doctors to diagnose patients (as opposed to just measuring once while in their office) and key data could easily be forwarded to physicians, enabling easier remote monitoring. Analyzing this type of data across large numbers of users could greatly improve the ability to predict health issues. There is also growing interest in sensors that can measure parameters such as pollution and pollen levels to alert the user so he or she can make choices that take those things into account. In general, it is believed that being able to easily measure key vital signs and parameters could help catch potential health issues early and improve overall health levels across the world. Growing Number of Biometric Sensors in Smartphones While all of the previously mentioned vital signs can be measured using bulky medical equipment in hospitals and doctors’ offices, the challenge is to implement the solutions in mobile devices or wearables, which are relatively small and require very low power consumption. Some companies have developed external add-on sensor solutions that can attach to a tablet or smartphone, but the general target is to make a sensor small enough so that it can eventually be integrated into a mobile phone. With MEMS and other advanced sensor technologies, this is becoming increasingly achievable and the number of sensors in portable devices is expected to continue to sharply increase over time. Select Biometric Solution Companies A variety of companies have announced sensors, software or wearable devices to measure health-related parameters and many of the new smart watches incorporate some vital sign sensors. In December 2015, Samsung indicated that it began selling a new processor specifically targeting health-focused wearable products and that it can take multiple measurements (including body fat, skin temperature and heart rate). The following are some companies focused on biometric solutions:
  • 63. Page 63 Leman Micro Devices Leman Micro Devices has developed advanced sensor solutions that can enable a smartphone to measure a variety of health parameters. Specifically, the company indicates that its “Smartphone Blood Pressure System (SBPS),” along with its Elemdy App and device driver software, enables smartphones to measure blood pressure with medical accuracy. It also enables accurate measurement of blood oxygen, heart rate, respiration rate, and body temperature, and includes a one-lead electrocardiogram (ECG). As high blood pressure is one of the most significant tell-tale signs of strokes and heart disease, Leman believes this could have a substantial impact on improving health and wellness around the world. While its SBPS module could be an add-on to a smart phone, it is designed to be small enough such that it could be directly integrated into smartphones, transforming phones into wellness devices. The company believes that just as cameras and GPS were initially a novelty in smartphones and later became a standard feature, health care functions such as blood pressure monitoring will also become commonplace. The company anticipates commercial shipments in 2016. Leman was founded in 2010 and is based in Lausanne, Switzerland. It has been funded by angel investors, VCs, and two unannounced major players in the smartphone sector. NeuroSky NeuroSky has introduced a number of solutions targeting both EEG sensors (brainwaves) and ECG sensors (heart rate). It has developed integrated chips, modules, and software/algorithms addressing these markets, and also offers a complete headset. The NeuroSky CardioChip is a system-on-a-chip (SoC) solution addressing the health and wellness market. It can be used to monitor ECG, heart rate, and heart rate variability. The chip integrates a low noise amplifier, a 16 bit ADC, sensor-off detection, hardwired DSP blocks, and a systems management unit. It targets bio-signals ranging from microvolts to millivolts. NeuroSky also provides a variety of complex algorithms to help determine heart rate, respiration rate, stress level, and other parameters based on the readings. NeuroSky has been a pioneer in developing solutions for the EEG/brainwave monitoring market. Its ThinkGear AM (TGAM) EEG sensor module senses the signals from the brain, filters out extraneous signals, and converts the results to digital for a variety of applications. The module includes NeuroSky’s own semiconductor chip which is optimized for this application. The chip includes A/D conversion, amplification off head detection, noise filtering for both EMG and 50/60Hz AC powerline interference, and NeuroSky’s eSense brainwave pattern technology. NeuroSky also provides advanced algorithms that can be used to determine mental activity/meditation, familiarity, and other parameters. In addition, NeuroSky offers a complete headset solution. NeuroSky indicates that its TGAM solution powers over 1 million consumer EEG devices around the globe. It recently announced that it is working with ARM on secure remote patient monitoring solutions.
  • 64. Page 64 In November 2013, NeuroSky announced that it received an investment from SoftBank to accelerate the development of mobile health/wellness solutions. NeuroSky was founded in 2006, and is headquartered in San Jose, California with offices in Asia and Europe. MC10 MC10 is a pioneer in flexible electronics, with a focus on biosensor applications. One of the major issues with bio-sensing is that traditional electronics are relatively bulky. Although a silicon die is very thin, traditional chip packaging (to protect the silicon chips) and printed circuit boards (to interconnect different chips) are thick and rigid. Ideally biosensors could be applied directly onto the skin without a person even noticing it while it is on. MC10 has developed technologies that allow extremely thin, flexible, conformal, transparent devices that can stretch, bend, and twist while on a person’s skin. This has the potential to greatly accelerate the penetration of biosensors into several markets. The MC10 solutions generally incorporate sensors to measure important parameters (temperature, heart rate, hydration level, UV levels, etc.) and then wirelessly transmit the data to a smartphone or other device. Because of the thinness of the MC10 solution, it can easily be incorporated into wrist bands or other accessory products, but even more significantly can be directly stamped onto the skin itself. MC10 developed a prototype called BioStamp, which can be stamped directly onto the skin and is then sealed with a spray-on bandage. The sensors can detect parameters such as body temperature, hydration levels, and UV exposure. This information can then be uploaded to a smartphone or sent to a doctor. The patch uses a thin film battery. The imprint can last for several days, withstanding stretching, flexing and even showering. In January 2016, the company unveiled its BioStamp Research Connect System, a system that provides researchers with the ability to gather physiological data through the use of its BioStamp sensors and a cloud-based software system. MC10 has also utilized its technology for a variety of other markets (e.g., it partnered with Reebok to develop CheckLight, a head impact indicator for contact sports). At CES, L'Oreal unveiled “My UV Patch” (using MC10’s technology), the first-ever stretchable skin sensor designed to monitor UV exposure and help consumers educate themselves about sun protection. MC10 believes there are a large number of additional applications for its flexible electronics, well beyond bio-sensing. MC10’s investors include North Bridge Venture Partners, Braemar Energy Ventures, Windham Venture Partners, Aberdare Ventures, Osage University Partners, Terawatt Ventures, Medtronic, Mitsui Ventures, and several undisclosed strategic investors. MC10 is headquartered in Lexington, MA.
  • 65. Page 65 Cambridge CMOS Sensors Cambridge CMOS Sensors (CCMOSS) has developed advanced gas sensor solutions for consumer applications. The company’s MEMS micro-hotplate technology enables metal oxide (MOX) gas sensors that are very small with very low power consumption and fast response times. CCMOSS indicates that its solution is about 97% lower in power consumption relative to conventional sensors and many of its sensors come in small 2x3mm packages. The company also indicates its solutions are more stable with less drift compared to traditional sensors. Applications include measuring carbon monoxide, ethanol (to determine blood alcohol level of a person), and general indoor air quality (volatile organic compounds, particulates, microbial contaminants, etc.). The company plans to introduce a variety of additional sensors over time using its core technologies. Its solutions can be used with smartphones, tablets, and IoT devices. The resistive micro-hotplates are fabricated using a silicon dioxide membrane and include an embedded tungsten heater acting as a heating element for the MOX based sensing material. The micro-hotplate can be used to heat the MOX material to up to 450°C and the electrical resistance of the MOX sensing material can be monitored to detect the target gas (e.g., if the sensing material reacts with carbon monoxide, the resistance decreases; the change in resistance can then be determined by the circuitry enabling the detection of carbon monoxide). The company was founded in 2008 and is based in Cambridge, UK. Xsensio Xsensio is an early stage Switzerland-based company that is developing an ultra-small sensing solution that is designed to sense electrolytes, metabolites, small molecules and proteins on the surface on a person’s skin. The company indicates that compared to existing solutions which rely on electrochemistry, its sensing technology platform is 10,000x smaller in size, and can achieve attomolar concentrations. Xsensio is a spinoff of the Nanolab at the Swiss Institute of Technology (EPFL) in Lausanne, Switzerland. Si-Ware Si-Ware is a mixed-signal ASIC company but it has developed its own family of infrared (FT-IR) spectrometers using MEMS technology. Si-Ware believes its solution provides the same functionality as standard bench top FT-IR spectrometer instruments, but with dramatically smaller size and cost. Spectrometers can quantify and identify materials based on their spectral response. There are many applications for the sensor ranging from helping farmers analyze soil to numerous industrial uses, but as cost is reduced with higher volumes, it could have a number of consumer applications. Si-Ware is based in Egypt.
  • 66. Page 66 HMicro Monitoring patient vital signs today often requires connecting numerous cables and wires to patients in a hospital which makes it difficult for them to move or sleep and hinders recovery. As such, there is growing interest in battery powered sensors that attach to the body, measure key vital signs, and then wirelessly transmit the data for further processing, monitoring, and analysis. This has many benefits but one of the historical challenges has been the wireless technology, which must be extremely robust (even with hundreds of patients and other wireless devices around) and must be highly power efficient (since the sensors often operate on a tiny battery). Traditional WiFi is generally viewed as too power hungry and other conventional wireless solutions have issues that make them ill-suited for this type of application. To address this, HMicro has developed a wireless solution that is well-suited for medical and bio-sensing applications. Its REACH SoC is a highly integrated chip that incorporates three different radios (its own ultra-low power Ultra-Wideband solution, a medical band radio, and WiFi) as well as a smart power management unit, a sensor subsystem with digital interfaces, and an application processor that customers can program. The triple radio provides significant redundancy and the chip intelligently switches among the radios to optimize efficiency while ensuring link integrity. Leveraging its SoC, the company has also developed module level solutions to enable easier customer integration. HMicro believes its technology will enable many “wired” medical applications to go “wireless” over the next several years and can transform the industry. While initially targeting medical, HMicro believes it can address many applications (auto, audio streaming, consumer, etc.). HMicro announced that it completed a $5.5 million capital raise in October 2015. In November 2015, it announced a partnership with Flextronics and that it became a member of the Flextronics Lab IX Start-Up Program. Flextronics and HMicro are jointly working together on a variety of projects including small sensor modules and an ECG patch and adaptor. Investors include Seraph Group, Reddy Capital Partners, and X/Seed Capital Partners. HMicro is based in Fremont, CA. Toumaz Group Toumaz Group has two divisions. The Frontier Silicon portion of the business focuses on digital radio chips and connected audio and was formed when Toumaz acquired Frontier Silicon in August 2012. The Toumaz Sensium Healthcare division has developed ultra-low power wireless solutions for monitoring patient vital signs, addressing the previously noted need for advanced wireless solutions for medical sensor applications. Sensium Vitals is an ultra-low power wireless system that tracks vital signs of general care patients using light weight wearable single-use patches that monitor heart-rate, respiration rate and temperature, and communicate the data wirelessly to any Web enabled device or smartphone. Sensium Vitals is the first ultra-low power, wearable, wireless patient monitoring product to have received FDA (510k) clearance. Toumaz also indicates that it developed the first radio chip to enable the wireless connection of medical devices in compliance to the latest IEEE 802.15.6 Medical Body Area Network (MBAN) standard. Toumaz is a public company listed on the AIM exchange and is based in London.
  • 67. Page 67 Linear Dimensions Linear Dimensions is a semiconductor and algorithm company that has developed a variety of solutions for bio-sensing related applications, including low power sensors. The company historically focused on developed analog and mixed-signal ASICs for customers but has increasingly developed a variety of its own innovative technologies. Its solutions can measure a broad range of vital signs (ECG, EKG, heart rate, pulse rate, etc.). The company indicates that its expertise in algorithms (including its advanced PEAL algorithm technology) provides significant advantages relative to other sensor solutions, which often work well in the lab but not when in motion (or require additional chips and sensors to compensate for motion). Linear indicates that its solutions remove noise and motion artifacts before they enter the system, rather than relying on additional power hungry devices to clean the signal up later. Some of its offerings include advanced photodetectors (which combined with its algorithms and VCSELs can measure a variety of vital signs); bio-impedance sensing solutions that measure ECG, IPG and EEG; bio-sensing reference designs; “lab on a chip” solutions; and a variety of precision analog and power management solutions. Linear Dimensions is based in Silicon Valley. Affective Computing/Emotion Analytics Affective computing is the concept of computing devices being able to recognize (and even simulate) human emotions. This is a type of biometric sensing, but typically, sensory input is used to determine the user’s emotional state. This could include optical imaging, voice input, or even brainwave patterns. By analyzing sensory data (the user’s facial expressions, body language and gestures, speech, vital signs, etc.), computing devices can determine a user’s mood and can use this information to better meet the user’s needs. For example, if a user is getting frustrated with a game, the device can adjust the difficulty level or provide some hints. If the user appears to be in a bad mood, it can play music to cheer the person up. Longer term, this should improve the ability of computers to interact with people, including in either virtual reality or augmented reality settings. Select Emotion Analytics M&A Transaction A recent example of an emotion analytics acquisition is given below:  Apple/Emotient – In January 2016, the Wall Street Journal reported that Apple acquired Emotient, an artificial intelligence company focused on analyzing faces to determine the user’s emotional state. According to the Journal, Emotient was unable to raise additional capital and its investors included Intel Capital.
  • 68. Page 68 “Every man takes the limits of his own field of vision for the limits of the world.” - Arthur Schopenhauer Introduction Image sensors are increasingly pervasive in a broad range of mobile devices. These sensors are now being used for more than just taking conventional photos. Earlier chapters discussed some potential applications for image sensors such as gesture recognition, eye tracking, and facial/eye recognition, which typically rely on optical image sensors. However, there are a variety of others. For example, there is growing interest in image recognition in which devices can recognize objects within captured images. Beyond basic photographs, there is growing demand for higher quality images and video as well as 360 panoramic images. Analyzing incoming video data to extract key information is extremely computationally intensive requiring advancements in both hardware and software. To address this, some companies have developed chips optimized for these applications. This chapter discusses a variety of technologies related to image/vision processing, many of which are also relevant for virtual and augmented reality (discussed in the next chapter). Image Capture Sensors A critical part of any digital camera or video camera is the sensor that captures the incoming light and converts it to electrical signals that can be stored or processed. Historically the two main types of image sensors were CCDs (Charge Couple Devices) and CMOS image sensors. For a number of years, CCDs were considered to have much better image quality and CMOS image sensors were used only for lower end applications. However, during the past couple of decades CMOS image sensor quality has significantly improved while costs have been sharply reduced (as CMOS is also used for the vast majority of semiconductor chips). As a result, CMOS image sensors have become pervasive in a broad range of consumer applications. CMOS image sensors are semiconductor components that incorporate tiny photodetectors which convert incoming photons into electrical charge and ultimately into an electrical signal that can be digitized and processed to represent the captured images. Color filters over the photodetectors enable the capture of color and tiny micro-lenses improve the image quality. The number of pixels that a CMOS image sensor can capture has continued to increase as process geometries shrink (e.g., from 1 megapixel a number of years ago to 16 megapixels or more). However, as the number of pixels increase the size of each pixel (photodetector) tends to decrease, which limits the amount of light that is captured by each pixel which can reduce image quality, creating engineering challenges. Chapter 9 – Image and Vision Processing
  • 69. Page 69 As image sensors are increasingly used for video, they must be able to capture many images per second (e.g., at least 30 frames per second, with growing interest in 45 and 60). To enable high quality images, there is demand for better quality sensors with high resolutions that can adjust to various lighting conditions and enable auto-focus and high dynamic range and other advanced features. Some of the major suppliers of CMOS image sensors include Sony, OmniVision, Samsung, ON Semiconductor, and Canon. An emerging area of interest for image sensing is light field cameras (plenoptic camera). These cameras capture a variety of information about a scene beyond what a normal RGB camera obtains (e.g., intensity of light, the direction the light rays are moving, etc.). This can enable functions such as refocusing an image after it has been captured. It also has advantages for creating 3-D images. A variety of light field cameras have been developed using different technologies (e.g., arrays of lenses, arrays of cameras, etc.), but they historically have had limitations and were very expensive. In November 2015, LYTRO (a pioneer in light field cameras) announced its Immerge family of light field cameras specifically targeting the high end professional market for capturing video for virtual reality applications. Over time, these types of cameras could become more commonplace if the technology improves and cost is reduced. Another area of interest is capturing 3-D images (rather than traditional 2-D images). This requires obtaining depth information about objects in the image. Technologies like time of flight (ToF) image sensors (in which the distance of an object can be determined by the time it takes light to travel from the camera to the object and back) were previously discussed in the chapter on gesture recognition. While gesture recognition is one important application for depth sensing, there are many others. For example, capturing 3-D information about an object or a room makes it significantly easier to implement a model of that image in virtual or augmented reality applications. A related topic to depth sensing is range detecting in which distances to objects are determined, often using image sensors (in some cases the application may not need to actually capture an image but just needs to obtain the distance). Applications for range detection include automotive (for collision avoidance), drones (collision avoidance), industrial (ensuring items are correctly positioned during manufacturing), law enforcement (speed detection), security, robotic guidance, and occupancy sensing. Select Image Capture / Sensor M&A Transactions Some recent image sensor acquisitions include:  AMS/CMOSIS – In November 2015, Austria-based AMS announced that it was acquiring Belgium-based CMOSIS from private equity firm TA Associates for 220 million euros. Revenue for CMOSIS for 2015 was projected to be approximately 60 million euros, per the announcement. CMOSIS sold a variety of high end CMOS image sensors, although its focus was mainly on non-consumer applications such as
  • 70. Page 70 medical, scientific, and machine vision. TA Associates had acquired CMOSIS less than two years earlier (TA announced that it was acquiring CMOSIS in December 2013). CMOSIS acquired AWAIBA (another image sensor company) in 2014.  Sony/Toshiba Image Sensor Unit – In October 2015, Sony announced that it would acquire Toshiba’s image sensor business for approximately 19 billion yen (about $155 million). Toshiba had a relatively small market share in image sensors but the transaction also included manufacturing facilities.  Hua Capital/OmniVision – OmniVision is a major supplier of CMOS image sensors. In April 2015, OmniVision agreed to be acquired by a consortium of Chinese private equity firms for $29.75 per share or about $1.9 billion. The consortium included Hua Capital Management, Citi Capital Holdings and GoldStone Investment Co. OmniVision had announced several months before that it received an unsolicited offer from the investor group.  Apple/LinX – In April 2015, the Wall Street Journal reported that Apple acquired LinX for $20 million. LinX was an Israeli company that specialized in small camera modules for smartphone cameras. Limited information was publicly available about the LinX solutions, but its website showed multi-lens modules that provided many competitive advantages (ability to refocus an image after it has been captured, 3-D object modeling, high image quality in low light environments, etc.).  ON Semiconductor/Aptina – In June 2014, ON Semiconductor announced that it was acquiring Aptina Imaging for approximately $400 million in cash. Aptina was a major supplier of CMOS image sensors for cameras, mobile devices, and gaming platforms. At the time of the announcement, ON indicated that Aptina had approximately $532 million in trailing revenue with gross and operating margins of approximately 29 percent and 3 percent, respectively. Aptina had previously been the image sensor business of Micron, which sold a majority of Aptina to private equity investors (Riverwood and TPG Capital) in 2009.  ON Semiconductor/Truesense Imaging – In April 2014, ON Semiconductor announced that it was acquiring Truesense Imaging for approximately $92 million in cash. Truesense sold image sensors primarily for industrial markets (machine vision, surveillance, traffic monitoring, medical and scientific). Truesense Imaging’s revenue for 2013 was approximately $79 million with gross and operating margins of approximately 44 percent and 23 percent, respectively. In addition to Aptina and Truesense, ON previously acquired a CMOS image sensor business from Cypress Semiconductor in 2011 for $31.4 million.
  • 71. Page 71 Select Image Sensor/Capture and Range Detection Companies Examples of image sensor, image capture, and range detection companies are below. InVisage InVisage has developed an innovative image capture technology called QuantumFilm which it believes has substantial advantages relative to conventional CMOS digital image sensor technology and utilizes quantum dot technology (quantum dot technology was previously discussed in the chapter on touch screens and displays). Specifically, the company indicates that silicon image sensors sense light linearly and saturate when the number of electrons a pixel can store exceeds a pixel’s fixed full well capacity (FWC), which is related to the size of a pixel. As pixel size continues to shrink, it results in poor dynamic range and image quality. In contrast, the QuantumFilm sensor provides a much higher FWC for a given pixel size and also has a non-linear response to light, similar to film. In addition, its QuantumCinema technology optimizes the response (linear in the darker regions, more film like in the brighter regions) to greatly improve image quality in a broad range of lighting environments, including those with a mixture of dark and bright regions. It also enables slow motion video and incorporates an electronic global shutter (full frame capture). In November 2015, InVisage introduced its initial product, the 13 megapixel Quantum13 camera sensor, which it indicated is the first image sensor that uses quantum dot film. It includes a global shutter and supports 4K video. InVisage stated that initial shipments to leading smartphone vendors would commence by the end of 2015. In 2014, InVisage announced an $18 million funding which brought its total funding to more than $100 million. Investors include GGV Capital, Nokia Growth Partners, RockPort Capital, InterWest, Intel Capital and OnPoint Technologies. Later that year, InVisage raised additional funding from China Oceanside. The company is based in Silicon Valley. Occipital Occipital originally developed a barcode scanning app and a 360 Panorama app, but more recently has shifted its focus to advanced 3-D scanning. Its Structure Sensor is an add-on to an iPad that enables a user to rapidly capture 3-D images/depth maps of objects or entire rooms. This enables easy creation of 3-D models. For example, a user could capture a 3-D image of a couch and then use that model in an application to see how the couch would look in a particular room. Rapidly creating 3-D images and models of objects greatly simplifies the creation of 3-D applications, including virtual and augmented reality. Its Structure Core solution is optimized for AR/VR, SLAM, and robot vision. While the company sells hardware scanners, it indicates that most of the core technology is in its software and SDK. Last year, Occipital acquired Lynx Laboratories, which had its own 3-D scanner technology but shifted to a software focus with its software running on Occipital hardware. Occipital indicated Lynx’s IP can reduce the error of 3-D sensors by up to 90%. In July 2015, the company raised $13 million from Intel Capital, Shea Ventures, Grishin Robotics and Foundry Group.
  • 72. Page 72 Sensl Sensl has developed a unique silicon photomultiplier (SiPM) sensor technology. Unlike traditional CMOS image sensors, photomultipliers are extremely sensitive to light (can multiply the current produced by incident light by more than 10 million times). There are a number of applications for which this type of sensitivity is required (e.g., medical imaging). Historically, this was accomplished using vacuum tube technology (one of the few remaining markets for vacuum tubes) for high end applications in which performance was important and by avalanche photodiodes for more cost-sensitive applications. However, Sensl believes its silicon sensor technology has enormous advantages relative to those traditional products. As would be expected, its solution is dramatically smaller, more robust, and less costly relative to vacuum tube-based photomultipliers and it doesn’t require the high voltages that traditional photomultipliers do. Prior to Sensl, however, silicon-based sensors could not achieve the performance of tube-based solutions. On the lower end, its solution provides substantially better performance and other technical advantages compared to avalanche photodiodes (high end performance at lower end cost). Sensl initially targeted medical applications (as well as military and scientific applications) and is in discussions with several major companies in those markets. However, more recently it has begun to leverage its technology for range detection/LiDAR type applications in which it believes its technology can provide compelling advantages due to its high level of sensitivity and performance advantages. Some of these applications include: automotive range detection, aerial surveying, robotics, and autonomous vehicles. Longer term, as Sensl drives the cost of its SiPM solutions down through higher volumes, it believes that it can penetrate high volume consumer markets such as gaming and augmented/virtual reality products. Sensl is based in Ireland. Leddartech Leddartech has developed advanced ranging and detection solutions that utilize image sensors and time of flight technology. Its “light-based radar” sends very short light pulses of (invisible) light about 100,000 times per second to actively illuminate an area of interest. The sensor captures the light backscattered from objects (either fixed or moving) over its detection area, and processes the signals to precisely map their location and other attributes. The company indicates it has the only light-based sensor technology with the capability to read all elements of the returned signal at once and to analyze them simultaneously. Its products include the LeddarCore chip and its family of LeddarOne complete modules. The company has indicated that it working with a variety of major partners in automotive and other markets. Leddartech was founded in 2007 and is based in Quebec City.
  • 73. Page 73 Image Recognition One area of growing importance for mobile devices is image recognition. Using image sensors a device might not only capture images but also recognize what those images are and take appropriate actions. So if a person, for example, captures an image of a certain product on a smartphone in the store, the smartphone might be able to recognize what the product is and bring up relevant information (reviews, coupons, etc.) about that product. However, recognizing images is no simple task. For example, Coke products can come in a variety of shapes and sizes and the user could be capturing images of the product from a number of different distances and angles with a variety of other objects around it. Image recognition is used in a variety of applications. Amazon’s iOS app, for example, incorporates its “Flow” technology, which enables users to capture images of products which are then recognized through visual search technologies. As a result, a user can simply point the iPhone camera at products the customer wants to buy (books, DVDs, grocery items, etc.) and the items register in the Amazon app and can then be purchased, if available on Amazon. Facial recognition (discussed in a previous chapter) is a type of image recognition. Image recognition is an important part of augmented reality solutions as the systems must be able to recognize and track objects from many positions. A variety of image recognition software solutions have been developed. Select Image Recognition M&A Transactions Some examples of recent image recognition M&A transactions are noted below. Although we are generally including only transactions from the past few years in this report, it is interesting to note that Google acquired an image recognition technology company named Neven Vision back in 2006. Neven’s technology could identify objects and people in images. We also note that some relevant transactions are included in the following chapter on virtual/augmented reality.  Google (DeepMind)/Vision Factory – In October 2014, Google indicated that it had acquired Vision Factory, a UK-based company focused on applying artificial intelligence techniques to enhance the accuracy and speed of object recognition and improving vision-based computer systems.  Qualcomm/Euvision – In September 2014, Qualcomm acquired Euvision, as reported by TechCrunch. Euvision was spun out of the University of Amsterdam and was focused on using artificial intelligence techniques for image recognition applications. Its initial product was an app that could analyze and organize photos on a smartphone into different categories.
  • 74. Page 74  Twitter/Madbits – In September 2014, Madbits announced that it had been acquired by Twitter. Madbits developed visual intelligence solutions that could analyze raw media and extract and understand relevant information. The company used deep learning to understand the content of an image.  Google/Jetpac – In August 2014, Google acquired Jetpac. Jetpac developed technology that could analyze images to obtain contextual information. For example, it could analyze the details in a photo to determine what location it was taken, or analyze photos of a restaurant to determine who it might appeal to. Beyond photos, the technology was also capable of examining and analyzing video.  Qualcomm/Kooaba – In January 2014, it was reported that Qualcomm Connected Experiences acquired Kooaba (per the 451 Group). Kooaba was a Switzerland- based company that developed a variety of image recognition technologies. That is, its technology could recognize objects within captured images and then bring up relevant information about the objects.  Pinterest/VisualGraph – In January 2014, Pinterest acquired VisualGraph, a small start-up focused on image recognition and visual search.  Google/Industrial Perception – In December 2013, a New York Times article reported that Google had acquired seven robotic related acquisitions, including Industrial Perception, which was focused on computer vision systems for controlling robotic arms and other applications.  Yahoo/IQ Engines and LookFlow –Yahoo acquired LookFlow (image recognition technology company) in October 2013 and IQ Engines (which had an image recognition platform that could help identify objects, landmarks, or people in an image for mobile and Web applications) in August 2013 (per the 451 Group).  Dropbox/Anchovi Labs – In September 2012, Dropbox acquired Anchovi Labs, which was focused on vision processing and image classification using learning and artificial intelligence technologies (per the 451 Group). Select Image Recognition Companies Below is an example of an image recognition company. Some image recognition related companies are included in the next chapter on virtual and augmented reality. LogoGrab LogoGrab has developed a browser-based image recognition solution focused on logo recognition. Its WebScanner solution eliminates the need for users to download a specific app, making it easier for marketing campaigns to achieve greater reach. Its LogoGrab technology can be easily integrated into mobile Web sites or customers’ apps. LogoGrab also has a Visual Listener solution that can accurately detect logos in social media utilizing LogoGrab’s large database of corporate logos.
  • 75. Page 75 LogoGrab has a variety of major customers. For example, in September 2015 it announced a digital campaign with Heineken in 85 countries related to the James Bond movie Spectre (by “grabbing” the logo on limited edition Heineken packaging, fans could unlock behind the scenes footage from the movie). LogoGrab has also done work with Coke and others. The company is based in Dublin, Ireland. Vision Processing Mobile devices are incorporating a growing number of image sensors. In addition, the complexity of image sensors is increasing (e.g., high pixel count sensors, 3-D/depth sensors, light field sensors, etc.). Augmented reality technologies are becoming more commonplace and there are many applications in which there is interest in not just capturing images but in being able to analyze and post-process images and determine what objects are in the image. Emerging technologies like simultaneous localization and mapping (SLAM), which enable devices to build internal maps of local environments without having prior knowledge of the local environment, require substantial processing power. In many cases, the processing required for these types of applications greatly exceeds the capabilities of mobile devices, which have limited processing power and cannot consume too much power. Traditional microprocessors are not specifically designed to handle vision processing. Just as digital signal processors (DSPs) and graphic processing units (GPUs) were developed to better handle tasks that were not well suited for ordinary processors (e.g., baseband cell phone processing, PC graphics), some companies have developed vision processing chips (VPUs) that are specifically optimized for the types of parallel processing required for vision/image applications for mobile devices. This can help enable a variety of advanced features and functions while also freeing up the main/application processor to better handle other tasks and greatly improve the user experience. One development in vision processing is Google’s Project Tango. Project Tango was announced by Google a few years ago and is focused on giving mobile devices a human- like understanding of space and objects by using computer vision and sensors. As an example, a device may be able to develop a complete internal map of the inside of a user’s house and then use that for a variety of applications such as interior design or even having game characters play, for example, “hide and seek” in images of the user’s house rather than in a generic house. An initial prototype looked like a traditional Android phone (although thicker) and incorporated a regular image sensor (4MP camera), a depth sensing sensor/receiver, and a lower resolution motion tracking camera, as well as dedicated vision processing chips. Subsequently, Google released a Project Tango development kit in mid- 2014, which was a 7 inch tablet with a motion tracking camera, depth sensing, and many other functions. A variety of major tech companies have announced support for Project Tango.
  • 76. Page 76 Select Vision Processing M&A Transactions Vision processors are a relatively new segment, but there has already been some M&A activity in the sector.  Apple/FlyBy – In January 2016, it was reported by the Financial Times that Apple acquired FlyBy Media, which developed vision-based software for smartphones and other devices. FlyBy was a vision-based software partner for Project Tango. Its focus was connecting the real world with digital content.  Freescale/CogniVue – In September 2015, Freescale announced that it had acquired CogniVue. CogniVue was highlighted in the first edition of this report and had developed software, IP, and an Image Cognition Processor (ICP) that were optimized for vision type applications. Its CV2201 APEX chip was designed specifically for vision processing and incorporates a RISC processor and local dedicated memory. CogniVue indicated it could achieve a 100X advantage in performance per power relative to conventional approaches. The APEX could either be a stand-alone chip or its IP core could be integrated into a larger SoC. CogniVue had previously announced a partnership with Freescale addressing vision processing in automotive applications. Select Vision Processing Companies The following describes a couple of companies that have developed vision processor chip solutions, as well as a company that provides middleware specifically targeting vision processing applications. Movidius Movidius is a leader in the emerging vision processing chip market. The company has developed advanced imaging chipsets and algorithms that enable always-on smart capture and continuous vision processing applications for mobile devices. As traditional processors and graphics processors are not specifically optimized for vision-type applications, Movidius believes that those types of chips are not well-suited for vision processing, especially for mobile devices which require not only extreme levels of computation but have significant power consumption limitations. Movidius’ second-generation chip, the Myriad 2 Mobile Vision Processor, is a Vision Processor SoC that incorporates a variety of differentiated features. Myriad 2 integrates 12 proprietary SHAVE (Streaming Hybrid Architecture Vector Engine) processors, 2 RISC processors, an optimized intelligence memory fabric, a variety of advanced imaging/vision hardware accelerators, and a number of I/O interfaces. The SHAVE processors incorporate a hybrid stream processor architecture combining the best features of GPUs, DSPs and RISC as well as unique features such as hardware support for sparse data structures. The architecture is designed to maximize performance-per-watt while
  • 77. Page 77 maintaining ease of programmability. The chip is produced on 28nm process technology. The result is extremely high performance for vision/imaging applications with ultra-low power consumption (e.g., two trillion 16-bit operations per second while consuming an average of less than 500 milliwatts). In addition to the Myriad 2 chip itself, Movidius has developed a broad range of vision-related algorithms (machine learning, 3-D depth, tracking/navigation, etc.) and an SDK to enable easy implementation for customers. In January 2016, Movidius announced that it is working with Google to accelerate the adoption of deep learning in mobile devices. As part of the agreement, Google will source Movidius processors and the entire Movidius software development environment, and Google will contribute to Movidius’ neural network technology roadmap. The target is to enable future products that have the ability to understand images and audio with incredible speed and accuracy, offering a more personal and contextualized computing experience. Movidius is headquartered in Silicon Valley with a major office in Ireland and a recently opened office in China. In April 2015, Movidius announced that it had raised $40 million in new funding. The investment was led by Summit Bridge Capital and included new contributions from ARCH Venture Partners and Sunny Optical, as well as early investors including Atlantic Bridge Capital, AIB Seed Capital Fund, Capital-E, DFJ Esprit, and Robert Bosch Venture Capital. Inuitive Inuitive has developed a 3-D imaging and computer vision multi-core processor (NU3000) and NU3000-based camera solutions. It indicates that its solution is specifically optimized for applications such as augmented reality as it incorporates not only 3-D image processing and computer vision, but also depth sensing processing. The NU3000 integrates a computer vision processor, an image processor, and a general purpose processor on one chip (two Ceva DSPs and an ARM Cortex A5 and ARM Neon Vector Processor), as well as a variety of proprietary hardware accelerators. The solution incorporates a 3-D Depth Engine based on a proprietary “Assisted Stereoscopy” technology and can provide a real-time localization and mapping solution. It can process 3 video streams in real time (2 depth cameras and 1 RGB image sensor). Its open architecture allows integration of a customer’s software into the solution and offloads the computationally intensive processing from a main CPU. Inuitive recently announced a reference design dedicated for Google’s Project Tango. The reference module incorporates the NU3000 with two cameras arranged in a stereoscopic setting, a fish-eye camera for 2-D wide-angle imaging, and an inertial motion unit. The company was founded in 2012 and is based in Israel. Its investors include T-Venture (the VC Company of Deutsche Telekom) and 7-Main.
  • 78. Page 78 Van Gogh Imaging VanGogh Imaging develops object recognition analysis and vision processing middleware for mobile and embedded devices. There is growing interest in implementing advanced 3-D vision-based solutions, but many companies lack the expertise to create some of the key functions. By using Van Gogh’s Starry Night software, customers can more rapidly develop solutions for a variety of applications. Starry Night is a computer vision Unity1 plugin that utilizes a patented shape-based registration technique which uses priori information about a scene or object. VanGogh indicates that it is highly tolerant to noise and can be fully automated, eliminating the need for manual post processing to create accurate 3-D models. It can examine a scene, recognize objects, and create 3-D models of those objects for applications ranging from 3-D printing, augmented reality, automotive, and robotics. In January, the company announced its Starry Night-MR computer vision software which is specifically optimized for augmented and virtual reality applications. Unlike many solutions, Starry Night can handle dynamic real world environments with moving objects and changing lighting conditions. VanGogh is based in McLean, Virginia. Video/Image Processing Another important aspect of video and image capture is the subsequent processing required to optimize and store the video. In many cases, the format of the video or images that are captured does not match with the format of how the video/image is displayed, and must be transformed. In some cases, the incoming images need to be “cleaned up” to improve quality and make adjustments based on lighting. There may be interest in zooming in or out, adding special effects, correcting geometrical distortions caused by the motion of the camera in 3-D space, or enhancing the captured images due to poor lighting conditions. Given the enormous amount of data required for video, there is often a need to compress the video before storing or transmitting it, and then subsequently decompress it before playing the video. All of these types of functions require significant processing power and advanced software, and in some applications this needs to be done in real time. As a result, there are a number of companies that have developed advanced software and/or chips to perform some of these tasks. Select Video/Image Processing Company Below are two companies that have solutions related to video and image processing. GEO Semiconductor GEO Semiconductor provides a variety of solutions related to image and video processing. GEO was formed in 2009 and acquired a number of key products and patents from Silicon Optix. In January 2013, GEO completed the acquisition of Maxim’s video compression business unit (much of which came from Mobilygen, a mobile compression start-up that Maxim acquired in 2008).
  • 79. Page 79 GEO has four core technologies: 1) its eWARP geometric processing engine (which can scale, transform, and rotate images to resolve a variety of alignment, calibration, and other issues associated with advanced camera and projection systems); 2) advanced video compression (virtualized H.264 compression solutions scalable to 4K resolutions); 3) Image Signal Processing (ISP) which provides a variety of advanced camera control features and algorithms (dynamic color and exposure control, advanced adaptive noise reduction, algorithms to optimize the representation of the scene including both dark and bright areas); and 4) massively parallel video processing architectures to optimize performance. GEO has developed a number of chips (video camera processors, geometric processors) that integrate one or more of these technologies. For example, its GW4300 camera processor integrates a high dynamic range ISP processor, an eWarp processor, an on- screen display controller and a Tensilica CPU running a variety of video processing algorithms. GEO also sells Realta, a highly integrated highly parallel SoC that was the first fully software programmable video array processor capable of performing over 1 trillion operations per second. GEO’s solutions are used in a broad range of markets including consumer, automotive, security, and IoT. In mid-2015, the company announced a line of video processors specifically optimized for automotive applications, such as a digital rear view mirror. GEO has operations in San Jose, Toronto, Orlando, and Bangalore. Irida Labs Irida Labs develops software and silicon IP for embedded video and vision processing applications such as video stabilization, face detection/recognition, low-light video enhancement, and pedestrian detection. Its target markets include mobile, wearables, surveillance, automotive and consumer electronics markets. The company has announced partnerships with CEVA, ARM, and Cadence/Tensilica. Irida was founded in 2007 and is based in Greece.
  • 80. Page 80 “Reality leaves a lot to the imagination.” - John Lennon Introduction A number of visionary HMI technologies appeared on Star Trek. In addition to the main Enterprise computer and Data (a lifelike android), one of the most interesting was the holodeck, a simulated reality facility in which people could interact with computer generated people and objects which appeared, communicated, and felt completely realistic to anyone inside the holodeck. While anything approaching the holodeck is still many decades away, there are a number of technologies under development that will move us in that direction. During the past couple of years, there have been a number of high profile virtual reality and augmented reality product announcements from some of the largest technology companies in the world. Virtual reality and augmented reality are sometimes used interchangeably but actually refer to somewhat different (although related) types of technologies. Specifically:  Virtual Reality – With virtual reality, the user is immersed in a computer simulated reality that is separate from the real world (other than the user’s movements). For an immersive virtual reality experience, the user typically wears some type of opaque eyewear/helmet such that he cannot see the real world and sees and hears only computer generated images and audio (although the images/audio could be recorded real world images or videos).  Augmented Reality – With augmented reality (also referred to as mixed reality), the user sees the real world (what is in the room around him) but computer generated images and other sensory input is added. For example, a user might see an actual table in the room he is standing in, but also see a computer generated alien sitting under that table. For an immersive augmented reality experience, real world images have to be combined with computer simulated images. Both of these technologies are expected to become increasingly commonplace over the next few years and could have an impact on a variety of markets (beyond the obvious gaming markets). Implementing immersive virtual or augmented reality simulations requires combining many of the technologies previously discussed in this report, in addition to highly sophisticated software and significant computational power. This chapter discusses both virtual and augmented reality, including a section at the end on holographic imaging which is still in its infancy but could become significant over time. Chapter 10 – Virtual and Augmented Reality
  • 81. Page 81 Virtual Reality As previously noted, virtual reality (VR) creates an entirely computer-simulated environment. At its simplest, a conventional video game can be considered virtual reality since it simulates another world that is completely computer generated. However, the term virtual reality is more commonly used to refer to immersive experiences in which the users have the sense (at least to some extent) that they are in a different virtual world. Historically, this was most commonly implemented through helmet mounted displays that provide 3-D video imaging and audio. Basic virtual reality products were developed as far back as the 1960s (although some of the early helmet models were so heavy they had to be suspended from the ceiling to wear). In the early 1990s, Sega introduced the Sega VR headset which incorporated LCD screens in the visor and sensors that enabled the system to track the user’s head movements (it was initially anticipated to be used for both arcade games and home consoles but was deployed only in arcades). A variety of basic virtual reality arcade games were subsequently developed but until recently the lack of processing power prevented systems from creating truly immersive virtual reality experiences. The virtual reality industry received a boost a couple of years ago when Facebook announced the acquisition of Oculus for over $2 billion. Given the large amount paid for the acquisition and that Facebook has historically focused on the mass market, the acquisition accelerated interest in virtual reality. For an interactive virtual reality experience to feel as real as possible, the images must adjust based on the user’s movements. For example, if the user turns her head or moves, the scene must rapidly adjust to show that 3-D view. To make the experience interactive, the system would ideally track the user’s gestures (e.g., if the user lifts his hands, his avatar in the virtual world also lifts his hands). One issue with VR is because the user may be moving around but cannot see the room he is actually physically in, there is a need to warn him if he will run into an actual object (e.g., a wall). All of these things require a significant amount of sensors, sensor processing, 3-D image processing and computational power. Oculus Rift Rift is the high profile virtual reality headset developed by Oculus, which is now part of Facebook. The Rift is an advanced virtual reality headset that connects to a PC via cable (the PC runs the virtual reality software so Rift is not a stand-alone device). The Rift headset incorporates a variety of technologies including high resolution OLED displays and some additional features to improve image quality. The headset also incorporates high quality lenses, headphones (for 3-D audio), and has full 6 degree of freedom rotational and positional tracking. The positional tracking is accomplished through an external infrared tracking sensor system called Constellation. The Rift comes with an Xbox One wireless gamepad which can be used as a controller with Rift-based games. Oculus has also developed its own Oculus Touch controller which is optimized for virtual reality applications. A variety of software and apps have been developed for Rift.
  • 82. Page 82 Several preliminary versions of Rift have been introduced, but Oculus has indicated that the commercial version of Rift for consumers will be released in 2016. Oculus recently began accepting pre-orders for Rift with a June 2016 estimated delivery date. The price is $599 and includes two games. In addition, the Oculus software has been used as part of the Samsung Gear VR solution which was already commercially released in late 2015 and is discussed below. Oculus has also indicated that it is already working on a next-generation VR solution beyond the Rift. Other Virtual Reality Products Besides the Oculus Rift, a number of other major technology companies have announced virtual reality solutions. Below are some of the high profile products. However, many other companies have also introduced innovative VR solutions (many shown at CES 2016).  Samsung Gear VR – The Gear VR is a head-mounted virtual reality accessory device developed in conjunction with Oculus (it uses software from Oculus). It is designed to work with Samsung’s smartphones (e.g., Galaxy Note 5, S6, etc.). The headset incorporates a number of sensors (accelerometer, gyro, proximity, etc.) and the Samsung smartphone directly snaps into the Gear headset (the smartphone produces the images and the headset has an advanced optical lens system to create the immersive experience). Samsung introduced a couple of preliminary development versions but formally began selling the consumer version in November 2015. The Gear VR enables 360 degree virtual reality viewing. A number of virtual reality games, videos, shows, and apps have been developed for Gear.  Sony PlayStation VR Headset – PlayStation VR (previously referred to as Project Morpheus) is a virtual reality headset developed by Sony that is expected to work with the PlayStation 4 game system. A prototype shown in 2015 included an OLED display capable of displaying images at 120 frames per second. It incorporates advanced head tracking and an output that enables others to watch what the user is seeing on a separate TV display. While the final product has not yet been released, Sony has indicated that it will likely be introduced during the first half of 2016.  HTC Vive – HTC announced the Vive in early 2015 with plans for commercial shipments in 2016. The Vive was developed in conjunction with Valve Corporation, a video game developer. It is a head-mounted display solution that incorporates two small high resolution screens (one for each eye) with a 90Hz refresh rate (90 frames per second) as well as over 70 sensors. Although it is a VR solution, it includes front facing cameras that can identify objects in a room to, at a minimum, prevent users from hitting a wall or an obstacle (its Chaperone technology which can alert users if they are getting too close to real world objects). A number of game developers have announced support.
  • 83. Page 83  Google Cardboard – Google Cardboard is an ultra-low cost virtual reality solution in which the headset is made from cardboard and relatively inexpensive lenses. Google doesn’t actually sell the product but provides the specs for free. The solution works with a smartphone which when inserted into the Cardboard headset provides the actual display. Google software divides the smartphone display into two sets of images (one for each eye), and creates a 3-D stereoscopic image. A number of major companies (Volvo, LG, Mattel, etc.) have released Cardboard-based solutions as part of promotional campaigns. Augmented Reality Augmented reality (AR) refers to combining real world images with digitally generated data and/or images. That is, in contrast to virtual reality in which all of the imagery is computer generated, augmented reality augments the real world images a user can see. A simple example of augmented reality is the yellow “first down line” that is added to the field during NFL TV broadcasts (i.e., you see the real time images of the field along with a computer generated yellow line that appears to be a real line on the field). A goal with AR is to integrate the digitally created content as seamlessly as possible such that it appears as part of the real world image. Augmented technologies can be implemented in a variety of ways. For example, real world images can be captured via image sensors and displayed on a screen while a variety of 3-D graphics are added to enhance the image. Alternatively, for more immersive experiences, augmented reality can be implemented through glasses or head mounted devices in which the user sees the real world but computer generated graphics are added to enhance the images that a person sees. Although Google Glass was not specifically developed as an augmented reality device, several developers have created apps to implement AR on Google Glass. The number of potential applications for augmented reality is enormous. For example, the real world image of a car engine can be augmented to show the user how to perform certain types of car maintenance (graphics and instructions overlaid on the actual engine image). AR can overlay a real time image of an object or place that a person is looking at with relevant information (what the object is, information from online resources about the object or place, etc.). It has many potential uses for field service (an “expert” can guide a non-expert in how to repair something without physically being there). AR is commonly used for advertising purposes (adding 3-D images to real world objects), and can be used for engineering design, navigation, and a variety of other applications. Automotive heads- up displays (HUDs) in which navigation or warning images are displayed on the driver’s windshield is a type of augmented reality. AR is also expected to become increasingly common for gaming applications in which gamers can play in their own house (e.g., shoot at images of characters moving around in the actual room they are in).
  • 84. Page 84 Implementing immersive augmented reality is very complex. For example, if an augmented reality game has an image of an alien sitting on a couch, the system has to be able to track where the user is in the room and where the couch is in order to ensure the computer simulated alien remains in the right spot regardless of where the user moves to in the room. Several major technology companies have developed AR-related software solutions and augmented reality has been incorporated into a variety of advertising campaigns. For example, these solutions might combine image recognition with AR such that when a camera from a smart phone recognizes a certain product, it creates computer simulated images or games around the product. A number of stores have used augmented reality to assist customers in deciding on different colors or what they would look like with different clothing/makeup or what furniture to buy (enabling users to see what a room might look like with different colors or different pieces of furniture). There have also been some augmented reality applications for fashion (e.g., allowing people to “try on” different clothing or make-up without physically putting anything on). As augmented reality improves, the list of potential applications is expected to significantly increase. Microsoft HoloLens A major augmented reality development was Microsoft’s HoloLens which the company announced in early 2015. HoloLens is a glasses-type headset that incorporates a variety of processors, sensors and imaging technology and enables users to see both the real world and 3-D computer generated images, which Microsoft refers to as “holograms” (technically they aren’t true holograms in that you need to be wearing the HoloLens glasses to see the images, but Microsoft believes the imaging goes beyond traditional augmented reality images). In one demo of HoloLens, a user sees an image of a large laser-type weapon appear on his hand, and it remains on his hand regardless of where his hand moves. Subsequently a computer generated alien ship appears to crash through the wall of the room the user is standing in and aliens begin to jump out and attack him, but he can shoot them with his laser weapon. In another demo, a 3-D Minecraft city appears on top of a real table in the room in which the user is standing in, and the user can then interact with the city. HoloLens is part of Microsoft’s Windows Holographic computing platform initiative. The HoloLens is still in development, but reportedly incorporates a high end CPU, a graphics processor (GPU) and a new holographic processing unit (HPU), which is a custom sensor fusion type coprocessor developed by Microsoft. The HPU processes sensor data (gesture recognition, mapping, speech processing, etc.) and according to Microsoft can handle terabytes of information in real time. HoloLens also integrates many different types of sensors (accelerometer, gyro, magnetometer), as well as depth cameras, video camera, and microphones. In addition, it contains an innovative “light engine” and multi-layer combiner lenses that enable users to see the computer generated images. As the HoloLens is still in development, the product is still subject to modifications. Microsoft has not set a date for the release of a commercial HoloLens solution, but indicated that it plans to introduce a development version in the first quarter of 2016.
  • 85. Page 85 Select Virtual Reality/Augmented Reality M&A Transactions Examples of virtual reality and augmented reality M&A transactions include:  Apple/Faceshift – In November 2015, TechCrunch reported that Apple acquired Faceshift, a facial analysis start-up. The Faceshift demos show how a user can move their facial features (smile, frown, open/close eyes, move head) and an avatar on a computer display performs the same movements. Faceshift reportedly developed technology used in the latest Star Wars film, and is based in Switzerland. This technology could potentially be used for virtual/augmented reality applications.  PTC/Vuforia(Qualcomm) – In October 2015, PTC announced that it was acquiring Qualcomm’s Vuforia business for $65 million. Vuforia was a leader in augmented reality solutions and software. Qualcomm indicated that Vuforia was used by over 175,000 registered developers and powered more than 20,000 apps.  Blippar/Binocular and Layar – Blippar is a private augmented reality company (highlighted later) but has already made acquisitions. In October 2015, it acquired Binocular, which developed augmented reality applications (virtual “try on”) for companies. For example, it created an app for Luxotica that allows users to try on glasses using a smartphone, without physically touching glasses. Blippar confirmed in June 2014 that it had acquired Layar, which developed a popular augmented reality platform. The Layar SDK enables easy mobile AR development.  Snapchat/Looksery – In September 2015, TechCrunch reported that Snapchat acquired Looksery. Looksery was a private company that developed animated “video lenses” which enable users to modify the look of their faces during video calls. For example, the lenses could make a person appear as a monster or a cat or a variety of other options. Alternatively it could also improve the image (eliminate blemishes, etc.). Ukrainian news agency Ain.ua indicated that although there was no confirmation, its sources indicated that the price was approximately $150 million.  Facebook/Surreal Vision – In May 2015, Oculus announced that it had acquired Surreal Vision. Surreal Vision was focused on real-time 3-D scene reconstruction (generating an accurate representation of the real world in the virtual world).  Apple/Metaio – In May 2015, it was reported (by Forbes, TechInsider, and several others) that Apple acquired Metaio. Metaio was highlighted in the first edition of this report and had been a major supplier of augmented reality solutions including Creator (an AR publishing tool that enables users to create a broad range of AR materials using drag and drop), SDK (a software development kit for AR), Cloud (lets users manage AR and content in a single place), Visual Search, Junaio (advanced AR Browser for iPhone and Android), and Engineer (hardware IP). Metaio reportedly had well over a thousand customers including Volkswagen, Mitsubishi, Audi, McDonalds, Macy’s, Adidas, Red Bull, and Lego. Metaio was based in Germany.
  • 86. Page 86  Daqri/ARToolsworks – In May 2015, private company Daqri acquired ARToolworks. ARToolworks was an early player in augmented reality and had been founded in 2001. It developed an open source AR toolkit and had a variety of early AR patents. Daqri is developing an AR headset mostly for industrial applications.  Google/Tilt Brush – In April 2015, Google acquired Tilt Brush, which enables users to paint in 3-D in virtual reality and then share the images with others. The Tilt Brush web site indicates that it works with the HTC Vive.  GoPro/Kolor – In April 2015, GoPro acquired Kolor, a French virtual reality software company that lets users to combine multiple photographs or videos to create 360- degree panoramas and videos ("spherical content"). GoPro indicated that its spherical videos can be viewed on Google Cardboard and would soon be available for Oculus VR and the Samsung Gear VR.  Facebook(Oculus)/13th Lab – In December 2014, Oculus announced that it had acquired 13th Lab, which developed advanced augmented reality tracking and 3-D construction framework technology. Its PointCloud SDK was a development tool focused on Simultaneous Localization and Mapping (SLAM), image detection, and tracking. The company was based in Sweden.  Google/Quest Visual – In May 2014, Google acquired Quest Visual. Quest Visual developed the Word Lens augmented reality translation application. Word Lens could capture images that contain foreign words (menus, signs, etc.), translate the words, and display the image with the translated words on it.  Facebook/Oculus – In March 2014, Facebook announced that it was acquiring Oculus Rift for more than $2 billion ($400 million in cash, about $1.6 billion in stock, and up to an additional $300 million if certain milestones are met). Oculus had developed an advanced virtual reality head-mounted display headset which was not yet shipping commercially. Oculus initially raised $2.5 million in Kickstarter campaign. Select Virtual Reality/Augmented Reality Private Companies There have been a large number of recently formed virtual and augmented reality companies. Below are just a few examples. In addition to companies that have created AR and VR products, we included some companies that have developed solutions for capturing 3-D video and images for virtual reality applications.
  • 87. Page 87 Magic Leap Magic Leap is high profile but stealth mode private technology company that has developed advanced augmented reality solutions. The company indicates that it has combined the physics of the visual world with how humans perceive things to create a Dynamic Digitized Lightfield Signal. Based on that core technology, the company has developed a variety of solutions, but has not yet released details to the general public. Some preliminary videos show impressive augmented reality scenarios (e.g., an animated robot hiding under a real world chair, a computer generated model of the solar system floating in an office). Magic Leap has provided only limited information about its solution, but has indicated that it actually projects images onto a user’s retina, as opposed to having the user look at a screen. It also indicated that it uses light field technology and that the user can move their eyes around to focus on different objects. In October 2014, it was reported by Reuters that Magic Leap had raised $542 million from investors that included Google, Qualcomm, Kleiner Perkins, Andreessen Horowitz, and Obvious Ventures. In February 2016, the company announced that it completed an additional $793.5 million funding round, led by Alibaba Group, with continued investment from Google and Qualcomm, and a variety of new investors including Warner Brothers and several major financial institutions. Magic Leap is headquartered in Dania Beach, Florida. Wikitude Wikitude is a major supplier of augmented reality software platform solutions. The company’s all-in-one AR solution includes image recognition and tracking, 3-D model rendering, video overlay, location based AR, and other features. It allows customers to easily develop apps and solutions that incorporate augmented reality. Its SDK supports a variety of products (Android, iOS, etc.). The latest version of the Wikitude SDK includes geo-based AR, image recognition and tracking services, extended image tracking, as well as 3-D modeling and presentation layers. The company recently implemented Cloud Recognition for massive image libraries, released the next-generation of its Wikitude SDK (v5), and has developed SLAM 3-D Tracking for tracking real world objects for augmented reality applications. Wikitude’s Cloud Recognition service allows customers to work with thousands of target images hosted in the cloud. Its Studio product enables customers to create their own AR solutions without requiring any programming. Wikitude indicates that its SDK is powering over 10,000 working apps, built by a community of over 50,000 active developers. Wikitude is based in Austria and was founded in 2009. Investors include Gamma Capital Partners and tecnet Capital.
  • 88. Page 88 Marxent Marxent is a virtual/augmented reality technology company focused on visual commerce applications. The company’s VisualCommerce SaaS platform solution powers apps from many major companies such as Lowes, ToysRUs, GE Healthcare, Simmons Bedding, Toyota, American Woodmark, Novartis, AZEK Building Products, Dell, NetApp, and many other major companies. For example, Lowe’s advanced Holoroom kiosk (which allows customers to select a variety of furniture or appliances, choose/modify colors, and then see the room in virtual reality using Oculus and/or Google Cardboard headsets) is based on Marxent’s VisualCommerce technology. Lowe’s recently indicated that it was rolling out the Holoroom to its stores in the U.S. after some initial testing. A variety of companies utilize Marxent’s technology to augment their catalogs to enable augmented/virtual reality experiences. These types of applications can significantly improve sales but require building a 3-D database of products and Marxent has expertise in building 3-D product databases for retail, manufacturing, and e-commerce. Marxent is based in Ohio with an office in Florida. Blippar Blippar is an image recognition and augmented reality solutions company. The Blippar app allows a user’s smartphone (or other mobile device) to recognize, through the smartphone’s camera, products or objects. The act of scanning an item with a device is called "blipping." When a user “blipps” something with the Blippar app, the app determines what the object is and then checks to see if there is content associated with the image. Utilizing augmented reality, the app displays this additional content on the display on top of the actual product image. For example, if you point a smartphone with the Blippar app at a box of Lucky Charms, it recognizes the product and then a 3-D AR game appears on the smartphone. Blippar has partnerships with a broad range of products including Heinz ketchup, Maybelline makeup, Jaguar cars, Perrier water, Star Wars Rebels, Heineken beer, Coke, Pepsi, and many others. The Blippar app is available on iOS, Android, Windows and wearables. In addition to the Blippar app, Blippar provides a variety of other solutions including Blippbuilder (self-service tool), Layar SDK (enabling others to include Blippar AR into their apps), and also offers strategy and customization services. Although private, Blippar has made several acquisitions (Layar, Binocular, etc.). Blippar is based in San Francisco and has several offices around the world. FaceCake FaceCake Marketing Technologies is an augmented reality company focused on retail applications that allow users to “try on” clothing or makeup without actually physically putting the products on. FaceCake’s Virtual Try-On solutions (for apparel, cosmetics, accessories and more) enables consumers to compare looks, get style advice, and make informed choices, in real time. It also can provide users with a variety of product recommendations based on the images. FaceCake’s solutions can work in-store or at- home on many types of devices (in-store kiosks, PCs, tablets, mobile phones).
  • 89. Page 89 FaceCake’s offerings include Swivel (which allows users to “virtually” try on a variety of different types of clothing and accessories), Swivel Close-Up (which enables users to try on different types of cosmetics and makeup colors), Swivel Smart Design Signage (augmented reality advertising for stores), and ShadeScout (a smartphone app that enables users to find makeup that matches a specific color they find). The privately held company is based in Calabasas, CA. uSens uSens has developed wireless headsets that support both virtual and augmented reality. The uSens Impression Pi headsets incorporate its internally developed gesture recognition, head tracking, and position tracking technologies. The company’s “Super Reality” technology enables a transition between virtual and augmented reality. The device includes an IR tracker and a sensor unit and works with a variety of smartphones that snap into the headset. In addition to gaming, the company believes its headset could be utilized in education, healthcare, entertainment, and training. In 2015, the company announced that it completed a $5.5 million pre-Series A round co-led by IDG Ventures, Lebox Capital, and Maison Capital, with Stone Capital, Fulcrum Capital, ChuanCheng Fund, and Chalor Capital also participating. ModiFace ModifFace has developed a variety of advanced facial visualization augmented reality solutions. The ModiFace Mirror is a photo-realistic real-time makeup and skin simulation that enables users to see what they would look like with makeup or other changes (e.g., a woman could “try on” many different types and colors of lipstick, eye shadow and other cosmetic products without having to actually physically put anything on her face). The technology can make recommendations based on the person’s face and can also simulate a variety of other changes (teeth whitening, anti-aging treatments, etc.). ModiFace recently introduced an HD version with a variety of advanced features. In addition, ModiFace provides a solution that enables users to upload a photo of their face to a Website and then modify how their face looks based on selecting various types of makeup. ModiFace also provides a broad range of apps that allow the user to see what they would look like with different hair color, hair styles (including celebrity hair styles), eye color (colored contacts), anti-aging treatments, and a variety of other changes. It can also help photo edit to implement a variety of changes. ModiFace’s technologies make it significantly easier for consumers to decide which product best suits them without having to try things out, and they can do it from home. Its technology is used by over 150 web and mobile apps, which have been downloaded nearly 60 million times. Some of the companies that have introduced ModiFace-based augmented reality apps include Allergan, Amore Pacific, L'Oreal, Yves Rocher, Jane Iredale, Matrix, Unilever, and Honest Beauty. ModiFace was founded in 2006.
  • 90. Page 90 Jaunt Jaunt provides an end-to-end solution for creating cinematic virtual reality experiences. The company has developed a suite of hardware and software tools to enable the creation of high quality immersive content that can be viewed on a variety of virtual reality platforms. Its professional grade stereographic NEO camera was specifically designed to create immersive video and includes 360 degree capture, 3-D sound-field microphones, and advanced optics. The company’s advanced rendering software and photography algorithms transform the recorded video data into an immersive viewing experience. The content can be viewed on a variety of VR headsets (Google Cardboard, Samsung Gear, Oculus Rift, etc.). Jaunt has created and distributed a broad range of VR content and has announced a long list of partners it is working with on virtual reality content. In September 2015, it announced a $65 million capital raise that included investors such as Walt Disney Company; Evolution Media Partners (CAA, TPG Growth, Participant Media), China Media Capital, Madison Square Garden Company, Google Ventures, Highland Capital, Redpoint Ventures, Sky and SV Angel. Matterport Matterport has developed an advanced professional camera that enables users to capture accurate 3-D representations of rooms and houses. The camera incorporates a variety of 2-D and 3-D sensors and can scan a room (spinning around) to capture a panoramic view. In addition, Matterport’s software can piece together the various room images to create a complete 3-D model of a house. So, for example, a real estate agent could use the Matterport camera and software and easily develop a comprehensive 3-D image of a house, enabling potential buyers to do a virtual “walkthrough” of the house (moving from room to room, looking in any direction) before visiting. Matterport provides a complete end- to-end 3-D media platform, enabling users to upload the data from the cameras to the cloud, after which Matterport processes the data to create and host immersive 3-D models, which can then be viewed anywhere with a browser. In addition to real estate, other initial applications include hospitality, insurance, education, and retailing (e.g., capture images of a home and try out different furniture, etc.). The company is also working with Google's Project Tango and Intel's RealSense and hopes to eventually bring its technology into a broad range of consumer mobile applications, especially as 3-D imaging becomes more commonplace in mobile devices. Matterport completed a $30 million Series C round in 2015 which was led by Qualcomm Ventures. Other investors include GIC (Singapore’s sovereign wealth fund), Lux Capital, DCM Ventures, Felicis Ventures, Greylock, Navitas Capital, AMD Ventures, AME Cloud Ventures, iGlobe Partners, Rothenberg Ventures, Blake Krikorian (Sling Media founder), and Gordon Segal (Crate & Barrel founder). It is based in Silicon Valley.
  • 91. Page 91 VideoStitch VideoStitch has developed technology that enables 360 degree video to be captured and then viewed, including viewing using virtual reality headsets. That is, by using multiple video cameras with overlapping views, a complete 360 degree view of an area can be captured. VideoStitch’s advanced software then rapidly stitches together the individual videos into a single, high resolution, seamless panoramic video, that can be streamed to the cloud or elsewhere. The software provides a regular video file or stream that can be watched, but it enables the user to modify the video view based on which direction the user wants to see. The technology can also be used with virtual reality headsets in which the user can turn to see different views just as in real life. Its products include Studio (360 video post-production) and Vahana VR (for virtual reality applications using Oculus Rift). VideoStitch also offers GoPro-based multi-camera rigs to help customers more easily capture video. VideoStitch indicates that it has over 1000 customers in 45 countries including entertainment, media and Fortune 500 companies (e.g., Facebook, Red Bull, Sony Entertainment). At CES 2016, VideoStitch and Intel announced that they are collaborating on high quality VR Video live streaming. The companies showcased 4K Live 360 video streaming using Intel hardware. The Live stream was taken from 6 action sports cameras that were stitched together live on a notebook PC. VideoStitch also announced that it is developing live stitching software that incorporates Intel’s Quick Sync technology, with an expected release mid-2016. VideoStitch is headquartered in Paris with a major office in San Francisco. Holographic Imaging There are a variety of technologies that provide advanced 3-D imaging without the aid of helmets or special eyewear. Magic Leap’s solution can apparently provide 3-D images without the aid of any type of glasses. Although Microsoft refers to the 3-D images generated by HoloLens as holograms, a true holographic image can be seen without the aid of specialized eyewear or headsets (and the viewer can walk around and see the image from different angles as if it were a real object). A traditional hologram records information about the light that comes from an object in all directions (rather than in just one direction like a photo). Typically two lasers are used – one is directed at the image itself (object beam) and the reflected light hits the recording medium, the other (reference beam) is directed at the recording medium. The interference pattern between the two lasers is recorded on the medium. The physical patterns on the recording medium appear to be random marks but when it is lit up in an appropriate manner, the light is diffracted in such a way that it creates a floating holographic image. The “father” of holography, Dennis Gabor, won the Nobel Prize in Physics in 1971.
  • 92. Page 92 As noted, holograms were historically created by capturing the reflected light from actual physical objects. However, there has been growing interest in digital holograms, in which holograms are created from 3-D computer images. There are currently several companies that can transform digital images to holograms and some even sell “holoprinters” which enable users to create their own holograms. Specifically, the holoprinter analyzes a digital image and prints out holographic film which when lit up produces a holographic image. While the ability to view a static holographic image has many applications (e.g., an architect can show a 3-D image of what a building complex might look like before it is built, a doctor can view a 3-D image of a patient’s organs, etc.), the longer term goal is to create solutions that enable computers, mobile devices and gaming systems to project real-time dynamic holographic images (i.e., holographic 3-D images of anything that would normally be displayed on a flat 2-D screen). The classic example is the Princess Leia hologram projected by R2-D2 in the original Star Wars movie (“Help me, Obi-Wan Kenobi!”). Although this level of sophistication is still years away, there are a number of companies working towards developing advanced dynamic holographic solutions. This requires implementing some type of technology to rapidly change the way light is diffracted (simulate the holographic film interference patterns, but change them in real-time to make the images dynamic). A variety of approaches have been used (digital light mirrors, addressable liquid crystal, advanced light guides in the backlight, etc.) to implement the required patterns. However, there are still numerous engineering challenges as the amount of data and computations required for holograms is extremely large, the optics are complex, the images need to rapidly change, and relevant costs have to be driven down. For many of the initial prototypes, image quality is mediocre, size is small and there are other issues, but the technology is expected to greatly improve over the next few years. Another application for holography is telepresence. Telepresence includes a number of technologies that simulate a person being present or allowing a person to feel as if he is present at a location other than the one he is at. This goes beyond simple video conferencing and may include, for example, creating holographic/3-D images of the people on the other end of the line. While there have been some examples of commercial holographic image deployments and several announced “breakthroughs,” the technology is still early stage with respect to consumer applications. However, over the next several years we anticipate that there will be significant interest in holograms as users would prefer seeing and interacting with 3-D images without having to wear headsets or special glasses. Longer term, it could become a critical mainstream technology for a variety of products.
  • 93. Page 93 Select Holography Companies A few examples of companies focused on holographic technology include: Zebra Imaging Zebra Imaging is a digital hologram company. Customers can upload computer images to the company and it converts the images into color or monochrome hologram prints that the customer can then use to create holographic images. Its holograms can be viewed using simple halogen or LED light sources. The company also sells its own line of holographic printers, enabling customers to create their own holographic prints and images. Zebra’s solutions are used across a variety of industries (construction, architecture, retail, etc.). Zebra recently announced the availability of a 3-D Hologram Creator plugin for Autodesk Revit software, enabling Revit users to quickly create designs that can be printed as 3-D holographic prints. Zebra also announced it signed a contract with the U.S. Department of Homeland Security to deliver high-resolution 3-D maps to Customs and Border Protection. Importantly, Zebra is currently focused on optimizing its ZScape holographic motion display solution, which is a self-contained system that links to a computer workstation via an Ethernet connection, and enables images displayed on a computer monitor to appear as 3- D holograms. It was initially developed with help from DARPA, but the technology has the potential for a broad range of markets, including high volume mainstream consumer applications. In November 2015, Zebra announced that it acquired Rattan Software which provided expertise to a variety of customers (Samsung, Virtuix Omni, etc.) on utilizing light field technology. Zebra is based in Austin, Texas. Holoxica Holoxica provides digital hologram prints to customers (i.e., customers provide digital image files and receive hologram prints that can produce holographic images when light is applied). It addresses a variety of markets including medical (holograms using data from CT and MRI scans), engineering design (holograms from CAD and other graphic models), and construction/architecture (holograms to show designs before physical construction). More significantly, the company is also developing advanced real-time interactive holographic displays. Its first generation solution enables simple dynamic holograms such as numbers, which can be used to create, for example, a holographic digital clock. Its second generation solution provides a variety of more complex images (but still flattish – similar to a heads-up display on a car windshield) and also enables the user to draw lines and figures in mid-air. Importantly, Holoxica is currently working on its third generation solution which would provide full 3-D volumetric holographic images and could be used for a variety of real commercial applications. The company is developing its solution such that the technology could be relatively easily commercialized within the current LCD supply chain.
  • 94. Page 94 While Holoxica will initially focus on certain niche target markets, the company believes that longer term its holographic displays will have many high volume consumer applications. Holoxica is located at the Scottish Microelectronics Centre at Edinburgh University. RealView Imaging RealView Imaging is an Israeli holographic imaging company, with an initial focus on medical applications. Their system is able to transform 3-D digital data from medical imaging equipment and project 3-D holographic still images or video streams that float in mid-air. The projected image is in high resolution color and can be viewed from a variety of angles. The company has also developed a number of technologies to enable interaction with the images. Its “image intimacy” technology allows users to, for example, mark up the holographic images, modify them, or add other digital images to them. The company is working on a variety of other medical applications (fetal imaging, etc.). Nvidia has indicated that the RealView technology uses a cluster of Nvidia Tesla GPUs and Quadro cards. RealView completed a clinical study with Philips demonstrating the feasibility of using live 3- D holographic visualization to guide minimally-invasive heart disease procedures. While RealView has primarily focused on medical, longer term it believes its technology has a variety of applications in other markets including consumer electronics. LEIA 3D LEIA 3D has developed holographic technology that enables 3-D without eyewear. The company replaces the light guide that is standard in the backlight of virtually all LCD displays with a more complex light guide with nanoscale gratings. This diffractive backlight allows the projection of different images in various directions of space. This enables content that looks 3-D from any viewpoint and that can be updated at video rates. The company sells a development kit (including a holographic display module) for $999. Although the development kit is monochrome, the company indicates it already has color technology available. LEIA has also developed a technology called Hover Touch, which can sense the presence of a finger at some distance from the screen, enabling interaction with 3-D images. While it provides a development kit, it is primarily focused on working with OEMs interested in integrating its technology. The company indicates that its technology is designed such that it can be implemented with relatively little incremental cost relative to current displays and is targeting a variety of high volume mainstream applications. The company was founded by David Fattal, who invented the technology at HP Labs. LEIA is headquartered in Silicon Valley.
  • 95. Page 95 Kino-Mo While true immersive hologram technology is still many years away, Kino-Mo has developed a solution that creates what appear to be holographic images. Specifically, it has developed a device with a rotating four arm propeller covered in LED lights and sensors. As the propeller rotates, microprocessors with advanced software algorithms track the LEDs and rapidly turn the LEDs on and off in such a way that it creates images that appear as true 3-D holographic images (the rotating blades themselves are not visible). At CES 2016, Kino-Mo displayed a broad range of complex 3-D images that appeared to be true (futuristic movie-like) holograms floating in mid-air and viewable from a variety of different angles (as long as you were in front of it). The company is initially targeting the technology for advertising but it could potentially have a variety of uses, including consumer applications. Kino-Mo’s initial product was a video bike in which advertisers can transfer images to the bike and the images are displayed on the bike’s wheels as the cyclist rides the bike (customers have included Intel, Samsung, GE, and BNP Paribas). The company is based in London.
  • 96. Page 96 “And in the end….the love you take is equal to the love you make.” - Paul McCartney The technology landscape can change very quickly. A decade ago (when the leading mobile phone OEMs included Nokia, Motorola, and Sony Ericsson), who would have predicted that the handset market would soon be dominated by companies such as Apple, Samsung, Huawei, Xiaomi, LG, and Lenovo (with Google providing a significant portion of the operating systems)? Over the next decade, it is certainly possible that we will see continued market share leadership changes, and this may be based on new products incorporating innovative human interface technologies that greatly improve the user’s experience. There is growing interest in HMI technologies from a broad range of different types of companies (e.g., smartphone OEMs, consumer electronics companies, software companies, chipmakers, Web companies, social media companies, etc.). It is interesting to note that some of the companies listed in this report that have made user interface related acquisitions and/or investments include Apple, Google, Yahoo, Intel, Qualcomm, Microsoft, Amazon, Samsung, and Facebook. With HMI technologies becoming increasingly important, a variety of start-ups focused on innovative user interface solutions, and strong interest from many of the largest technology companies, there may continue to be a robust M&A market in this sector over the next several years. The technologies discussed in this report only scratch the surface of the types of human interface technologies that are being developed. It will be very interesting to see the number of new and innovative technologies that are created over the next decade! We hope this report has been useful. We tried to highlight a variety of technologies and companies, but obviously had to limit the number of topics covered, the amount of space devoted to each topic, and the number of companies covered. Our hope is to periodically issue future editions with updated data and additional topics. As a result, if there are any comments or recommendations, let us know. We will attempt to address any comments we receive in future editions. Finally, if you have, or know anyone that has, M&A interest (sell side or buy side) in this sector, or any area within technology for that matter, please give us a call. We look forward to speaking with you! Chapter 11 – Some Final Thoughts
  • 97. Page 97 “I am a brain, Watson. The rest of me is a mere appendix.” - Sherlock Holmes The following table is a summary of many of the M&A transactions noted throughout this report. In many cases, the transaction itself and/or the valuation of the deal were not publically announced by the buyers or seller, and we relied on third party data sources for this information when available (which may not necessarily be completely accurate). In some cases, the Enterprise Value listed does not include earn-outs. Pagemill was an advisor on a number of the M&A transactions listed below, but we include data only if the data is publicly available. Sources: Capital IQ, 451 Group, company reports, and media reports. Date Acquirer Target Target Description Enterprise  Value ($M) Jan‐2016 Apple FlyBy Media Vision processing software for smartphones.     Jan‐2016 Apple Emotient Artifical intelligence technology to analyze user faces to determine  emotional state.   Dec‐2015 Atheer OntheGO  Platforms Gesture interfaces using standard cameras in mobile devices.   Nov‐2015 AMS CMOSIS CMOS image sensors. $235 Nov‐2015 Zebra Imaging Rattan Software Expertise in light field technology for a variety of applications,  including holographic imaging.   Nov‐2015 Apple Faceshift Facial analysis software that can enable an avatar to have the  same expressions as the user.   Oct‐2015 Sony SoftKinetic  Systems Gesture recognition solutions including ToF CMOS image sensors,  camera modules, middleware, and cameras.   Oct‐2015 Apple Vocal IQ Natural language self‐learning technology which can learn from  interactions with people over time.   Oct‐2015 Intel Saffron Cognitive computing platforms for analytics and decision making,  and potentially for consumer interaction. Oct‐2015 Sony Toshiba Image  Sensor unit Toshiba's image sensor unit and manufacturing facilities. $155 Oct‐2015 PTC Qualcomm  Vuforia unit Major provider of augmented reality technology solutions. $65 Oct‐2015 Blippar Binocular Augmented reality applications for customers.   Sep‐2015 Freescale CogniVue Vision processor chips and IP.     Appendix: M&A Transaction Summary
  • 98. Page 98 Sources: Capital IQ, 451 Group, company reports, and media reports. Date Acquirer Target Target Description Enterprise  Value ($M) Sep‐2015 Snapchat Looksery Augmented reality video "lenses" that enable users to change their  appearance during video calls. $150 Aug‐2015 Voxx EyeLock Iris recognition technology including more than 70 patents. $20 Jul‐2015 Facebook Pebbles Interface Optics, sensor systems, and algorithms to detect and track hand  movement. $60 Jul‐2015 Cirrus Logic AGNITO's KIVOX  unit KIVOX voice authentication technologies for mobile devices for on  device voice matching.   Jun‐2015 Parade Cypress Mobile  Touch Unit Cypress' touch controller business unit for mobile devices  (Cypress continues to supply auto and appliances). $100 May‐2015 Facebook Surreal Vision Real‐time 3D scene reconstruction (accurate representation of the  real world in the virtual world). May‐2015 Apple Metaio Major supplier of augmented reality solutions, including an AR  publishing tool, an SDK, and other AR solutions.   May‐2015 Daqri ARToolworks Open source AR toolkit and a variety of AR patents.   Apr‐2015 Microsoft N‐Trig Stylus used for writing on tablets, including the Surface Pro 3. $30 Apr‐2015 Hua Capital  and others OmniVision Major supplier of CMOS image sensors. $1,900 Apr‐2015 Apple LinX Small camera modules for smartphones. $20 Apr‐2015 Google Tilt Brush Enables users to paint in 3D in virtual reality environments.   Apr‐2015 GoPro Kolor Virtual reality software that enables users to combine multiple  images or videos to create 360 degree videos. Mar‐2015 Intel Lemoptix MEMS scanning based pico‐projection solutions for handsets,  auto HUDs, gesture recognition, and wearables.   Jan‐2015 Facebook Wit.ai API for developing voice activated interfaces making it easier to  implement speech recognition/voice activation.    Dec‐2014 Facebook Nimble VR 3D camera for skeletal hand tracking for virtual reality type  applications.   Dec‐2014 Apple Privaris Patents A variety of patents related to fingerprint sensing technology.   Dec‐2014 Tessera Smart Sensors Iris recognition technology.   Dec‐2014 Facebook 13th Lab Augmented reality tracking and 3D construction framework  technology for SLAM and image detection.   Oct‐2014 Google Dark Blue Labs Deep learning technologies for understanding natural language  and what users are asking for.  
  • 99. Page 99 Sources: Capital IQ, 451 Group, company reports, and media reports. Date Acquirer Target Target Description Enterprise  Value ($M) Oct‐2014 InvenSense Analog Devices'  MEMS Mic unit Analog Devices' MEMS microphone business unit. $100 Oct‐2014 Google Vision Factory Artifical intelligence technology to enhance the speed of object  recognition and improve vision based systems.     Sep‐2014 Dropbox KBVT Facial recognition software.   Sep‐2014 Qualcomm Euvision Artificial intelligence for improving image recognition including  an app that could analyze photos on a smartphone.   Sep‐2014 Twitter Madbits Deep learning technologies to understand the content of an image.   Aug‐2014 Google Jetpac Technology to analyze images and video and extract contextual  data.   Jul‐2014 InvenSense Movea Sensor fusion middleware solutions including technologies for  gesture control and activity monitoring. $61 Jun‐2014 Synaptics Renesas SP  Drivers Display drivers for small and medium size displays. $515 Jun‐2014 Parker Hannifin Bayer/Artifical  Muscle Unit Bayer MaterialScience's Artificial Muscle haptics business  (electroactive polymer based).    Jun‐2014 Audience Sensor Platforms Sensor fusion solutions for mobile devices.   Software to combine  and process sensor data. $41 Jun‐2014 ON Semi Aptina Major supplier of CMOS image sensors. $400 Jun‐2014 Blippar Layar Augmented reality platform.   May‐2014 Google Quest Visual Augmented reality solution that can capture images of words and  translate them and then display the translated words.   Apr‐2014 FocalTech Orise Merger of FocalTech (touch controllers) and Orise (display  drivers). $288 Apr‐2014 Cirrus Logic Wolfson Primarily audio analog chips but included a small MEMS  microphone unit. $467 Apr‐2014 ON Semi Truesense  Imaging Image sensors primarily for industrial markets. $92 Mar‐2014 Facebook Oculus Major supplier of virtual reality headset (Rift) and related  technologies. $2,000 Jan‐2014 Google DeepMind  Technologies General‐purpose, artificial learning, algorithm‐based simulation  software. $400 Jan‐2014 Qualcomm Kooaba  Mobile image recognition software application for software  developers and businesses to integrate into mobile applications. Jan‐2014 Fairchild Xsens 3D motion tracking using MEMS inertial sensing solutions. $58
  • 100. Page 100 Sources: Capital IQ, 451 Group, company reports, and media reports. Date Acquirer Target Target Description Enterprise  Value ($M) Jan‐2014 Apple Novauris Automatic speech recognition technology that could process  several simultaneous voice access requests.   Jan‐2014 Pinterest VisualGraph Image recognition and visual search.   Dec‐2013 Google Industrial  Perception 3D vision systems for robotics and other applications.   Dec‐2013 Yahoo SkyPhrase Speech recognition software that enables users to more easily  communicate with computers by using natural language. Dec‐2013 TA Associates CMOSIS CMOS image sensors.   Nov‐2013 Apple PrimeSense 3D gesture recognition/machine vision technologies for digital  devices; provides SoC solutions and software. $345 Oct‐2013 Verint Systems Victrio Biometric voice recognition and authentication SaaS that enables  secure access to accounts and passive background monitoring. Oct‐2013 Yahoo LookFlow Image recognition and machine learning technology.   Oct‐2013 Synaptics Validity Sensors Fingerprint biometric sensors for use in identity authentication. $93 Oct‐2013 Google Flutter Gesture recognition software application that enables PC users to  control a range of applications via a built‐in webcam. $40 Sep‐2013 IDEX PicoField  Technologies Biometric consumer device fingerprint scanning and sensor  systems for mobile device manufacturers. Sep‐2013 Intel Indisys Natural language recognition software and Web and mobile  artificial intelligence interactive customer service software. $26 Aug‐2013 Facebook Jibbigo (Mobile  Technologies) Speech‐to‐speech mobile device based translation software that  translates spoken words for use with iOS and Android devices. Aug‐2013 Yahoo IQ Engines Image recognition technology that can determine which objects  are in a scene for photo tagging and other applications. Jul‐2013 Google SR Tech Group,  LLC ‐ Patents Portfolio of speech‐related patents including "Speech interface for  search engines".   Jul‐2013 Intel Omek Interactive Gesture recognition software and hardware for use in consumer  electronics, auto, gaming and healthcare. $50 Apr‐2013 Google wavii Smart phone news aggregation and reader application that  delivers customized news feeds . $30 Apr‐2013 Amazon Evi Technologies  Mobile speech‐recognition application that enables mobile device  users to receive answers to spoken questions and instructions. $26 Jan‐2013 Amazon Ivona Multi‐lingual speech synthesis systems. Jan‐2013 GEO  Semiconductor Maxim's  Compress. Unit Video compression technologies business unit.
  • 101. Page 101 Sources: Capital IQ, 451 Group, company reports, and media reports. Date Acquirer Target Target Description Enterprise  Value ($M) Jan‐2013 TPK MasTouch  Optoelectronics Projected capacitive mode touch‐and‐control panel boards. Jan‐2013 Nuance VirtuOz Web‐based virtual assistant customer service software. Nov‐2012 Alcon SMI's Opthalmic  Unit Eye tracking solutions for opthalmic use (ocular surgery  guidance).   Oct‐2012 Google Viewdle Facial recognition software for mobile devices.  Also developing  image recogntion solutions (smiles, gestures, etc.). $40 Sep‐2012 Dropbox Anchovi Labs Computer vision and articifial intelligence technologies for  classifying objects.   Sep‐2012 Nuance Ditech Networks Voice‐to‐text conversion and voice quality assurance software. $23 Sep‐2012 Qualcomm EPOS  Development Embedded semiconductor software that uses digital positioning  technology to control user input movements. Aug‐2012 STMicro‐ electronics bTendo Embedded MEMS for use in video‐projection applications for  mobile device and digital camera manufacturers. Jul‐2012 Apple AuthenTec Biometric fingerprint authentication sensor systems and software  for computer and electronic device manufacturers. $356 Jul‐2012 Microsoft Perceptive Pixel Multi‐touch display technologies and large screen displays.   Jun‐2012 Facebook Face.Com Facial recogntion (primarily for recognition of faces in uploaded  photos). $60 Apr‐2012 Gores Group TE's Touch Screen  Unit Touch screen solutions business (sensor‐based touch screen  displays and computers sold under EloTouch brand). $380 Mar‐2012 Facebook SiteHawk Eye tracking solutions using conventional Webcams (acquisition  of the team).   Feb‐2012 VoiceTrust Perceive  Solutions Biometric speech‐recognition software for the purpose of  authenticating users. Dec‐2011 Nuance Vlingo Mobile speech‐recognition voice command application (Nuance  paid $196.3M, but already had an equity stake in Vlingo). $225 Nov‐2011 Amazon Yap White‐label voice‐to‐speech recognition SaaS. Aug‐2011 Nuance Loquendo Multi‐language recognition and text‐to‐speech conversion  software. $75 Jul‐2011 Qualcomm GestureTek Camera‐enabled gesture recognition software. Jul‐2011 Google PittPatt Facial recognition software that identifies human faces in  photographs and tracks the motion of human faces in video. $38 Jun‐2011 Nuance SVOX Desktop and mobile voice automation and authentication, speech  recognition and text‐to‐speech conversion software. $123
  • 102. Page 102 Sources: Capital IQ, 451 Group, company reports, and media reports. Date Acquirer Target Target Description Enterprise  Value ($M) May‐2011 TPK Cando  Multi‐touch capacitive touch panel products (TPK acquired only  about 20% of the equity for about $190 million). $955 Feb‐2011 TandemLaunch  Technologies Mirametrix Gaze tracking systems for use in vision research and content  analytics applications. Jan‐2011 Google SayNow Voice/telephony related solutions including conference‐call type  voice meetings. Jan‐2011 Nuance PerSay Biometric voice authentication software. Dec‐2010 Google Phonetic Arts Speech synthesis (speech to text) technologies and solutions. Oct‐2010 Microsoft Canesta 3D image sensor semiconductors and software for consumer  electronics and videogames and auto. Sep‐2010 Apple Polar Rose Facial recognition for photo sharing sites. $29 May‐2010 Tessera Siimpel MEMS auto‐focus chips for smartphone cameras. $15 Apr‐2010 Apple Siri Mobile application that enables iPhone and iPod Touch users to  ask questions and receive answers and recommendations. $200 Mar‐2010 Bayer Material  Science Artificial Muscle User touch‐based interface devices and haptic actuators for use in  consumer electronics and medical devices. Jan‐2010 AuthenTec UPEK Biometric security and access control systems for computer and  electronics manufacturers. $32 Aug‐2009 Bosch Akustica Major supplier of MEMS microphones.   Mar‐2009 Microsoft 3DV Motion‐detection cameras used in gaming and web‐conferencing  by both consumers and enterprises. $35 Aug‐2005 Dover Knowles Major supplier of MEMS microphones. $750
  • 104. Page 104 Pagemill Partners Pagemill Partners A Division of Duff & Phelps Securities, LLC 1950 University Circle, Suite 400, East Palo Alto, CA 94303 Phone: +1-650-354-4000 Fax: +1-650-319-8791 This document is published solely for informational purposes, and is not to be construed as legal advice or tax advice. In addition, nothing contained herein is to be construed as an offer or recommendation, or the solicitation of an offer, to sell or buy a specific security, nor is any of the information contained herein intended to constitute an analysis of any company or security reasonably sufficient to form the basis for any investment decision. You are strongly encouraged to seek advice from qualified legal counsel for specific questions you may have regarding M&A transactions. The information herein is based in part on third party sources that we believe to be reliable, but have not been verified by us to be accurate. As such, we do not guarantee that the information is accurate or complete. Pagemill Partners is engaged in the business of providing M&A, private placement, and valuation advisory services to technology companies. Pagemill Partners may have provided in the past, may currently be providing, or may in the future provide advisory services to one or more of the companies mentioned in this document. The contents of this document are under copyright to Pagemill Partners. The material should not be copied nor distributed to other parties without the prior express written permission of Pagemill Partners. The Pagemill Partners logo is a trademark of Pagemill Partners. Copyright 2016 Pagemill Partners, a Division of Duff & Phelps Securities, LLC. All rights reserved. Member FINRA, SIPC.