The Untold Story Of Microsoft’s Surface Hub – part 1

A man with a dream. A company in flux. A secret factory outside Portland. And a hyper-ambitious gambit to reimagine how meetings happen.

By Harry McCracken re-posted from Fast Company

Jeff Han’s fingers are dancing across an expansive, wall-mounted touch screen. Planet Earth spins in front of him in computer-generated form; he grabs it with both hands and starts to zoom in. He keeps going—Western Hemisphere, North America, United States, Pacific Northwest—until we’re finally staring at a prosaic industrial area alongside a highway.

“Here’s our building, right here,” he declares.

The conference room where this demo is going on is named after Bill Gates, so it shouldn’t come as a complete shock that we’re on Microsoft premises. But this isn’t corporate headquarters in Redmond, Washington. Han and I are 200 miles away, across the Oregon border in the Portland suburb of Wilsonville, inside a 4-acre structure flanked by a manufacturer of industrial shredders and a storage facility for boats and RVs.

Microsoft hasn’t played up the fact that it has a major operation in Wilsonville. Actually, it’s been downright stealthy about it. (The roadside signage directing visitors to the main entrance doesn’t even mention a company name.) But since March 2014, the building is where the company has been engineering the device Han has been showing me, the Surface Hub.

The company unveiled the device during its Windows 10 launch extravaganza in Redmond last January. It showed how the 84-inch all-in-one computer can be used for tasks such as brainstorming, videoconferencing, and PowerPoint presentations, all conducted via touch and pen input rather than mouse and keyboard.

Microsoft held back one of the most intriguing facts about this Windows 10 machine until now: It not only designed the Surface Hub but is about to begin manufacturing the thing itself, in 70,000 square feet of factory space in the Wilsonville building, steps away from where the hardware was engineered. “We don’t actually manufacture the LCD panel,” Han explains. “But that’s pretty much the only thing we don’t make here.” Two models, a $19,999 unit with an 84-inch display and a $6999 version with a 55-inch display, will roll off the assembly line.

Microsoft has done such a crafty job of keeping its Wilsonville plans on the QT that a September 2014 Oregonian article on the facility simply assumed that it was all but a given that the company would go offshore when it came time to build anything. “[M]ass production would almost certainly take place in a country with lower labor costs,” the story helpfully explained. “It’s that way with nearly all high-volume electronics manufacturing.”

For the 39-year-old Han, the impending release of the Surface Hub is the fullest expression yet of a mission he’s been pursuing for more than a decade. The world first took note of it in 2006, when he was a computer-science researcher giving a TED talk in Monterey, California, about an intuitive new computing interface called multi-touch. He was an exuberant, charismatic presenter—and still is, as I learned when he showed off the Surface Hub to me in Wilsonville.

In the pre-iPhone era of 2006, what he showed was mind-bending; the video version of his presentation became one of the first TED talks to go viral, and Han’s sheer enthusiasm made multi-touch feel like it might indeed be the next big thing. He then parlayed his fame into a startup, Perceptive Pixel (PPI), which sold pricey screens to everyone from the Department of Defense to Disney. Microsoft acquired the company in 2012.

When? How Much?

Microsoft will begin taking Surface Hub orders on July 1—globally, in 24 markets—and expects the first units to reach customers in September. The 84-inch model, with a 4K screen and an Intel Core i7 processor, will have a suggested price of $19,999. It’s designed for conference rooms that accommodate seven to 10 people. A 55-inch version with a 1080p screen and Core i5 chip, meant for smaller “huddle spaces,” will go for $6999. Those prices are not out of whack with the investments that large companies make in meeting-space technology, especially since the Surface Hub does videoconferencing as well as collaboration. “You can compare it to Cisco telepresence or HP telepresence, which is more expensive,” says J.P. Gownder, an analyst at Forrester. “It’s fairly inexpensive, because you’re solving that problem on some level, as well as the collaboration piece.”

Today, Han is general manager of Surface Hub. Without his research, vision, and persistence, the device might not exist. And yet at the same time, it’s also a profoundly Microsoftian product. PPI’s products were displays you plugged into a PC; the Surface Hub is a computer, running big-screen editions of Windows 10 and Office. It doubles as a videoconferencing system equipped with two high-resolution cameras and Microsoft’s Skype for Business service. There are even signs of Xbox influence, such as the noise-canceling four-element microphone, which originated as part of the game console’s Kinect sensor.

Unlike the other hardware device Microsoft unveiled at the January Windows 10 event—the mixed-reality HoloLens headset—the Surface Hub doesn’t have a whiff of science fiction about it. It’s about polish and technical sophistication more than raw futurism, and in a way, it feels like Microsoft tackling a bit of unfinished business.
“We don’t do things small at Microsoft. We do things big.”

“I’ve been working on product development around productivity forever,” says Mike Angiulo, who, as Microsoft’s corporate VP for hardware, is Han’s boss. (“Forever,” in this case, means since 1993.) “If I think about what’s changed since I’ve been there in terms of individuals doing work, your ability to create and communicate has gone up exponentially, with PCs and mobility and phones. But the conference rooms are exactly the same as they were when I started. They’re like time capsules. There’s a projector, there’s a whiteboard, there’s a conference phone.”

With the Surface Hub’s collaborative tools, Microsoft is making a hyper-ambitious attempt to yank those time capsules into the present day. Which is not anything like a guarantee that it will be successful. In 2002, for instance, the company launched Tablet PCs and said it expected them to displace conventional laptops within a half decade; they didn’t. And its first foray into multi-touch computing—2007’s original Surface, which built a computer into a table for use in retail environments—didn’t go anywhere.

Still, no matter how the market responds to the Surface Hub, the fact that Microsoft is behind it raises the stakes far beyond anything Han has attempted in the past. “This isn’t a trivial little experiment,” he says. “We’re committed.” He pauses. “We don’t do things small at Microsoft. We do things big.”

Han first noodled around with multi-touch interfaces—a concept dating back to the 1970s—in 2002, as a researcher at New York University. Then he put the project aside in favor of other experiments. (Though his name is practically synonymous with multi-touch, he’s also explored everything from autonomous navigation to eye tracking to motion capture.)

In 2005, he turned his attention back to multi-touch, and stuck with it. By February 2006, he was ready to show his work in progress at the TED conference, using a 36-inch screen set up like a drafting table, with cameras that tracked his finger movements. Viewed today, his demo remains engaging. But for a 2015 viewer, the most striking thing about it is the sound of attendees gasping and applauding at now-familiar gestures such as Han pinching a photo to resize it.

Han began his presentation by explaining that the interface he was about to show was ready to come out of the lab. He wasn’t exaggerating. Perceptive Pixel, he told me, “was founded shortly after the response I got from TED.” The startup’s goal was to turn the technology into useful commercial products.

However, the next meaningful moment in the popularization of multi-touch happened on January 9, 2007, when Steve Jobs announced the first iPhone at Macworld Expo in San Francisco. “We have invented a new technology called multi-touch,” he boasted, in one of the more reality-distorting things ever said during an Apple keynote. Han, by contrast, had been careful to share credit in his TED talk: “I’m not the only one doing it, there are a lot of people doing it.”

Among the other organizations that had been working on multi-touch was Microsoft. Four months after the iPhone reveal, it announced Surface, a Windows Vista computer built into a table with a 30-inch screen. The interface bore decided similarities to Han’s efforts. But instead of touting the table as a tool for workplace productivity, Microsoft said it was going to roll it out in public places such as hotels, casinos, and phone stores.
“I wanted to focus on upright productivity. Ever since TED, that’s what it was about for me.”

A press release quoted Microsoft CEO Steve Ballmer: “We see this as a multibillion dollar category, and we envision a time when surface computing technologies will be pervasive, from tabletops and counters to the hallway mirror. Surface is the first step in realizing that vision.” But the billions never rolled in. Microsoft gave the table a vote of no confidence in 2012 by stripping it of its name, redubbing it PixelSense and turning the Surface moniker over to a line of Windows 8 tablets. Samsung continues to sell a PixelSense-based table computer, but if you’ve never seen one in the wild, you’ve got plenty of company.

While the Surface table was failing to catch on, the iPhone was putting multi-touch into the hands of people around the world, transforming the phone business forever. Today, Han says that neither product led him to question whether he was on the right track. “I wanted to focus on upright productivity,” he explains. “Ever since TED, that’s what it was about for me.”

Perceptive Pixel’s earliest screens found an unexpected killer app during the 2008 presidential campaign. News broadcasters—most famously John King of CNN—used them to swoop around maps and tally up votes via software handcrafted for the purpose by PPI.

The company sold these screens for six figures, but better still, they served as potent free advertising. “John King knows every single one of those counties,” he says. “He’s not reading off the screen. He’s using this as a storytelling tool. At that time, whenever a customer saw that, they were like, ‘This is exactly what I want to do.’ This is what a teacher does. This is what a commander does in a mission briefing. This is what a surgeon does before they actually go into surgery.”

“When I met Bill Gates, he literally said to me, ‘I want to see these things rain from the skies.'”

Han’s big-ticket high-end screens attracted attention, won awards, and generated revenue. But on some level, he remained unsatisfied. “It was good to put food on the table, but it was not going to be a huge, huge business,” he says. “We got a lot of criticism as a startup: ‘Great stuff, but when are you going to make it cheaper for the masses?’ That’s exactly why we started talking with Microsoft.”

Those talks included both Steve Ballmer and Microsoft’s cofounder and then-chairman, Bill Gates. “We started talking with them and realized they actually have the same vision,” Han says. “When I met Bill Gates, he literally said to me, ‘I want to see these things rain from the skies.'”

As for Ballmer, “he put one in his office right away. He was the first guy in the company to do it. He had a giant corner office. He took down the only whiteboard he had and put up the PPI, and said, ‘This is how we’re doing things.'”

When Perceptive Pixel became part of Microsoft in July 2012, it joined a company in more or less continuous flux. Ballmer was energetically trying to reposition the world’s most famous software developer by delving into the hardware business. A few weeks before announcing the PPI deal, the company had unveiled its Surface tablets, the first PCs ever to be sold under the Microsoft name.

At first, PPI was part of Microsoft’s Office division, placing it as close as possible to the nerve center of the company’s productivity efforts. A year later, Ballmer instigated a sweeping reorganization that clustered Han’s team, more conventionally, alongside other hardware products such as Xbox and Surface tablets (and, eventually, the former Nokia phone business, which Microsoft acquired in 2014).

Only six weeks after initiating the reorg, Ballmer announced he would step down as CEO once his successor had been found. The search dragged into February 2014, when longtime Microsoft executive Satya Nadella got the gig.

Nadella moved swiftly to rejigger how the company articulated its take on its own future, dropping Ballmer’s “devices and services” mantra in favor of emphasizing mobility, the cloud, and productivity. Despite being the least portable hardware Microsoft has ever offered—the 84-inch model weighs 280 pounds—the Surface Hub project’s focus on Net-connected collaboration had a place in the new CEO’s vision, especially as he clarified that mobility was about more than phones and tablets. As he began putting it: “We want to build experiences that are about the mobility of the experience—not the mobility of the device.”

Han with a Perceptive Pixel screen at Microsoft’s Worldwide Partner Conference in July 2012

Today, Han says, “the management support for this is even bigger than it was ever before. That’s not a ding on Ballmer. Every time you have a regime change, you wonder. But this perfectly fits into Satya’s vision of cloud-connected mobile workflows.”

The personal touches that contributed to making Han’s 2006 TED demo so engaging have helped him find his way inside the Microsoft labyrinth. “Jeff’s a very talented, energetic guy,” says hardware VP Angiulo. “And one of the few people who have made the transition from being a founder—where he was running the Jeff Show—to now being an important part of a cross-company initiative. Skype is the one writing the Skype client. OneNote is writing the whiteboard. Windows has an entire team of people creating team Windows now, that’s designed for sharing.”

Han, Angiulo says, “shows up with intelligence—deep intelligence, he really knows what he’s talking about. And the passion for the product, and the vision. He gets people to want to work with him.”

Microsoft won’t disclose how many staffers it has working on the Surface Hub, but does allow that the number is in the hundreds. They include Perceptive Pixel crew members such as David Slobodin, who supervises hardware, and Microsoft veterans such as Hayete Gallot, who heads up business strategy and marketing. Stevie Bathiche, director of research for Microsoft’s Applied Sciences Group, is also involved; even Alex Kipman, best known for spearheading the Kinect and HoloLens, is contributing to the Windows 10 side of things.

The device that Han, former Perceptive Pixel staffers, and new Microsoft colleagues ended up designing isn’t exactly in a category of one. InFocus’s Mondopad and Clary Icon’s OneScreen, for instance, combine large touch screens, videoconferencing, and Windows into conference-room collaboration tools similar, in principle, to the Surface Hub.

“I know it sounds arrogant, but we don’t have competitors,” says Han when I ask about such machines. “Sure, we’ve seen large touch screens before, but they’re in a different league, to be honest.”

Arrogant, maybe, but also a defensible stance. There’s a lot about the Surface Hub that’s new, starting with its touch-screen technology. Like an iPad or Surface tablet, it uses an optically bonded capacitive touch screen that gives it the same sort of buttery responsiveness and ink-on-paper clarity that smaller-screen devices achieve. The screen’s refresh rate of 120Hz—twice that of smaller devices—adds to the general smoothness of the visual experience.
“To bring multi-touch to its fullest potential, we’re going multi-user.”

The Surface Hub’s capacitive screen also works with a pressure-sensitive pen for notetaking, sketching, and other purposes. Though it’s not the only big screen to offer a pen, it uses Perceptive Pixel technology, which permits a single capacitive touch-sensor system to deal with both fingers and pens. Han says that PPI was one of only two companies to have worked this out; in May, Microsoft announced that it had acquired the other one, Israel-based N-Trig, whose technology is used in Surface tablets.

So much about the Surface Hub experience is familiar that it’s tempting to think of it as an overgrown Surface tablet—a perception that the “Surface” in its name doesn’t do anything to discourage. But the ways in which it differs from its dinkier counterparts are at least as important as the similarities. For one thing, it’s designed to be used by multiple people at once—which, especially with the sprawling 84-inch model, makes sense.

“Multi-touch was about breaking the tyranny of the one, to me, the single cursor. But to bring multi-touch to its fullest potential, we’re going multi-user,” Han says. “A lot of plumbing needs to change in order to really accommodate what it means for multiple people to be working together. And there are some hard problems in there.” For instance, the Surface Hub must be able to to register not up to 10 fingers at a time, but 20 or more—and to figure out if two people are working in different apps at the same time.

The Surface Hub’s approach to videoconferencing is also new. Instead of plunking a single camera above the screen like a tablet would do, it’s got two side-mounted 1080p ones which are angled to capture the entire room and are “located at eye level, not God level,” Han says. The device’s special version of Skype for Business intelligently cuts between the two cameras to provide what Microsoft hopes is an optimal view of what’s going on at any given moment.

Speaking of software, it isn’t a coincidence that the Surface Hub will reach its first paying customers a few weeks after Windows 10 is available in final form. The device and its operating system were built with each other in mind, and though the hardware can run the same Windows apps that will work on a PC, tablet, or phone, it adjusts the interface to make sense on a huge display. The Start menu, for example, sits in the middle of the screen, rather than popping up in the lower left-hand corner, and some controls get duplicated alongside the left- and right-hand edges.

The Mondopad and OneScreen, by contrast, both ship with Windows 7—an operating system that, though still beloved by risk-averse large enterprises, was never designed with gigantic screens or serious touch input in mind.
“People have tried PCs in conference rooms before. They suck.”

So much of the Surface Hub’s hardware and software is designed around the fact that’s meant to be used by multiple people in a collaborative environment that Han bristles at the notion that it’s merely a really nice PC, or even a PC at all. “People have tried PCs in conference rooms before,” he says. “They suck.”

The Surface Hub aims to eliminate typical conference-room PC hassles. Using Microsoft Exchange calendaring features, it automatically credentials meeting attendees so they don’t have to log in to get access to their files and the network. A motion sensor allows it to sit unobtrusively until someone approaches, whereupon the screen lights up. Remove the pen from its holder, and the device automatically goes into whiteboard mode. And when your meeting ends, it wipes away all evidence of the activities you conducted, so it’s ready for the next user.

Microsoft has also put a lot of thought into how the device works with the PCs, tablets, and phones that people bring into meetings. Using the Wi-Fi Alliance’s Miracast wireless technology, attendees can fling their gadgets’ displays onto the big screen. Unlike garden-variety Miracast, the Surface Hub’s version is bidirectional: If you use its pen to scribble notes on a PowerPoint slide, your annotations travel back to the version on a Windows 10 laptop.

The Surface Hub will do a lot out of the box, and will run standard Windows 10 apps. But it’s pretty clear that it will become much more useful if third-party developers start to build software with it in mind. And some of that software will need to cater to specific industries.

“We digitally model everything, but our interface is a mouse and keyboard, and that’s not necessarily the most intuitive way,” says John Cerone, director of digital design and construction at New York-based SHoP Architects, whose projects include Uber’s new headquarters. SHoP has used Perceptive Pixel displays and has just gotten in a preproduction Surface Hub. With the right applications, it sees the potential for Microsoft’s new device to be transformative. “Anytime we have the opportunity to touch or draw or pull or push a 3-D surface, we want to do that,” Cerone says.