Sci-Fi Reality

Carnegie Mellon University is at work to create 3-D representations of people and objects through the magic of claytronics.

In your hand, imagine a lump of clay, an earthy, supple material that children love to squeeze between their fingers, potters fire into bowls and artists shape into sculptures. This simple clay comprises hundreds of thousands of microscopic particles.

Now, imagine that each particle contains a computer that enables all of them to move together and form representations of objects and even people in three-dimensional space. This is claytronics, an amazing new technology for the display of information, being created at Carnegie Mellon University.

When the technology is perfected, the images of people and other objects that you see represented today inside a television will take forms in the space that you occupy. Moreover, these completely realistic representations will interact with you as if they were the actual object, animal or person. You could touch this image, talk to it, manipulate it—treat it the way you would a real-life object. Video cameras would capture the original images and then transmit them in real time to be converted into the 3-D representation, which takes shape from the millions of catoms in a claytronic “ensemble.”  That representation would be essentially indistinguishable from the original object.

Claytronics is the future of information technology, conceived by Seth Goldstein and Todd Mowry, computer scientists at CMU’s School of Computer Science. They created the idea for the project at a Grand Challenge workshop of the Computer Research Association in June 2002.  CMU has partnered with microchip manufacturer Intel, which is supporting the research and where Intel senior researcher Jason Campbell is managing the project.

Goldstein, Mowry and Campbell lead a 20-member team of faculty, researchers, staff and students whose goal is to create the electronic specks—called claytronic atoms or “catoms”—that would enable images now seen inside a video screen to take a tangible form in front of you. As conceived in an advanced stage of development, perhaps a few decades away, this radical new concept in display technology will change the way you handle just about everything presented to you by a computer.

Each catom will carry the equivalent of the data-storage-potential capability of a 1980s-vintage personal computer, and the combined computing power of hundreds of thousands of catoms will make possible the real-life qualities of the representations.

Here’s a striking example of how it would work: Suppose you are unable to attend an important event thousands of miles away. You could send real-time images of yourself across the Internet to a claytronic device—essentially a pile of catoms. There, the tiny robotic devices would assemble a 3-D version of you that would interact with real people at the event. 

Your sensory-rich 3-D replica would be capable of two-way communication. Moreover, your distant participation would be much more realistic than attending through televised video-conferencing. People with whom you want to meet could also send claytronic versions of themselves to you to close the loop in your conference room and to make transactions among catom-based and atom-based individuals seem completely real. As colleagues interact with your claytronic presence in their space, you would conduct the business of the meeting with their claytronic representatives in your space.

Reproductions from catoms would have a look, feel and movement that will evolve with the technology toward greater and greater realism. However, the surface characteristics of these reproductions will have no underlying anatomical or physical structure. The catoms will assemble a surface upon which features will be painted electronically, similar to the way that video displays today use individual pixels to contribute tiny dots of color and shadow to paint a much larger moving image.

Electrostatic forces will hold the catoms together in the form on which the image will be “painted” and, if you were to penetrate the object with a knife, you would find the object to be hollow inside. If you turn off the juice that powers the pile of catoms, the catoms return to a formless pile—ready for another charge and the software instructions they need to form another object.

How did this concept come about? Goldstein is particularly interested in the study and creation of nanotechnology, which involves very small-scale devices. He saw the potential for nanotechnology to build some of the world’s tiniest robots into a new electronic medium that could create 3-D representations of almost any object.

“We wanted to come up with a project that would be equivalent to the challenges faced in the days when people came up with the idea of sending a person to the moon,” Mowry says. Mowry envisions a world in which claytronic conferencing would reduce much of the need for business travel. Claytronics also would support an entirely new realm of interactivity in games and entertainment and even overcome the physical distances separating families and friends.

“We already play tricks with sight and sound,” Mowry says. “Claytronics will let us play tricks with space and make it feasible to work and play in locations that we don’t actually occupy.”

Does this human fax machine sound like science fiction? The horseless carriage probably did, too. Computers already have the capacity to handle the heavy lifting of information processing that will be required to form reproductions of real people. You can see this processing capacity in the realism of computer-created characters in movies such as Shrek, Happy Feet and Polar Express.

Needed now is the hardware—a manufacturable version of the tiny catom that can carry and respond to microelectronic data—and the software engineering to develop programs that will enable such a vast number of tiny computing devices to work together.

“I consider claytronics to be the most prominent instance of our strategic focus on ’big intellectual bets,’” says Randall Bryant, dean of CMU’s School of Computer Science. “Looking back over the 50 years of computer science at CMU, we can see that the reason we’re the top computer-science school today is that a number of visionary people were able to see the potential for computer technology far past the horizon most people were viewing. It’s important to have far-reaching projects like claytronics, which force us to think about what we could do with computer technology once it reaches the point that a computer can be the size of a grain of sand.”

Bryant goes on to say that CMU has been at the forefront of key conceptual insights that have shaped the world of computing, including machines that exhibit intelligent behavior and recognize spoken and written natural languages. The claytronics project, he adds, also reflects the kind of thinking in the university that conceived the assembly of large numbers of commodity microprocessors into supercomputers. Most of all, he believes that claytronics presents the type of scientific and engineering challenge that will keep CMU’s classrooms filled with the world’s brightest students for decades.

Imagine this potential use of claytronics: Cameras capture the image and motion of a doctor in a clinic and transmit his or her sensory-rich reproduction to a private home, where there is a claytronic device to reproduce the actions of the doctor. The 3-D reproduction of the doctor making this claytronic “house call” can then help a person in the house who is having chest pain—or other unusual symptom. As the real doctor in the clinic monitors the replica’s every move and guides it from a distance, the claytronic reproduction can take vital signs and provide assistance to the ailing person, even before a live crew of emergency medical technicians arrives in an ambulance. The patient is taken to a hospital, where the physician determines he needs a heart transplant.

Today, a surgeon prepares to operate by studying flat-screen images of a patient’s anatomy. However, claytronics could provide a 3-D replica of a person’s internal organs assembled from body scans. Long before making the first incision, the doctor could review a replica of the tissue and determine the best approach for a surgery.

While  the real doctor in Pittsburgh, a world-renowned specialist, is treating a patient here, he or she could pay a visit via transmission of a claytronic representation over the Internet to consult with colleagues on the condition of another patient in a hospital room in Japan.

Catoms will also find many uses for building amazingly flexible electronic gadgets.  Today’s palm device for computing, communicating and personal organizing could be transformed into an adaptable, fluid design. Flip it open to handle a call. Stretch it wide to provide a finger-friendly keyboard. Shape it long like a calendar to post an upcoming appointment.

Architects would throw away their balsa wood and construction paper and design from micro-electronic models. A nudge of the fender on a new auto design could alter the vehicle’s shape, color or other features. Because every claytronic model would be transmissible, design teams could work from remote locations with a version of the same model synchronizing in real time with an original.

The CMU-Intel researchers write about a machine that would transmit full-dimension facsimiles of inanimate objects. Paleontologists could use this portable 3-D fax machine from a remote location to transmit the replica of a freshly unearthed fossil to academic peers elsewhere. 

Quickly sharing a tangible, virtually identical, fully dimensioned copy of a rare artifact with a worldwide community of experts would hasten the creation of knowledge. Just as the 2-D fax machine forced a revolution in letter delivery, a reasonably priced, easy-to-handle 3-D fax could shift the roadway under package delivery. 


Intel’s Jason Campbell, one of the authors of the paper on the 3-D fax machine, says that claytronic devices will require software language that instructs hundreds of thousands of catoms to work with a level of virtually instinctive cooperation. Like bees or ants searching for food and finding their way around every obstacle, computer devices in such numbers need to operate from instructions that define broad goals for their motion rather than giving detailed, top-down instructions describing every step, which is the norm for today’s machine languages.

Designing such software is the greatest challenge the team faces.

“If we can really do it,” says Campbell, “then this type of computing will radically change the way we use computers and shift our relationship to computing across the board.”

While human replicas offer vivid examples of what claytronics might become, Goldstein, a former software entrepreneur who returned to graduate school at Berkeley to become a computer scientist, says the manufacturing of tiny particles of programmable matter will open a vast frontier in the design and use of computers. “Imagine having millions of tiny computers working together in one small space,” he says.

Because the catoms would function both independently and cooperatively, they could be programmed to model activities in small spaces that in real life now occur literally spread out across the planet. ”You could have a model of the Internet in front of you on your desk,” he says. Or you could begin to build much better simulations of organic processes. “Catoms in large numbers could be used for many things, even to simulate the rich connectivity of neurons inside a brain,” he says.

These dust-sized particles could give everyone the means to turn an idea into a 3-D object as quickly and easily as we download documents. In principle, it would close the gap between the fresh idea and its finished product.

Using this dynamic material to build things, creative minds could design consumer applications that cover the spectrum from serious to silly. Imagine a lounge chair that reconfigures as a writing table at the command of the user. Where earpieces now stick out from the faces of people on cell calls, a designer of claytronic phones might simulate earlobes, with skin-toned extensions for the talking end that match and enhance jabbering jawbones.

A culture accustomed to electronic illusions in sight and sound would awaken to the tricks of electronic touch and the 3-D excitement of home entertainment that plays the Super Bowl on your dining-room table. You might rent Yo-Yo Ma in replica to play cello in the corner during a romantic candlelight dinner you share with the representation of a new friend in Spain whom you’ve met on the Internet. All sorts of arrangements are possible when a claytronic double can stand in.

You might invite the neighbors into your greatly enhanced Steelers room for the scaleable 3-D experience that almost beats season tickets at Heinz Field. In addition, makers of claytronic furniture could provide people who live in small apartments or homes with an all-purpose furniture piece that can reform from worktable to love seat to breakfast table.

To turn this vision into reality, the CMU-Intel team needs to create the sub-millimeter-sized catom that will compute, move, communicate and carry power. The making of the first batch of 1-millimeter catoms promises to be the equivalent of the Big Bang for the claytronics universe. For the first time ever, electronic information would form objects and interact with users of the information without the need for a “box”—such as a video display—to contain the message.

Silicon-chip makers today can place the processing power of a vintage early-1980s personal computer—the IBM   PC-AT, for example—in the space of a punctuation mark. When that computing power is manufactured inside one period-sized sphere, claytronic devices made from millions of catoms will harness the multiplied effect of millions of personal-computers working together in very small spaces. How quickly will these wonders materialize? The CMU-Intel researchers developing this remarkable technology are moving as fast as they can—and making notable progress. Building on the work of a graduate student on the team, Michael DeRosa, the team has offered one possible solution for one of the project’s biggest software riddles—how to instruct millions of catoms to shape the surfaces of claytronic ensembles into recognizable forms.   Other complicated software-programming issues, which pose the most difficult challenges facing claytronics, are also yielding to promising solutions.

Early results have challenged the skepticism of those who believe claytronics is impossible. Still, like television and other electronic media, claytronic appliances no doubt will require decades to evolve to peak performance.

The capacity for claytronic shape shifting, for example, might appear early in smart antennas that alter shape to capture broadcast frequencies. Such devices would have a revolutionary impact on the cost of telecommunication, which today depends on antennas that pick up bandwidth with less efficiency than a shortstop can pick up grounders.

Some may wonder, “Would there be security problems—such as identify theft—associated with this new medium?” Because a claytronic device would represent the characteristics of a person directly from gestures and expressions of the individual in real time, says Goldstein, it would be almost impossible to capture identity without the person’s knowledge or cooperation. Would there be a credible threat from another scenario? Could a pile of formless catoms, resting like an idle computer, somehow be activated by a remote intruder and instructed to gather private information by rifling through papers on a desk? Or could those catoms be instructed by a virus to take another form that would enable it to do something worse?

These scenarios are conceivable to Goldstein. However, he says, as a concern for computer security, it resembles issues that already exist today and that people are working diligently to resolve. Solutions for computer crimes in their claytronics versions will come with the development of strategies to control similar concerns with two-dimensional computing—improved authentication of the users of remote systems and greater security built into the systems themselves so that control remains in the hands of legitimate users.

One thing is clear: Just as sight and sound virtually come alive through computing, claytronics promises to bring the sense of touch into a realm that only yesterday was pure science fiction.

“This is an exciting technology,” Goldstein says. “It has great potential to transform the experience of computing and to enable people to experience in three dimensions images and representations of information that they now receive in two dimensions. Just as the television has been a work-in-progress for more than a half-century, there are going to be many steps in the development of new versions of claytronics devices before we see human-looking representations of people present for remote teleconferences. Someday, although they won’t have intelligence or behavior that is independent of the original, I’m convinced claytronics will reproduce images of individuals that will have the look and feel of a real person—with a shirt on a claytronic image that appears to ripple in the wind and a hand that feels like real skin extended in greeting.”

Categories: Community Feature