A definition of real-time might be attempted, however, certain difficulties arise since there are few good examples one can refer to. Technical explanations are relatively simple but to convey the experience of a new application is quite another matter. Recently at the 1970 Spring Joint Computer Conference the Evans and Sutherland Corporation demonstrated several real-time computer programs on their graphic console. One such program is a simulation of an aircraft taking off and landing on an aircraft carrier. The user can graphically fly his aircraft in a real-time environment communicating with the program through an electronic tablet and light pen. He can make changes in the elevation of the movement of the aircraft as the event occurs in time. He has the visual representation of these relationships on the oscilloscope as if it were a real physical event of flying the aircraft. To the observer it appears as if the user takes off in his aircraft from the aircraft carrier and the program shrinks the carrier's size and tilts the horizon line depending upon the user's maneuvers. What the observer does not usually appreciate are the utility routines and the perspective program and the clipping divider program which handle the transformations. A very dynamic and interactive mode of communication is offered in which one can test out maneuvers and experiment with alternatives without serious consequences.
In our real-time system at Ohio State the user can draw images on the CRT or he can use mathematically generated images for film animation. Not only do his images move in real-time but the images can be controlled as they move with the light pen and switches and buttons on the function keyboard. An artist can draw key frames for an animation sequence and a computer program fills in the additional drawings required to make the images move. To control speed one can adjust switches on the function keyboard which represent different rates of speed, and the moving images instantaneously speed up or slow down. The axis of rotation of a moving drawing can be adjusted through the function keys and light pen. The size of a moving drawing can be changed by pressing a function key to reduce the size of the drawing while another function key is used to increase its size. These function keys are programmed to behave like potentiometers. With the light pen one can draw the path they wish the drawing to follow, and as soon as the path is completed and one presses a function key, the drawing moves at motion picture speed. Several drawings can be displayed simultaneously on the CRT and the user has independent control over the path, the speed of movement about the path, and the size of each drawing. By pressing keys on the 2250 keyboard that represent x, y, z, directions, a software joystick can be used to control the path of 3D drawings. For instance, a 3D drawing representing a turtle can be walked in real-time throughout three dimensional space. The animator also has independent control over the movement of the turtle's head, the tail, and each of the four legs in real-time. Alphanumeric characters for words and sentences can be typed in through the 2250 keyboard, and as soon as the word or sentence is typed, the light pen can be used to sketch the path, and then the word will move about the path. Essentially in our real-time environment the user can fly, walk or model his drawings through two and three dimensional space.
At the risk of being redundant this concept of real-time might be described in the following way to artists and film makers. The process of creating images which we have just described involves an exciting new concept of time. Essentially the computer is only programmed so that it can respond to the decisions of the artist about the image and its movement. Then whatever the artist decided to do is transmitted almost instantaneously to the screen. This allows for a full interaction between the artist and his images. It occurs in what we call real-time, that is, time which is real because the moment of the artistic idea is also the moment of its materialization. This is a revolutionary idea in film-making, rather like editing the film before it is made. It makes the creative process a very spontaneous thing, for all the problems of execution are solved in advance. It is also by no means as mechanical as it might seem, for different artists will produce a variety of images and different relationships between images to portray their concept of reality.
In a real-time environment there are limitations as to the number of transformations that can be linked together before the real-time display is lost. Even with a sophisticated use of a compound transformation, and special approaches to the relationships between transformations and data structures, the computer's storage size and cycle speed, with its word length and instruction set, will impose additional constraints.
One of our concerns is not only one of developing new tools or options in graphics but how does the non-specialist use them in a natural way. Perhaps the question might be stated in the following manner. How can computer graphic tools involving chains or links of transformations be integrated for real-time animation so that the user can be involved in a spontaneous and interactive mode of communication to generate animated film sequences? Here one would hope to minimize the delays in response time and the computer programs organized in such a way that the user can give his full attention to the concepts and ideas he wishes to illustrate.
An example of the difficulties one can encounter with a rigid relationship between computer programs for real-time animation is as follows. The user has available to him the following programs that may represent only a small fraction of a graphics software package. (1) A 3D rotation program which handles hidden line removal (the data described as edges and intersecting planes) and which simultaneously handles straight line path zoom effects, as images are moving in real-time. (2) A 3D rotation program which handle hidden point removal (the logical structure of the data is a description of surfaces as points) and it has a software joystick and also the option to draw curves on the CRT with the light pen which can represent the zoom path of data. If the user wants the data described as edges and intersecting planes in program (1), and only the software joystick to control its movement from program (2), another new program will need to be written by a programmer to accommodate a new relationship between these routines. The solution to this problem may be a simple one but it is an inefficient approach and the storage space and speed of the computer system can soon be exhausted. Each user in the computer film animation environment has a different need and an approach is highly desirable which offers flexibility and new options and one which is not programmer dependent.
An alternative to the preceding approach is to treat the transformations as discrete units or modules. Conceptually, at one level they would appear to the user as items on a shelf or as a menu that he could string together as a chain of transformations. There would be a discrete unit for 3D to 2D or 2D to 3D data conversion, straight line zoom, drawn curve zoom, joystick, random walk, hidden line program, hidden surface program, scaling, 3D rotation, data conversion of hidden surface and hidden line, and many other units. At another and more important level transformations might be treated as building blocks by the user to formulate new transformations. A modular approach to computer programming is not new and it may appear simplistic but in real-time animation there are a number of special considerations. These considerations involve the relationship between data structures, transformations, and data management, and storage allocation.
The following example might illustrate a more complex problem for real-time animation and it is one which we have solved with software. Let us use as a data deck, a line drawing representation (x, y, z, coordinates) of a helicopter. The helicopter has two rotors, one for the main assembly section and another one on the tail section. We have a 3D rotation program which has as part of its structure, transformations to change the axis and the center of rotation of the helicopter. The problem is to make the helicopter move on the CRT and the computer must calculate and display pictures at a rate fast and smooth enough to appear to the user as a motion picture or video film. It should appear to move about an axis of rotation and the effect is as if the helicopter is moving in perspective about the path of an ellipse. While it is moving about this path each rotor is to move at a different speed in relationship to the helicopter itself. Then in addition, hidden lines are to be removed as the helicopter is changing its position in relationship to the user of the program. The user should be able to communicate to the program with a light pen and function keyboard to immediately change the speed of the helicopter's movement and the speed of each of the rotors. He can at any stage change the axis of rotation of the helicopter or even draw a new path.
In this example, let us give our attention to the most important problems and perhaps set aside a number of minor details about programming in a real-time environment. For instance, the transformations involved in a 3D rotation program are important but not the issue at the moment. Neither are the utility routines which handle the interrupts through the light pen or the function keyboard. There are a number of questions we must ask ourselves. (1) How do we define the data structure so that the program can recognize the helicopter as a whole and the individual rotors as data? (2) Once a transformation is developed to move the helicopter, how is another transformation introduced which will take into account that the helicopter is already moving at some rate of speed constantly changing its location, but now the main rotor must move at another rate of speed? (3) How do we link the transformation to the data structure and other transformations which will look at the main assembly that is moving and at the same time have another speed for the tail rotor? (4) While we are dealing with three motion variables, how can hidden lines be removed as the helicopter is changing its position? (5) How do these transformations look at the data structures to recognize the helicopter, main assembly and tail, main assembly rotor, and tail rotor? How does one set up the pointers to recognize data and establish the appropriate links between the transformations? At the same time, how does one allocate space for programs and data and move them in and out of the computer's core memory fast enough to provide a real-time display?
Algorithms to solve the hidden line problem in a batch processing environment have been in existence for several years. Most of them have been written in a higher level language such as FORTRAN. The calculation time for one view of an object may vary from 30 seconds to several minutes depending upon the complexity of the object and the efficiency of the algorithm - one would assume access to a high speed computer. If the programmer could reduce the calculation time from seven minutes to two minutes, he would naturally be enthusiastic about the improvement. There are differences in the algorithms used by researchers Bouknight, Warnock, and Loutrel. Although these are excellent solutions to the hidden line problem, they do not seem fast enough as solutions to produce a real-time hidden line display (the convex case), at least on 32K machines with medium speed core.
At Ohio State on our 1130/2250 system we have a real-time hidden line program which can handle the convex case (600 edges). In the process of doing an analysis of the hidden line problem, we discovered some procedures which reduced the number of tests that had to be made with the data. Another significant factor in our real-time hidden line display involves an extremely fast three dimensional rotation program. Our hidden line routine is also fast enough to permit several drawings to be displayed simultaneously moving at different rates of speed, and at the same time each drawing has a different zoom path and zoom rate as hidden lines are removed. The technique and concept used for this rotation program coupled with an improved hidden line algorithm accounts for our success with real-time display. Our results were produced on a 32K machine with a 3.6 microsecond cycle time but the same techniques and concepts used on a machine such as the PDP10 with its greater speed and better instruction set would produce far more complex figures moving in real-time. It would be reasonable to expect that many figures in the concave case could be displayed in real-time on a PDP10 machine interfaced to an Evans and Sutherland graphics display processor.
Extensions or projected plans of our work into the hidden line problem include not only refinements on the convex case but solutions to the concave case which will make it possible for us to do the following type of animation sequence. It would probably not be possible for us to achieve this example in real-time but the display might move at stop action speed.
Under computer program control we would like to graphically move throughout this landscape. As the user communicates his position relative to the landscape, the program provides a display of his position at that location. The house, the two cars, and the airplane are three dimensional models which have a representation of their interior structure as well as their exterior structure. The user first adjusts with the light pen and function keyboard, the path and the speed of the two cars, and the airplane. Then he establishes a location x, y, z, coordinate in the scene from which he can view the scene. Let us assume, that he starts at a position several hundred feet above the ground level. Within a matter of seconds or less the CRT will display a representation of the scene from his viewing position. Through the use of a software joystick he can move in rapid incremental steps down to the house with the program changing size relationships, the perspective, clipping parts of the landscape on the x, y, and z planes, and at the same time removing hidden lines. As he reaches the wall of the house and then passed through it, the program would display the interior of the house. He could spend some time looking at various features of the interior. Perhaps he might look outside through a window. As soon as he passed through the house the outside scene would now be displayed. He could shift his position and drive the car down the highway. It is expected that, once we could display and control information at that level, the computer program could have addition options of a multiple point light source to provide gray scale representations and also to indicate the translucent and opaque appearance of surfaces depending upon the user's viewing angle.
The 2250 display has only one intensity level. In order for us to do gray scale representations of solid objects, a point density subroutine was written to achieve the variations of gray. The spacing of points on a given face of a geometric solid determines the shade of gray. Data structures were defined to have a logical description of surfaces as points. A solution to the hidden surface problem is part of this program. At this stage of the project we are restricted to the convex case and the representation of data must accommodate this constraint. Although our gray scale objects cannot function as real-time animation, a relatively fast algorithm has been devised to paint the object fast enough to make it reasonable to make animated film. We simulate a Flying Spot Scanner through software and control the CRT beam as it is exposed to motion picture film. Two representations of the data are involved to make a real-time animation a useful technique for gray scale figures. One data structure is made which described the geometric solid as edges, while another one described it as surfaces with points and specific shades of gray. With the line drawing to represent data, judgments can be made about the size, position, motion, and speed of the objects. Adjustments can easily be made in the real-time environment with those parameters. Such important decisions would be the precise path, size, speed, and time duration about the path. Then the values representing those parameters are used on the second data deck to make the final film sequence. The individual frames are painted in shades of gray fast enough to make stop action filming a feasible approach.
Our algorithm to write, points on the CRT representing geometric solids (convex) and simultaneously to do a double rotation while removing hidden surfaces, can display 225,000 points in less than 12 seconds. Once the first view is painted there is less than one second delay between the first view and the second view. It seems reasonable for us to make short sequences of gray scale motion picture of figures which can be represented by 75,000 - 100,000 points. Here one would have a new frame every 5 or 6 seconds.
Interactive graphic displays for real-time film animation and model simulation can be a dynamic and productive process for the user. The computer programming requirements to support and extend the capabilities of such a system is a sophisticated problem which involves basic research in computer graphics. The frontiers of research into the relationship between man and machine, as it is applied to interactive graphic display systems, indicates that additional modes of communication might be brought to bear upon this problem. New relationships between man and machine could involve a more total sensory-electronic communication system. They could influence his definitions of problems and how these problems could be solved. All of these modes of communication include special purpose electronic hardware and the appropriate computer programs to implement them. (1) Visual communication systems which involve the user's eye and body movements offer some interesting possibilities. Ivan Sutherland's video helmet for 3D perception of computer generated images in a real-time system is one example. Another possibility makes use of a special optical headgear electronically connected to the computer which scans the user's eye movements. At The Ohio State University a major research project (1966-1969) did research in eye movement studies on subjects as they watched commercial television programs. The subjects wore a head mounted electronically operated optical system which tracked their eye movements and these eye movements were recorded on video tape and then analysed. Manfred Knemeyer, my research associate, designed the electronic system for the eye movement studies and he also wrote the computer programs which did the analysis of their data. It seems reasonable to us to electronically wire the human so that he could communicate to the computer with his eye movements, information about the path, the speed, and even the size of the drawings displayed upon the CRT screen. Additional visual commands or instructions could be devised as a visual code to communicate to the computer program. (2) Computers that can recognize human speech are not as yet practical but there is evidence to suggest such a mode of communication may soon be a reality. This method of communication may initially be confined to simple commands, to control a graphic display but the potential for a more complex relationship between the human and the machine is an exciting prospect. (3) A tactile-kinesthetic mode of communication with a computer may seem somewhat bizarre but there is research already moving in that direction at Bell Telephone Laboratories, and at The University of North Carolina. The potential for form generation, form manipulation, and form perception is a promising area to investigate.
In the man and machine communication problem as it applies to visual representation by computer different modes of communication need to be examined to improve the quality of this relationship. Concepts about computer equipment design and the need for better communication systems are already emerging in the technology. With the advent of such a communication system (visual, tactile-kinesthetic, and verbal) basic questions of language description about a problem and problem definition itself are still difficult ones to resolve. In fact, additional modes of communication might further complicate these questions, at least, if one continues to use conventional approaches to data structure definition and linkages to transformations, data management, and the structure of transformations. However, these modes of communication and the ideas they will stimulate may well provide the conceptual breakthroughs that can make the computer process a more dynamic one.
In many ways the computer simulation of pertinent information from the real world is still an art and not a science. The effectiveness of any simulation rests on the human simulator's ability to abstract only those factors that affect the system or process he wishes to duplicate. Our research in computer graphics for real-time film animation, especially our success with the helicopter problem which involved a compound transformation, suggests that we may be able to develop additional computer graphic tools to deal with simulation models. It may be an ambitious goal but it is certainly a highly desirable goal, to expect that computer technology may be able to provide the appropriate tools to allow the scientist, the educator, and the artist to simulate their models of reality.