Computer graphics, the use of computers to generate and interpret pictures, is an extremely broad subject. A complete discussion of computer graphics would include such areas as pattern recognition, display equipment, and schemes for plotting results from numerical computations, as well as a discussion of the on-line use of computer-generated pictures. The articles in this issue cover a part of this subject area. This article, concerning the on-line use of computer pictures, discusses a representative sample of the topics of interest to today's researchers in on-line computer graphics.
Because there are very good people working on each of these unsolved problems, I expect that the next few years will see interesting solutions to many of them. As our use of on-line computer graphics improves, we gain new capability to visualize and thus understand complex processes. I feel that the gain will be worth the research costs.
The first problem is to get more people involved in online graphical systems. The only way to involve many more people is to bring the cost of on-line display devices down to a more reasonable level. The first and the biggest unsolved problem, then, is to build a low-cost display device suitable for on-line graphical use. The solutions to all the other problems may depend on solving this first one.
But the ideal low-cost display must have at least the following characteristics, outlined by Roberts at the 1965 IFIP Congress [1] ( see Table). First, a display device must have a certain minimum standard of quality. A mathematician working on a geometrical problem might draw three lines on his display as shown in Fig. 1. If there is a little triangular space between the three lines he may wonder, Do those three lines really meet at a single point ?
Too many of today's displays have line generators that produce slightly curved lines. Such errors in display equipment undermine users' confidence in what they see.
The second characteristic is flicker rate. There is a lot of debate about what flicker rates are acceptable in a display. No one, however, will debate that refreshing a display once every five minutes is not acceptable. Flicker rates in the range of 15 to 45 cps are useful.
The third characteristic is modification time. People who have been waiting for three hours or three days for a result from conventional computing centers get impatient if they have to wait three minutes for trivial results from an online system. There is a point, between five and 20 seconds, where instead of waiting at the console ready to do the next thing, you will get up to stretch your legs, or have a cup of coffee. A display system must respond to modification quickly.
Fourth, there must be some kind of stylus input. You have learned over years of experience to take your pen in hand and sign your name. Any other way of signing is unusable. A display must have some kind of device that you can take in your hand and which tells the computer where you are writing. DAC-I's [2] voltage pencil stylus plate and the RAND Tablet [3] give this capability directly. A light pen can give position inputs with appropriate pen-tracking hardware or software.
The final and very important capability is pointing. You must have some kind of device which lets you talk about a picture in pointing terms. You have to be able to say to the computer, in effect, I want to erase this letter, by pointing to it. The light pen gives this ability directly. A RAND Tablet or DAC-I's voltage pencil stylus plate must be augmented with coincidence detecting circuits [4] or software to provide this information.
Quality:-How accurate is the picture produced?
Character legibility
Line straightness
Focus
Uniformity
Resolution
Refresh rate - How flicker-free is the picture?
Modification time - How long does it take to get a new picture?
Stylus coordinate input: - Where is the stylus?
Direct from:
RAND Tablet
DAC-I voltage pencil stylus
Light pen requires tracking
Stylus pointing input: - What is he pointing at?
Direct when light pen sees character
Position coincidence detector required for:
RAND Tablet
DAC-I voltage pencil stylus
How we are going to get a low-cost display that has all of these characteristics I haven't the slightest idea. It is an unsolved problem. Nevertheless, I feel sure that the next year or two will see a solution in one form or another.
There are a large number of little gimmicks that make the difference between a highly-usable and a difficult-to-use on-line system. People will approach difficult-to-use on-line systems only if the advantages are immediate and obvious. If on-line systems are to be used for simple tasks, they must operate smoothly enough to be useful. A draftsman, for instance, with a choice of using a computer system or his old familiar paper and pencil will choose to use the computer only if it actually makes his job easier. It will be some time before we learn to use enough techniques to make the on-line systems as smooth in operation as they should be.
Here are a few known techniques. In SKETCHPAD [5] I used a technique which we can call Rubber-Band Line. As you draw a line, the line stretches like a rubber band between where you left its starting point and wherever you have your stylus. What gets left in the computer is not the path of the light pen, but a single straight line as shown in Fig. 2.
Another technique I used in SKETCHPAD we can call the Pseudo Pen Location. When you bring the light pen near an existing line in a drawing, the little bright dot, which is the point of your pencil, jumps to sit on the line. Anywhere near a line, the point of your pencil sits exactly on the line with mathematical precision. Pseudo Pen Location proved to be a useful little feature.
The RAND Corp is pioneering a technique in which characters are recognized as they are drawn on a RAND Tablet. I think this is going to be a very important technique because you cannot type while holding a stylus in one hand. In systems which involve a little lettering, as well as other operations, it will be very useful to print the letters you want directly where you want them.
We need not be limited to recognizing letters. For instance, it should be possible to erase a curve in a picture by making some little wiggly erasing motion over it, instead of finding and then pushing the erase button. In editing text, we should be able to use motions familiar to typesetters such as ∼, which says, transpose those two characters. The characters should instantly interchange. We should be able to draw balloons around sentences with little arrows to say, Take out all this junk and put it up here. These balloons would be recognized by a computer.
Timing and rate functions are important on-line. For example, a rather simple timing technique can vastly improve the feel of a knob. I want to move a cursor around on a cathode ray tube, to position it on top of a letter. A good way to position the cursor is to connect a couple of knobs to the x and y coordinates of the cursor. The cursor will move around on the screen when I turn the knobs. Such arrangements are common.
I tried a program which moves the cursor quadratically with the speed of the knobs. You naturally give the knob a good healthy twist whenever you want to move the cursor a long way. Because of the quadratic connection, the cursor moves extra rapidly. You will naturally turn the knob slowly when you are close, and then the cursor moves slowly and smoothly. This quadratic technique makes the cursor seem to understand what you want.
The third unsolved problem is how to connect computer graphic devices and systems to substantive computation. It is easy to draw a circuit or a bridge or a diagram of a part with a graphical device. It is not nearly so easy to make use of that drawing in doing computations about the thing that you have drawn. For instance, I should be able to draw a circuit diagram, and then observe the operation of the circuit as shown by some waveforms. I should be able to draw a complicated part, apply forces to this part by drawing them, and observe the stress distributions in the part. We have some computer programs which can do the computations. We have computer programs which can generate the drawing. But we don't have these separate computer programs put together. That is the problem of coupling. (This problem is near solution, see Fig. 3)
For any specific system you can think of (electric circuit diagrams and stress analysis are just two of the most obvious one to attack), some straightforward programming will achieve coupling. There are, however, hundreds of programs which could use graphical input and output usefully. The problem of achieving coupling in a very general way, so as to apply to a wide variety of tasks, will require much thoughtful effort.
Let me mention a place where coupling does exist and where it will be useful. It is possible to draw shapes in two dimensions with computer systems rather well. It proved to be very easy to take the descriptions of those shapes and turn them into commands for numerically-controlled machine tools. Marvin Ling [6] from GE provided me with the part shown in Fig. 5, which was actually produced from the computer-made drawing shown in Fig. 4. It is exciting to think that a light pen can, in effect, carve metal.
There are many unsolved problems having to do with the language of discourse between human beings and computers. Although we speak English every day, we don't often think of it as a language in the same sense that we think of FORTRAN as a language. In the computer business when we talk about languages, we usually mean the languages of written symbol strings. But whenever you interact with a computer, the commands you can give the computer and the cues that the computer gives you back, whether they are simple written strings or noises or flashes of light or clicks or presses of buttons or motions of the light pen, - whatever they are, they constitute a language. It is sometimes a little more difficult to recognize complex drawing languages as languages than it is to recognize and talk about the symbol-string languages. There is a lot of theoretical work to be done in understanding the more esoteric languages such as Culler-Fried [8], SKETCHPAD [5] and the better ones to come.
There is one thing which does not exist in any of today's interactive graphical languages: a way to describe motion. Bell Telephone Laboratories has several computer-produced movies. One of these shows the motion of two masses interacting through gravity [9]. In it you can see the two masses rotating about each other to form orbits of different kinds. The computer traces the path of the masses so that you can see the shape of the orbits.
The amount of information required to specify such a movie is really very small. For instance, the movie shows the motion of a mass under just the influence of a down-ward force. The mass starts at the bottom of the screen, moving up. The mass slows down because of the down-ward force. The movie shows that the mass is moving in a parabola because each second it leaves a mark behind. When the mass gets right up to the vertex of the parabola, the force is turned off. Without the force, of course, the object continues in a straight line.
To make this scene, the computer must be told "Draw me an object which observes F=MA. Apply a downward force to it. Give it initial velocity like this. When it reaches here, remove the force. I want velocity and force indicators like this". We could express that in FORTRAN, but I should be able to express it more easily graphically and on-line. Knowlton's movie language [10] lets you describe moving figures of the kind shown in Fig. 6, but only in a written language. You can make animated cartoons of a type now made by hand, at a cost which is comparable to conventional means. The hard part of using Knowlton's language is figuring out what the picture is going to look like as you write. Since you want an artistic picture, you have to make a little sketch ahead of time on graph paper. You then pick off the coordinates from the sketch and jot the coordinates down on a piece of paper to give to the computer. Knowlton's language has the motion capability but it doesn't have the on-line characteristics. I should be able to sit at the console and draw what I want. I should be able to adjust it until it is artistic. I don't care about the exact coordinates and I shouldn't have to write them down.
Any computer display that you look at today will be 90% blank space. The lines or characters will occupy only a tiny part of the display area. Most of the display area on that cathode ray tube will be unused. Today's displays characteristically produce the same kind of drawings that I can produce on a blackboard: drawings composed of narrow lines and little dots. There are many interesting things which computers could represent if they had some half-tone capability. Knowlton's movie language produces half-tone movies, but it cannot show them in real-time. I would like to see a half-tone display change as I watch. It should be possible to build hardware for such a display, perhaps even in color.
If I had such a gadget there would be a whole host of unsolved problems in graphics about how to use it. The entities that I would have to talk about would no longer be just simple lines or characters, but things with body: a shaded rectangle, or a rectangle outlined in dark and shaded in the middle, or shaded dark on one side and lightly shaded in the middle, or shaded dark on one side and lightly toward the other side. We haven't even begun to have languages that are capable of describing such things.
The sixth unsolved problem has to do with the strueture of drawings. We human beings are able to look at drawings and detect symmetry in them. For example, the drawing shown in Fig. 7 has a rather obvious structure or symmetry to it.
Now suppose that a man who is using an on-line system points his light pen or his RAND Tablet stylus at this drawing as shown in Fig. 7, and then says, delete. The correct computer response to the delete command depends upon the real underlying structure of the drawing. If the drawing is intended to represent five independent boxes, the correct response might be to erase one of them. Or the correct response might be to erase the dot from each box. If, however, the underlying structure of the drawing is really two groups of two boxes each, with an additional box in the middle as shown in Fig. 8, then the correct response might be to erase the righthand dot in each pair.
For people, the underlying structure is made clear by context. In fact, we usually do not even think explicitly about the underlying structure. But if a computer is to make good use of the underlying structure of drawings, then we must have languages powerful enough to explicitly represent the structure of pictures. Moreover, the languages must represent the structure of pictures. Moreover, the languages must represent the structure naturally enough so that it doesn't get in our way. We must represent the structure so users can see it. In Fig. 8, a dotted line around the groupings shows the structure, but that would not be appropriate in every case. We must learn how to represent structure for human consumption.
We must also build languages which can construct structure. Dialogue may help. If you, for instance, tell an electrical draftsman, Remove the gate, while pointing to one of a parallel set of gates, he will immediately ask you, Do you want me to remove them all or just that particular one? A computer could ask that same question too, given an appropriate program. We have much to learn before it will be possible to write such a program.
There are three more problems, which have to do with the computations involved in using computer graphics, rather than the graphics themselves. The first of these is the hidden line problem. When we look around the world we see opaque objects and we don't see what is behind the opaque objects. It is hard to make objects displayed by a computer look similarly opaque. It is easy to make a perspective presentation of any individual point in space. It takes a few multiplications and a division or two to implement the coordinate transformation from the three-dimensional space coordinates to the two-dimensional display coordinates. By programming this transformation you can easily display transparent or wire frame views of your object, as shown in Fig. 9.
It is much harder to decide whether a point ought to show or not. It is a major task to eliminate hidden lines from the drawing. For instance, in the display of a cube I ought either to see the rear lines dotted, or I ought not to see them at all. It is not too hard to eliminate the hidden lines for a single cube. You merely show all parts of any face of the cube that faces the viewer. It is much harder to eliminate parts. of lines from one object that is partly hidden by another. You must decide how much of each edge, if any, to show for partly-hidden objects.
It is conceptually easy to eliminate hidden lines by brute force. You represent each line as thousands of little points and you eliminate any point hidden by any of the thousands of objects which might be in front of that point. Unfortunately, the amount of computation involved gets rapidly out of hand. A more elegant solution is possible. Roberts of the MIT Lincoln Laboratory generates views of solid three-dimensional objects with all the hidden lines removed [11] (see Fig. 10). Roberts' technique, which uses a method of linear inequality, is limited to plain-faced objects. Applying Roberts' technique to surfaces more complex than plain faces - to the conics or cubics, for instance - is going to be quite a difficult task. It is essential to do this task if we intend to look at displays of solid objects.
The eighth problem has to do with making computer programs less inscrutable. You can see all the pieces of a clock you have taken apart. Each has a characteristic shape. You can look at a gear and easily pick out the only other gear it could possibly match with by just looking at the size of its teeth. Because you can see what the parts do, you can look at a fairly complicated mechanical gadget, and understand it.
Have you ever tried to pick up somebody else's computer program and understand what it does? It ought to be possible to display whatever it is about a computer program that makes it unique. It ought to be possible to represent whatever it is that makes an individual computer program individual. I want to look at a display of a computer program and recognize that this part must go with that part just as I can match up the gears of a clock.
I ought also to be able to see a computer program running. I want to get a display of information about how the computer program is running and what it is doing. For instance, I might use a special program to count how many times each branch is used. I could look at these counters and know what parts of the computer program are used a lot and what parts are not used very much.
I could make a graphic presentation of such information. Imagine, for instance, a flowchart or a block diagram drawn in three dimensions. Two dimensions might show the diagram and the third dimension might show the number of times that this part has been used. All of the parts that have been used a lot would stick way up.
I am not suggesting that three-dimensional flowcharts are the answer for program instrumentation. I merely suggest them as an example. Only when we have actually had experience with several dozen such examples will we have any idea which kinds of presentations help. With such presentations, computer programs would be more scrutable.
The last two problems are unlike the others because they will be with us forever.
Problem number nine has to do with the logical arrangement of drawings. When we draw an organization chart, we draw organizations of similar function or responsibility beside each other, rather than sprinkled all over the page. In drawing an electrical circuit we may use a convention that relates de voltage level to position on the page. If I draw a block diagram of a multiplier, I try to show the different parts of the multiplier lined up together. If there is a register that consists of a bunch of flip flops, I try to line them up in a row. If there is a set of gates that enter information into the register, I try to show them below the flip flops. If there is some other logic net, I try to show it collected together by function.
All of these things have to do with the logical arrangement of the drawing. They have nothing to do with its topology, which would be the same no matter how the drawing was arranged. By arranging the parts of the drawing in a sensible or logical way, the meaning of that drawing can become much more apparent. The ninth unsolved problem in computer graphics, then, is to get computers to make drawings with logical layouts.
Although there have been some fine attacks on this problem, each is limited to a particular area, and falls down in some cases. Because for each particular case, logical takes on a different meaning, we may never find a complete solution. Perhaps the best we can expect is an interactive system in which a man and a computer, together, can come up with an arrangement that satisfies the man.
The tenth and final problem has to do with abstractions. In all fields of scientific endeavor, the man who makes the scientific progress is the man who understands. Such a man has either a mental model, or a notation, that lets him represent and, therefore, think about a mass of complexities that nobody ever understood before. In one or two words he is able to say, This is the way it is. He puts together a beautifully simple little theory which explains all the complexity.
For instance, we can think about a bell-shaped curve. The heights of people probably form a pretty good bell-shaped curve. But the bell-shaped curve isn't anything that exists in nature. It exists only because somebody chose long ago to represent heights in Cartesian coordinates and his graph came out with a characteristic shape. A lot of other things represented in Cartesian coordinates came out to have the same shape. People started to talk about the properties of the shape and useful predictions resulted. The bell-shaped curve is a fiction. It isn't anything that exists in nature. It is a description of the fact that certain things which do exist in nature come in different sizes: people come tall and short, but most people are of medium height.
What are some other examples of abstractions? We can speak of a sawtooth oscillator. A sawtooth oscillator doesn't produce sawteeth (I can just see a sawtooth oscillator spitting out teeth!). It produces an output voltage that looks like sawteeth only when you graph it as a function of time. Another abstraction might be useful in representing an automobile sale in a computer. If you buy an automobile you have a choice of getting either white wall tires or black wall tires. Let me call that the tire option. The tire option has properties. For instance, one of its properties is a price differential of $30 or $40. When you buy a car you also have an engine option. You can either get a six or a big V-8. The engine option also has a price differential associated with it. I can ask some questions which have nothing to do with automobiles at all but which ask you about these abstractions. I can ask, for instance, is the engine option bigger than the tire option?
I have some of my own useful abstractions. When I think about electric circuits, I represent dc voltage as a position on the page. I also represent ac voltage as a position on the page. I visualize an operating circuit as wiggling up and down. I think of resistors as springs, and wires as pieces of string. I think of transistors as containing a little bellrope puller who doesn't care how much rope he has let out. All he cares about is pulling on his string; he is a current source. This abstract model helps me to analyze simple circuits instantly.
Unfortunately, my abstract model tends to fade out when I get a circuit that is a little bit too complex. I can't remember what is happening in one place long enough to see that is going to happen somewhere else. My model evaporates. If I could somehow represent that abstract model in the computer to see a circuit in animation, my abstraction wouldn't evaporate. I could take the vague notion that fades out at the edges and solidify it. I could analyze bigger circuits.
In all fields, there are such abstractions. We haven't yet made any use of the computer's capability to firm up these abstractions. The scientist of today is limited by his pencil and paper and mind. He can draw abstractions, or he can think about them. If he draws them, they will be static, and if he just visualizes them they won't have very good mathematical properties and will fade out. With a computer we could give him a great deal more. We could give him drawings that move, drawings in three or four dimensions which he can rotate, and drawings with great mathematical accuracy. We could let him represent all kinds of very complex and very abstract notions, and we could let him work with them in a way that he has never been able to before.
I think that really big gains in the substantive scientific areas are going to come when somebody invents new abstractions which can only be represented in computer graphical form. As he begins to work with his new abstraction, he is suddenly going to discover something brand new and really exciting which may revolutionize the world. How do you go about inventing new abstractions? I am afraid I really don't know.
1. Roberts, Laurence G, Graphical Communication in a Time Sharing Environment.
2. Design Augmented by Computers System at General Motors Research Laboratories.
3. Davis M R, Ellis TO, The Rand Tablet: A Man-Machine Graphical Communication Device, RAND Memorandum RM-4122-ARPA.
4. Hornbuckle, Gary D, Display System Specifications, Project GENIE Document 20.60.10, Computer Center, Univ of California, Berkeley, 1965.
5. Sutherland, I E, Sketchpad: A Man-Machine Graphical Communication System,, SJCC,1963 and MIT Lincoln Laboratory Technical Report No 296, 1963, reissued 1965.
6. Ling, Marvin, The Logical and Analytical Structure of the Computer-Aided Design Process as Applied to a Class of Mechanical Design Problems, Univ of Michigan, 1964.
7. Sutherland, William R, The On-Line Graphical Specification of Computer Procedures, MIT Lincoln Laboratory Technical Report No 405.
8. Culler, Glenn J, Fried, Burton D, The TRW Two-Station On-Line Scientific Computer, Proceedings of the Symposium on Computer Augmentation, DC, 1964.
9. Sinden F W, Force, Mass and Motion, Bell Telephone Laboratories Movie.
10. Knowlton K C, A Computer Technique for Producing Animated Movies, SJCC 1964 .
11. Roberts, Laurence G, Graphical Communication in a Time Sharing Environment, MIT Lincoln Laboratory Technical Report No. 315, 1963.