Do you ever wish you could just see some table or tree (or graph, list or map) while programming?

The goal of Lucidity is to make this continually possible in a flexible, easy to use, language-agnostic manner. More generally, it aims to make the state of executing programs visible at a level of abstraction useful to programmers, either for debugging or for instructional algorithm visualization.

The following is a demo of the latest version.

How to use it:

The simplest way of getting your program state visualized is to use Lucidity's object monitoring capabilities: instead of using a 'println' statement, use MObject.observe(yourObject), at which point Lucidity will take a snapshot of your object (and all objects which it references), track any changes to any of these snapshotted properties, and generate an interactive visualization of your object changing in time. You can step through the individual modifications to your object one at a time, forward and/or backward; or you can watch the state evolve in real-time (or scaled time).

The other possibility is to write your algorithm on a 'monitored data structure' of a specific type. For instance, if your algorithm is operating on a tree, you would write it on MTreeNodes (provided by the Lucidity client); or your node class could subclass MTreeNode; or you could add calls to MTreeNode.add/remove etc. into your node class' add/remove etc. methods. The advantage to using it this way is that the visualization will be highly specialized for either lists, tables, graphs, trees, or maps (i.e. associative arrays/dictionaries); with object monitoring, a tree visualization is always used.

Lucidity uses a client/server architecture. The server processes data structure operations and provides interactive visualizations. The client tracks changes to monitored objects/data structures, and sends operation data to the server. So far a fairly complete client has been written for Java and a less complete one for javascript, but one could be written for most programming languages.

Current state of the project:

  • List, map, and tree visualizations are available.

  • General object monitoring is about 90% complete.

  • Typical media player-like controls are available for navigating visualizations (play/pause, next/previous operation).

  • Large numbers of data structures can be monitored at once, and the system manages how they're laid out in the 3D - scene; it groups them into 'pages' which can be switched between using the arrow keys.

  • While observing some single page, the camera moves automatically to optimally fit your data on screen.

  • Three 'playback modes' are available which play back operations in different ways: in real-time, using uniform delays between operations, using delays proportional to actual delays in program execution. Operation playback rate can be scaled in the GUI.

  • There's a simple way of dynamically coloring data structure regions (as a sort of high-level algorithm annotation system).

  • A client has been written in Java, and there's a partially complete Javascript client which I've been using to build an Electron app that allows users to write javascript in a sandboxed envirnoment and get immediate visualizations of how their code affects the data it manipulates.

Where this came from:

My favorite debugging technique is to create a dynamic visualization of whichever datastructure(s) I have found to be causing me troubles. I got tired of writing these from scratch each time, and I was thinking about a project that was going to require a bunch of different visualizations, so, this is an effort at a general solution. You can see my use of a sort of 'one off' data structure visualization here (was used for debugging Tiled Text; it's the sideways tree-thing on the right):

I used it to watch the abstract syntax tree as I performed operations on the 'document.' It took probably a good five hours to code, and there were certain things about the nature of it that made it easier than it would be in many cases. With Lucidity, it would take virtually no time: I would just code the abstract syntax tree on a MonitoredTreeNode instead of a TreeNode.

Another way of thinking about it:

The fundamental activity of all computer programs is manipulating data. We programmers issue huge, complex, sets of instructions describing these manipulations, and because the activity is so complex, we have to revise our instructions in iterations: we watch our programs execute, see in which ways the results are off, revise our programs, repeat. Could this process not be improved if we were able to watch the data being manipulated, so that we could actually see the effect our instructions are having?

I think we're in a similar position now to astronomers before telescopes were around: there's something we want to observe that we can't look at directly. We need something that automatically transforms the object of our observations into something which human minds and senses can actually use.

The transformation required in software is not magnification, however. Our problem is that we have too much data, too rapidly evolving—and we need ways of automatically eliminating less relevant subsets of that data, retaining only the most essential parts. Lucidity is an attempt to build one of these 'abstractoscopes.' (Though, granted, its attainments will likely be more modest than what I think 'abstractoscopes' are in principle capable of—it's just something in that direction.)

Feel free to send me any questions/comments at westoncb[at Google's mail service].