Monthly Archives: March 2014

D3 and jQuery Interoperability

D3 and jQuery utilize distinctly different approaches to node manipulation and internal array storage. However, there are mechanisms available for converting between D3 and jQuery, and vice versa.

D3 and jQuery Internals

Let’s assume the following HTML structure:

<div class="test-node">Test One</div>
<div class="test-node">Test Two</div>
<div class="test-node">Test Three</div>
<div class="test-node">Test Four</div>

Now let’s retrieve the “test-node” elements with jQuery:

var testNodes = $(".test-node");
console.log(testNodes);

Console Log

As can be seen above, jQuery stores the selected elements in a first level array, making elements accessible by testNodes[0…3] as expected.  Now let’s examine the D3 equivalent:

var testNodes = d3.selectAll(".test-node");
console.log(testNodes);

Console Log

When using D3, the selected elements get stored in a second level array making the elements accessible by testNode[0][0…n].

Converting from jQuery to D3

var $testNodes = $(".test-node");  //jquery
var testNodes = d3.selectAll( $testNodes.toArray() ); //d3
console.log(testNodes);

Console Log

The above first retrieves the nodes using standard jQuery selector syntax.  Next the “toArray()” method is called returning the native array of selected elements.  Finally the “d3.selectAll()” method is invoked using the native array as the argument.  This forces the native element array into the second level array of D3.

Converting from D3 to jQuery

var testNodes = d3.selectAll(".test-node");
var $testNodes = $(testNodes[0]);
console.log($testNodes);

Console Log

The above first retrieves the nodes using D3 selector syntax.  Next the first level D3 array is used as the argument passed into the jQuery selector.

Class Addition/Removal

Adding and removing a class in jQuery is pretty straightforward:

$(".test-node").addClass("selected"); //add class "selected"
$(".test-node").removeClass("selected"); //remove class "selected"

The same performed in D3 is performed using the following:

d3.selectAll(".test-node").classed("selected", true); //add class "selected"
d3.selectAll(".test-node").classed("selected", false); //remove class "selected"

Data/Datum

D3 has a very elegant data system whereby information can be bound to elements by use of the “datum()” method.

D3 Datum

Let’s examine the following:

var nodes = d3.selectAll(".test-node")
    .datum({key: "hello", value: "world"})
    .append("span");
 
console.log(nodes);

Console Log

The above example selects all of the “.test-node” div classed elements utilizing D3, initializes a data structure/object and passes this structure/object to the “span” elements which get appended to all the matching “.test-node” div classed elements.  This “datum()” method passes the data structure/object to the “span” elements using a property called “__data__”.  As displayed above, the __data__ property has the structure of:

__data__: {
    key: "hello",
    value: "world"
}

The “__data__” property can be accessed directly using second level array syntax (ie. d3.selectAll(“span”)[0][0…n].__data__) or it can be utilized as a default argument passed into nearly all D3 selector instance methods.  Let’s examine an instance method where this takes place:

var nodes = d3.selectAll(".test-node")
    .datum({key: "hello", value: "world"})
    .append("span")
    .text(function(d){
        return d.key + " " + d.value;
    });

In the above example, the only difference from the previous example is the instance method “text()”, whereby the argument passed is a function.  This function defines an argument “d” which D3 passes to these type of instance methods automatically.  The “d” argument being the actual data/datum used to initialize the “span” elements.  The body of the text function returns the string “hello world” using the data/datum, thusly setting the text of the “span” elements to “hello world”.

jQuery Data

jQuery has a simple mechanism for adding arbitrary data to elements by use of the “data()” method.  Let’s examine the following:

var nodes = $(".test-node")
    .append("span")
    .data("datum", {key: "hello", value: "world"});
 
console.log(nodes);
console.log($.cache);

Console Log

In the above example, the “.test-node” elements are retrieved, “span” elements are appended and the datum {key: “hello”, value: “world”} is added to the data.  The console output reveals that the datum is stored in the $.cache in jQuery.

Exchanging D3 Datum with jQuery

Now that we’ve seen how both D3 and jQuery store data/datum, let’s exchange D3 datum with jQuery:

//D3
var nodes = d3.selectAll(".test-node")
    .datum({key: "hello", value: "world"})
    .append("span");
 
console.log(nodes);
 
//jQuery
var $nodes = $(nodes[0]);
$nodes.each(function(){
    $(this).data("datum", $(this).prop("__data__"));
});
 
console.log($nodes);
console.log($.cache);

Console LogIn this example, we first select the “.test-node” classed elements using D3, append “span” elements using the datum/data specified.  Next, we create a jQuery selector using the first level array of the D3 selector.  Using the jQuery selector, the nodes are iterated over in “each” and the “data()” of the jQuery element is set to the “__data__” property of the D3 element.

Exchanging jQuery Data With D3

Now, let’s exchange jQuery data with D3 using a similar technique to the previous example:

//jQuery
var $nodes = $(".test-node")
    .append("span")
    .data("datum", {key: "hello", value: "world"});
 
console.log($nodes);
console.log($.cache);
 
//D3
var nodes = d3.selectAll($nodes.toArray())
    .datum({})
    .each(function(d){
        this.__data__ = $(this).data("datum");
    });
 
console.log(nodes);

Console LogIn the above example, the “.test-node” elements are selected using jQuery, the span child elements are appended and the “data()” method is used to store the “datum” {key: “hello”, value: “world”}.  Next, the D3 selector is populated using the “toArray()” method of the jQuery selector, the “datum()” method is used to initialize an empty data object and finally the nodes are iterated over and the __data__ property is populated with the jQuery data “datum”.

Conclusion

Both frameworks are very powerful and can live together in peace.  Information can be exchanged freely between the the two frameworks utilizing these or similar techniques.

Visualizing 3D Network Topologies Using Unity

Network topology visualization can be performed using a wide variety of technologies ranging from simple 2D applications to complex 3D applications.  This approach utilizes the Unity (http://unity3d.com/) game engine to develop a network topology visualization in 3D complete with FPS controls in zero gravity and warp to zones.

Demo

Here’s an example of what the end product will look like: View Demo

Screenshots

Unity Topology Screenshot 4 Unity Topology Screenshot 3 Unity Topology Screenshot 2 Unity Topology Screenshot 1

Source Files

The entire Unity project and source files are available for download: Download Unity Topology Source

Layout File Format

Let’s get started, first a data format (or formats) needs to be chosen for loading the layout or topology data.  For my example I have chosen to use GraphML (http://graphml.graphdrawing.org/) being that Unity in C# directly supports XmlDocument.

The GraphML format I am using will support a collection of <node> and <edge> with standard and custom attributes:

<node id="node_1" x="0" y="0" z="0" name="156.145.220.128/25" ... />
<edge id="link_1" source="node_1" target="node_2" ... />

The node defines at the very least the node identifier which will uniquely identify the node in the topology map, the x, y and z which will become our Vector3 coordinates and the name attribute which will become the node label.

The edge defines at the very least the link identifier which will uniquely identify the link in the topology map, the source which identifies the source node of the link and the target which identifies the target node of the link.

Unity Setup

Next let’s define our basic Unity project structure.  Under the Assets folder, create the following subfolders:

  • Data
  • Materials
  • Prefabs
  • Scenes
  • Scripts

Unity - Topology Folders Once we have our basic folder structure setup, let’s create our 2 prefabs we will be using in our project, the Node prefab and the Link prefab.

The Node

Start by creating a new cylinder (GameObject -> Create Other -> Cylinder), size the cylinder down a little to make it look like and adding a Self-Illumin/Diffuse shader.  I made mine blue, however feel free to choose whatever color you wish.  Name the new object “Node”.

Unity - Node

Next add a new script and call it “Node” (Node.cs).  For the script we want to define a basic bit of logic to have our node text always facing the camera and a public string id:

 

using UnityEngine;
using System.Collections;
 
namespace Topology {
 
    public class Node : MonoBehaviour {
 
        public string id;
        public TextMesh nodeText;
 
        void Update () {
            //node text always facing camera
            nodeText.transform.LookAt (Camera.main.transform);
        }
    }
 
}

Add a 3D Text object (GameObject -> Create Other -> 3D Text), move the 3D Text just below the cylinder and move the 3D Text as a child of the Node object.  Next drag the reference into the Node Text (Text Mesh) property.

Unity - Node Properties

Finally drag the “Node” from the Hierarchy window to the “Assets/Prefabs” folder.  Then remove the “Node” object from the Hierarchy view.

The Link

For the link, create an empty game object (GameObject -> Create Empty), name it “Link” and add a new script called “Link” (Link.cs).  Within the script we’ll expose a few public properties such as id, source_id, target_id, etc; and define a LineRenderer which will be used to draw the line connecting the nodes.

 

 

using UnityEngine;
using System.Collections;
 
namespace Topology {
 
    public class Link : MonoBehaviour {
 
        public string id;
        public Node source;
        public Node target;
        public string sourceId;
        public string targetId;
        public string status;
        public bool loaded = false;
 
        private LineRenderer lineRenderer;
 
        void Start () {
            lineRenderer = gameObject.AddComponent<LineRenderer>();
 
            //color link according to status
            Color c;
            if (status == "Up")
                c = Color.gray;
            else
                c = Color.red;
            c.a = 0.5f;
 
            //draw line
            lineRenderer.material = new Material (Shader.Find("Self-Illumin/Diffuse"));
            lineRenderer.material.SetColor ("_Color", c);
            lineRenderer.SetWidth(0.3f, 0.3f);
            lineRenderer.SetVertexCount(2);
            lineRenderer.SetPosition(0, new Vector3(0,0,0));
            lineRenderer.SetPosition(1, new Vector3(1,0,0));
        }
 
        void Update () {
            if(source && target && !loaded){
                //draw links as full duplex, half in each direction
                Vector3 m = (target.transform.position - source.transform.position)/2 + source.transform.position;
                lineRenderer.SetPosition(0, source.transform.position);
                lineRenderer.SetPosition(1, m);
 
                loaded = true;
            }
        }
    }
 
}

Being that this script was a bit larger, I’ll walk through what it does.  First, the public properties; the “id” is the link identifier (ie. “link_1”), the “source” will become the source node reference, the “target” will become the target node, the “sourceId” will hold the id of the source node until the “source” property is populated and the same goes for the “targetId”.  The “status” will hold a value of either “Up” or “Down” and will be used to color the LineRenderer.  Within the “Start()” function, we create a new LineRenderer and color it according to the “status” property value setting its initial line positions to vectors (0,0,0) and (1,0,0) respectively.  The “Update()” method waits until the “source” and “target” properties are populated then sets the LineRenderer end points.  The “loaded” property makes sure this only happens once.

Finally drag the “Link” object from the Hierarchy window to the “Assets/Prefabs” folder.  Then remove the “Link” object from the Hierarchy window.

Controller

Add a new empty game object (GameObject -> Create Empty) and name it “GameController”.  Create a new script called “GameController” (GameController.cs).  This script will be responsible for loading the layout file, creating Nodes and Links and handling general UI updates.

 

using UnityEngine;
using System.Collections;
using System.Xml;
using System.IO;
 
namespace Topology {
 
    public class GameController : MonoBehaviour {
 
        public Node nodePrefab;
        public Link linkPrefab;
 
        private Hashtable nodes;
        private Hashtable links;
        private GUIText statusText;
        private int nodeCount = 0;
        private int linkCount = 0;
        private GUIText nodeCountText;
        private GUIText linkCountText;
 
        //Method for loading the GraphML layout file
        private IEnumerator LoadLayout(){
 
            string sourceFile = Application.dataPath + "/Data/layout.xml";
            statusText.text = "Loading file: " + sourceFile;
 
            //determine which platform to load for
            string xml = null;
            if(Application.isWebPlayer){
                WWW www = new WWW (sourceFile);
                yield return www;
                xml = www.text;
            }
            else{
                StreamReader sr = new StreamReader(sourceFile);
                xml = sr.ReadToEnd();
                sr.Close();
            }
 
            XmlDocument xmlDoc = new XmlDocument();
            xmlDoc.LoadXml(xml);
 
            statusText.text = "Loading Topology";
 
            int scale = 2;
 
            XmlElement root = xmlDoc.FirstChild as XmlElement;
            for(int i=0; i<root.ChildNodes.Count; i++){
                XmlElement xmlGraph = root.ChildNodes[i] as XmlElement;
 
                for(int j=0; j<xmlGraph.ChildNodes.Count; j++){
                    XmlElement xmlNode = xmlGraph.ChildNodes[j] as XmlElement;
 
                    //create nodes
                    if(xmlNode.Name == "node"){
                        float x = float.Parse(xmlNode.Attributes["x"].Value)/scale;
                        float y = float.Parse (xmlNode.Attributes["y"].Value)/scale;
                        float z = float.Parse(xmlNode.Attributes["z"].Value)/scale;
 
                        Node nodeObject = Instantiate(nodePrefab, new Vector3(x,y,z), Quaternion.identity) as Node;
                        nodeObject.nodeText.text = xmlNode.Attributes["name"].Value;
 
                        nodeObject.id = xmlNode.Attributes["id"].Value;
                        nodes.Add(nodeObject.id, nodeObject);
 
                        statusText.text = "Loading Topology: Node " + nodeObject.id;
                        nodeCount++;
                        nodeCountText.text = "Nodes: " + nodeCount;
                    }
 
                    //create links
                    if(xmlNode.Name == "edge"){
                        Link linkObject = Instantiate(linkPrefab, new Vector3(0,0,0), Quaternion.identity) as Link;
                        linkObject.id = xmlNode.Attributes["id"].Value;
                        linkObject.sourceId = xmlNode.Attributes["source"].Value;
                        linkObject.targetId = xmlNode.Attributes["target"].Value;
                        linkObject.status = xmlNode.Attributes["status"].Value;
                        links.Add(linkObject.id, linkObject);
 
                        statusText.text = "Loading Topology: Edge " + linkObject.id;
                        linkCount++;
                        linkCountText.text = "Edges: " + linkCount;
                    }
 
                    //every 100 cycles return control to unity
                    if(j % 100 == 0)
                        yield return true;
                }
            }
 
            //map node edges
            MapLinkNodes();
 
            statusText.text = "";
        }
 
        //Method for mapping links to nodes
        private void MapLinkNodes(){
            foreach(string key in links.Keys){
                Link link = links[key] as Link;
                link.source = nodes[link.sourceId] as Node;
                link.target = nodes[link.targetId] as Node;
            }
        }
 
        void Start () {
            nodes = new Hashtable();
            links = new Hashtable();
 
            //initial stats
            nodeCountText = GameObject.Find("NodeCount").guiText;
            nodeCountText.text = "Nodes: 0";
            linkCountText = GameObject.Find("LinkCount").guiText;
            linkCountText.text = "Edges: 0";
            statusText = GameObject.Find("StatusText").guiText;
            statusText.text = "";
 
            StartCoroutine( LoadLayout() );
        }
 
    }
 
}

So now time for the walk through.  The 2 public properties nodePrefab and linkPrefab define the prefabs to be used when creating nodes and links.  The 2 Hastables, nodes and links define the collection which will hold the live instances of the prefabs “Node” and “Link”.  The GUIText objects will reference the loading status displayed on the screen, the count of nodes and the count of links.  The nodeCount and linkCount hold the actual numeric values for the count of nodes and links.  The “LoadLayout()” method handles the loading of the GraphML file, it loads the xml file into a string, populates an XmlDocument object and iterates over the <node/> and <edge/> elements.  When a <node /> is encountered we instantiate a Node object and populate its members and same goes for <edge /> with respect to populating a Link.  Once all the “Node” and “Link” objects have been created, the “source” and “target” members of the “Link” object are replaced with the living references of “Node” through the call to “MapLinkNodes()”.  The “Start()” method instantiates the nodes and links Hastables, resets the GUIText objects and starts a coroutine call to “LoadLayout()” which allows for returning control back to the Unity platform and preventing UI lock up.

Next, lets’ add the GUIText objects.  Create a GUIText object for LinkCount, NodeCount and StatusText at a minimum.  Next place the GUIText objects in the Game view to their relative positions.

Unity - Game View

Make these GUIText objects a child of GameController and add drag the references to the GameController script references.

Unity - GameController Hierarchy

Unity - GameController

 Camera

Click on the Main Camera object and add a new script “CameraController” (CameraController.cs).  In this script we’ll add basic controls for WASD, Control, Space and Wheel/Scroll controls.

 

using UnityEngine;
using System.Collections;
 
[AddComponentMenu("Camera-Control/Move ZeroG")]
public class CameraControlZeroG : MonoBehaviour {
 
    public float speed = 5f;
    public GUIText movementSpeed;
 
    private Vector3 move = new Vector3();
    private Vector3 cluster1 = new Vector3(1960, 1791, 2726);
    private Vector3 cluster2 = new Vector3(2042, 1579, 4254);
    private Vector3 cluster3 = new Vector3(2692, 81, 2526);
    private Vector3 cluster4 = new Vector3(531, 2317, 3776);
    private Vector3 cluster5 = new Vector3(-587, 2043, 2194);
 
    void Start(){
        //set to first cluster position
        transform.position = cluster1;
    }
 
    void Update () {
        move.x = Input.GetAxis("Horizontal") * speed * Time.deltaTime;
        move.z = Input.GetAxis("Vertical") * speed * Time.deltaTime;
 
        move.y = 0;
        if (Input.GetKey ("space")) {
            move.y = speed * Time.deltaTime;
        }
 
        if (Input.GetKey ("left ctrl")) {
            move.y = -speed * Time.deltaTime;
        }
 
        //adjust speed with mouse wheel
        speed += Input.GetAxis("Mouse ScrollWheel");
        if (speed < 5)
            speed = 5;
 
        movementSpeed.text = "Move Speed: " + speed;
 
        move = transform.TransformDirection(move);
        transform.position += move;
 
        //set warp to cluster controls
        if(Input.GetKey("1")){
            transform.position = cluster1;
        }
 
        if(Input.GetKey("2")){
            transform.position = cluster2;
        }
 
        if(Input.GetKey("3")){
            transform.position = cluster3;
        }
 
        if(Input.GetKey("4")){
            transform.position = cluster4;
        }
 
        if(Input.GetKey("5")){
            transform.position = cluster5;
        }
    }
}

Walking through the above code, we define 2 public properties, speed which will hold the speed of movement through the zero gravity environment and GUIText movementSpeed which displays our current movement speed.  Next we define a Vector3 for move which will be used as the movement vector.  The following cluster[1-5] members define areas of the Topology which we’ll allow the camera to quickly jump to, feel free to replace these with your own coordinates.  The “Start()” function places our camera at the first cluster position (the largest cluster).  The “Update()” function first updates the move vector to move the camera in a direction.  The “space” check will change our “y” movement upward.  The “left ctrl” changes our “y” movement downward.  The “Mouse ScrollWheel” changes our overall movement speed, then we display the speed in the GUIText.  The final modifications are made to the camera’s transform position.  The GetKey() calls for 1 through 5 warp our camera to the cluster[1-5] positions.

We’ll also want to import a standard asset from the Unity “Character Controller” set.  Click Assets -> Import Package -> Character Controller, deselect everything, then online select “MouseLook.cs”.

Unity - Import Asset

 

This will import the “MouseLook.cs” script which we’ll need to add to the Main Camera (Add Component -> Camera-Control -> Mouse Look).

Putting It All Together

Lastly, build, deploy and check it out.  One thing to keep in mind is that the layout.xml in the Data directory (/Assets/Data/layout.xml) does not get deployed with the build.  You will need to manually copy this file out to your build/deploy directory.  Hopefully you won’t have any errors and will be able to view the 3D topology in all of its beauty and splendor.  Have fun!

Edit Over SSH Using Any Locally Installed Editor

Ever find yourself needing to edit remotely over SSH and just wish you could use your favorite locally installed editor?  Yeah VI and EMACS are great, but sometimes don’t you just wish you could have your full workspace open without having to SCP/FTP files around.  Well, you can and here’s how you do it.

Mac/Linux Instructions:

Install the sshfs client according to your specific Linux version

CentOS/Fedora

root@computer:$ yum install sshfs

Debian

root@computer:$ apt-get install sshfs

Mac OS X

This command assumes that you have MacPorts installed. If you do not, install from here: Install MacPorts

user@computer:$ sudo port install sshfs

 

Next create a directory which will be used to access the remote files:

root@computer:$ mkdir remote_files

 

Finally mount the remote server and directory to our local directory “remote_files” replacing “username”, “remote-server” and “some-directory” with your real world values.  This syntax is very similar to that used by scp:

root@computer:$ sshfs username@remote-server:/some-directory remote_files

 

Assuming that the credentials supplied to the sshfs command were correct, you should now have a remotely mounted drive by the name of “remote_files”.

Now open your favorite editor and browse to “remote_files”, all your remote files are available to you as if they were local.  Code on!

Windows Instructions:

Begin by downloading win-sshfs at: https://code.google.com/p/win-sshfs/downloads/list

Install and be aware that a reboot may be required (you may want to bookmark this page so you can get back to it after the reboot).  You can just leave the default install options the way they are.

After it has been fully installed and your machine possibly rebooted, lauch wins-sshfs if it hasn’t been launched already.  Next click on the wins-sshfs icon in the task tray.

wins-ssh tray

Next you’ll be presented with a dialog, click the “Add” button and fill out the details of the remote server to connect to.  Ensure you select the correct Drive Letter to use for this remote connection.

wins-ssh-window-1

Next click “Save”, then click “Mount”.

Assuming you didn’t have any credential errors you will have a newly mapped drive available to you.  Open your favorite editor and begin editing remote files on the drive like they are local.  Code On!