Tag Archives: code

Visualizing 3D Network Topologies Using Unity

Network topology visualization can be performed using a wide variety of technologies ranging from simple 2D applications to complex 3D applications.  This approach utilizes the Unity (http://unity3d.com/) game engine to develop a network topology visualization in 3D complete with FPS controls in zero gravity and warp to zones.

Demo

Here’s an example of what the end product will look like: View Demo

Screenshots

Unity Topology Screenshot 4 Unity Topology Screenshot 3 Unity Topology Screenshot 2 Unity Topology Screenshot 1

Source Files

The entire Unity project and source files are available for download: Download Unity Topology Source

Layout File Format

Let’s get started, first a data format (or formats) needs to be chosen for loading the layout or topology data.  For my example I have chosen to use GraphML (http://graphml.graphdrawing.org/) being that Unity in C# directly supports XmlDocument.

The GraphML format I am using will support a collection of <node> and <edge> with standard and custom attributes:

<node id="node_1" x="0" y="0" z="0" name="156.145.220.128/25" ... />
<edge id="link_1" source="node_1" target="node_2" ... />

The node defines at the very least the node identifier which will uniquely identify the node in the topology map, the x, y and z which will become our Vector3 coordinates and the name attribute which will become the node label.

The edge defines at the very least the link identifier which will uniquely identify the link in the topology map, the source which identifies the source node of the link and the target which identifies the target node of the link.

Unity Setup

Next let’s define our basic Unity project structure.  Under the Assets folder, create the following subfolders:

  • Data
  • Materials
  • Prefabs
  • Scenes
  • Scripts

Unity - Topology Folders Once we have our basic folder structure setup, let’s create our 2 prefabs we will be using in our project, the Node prefab and the Link prefab.

The Node

Start by creating a new cylinder (GameObject -> Create Other -> Cylinder), size the cylinder down a little to make it look like and adding a Self-Illumin/Diffuse shader.  I made mine blue, however feel free to choose whatever color you wish.  Name the new object “Node”.

Unity - Node

Next add a new script and call it “Node” (Node.cs).  For the script we want to define a basic bit of logic to have our node text always facing the camera and a public string id:

 

using UnityEngine;
using System.Collections;
 
namespace Topology {
 
    public class Node : MonoBehaviour {
 
        public string id;
        public TextMesh nodeText;
 
        void Update () {
            //node text always facing camera
            nodeText.transform.LookAt (Camera.main.transform);
        }
    }
 
}

Add a 3D Text object (GameObject -> Create Other -> 3D Text), move the 3D Text just below the cylinder and move the 3D Text as a child of the Node object.  Next drag the reference into the Node Text (Text Mesh) property.

Unity - Node Properties

Finally drag the “Node” from the Hierarchy window to the “Assets/Prefabs” folder.  Then remove the “Node” object from the Hierarchy view.

The Link

For the link, create an empty game object (GameObject -> Create Empty), name it “Link” and add a new script called “Link” (Link.cs).  Within the script we’ll expose a few public properties such as id, source_id, target_id, etc; and define a LineRenderer which will be used to draw the line connecting the nodes.

 

 

using UnityEngine;
using System.Collections;
 
namespace Topology {
 
    public class Link : MonoBehaviour {
 
        public string id;
        public Node source;
        public Node target;
        public string sourceId;
        public string targetId;
        public string status;
        public bool loaded = false;
 
        private LineRenderer lineRenderer;
 
        void Start () {
            lineRenderer = gameObject.AddComponent<LineRenderer>();
 
            //color link according to status
            Color c;
            if (status == "Up")
                c = Color.gray;
            else
                c = Color.red;
            c.a = 0.5f;
 
            //draw line
            lineRenderer.material = new Material (Shader.Find("Self-Illumin/Diffuse"));
            lineRenderer.material.SetColor ("_Color", c);
            lineRenderer.SetWidth(0.3f, 0.3f);
            lineRenderer.SetVertexCount(2);
            lineRenderer.SetPosition(0, new Vector3(0,0,0));
            lineRenderer.SetPosition(1, new Vector3(1,0,0));
        }
 
        void Update () {
            if(source && target && !loaded){
                //draw links as full duplex, half in each direction
                Vector3 m = (target.transform.position - source.transform.position)/2 + source.transform.position;
                lineRenderer.SetPosition(0, source.transform.position);
                lineRenderer.SetPosition(1, m);
 
                loaded = true;
            }
        }
    }
 
}

Being that this script was a bit larger, I’ll walk through what it does.  First, the public properties; the “id” is the link identifier (ie. “link_1”), the “source” will become the source node reference, the “target” will become the target node, the “sourceId” will hold the id of the source node until the “source” property is populated and the same goes for the “targetId”.  The “status” will hold a value of either “Up” or “Down” and will be used to color the LineRenderer.  Within the “Start()” function, we create a new LineRenderer and color it according to the “status” property value setting its initial line positions to vectors (0,0,0) and (1,0,0) respectively.  The “Update()” method waits until the “source” and “target” properties are populated then sets the LineRenderer end points.  The “loaded” property makes sure this only happens once.

Finally drag the “Link” object from the Hierarchy window to the “Assets/Prefabs” folder.  Then remove the “Link” object from the Hierarchy window.

Controller

Add a new empty game object (GameObject -> Create Empty) and name it “GameController”.  Create a new script called “GameController” (GameController.cs).  This script will be responsible for loading the layout file, creating Nodes and Links and handling general UI updates.

 

using UnityEngine;
using System.Collections;
using System.Xml;
using System.IO;
 
namespace Topology {
 
    public class GameController : MonoBehaviour {
 
        public Node nodePrefab;
        public Link linkPrefab;
 
        private Hashtable nodes;
        private Hashtable links;
        private GUIText statusText;
        private int nodeCount = 0;
        private int linkCount = 0;
        private GUIText nodeCountText;
        private GUIText linkCountText;
 
        //Method for loading the GraphML layout file
        private IEnumerator LoadLayout(){
 
            string sourceFile = Application.dataPath + "/Data/layout.xml";
            statusText.text = "Loading file: " + sourceFile;
 
            //determine which platform to load for
            string xml = null;
            if(Application.isWebPlayer){
                WWW www = new WWW (sourceFile);
                yield return www;
                xml = www.text;
            }
            else{
                StreamReader sr = new StreamReader(sourceFile);
                xml = sr.ReadToEnd();
                sr.Close();
            }
 
            XmlDocument xmlDoc = new XmlDocument();
            xmlDoc.LoadXml(xml);
 
            statusText.text = "Loading Topology";
 
            int scale = 2;
 
            XmlElement root = xmlDoc.FirstChild as XmlElement;
            for(int i=0; i<root.ChildNodes.Count; i++){
                XmlElement xmlGraph = root.ChildNodes[i] as XmlElement;
 
                for(int j=0; j<xmlGraph.ChildNodes.Count; j++){
                    XmlElement xmlNode = xmlGraph.ChildNodes[j] as XmlElement;
 
                    //create nodes
                    if(xmlNode.Name == "node"){
                        float x = float.Parse(xmlNode.Attributes["x"].Value)/scale;
                        float y = float.Parse (xmlNode.Attributes["y"].Value)/scale;
                        float z = float.Parse(xmlNode.Attributes["z"].Value)/scale;
 
                        Node nodeObject = Instantiate(nodePrefab, new Vector3(x,y,z), Quaternion.identity) as Node;
                        nodeObject.nodeText.text = xmlNode.Attributes["name"].Value;
 
                        nodeObject.id = xmlNode.Attributes["id"].Value;
                        nodes.Add(nodeObject.id, nodeObject);
 
                        statusText.text = "Loading Topology: Node " + nodeObject.id;
                        nodeCount++;
                        nodeCountText.text = "Nodes: " + nodeCount;
                    }
 
                    //create links
                    if(xmlNode.Name == "edge"){
                        Link linkObject = Instantiate(linkPrefab, new Vector3(0,0,0), Quaternion.identity) as Link;
                        linkObject.id = xmlNode.Attributes["id"].Value;
                        linkObject.sourceId = xmlNode.Attributes["source"].Value;
                        linkObject.targetId = xmlNode.Attributes["target"].Value;
                        linkObject.status = xmlNode.Attributes["status"].Value;
                        links.Add(linkObject.id, linkObject);
 
                        statusText.text = "Loading Topology: Edge " + linkObject.id;
                        linkCount++;
                        linkCountText.text = "Edges: " + linkCount;
                    }
 
                    //every 100 cycles return control to unity
                    if(j % 100 == 0)
                        yield return true;
                }
            }
 
            //map node edges
            MapLinkNodes();
 
            statusText.text = "";
        }
 
        //Method for mapping links to nodes
        private void MapLinkNodes(){
            foreach(string key in links.Keys){
                Link link = links[key] as Link;
                link.source = nodes[link.sourceId] as Node;
                link.target = nodes[link.targetId] as Node;
            }
        }
 
        void Start () {
            nodes = new Hashtable();
            links = new Hashtable();
 
            //initial stats
            nodeCountText = GameObject.Find("NodeCount").guiText;
            nodeCountText.text = "Nodes: 0";
            linkCountText = GameObject.Find("LinkCount").guiText;
            linkCountText.text = "Edges: 0";
            statusText = GameObject.Find("StatusText").guiText;
            statusText.text = "";
 
            StartCoroutine( LoadLayout() );
        }
 
    }
 
}

So now time for the walk through.  The 2 public properties nodePrefab and linkPrefab define the prefabs to be used when creating nodes and links.  The 2 Hastables, nodes and links define the collection which will hold the live instances of the prefabs “Node” and “Link”.  The GUIText objects will reference the loading status displayed on the screen, the count of nodes and the count of links.  The nodeCount and linkCount hold the actual numeric values for the count of nodes and links.  The “LoadLayout()” method handles the loading of the GraphML file, it loads the xml file into a string, populates an XmlDocument object and iterates over the <node/> and <edge/> elements.  When a <node /> is encountered we instantiate a Node object and populate its members and same goes for <edge /> with respect to populating a Link.  Once all the “Node” and “Link” objects have been created, the “source” and “target” members of the “Link” object are replaced with the living references of “Node” through the call to “MapLinkNodes()”.  The “Start()” method instantiates the nodes and links Hastables, resets the GUIText objects and starts a coroutine call to “LoadLayout()” which allows for returning control back to the Unity platform and preventing UI lock up.

Next, lets’ add the GUIText objects.  Create a GUIText object for LinkCount, NodeCount and StatusText at a minimum.  Next place the GUIText objects in the Game view to their relative positions.

Unity - Game View

Make these GUIText objects a child of GameController and add drag the references to the GameController script references.

Unity - GameController Hierarchy

Unity - GameController

 Camera

Click on the Main Camera object and add a new script “CameraController” (CameraController.cs).  In this script we’ll add basic controls for WASD, Control, Space and Wheel/Scroll controls.

 

using UnityEngine;
using System.Collections;
 
[AddComponentMenu("Camera-Control/Move ZeroG")]
public class CameraControlZeroG : MonoBehaviour {
 
    public float speed = 5f;
    public GUIText movementSpeed;
 
    private Vector3 move = new Vector3();
    private Vector3 cluster1 = new Vector3(1960, 1791, 2726);
    private Vector3 cluster2 = new Vector3(2042, 1579, 4254);
    private Vector3 cluster3 = new Vector3(2692, 81, 2526);
    private Vector3 cluster4 = new Vector3(531, 2317, 3776);
    private Vector3 cluster5 = new Vector3(-587, 2043, 2194);
 
    void Start(){
        //set to first cluster position
        transform.position = cluster1;
    }
 
    void Update () {
        move.x = Input.GetAxis("Horizontal") * speed * Time.deltaTime;
        move.z = Input.GetAxis("Vertical") * speed * Time.deltaTime;
 
        move.y = 0;
        if (Input.GetKey ("space")) {
            move.y = speed * Time.deltaTime;
        }
 
        if (Input.GetKey ("left ctrl")) {
            move.y = -speed * Time.deltaTime;
        }
 
        //adjust speed with mouse wheel
        speed += Input.GetAxis("Mouse ScrollWheel");
        if (speed < 5)
            speed = 5;
 
        movementSpeed.text = "Move Speed: " + speed;
 
        move = transform.TransformDirection(move);
        transform.position += move;
 
        //set warp to cluster controls
        if(Input.GetKey("1")){
            transform.position = cluster1;
        }
 
        if(Input.GetKey("2")){
            transform.position = cluster2;
        }
 
        if(Input.GetKey("3")){
            transform.position = cluster3;
        }
 
        if(Input.GetKey("4")){
            transform.position = cluster4;
        }
 
        if(Input.GetKey("5")){
            transform.position = cluster5;
        }
    }
}

Walking through the above code, we define 2 public properties, speed which will hold the speed of movement through the zero gravity environment and GUIText movementSpeed which displays our current movement speed.  Next we define a Vector3 for move which will be used as the movement vector.  The following cluster[1-5] members define areas of the Topology which we’ll allow the camera to quickly jump to, feel free to replace these with your own coordinates.  The “Start()” function places our camera at the first cluster position (the largest cluster).  The “Update()” function first updates the move vector to move the camera in a direction.  The “space” check will change our “y” movement upward.  The “left ctrl” changes our “y” movement downward.  The “Mouse ScrollWheel” changes our overall movement speed, then we display the speed in the GUIText.  The final modifications are made to the camera’s transform position.  The GetKey() calls for 1 through 5 warp our camera to the cluster[1-5] positions.

We’ll also want to import a standard asset from the Unity “Character Controller” set.  Click Assets -> Import Package -> Character Controller, deselect everything, then online select “MouseLook.cs”.

Unity - Import Asset

 

This will import the “MouseLook.cs” script which we’ll need to add to the Main Camera (Add Component -> Camera-Control -> Mouse Look).

Putting It All Together

Lastly, build, deploy and check it out.  One thing to keep in mind is that the layout.xml in the Data directory (/Assets/Data/layout.xml) does not get deployed with the build.  You will need to manually copy this file out to your build/deploy directory.  Hopefully you won’t have any errors and will be able to view the 3D topology in all of its beauty and splendor.  Have fun!

Understanding JavaScript Promises In jQuery

You may or may not have heard the new buzz about promises being introduced into JavaScript and if you don’t know what they are, you’re not alone.  I’ll attempt to explain what they are and why you should or maybe shouldn’t care about them.  First let’s understand what promises are.

Promise defined: “The Promise interface represents a proxy for a value not necessarily known at its creation time. It allows you to associate handlers to an asynchronous action’s eventual success or failure. This let asynchronous methods to return values like synchronous methods: instead of the final value, the asynchronous method returns a promise of having a value at some point in the future.”

Basically, it allows you to setup chaining for a value or values and determine a functional response.  For example, let’s say you need to get two values from two separate AJAX calls, but you don’t want to proceed unless both have loaded correctly and if either fails, you want to bail out of the entire thing.  Although not an optimum approach, this will serve as a more or less real world problem.  Consider the following code:

 

var first_call = false;
var second_call = false;
 
//make the first call
$.ajax({
    url: "/some/page",
    success: function(data){
        //set a success variable
        first_call = true;
    }
});
 
//make the second call
$.ajax({
    url: "/some/other-page",
    success: function(data){
        //set a success variable
        second_call = true;
    }
});
 
if(first_call && second_call){
    //we're good to go - do something
}
else{
    //ruh roh - bail out!
}

 

Now obviously, the above code isn’t fantastic but it will suit our needs for now.  Let’s walk through it.  The first ajax call if successful sets the first_call variable to true and the same goes for the second.  If all is good, we can proceed and do what we need to, otherwise we execute our bail out code.  So now let’s check out the same functional code executed with promises:

$.when( $.ajax("/some/page"), $.ajax("/some/other-page") )
    .done(function(first_call, second_call){
        //we're good to go - do something
    })
    .fail(function(){
        //ruh roh - bail out!
    });

So, looking at this latest example we see not only is the code much more compact, but now we can chain even more calls together, no use of outside variables for state checking and still we end up in the same place.  This is the power that promises provide you.  In the above example, we have implemented jQuery’s $.when call which will take a list of operations as the arguments.  For each operation, it will return the value into the done() function as the ordered arguments (ie. done(first_call_data, second_call_data, …)).  If a failure occurs for any of the calls, the fail() method is invoked where we can make use of our bail code and either try again, display an error message, etc.  If we wanted to add more calls to this promise we would do so by adding them directly to the when() function and creating the corresponding argument in the done() method.

$.when(
    $.ajax("/some/page"),
    $.ajax("/some/other-page"),
    $.ajax("/some/other-other-page"),
    $.ajax("/and/so/on")
    )
    .done(function(first_call, second_call, third_call, fourth_call){
        //we're good to go - do something
    })
    .fail(function(){
        //ruh roh - bail out!
    });

We can continue to chain $.when calls together to simply an otherwise complex system of dependencies.  Promises also provide us with a way of implementing our own custom promise behavior by taking advantage of methods such as .reject() .resolve() and .state().  However, this is a bit beyond the scope of this article.  For more information regarding jQuery promises have a look at:

Promise – http://api.jquery.com/promise/

Deferred Object – http://api.jquery.com/category/deferred-object/

D3 Drag And Drop

Implementing drag and drop in D3 (http://d3js.org/) is a pretty simple task when you know how to do it, the following is a walkthrough from setup to execution and the events along the way.

First thing we are going to want to do is setup our drag behavior.  This is the object that will be responsible for handling the actual drag and drop, events and will be added to the call chain of the objects of which we want to allow the drag and drop behavior.

var drag = d3.behavior.drag();

The above statement initializes a new instance of the drag behavior.  We can use it to bind this behavior to the call chain of other objects.  Let’s assume we want to allow all nodes having class “draggable” to have this drag and drop behavior.  To do this we simply add the drag behavior we just instantiated to the call chain of the objects:

var drag = d3.behavior.drag();
d3.selectAll(".draggable").call(drag);

It’s that simple, now all objects containing the “draggable” class will now have this behavior.  But what if we need to do something a bit more complex?  What if we need to have other things happen along the way, update other areas during a drag, initialize or change variables on start and end drag.

var drag = d3.behavior.drag()
    .on("dragstart", function(){
        //do some drag start stuff...
    })
    .on("drag", function(){
        //hey we're dragging, let's update some stuff
    })
    .on("dragend", function(){
        //we're done, end some stuff
    });

As you would expect, each on(…) call defines a hook into the event chain of the drag behavior allowing you to customize the behavior as you see fit.

Getting Started With Windows Azure Blob Storage Using Java

The following is a quick start guide to getting up and running with Windows Azure and Blob storage using Java.  First thing you’ll need to do is setup a Windows Azure account, you can do that here: https://www.windowsazure.com

Next you’ll need to download the Azure SDK for Java which is available here: https://www.windowsazure.com/en-us/develop/java/

Azure SDK Download

Azure SDK Download

I found it easier to simply download the libraries directly and include the azure jar file (microsoft-windowsazure-api-0.3.2.jar) directly.

Windows Azure Libraries

Windows Azure Libraries

Next you’ll need to capture your Account Name and Account Key.

Windows Azure Dashboard

Windows Azure Dashboard

Windows Azure Keys

Windows Azure Keys

Now for the fun part, the code.  I’ve already done a lot of the heavy lifting for you (you’re welcome) so all you’ll really need to do is modify a few values and implement as needed.  So the first thing you’ll need is to take the Key values from above and create an Azure connection string like so:

// Replace your-account-name and your-account-key with your account values
public static final String storageConnectionString = 
"DefaultEndpointsProtocol=http;" + 
"AccountName=your-account-name;" + 
"AccountKey=your-account-key";

Now let’s create a client which we’ll use to communicate with the Azure blob storage.
// Connection String
CloudStorageAccount storageAccount = CloudStorageAccount.parse(storageConnectionString);
 
// Create the client
CloudBlobClient blobClient = storageAccount.createCloudBlobClient();

Now that we have a client we can begin performing actions like upload, listing and retrieving files.  Let’s start with listing files.  You’ll need a root container name, this is essentially the name of the bucket that you’re going to be sticking your files in.  Now let’s list some files:
// Change this to your container name
String containerName = "your-container-name";
 
// Create the client
CloudBlobClient blobClient = storageAccount.createCloudBlobClient();
 
// Get a reference to the container
CloudBlobContainer container = blobClient.getContainerReference(containerName);
 
// Create the container if it does not exist
container.createIfNotExist();
 
// List them
for (ListBlobItem blobItem : container.listBlobs())
   System.out.println(blobItem.getUri());

So here’s what’s going on, we’re using the blob client we created which is looking at the container (or root bucket) and generating a list of the blobs there, if it container doesn’t exist we create it. Next we’re iterating the blobs and displaying the URI for each.  Now let’s say we had a folder structure on that container.  Something to the extent of an “images” directory with image files inside it.  Here’s how you’d list those files:

// Change this to your container name
String containerName = "your-container-name";
 
// Create the client
CloudBlobClient blobClient = storageAccount.createCloudBlobClient();
 
// Get a reference to the container
CloudBlobContainer container = blobClient.getContainerReference(containerName);
 
// Create the container if it does not exist
container.createIfNotExist();
 
// Get a reference to the images directory
CloudBlobDirectory directory = container.getDirectoryReference("images/");
 
// List them
for (ListBlobItem blobItem : directory.listBlobs())
   System.out.println(blobItem.getUri());

The only real difference between this example and the previous one is simply the list of either the container or the directory. In this case we get a reference to the cloud directory then list.  Now that we can list container and directory contents, let’s upload some files:

// Change to your file to upload
File source = new File("path/to/your/file");
 
// Where are we going to store the files?
String uri = "your-container-name/images";
 
// Change this to your container name
String containerName = "your-container-name";
 
// Create the client
CloudBlobClient blobClient = storageAccount.createCloudBlobClient();
 
// Retrieve reference to a previously created container
CloudBlobContainer container = blobClient.getContainerReference(uri);
 
// Create the container if it does not exist
container.createIfNotExist();
 
// Let's upload the file.
CloudBlockBlob blob = container.getBlockBlobReference(source.getName());
blob.upload(new FileInputStream(source), source.length());

The above example will upload the source file into the images directory of your container. The only thing left to do now is download a file and pattern is complete.

// Where are we going to save the downloaded file?
File destination = new File("path/to/your/file");
 
// Change this to your container name
String containerName = "your-container-name";
 
// Get a reference to the container
CloudBlobContainer container = blobClient.getContainerReference(containerName);
 
// Create the container if it does not exist
container.createIfNotExist();
 
// Get the actual Azure storage uri
CloudBlobDirectory directory = container.getDirectoryReference("images/");
String fileUri = String.format("%s/%s", directory.getUri(), destination.getName());
 
// Download it
blobItem.download(new FileOutputStream(destination));

And that’s the whole enchilada, hopefully this helped you save a little bit of time and got you up to speed quickly…and I’m out.

Web Developer Essential Tools

Foreword

The following highlights what I believe to be the best tools for web development and best of all, they’re all free.  This is a living reference which will be updated as new and better tools are found.  Feedback and suggestions are always welcome.

The List

Google Chrome

Google Chrome

Google Chrome

Light, extremely fast, jam packed with features and a debugger that makes short work of CSS and JavaScript.  The ability to right click inspect an element, change CSS values live in the DOM using up, down arrows or by entering values directly make this an essential tool for web development.

Get Google Chrome

ColorPic

ColorPic

ColorPic

Fantastic color picking tool which allows for obtaining the color values for anything on the screen.  Creation and loading of palettes, values can be slider modified for easy lighting or darkening of tones.  Shortcuts make color selection a cinch.  A must have tool.

Get ColorPic

Tape: Google Chrome Extension

Tape: Google Chrome Extension

Tape: Google Chrome Extension

You know that problem you have in CSS where you’re trying to line up elements either horizontally or vertically and you end up using paper to see if the alignment is correct, well you can forget about doing that ever again.  The Tape plugin allows you to easily place lines on the screen, then you can adjust your CSS to match.

Get Tape Extension

XML Tree: Google Chrome Extension

XML Tree: Google Chrome Extension

XML Tree: Google Chrome Extension

Display XML, RSS any form of node tree in a collapsible human readable format.

Get XML Tree Extension

Awesome Screenshot: Google Chrome Extension

Awesome Screenshot: Google Chrome Extension

Awesome Screenshot: Google Chrome Extension

Quickly screen grab what’s being displayed in the browser, draw, blur, annotate, write, etc. all from within the browser. Save when you’re done either locally or in the cloud.

Get Awesome Extension

LightTable

LightTable

LightTable

LightTable is one of the most robust and unique editors I have ever used.  Extremely powerful, easily customized with more features than I can list here.

Get LightTable

GIMP

GIMP

GIMP

If you’re used to Photoshop, it will take a little bit of a learning curve to get used to GIMP’s shortcuts and approach to image manipulation. However, once you get used to the shortcuts and image editing as a whole, you won’t want to go back. Select an area, resize the selection area by mouse or by shrink or grow amounts, copy then shortcut paste into a new image frame, fast. No more, click save, select extension type from a drop down, confirm, confirm, confirm. Instead, shortcut for save, enter filename with the extension (.png, .jpg, .gif, etc.), hit enter, enter, done. Made for rapid manipulation of images, tons of plugins, filters, great tool.

Get GIMP

Encryption and Decryption Between .NET and PHP

I recently worked on a project that required encryption and decryption by and between .NET and PHP. By default, the 2 technologies don’t mesh very well. Being that the data was originally being encrypted and decrypted by .NET, I had to write PHP code that worked with the encryption schemas being used. One of the main problems I ran into was the use of padding, in my case pkcs7 which was used by default in .NET. First thing to do was to make sure the encyption schemas were the same. For example, when using DES, the .NET default mode is MCRYPT_MODE_CBC. Once that was setup, I could initialize the mcrypt libraries.

 

$module = mcrypt_module_open(MCRYPT_DES, '', MCRYPT_MODE_CBC, '');
 
if($module === false)
die("DES module could not be opened");
 
$blockSize = mcrypt_get_block_size(MCRYPT_DES, MCRYPT_MODE_CBC);

 

The $blockSize variable is used later for padding and padding removal using pkcs7. Next to encrypt data I had to implement the following:

 

//encryption
$key = substr($key, 0, 8);
 
$iv = $key;
$rc = mcrypt_generic_init($module, $key, $iv);
 
//apply pkcs7 padding
$value_length = strlen($value);
$padding = $blockSize - ($value_length % $blockSize);
$value .= str_repeat( chr($padding), $padding);
 
$value = mcrypt_generic($module, $value);
$value = base64_encode($value);
mcrypt_generic_deinit($module);

 

//value now encrypted

Basically, the encryption scheme the .NET side was using was set the iv to the key, pad data, encrypt data, then base64 encode data. So here I’ve done the same thing in PHP. Now I needed to do the exact same thing for decryption:

//Decryption
$key = substr($key, 0, 8);
$iv = $key;
$rc = mcrypt_generic_init($module, $key, $iv); 
 
$value = base64_decode($value);
$value = mdecrypt_generic($module, $value); 
 
//apply pkcs7 padding removal
$packing = ord($value[strlen($value) - 1]);
if($packing && $packing < $this->_blockSize){
    for($P = strlen($value) - 1; $P >= strlen($value) - $packing; $P--){
        if(ord($value{$P}) != $packing){
            $packing = 0;
        }//end if
    }//end for
}//end if 
 
$value = substr($value, 0, strlen($value) - $packing); 
 
mcrypt_generic_deinit($module); 
 
//value now decrypted

This is basically the same as encryption but in reverse. The only real difference is the pkcs7 padding removal. Hopefully this tidbit helps a few others out there who run into encrypt and decryption issues between .NET and PHP.

Creating Fluid Fullscreen XHTML CSS Layouts

Making the transition from old school html to new school xhtml can be quite painful, especially if you’re not familiar with how to do this without the use of tables. In this article, I will discuss the techniques and ideas necessary to create fluid xhtml/css layouts. Let’s examine the following XHTML:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
    <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> 
 
    <head>
    <title></title>
    <meta http-equiv="Content-Type" content="text/xhtml;charset=utf-8" />
    <style type="text/css">@import "style.css";</style>
    </head> 
 
    <body> 
 
    <div class="container">
        <div class="header">
            <div class="header-content">Header</div>
        </div> 
 
        <div class="body">
            <div class="left-sidebar">
                <div class="left-content">left</div>
            </div>
            <div class="right-sidebar">
                <div class="right-content">right</div>
            </div>
            <div class="content">Content</div>
        </div> 
 
        <div class="footer">
            <div class="footer-content">Footer</div>
        </div>
    </div> 
 
    </body> 
 
    </html>

Now let’s begin dissecting it. The first line of the XHTML document is the doctype (document type) specifier. This tells the browser that the following markup is to adhere to the standards of XHTML strict, as defined by the W3C (World Wide Web Consortium www.w3.org). The xhtml strict doctype basically states that the document must be XML compliant and follow the rules of the doctype, the root element in the document must be “html” and the root element must contain an xml namespace (xmlns) attribute.

Now as you see, the document does in fact meet those rules. The first node in the document is “html” and it does have the xmlns attribute…BUT WHAT DOES IT ALL MEAN? Don’t worry, we’ll go line by line and break it down.

The first line in the document is the doctype, which we’ve all ready talked about, the second line is the “html” node. This has an XML namespace attribute on it that basically states that all the nodes defined in this document, are by default, XHTML. The xml:lang=”en” attribute specifies that the xml in the document is English in language as does the lang=”en” which specifies the language content of the document is also in English.

The next node is the “head” node, which is basically the same as in old school HTML. It contains a “title” node (document title), a “meta” node and a “style” node.

The “meta” node in this case is specifying the content type of the document as well as the encoding/charset type. The content type is pretty self expanitory, it’s xhtml/text, the encoding/charset for the document is Unicode (aka ISO 10646, for more information check out http://www.w3.org/International/tutorials/tutorial-char-enc/#Slide0040).

The “style” node is specifying where we are going to get the style or CSS from? In this case it’s a document called “style.css”. So what have it included in this way? The answer is, to ensure that the browser type reading this document is CSS1 compliant or better. We’ll check out the details of the CSS later, right now let’s just check out the XHTML.

Next we get to the guts of the operation, the “body” tag. Within the body is the basic layout of the page. Now, the idea behind this layout is to achieve a fluid layout that will take up the entire height of the page, without adding scrollbars and without the need for any “fixing” javascript to adjust the sizing for us. In short, we are defining the basic structure for a tableless layout that will give us a header, footer, left pane, content area and a right pane. This may be a little more that you’re looking for, but I figured this would be the best way to demonstrate this fluid layout technique.

This is basically what we want for this example:

XHTML Layout

XHTML Layout

So now the important part, if you haven’t already, you might want to take a moment and create an index.html page and copy the example XHTML document into it, then create a style.css page at the same level. It’s important to understand what’s going on as we make changes to the CSS and it’s a lot easier to understand if you can see the changes as they happen.

Now, let’s create the CSS to layout our XHTML document in the manner we want. First thing we need to do which may seem a bit odd if you’re first making the transition from HTML to XHTML is set the height of the “html” and “body” nodes. In HTML, these are already set to 100% height of the current window, however in XHTML they are not. So let’s add the following to our CSS:

html, body {
height: 100%;
width: 100%;
padding: 0px;
margin: 0px;
}

So now, we’ve set the “body” and “html” up to take up the entire screen and not pad or margin the edges. Ok, well that’s a good start, next let’s add a little background color and sizing to various parts of the document so we can see what’s where.

.header, .footer{
height: 80px;
background-color: #EFEFEF;
} 
 
.body{
background-color: #C7DFFA;
} 
 
.left-sidebar{
background-color: gray;
} 
 
.right-sidebar{
background-color: gray;
}

Ok, so now let’s look at what we have. YIKES!

XHTML Example

XHTML Example

At this point you should have something similar to the above example. We can see the individual divs, but we’ve got some work to do to get them layed out like we want.

The first thing we’re going to want to do is set the container to be 100% in height, that way we can have the full height of the window to layout our divs.

.container{
height: 100%;
}

Next, let’s set the height of our body class “.body” (not the “body” tag) to 100%. We want this area to expand and contract with the window and still take up the entire height of the screen. Let’s also just for clarity, hide the left and right sidebars (we’ll show them again later)

.body{
height: 100%;
background-color: #C7DFFA;
} 
 
.left-sidebar{
display: none;
} 
 
.right-sidebar{
display: none;
}

XHTML Example

XHTML Example

Well, that’s a little better, but there are scrollbars on the window now. We wanted 100% of the available screen height but instead of we got 100% of the window height, what’s going on? This is exactly what’s supposed to happen in XHTML the height is based up the actual window height, not the available height. So to fix this we need to do a little finessing. He’re what we need to do. We need to set the top and bottom margins of the “body” class (not the “body” tag) to be negative the height of the header and footer respectively. Being that our header and footer are the same height (80px) we need to subtract that area from the “body” class (not the “body” tag). Like so:

.body{
height: 100%;
margin-top: -80px;
margin-bottom: -80px;
background-color: #C7DFFA;
}

XHTML Example

XHTML Example

Well, that took care of the height problem, but wait…what happened to the header, it’s gone. Actually, it’s behind the “body” class (not the “body” tag), we need to bring it forward. To do that we set the position to relative. This allows us the position the tag using coordinates if we want and forces the browser to paint these above other non-relative/absolute layers.

.header, .footer{
height: 80px;
background-color: #EFEFEF;
position: relative;
}

Ahhhh, that’s better. The “header” and “footer” are visible, the “body” class (not “body” tag) are acting correctly. You may have also noticed at this point that the “content” is not longer visible within the “body” class (not the “body” tag). Don’t worry about that, we’ll address that later. So now, we can resize the window all we want and the height is correct, no scrollbars, life is peachy. Now let’s tackle those sidebars. Let’s start by making them visible again.

.left-sidebar{
background-color: gray;
} 
 
.right-sidebar{
background-color: gray;
}

XHTML Example

XHTML Example

Wait, where are they? They’re still not visible, let’s make them visible and put them in the right spot. First let’s set the height to 100%, next let’s give it a width of 100px, then let’s set to the left by using float left.

.left-sidebar{
height: 100%;
background-color: gray;
width: 100px;
float: left;
}

XHTML Example

XHTML Example

GEORGEOUS! That’s what we want, now let’s do the same to the other side but float it right.

XHTML Example

XHTML Example

PERFECTION! Now all we have left to do is situate our content layers properly and we’re done. So let’s start with the “left-content”. To make the content of this layer visible we need to pad enough to account for the header and the footer. Remember, we set the height of this div to 100%, so the top and the bottom are hidden behind the “header” and “footer”. So let’s pad the top by 100px, bottom by 100px, left by 10px and right by 10px, that should look nice. Also, let’s apply this to both the left and right content areas. I’m going to also set the “right-content” to text align right, just for looks.

.left-content{
padding: 100px 10px 100px 10px;
} 
 
.right-content{
padding: 100px 10px 100px 10px;
text-align: right;
}

XHTML Example

XHTML Example

Looking good, now let’s finish up by getting the “content” visible and formatted. So let’s do basically the same thing and pad the “content” div, let’s set the top padding to 100px, left to 120px (100px plus 20px for looks), bottom to 100px and right to 120px.

.content{
padding: 100px 120px 100px 120px;
}

And while we’re at it, why not format the header and footer content nicely too.

.header-content{
padding: 20px;
text-align: center;
} 
 
.footer-content{
padding: 20px;
text-align: center;
}

XHTML Example

XHTML Example

Now we’re done, we have a fully fluid layout that takes up 100% of the height and width. The content and sides resize correct. We only get scrollbars when the content is crushed, SUCCESS! Hopefully this helps some of you out there, belows is the complete CSS just in case you need it or the examples didn’t work out for you.

html, body {
    height: 100%;
    width: 100%;
    padding: 0px;
    margin: 0px;
    } 
 
    .header, .footer{
    height: 80px;
    background-color: #EFEFEF;
    position: relative;
    } 
 
    .header-content{
    padding: 20px;
    text-align: center;
    } 
 
    .footer-content{
    padding: 20px;
    text-align: center;
    } 
 
    .container{
    height: 100%;
    } 
 
    .body{
    height: 100%;
    margin-top: -80px;
    margin-bottom: -80px;
    background-color: #C7DFFA;
    } 
 
    .content{
    padding: 100px 120px 100px 120px;
    } 
 
    .left-sidebar{
    height: 100%;
    background-color: gray;
    width: 100px;
    float: left;
    } 
 
    .right-sidebar{
    height: 100%;
    background-color: gray;
    width: 100px;
    float: right;
    } 
 
    .left-content{
    padding: 100px 10px 100px 10px;
    } 
 
    .right-content{
    padding: 100px 10px 100px 10px;
    text-align: right;
    }

Optimizing memcpy Routines

First let’s understand what the problem is. Memcpy copies memory one byte at a time. Which is fine as long as you don’t have a lot of memory to copy. If you find yourself needing to copy large chunks of memory from one spot to the next you may very well find yourself loosing a lot of process time in the memcpy routine. One possible way to optimize this call is to define a copy routine of your own that is optimized for the data types you need to copy. For example, let’s say that you have data that will always be in 8 byte sections. A good way to optimize this would be to copy the data 8 bytes at a time versus 1 byte at a time via memcpy. Consider the following code:

void uint8copy(void *dest, void *src, size_t n){
    uint64_t * ss = (uint64_t)src;
    uint64_t * dd = (uint64_t)dest;
    n = n * sizeof(uint8_t)/sizeof(uint64_t); 
 
    while(n--)
        *dd++ = *ss++;
}//end uint8copy()

The code is pretty simple, it takes in a dest, src, and n (size) the same as memcpy. But, it converts the src and dest pointers to uint64_t or 64 bit pointers before making the copy. Next it determines how many times to iterate and begins copying data 64 bits or 8 bytes at a time. This approach works well for this particular scenario and may assist in being a starting point for your code as well.

Finding The Fastest Write Method

I was recently tasked to develop a system that would need to write data in the Gigabytes per second range directly to disk. While doing so, I needed to spec out and build a machine that allowed for that type of bandwidth to be persisted as well as find the most optimized filesystem and write method available in C. Once the specs were done and the machine arrived, it was time to determine which filesystem and write method under Linux would give the best results for writing. I initially tested with bonnie++, but soon became aware that the results I was receiving were not a true test of the raid configuration and filesystem, but were instead block writes and reads. For my purposes I need to benchmark the write speed of several different writing types including asynchronous writing, standard IO and generic (open,write,read) IO. To do so, I created a simple writetest program:

#include <aio.h>
#include <time.h>
#include <stdio.h>
#include <malloc.h>
#include <stdlib.h>
#include <string.h>
#include <errno.h>
#include <sys/stat.h>
#include <sys/mman.h>
#include <math.h> 
 
int nVersionMajor = 1;
int nVersionMinor = 0;
int nVersionBuild = 1;
int64_t mb = 1048576;
char szOutputDir[1024];
long nSize = 0;
int nIteration;
short nUseAio;
short nUseStd;
short nUseGen;
unsigned long long nStdioAvg = 0;
unsigned long long nAioAvg = 0;
unsigned long long nGenAvg = 0;
double nSec = 1000000.0; 
 
char * pBytes; 
 
void stdioWrite(char * filename){
    FILE * pFile;
    struct timeval tvStart;
    struct timeval tvEnd;
    int64_t nEnd;
    int64_t nStart; 
 
    gettimeofday(&tvStart);
    nStart = tvStart.tv_sec * 1000000 + tvStart.tv_usec; 
 
    pFile = fopen(filename,"w");
    if(!pFile){
        fprintf(stderr, "[Error] could not open %s for writing.n", filename);
        exit(1);
    }//end if 
 
    fwrite(pBytes, strlen(pBytes), 1, pFile);     
 
    fclose(pFile); 
 
    gettimeofday(&tvEnd);
    nEnd = tvEnd.tv_sec * 1000000 + tvEnd.tv_usec; 
 
    nStdioAvg += (nEnd-nStart);
    printf("stdio->fopen,fwrite,fclose t%f sec.n", (nEnd-nStart)/nSec);
}//end stdioWrite() 
 
void aioWrite(char * filename){
    struct aiocb aio;
    int outFile;
    struct timeval tvStart;
    struct timeval tvEnd;
    int64_t nEnd;
    int64_t nStart; 
 
    gettimeofday(&tvStart);
    nStart = tvStart.tv_sec * 1000000 + tvStart.tv_usec; 
 
    outFile = open(filename, O_WRONLY|O_CREAT, 0666); 
 
    if(outFile == -1){
        fprintf(stderr, "[Error] Could not open %s for writing.n", filename);
        exit(1);
    }//end if 
 
    aio.aio_offset = 0;
    aio.aio_fildes = outFile;
    aio.aio_reqprio = 0;
    aio.aio_buf = pBytes;
    aio.aio_nbytes = strlen(pBytes);
    aio.aio_sigevent.sigev_notify = SIGEV_NONE; 
 
    aio_write(&aio); 
 
    while(aio_error(&aio) == EINPROGRESS);
    close(outFile); 
 
    gettimeofday(&tvEnd);
    nEnd = tvEnd.tv_sec * 1000000 + tvEnd.tv_usec; 
 
    nAioAvg += (nEnd-nStart);
    printf("aio->open,aio_write,close t%f sec.n", (nEnd-nStart)/nSec);
}//end aioWrite() 
 
void genWrite(char * filename){
    int outFile;
    struct timeval tvStart;
    struct timeval tvEnd;
    int64_t nEnd;
    int64_t nStart; 
 
    gettimeofday(&tvStart);
    nStart = tvStart.tv_sec * 1000000 + tvStart.tv_usec; 
 
    outFile = open(filename, O_WRONLY|O_CREAT, 0666); 
 
    if(outFile == -1){
        fprintf(stderr, "[Error] Could not open %s for writing.n", filename);
        exit(1);
    }//end if 
 
    write(outFile, pBytes, strlen(pBytes)); 
 
    close(outFile); 
 
    gettimeofday(&tvEnd);
    nEnd = tvEnd.tv_sec * 1000000 + tvEnd.tv_usec; 
 
    nGenAvg += (nEnd-nStart);
    printf("gen->open,write,close     t%f sec.n", (nEnd-nStart)/nSec);
}//end genWrite() 
 
void displayUsage(){
    printf("writetest - GodLikeMouse file write testing Version %d.%d.%dn", nVersionMajor, nVersionMinor, nVersionBuild);
    printf("tCopyright 2008 GodLikeMouse (www.GodLikeMouse.com)n");
    printf("n");
    printf("Usage:n");
    printf("tnqpcapd [options]n");
    printf("n");
    printf("Options:n");
    printf("t--output-dir [directory]n");
    printf("ttThe directory to write to for testing (default .test).n");
    printf("t--i [iterations]n");
    printf("ttThe amount of times to write (default 1).n");
    printf("t--mb [megabytes to write]n");
    printf("ttThe size of the files to write in megabytes.n");
    printf("t--b [bytes to write]n");
    printf("ttThe size of the files to write in bytes.n");
    printf("ttwritten (default /data).n");
    printf("t--stdion");
    printf("ttUse fopen,fwrite,fclose.n");
    printf("t--aion");
    printf("ttUse open,aio_write,close.n");
    printf("t--genion");
    printf("ttUse open,write,close.n");
    printf("t--helpn");
    printf("ttDisplay this help message.n");
    printf("t--versionn");
    printf("ttDisplay the version information.n");
    printf("n");
}//end displayUsage() 
 
void parseArgs(int argc, char ** argv){
    int i; 
 
    for(i=0; i<argc; i++){ 
 
        if(strstr(argv[i], "--output-dir")){
            sprintf(szOutputDir, "%s", argv[++i]);
            continue;
        }//end if 
 
        if(strstr(argv[i], "--mb")){
            nSize = mb * atol(argv[++i]);
            continue;
        }//end if 
 
        if(strstr(argv[i], "--b")){
            nSize = atol(argv[++i]);
            continue;
        }//end if 
 
        if(strstr(argv[i], "--help")){
            displayUsage();
            exit(0);
        }//end if 
 
        if(strstr(argv[i], "--i")){
            nIteration = atoi(argv[++i]);
            continue;
        }//end if 
 
        if(strstr(argv[i], "--aio")){
            nUseAio = 1;
            continue;
        }//end if 
 
        if(strstr(argv[i], "--stdio")){
            nUseStd = 1;
            continue;
        }//end if 
 
        if(strstr(argv[i], "--genio")){
            nUseGen = 1;
            continue;
        }//end if 
 
        if(strstr(argv[i], "--version")){
            printf("writetest - GodLikeMouse file write testing Version %d.%d.%dn", nVersionMajor, nVersionMinor, nVersionBuild);
            exit(0);
        }//end if
    }//end for
}//end parseArgs() 
 
void printSeparator(){
    printf("---------------------------------------------n");
}//end printSeparator() 
 
void printAverages(){
    double nTemp; 
 
    if(nUseStd){
        nTemp = ((double)nStdioAvg/nIteration)/nSec;
        printf("stdio average write time: t%f sec.n", nTemp);
        printf("stdio average throughput: t%.0f bytes/sec ", nSize/nTemp);
        printf("%.0f MB/secn", (nSize/nTemp)/mb);
    }//end if 
 
    if(nUseAio){
        nTemp = ((double)nAioAvg/nIteration)/nSec;
        printf("aio average write time: t%f sec.n", nTemp);
        printf("aio average throughput: t%.0f bytes/sec ", nSize/nTemp);
        printf("%.0f MB/secn", (nSize/nTemp)/mb);
    }//end if 
 
    if(nUseGen){
        nTemp = ((double)nGenAvg/nIteration)/nSec;
        printf("gen average write time: t%f sec.n", nTemp);
        printf("gen average throughput: t%.0f bytes/sec ", nSize/nTemp);
        printf("%.0f MB/secn", (nSize/nTemp)/mb);
    }//end if
}//end printAverages() 
 
int main(int argc, char ** argv){
    int i;
    char szFile[2048]; 
 
    nIteration = 1;
    nSize = 1024;
    strcpy(szOutputDir, ".test");
    nUseStd = 0;
    nUseAio = 0;
    nUseGen = 0; 
 
    parseArgs(argc, argv); 
 
    printf("n");
    printf("Beginning cycle writen");
    printf("Writing %ld bytes, %ld MBn", nSize, (nSize/mb)); 
 
    printSeparator(); 
 
    for(i=0; i<nIteration; i++){
        if(nUseStd){
            pBytes = (char *)malloc(nSize);
            memset(pBytes, 'X', nSize);
            sprintf(szFile, "%s/%s.%d", szOutputDir, "stdio", i);
            stdioWrite(szFile);
            free(pBytes);
        }//end if 
 
        if(nUseAio){
            pBytes = (char *)malloc(nSize);
            memset(pBytes, 'X', nSize);
            sprintf(szFile, "%s/%s.%d", szOutputDir, "aio", i);
            aioWrite(szFile);
            free(pBytes);
        }//end if 
 
        if(nUseGen){
            pBytes = (char *)malloc(nSize);
            memset(pBytes, 'X', nSize);
            sprintf(szFile, "%s/%s.%d", szOutputDir, "gen", i);
            genWrite(szFile);
            free(pBytes);
        }//end if 
 
        printSeparator();
    }//end for 
 
    printf("n");
    printAverages();
    printf("n");
    printf("n"); 
 
    printf("Beginning sequential writen");
    printf("Writing %ld bytes, %ld MBn", nSize, (nSize/mb)); 
 
    printSeparator(); 
 
    nStdioAvg = 0;
    nAioAvg = 0;
    nGenAvg = 0; 
 
    if(nUseStd){
        for(i=0; i<nIteration; i++){
            pBytes = (char *)malloc(nSize);
            memset(pBytes, 'X', nSize);
            sprintf(szFile, "%s/%s.%d", szOutputDir, "stdio", i);
            stdioWrite(szFile);
            free(pBytes);
        }//end for
        printSeparator();
    }//end if 
 
    if(nUseAio){
        for(i=0; i<nIteration; i++){
            pBytes = (char *)malloc(nSize);
            memset(pBytes, 'X', nSize);
            sprintf(szFile, "%s/%s.%d", szOutputDir, "aio", i);
            aioWrite(szFile);
            free(pBytes);
        }//end for
        printSeparator();
    }//end if 
 
    if(nUseGen){
        for(i=0; i<nIteration; i++){
            pBytes = (char *)malloc(nSize);
            memset(pBytes, 'X', nSize);
            sprintf(szFile, "%s/%s.%d", szOutputDir, "gen", i);
            genWrite(szFile);
            free(pBytes);
        }//end for
        printSeparator();
    }//end if 
 
    printf("n");
    printAverages();
    printf("n"); 
 
}//end main()

This simple program took in a set of parameters to determine which write methods to use to write, how much data, how many times and where to write it. Using this I began testing all the available free filesystems I could find for Linux. The fastest write speeds were given by XFS with the following make and mount options.

root@localhost:$ mkfs.xfs -l size=32m -d agcount=4 /dev/sda4
mount /dev/sda4 /data -o noatime,nodiratime,osyncisdsync

The first round metrics I received using this method to test are as follows:

root@localhost:$ writetest # src/writetest --output-dir /data/ --stdio --aio --genio --i 3 --mb 500

Beginning cycle write
Writing 524288000 bytes, 500 MB
---------------------------------------------
stdio->fopen,fwrite,fclose 1.047127 sec.
aio->open,aio_write,close 5.724001 sec.
gen->open,write,close 2.580516 sec.
---------------------------------------------
stdio->fopen,fwrite,fclose 2.063382 sec.
aio->open,aio_write,close 5.954485 sec.
gen->open,write,close 1.351529 sec.
---------------------------------------------
stdio->fopen,fwrite,fclose 1.729958 sec.
aio->open,aio_write,close 6.457802 sec.
gen->open,write,close 1.754718 sec.
---------------------------------------------

stdio average write time: 1.613489 sec.
stdio average throughput: 324940548 bytes/sec 310 MB/sec
aio average write time: 6.045429 sec.
aio average throughput: 86724693 bytes/sec 83 MB/sec
gen average write time: 1.895588 sec.
gen average throughput: 276583357 bytes/sec 264 MB/sec

Beginning sequential write
Writing 524288000 bytes, 500 MB
---------------------------------------------
stdio->fopen,fwrite,fclose 1.479412 sec.
stdio->fopen,fwrite,fclose 1.597659 sec.
stdio->fopen,fwrite,fclose 1.851250 sec.
---------------------------------------------
aio->open,aio_write,close 6.575862 sec.
aio->open,aio_write,close 7.003166 sec.
aio->open,aio_write,close 6.991679 sec.
---------------------------------------------
gen->open,write,close 1.465857 sec.
gen->open,write,close 0.980143 sec.
gen->open,write,close 0.841066 sec.
---------------------------------------------

stdio average write time: 1.642774 sec.
stdio average throughput: 319148043 bytes/sec 304 MB/sec
aio average write time: 6.856902 sec.
aio average throughput: 76461349 bytes/sec 73 MB/sec
gen average write time: 1.095689 sec.
gen average throughput: 478500888 bytes/sec 456 MB/sec

Writing 500MB of data directly to disk using stdio (fopen,fwrite,fclose) aio (open,aio_write,close) and genio (open,write,close) for 3 iterations yeilded 456 when calling open,write,close consecutively; which is great, it meant I’m on the right track. After a few more tweaks and using genio (open,write,close) I started seeing:

root@localhost:$ src/writetest --output-dir /data/ --genio --i 3 --mb 500

Beginning cycle write
Writing 524288000 bytes, 500 MB
---------------------------------------------
gen->open,write,close 0.804808 sec.
---------------------------------------------
gen->open,write,close 0.850169 sec.
---------------------------------------------
gen->open,write,close 0.867514 sec.
---------------------------------------------

gen average write time: 0.840830 sec.
gen average throughput: 623536021 bytes/sec 595 MB/sec

Beginning sequential write
Writing 524288000 bytes, 500 MB
---------------------------------------------
gen->open,write,close 0.855086 sec.
gen->open,write,close 0.947534 sec.
gen->open,write,close 0.842693 sec.
---------------------------------------------

gen average write time: 0.881771 sec.
gen average throughput: 594585215 bytes/sec 567 MB/sec

Perfect, now I getting where I need to go and I know that XFS with the options I specified along with using the generic open,read,write approach would give me the best write times. Feel free to take the above writetest program and use it to tune your filesystem and to make sure that you’re using the fastest possible write method for your chosen filesystem. Be sure to compile with -laio. I thought these findings were worth mentioning.

**************** Update ********************

After changing the OS from a 32-bit to 64-bit system and with a few additional modifications to the kernel, new and even more impressive speeds have been reached

root@localhost:$ src/writetest --mb 500 --i 10 --output-dir /data --genio

Beginning cycle write
Writing 524288000 bytes, 500 MB
---------------------------------------------
gen->open,write,close 0.460466 sec.
---------------------------------------------
gen->open,write,close 0.493884 sec.
---------------------------------------------
gen->open,write,close 0.494179 sec.
---------------------------------------------
gen->open,write,close 0.492141 sec.
---------------------------------------------
gen->open,write,close 0.491546 sec.
---------------------------------------------
gen->open,write,close 0.490171 sec.
---------------------------------------------
gen->open,write,close 0.490388 sec.
---------------------------------------------
gen->open,write,close 0.489386 sec.
---------------------------------------------
gen->open,write,close 0.489673 sec.
---------------------------------------------
gen->open,write,close 0.491476 sec.
---------------------------------------------

gen average write time: 0.488331 sec.
gen average throughput: 1073632434 bytes/sec 1024 MB/sec

Beginning sequential write
Writing 524288000 bytes, 500 MB
---------------------------------------------
gen->open,write,close 0.484176 sec.
gen->open,write,close 0.489009 sec.
gen->open,write,close 0.491489 sec.
gen->open,write,close 0.487558 sec.
gen->open,write,close 0.489555 sec.
gen->open,write,close 0.486413 sec.
gen->open,write,close 0.492973 sec.
gen->open,write,close 0.489842 sec.
gen->open,write,close 0.487276 sec.
gen->open,write,close 0.489895 sec.
---------------------------------------------

gen average write time: 0.488819 sec.
gen average throughput: 1072561478 bytes/sec 1023 MB/sec

CSS From A Single Source

When you have the need to write browser specific CSS but don’t want to resort to serving up multiple CSS files from the server, scripting styles or inlining tag styles, keep this in mind. There is a way to write browser specific CSS from a single CSS file and here’s how you do it. Let’s assume that you have a style that you only want picked up by Internet Explorer and not a Mozilla flavored browser. For testing purposes, let’s change the color of text to red for IE:

/* Assume this is a style block */ 
 
P{
#color: red;
_color: red;
}

If you haven’t seen this syntax before, you’re probably thinking…what the hell is that? The “#” and “_” sign prefixing styles will only be picked up by Internet Explorer browsers, Mozilla based browsers will discard the styles altogether and move on. The reason we use both the “#” and the “_” is due to Internet Explorer versions, certain versions will only acknowledge the “#” sign, while others only see the “_” sign. So to be safe we use both. Now try the above code in your own page, create a paragraph tag, stick some text in it and open it with Internet Explorer. Booyah! red text, so now what about Mozilla flavored browsers:

/* Assume this is a style block */ 
 
P{
color: green;
#color: red;
_color: red;
}

In CSS we have the general rule of “last one wins”. In this case Internet Explorer will see the “color: green;” style and apply it, then depending on the version of IE, it will see on of the “#” or “_” prefixed styles and apply that. Being that last one wins, the paragraph block will be styled red. Mozilla on the otherhand will only see the “color: green;” style, then completely discard the “#” and “_” prefixed styles, leaving the Mozilla paragraph block green. Try it yourself in both browsers and you’ll see what I mean, hope this helps the next time you need to write browser specific CSS.

Another way to directly target a specific browser using only CSS is by use of Browser Specific class selectors. Browser specific class selectors work very much in the same way as CSS hacks, but can offer more direct browser targetting and feature selection. Take for an example, the following HTML:

<div class="example"></div> 
 
<style type="text/css">
.ie .example {
background-color: yellow
}
.ie7 .example {
background-color: orange
}
.gecko .example {
background-color: gray
}
.win .gecko .example {
background-color: red
}
.linux .gecko .example {
background-color: pink
}
.opera .example {
background-color: green
}
.konqueror .example {
background-color: blue
}
.webkit .example {
background-color: black
}
.example {
width: 100px;
height: 100px;
}
.no_js { display: block }
.has_js { display: none }
.js .no_js { display: none }
.js .has_js { display: block }
</style>

Each class selector specifically targets a particular browser or feature of the browser.