carmack on Java

by Damon 29. March 2005 06:00
I don't think I'd be out of line to say that most of the Java community is adamantly anti-Microsoft. Microsoft is a big evil corporation that does Bad Things™ and the .NET platform is inferior because of this.

I used to write Java; it was, at the time, a good career progression from the C++ work I was doing. I still know tons of Java developers and I've even defended Java as having its merits (and taken a beating for it) to some MSFT folks. The sheer childishness of some people on both sides in regards to trashing the other platform is funny sometimes but most often sad. Technology platform may as well join Religion and Politics as things you don't discuss among friends.

Many times, I've tried to tell a friend about a feature of the .NET platform that is so cool it makes me want to break into song, like designer support. After all this feature is very Component Engineering friendly, and should be a big turn-on to the average Java dude. However, their ears are sealed because MSFT made it. Your loss, my friends.

Despite not being a .NET developer John Carmack is my hero. Technology is his life and despite being a huge fan of LINUX and OpenGL and various non-Microsoft platforms he is 100% fair in his statements. You could say he's equally critical of everything. I respect the fact that despite not being a big MSFT fan Carmack uses Visual Studio to develop because he feels it is without compare as a tool for any serious developer.

On his blog, he finally changed to blogging from .plan updates, he recently described his experiences working with Java. He talks about the woefull inadequecies of the platform where performance is concerned and comes right out and says "Write once and run anywhere" is a joke, especially for mobile platforms. Being a bit of a mobile developer myself, I appreciated his comments. Check them out here.

Tags:

CF Message Queue

by Damon 17. March 2005 06:00
Moral of the story: If you're going to write an app with some threading, test with 3 or more threads at least, even if you don't plan on using more than 2. In .NET, you can get away with murder if you only have 2 threads. Problems with your threading code are much more apparent when 4 threads are in contention for the same resource. In this scenario that resource is the UI message loop.

I support a mobile project that synchronizes a great deal of data over the internet (60meg all told) between a Compact Framework windows app and a PHP/PERL/BSD backend. MSFT tools were not an option for the backend, so using RDA Pull commands to get data is out. The syncrhonization is done via web services, essentially the server sends a DataSet friendly format back and it is merged locally. They recently asked if the sync could be made faster. My first reply was "Yeah, if you could tell me which rows are Inserts vs. Updates so I don't have to do a row-by-row merge" but apparently that is not going to happen either. While working on a completely different application and trying to fight through "medicine head" brought on by DayQuil a good idea came to me.

Like many good ideas it was simple and obvious. The current sync process was:
  1. Ask for data
  2. Get DataSet back from server
  3. Process DataTables row by row (very time consuming in SQL CE 2.0)
  4. Ask for more data
Lather, rinse, repeate. I had already threaded out the Sync process so the UI could keep drawing itself during this time, but now it was clear that I could also implement sort of a local Message Queue so each individual bundle of data could be processed Asynchronous to the other thread which was actually pulling the data from the server. So, I came up with a simple class to queue and dequeue messages, came up with classes and interfaces to represent a Queue Work Item and now performance is considerably better. Messages are being processed while the next message is fetched from the server, so the processor on the mobile device and the bandwidth it can suck from the server are both maxed out at all times.

The actual work queue itself is not rocket science either. Here's the code, complete with all my debug nonsense:

using System;

using System.Collections;

using System.Threading;

using OpenNETCF.Windows.Forms;

namespace Mobile.Sync

{

/// <summary>

/// Summary description for SyncWorkItemQueue.

/// </summary>

public class SyncWorkItemQueue

{

private object _syncRoot;

private System.Collections.Queue _queue;

private bool _run;

public const int QUEUE_THRESHOLD = 50;

public const int CATCH_UP_THRESHOLD = 8;

public SyncWorkItemQueue()

{

_syncRoot = new object();

_queue = new Queue();

_run = true;

}

/// <summary>

/// Default to true

/// </summary>

public bool Run

{

get

{

lock(_syncRoot)

{

return _run;

}

}

set

{

lock(_syncRoot)

{

_run = value;

}

ApplicationEx.DoEvents();

}

}

/// <summary>

/// Get the number of SyncWorkItem s in the queue

/// </summary>

public int ItemCount

{

get

{

lock(_syncRoot)

{

return _queue.Count;

}

}

}

/// <summary>

/// (Synchronized) Add an item to the end of the queue

/// </summary>

/// <param name="workItem"></param>

public void AddToQueue(SyncWorkItem workItem)

{

lock(_syncRoot)

{

Console.WriteLine("Adding: " +workItem);

_queue.Enqueue(workItem);

}

ApplicationEx.DoEvents();

}

public void StartProcessing()

{

ThreadStart threadStart = new ThreadStart(StartWork);

Thread workerThread = new Thread(threadStart);

workerThread.Start();

}

protected void StartWork()

{

while(Run)

{

System.Windows.Forms.Application.DoEvents();

if (_queue.Count > 0 )

{

Console.WriteLine("Processing Item, Count=" + _queue.Count);

SyncWorkItem workItem = (SyncWorkItem)_queue.Dequeue();

try

{

workItem.Updater.UpdateTable();

}

catch(Exception ex)

{

Console.WriteLine(ex);

Console.WriteLine("Error processing work item");

Console.WriteLine(workItem.Updater);

ApplicationEx.DoEvents();

}

Console.WriteLine("Processed: " + workItem.Updater);

}

//Allow everything to update

ApplicationEx.DoEvents();

//Now check for greater than threshold amount, to let everything catch up

if (_queue.Count > QUEUE_THRESHOLD)

{

Console.WriteLine("Catching up");

lock(_syncRoot)

{

for(int i = 0; i < CATCH_UP_THRESHOLD; ++i)

{

SyncWorkItem workItem = (SyncWorkItem)_queue.Dequeue();

workItem.Updater.UpdateTable();

Console.WriteLine("CATCH_UP_THRESHOLD item processed, Count=" + _queue.Count);

ApplicationEx.DoEvents();

}

}

ApplicationEx.DoEvents();

}

}

Console.WriteLine("Ending Startwork");

}

}

}


At the end, if there are work items to process, I just do the following to let the sync thread catch up:

while (_syncQueue.ItemCount > 0)

{

Thread.Sleep(10);

}

_syncQueue.Run = false;



From where I'm standing, this is not too hard to do. I did run into a thread race condition while implementing this though. Even so, I still don't know why the average developer/architect is so afraid of using threading. The dreaded thread race issue was actually relatively easy to track down using the debugger: just hit "Pause" and it displayed all my threads and I could clearly see what thread was blocking where. In my case, the issue was sloppy coding. At one point the sync was not on its own thread and it directly updated the UI. When I put it on its own thread I did not change these messages to fire through Control.Invoke(delegate); I got away with it until more threads and were added to the mix, thus my comment about getting away with murder when you have only two threads: Application.DoEvents is often enough to resolve a lockup but not once you get beyond 2 threads.

Tags:

Headphones

by Damon 12. March 2005 06:00
I really like coding with headpones on. Obviously there are phases of a project where human interaction is constant and necessary, but there are also phases of the project where you just need to get some work done. During those times, music is my best friend. From Chopin to Chemlab, music helps me focus somehow.

After many years of loyal service my ancient Sony headphones broke recently. I ended up settling for the Senheiser HD 212 phones shown here. Next week I can get some serious work done now.

The refactorings needed for the TRAP designer are done, as is all the other config coded. The next step is simply to get the Engine and Provider Assemblies generating SQL. Most of the Provider code is already written. I also added Log4N as the logger last night.

The designer writeup is in the works, but its going to take a while. I need to do work on three other projects this weekend.

Tags:

Designer Support

by Damon 8. March 2005 06:00
I have not been idle with the TRAP project. I merely hit a snag this weekend that I have not had time to unsnag. I have been working on the visual interface to creating the mapping files. Given that this is a code-like operation, I decided to use the built in Designer functionality. After all, Microsoft spends billions on usability studies before they release a product, so I'm just going to ride that wave. If they say the PropertyGrid is the best way to edit a set of attributes, then I say use the PropertyGrid.

There's the problem though, "Designer Support",Browsable, EditorBrowsable, PropertyGrid, DesignerSerializationVisibility, CollectionEditor, UITypeEditor, TypeConverter, StandardValuesCollection. Ever use those? Many .NET developers, even the best ones I know (yes far better than me) know little or nothing about what you can do with the built in designer support and designer classes in .NET. There's really three categories of things you can do with all this great stuff.
  1. Create your own visual designer to build something visually. (Like my WYSIWYG Mobile printing code)
  2. Add designer attributes to your code, if you are developing a control that many people will use or that you plan to sell. There are some differences in the designer support for Asp.Net versus WinForms.
  3. Use the built in Designer code to "configure" something within your UI. This is what I am doing with the TRAP UI.
I have shown once or twice a screenshot of some Mapping related class inside a property grid. Big deal, right? Well, as soon as you increase the complexity by only showing some of the properties, or editing collections, or wanting to support a list of acceptible values in a dropdown, or wanting to customize the way the editor looks, or wanting to edit nested properties of types with their own properties, things get somewhat complicated fast. There is also practically no documentation or examples to be found, making this an obscure area of .NET knowledge.

I have most of what I want for designer code working in the TRAP UI now, and that alone could make a pretty decent Designer support tuturial. However, I ran into an issue and the code needs to be cleaned up. On the component diagram I showed a UI assembly and a Types Assembly, where Types is not allowed to depend on another TRAP assembly. Well, once I started adding all of the attributes and custom collections (and there's a lot) needed to give me the designer behavior I want I hit a snag and need to refactor. There is a designer class called StandardValuesCollection that you can create from within a TypeConverter; this class makes it so that if you edit a certain value in the designer you get a ComboBox with your value list in it rather than a TextBox where the user could type anything. Well, by placing all these designer attributes and classes within my Types, I eventually found a dependency on the UI project, in this case it was getting the Project context from my Component Director so I could list the database Tables as StandardValues. I now need to completely clean the types of all this knowledge and create "Designable Types", and use a Decorator pattern or something like it to have designable versions of the types. When that is done I will show some designer code samples.

Tags:

Multi key sort with custom Types

by Damon 7. March 2005 06:00
I often revist my own code experiments when the need arises. This weekend at Deeper in .NET someone asked me about my Sorting Types in Code pattern and if it suppoted multi-key sorts. I said no, but it could. The only weakness is that it only supports Ascending sort order for the "extra" keys beyond the first one. So, if you want to sort your data types in C# code, and you want to produce results ala Order by Property, Property2, Property3 then this should work for you. Suppose you have a Type Foo:

public class Foo

{

private string one;

private string two;

private string three;

public Foo(string f1, string f2, string f3)

{

one = f1;

two = f2;

three = f3;

}

public string F1

{

get{return one;}

set{one = value;}

}

public string F2

{

get{return two;}

set{two = value;}

}

public string F3

{

get{return three;}

set{three = value;}

}

}

Create some instances of type Foo to sort. In order to use the new TypeComparer you need the following code

ArrayList fooz = new ArrayList();

Foo f1 = new Foo("z", "z", "c");

Foo f2 = new Foo("a", "m", "z");

Foo f3 = new Foo("a", "m", "c");

Foo f4 = new Foo("a", "c", "c");

fooz.Add(f1);

fooz.Add(f2);

fooz.Add(f3);

fooz.Add(f4);

string[] names = new string[]{"F1","F2","F3"};

TypeComparer tc = new TypeComparer(names, typeof(Foo));

fooz.Sort(tc);

And last but not least, if you are actually interested in using this, here is the code, only slight modifications from the old TypeComparer:

using System;

using System.Collections;

using System.Reflection;

 

namespace DamonPayne.Trap.Engine

{

          /// <summary>

          /// Make it easy to Sort Custom Types

          /// </summary>

          public class TypeComparer : IComparer

          {

                   private delegate int DoCompare(object x, object y);

                   private DoCompare _compareMethod;

                   private PropertyInfo _propInfo;

                   private Hashtable _methods;

                   private bool _caseSensitive;

                   private string _sortProperty;

                   private bool _multiField;

                   private PropertyInfo[] _propInfos;

                   private DoCompare[] _multiMethods;

                                     

                   public TypeComparer(string propertyName, Type objectType)

                   {

                             _caseSensitive = true;

                             _sortProperty = propertyName;

                             _methods = BuildSortMethods();

                             _propInfo = objectType.GetProperty(_sortProperty);

                             if (null == _propInfo)

                             {

                                      throw new ArgumentException(string.Format("Type {0} does not have a Property '{1}'", new Object[] {objectType.ToString(), propertyName}));

                             }

                             _compareMethod = (DoCompare)_methods[_propInfo.PropertyType];

                             if (null == _compareMethod)

                             {

                                      throw new ArgumentException( string.Format("Type not supported: {0}", _propInfo.PropertyType) );

                             }

                   }

 

                   public TypeComparer(string[] names, Type objectType)

                   {

                             _caseSensitive = true;

                             _multiField = true;

                             _sortProperty = names[0];

                             _methods = BuildSortMethods();

                             _propInfos = new PropertyInfo[names.Length];

                             _multiMethods = new DoCompare[names.Length];

                             for(int i = 0; i < names.Length; ++i)

                             {

                                      _propInfos[i] = objectType.GetProperty(names[i]);

                                      if (null == _propInfos[i])

                                      {

                                                throw new ArgumentException(string.Format("Type {0} does not have a Property '{1}'", new Object[] {objectType.ToString(), names[i]}));

                                      }

                                      _multiMethods[i] = (DoCompare)_methods[_propInfos[i].PropertyType];

                                      if (null == _multiMethods[i])

                                      {

                                                throw new ArgumentException( string.Format("Type not supported: {0}", _propInfo.PropertyType) );

                                      }

                             }

 

                   }

 

                   protected Hashtable BuildSortMethods()

                   {

                             Hashtable ht = new Hashtable();

                             ht.Add(typeof(int), new DoCompare(CompareInt));

                             ht.Add(typeof(double), new DoCompare(CompareDouble));

                             ht.Add(typeof(string), new DoCompare(CompareString));

                             ht.Add(typeof(DateTime), new DoCompare(CompareDate));

                             return ht;

                   }

 

                   public bool CaseSensitive

                   {

                             get{return _caseSensitive;}

                             set{_caseSensitive = value;}

                   }

 

                   protected int CompareInt(object x, object y)

                   {

                             int xInt = (int)x;

                             int yInt = (int)y;

                             return xInt.CompareTo(yInt);

                   }

 

                   protected int CompareDouble(object x, object y)

                   {

                             double xDouble = (double)x;

                             double yDouble = (double)y;

                             return xDouble.CompareTo(yDouble);

                   }

 

                   protected int CompareDate(object x, object y)

                   {

                             DateTime xDate = (DateTime)x;

                             DateTime yDate = (DateTime)y;

                             return xDate.CompareTo(yDate);

                   }

 

                   protected int CompareString(object x, object y)

                   {

                             string xString = (string)x;

                             string yString = (string)y;

                             if (!CaseSensitive)

                             {

                                      xString = xString.ToUpper();

                                      yString = yString.ToUpper();

                             }

                             return xString.CompareTo(yString);

                   }

 

                   #region IComparer Members

 

                   public int Compare(object x, object y)

                   {

                             if (_multiField)

                             {

                                      return MultiKeyCompare(x, y);

                             }

                             object xVal = _propInfo.GetValue(x, null);

                             object yVal = _propInfo.GetValue(y, null);

                            

                             return _compareMethod(xVal, yVal);

                   }

 

                   public int MultiKeyCompare(object x, object y)

                   {

                             int index = 0;

                             object xVal = null;

                             object yVal = null;

                             int comp = 0;

                             while(0 == comp && index < _propInfos.Length)

                             {

                                      xVal = _propInfos[index].GetValue(x, null);

                                      yVal = _propInfos[index].GetValue(y, null);

                                      comp = _multiMethods[index](xVal,yVal);

                                      ++index;

                             }

                             return comp;

                   }

 

                   #endregion

          }

}

Happy coding

Tags:

Choosing an OR mapper

by Damon 3. March 2005 06:00
Well, OR mapping is not as un-popular in the .NET world as I thought it was. Only a week or two after I post my target feature list for trap I run across this article on TheServerSide.Net. It seems like I was barking up the right tree, at least according to this article's author. I have to admit I had not thought about supporting OQL, mostly because I personally didn't find I cared much one way or another when I was using other OR tools.

Tags:

Trap Codegen

by Damon 2. March 2005 06:00
TRAP Typegen More code less talk
Late last night I was working on the TRAP type mapping design and I realized I've reached the point where experimentation and refactoring will do more good than thinking deep thoughts about the design even more. I've decided to take a crawl --> walk --> fly approach, by which I mean the easiest feature will be implemented first, moving up to the most complex features. The simplest feature is a select field1, field2, field3 from table operation, quickly followed by adding the Criteria object to the query. From there, 1:1 relationships, then 1:n relationships, then m:n relationships, then the stored procedure execution path, then some of the relfection emit optimizations, the stored procedure execution path, etc.

I added some simple code generation to the Engine assembly the other day. Reducing tedious data access tasks is the goal of this tool and that includes typing get{} set{} code for properties for classes which closely mirror the database schema a great deal of the time. Why not just use public fields?, you ask, Far less typing. From a Reflection standpoint, Properties and Fields must be treated differently in code, and for TRAP I don't want to create a second execution path for Fields vs. Properties. DataBound UI controls in both WinForms and WebForms are only set up to look for Indexers (dataRow["field"]) and Properties, and this makes a second good reason to use Properties, even if they have no intelligence in them. So, here's a shot of the UI generating custom Types from my database schema, which saves me a lot of typing.
br> Unless you've been under a rock (or are new to MSFT tech) you'll recognize the good ole Northwind database there. I decided to use this for testing and examples, rather than show more than I'm ready to show about my other projects that are going to be using this tool. I find this sort of ironic, since the Northwind database has been used in all of the "DataSets and DataReader are the best!" examples I so despise.

When next I open my yap to rap about TRAP I will show the OR Mapper UI in action and some working code.

Tags:

TRAP progress

by Damon 27. February 2005 06:00
Before moving on to some client related work I did some more work on TRAP this weekend. Its actually going about as fast as I expected so far. As the design fleshes itself out, I can start to see some of the challenges involved in making this kind of tool. As is typically the case when getting involved in a project, some things turn out to be far easier than anticipated and some things are more difficult, or just annoying. For example, supporting multiple databases with the same engine will be easy, all encapsulated and behind the provider interfaces.

I see performance as the biggest challenge on this project. Its plain to me now that there's no reason why I can't support all the features I want to support and have a robust tool. There are two performance issues that I need to address, not necisarily right now but they need to stay in the back of my mind so that the design doesn't stand in the way of these changes later. First, the engine is heavily reflection based right now. In a data intensive application the performance hits incurred from inspecting classes at runtime could start to add up. The idea I have in mind to address this down the line is to borrow a cue from the core .NET framework and use Reflection Emit to build some helper assemblies on the fly that implement the same functionality in non-relfective code. The second performance challenge is of course generating efficient SQL.

When Type Manager has a one to many relationship with Type Employee, the tool has to decide how to load that. By turning on the maximum debug levels for OR tools I've used in the past I often found that when loading relationships the engine would:
  1. Issued one statement to load Manager data meeting the criteria.
  2. Then issued one statement per instance of Manager to read its list of employees
Obviously this is much simpler than writing code that can do the table join to get all the information in one call. Writing code that can handle the Manger-to-Employee situation is fairly easy, but what about when Employee has related types, and those types have related types, and so forth? Writing an engine that can generate all of this may be complex, or not possible. The "one statement per parent" method and the performance hit was infamously known in the EJB CMP community as the "1+", meaning essentially it is an Order(n) + 1 operation. I've been thinking quite a bit about the three relationship load options I plan to offer:
  • Lazy Load - related types are not loaded until the property is referenced. A proxy object is created that knows how to load its data once it is deferenced. This would likely require some Reflection emit code to extend a Type on the fly with a proxy, if used for 1:1 relationships. For 1:n and m:n relationships, a proxy class that extends a built in type is easy enough to create.
  • Eager Load - The engine will populate the entire type tree with one call. In most cases the number of calls could be drastically reduced by using a single subselect for each level of relationships.
  • Semi-Eager-Threaded Load (need a better name for this one)
  • - Execute the main call, with the "1+" logic being handheld by an asynchronous delegate, giving the user the illusion of faster performance. To keep using the same example, the main thread returns as soon as Managers are loaded but a threadpool thread immediately begins the work of loading Employees, the collection of Employees is likey already populated before it is referenced. This would still beat up the database pretty good.

I have quite a bit of the design and 2,000 lines of code done for TRAP right now. The next step is to finish the Types assembly design, specifically to determine exactly how I want the mapping schema to look. The stuff in the Core assembly and Types assembly comprise the interface projects using TRAP will interface with. At this point I could share a little bit of what that looks like right now:
So, essentially to find some instances of a type you would get a UnitOfWork, which represents both a connection to a data store and a transaction. You would pass a Criteria object and a System.Type to find instances of objects, and then possibly update these objects via the same unit of work.

I'm also showing a preview of the main mapper form from the UI. Hopefully after looking at this you will think "Oh yeah, a Designer like that would make me very liley to use this tool to get an application up and running quickly." The essential idea is to use reflection to display your types and use the connection information from your Project to display database schema, and the Mapping Types are displayed in a property grid when an item on either side is selected. Everything I show from the UI is working code, soup to nuts, so really this is likely to be in an "Alpha" stage in another week or two.
I have two websites to finish, next up I will display the design for the Types assembly. After that, Provider and Engine, these two are by far the most complex and closely associated. Stay tuned.

Tags:

PInvoke keybd_event

by Damon 25. February 2005 06:00
Today's tip for the compact framework is a simple one. Suppose you are working on a PPC device that only has two or 3 buttons, most likely the built in "calendar" and "task" type buttons. Your users would rather not use the screen/stylus as much as possible. Yes, you still have to build a user interface with tabbing, space bar, enter, etc. For example, suppose you use MessageBox to indicate an error; with no space bar or enter key you can clear it by using OpenNetCF's windows message filter to catch the key event. Then, you can PInvoke keybd_event to send whatever other message you want to your application. For example, the enter key (Key code 13) to hit "OK" on your message box.

[DllImport("coredll.dll")]

public static extern void keybd_event(byte vKey, byte bScan, uint dwFlags, uint dwExtraInfo);

That's all for today. Hopfully my other furnace doesn't blow up this weekend and I get some work done on TRAP.

Tags:

Will code for food

by Damon 21. February 2005 06:00
I typically avoid personal things here and leave for my other site. However, just to reach a wider audience: This weekend my furnace stopped working. The timing couldn't be worse, either. (Not that there's ever a good time)

So, just letting the general population know that I am available for work on the side. I am pretty much always working on some small system, but given the $$ I just spent keeping my house warm I will probably take on another project right now. You can establish contact here for starters.

Tags:

About the author

Damon Payne is a Microsoft MVP specializing in Smart Client solution architecture. 

INETA Community Speakers Program

Month List

Page List

flickr photostream