CF Message Queue

by Damon 17. March 2005 06:00
Moral of the story: If you're going to write an app with some threading, test with 3 or more threads at least, even if you don't plan on using more than 2. In .NET, you can get away with murder if you only have 2 threads. Problems with your threading code are much more apparent when 4 threads are in contention for the same resource. In this scenario that resource is the UI message loop.

I support a mobile project that synchronizes a great deal of data over the internet (60meg all told) between a Compact Framework windows app and a PHP/PERL/BSD backend. MSFT tools were not an option for the backend, so using RDA Pull commands to get data is out. The syncrhonization is done via web services, essentially the server sends a DataSet friendly format back and it is merged locally. They recently asked if the sync could be made faster. My first reply was "Yeah, if you could tell me which rows are Inserts vs. Updates so I don't have to do a row-by-row merge" but apparently that is not going to happen either. While working on a completely different application and trying to fight through "medicine head" brought on by DayQuil a good idea came to me.

Like many good ideas it was simple and obvious. The current sync process was:
  1. Ask for data
  2. Get DataSet back from server
  3. Process DataTables row by row (very time consuming in SQL CE 2.0)
  4. Ask for more data
Lather, rinse, repeate. I had already threaded out the Sync process so the UI could keep drawing itself during this time, but now it was clear that I could also implement sort of a local Message Queue so each individual bundle of data could be processed Asynchronous to the other thread which was actually pulling the data from the server. So, I came up with a simple class to queue and dequeue messages, came up with classes and interfaces to represent a Queue Work Item and now performance is considerably better. Messages are being processed while the next message is fetched from the server, so the processor on the mobile device and the bandwidth it can suck from the server are both maxed out at all times.

The actual work queue itself is not rocket science either. Here's the code, complete with all my debug nonsense:

using System;

using System.Collections;

using System.Threading;

using OpenNETCF.Windows.Forms;

namespace Mobile.Sync

{

/// <summary>

/// Summary description for SyncWorkItemQueue.

/// </summary>

public class SyncWorkItemQueue

{

private object _syncRoot;

private System.Collections.Queue _queue;

private bool _run;

public const int QUEUE_THRESHOLD = 50;

public const int CATCH_UP_THRESHOLD = 8;

public SyncWorkItemQueue()

{

_syncRoot = new object();

_queue = new Queue();

_run = true;

}

/// <summary>

/// Default to true

/// </summary>

public bool Run

{

get

{

lock(_syncRoot)

{

return _run;

}

}

set

{

lock(_syncRoot)

{

_run = value;

}

ApplicationEx.DoEvents();

}

}

/// <summary>

/// Get the number of SyncWorkItem s in the queue

/// </summary>

public int ItemCount

{

get

{

lock(_syncRoot)

{

return _queue.Count;

}

}

}

/// <summary>

/// (Synchronized) Add an item to the end of the queue

/// </summary>

/// <param name="workItem"></param>

public void AddToQueue(SyncWorkItem workItem)

{

lock(_syncRoot)

{

Console.WriteLine("Adding: " +workItem);

_queue.Enqueue(workItem);

}

ApplicationEx.DoEvents();

}

public void StartProcessing()

{

ThreadStart threadStart = new ThreadStart(StartWork);

Thread workerThread = new Thread(threadStart);

workerThread.Start();

}

protected void StartWork()

{

while(Run)

{

System.Windows.Forms.Application.DoEvents();

if (_queue.Count > 0 )

{

Console.WriteLine("Processing Item, Count=" + _queue.Count);

SyncWorkItem workItem = (SyncWorkItem)_queue.Dequeue();

try

{

workItem.Updater.UpdateTable();

}

catch(Exception ex)

{

Console.WriteLine(ex);

Console.WriteLine("Error processing work item");

Console.WriteLine(workItem.Updater);

ApplicationEx.DoEvents();

}

Console.WriteLine("Processed: " + workItem.Updater);

}

//Allow everything to update

ApplicationEx.DoEvents();

//Now check for greater than threshold amount, to let everything catch up

if (_queue.Count > QUEUE_THRESHOLD)

{

Console.WriteLine("Catching up");

lock(_syncRoot)

{

for(int i = 0; i < CATCH_UP_THRESHOLD; ++i)

{

SyncWorkItem workItem = (SyncWorkItem)_queue.Dequeue();

workItem.Updater.UpdateTable();

Console.WriteLine("CATCH_UP_THRESHOLD item processed, Count=" + _queue.Count);

ApplicationEx.DoEvents();

}

}

ApplicationEx.DoEvents();

}

}

Console.WriteLine("Ending Startwork");

}

}

}


At the end, if there are work items to process, I just do the following to let the sync thread catch up:

while (_syncQueue.ItemCount > 0)

{

Thread.Sleep(10);

}

_syncQueue.Run = false;



From where I'm standing, this is not too hard to do. I did run into a thread race condition while implementing this though. Even so, I still don't know why the average developer/architect is so afraid of using threading. The dreaded thread race issue was actually relatively easy to track down using the debugger: just hit "Pause" and it displayed all my threads and I could clearly see what thread was blocking where. In my case, the issue was sloppy coding. At one point the sync was not on its own thread and it directly updated the UI. When I put it on its own thread I did not change these messages to fire through Control.Invoke(delegate); I got away with it until more threads and were added to the mix, thus my comment about getting away with murder when you have only two threads: Application.DoEvents is often enough to resolve a lockup but not once you get beyond 2 threads.

Tags:

Pingbacks and trackbacks (523)+

Add comment




  Country flag
biuquote
  • Comment
  • Preview
Loading


About the author

Damon Payne is a Microsoft MVP specializing in Smart Client solution architecture. 

INETA Community Speakers Program

Month List

Page List

flickr photostream