RDA, sometimes so good, sometimes so bad.  I post about data synchronization to mobile systems a lot because its one of those interesting problems to me.  There are many ways to do it, Merge Replication is one.  Merg Rep comes with so many drawbacks I could write a decent sized essay about it.  For now I'll just say that if you are considering merge replication you should try to talk to someone with experience implementing it and make sure you want to go down that path.  Synchronizing a read-only database is easy with merg rep or without using custom code.  Synchronizing a read-write database with custom code is a non-trivial endeavor.

Having written and re-written such things several times I have tried various methods.  After my first attempts at remote rebuilding a 100 meg database on a 200mhz strong arm over the internet using web services I decided that RDA needs to be a part of any solution.  RDA allows one to pull down SQL Sever data in a nice binary format that automagically turns into an SQLCE table schema and table data as it streams down to a mobile device.  You can push SQL data using RDA as well, but I don't like doing so for various reasons.  Since RDA is streaming data over HTTP it also gets you around some ugly memory issues.  Consider your woe when your XML message containing 10,000 rows of data is easily spit out by your server but runs the mobile systems out of memory every time.  Segmenting large messages using many XML calls quite frankly feels dirty as well.  As far as rebuilding that 100meg database over the internet, RDA is great.

On my current project we do frequent data synchronization over an AirLink modem.  These things are truly great but they have data rates that are woefully slow in a world that has become accustomed to cable modems and burstable ethernet.  We were having situations where in-office testing would have a sync time of say a minute and a half for a good amount of data.  However in the field some locations were so bad (-110 db signal strength or worse) that this same data packet was taking 20 minutes to process.  This is unacceptable.  Since I happen to have a method to FTP the handheld log files to myself I started doing some log file forensics.

In addition to low data rate, these CDMA modems have high latency, 500ms ping and such.  Every additional trip to the server was incurring large overhead in just making the request.  My data sync that was making an RDA.Pull("select blah blah from blah blah where DateLastChanged > myLastSync") for each table that was fine in our office (where the connection SUCKS butt apparently not as suck as some of our remote locations) but not fine over a low data rate, high latency, in-a-black-hole parking lot where our trucks must operate.  In testing with the SQL server DBA we have noticed an inexplicable overhead for each RDA operation that is about six seconds over a good connection and skyrockets exponentially on a bad one.  While we work out issues with other modem providers (Sprint, Verizon, and Cingular are the big players, Sprint is currently working but connectivity sucks in many areas) a solution was needed.  In the process of making my remote operationgs more "transactional" (after studying F# and what it's all about a tiny bit I dislike the common usage of the term idempotent, which I will explain in a future post) I ended up building multiple code paths for the data sync based on how much data was to be exchanged.  A full rebuild or "very large messages" will still use RDA and execute one pull per table, but in many cases the data package can be sent and received as one large XML message.  I had avoided this approach as long as possible due to the bloat of encoding everything as XML and the potential for memory problems on the clients.  I've found that an "average" XML message is about 179kb which makes me nervous given the CF 1.0 max stack size of 640k and my assumption that the webservice client must load the entire message into memory before it can deserialize it.  If this does become an issue I will do some column mapping such that <FuelDeliveryTransaction><FuelDeliveryTransactionId>12345</FuelDeliveryTransactionId><TotalPricePerGallon>$2.78</TotalPricePerGallon></FuelDeliveryTransaction> in a message becomes <FuelDeliveryTransaction><0>12345</0><1>$2.78</1></FuelDeliveryTransaction> and thereby saves many bytes of data over hundreds or thousands of rows.

I would say "Bring on the high speed wireless connection to every corner of the globe", but Johnny Mneumonic had a traumatizing effect on me.

I will follow up on my solution for the "transaction" post later, however with that problem solved I have a more immediate need: I'm looking for a free Code Coverage tool for .NET.  I found a lot of blog posts pointing to CoverageEye but I wanted to know if anyone is using anything else and having good experiences?

I usually do not post my unqualified ramblings on here but a thought occurred to me today regarding a problem I'm having with on of my mobile systems.  There is a great deal of data exchange and I have taken pains to ensure that no data is ever "lost" in the process.  If a network connection dies at the wrong time the worst thing that happens is that the data exchanged did not complete and the PDA OR server (never both) each have an incomplete picture of the data in question.  When this occurs, I recover the complete picture from a transaction log and go on with my life.

However, I was thinking that it would sure be great to be able to treat the whole data synchronization process as one great big distrubuted transaction.  A distributed transaction involved a wifi network, an over-the-air CDMA connection, web services and SQL server.  There are a hundred sticky little issues that would go along with that strategy and at least a hundred very good reasons not to pursue the idea but it certainly has some allure in my mind as well.  The way my data sync is set up, unfortunately, the transaction cannot be just on the server nor just on the client, they must both agree to commit or roll back if the agreement cannot be procured in a reasonable amount of time; it would be much easier if I could just "make sure the handheld was done", etc.

If anyone out there has mused before on distrubuted transactions involving compact framwork applications I'd be very interested in where your thoughts led.

If you are in the Milwaukee area you need to think about attending Deeper in .NET 2006 on April 22nd this month.  You can see the information at http://www.wi-ineta.org/DesktopDefault.aspx?tabid=104

All of the speakers are good, but I should call out two of them for special consideration.  Scott Hanselman is someone I admire for his technical expertise, attitude, and sense of humor.  I've subscribed to his blog for over a year now, and I got a chance to see him speak about Software Factories at TechEd last year.  He's a great speaker.

Jason Beres is also a great speaker and a very technical and fun guy.  Also, I can attest that he is a formidable drinker of beer and has excellent taste in steaks. 

If you are in the area, you must come to Deeper in .NET 2006.

This was somewhat annoying.

Regular expressions are one huge area where I know absolutely nothing.  I used to get by in my unix days with stuff like "find . -name *fork* -ls {} " but that's about it.  Tonight I was working on a toy website and got my PasswordRegularExpression right in my <asp:CreateUserWizard/>, however it would still complain about the password.  As it turns out the AspMembershipProvider can specify its own password requirements which trump what you put in your Controls.  Clear as mud.  You can see this in machine.config and override it in your own app if you please:

<membership>

<providers>

<clear/>

<add name="AspNetSqlMembershipProvider"

type="System.Web.Security.SqlMembershipProvider, System.Web, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a"

connectionStringName="LocalSqlServer"

enablePasswordRetrieval="false"

enablePasswordReset="true"

requiresQuestionAndAnswer="true"

applicationName="/"

requiresUniqueEmail="false"

passwordFormat="Hashed"

maxInvalidPasswordAttempts="5"

minRequiredPasswordLength="5"

minRequiredNonalphanumericCharacters="0"

passwordAttemptWindow="10"

passwordStrengthRegularExpression="" />

</providers>

</membership>

Seems somewhat annoying for "configurable things that are there to save me time."

Also, I must get a regex book immediately as my lack of knowledge on this topic wasted all of my playtime tonight.

This is a follow up to my previous post.  After today, I have a small distant idea of how NASA must feel when a "trivial" item causes a big problem.  My trucks in Detroit were having issues today, and long support calls were getting me nowhere.  I finally came around to thinking that our data synchronization is time-based (rather than GUID based like Merge Replication) and wondered if I had not setup the Detroit units properly.  All times are convered to UTC on the units so there is no hoo-ha for trucks outside of good ole  GMT - 6 where the server lives, and on my checklist for building new handhelds is a bullet point "Verify time zone".  I did this however forgot to verify Time as well so my units had GMT -5 but the hour set to the same hour as current local time.  A small thing caused a very big SNAFU today.  Obviously I can't trust truck drivers (nor myself, clearly) to not screw up the time on the units and I should automate the system time from the backend.  Things like time-zones for nation-wide deployments will play a larger role in my testing strategy in the future...

 

One of the cool aspects of doing Mobile systems is that you often get to do a different kind of programming.  There are plenty of business problems to solve with ASP.NET and SQL but I've recently had a very cool opportunity.

A local company is in the fleet fueling business, this means if you are, say, FedEx in Atlanta, they bring fuel trucks around to your lot at night and re-fuel all your Coca Cola delivery trucks.  The fuel is dispensed using control systems in the truck comprised of fuel meters, printers, and other devices.  As you can imagine with oil-based products being very expensive right now, there are a lot of fraud-prevention mechanisms and business rules related to fuel pricing.  I recently completed, and am now supporting in Production, a new Compact Framework based application that controls the control systems on the truck.  There were many cool-yet-annoying requirements on this effort that forced me to defend-or-change some of my software engineering practices: they system has to talk to several different brands of fuel meters to record gallons, meter totalizers, preset delivery amounts, etc.  Each manufacturer has its own protocol and there are no well published APIs or Web Services: binary hex instead.  We also could not be tied to any specific make/model for any of our other hardware: PDAs, wireless to serial bridges, port replicators, CDMA modems, mobile printers.  Supporting Windows Mobile 2003 and raw CE.NET 4.2 on the same codebase along with 4 different flavors of different OEMs' barcode scanners, cold boot procedures, and WiFi registry settings was also a "Lesson in configurability".

 

Here's a shot of my oh-so-clean and ergonomic development environment with various handheld units:

 

The most excellent part of this system was interacting with the hardware on a real truck once everything was working.  One great thing about being a developer is changing code and then seeing it dance on the screen; pressing a button to calibrate a Fuel Meter, writing exception handling code that tells hardware to reboot itself, scanning a barcode and then watching Low Sulpher Diesel flow out of a real truck at a pre-determined flow rate, now that's fun, and realy damn cool. I recall a conversation with one great architect who was telling me about his experiences with writing warehouse control code many years ago.  He said words to the effect of "Design and testing take on new meaning when a 'bug' results in boxes piling up somewhere or flying off the end of a conveyor belt with nothing there to catch them."  Working in the hardware world occasionally helps make code and the things the code does more real.  If you ever get a chance to work with a hardware system, don't pass it up, you'll probably be thinking and talking about it for a long time.  I'll never forget my first hardware control system experience working for a local manufacturer.  We had a large "test bench" setup so that we could watch lights blink.  One aspect of the software we wrote was the notion of an Alarm that would occur on a client application if hardware operating parameters were outside norms.  The canonical alarm example was that a processor could overheat.  While we stood around deciding how we could accurately test a real alarm situation, one of the testers pried the heatsync off of said processor with a pocketknife to create the alarm state.  Contrast that with the types of testing we normally see on projects, "If I leave this field blank, it doesn't save" and things like that.

Speaking of test benches, here is a shot of some of my test hardware:

All in all, this is one of the coolest projects I've gotten to work on.  I'll ask for permission to post pictures of the actual fuel trucks. Now I just have to convince them to upgrade to CF2 and SQL Mobile...

Clemens Vasters has posted an interesting article about O/R mapping.  I can admit that I am one of those people who is drawn to the beauty of an O/R abstraction, and have been since first experimenting with Jasmine and PostgreSQL years ago, and the ideas are obviously much older than that.

After exploring my own O/R implementation I must admit that there are problems with the "aesthetics of the abstraction" as well.  You are seldom working with a fully populated domain model, and lazy loading and the like carries with it its own trade offs.  I'm a bigger fan of Code Generation than pure OR mapping these days.  At any rate, read the article if you are interested in O/R mapping