Serializing Data

Summary

In this blog post I’m going to talk about some tricky problems with serializing and deserializing data.  In particular, I’m going to demonstrate a problem with the BinaryFormatter used in C# to turn an object into a byte array of data.

Using the BinaryFormatter Serializer

If you are serializing an object inside your project and storing the data someplace, then deserializing the same object in your project, things will work as expected.  I’ll show an example.  First, I’ll define a generic object called AddressClass.  Which stores address information:

[Serializable]
public class AddressClass
{
    public string Address1 { get; set; }
    public string Address2 { get; set; }
    public string City { get; set; }
    public string State { get; set; }
    public string Zip { get; set; }
}

The first thing you’ll notice is that there is a [Serializable] attribute.  This is needed in order for BinaryFormatter to serialize the object.  Next, I’ll create an instance of this object in my console application and populate with some dummy data:

// create an instance and put some dummy data into it.
var addressClass = new AddressClass
{
    Address1 = “123 Main st“,
    City = “New York“,
    State = “New York“,
    Zip = “12345
};

Now we’re ready to serialize the data.  In this example, I’ll just serialize this object into a byte array:

// serialize the object
using (var memoryStream = new MemoryStream())
{
    var binaryFormatter = new BinaryFormatter();
    binaryFormatter.Serialize(memoryStream, addressClass);

    storedData = memoryStream.ToArray();
}

There’s nothing fancy going on here.  You can even use a compressor inside the code above to compress the data before saving it someplace (like SQL or Redis or transmitting over the wire).  Now, I’m going to just deserialize the data into a new object instance:

//deserialize the object
AddressClass newObject;
using (var memoryStream = new MemoryStream())
{
    var binaryFormatter = new BinaryFormatter();

    memoryStream.Write(storedData, 0, storedData.Length);
    memoryStream.Seek(0, SeekOrigin.Begin);

    newObject = (AddressClass)binaryFormatter.Deserialize(memoryStream);
}

If you put a break-point at the end of this code as it is, you can see the newObject contains the exact same data that the addressClass instance contained.  In order to make all the code above work in one program you’ll have to include the following usings at the top:

using System;
using System.IO;
using System.Runtime.Serialization.Formatters.Binary;



Deserializing in a Different Program

Here’s where the trouble starts.  Let’s say that you have two different programs.  One program serializes the data and stores it someplace (or transmits it).  Then another program will read that data and deserialize it for its own use.  To simulate this, and avoid writing a bunch of code that will distract from this blog post, I’m going to dump the serialized data as an array of integers in a text file.  Then I’m going to copy that raw text data and then use it in my second program as preset data of a byte array.  Then I’m going to copy the AddressClass code above and the deserialize code above and put it in another program.  This should deserialize the data and put it into the new object as above.  But that doesn’t happen.  Here’s the error that will occur:

Unable to find assembly ‘SerializatingDataBlogPost, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null’.

This error occurs on this line:

newObject = (AddressClass)binaryFormatter.Deserialize(memoryStream);

Inside the serialized data is a reference to the dll that was used to serialize the information.  If it doesn’t match, then the assumption is that BinaryFormatter will not be able to convert the serialized data back into the object defined.

If you dig around, you’ll find numerous articles on how to get around this problem.  Using a BindToType object is one method as shown here:

Unable to find assembly with BinaryFormatter.Deserialize

And it goes down a rabbit hole from there.  

Another Solution

Another solution is to serialize the data into JSON format.  The Newtonsoft serializer is very good at serializing objects into JSON.  After that, the deserialized data can be cast back into the same object inside another dll.  Use the NuGet manager to add Newtonsoft to your project.  Then use the following code to serialize your addressClass object:

// serialize the object
var serializer = new JsonSerializer();
string resultSet = JsonConvert.SerializeObject(addressClass);


This will convert your object into the following string:

{“Address1″:”123 Main st”,”Address2″:null,”City”:”New York”,”State”:”New York”,”Zip”:”12345″}

Inside another project, you can deserialize the string above using this:

// deserialize object
AddressClass newObject;
newObject = JsonConvert.DeserializeObject<AddressClass>(resultSet);



Compressing the Data

JSON is a text format and it can take up a lot of space, so you can add a compressor to your code to reduce the amount of space that your serialized data takes up.  You can use the following methods to compress and decompress string data into byte array data:

private static byte[] Compress(string input)
{
    byte[] inputData = Encoding.ASCII.GetBytes(input);
    byte[] result;

    using (var memoryStream = new MemoryStream())
    {
        using (var zip = new GZipStream(memoryStream, CompressionMode.Compress))
        {
            zip.Write(inputData, 0, inputData.Length);
        }

        result = memoryStream.ToArray();
    }

    return result;
}

private static string Decompress(byte[] input)
{
    byte[] result;

    using (var outputMemoryStream = new MemoryStream())
    {
        using (var inputMemoryStream = new MemoryStream(input))
        {
            using (var zip = new GZipStream(inputMemoryStream, CompressionMode.Decompress))
            {
                zip.CopyTo(outputMemoryStream);
            }
        }

        result = outputMemoryStream.ToArray();
    }

    return Encoding.Default.GetString(result);
}

You’ll need the following usings:

using System.IO.Compression;
using System.IO;


Then you can pass your JSON text to the compressor like this:

// compress text
byte[] compressedResult = Compress(resultSet);


And you’ll need to decompress back into JSON before deserializing:

// decompress text
string resultSet = Decompress(compressedResult);



Where to Get the Code

As usual, you can go to my GitHub account and download the projects by clicking here.

 

Exporting XML From MS SQL Server

Summary

This is going to be a short post on how to query an MS SQL Server database and produce XML output.

The Query

There are two formats that you can use when you want to convert a query result into XML, Elements format and Properties Format.  The following query will select from a table named “product” and put the result set into elements:

SELECT * FROM product FOR XML RAW ('product'), ROOT('data'), ELEMENTS
 
 

The results window will look something like this:
 

Click on the url link and the output should look like this:

<data>
  <product>
    <ProductId>1</ProductId>
    <ProductType>1</ProductType>
    <Name>Matchbox Car</Name>
    <store>1</store>
  </product>
  <product>
    <ProductId>2</ProductId>
    <ProductType>1</ProductType>
    <Name>Baby Rattle</Name>
    <store>1</store>
  </product>
  <product>
    <ProductId>3</ProductId>
    <ProductType>1</ProductType>
    <Name>Lego Bricks</Name>
    <store>1</store>
  </product>
  <product>
    <ProductId>4</ProductId>
    <ProductType>2</ProductType>
    <Name>Steak</Name>
    <store>1</store>
  </product>
</data>

If you want to flatten out your XML, you can use this syntax:

SELECT * FROM product FOR XML RAW ('product')

Which will produce this output:

<product ProductId=”1ProductType=”1Name=”Matchbox Carstore=”1” />
<product ProductId=”2ProductType=”1Name=”Baby Rattlestore=”1” />
<product ProductId=”3ProductType=”1Name=”Lego Bricksstore=”1” />
<product ProductId=”4ProductType=”2Name=”Steakstore=”1” />

As you can see by the last example, you can leave the “Root” keyword off the query if you don’t want to the root data tags added.

Producing JSON

If you need to produce JSON, you will need to do it using another tool.  Use the For XML RAW command above, including the root tag (the output must be well-formed.  Then go to this site (or any other free xml to json converter website): thomasfrank.se XML to JSON – a converter

Paste your xml into the window and hit the convert button.

 

Returning XML or JSON from a Web API

Summary

In my last blog post I demonstrated how to setup a Web API to request data in JSON format.  Now I’m going to show how to setup your API so that a request can be made to ask for XML as well as JSON return data.  To keep this simple, I’m going to refactor the code from the previous blog post to create a new retriever method that will set the Accept parameter to xml instead of json.

Changes to the Retriever

I copied the retriever method from my previous code and created a retriever that asks for XML data:

public void XMLRetriever()
{
    var xmlSerializer = new XmlSerializer(typeof(ApiResponse));

    var apiRequest = new ApiRequest
    {
        StoreId = 1,
        ProductId = { 2, 3, 4 }
    };

    var request = (HttpWebRequest)WebRequest.Create(
        apiURLLocation + “”);
    request.ContentType = “application/json; charset=utf-8“;
    request.Accept = “text/xml; charset=utf-8“;
    request.Method = “POST“;
    request.Headers.Add(HttpRequestHeader.Authorization, 

            apiAuthorization);
    request.UserAgent = “ApiRequest“;

    //Writes the ApiRequest Json object to request
    using (var streamWriter = new  
           StreamWriter(request.GetRequestStream()))
    {
        streamWriter.Write(JsonConvert.SerializeObject(apiRequest));
        streamWriter.Flush();
    }

    var httpResponse = (HttpWebResponse)request.GetResponse();

    // receives xml data and deserializes it.
    using (var streamreader = new  
           StreamReader(httpResponse.GetResponseStream()))
    {
        var storeInventory = 

            (ApiResponse)xmlSerializer.Deserialize(streamreader);
    }
}

There are two major changes in this code: First, I changed the “Accept” to ask for xml.  Second, I recoded the return data to deserialize it as xml instead of json.  I left the api request to be in JSON.


Changes to the API Application

I altered the API controller to detect which encoding is being requested.  If Accept contains the string “json” then the data is serialized using json.  If Accept contains the string “xml” then the data is serialized using xml.  Otherwise, an error is returned.

Here is the new code for the API controller:

var encoding = ControllerContext.Request.Headers.Accept.ToString();
if (encoding.IndexOf(“json“,  

    StringComparison.OrdinalIgnoreCase) > -1)
{
    // convert the data into json
    var jsonData = JsonConvert.SerializeObject(apiResponse);

    var resp = new HttpResponseMessage();
    resp.Content = new StringContent(jsonData, Encoding.UTF8, 
        “application/json“);
    return resp;
}
else if (encoding.IndexOf(“xml“,  

         StringComparison.OrdinalIgnoreCase) > -1)
{
    // convert the data into xml
    var xmlSerializer = new XmlSerializer(typeof(ApiResponse));

    using (StringWriter writer = new StringWriter())
    {
        xmlSerializer.Serialize(writer, apiResponse);

        var resp = new HttpResponseMessage();
        resp.Content = new StringContent(writer.ToString(), 
             Encoding.UTF8, “application/xml“);
        return resp;
    }
}
else
{
    return Request.CreateErrorResponse(HttpStatusCode.BadRequest, 

           “Only JSON and XML formats accepted“);
}


Compile the API application and then startup fiddler.  Then run the retriever.  In fiddler, you should see something like this (you need to change your bottom right sub-tab to xml):




Download the Source


You can go to my GitHub account and download the source here: https://github.com/fdecaire/WebApiDemoJsonXml

 

 

Web API and API Data Retreiver

Summary

In this blog post I’m going to show how to create a Web API in Visual Studio.  Then I’m going to show how to setup IIS 7 to run the API on your PC (or a server).  I’m also going to create a retriever and show how to connect to the Web API and read data.  I’ll be using JSON instead of XML so I’ll also show what tricks you’ll need to know in order to implement your interface correctly.  Finally, I’ll demonstrate how to troubleshoot your API using fiddler.


This is a very long post.  I had toyed with the idea of breaking this into multiple parts, but this subject turned out to be too difficult to break-up in a clean manner.  If you are having trouble getting this to work properly or you think I missed something leave a message in the comments and I’ll correct or add to this article to make it more robust.

Web API Retriever

I’m going to build the retriever first and show how this can be tested without an API.  Then I’ll cover the API and how to incorporate IIS 7 into the whole process.  I’ll be designing the API to use the POST method.  The reason I want to use a POST instead of GET is that I want to be able to pass a lot of variables to request information from the API.  My demo will be a simulation of a chain of stores that consolidate their inventory data into a central location.  Headquarters or an on-line store application (i.e. website) can send a request to this API to find out what inventory a particular store has on hand.  

The retriever will be a simple console application that will use an object to represent the request data.  This object will be serialized into a JSON packet of information posted to the API.  The request object will look like this:

public class ApiRequest
{
  public int StoreId { get; set; }
  public List<int> ProductId { get; set; }
}


This same object will be used in the API to de-serialize the JSON data.  We can put the store id in this packet as well as a list of product ids.  The data received back from the API will be a list of inventory records using the following two objects:

public class InventoryRecord
{
  public int ProductId { get; set; }
  public string Name { get; set; }
  public int Quantity { get; set; }
}


public class ApiResponse
{
  public List<InventoryRecord> Records = new  

         List<InventoryRecord>();
}

As you can see, we will receive one record per product.  Each record will contain the product id, the name and the quantity at that store.  I’m going to dummy out the data in the API to keep this whole project as simple as possible.  Keep in mind, that normally this information will be queried from a large database of inventory.  Here’s the entire retriever:

public class WebApiRetriever
{
  private readonly string apiURLLocation =  

             ConfigurationManager.AppSettings[“ApiURLLocation“];
  private readonly string apiAuthorization =  

          ConfigurationManager.AppSettings[“ApiCredential“];

  public void Retreiver()
  {
    var serializer = new JsonSerializer();

    var apiRequest = new ApiRequest
    {
      StoreId = 1,
      ProductId = { 2, 3, 4 }
    };

    var request = 
        (HttpWebRequest)WebRequest.Create(apiURLLocation + “”);
    request.ContentType = “application/json; charset=utf-8“;
    request.Accept = “application/json“;
    request.Method = “POST“;
    request.Headers.Add(HttpRequestHeader.Authorization, 

         apiAuthorization);
    request.UserAgent = “ApiRequest“;

    //Writes the ApiRequest Json object to request
    using (var streamWriter = new 
           StreamWriter(request.GetRequestStream()))
    {             

      streamWriter.Write(
           JsonConvert.SerializeObject(apiRequest));
      streamWriter.Flush();
    }

    var httpResponse = (HttpWebResponse)request.GetResponse();

    using (var streamreader = new  
           StreamReader(httpResponse.GetResponseStream()))
    using (var reader = new JsonTextReader(streamreader))
    {
      var storeInventory = 

          serializer.Deserialize<ApiResponse>(reader);
    }
  }
}


Some of the code shown is optional. I put the URL location and credentials into variables that are stored in the app.config file.  You can add this to your app.config file:


<appSettings>
    <add key=”ApiURLLocationvalue=” 

       http://www.franksdomain.com/WebApiDemo/api/MyApi/“/>
    <add key=”ApiCredentialvalue=”ABCD“/>
</appSettings>

The URL will need to be changed to match the URL that you setup on your IIS server (later in this blog post).  For now you can setup a redirect in your “hosts” file to match the domain in the app setting shown above.

Navigate to C:WindowsSystem32driversetc and edit the “hosts” file with a text editor.  You’ll see some sample text showing the format of a URL.  Create a domain name on a new line like this:

127.0.0.1        www.franksdomain.com

You can make up your own URL and you can use a URL that is real (technically, franksdomain.com is a real URL and it’s not mine).  If you use a real URL your computer will no longer be able to access that URL on the internet, it will redirect that URL to your IIS server (so be aware of this problem and try to avoid using real URLs).  The IP address 127.0.0.1 is a pointer to your local machine.  So we’re telling your browser to override www.franksdomain.com and redirect the request to the local machine.

Now you should be able to test up to the request.GetResponse() line of code.  That’s where the retriever will bomb.  Before we do this, we need to download and install Fiddler (assuming you don’t already have fiddler installed).  Click here to download fiddler and install it.  Now start up fiddler and you’ll see something like this:


Now run the retriever application until it bombs.  Fiddler will have one line in the left pane that is in red.  Click on it.  In the right pane, click on the “Inspectors” tab and then click on “JSON” sub-tab.  You should see something like this:


In the right side top pane, you’ll see your JSON data.  If your data is shown as a tree-view control then it is formatted correctly as JSON and not just text.  If you serialized your object incorrectly, you will normally see an empty box.  Notice that the store is set to “1” and there are three product ids being requested.

The Web API

The web API will be an MVC 4 application with one ApiController.  The API controller will use a POST method. 

So let’s start with a class that defines what information can be posted to this API.  This is the exact same class used in the retriever:

public class ApiRequest
{
  public int StoreId { get; set; }
  public List<int> ProductId { get; set; }
}


Make sure this class is not decorated with the [Serializable] attribute.  We’re going to use a [FromBody] attribute on the API and the object variables will not bind if this object is setup as serializable (I discovered this fact the hard way).  As you can see by the list definition we can pass a long list of product ids for one store at a time.  We expect to receive a list of inventory back from the API.


The response back to the calling application will be a list of inventory records containing the product id (which will be the same number we passed in the request), the name of the product and the quantity.  These are also the same objects used in the retriever:


public class InventoryRecord
{
  public int ProductId { get; set; }
  public string Name { get; set; }
  public int Quantity { get; set; }
}


public class ApiResponse
{
  public List<InventoryRecord> Records = new  

         List<InventoryRecord>();
}



The entire API controller in the MVC application looks like this:


public class MyApiController : ApiController
{
  [HttpPost]
  [ActionName(“GetInventory“)]
  public HttpResponseMessage GetInventory([FromBody] ApiRequest request)
  {
    if (request == null)
    {
      return Request.CreateErrorResponse(HttpStatusCode.BadRequest, 

             “Request was null“);
    }
       
    // check authentication
    var auth = ControllerContext.Request.Headers.Authorization;
       
    // simple demonstration of user rights checking.
    if (auth.Scheme != “ABCD“)
    {
      return Request.CreateErrorResponse(HttpStatusCode.BadRequest, 

             “Invalid Credentials“);
    }
   
    ApiResponse apiResponse = new ApiResponse();

    // read data from a database
    apiResponse.Records = DummyDataRetriever.ReadData(request.ProductId);

    // convert the data into json
    var jsonData = JsonConvert.SerializeObject(apiResponse);

    var resp = new HttpResponseMessage();
    resp.Content = new StringContent(jsonData, Encoding.UTF8, 
                   “application/json“);
    return resp;
  }
}


The controller is a post method controller that looks for an ApiRequest JSON object in the body of the posted information.  The first thing we want to check for is a null request.  That seems to occur most often when a bot crawls a website and hits an API.  If we’re lucky, then bots will not find their way in, but I always code for the worse case situation.  The next part will check the header for the authorization.  I didn’t cover this in the retriever, but I stuffed a string of letters in the authorization variable of the header.  This was setup to be “ABCD” but in a real application you’ll need to perform a database call to a table containing GUIDs.  These GUIDs can be assigned to another party to gain access to your API.  In this example the shopping website application will have its own GUID and each store will have a GUID that can be setup to restrict what each retriever can access.  For instance, the website GUID might have full access to every store to lookup information using this API, but a store might only have access to their own information, etc.  I’m only showing the basics of this method in this article.  I’ll cover this subject more thoroughly in a future blog post.

Next in the code is the dummy lookup for the information requested.  If you navigate to my sample dummy data retriever you’ll see that I just check to see which product is requested and stuff a record in the list.  Obviously, this is the place where you’ll code a database select and insert records into the list from the database.

Next, the list of inventory records are serialized into a JSON format and then attached to the content of the response message.  This is then returned.

Next, you’ll need to setup an IIS server to serve your API.

 

Setting up the IIS Server

I’m going to setup an application in IIS 7 to show how to get this API working from your PC.  Eventually, you’ll want to setup an IIS server on your destination server that you will be deploying this application to.

I’ll be using IIS 7 for this demo, if you’re using Windows 7 or earlier, you will probably need to download and install IIS 7 (the express version works too).  When you open IIS 7 you should see a treeview on the left side of the console:


Right-click on “Default Web Site” and “Add Application“.  Name it “WebApiDemo“.  You should now see this:


Now click on the “WebApiDemo” node and you’ll see a panel on the right side of the control panel named “actions“.  Click on the “Basic Settings” link.  Change the physical path to point to your project location for the API project (this is the MVC4 project that you created earlier or downloaded from my Github account).



You’ll notice that the application pool is set to “DefaultAppPool“.  We’ll need to change this to a .Net 4 pool.  So click on the “Application Pools” node and double-click on the “DefaultAppPool” line.  Change the .Net Framework Version to version 4:


At this point, if you try to access your API, you’ll discover that it doesn’t work.  That’s because we need to give IIS server access to your project directory.  So navigate to your project directory, right-click and go to properties, then click on the “Security” tab.  Click the “advanced” button and “Change Permissions” button.  Then click the “Add” button.  The first user you’ll need to add will be the IIS_IUSRS permission.  Click the “Check Names” button and it should add your machine name as the domain of this user (my machine name is “FRANK-PC”, yours will be different):


You’ll need to give IIS permissions.  I was able to make the API work with these permissions:

I would recommend doing more research before setting up your final API server.  I’m not going to cover this subject in this post.

Now click “OK”, “OK”, “OK”, etc.

Now run through those steps again: to add a user named IUSR, with the same permissions:




If you didn’t setup your URL in the hosts file, you’ll need to do that now. If you did this in the retriever section above, then you can skip this step.  

Navigate to C:WindowsSystem32driversetc and edit the “hosts” file with a text editor.  You’ll see some sample text showing the format of a URL.  Create a domain name on a new line like this:

127.0.0.1        www.franksdomain.com

Remember, you can makeup your own domain name.

Now, let’s test the site.  First, we need to determine what the URL will be when we access our API.  The easy method is to go into the IIS control panel and click on the “Browse urlname.com on *80 (http)” link:

Now you can copy the URL in the browser that popped up:

http://www.franksdomain.com/WebApiDemo


This URL is actually the URL to the MVC website in your application.  In order to access your API, your going to have to add to this URL:

http://www.franksdomain.com/WebApiDemo/api/MyApi

How did I get the “api/MyApi“?  If you go to your MVC application and navigate to the “App_Start/WebApiConfig.cs” file, you’ll see this default setup:

config.Routes.MapHttpRoute(
        name: “DefaultApi“,
        routeTemplate: “api/{controller}/{id}“,
        defaults: new { id = RouteParameter.Optional }
);



So the URL contains “api” and then the controller name.  Remember the previous code for the API controller:

public class MyApiController : ApiController

Ignore the “Controller” part of the class name and you’ll have the rest of your path.  Put this path string in the app.config file of your retriever application and compile both applications.  

Startup fiddler.  Run your retriever.  Your retriever should run through without any errors (unless you or I missed a step).  In fiddler, select JSON for the top right and bottom right panes. You should see something like this:

Notice that the retriever sent a request for product ids 2,3,4 and the API returned detailed product information for those three products (Spoon, Fork and Knife).  You might need to expand the tree-views in your JSON panels to see all the information like that shown in the screen-shot above.


Setting a Breakpoint in the API Application


Your API application doesn’t really run until it is accessed by IIS.  So we need to run the retriever once and then we can set our debugger to attach to the application process.  Once you run your retriever, go to your API application, in Visual Studio click on the “Debug” menu, then select “attach to process“.  Then make sure the “Show processes from all users” is checked at the bottom of the dialog box.  Then find the process named “w3wp.exe” and click on it. 



Click the “attach” button.  Then click the “attach” button in the pop-up dialog.  You’ll notice that your application is now “running” (you can tell by the red stop button in your tool bar):


Put a break-point in your code.  Run the retriever program and you see your API project pop-up with your break-point waiting for you.  Now you can step through your program and inspect variables (such as the request variables that were received, hint, hint) just like you ran this program using the F-5 key.



Summary

There are a lot of little things that must occur correctly for this API to work properly.  Here are a few things to look out for:

– Your firewall could also block requests, so be on the lookout for that problem.  
– If you don’t have the correct URL setup in your retriever, you will not get a connection.  
– The objects you use to serialize and de-serialize JSON data must match between your retriever and API applications.  
– The IIS server must have the correct rights to the API application directory.  
– You need to setup the correct .NET type in your IIS application pool.
– Verify that IIS is running on your PC.
– Verify your inputs on your API by attaching to the w3wp.exe process and breaking on the first line of code.
– Verify your output from your retriever using fiddler.

Getting the Sample Projects

You can download the two sample projects from my GitHub account:  
https://github.com/fdecaire/WebApiDemo 

 

SOA, JSON, WCF, REST

SOA

I’m currently working my way through the subject of Service Oriented Architecture, or SOA.  I have a book on the way, but I’m getting a jump on the subject.  In previous blog posts I’ve talked about the necessity to do research.  This is one of those instances where I have applications to do some task and I need to learn the technology to make it happen.  Sometimes it can take a couple weeks to learn a new technology, type in a few test programs and then obtain that “aha” moment.  Once that occurs, then it’s time to design and implement.  In the case of SOA, I haven’t quite reached the implementation “aha” moment yet.  I understand the overall purpose and what I will be using SOA for is to transfer data between iPhone apps, iPad apps and our database.

So I’ve done a bit of digging and I’ve stumbled across a few technologies that I’ve seen in the past but paid no attention to (due to the fact that I had no purpose for them before).  One technology I’m focusing my attention on today is WCF.  WCF is the Windows Communication Foundation and allows two apps to communicate.  Primarily an app (any application) with a service running on a different server.  I plan to run a service on a server in the computer room and make a secured connection (probably SSL) between a mobile device and the server.  The service will be able to read and write data to the database.  This will enable our field personnel to record information with their iPhone or iPad.

WCF

Before I get to the whole design and implementation, I have to test a few technologies to see how I can apply them.  I’ve already tested JSON and I’m satisfied with my knowledge of JSON for now.  At this point, I can design a software specification that indicates how JSON will be used and I can either code this myself or pass on the information to a programmer that will be implementing the actual coding.

I found this very simple example of a client and server test setup for WCF:

A truely simple example to get started with WCF

So I copied the code into a console application and I had to mess around with it a bit to get it to work right.  One thing I changed that was not indicated in the article is that I put the server and contract into a separate project of the same solution.  I had to do that in order to keep the complier from complaining about two entry points.  Technically, I have a solution with two console applications in it.  One is used as a module for the other.  It works, and it’s just a R&D test case.  My next step is to separate these sections and run them on separate PC’s to see if they’ll communicate correctly.  After that, I’ll put the server inside a service component and test it under that condition. 

REST

Representational State Transfer (REST) is not a subject that I am currently familiar with.  That’s my next subject to study.  I will be focusing on this subject next.  Rest is supposed to be lightweight and simpler than RPC.  I will be judging for myself.  Not that I’m skeptical, I just want to see how this works.

JSON

As I mentioned before, I’ve already done some R&D on JSON and I’m impressed.  My plans are to use JSON instead of XML for structured data transfers. 

Summary

My point in this posting is to show that nobody is an expert in every subject.  They best way to become an expert in a subject is to keep your eyes open to what technologies are available.  Sometimes I see an acronym that I’ve never seen before.  It starts to get a lot of hype in magazines and news articles.  So I lookup the definition or wiki on the acronym and see what the actual purpose is.  If it’s something that could revolutionize my projects, then I start to investigate it more (or I’ll relegate a software developer to R&D it and report back to me with a small demo).  Once a demo has been created, then I can evaluate the usage of the new technology and incorporate it into our business plans. 

Other times, I have a problem to solve and I know that other companies are doing this very task (i.e. talking between iPhone apps and their databases).  So I start searching for information on the Internet.  It usually doesn’t take me long to come up with one or more technologies that are candidates to becoming my solution.  After that, I just work my way through each innovation until I’m satisfied with the solution I want to work with.

In this business you can never stop learning new things.

 

JSON

So my latest endeavor is to learn JSON.  I mentioned in an earlier post how I do research all the time.  Well, I stumbled across a subject called SOA (Service Oriented Architecture) and I started to read articles on the Internet and decided that this is a major subject that I wanted to know about.  So I bought a book (and of course, I’m still waiting for the book to arrive).  In the mean time, I discovered this data exchange format called JSON.  One of the first projects I tested reads data from a currency service.  The code and instructions are located here:

Use C# to get JSON Data from the Web and Map it to .NET Class => Made Easy!

It’s from codeproject, and it worked the first time.  One step that was different involved the currency exchange service.  They changed their requirements and they require an id to download their data.  They also have a different link, but it’s easy to get this information by going to their site:

Open Exchange Org

Click the “Get Instant Access” button and look for the little “free” link at the bottom. You can enter your email information and get an account for development purposes for free.  This will then lead you to a webpage that gives you an “App ID” and instructions including the actual link you need to call to receive data in JSON format.


Back to JSON

So I’m looking at how JSON is able to get data from a remote website and use it within my own website.  Then I discovered a weather service that is free and provides a JSON API.  The reason this interested me is that my company does site management for roofing.  What site managers do is they watch the roofing contractors install the roof and they mark of the information on a daily quality report.  The quality report includes information about the local weather.  The purpose of recording all this information is to resolve disputes in the future if a roof system fails.  Here’s a list of weather services that provide JSON data:

Weather services using JSON

and here’s the free one that I’m currently looking at:

Free Weather API for developers

What’s the purpose?

The weather information that can be read from this website can be transferred to my website by clicking a button.  My intention is to be able to read the weather for the current location at a designated date (site managers can create a quality report after they have performed the site inspection, so they might want to record yesterday’s data, for instance), then pre-populate the weather fields in the quality report.  This relieves the site manager from going to a weather service and looking up what the official weather was for that date (site managers don’t carry weather measuring equipment like thermometers and anemometers).

My real purpose in doing these exercises is to be able to communicate between two processes on different computers.  I have developed “Service” type applications that sit on a dedicated machine and poll a database queue waiting for some data to crunch.  The purpose of doing this is to be able to process lengthy data from a website without requiring the web consumer to stay logged in while the data is being processed.  Web users can process huge numbers of data points and come back and check the progress later.  Meanwhile the data is being processed by a machine in the computer room that is running a service type program.  One major inefficiency of polling in general, is that it is wasting time and resource constantly reading the database to see if there is something in the queue.  Over 99% of the time, there is nothing in the queue to process, so this constant querying (usually every 2 seconds) is consuming resources.  Also, the poll time determines how fast the process will respond to a request.  In order to make this all more efficient, it would be better to queue the items needed to be processed, then send a JSON message to the service to indicate that it needs to process a bunch of data.  That way the remote service application doesn’t have to constantly poll the database.


So what is JSON anyway?

JSON is a lot like XML.  It’s a data transmission format.  It’s very clever in that it’s more compact than XML and it can be directly translated into an object in C# or JavaScript (and many other languages).  It’s also language independent.  For example, a service running Python could provide a JSON encoded data string that can be consumed by a C# program.  This means that a mainframe computer system could communicate with an IIS server without requiring a programmer with knowledge of both languages used.  Two programmers could agree on the JSON format between the two machines and they could each create an interface to communicate between machines.

My motto: Never stop learning.