XML Serialization

Summary

In this post I’m going to demonstrate the proper way to serialize XML and setup unit tests using xUnit and .Net Core.  I will also be using Visual Studio 2017.

Generating XML

JSON is rapidly taking over as the data encoding standard of choice.  Unfortunately, government agencies are decades behind the technology curve and XML is going to be around for a long time to come.  One of the largest industries industries still using XML for a majority of their data transfer encoding is the medical industry.  Documents required by meaningful use are mostly encoded in XML.  I’m not going to jump into the gory details of generating a CCD.  Instead, I’m going to keep this really simple.

First, I’m going to show a method of generating XML that I’ve seen many times.  Usually coded by a programmer with little or no formal education in Computer Science.  Sometimes programmers just take a short-cut because it appears to be the simplest way to get the product out the door.  So I’ll show the technique and then I’ll explain why it turns out that this is a very poor way of designing an XML generator.

Let’s say for instance we wanted to generate XML representing a house.  First we’ll define the house as a record that can contain square footage.  That will be the only data point assigned to the house record (I mentioned this was going to be simple right).  Inside of the house record will be lists of walls and lists of roofs (assume a house could have two or more roofs like a tri-level configuration).  Next, I’m going to make a list of windows for the walls.  The window block will have a “Type” that is a free-form string input and the roof block will also have a “Type” that is a free-form string.  That is the whole definition.

public class House
{
  public List Walls = new List();
  public List Roofs = new List();
  public int Size { get; set; }
}

public class Wall
{
  public List Windows { get; set; }
}

public class Window
{
  public string Type { get; set; }
}

public class Roof
{
  public string Type { get; set; }
}

The “easy” way to create XML from this is to use the StringBuilder and just build XML tags around the data in your structure.  Here’s a sample of the possible code that a programmer might use:

public class House
{
  public List<Wall> Walls = new List<Wall>();
  public List<Roof> Roofs = new List<Roof>();
  public int Size { get; set; }

  public string Serialize()
  {
    var @out = new StringBuilder();

    @out.Append("<?xml version=\"1.0\" encoding=\"utf-8\"?>");
    @out.Append("<House xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xmlns:xsd=\"http://www.w3.org/2001/XMLSchema\">");

    foreach (var wall in Walls)
    {
      wall.Serialize(ref @out);
    }

    foreach (var roof in Roofs)
    {
      roof.Serialize(ref @out);
    }

    @out.Append("<size>");
    @out.Append(Size);
    @out.Append("</size>");

    @out.Append("</House>");

    return @out.ToString();
  }
}

public class Wall
{
  public List<Window> Windows { get; set; }

  public void Serialize(ref StringBuilder @out)
  {
    if (Windows == null || Windows.Count == 0)
    {
      @out.Append("<wall />");
      return;
    }

    @out.Append("<wall>");
    foreach (var window in Windows)
    {
      window.Serialize(ref @out);
    }
    @out.Append("</wall>");
  }
}

public class Window
{
  public string Type { get; set; }

  public void Serialize(ref StringBuilder @out)
  {
    @out.Append("<window>");
    @out.Append("<Type>");
    @out.Append(Type);
    @out.Append("</Type>");
    @out.Append("</window>");
  }
}

public class Roof
{
  public string Type { get; set; }

  public void Serialize(ref StringBuilder @out)
  {
    @out.Append("<roof>");
    @out.Append("<Type>");
    @out.Append(Type);
    @out.Append("</Type>");
    @out.Append("</roof>");
  }
}

The example I’ve given is a rather clean example.  I have seen XML generated with much uglier code.  This is the manual method of serializing XML.  One almost obvious weakness is that the output produced is a straight line of XML, which is not human-readable.  In order to allow human readable XML output to be produced with an on/off switch, extra logic will need to be incorporated that would append the newline and add tabs for indents.  Another problem with this method is that it contains a lot of code that is unnecessary.  One typo and the XML is incorrect.  Future editing is hazardous because tags might not match up if code is inserted in the middle and care is not taken to test such conditions.  Unit testing something like this is an absolute must.

The easy method is to use the XML serializer.  To produce the correct output, it is sometimes necessary to add attributes to properties in objects to be serialized.  Here is the object definition that produces the same output:

public class House
{
  [XmlElement(ElementName = "wall")]
  public List Walls = new List();

  [XmlElement(ElementName = "roof")]
  public List Roofs = new List();

  [XmlElement(ElementName = "size")]
  public int Size { get; set; }
}

public class Wall
{
  [XmlElement(ElementName = "window")]
  public List Windows { get; set; }

  public bool ShouldSerializenullable()
  {
    return Windows == null;
  }
}

public class Window
{
  public string Type { get; set; }
}

public class Roof
{
  public string Type { get; set; }
}

In order to serialize the above objects into XML, you use the XMLSerializer object:

public static class CreateXMLData
{
  public static string Serialize(this House house)
  {
    var xmlSerializer = new XmlSerializer(typeof(House));

    var settings = new XmlWriterSettings
    {
      NewLineHandling = NewLineHandling.Entitize,
      IndentChars = "\t",
      Indent = true
    };

    using (var stringWriter = new Utf8StringWriter())
    {
      var writer = XmlWriter.Create(stringWriter, settings);
      xmlSerializer.Serialize(writer, house);

      return stringWriter.GetStringBuilder().ToString();
    }
  }
}

You’ll also need to create a Utf8StringWriter Class:

public class Utf8StringWriter : StringWriter
{
  public override Encoding Encoding
  {
    get { return Encoding.UTF8; }
  }
}

Unit Testing

I would recommend unit testing each section of your XML.  Test with sections empty as well as containing one or more items.  You want to make sure you capture instances of null lists or empty items that should not generate XML output.  If there are any special attributes, make sure that the XML generated matches the specification.  For my unit testing, I stripped newlines and tabs to compare with a sample XML file that is stored in my unit test project.  As a first-attempt, I created a helper for my unit tests:

public static class XmlResultCompare
{
  public static string ReadExpectedXml(string expectedDataFile)
  {
    var assembly = Assembly.GetExecutingAssembly();
    using (var stream = assembly.GetManifestResourceStream(expectedDataFile))
    {
      using (var reader = new StreamReader(stream))
      {
        return reader.ReadToEnd().RemoveWhiteSpace();
      }
    }
  }

  public static string RemoveWhiteSpace(this string s)
  {
    s = s.Replace("\t", "");
    s = s.Replace("\r", "");
    s = s.Replace("\n", "");
  return s;
  }
}

If you look carefully, I ‘m compiling my xml test data right into the unit test dll.  Why am I doing that?  The company that I work for as well as most serious companies use continuous integration tools such as a build server.  The problem with a build server is that your files might not make it to the same directory location on the build server that they are on your PC.  To ensure that the test files are there, compile them into the dll and reference them from the namespace using Assembly.GetExecutingAssembly().  To make this work, you’ll have to mark your xml test files as an Embedded Resource (click on the xml file and change the Build Action property to Embedded Resource).  To access the files, which are contained in a virtual directory called “TestData”, you’ll need to use the name space, the virtual directory and the full file name:

XMLCreatorTests.TestData.XMLHouseOneWallOneWindow.xml

Now for a sample unit test:

[Fact]
public void TestOneWallNoWindow()
{
  // one wall, no windows
  var house = new House { Size = 2000 };
  house.Walls.Add(new Wall());

  Assert.Equal(XmlResultCompare.ReadExpectedXml("XMLCreatorTests.TestData.XMLHouseOneWallNoWindow.xml"), house.Serialize().RemoveWhiteSpace());
}

Notice how I filled in the house object with the size and added one wall.  The ReadExpectedXml() method will remove whitespaces automatically, so it’s important to remove them off the serialized version of house in order to match.

Where to Get the Code

As always you can go to my GitHub account and download the sample application (click here).  I would recommend downloading the application and modifying it as a test to see how all the piece work.  Add a unit test to see if you can match your expected xml with the xml serializer.

 

 

 

Exporting XML From MS SQL Server

Summary

This is going to be a short post on how to query an MS SQL Server database and produce XML output.

The Query

There are two formats that you can use when you want to convert a query result into XML, Elements format and Properties Format.  The following query will select from a table named “product” and put the result set into elements:

SELECT * FROM product FOR XML RAW ('product'), ROOT('data'), ELEMENTS
 
 

The results window will look something like this:
 

Click on the url link and the output should look like this:

<data>
  <product>
    <ProductId>1</ProductId>
    <ProductType>1</ProductType>
    <Name>Matchbox Car</Name>
    <store>1</store>
  </product>
  <product>
    <ProductId>2</ProductId>
    <ProductType>1</ProductType>
    <Name>Baby Rattle</Name>
    <store>1</store>
  </product>
  <product>
    <ProductId>3</ProductId>
    <ProductType>1</ProductType>
    <Name>Lego Bricks</Name>
    <store>1</store>
  </product>
  <product>
    <ProductId>4</ProductId>
    <ProductType>2</ProductType>
    <Name>Steak</Name>
    <store>1</store>
  </product>
</data>

If you want to flatten out your XML, you can use this syntax:

SELECT * FROM product FOR XML RAW ('product')

Which will produce this output:

<product ProductId=”1ProductType=”1Name=”Matchbox Carstore=”1” />
<product ProductId=”2ProductType=”1Name=”Baby Rattlestore=”1” />
<product ProductId=”3ProductType=”1Name=”Lego Bricksstore=”1” />
<product ProductId=”4ProductType=”2Name=”Steakstore=”1” />

As you can see by the last example, you can leave the “Root” keyword off the query if you don’t want to the root data tags added.

Producing JSON

If you need to produce JSON, you will need to do it using another tool.  Use the For XML RAW command above, including the root tag (the output must be well-formed.  Then go to this site (or any other free xml to json converter website): thomasfrank.se XML to JSON – a converter

Paste your xml into the window and hit the convert button.

 

XML Serializing Nullable Optional Attribute

Summary

The title of this blog post is a bit of a mouth-full.  I do a lot of xml serialization and de-serialization.  It’s all part of the new paradigm of using APIs to communicate with other systems over the Internet.  One of the annoying “features” of the xml serializer is that it doesn’t support nullable attributes.  It’ll serialize nullable elements, but not attributes.  So I’m going to show how to serialize nullable attributes and make the attribute optional.

The Problem

Here’s the example code of an XML serializer that will not work

public class House
{
    [XmlElement]
    public List<Room> rooms = new List<Room>();
}

public class Room
{
    [XmlAttribute(AttributeName = “name“)]
    public string Name { get; set; }

    [XmlAttribute(AttributeName = “windows“)]
    public int? NumberOfWindows { get; set; }       
}

In this instance I’m attempting to serialize a nullable integer named “NumberOfWindows”.  My goal is to produce an XML file that looks something like this (I removed the schema info to make this easier to read):

<?xml version=”1.0encoding=”utf-8“?>
  <rooms name=”kitchenwindows=”2” />
  <rooms name=”bathroomwindows=”0” />
  <rooms name=”closet” />
</House>

Notice how the “closet” doesn’t have any windows.  For a closet, we’ll assume that a window does not apply.  So the windows property must be nullable and it must be optional.

How to Fix it
 
First, let’s change the Room class so that it will serialize without getting an error.  The first thing to note is that we can turn the NumberOfWindows parameter into a string, and treat an empty string as the null value:

public class Room
{
    [XmlAttribute(AttributeName = “name“)]
    public string Name { get; set; }

    [XmlIgnore]
    public int? NumberOfWindows { get; set; }

    [XmlAttribute(AttributeName = “windows“)]
    public string WindowsSerializable 
    {
        get
        {
            if (NumberOfWindows != null)
            {
                return NumberOfWindows.ToString();
            }
            else
            {
                return “”;
            }
        }
        set
        {
            if (WindowsSerializable != null)
            {
                NumberOfWindows = int.Parse(WindowsSerializable);
            }
        }
    }
}

So the NumberOfWindows variable is checked for null and if it is then return an empty string.  That will cause the closet to return: windows=””, which is not quite what we want.  But at least it will execute and generate an xml output without causing an error.  Also, notice that I put an XmlIgnore on the variable that will be populated, but not used to generate the serialized output.

Now we need to make the NumberOfWindows attribute optional.  To make it optional we can add this to the end of the WindowsSerializable getter:

public bool ShouldSerializeWindowsSerializable()
{
    return NumberOfWindows.HasValue;
}


You can also do something like this:

public bool ShouldSerializeWindowsSerializable()
{
    return WindowsSerializable != “”;
}

The “ShouldSerialize{varname}” method will output your results for the variable indicated if it returns true.  So you can put any fancy logic in this method that you want to show or hide the attribute of your choice.


Where to Get the Code

You can go to my github account and download the sample code by clicking here.


 

 

Returning XML or JSON from a Web API

Summary

In my last blog post I demonstrated how to setup a Web API to request data in JSON format.  Now I’m going to show how to setup your API so that a request can be made to ask for XML as well as JSON return data.  To keep this simple, I’m going to refactor the code from the previous blog post to create a new retriever method that will set the Accept parameter to xml instead of json.

Changes to the Retriever

I copied the retriever method from my previous code and created a retriever that asks for XML data:

public void XMLRetriever()
{
    var xmlSerializer = new XmlSerializer(typeof(ApiResponse));

    var apiRequest = new ApiRequest
    {
        StoreId = 1,
        ProductId = { 2, 3, 4 }
    };

    var request = (HttpWebRequest)WebRequest.Create(
        apiURLLocation + “”);
    request.ContentType = “application/json; charset=utf-8“;
    request.Accept = “text/xml; charset=utf-8“;
    request.Method = “POST“;
    request.Headers.Add(HttpRequestHeader.Authorization, 

            apiAuthorization);
    request.UserAgent = “ApiRequest“;

    //Writes the ApiRequest Json object to request
    using (var streamWriter = new  
           StreamWriter(request.GetRequestStream()))
    {
        streamWriter.Write(JsonConvert.SerializeObject(apiRequest));
        streamWriter.Flush();
    }

    var httpResponse = (HttpWebResponse)request.GetResponse();

    // receives xml data and deserializes it.
    using (var streamreader = new  
           StreamReader(httpResponse.GetResponseStream()))
    {
        var storeInventory = 

            (ApiResponse)xmlSerializer.Deserialize(streamreader);
    }
}

There are two major changes in this code: First, I changed the “Accept” to ask for xml.  Second, I recoded the return data to deserialize it as xml instead of json.  I left the api request to be in JSON.


Changes to the API Application

I altered the API controller to detect which encoding is being requested.  If Accept contains the string “json” then the data is serialized using json.  If Accept contains the string “xml” then the data is serialized using xml.  Otherwise, an error is returned.

Here is the new code for the API controller:

var encoding = ControllerContext.Request.Headers.Accept.ToString();
if (encoding.IndexOf(“json“,  

    StringComparison.OrdinalIgnoreCase) > -1)
{
    // convert the data into json
    var jsonData = JsonConvert.SerializeObject(apiResponse);

    var resp = new HttpResponseMessage();
    resp.Content = new StringContent(jsonData, Encoding.UTF8, 
        “application/json“);
    return resp;
}
else if (encoding.IndexOf(“xml“,  

         StringComparison.OrdinalIgnoreCase) > -1)
{
    // convert the data into xml
    var xmlSerializer = new XmlSerializer(typeof(ApiResponse));

    using (StringWriter writer = new StringWriter())
    {
        xmlSerializer.Serialize(writer, apiResponse);

        var resp = new HttpResponseMessage();
        resp.Content = new StringContent(writer.ToString(), 
             Encoding.UTF8, “application/xml“);
        return resp;
    }
}
else
{
    return Request.CreateErrorResponse(HttpStatusCode.BadRequest, 

           “Only JSON and XML formats accepted“);
}


Compile the API application and then startup fiddler.  Then run the retriever.  In fiddler, you should see something like this (you need to change your bottom right sub-tab to xml):




Download the Source


You can go to my GitHub account and download the source here: https://github.com/fdecaire/WebApiDemoJsonXml

 

 

Visual Studio Project Files

Summary

In this blog post I’m going to talk about Visual Studio’s proj file which is nothing more than an XML file used to keep track of the resources inside a project.  


The Setup

Open your Visual Studio (I’m currently using VS 2012) and create a console application.  If you have your solution explorer window open you should see something like this:


Now, let’s create a cs file in the solution and make an empty class.  I just right-clicked on the project (ProjectFileBlogPost in this example)  then selected “Add” then “Class”.  Now click on your project to make sure it’s selected then go to your project menu (on the main menu bar) and select “Show All Files”.  Right-click on your TestClass.cs file and select “Exclude From Project”.  The file will turn white, indicating that it is not part of the project and will not be compiled with your solution.

If you’ve been working with Visual Studio for any length of time, you probably already know how all this works because you can copy a cs file from another project from Windows Explorer and then include it in your project to make it appear in the solution explorer.  Now I’m going to show how the XML project file works.


The Project File

Now we’re going to use Visual Studio to edit the xml file.  First, you need to close your console application (because it locks the file).  Next you’ll need to go to the File menu and open a file (just a plain file, not a project).  Once you open the file, VS will recognize it as an XML file and color code the text accordingly.  

If you scroll through the XML, you’ll noticed that there are a bunch of ItemGroup sections.  The one we’re interested in is one that contains a bunch of “Compile” nodes:


As you can see only the Program.cs file and the AssemblyInfo.cs files are in this group.  If you copy the Program.cs node and change the file name to TestClass.cs and save this file, you’ll be able to open this project in VS and the TestClass.cs file will be included in the project.


If you delete any of these “Compile” nodes, then the file will no longer be included in the project by VS.


Folders

If your cs file is in a folder then the full path can be spelled out in the “Compile” node and the folder will be included in your project as well as the file.  However, if you want an empty folder to be included in your project, you’ll need to add it to a different ItemGroup node set.  These nodes are named Folder nodes:


If a folder node exists and there are references in the group of Compile nodes, VS will be OK with that, but it’s not necessary.


Manipulating The Project File

At this point, you probably have a program that generates some sort of C# files and you want those files to be automatically added to some utility project.  It’s best to use the XmlDocument object to manipulate the data in the project file to add any of your C# files and/or directories.  Text parsing can be a nightmare.  You’ll need to use a name space to make it work:

XmlDocument doc = new XmlDocument();
doc.Load(projectFileName);

var nsmgr = new XmlNamespaceManager(doc.NameTable);
nsmgr.AddNamespace(“a“, 
        “http://schemas.microsoft.com/developer/msbuild/2003“);
XmlNodeList itemGroupNodes = doc.SelectNodes(

        “//a:Project/a:ItemGroup“, nsmgr);

// add and delete nodes here

doc.Save(projectFileName);


You can spin through all the ItemGroup nodes and look for child nodes that have the name “Folder” or “Compile”, then add Folder or Compile nodes as desired.  You’ll have to account for the possibility that there are no ItemGroup nodes with Folder or Compile nodes in them.  You can use something like this to find the proper ItemGroup nodes:

foreach (XmlNode itemGroupNode in itemGroupNodes)
{
    XmlNodeList childNodes = itemGroupNode.ChildNodes;

    foreach (XmlNode childNode in childNodes)
    {
        if (childNode.Name == “Folder“)
        {
            // your code here
        }
    }
}

In order to append a new node to your Folder node, you can do something like this:

XmlNode folderNode = doc.CreateNode(XmlNodeType.Element, 
     “Folder“, 
     “http://schemas.microsoft.com/developer/msbuild/2003“);
XmlAttribute xKey = doc.CreateAttribute(“Include“);
xKey.Value = “TestFolder“;
folderNode.Attributes.Append(xKey);
itemGroupNode.AppendChild(folderNode);



Another thing to note: You should check the namespace name in the top of your project file.  I copied that text into these sample snippets, but I’m using VS 2012 and there is no guarantee that the namespace url won’t change for newer versions of VS.