REST easy with kbmMW #24 – XML, JSON or YAML to object conversion

How often have you received an XML or JSON or for that matter a YAML file that you need to interpret?

I know I personally have received many from customers or customer partners who would like a new customer system to integrate nicely with them, transferring XML or JSON files.

Sometimes you may get an XSD (an XML Schema Definition file), which explains how the XML document is formatted, which is great, because then you can use kbmMW’s XSD importer to interpret the XSD and generate Delphi class definitions making it easy to read and/or write the XML file according to the format given in the XSD. In fact, having the XSD, makes it easy to also read and write the data from and to JSON and other kbmMW supported object notation formats.

However I can’t count the times I have received an example of the actual data file, rather than the XSD definition.

To interpret the file in such case, there are two options

• to use a generic XML or JSON or YAML reader to read the file, and manually extract required data from it
• or to handcode Delphi class definitions with relevant kbmMW attributes, making it easy for kbmMW to both parse and generate such files

However the first option is not a very generic way to do things, and it does not help with writing a compatible file. The second option can be time consuming, because you need to analyse the file to determine its structure, to be able to generate the relevant object classes and lists, not to mention augmenting the classes and lists with relevant attributes.

Fortunately kbmMW comes to the rescue…again 🙂 That’s the whole point of programming… to make life easier… so when I notice stuff that could make my life easier, kbmMW receives a new feature, hopefully making the (programming) life of kbmMW users easier too.

So next release of kbmMW contains built in support for parsing and analysing data files (XML, JSON and YAML for the time being), and generates relevant attribute augmented Delphi classes, making it easy to use kbmMW’s marshalling framework to read and write such files.

Lets look at a couple of files and the code generated for handling them.

One file is called auctions.json. I found it somewhere on the internet, but it is just an example. Any other JSON file could have been used.

{
"realm":{"name":"Aegwynn","slug":"aegwynn"},
"alliance":{"auctions":[
	{"auc":1972333274,"item":22574,"owner":"Schäuble","bid":51300,"buyout":54000,"quantity":1,"timeLeft":"VERY_LONG","rand":0,"seed":30355200},
	{"auc":1970925966,"item":22446,"owner":"Aenni","bid":1890000,"buyout":2450000,"quantity":10,"timeLeft":"LONG","rand":0,"seed":1280052352},
	{"auc":1972187088,"item":82800,"owner":"Ainshu","bid":4329000,"buyout":4329000,"quantity":1,"timeLeft":"VERY_LONG","rand":0,"seed":955897088,"petSpeciesId":122,"petBreedId":5,"petLevel":1,"petQualityId":2},
	{"auc":1972201216,"item":51950,"owner":"Ulath","bid":119157,"buyout":121590,"quantity":3,"timeLeft":"VERY_LONG","rand":0,"seed":1109650432},
... +60000 entries
]},
"horde":{"auctions":[
	{"auc":1970970808,"item":82952,"owner":"Guldarak","bid":4004290,"buyout":4004426,"quantity":1,"timeLeft":"LONG","rand":0,"seed":950078272},
	{"auc":1971973942,"item":4306,"owner":"Blutrabé","bid":94500,"buyout":99999,"quantity":20,"timeLeft":"VERY_LONG","rand":0,"seed":956528896},
	{"auc":1971973992,"item":4306,"owner":"Blutrabé","bid":94500,"buyout":99999,"quantity":20,"timeLeft":"VERY_LONG","rand":0,"seed":1476867968},
	{"auc":1972449149,"item":87893,"owner":"Thêlon","bid":2375000,"buyout":2500000,"quantity":1,"timeLeft":"VERY_LONG","rand":0,"seed":848616448},
	{"auc":1971763986,"item":74705,"owner":"Gondoline","bid":15558129,"buyout":16662661,"quantity":1,"timeLeft":"LONG","rand":0,"seed":577727872},
... +2000 entries
]},
"neutral":{"auctions":[
	{"auc":1971600068,"item":72145,"owner":"Mellkore","bid":59990000,"buyout":59990000,"quantity":1,"timeLeft":"VERY_LONG","rand":0,"seed":436886144},
	{"auc":1972336265,"item":8485,"owner":"Zaruman","bid":351500,"buyout":450000,"quantity":1,"timeLeft":"VERY_LONG","rand":0,"seed":0},
	{"auc":1972336316,"item":8487,"owner":"Zaruman","bid":351500,"buyout":450000,"quantity":1,"timeLeft":"VERY_LONG","rand":0,"seed":0},
	{"auc":1972336611,"item":8487,"owner":"Zaruman","bid":351500,"buyout":450000,"quantity":1,"timeLeft":"VERY_LONG","rand":0,"seed":0},
	{"auc":1972336922,"item":8488,"owner":"Zaruman","bid":351500,"buyout":450000,"quantity":1,"timeLeft":"VERY_LONG","rand":0,"seed":0},
	{"auc":1972336986,"item":8488,"owner":"Zaruman","bid":351500,"buyout":450000,"quantity":1,"timeLeft":"VERY_LONG","rand":0,"seed":0},
	{"auc":1971335148,"item":76085,"owner":"Splatthy","bid":4418750,"buyout":4950000,"quantity":5,"timeLeft":"VERY_LONG","rand":0,"seed":2058173312},
	{"auc":1972336837,"item":8487,"owner":"Zaruman","bid":351500,"buyout":450000,"quantity":1,"timeLeft":"VERY_LONG","rand":0,"seed":0},
	{"auc":1972336524,"item":8486,"owner":"Zaruman","bid":351500,"buyout":450000,"quantity":1,"timeLeft":"VERY_LONG","rand":0,"seed":0},
	{"auc":1971335152,"item":76085,"owner":"Splatthy","bid":4418750,"buyout":4950000,"quantity":5,"timeLeft":"VERY_LONG","rand":0,"seed":2058173312},
	{"auc":1972336351,"item":8485,"owner":"Zaruman","bid":351500,"buyout":450000,"quantity":1,"timeLeft":"VERY_LONG","rand":0,"seed":0},
	{"auc":1972386885,"item":4338,"owner":"Huntez","bid":90284,"buyout":95038,"quantity":15,"timeLeft":"VERY_LONG","rand":0,"seed":944449152},
	{"auc":1971335163,"item":76085,"owner":"Splatthy","bid":17675000,"buyout":19800000,"quantity":20,"timeLeft":"VERY_LONG","rand":0,"seed":1033557888},
	{"auc":1972336774,"item":8487,"owner":"Zaruman","bid":351500,"buyout":450000,"quantity":1,"timeLeft":"VERY_LONG","rand":0,"seed":0},
	{"auc":1971335155,"item":76085,"owner":"Splatthy","bid":4418750,"buyout":4950000,"quantity":5,"timeLeft":"VERY_LONG","rand":0,"seed":2058173312},
	{"auc":1972336383,"item":8485,"owner":"Zaruman","bid":351500,"buyout":450000,"quantity":1,"timeLeft":"VERY_LONG","rand":0,"seed":0},
	{"auc":1972336563,"item":8486,"owner":"Zaruman","bid":351500,"buyout":450000,"quantity":1,"timeLeft":"VERY_LONG","rand":0,"seed":0},
	{"auc":1972386607,"item":12808,"owner":"Huntez","bid":71499,"buyout":79999,"quantity":1,"timeLeft":"VERY_LONG","rand":0,"seed":266490112},
	{"auc":1971335149,"item":76085,"owner":"Splatthy","bid":4418750,"buyout":4950000,"quantity":5,"timeLeft":"VERY_LONG","rand":0,"seed":2058173312},
	{"auc":1972336707,"item":8486,"owner":"Zaruman","bid":351500,"buyout":450000,"quantity":1,"timeLeft":"VERY_LONG","rand":0,"seed":0},
	{"auc":1972386682,"item":16249,"owner":"Huntez","bid":11250,"buyout":0,"quantity":1,"timeLeft":"VERY_LONG","rand":0,"seed":747154816},
	{"auc":1972386894,"item":4338,"owner":"Huntez","bid":90284,"buyout":95038,"quantity":15,"timeLeft":"VERY_LONG","rand":0,"seed":944449152},
	{"auc":1972337130,"item":8488,"owner":"Zaruman","bid":351500,"buyout":450000,"quantity":1,"timeLeft":"VERY_LONG","rand":0,"seed":0},
	{"auc":1972336425,"item":8485,"owner":"Zaruman","bid":351500,"buyout":450000,"quantity":1,"timeLeft":"VERY_LONG","rand":0,"seed":0}]}
}

As you can see, its fairly big, and relatively complex with multiple arrays of objects.

We make kbmMW analyse it and output the relevant Delphi based source code, that makes it easy to marshal/unmarshal such data:

var
   s:string;
...
   s:=TkbmMWJSONMarshal.GenerateDelphiClassFromUTF8File('auctions.json','Unit2','JSONData');
...

What happens is that we ask the JSON marshalling class to generate the source code for the classes needed for us, matching the contents of auctions.json, producing a unit called Unit2 and a main class called TJSONData. The reason for the main class, is that the main object in the json file is unnamed or anonymous, hence we need to provide some name for the generator to use in the Delphi code.

Simply write the contents of s to a file called Unit2.pas and add it to the projects where you need to read or write auctions.json files.

This is the produce:

unit Unit2;

// ==========================================================================
// Generated by kbmMW ObjectNotation marshalling converter
// 11/03/2019 00:07:12
// ==========================================================================

interface

uses
   Classes,
   Generics.Collections,
   kbmMWRTTI,
   kbmMWObjectMarshal,
   kbmMWDateTime,
   kbmMWNullable;

type

   TJSONData=class;
   Trealm=class;
   Talliance=class;
   TauctionsList=class;
   Tauctions=class;
   Thorde=class;
   Tneutral=class;
   [kbmMW_Root('TJSONData',[mwrfIncludeOnlyTagged])]
   TJSONData=class
   private
      Frealm:Trealm;
      Falliance:Talliance;
      Fhorde:Thorde;
      Fneutral:Tneutral;
   protected
      procedure Setrealm(const AValue:Trealm); virtual;
      procedure Setalliance(const AValue:Talliance); virtual;
      procedure Sethorde(const AValue:Thorde); virtual;
      procedure Setneutral(const AValue:Tneutral); virtual;
   public
      destructor Destroy; override;

      [kbmMW_Element('realm')]
      property realm:Trealm read Frealm write Setrealm;

      [kbmMW_Element('alliance')]
      property alliance:Talliance read Falliance write Setalliance;

      [kbmMW_Element('horde')]
      property horde:Thorde read Fhorde write Sethorde;

      [kbmMW_Element('neutral')]
      property neutral:Tneutral read Fneutral write Setneutral;
   end;

   [kbmMW_Root('realm',[mwrfIncludeOnlyTagged])]
   Trealm=class
   private
      Fname:kbmMWNullable<string>;
      Fslug:kbmMWNullable<string>;
   public
      [kbmMW_Element('name')]
      property name:kbmMWNullable<string> read Fname write Fname;

      [kbmMW_Element('slug')]
      property slug:kbmMWNullable<string> read Fslug write Fslug;
   end;

   [kbmMW_Root('alliance',[mwrfIncludeOnlyTagged])]
   Talliance=class
   private
      Fauctions:TauctionsList;
   protected
      procedure Setauctions(const AValue:TauctionsList); virtual;
   public
      destructor Destroy; override;

      [kbmMW_Element('auctions')]
      property auctions:TauctionsList read Fauctions write Setauctions;
   end;

   [kbmMW_Child('auctions',[mwcfFlatten])]
   TauctionsList=class(TObjectList<Tauctions>);
   [kbmMW_Root('auctions',[mwrfIncludeOnlyTagged])]
   Tauctions=class
   private
      FpetSpeciesId:kbmMWNullable<double>;
      FpetBreedId:kbmMWNullable<double>;
      FpetLevel:kbmMWNullable<double>;
      FpetQualityId:kbmMWNullable<double>;
      Fauc:kbmMWNullable<double>;
      Fitem:kbmMWNullable<double>;
      Fowner:kbmMWNullable<string>;
      Fbid:kbmMWNullable<double>;
      Fbuyout:kbmMWNullable<double>;
      Fquantity:kbmMWNullable<double>;
      FtimeLeft:kbmMWNullable<string>;
      Frand:kbmMWNullable<double>;
      Fseed:kbmMWNullable<double>;
   public
      [kbmMW_Element('petSpeciesId')]
      property petSpeciesId:kbmMWNullable<double> read FpetSpeciesId write FpetSpeciesId;

      [kbmMW_Element('petBreedId')]
      property petBreedId:kbmMWNullable<double> read FpetBreedId write FpetBreedId;

      [kbmMW_Element('petLevel')]
      property petLevel:kbmMWNullable<double> read FpetLevel write FpetLevel;

      [kbmMW_Element('petQualityId')]
      property petQualityId:kbmMWNullable<double> read FpetQualityId write FpetQualityId;

      [kbmMW_Element('auc')]
      property auc:kbmMWNullable<double> read Fauc write Fauc;

      [kbmMW_Element('item')]
      property item:kbmMWNullable<double> read Fitem write Fitem;

      [kbmMW_Element('owner')]
      property owner:kbmMWNullable<string> read Fowner write Fowner;

      [kbmMW_Element('bid')]
      property bid:kbmMWNullable<double> read Fbid write Fbid;

      [kbmMW_Element('buyout')]
      property buyout:kbmMWNullable<double> read Fbuyout write Fbuyout;

      [kbmMW_Element('quantity')]
      property quantity:kbmMWNullable<double> read Fquantity write Fquantity;

      [kbmMW_Element('timeLeft')]
      property timeLeft:kbmMWNullable<string> read FtimeLeft write FtimeLeft;

      [kbmMW_Element('rand')]
      property rand:kbmMWNullable<double> read Frand write Frand;

      [kbmMW_Element('seed')]
      property seed:kbmMWNullable<double> read Fseed write Fseed;
   end;

   [kbmMW_Root('horde',[mwrfIncludeOnlyTagged])]
   Thorde=class
   private
      Fauctions:TauctionsList;
   protected
      procedure Setauctions(const AValue:TauctionsList); virtual;
   public
      destructor Destroy; override;

      [kbmMW_Element('auctions')]
      property auctions:TauctionsList read Fauctions write Setauctions;
   end;

   [kbmMW_Root('neutral',[mwrfIncludeOnlyTagged])]
   Tneutral=class
   private
      Fauctions:TauctionsList;
   protected
      procedure Setauctions(const AValue:TauctionsList); virtual;
   public
      destructor Destroy; override;

      [kbmMW_Element('auctions')]
      property auctions:TauctionsList read Fauctions write Setauctions;
   end;


implementation

procedure TJSONData.Setrealm(const AValue:Trealm);
begin
     if Assigned(Frealm) then
        Frealm.Free;
     Frealm:=AValue;
end;

procedure TJSONData.Setalliance(const AValue:Talliance);
begin
     if Assigned(Falliance) then
        Falliance.Free;
     Falliance:=AValue;
end;

procedure TJSONData.Sethorde(const AValue:Thorde);
begin
     if Assigned(Fhorde) then
        Fhorde.Free;
     Fhorde:=AValue;
end;

procedure TJSONData.Setneutral(const AValue:Tneutral);
begin
     if Assigned(Fneutral) then
        Fneutral.Free;
     Fneutral:=AValue;
end;

destructor TJSONData.Destroy;
begin
     Frealm.Free;
     Falliance.Free;
     Fhorde.Free;
     Fneutral.Free;
     inherited;
end;

procedure Talliance.Setauctions(const AValue:TauctionsList);
begin
     if Assigned(Fauctions) then
        Fauctions.Free;
     Fauctions:=AValue;
end;

destructor Talliance.Destroy;
begin
     Fauctions.Free;
     inherited;
end;

procedure Thorde.Setauctions(const AValue:TauctionsList);
begin
     if Assigned(Fauctions) then
        Fauctions.Free;
     Fauctions:=AValue;
end;

destructor Thorde.Destroy;
begin
     Fauctions.Free;
     inherited;
end;

procedure Tneutral.Setauctions(const AValue:TauctionsList);
begin
     if Assigned(Fauctions) then
        Fauctions.Free;
     Fauctions:=AValue;
end;

destructor Tneutral.Destroy;
begin
     Fauctions.Free;
     inherited;
end;


initialization
   kbmMWRegisterKnownClasses([TMainClass,Trealm,Talliance,TauctionsList,Tauctions,Thorde,Tneutral]);

end.

To actually read the file, create a program, add Unit2.pas to it, and write this code:

procedure TForm1.Button1Click(Sender: TObject);
var
   m:TkbmMWCustomRTTIMarshal;
   o:TJSONData;
begin
     m:=TkbmMWJSONMarshal.Create;
     try
        TkbmMWJSONMarshal(m).AnonymousRoot:=true;
        o:=TkbmMWJSONMarshal(m).ValueFromUTF8File<TJSONData>('auctions.json');
        // Your complete parsed file is now contained in the object instance o.
     finally
        o.Free;
        m.Free;
     end;
end;

It is as simple as that.

Another example is the XML file courses.xml, also picked up from the internet somewhere.

<?xml version="1.0"?>
<root>
	<course>
		<reg_num>10577</reg_num>
		<subj>ANTH</subj>
		<crse>211</crse>
		<sect>F01</sect>
		<title>Introduction to Anthropology</title>
		<units>1.0</units>
		<instructor>Brightman</instructor>
		<days>M-W</days>
		<time>
			<start_time>03:10PM</start_time>
			<end_time>04:30</end_time>
		</time>
		<place>
			<building>ELIOT</building>
			<room>414</room>
		</place>
	</course>
	<course>
		<reg_num>20573</reg_num>
		<subj>ANTH</subj>
		<crse>344</crse>
		<sect>S01</sect>
		<title>Sex and Gender</title>
		<units>1.0</units>
		<instructor>Makley</instructor>
		<days>T-Th</days>
		<time>
			<start_time>10:30AM</start_time>
			<end_time>11:50</end_time>
		</time>
		<place>
			<building>VOLLUM</building>
			<room>120</room>
		</place>
	</course>
	<course>
		<reg_num>10624</reg_num>
		<subj>BIOL</subj>
		<crse>431</crse>
		<sect>F01</sect>
		<title>Field Biology of Amphibians</title>
		<units>0.5</units>
...
	</course>
...
</root>

To create Delphi source code to marshal and unmarshal that type of file

var
   s:string;
...
    s:=TkbmMWXMLMarshal.GenerateDelphiClassFromFile('courses.xml','Unit3','XMLData');
...

That will analyse courses.xml and produce a string containing Delphi source code for a unit called Unit3. We also, by convention, provide a main class name, but it will not be used since XML do not, like JSON, operate with an anonymous root datastructure.

The generated unit will look like this:

unit Unit3;

// ==========================================================================
// Generated by kbmMW ObjectNotation marshalling converter
// 10/03/2019 23:54:52
// ==========================================================================

interface

uses
   Classes,
   Generics.Collections,
   kbmMWRTTI,
   kbmMWObjectMarshal,
   kbmMWDateTime,
   kbmMWNullable;

type

   Troot=class;
   TcourseList=class;
   Tcourse=class;
   Ttime=class;
   Tplace=class;
   [kbmMW_Root('root',[mwrfIncludeOnlyTagged])]
   Troot=class
   private
      Fcourse:TcourseList;
   protected
      procedure Setcourse(const AValue:TcourseList); virtual;
   public
      destructor Destroy; override;

      [kbmMW_Element('course')]
      property course:TcourseList read Fcourse write Setcourse;
   end;

   [kbmMW_Child('course',[mwcfFlatten])]
   TcourseList=class(TObjectList<Tcourse>);
   [kbmMW_Root('course',[mwrfIncludeOnlyTagged])]
   Tcourse=class
   private
      Ftime:Ttime;
      Fplace:Tplace;
      Freg_num:kbmMWNullable<string>;
      Fsubj:kbmMWNullable<string>;
      Fcrse:kbmMWNullable<string>;
      Fsect:kbmMWNullable<string>;
      Ftitle:kbmMWNullable<string>;
      Funits:kbmMWNullable<string>;
      Finstructor:kbmMWNullable<string>;
      Fdays:kbmMWNullable<string>;
      Fxml_repository:kbmMWNullable<string>;
   protected
      procedure Settime(const AValue:Ttime); virtual;
      procedure Setplace(const AValue:Tplace); virtual;
   public
      destructor Destroy; override;

      [kbmMW_Element('time')]
      property time:Ttime read Ftime write Settime;

      [kbmMW_Element('place')]
      property place:Tplace read Fplace write Setplace;

      [kbmMW_Element('reg_num')]
      property reg_num:kbmMWNullable<string> read Freg_num write Freg_num;

      [kbmMW_Element('subj')]
      property subj:kbmMWNullable<string> read Fsubj write Fsubj;

      [kbmMW_Element('crse')]
      property crse:kbmMWNullable<string> read Fcrse write Fcrse;

      [kbmMW_Element('sect')]
      property sect:kbmMWNullable<string> read Fsect write Fsect;

      [kbmMW_Element('title')]
      property title:kbmMWNullable<string> read Ftitle write Ftitle;

      [kbmMW_Element('units')]
      property units:kbmMWNullable<string> read Funits write Funits;

      [kbmMW_Element('instructor')]
      property instructor:kbmMWNullable<string> read Finstructor write Finstructor;

      [kbmMW_Element('days')]
      property days:kbmMWNullable<string> read Fdays write Fdays;

      [kbmMW_Element('xml_repository')]
      property xml_repository:kbmMWNullable<string> read Fxml_repository write Fxml_repository;
   end;

   [kbmMW_Root('time',[mwrfIncludeOnlyTagged])]
   Ttime=class
   private
      Fstart_time:kbmMWNullable<string>;
      Fend_time:kbmMWNullable<string>;
   public
      [kbmMW_Element('start_time')]
      property start_time:kbmMWNullable<string> read Fstart_time write Fstart_time;

      [kbmMW_Element('end_time')]
      property end_time:kbmMWNullable<string> read Fend_time write Fend_time;
   end;

   [kbmMW_Root('place',[mwrfIncludeOnlyTagged])]
   Tplace=class
   private
      Fbuilding:kbmMWNullable<string>;
      Froom:kbmMWNullable<string>;
   public
      [kbmMW_Element('building')]
      property building:kbmMWNullable<string> read Fbuilding write Fbuilding;

      [kbmMW_Element('room')]
      property room:kbmMWNullable<string> read Froom write Froom;
   end;


implementation

procedure Troot.Setcourse(const AValue:TcourseList);
begin
     if Assigned(Fcourse) then
        Fcourse.Free;
     Fcourse:=AValue;
end;

destructor Troot.Destroy;
begin
     Fcourse.Free;
     inherited;
end;

procedure Tcourse.Settime(const AValue:Ttime);
begin
     if Assigned(Ftime) then
        Ftime.Free;
     Ftime:=AValue;
end;

procedure Tcourse.Setplace(const AValue:Tplace);
begin
     if Assigned(Fplace) then
        Fplace.Free;
     Fplace:=AValue;
end;

destructor Tcourse.Destroy;
begin
     Ftime.Free;
     Fplace.Free;
     inherited;
end;


initialization
   kbmMWRegisterKnownClasses([Troot,TcourseList,Tcourse,Ttime,Tplace]);

end.

And to read and write such XML data, this code can be used

procedure TForm1.Button2Click(Sender: TObject);
var
   m:TkbmMWCustomRTTIMarshal;
   o:Troot;
   s:string;
begin
     m:=TkbmMWXMLMarshal.Create;
     try
        o:=TkbmMWXMLMarshal(m).ValueFromFile<Troot>('courses.xml');
        // o now contains all courses.

        s:=TkbmMWXMLMarshal(m).ValueToString(o);
        // s now contains XML generated from o.
     finally
        o.Free;
        m.Free;
     end;
end;

If the kbmMW file analyser detects a type it does not know what to do with, it will auto generate one called TUNKNOWN, which will need to be manually handled by the developer, by replacing it with more relevant class definitions to be able to marshal and unmarshal that particular part of the data.

If you like what you see, please share the word. Reshare blog posts and let others know about the product!

Essentially help us to help you 🙂

 23,658 total views,  1 views today