kbmMW LINQ #1

LINQ… what is LINQ? Well its a term used in C# which means Language Integrated  Query.

Next version of kbmMW will support our own variant of LINQ. In reality we can’t make true C# LINQ functionality, because it require the compiler to be aware about the fundamentals of LINQ, and Delphi is blissfully unaware about such language integrated features.

However, my interpretation of the purpose of LINQ is that its designed to make certain programming tasks easier for the programmer. To get rid of boiler plate code, which is something that I have focused quite allot on in kbmMW v5 and continues to focus on.

So what does LINQ do for us?

It allows us to query, filter, order, calculate, compare, group etc. various types of data in an easy way using the same syntax regardless of what type the (supported) source data is.

Is kbmMW’s LINQ fast? Yes and no. Since it hides much functionality from us, more CPU cycles are usually spend than would otherwise have been needed if you coded an optimized algorithm yourself. However since it use optimized kbmMW features underneath, then some scenarios will probably perform just as well as manually written code.

So the advantage of using kbmMW’s LINQ is not as such performance, but rather provides the ability to do quite complex and advanced things with very simple code.

To use kbmMW’s LINQ, simply add kbmMWLinq to the uses clause. It will give you a global threadsafe object instance, named Linq, thru which all LINQ functionality originates.

In the following I will show various scenarios that are possible with kbmMW.

Operating a TStringList using LINQ

This is an example of using LINQ with a regular TStringList.

First we build a string list.

var
   sl,sl2:TStrings;
begin
     sl:=TStringList.Create;
     sl.Add('1');
     sl.Add('2');
     sl.Add('3');
     sl.Add('4');
     sl.Add('5');
     sl.Add('6');
     sl.Add('7');
     sl.Add('8');
     sl.Add('9');
     sl.Add('20');

Then we use Linq to give us the first 5 of them sorted descending and returning the result as a new TStringlist:

     sl2:=Linq.Using(sl).First(5).Sort('value:D').AsStrings;
...
     sl2.Free;

Next we show to how make multiple operations on the same data, without the overhead of re-parsing the source data. First we define the initial Linq stage (the one preparing the source data) as shared, then we return the last 8 items as a TStringList, looks for the Max value, and then calculates a SUM. Since the string list is strings, kbmMW’s LINQ assumes that Max/Min functions should operate on a string level, not numeric. However the SUM function can only work on numeric data, and thus will always operate as such:

type
   lq:IkbmMWLinqStage;
   s:string;
   d:double;
...
     lq:=Linq.Using(sl).Shared;
     sl2:=lq.Last(8).AsStrings;
...
     sl2.Free;

     s:=lq.Max; // Returns the string 9
     d:=lq.Sum; // Returns 65

When a IkbmMWLinqStage goes out of scope, it will automatically be freed along with any data it holds.

Next we make a more complex SQL like query on the data, returning all even numbers as a list of strings.

     sl2:=lq.Where('value MOD 2 = 0').AsStrings;
...
     sl2.Free;

Operating class instances using LINQ

The next example shows how to use Linq on lists of class instances. For a class to be “Linqable” it must be tagged with the kbmMW_Linq attribute as seen below. In addition the class should be registered as a kbmMW known type and RTTI must be enabled for it.

  [kbmMW_Linq]
  TMyData = class
  private
     FName:string;
     FAddress:string;
     FAge:integer;
  public
     constructor Create(const AName:string;
                        const AAddress:string;
                        const AAge:integer);
     property Name:string read FName write FName;
     property Address:string read FAddress write FAddress;
     property Age:integer read FAge write FAge;
  end;

One place to register the class to kbmMW is in the units initialization section. Notice that both TMyData and TObjectList are being registered, since we will use both types.

initialization
   TkbmMWRTTI.EnableRTTI([TMyData,TObjectList<TMyData>]);
   kbmMWRegisterKnownClasses([TMyData,TObjectList<TMyData>]);

Lets prepare some data to play with:

var
   lst:TObjectList<TMyData>;
...
     lst:=TObjectList<TMyData>.Create;
     try
        lst.Add(TMyData.Create('Kim','Address 1',49));
        lst.Add(TMyData.Create('Hans','Address 2',22));
        lst.Add(TMyData.Create('Jens','Address 3',33));
        lst.Add(TMyData.Create('Joe','Address 3',77));
        lst.Add(TMyData.Create('John','Address 4',33));

If we want to locate the maximum age, we write

        i:=Linq.Using(lst).Max('Age');

or the alphabetically smallest name

        s:=Linq.Using(lst).Min('Name');

or we can return a key/value string list of sorted names with the age

        sl:=Linq.Using(lst).Sort('Name').AsStrings('Name','Age');

And then we clean up. Also remember to free the returned TStringList (sl) when you don’t need it any longer.

     finally
        lst.Free;
     end;

Operating JSON documents using LINQ

First lets prepare some data. This time we use the UsingJSON method. It can take a JSON string, a stream or you can use UsingJSONFile to load the JSON document from a file.

var
   sl:TStrings;
   lq:IkbmMWLinqStage;
begin
     lq:=Linq.UsingJSON('{"result":['+
                        '{"ID":1,"name":"kim","date":"2018-01-05T19:05:00.000+08:00"},'+
                        '{"ID":2,"name":"kim","date":"2018-01-05T20:05:30.000+01:00"},'+
                        '{"ID":3,"name":"kim","date":"2018-01-05T20:05:45.000+01:00"},'+
                        '{"ID":4,"name":"kim","date":"2018-01-05T20:06:15.000+01:00"},'+
                        '{"ID":5,"name":"kim","date":"2018-01-05T20:06:30.000+01:00"},'+
                        '{"ID":6,"name":"kim","date":"2018-01-05T20:06:45.000+01:00"},'+
                        '{"ID":7,"name":"kim","date":"2018-01-05T20:07:15.000+01:00"},'+
                        '{"ID":8,"name":"kim","date":"2018-01-05T21:07:30.000+01:00"},'+
                        '{"ID":9,"name":"kim","date":"2018-01-05T21:07:45.000+01:00"},'+
                        '{"ID":10,"name":"kim","date":"2018-01-05T21:08:00.000+01:00"}]}',
                  '/result/.*',
                  'ID,date');

UsingJSON takes 2 and optionally 3 arguments, the JSON data string, a subset pattern match( ‘/result/.*’), and optionally a list of field names or expressions (‘ID,date’).

The subset is actually a regular expression which is applied to the JSON data, to only select the relevant data on which the Linq methods should operate. In this case we have one property (result) with a sub array, so we use the expression to accept everything starting (path wise) with /result/. We have also told kbmMW that we only want to access the ID and date fields of the JSON document. If we didn’t specify that, kbmMW would automatically have figured out all relevant fields that should be accessible. The field names can include expressions, so its thus possible to add fields together or do calculations.

     i:=lq.Max('ID');
     i:=lq.Min('ID');
     sl:=lq.Sort('date:D').AsStrings('date');

In similar way you can query YAML, BSON and MessagePack documents.

Operating XML documents using LINQ

XML documents are structurally more complex than JSON and YAML documents, in the sense that each node in the document can have attributes in addition to child nodes and data.

We must still specify a subset we want to operate on like above, but if we want access to the attributes, we must use the XMLAttr function, that takes two arguments: the node holding the attribute, and the attribute name itself. Since attributes by definition are strings, we have the ability to automatically have the values casted to some other types, like TEXT(size), INTEGER etc.

kbmMW supports the following casts: INT/INTEGER, VARCHAR2(n), VARCHAR(n), CHAR(n), BOOL, BOOLEAN, AUTOINC, FLOAT, DOUBLE, NUMERIC, REAL, DATETIME, TIMESTAMP, DATE, TIME, LARGEINT, INT64, BLOB, GRAPHIC, CLOB, TEXT(n), CURRENCY, WORD, MEMO, WIDEMEMO and GUID.

If n is not given the default value is 20.

var
   sl:TStrings;
   lq:IkbmMWLinqStage;
begin
     lq:=Linq.UsingXML('<?xml version="1.0" ?>'+
                       '<Dictionary>'+
                       ' <Parameters>'+
                       '   <Parameter SymbolName="CoDeviceType"'+
                       '              ObjectType="VAR"'+
                       '              Index="0x1000"'+
                       '              SubIndex="0"'+
                       '              DataType="UNSIGNED32"'+
                       '              AccessType="const" />'+

                       '   <Parameter SymbolName="CoErrorRegister"'+
                       '              ObjectType="VAR"'+
                       '              Index="0x1001"'+
                       '              SubIndex="0"'+
                       '              DataType="UNSIGNED8"'+
                       '              AccessType="ro" />'+

                       '   <Parameter SymbolName="CoClearErrorLog"'+
                       '              ObjectType="VAR"'+
                       '              Index="0x1003"'+
                       '              SubIndex="0"'+
                       '              DataType="UNSIGNED8"'+
                       '              AccessType="rw"'+
                       '              Remarks="Write 0 to clear"/>'+
                       ' </Parameters>'+
                       '</Dictionary>'
                      ,'/Dictionary/Parameters/.*/'
                      ,'XMLAttr(Parameter,"SymbolName") as "SymbolName->TEXT(40)"'+
                      ,'XMLAttr(Parameter,"SubIndex") as "SubIndex->INTEGER"');

     sl:=lq.Sort('SymbolName').AsStrings('SymbolName','SubIndex');

sl will now contain a sorted key/value list:

CoClearErrorLog=0
CoDeviceType=0
CoErrorRegister=0

More LINQ features

kbmMW’s LINQ also supports:

• Count – Return the number of items in the given Linq stage.
• Distinct(fieldnames) – Return only items that have unique values in the fields specified by fieldnames.
• GroupBy(groupfieldnames,aggregatefieldnames) – Return records grouped by the groupfieldnames (required), and optionally aggregated values on the fields specified in aggregatefieldnames. You specify aggregation method as a modifier to the field name.
  Eg. field1:COUNT, field2:MAX
  The output of aggregated fields will be named ‘originalfieldname_COUNT/AVG/SUM/MIN/MAX/STDDEV’ (eg. field1_COUNT)
• Select(fieldexpressions) – Return the items exposed by the given expressions. Eg. Select(‘SIN(fld1) as fld1, fld2|fkd3 as newfield’). When expressions are used, the resulting field will be named Fn where n is the index in the resulting item starting with 1. To ensure that you have full control over the names, you can specifically name them using the “as name” method as shown.
• AsString(fieldname) – Return the first item’s field value as a string.
• AsInteger(fieldName) – Returns the first item’s field value as an integer.
• AsFloat(fieldName) – Returns the first item’s field value as a double.
• AsVariant(fieldname) – Returns the first item’s field value as a variant.
• AsDataset – Returns the data as a dataset. The ownership of the dataset belongs to the Linq stage. Thus when the Linq stage goes out of scope, the dataset is also destroyed.

Functions like Min, Max, Avg, Sum and StdDev can take zero or one field name. If zero field names are given, the first known internal field column is used.

In functions like Distinct, Sort and GroupBy which takes multiple fields, the fields must be separated by comma (,).

Functions like As…..(fieldname) can take 0 or 1 string argument, or alternatively an integer value. If no argument is given, the first field is assumed. If a string argument is given, the field with the given name is returned. If an integer value is given, the values for the field with the given index (first field is 0) is returned.

Feel free to come with ideas and input for the new Linq look alike features in kbmMW.

Also remember that Linq works with all compilers supported by kbmMW, so you can go “Linq nuts” on Android, IOS, Linux, OSX, Windows and Linux.

And remember… if you like our products, let others know! Please share the posts, blog about kbmMW or kbmMemTable or link to our blogs.