Compile Tool #1 – An easier way to compile projects

Petit Louis

Some people may have wondered if I have fallen off the face of the earth as I have been less vocal the last couple of weeks.

It has nothing to do with the dreadful COVID-19 infection I suppose most of us, one way or the other, are affected by. It rather has to do with being overly busy with various things.

One of the things, that relate to kbmMW and kbmMemTable, is the development of a brand new CompileTool which will be included in next release of kbmMemTable and kbmMW.

The purpose of the CompileTool is … TA DAAA…. to compile stuff ๐Ÿ™‚

So what’s new about that? Not really much… but let me explain the rationale behind my apparant brain damage.

The Compile Tool

Compiling and installing kbmMemTable, has in Delphi always been fairly easy. In C++Builder only mode, not so much, partly because the C++ only environment diverge more and more from what kbmMemTable originally supported, and the matching C++ project files.

To complicate matters even more, kbmMW can be a pain to install in Delphi due to kbmMW’s ability to seemlessly integrate with loads of 3rdparty stuff. Paths and requirements and more needs to be provided. In C++ only mode it is even more complex to get it going, and despite the Compile Tool helping much on the situation, it is not fully solved with kbmMW yet, because C++Builder exhibits random crashes and unexplained compile/link errors (internal errors).

But one of the things Delphi did very well.. in the early days (I suppose until XE got to see the light), was to consistently tell you that some 3rdparty packages should be referenced to compile kbmMW’s packages nicely. Delphi was even nice enough to add the relevant requirements to the kbmMW package so everything just worked.

Unfortunately, Delphi has stopped to do that stabily since many years. I have reported it to Embarcadero on numerous occations, and they have acklowledged the issue, but have not been able to figure out why it has stopped working. Mind you.. sometimes it works… but then suddenly it does not, usually when that happens, it stops working for good in my experience.

So the Compile Tool has as goal to:

  1. Know everything about the units and requirements of the project in hand
  2. Be able to manage knowledge about 3rdparty libraries/packages
  3. Produce correct valid project files for the project that the Compile Tool has been prepared for.
  4. Compile and install the projects automatically
  5. Be able to recompile and restart itself, so it is up to date with whatever settings you may have made in for example kbmMWConfig.inc and thus based on those settings, is able to produce correct project files.

At first I made it to make compilation and installation of kbmMW easier, but it soon dawned to me that kbmMemTable should be supported too (standalone) and that it as such could be a generic tool that will work for other developers too. (Currently it is not released with license for other 3rdparty developers to use it, but ping me if you have interest in that.)

Compile Tool for kbmMemTable

Let us have a look at the Compile Tool for kbmMemTable. It will usually be found, as CompileTool.exe, in the source directory of the project for which it is supposed to support. Further the source of the Compile Tool will be found in the subdirectory CompileTool under the managed projects source directory.

If you loose CompileTool.exe, it can be recompiled by opening and building the project in the CompileTool directory. You can not use the CompileTool source/executable from a different project (like kbmMW), because they contain different settings, specially in the uCompileToolFeatures.pas file, which is specialized for each project.

If you start CompileTool.exe on a computer on which Delphi, C++Builder or RAD Studio is not installed, you will get an exception and the tool will shut down.

So let us start it on a computer with RAD Studio installed.

On the left side, a list of supported features for the project, is listed. In the case of kbmMemTable it is pretty simple, and will not change, since all features are available for all versions of kbmMemTable. However look later for how the Compile Tool for kbmMW looks.

In the center, the discovered versions of Delphi, C++Builder or RAD Studio is listed. You can select, in which of them, kbmMemTable should be installed.

On the right side, you can follow the status of what is happening.

At the bottom, various buttons are available.

Update compile tool – Will rebuild the Compile Tool itself, and restart it with the newly compiled executable. For kbmMemTable it is not often needed to do, but in the case of kbmMW, you will want to do that, everytime you have changed something in kbmMWConfig.inc.

Validate requirements – It will show a dialog, where 3rdparty requirements, which the Compile Tool has not been able to resolve itself, can be defined. The definitions made here will be remembered for next time in the file CompileTool.ini, making it easier to recompile/install without having to reconfigure each time. The .ini file will never be overwritten by kbmMemTable or kbmMW installations. kbmMemTable will usually not need any settings in this dialog. See section about kbmMW installation further down for more information.

Generate projects – It will produce new kbmMemTable project files matching the selected IDE. The project files will automatically be written to the parent directory (which is the kbmMemTable source directory).

Generate, compile and install – It will generate projects, as described above, compile the projects and automatically install the resulting packages in the IDE, for all selected IDE’s. Usually you will be required to close the selected IDE before being able to compile and install. You can start with the option -F:ย  CompileTool.exe -F to override the requirement to stop the IDE. However the packages will not show up until you restart the IDE later on, and if the packages already was in use by the IDE you will get compile/linker errors.

This is an example the result. You may notice that there are various paths shown in the status. Those paths are automatically picked up from your current installation, and provided for the compiler by the Compile Tool.

After it succeeds, you can close the tool, and start the IDE. Now kbmMemTable will be available and installed with all paths for Windows 32 compilation, correctly setup automatically.

So let us look at how it works when installing kbmMW.

Compile Tool for kbmMW

In this case, I have, for the demo, opened kbmMWConfig.inc, uncommented the line:

{$DEFINE KBMMW_DBISAM3_SUPPORT} // DBISAM 3 support.

and saved the file again, which essentially tells kbmMW that we want full support for DBISAM v3. (FTR there are similar defines for 36 other databases too, incl. DBISAM v4, ElevateDB, NexusDB and many many more).

Since this is a new setting, I first start the Compile Tool, where the Feature support at this time do not include DBISAM3, and let it recompile itself.

When it restarts, it looks like this:

Now the Compile Tool recognize the request for feature support for ElevateSoft DBISAM v3.
Since it is a new 3rdparty tool that kbmMW should support, compared to previous settings, we want to check the requirements dialog by clicking on the Validate requirements button.

You can see that there are a runtime package requirement that is currently unresolved. To resolve it, simply type the name of the DBISAM v3 runtime package, typically something like db300d20 or something along those lines. As the structure of the package name can vary wildly between various 3rdparty projects, it is left for you to type the right value. Remember that this value will be used for all the IDE’s that the Compile Tool will compile for.

You can include parameters in the name. Eg. DBISAM3=db300[!–IDESHORTYPE–!][!–IDESHORTID–!] which will replace [!–IDESHORTYPE–!] with D for Delphi or C for C++Builder and [!–IDESHORTID–!] with 20 for Delphi 10.3.

If you need to refer to multiple packages/requirements for the DBISAM3 selection, you can separate those with a semicolon ;

Click save, and the Compile Tool will remember your settings, also for next time you start the Compile Tool.

Clicking either Create projects or Create, compile and install, will ensure the kbmMW project files contains the relevant requirements.

Also notice that there is an extra Prerequisites section at the top of the Compile Tools window. It is there because kbmMW require compilation and installation of kbmMemTable beforehand. You can point out where it’s source is, and click the checkbox, then it will automatically recompile and install it when you recompile and install kbmMW via the Create, compile and install button.

Behind the scenes

So what is happening behind the scenes? Well… I told about the uCompileToolFeatures.pas file which is special for each project. It is in that file of the Compile Tool sources, where the project specialities are defined.

It contains a class definition which groups a few methods which should be defined for a project.

The above specifies the texts to be shown and thus can be configured for other projects (like kbmMemTable which Compile Tool has a similar section).

This method builds relevant parameters that must exist for the project file generation. It includes version numbers, project names and descriptions and more. The method will be called multiple times during project generation.

This section defines all the features that can exist in a kbmMW runtime package, and their library requirements and units, including the DBISAM v3 option.

AddProjectInfo takes 5 arguments:

  1. The unique ID of the project part. For example KBMMEMTABLE. Any ID can be used, as long as it is unique.
  2. The descriptive name of the project part.
  3. The libraries that are required for the project part, separated by semicolon and without file extensions. If it is an empty string, there are no requirements for that particular project part. If it is a question mark, it is unknown, and thus can be handled by the user in the Compile Tool package resolver dialog. It is allowed to include paths if needed, but recommended to only use relative paths from the Source directory.
    Further it is legal to prefix each library with either < or }. Doing so will ensure to sort the item first (<) or last (}), rather just according to its regular name, when project files are generated.
  4. The unit names (without extension) that are to be part of this project part. Multiple unit names can be specified separated by semicolon (;). If the unit also encompasses a form or datamodule file, use this syntax:
    unitname=formname:formclass. Eg. kbmMWCustomJavaService=kbmMWCustomJavaService:TkbmMWSimpleService;}JNI
  5. A boolean indicating if a requirement is mandatory for this project part. It is used for validation/warning that the project file may not have been generated correctly, if the requirement value has not been made available.

This section defines all the features that can exist in a kbmMW designtime package, and their library requirements and units.

It follows the same explanation as for the runtime part.

The above code is used for creating the relevant project file name for a specific IDE. The file name should not include any file extensions. The method will be called multiple times during project creation.

Final plea

If you have reached here, then you are qualified to assist me making the Compile Tool better. Since kbmMW supports so many 3rdparty tools, I have for now only specified library requirements in the Compile Tool for some of them. For the remaining, I have left it for you to define, simply because I do not know the naming rules for all the various 3rdparty libraries. Please comment here on this thread if you have some pet libraries that kbmMW supports and that you would like the Compile Tool to know about, preferably with the complete description of how the library name is defined. I.e. what each part of the name consists of.

If the name do not change based on the version of the 3rdparty library, it most likely will be possible to add automatic support in the Compile Tool for that particular library, making it even easier to compile and install kbmMW.

 12,316 total views,  1 views today

Author: kimbomadsen

16 thoughts on “Compile Tool #1 – An easier way to compile projects

  1. Kim,
    Including these libraries helped me to build kbmMW:
    requires
    IndyCore,
    IndySystem,
    rtl,
    kbmMemRunD103Pro,
    vcl,
    FIBPLUS_D26,
    dclib,
    dclie,
    FMX,
    DbxCommonDriver,
    dbexpress,
    vclFireDAC,
    FireDAC,
    FireDACCommonDriver,
    FireDACCommon,
    dac260,
    ibdac260,
    vcldesigner,
    adortl,
    ibxpress;
    Some are probably overkill.
    Looking in Computer\HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Embarcadero\BDS\20.0\Known Packages helped me find exact package names.

  2. Hi,
    Thank you for the list. I have extracted and added support for FIB and IBDAC, MYDAC and UNIDAC from your example.

    Keep them coming ๐Ÿ™‚

    best regards
    Kim/C4D

  3. Where can we download the compile tool? Iโ€™m finally updating to RIO, and I tried to leave everything default (paths Delphi uses) this time regarding where Delphi places everything (bpl, dcu, etc). kgmMemTable installed without any issues. kbmMW, fails miserably with 100s of imports that shouldnโ€™t be imported. The error I ultimately get is โ€œ[dcc32 Error] E2621 Export table in output file too large: exceeds 65536 limits]. Very frustrating. It seems so un-related to what Iโ€™m actually trying to accomplish. I get a few of these: [dcc32 Hint] H2161 Warning: Duplicate resource: Type 12 (CURSOR GROUP), ID 32761; File c:\program files (x86)\embarcadero\studio\20.0\lib\Win32\release\FMX.Controls.Win.res resource kept; file c:\program files (x86)\embarcadero\studio\20.0\lib\Win32\release\Controls.res resource discarded.
    If Embarcadero needs to work on anything, they need to work on this. Adding third party components should be a no-brainer, but every vendor puts their code in different places, modifies library paths, modifies environment paths, one has a batch file, on and on. Just so frustrating. Iโ€™m hoping your compiler tool will help. Can it be downloaded yet or am I missing the obvious and itโ€™s here and Iโ€™m not seeing it?

  4. Hi, I feel your pain.. .literally ๐Ÿ™‚

    The “exceeds 65536 limits” is because the linker imports all requirements, that are not in the requires section, as internal entrypoints in the BPL it is about to create. A BPL is just a DLL, and MS has defined a max of 65536 exports in a DLL.

    This link explains some of the things you can add as requirements:

    https://stackoverflow.com/questions/61061554/compiling-kbmmw-design-package-in-delphi-10-3-vcl-conflict-error-e2199

    If you are using ReportBuilder, it also adds a huge amounts of exports to the DLL. Make sure you have all ReportBuilder bpl’s listed in the requires section of the kbmMW runtime package. (If you indeed are using ReportBuilder, let me know what you have added, so I can add that knowledge to the Compile Tool)

    The Compile Tool needs a few minor updates in the kbmMemTable and kbmMW sources, so it will be released along with the next kbmMemTable and kbmMW release.

    best regards
    Kim/C4D

  5. Hi,

    I tried to use the CompileTool but always getting build errors, while the IDE can build the packages and install just fine.

    The error which I got:

    “f:\work\Delphi\pkg\kbmMW\Source\kbmMWRunD103Ent.dproj” (Build cl) (1) ->
    (_PasCoreCompile cl) ->
    kbmMWRunD103Ent.dpk(33): error E2202: Required package ‘vcldb’ not found [f:\work\Delphi\pkg\kbmMW\Source\kbmMWRunD103Ent.dproj]

    which is strange because the Runtime package does not require the vcldb package, but the build system wants it and fail.

    BTW
    NEXUSDB v4.5 = NexusDB450ll260;NexusDB450sr260;NexusDB450sd260;NexusDB450db260;NexusDB450se260

    DBISAM4 = dbisamr

    Best Regards

  6. Hi,
    I have seen that behaviour on my 10.2 installation, where I suspect unusual long search path’s to be the reason… but I have not found a solution as of yet.

    Thank you for the info about some 3rdparty package names. The NexusDB one is one of the tricky ones, as it changes with each version released as far as I know?

    1. I have altered the compile tool so its valid to have question marks in the middle of requirements to indicate that something needs to be filled out by the user.

      Hence for NexusDB, this would be the default requirement string:
      NexusDB???ll[!–IDEPACKAGEVERSION–!];NexusDB???sr[!–IDEPACKAGEVERSION–!];NexusDB???sd[!–IDEPACKAGEVERSION–!];NexusDB???db[!–IDEPACKAGEVERSION–!];NexusDB???se[!–IDEPACKAGEVERSION–!]

  7. Hi,

    I think the new/latest version is not compiled with sqldir. I tried to use compiletool but it also gives error. When I just want to re-create the projects, newly prepared projects cannot be compiled. Does anyone have any suggestions?

  8. Hi All,
    I read your article in Blaise Magazine. Iโ€™m not a C4D user/customer and Iโ€™m wondering if itโ€™s possible to try and test your compile tool. In case of, than you to provide me an download link.
    Best regards,
    Laurent

    1. Hi,
      Thank you for your interest in the compile tool. It is currently only bundled with source versions of kbmMemTable and kbmMW. What specifically would you like to test?
      best regards
      Kim/C4D

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.