[kimbomadsen]

Hi,

There have been several fixes since 5.15 related to the lockfree hash array. So I would recommend to update to a newer version of kbmMW.

best regards
Kim/C4D

[kimbomadsen]

Logically, yes. The change certainly is not wrong, and I will implement it.
However free is actually supposed to ignore a null self.

It may more indicate that there is a lingering TkbmMWCustomTCPIPIndyClientTransport instance somewhere that may be attempted to be freed by two threads at the same time?

/KBM

[kimbomadsen]

I think the problem has been solved in 5.24.00 by rewriting the kbmMWDependency package and cleaning up dependencies.

/Kim/C4D

[kimbomadsen]

Hi,

I have reviewed your suggestion and decided to include it in the code base. Thank you!

/Kim

[kimbomadsen]

Hi,

I cant from the screenshot you present see any leak.
I can see the two message queue processors are allocated (one for the inqueue and one for the outqueue).
Have you set InboundProcessor and/or OutboundProcessor anywhere in your code?
If you have, then kbmMW will not try to deallocate them upon close.

Further make sure you close transports upon closing your application.

/Kim

[kimbomadsen]

Have you installed the Swagger GUI and the index.html file mentioned in the blog post?

/Kim

[kimbomadsen]

Hi

It has just been released.

/Kim

[kimbomadsen]

Hi,
I will look into it.
Thanks for the bug report.
/Kim

[kimbomadsen]

Hi,

Perfect 🙂

/Kim

[kimbomadsen]

Hi,

It sounds like you are trying to recompile kbmMW CE.
You cant do that since no source files are included.
kbmMW’s installer will automatically install the packages and dcu files for you.
Always make sure your projects can locate the dcu files when compilling.

/Kim

[kimbomadsen]

Hi,

Send me the smallest sample project possible showing what should work and what does not work, and I will have a look at it.

/Kim
kbm@components4developers.com

[kimbomadsen]

Hi,

It was designed in such way that you should create your own descendant of TkbmMWI18NCustomStorage and override InternalGetLanguages, InternalLoad, InternalSave and InternalDelete.

InternalGetLanguages should return an array of strings with the found language names.
InternalLoad will load one of the langauges (based on the name), InternalSave will save changes to one language, InternalDelete should remove the language.

/Kim

[kimbomadsen]

Hi,

Yes kbmMW supports that. If you will need the ORM or not depends.

kbmMW natively supports dynamic SQL and you can retrieve info about tables etc using its metadata system.

The ORM however allows you to “surface” a true database table as an ORM table, which means you have full access to manipulate it via the ORM and via your own objects.

Combine this with kbmMW SmartBind and you can build very dynamic user interfaces.

/Kim

[kimbomadsen]

Hi,

Thank you for your bugreport.

Parsing scientific numbers will be improved in next release for both JSON and YAML.

/Kim

[kimbomadsen]

I replied to the first post I received from you at April 24 2023.

I hope you did receive it. If not the reply is here:

“Actually I can only see you having an original Enterprise Edition new license purchase Sep 18 2019.
As the SAU is 12 months _extension_ it will extend from the expiration of the previous license period, which were April 20 2020 (6 months + a few days).
So your new SAU extension would usually have an expiration at April 20 2021.
The alternative is to purchase enough extensions to get back on track, or purchase a new license with a 10% back on track discount, which again would include 6 months SAU.
Instead I have (actually wrongly.. but my mistake) acted as you have purchased a new license, which thus  contains 6 months SAU, but essentially with 50% discount. ”

/Kim

[kimbomadsen]

I have had a look at it.
To fix the disconnection issue do the following:

At line 352 in kbmMWTCPMessagingClientTransport.pas, add line:

if not FConnection.IsClosed then

before

FConnection.Close;

Remove line 362 in the same unit. The line reads:

InternalCloseSocket;

/Kim

[kimbomadsen]

Hi,

I do not know why MSSQL or DevArt do not respect your setting, or if it really takes that long to process the records that the timeout happens anyway. You could try to set it to -1 as that should make it default to MSSQLs default setting which is infinite.
However I have added a BatchSize property (default 0) to the resolver. It will autocommit every n records. It will however work against the allornothing=true setting, as that will only be related on the latest batch, and not on all previous already committed batches.

/Kim

[kimbomadsen]

Well… one thing might be that the initial free is running on a bad pointer.
But it would be helpful to see how the definition of TPMItem and TPMRequirementItem actually is.

/Kim

[kimbomadsen]

Hi,

I have fixed it for next release.

/Kim

[kimbomadsen]

Hi,
Hmm… I wonder if FPC has added a few more TFieldType entries than before?

/Kim

[kimbomadsen]

Hi,

You can use SQL function DATE(col). It will essentially do the same (use trunk)

/Kim

[kimbomadsen]

Hi,

In principle yes. I have however not released any software for OS/X 64 bit, but have released software for Android and IOS 32/64 bit including kbmMemTable.

I would think it will work fine in 64bit OS/X.

/Kim

[kimbomadsen]

Hi,
Is it with any CSV file of any length? Can you please try to send an extremely simple sample for me to test?

/Kim

[kimbomadsen]

Hi,

I honestly no longer test for compatibility for pre XE7, why I would recommend using XE7 or newer, although it probably will compile with XE5 too.

The reason for it, is that generics, which I am taking advantage of especially in kbmMW, but also in kbmMemTable, cant be considered stable until XE7.

You can disable compiling kbmMemTable with support for generics by undefining KBMMEMTABLE_SUPPORT_GENERICS.

I however do not test without that defined except in relation to FPC.

/Kim

[kimbomadsen]

Hi,

You need to define an index first and then switch to it.

Use AddIndex. Then set IndexName:=your defined indexname.

Then you can make case insensitive lookups.

/Kim

[kimbomadsen]

Hi,
It is using TPerlRegEx under the hood to do the LIKE wildcard comparisons.
TPerlRegEx is default not case sensitive. Currently there are no options in kbmMemTable to make LIKE case insensitive, except to do an UPPER or LOWER on the value to compare with to force the same case.

/Kim

[kimbomadsen]

Hi,

Hmm… replied and my reply got lost… trying again (just if someone notice that there are two replies at some point in time).
I honestly do not test with D2010 and have not for quite a while.
I would recommend XE7 and up, although it will probably compile with XE5 too.

/Kim

[kimbomadsen]

Is it possible to make a small sample provoking the error?

I do not see that problem while shutting down my own applications using latest kbmMW.
So probably there is a difference (one difference is that I do not use SSL in the most acticely worked on applications. I instead use AES256 directly of various reasons).

/Kim

[kimbomadsen]

It does indeed clean it up at finalization, if the depending units have first been finalized.
It is all governed by the kbmMWDependency unit.
If it does not call the RunFinalization procedure in the kbmMWSmartEvent.pas unit during shutdown, there is something preventing it to do so.
If you can provide a (small) sample not requiring 3rdparty (except kbmMW/MemTable) that do not clean up, then I would be interested in getting that.

It could also be that MadExcept is building its trace a bit to early in the shutdown phase.

[kimbomadsen]

Hi,

 

Check this whitepaper. It have lots of goodies, including what you are asking for.

Click to access Tips%20and%20tricks%20with%20kbmMemTable.pdf

/Kim

[kimbomadsen]

Hi,

As far as I can see it is TkbmMWEvents that are leaked?
It is usually a singleton, unless you defined it yourself somewhere in your code?

Have you tried to issue a kbmMWShutdown before closing your application?

/Kim

[kimbomadsen]

Hi,

The reason is that kbmMW do not know the parameter definitions or the returned field definitions in advance, and have various ways to ask for that.

The default is to first ask for the parameter definitions from the database so you can fill them out.
That will result in the first call.
After that it will ask for the field definitions from the database as they can change based on your parameters. That will result in a 2nd call.
Then it will ask for the actual data from the database from the stored procedure. That is the 3rd call.

Then you may have a client side stored procedure component, which are doing exactly the same, which is then resulting in a flood of calls to the server.

To avoid that predefine parameters, and set AutoFieldDefsOnOpen to mwafoWithData or predefine fielddefs and set it to mwafoNever.

Do the same on the client side.

Further, if you do not predefine the fields/parameters on the client side and use mwafoNever, then you might want to enable cache on the connection pool for the client side, to handle some of the redundant calls to the server for definitions.

/Kim

[kimbomadsen]

What have you setup the connection pool as?
And what metadata component are you using with it?

/Kim

[kimbomadsen]

Hi,

I am having a look at it. It seems FPC do not like VarAsType very well. I have tried to change it to VarToStr which makes it not barf over it, but later the IsWild function or something after that (I have serious problems stepping into library code using trunk version of FPC also with new debugger) makes it look as there is a match even if there is not.

/Kim

[kimbomadsen]

Hi,

It is done so to explicitly disallow case sensitive sequence names. So I could choose if it should all be upper case or lower case. I decided for uppercase 🙂

When would you want to use lower case or mixed cases?

/Kim

[kimbomadsen]

Hi,

You can do:

ResponseTransportStream.Params.Values[KBMMW_HTTP_PARAM_OK_STATUS_CODE]:=’201′;

/Kim

[kimbomadsen]

Hi,

Check this link for some background info:
https://forums.devart.com/viewtopic.php?t=35949

You can set OLEDBCompatible property to true on the TkbmMWDBXConnectionPool instance.

/Kim

[kimbomadsen]

Yes,

There is actually a sample included doing exactly that… (and a bit more showing Websocket support).

/Kim

[kimbomadsen]

I will in the fix release also provide a new global kbmMWShutdown method, which will be used for signalling all long running parts of kbmMW to shut down as soon as possible.

 

/Kim

[kimbomadsen]

Hi,

The ASettings argument is simplified YAML format. It use a property: value scheme where the property name and value follows the YAML syntax for strings.

YAML strings only needs to be quoted if there can be a misunderstanding of the strings composition.
Check this link for more info.
https://stackoverflow.com/questions/19109912/yaml-do-i-need-quotes-for-strings-in-yaml

The ASettings content depends on the storage type.
In case of the JSON and YAML storage types currently bundled with kbmMW, they accept property names: file, encoding and ignoreMissingFile, where file is a string path to the file to contain the i18n storage, encoding either utf-8 or utf-16 and ignoreMissingFile either true or false (default false). If true, no exception will be thrown trying to load a non existing storage file.

If you would want to make a new custom i18n database bound storage, the ASettings property would probably include a connection string to the database along with other relevant properties.

[kimbomadsen]

Hi,

I suppose you are talking about Indy Client messaging transport vs native Client messaging transport?

/Kim

[kimbomadsen]

Hi,

Fixed in fix release.

/Kim

[kimbomadsen]

Hi,

I do not see server freezing in my setup, but I do see it no longer grabbing screenshots.
In my case it is because the DirectX stuff is running out of resources of some reason (code 0x80070057).

One reason could  be that my screen is 5120×2500.

Switching back to GDI makes it capture fine again without problems.

If indeed your server freezes, it is most likely due to problems with the DirectX GPU drivers.

/Kim

[kimbomadsen]

Hi,

Will fix in fix release.

/Kim

[kimbomadsen]

Damn!

Well.. it will be in the bugfix update to be released shortly.

/Kim

[kimbomadsen]

Hi,

I have changed it for 5.20.

/Kim

[kimbomadsen]

Grr…. it is a problem due to the use of TBitmap in threads.

It has always been somewhat scary to use TBitmap in a multithreaded environment. But it usually worked, when one remembered to lock the canvas before use.

However I think something has changed in the locking mechanism, since TBitmap.Canvas.Lock quite often ends up in a deadlock situation. As I remember, it did not do that with an earlier Delphi compiler.

Anyway…. it has resulted in almost 3 days and nights without much sleep, adding a new kbmMWDIB.pas unit, and completely dropping the use of TBitmap on the server side.

The good thing is that the new TkbmMWDIB bitmap class is much faster for what we use it for, because it does not contain any fluff like TBitmap and TCanvas, and specifically it does not require locking to operate in its own thread.

In addition I have changed the transport on the server from Indy to our own native IOCP transport, because it scales vastly better and performs very well.

All changes in the release being prepped right now!

/Kim

[kimbomadsen]

Hi,

Yes it will automatically attempt a lookup of the property text in the phrase section.

Sp it would look for sCancel in phrases.

/Kim

[kimbomadsen]

Hi,

You also need to enable the “masterswitch”:

i18n.LearnProperties := True;
i18n.Load(”, ‘test.yaml’,true);

/Kim

[kimbomadsen]

Thx
Is in upcoming release.
/Kim

[kimbomadsen]

Check new release about to be released to see if it solves the problem for you.
/Kim

[kimbomadsen]

FYI next release will include native MetroHash64 and MetroHash128 implementations.

I have not made any changes currently for replacing FNV1A with another hash in the codebase, but will keep it in mind at least for a performance benchtest to see if there is a significant impact or not.

/Kim

[kimbomadsen]

Hi,

Thanks for the pointer…. I have just had a look…. I may look into supporting it natively at some point, but one of the drawbacks performance wise, is that for multiplatform support, I would need to use Pascal only implementation at least for some platforms, which will make it slower than others. It performs really well when using the SSE2 variant

Other options that I have been referred to is Metro64, which seemingly is faster (using non assembler code) and has a better hashing quality.

I will have a look at them all… if better performance for specifically those places where hashes are used extensively (memory debugging lists etc.) can be achieved, it is worth a look 🙂

/Kim

[kimbomadsen]

TkbmMWScheduledEvent.Terminate contains locking to prevent the scheduled event being executed or being scheduled for execution if it is about to be terminated vs. termination setting awaits execution finishing.

/Kim/C4D

[kimbomadsen]

Hi,

Thank you for your bugreport and fix. It will be in next release!

/Kim/C4D

[kimbomadsen]

1) Variable capturing is a feature of Delphi, and it should work regardless of how the anonymous function is used, as long as it is within scope of the running code. Since the function must be in scope when you run it (otherwise you would get an A/V), the captured variables ought to be live too. If not it is a bug in Delphi.

However you can use the Withxxxx(..) methods to include data to your scheduled task. You can access that data from the AEvent instance’s Data, &Object or &Interface properties in the event procedure.

2) That is what RunNow does. It schedules an event to run async as soon as possible. You can force it to run sync using the Synchronized or SyncQueued setting, but you can also use InAMoment which is the same as RunNow but which will queue the event on the main thread (typically for visual updates).

3) Hmm… there are 3 types of threads handled by the Scheduler: a) The main VCL thread, b) pool of threads for precise events, c) pool of threads for relaxed events

a is governed by the VCL. b and c respectively require all threads to run with same priority. b has no limits on the number of allocated system threads, while c has.

Generally it is not recommended to make changes to scheduled events active state from within a scheduled event.
Instead set its pause property, as this will not interfere with its spooling up/down, but will prevent it from being run if not already running.

[kimbomadsen]

Since kbmMW is dual use software (used for military purposes as well as civilian), it has been necessarily to vet each download of kbmMW Community Edition, to filter out users from Russia and Belarusia.

For that reason it no longer automatically is available for download.

/Kim/C4D

[kimbomadsen]

Hi,

You inherit from it and override its InternalProcessMessage(const ASender:TObject; const AMessage:IkbmMWCustomTransportStream; var AAction:TkbmMWMessageProcessingSuccessAction) procedure or put an event handler in the OnProcess event. The procedure or event is called for each message to process.

You should set the AAction property to the status of your processing.

mwpsaRollback: Message will be rolled back, and whatever rollback processing will happen. It means that the message will be retried later, unless rolled back too often where stale detection kicks in.

mwpsaReject: Means you specifically reject the message. It will be placed on the reject queue if such is setup on the queue that is being processed.

mwpsaDelete: Message is deleted.

mwpsaCommit: You have done with the message what you want to do, successfully. After commit, the message is basically gone.

mwpsaRetry: Will retry in a short moment. Better use rollback to ensure that stale detection can kick in.

mwpsaSkip: Will skip the message. No rollback nor commit will happen. Rarely use.

Then instantiate an instance of the processor, set its Queue property to point on a queue you want it to process, and call Start. When you no longer want it to process messages call Stop.

/Kim/C4D

[kimbomadsen]

Hi,

What type transport are you using?

/Kim/C4D

[kimbomadsen]

You can enable use of TMonitor for locking instead of kbmMW’s own fast MREW locking mechanism by defining: KBMMW_SUPPORT_MONITORLOCK

/Kim/C4D

[kimbomadsen]

It is definitely not better 🙂
The compile problem comes due to that Delphi MREW do not support timeout for BeginRead or BeginWrite methods.
As some parts of code will want to support timeout, I am not so sure if it is worth retrofitting it to support the parameter but ignore it.
/Kim/C4D

[kimbomadsen]

As they have now been running in production for some time, the kbmMW Socket WIB server side transport will be included in next release.
/Kim/C4D

[kimbomadsen]

Actually disregard that. It should be working even with that definition.
LMarshaller is defined in kbmMWGlobal.pas as long as ANDROID is defined and KBMMW_NODEBUG is not defined.

So I wonder if it is finding an older kbmMWGlobal.pas compiled unit?

/Kim

[kimbomadsen]

Hi,

Have you defined KBMMW_NODEBUG somewhere before compiling? If yes, try to remove the definition.

/Kim

[kimbomadsen]

Hi,

Thank you for the report. I will have a look at it….

/Kim

[kimbomadsen]

Hi,

I would not expect a component owned by the form to be manually freed. What happens if you instantiate it without ownership and then free it?

/Kim

[kimbomadsen]

Hi,

Delphi is a 32 bit IDE, so you only install the 32 bit variant of kbmMemTable designtime in the IDE.
Having done that, you can easily compile your applications for 64 bit by just setting the appropriate target in your project settings.

/Kim

[kimbomadsen]

They have still not been released. But they are getting more and more relevant for some projects for me, why they are getting more and more attention 😉

[kimbomadsen]

Yeh.. but it could read that I unfortunately reply full stop 😉

[kimbomadsen]

The TkbmMWFilePool should in itself be threadsafe, as long as you are using the Access and ReleaseAccess methods and never manually trying to free any TkbmMWFileReference instance returned by Access.

/Kim

[kimbomadsen]

Hi,

You can set events.ExceptionHandling to one of mweehIgnore,mweehLog,mweehExcept. It is default set to mweehIgnore.
Even when set to ignore, you can use the event handler OnException to silently catch the exception at runtime.

/Kim

[kimbomadsen]

Hi

There will be an additional set of events which will include the context.
The first context less event will be called first, then the event with context.

/Kim

[kimbomadsen]

Unfortunately? 😉

[kimbomadsen]

Hi,

The reason is that IndexFieldNames is empty before the FindKey call. It will become that if you switch from another index back to the row order index for example.
There were a bug which were fixed in 7.93 which probably could be the reason you see this, as previous versions incorrectly did not reset it to an empty string when switching to the roworder index.

/Kim

[kimbomadsen]

Hi,

The fix will be included in upcoming release.
/Kim

[kimbomadsen]

In the URL, http://127.0.0.1:888/xxx/yyy/index.html?url=/xxx/yyy

xxx must be the same on both sides, and similarly with yyy.
It is a “feature” of the index.html file and/or Swagger itself from what I can tell.

So if you want to have another Swagger entry into a different side of your server, you will need to add another OpenAPI method on that additional service, and make sure it responds to another xxx path.

Eg. http://127.0.0.1:888/myotherentry/api/index.html?url=/myotherentry/api

Then your additional service should have a kbmMW_REST attribute that names it to be called as myotherentry, in same syntax style as the first REST’able service.

/Kim

[kimbomadsen]

Hi,

Seems like the BPL files may not be in the Windows PATH or in Delphi’s library search path.

/Kim

[kimbomadsen]

Hi,

Visit: https://blog.csdn.net/qq_18932003/article/details/103253217

You may need Google Translate as it is a Chinese article.

/Kim

 

 

[kimbomadsen]

Well CopyMemory does internally use Move and nothing else.

But the problem most likely is that DrawData should be changed to:

procedure TkbmMWDIB.DrawData(const ASourceData:pointer);
begin
if ASourceData<>nil then
Move(ASourceData^,FData^,FBytes);
end;

Regarding order of monitors.

According to the Win32 docs, then EnumAdapters1 will first return the adapter holding the primary monitor (index 0).

On that adapter, EnumOutputs will first return the output that is connected to the primary monitor (again index 0).

So there should be no reversal in the way that kbmMW remote server DXGI retrieves monitors as far as I can see.

/Kim

[kimbomadsen]

Hi,

At any point in time you want to.
And remember to switch to it, then your deleted records will show up too.

Example from the Win32 demo:

// Define special index for deleted records (index must include at least one field name to index on, in this case PERIOD)
kbmMemTable1.AddIndex2(‘deleted’,’PERIOD’,[],[usDeleted]);
kbmMemTable1.IndexName:=’deleted’;

/Kim

[kimbomadsen]

Thanks! Received it. I will check it out!

/Kim

[kimbomadsen]

Can you show me a small sample here of how you were creating the tables with quotes, so I get an understanding of if it is something that I need to look into 🙂
/Kim

[kimbomadsen]

Try to set it to afoWithData.

Amd you may choose to do a FetchDefinitions for relevant server side StoredProc/Query components before first use. Then they will know what parameters are available.

/Kim

[kimbomadsen]

Yes, adding kbmMWSmartEvent to the uses clause makes a singleton Event variable available, which is used for subscribing for events and notifying others about events happening.

 

/Kim

 

[kimbomadsen]

Hi,

Ok… so you want to download a file using kbmMW from another, non kbmMW HTTPS site?
kbmMW primarily focus on the server side of things when we are talking about HTTP/REST etc. so it does not include a generic HTTP client side component as it is today.
However, now that we have a native socket library that is getting to be stable included, it is something that I most likely will put some focus on in the future, as client/consumer side support for various things is a missing feature.

/Kim

[kimbomadsen]

Hi,

Just set it to REST.

It supersedes HTTP.

/Kim

[kimbomadsen]

?? Not understood.

What do you mean?

/Kim

[kimbomadsen]

Hi,

You should be able to do the following in the method that handles your request in the server:

ResponseStream.Error(KBMMW_ERR_HTTP_EXCEPTIONS_MIN+310,’Returning HTTP status 310′); // This will not raise an exception.

Or if you want to signal an error and thus want to signal a specific error code, but not return data. It will actually raise an exception.

kbmMWRaiseHTTPError(310.’Returning HTTP status 310′);

/Kim

[kimbomadsen]

Hi,

Ok.. I assumed you were talking about our native transports and REST.

/Kim

[kimbomadsen]

Hi,

My guess is that Indy do not detect disconnection until the time when you try to send something due to it being a blocking non async transport, while the native transports are pure async transports, and any changes from the stack will be signalled back as soon as possible.

/Kim

[kimbomadsen]

Hi,

I am looking into the leak.

/Kim

[kimbomadsen]

It will fix the state data leak too yes.

/Kim

[kimbomadsen]

When using messaging for request/response, it is in reality not the transport that is blocking since that is by definition non blocking, and in the native transports even more so (since the none blocking appearance is simulated using Indy which internally really _is_ blocking…. confused?? 🙂 )

But what is really blocking it is the TkbmMWCustomClient (and its descendants), who will ask the transport to maintain a list of outstanding requests and await the response of the one matching the client’s current request.

You can hook into the wait behaviour via the client event OnClientWaitingResponse.
You will be getting info in the arguments about how long time is left waiting for the client, roughly 10 times a second.

You will also be able to set the AAbort argument to true, in which case the wait will immediately stop.

So you could add this line to the event:

AAbort:=Application.Terminated;

Then the client will not await any responses from the server if the application is being shut down.

/Kim

[kimbomadsen]

Hi,

Problem is that I do not see a server unresponsive situation. In fact the server continues fine, but no screens are forwarded in my case and I can switch on the fly to GDI.

I’m running on dual Titan V cards.

So it will probably need some debugging on your specific setup to figure out where it hangs.

/Kim

[kimbomadsen]

Those messages may somehow throw exceptions and be rolled back to the queue. If no stall detection is defined, then they will stick there.

However I have seen relatively rare situations where the Indy transport implementation may not provide a complete message to the messaging framework. It is changed with the native messaging transport, which handles all the packet fragment collection itself.

/Kim/C4D

[kimbomadsen]

Thank you. It is changed in next release.
/Kim/C4D

[kimbomadsen]

Yes… only one of them can be enabled at any time.
/Kim/C4D

[kimbomadsen]

Hi,

Try to open kbmMWCore.pas, locate kbmMWGetCurrentThreadID and change its return value type to TkbmNativeUInt instead of TkbmNativeInt. Then recompile and check if your application works then.
Please report result back to me.

/Kim/C4D

[kimbomadsen]

I agree on your last suggestion so will remove the call to DoOnLoginSuccess when the user is already logged in.

/Kim/C4D

[kimbomadsen]

Hi,

There will be some improvements in next version, which _may_ affect your observation.

/Kim

[Author]

Posts