Bayesean Blog - Desktop, Mobile and IOT Developer Blog


Delphi Delimited String to Fields

Posted on 22nd Aug 2019 in Delphi > XE4, Delphi VCL


strtofields6.png

As part of a recent project, I was converting a KML (Keyhole Markup Language) file to a GPX (GPS Exchange Format) file which required the extraction of longitude, latitude and elevation coordinates into a comma delimited string format. After extracting the into a string, the string needed to be parsed into three fields. As data could vary in size up to millions of coordinates, this necessitated a method that would not impact on process time nor create memory issues when converting a delimited string into fields.

Whilst it is simple to convert a text file to fields easily, it is converting an available delimited string into different text fields that requires a changed approach.  Other scripting languages’ built in techniques makes it really easy to parse text into different formats. Delphi requires a slight difference in approach to achieve the same.

I will share the various methods that were tried and the most effective method chosen. The code can be downloaded at the end of the blog.

parse40.png

Four Methods Tried

So I tried a few direct methods, all with large test data and was rather surprised and disappointed by the results.

Firstly I streamed the string into a string array and parsed it directly to a multi-string array  using the standard maths formula “(i + j ) * ncols”. Straightforward but perhaps a little overkill however anything over 3000 records and memory issues kills the process.

Secondly extracted the sting straight into a stringlist and then parsed this directly to a multi-string Array using the above process. Whilst it improved, memory issues still persisted.

So it seems that the methods above may be exceeding the physical memory limits and processor address space limits.

I tried the TBatchmove component  with the FDBatchMoveTextReader.  It has the option to load from a stream method. Whilst loading a file is always effective, the stream method would not parse the string, with or without sent headers into fields?

Chosen Coding Method

Parsing the string into a base Tstringlist and thereafter parsing the data into three other Tstringlists as fields is the most efficient method over all the other methods that were tried.

I will only share the important parts of the code as the full downloadable code is available via Github at the end of this blog.

This method makes use of four stringlists. The first stringlist holds all the original string items split out from the delimited text. The other three stringlists are used to receive the parsed data and will act as the three individual field’s data. This seems to be the easiest process that reduces the chances of receiving memory issues for large string data as each stringlist is able to hold up to 134,217,728 strings.

After creating the four stringlists the temporary stringlist it is important to set the StrictDelimitor to true with the delimiter char set and the Delimitedtext routine loads the string directly into the stringlist.

The StrictDelimitor forces the use of the delimiter and the Delimitedtext is used to receive the string. The use of the trim function is used to remove excess white spaces.

To parse the sTemp (stringlist) into the three fields the “for ..do” is used to loop through all of the sTemp (stringlist) rows.

We need to extract every third item which is determined by the Maths mod function.

We use this function to extract every third value i.e.  The Value mod three (3) must equal to zero (0).

As we are looping from zero (0) will result in dividing by zero (0) such as “0 Mod 0” and hence will result in creating a calculation error. Adding a simple if statement will take care of this.

Whilst not apply to this example and some info, be aware that the Delphi Mod much like C# does not return the correct values for negative integers. Most other languages C, C++ take this into account.

 Each field row is updated when stringlist.add() is called and adds the following respective field values

 

  • first field STemp[i] + 0 
  • second field STemp[i] + 1
  • third field STemp[i] + 2

 The code for this is as follows:

for i := 0 to sTemp.Count - 1 do
  begin
  //first field
   if i = 0 then
      sfield1.add(sTemp[0]); // extract first field name
    if (i > 2) and (i < sTemp.Count) and (i mod 3 = 0) then
      sfield1.add(sTemp[i]+ 0);
  //second field
  if (i <= sTemp.Count - 2) and (i mod 3 = 0) then
      sfield2.add(sTemp[i + 1]);
//third field
    if (i <= sTemp.Count - 3) and (i mod 3 = 0) then
      sfield3.add(sTemp[i + 2]);

From now on it is easy to parse the three fields directly to a Database, XML file or Json by looping through the length of any one of the three fields

For j = 0 to Sfield1.Count – 1 do
Begin
Field1 := sfield1[i];
Field2 := sfield2[i];
Field3:=  sfield3[i];
End;
// where Field1, Field2, Field3 can be a DB or in my case Longitude, Latitude and elevation //data.

Well this sums up the best method for parsing a delimited text string into fields.

Feel free to download the code here

Download Code


Reg      Commented   3 years ago Reply

I found in Stack Overflow that they use a foreach statement to pull the data. Your method is different. Which is the best method?

Admin      Replied    Moderator   3 years agoReply


Yes I am aware of the Stack Overflow method suggested. The problem is that we are dealing with server side and client side script. The foreach method updates the Client on every call. So if you has 500 rows of data, then you will be persistently updating the client 500 times and as it is a single threaded call, the client will simply stop working until the process is complete. Loading the data into Json in the Server creates a lightweight data object. This is only sent once to the client and the Json object (string) is quickly unpacked in the client allowing a quicker response. I hope that this clears up your concerns.

Add a Comment

9+5

Recent News

Delphi Delimited String to Fields
Delphi A Professional VCL DBGrid Part Four
Delphi A Professional VCL DBGrid Part Three
Delphi A Professional VCL DBGrid Part Two
Delphi A Professional VCL DBGrid Part One
Delphi VCL Buttons in DBGrid
Two Helper Apps for Delphi LibUSB
Delphi Libusb Library Introduction

Categories

Bootstrap 4
Delphi VCL
Delphi FMX
Ajax
Bootstrap 3
CSS
XE4>Delphi > XE4
Delphi < XE4
PHP

Archives

August 2019

Delphi Delimited String to Fields

June 2019

Delphi A Professional VCL DBGrid Part Four

May 2019

Delphi A Professional VCL DBGrid Part Three

April 2019

Delphi A Professional VCL DBGrid Part Two

March 2019

Delphi A Professional VCL DBGrid Part One

November 2018

Delphi VCL Buttons in DBGrid

October 2018

Two Helper Apps for Delphi LibUSB

September 2018

Delphi Libusb Library Introduction

August 2018

Delphi Object directly to a Json string in a REST Client
Delphi using Environment Variables in your App

July 2018

Delphi FMX Leaflet Plotter using OSM Maps

June 2018

C2PAS32 Convertor Application
C to Delphi Open Source Convertors Shootout
Delphi command-line programs with DOSCommand

May 2018

Delphi PDF Embedded viewer with PDF.js

March 2018

Delphi FMX - Changing TCharacter to TCharHelper
Make Your Delphi App POP using Javascript!

January 2018

Delphi FMX Dashboard using Chart.JS
Delphi FMX Form Docking

December 2017

PHP Slim REST Server & Delphi Auth Part 5

November 2017

Delphi FMX REST Client App Part 4

October 2017

Delphi VCL REST Pricing Client App Part 3

September 2017

Delphi REST VCL Client Basic Auth Part 2B

August 2017

Delphi REST Client Part 2A
PHP PDO REST Server Part 1

July 2017

PHP REST Server and Delphi Client Intro

June 2017

Delphi SQLite Encryptor-Decryptor Tool
Updating Applications Manifest using Delphi

May 2017

Create a Visual IP Address Geolocation with PHP

March 2017

PHP Downloader using Countdown timer
PHP File Downloader from a Inbox Selection

February 2017

Javascript Image-File Uploader with ThumbViewer

January 2017

Morris Charts and PHP-PDO

December 2016

CSS to create a functional Toggle Button