No announcement yet.

Handling of large files?

  • Filter
  • Time
  • Show
Clear All
new posts

  • Handling of large files?

    Hi @All

    I like to build a scope mount drive for astronomical telescopes.
    The device consists of a base device which contains the Power electronic, stepper motor drivers and the main controller. IN the hand control I want to use a 4DGL display to diplay all the informations. It is controlled by the base device via serial commands but uses itself 4DGL. Now I have a problem.
    I want to store catalogs of astronomical objects onto the uSD card. A special catalog program (KATALOG.4XE) opens the desired files for the catalog, parses the information, displays the info and a picture of the object( if exist on uSD). Sample code is attached. A record for each astronomical object in the file is about 100 bytes large. For small cataloges like the "Messier"-Katalog which only contains 109 entries ther is no problem. The file size is small so that I can use a word (16bit) counter to parse through all records. But my (smallest) version of the SAO catalogue contains about 400.000 records. So it is not possible to count all with a WORD counter. A DWORD counter will do this but DWORD is not implemented in 4DGL yet. How can I 'simulate' it with 2 DWORDs? Sorry for my question but i never had experiences with languages which only supportd WORD and no FLOATs, DWORDs and so on.

    An other problem is the file_Seek(F,0,(idx*RECSIZE)) command which uses a HIWORD and a LOWORD to address the offset in the file. How can I discover when I an overflow of this expression occurs and I have to increase or decrease the HIWORD of the file_Seek command?

       var BasImg,KatImg;
       var ex;
       var p;                  // buffer pointer   
       var i,m,j,n,k,h,c;   
       var key; 
       var combuf[2];          //buffer for serial Communication
       var F;                  //file
       var KatSizeHi, KatSizeLo,nrec;
       var rec[100];
       var mysub[30];
       var ObjInfo[10];       // array for infos
       var ObjInfo0[20];     //arrays for Strings
       var ObjInfo1[20];
       var ObjInfo2[20];
       var ObjInfo3[20];
       var ObjInfo4[20];
       var ObjInfo5[20];
       var ObjInfo6[20];
       var ObjInfo7[20];
       var ObjInfo8[20];  
    func LoadBasisImages()
        print ("Lade Programm...\n");   
        BasImg := file_LoadImageControl("Basimg.dat", "basimg.gci", 1);  // build the image control, returning a pointer to the structure allocation   
        if (!BasImg)   
            putstr("Fehler: Kann Basisbilder nicht laden...\n");   
    func LoadKatalogImages()
        print ("Lade Katalogbilder...\n");   
        KatImg := file_LoadImageControl("Messier.dat", "Messier.gci", 1);  // build the image control, returning a pointer to the structure allocation   
        if (!KatImg)   
            putstr("Kann Messierbilder nicht laden...\n");   
        F := file_Open("MESSIER.KAT", 'r');  //und den zugehörigen Katalog
        nrec:= KatSizeLo/RECSIZE;
    func readKatalog(var idx)
    var bpnext; 
    var ind;
        file_Seek(F,0,(idx*RECSIZE));  //Problem with large files!!
        ind := 0;
        // Point to string to extract sub-strings from
        bpnext := str_Ptr(rec);
            if((bpnext:=GetNextSub(bpnext, mysub, ';')) != 0)
                to(ObjInfo[ind]); print([STR] mysub);
        until(bpnext == 0);
        sys_SetTimer(TIMER5, 500);
    func main()   
        txt_Set(FONT_SIZE, FONT2);   
        txt_Set(TEXT_OPACITY, OPAQUE); 
        txt_Set(TEXT_COLOUR, 0x07E0);
        ObjInfo[0]:= &ObjInfo0[20];
        ObjInfo[1]:= &ObjInfo1[20];
        ObjInfo[2]:= &ObjInfo2[20];
        ObjInfo[3]:= &ObjInfo3[20];
        ObjInfo[4]:= &ObjInfo4[20];
        ObjInfo[5]:= &ObjInfo5[20];
        ObjInfo[6]:= &ObjInfo6[20];
        ObjInfo[7]:= &ObjInfo7[20];
        ObjInfo[8]:= &ObjInfo8[20];
        com_Init(combuf, 1, ':' );  //Com Buffer initilize for one Key
        if (!file_Mount()) print("File error ",file_Error());   
        n := 0;   
         key:=readcom();   //Waits for a Keypad Command from Host Controller
         if (key ==KEYDOWN)     //Selects the next catalog element
           if (n<nrec) n++;
         if (key ==KEYUP)     //selects the proir catalog element
           if (n>0) n--;
         if (key ==KEYOK) 
           print("Enter");     // here Will be the routine to send Objectinfo to host conbtroller
        until (ex == 1); 

  • #2

    From the doco it would appear that file seq is expecting two unsigned 16 bit numbers as highword and loword. This means the compiler will be printing negative numbers for half the lowword values

    I presume if you add 1 to 32767 and so on, you end up with -32768, -32767.... -1, 0 at least that's the normal behaviour.

    Also, I'd start playing with 8 bit numbers, (use 3 to make 24 bits) so if you multiply the low byte (only, easily, by an 8 bit number) you can extract out the high order byte to move into the Mid byte, etc. similiar for addition.

    Quite messy, so you just tend to use what you need for any given application.

    Hope that's helpful, and makes sense.


    • #3

      From your last posting regarding extracting sub-strings, this sort of points me towards the fact that you are using variable length fields (which usually implies variable length records.) If you are using fixed length fields then it is actually more efficient to extract them from their known offsets, and not do any string parsing.

      But getting back on-thread, given that you want to read fixed length records from your file you can easily manipulate 2 x 16 bit unsigned Words as a single 32 bit value for passing into file_Seek(), but as you point out 4DGL doesn't have native support for 32 bit integers.

      I have a few Int32 manipulation functions you can have (assignment, inc, dec, compare etc.) but no multiply/division. However, it is trivial to do multiplication and division when multiplying or dividing by binary factors (i.e. 1, 2, 4, 8, etc.)

      If you have the opportunity to set the record length of your catalogue to a binary factor then my functions (slightly extended) would be of use to you.



      • #4

        Hello Steve

        I have downloaded several catalog files (Messier, NGC, SAO,Sharpless, Melotte ...) from the Internet and all had a different format.
        The I decided to set all records to a fixed length so that it is possible to read them at once in a buffer rec
        Each record is terminated by a CR.

        Of course I can change the format because I have to convert all catalogs.
        With Excel and Ultraedit I was formating the records fields so that they also have a fixed length. Th first entry, the object name has a length of 18 characters because I want to diplay it in a List with buttons (see attached Picture on TOP of the screen Test1, Test2 and Test3). Because the gfx_Button command has no possibilty to set with and Height of the button you only can handle this via the caption length and the font. To add spaces in the katalog file was for me the easiest way.;-) It works fine.

        I'm highly interested in 32bit funtions. because I some catalogs have more than 65.000 records. Division and multiplication is not so important at the moment. But addition/substraction is very useful for me. Maybe when I will be more familiar with 4DGL in the future, I can develop division and multiplication function by myself. At the moment I'm a beginner. With Borland Delphi, which we use at work, I'm a Pro :-)

        Attached files


        • #5
          Int32 functions attached + new rotate left (multiply) and rotate left (divide.) main() demonstrates usage.

          With the functions in place your calling code will look something like this...

          var _i32fmax[2]; // Int32 total number of records
          var _i32frec[2]; // Int32 file record number (32 bit unsigned)
          var _i32fptr[2]; // Int32 file offset pointer


          I32Set(_i32frec, 0x000F, 0xFFF0); // Current record number: 0x000FFFF0
          I32Set(_i32fmax, 0x0010, 0x0000); // Last record number: 0x00100000 (>1mil)

          // Calculate 32 bit file offset for record 'rec'
          RecOffset(_i32fptr, _i32frec, FILE_RECL);

          // Print the calculated offset in _i32fptr[HIGH] / _i32fptr[LOW]
          print("Rec: 0x", [HEX4] _i32frec[HIGH], [HEX4] _i32frec[LOW], " - ", "Offset: 0x", [HEX4] _i32fptr[HIGH], [HEX4] _i32fptr[LOW], "\n");

          // I32Inc(_i32frec); // Next record

          until(I32IncComp(_i32frec, _i32fmax) >= 0);



          • #6

            Thank you for the rountines but I have some small errors.
            [quote]// Perform bitwise rotate right of Int32 (divide by power of 2)
            // i32p = Int32 to rotate
            // numbits = number of bits to rotate (1


            • #7

              This operation is as designed. i32p is rotated in place and the bits rotated out of i32p are placed in i32pc ('c' standing for carry.) i32pc therefore needs to be primed with '0's. For your purposes you can ignore i32pc.

              E.g. rotating i32p 0x87654321 left by 4 bits will leave 0x76543210 in i32p and 0x00000008 in i32pc.

              main() demonstrates the calling of the code.

              Not sure whether you realise but you will need to change your record length from 100 bytes to 128. By feeding in a RECL of 128 bytes to RecOffset() it will calculate the required bitwise rotation.


              • #8

                Further to my initial response I realised that you are talking about the rotate right function. It should still work as I described but is not of use for multiplication. I assume that you are using it to divide by 128 to calculate the total number of records in your catalogue.

                As an example, to use it for division (file_size / 128) then you need to shift the file size right by 7 bits (2^7 = 128.) To do this just declare an Int32 called _i32size (or reuse a spare one) to hold the file size and then call I32ShiftR(_i32size, _i32c, 7). Note that if _i32c is non-zero then there is a remainder and the file size is not exactly divisible by 128.



                • #9

                  Oh, sorry.
                  English is not my native language.
                  I should have to read mor carefully.
                  I understood it so that i32pc contains the rotated "result" of i32p.


                  • #10

                    Don't worry about it!

                    The Int32Dec() function should read...

                       i32p[LOW]--; if(i32p[LOW]==0xFFFF) i32p[HIGH]--;


                       i32p[LOW]--; if(i32p[LOW]==0
                    ) i32p[HIGH]--;

                    The version you have does not wrap correctly at the 64k boundary. (One of the perils of cut and paste!) I'd never exercised the function (I had only used the increment ones.)