Reading/writing ints and floats from/to a binary file.

[saiph]

Hi all.

I use a historical motor racing sim called "Grand Prix Legends" on my PC, and I'm trying to write a utility to edit and compare car setup files outside the game. These setup files contain data for each car, detailing suspension values, tyre pressures, gear ratios etc. and they define how the car will behave on-track. The in-game interface for editing the car data isn't very good, and doesn't allow comparison between different setup files. Also, I very often like to tinker with new setup ideas without going to the trouble of starting up the whole game. So I thought I would jump in the deep end and try to create my own editor.

However, I seem to be having problems at the first hurdle. I'm trying to read in data from a setup file, to translate it into a meaningful format so I can display the setup values. The setups are contained in fairly small (272-byte) binary files, and the Grand Prix Legends community has published details of the data format. I've created a shortened (16-byte) file to act as a test, but I'm having trouble translating the values from the file into their proper values in my program.

Here is a hex dump of the 16-byte file and the values I'm looking for:


50 47 54 53    04 00 00 00    01 01 00 00    C4 4E 2C C0

P  G  T  S     4              257            -2.6923

4-byte ASCII text "PGTS": standard header for all Grand Prix Legends setup files - helps confirm I'm looking at the right type of file.

32-bit INT - value 4: represents the file subtype.

32-bit INT - value 257: in the full-sized file this gives the length of data in the file, after the file header bytes.

32-bit FLOAT - value -2.6923: ratio of reverse gear

I've tried various ways of reading the data. My first thought was to use the "FIELD" keyword:

FIELD #CarSetup, 4 as HeaderTag$, 4 as SubType, 4 as DataLen, 4 as ReverseRatio

And then just use

GET #CarSetup, 1

to read the data as if it were the first record of a random-access file. But unfortunately, although HeaderTag$ was correct, the numeric variables didn't get the right values.

I then thought I would have to resort to this pseudocode algorithm for reading the INT variables:

Value = 1st Byte

Value = Value + (2nd Byte * 256)

Value = Value + (3rd Byte * 65535)

Value = Value + (4th Byte * 16777216)

But that still leaves the problem of the 4-byte Floats. I've also tried to use INPUT$(#CarSetup, 4) in various ways, but that didn't work either.

If anyone could give me some pointers (hang on, only C/C++ has pointers, JustBasic doesn't ) on how to solve this I would be very grateful. I know that it should be possible to achieve what I want, as a fellow GPL player has released a setup viewer a few years ago written in JustBasic. That's what actually alerted me to the existence of JustBasic, as his program has a very neat GUI, and I wondered how he had written it. He mentions JustBasic in his readme file, so I Googled for it and was impressed by what I found, especially the GUI editor which is included with the language.

I have left a private message for my fellow player on a Grand Prix Legends forum, but it looks like he's only an infrequent visitor these days, so it might be quite some time before I get a reply there. So I thought I would ask here.

Thanks in advance for any help that's offered.

[B+]

With files so small, I'd use sequential access. You could make each line a field or you could use a whole line for 1 record, maybe separate the fields with commas. With sequential access you don't have to manage bytes and you can use any Word Processor to write or edit data.

Here is a tip for using a sequential file as a Random Access file: justbasiccom.proboards.com/thread/155/random-access-sequential-files

[Rod]

Getting the float value back will be possible. It’s just bytes, the format is the thing to tackle.dont have much time right now but there are many members here that will be able to puzzle out the four byte float value.

[tenochtitlanuk]

Here's the easy bit.. as Rod says, the 32 bit float is more complex..

open "test.fil" for input as #fIn
   content$ =input$( #fIn, lof( #fIn))
close #fIn

for i =1 to 4
   print  mid$( content$, i, 1);
next i

print

v =0
v =v +asc( mid$( content$,  5))+_
     asc( mid$( content$,  6)) *2^8+_
     asc( mid$( content$,  7)) *2^16+_
     asc( mid$( content$,  8)) *2^24
print v

v =0
v =v +asc( mid$( content$,  9))+_
     asc( mid$( content$, 10)) *2^8+_
     asc( mid$( content$, 11)) *2^16+_
     asc( mid$( content$, 12)) *2^24
print v

[jump]
open "test.fil" for input as #fIn
   content$ =input$( #fIn, lof( #fIn))
close #fIn

for i =1 to 4
   print  mid$( content$, i, 1);
next i

print

v =0
v =v +asc( mid$( content$,  5))+_
     asc( mid$( content$,  6)) *2^8+_
     asc( mid$( content$,  7)) *2^16+_
     asc( mid$( content$,  8)) *2^24
print v

v =0
v =v +asc( mid$( content$,  9))+_
     asc( mid$( content$, 10)) *2^8+_
     asc( mid$( content$, 11)) *2^16+_
     asc( mid$( content$, 12)) *2^24
print v


end


[tenochtitlanuk]

I'll try to write a routine but don't have time now.
The image should show how it is broken down- three parts- sign, exponent and 'significand. Plus you need to understand binary representation of negative numbers...


EDIT
AND note the bytes are used in reverse order.

[saiph]

Thanks for the replies so far, chaps.

I've had time to do a bit more searching around myself, and it seems the easiest way to do what I want would be to use the BASIC functions CVS and MKS$. After reading the 4-byte (single-precision float) binary string from my file, CVS converts it into a floating point value. MKS$ does the reverse, taking a floating point value and converting it to a 4-byte string, ready to be written to a binary file.

However, it appears that CVS, MKS$ (and other functions which deal with other data types) do not exist in JustBasic 2.0. They are mentioned in the HELP section of the FIELD statement, and there it implies that they are unnecessary. But in my case they clearly are necessary.

I have found a section in Wikipedia which describes floating point format, and how to encode/decode numbers, but it is a little complex. I will continue to work on it to see if I can use JustBasics' bitwise functions to extract and decode my binary floats. But I will also continue to check back here in case anyone comes along with an easier solution.

OOPS! Looks like we cross-posted tenochtitlanuk! That looks interesting - I'll add it to the knowledge I'm picking up from Wikipedia and see if I can do anything with it. Thanks!

EDIT: I've had another look at the JustBasic program written by my fellow Grand Prix Legends player. He supplied the program in runtime form, with the JustBasic DLL files, a .TKN file, and a renamed version of the runtime engine. I loaded the runtime engine file into a hex viewer, and from version strings in the file, it seems that he was using JustBasic v1.01 ("jbrun101.exe"). He stated in his readme file that he was only a "novice" programmer, so I find it hard to believe that he would have needed to resort to bit-twiddling to read in the float values from a binary file. Did JustBasic v1.01 have the 'CVS' and 'MKS$' functions available? Were they not implemented in JustBasic v.2 for some reason?

[tsh73]

EDIT: I've had another look at the JustBasic program written by my fellow Grand Prix Legends player. He supplied the program in runtime form, with the JustBasic DLL files, a .TKN file, and a renamed version of the runtime engine. I loaded the runtime engine file into a hex viewer, and from version strings in the file, it seems that he was using JustBasic v1.01 ("jbrun101.exe"). He stated in his readme file that he was only a "novice" programmer, so I find it hard to believe that he would have needed to resort to bit-twiddling to read in the float values from a binary file. Did JustBasic v1.01 have the 'CVS' and 'MKS$' functions available? Were they not implemented in JustBasic v.2 for some reason?

No, JB 1.01 have no 'CVS' and 'MKS$' functions.
But bit twiddling is not *that* hard.
Or he could pick up function on a forum.

Here I utilized some hex/bin functions I already had:
(it is in-effective as to speed, but gets job done)
'https://en.wikipedia.org/wiki/Single-precision_floating-point_format
'C4 4E 2C C0
a=hex2Dec("C4")
b=hex2Dec("4E")
c=hex2Dec("2C")
d=hex2Dec("C0")

'c=hex2Dec("80")
'd=hex2Dec("3F")

'backward bytes
print d,c,b,a
print eightBitPattern$(d),
print eightBitPattern$(c),
print eightBitPattern$(b),
print eightBitPattern$(a)

bits$=eightBitPattern$(d);eightBitPattern$(c);eightBitPattern$(b);eightBitPattern$(a)
'31
print bits$
sign = val(mid$(bits$,1,1))
print "Sign: (1-negative) ";sign
'30-23
pow$=mid$(bits$, 32-30, 8)
print "Power: ";pow$
pow = bin2num(pow$)-127
print "Power: ";pow
'rest is 22 to 0
rest$=mid$(bits$, 32-22)
print rest$
'1 is assumed
'Wikipedia:
'The true significand includes 23 fraction bits to the right of the binary point
'and an implicit leading bit (to the left of the binary point) with value 1,
'   unless the exponent is stored with all zeros.
if pow$<>"00000000" then mantissa = 1   'else 0
for i = 1 to len(rest$)
   mantissa = mantissa + val(mid$(rest$,i,1))/2^i
next
print "Mantissa =";mantissa
print "--------------------"
num = (1-sign*2)*mantissa*2^pow
print num



'reading from binary file #1
'and some conversions
' some functions ********************************************8
function readByte()
   readByte = asc(input$(#1, 1))
end function

function read2Bytes()
   read2Bytes = byte2Num(input$(#1, 2))
end function

function readLong()
   readLong = byte2Num(input$(#1, 4))
end function

function readStr$(n)
    readStr$ = input$(#1, n)
end function
'-----------------------------------------------------------------
'type conversions
function byte2Num(c$)
   res = 0
   for i = len(c$) to 1 step -1    'lower byte first
       res = res * 256+asc(mid$(c$,i,1))
   next i
   byte2Num = res
end function

function bin2num(bin$)
   bin2num = 0
   for i = 1 to len(bin$)
       c$=mid$(bin$, i, 1)
       if instr("01",c$)=0 then exit function  'like VAL(), break on first non-valid digit
       bin2num = bin2num*2+(instr("01",c$)-1)
   next
end function

function eightBitPattern$(num)
   for i = 0 to 7
       eightBitPattern$ = str$((num and 2 ^ i) > 0) + eightBitPattern$
   next i
end function

function BitPattern$(num)
   for i = 0 to 31
       if i mod 8 = 0 then BitPattern$ = " " + BitPattern$
       BitPattern$ = str$((num and 2 ^ i) > 0) + BitPattern$
   next i
end function

function bitwiseNot(num)
    bitwiseNot = num XOR -1
end function

'Converts HEX string to (decimal) number. So, "FF"->255
function hex2Dec(s$)
   hex2Dec = 0
   For I = 1 To Len(s$)
       c$ = upper$(Mid$(s$, I, 1))
       If InStr("0123456789ABCDEF", c$) = 0 Then exit function    'like VAL, it stops on first non-valid symbol
       hex2Dec = hex2Dec * 16 + InStr("0123456789ABCDEF", c$) - 1
   Next I
End Function

function hex2Dec4(s$)
   hex2Dec = 0
   'supposed valid value - up to 4 digits
   If Len(s$) > 4 Then Exit Function
   For I = 1 To Len(s$)
       c$ = upper$(Mid$(s$, I, 1))
       'safeguard? Looks plain WRONG
''        If InStr("0123456789ABCDEF", c$) = 0 Then c$ = "0"
       hex2Dec = hex2Dec * 16 + InStr("0123456789ABCDEF", c$) - 1
   Next I
End Function

Function hex$(n)
   h$ = ""
   do
       h$ = mid$("0123456789ABCDEF", (n mod 16)+1, 1) + h$
       n = int(n/16)
   loop while n > 0
   hex$ = h$
End Function

Function hex2$(n)
   hex2$ = hex$(n)
   If n < 16 Then hex2$ = "0" + hex2$
End Function

Output:

192           44            78            196
11000000      00101100      01001110      11000100
11000000001011000100111011000100
Sign: (1-negative) 1
Power: 10000000
Power: 1
01011000100111011000100
Mantissa =1.34615374
--------------------
-2.69230747


[Rod]

Ah beaten by Anatoly. I was struggling with getting the bits back to decimal. Neither Liberty BASIC nor Just BASIC have supported single precision. We have three data types Double floats, integers or strings. That said in Liberty BASIC you can call API resources to make the conversion. But is as near complex as Anatolys solution, mine looks even worse and wasn't giving the correct answer! Go to Bay6 Software and look at the snips tab Brent has both functions you speak of. But you would need Liberty BASIC.

www.b6sw.com/forum/content.php?mode=snips

[saiph]

Thanks Anatoly and Rod. You have some neat and clever functions there, Anatoly.

[tenochtitlanuk]

Looks like we all enjoyed playing with this.
My messy code follows- basically automating what is actually going on, and not cleaned nor optimised.

   '        C02C4EC4
   i$ ="asdfC44E2CC0zyxw" '    We are interested in 4 bytes in reversed order at position 5 in a longer string..

   print Hex2Float( mid$( i$, 5, 8))

   end

   function Hex2Float( s$)   '   4 hex chars from position p in supplied string of bytes ( file)
       print s$
       for i =3 to 0 step -1
           t$ =t$ +mid$( s$, 1 +2 *i, 2)
       next i
       print t$

       for i =1 to 16
           bin$ =bin$ +hex2bin$( mid$( t$, i, 1))
       next i

       print bin$

       exponentBin$    =""
       significand     =1

       for i =1 to 32
           if i = 1                                then print " Sign bit = ";
           if sign$ <>"" and mid$( bin$, 1, 1) ="1"  then sign$ ="-" else sign$ ="+"
           if i = 2                                then print "   Exponent = ";
           if i >1 and i <10                       then exponentBin$ =exponentBin$ +mid$( bin$, i, 1)
           if i =10                                then print "   Significand = 1.";
           if i >=10                               then significand =significand +val( mid$( bin$, i, 1)) /2^( i -9)
           print mid$( bin$, i, 1);
           if i =1                                 then print " ";
       next i

       print

       print sign$, bin2dec( exponentBin$), significand
       print

       Hex2Float =significand *2^bin2dec( exponentBin$)
       if sign$ ="-" then Hex2Float =0 -Hex2Float
   end function

   function bin2dec( j$)
       for m =len( j$) to 1 step -1
           b =val( mid$( j$, m, 1))
           if b =1 then expDecimal =expDecimal +2^( 8 -m)
       next m
       if val( mid$( j$, 1, 1)) =1 then bin2dec =expDecimal -127 else bin2dec =expDecimal
   end function

   function hex2bin$( a$)
       select case a$
           case "0"
               o$ ="0000"
           case "1"
               o$ ="0001"
           case "2"
               o$ ="0010"
           case "3"
               o$ ="0011"
           case "4"
               o$ ="0100"
           case "5"
               o$ ="0101"
           case "6"
               o$ ="0110"
           case "7"
               o$ ="0111"
           case "8"
               o$ ="1000"
           case "9"
               o$ ="1001"
           case "A"
               o$ ="1010"
           case "B"
               o$ ="1011"
           case "C"
               o$ ="1100"
           case "D"
               o$ ="1101"
           case "E"
               o$ ="1110"
           case "F"
               o$ ="1111"

       end select
       hex2bin$ =o$
   end function


Producing...
C44E2CC0
C02C4EC4
11000000001011000100111011000100
Sign bit = 1    Exponent = 10000000   Significand = 1.01011000100111011000100
-             1             1.34615374

-2.69230747


[saiph]

Ok, after some time carefully reading the Wikipedia entry on floating point, looking at tenochtitlanuk's post, and carefully stepping though Anatoly's code, I've come up with this compacted version, and (miracle upon miracle ) it seems to work! I'm now going to include this code in my experimental file reader, and see if I still get the results I'm expecting.

Thanks everybody for your help!

EDIT: LOL! Looks like we cross-posted again tenochtitlanuk!! But thanks for your code too!

'
' First, create two masking constants for use later...
'
B23to30 = 0                    '
for n = 23 to 30               ' Used for masking bits 23-30
   B23to30 = B23to30 + 2^n    '
next n                         '
print "B23to30 = ";B23to30     '

B0to22 = 0                     '
for n = 0 to 22                ' Used for masking bits 0-22
   B0to22 = B0to22 + 2^n      '
next n                         '
print "B0to22 = ";B0to22       '
print

' Now let's accumulate the value of our 4-byte FLOAT in variable 'a'...                      
a=hexDec("C4")                 'Simulate reading 1st byte of 32-bit float....
a=a+((2^8)*hexDec("4E"))       ' Simulate reading 2nd byte of 32-bit float....
a=a+((2^16)*hexDec("2C"))      ' Simulate reading 3rd byte of 32-bit float....
a=a+((2^24)*hexDec("C0"))      ' Simulate reading 4th byte of 32-bit float.

' Mask out all but bit 31, and shift down by 31 bits to give Sign bit
SignBit = (a AND 2^31)/2^31
print "Sign bit = ";SignBit

' Mask out all but bits 23-30, and shift down by 23 bits to give Exponent
Exponent = (a AND B23to30)/2^23
Exponent = Exponent - 127               ' Remove exponent bias
print "Exponent = ";Exponent

' Mask out all but bits 0-22 to give Significand (no shift required)
Significand = (a AND B0to22)
print "Significand = ";Significand
print "Binary representation: ";
for i = 22 to 0 step -1
   print (Significand AND 2^i)/2^i;
next i
print

' Ok, now let's calculate the Mantissa.
' First, take account of implicit leading bit.
if Exponent = -127 then     ' Was Exponent stored as all-zeroes?
   Mantissa = 0            ' Yes, implicit bit = 0
else
   Mantissa = 1            ' No, implicit bit = 1
end if

for i = 0 to 22
   Mantissa = Mantissa + (((Significand AND 2^i)/2^i)/2^(23-i))
next i
print "Mantissa = ";Mantissa
print
Num = (1-SignBit*2)*Mantissa*2^Exponent
print "Final value = ";Num
print "Should be ~= -2.6923"