string length

[jlthompson]

How much can a single string hold before it generates an out of memory error?

[tsh73]

Go ahead and try
For me, JB 1.01
a$=space$(20*1e6)
print len(a$)
a$=""
print len(a$)

OK
a$=space$(30*1e6)
print len(a$)
a$=""
print len(a$)

"system primitive failed" error.

JB 2.0 should have that limit higher I guess.

Also, you'll probably need some memory for *working* with that string, too.

[Rod]

I suspect that behind the scene we have some accumulation going on. "how much can a single string hold" well keeping the string single might be the key. Consider that a$=space$(20*1e6) actually creates two 20mb strings in memory, the left and right side of the equation. Also consider a$=a$+a$ might actually use four times the string size to compute! On top of that we probably have garbage collection, when we say a$="" a$ instantly becomes "" or 0 but the memory that it once occupied takes a little longer to be released back to the pool.

So processing large strings can quickly eat up resources. Best to tackle large strings in manageable chunks. Keep most data out on disc and bring in what you can handle in reasonable time. That said 20mb strings are quite substantial are they not?

To see what I mean about garbage collection try running the sample code twice. It runs first time and gives 20mb string but it wont run a second time. Probably because the garbage collection is needing to happen.

[jlthompson]

So when I am reading a file with a longish line (around 100,000 KB), and immediately copy that into a couple of strings, and then use the word$() command on one of them, and my program is already so large that it eats up most of the allotted memory, that is likely why I get a "low on memory - process terminated" message, and so breaking that line up into smaller chunks will likely solve the problem then.

[Rod]

Well no not really. Anatoly was using 20Mb strings and we are supposed to have 256Mb available. A really small 100Kb string should be no problem at all.


But there are other ways to use up memory and particularly if you are looping because Just BASIC loops very fast. Graphics can use memory. Drawing to the screen in a loop can use up all memory almost instantly if you are not managing memory.


You can see what memory Just BASIC is using by pressing CtrlAltDel and selecting Task Manager. It will show you the memory increase as you run your program.


For best advice you should really let us see your program.

[Rod]

Oh wait I read that string size incorrectly 1kb 10kb 100kb 1mb 10mb 100mb 1gb is the convention so 100,000kb = 100mb That is a very big string.

[Rod]

I have read back over a few of your posts. It seems you have a very large program and it deals with very large strings and keeps multiple copies of those strings "open". So its back to my original advice. Keep most of the data out on disc and try to deal with manageable chunks. Say you split your files into alphabetical segments. By doing this you might find you get a speed increase. Word$() is a super command to use but on very long strings it can be slow. So loading segments from disc can be fast and using word$() on a smaller segment will speed it up also.

If you gave an overview of what the string handling process is, you might get better advice.

[jlthompson]

I cut down on the size of the strings and this did fix the problem.

I do actually keep most of the data on disc. Long as my program is, the vast majority of the data is stored in literally millions of lines' worth of files.

Running this program requires several computers working in parallel, connected through an ethernet hub, and they must constantly send a lot of data to each other. This does eat up a lot of memory, though. I suppose I should go through the program and dump the contents of variables once they have outlived their usefulness.

Thank you for looking into this all, and your advice.