Click here to Skip to main content
15,885,546 members
Please Sign up or sign in to vote.
0.00/5 (No votes)
See more:
We have an old old ASP web page that reads multiple text files from disk and processes them.

It's incredibly slow, of course, and crashes almost daily.

Currently, this is the code:
For i = 0 To tNumberSites

  Set TStream = fs.OpenTextFile(SiteName)
  FileOpen = True

  Do While (TStream.AtEndOfStream <> True)
    tLine = TStream.readLine

[removed tons of processing code here that is executed for each line of each file]

LOOP


As you can probably guess, efficiency is nowhere to be seen, and the crashes are probably due to over-taxing the server more than anything else.

What I'd like to do is read all the files into one big TextStream and just do the processing once. Hopefully, it will speed things up a bit.

But FileStreamObject and TextStream don't seem to have a native Append method for doing this.

What would be my best plan of attack?

Read a file into a stream, loop through it and write the lines to a another stream (call it BigStream for now), then read the second file, loop through it and continue writing to BigStream, rinse and repeat, then run the processing against BigStream when all the files have been loaded?

Or is there an easier way to do an in-memory Append that I can't find anywhere?

NOTE: I am NOT an ASP/VBScript guy, and none exist here anymore. And NO, we do NOT have a test environment, so any code I try is going to be live. (Yay! Such fun)

What I have tried:

As I said above, there is no test/development environment for this, so all my tries have been mental exercises. The only thing I can think of is the process I outlined above, which should work, but I'm only 97% sure of it.

I'm hoping someone has a better (and hopefully bulletproof) idea. I don't need code. Despite being a non VB kind of guy, I can figure out what to do if given enough of a hint. Or pseudocode. Unless it's in BNF, in which case I will take to drink.
Posted
Comments
njammy 2-Sep-16 11:50am    
Are the files in any particular format?
GenJerDan 2-Sep-16 12:32pm    
Caret-delimited text, like^this.
njammy 2-Sep-16 12:42pm    
Without knowing your tool set skills set etc I'd suggest using DTS package or SSIS package to load txt file data to a database table then query only necessary data to build the output or processing. Much more efficient and parallel mechanics.
GenJerDan 2-Sep-16 12:45pm    
It's in the pipeline; we just don't have the people to get everything done that needs doing in a timely manner. :(
Richard Deeming 2-Sep-16 12:25pm    
I doubt combining the files into one giant file would make any significant difference, unless your issue relates to file locking.

What's the error you get when the code crashes?

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)



CodeProject, 20 Bay Street, 11th Floor Toronto, Ontario, Canada M5J 2N8 +1 (416) 849-8900