Comments by Eugene Efimov

Eugene Efimov 22-May-13 9:17am View

> but generally what you describe is a working scenario
Do you have experience working with sockets? Cuz if so, you know there is no way to know the size of incoming packages. And the scenario I described not just "generally working", but a valid and preferable approach (instead of fixing package size and read fix length pachages, what you suggest for in analogy to block-devices).

Eugene Efimov 22-May-13 9:13am View

If you claim the stream is "UNRELIABLE", how can you rely on it to download file? What's the difference between downloading file to local file or to memory for on-the-fly analysis?

Eugene Efimov 21-May-13 10:06am View

>Doing it line-by-line is somewhat unpredictable, because you don't know the line size before you read, so it's generally problematic and bad for performance. I would not do it at all.
Why is this unpredictable? We need to receive a chunk of data, write it to buffer, scan buffer for newline separators, extract fully-received lines, yield them and remove them from buffer.

In fact, that is that StreamReader does as far I can see. The code above works fine in most cases, and it actually gets all content. Except for throwing exceptions some times in the end of file instead of returning null.

My current workaround is just to ignore ObjectDisposedException if it occurs in ReadLine(). And it works fine, processing date on the fly with unnecessary caching data on local drive. I'm just curious about the reasons of occasionally occuring ObjectDisposedException exceptions.

Eugene Efimov 21-May-13 9:54am View

I did not mean storing all data in memory at once. I get your point about temporary storing data in file.

But we already have stream to read from! I see no real reason to write that stream to file, and then create another stream (only now its from file not network). We can read and process data as it comes, on the fly! The problem is that I may be doing I am doing it wrong, but let me emphasize again - on the fly processing is the right way to do it. Using temporary storage must be justified with more than "This way it works and we have no idea about what is wrong with the previous code".

Eugene Efimov 21-May-13 9:43am View

Please, check again. All brackets are balanced and code complies fine. The first 'using' statement has no brackets at all, see nested using example here: http://stackoverflow.com/questions/1329739/nested-using-statements-in-c-sharp

Downloading all data at once to a single byte array could be painful for big files, besides question clearly states that line by line processing required, so no need to store whole data at once.

P.S. Current example does not contain part of processing streamReader.ReadLine() non-null result just to make example smaller.

Eugene Efimov 21-May-13 7:34am View

Could you be more specific about why not the whole file is read? Point out some code, cuz I expect ReadLine method in the loop to read all data.

And how can I get snapshot? The file can be modified during download process. I just can't get why should one download file content to memory, then flush it on disk and then again load to memory.

It may be a workaround for this particular task, but the true reason of the problem is still hidden.

Eugene Efimov 17-May-13 4:36am View

> while (true)
and
> if (streamReader.ReadLine() == null) break;
Will assure reading stream until the end (and until the end of underlying stream). In fact, that is what happens in my case - program reads file until the end, but then instead of returning null in ReadLine() it throws exception.

Eugene Efimov 16-May-13 4:42am View

I do transferring the whole file, do i not?

Eugene Efimov 15-May-13 2:58am View

I need to analyze text log located on a remote server accessible only via FTP. Text log structure requires analyzing it line by line. Is there a better way?

Comments by Eugene Efimov (Top 9 by date)