System.IO.File.WriteAllLines exception - C# Discussion Boards

Re: Weird problem with Custom StreamReader with Read override not reading correct number of bytes during base method call

7-Nov-20 13:25

Is your point to help or to show how superior your knowledge of programming is? No it isn't a psychological limitation. It is a practical one. The file stream behind the reader is used as both an indexer to keep instance data separate in a static dictionary structure AND as a way to call required additional functionality on the stream. What you suggest would require me to change a significant amount of code to pass currently unnecessary data, making it more complex that it currently is since currently I can handle separate use cases with the same sections of code (that was the reason I used the current approach). Either way the solution is complicated so why not try and fix the problem first? Unless you are saying that the problem can't be fixed using an inherited FileStream because of its "complex" nature.

Gerry Schmitz8-Nov-20 5:53

Gerry Schmitz

8-Nov-20 5:53

An "adapter" pattern lets you manipulate the filestream; instead, you want to "act" like a filestream. Since you didn't write filestream, you really have less control.

You also outlined 2 distinct functions. So, filestream in fact, is now schizophrenic.

Anyway, your problem seems to be you "request" "count" bytes, read, then don't bother to check "result" (actual bytes read) against "count"; and then proceed to calculate a new offset using "count" (instead of result).

Not "superior"; I'm just not the coddling kind. Have a nice day.

It was only in wine that he laid down no limit for himself, but he did not allow himself to be confused by it.
― Confucian Analects: Rules of Confucius about his food

Re: Weird problem with Custom StreamReader with Read override not reading correct number of bytes during base method call

pr1mem0ver16-Nov-20 14:58

Re: Weird problem with Custom StreamReader with Read override not reading correct number of bytes during base method call

16-Nov-20 14:58

I don't need to be coddled lol. I just need something beyond vague references as I did not major in CS and there are a few gaps in my learning (mostly self taught). And yes... I DID check "result" which was reported as 4 bytes. So that is not the issue.

The issue is that for some reason it is filling the buffer and counting it as being read when it isn't supposed to. I HAVE checked into that and confirmed it. I reported the issue on MSDN forums as something for them to look into and was told that inheriting/overloading FileStream itself can be unreliable if mixing in other unrelated functionality. I was advised to inherit Stream instead. This is similar to your original suggestion but serves my other needs as well.

OriginalGriff8-Nov-20 1:35

OriginalGriff

8-Nov-20 1:35

We can't necessarily duplicate your problem from that code: we are lacking half the stuff that makes it work, such as the construction of your BinaryReader and it's FileStream.

So start by creating a minimum subset app which demonstrates the problem: uses the appropriate (cut-down) classes, dumps all the extraneous code, and shows the problem in as short a code as possible (but reflecting the structure you have in your existing code).
If that still shows the fault, show us the complete code so we can run it ourselves. If it doesn't, start looking at what you removed and put it back until it does ... then we can run it or it'll be fairly obvious what causes it.

Me? Without running it - and I can't - I'd guess that it's working but looks like it's failing for other reasons: if the return value is count then it's unlikely that the read is fetching considerably more data than that (somebody would have noticed by now). So you - and we - need a minimum app to show the problem so it can be looked at closely.

Sorry, but there is nothing I can do with those code fragments to prove your problem even exists!

"I have no idea what I did, but I'm taking full credit for it." - ThisOldTony
"Common sense is so rare these days, it should be classified as a super power" - Random T-shirt
AntiTwitter: @DalekDave is now a follower!

Re: Weird problem with Custom StreamReader with Read override not reading correct number of bytes during base method call

pr1mem0ver16-Nov-20 15:10

Re: Weird problem with Custom StreamReader with Read override not reading correct number of bytes during base method call

16-Nov-20 15:10

Yes... it is even hard for ME to duplicate as it doesn't happen in all circumstances. I have tried my class on other files successfully. The one in this particular debug was the only one giving me the issue at the time and there is nothing I can isolate. See my other recent posts if you are curious but I did solve the issue with a workaround while I change the class to inherit from Stream instead of FileStream for the final solution.

Richard Deeming8-Nov-20 21:39

Richard Deeming

8-Nov-20 21:39

By "several thousand", do you mean 4096?

The FileStream class buffers the file as it reads it. By default, it uses a 4Kb buffer.

FileStream Class (System.IO) | Microsoft Docs[^]:
FileStream buffers input and output for better performance.

You can pass a custom buffer size to the constructor if you want to override this behaviour. However, if you make the buffer too small, it could have a negative impact on performance.

"These people looked deep within my soul and assigned me a number based on the order in which I joined."
- Homer

Re: Weird problem with Custom StreamReader with Read override not reading correct number of bytes during base method call

pr1mem0ver16-Nov-20 15:06

System.IO.File.WriteAllLines exception

16-Nov-20 15:06

Yes. Exactly. I noticed that when I was debugging it fills the buffer and moves the Position to match the location that it WOULD be at if I was reading to the end of the buffer even though it returns that it only read 4 bytes and should have only advanced the position 4 bytes. It seems to be some behavioral bug. I posted the issue on MSDN and they said that inheriting FileStream for any complex situation such as mine is a bad idea and may result in strange behavior... I guess it did. I am now inheriting Stream instead.

_Flaviu5-Nov-20 23:52

_Flaviu

5-Nov-20 23:52

I have compiled successfully a C# console project. And when I run this app is stopping at the line:

System.IO.File.WriteAllLines(@destFile, elem);

where @destFile is

string destFile

and elem is

string[] elem = new string[...]

The error is:

System.UnauthorizedAccessException
  HResult=0x80070005
  Message=Access to the path 'e:\' is denied.
  Source=mscorlib
  StackTrace:
   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
   at System.IO.StreamWriter.CreateFile(String path, Boolean append, Boolean checkHost)
   at System.IO.StreamWriter..ctor(String path, Boolean append, Encoding encoding, Int32 bufferSize, Boolean checkHost)
   at System.IO.StreamWriter..ctor(String path, Boolean append, Encoding encoding)
   at System.IO.File.WriteAllLines(String path, String[] contents)

Of course, I have ran this app as administrator rights, with the same results. What should I understand from this error ? And how can I solve it ? My C# knowledge is thin ...

Richard MacCutchan6-Nov-20 0:01

Richard MacCutchan

6-Nov-20 0:01

You cannot write a file to "E:\"; that is not a valid file name.

OriginalGriff6-Nov-20 0:18

OriginalGriff

6-Nov-20 0:18

What do you expect @destFile to be?
You do not need an "@" in front of a variable name unless you are trying to use a keyword as a variable name - and if you are doing that, then you need to rethink your coding style. To call something public or string would require the "@" prefix, because they are both specific words that the compiler understands - but destFile is not, and does not need it.

int @public = 666;
Console.WriteLine(@public);
...
string destFile = "...";
Console.WriteLine(destFile);

And if you don't assign a value to destFile before you use it as an output location, you will get an error - it needs to contain a valid filename, in a folder that is not restricted for security reasons:

string destFile = "D:\Test Data\byList.csv";
File.WriteAllLines(destFile, elem);

So use the debugger to examine exactly what you have in destFile before you try to write to the file and check it's a fully valid filename!
If you don't know how to use it then a quick Google for "Visual Studio debugger" should give you the info you need.

Put a breakpoint on the first line in the function, and run your code through the debugger. Then look at your code, and at your data and work out what should happen manually. Then single step each line checking that what you expected to happen is exactly what did. When it isn't, that's when you have a problem, and you can back-track (or run it again and look more closely) to find out why.

Sorry, but we can't do that for you - time for you to learn a new (and very, very useful) skill: debugging!

_Flaviu6-Nov-20 2:33

_Flaviu

6-Nov-20 2:33

Thank you for detailed explanation, the code is not written by me, so I try to adapt it as much as I can.

Richard MacCutchan6-Nov-20 5:21

Richard MacCutchan

6-Nov-20 5:21

I don't know what happened to the reply you sent to me, so I reproduce it below:

Quote:
Believe me I have tried a lot of combination (e.g. "D:\\tempx\\" or "d:\\tempx"), the same result.

It seems to me you are confused between directory names and file names. You cannot use Write of any sort to write into a directory, you must use a proper file name. So something like "E:\\mytextfile.txt", "D:\\tempx\\mytextfile.txt", is what is needed.

How to extract part of HTML source?

_Flaviu7-Nov-20 21:08

_Flaviu

7-Nov-20 21:08

Yes, I figure out that and now I am trying to move on.

Member 5697394-Nov-20 7:53

Member 569739

4-Nov-20 7:53

I have a website that I am trying to extract some data from and then run some things on it. They do not seem to have any API's to do this. So take this site: abc

if I view the source and look down to line 71 at this time I basically want this piece:

JavaScript

<script data-react-helmet="true" type="application/ld+json">
                [{
                    "@context": "http://schema.org/",
                    "@type": "ItemList" ,
                    "itemlistElement":
                        [ [{
                                "@type": "ListItem",
                                "position" : 1,
                                "name" : "November 2020",
                                "item":  [{
                                    "@type": "Thing",
                                    "name" : "General Hospital 11/03/20",
                                    "url" : "www.abc.com/shows/general-hospital/episode-guide/2020-11/03-general-hospital-110320"
                                },{
                                    "@type": "Thing",
                                    "name" : "General Hospital 11/02/20",
                                    "url" : "www.abc.com/shows/general-hospital/episode-guide/2020-11/02-general-hospital-110220"
                                }]
                            }],[{
                                "@type": "ListItem",
                                "position" : 2,
                                "name" : "October 2020",
                                "item":  [{
                                    "@type": "Thing",
                                    "name" : "General Hospital 10/30/20",
                                    "url" : "www.abc.com/shows/general-hospital/episode-guide/2020-10/30-general-hospital-103020"
                                },{
                                    "@type": "Thing",
                                    "name" : "General Hospital 10/29/20",
                                    "url" : "www.abc.com/shows/general-hospital/episode-guide/2020-10/29-general-hospital-102920"
                                },{
                                    "@type": "Thing",
                                    "name" : "General Hospital 10/28/20",
                                    "url" : "www.abc.com/shows/general-hospital/episode-guide/2020-10/28-general-hospital-102820"
                                },{
                                    "@type": "Thing",
                                    "name" : "General Hospital 10/27/20",
                                    "url" : "www.abc.com/shows/general-hospital/episode-guide/2020-10/27-general-hospital-102720"
                                },{
                                    "@type": "Thing",
                                    "name" : "General Hospital 10/26/20",
                                    "url" : "www.abc.com/shows/general-hospital/episode-guide/2020-10/26-general-hospital-102620"
                                },{
                                    "@type": "Thing",
                                    "name" : "General Hospital 10/23/20",
                                    "url" : "www.abc.com/shows/general-hospital/episode-guide/2020-10/23-general-hospital-102320"
                                },{
                                    "@type": "Thing",
                                    "name" : "General Hospital 10/22/20",
                                    "url" : "www.abc.com/shows/general-hospital/episode-guide/2020-10/22-general-hospital-102220"
                                },{
                                    "@type": "Thing",
                                    "name" : "General Hospital 10/21/20",
                                    "url" : "www.abc.com/shows/general-hospital/episode-guide/2020-10/21-general-hospital-102120"
                                }]

So in theory I could grab the whole source into a file like:

string htmlContent = new System.Net.WebClient().DownloadString("https://abc.com/shows/general-hospital");

But then how do I filter out just the block shown or just the JSON piece so I can work with it?

Thanks.

JR

modified 4-Nov-20 14:11pm.

OriginalGriff4-Nov-20 8:37

OriginalGriff

4-Nov-20 8:37

Have a look at the HtmlAgilityPack: it makes scraping sites a whole load easier: Html Agility pack | Html Agility Pack[^]

For example, you can extract all the links from a page with one line of code:

foreach (HtmlNode link in doc.DocumentNode.SelectNodes("//a[@href]"))
   {
   ...
   }

It's pretty easy to use, and very powerful when you get used to it.

Member 5697394-Nov-20 9:12

Member 569739

4-Nov-20 9:12

Thanks for the reply. I believe I actually tried just that but the problem is they aren't href's, they are JSON in a script so I believe that returned nothing unless I'm missing it. So like this from it:

"url" : "www.abc.com/shows/general-hospital/episode-guide/2020-10/26-general-hospital-102620"

There is no href in it so it doesn't retrieve it.

Richard Andrew x644-Nov-20 14:11

Richard Andrew x64

4-Nov-20 14:11

It looks to me as if that library parses the HTML into a Document Object Model that you can navigate and extract any component parts from.

Have a look at the documentation. I'm sure there's a way to extract JSON, you just have to find it. Griff was only giving an example of how useful the library is.

The difficult we do right away...
...the impossible takes slightly longer.

DerekT-P4-Nov-20 23:29

DerekT-P

4-Nov-20 23:29

Bear in mind that a large proportion of websites have T+Cs that explicitly stop you doing that.

I used to do a lot of small web-scraping jobs for a variety of clients; first task was always scour the site T+Cs to check it was permitted.

When I was doing it a lot there weren't any 3rd party tools available, so it's just a case of pulling out the parts of the string you want. In your case I'd search for

Member 569739 wrote:
type="application/ld+json"

(either using a simple ".indexOf()" call or using RegEx), check how many such script blocks there were, and continue likewise to pull out the bits I need. Pretty basic string manipulation.

In this case, though, since the whole of what you want is JSON, just load it up using Newtonsoft.JSON and then access the objects directly:

JObject myJsonObject = JObject.Parse(myJsonText);

Richard Deeming4-Nov-20 23:36

Richard Deeming

4-Nov-20 23:36

The application/ld+json block is meant to be scraped by other sites - at the very least by search engines:
JSON-LD - JSON for Linking Data[^]

I think it's unlikely that the site would include that block if they didn't want anyone to be able to use it. Smile | :)

"These people looked deep within my soul and assigned me a number based on the order in which I joined."
- Homer

DerekT-P5-Nov-20 0:26

DerekT-P

5-Nov-20 0:26

"meant to be scraped" - yes.
By whom? Always check the T+Cs. Smile | :)