ASP.NET

Scrape with htmlagilitypack problems

17-Dec-19 6:07

You haven't configured your virtual directory as an application:
Understanding Sites, Applications, and Virtual Directories on IIS 7 | Microsoft Docs[^]

You'd have found that out quicker if you'd pasted the error message into Google. Smile | :)

"These people looked deep within my soul and assigned me a number based on the order in which I joined."
- Homer

Member 175325016-Dec-19 7:02

Member 1753250

16-Dec-19 7:02

Searched and worked hours trying to pull off/identify when a work/project is available on a web page. Below is the source of the page and my code. I can pull the tables, which are 7, but rows comes up as nothing. Want to pull when table id="workorders" and then = Project Name/work order

Thanks
Jim

Current Work Orders

No assigned work orders found.

-------if present, want to identity/capture this

My code
Get all tables in the document
Dim k As Integer
'
' Get all tables in the document
Dim tables As HtmlAgilityPack.HtmlNodeCollection = main.DocumentNode.SelectNodes("//table").innertext
' Iterate all rows in the first table

Dim rows As HtmlAgilityPack.HtmlNodeCollection = main.DocumentNode.SelectNodes("th")
' Iterate all rows in the first table
For k = 0 To rows.Count - 1
' Iterate all columns in this row
Dim cols As HtmlAgilityPack.HtmlNodeCollection = rows(k).SelectNodes("th")
If cols IsNot Nothing Then
For j As Integer = 0 To cols.Count - 1
' Get the value of the column and print it
Dim value As String = cols(j).InnerText
Console.WriteLine(value)
Next
End If
Next

Project Name/Work Order	Date	Time	Location	Tech	Pay Rate	Pay Type	Actions

Re: Scrape with htmlagilitypack problems

Richard Deeming16-Dec-19 9:41

Re: Scrape with htmlagilitypack problems

16-Dec-19 9:41

Member 1753250 wrote:

VB.NET

Dim rows As HtmlAgilityPack.HtmlNodeCollection = main.DocumentNode.SelectNodes("th")

You're iterating over all <th> nodes which are direct descendants of the document. There probably aren't any.

Instead, based on the comments, you probably want to iterate over the <tr> nodes which are direct descendants of the <table>:

VB.NET

Dim rows As HtmlAgilityPack.HtmlNodeCollection = tables(0).SelectNodes("tr")

However, this will depend on the markup. The rows might not be direct descendants of the <table>; they could be nested in one or more <thead>, <tbody>, or <tfoot> elements.

When you query for cells, you're also only looking for <th> elements. You're skipping ordinary <td> elements.

If you don't have to worry about nested tables, you could try:

VB.NET

Dim rows As HtmlAgilityPack.HtmlNodeCollection = main.DocumentNode.SelectNodes("//table[@id='workorders']//tr")
For i As Integer = 0 To rows.Count - 1
    Dim cols As HtmlAgilityPack.HtmlNodeCollection = rows(i).SelectNodes("th|td")
    ...
Next

"These people looked deep within my soul and assigned me a number based on the order in which I joined."
- Homer

Member 175325018-Dec-19 12:22

Member 1753250

18-Dec-19 12:22

Richard Deeming, you da Man!! Got it working afer spending hours on this. Thanks SO much.
Jim

ASP.Net MVC: How data is serialize into model when pass to client side from action

Mou_kol15-Dec-19 10:33

Mou_kol

15-Dec-19 10:33

I am sending list of data from action to client side. in client side i am binding jquery data table with pass data. column name is extracted because column is dynamic in my case. i got a below code which is not very clear how it is working.

My question is very basic question. see how list of data is pass from action to client side.

[HttpGet]
public ActionResult Index()
{

    List<UserData> data = new List<UserData>
    {
    new UserData
    {
    ID = 1,
    Name = "Simon"
    },
    new UserData
    {
    ID = 2,
    Name = "Alex"
    }
    };

    return View(data);
}

passed model data is captured like below way which is not clear to me

var keys = Object.keys(@Html.Raw(Json.Encode(Model[0])));
    var columns = keys.map(function (key) {
    return { title: key };
});

var data = @Html.Raw(Json.Encode(Model));
    var dataSet = data.map(function (d) {
    return Object.values(d);
});

console.log(columns);
console.log(dataSet);

see this code first

var keys = Object.keys(@Html.Raw(Json.Encode(Model[0])));
    var columns = keys.map(function (key) {
    return { title: key };
});

1) we know list is kind of array then why keys are extracted from zeroth position ? Model[0] why not Model[1]

i like to know in what form data is stored into model when pass from action to client side ?
see this code again

var data = @Html.Raw(Json.Encode(Model));
    var dataSet = data.map(function (d) {
    return Object.values(d);
});

2) in above code no ordinal position mention during extracting data. why?

3) what is Model[0] ? how it comes? because in asp.net mvc we write model like @model not Model

4) is this Model is Asp.Net MVC related model or javascript Model object ?

During keys extraction ordinal position is mentioned but in case of data extraction no ordinal position mentioned. please explain if some one understand what i am trying to know.

My main question is i have only one question that what is Model[0] here in js? is it any JavaScript related object or server side object ? if it is javascript then how server side model data comes into js model ?

please explain how above code is working at client side. thanks

Re: ASP.Net MVC: How data is serialize into model when pass to client side from action

Richard Deeming16-Dec-19 9:30

Re: ASP.Net MVC: How data is serialize into model when pass to client side from action

16-Dec-19 9:30

Both @Html.Raw(Json.Encode(Model[0])) and @Html.Raw(Json.Encode(Model)) are evaluated on the server. The Model refers to the server-side model of the view.

Not to be confused with the @model directive, which declares the type of the model for the view. @model declares the type; Model refers to the instance of that type.

@Html.Raw(Json.Encode(Model[0])) will generate a Javascript object representing the first item in the list. You specify the index because you only need the keys from one item. You don't use Model[1] in case there isn't a second item in the list.

For example, the rendered Javascript will look something like:

JavaScript

var keys = Object.keys({ "ID": 1, "Name": "Simon" });

NB: If the list is empty, you will get an exception when ASP.NET tries to evaluate the view to generate this code.

@Html.Raw(Json.Encode(Model)) will generate a Javascript array containing all of the items from the list. For example:

JavaScript

var data = [
    { "ID": 1, "Name": "Simon" },
    { "ID": 2, "Name": "Alex" }
];

"These people looked deep within my soul and assigned me a number based on the order in which I joined."
- Homer

Mou_kol24-Dec-19 22:14

Mou_kol

24-Dec-19 22:14

thanks a lot for your answer sir.

if i need to print at server side what is stored in Model or what is stored in first element of Model then what kind of code i need to write ?

i did it this way but getting error

[HttpGet]
        public ActionResult PrintModel()
        {
            List<UserData> data = new List<UserData>
                {
                    new UserData
                    {
                        ID = 1,
                        Name = "Simon"
                    },
                    new UserData
                    {
                        ID = 2,
                        Name = "Alex"
                    }
                };

            return View(data);
        }

in my razor i did it this way

@Html.Raw(Json.Encode(Model))

but above code giving error....how to print raw model ?
please guide. thanks

modified 25-Dec-19 4:01am.

Re: ASP.Net MVC: How data is serialize into model when pass to client side from action

Richard Deeming7-Jan-20 9:09

How do I change this code using Inheritance?

7-Jan-20 9:09

What error?

"These people looked deep within my soul and assigned me a number based on the order in which I joined."
- Homer

Marc Hede12-Dec-19 22:39

12-Dec-19 22:39

I have finally gotten my code to work, but there is one thing left that I want to do.
In my code I had made a Keycard class that had a name and a keynumber.
What I would like to do is to change the Keycard class to Person Class, which keeps the Name string, and then add the Keycard class as a child to the Person class, which keeps the Mykey int.

I have done inheritance before, but I have not tried doing it in a file that also uses Serialization.
How should the code look?

Also, a small bonus question. As you can see, my code gives a binary output. Like with XML, is there a way to show this binary output on a web application?

using System;
using System.Collections.Generic;
using System.Linq;
using System.Web;
using System.Runtime.Serialization;
using System.Runtime.Serialization.Formatters.Binary;

namespace Keycard
{
    [Serializable()]
    public class Keycard : ISerializable
    {
        protected string name;
        protected int mykey;

        public Keycard(string name, int mykey)
        {
            this.Name = name;
            this.Mykey = mykey;
        }

        public string Name
        {
            get { return name; }
            set { name = value; }
        }

        public int Mykey
        {
            get { return mykey; }
            set { mykey = value; }
        }

        public void GetObjectData(SerializationInfo info, StreamingContext context)
        {

            info.AddValue("Name", name);
            info.AddValue("Keynumber", mykey);

        }
        public Keycard(SerializationInfo info, StreamingContext context)
        {
            Name = (string)info.GetValue("Name", typeof(string));
            Mykey = (int)info.GetValue("Keynumber", typeof(int));
        }

        public override string ToString()
        {
            return "Name: " + name + " ---- " + " Keynumber: " + mykey;
        }
    }
}

using System;
using System.Collections;
using System.Collections.Generic;
using System.Linq;
using System.Xml.Serialization;
using System.IO;
using System.Runtime.Serialization.Formatters.Binary;

namespace Keycard
{
    class Class1
    {
        public static void Main(string[] args)
        {
            Keycard d1 = new Keycard("John", 102030);

            Stream stream = File.Open("KeycardData.dat",
                FileMode.Create);

            BinaryFormatter bf = new BinaryFormatter();

            bf.Serialize(stream, d1);
            stream.Close();

            d1 = null;

            stream = File.Open("KeycardData.dat", FileMode.Open);

            bf = new BinaryFormatter();

            d1 = (Keycard)bf.Deserialize(stream);
            stream.Close();
            Console.WriteLine(d1.ToString());
            Console.ReadLine();

        }
    }
}

Richard MacCutchan12-Dec-19 23:15

Richard MacCutchan

12-Dec-19 23:15

Creating a Person class is easy. Just change the name of your Keycard class and add the remaining details that a Person object needs. If the Keycard class only has a single property then it becomes redundant. Just leave the Keycard value as a property of the Person class.

Marc Hede wrote:
Like with XML, is there a way to show this binary output on a web application?

I am not sure what you mean here. If you want to display the value somewhere then just use its ToString method.

phil.o12-Dec-19 23:23

phil.o

12-Dec-19 23:23

You can do it (creating a person class then a keycard class inheriting from person class), that is not hard at all. Give it a try. For serialization, you just have to mark both classes with Serializable attribute. Schematically:

[Serializable]
public class Person
{
   // your implementation here
}

[Serializable]
public class KeyCard : Person
{
   // your implementation here
}

Regarding the serialization format, you could use a XmlFormatter instead of a BinaryFormatter. Or a XmlSerializer which allows to customize the formatting process even further. Finally, you could also opt in for a JSON serialization.
Showing a binary output in a web application would not be very usefull: do you know anyone willing to read binary format and make sense out of it? Smile | :)

Cheers.

"Five fruits and vegetables a day? What a joke!
Personally, after the third watermelon, I'm full."

Marc Hede12-Dec-19 23:57

12-Dec-19 23:57

Yeah, that is also how I thought I should do it.
But I am getting a ton of errors when doing so.

"Person does not contain a constructor that takes 0 arguements"
"mykey does not exist within current context"
"method must have a return type"

This is how I wrote it

using System;
using System.Collections.Generic;
using System.Linq;
using System.Web;
using System.Runtime.Serialization;
using System.Runtime.Serialization.Formatters.Binary;

namespace Keycard
{
    [Serializable()]
    public class Person : ISerializable
    {
        protected string name;

        public Person(string name)
        {
            this.Name = name;
        }

        public string Name
        {
            get { return name; }
            set { name = value; }
        }

        [Serializable()]
        public class Keycard : Person
        {
            protected int mykey;

            public Keycard(int mykey)
            {
                this.Mykey = mykey;
            }
            public int Mykey
            {
                get { return mykey; }
                set { mykey = value; }
            }
        }

        public void GetObjectData(SerializationInfo info, StreamingContext context)
        {

            info.AddValue("Name", name);
            info.AddValue("Keynumber", mykey);

        }
        public Person(SerializationInfo info, StreamingContext context)
        {
            Name = (string)info.GetValue("Name", typeof(string));
        }
        public Keycard(SerializationInfo info, StreamingContext context)
        {
            Mykey = (int)info.GetValue("Keynumber", typeof(int));
        }

        public override string ToString()
        {
            return "Name: " + name + " ---- " + " Keynumber: " + mykey;
        }
    }
}

phil.o13-Dec-19 0:26

phil.o

13-Dec-19 0:26

Marc Hede wrote:
"Person does not contain a constructor that takes 0 arguements"

The Person class has a single constructor accepting a string value as argument. Defining a constructor also means that the class does not have any default parameterless constructor. Since the KeyCard class inherits from Person, it has to provide its base class a name argument (so that the base class can be constructed).
There are several solutions to this:

you can define a parameterless constructor in the base class.
you can delete defined constructor in base class (so that compiler generate a default, parameterless one); but this "solution" is not really clever.
you can provide a default name argument in KeyCard's constructor.

In clear:

#region Solution 1 : Defining a parameterless constructor

public class Person
{
   // ...
   public Person() : this("unnamed") { }
}

#endregion

#region Solution 3 : providing a default name in inherited classs' constructor

public class KeyCard : Person
{
   // ...
   public KeyCard(int mykey) : base("unnamed")
   {
      // ...
   }

   // and/or

   public KeyCard(string name) : base(name)
   {
      this.Mykey = -1;
   }

   public KeyCard(string name, int mykey) : base(name)
   {
      this.Mykey = mykey;
   }
}

#endregion

Marc Hede wrote:
"mykey does not exist within current context"

The compiler does not how to differentiate the mykey parameter of the constructor from the protected mykey field. You can either rename one of them, or just define an auto-implemented property and get rid of the protected field:

public int Mykey { get; set; }

Marc Hede wrote:
"method must have a return type"

Probably because you defined the KeyCard class inside the Person class, and then defined a KeyCard constructor inside Person (which is incorrect). Please get the KeyCard class out of the Person (you do not need to nest inheriting classes), and put constructors and methods in the class to which they properly belong.
Good work Smile | :)

"Five fruits and vegetables a day? What a joke!
Personally, after the third watermelon, I'm full."

modified 13-Dec-19 5:32am.

Marc Hede13-Dec-19 0:49

13-Dec-19 0:49

Hey Phil.
Thanks a lot
I almost got it to work. Only one error left thankfully.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Web;
using System.Runtime.Serialization;
using System.Runtime.Serialization.Formatters.Binary;

namespace Keycard
{
    [Serializable()]
    public class Person : ISerializable
    {
        protected string name;

        public Person(string name)
        {
            this.Name = name;
        }

        public string Name
        {
            get { return name; }
            set { name = value; }
        }

        public void GetObjectData(SerializationInfo info, StreamingContext context)
        {

            info.AddValue("Name", name);

        }
        public Person(SerializationInfo info, StreamingContext context)
        {
            Name = (string)info.GetValue("Name", typeof(string));
        }
    }
    [Serializable()]
    public class Keycard : Person
    {
        protected int mykey;

        public Keycard(string name)
            : base (name)
        {
            this.Mykey = -1;
        }
        public Keycard(string name, int mykey) : base(name)
        {
            this.Mykey = mykey;
        }
        public int Mykey
        {
            get { return mykey;  }
            set { mykey = value; }
        }

        public new void GetObjectData(SerializationInfo info, StreamingContext context)
        {

            info.AddValue("Keynumber", mykey);

        }
        public Keycard(SerializationInfo info, StreamingContext context)
        {
           Mykey = (int)info.GetValue("Keynumber", typeof(int));
        }
        public override string ToString()
        {
            return "Name: " + name + " ---- " + " Keynumber: " + mykey;
        }
    }
}

The only problem here is this line

public Keycard(SerializationInfo info, StreamingContext context)

I get the constructor error again.
I understood your solution with the default name arguement, but how do I apply that logic here?

F-ES Sitecore13-Dec-19 0:59

F-ES Sitecore

13-Dec-19 0:59

public Keycard(SerializationInfo info, StreamingContext context)
    : base ("unnamed")
{
    Mykey = (int)info.GetValue("Keynumber", typeof(int));
}

Marc Hede13-Dec-19 1:17

13-Dec-19 1:17

Ah of course.
Sometimes it surprises me how simple the solution is. I tend to overthink things Big Grin | :-D

The only issue now is that I get a

System.Runtime.Serialization.SerializationException

Mykey = (int)info.GetValue("Keynumber", typeof(int));

Do I need to add Keynumber in my other class file?

using System;
using System.Collections;
using System.Collections.Generic;
using System.Linq;
using System.Xml.Serialization;
using System.IO;
using System.Runtime.Serialization.Formatters.Binary;

namespace Keycard
{
    class Class1
    {
        public static void Main(string[] args)
        {
            Keycard k1 = new Keycard("John", 123);

            Stream stream = File.Open("KeycardData.dat",
                FileMode.Create);

            BinaryFormatter bf = new BinaryFormatter();

            bf.Serialize(stream, k1);
            stream.Close();

            k1 = null;

            stream = File.Open("KeycardData.dat", FileMode.Open);

            bf = new BinaryFormatter();

            k1 = (Keycard)bf.Deserialize(stream);
            stream.Close();
            Console.WriteLine(k1.ToString());
            Console.ReadLine();
        }

    }
}

phil.o13-Dec-19 1:52

phil.o

13-Dec-19 1:52

Technically, you do not need to provide any customized serialization code for an integer member variable; the Serializable attribute on its own will do just fine.

"Five fruits and vegetables a day? What a joke!
Personally, after the third watermelon, I'm full."

F-ES Sitecore13-Dec-19 1:54

F-ES Sitecore

13-Dec-19 1:54

Ok, I understands what you're doing a bit better now. The GetObjectData method on Person is

public void GetObjectData(SerializationInfo info, StreamingContext context)
{

    info.AddValue("Name", name);

}

and on Keycard it is

public new void GetObjectData(SerializationInfo info, StreamingContext context)
{
    info.AddValue("Keynumber", mykey);

}

The base class (Person) is a normal method and the inherited class (Keycard) is marked "new" which means you're breaking the inheritance chain and saying that the two methods may have the same name but they are their own thing. When you serialise your data as Person is the class that is ISerializable that is the class your GetObjectData is called on, the version on Keycard is never called. What you need to do is mark the version on Person as virtual and the one on Keycard as override. That means that even though GetObjectData is being called on Person, the method on Keycard will be called instead as that version overrides the one on Person. Now your keycard version is being called you can then call the version on Person yourself.

Person;

public virtual void GetObjectData(SerializationInfo info, StreamingContext context)
{
    info.AddValue("Name", name);
}

Keycard;

public override void GetObjectData(SerializationInfo info, StreamingContext context)
{
    base.GetObjectData(info, context);
    info.AddValue("Keynumber", mykey);
}

Now when serialisation happens both GetObjectData methods will be called.

Your next problem is that when you deserialise Keycard the constructor called is

public Keycard(SerializationInfo info, StreamingContext context)
    : base ("unnamed")
{
    Mykey = (int)info.GetValue("Keynumber", typeof(int));
}

The problem here (admittedly I did say to code that but I didn't fully understand what you're trying to do) is that your Keycard constructor is called the non-ISerializable constructor on Person;

public Person(string name)
{
    this.Name = name;
}

So the keycard constructor is reading the Mykey value from the SerializationInfo object, but the Person constructor is hard-coding the name to "unnamed", it isn't reading from SerializationInfo. So you need to call the ISerializable constructor on Person instead;

public Keycard(SerializationInfo info, StreamingContext context)
    : base (info, context)
{
    Mykey = (int)info.GetValue("Keynumber", typeof(int));
}

Now both the Keycard and Person parts of your object will initialise themselves from the serialization info.

Richard Deeming13-Dec-19 1:58

13-Dec-19 1:58

Make Person.GetObjectData a virtual method. Change Keycard.GetObjectData to override the method instead of shadowing the method, and have it call the base method.

[Serializable()]
public class Person : ISerializable
{
    ...
    public virtual void GetObjectData(SerializationInfo info, StreamingContext context)
    {
        info.AddValue("Name", name);
    }
    ...
}

[Serializable()]
public class Keycard : Person
{
    ...
    public override void GetObjectData(SerializationInfo info, StreamingContext context)
    {
        base.GetObjectData(info, context);
        info.AddValue("Keynumber", mykey);
    }
    ...
}

Your code will now work.

However, you're creating Stream objects, which should be disposed of when you're finished with them. The simplest way to do that is with a using block[^].

Keycard k1 = new Keycard("John", 123);
BinaryFormatter bf = new BinaryFormatter();

using (Stream stream = File.Open("KeycardData.dat", FileMode.Create))
{
    bf.Serialize(stream, k1);
}

k1 = null;

using (Stream stream = File.Open("KeycardData.dat", FileMode.Open))
{
    k1 = (Keycard)bf.Deserialize(stream);
}

Console.WriteLine(k1);
Console.ReadLine();

"These people looked deep within my soul and assigned me a number based on the order in which I joined."
- Homer