|
I have a website that I am trying to extract some data from and then run some things on it. They do not seem to have any API's to do this. So take this site: abc
if I view the source and look down to line 71 at this time I basically want this piece:
<script data-react-helmet="true" type="application/ld+json">
[{
"@context": "http://schema.org/",
"@type": "ItemList" ,
"itemlistElement":
[ [{
"@type": "ListItem",
"position" : 1,
"name" : "November 2020",
"item": [{
"@type": "Thing",
"name" : "General Hospital 11/03/20",
"url" : "www.abc.com/shows/general-hospital/episode-guide/2020-11/03-general-hospital-110320"
},{
"@type": "Thing",
"name" : "General Hospital 11/02/20",
"url" : "www.abc.com/shows/general-hospital/episode-guide/2020-11/02-general-hospital-110220"
}]
}],[{
"@type": "ListItem",
"position" : 2,
"name" : "October 2020",
"item": [{
"@type": "Thing",
"name" : "General Hospital 10/30/20",
"url" : "www.abc.com/shows/general-hospital/episode-guide/2020-10/30-general-hospital-103020"
},{
"@type": "Thing",
"name" : "General Hospital 10/29/20",
"url" : "www.abc.com/shows/general-hospital/episode-guide/2020-10/29-general-hospital-102920"
},{
"@type": "Thing",
"name" : "General Hospital 10/28/20",
"url" : "www.abc.com/shows/general-hospital/episode-guide/2020-10/28-general-hospital-102820"
},{
"@type": "Thing",
"name" : "General Hospital 10/27/20",
"url" : "www.abc.com/shows/general-hospital/episode-guide/2020-10/27-general-hospital-102720"
},{
"@type": "Thing",
"name" : "General Hospital 10/26/20",
"url" : "www.abc.com/shows/general-hospital/episode-guide/2020-10/26-general-hospital-102620"
},{
"@type": "Thing",
"name" : "General Hospital 10/23/20",
"url" : "www.abc.com/shows/general-hospital/episode-guide/2020-10/23-general-hospital-102320"
},{
"@type": "Thing",
"name" : "General Hospital 10/22/20",
"url" : "www.abc.com/shows/general-hospital/episode-guide/2020-10/22-general-hospital-102220"
},{
"@type": "Thing",
"name" : "General Hospital 10/21/20",
"url" : "www.abc.com/shows/general-hospital/episode-guide/2020-10/21-general-hospital-102120"
}]
So in theory I could grab the whole source into a file like:
string htmlContent = new System.Net.WebClient().DownloadString("https://abc.com/shows/general-hospital");
But then how do I filter out just the block shown or just the JSON piece so I can work with it?
Thanks.
JR
modified 4-Nov-20 14:11pm.
|
|
|
|
|
Have a look at the HtmlAgilityPack: it makes scraping sites a whole load easier: Html Agility pack | Html Agility Pack[^]
For example, you can extract all the links from a page with one line of code:
foreach (HtmlNode link in doc.DocumentNode.SelectNodes("//a[@href]"))
{
...
}It's pretty easy to use, and very powerful when you get used to it.
"I have no idea what I did, but I'm taking full credit for it." - ThisOldTony
"Common sense is so rare these days, it should be classified as a super power" - Random T-shirt
AntiTwitter: @DalekDave is now a follower!
|
|
|
|
|
Thanks for the reply. I believe I actually tried just that but the problem is they aren't href's, they are JSON in a script so I believe that returned nothing unless I'm missing it. So like this from it:
"url" : "www.abc.com/shows/general-hospital/episode-guide/2020-10/26-general-hospital-102620"
There is no href in it so it doesn't retrieve it.
|
|
|
|
|
It looks to me as if that library parses the HTML into a Document Object Model that you can navigate and extract any component parts from.
Have a look at the documentation. I'm sure there's a way to extract JSON, you just have to find it. Griff was only giving an example of how useful the library is.
The difficult we do right away...
...the impossible takes slightly longer.
|
|
|
|
|
Bear in mind that a large proportion of websites have T+Cs that explicitly stop you doing that.
I used to do a lot of small web-scraping jobs for a variety of clients; first task was always scour the site T+Cs to check it was permitted.
When I was doing it a lot there weren't any 3rd party tools available, so it's just a case of pulling out the parts of the string you want. In your case I'd search for Member 569739 wrote: type="application/ld+json"
(either using a simple ".indexOf()" call or using RegEx), check how many such script blocks there were, and continue likewise to pull out the bits I need. Pretty basic string manipulation.
In this case, though, since the whole of what you want is JSON, just load it up using Newtonsoft.JSON and then access the objects directly:
JObject myJsonObject = JObject.Parse(myJsonText);
|
|
|
|
|
The application/ld+json block is meant to be scraped by other sites - at the very least by search engines:
JSON-LD - JSON for Linking Data[^]
I think it's unlikely that the site would include that block if they didn't want anyone to be able to use it. 
"These people looked deep within my soul and assigned me a number based on the order in which I joined."
- Homer
|
|
|
|
|
"meant to be scraped" - yes.
By whom? Always check the T+Cs. 
|
|
|
|
|
Thanks folks. yeah it gets more complicated by the fact that there are multiple of those same blocks other than right now the one I want is the third one. That said, after hours of digging, I did find API's that return JSON for this so coding around that as obviously it's going to be a lot cleaner and more reliable. Haven't run the code yet but fingers crossed this works better. Not sure why ABC hides references to their API's like they do.
|
|
|
|
|
Hi friends,
I want to switch the whole DataGridView Left-To-Right so that the first column is in the right side, like what you see in Excel.
How is it possible?
Please Guide me.
Thanks.
|
|
|
|
|
Alex Dunlop wrote: first column is in the right side, like what you see in Excel.
I assume you are dealing with binding a DataSet to a DataGridView where you have no control over the Column order.
0) if you set the RightToLeft property of the Form, the DataGridView will show reverse column order.
1) if you set the RightToLeft property of the DataGridView, you reverse the column order.
2) if you have set the AllowUserToOrderColumns to true, the run time user can reorder columns by dragging.
3) you can change column order in code using the DisplayIndex property of the column [^].
Code like this will work, but, you must execute it either after the form calls InitializeComponent(); ... or, in the Shown event of the form:
DataGridViewColumnCollection columns = dataGridView1.Columns;
int nColumns = columns.Count - 1;
for (int i = 0; i <= nColumns; i++)
{
columns[i].DisplayIndex = nColumns - i;
}
«One day it will have to be officially admitted that what we have christened reality is an even greater illusion than the world of dreams.» Salvador Dali
|
|
|
|
|
And last of all you can reconstruct your datasource to match the column order you require and bind that you the DGV.
Never underestimate the power of human stupidity -
RAH
I'm old. I know stuff - JSOP
|
|
|
|
|
Hello All,
I am tasked with performing researching of moving on-premise .net websites(4 different websites) to cloud.
I have to perform comparison of Azure, AWS, Google Cloud Service. Is there any other solutions?
What would be the best way to perform this research and document side-by-side pros and cons?
Thank you!!
|
|
|
|
|
I think the best way would be to learn as much as you can about each cloud provider and compare their offerings.
Don't forget the IBM cloud. That leverages their famous Watson technology.
The difficult we do right away...
...the impossible takes slightly longer.
|
|
|
|
|
Hi!
I have a DataGridView in a Windows Form application where I want to localize the columns in the DataGridView. I use the DataGridView 's DataSource property to populate the DataGridView from a list. Therefore the columns in the DataGridView are genereated automatically after the properies in the list.
Here is the code to populate the DataGridView.
public partial class Form1 : Form
{
public Form1()
{
InitializeComponent();
var p1 = new Person() { Name = "Foo", Age = 1337 };
var p2 = new Person() { Name = "Bar", Age = 42 };
var people = new List<Person>() { p1, p2 };
dataGridView1.DataSource = people;
}
}
public class Person
{
public string Name { get; set; }
public int Age { get; set; }
}The generated columns in the DataGridView are Name and Age, the same as the properties in the Person class.
I have set the Form's Localizable property to True.
With a Label in the Form, I have verified that I can change the Label's Text property to different values based on the CurrentCulture / CurrentUICulture I set. I switch between different cultures in the Program.cs file:
static class Program
{
[STAThread]
static void Main()
{
var eng = "en";
var swe = "sv-SE";
var culture = eng;
Thread.CurrentThread.CurrentCulture = new System.Globalization.CultureInfo(culture);
Thread.CurrentThread.CurrentUICulture = new System.Globalization.CultureInfo(culture);
Application.EnableVisualStyles();
Application.SetCompatibleTextRenderingDefault(false);
Application.Run(new Form1());
}
}My project have generated Form1.resx and Form1.sv.resx files to handle both English and Swedish. In these files I can find the label's text for the English and Swedish languages accordingly.
I just don't know how to set the DataGridView's column to different languages. Since the columns are not manually created by me I don't have the appropriate values in the Form1.resx and Form1.sv.resx to set the columns based on the selected language.
Is there any solution to this?
|
|
|
|
|
Because of the way DGV columns are created, you are going to have to iterate over them to localize them: [^].
Look at this excerpt from Hans Passant's code in the link:
foreach (DataGridViewColumn col in dgv.Columns) {
resources.ApplyResources(col, col.Name);
}
«One day it will have to be officially admitted that what we have christened reality is an even greater illusion than the world of dreams.» Salvador Dali
|
|
|
|
|
Hello,
I have a simple Arduino robot that communicates through a Bluetooth device (HC-05).
First I made an application in Microsoft Visual C# 2008 that continuously could receive data from the HC-05.
In this Csharp application I just have to place the serialPort1 object and use the serialPort1_DataReceived event. This is working great ! Each time my robot sends three bytes of data, my application successfully receives this.
Now I have Microsoft Visual Studio Professional 2019 and it seems that I could not use the serialPort1 any more...
So I looked on internet and I found a solution with the package "In The Hand Bluetooth from 32feet.NET".
I installed and used successfully this package. Now I can sent and receive data with the HC-05 device.
However, I can only let this work on the event of a button...
So the big problem that I have now is that I cannot receive continuously data like my old application.
I was trying to make a thread that continuously reads data but it doesn't work:
<pre>
FormManager.BluetoothStream = FormManager.bluetoothClient.GetStream();
try
{
Array.Clear(bufferBluetooth, 0, bufferBluetooth.Length);
do
{
FormManager.BluetoothStream.Read(bufferBluetooth, 0, bufferBluetooth.Length);
}
while (FormManager.BluetoothStream.DataAvailable);
System.Diagnostics.Debug.WriteLine("--> bytes received " + bufferBluetooth[0] + " " + bufferBluetooth[1] + " " + bufferBluetooth[2]);
}
catch
{
System.Diagnostics.Debug.WriteLine("Receiving Error");
}
|
|
|
|
|
It looks like you are expecting a continuous stream of data coming in. As soon as your application has caught up (or if there is no data initially to read), your do...while loop will exit and nothing will then read further incoming data. It is improbable that your bluetooth device will be sending data continuously fast enough to satisfy your loop. In comparison, your previous version, being event driven, could wait for further data. You need to see if there is any event driven equivalents in the new package or if there is any way of suspending until new data arrives.
Note: I know nothing about Bluetooth or about the old or new packages that you have been using, so my thoughts could be wrong!
|
|
|
|
|
Hello,
I got it working by using the read and try command.
And this in a loop that I can put later into a thread:
byte[] buffer = new byte[3];
Stream mStream = FormManager.bluetoothClient.GetStream();
while (FormManager.bluetoothClient.Connected)
{
try
{
mStream.Read(buffer, 0, buffer.Length);
System.Diagnostics.Debug.WriteLine("read status "+buffer[0] + " " + buffer[1] + " " + buffer[2] + " --> Status " + FinalResultBT);
}
catch
{
System.Diagnostics.Debug.WriteLine("read status fail ");
}
}
|
|
|
|
|
I'm using a typical 'n-tier' design. In most cases the BL just passes through called to and from the UI and DAL. But I do have some business rules that I'd like to check in the BL. I want to put the code there, versus in the UI because I have a WPF app and a Mobile app both calling the Web API.
So, the question is, how would you handle a rule violation in the BL? Would you throw an exception? Or return a string or other value?
Thanks
If it's not broken, fix it until it is.
Everything makes sense in someone's mind.
Ya can't fix stupid.
|
|
|
|
|
|
Richard Deeming wrote: Return an error status code
Thanks. I was leaning in that direction but wanted to get opinions.
I also posted a question in Arch & Design. I would welcome your thouhts on that.
K
If it's not broken, fix it until it is.
Everything makes sense in someone's mind.
Ya can't fix stupid.
|
|
|
|
|
HI im new to C#
im creating an test app to connect to a local Database, i have created a form which connects to my DB and server, once connected this opens a new form. So far so good. what i need to know now is how do i use the connection in the new form?
thanks
|
|
|
|
|
Exactly how depends on the "relationship" between the two forms.
Have a look at these, one of them will fit your circumstances.
The form that creates an instance of another:
MyForm mf = new MyForm();
mf.Show(); Is the "parent", the other form is the "child".
(This doesn't imply any formal MDI relationship)
Transferring information between two forms, Part 1: Parent to Child[^]
Transferring information between two forms, Part 2: Child to Parent[^]
Transferring information between two forms, Part 3: Child to Child[^]
But ... you shouldn't create a SqlConnection and "pass it around" your app - that's a bad idea as some operations such as a DataReader that fails to be closed and disposed properly can "block" the connection object from doing anything else. Instead, create a connection string and pass that around (or create it as a public static property and just use it everywhere). Then create your Connection within a using block each time you need one.
"I have no idea what I did, but I'm taking full credit for it." - ThisOldTony
"Common sense is so rare these days, it should be classified as a super power" - Random T-shirt
AntiTwitter: @DalekDave is now a follower!
|
|
|
|
|
Thanks for your reply i will take look at the parent child
this is my first app, advancing from hello world . passing the connection string between forms is what i am trying to do. I am a complete Novice.
thanks
|
|
|
|
|
You don't have to "pass" anything. The connection string can be made available to any methods that want it by exposing it as a getter-only property in a static class.
Seriously, if you're moving up from Hello World to this, you may have taken your steps a bit too long.
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
