|
All,
I am looking for a code which retrieves a HTML source for a given website. I got few but none of the code really works for all websites. Some has security issues, some has timeout issues, some has redirect issues, some has session management issues. I am looking for the code which works for all kinds of websites.
given a website say www.xyz.com/default.aspx I want to retrieve the HTML source(same as when this page get rendered). I want to achieve this without any UI.
Does anyone knows where I can get one?
Thanks,
Anand
|
|
|
|
|
Unless you have knowledge of the specific site, you're always going to run into problems with a secured site or one with complex session management. Implementations will differ from site to site, making a one-size-fits-all component very difficult.
For example, lets assume that have valid credentials to log into a secured site, how does that site manage them? Do you need to pass them in the form of a cookie, or do you have to post to a specific page? What happens if the site changes their implementation details?
Creating a component that could handle all of the different variations would most likely require a constantly updated data source with the implementation details for each non-public site. I can't imagine that anyone would argue that disclosing the inner workings of your authentication mechanism is a good security practice.
I don't think that you're going to find a component that meets your requirements. Your best bet would be to see if the desired sites offer an RSS or ATOM syndication, or perhaps a web service to supply details to your application.
Not the answer that you were looking for, but I hope it helps a bit. 
--Jesse
|
|
|
|
|
Thanks Jesse. That really helps.
I guess there are two major things which need to be taken care of
1. Security/Credentials.
2. Session Management.
How about using AXWebBrowser or InternetBrowser to load the page and then retrieve the HTML Source (IHTMLDocument2), since it is as good as loading it in browser.
1. Assuming that for all the site we have only one credential. How about passing credentials in all possible ways(cookies, setting the credential property, and many (I don't know how many  ). If we do that we can handle this. What you say?
2. Session Management - Might not be an issue if we use AXWebBrowser or InternetBrowser to load the page and then retrieve the HTML Source.
I am writing a WebSpider kind of application, which will parse HTML source and try to download each link in the HTML page (n - level).
Thanks,
Anand
|
|
|
|
|
Anand Mudliar wrote:
How about using AXWebBrowser or InternetBrowser to load the page and then retrieve the HTML Source (IHTMLDocument2), since it is as good as loading it in browser.
If you are looking for a non-UI solution there is no need to include this as it packages more overhead that what you really need. See my response for more detail.
- Nick Parker
My Blog | My Articles
|
|
|
|
|
One thing that I failed to mention in my original response is that I'm looking at the problem from the perspective of crawling a public zone, with numerous sites that you know nothing about. If instead you're writing a crawler to work against sites that you know the details of, such as an intranet... it becomes a much more doable task. I'm still writing with the assumption that you're looking to implement a public crawler.
Assuming that you aren't displaying the UI, using the HttpWebRequest, as Nick suggests, will accomplish the same thing as the AXWebBrowser and be a good deal lighter.
The problem with security is that there isn't a standard way for it to be handled for a public site. Nick's example will work fantastic if you're crawling a site that uses NT security with an IIS web server. As far as passing authentication details in cookies and/or posting them... the problem is that the details will differ from site to site. Different authors will handle the details in different ways. For example, one site may have a cookie that stores the username and password in a cookie, then checks them against a database. Another site may store a hashed version of the password. And so on... Sadly, if there is a way that you could reliably automate every possible implementation, I'm not aware of it.
As for the session management issues, as far as I can figure, you're going to run into the same problem as security. The HttpWebRequest object makes the same requests as the AXWebBrowser, so there isn't an issue there. The problem area is going to be the case where the server expects some kind of persisted (cookies, etc) or supplied (post, querystring) state-related information. Just like with authentication/security, the implementation from site to site will vary.
Two final things that I think are worth mentioning. If a site is protected by some authentication/security or session verification method, then there is a very good chance that the owner does not wish to be crawled by an automated spider. Also, be sure to adhere to the wishes of the robots.txt file when it comes to crawling.
Hope that helps.
--Jesse
|
|
|
|
|
That really helped
I am writing a spider for the sites where I have access, but I am not aware of the kind of authentication mechanism/sesion management they use. I tried using HttpWebRequest with credentials but I was runningg into problems of multiple redirection, operation timeout, underlying connection closed.
When I access the sites from the IE browser it works fine but with my crawler it fails 
Thanks,
Anand
|
|
|
|
|
Ahhh... so I've basically been spouting off under a false assumption.  Sorry about that.
The HttpWebRequest can do anything that the AXWebBrowser can, it just sometimes requires a bit more code. I would definately recommend using the HttpWebRequest, as the difference in performance and resource consumption is worth it, IMHO.
As far as the timeouts and closed connections, they're usually due to network or remote site issues. I would think your best bet would simply be to trap the exception and retry. The redirections are where some work comes in.
Assuming that you retrieve a HttpWebResponse object from the request by calling the GetResponse method, redirection will cause the ContentLength property of the response to be set to -1. By parsing out the headers of the response, you can determine where the redirection was to, and use your HttpWebRequest to query the redirection URI. You can reference the headers here[^]. Search for the "10.3 Redirection 3xx" section.
I had some code that followed redirection using the HttpWebRequest, but, unfortunately I can't seem to find it at the moment. If I can remember where the heck I put it, I'll post it in another reply.
Hope that helps.
--Jesse
|
|
|
|
|
Here is a quick example, you should be able to fit this into your codebase and it requires no UI.
public string GetPageContent(string user, string pass, string url)
{
WebRequest wreq;
WebResponse wres;
WebProxy proxy;
StreamReader sr;
String content = string.Empty;
try
{
proxy = new WebProxy("someproxy", false);
NetworkCredential cred = new NetworkCredential(user, pass,
"somedomain");
wreq = HttpWebRequest.Create(url);
wreq.Proxy = proxy;
proxy.Credentials = cred;
wres = wreq.GetResponse();
sr = new StreamReader(wres.GetResponseStream());
content = sr.ReadToEnd();
sr.Close();
}
catch(Exception e)
{
Console.WriteLine("Error:" + e.Message);
}
return content;
}
- Nick Parker
My Blog | My Articles
|
|
|
|
|
When you righ-click on the executable/properties, a version tab is available. How do I set some of this parameters (e.g. Company Name, product version,..etc)
|
|
|
|
|
Take a look at the attributes in the AssemblyInfo.cs file. Most of the file properties can be set there.
Hope that helps.
--Jesse
|
|
|
|
|
Hello, all.
I am using a non-standard color scheme for my app. My problem (well, one of them...) is that I can't seem to set the color of the scroll bar, or the scroll arrows for a TabControl. No matter what I set the BackColor for the app to be, the scroll bar and scroll arrows seem to refuse to be anything but the default color for controls in the entire system. Is there any way to override these colors without painting them myself?
Thanks! And happy holidays!
|
|
|
|
|
I am trying to create a simple encrypted licence file for my software. The software will have the decryption keys built in - just strings - but anybody can use .Net Reflector to decompile my code and look for the Key/IV strings.
I am sure I am missing something as I am new to all this encryption malarkey and would appreciate any help you can offer on how I should be doing this.
Thanks in advance
|
|
|
|
|
You could use Dotfuscator to make your source a bit more secured if they try to decompile it. Just a thought.
~javier lozano
(blog)
|
|
|
|
|
First off, consider if your software is worth somebody spending the time to decompile your code just to be able to use your application and then publishing a license generator in the web. (Take into account that the average user will not have the skill to do this)
If the answer is yes, then I wouldnt waste too much time...it'll get cracked anyways...just do a search in google for 'Keygen' and any software developer will cry himself to death. :p and I'm sure most of those apps have spend hours and money way beyond what ur will to try and stop that from happening.
|
|
|
|
|
Hi,
I need print a data from an big array in severals columns.
I can not find in C# a way to to "write " an array similar to
Fortran statement like:
WRITE (*;100) (a(i) , i =1,20)
100 FORMAT (1X, 'a= '; 20F10.2)
Thanks
Ted
|
|
|
|
|
I havent worked in Fortran. Can u please tell what these statements are exactly doing. Maybe then i can help u
|
|
|
|
|
I need write some values from an array in columns like below
for (int np=1; np < NP1 ;np++)
{
writer1.WriteLine ("{0,8},{1,8},{2,8},{3,8},{4,8},{5,8},{6,8},{7,8},
PARAM[1],PARAM[2],PARAM[3],PARAM[4],etc...
}
The problem is that the number of columns is calculated by the code
/it can varied/ . Second only a part of array should will be printed.
Regards
Ted
|
|
|
|
|
Can't you just nest another loop? Like:
int cols=3;
int testCount=PARAMS.length;
for (int rowIndex=0;rowIndex < Math.Ceiling (Convert.ToDouble (testCount) / Convert.ToDouble (cols));rowIndex++)
{
for (int colIndex=0;colIndex < cols;colIndex++)
{
if (rowIndex * cols +colIndex < testCount)
{
writer1.Write(PARAMS[rowIndex * cols +colIndex].ToString () + " ");
}
}
writer1.WriteLine();
}
That should write it one row at a time:
0 1 2
3 4 5
6 etc.
Hope this helps
Bill
|
|
|
|
|
My problem is now much more trivial:
how I can get my output as a row?, instead for column from example as below:
int []PARAM = {1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17};
int zmax =11;
for (int z = 0; z < zmax+1 ; z++)
writer1.WriteLine(PARAM[z].ToString()+ " ");
=== output
1
2
3
4
5
6
7
8
9
10
11
12
//Ted
teja
|
|
|
|
|
Oh! That is easier.
I do not know what writer1 is, since none of your previous posts contain a declaration...but in many cases there is a .Write() method as well as a .WriteLine. Is so you can do:
for (int i=0;i < PARAMS.Length; i++)
writer1.Write(PARAMS[i].ToString() + " ");
writer1.WriteLine()
If you have no Write, just build the string up before you print it.
System.Text.StringBuilder sb = new System.Text.StringBuilder();
for (int i=0;i < PARAMS.Length; i++){
SB.Append(PARAMS[i].ToString() );
SB.Append(" ");
}
writer1.WriteLine(SB.ToString())
|
|
|
|
|
I've got a small utility that occasionaly locks up in itself if a client does not respond, so I have added a time out clause by creating a side thread for it to run on, and killing the thread when the timeout condition is reached OR the thread completes.
However, creating an entirely new thread to run this on just seems daft, so i'd like to do it on a ThreadPool. Is there any way to kill a thread pulled from a thread pool after a certain amount of time?
Cheers
Cata
|
|
|
|
|
I solved a similar problem another way. I created a class to hold each thread's info. i.e. ThreadID,StartDateTime,InUse,Finished etc...
Then when a new thread is required I add an instance of this class to an array which is monitored from a single worker thread. In the worker thread I interate through the array of threads and kill the ones that have the Finished flag set or that have exceeded my timout based on StartDateTime, and InUse flags. That way you can handle the termination of each thread based on what type of operation it is.
Don't know if this is applicable to your solution or not, but it worked great for me.
|
|
|
|
|
My solution is very similar, but i use the ThreadState of the thread I create. THe thread is only active for at most 1200ms and on the median about 15ms, before it is terminated. Therefore, creating and disposing new threads is a waste.
I was looking for a way to use the thread pool as it would be a more appropriate solution.
|
|
|
|
|
Hello, please can anyone help. I need the article and source code for the C# Sorted ListView by Carlos H. Perez, as it is not at the following link anymore. I use a listview which continuously updates two of the subitems, which is causing bad flickering. I need to know how to update subitems on a listview, without redrawing the whole row.
http://www.codeproject.com/cs/miscctrl/sortedlistview.asp
If anyone has the sourcecode please can they send it me.
Kindest Regards
Mike
|
|
|
|
|
Hello, please can anyone help. I need the article and source code for the C# Sorted ListView by Carlos H. Perez, as it is not at the following link anymore. I use a listview which continuously updates two of the subitems, which is causing bad flickering. I need to know how to update subitems on a listview, without redrawing the whole row.
http://www.codeproject.com/cs/miscctrl/sortedlistview.asp
If anyone has the sourcecode please can they send it me.
Kindest Regards
Mike
|
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
