Click here to Skip to main content
15,890,512 members
Home / Discussions / C#
   

C#

 
QuestionRe: C# tooltip Pin
jojoba201024-Jan-10 19:50
jojoba201024-Jan-10 19:50 
AnswerRe: C# tooltip Pin
Mycroft Holmes24-Jan-10 19:54
professionalMycroft Holmes24-Jan-10 19:54 
AnswerRe: C# tooltip Pin
sanforjackass24-Jan-10 20:01
sanforjackass24-Jan-10 20:01 
GeneralRe: C# tooltip Pin
Mycroft Holmes24-Jan-10 19:50
professionalMycroft Holmes24-Jan-10 19:50 
QuestionCan a Java application call a DotNet WCF service? Pin
ashishtango24-Jan-10 17:24
ashishtango24-Jan-10 17:24 
AnswerRe: Can a Java application call a DotNet WCF service? Pin
Abhinav S24-Jan-10 18:28
Abhinav S24-Jan-10 18:28 
GeneralRe: Can a Java application call a DotNet WCF service? Pin
ashishtango24-Jan-10 22:17
ashishtango24-Jan-10 22:17 
Questioncrawling a website Pin
uglyeyes24-Jan-10 15:52
uglyeyes24-Jan-10 15:52 
Hi,
I need to crawl a website of below url format and store the content locally in a csv file
now, I only need to crawl URL of below format. I should be able to do a loop with query string ranging from 0 to 1000000(or a max value present).
http://www.mysite.com.au/products/products.asp?p=101

the only way i think will be to do a loop from 0 to 1000000 but again there are some id that doesnt exists and redirects to main page which i need to exclude from crawl list i.e it redirects to http://www.mysite.com.au/products page.
below is the code so far i have coded. please assist me how can i achieve this.

 public static void CrawlSite()
        {
            Console.WriteLine("Beginning crawl.");
            CrawlPage("http://www.mysite.com.au/products/products.asp?p=");                     Console.WriteLine("Finished crawl.");

        }

private static void CrawlPage(string url)
{
	for(int i=0; i<=100000; i++)
	{
		Console.WriteLine("Crawling " + url+i.tostring());

		HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(url);
		request.UserAgent = "blah!";

		WebResponse response = request.GetResponse();

		Stream stream = response.GetResponseStream();

		StreamReader reader = new StreamReader(stream);
		string htmlText = reader.ReadToEnd();
		
		//logic to check if it sends to home page, if it doesnt dont write it to a file
		using (StreamWriter sw = File.AppendText(@"c:\logs\data.txt"))
		{
		
			sw.WriteLine(htmlText);
			sw.Close();
		}
		
		
	}
	
}

AnswerRe: crawling a website Pin
RyanEK24-Jan-10 17:05
RyanEK24-Jan-10 17:05 
QuestionInteger and null value Pin
Jassim Rahma24-Jan-10 12:27
Jassim Rahma24-Jan-10 12:27 
AnswerRe: Integer and null value Pin
Dan Mos24-Jan-10 12:43
Dan Mos24-Jan-10 12:43 
AnswerRe: Integer and null value Pin
Graham Shanks24-Jan-10 12:44
Graham Shanks24-Jan-10 12:44 
QuestionProcessor Affinity and Memory Allocation Limit+Tracking on per-process and per-thread basis Pin
devvvy24-Jan-10 11:49
devvvy24-Jan-10 11:49 
AnswerRe: Processor Affinity and Memory Allocation Limit+Tracking on per-process and per-thread basis Pin
Luc Pattyn24-Jan-10 12:02
sitebuilderLuc Pattyn24-Jan-10 12:02 
QuestionMessage Removed Pin
24-Jan-10 8:57
Bizcrack24-Jan-10 8:57 
AnswerRe: C# - get data from MySQL DB into DataGridView Pin
Som Shekhar24-Jan-10 9:06
Som Shekhar24-Jan-10 9:06 
AnswerRe: C# - get data from MySQL DB into DataGridView Pin
Luc Pattyn24-Jan-10 9:48
sitebuilderLuc Pattyn24-Jan-10 9:48 
AnswerRe: C# - get data from MySQL DB into DataGridView Pin
Bizcrack26-Jan-10 5:15
Bizcrack26-Jan-10 5:15 
GeneralRe: C# - get data from MySQL DB into DataGridView Pin
Bizcrack27-Jan-10 4:12
Bizcrack27-Jan-10 4:12 
QuestionAttach scrollbar to listview Pin
Eli Nurman24-Jan-10 8:11
Eli Nurman24-Jan-10 8:11 
AnswerRe: Attach scrollbar to listview Pin
DeepakMega25-Jan-10 1:02
DeepakMega25-Jan-10 1:02 
QuestionContext Menu Not Appearing in Correct Location Pin
kodysumter24-Jan-10 8:10
kodysumter24-Jan-10 8:10 
AnswerRe: Context Menu Not Appearing in Correct Location Pin
Som Shekhar24-Jan-10 8:18
Som Shekhar24-Jan-10 8:18 
GeneralRe: Context Menu Not Appearing in Correct Location Pin
kodysumter24-Jan-10 14:08
kodysumter24-Jan-10 14:08 
GeneralRe: Context Menu Not Appearing in Correct Location Pin
Som Shekhar24-Jan-10 18:02
Som Shekhar24-Jan-10 18:02 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.