|
Anand Mudliar wrote:
How about using AXWebBrowser or InternetBrowser to load the page and then retrieve the HTML Source (IHTMLDocument2), since it is as good as loading it in browser.
If you are looking for a non-UI solution there is no need to include this as it packages more overhead that what you really need. See my response for more detail.
- Nick Parker
My Blog | My Articles
|
|
|
|
|
One thing that I failed to mention in my original response is that I'm looking at the problem from the perspective of crawling a public zone, with numerous sites that you know nothing about. If instead you're writing a crawler to work against sites that you know the details of, such as an intranet... it becomes a much more doable task. I'm still writing with the assumption that you're looking to implement a public crawler.
Assuming that you aren't displaying the UI, using the HttpWebRequest, as Nick suggests, will accomplish the same thing as the AXWebBrowser and be a good deal lighter.
The problem with security is that there isn't a standard way for it to be handled for a public site. Nick's example will work fantastic if you're crawling a site that uses NT security with an IIS web server. As far as passing authentication details in cookies and/or posting them... the problem is that the details will differ from site to site. Different authors will handle the details in different ways. For example, one site may have a cookie that stores the username and password in a cookie, then checks them against a database. Another site may store a hashed version of the password. And so on... Sadly, if there is a way that you could reliably automate every possible implementation, I'm not aware of it.
As for the session management issues, as far as I can figure, you're going to run into the same problem as security. The HttpWebRequest object makes the same requests as the AXWebBrowser, so there isn't an issue there. The problem area is going to be the case where the server expects some kind of persisted (cookies, etc) or supplied (post, querystring) state-related information. Just like with authentication/security, the implementation from site to site will vary.
Two final things that I think are worth mentioning. If a site is protected by some authentication/security or session verification method, then there is a very good chance that the owner does not wish to be crawled by an automated spider. Also, be sure to adhere to the wishes of the robots.txt file when it comes to crawling.
Hope that helps. 
--Jesse
|
|
|
|
|
That really helped
I am writing a spider for the sites where I have access, but I am not aware of the kind of authentication mechanism/sesion management they use. I tried using HttpWebRequest with credentials but I was runningg into problems of multiple redirection, operation timeout, underlying connection closed.
When I access the sites from the IE browser it works fine but with my crawler it fails 
Thanks,
Anand
|
|
|
|
|
Ahhh... so I've basically been spouting off under a false assumption.  Sorry about that.
The HttpWebRequest can do anything that the AXWebBrowser can, it just sometimes requires a bit more code. I would definately recommend using the HttpWebRequest, as the difference in performance and resource consumption is worth it, IMHO.
As far as the timeouts and closed connections, they're usually due to network or remote site issues. I would think your best bet would simply be to trap the exception and retry. The redirections are where some work comes in.
Assuming that you retrieve a HttpWebResponse object from the request by calling the GetResponse method, redirection will cause the ContentLength property of the response to be set to -1. By parsing out the headers of the response, you can determine where the redirection was to, and use your HttpWebRequest to query the redirection URI. You can reference the headers here[^]. Search for the "10.3 Redirection 3xx" section.
I had some code that followed redirection using the HttpWebRequest, but, unfortunately I can't seem to find it at the moment. If I can remember where the heck I put it, I'll post it in another reply.
Hope that helps.
--Jesse
|
|
|
|
|
Here is a quick example, you should be able to fit this into your codebase and it requires no UI.
public string GetPageContent(string user, string pass, string url)
{
WebRequest wreq;
WebResponse wres;
WebProxy proxy;
StreamReader sr;
String content = string.Empty;
try
{
proxy = new WebProxy("someproxy", false);
NetworkCredential cred = new NetworkCredential(user, pass,
"somedomain");
wreq = HttpWebRequest.Create(url);
wreq.Proxy = proxy;
proxy.Credentials = cred;
wres = wreq.GetResponse();
sr = new StreamReader(wres.GetResponseStream());
content = sr.ReadToEnd();
sr.Close();
}
catch(Exception e)
{
Console.WriteLine("Error:" + e.Message);
}
return content;
}
- Nick Parker
My Blog | My Articles
|
|
|
|
|
When you righ-click on the executable/properties, a version tab is available. How do I set some of this parameters (e.g. Company Name, product version,..etc)
|
|
|
|
|
Take a look at the attributes in the AssemblyInfo.cs file. Most of the file properties can be set there.
Hope that helps.
--Jesse
|
|
|
|
|
Hello, all.
I am using a non-standard color scheme for my app. My problem (well, one of them...) is that I can't seem to set the color of the scroll bar, or the scroll arrows for a TabControl. No matter what I set the BackColor for the app to be, the scroll bar and scroll arrows seem to refuse to be anything but the default color for controls in the entire system. Is there any way to override these colors without painting them myself?
Thanks! And happy holidays!
|
|
|
|
|
I am trying to create a simple encrypted licence file for my software. The software will have the decryption keys built in - just strings - but anybody can use .Net Reflector to decompile my code and look for the Key/IV strings.
I am sure I am missing something as I am new to all this encryption malarkey and would appreciate any help you can offer on how I should be doing this.
Thanks in advance
|
|
|
|
|
You could use Dotfuscator to make your source a bit more secured if they try to decompile it. Just a thought.
~javier lozano
(blog)
|
|
|
|
|
First off, consider if your software is worth somebody spending the time to decompile your code just to be able to use your application and then publishing a license generator in the web. (Take into account that the average user will not have the skill to do this)
If the answer is yes, then I wouldnt waste too much time...it'll get cracked anyways...just do a search in google for 'Keygen' and any software developer will cry himself to death. :p and I'm sure most of those apps have spend hours and money way beyond what ur will to try and stop that from happening.
|
|
|
|
|
Hi,
I need print a data from an big array in severals columns.
I can not find in C# a way to to "write " an array similar to
Fortran statement like:
WRITE (*;100) (a(i) , i =1,20)
100 FORMAT (1X, 'a= '; 20F10.2)
Thanks
Ted
|
|
|
|
|
I havent worked in Fortran. Can u please tell what these statements are exactly doing. Maybe then i can help u
|
|
|
|
|
I need write some values from an array in columns like below
for (int np=1; np < NP1 ;np++)
{
writer1.WriteLine ("{0,8},{1,8},{2,8},{3,8},{4,8},{5,8},{6,8},{7,8},
PARAM[1],PARAM[2],PARAM[3],PARAM[4],etc...
}
The problem is that the number of columns is calculated by the code
/it can varied/ . Second only a part of array should will be printed.
Regards
Ted
|
|
|
|
|
Can't you just nest another loop? Like:
int cols=3;
int testCount=PARAMS.length;
for (int rowIndex=0;rowIndex < Math.Ceiling (Convert.ToDouble (testCount) / Convert.ToDouble (cols));rowIndex++)
{
for (int colIndex=0;colIndex < cols;colIndex++)
{
if (rowIndex * cols +colIndex < testCount)
{
writer1.Write(PARAMS[rowIndex * cols +colIndex].ToString () + " ");
}
}
writer1.WriteLine();
}
That should write it one row at a time:
0 1 2
3 4 5
6 etc.
Hope this helps
Bill
|
|
|
|
|
My problem is now much more trivial:
how I can get my output as a row?, instead for column from example as below:
int []PARAM = {1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17};
int zmax =11;
for (int z = 0; z < zmax+1 ; z++)
writer1.WriteLine(PARAM[z].ToString()+ " ");
=== output
1
2
3
4
5
6
7
8
9
10
11
12
//Ted
teja
|
|
|
|
|
Oh! That is easier. 
I do not know what writer1 is, since none of your previous posts contain a declaration...but in many cases there is a .Write() method as well as a .WriteLine. Is so you can do:
for (int i=0;i < PARAMS.Length; i++)
writer1.Write(PARAMS[i].ToString() + " ");
writer1.WriteLine()
If you have no Write, just build the string up before you print it.
System.Text.StringBuilder sb = new System.Text.StringBuilder();
for (int i=0;i < PARAMS.Length; i++){
SB.Append(PARAMS[i].ToString() );
SB.Append(" ");
}
writer1.WriteLine(SB.ToString())
|
|
|
|
|
I've got a small utility that occasionaly locks up in itself if a client does not respond, so I have added a time out clause by creating a side thread for it to run on, and killing the thread when the timeout condition is reached OR the thread completes.
However, creating an entirely new thread to run this on just seems daft, so i'd like to do it on a ThreadPool. Is there any way to kill a thread pulled from a thread pool after a certain amount of time?
Cheers
Cata
|
|
|
|
|
I solved a similar problem another way. I created a class to hold each thread's info. i.e. ThreadID,StartDateTime,InUse,Finished etc...
Then when a new thread is required I add an instance of this class to an array which is monitored from a single worker thread. In the worker thread I interate through the array of threads and kill the ones that have the Finished flag set or that have exceeded my timout based on StartDateTime, and InUse flags. That way you can handle the termination of each thread based on what type of operation it is.
Don't know if this is applicable to your solution or not, but it worked great for me.
|
|
|
|
|
My solution is very similar, but i use the ThreadState of the thread I create. THe thread is only active for at most 1200ms and on the median about 15ms, before it is terminated. Therefore, creating and disposing new threads is a waste.
I was looking for a way to use the thread pool as it would be a more appropriate solution.
|
|
|
|
|
Hello, please can anyone help. I need the article and source code for the C# Sorted ListView by Carlos H. Perez, as it is not at the following link anymore. I use a listview which continuously updates two of the subitems, which is causing bad flickering. I need to know how to update subitems on a listview, without redrawing the whole row.
http://www.codeproject.com/cs/miscctrl/sortedlistview.asp
If anyone has the sourcecode please can they send it me.
Kindest Regards
Mike
|
|
|
|
|
Hello, please can anyone help. I need the article and source code for the C# Sorted ListView by Carlos H. Perez, as it is not at the following link anymore. I use a listview which continuously updates two of the subitems, which is causing bad flickering. I need to know how to update subitems on a listview, without redrawing the whole row.
http://www.codeproject.com/cs/miscctrl/sortedlistview.asp
If anyone has the sourcecode please can they send it me.
Kindest Regards
Mike
|
|
|
|
|
Hi
I have done a c# program such that , it reads a xml schema and passes values
to the dataset and create a xml file appropriately for their datatypes
( for ex: john for string, 67 for int)
What I have to do is . At present it creates only 1 row (record) in the xml file.
should loop and create atleast 200 records. Also, think of ways where the 200 records
can be different from each other.
Here is the coding which i have done.
please help me to do my requirement
using System;
using System.Drawing;
using System.Collections;
using System.ComponentModel;
using System.Windows.Forms;
using System.Data;
using System.Xml;
using System.Xml.Schema;
namespace testdatagenerator
{
///
/// Summary description for Form1.
///
public class Form1 : System.Windows.Forms.Form
{
private System.Windows.Forms.TextBox txtSchema;
private System.Windows.Forms.Button button1;
private System.Windows.Forms.TextBox txtXml;
private System.Windows.Forms.Button button2;
private System.Windows.Forms.Button button3;
private System.Windows.Forms.DataGrid dataGrid1;
private System.Data.DataSet dataSet1;
///
/// Required designer variable.
///
private System.ComponentModel.Container components = null;
public Form1()
{
//
// Required for Windows Form Designer support
//
InitializeComponent();
//
// TODO: Add any constructor code after InitializeComponent call
//
}
///
/// Clean up any resources being used.
///
protected override void Dispose( bool disposing )
{
if( disposing )
{
if (components != null)
{
components.Dispose();
}
}
base.Dispose( disposing );
}
#region Windows Form Designer generated code
///
/// Required method for Designer support - do not modify
/// the contents of this method with the code editor.
///
private void InitializeComponent()
{
this.txtSchema = new System.Windows.Forms.TextBox();
this.button1 = new System.Windows.Forms.Button();
this.txtXml = new System.Windows.Forms.TextBox();
this.button2 = new System.Windows.Forms.Button();
this.button3 = new System.Windows.Forms.Button();
this.dataGrid1 = new System.Windows.Forms.DataGrid();
this.dataSet1 = new System.Data.DataSet();
((System.ComponentModel.ISupportInitialize)(this.dataGrid1)).BeginInit();
((System.ComponentModel.ISupportInitialize)(this.dataSet1)).BeginInit();
this.SuspendLayout();
//
// txtSchema
//
this.txtSchema.Location = new System.Drawing.Point(88, 24);
this.txtSchema.Name = "txtSchema";
this.txtSchema.Size = new System.Drawing.Size(440, 20);
this.txtSchema.TabIndex = 0;
this.txtSchema.Text = "";
//
// button1
//
this.button1.Font = new System.Drawing.Font("Microsoft Sans Serif", 8.25F, System.Drawing.FontStyle.Bold, System.Drawing.GraphicsUnit.Point, ((System.Byte)(0)));
this.button1.Location = new System.Drawing.Point(536, 24);
this.button1.Name = "button1";
this.button1.TabIndex = 1;
this.button1.Text = "Browse";
this.button1.Click += new System.EventHandler(this.button1_Click);
//
// txtXml
//
this.txtXml.Location = new System.Drawing.Point(88, 64);
this.txtXml.Name = "txtXml";
this.txtXml.Size = new System.Drawing.Size(440, 20);
this.txtXml.TabIndex = 2;
this.txtXml.Text = "";
//
// button2
//
this.button2.Font = new System.Drawing.Font("Microsoft Sans Serif", 8.25F, System.Drawing.FontStyle.Bold, System.Drawing.GraphicsUnit.Point, ((System.Byte)(0)));
this.button2.Location = new System.Drawing.Point(536, 64);
this.button2.Name = "button2";
this.button2.TabIndex = 3;
this.button2.Text = "Load";
this.button2.Click += new System.EventHandler(this.button2_Click);
//
// button3
//
this.button3.Font = new System.Drawing.Font("Microsoft Sans Serif", 8.25F, System.Drawing.FontStyle.Bold, System.Drawing.GraphicsUnit.Point, ((System.Byte)(0)));
this.button3.Location = new System.Drawing.Point(616, 64);
this.button3.Name = "button3";
this.button3.TabIndex = 4;
this.button3.Text = "Save";
this.button3.Click += new System.EventHandler(this.button3_Click);
//
// dataGrid1
//
this.dataGrid1.DataMember = "";
this.dataGrid1.DataSource = this.dataSet1;
this.dataGrid1.HeaderForeColor = System.Drawing.SystemColors.ControlText;
this.dataGrid1.Location = new System.Drawing.Point(0, 120);
this.dataGrid1.Name = "dataGrid1";
this.dataGrid1.Size = new System.Drawing.Size(776, 320);
this.dataGrid1.TabIndex = 5;
//
// dataSet1
//
this.dataSet1.DataSetName = "NewDataSet";
this.dataSet1.Locale = new System.Globalization.CultureInfo("en-US");
//
// Form1
//
this.AutoScaleBaseSize = new System.Drawing.Size(5, 13);
this.ClientSize = new System.Drawing.Size(808, 454);
this.Controls.Add(this.dataGrid1);
this.Controls.Add(this.button3);
this.Controls.Add(this.button2);
this.Controls.Add(this.txtXml);
this.Controls.Add(this.button1);
this.Controls.Add(this.txtSchema);
this.Name = "Form1";
this.Text = "TestDatagenerator";
((System.ComponentModel.ISupportInitialize)(this.dataGrid1)).EndInit();
((System.ComponentModel.ISupportInitialize)(this.dataSet1)).EndInit();
this.ResumeLayout(false);
}
#endregion
///
/// The main entry point for the application.
///
[STAThread]
static void Main()
{
Application.Run(new Form1());
}
private void button1_Click(object sender, System.EventArgs e)
{
OpenFileDialog fdlg = new OpenFileDialog();
fdlg.Title = "select schema file" ;
fdlg.InitialDirectory = @"c:\CDEV\testdatagenerator" ;
fdlg.Filter = "schema files(*.xsd) |*.xsd| all files(*.*) | *.*";
fdlg.FilterIndex = 2 ;
fdlg.RestoreDirectory = true ;
if(fdlg.ShowDialog() == DialogResult.OK)
{
txtSchema.Text = fdlg.FileName ;
}
}
private void button2_Click(object sender, System.EventArgs e)
{
dataSet1.ReadXmlSchema(txtSchema.Text);
foreach (DataTable dTbl in dataSet1.Tables)
{
object[] oValues = new object[dTbl.Columns.Count];
int i = 0;
foreach (DataColumn dColmn in dTbl.Columns)
{
switch(dColmn.DataType.ToString())
{
case "System.String":
oValues[i] = (string) "This is string";
break;
case "System.Int32":
oValues[i] = (int) 100;
break;
case "System.DateTime":
oValues[i] = new DateTime(2004,10,12);
break;
case "System.Decimal":
oValues[i] = new Decimal(999999999.99);
break;
case "System.Int16":
oValues[i] = (short) 32767;
break;
case "System.Int64":
oValues[i] = (long) 9223372036854775807;
break;
case "System.Double":
oValues[i] = (double) 888888888888888;
break;
}
i += 1;
}
dTbl.Rows.Add(oValues);
}
}
private void button3_Click(object sender, System.EventArgs e)
{
dataSet1.WriteXml(txtXml.Text,XmlWriteMode.WriteSchema );
MessageBox.Show("Saved");
}
}
}
|
|
|
|
|
here is the problem let say that I 've the following DataGrid....
_______ ---------------------------------------------
Header _ ||col_01 || col_02 || col_03 || col_04 ||
_______ ---------------------------------------------
Row 0__ || __a__ || __b__ || __c__ || __d__ ||
_______ ---------------------------------------------
I NEED TO PAINT SO IT LOOKS LOKE THE FO::OWING
_______ -----------------------
Header _ ||col_01 || col_03 ||
_______ -----------------------
Header_ ||col_02 || col_04 ||
_______ -----------------------
Row 0__ || __a__ || __b__ ||
_______ -----------------------
Row 0__ || __c__ || __d__ ||
_______ -----------------------
I Hope The Idea Is now Clear ... so plz help me I need it yesterday............
|
|
|
|
|
hi
i want to check what key on keyboard is presed.
how kan i get the key pressed char code?
thanks
|
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
