|
OK,
1.
Generating a huge file, then search for duplicates, is a waste of time. If the code that generates such list can be modified, modifying it will always be a much better approach.
It will take less code, less effort to develop, and less CPU cylces to execute.
2.
One way to reduce the postprocessing effort (but not code) is by generating not one but many files, i.e. by applying some sorting. Say you calculate an 8-bit hash value on the URLs, and sort them accordingly in 256 different files. That is easy, and reduces your postprocessing to 256 much smaller jobs (average file size now is 2MB, and now you can do a ReadAllLines).
3.
The ultimate solution is to avoid duplicates right away.
I have written web crawlers before; obviously they need a way to avoid visiting the same URL twice, so they contain some logic for that.
Furthermore, you typically want to search breadth-first, i.e. read and process all of a page before switching to another page; that is more efficient than depth-first, since in the latter you have to either keep in memory or refetch the content of all the pages you are encountering while digging deeper.
The natural solution to these two issues is to organize the crawler around a list containing all the URLs you have encountered (and should visit). Every anchor, image source, style sheet encountered gets added, if not already in the list; when a page is done, move to the next in the list (use an index). When the index gets to the end of the list, you are done, and the list now holds all the unique URLs.
|
|
|
|
|
Thank you for your solution. I like the idea of 256 file and checksum. I will try it now.
|
|
|
|
|
To make it fairly simple, us a generic dictionary where each url key and an entry into the database. Read each line of the textfile one at a time and verify that the key doesn't exist if it doesn't add it to the dictionary. If the key does exist then ignore it. When you are all done use the foreach to go through each key writing it to a file as text.
Something like
Dictionary<string, string=""> Urls = new Dictionary<string, string="">();
while(TextReader.Read())
{
string key = TextReader.ReadLline();
if(!Urls.ContainsKey(key))
{
Urls.Add(key,key);
}
}
foreach(string s in Urls.Keys)
{
TextWriter.WriteLine(s);
}
Obviously you need to do all of the opening and closing of files, but this is probably the simplest way.
|
|
|
|
|
|
I think the easiest way to do this would be to provide your own Import/Export settings UI and handle the loading of data and saving to settings/saving to export file in your code. It should be fairly trivial.
|
|
|
|
|
I don't know if this will help you, but take a look at this CP Article[^]. I have found it useful and it contains some links to other articles/resources which may assist.
|
|
|
|
|
Hi,
I have a simple solution with a windows client and a website. The website exposes a web service that the windows client consumes. So far so good. Here's the weird part. When running my solution in VS, the call to the web service doesn't work when I hit CTRL + F5 unless I run it with a simple F5 first. So each time I open the solution in a new VS session, I have to run it in debug (F5) at least once before it works without debugging (CTRL + F5).
This makes no sense to me. Has anyone else run into an issue like this? Any ideas would be greatly appreciated!
Thanks!
Ian
|
|
|
|
|
It should respond with a (recompile if neccesary) during <CTRL>+<F5>. Have you checked/resetted your keymappings?
|
|
|
|
|
The key mappings are fine. Again, the application runs with a CTRL + F5 - it just doesn't work unless I've already run it with a simple F5. I wondering if it's a VS bug? I'm running it in a VM on a 64 bit test environment...
|
|
|
|
|
This is a very trivial question, but I have not had much chance in finding a solution. I have searched Google / blogs, but apparantly either there is no solution or I searched for the wrong keyword. Any help will be greatly appreciated.
I have an application (similar to a wizard) and I need to validate whether the user has the permission to write/read from a particular location.
I the following function (below), but I am always getting the value "true" eventhough from Windows Explorer when I try to navigate to the location, I'm always getting an error (eg. "You need permission to perform this action. [Retry] [Cancel]").
static bool CanReadAndWrite(string path)
{
var perm = new FileIOPermission(
FileIOPermissionAccess.Write &
FileIOPermissionAccess.Read &
FileIOPermissionAccess.PathDiscovery,
path);
try
{
perm.Demand();
return true;
}
catch
{
return false;
}
}
Many of the blogs/articles suggest that I should attemt to write to the location anyways, and if the write operation is successful, then I can proceed to write the main data. If there is an exception creating the stram then there is a perimission issue.
But this solution would NOT work for me. The application has two "pages".
- First wizard page: user selects the file path
- Second wizard page: specify options
- Third wizard page: write data
Each of the pages has Cancel button and the expectation is that user NOT be able to continue to the 2nd page if permission is denied. The above suggested solution would WORK fine if the user does not have permission. But if the user DOES have permission, then the validation check in Page # 1 would have created new file and/or overwrote an existing file with 0 bytes -- and if user cancels on Page #2 then there is a useless file on the file system.
So anyways, what should I do (clean implementation) to check for file permision?
I'm using Vista, .Net 3.5.
Thanks,
- Malhar
|
|
|
|
|
Create the temporary file to test the permissions and then immediately delete it. Whether the user continues the process or cancels it the file isn't needed at all after you've verified the permissions of the selected path.
|
|
|
|
|
Shouldn't you use an OR operator ('|') to request all those permissions?
I guess the AND result is plain zero.
|
|
|
|
|
Sorry for overlooking the detail when I pasted the code. But my original attempt included "|". I just tried now instantiating "perm" object like below:
var perm = new FileIOPermission(FileIOPermissionAccess.Write, directory);
But still no luck.
- Malhar
|
|
|
|
|
Hello,
I'm building my first web service in C#. I have met most of my needs for this service so far up to sending files. If this needs to go to another section. Please let me know which section to post in.
The basic steps of the service are:
1. receive an xml file through asp.net page
2. read xml and save to database.
3. read xml and retrieve file nodes and location for other files.
4. upload other files to directory on server.
This last step is where I need a tutorial or some suggestions.
As it stands I would just like to pull the files from client in some fashion after reading the xml file information. If anybody has a link to an article that would be great. I have looked at the Mtom solution but that is not viable for a web application.
Thanks to anybody that responds.
chuck
|
|
|
|
|
|
Have a little patience, you have only been waiting 4 hours. Instant answers cost money. 
Henry Minute
Do not read medical books! You could die of a misprint. - Mark Twain
Girl: (staring) "Why do you need an icy cucumber?"
“I want to report a fraud. The government is lying to us all.”
|
|
|
|
|
I think I found something[^]  FOR FREE
And also another thing[^]  FOR 12,95 $/Month
To view the Experts-Exchange solutions for free use the techinique described in this web site[^].
modified on Friday, May 1, 2009 5:49 AM
|
|
|
|
|
I don't know why pictures aren't shown in listview.. i would like to add them like thumbnails..or small icons..so when i will click on them they will resize in picturebox.. Here is code for adding to listview which isn't working.
private void Form1_Load(object sender, EventArgs e)
{
DirectoryInfo dir = new DirectoryInfo(@"d:\\feri\\ičr\\galerija\\galerija\\galerija");
foreach (FileInfo file in dir.GetFiles())
{
try
{
this.imageList1.Images.Add(Image.FromFile(file.FullName));
}
catch
{
Console.WriteLine("This is not an image file");
}
}
this.listView1.View = View.LargeIcon;
this.imageList1.ImageSize = new Size(32, 32);
this.listView1.LargeImageList = this.imageList1;
for (int j = 0; j < this.imageList1.Images.Count; j++)
{
ListViewItem item = new ListViewItem();
item.ImageIndex = j;
this.listView1.Items.Add(item);
}
}
|
|
|
|
|
Speaking purely from memory (can't get to VS to test at the moment) resizing an ImageList clears it. Your code resizes the ImageList after loading the images.
Try moving the resize to be before loading the images.
Henry Minute
Do not read medical books! You could die of a misprint. - Mark Twain
Girl: (staring) "Why do you need an icy cucumber?"
“I want to report a fraud. The government is lying to us all.”
|
|
|
|
|
|
Now its duplicating images in listview...how could everytime someone adds image to it..that content before would be erased and new images would be added then..
private void btn_browse_Click()
{
this.listView1.View = View.LargeIcon;
this.imageList1.ImageSize = new Size(60, 60);
this.listView1.LargeImageList = this.imageList1;
try
{
OpenFileDialog open = new OpenFileDialog();
open.Filter = "Image Files(*.jpg; *.jpeg; *.gif; *.bmp)|*.jpg; *.jpeg; *.gif; *.bmp";
if (open.ShowDialog() == DialogResult.OK)
{
picture.Image = new Bitmap(open.FileName);
this.imageList1.Images.Add(Image.FromFile(open.FileName));
}
}
catch (Exception)
{
throw new ApplicationException("Failed loading image");
}
for (int j = 0; j < this.imageList1.Images.Count; j++)
{
ListViewItem item = new ListViewItem();
item.ImageIndex = j;
this.listView1.Items.Add(item);
}
}
|
|
|
|
|
Aljaz111 wrote: Now its duplicating images in listview...how could everytime someone adds image to it..that content before would be erased and new images would be added then..
It would be nice if the image collection had a method to clear itself.
It is a truth universally acknowledged that a zombie in possession of brains must be in want of more brains.
-- Pride and Prejudice and Zombies
|
|
|
|
|
dan neely wrote: It would be nice if the image collection had a method to clear itself.
Gosh, wouldn't that be a good idea. You'd have thought Microsoft would have thought of that, wouldn't you? 
Henry Minute
Do not read medical books! You could die of a misprint. - Mark Twain
Girl: (staring) "Why do you need an icy cucumber?"
“I want to report a fraud. The government is lying to us all.”
|
|
|
|
|
And how this method for cleaning would be?
|
|
|
|
|
private void btn_browse_Click()
{
imageList1.Clear();
}
Unless they are likely to be changed somewhere else in your application, you would be better off setting the following three properties in the designer.
this.listView1.View = View.LargeIcon;
this.imageList1.ImageSize = new Size(60, 60);
this.listView1.LargeImageList = this.imageList1;
You should be OK now. Good Luck!
Henry Minute
Do not read medical books! You could die of a misprint. - Mark Twain
Girl: (staring) "Why do you need an icy cucumber?"
“I want to report a fraud. The government is lying to us all.”
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
