Fastest Index-Based XML Search

Question

5.00/5 (1 vote)

See more:

I have an XML-file with around 20Lakhs records.
I wish generate an Auto-Complete method for a textBox.
For Performing faster Index-based-Search in XML-file which will be the better option.
1.) XPath (or)
2.) LINQ
Or Any other technique is there.. for more faster search? Plz suggest and Explain...

Posted 1-Apr-11 20:50pm

SHAJANCHERIAN

Add a Solution

2 solutions

Solution 1

If you're searching for a small portion of data in you XML document, I'd say that using XmlReader[^] provides the fastest access to the data.

However since you're using it to create an auto-complete text box I guess that the search is going to be repeated often. Because of this it could be reasonable to load and parse the XML document and have an in-memory copy of the data in such format which is both fast to search and compact in size. Utilizing a HashSet[^] could be one option.

Posted 1-Apr-11 21:15pm

Wendelius

Comments

Albin Abel 2-Apr-11 3:21am

my 5. Added point. If it is a asp.net application then need to keep the collection a static one. Otherwise the collection has to be reconstructed for every page requests. OP hasn't mentioned it is a windows or asp.net application

Wendelius 2-Apr-11 3:55am

Thank you :) Yes, you're right, if ASP then the collection should be static.

SHAJANCHERIAN 2-Apr-11 4:27am

I am using Ajax-AutoComplete Extender and a WebService: Hope It will Manage In-memory collection.
Request for your suggestion/ advice..
------------------------------------------------------------------------------------------------
<cc1:AutoCompleteExtender ID="AutoCompleteExtender2" runat="server" TargetControlID="TextBox2"
ServiceMethod="GetItems" MinimumPrefixLength="1" CompletionSetCount="10" EnableCaching="true"
ServicePath="WebService.asmx">

-------------------------------------------------------------------------------
[WebMethod]
public string[] GetItems(string prefixText, int count)
{
List<string> suggestions = new List<string>();
using (XmlTextReader reader = new XmlTextReader(HttpContext.Current.Server.MapPath("~/XmlFiles/Items.xml")))
{
while (reader.Read())
{
if (reader.NodeType == XmlNodeType.Element && reader.Name == "item")
{
string itemName = reader.ReadInnerXml();
if (itemName.StartsWith(prefixText, StringComparison.InvariantCultureIgnoreCase))
{
suggestions.Add(itemName);

if (suggestions.Count == count) break;
}
}
}
}
return suggestions.ToArray();
}
------------------------------------------------------------------------------------------

Wendelius 2-Apr-11 8:26am

One possible bottleneck in this scenario is the web service. Consider if you could searxh for the data without having to call the web service each time the data changes (if I understood your approach correctly).

Albin Abel 3-Apr-11 15:16pm

Hi there is a over head of a web service call. Web service is fine for a layered application where many clients need to access a common resource. But again in your implementation the results are not cached in memory. Every time it reads the xml file and do all blah blah. Keep the suggestions as a static list that would be the best suggestion for this scenario. If multiple web application going to use this list then you can serve this list through a web service, else move it to your own application will give you performance.

SHAJANCHERIAN 14-Apr-11 2:29am

If I move this to (say class name: Class1 nad methodNmae: GetItems..)
Can you provide the code for Autocomplete extender instead of previous code . ie.
<cc1:AutoCompleteExtender ID="AutoCompleteExtender2" runat="server" TargetControlID="TextBox2"
ServiceMethod="GetItems" MinimumPrefixLength="1" CompletionSetCount="10" EnableCaching="true"
ServicePath="WebService.asmx">

Albin Abel 15-Apr-11 4:37am

Hi ShajanCherian, you asked for code. It is very difficult to guess the mess on your table to give a perfect code. We can give you guidelines or else if you asked for code then we need to handle your project. Look at my answer.

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Albin Abel · Accepted Answer · 2011-04-14T22:34:00

Hi
The autocomplete extender works with a webservice or a page method decorated with webservice method and script service method attributes. To read an XML with that much record from file and search for each character typed is a huge work. An in memory collection may help a little. Whether xpath or linq, it has to be executed after the data loaded into memory. So if you have that already in memory this overhead can be avoided. But remember this is again a trade off between speed and memory consumption. The in memory collection permanently occupy some memory space. Again it is a string collection definitely occupy more space. The key parameter you can set in the autocomplete markup is MinimumPrefixLength. It should be minimum 3 to avoid a costly search.

Another way store the xml in sql server and when needed retrieve it and can use the xpath. Which will be more faster than file based. Refer this link. http://msdn.microsoft.com/en-us/library/ms187508%28v=sql.90%29.aspx[^]

Well google give more information if you use key words "ado.net xml best practice".

You asked for a code. Well here is the code for in memory collection

C#

[WebService(Namespace = "http://tempuri.org/")]
[WebServiceBinding(ConformsTo = WsiProfiles.BasicProfile1_1)]
[System.Web.Script.Services.ScriptService]
public class TestService : System.Web.Services.WebService {
    //it is a static array which will be available at the application scope. Not session based.
    private static string[] suggestions;

    [WebMethod]
    public string[] getSuggestions(string prefixText, int count)
    {
        //static array will be populated only one time.
        if (suggestions == null)
        {
            setSuggestions();
        }
        return suggestions.Where(x => x.StartsWith(prefixText)).ToArray<string>();
    }
    //if some instance the list corrupted can be reset.
    [WebMethod]
    public void ResetSuggestions()
    {
        setSuggestions();
    }
    private void setSuggestions()
    {
        IList<string> templist = new List<string>();
        //have your xml reading code here to populate the array.
        templist.Add("happy");
        templist.Add("world");
        suggestions = templist.ToArray();
    }
}

But the important thing

 don't just rely on the code blindly. Follow my guidelines above, read more and decide which one fit for your case. Try yourself and then come back if you have any problem with code.