Re: String manipulation with large strings - C# Discussion Boards

8-Apr-09 22:17

Hey guys, please consider the following code

...
RequestSize = Int32.Parse(FinalLength);

//
// remove the terminator at the start
strMessage = strMessage.Remove(0, 1);

//
// get the individual command
string command = strMessage.Substring(0, RequestSize);

//
// remove command from the message
strMessage = strMessage.Remove(0, RequestSize);

//
// send command away for proccessing
ProcessCommand(command);

This works perfectly, unless the command variable is upwards of 30000 characters long Confused | :confused:

If the command is that big, both the SubString and Remove methods just do nothing. When debugging this and hitting those calls, its like it just breaks away from those methods and then nothing. The UI thread is not getting tied up so its not just taking some time to do it...

How can I fix this?

Thanks

<edit>Please see reply to Rob Philpott for more detail</edit>

Harvey Saayman - South Africa
Software Developer
.Net, C#, SQL
you.suck = (you.Passion != Programming & you.Occupation == jobTitles.Programmer)
1000100 1101111 1100101 1110011 100000 1110100 1101000 1101001 1110011 100000 1101101 1100101 1100001 1101110 100000 1101001 1101101 100000 1100001 100000 1100111 1100101 1100101 1101011 111111

modified on Thursday, April 9, 2009 5:14 AM

Rob Philpott8-Apr-09 22:52

Rob Philpott

8-Apr-09 22:52

Strange, my understanding is that strings can be huge in .NET, so I don't know why this 30,000 ceiling comes in.

I can offer no solution to that, but might suggest that rather than 'Removing' parts from such a huge string you either iterate over it to get what you need or just copy substrings.

Regards,
Rob Philpott.

Harvey Saayman8-Apr-09 23:13

8-Apr-09 23:13

Rob Philpott wrote:
I can offer no solution to that, but might suggest that rather than 'Removing' parts from such a huge string you either iterate over it to get what you need or just copy substrings.

Here's what happens exactly

After i get a server response from a TCP connection, the listener thread sends back all that data via a callback. Now one of these callbacks may have more than one command in it.

Each command starts with ln=[length of command][terminator]

So I loop to find a ln=, when I do I remove it, and then parse the length of the command to come. Then I take that command out (sub string it so i can pass it to another method), and remove it from the pool of commands so that the loop may continue in case there's another command to execute. Here's the full methods code

// This method is called from a different thread
private void ProcessMessage(string CommandPool)
{
    RawResponseReceived(CommandPool);

    while (CommandPool != string.Empty)
    {
        //
        // If the command starts with a ln=
        if (CommandPool.StartsWith("ln="))
        {
            //
            // Remove the ln=
            CommandPool = CommandPool.Remove(0, 3);

            //
            // Get the length of the command

            int RequestSize = -1;
            string FinalLength = string.Empty;

            while (Int32.TryParse(CommandPool[0].ToString(), out RequestSize))
            {
                FinalLength += RequestSize.ToString();
                CommandPool = CommandPool.Remove(0, 1);
            }

            //
            // Parse the request size
            RequestSize = Int32.Parse(FinalLength);

            //
            // remove the terminator at the start
            CommandPool = CommandPool.Remove(0, 1);

            //
            // get the individual command
            string command = CommandPool.Substring(0, RequestSize);

            //
            // remove command from the command pool
            CommandPool = CommandPool.Remove(0, RequestSize);

            //
            // send command away for proccessing
            ProcessCommand(command);
        }
        else
        {
            Console.Write("Char removed - " + CommandPool[0]);
            CommandPool = CommandPool.Remove(0, 1);
        }
    }
}

NOTE: I just changed some of the variable names to make it more readable

The reason these strings are so big sometimes, is that I'm getting binary data in it aswel. Can this be causing the problems?

Rob Philpott8-Apr-09 23:24

Rob Philpott

8-Apr-09 23:24

Hmm. Ok. I wouldn't have thought binary would have caused an issue, but if you're receiving data back from a TCP stream, why don't you process it in a stream manner?

So, rather than waiting for the whole response and turning into a huge string which your iteratively break down into seperate parts, you do this as the response comes in. This way, you don't have the memory footprint of the large string, and you can start processing when the data starts arriving rather than when its finished.

Something like ReadByte() check = 'l', ReadByte() check = 'n', ReadByte() check = '=', then read to terminator to extract the command length, parse that. Then, keep reading each command.

Only a suggestion, and I'm still at a loss why your having problems with your current solution.

Regards,
Rob Philpott.

Harvey Saayman8-Apr-09 23:38

8-Apr-09 23:38

Thanks, I'll have a look at doing it like that...

I'd prefer this way though, it seems easier and less error prone

dojohansen8-Apr-09 23:58

8-Apr-09 23:58

I for one have difficulty believing that what you say happens actually does happen.

But, even if your code is correct and the string you're interested in should eventually be extracted from it you should never ever manipulate large strings in .NET. They are immutable. And that means that whenever you modify a string you are in fact allocating a new string and copying all the not-modified characters from the original. That's not cool.

For an extreme display of the difference in performance, try running this code in a console.

Stopwatch w;

static void time(string msg)
{
   Console.WriteLine(msg + ", elapsed time = " + w.Elapsed.ToString());
   w = Stopwatch.StartNew();
}

static void Main(string[] args)
{
   int count = 1000*1000; 
   string s = "";
   StringBuilder sb = new StringBuilder();
   
   w = Stopwatch.StartNew();
   for (int i=0; i < count; i++) s += ".";
   time("Using string");
   for (int i=0; i < count; i++) sb.Append(".");
   time("Using StringBuilder");
}

You may find you wish to adjust the count so that the "string" implementation will ever finish, although the StringBuilder will do this pretty quickly.

Obviously it is a very extreme example - the statement s += "."; hides an inner loop copying all the characters of the old s into the new s. Even worse, it puts huge pressure on the memory manager as it allocates a million strings (count), and these eventually become huge, forcing the GC to collect and reallocate, collect and reallocate all the time. You will see that with small strings both methods run in a short time, but percentage-wise the StringBuilder will always be a lot faster.

The way the StringBuilder does this internally is by representing the string as a char[] and keeping track of which chars actually have valid data in them. Just like an ArrayList it doubles in size when it reallocates, so starting from 16 (you'd get slightly better performance if you passed the correct capacity to the StringBuilder constructor as then it wouldn't need to reallocate ever) it must reallocate just fourteen times to get to a million, against one million for the string method.

Harvey Saayman9-Apr-09 0:30

9-Apr-09 0:30

Hey there

Yeah i realize the string builder is allot more efficient, i usually just get the concept working using stings and then if necessary replace it with string builders.

To try and fix this problem replaced the string with a string builder, here is the current code

private void ProcessMessage(string CommandPool)
{
    RawResponseReceived(CommandPool);
    StringBuilder sbCommandPool = new StringBuilder(CommandPool);

    while (sbCommandPool.ToString() != string.Empty)
    {
        //
        // If the command starts with a ln=
        if (sbCommandPool.ToString().StartsWith("ln="))
        {
            //
            // Remove the ln=
            sbCommandPool = sbCommandPool.Remove(0, 3);

            //
            // Get the length of the command

            int RequestSize = -1;
            string FinalLength = string.Empty;

            while (Int32.TryParse(sbCommandPool[0].ToString(), out RequestSize))
            {
                FinalLength += RequestSize.ToString();
                sbCommandPool = sbCommandPool.Remove(0, 1);
            }

            //
            // Parse the request size
            RequestSize = Int32.Parse(FinalLength);

            //
            // remove the terminator at the start
            sbCommandPool = sbCommandPool.Remove(0, 1);

            //
            // get the individual command
            string command = sbCommandPool.ToString().Substring(0, RequestSize); //It breaks here

            //
            // remove command from the command pool
            sbCommandPool = sbCommandPool.Remove(0, RequestSize);

            //
            // send command away for proccessing
            ProcessCommand(command);
        }
        else
        {
            Console.Write("Char removed - " + sbCommandPool[0]);
            sbCommandPool = sbCommandPool.Remove(0, 1);
        }
    }
}

the problem persists, when the debugger hits string command = sbCommandPool.ToString().Substring(0, RequestSize); the UI thread becomes active again and the next lines aren't getting executed Confused | :confused:

dojohansen9-Apr-09 1:14

9-Apr-09 1:14

But... when you call ToString().SubString() you are of course using string again.

The fact that the UI thread becomes active is probably no indication of the state of your program. Visual Studio need not hang in order for your program to be busy.

However, doing this a single time on a string of size on the order of thirty thousand characters should not take long at all.

I wrote a simple console app to test and with a string of 30000 characters and taking 28000 of them for the substring it takes about 0.01 seconds on my (cheapass) machine.

class Program
{
    static void Main(string[] args)
    {
        int size = 30000;
        int take = 28000;

        char[] chars = new char[size];
        for (int i = 0; i < size; i++) chars[i] = '.';
        
        string s = new String(chars);
        Stopwatch w = Stopwatch.StartNew();
        s = s.Substring(0, take);
        Console.WriteLine(w.Elapsed);
        Console.ReadKey(true);
    }
}

I'm at a complete loss as to how this can happen. Nor can I reproduce the problem you're having. So it seems like it might be time to reinstall your system... or at least try to take the code to another computer and compile it there and see if you still have the problem.

Should all this fail, you can always attempt using a char[] instead and implement "substring" yourself; after all it's a simple matter.

void substring(char[] src, char[] dest, int startIndex, int count)
{
   for (int i=0; i < count; i++)
     dest[i] = src[i + startIndex];
}

Obviously you should add range checking to that if you want it proper.

dojohansen9-Apr-09 1:25

9-Apr-09 1:25

Re-reading your previous postings makes me wonder if what really happens is that your program terminates. I had not noticed that you're calling this on another thread, but depending on how you do this, what type of application it is, and what framework version you are using, it migth be possible.

I don't know all the different possibilities and how they're handled, but I do remember we created threads ourselves in our asp.net application like this: Thread t = new Thread(new ThreadStart(method)); t.Start(); and this actually killed the application if any unhandled exception occured during method. [EDIT] I forgot to say this was on the 1.x Framework and I believe it has changed since. [/EDIT]

In other words, if we invoked String.SubString and passed any invalid arguments, such as a length that requires going beyond the end of the string (try to take 30001 from the 30000 long string), it would immediately terminate the application. I don't remember how VS would behave if debugging this but it seems possible it would just go all blank and not show any messages or anything.

So you should at least test this hypothesis by making sure you wrap the entire method in a try-block and add a catch at the end where you break into the debugger and dump the exception to debug output, like this:

try 
{
    .... 
}
catch (Exception ex)
{
   Debug.WriteLine(ex.ToString());
   Debugger.Break();
}

dojohansen9-Apr-09 1:37

how to select the text from the dropdownlist

9-Apr-09 1:37

By the way, your signature uses the bitwise and where a logical and is required - no doubt caused by the fact that & is the character that marks an entity reference in HTML, so that && actually becomes && in source... Wink | ;)

mdazeemuddin8-Apr-09 22:03

mdazeemuddin

8-Apr-09 22:03

hi
i have one dropdown list and i get the data from the database in
dropdownlist. now i need to select the data from the dropdown list
and after selecting i need to insert that text into the database.

i try ddlproduct.selecteditem.text where ddlproduct is a drodownlist
id but i get the first value from the dropdown if i select some other
value i am getting the first value only

i forget to tell one thing i am using c#
please help me with some code and example

thanks

Penel in Status Bar [modified]

Sajjad Leo8-Apr-09 21:16

Sajjad Leo

8-Apr-09 21:16

I want to know, how can I,

How can i add penel in status bar on design time not with code in window application.

I tried to add columns through property but i couldn't, i did't get any option regarding to penel in property window.

Please do replay.

thanks

modified on Thursday, April 9, 2009 3:43 AM

DaveyM698-Apr-09 22:09

DaveyM69

8-Apr-09 22:09

It depends on what you mean. I'm assuming you mean the StatusStrip.

If you just want a new label on it, then click it in the IDE and click the dropdown button that appears and choose Status Label.

If you want to add an actual panel to it, then that's a little more difficult. All items hosted in the StatusStrip derive from ToolStripItem. I've never tried this tutorial[^] but it might be what you need.

Dave
BTW, in software, hope and pray is not a viable strategy. (Luc Pattyn)
Visual Basic is not used by normal people so we're not covering it here. (Uncyclopedia)
Why are you using VB6? Do you hate yourself? (Christian Graus)

Sajjad Leo8-Apr-09 23:19

Sajjad Leo

8-Apr-09 23:19

Yeah i am trying Status Strip for status bar where i can show status i want to add penal one left one right.

For doing this what i should do???

DaveyM699-Apr-09 0:11

DaveyM69

9-Apr-09 0:11

Add three labels as I described above and set the second one's Spring property to true and clear it's Text.