C# Discussion Boards - CodeProject

BillWoodruff17-Jul-13 17:25

17-Jul-13 17:25

Dave, fyi, the expression "do the needful" is a Hindi-English idiom often used in correspondence; in this case it may not have the meaning of "gimme codez" ... or, it might Smile | :)

yours, Bill

“Human beings do not live in the objective world alone, nor alone in the world of social activity as ordinarily understood, but are very much at the mercy of the particular language which has become the medium of expression for their society. It is quite an illusion to imagine that one adjusts to reality essentially without the use of language and that language is merely an incidental means of solving specific problems of communication or reflection." Edward Sapir, 1929

Dave Kreskowiak17-Jul-13 18:00

Dave Kreskowiak

17-Jul-13 18:00

I know where it's from and it can be traced back to about the 1700's in British English.

But, since the OP doesn't specify anything at all about what he's looking for, I just had to guess and us an "if" in preface to the most likely possibility.

A guide to posting questions on CodeProject[^]

Dave Kreskowiak

BillWoodruff17-Jul-13 18:11

17-Jul-13 18:11

Hi Dave,

I am delighted to know you are familiar with such constructions (the more common form, I believe, is "do the necessary") [^]. I'm not sure whether, technically, these (now archaic) forms qualify as calques, and/or Anglicisms.

Have you lived in India, or is there a family connection with the Raj ?

Hope my response to you did not seem patronizing Smile | :)

yours, Bill

Dave Kreskowiak18-Jul-13 2:09

Dave Kreskowiak

18-Jul-13 2:09

BillWoodruff wrote:
Have you lived in India, or is there a family connection with the Raj ?

Oh God, no! I just went and did the homework to find out my it's so popular in India, and, to a lesser extent, South Asia English.

BillWoodruff wrote:
Hope my response to you did not seem patronizing

No, no. We're good! Laugh | :laugh:

A guide to posting questions on CodeProject[^]

Dave Kreskowiak

BillWoodruff17-Jul-13 17:28

Using Binary Serialization and GZip ?

17-Jul-13 17:28

Hi Meetnneel,

Please describe what you have done so far, post the code you have written so far, and discuss what specific problems you are having with it.

yours, Bill

BillWoodruff17-Jul-13 3:14

17-Jul-13 3:14

I am successfully using .NET's BinaryFormatter to serialize a very complex object (an instance, at run-time) into a MemoryStream, and I can save that to a file, and then de-serialize the file back into an instance of the object.

Then, I wanted to add compressing the MemoryStream using GZip [1], before saving it to a file.

I am, without compile- or run- time error, using .NET's GZip facility to GZip the created MemoryStream, and then save it to a disk file.

To summarize the save-to-file sequence:

1. use BinaryFormatter to serialize the object to a MemoryStream

2. use GZip to compress the MemoryStream

3. write the Gzipped MemoryStream to disk

I've already learned the hard way that there is a quirk in using GZip in .NET in compressing [^].

I verify that the file is written, and it appears to have been "shrunk" by the GZip facility.

But, when I try to reverse the process, to essentially deserialize (recreate) the object by:

1. reading the Gzipped saved file into a MemoryStream

2. using GZip's Decompress on the MemoryStream

3. using BinaryFormatter to deserialize the MemoryStream, which I then cast into the Type of the original object: fail.

The point I am stuck at is expressed in this code:

// un GZip

// at this point 'newMemoryStream holds the result
// of opening the saved serialized/Gzipped file

using (GZipStream deCompressionStream = new GZipStream(new MemoryStream(), CompressionMode.Decompress))
{
   // here 'deCompressionStream is 'CanRead = 'true, CanWrite = 'false
   // Length property examined at break-point indicates some Exception
   // so you can't write anything into 'deCompressionStream
   // like this:
   // newMemoryStream.CopyTo(deCompressionStream);

   // yes, this will not fail, but doesn't make sense
   // it's inconsistent with the way that Gzipping compression works !
   deCompressionStream.CopyTo(newMemoryStream);
}

BinaryFormatter bFormatter = new BinaryFormatter();

// if you try this using the "doesn't make sense"
// attempt above:
// you'll get a 'stream was not readable' error

object obj = bFormatter.Deserialize(newMemoryStream);

Thanks in advance for any advice.

yours, Bill

[1] I set out to explore using GZip as a way to increase my general knowledge of .NET's stream facilities, and after learning how to successfully use Mehdi Gholam's fastBinaryJSON work here on CP, and using the MiniLZO class in Mehdi's Raptor Document Store.

Richard Deeming17-Jul-13 4:02

Richard Deeming

17-Jul-13 4:02

You're passing an empty MemoryStream to the GZipStreams constructor. You need to pass the stream containing the previously compressed data instead.

using (var fileStream = File.OpenRead(@"Path\To\Your\File.bin"))
using (var deCompressionStream = new GZipStream(fileStream, CompressionMode.Decompress))
{
   var bFormatter = new BinaryFormatter();
   object obj = bFormatter.Deserialize(deCompressionStream);
}

"These people looked deep within my soul and assigned me a number based on the order in which I joined."
- Homer

BillWoodruff17-Jul-13 15:07

17-Jul-13 15:07

Richard, thanks very much for your example !

Your deserialization code is one of the first variations I tried, and it fails to deserialize with an error: "The magic number in GZip header is not correct. Make sure you are passing in a GZip stream."

Clearly my Gzipping is not being done properly. This is confirmed by this experiement: if I modify your code like this:

using (var fileStream = File.OpenRead(filePath))
using (var deCompressionStream = new GZipStream(fileStream, CompressionMode.Decompress))
{
    var bFormatter = new BinaryFormatter();

    // put a break-point here: examine obj
    // this works !
    object obj = bFormatter.Deserialize(fileStream);

    // for testing only
    return null;
}

That confirms to me I did not GZip properly.

I tried several variations of GZipping found on MSDN, and StackOverFlow: almost all these examples use source.CopyTo(destination) in some form, within the using statements bracketing the use of GZipStream, evidently for the reason there is a known problem with GZipping not being completed fully until after it exits the 'using statement closure, as referenced here: [^].

So, the burden shifts back to finding the correct way for Gzipping.

thanks again for your time, Bill

Eddy Vluggen17-Jul-13 5:07

Eddy Vluggen

17-Jul-13 5:07

You're copying streams which isn't required; they're based on the decorator-pattern. Below example should work (albeit the GZipped version will be bigger since it's a small class)

using System;
using System.Runtime.Serialization.Formatters.Binary;
using System.IO;
using System.IO.Compression;

[Serializable()]
class SumTestClass
{
    public int TestInt { get; set; }
    public string TestString { get; set; }
    public SumTestClass()
    {
        TestInt = 42;
        TestString = "Hello World";
    }
}

static class Program
{
    static BinaryFormatter bf = new BinaryFormatter();
    static void Main(string[] args)
    {
        object thingyToSerialize = new SumTestClass();

        // ..or use a memorystream :)
        using (var ms = new FileStream("C:\\Temp\\Test.zip", FileMode.Create)) 
        using (var zs = new GZipStream(ms, CompressionMode.Compress, leaveOpen: true))
        {
            bf.Serialize(zs, thingyToSerialize);
            Console.WriteLine("Stuff saved");
        }
        
        using (var ms = new FileStream("C:\\Temp\\Test.zip", FileMode.Open)) 
        using (var zs = new GZipStream(ms, CompressionMode.Decompress, leaveOpen: true))
        {
            SumTestClass testResult = bf.Deserialize(zs) as SumTestClass;
            Console.WriteLine("String: {0}\nInt: {1}", testResult.TestString, testResult.TestInt);
        }

        Console.ReadLine();
    }
}

Hope it helps a bit, yell if it doesn't Smile | :)

Bastard Programmer from Hell Suspicious | :suss:

If you can't read my code, try converting it here[^]

BillWoodruff17-Jul-13 16:52

17-Jul-13 16:52

Thanks, Eddy !

Your code works with a MemoryStream that has been Gzipped, Serialized, and written to a file.

The 'leaveopen parameter ... in both gzip, and un-gzip ... appears to be redundant: my implementation of your code works with, or without, them.

It compresses a 9mb object=>source-stream to a little over 3mb disk file, whether Compression setting is 'Fastest, or 'Optimal.

"Raising the stakes" from 100k to 1 million total objects yielded a 98+ megabyte MemoryStream that Gzipped to a 30.6mb file with both 'Optimal, and 'Fastest compression. That does make me question the efficiency of Gzip in .NET.

But (hypothesis), perhaps the complex object I am Gzipping and Serializing is, by its nature not highly compressible ?

It is much better to have working code you don't quite understand, than to have code you believe you understand, and expect to work, that does not work Smile | :)

When I feel like I fully understand how I "strayed onto the wrong path" for so many hours, yesterday, and understand exactly why your code works, I feel I should write up a Tip-Trick for CP.

I didn't realize that by using a ".gz" extension to the files I created I was creating something that could be opened with WinRar !

yours, Bill

~

"This isn't right. This isn't even wrong." Wolfgang Pauli"

Eddy Vluggen18-Jul-13 6:32

Eddy Vluggen

18-Jul-13 6:32

You're welcome Smile | :)

BillWoodruff wrote:
"Raising the stakes" from 100k to 1 million total objects yielded a 98+ megabyte MemoryStream that Gzipped to a 30.6mb file with both 'Optimal, and 'Fastest compression. That does make me question the efficiency of Gzip in .NET.

Alternatively, one could launch an external process and wait for the command to finish. And suddenly, one can easily switch out the ZIP and try an ARJ, or a LHArc Smile | :)

BillWoodruff wrote:
I feel I should write up a Tip-Trick for CP.

Please do

Bastard Programmer from Hell Suspicious | :suss:

If you can't read my code, try converting it here[^]

BillWoodruff18-Jul-13 17:00

18-Jul-13 17:00

Eddy Vluggen wrote:
Alternatively, one could launch an external process and wait forthe command to finish. And suddenly, one can easily switch out the ZIP and try an ARJ, or a LHArc

I have thought about experimenting with launching something like 7-Zip via Process.Start, as suggested (but never detailed) on the 7-Zip discussion forums, but I wonder if, in using any externally launched process, the (possibly required) transfer of the stream to be compressed, or the creation of a "temp" file to hold the stream, before invoking the external process' facilities, would add an "overhead" that would negate its benefit.

I have SharpCompress, found here on CodeProject [^], working, but am not seeing any radical improvement in the current test case over .NET's built-in GZip.

Yesterday I spent a few hours trying to get SharpZipLib [^] working in VS 2012, FrameWork 4.5, without success. But, I'll try that again today.

thanks, again, Bill

~
“This isn't right; this isn't even wrong." Wolfgang Pauli, commenting on a physics paper submitted for a journal

modified 19-Jul-13 1:10am.

Eddy Vluggen19-Jul-13 7:30

Eddy Vluggen

19-Jul-13 7:30

BillWoodruff wrote:
would add an "overhead" that would negate its benefit.

To answer that, one needs to investigate both overhead and benefit. Yes, it does introduce a slight overhead - but not very much if you hide the console-window. Consider it an isolated, switchable, heavyweight instance of a thread.

BillWoodruff wrote:
SharpCompress, SharpZipLib

Let's also add DotNetZip [^]. And LZMA[^].

Bastard Programmer from Hell Suspicious | :suss:

If you can't read my code, try converting it here[^]

harold aptroot18-Jul-13 7:17

harold aptroot

18-Jul-13 7:17

See http://www.virtualdub.org/blog/pivot/entry.php?id=335[^] for how much it sucks.
There are various 3rd party deflate libs that suck less, I recommend SharpZipLib (nice license, too).

Though personally I recommend not using the built-in serialization in the first place and just dumping your raw data in a stream. It's a little more work up-front, but it's more flexible, a lot faster, and your data format will actually make sense (which is not always important of course). You can still put it through Deflate of course, I tend to do that (easier than explicitly storing sparse arrays and such).

BillWoodruff18-Jul-13 17:04

18-Jul-13 17:04

That's an excellent analysis of GZip's limitations, in the article you cite, on virtualdub.com !

As I just commented to Eddy (post above): "Yesterday I spent a few hours trying to get SharpZipLib working in VS 2012, FrameWork 4.5, without success. But, I'll try that again today."

thanks, Bill

~
“This isn't right; this isn't even wrong." Wolfgang Pauli, commenting on a physics paper submitted for a journal

Ennis Ray Lynch, Jr.17-Jul-13 7:11

Ennis Ray Lynch, Jr.

17-Jul-13 7:11

GZip Stream writes information to the stream after it is "closed" when creating the compressed zip stream. Do not wrap your compression in the using block. Call flush before closing out the stream when compressing. Then your decompress code may work.

Need custom software developed? I do custom programming based primarily on MS tools with an emphasis on C# development and consulting.

"And they, since they Were not the one dead, turned to their affairs" -- Robert Frost

"All users always want Excel" --Ennis Lynch

BillWoodruff17-Jul-13 17:02