Welcome to the Lounge

User 274316210-Dec-17 20:34

User 2743162

10-Dec-17 20:34

Buy an SSD already Wink | ;)

modified 19-Nov-18 21:01pm.

kmoorevs11-Dec-17 6:34

kmoorevs

11-Dec-17 6:34

But it already has an SSD...and it still took over an hour! Laugh | :laugh:

"Go forth into the source" - Neal Morse

Johnny J.11-Dec-17 3:31

Johnny J.

11-Dec-17 3:31

kmoorevs wrote:
you may want to put this update off until you have plenty of time.

I don't think you get to make that decision - I know that I didn't! Mad | :mad:

charlieg11-Dec-17 9:15

charlieg

11-Dec-17 9:15

Working on an update?

Charlie Gilley
<italic>Stuck in a dysfunctional matrix from which I must escape...

"Where liberty dwells, there is my country." B. Franklin, 1783
“They who can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety.” BF, 1759

kmoorevs11-Dec-17 9:41

kmoorevs

11-Dec-17 9:41

charlieg wrote:
Working on an update?

You mean an update on the update? If so, then the sucker managed to finish after about an hour. Then after logging back in, I am presented with 'We have updates' and have to wait another 5 minutes until I can use the computer. Laugh | :laugh:

(It usually boots to usable in < 30 secs.)

The only thing broken (afaik) were hklm registry permissions that apparently WU keeps resetting for me as I've had to fix them after every major update. WTF | :WTF:

Other than that, no problems. Smile | :)

"Go forth into the source" - Neal Morse

FLISP ?

PIEBALDconsult10-Dec-17 7:15

10-Dec-17 7:15

(Almost a programming question.)

Most of my current work involves ETLs moving data from place to place (currently in SSIS Dead | X|

).
Often that means copying files from place to place -- primarily copying from NAS to local disk, but also extracting from a ZIP or TARBALL to local disk.
Sometimes it means determining which of a group of files is newest and giving it a generic name when it's copied -- e.g. foo_2017-12-10.csv ==> foo.csv

I find myself on the brink of defining a file management scripting language so we can describe the desired action in a high-level way.
Among the various techniques we've used so far include:
0) Using SSIS' built-in file handling component which is very limited in its abilities.
1) Writing a C# script and either:
1.1) Using System.IO.File.Copy and its ilk.
1.2) Using a System.Diagnostics.Process to execute COPY and 7ZIP commands Dead | X|

.

What I've come up with so far is beginning to resemble LISP. WTF | :WTF:

EXTRACT ( COPY ( NEWEST ( LIST ( DIR ( "\\serverx\widgetco\" ) , "foo*.zip" ) ) , "foo.zip" ) , "foo.csv" )

There are several other features I am considering, particularly only copying a source file when it is newer than the file currently at the destination and raising an alarn if it's older.

I need to be either pushed over the edge or pulled back.

Do I need to proceed to develop this thing?
Does anyone out there know of some existing tool (with API so I can integrate it into SSIS) that already does this sort of thing?

Daniel Pfeffer10-Dec-17 7:44

Daniel Pfeffer

10-Dec-17 7:44

I vaguely remember a Microsoft tool called BizTalk that could be used for developing routing for files. The idea IIRC was to enter the CRM market.

They used to have a trial edition for this, so testing it should not cost anything out of pocket.

If you have an important point to make, don't try to be subtle or clever. Use a pile driver. Hit the point once. Then come back and hit it again. Then hit it a third time - a tremendous whack.

--Winston Churchill

abmv10-Dec-17 8:09

abmv

10-Dec-17 8:09

You could call PowerShell,if you are on 2012r2/2016 windows server...

Best option for working with Files in SSIS - SQLServerCentral[^]

You could store the processed file names in a db or log the errors to a txt file...

Caveat Emptor.

"Progress doesn't come from early risers – progress is made by lazy men looking for easier ways to do things." Lazarus Long

modified 10-Dec-17 14:18pm.

PIEBALDconsult10-Dec-17 8:40

10-Dec-17 8:40

Sure, but I'd still have to implement a higher-level language, and I'm not about to do that in PowerShell.

And we already log to a database.

To be clear, I have very little experience with PowerShell. However, the idea of using PowerShell as the front-end might be worth investigation.

Edit:
Looking into developing cmdlets now -- it may be the way to go.

modified 10-Dec-17 15:25pm.

Mike Hankey10-Dec-17 8:11

Mike Hankey

10-Dec-17 8:11

Whoa man you're over thinking this thing. Get a grip man this [^] is what you need!

Someone's therapist knows all about you!

PIEBALDconsult10-Dec-17 8:37

10-Dec-17 8:37

I began working on PIEBALDBASIC a few years back, before common sense called a halt.

Mike Hankey10-Dec-17 8:42

Mike Hankey

10-Dec-17 8:42

Common sense or hair lose? Smile | :)

Someone's therapist knows all about you!

PIEBALDconsult10-Dec-17 8:48

10-Dec-17 8:48

I don't anticipate hair loss. My father still has his in his 80s and his father had his into his 90s.

Mike Hankey10-Dec-17 8:50

Mike Hankey

10-Dec-17 8:50

I was thinking more like pulling it out due to frustration?

Someone's therapist knows all about you!

PIEBALDconsult10-Dec-17 9:01

10-Dec-17 9:01

Nah, I give up pretty easily.

Marc Clifton10-Dec-17 9:44

Marc Clifton

10-Dec-17 9:44

PIEBALDconsult wrote:
What I've come up with so far is beginning to resemble LISP.

Ew, no! Consider using a fluent syntax:

Dir("\serverx\widgetco\").List("foo*.zip").Newest().Copy("foo.zip").Extract("foo.csv");

Consider that your nudge over the edge. Wink | ;)

Latest Article - Class-less Coding - Minimalist C# and Why F# and Function Programming Has Some Advantages

Learning to code with python is like learning to swim with those little arm floaties. It gives you undeserved confidence and will eventually drown you. - DangerBunny

Artificial intelligence is the only remedy for natural stupidity. - CDP1802

PIEBALDconsult10-Dec-17 10:28

10-Dec-17 10:28

Not a bad idea, I thought about it and thought maybe it would be harder to parse, but maybe it would be easier. So maybe FLINQ ? OMG | :OMG:

On the other hand, my fevered imagination has spewed forth the possibility of merging two pipelines together:

COPY ( MERGE ( LIST ( DIR ( "\\serverx\widgetco\" ) , "foo*.csv" ) , LIST ( DIR ( "\\servery\acme\" ) , "bar*.csv" ) ) )

But that rabbit hole leads to implementing UNION and INTERSECTION, and my eyes glaze over. YAGNI!!

Marc Clifton10-Dec-17 10:35

Marc Clifton

10-Dec-17 10:35

PIEBALDconsult wrote:
maybe FLINQ ?

Exactly: LINQ and File Directories (C#) | Microsoft Docs

Though I haven't looked into any of their linqs (harhar) it might be a start, throw in some extension methods, and voila.

PIEBALDconsult wrote:
But that rabbit hole leads to implementing UNION and INTERSECTION, and my eyes glaze over.

Heck, why not -- if you stick with IEnumerable<File> or something like that, then Union, Join, and Intersect should "just work".

This could turn into a cool little library. A quick google on "file linq" did not discover any existing implementations other than that first link which seems to be cloned here.

PIEBALDconsult10-Dec-17 13:07

10-Dec-17 13:07

Marc Clifton wrote:
throw in some extension methods, and voila.

Hmmm, interesting idea.

Allow the "script" to be actual calls to the Extension Methods and compile to a .net asembly -- Compiling Source Code from a String[^] -- sounds dangerous. Cool | :cool:

It doesn't require parsing on my part at all. I like it.

Edit:
Further thought leads to considering a class with the required methods and then tack the "script" onto it in a dynamically-compiled class:

using ( Flinq f = new Flinq() ) { try { f.DIR("...").LIST("...") ... ; } catch { f.LOG ( ... ) } }

Need to protect against Little Bobby Tables, but otherwise what could possibly go wrong?

modified 10-Dec-17 23:31pm.