|
Hello,
My problem.
we have a legacy unix system that perdiodically writes files to a subdirectory ( a repeating job runs every few minutes and drops the files).
There are sets of files so there may be setA File1to3 and SetB file1to8.
This is absoluely THE ONLY WAY the unix system will transfer the files (no http pushes or web services or anything else)
I have to write an interface that runs a periodic VB net job which will read the contents of the subdirectory and if a complete set is present move that set from the subdir.
Of course my job and the unix job could be running at exactly the same time so I cant just find a find and move it out , because the file may still be being copied from unix as I try and copy it.
I am looking for a safe way of handshaking those files.
I was thinkinng the unix system could write a lock file, which stops my job looking in the subdir,copies the unix files and deletes the locking file.
BUT ok here is where I admit to lying slightly - it isnt one unix system its 4 and there is also 1 windows xp system that works in the same way (sigh!)
So I could use 5 locking files and if any are present not do anything But what if one of those unix systems crashes after writing a lock file , then the whole process is stuffed (er thats a technical term I believe!).
So what I am really looking for is process logic which is a safe and simple way of getting my VB Net interface to read files in sets which have been delivered to the same subdirectory from different sources (and yes it must be the same subdirectory - The network security people have only opened this 1 subdir and no child subdirs are permitted - They must be a different breed of people - still...)
is there a way of telling if a file has finished being copied to a subdir - ie its 12M in length - I look at it whene only 4 M has been copied......
Thanks in advance
Martin
life is a bowl of cherries
go on take a byte
|
|
|
|
|
This will only work if each of the files are written all at once... Your app could look for a complete set of files, and if there isn't one, reset and wait for a bit to check again. If there IS a complete set of files, wait an appropriate amount of time for the files to be completely written.
There are problems with trying to open files to check if another process is writing to them. If the other process opens the files, allowing shared access to it, your attempt at opening the file will always succeed, so that's useless in trying to find out if the other process is finished.
Also, if the other process opens the file, writes some data, closes it, then reopens the file, writes some more data, closes it, ... Rinse and Repeat ad nauseum ... then this method will also fail again because you might successfully open the file between writes by the other app.
You've already pointed out the flaw of using lock files, but it's not as bad as you think. When the unix app restarts, it should NOT look for and kill it's lock file if there is one. It should maintain the lock so the data can be backed out or rewritten, THEN kill the lock file.
MartyK2007 wrote: is there a way of telling if a file has finished being copied to a subdir
There's no "100% safe" way of doing this using files...
|
|
|
|
|
hello,
Dave Kreskowiak wrote: wait an appropriate amount of time for the files to be completely written
can you define and appropriate amount of time. Files can be up to 6M , network drive response could be crap due to bandwidth issues etc.
Also does having 10 files mean the codes waits 10 periods or waits 1 period and hopes all 10 files have finished copying.
if the latter:
What happens if you see set A File 1 , wait 10 minutes (say for this example - I know in reality it will be a lot less) and Set B File 9 starts to arrive in the tenth minute. You can now copy set A file 1 no problem but then Set B file 9 is still copying so you crash whne you try and grab it.
If the former then it slows the overall throughput down dramticaly
Dave Kreskowiak wrote: Also, if the other process opens the file, writes some data
In theory that wont happen in my scenario all, the Set A wont be copied to the directory until all the files have been updated.
Dave Kreskowiak wrote: It should maintain the lock so the data can be backed out or rewritten
that implies that some thought has been given by our unix team to crashes etc and that they have coded something like that - for which I am doubtfull.
It would mean a change to the reboot process of all the unix systems and the windows system. I am not sure I can get the resource authorised for that.(you know how it is)
You dont have a decent "Pattern" for doing this anywhere do you??
Thanks
Martin
life is a bowl of cherries
go on take a byte
|
|
|
|
|
MartyK2007 wrote: can you define and appropriate amount of time.
Nope. That's where you have to define it based on your environment, the number of files in the set, network speeds, ...
MartyK2007 wrote: Also does having 10 files mean the codes waits 10 periods or waits 1 period and hopes all 10 files have finished copying.
Again, that's up to you. Only you know how long it takes for a complete set to show up. How long has the biggest set taken to show up?
Basically, you're looking for an entire set of files to show up before you start copying/moving the files in that set. If you want the ral answers as to how this is going to work, and not work, you're going to have to write some test code to test to see if you can open these files for exclusive access WHILE they're being written to this folder. This is the only way you're going to know if your test will break the other app, or if the technique will work at all.
MartyK2007 wrote: In theory that wont happen in my scenario
Say that again around professional developers and watch how fast they point and laugh at you.
MartyK2007 wrote: that implies that some thought has been given by our unix team to crashes etc and that they have coded something like that - for which I am doubtfull.
This statement alone turns this project into an impossible task to make run "100% safe".
MartyK2007 wrote: You dont have a decent "Pattern" for doing this anywhere do you??
I'd do some testing before I get to any design patterns. It's pretty much a waste of time if the technique you're trying to use doesn't work. Figure out what technique is going to work, then ytou can design a framework around it.
|
|
|
|
|
Hi Martin
We get our mainframe to follow a strict protocol:- FTP (or file-copy) the big data file into the folder (e.g. "MyData20070821_1.dta").
- FTP a zero-sized flag file with the same basename (e.g. "MyData20070821_1.flg").
Our reeeeally cutting-edge VbScript program then periodically looks for flag files in the input folder. It can then safely assume that the associated data file(s) have been successfully uploaded.
I would recommend that the file names should be uniquely-named. Ours weren't so we appended a unique ID number into each data file's name (and we tagged any resulting database records with that id too). We move all processed files into an "archive" folder as an audit trail - and also so we can potentially rerun files if we need to restore our database for any reason.
We prototyped an alternate mechanism a couple of years. We found that FTP placed an exclusive-write lock on the uploading data file - so we added some logic to obtain a read-lock on the file. If our lock failed then we could assume that FTP was still manfully trying to upload the file.
Hope that helps.
Andy
If you want to thank me for my help, please vote my message by clicking one of numbers beside "Rate this message".
|
|
|
|
|
hey
that makes sense and may help
thanks
Martin
life is a bowl of cherries
go on take a byte
|
|
|
|
|
Hi,
My proposal is along the lines of Andy's:
- let the first system create the files of a set, using unique file names
(every OS has a function to provide these, if not include datetime up to
milliseconds and wait 1 millisecond after creating a file)
- when the set is complete, let the first system create a content file
listing all the files of a set;
this file too has a random filename but needs to be recognizable (maybe
by its first char, or by an extension)
- let the second system periodically look for content files; if one is found,
"execute" it, and delete all related files.
- to make sure the content file is complete, have it end on a sentinel
(a fixed line of text, file is invalid if sentinel is missing);
alternatively keep it as short as a single disk sector (512B), then it
can not be a partial file!
Luc Pattyn [Forum Guidelines] [My Articles]
this weeks tips:
- make Visual display line numbers: Tools/Options/TextEditor/...
- show exceptions with ToString() to see all information
- before you ask a question here, search CodeProject, then Google
|
|
|
|
|
hey
thanks luc
thats another good idea
Martin
life is a bowl of cherries
go on take a byte
|
|
|
|
|
I have a report in Crystal that gets data from SQL queries and it's slow. How do I use a stored procedure to make the report access data faster?
|
|
|
|
|
Create the stored procedure to return the data from the query that you are using.
CREATE PROCEDURE [dbo].[MySelectProcedure]
@Parm1 int
AS
SELECT *
FROM MainTable MT
LEFT JOIN MyOtherTable OT ON MT.KeyField1 = OT.KeyField1
WHERE MT.MyParmField = @Parm1
Add parameters if you need them. When you are defining a new report in Crystal, you can select a stored procedure exactly like selecting a table. Add your stored procedure as the datasource. If you used parameters in it, they will show up in your parameters list in the report.
Well written stored procedures are absolutely the BEST datasource that you can use for a Crystal Report.
Tom Garth
Developer
R. L. Nelson and Associates, Inc., Virginia
|
|
|
|
|
How can I configure my setup and deployment project to install a barcode font needed for the application it installs?
|
|
|
|
|
I have one dll(Trial.dll) developed in vb.net 2.0 and I gave a strong name through sn.exe -k Trial.snk
and then I created a tlb file using cmd
regasm -tlb Trial.dll tlb:Trial.tlb
and even
regasm -codebase -tlb Trial.dll tlb:Trial.tlb
now I want this registered tlb or dll to be used in foxpro 9.0
but in foxpro when I go to tools->objectBrower->open TypeLibrary and select my tlb I am not able to see any class or methods or properties...
I don't know why
Please help
Thanks
Mital
|
|
|
|
|
FoxPro can't see a straight .NET DLL. It has to be attributed properly so that it is exposed through COM Interop. That's the only way FoxPro is going to be able to use that DLL.
Exposing .NET Framework Components to COM[^]
|
|
|
|
|
Is this possible to scroll the form's title?
Thanks In advance
Ejaz
|
|
|
|
|
It's not supported natively. Depending on how you want the title to scroll, you'd have to add your own drawing code to the form and override the WndProc method, looking for the window messages to tell you to draw your form's non-client area yourself, including the title bar. Warning! This is NOT for the person who does not know how windows work and are drawn.
|
|
|
|
|
If you are just looking to scroll the form's title then you can do it this way...
*
Add a timer to your form (control or class -- doesn't matter)
Set the interval to something like 500
*Declare 2 module variables:
Private msWindowText As String
Private miScrollPos As Int16 = 1
*Place the form's title in the msWindowText string
* In the elapsed event:
Private Sub tmrFlash_Tick(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles tmrFlash.Tick
If miScrollPos > msWindowText.Length Then miScrollPos = 1
Me.Text = Mid(msWindowText, miScrollPos) & " " & IIf(miScrollPos = 1, String.Empty, Mid(msWindowText, 1, miScrollPos))
miScrollPos += 1
End Sub
|
|
|
|
|
hi
When I import any picture on my project and use it with any object ,
it I del it form Resources solution folder . a problem occure .. how can i deal with it ..
jooooo
|
|
|
|
|
What problem? Be more specific.
_____________________________________________
Flea Market! It's just like...it's just like...A MINI-MALL!
|
|
|
|
|
the problem is >>
you add a picture to a form .. then you indicate the image of it .. the solution add it to Resourses folder . then you del that image from the folder with/without by mistake .. then the programm has error ..
i want to know how to repair that problem withou re add that image to the folder
i hope it's clear
jooooo
|
|
|
|
|
This depends on the version of Visual Studio you're using. But, for the straight forward answer, you have to put the file back in the Resources folder if you want this fixed.
|
|
|
|
|
Hi ..
I designed an applicatoin with two language interface .
then when i changed Language propery . then it's ok .. but when make any changes on the form the VS program hang up ..
but on other solutions it's ok ..
it's only on that solution .
i delete Bin/Debug folder contents and and alos Obj/Debug
but it's still >>>
jooooo
|
|
|
|
|
How do I control a messageboxes results? I know of the msgboxresult.yes/no/ok etc. But I am missing something because when I use this no matter what the result is it still runs the same code when it shouldnt. If someone could please give me a quick example on how to use messageboxes and have different things happen depending on the msgboxresult I would really appreciate it. Thank you...
|
|
|
|
|
Are you using Msgboxresult by itself or comparing what msgbox is returning?
If MsgBox("place question here", MsgBoxStyle.OkCancel) = MsgBoxResult.Cancel Then
End If
|
|
|
|
|
you can try this
If MessageBox.Show("Do you want Exit?", "titleBar", MessageBoxButtons.YesNo, MessageBoxIcon.Question) = DialogResult.Yes Then
'quiting the Application
Else
'Dont Quit
End If
Vuyiswa Maseko
|
|
|
|
|
Dim msgResult As MsgBoxResult = MsgBox("Do you want to SAVE this record?", MsgBoxStyle.YesNoCancel, "DATA HAS CHANGED")
Select Case msgResult
Case MsgBoxResult.Yes
' Save Data
Case MsgBoxResult.No
' Clear the form
Case MsgBoxResult.Cancel
' Do nothing
End Select
Tom Garth
Developer
R. L. Nelson and Associates, Inc., Virginia
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
