|
Well Bill, I respectfully disagree with your disagreement. This forum is for questions on C#, not reviews of people's projects. And there is a perfectly adequate Database forum further down, which is still fairly active at times.
|
|
|
|
|
@chrismaunder
i do see your POV !
for me, the salient questions are:
1) if (your words) "This forum is for questions on C#" ... how does that make it any different from the C# QA forum ? why have two fora ?
2) if there's all kinds of questions that appear here that could just as well be appropriate on the C# QA forum, and that's okay, then why not tolerate other types of posts that do have some C# related content, like Michael's ?
ideally ... for me ... this would be a forum for serious discussions of C# language issues: new features; work-arounds, optimal techniques, and yes, code reviews. Questions/problems ? Yes: but, hopefully those that invite in-depth discussions and debate. Kevin Marois might have to use QA to get his code written for him 
But, as i said to Honey,,,Witch on the Insider News forum recently: "i can't afford 'ideals' at my age; i have to get by living off 'values'."
And, speaking of "values:" allow me to propose that when we respond to a post like Michael's that may, indeed, be an "edge case" for content on this forum, based on perceptions that he's smart, that he's written some CP articles, that with encouragement/recognition (who doesn't want that ?) he may bring more of his knowledge and experience to CP ... that's valuable.
i call that value "collegialty."
cheers, past-his-use-by date-Bill
«The mind is not a vessel to be filled but a fire to be kindled» Plutarch
|
|
|
|
|
1. The two are different, and you have to go back a few years to when QA was introduced to understand why it was introduced.
2. I did not see anything related to C# in the original question.
Michael Balloni wrote: My first time posting on this forum.
So why say that when he has been, as you pointed out, a contributor of many years standing? And that was why I suggested he was in the wrong place.
|
|
|
|
|
Richard MacCutchan wrote: go back a few years to when QA was introduced to understand why it was introduced.
If you have time, i'd appreciate knowing your perceptions/memories of "why."Richard MacCutchan wrote: 2. I did not see anything related to C# in the original question.
i interpreted this more narrowly: as meaning he had not posted in this forum before. But, i agree that his database article should have been mentioned, and that might have made the post less "ambiguous."
cheers, Bill
«The mind is not a vessel to be filled but a fire to be kindled» Plutarch
|
|
|
|
|
BillWoodruff wrote: perceptions/memories of "why.
IIRC the idea was to attract 'quick' questions. That is to say ones that could be answered fairly easily without long discussion threads as tend to exist in the forums.
|
|
|
|
|
Hi, i'm not trying to go "legal" on you, and i am getting all i ever hoped for, and more  , from CP; but, in your opinion, is it optimum to have (what i perceive as) the kind of overlap we have now ?
If i am the only one beating the gong, i'll just adjust my attitude and perceptions.
cheers, Bill
«The mind is not a vessel to be filled but a fire to be kindled» Plutarch
|
|
|
|
|
Well it is what Chris and the team wanted, and in some ways it works well for both questioner and responder. But the problem with an open forum is getting all the users (myself included) to follow the rules correctly.
|
|
|
|
|
... understood ... thanks !
i've been told i have a real gift for thinking up things other people should change 
«The mind is not a vessel to be filled but a fire to be kindled» Plutarch
|
|
|
|
|
BillWoodruff wrote: a real gift for thinking up things other people should change
I think we can all do that.
|
|
|
|
|
Hi Michael, i'm surprised you didn't mention your 2021 CP article: [^] to which i and other people responded.
imho, your GitHub site has no Wiki; the intro (.md file) is a single statement that does not effectively "advertise" what the project is about, and motivate the viewer to explore it. i suggest adding much more information for the visitor. And, aren't there ways to promote a GitHub site ?
i have no quibble with your posting this here; over the years, this forum has mutated from being a place for C# language issue discussion to being a QA forum, and tolerating whatever else walks through the door.
There is a (relatively inactive) DB forum here: [^] ... why not post there, also ?
imho, code-review and concept-review, to any real depth, are rara avis in this section of the zoo (code-monkey-house ?) That's not a complaint ! Perhaps StackExchange's Code Review forum [^] might be fertile ground for you ?
Wish I had the database experience and depth (and youth !) to evaluate your implementation, and ask intelligent questions.
cheers, Bill
«The mind is not a vessel to be filled but a fire to be kindled» Plutarch
modified 17-Oct-21 5:44am.
|
|
|
|
|
Hey Bill,
Thanks for taking the time. Sorry this is so late, I don't know why CP wasn't emailing me with y'all's responses. Just got a comment here about the general premise, tried to speak to that. Yeah, I haven't fleshed out the GitHub "sites" at all yet, was hoping to get some direction based on my question as to how this tool might best be used, and a code review is always much appreciated. I posted to the Database group, no response there. I'm porting it to C++ for fun, that's been a blast, should make for another fun article.
Cheers, -Michael
|
|
|
|
|
Hi Michael,Michael Sydney Balloni wrote: was hoping to get some direction based on my question as to how this tool might best be used
i think that without more of an overview of the project, and summary details of its architecture, api's, and work-flow, its features, its limitations, etc., you may not get the responses that would be useful.
imho, JSchell's reaction, posted here, probably reflects the naturally skeptic attitude people with significant db experience will take tom such innovation.
People like me, without pro-level db experience, may compare what you are doing with Mehdi Gholam's RaptorDB here on CP: [^] a major project developed over years, with extensive performance testing/timing.
If your project has a limited range of functionality ... and, i understand what that is ... and, performance in its range of use-cases is faster, more efficient, easier to use ... that's fine with me !
«The mind is not a vessel to be filled but a fire to be kindled» Plutarch
|
|
|
|
|
|
Michael Sydney Balloni wrote: general premise
"A simple, easy to use, highly productive,"
My feedback is that I doubt that statement.
Mainstream databases have massive amounts of resources available to them. Productivity for the persisted data is often limited by the complexity of the requirements and how to fit that complexity into the generalities of the common persisted data solutions. However that complexity is what one allows one to craft solutions in the first place without doing a deep dive into how persisted data might solve be used to solve that.
Every 'new' persisted data store solution that I have seen introduced in the past 10 years or so has been done so to support a very specific type of data driven need and none of them are actually anything that could not at least be implemented by traditional mainstream persisted solutions (naturally one might suppose there are speed/cost differences.)
Certainly nothing in the above suggests what is the exact need that you think your solution will address? Does it attempt to replace enterprise Oracle installs? Or is it just another way to manage configuration information for a single app? Maybe it is supposed to compete with Elasticsearch because it is faster? Or something completely new that has only been identified recently by changing user driven needs?
|
|
|
|
|
Thanks for having a look, I really appreciate it.
Taking apart the premise...
- File-based is simpler than server-based.
- Four operations are easier to use than dozens.
- And something simpler and easier to use is more productive.
4db.net builds on SQLite, and provides the database functionality needed for basic applications. It's not fast, it's not sophisticated or robust, but if you only need those four statements, 4db.net is the way to go. To see what I'm talking about, check out this small cars-based sample:
4db.net/Program.cs at main · michaelsballoni/4db.net · GitHub
Imagine how much SQLite code that would take, let alone the server setup for MySQL or some other RDBMS.
Sometimes you need a yacht. But sometimes you just need a canoe. What do you think?
|
|
|
|
|
Michael Sydney Balloni wrote: Sometimes you need a yacht. But sometimes you just need a canoe.
and, some people may need to understand what a paddle is, and why a canoe is a good design for a boat
«The mind is not a vessel to be filled but a fire to be kindled» Plutarch
|
|
|
|
|
Michael Sydney Balloni wrote: Imagine how much SQLite code that would take, let alone the server setup for MySQL or some other RDBMS.
Not sure you really understood my point though.
I work on large applications. So the amount of code and even management of the persisted store data systems is not significant because the applications are going to be large regardless.
And even midsize solutions usually have aspirations to be bigger and will likely need more functionality than provided.
So based on the requirements presented here this only works for very small systems. And for those systems any solution is probably going to be adequate.
So as I suggested before without a specifically identified niche I don't see the need for what you are suggesting.
|
|
|
|
|
Point taken. In .NET, coding directly against SQLite is easy and you get great functionality. No brainer there.
So I ported the .NET metastrings / 4db stuff to C++, where coding against the C SQLite library is a pain.
I wrote this:
4db: A Dynamic File-based NoSQL Database for C++
For a C++ programmer wanting basic record persistence and not much else, this is a good alternative to fprintf or hacking SQLite by hand. In that article, in its code, there are a couple wrapper classes for SQLite. Those might have more valuable than 4db. It's been a wild ride...
|
|
|
|
|
I have some folders having files, its count like 40Lac, 70Lac, 1Cr, 1.5cr. And filetype like .png, .xlsx, .txt, .msg, .ico, .jpg, .bmp etc...
Now i want to insert this filename into my database table with its size, but when i implement it take too much time to scanning the folder, and then after it throws memory exception.
can anyone please help me out here, how can i implement this scenario in a better manner.
how can insert faster in table ?
I am using C#.net with PostgreSQL database.
Ankur B. Patel
|
|
|
|
|
First of all, what is a lac and cr?
This is never going to be "fast", more important is that you do it correct. I assume your putting all your sh*t in a blob. Don't, txt should be archived as memo, so you can later use search form DB.
If you just want to archive names and their sizes, read the entire folders' contents and spawn some threads to save chuncks of that.
Bastard Programmer from Hell 
"If you just follow the bacon Eddy, wherever it leads you, then you won't have to think about politics." -- Some Bell.
|
|
|
|
|
|
Yah. So them like Americans with their pounds.
In international communication we use the SI system. If you can't, better learn
Bastard Programmer from Hell
"If you just follow the bacon Eddy, wherever it leads you, then you won't have to think about politics." -- Some Bell.
|
|
|
|
|
Thanks but, I just correct here that forget about lac and cr (it's Lakh and Crore), keep in mind that you have millions of files in a folder and I just want to insert the files name and its size in bytes into the database.
how do I bulk insert? and also keep in mind that system memory doesn't go high.
please suggest to me, how I can do it quickly and securely.
|
|
|
|
|
Pipe a DIR listing to a text file. That'll give you the list you need; which you can then "substring" and load into a table of files names, etc.
Displays a list of files and subdirectories in a directory.
DIR [drive:][path][filename] [/A[[:]attributes]] [/B] [/C] [/D] [/L] [/N]
[/O[[:]sortorder]] [/P] [/Q] [/R] [/S] [/T[[:]timefield]] [/W] [/X] [/4]
[drive:][path][filename]
Specifies drive, directory, and/or files to list.
/A Displays files with specified attributes.
attributes D Directories R Read-only files
H Hidden files A Files ready for archiving
S System files I Not content indexed files
L Reparse Points O Offline files
- Prefix meaning not
/B Uses bare format (no heading information or summary).
/C Display the thousand separator in file sizes. This is the
default. Use /-C to disable display of separator.
/D Same as wide but files are list sorted by column.
/L Uses lowercase.
/N New long list format where filenames are on the far right.
/O List by files in sorted order.
sortorder N By name (alphabetic) S By size (smallest first)
E By extension (alphabetic) D By date/time (oldest first)
G Group directories first - Prefix to reverse order
/P Pauses after each screenful of information.
/Q Display the owner of the file.
/R Display alternate data streams of the file.
/S Displays files in specified directory and all subdirectories.
It was only in wine that he laid down no limit for himself, but he did not allow himself to be confused by it.
― Confucian Analects: Rules of Confucius about his food
|
|
|
|
|
If this is your idea of a backup mechanism, don't bother: it'll never be quick, it'll never be efficient, and it will always risk running out of memory.
Instead, think about using a "proper" backup system which archives the disk as sectors instead of files - it's a lot quicker, a lot more efficient, and much, much safer.
Remember, backups should be air-gapped for safety: DB's and suchlike are just files and as such are just as much at risk from ransomware and any other data (more so, in some cases as they are a "Prime target" for ransomware to exploit).
I use AOMEI Backupper - it has a free Standard version and it allows you to mount the backup images as a disk and retrieve individual files if necessary.
"I have no idea what I did, but I'm taking full credit for it." - ThisOldTony
"Common sense is so rare these days, it should be classified as a super power" - Random T-shirt
AntiTwitter: @DalekDave is now a follower!
|
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
