Welcome to the Lounge

2-Jan-20 14:07

After using them and building several of them, I still prefer building my parsers by hand.

The only reason I don't is error handling and maintenance, although maintenance with generated parsers is a double edged sword. For starters, it generates a lot more code than you would if you hand rolled a parser, typically. Second, with a hand rolled parser you can parse directly into your final tree. With a generated parser you must first parse into a parse tree, before converting *that* to your final tree, and so maintenance becomes an issue again because there's that second phase which is usually hand-written. Although to be fair, the data you're working with in the second phase has already been validated and error checked for syntax and structure.

I provide macros to help with the second phase, but it doesn't help that much. It would be nice if i could make a grammar description that could completely shape a tree, and create arbitrary objects off of it, to mimic the creation-into-the-final-tree that hand rolled parsers can do.

I don't like that the generated parser will take my codedomgokit source from 500KB to about 1.5-2MB

But I also don't like how sketchy my hand rolled parser is.

Is the extra 1+MB worth it? I guess that's probably 400-500k compiled based on previous experience. Currently CodeDOM Go Kit is about 250K compiled.

I don't know. It's something to think about. I really want a Better Way(TM) - something that handles that second phase and rolls it into the first.

When I was growin' up, I was the smartest kid I knew. Maybe that was just because I didn't know that many kids. All I know is now I feel the opposite.

Greg Utas2-Jan-20 15:04

Greg Utas

2-Jan-20 15:04

honey the codewitch writes:
I still prefer building my parsers by hand.

I even built lexical code by hand, so no arguments from this 🦕.

With a generated parser you must first parse into a parse tree, before converting that to your final tree...

🤮

I provide macros to help with the second phase...

Naughty! I'd deprecate macros with prejudice. No, not even deprecate: they'd just vanish overnight. I've only seen one instance where they were used elegantly: to generate assembly opcodes of all things. Maybe I've led a sheltered life. But I'll confess that I never "got" templates until I read a comment describing them as "macros on steroids".

But I also don't like how sketchy my hand rolled parser is.

Would you like to borrow my rubber 🦆, who's a refactoring taskmaster? He promises to be gentle.

Quote:
Is the extra 1+MB worth it?

I'd bet this is a problem with all non-trivial generated code.

honey the codewitch2-Jan-20 15:10

2-Jan-20 15:10

I'm glad you read this post. I was thinking of you when I wrote it. I enjoy our conversations, as we have similar passions, it seems.

My "Macros" are actually type checked. After the code gets parsed and dumped into an AST then I go back and resolve macros and I do it in a type safe manner. They resolve to function calls. It isn't a simple preprocessor, so doesn't have a lot of those traditional drawbacks.

Sure, templates are "macros on steroids" but more importantly perhaps, they're *type checked* "macros".

I love templates in C++. Once you learn Generic Programming(TM) you never go back unless you have to.

By the way, you might want to consider using a lexer like flex or lex for your parser.

The reason is speed, not just ease.

If you learn how to use it (the learning curve isn't big, lex is stupid simple and mostly you write it in C/C++ code) - it will do things like shortest string discovery. It uses these optimizations to skip examining characters in the input stream, speeding up the parse. This is near impossible in a hand built lexer, because you need a lot of tabular data to do it properly.

It also makes your parser less buggy than hand lexing, in my experience. Evaluating tokens is a hell of a lot less error prone than evaluating individual characters, although I'm only assuming you're doing scannerless/lexerless parsing.

When I was growin' up, I was the smartest kid I knew. Maybe that was just because I didn't know that many kids. All I know is now I feel the opposite.

Greg Utas2-Jan-20 15:59

Greg Utas

2-Jan-20 15:59

If you could recommend a link for lex or flex, I'll take a look, though it'd probably be hard to retrofit it at this point. The shortest string stuff sounds nice. Even worse is having to back up and re-parse an entire identifier. I ran into this today while writing code to support a label that's the target of a goto. Not that they're used much, but goto is on the list of keywords that I never got around to supporting.

My lexer code is very string oriented, so there aren't distinct lexer and parser phases, which is what I assume you mean by "scannerless/lexerless". It's probably a trade-off between things like the shortest string optimization that you mentioned, and the ability to override rules (e.g. to allow punctuation in a name so that a template instance can be referred to by its verbatim name rather than some mangled monstrosity).

honey the codewitch2-Jan-20 16:13

2-Jan-20 16:13

The man pages are actually VERY good. But also Using <CODE>flex</CODE> - An Overview of @code{flex}, with Examples[^]

That's a very basic tutorial on using flex.

Download Flex[^]

Now, here's something for some reason nobody really talks about.

In practice, forget those samples. They give you basics but nothing real world. Real world though, is even easier.

Have a set of constants for all your symbols. If you intend to parse >= you need a GREATERTHANOREQUAL/GTE constant. Give it an int value. Do it for all your terminals. It doesn't matter what the values are. You're going to use these to distinguish your tokens.

In your scanner:

C++

%option unicode, codepage:raw, out: outputfile.c


%{
// extra code.
%}

%%
<<EOF>>		{ return EOS; }

"int" {return INT_TYPE; }
"uint" { return UINT_TYPE; }
// all keywords BEFORE IDENT
(_|[[:IsLetter:]])(_|[[:IsLetterOrDigit:]])* { return IDENT; }


// ignore whitespace
[\r\n\t\v\f ] 

// capture anything we didn't get upstream and report it as an error
[\n]|[^\n] { return ERROR; }

basically you just return a constant a value for every token you capture. You can do more - whatever code you need in those brackets is fine. I use them for building my skip lists inside those little brace breakouts above where we return consts. Like
{ yyskip(LINE_COMMENT); }
since i didn't return a value, flex simply "hides" it. It only gives you something back when you return. otherwise it just keeps going. yyskip() is a function I made that would have been included in the "extra code" part above. It adds the current value to my skiplist.

Oh to get access to these values you use int symid = yylex();
and the text returned is in yytext

It's all pretty simple. It's a bit of work defining your tokens and consts up front but hey, it's a lot less work than parsing them yourself.

And it will make this FAST.

I actually generate these files using Parsley Laugh | :laugh:

When I was growin' up, I was the smartest kid I knew. Maybe that was just because I didn't know that many kids. All I know is now I feel the opposite.

honey the codewitch2-Jan-20 15:54

2-Jan-20 15:54

One thing I forgot to mention to you - regarding skipping comments.

I learned the hard way, if you're going to skip, you need to manually clear the skip list periodically.

I was trying to clear it after every read, and it doesn't work, because trailing items get lost if you aren't checking *everywhere*

I learned the hard way as the way I exposed these things doesn't leave me room to do this. (Part my fault, and part limitations of the lexer generator i'm using - skipped lists weren't really in the design Big Grin | :-D

)

When I was growin' up, I was the smartest kid I knew. Maybe that was just because I didn't know that many kids. All I know is now I feel the opposite.

Mark_Wallace2-Jan-20 23:55

Mark_Wallace

2-Jan-20 23:55

I used to build databases by hand, and use PostScript and mark-up languages (the vast majority of which were far better suited to the job than XML) to produce documents.

... But then better tools came along, which allowed me to spend my time and energy on doing the work, rather than on setting up the infrastructure for it.

I wanna be a eunuchs developer! Pass me a bread knife!

Visual Studio and WINE?

honey the codewitch2-Jan-20 12:00

2-Jan-20 12:00

There's one main reason I don't switch back to linux and that is Visual Studio. MonoDevelop is good, but not great. Linux C++ IDE offerings are bunk on linux even if gcc itself is good. KDevelop had potential way back when but oh well.

Has anyone ever tried to get Visual Studio (not VSCode!) running in linux in something other than a virtual machine?

I don't want all that hassle of running windows at all if I don't have to.

I just got my computer back from microsoft. I had to wait for it to install some unplanned updates that rebooted the machine 3 times and took almost 10 minutes. And MS says windows boot times are fast. Mad | :mad:

When I was growin' up, I was the smartest kid I knew. Maybe that was just because I didn't know that many kids. All I know is now I feel the opposite.

Kornfeld Eliyahu Peter2-Jan-20 12:07

Kornfeld Eliyahu Peter

2-Jan-20 12:07

The last VS version was better than garbage on wine was 2008... with a bronze... 2005 got gold...

Maybe here you could find something... .NET tools & editors for Windows, Linux and macOS[^] (I'm using VS Code)

"The only place where Success comes before Work is in the dictionary." Vidal Sassoon, 1928 - 2012

Eddy Vluggen2-Jan-20 12:55

Eddy Vluggen

2-Jan-20 12:55

honey the codewitch wrote:
Has anyone ever tried to get Visual Studio (not VSCode!) running in linux in something other than a virtual machine?

Little chance for that; too much components in there that will fail the install.

honey the codewitch wrote:
I had to wait for it to install some unplanned updates that rebooted the machine 3 times and took almost 10 minutes. And MS says windows boot times are fast.

..an update is not a normal boot. I love my Kubuntu-machine, but to be fair, I also love the VS on Windows, and it certainly doesn't take 10 minutes for a normal boot.

In Linux you'd be loosing more time after each update, since not every packages' dependence will update nicely, going through weird errors and copy/pasting a lot of commands in the console window.

Bastard Programmer from Hell Suspicious | :suss:

If you can't read my code, try converting it here[^]
"If you just follow the bacon Eddy, wherever it leads you, then you won't have to think about politics." -- Some Bell.

honey the codewitch2-Jan-20 12:59

2-Jan-20 12:59

Eddy Vluggen wrote:
In Linux you'd be loosing more time after each update, since not every packages' dependence will update nicely, going through weird errors and copy/pasting a lot of commands in the console window.

You know, for some reason linux and I just get along. I don't tend to have a lot dependency problems, and I understand the various package tools pretty well.

It tends to be less cranky for me than windows overall. Usually if it doesn't work it's because i did something wrong. Windows won't work if I look at it sideways.

Not that I'm down on windows. I love windows, but I get sick of windows' $*(#@& sometimes and I want a good old ubuntu machine or something.

When I was growin' up, I was the smartest kid I knew. Maybe that was just because I didn't know that many kids. All I know is now I feel the opposite.

Eddy Vluggen2-Jan-20 13:38

Eddy Vluggen

2-Jan-20 13:38

honey the codewitch wrote:
Usually if it doesn't work it's because i did something wrong.

That's not what most people report, as you already know.

honey the codewitch wrote:
I love windows, but I get sick of windows' $*(#@& sometimes and I want a good old ubuntu machine or something.

And you can, but Linux is not stable enough for VS.

I'd be surprised if VS came past the installation of .NET; it's not a question of whether it runs, it won't even install.

Bastard Programmer from Hell Suspicious | :suss:

If you can't read my code, try converting it here[^]
"If you just follow the bacon Eddy, wherever it leads you, then you won't have to think about politics." -- Some Bell.

theoldfool2-Jan-20 13:51

theoldfool

2-Jan-20 13:51

Wouldn't cost you anything to try it with a trial of codeweavers. I choose to run it in a W10 VM in Mint. Have not had any stability problems with many versions of Linux over the past 4 or 5 years. I have been using it for servers and my main system for some 10 years or more. Used to be problematic on my old Thinkpad but ran OK if I didn't suspend it. That was some 5 years ago. Run a MacBook now, with Fusion. VM's run fine.
I don't hate Windows... I don't hate anything. I am just a contrarian. Smile | :)

Lou

If you can keep your head while those about you are losing theirs, perhaps you don't understand the situation.

honey the codewitch2-Jan-20 13:56

2-Jan-20 13:56

I'll maybe try a VM since my latest machine has hardware support for it, but it doesn't have a lot of RAM which is why I wanted to avoid running one.

I don't think linux has stability issues, though some of the other commenters think that. I think it's more of an issue of the stability and interoperability of the packages you install. That's really what's at issue since linux is less of a "closed loop" than windows is, there's more opportunity to *#$%@# it up. That's not an issue though if you know what you're doing. It's a bit like coding in C++ vs. C#. I like both. C++ is more powerful but when you screw up you blow your whole leg off.

And for the record, i love windows. I even wrote a small part of it. (Click on "This PC" in a file explorer, and then Manage...)

But it can frustrate me. Linux is more my speed, but Visual Studio is my jam. Monodevelop and the various other IDEs on linux just don't cut it. I've been thinking of looking into JetBrains Ride to see how good it is, but I don't know if I want to buy yet another IDE after I've already invested so much into the VS stack, you know?

When I was growin' up, I was the smartest kid I knew. Maybe that was just because I didn't know that many kids. All I know is now I feel the opposite.

Eddy Vluggen3-Jan-20 2:40

Eddy Vluggen

3-Jan-20 2:40

honey the codewitch wrote:
though some of the other commenters think that.

Note some, just me, and you are allowed to name me.

honey the codewitch wrote:
That's really what's at issue since linux is less of a "closed loop" than windows is, there's more opportunity to *#$%@# it up.

More dependencies that get changed then under Windows, where they are very reluctant to change anything that may break something, and where they can spend money on keeping stuff working.

honey the codewitch wrote:
That's not an issue though if you know what you're doing.

So, the mismatch in the packages I install, and the updater killing itself is my own fault? Roll eyes | :rolleyes:

Bastard Programmer from Hell Suspicious | :suss:

If you can't read my code, try converting it here[^]
"If you just follow the bacon Eddy, wherever it leads you, then you won't have to think about politics." -- Some Bell.

honey the codewitch3-Jan-20 4:04

3-Jan-20 4:04

> So, the mismatch in the packages I install, and the updater killing itself is my own fault?

As much as my VS2017 installation failing is mine

When I was growin' up, I was the smartest kid I knew. Maybe that was just because I didn't know that many kids. All I know is now I feel the opposite.

dandy723-Jan-20 3:44

dandy72

3-Jan-20 3:44

honey the codewitch wrote:
And for the record, i love windows. I even wrote a small part of it. (Click on "This PC" in a file explorer, and then Manage...)

The MMC host?

honey the codewitch3-Jan-20 4:02

3-Jan-20 4:02

no. the computer management snap-in for the MMC host

When I was growin' up, I was the smartest kid I knew. Maybe that was just because I didn't know that many kids. All I know is now I feel the opposite.

#realJSOP2-Jan-20 14:24

#realJSOP

2-Jan-20 14:24

Not possible, because VS is a WPF application.

".45 ACP - because shooting twice is just silly" - JSOP, 2010
-----
You can never have too much ammo - unless you're swimming, or on fire. - JSOP, 2010
-----
When you pry the gun from my cold dead hands, be careful - the barrel will be very hot. - JSOP, 2013

Mark_Wallace2-Jan-20 23:57

Mark_Wallace

2-Jan-20 23:57

#realJSOP wrote:
Not possible, because VS is a WPF application.

"because shooting twice is just silly"

I wanna be a eunuchs developer! Pass me a bread knife!