Click here to Skip to main content
15,885,782 members

Welcome to the Lounge

   

For discussing anything related to a software developer's life but is not for programming questions. Got a programming question?

The Lounge is rated Safe For Work. If you're about to post something inappropriate for a shared office environment, then don't post it. No ads, no abuse, and no programming questions. Trolling, (political, climate, religious or whatever) will result in your account being removed.

 
GeneralRe: My cat continues to teach me about my computer. Pin
rnbergren4-Jan-21 4:18
rnbergren4-Jan-21 4:18 
GeneralRe: My cat continues to teach me about my computer. Pin
Gary R. Wheeler4-Jan-21 15:45
Gary R. Wheeler4-Jan-21 15:45 
GeneralRe: My cat continues to teach me about my computer. Pin
Rage5-Jan-21 2:28
professionalRage5-Jan-21 2:28 
GeneralRe: My cat continues to teach me about my computer. Pin
Daniel Will5-Jan-21 2:57
Daniel Will5-Jan-21 2:57 
GeneralRe: My cat continues to teach me about my computer. Pin
Member 106770245-Jan-21 5:20
professionalMember 106770245-Jan-21 5:20 
GeneralRe: My cat continues to teach me about my computer. Pin
wreckless5-Jan-21 5:56
wreckless5-Jan-21 5:56 
GeneralRe: My cat continues to teach me about my computer. Pin
Dan Sutton5-Jan-21 5:39
Dan Sutton5-Jan-21 5:39 
GeneralWell that was satisfying... Pin
honey the codewitch3-Jan-21 14:41
mvahoney the codewitch3-Jan-21 14:41 
Edit: To be clear, the reason this stuff below makes me happy is not because I want to be the best at something, but because one of the goals of my project was to rely mostly on cross platform features and downgrade gracefully. The only platform specific magic i use is mem mapped files, for which my code supports windows and linux currently. I set out to prove that you could largely rely on the stdlib and just design algorithmically better ways to process JSON and I feel I did that. I also think that given the portability and memory usage of my library, it pays for itself even in areas where it slightly underperforms simdjson in raw speed. certain cases - many in fact - will see simdjson slay my library, particulary with random access, which mine can't even do, but for the scenario i designed it for, i designed it to be competitive with fast offerings and it exceeded my expectations so far. I need more benchmarking though

I'm looking at possibly integrating my JSON(C++) bulk loader with simdjson. I've been talking with some of the contributors, but we've been running into a wall because of their reliance on upfront indexing of the entire document.

I just ran my first benchmark of the two performing the same operation - retrieving the name field off of a 20MB json object.

simdjson took 46.231 ms to find Burn Notice
JSON(C++) took 0.055 ms to find Burn Notice


The reason mine is so much faster in this case is I don't index anything. That's a linear search.

I need to run a full suite of benchmarks but the above doesn't surprise me. Simdjson will outperform my library if you need most of the data in the document. If you don't, then mine may win out.

simdjson claims to be the fastest json processor in the world.

the above wobbles that claim, even if it's not exactly a real world benchmark

Simdjson runs on 64bit only.

Mine runs on 8bit all the way up to 64 (and beyond)

Mine is a lot smaller too.

I'm not usually competitive and I didn't design this to beat simdjson, but it's nevertheless satisfying that i'm getting results like this.

Edit: The more benchmarks i create, the happier I get.

Benchmark presently, extracting 20000 episodes from the document.
JSONPath equiv would be $..episodes[*].season_number,episode_number,name
i had to use a different path for simdjson because i couldn't figure out how to use a different search axis:
JSONPath for simdjson was $.seasons[*].episodes[*].season_number,episode_number,name

simdjson took 49.669 ms to extract episode data
JSON(C++) took 55.921 ms to extract episode data


That's the second benchmark, which i'm actually thrilled by. The first one just gets the "status" field off the root object - it's toward the end of the document:

simdjson took 45.685 ms to find Canceled
JSON(C++) took 30.12 ms to find Canceled


Each time I'm closing and reopening and doing a fresh read. The reason being is that my stuff was designed for forward only bulk loading - you get everything you need in a single pass. period. it just wasn't designed to work otherwise. so i wanted to compare apples to apples.

simdjson took over 100MB of RAM to get those numbers, and generally requires 4-5x the amount of RAM as the document size on disk.

Mine did it with under 1kB not counting platform specific incidentals like the mapped vmem page(s) i'm using.
Real programmers use butterflies


modified 3-Jan-21 22:55pm.

GeneralRe: Well that was satisfying... Pin
Daniel Pfeffer3-Jan-21 21:50
professionalDaniel Pfeffer3-Jan-21 21:50 
GeneralRe: Well that was satisfying... Pin
honey the codewitch4-Jan-21 1:47
mvahoney the codewitch4-Jan-21 1:47 
GeneralRe: Well that was satisfying... Pin
Randor 3-Jan-21 22:21
professional Randor 3-Jan-21 22:21 
GeneralRe: Well that was satisfying... Pin
honey the codewitch4-Jan-21 1:43
mvahoney the codewitch4-Jan-21 1:43 
PraiseRe: Well that was satisfying... Pin
Randor 4-Jan-21 2:22
professional Randor 4-Jan-21 2:22 
GeneralRe: Well that was satisfying... Pin
honey the codewitch4-Jan-21 2:27
mvahoney the codewitch4-Jan-21 2:27 
GeneralRe: Well that was satisfying... Pin
Randor 4-Jan-21 8:46
professional Randor 4-Jan-21 8:46 
GeneralRe: Well that was satisfying... Pin
honey the codewitch4-Jan-21 11:12
mvahoney the codewitch4-Jan-21 11:12 
GeneralRe: Well that was satisfying... Pin
Randor 4-Jan-21 11:26
professional Randor 4-Jan-21 11:26 
GeneralRe: Well that was satisfying... Pin
honey the codewitch4-Jan-21 11:30
mvahoney the codewitch4-Jan-21 11:30 
GeneralRe: Well that was satisfying... Pin
Randor 4-Jan-21 11:44
professional Randor 4-Jan-21 11:44 
GeneralRe: Well that was satisfying... Pin
honey the codewitch4-Jan-21 11:48
mvahoney the codewitch4-Jan-21 11:48 
GeneralAre hardware questions allowed here? Pin
CodeWraith3-Jan-21 11:35
CodeWraith3-Jan-21 11:35 
GeneralRe: Are hardware questions allowed here? Pin
Mike Hankey3-Jan-21 13:31
mveMike Hankey3-Jan-21 13:31 
GeneralRe: Are hardware questions allowed here? Pin
CodeWraith3-Jan-21 14:28
CodeWraith3-Jan-21 14:28 
GeneralRe: Are hardware questions allowed here? Pin
enhzflep3-Jan-21 16:08
enhzflep3-Jan-21 16:08 
GeneralRe: Are hardware questions allowed here? Pin
honey the codewitch3-Jan-21 17:14
mvahoney the codewitch3-Jan-21 17:14 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.