Welcome to the Lounge

Re: Just one well placed virtual function saves the day

28-Dec-20 4:08

I've been trying to speed up my JSON processing and the bottleneck was my I/O class.

So I added a function to my I/O classes called skipToAny() which takes a string of characters and advances the input until it finds one of them.

Depending on the kind of source you use (like memory mapped file sources) it's *fast*

apparently strpbrk() is optimized for SSE and AVX instructions w/ some stdlib implementations.

So using the builtin C function to scan over my memory mapped buffer increases my performance by a factor of almost 4. I'm in spitting distance of 600MB/s now.

And I didn't even need to bit twiddle.

Higher level optimizations > bit twiddling

Real programmers use butterflies

Josh Gray228-Dec-20 4:57

Josh Gray2

28-Dec-20 4:57

This is a very specific topic but oddly I received an email from a colleague today about a library called simdjson which gives a significant performance improvement over other c++ json libraries.

simdjson claim 'Standalone UTF8 Validation' at 13/GB/s

Re: Just one well placed virtual function saves the day

honey the codewitch28-Dec-20 5:18

Re: Just one well placed virtual function saves the day

28-Dec-20 5:18

simdjson is cool, but it doesn't run everywhere. My JSON thing will even run on arduinos

Real programmers use butterflies

DRHuff28-Dec-20 5:27

DRHuff

28-Dec-20 5:27

What are you doing on an Arduino that needs to process large Json files so quickly?

Or is this just a case of seeing how many people can fit in a phone booth?

If you can't laugh at yourself - ask me and I will do it for you.

Re: Just one well placed virtual function saves the day

honey the codewitch28-Dec-20 5:33

Synchro/Servo? Stepper? What?

28-Dec-20 5:33

It's not fast on an 8 bit Arduino, it's just that it scales to workstations and servers.

The advantage of it over ArduinoJSON is the same library also blazes on traditional computers, and it actually uses far less RAM than ArduinoJSON, which cannot process large online dumps from say, a mongoDB backed repository, like the one tbdb.com runs.

ETA: Originally I wrote it to try out a novel way of parsing and running queries, but it turns out simdjson just recently invented something similar. =)

Real programmers use butterflies

GenJerDan28-Dec-20 0:11

28-Dec-20 0:11

I want to build something, basically an analog clock where, when power is interrupted, the clock stops. But then, when power is restored, the clock zips around to the correct time.

Back in my SATCOM days, the az/el motors would follow the back-in-the-controlroom position controls, but wouldn't move if no power was applied to them. Then slew around to the correct position when the power was turned on.

So, basically, like that, in principle.

I have no idea what I'd need to get this to work these days. A digitally-controlled analog clock is probably the simplest...if I can figure out how to do it.

Suggestions?

We won't sit down.
We won't shut up.
We won't go quietly away.

YouTube, and My Mu[sic], Films and Windows Programs, etc. and FB

Mircea Neacsu28-Dec-20 0:42

Mircea Neacsu

28-Dec-20 0:42

A Stepper motor[^] is your friend.

Mircea

GenJerDan28-Dec-20 1:05

28-Dec-20 1:05

Thanks. I was thinking that. Way simpler than a synchro.

We won't sit down.
We won't shut up.
We won't go quietly away.

YouTube, and My Mu[sic], Films and Windows Programs, etc. and FB

glennPattonWork328-Dec-20 2:43

glennPattonWork3

28-Dec-20 2:43

So if you have a digital battery backed 'clock' keeping time, you could feed out to a couple of octal counters for the hour, six decades counters for the minutes. The time is read from from the counters and translated to a signal for the clock motor, if the power is lost the clock face just stops. Once the power is back up the signal will be a series of steps on from when power was lost the counters 'jump' to the correct time, the hands are not so will spin as fast as they can to show the right time. I think the issue will be getting the correct ammount and style of counters. This is sort of silly thing I enjoy give me chance I'll have a think.

GenJerDan29-Dec-20 0:13

29-Dec-20 0:13

That sounds simplest, so far. Smile | :)

We won't sit down.
We won't shut up.
We won't go quietly away.

YouTube, and My Mu[sic], Films and Windows Programs, etc. and FB

glennPattonWork329-Dec-20 1:14

glennPattonWork3

29-Dec-20 1:14

It is my task in life to find the simple way to do things! Engineers are essance lazy, they find generally find a simple cheap way of doing things, managment complicate Big Grin | :-D

trønderen28-Dec-20 4:08

trønderen

28-Dec-20 4:08

If you plan the clock for the Central European market, you are within the coverage of DCF77[^] - it reaches half way up Norway.

The DFC77 signals, at 77.5 kHz (really longwave!) are so primitive, both at the physical modulation level and the data format, that I often refer to them as 'Morse code Mark II'. The frequency is so low that you could probably sample the signal from the antenna directly (after a simple tuning circuit) using any standard microprocessor that can take an analog input.

(My PC sound card is capable of producing 192 kHz sample rate sound, sufficient for generating the right on-the-air waveforms. I have jokingly suggested that once I get myself a new stereo amp capable of handling analog 192 kHz, I will plug an antenna into the speaker connectors and have my PC generate DCF77 signals to adjust all the DCF77 clocks in my neighborhood by an hour or two Smile | :)

.)

The DCF77 data gives you exact time and date directly in binary format. The simpler way to do it is to feed these digital values directly to a numeric LED display. If you very much want an analog clock, you would use a stepper motor (moving arms 6 degrees/pulse) that at reset moves the arms to 00:00:00, and then moves each arm as many steps as the digital time value indicates. After a power down, you would have to do a reset (which may cause each arm to move all the way around - in theory almost twice: One to go from 1 to 00, and then from 00 to 59, if the arms were at 01:01:01 and the correct time is 11:59:59), but you need not worry about anything else to keep it in sync.

If you have battery backup for the stepper motors and the (very simple) electronics - it is as if the smallest Arduino is overkill - the clock doesn't have to stop, unless you want it to. One of my DCF77 clocks says in the documentation that to save battery power, it turns on the receiver once an hour checking for drifts of the internal oscillator, possibly holding back one pulse or giving one extra to the seconds arm, but even the cheapest oscillators nowadays are so stable that hourly check is probably overkill. You can just turn off the receiver completely if external power goes out.

I haven't been using a soldering iron for years, so I never build a DCF77 clock like this (but I have come as far as to buy the stepper motors for it Smile | :)

) - I guess you know how to handle the electronics much better than I can. Now that you post at CP, I take it that you also handle the software part well Smile | :)

.

ps. If you do this, write a CP article about it!

GenJerDan29-Dec-20 0:11

29-Dec-20 0:11

Never even considered that. Even though I'm old and am used to things like LORAN-C (we used it just for the pulses, not positioning).

We won't sit down.
We won't shut up.
We won't go quietly away.

YouTube, and My Mu[sic], Films and Windows Programs, etc. and FB

honey the codewitch28-Dec-20 5:50

28-Dec-20 5:50

You might be able to use an Arduino or better yet, an ESP32 to control the steppers on the thing

An IoT Smart Clock Using an ESP32 Development Board[^]

IoT Smart Clock using the Mega 2560+WiFi R3[^]

These clocks use WiFi and NTP rather than a radio based sync mechanism so they don't support time zones but could be made to be configurable or modified to use the radio system.

Anyway, some of the principles therein might be useful. Get an ESP32 as they are cheaper and more capable than arduinos + smaller. I included both for the sake of completeness although the arduino one uses an ESP8266 as it's primary worker CPU, not the ATmega2560

Real programmers use butterflies

GenJerDan29-Dec-20 0:13

29-Dec-20 0:13

Thank you. More things to read about. Big Grin | :-D

We won't sit down.
We won't shut up.
We won't go quietly away.

YouTube, and My Mu[sic], Films and Windows Programs, etc. and FB

RedDk28-Dec-20 6:59

RedDk

28-Dec-20 6:59

[^]555 is also 'yer friend ...

Joan M28-Dec-20 10:34

Joan M

28-Dec-20 10:34

Going from simple to complex:

Asynchronous motors with a good reduction and an external encoder can work.
Steppers will do it if you don't need much torque and you don't need them to rotate fast (they can loose position feedback if they turn too fast) (you would need an external encoder to prevent that).
Servomotors will give you a bigger range of speeds and torque while not suffering from the loss of precision at fast speeds.

Beer | [beer]

www.robotecnik.com[^] - robots, CNC and PLC programming

A little light reading

honey the codewitch27-Dec-20 13:05

27-Dec-20 13:05

Implementing strcmp, strlen, and strstr using SSE 4.2 instructions - strchr.com[^]

strchr.com is cool.

I found it while looking for SSE string comparison instructions

Real programmers use butterflies

PIEBALDconsult27-Dec-20 13:59

PIEBALDconsult

27-Dec-20 13:59

But do they still suffer from the "Shlemiel the painter’s algorithm"?

Back to Basics – Joel on Software[^]

Rick York27-Dec-20 17:09

Rick York

27-Dec-20 17:09

The last paragraph of that is pretty good and it's about twenty years old now.

"They have a consciousness, they have a life, they have a soul! Damn you! Let the rabbits wear glasses! Save our brothers! Can I get an amen?"

Chris Maunder27-Dec-20 14:46

Chris Maunder

27-Dec-20 14:46

Interesting that you post this since I was going to suggest, to your previous thread about perf, that you dip down into ASM to wring out some speed.

And here you are.

(I've always admire those fluent in any dialect of ASM, but have never actually bothered trying to learn a single instruction. Maybe about time I just spent a weekend diving in)

cheers
Chris Maunder

honey the codewitch27-Dec-20 15:25

27-Dec-20 15:25

I haven't even started to use that yet, because I'm trying the portable strpbrk() function over a memory mapped file first.

My results are fire.

Approx stack size of local JSON stuff is 176 bytes
Read 1290495 nodes and 20383269 characters in 249.894000 ms at 76.032238MB/s
Skipped 1290495 nodes and 20383268 characters in 33.278000 ms at 570.947773MB/s
utf8 scanned 20383269 characters in 75.141000 ms at 252.857960MB/s
raw ascii i/o 20383269 characters in 58.162000 ms at 326.673773MB/s
raw ascii block i/o 19 blocks in 3.130000 ms at 6070.287540MB/s

Bold line is where I search fast through a document

Edit: Fixed a fencepost error in counting the position.

Edit 2: More complete benchmarks:

Query is $.season[7].episode[2].overview
Approx stack size of local JSON stuff is 152 bytes
Found "Labore magna sint occaecat ea officia labore sit voluptate ut fugiat. Nisi qui commodo consectetur officia incididunt anim do culpa eu. Eu ea magna aliqua excepteur et. Qui eiusmod irure adipisicing enim aute nostrud deserunt eiusmod quis culpa id.rn" and scanned 7149420 characters in 12.034000 ms at 498.587336MB/s

Query is $..id:
Approx stack size of local JSON stuff is 152 bytes
Found 40008 fields and scanned 20383269 characters in 65.563000 ms at 289.797599MB/s

Approx stack size of local JSON stuff is 176 bytes
Read 1290495 nodes and 20383269 characters in 256.696000 ms at 74.017515MB/s
Skipped 1290495 nodes and 20383269 characters in 33.527000 ms at 566.707430MB/s
utf8 scanned 20383269 characters in 72.913000 ms at 260.584532MB/s
raw ascii i/o 20383269 characters in 57.787000 ms at 328.793673MB/s
raw ascii block i/o 19 blocks in 3.106000 ms at 6117.192531MB/s

Edit: Found and fixed a bug with some escape characters not getting translated. (regression when I introduced my fast scanning)

Real programmers use butterflies

modified 27-Dec-20 22:19pm.