|
I've been trying to speed up my JSON processing and the bottleneck was my I/O class.
So I added a function to my I/O classes called skipToAny() which takes a string of characters and advances the input until it finds one of them.
Depending on the kind of source you use (like memory mapped file sources) it's *fast*
apparently strpbrk() is optimized for SSE and AVX instructions w/ some stdlib implementations.
So using the builtin C function to scan over my memory mapped buffer increases my performance by a factor of almost 4. I'm in spitting distance of 600MB/s now.
And I didn't even need to bit twiddle.
Higher level optimizations > bit twiddling
Real programmers use butterflies
|
|
|
|
|
This is a very specific topic but oddly I received an email from a colleague today about a library called simdjson which gives a significant performance improvement over other c++ json libraries.
simdjson claim 'Standalone UTF8 Validation' at 13/GB/s
|
|
|
|
|
simdjson is cool, but it doesn't run everywhere. My JSON thing will even run on arduinos
Real programmers use butterflies
|
|
|
|
|
What are you doing on an Arduino that needs to process large Json files so quickly?
Or is this just a case of seeing how many people can fit in a phone booth?
If you can't laugh at yourself - ask me and I will do it for you.
|
|
|
|
|
It's not fast on an 8 bit Arduino, it's just that it scales to workstations and servers.
The advantage of it over ArduinoJSON is the same library also blazes on traditional computers, and it actually uses far less RAM than ArduinoJSON, which cannot process large online dumps from say, a mongoDB backed repository, like the one tbdb.com runs.
ETA: Originally I wrote it to try out a novel way of parsing and running queries, but it turns out simdjson just recently invented something similar. =)
Real programmers use butterflies
|
|
|
|
|
I want to build something, basically an analog clock where, when power is interrupted, the clock stops. But then, when power is restored, the clock zips around to the correct time.
Back in my SATCOM days, the az/el motors would follow the back-in-the-controlroom position controls, but wouldn't move if no power was applied to them. Then slew around to the correct position when the power was turned on.
So, basically, like that, in principle.
I have no idea what I'd need to get this to work these days. A digitally-controlled analog clock is probably the simplest...if I can figure out how to do it.
Suggestions?
|
|
|
|
|
|
Thanks. I was thinking that. Way simpler than a synchro.
|
|
|
|
|
So if you have a digital battery backed 'clock' keeping time, you could feed out to a couple of octal counters for the hour, six decades counters for the minutes. The time is read from from the counters and translated to a signal for the clock motor, if the power is lost the clock face just stops. Once the power is back up the signal will be a series of steps on from when power was lost the counters 'jump' to the correct time, the hands are not so will spin as fast as they can to show the right time. I think the issue will be getting the correct ammount and style of counters. This is sort of silly thing I enjoy give me chance I'll have a think.
|
|
|
|
|
That sounds simplest, so far. 
|
|
|
|
|
It is my task in life to find the simple way to do things! Engineers are essance lazy, they find generally find a simple cheap way of doing things, managment complicate 
|
|
|
|
|
If you plan the clock for the Central European market, you are within the coverage of DCF77[^] - it reaches half way up Norway.
The DFC77 signals, at 77.5 kHz (really longwave!) are so primitive, both at the physical modulation level and the data format, that I often refer to them as 'Morse code Mark II'. The frequency is so low that you could probably sample the signal from the antenna directly (after a simple tuning circuit) using any standard microprocessor that can take an analog input.
(My PC sound card is capable of producing 192 kHz sample rate sound, sufficient for generating the right on-the-air waveforms. I have jokingly suggested that once I get myself a new stereo amp capable of handling analog 192 kHz, I will plug an antenna into the speaker connectors and have my PC generate DCF77 signals to adjust all the DCF77 clocks in my neighborhood by an hour or two .)
The DCF77 data gives you exact time and date directly in binary format. The simpler way to do it is to feed these digital values directly to a numeric LED display. If you very much want an analog clock, you would use a stepper motor (moving arms 6 degrees/pulse) that at reset moves the arms to 00:00:00, and then moves each arm as many steps as the digital time value indicates. After a power down, you would have to do a reset (which may cause each arm to move all the way around - in theory almost twice: One to go from 1 to 00, and then from 00 to 59, if the arms were at 01:01:01 and the correct time is 11:59:59), but you need not worry about anything else to keep it in sync.
If you have battery backup for the stepper motors and the (very simple) electronics - it is as if the smallest Arduino is overkill - the clock doesn't have to stop, unless you want it to. One of my DCF77 clocks says in the documentation that to save battery power, it turns on the receiver once an hour checking for drifts of the internal oscillator, possibly holding back one pulse or giving one extra to the seconds arm, but even the cheapest oscillators nowadays are so stable that hourly check is probably overkill. You can just turn off the receiver completely if external power goes out.
I haven't been using a soldering iron for years, so I never build a DCF77 clock like this (but I have come as far as to buy the stepper motors for it ) - I guess you know how to handle the electronics much better than I can. Now that you post at CP, I take it that you also handle the software part well .
ps. If you do this, write a CP article about it!
|
|
|
|
|
 Never even considered that. Even though I'm old and am used to things like LORAN-C (we used it just for the pulses, not positioning).
|
|
|
|
|
You might be able to use an Arduino or better yet, an ESP32 to control the steppers on the thing
An IoT Smart Clock Using an ESP32 Development Board[^]
IoT Smart Clock using the Mega 2560+WiFi R3[^]
These clocks use WiFi and NTP rather than a radio based sync mechanism so they don't support time zones but could be made to be configurable or modified to use the radio system.
Anyway, some of the principles therein might be useful. Get an ESP32 as they are cheaper and more capable than arduinos + smaller. I included both for the sake of completeness although the arduino one uses an ESP8266 as it's primary worker CPU, not the ATmega2560
Real programmers use butterflies
|
|
|
|
|
Thank you. More things to read about.
|
|
|
|
|
[^]555 is also 'yer friend ...
|
|
|
|
|
Going from simple to complex:
Asynchronous motors with a good reduction and an external encoder can work.
Steppers will do it if you don't need much torque and you don't need them to rotate fast (they can loose position feedback if they turn too fast) (you would need an external encoder to prevent that).
Servomotors will give you a bigger range of speeds and torque while not suffering from the loss of precision at fast speeds.
![Beer | [beer]](https://www.codeproject.com/script/Forums/Images/beer.gif)
|
|
|
|
|
|
|
The last paragraph of that is pretty good and it's about twenty years old now.
"They have a consciousness, they have a life, they have a soul! Damn you! Let the rabbits wear glasses! Save our brothers! Can I get an amen?"
|
|
|
|
|
Interesting that you post this since I was going to suggest, to your previous thread about perf, that you dip down into ASM to wring out some speed.
And here you are.
(I've always admire those fluent in any dialect of ASM, but have never actually bothered trying to learn a single instruction. Maybe about time I just spent a weekend diving in)
cheers
Chris Maunder
|
|
|
|
|
I haven't even started to use that yet, because I'm trying the portable strpbrk() function over a memory mapped file first.
My results are fire.
Approx stack size of local JSON stuff is 176 bytes
Read 1290495 nodes and 20383269 characters in 249.894000 ms at 76.032238MB/s
Skipped 1290495 nodes and 20383268 characters in 33.278000 ms at 570.947773MB/s
utf8 scanned 20383269 characters in 75.141000 ms at 252.857960MB/s
raw ascii i/o 20383269 characters in 58.162000 ms at 326.673773MB/s
raw ascii block i/o 19 blocks in 3.130000 ms at 6070.287540MB/s
Bold line is where I search fast through a document
Edit: Fixed a fencepost error in counting the position.
Edit 2: More complete benchmarks:
Query is $.season[7].episode[2].overview
Approx stack size of local JSON stuff is 152 bytes
Found "Labore magna sint occaecat ea officia labore sit voluptate ut fugiat. Nisi qui commodo consectetur officia incididunt anim do culpa eu. Eu ea magna aliqua excepteur et. Qui eiusmod irure adipisicing enim aute nostrud deserunt eiusmod quis culpa id.rn" and scanned 7149420 characters in 12.034000 ms at 498.587336MB/s
Query is $..id:
Approx stack size of local JSON stuff is 152 bytes
Found 40008 fields and scanned 20383269 characters in 65.563000 ms at 289.797599MB/s
Approx stack size of local JSON stuff is 176 bytes
Read 1290495 nodes and 20383269 characters in 256.696000 ms at 74.017515MB/s
Skipped 1290495 nodes and 20383269 characters in 33.527000 ms at 566.707430MB/s
utf8 scanned 20383269 characters in 72.913000 ms at 260.584532MB/s
raw ascii i/o 20383269 characters in 57.787000 ms at 328.793673MB/s
raw ascii block i/o 19 blocks in 3.106000 ms at 6117.192531MB/s
Edit: Found and fixed a bug with some escape characters not getting translated. (regression when I introduced my fast scanning)
Real programmers use butterflies
modified 27-Dec-20 22:19pm.
|
|
|
|
|
Too high level. Anyone remember:
C:\>debug
-D
0B06:0100 75 60 C6 46 00 00 8A 7E-04 F6 C7 04 74 E6 C6 46 u`.F...~....t..F
0B06:0110 00 02 8B 76 02 80 3C 00-74 4B B3 2E 34 00 F5 0A ...v..<.tK..4...
0B06:0120 B3 3A 38 5C FE 74 05 C6-46 00 01 4E 32 DB 86 1C .:8\.t..F..N2...
0B06:0130 E8 39 EB 3B D6 73 1B 56-51 8B CE 8B F2 AC E8 B2 .9.;.s.VQ.......
0B06:0140 E1 74 09 AC 3B F1 72 F5-59 5E EB 0B 3B F1 72 ED .t..;.r.Y^..;.r.
0B06:0150 59 5E 3A 5C FF 74 0E B4-3B CD 21 86 1C 73 95 E8 Y^:\.t..;.!..s..
0B06:0160 9B DA E9 C9 D7 E9 C3 D7-89 7E 02 80 46 01 0C B8 .........~..F...
0B06:0170 3F 2E B9 08 00 F3 AA 86-C4 AA 86 C4 B1 03 F3 AA ?...............
-
Now, those were the (so-called) good old days. :)
If you can keep your head while those about you are losing theirs, perhaps you don't understand the situation.
|
|
|
|
|
Yes. That reminds me of when i learned 6502 bytecode before i realized i had a built in mini-assembler.
Real programmers use butterflies
|
|
|
|
|
The first assembler I used was simply adding symbols. The instruction set was very regular (the CPU architecture from the days long before microcode), so opcode, modifiers and offsets all had their fixed place in the instruction word.
We played around with this: To generate a MUL (multiply) instruction, you could rather use ADD ADD, as the opcode for MUL was twice the opcode of ADD
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
