I recently finished a rewrite of the code at the core of my project Terrain2STL. The main changes I made were to reduces memory usage from being quadratic with model size to merely linear (important for the cheap virtual server the site runs on) and rewriting the entire STL file generator program in pure C. The original was C++ I managed to cobble together, but I've been programming in C a lot lately for work so new code is probably a lot more concise and memory-safe than the original.
This document is a very very loose collection of my notes on the rewrite and retest process. This is also my
first time using the valgrind tools to run a memory-usage profile on a program,
which was super useful and a lot easier than I thought it would be. I intend to come back and reread these notes
the next time I use valgrind as a refresher - maybe they'll be a useful example
for someone else too!
For reference, the code I was testing is available on GitHub and you can play around with making your own 3D-printable elevation maps over at Terrain2STL.
June 7
New goal: verify model production in the new C verion of elevstl. I already
know the C version is uses much less memory, but I need to make sure the models it creates look the same as the
ones produced by the original code.
- Modified
terrainServer.json the server to runelevstlandcelevstlwith the same parameters on each requrest. - I'll go through and make some models, download the copies made by both programs, and compare the results
First Test: Crossing tiles
- Chose a NW corner and box size that crosses a line of latitude and a line of longitude
./elevstl 44.1052 -71.1024 200 200 2 0 0 1 1 > ./stls/rawmodel-0.stl; zip -q ./stls/terrain-0 ./stls/rawmodel-0.stl; ./celevstl 44.1052 -71.1024 200 200 2 0 0 1 1; mv out.stl ./stls/crawmodel-0.stl; zip -q ./stls/cterrain-0 ./stls/crawmodel-0.stl
./elevstl 44.1052 -71.1024 200 200 2 0 0 1 1 > ./stls/rawmodel-0.stl;
zip -q ./stls/terrain-0 ./stls/rawmodel-0.stl;
./celevstl 44.1052 -71.1024 200 200 2 0 0 1 1;
mv out.stl ./stls/crawmodel-0.stl;
zip -q ./stls/cterrain-0 ./stls/crawmodel-0.stl
Looks good! No discernible latitude or longitude lines.
Second Test: Rotations
- Rotation of 53 degrees
./elevstl 44.0500 -71.1448 200 200 2 53 0 1 1 > ./stls/rawmodel-4.stl; zip -q ./stls/terrain-4 ./stls/rawmodel-4.stl; ./celevstl 44.0500 -71.1448 200 200 2 53 0 1 1; mv out.stl ./stls/crawmodel-4.stl; zip -q ./stls/cterrain-4 ./stls/crawmodel-4.stl
./elevstl 44.0500 -71.1448 200 200 2 53 0 1 1 > ./stls/rawmodel-4.stl;
zip -q ./stls/terrain-4 ./stls/rawmodel-4.stl;
./celevstl 44.0500 -71.1448 200 200 2 53 0 1 1;
mv out.stl ./stls/crawmodel-4.stl;
zip -q ./stls/cterrain-4 ./stls/crawmodel-4.stlLooks good! Couldn't tell a difference between the models.
Third Test: Box Scaling, Water Drop, and Base Height
./elevstl 44.6300 -68.5734 200 200 3.2 53 4 10 3 > ./stls/rawmodel-6.stl; zip -q ./stls/terrain-6 ./stls/rawmodel-6.stl; ./celevstl 44.6300 -68.5734 200 200 3.2 53 4 10 3; mv out.stl ./stls/crawmodel-6.stl; zip -q ./stls/cterrain-6 ./stls/crawmodel-6.stl
./elevstl 44.6300 -68.5734 200 200 3.2 53 4 10 3 > ./stls/rawmodel-6.stl;
zip -q ./stls/terrain-6 ./stls/rawmodel-6.stl;
./celevstl 44.6300 -68.5734 200 200 3.2 53 4 10 3;
mv out.stl ./stls/crawmodel-6.stl;
zip -q ./stls/cterrain-6 ./stls/crawmodel-6.stlTest passed! Models look identical.
The one thing that's different about elevstl and celevstl is that celevstl always writes its file output to
out.stl, while elevstl always write to STDOUT.
Neither of these is a good option, so I changed celevstl to take another
argument (the 10th) as the output file name. That version is running on the site now and seems to working
well!
With these successes, I'd like to move the server back to the smaller, cheaper, low-RAM virtual server I originally used.
June 1
Commands for analyzing elevstl memory usage are:
valgrind --tool=massif ./elevstl 44.267792 -69.135099 600 600 2.5 0 1 3 1
ms_print massif.out.
Test Procedure:
Run the following STL generation command on the server, for the C++ and C versions of elevstl, with and without valgrind.
- @ master:
valgrind --tool=massif ./elevstl 51.9038 -58.8139 1320 1320 15 2 -1 4 1 > stls/test-cpp.stl - Checkout branch
justc - Compile new
elevstl.ctocelevstl - @ justc:
valgrind --tool=massif ./celevstl 51.9038 -58.8139 1320 1320 15 2 -1 4 1 > stls/test-c.stl
Results
I hope to see lower memory usage the justc branch version of the code, but it's
also important that the files produced are the same.
- master branch:
massif.out.7446 - justc branch:
massif.out.8121 stls/test-cpp.stl: 174503884 bytesout.stl: 174635884 bytes (celevstlwrite all files to out.stl - this needs to be changed)- The two files have different md5 hashes, so they are not identical.
master branch massif results:
MB
13.37^ :
|#::::@@:@::::@@:@:@::@::@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
|#::::@@:@::::@@:@:@::@::@
|#::::@@:@::::@@:@:@::@::@
|#::::@@:@::::@@:@:@::@::@
|#::::@@:@::::@@:@:@::@::@
|#::::@@:@::::@@:@:@::@::@
|#::::@@:@::::@@:@:@::@::@
|#::::@@:@::::@@:@:@::@::@
|#::::@@:@::::@@:@:@::@::@
|#::::@@:@::::@@:@:@::@::@
|#::::@@:@::::@@:@:@::@::@
|#::::@@:@::::@@:@:@::@::@
|#::::@@:@::::@@:@:@::@::@
|#::::@@:@::::@@:@:@::@::@
|#::::@@:@::::@@:@:@::@::@
|#::::@@:@::::@@:@:@::@::@
|#::::@@:@::::@@:@:@::@::@
|#::::@@:@::::@@:@:@::@::@
|#::::@@:@::::@@:@:@::@::@
0 +----------------------------------------------------------------------->Gi
0 16.31
MB
13.37^ :
|#::::@@:@::::@@:@:@::@::@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
|#::::@@:@::::@@:@:@::@::@
|#::::@@:@::::@@:@:@::@::@
|#::::@@:@::::@@:@:@::@::@
|#::::@@:@::::@@:@:@::@::@
|#::::@@:@::::@@:@:@::@::@
|#::::@@:@::::@@:@:@::@::@
|#::::@@:@::::@@:@:@::@::@
|#::::@@:@::::@@:@:@::@::@
|#::::@@:@::::@@:@:@::@::@
|#::::@@:@::::@@:@:@::@::@
|#::::@@:@::::@@:@:@::@::@
|#::::@@:@::::@@:@:@::@::@
|#::::@@:@::::@@:@:@::@::@
|#::::@@:@::::@@:@:@::@::@
|#::::@@:@::::@@:@:@::@::@
|#::::@@:@::::@@:@:@::@::@
|#::::@@:@::::@@:@:@::@::@
|#::::@@:@::::@@:@:@::@::@
0 +----------------------------------------------------------------------->Gi
0 16.31justc branch massif results:
KB
23.46^##
|#
|#
|#
|# :::::::::::@:::::::::::::::::::::::::::::::::::@::::::::::::::::::::::
|# :: :: : :: @: :: :: :: : :: : ::: ::: :::: :::@:: : :::: ::: : : :::
|# :: :: : :: @: :: :: :: : :: : ::: ::: :::: :::@:: : :::: ::: : : :::
|# :: :: : :: @: :: :: :: : :: : ::: ::: :::: :::@:: : :::: ::: : : :::
|# :: :: : :: @: :: :: :: : :: : ::: ::: :::: :::@:: : :::: ::: : : :::
|# :: :: : :: @: :: :: :: : :: : ::: ::: :::: :::@:: : :::: ::: : : :::
|# :: :: : :: @: :: :: :: : :: : ::: ::: :::: :::@:: : :::: ::: : : :::
|# :: :: : :: @: :: :: :: : :: : ::: ::: :::: :::@:: : :::: ::: : : :::
|# :: :: : :: @: :: :: :: : :: : ::: ::: :::: :::@:: : :::: ::: : : :::
|# :: :: : :: @: :: :: :: : :: : ::: ::: :::: :::@:: : :::: ::: : : :::
|# :: :: : :: @: :: :: :: : :: : ::: ::: :::: :::@:: : :::: ::: : : :::
|# :: :: : :: @: :: :: :: : :: : ::: ::: :::: :::@:: : :::: ::: : : :::
|# :: :: : :: @: :: :: :: : :: : ::: ::: :::: :::@:: : :::: ::: : : :::
|# :: :: : :: @: :: :: :: : :: : ::: ::: :::: :::@:: : :::: ::: : : :::
|# :: :: : :: @: :: :: :: : :: : ::: ::: :::: :::@:: : :::: ::: : : :::
|# :: :: : :: @: :: :: :: : :: : ::: ::: :::: :::@:: : :::: ::: : : :::
0 +----------------------------------------------------------------------->Gi
0 6.454
KB
23.46^##
|#
|#
|#
|# :::::::::::@:::::::::::::::::::::::::::::::::::@::::::::::::::::::::::
|# :: :: : :: @: :: :: :: : :: : ::: ::: :::: :::@:: : :::: ::: : : :::
|# :: :: : :: @: :: :: :: : :: : ::: ::: :::: :::@:: : :::: ::: : : :::
|# :: :: : :: @: :: :: :: : :: : ::: ::: :::: :::@:: : :::: ::: : : :::
|# :: :: : :: @: :: :: :: : :: : ::: ::: :::: :::@:: : :::: ::: : : :::
|# :: :: : :: @: :: :: :: : :: : ::: ::: :::: :::@:: : :::: ::: : : :::
|# :: :: : :: @: :: :: :: : :: : ::: ::: :::: :::@:: : :::: ::: : : :::
|# :: :: : :: @: :: :: :: : :: : ::: ::: :::: :::@:: : :::: ::: : : :::
|# :: :: : :: @: :: :: :: : :: : ::: ::: :::: :::@:: : :::: ::: : : :::
|# :: :: : :: @: :: :: :: : :: : ::: ::: :::: :::@:: : :::: ::: : : :::
|# :: :: : :: @: :: :: :: : :: : ::: ::: :::: :::@:: : :::: ::: : : :::
|# :: :: : :: @: :: :: :: : :: : ::: ::: :::: :::@:: : :::: ::: : : :::
|# :: :: : :: @: :: :: :: : :: : ::: ::: :::: :::@:: : :::: ::: : : :::
|# :: :: : :: @: :: :: :: : :: : ::: ::: :::: :::@:: : :::: ::: : : :::
|# :: :: : :: @: :: :: :: : :: : ::: ::: :::: :::@:: : :::: ::: : : :::
|# :: :: : :: @: :: :: :: : :: : ::: ::: :::: :::@:: : :::: ::: : : :::
0 +----------------------------------------------------------------------->Gi
0 6.454- Resulting models looks very similar - indistinguishable at least overall:
celevstlmodel:
elevstlmodel:
Takeaways:
celevstlruns in half (almost one third) of the instructions aselevstlelevstluses 13.37 MB of memory in this case (unusually large requested file size) whilecelevstluses 23.46 KB memory - thats 0.17%! Wow!- Models are visually indistiguishable, but do have different contents. A more thorough small-scale analysis nees to be done to establish model accuracy.
May 30
Over the past year I've made very intermittent progress on a write of elevstl, the core part of Terrain2STL, in C. The goals of the rewrite are to:
- Write better code (I know C better now)
- Reduce memory usage (high memory usage caused crashes on a small server, so now I'm paying more for a server with extra RAM)
- Produce results identical to the original C++ version of the program
To measure memory usage on both the old and new version of elevstl, I'm going
to use valgrind, which I learned about from a
Hackaday post on the subject. In particular, the massif tool seems like it
will be very useful in profiling memory usage.
This Stack Exchange answer also points
to massif as the useful tool for this task. Another comment on that page
points out that valgrind and its associated tools won't work very well on
statically-linked executables, but that won't be a problem for me because I'm not using gcc with the -static flag as shown here.
The Massif heap profiler manual is also a really good guide, with lots of nice example outputs.
All the C rewrite work is in the justc branch of the Terrain2STL
repository.