<feed xmlns='http://www.w3.org/2005/Atom'>
<title>hittekaart/src/renderer.rs, branch osmand-sqlite</title>
<subtitle>A GPX track heatmap renderer.</subtitle>
<link rel='alternate' type='text/html' href='https://git.kingdread.de/cgit.cgi/hittekaart/'/>
<entry>
<title>abstract away tile rendering logic</title>
<updated>2023-03-11T18:38:16+00:00</updated>
<author>
<name>Daniel Schadt</name>
<email>kingdread@gmx.de</email>
</author>
<published>2023-03-11T18:38:16+00:00</published>
<link rel='alternate' type='text/html' href='https://git.kingdread.de/cgit.cgi/hittekaart/commit/?id=4e8ce5bbaf5aa71f7e00e7a131fc6b25e623c992'/>
<id>4e8ce5bbaf5aa71f7e00e7a131fc6b25e623c992</id>
<content type='text'>
This is in prepration for the tilehunt mode, where we want to render
tiles differently.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
This is in prepration for the tilehunt mode, where we want to render
tiles differently.
</pre>
</div>
</content>
</entry>
<entry>
<title>add some more docstrings</title>
<updated>2023-01-18T19:46:26+00:00</updated>
<author>
<name>Daniel Schadt</name>
<email>kingdread@gmx.de</email>
</author>
<published>2023-01-18T19:46:26+00:00</published>
<link rel='alternate' type='text/html' href='https://git.kingdread.de/cgit.cgi/hittekaart/commit/?id=a6e3e58877307d7c9113fb229d46ada5f165eadc'/>
<id>a6e3e58877307d7c9113fb229d46ada5f165eadc</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>add support for SQLite output</title>
<updated>2023-01-18T17:46:39+00:00</updated>
<author>
<name>Daniel Schadt</name>
<email>kingdread@gmx.de</email>
</author>
<published>2023-01-18T17:46:39+00:00</published>
<link rel='alternate' type='text/html' href='https://git.kingdread.de/cgit.cgi/hittekaart/commit/?id=ac3afadba547b4b9a4063da567acd6d2f4f74554'/>
<id>ac3afadba547b4b9a4063da567acd6d2f4f74554</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>factor out saving logic</title>
<updated>2023-01-17T22:06:54+00:00</updated>
<author>
<name>Daniel Schadt</name>
<email>kingdread@gmx.de</email>
</author>
<published>2023-01-17T22:06:54+00:00</published>
<link rel='alternate' type='text/html' href='https://git.kingdread.de/cgit.cgi/hittekaart/commit/?id=160aba6258e1979ef85d66b2c27c33f6f28a7e38'/>
<id>160aba6258e1979ef85d66b2c27c33f6f28a7e38</id>
<content type='text'>
Since we want to support SQLite at some point, it makes sense to have
the exact storage method abstracted away.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Since we want to support SQLite at some point, it makes sense to have
the exact storage method abstracted away.
</pre>
</div>
</content>
</entry>
<entry>
<title>replace std::mpsc with crossbeam_channel</title>
<updated>2023-01-17T19:04:19+00:00</updated>
<author>
<name>Daniel Schadt</name>
<email>kingdread@gmx.de</email>
</author>
<published>2023-01-17T19:04:19+00:00</published>
<link rel='alternate' type='text/html' href='https://git.kingdread.de/cgit.cgi/hittekaart/commit/?id=088f292c1101adb15716705fd0e753de9668f9cf'/>
<id>088f292c1101adb15716705fd0e753de9668f9cf</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>use a single thread to write out files</title>
<updated>2023-01-16T20:01:41+00:00</updated>
<author>
<name>Daniel Schadt</name>
<email>kingdread@gmx.de</email>
</author>
<published>2023-01-16T19:58:53+00:00</published>
<link rel='alternate' type='text/html' href='https://git.kingdread.de/cgit.cgi/hittekaart/commit/?id=f81eb4882a015ff19cb7209980ca3c7dd33bbae1'/>
<id>f81eb4882a015ff19cb7209980ca3c7dd33bbae1</id>
<content type='text'>
It seems like this does not make the encoding slower, and the main point
is that we might want to support SQLite storage for the tiles, in which
case it might be good to have only one writer. Even with the FS-based
approach, maybe it's good to have a single thread responsible for
writing everything, and not hammer the OS with 16 write requests at
once.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
It seems like this does not make the encoding slower, and the main point
is that we might want to support SQLite storage for the tiles, in which
case it might be good to have only one writer. Even with the FS-based
approach, maybe it's good to have a single thread responsible for
writing everything, and not hammer the OS with 16 write requests at
once.
</pre>
</div>
</content>
</entry>
<entry>
<title>parallelize PNG encoding</title>
<updated>2023-01-12T22:59:46+00:00</updated>
<author>
<name>Daniel Schadt</name>
<email>kingdread@gmx.de</email>
</author>
<published>2023-01-12T22:59:46+00:00</published>
<link rel='alternate' type='text/html' href='https://git.kingdread.de/cgit.cgi/hittekaart/commit/?id=7bbba155f10ce1344724ea00ca70c4d3bb469272'/>
<id>7bbba155f10ce1344724ea00ca70c4d3bb469272</id>
<content type='text'>
This gives a massive speedup
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
This gives a massive speedup
</pre>
</div>
</content>
</entry>
<entry>
<title>clean up some unused code</title>
<updated>2023-01-12T21:35:44+00:00</updated>
<author>
<name>Daniel Schadt</name>
<email>kingdread@gmx.de</email>
</author>
<published>2023-01-12T21:35:44+00:00</published>
<link rel='alternate' type='text/html' href='https://git.kingdread.de/cgit.cgi/hittekaart/commit/?id=def44c6e969f1027da82e4130a82db7e4916ba86'/>
<id>def44c6e969f1027da82e4130a82db7e4916ba86</id>
<content type='text'>
Some of it was written because it fit the API, but we didn't end up
using it in main.rs.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Some of it was written because it fit the API, but we didn't end up
using it in main.rs.
</pre>
</div>
</content>
</entry>
<entry>
<title>add some first benchmarks</title>
<updated>2023-01-11T22:37:46+00:00</updated>
<author>
<name>Daniel Schadt</name>
<email>kingdread@gmx.de</email>
</author>
<published>2023-01-11T22:37:46+00:00</published>
<link rel='alternate' type='text/html' href='https://git.kingdread.de/cgit.cgi/hittekaart/commit/?id=3826d00f339e87698f95dc24c33739e2880aac65'/>
<id>3826d00f339e87698f95dc24c33739e2880aac65</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>considerably speed up the rendering process</title>
<updated>2023-01-10T20:46:13+00:00</updated>
<author>
<name>Daniel Schadt</name>
<email>kingdread@gmx.de</email>
</author>
<published>2023-01-10T20:40:02+00:00</published>
<link rel='alternate' type='text/html' href='https://git.kingdread.de/cgit.cgi/hittekaart/commit/?id=6c547a24cea3422d934547bc9baf28a5f9ecf139'/>
<id>6c547a24cea3422d934547bc9baf28a5f9ecf139</id>
<content type='text'>
Most of the time was spent doing hashmap lookups because all of our
operations were done pixel by pixel, and layer.get_pixel_mut always went
through the hashmap lookup. This was true for render_circle, render_line
*and* merge_heat_counter - the last of which iterated over the full
layer every time.

The biggest change now is that we try to do accesses tile-by-tile. For
the drawing functions, this means that we render the image on a small
patch locally, and then blit the image onto the base - tile by tile,
instead of pixel by pixel.

For merge_heat_counters, we do the same: We iterate over tiles first,
keeping a reference, and then iterate over the tile's pixels - that way
we get a *huge* speedup. I can now render level 19 in 9 seconds,
compared to before when it took 20s for level 17.

Another benefit now is that we save the heatmap as u8 instead of u32.
For a single track, we could even use a single bit (though that brings
other problems with it). For the complete heatmap, u8 is probably too
small (having 256 tracks is realistic), but we can change the merged one
to be u16 later. This allows us to cut down on the RAM the program needs
considerably, as we basically only use a fourth of the space now.

A bit of noise is introduced in this patch since I ran cargo fmt.

Side note: The bottleneck now seems to be the PNG compression, so that
would be the next area to improve upon. Either by toning down the
compression ratio (at the cost of higher storage needs), or by
leveraging multithreading to deal with that.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Most of the time was spent doing hashmap lookups because all of our
operations were done pixel by pixel, and layer.get_pixel_mut always went
through the hashmap lookup. This was true for render_circle, render_line
*and* merge_heat_counter - the last of which iterated over the full
layer every time.

The biggest change now is that we try to do accesses tile-by-tile. For
the drawing functions, this means that we render the image on a small
patch locally, and then blit the image onto the base - tile by tile,
instead of pixel by pixel.

For merge_heat_counters, we do the same: We iterate over tiles first,
keeping a reference, and then iterate over the tile's pixels - that way
we get a *huge* speedup. I can now render level 19 in 9 seconds,
compared to before when it took 20s for level 17.

Another benefit now is that we save the heatmap as u8 instead of u32.
For a single track, we could even use a single bit (though that brings
other problems with it). For the complete heatmap, u8 is probably too
small (having 256 tracks is realistic), but we can change the merged one
to be u16 later. This allows us to cut down on the RAM the program needs
considerably, as we basically only use a fourth of the space now.

A bit of noise is introduced in this patch since I ran cargo fmt.

Side note: The bottleneck now seems to be the PNG compression, so that
would be the next area to improve upon. Either by toning down the
compression ratio (at the cost of higher storage needs), or by
leveraging multithreading to deal with that.
</pre>
</div>
</content>
</entry>
</feed>
