tag:blogger.com,1999:blog-6469704299235308349.post3522966461340861026..comments2024-03-28T15:50:13.644+11:00Comments on -ck hacking: lrzip-0.544 and fudging with memory allocation on 32 bitsckhttp://www.blogger.com/profile/02904761195451530213noreply@blogger.comBlogger4125tag:blogger.com,1999:blog-6469704299235308349.post-79501955721337961162010-12-08T10:15:14.857+11:002010-12-08T10:15:14.857+11:00Thanks for your comments. Knowing it needs 12x the...Thanks for your comments. Knowing it needs 12x the ram is VERY helpful. The rzip first stage already does use very little ram to make the most of big windows as you described with your own app which sounds very interesting. Compressing with bzip2 as a backend on lrzip is pretty efficient with ram usage but I'm trying for bigger and better :P I'll certainly take this extra 12x info on board.ckhttps://www.blogger.com/profile/02904761195451530213noreply@blogger.comtag:blogger.com,1999:blog-6469704299235308349.post-55773014820537784572010-12-08T10:07:20.953+11:002010-12-08T10:07:20.953+11:00As a side comment, regarding your point on memory ...As a side comment, regarding your point on memory usage with lot of threads,<br />it reminds me of an old compression project which was based on efficient memory usage with large dictionary sizes.<br /><br />It's called Etincelle, with a working win32 demo available here :<br />http://phantasie.tonempire.net/pc-compression-f2/etincelle-fast-efficient-compression-x280-82mb-s-t102.htm#160<br /><br />The nice thing about Etincelle is that it uses a 2MB table, whatever the dictionary size (which could be the size of chunk it is fed with). Therefore, its memory footprint is pretty much under control, even with many threads : it will simply need about 2MB per thread, (plus obviously input & output buffers).<br /><br />Etincelle still needs some code clean up to reach open source status; but obviously, if you think it could be an interesting back end test for your project, it's reason enough to speed-up its release.Cyanhttps://www.blogger.com/profile/02905407922640810117noreply@blogger.comtag:blogger.com,1999:blog-6469704299235308349.post-57629032918251180732010-12-08T08:48:40.117+11:002010-12-08T08:48:40.117+11:00This comment has been removed by the author.Cyanhttps://www.blogger.com/profile/02905407922640810117noreply@blogger.comtag:blogger.com,1999:blog-6469704299235308349.post-57175483416495582142010-12-08T08:38:21.016+11:002010-12-08T08:38:21.016+11:007zip needs really a lot of memory to compress a ch...7zip needs really a lot of memory to compress a chunk. As a baseline, you can consider it needs roughly x12 memory, which means for a 60MB chunk that it will need about 700MB.<br />This is really a lot for a 32bit system, especially if you want to scatter several threads, then memory usage adds up.Cyanhttps://www.blogger.com/profile/02905407922640810117noreply@blogger.com