----------------------------------------------------------------------------- 2nd Official Valgrind Survey, September 2005: full report ----------------------------------------------------------------------------- Survey was opened 4pm CDT, Thursday September 22, 2005. Survey was extended at 11.30pm CDT, Thursday September 29, 2005. Survey was closed 1pm CDT, Saturday October 08, 2005. I received surveys on the following days (date is BST): - Thu Sep 22 16 - Fri Sep 23 53 - Sat Sep 24 5 - Sun Sep 25 5 - Mon Sep 26 9 - Tue Sep 27 7 - Wed Sep 28 6 - Thu Sep 29 9 - Fri Sep 30 27 - Sat Oct 01 7 - Sun Oct 02 1 - Mon Oct 03 11 - Tue Oct 04 3 - Wed Oct 05 6 - Thu Oct 06 11 - Fri Oct 07 2 - Sat Oct 08 1 --- - total 179 We got surges in response to the emails to the three lists on Sep 22 and Sep 30, and then a steady trickle the rest of the time, presumably from people just seeing the notice of the front page of valgrind.org. Not everybody answered every question. (A few people barely answered any.) Here is the number of responses to each question. 1. How long have you used Valgrind? 177 99% 2. How often do you use Valgrind? 178 99% 3. Use on different hardware? 177 99% 4. Use on different OSes? 179 100% 5. How do you obtain Valgrind? 179 100% 6. Which tools do you use? 179 100% 7. Command-line options almost always used? 147 82% 8. Command-line options sometimes used? 107 60% 9. Platform preferences? 156 87% 10. Good software features? 145 81% 11. Bad software features? 130 73% 12. Missing software features? 104 58% 13. Ideas for new tools? 57 32% 14. Good development aspects 66 37% 15. Bad development aspects 35 20% 16. Project details 119 66% 17. Comments about the survey? 26 15% 18. Other comments? 72 40% 19. Name/email/country 139 78% For each question below, I give a detailed breakdown of the results, summarise under the "Conclusions" heading, and then list "Action items" which are things that we the developers should consider doing/changing/acting on. Some of the questions don't have any action items because they aren't relevant; I have dubbed these "curiosity questions". I also evaluate how good each question was (under the "Next time" heading), and whether it should be kept/changed/removed for the next survey, in particular focussing on how automation could be used more effectively. ============================================================================= Using Valgrind ============================================================================= ----------------------------------------------------------------------------- 1. Approximately how long (in months) have you been using Valgrind? ----------------------------------------------------------------------------- 177 people answered, two did not. - 175 numerical answers, 3 of which were uncertain - 2 non-numerical answers. - One answer of zero from someone who doesn't use Valgrind becauses it's not available on his/her platform, but wishes it was. - One answer of 72 which is much longer than Valgrind has been around (it was first publically available around the end of 2001, ie. about 48 months ago). The exact responses were, after grouping: 48+ months: 8 -------------- 72 1 50 1 48 6 36--47 months: 27 ----------------- 43 1 40 2 37 1 36 22 almost from the beginning of the project 1 24--35 months: 65 ----------------- 30 14 29 1 28 1 27 1 26 1 25 2 24 44 a long time (years) 1 12--23 months: 45 ----------------- 20 5 18 12 16 3 13 2 12 23 0--11 months: 32 ---------------- 10 2 8 2 6 6 5 2 4 5 3 4 2 4 1 5 0.25 1 0 1 The percentages are: 48+ 8 5% 36--47+ months: 27 15% 24--35 months: 65 37% 12--23 months: 45 25% 0--11 months: 32 18% The arithmetic mean of the numeric answers (including the 0 and the 72) is 21.7 months. Conclusions: - Big range, fairly even distribution - Didn't tell us that much Action items: - None, curiosity question. Next time: - Give ranges, and use check boxes, for more automation - Probably better would have been to give the year of first use - And mention that the first public releases were at the start of 2002, for those who were using it right from the start. - Or ditch altogether... well, it doesn't tell us much, but it can be done so that analysing it is very easy. ----------------------------------------------------------------------------- 2. Which of the following best describes how often you use Valgrind? - hourly - daily - weekly - monthly ----------------------------------------------------------------------------- 178 people answered, 1 did not. Of the 178 responses: hourly 2 1% daily 23 13% weekly 84 47% monthly 69 38% Although one person said in q18 that they actually use it quarterly. Conclusion: - 85% of people use it weekly or less often, that's less than I expected. Action items: - None (curiosity question). Next time: - this was pretty good, easily automated - should have a "less than monthly" option ----------------------------------------------------------------------------- 3. What hardware do you use Valgrind on? Estimate the proportion of your Valgrind usage on different machines. (Example: Pentium 4 70%, Athlon64 (32-bit mode) 20%, Athlon64 (64-bit mode) 10%) ----------------------------------------------------------------------------- 177 people answered, 2 did not. A dumb question -- we got way too much detail about different processor types which isn't really that interesting. Also, it's hard to tell with a lot of the answers whether they're using x86 or AMD64 -- an answer like "Opteron" (or even worse "AMD") is not much help with that. Having said that, here is the detailed breakdown. A 2-way "X" or 4-way "X" machines I reported as just "X". I ignored MHz values where given. Sometimes people didn't give numbers, so I gave even proportions to all those named. Some people can't add up to 100%, so I scaled their proportions to total 100% AMD 0.4% AMD (32-bit mode) 0.5% AMD64 1.3% AMD64 (64-bit mode) 0.4% Athlon 2.9% Athlon (32-bit mode) 0.9% Athlon (Opteron) 0.6% Athlon32/classic/K7 1.9% Athlon64 2.5% Athlon64 (32-bit mode) 10.0% AthlonXP 2.5% AthlonXP (32-bit mode) 0.3% Celeron 1.1% Duron 0.2% EM64T/64-bit Intel 0.9% EM64T (64-bit mode) 0.2% Intel 0.6% K6-II/K6-III 1.1% Opteron/Opteron64 1.8% Opteron (32-bit mode) 0.1% Opteron (64-bit mode) 0.8% P2 0.1% P3 5.2% P3 (32-bit mode) 0.0% P4 39.7% P4 (32-bit mode) 0.6% P4 Nocona (64-bit mode) 0.6% P4M 2.3% Pentium 3.9% PentiumD 0.3% PentiumM 2.6% PowerPC 0.0% Sempron 0.3% Xeon 6.7% i386/i686/x86/intel-32 4.4% x86_64 0.1% x86_64 (64-bit mode) 0.4% Wants IA64 0.6% literally everywhere it works 0.6% (Mixed cluster of all sorts of machines...) 0.6% The biggest ones are P4, Athlon64 (32-bit mode), Xeon, P3. To answer the more interesting question of which Valgrind platforms are they using, I grouped them. Although some answers were clearly x86 or AMD64, many were not (eg. "Opteron" could be run in 32-bit or 64-bit modes, some P4s support 64-bit mode, etc). So take this with a grain of salt. Probably x86 48.6% Definitely x86 29.2% x86 or AMD64 10.5% Probably AMD64 7.6% Definitely AMD64 2.3% Wants IA64 0.6% (Mixed cluster of all sorts of machines...) 0.6% literally everywhere it works 0.6% PowerPC 0.0% Conclusions: - x86 is still far more widely used than AMD64 Action items: - None (curiosity question) Next time: - Don't ask for specific machine details (eg. P3 vs. P4), it's not that interesting and many responses are too vague. - Don't bother differentiating between 32- and 64-bit modes on AMD64, it doesn't add anything but confusion. - Better would have been: "What proportion of your Valgrind use is on each platform? * x86/Linux * AMD64/Linux * PPC32/Linux * Other (please specify) - That would have made a good partner for Q9. - Having pre-canned responses would also have made automation easier. - Forcing the input to be numeric and sum to 100 would also make it easier (although we don't want to reject a survey and say "try again" -- that would annoy people). Could have a "don't know" option that starts at 100, and make all the boxes auto-sum to 100. - Perhaps the percentages aren't so important, just ticking checkboxes would be enough. ----------------------------------------------------------------------------- 4. What operating systems(s) do you use Valgrind on? Estimate the proportion of your Valgrind usage on different operating systems. Please give version numbers if possible. (Example: Linux RedHat 9.0 90%, Linux SuSe 8.1 10%) ----------------------------------------------------------------------------- 179 people (every person) answered. Lots of different versions here. I aggregated a lot of them. FC (plain/2/3/4) 17.3% SuSE (plain/7.3/8.1/9/9.1/9.2/9.3/SLES8/SLES9) 16.0% RHEL (plain/3/4/AS2.1/AS3/AS4/ES3/ES3.X/ES/ES4/WS/WS4) 15.6% RH (5.1-hand-upgraded/7/7.1/7.2/7.3/7.x/8/9/other) 11.2% Gentoo (plain/2004.4/unstable) 7.6% Debian (plain/testing/3.0/3.1/3.x/Sarge/Stable/Unstable) 7.2% Linux (2.4/2.6/other/many/custom/unspecified) 7.2% RH 4.0% Mandrake (plain/9.0/10.0/10.1/Cooker) 3.4% Ubuntu (plain/5.01/5.04/5.10/Hoary) 2.8% Slackware 2.1% Mandriva (plain/9.2/10.2/10.x/2005LE/2006) 1.7% CentOS (3/4/4.1/4.x) 1.1% FreeBSD (4.11/6/7) 0.6% Other (ECGL/L-from-scratch/Kurumin/MontaVista/Scientific/Win 2.4% I distinguished older Red Hats (up to 9.0) from RH Enterprise. The "RH" category is for answers just saying "Red Hat", ie. I couldn't tell if it was an older one or an Enterprise one (or even Fedora). Conclusions: - All the Red Hat ones (including Fedora) accont for 48.1%. - The rest is a mix of much smaller proportions. Action items: - None (curiosity question). Next time: - Give a selection of check boxes, plus an "other" box, eg: * Fedora Core * SuSE * Red Hat Enterprise Linux * Red Hat (9.0 or earlier) * Gentoo * Debian * Mandrake * Ubuntu * Slackware * Other - Again, maybe proportions aren't important, just ticking check boxes might be enough. ----------------------------------------------------------------------------- 5. In what form did you/do you obtain Valgrind? - source, from CVS/Subversion repository; - source, from website; - pre-built version (eg. RPM); - already installed on system; - other (please specify below); ----------------------------------------------------------------------------- 179 people (every person) answered. Nb: I intended "source, from website" to mean www.valgrind.org, but I see now that it could mean any website that mirrored the sources. The total number of responses follow. The percentages add up to more than 100% because lots of people mentioned more than one source. source from website 140 78% pre-built (eg. RPM) 39 23% source from CVS/SVN repository 35 21% already installed 28 16% other: gentoo ebuild 9 5% other/pre-built: debian package 5 3% other: FreeBSD ports system 1 <1% "pre-built" was intended to include Debian packages, but 5 people explicitly mentioned Debian in the "other" field, so I've put them separately. Conclusions: - A surprisingly high proportion of people are building from source, from the website. That's interesting, although it doesn't tell us to do anything differently. Action items: - None (curiosity question). Next time: - pretty good - add Gentoo eBuild - change "from website" to "from www.valgrind.org" - clarify pre-built to include Debian packages ----------------------------------------------------------------------------- 6. Which Valgrind tools do you use? Estimate the proportion of your Valgrind usage each tool accounts for. (Example: Memcheck 80%, Addrcheck 10%, Cachegrind 10%) ----------------------------------------------------------------------------- 179 people (every person) answered. Summing the percentages, the total proportion of tool usage is as follows. memcheck 80.7% addrcheck 8.0% callgrind 6.1% massif 1.8% cachegrind 1.8% helgrind 1.4% vprof 0.2% valgui 0.0% Six people made comments about Helgrind along the lines of "we'd use Helgrind if it was currently working". The tools other than Memcheck are used more widely than this indicates. The following is the proportion of people who mentioned using a tool (even if accounts for only 1% of their use). memcheck 178 99.4% addrcheck 47 26.2% callgrind 43 24.0% massif 30 16.8% cachegrind 27 15.0% helgrind 19 10.6% vprof 2 1.1% valgui 1 0.6% It's quite possible that people are using GUIs such as Valgui, but didn't mention it. Conclusions: - Memcheck is still by far the most popular and important tool. - Addrcheck is next most popular, despite it currently not working. The answers don't tell us if people use it rather than Memcheck because it's (a) faster, (b) uses less memory, (c) doesn't report definedness errors. This would be good to know, since there is an idea kicking around of making Memcheck use less memory, and ditch Addrcheck altogether. This would be ok if (b) is the main reason people use Addrcheck, but not if it's (a). ((c) could be done with a command-line switch to Memcheck.) - There are enough Helgrind users that it would be worthwhile to get it working again. - All of the tools in the distro are being used sufficiently that it's worth keeping them in. Conclusions: - Memcheck still dominates greatly - Addrcheck and Helgrind are missed - People like Callgrind Action items: - We should try to bring Addrcheck back, or give Memcheck the characteristics that people like about Addrcheck (unfortunately this doesn't tell us what they are) - We should try to bring Helgrind back Next time: - pretty good - would be nice to force input to be numeric - would be nice to force numbers to add up to 100 ----------------------------------------------------------------------------- 7. Which of Valgrind's command-line options do you always, or almost always, use? ----------------------------------------------------------------------------- 147 people gave an answer, 32 did not. Totalled results: --num-callers 67 45.6% --memcheck:leak-check=yes/full 61 41.5% --tool=memcheck 33 22.4% -v/--verbose 31 21.1% --memcheck:show-reachable=yes 28 19.0% --suppressions 22 15.0% --log-file 22 15.0% --error-limit=no 20 13.6% --trace-children=yes 16 10.9% --db-attach=yes 15 10.2% --tool 8 5.4% --memcheck:leak-resolution=high 8 5.4% -q/--quiet 7 4.8% --memcheck:leak-check 6 4.1% --log-fd 5 3.4% --track-fds=yes 5 3.4% (none) 4 2.7% --time-stamp=yes 4 2.7% --db-attach=no 2 1.4% --memcheck:leak-check=no 2 1.4% --memcheck:leak-resolution=med 2 1.4% --memcheck:show-reachable 2 1.4% --alignment 1 0.7% --alignment=8 1 0.7% --callgrind:base= 1 0.7% --callgrind:dump-instr=yes 1 0.7% --callgrind:simulate-cache=no 1 0.7% --callgrind:trace-jump=yes 1 0.7% --db-command 1 0.7% --demangle=yes 1 0.7% --freelist-vol=10000000 1 0.7% --freelist-vol=5000000 1 0.7% --gen-suppressions 1 0.7% --help 1 0.7% --log-file-qualifier 1 0.7% --massif:format=html 1 0.7% --memcheck:avoid-strlen-errors=yes 1 0.7% --memcheck:workaround-gcc296-bugs=yes 1 0.7% --run-libc-freeres=no 1 0.7% --show-below-main=yes 1 0.7% --tool=addrcheck 1 0.7% --tool=helgrind 1 0.7% use valkyrie GUI front end 1 0.7% Many of the --num-callers answers included a number. The breakdown was: 8.. 16 20 20.. 40 22 50..200 13 unspecified 12 A number of people made comments about 4 being too small for the default --num-callers value, even though we changed it to 12 a while back. I assume the people who set it to 8 or 10 also still thought that the default was 4 (and not that 12 is too high). Conclusions: - Increasing the default for --num-callers from 4 to 12 was definitely a good idea, it might be worth increasing it again, eg. to 20. - Memcheck has --leak-check=summary as the default. --leak-check=full is very popular, perhaps it should be the default. - Making --tool=memcheck the default option was a good thing to do, but some people probably haven't realised that it's no longer necessary. - --show-reachable=yes is popular, but doesn't seem popular enough to make the default, since 61 people use --leak-check=full but only 28 use --show-reachable. - A surprising number of people run with -v. I'm not sure why, only reason I'm aware of is that one person said it gives how many times each error happened (ie. how many duplicates per error), although another person complained about -v output being too verbose (especially the redirection info). So for -v perhaps we should separate the things that users might be interested in (eg. per-error counts) from the things that only developers would be interested in (eg. redirection info). Action items: - Should make --leak-check=full as the default for Memcheck. - Should increase --num-callers default from 12 to 20. - Perhaps change what gets shown with -v. - Advertise some of the changes that people haven't realised in the release notes of the next release (eg. --tool no longer mandatory). Next time: - Can probably merge this with Q8, the distinction isn't that important. - could list all options with a check box, but that's a headache, especially if it's on the site over a period of time (as options get added/removed) - but letting users specify them means they sometimes get them wrong - also, for some you want the values (eg. --num-callers) - so, tricky... ----------------------------------------------------------------------------- 8. Which of Valgrind's command-line options do you sometimes use? ----------------------------------------------------------------------------- 107 people answered, 73 did not. Totalled results: --db-attach 32 29.9% --memcheck:leak-check=full 23 21.5% --memcheck:show-reachable=yes 16 15.0% --memcheck:leak-check 13 12.1% -v 11 10.2% --track-fds=yes 10 9.3% --gen-suppressions 9 8.4% --trace-children=yes 7 6.5% --log-file 6 5.6% --num-callers 6 5.6% --db-command 5 4.7% --suppressions 5 4.7% --leak-resolution=high 4 3.7% --error-limit=no 3 2.8% --log-fd 3 2.8% --tool 3 2.8% --callgrind:collect-jumps=yes 2 1.9% --callgrind:collect-systime=yes 2 1.9% --callgrind:separate-threads=yes 2 1.9% --help 2 1.9% --leak-resolution=med 2 1.9% --workaround-gcc296-bugs=yes 2 1.9% None 2 1.9% --callgrind:collect-alloc=yes 1 0.9% --callgrind:dump-instr 1 0.9% --callgrind:separate-callers=4 1 0.9% --callgrind:separate-recs=30 1 0.9% --freelist-vol=200000 1 0.9% --gen-suppressions=all 1 0.9% --leak-resolution 1 0.9% --log-socket 1 0.9% --massif:alloc-fn 1 0.9% --massif:depth=20 1 0.9% --max-stackframe 1 0.9% --memcheck:leak-check=no 1 0.9% --memcheck:partial-loads-ok=no 1 0.9% --num-callers=15 1 0.9% --num-callers= 1 0.9% --smc-check 1 0.9% --stack-limit 1 0.9% --time-stamp 1 0.9% --tool=cachegrind 1 0.9% --trace-malloc=yes 1 0.9% --trace-syscalls=yes 1 0.9% --use-xml=yes 1 0.9% --verbose 1 0.9% --version 1 0.9% --workaround-gcc296-bugs 1 0.9% -q 1 0.9% -vv 1 0.9% All of them 1 0.9% The following are the core and Memcheck options not mentioned in any answers to questions 7 or 8: --log-file-exactly --weird-hacks (now --sim-hints) --pointercheck (since removed) --show-emwarns --xml-user-comment --input-fd I guess an argument could be made for removing these, but in each case there's a good reasons for having it. Conclusion: - not much that question 7 didn't tell us Action items: - See question 7 Next time: - See question 7 ============================================================================= Valgrind on Other Platforms ============================================================================= ----------------------------------------------------------------------------- 9. If Valgrind was available on all hardware/OS platforms, estimate what proportion of your usage would be on each platform. Please describe each platform as precisely as possible. Example answer: - PowerPC/MacOS X : 50% - SPARC9/Solaris : 25% - x86/Linux : 20% - IA64/Linux : 5% ----------------------------------------------------------------------------- 156 people answered, 23 did not. x86/Linux 46.4% AMD64/Linux 10.7% existing 8.1% IA64/Linux 4.8% x86/Windows 3.9% AMD64 3.2% PPC/MacOS X 2.7% SPARC9/Solaris 2.3% SPARC/Solaris 2.2% Linux 1.6% PPC/Linux 1.4% Windows 1.2% ARM 1.0% PA-RISC/HPUX 0.9% PPC/AIX 0.8% Solaris 0.6% x86/FreeBSD 0.6% Symbian 8.0 0.6% Alpha/Tru64 0.6% IA64/HPUX 0.5% FreeBSD 0.5% PPC 0.5% SPARC 0.4% AMD64/Windows 0.3% ARM/Linux 0.3% OpenBSD 0.3% HPUX 0.3% MIPS/IRIX 0.3% SPARC/SunOS 8/9 0.3% x86 0.3% x86/OpenBSD 0.3% AIX 0.2% SunOS 5.8 0.2% SPARC9 0.2% zSeries/zOS 0.2% IA64/BSD 0.2% SPARC8/Solaris 0.2% x86/MacOS X 0.1% Power/AIX 0.1% SPARC9/Linux 0.1% x86/Solaris 0.1% IRIX 0.1% MIPS32/HPUX 0.1% Windows (MinGW) 0.1% Windows (cygwin) 0.1% x86/WinXP (cygwin) 0.1% Various PDAs (Linux & Windows) 0.0% x86/UnixWare 0.0% s390/s390x/Linux 0.0% "existing" means one of the platforms Valgrind already supports (x86/Linux or AMD64/Linux), but it was unclear which one. Since some people mentioned only a CPU or only an OS, it's worth considering the CPU and OS in isolation also. x86 51.9% AMD64 14.1% SPARC 5.7% IA64 5.5% PPC 5.3% Linux 65.3% Windows 5.7% Solaris 5.5% MacOS X 2.8% HPUX 1.8% These percentages are out of less than 100%, because less than 100% of responses mentioned both a CPU and an OS. Conclusions: - x86 and AMD64 are easily the most important - IA64 is more popular than I expected - MacOS X is less popular than I expected - This question doesn't tell us as much as it might seem. For example, if Valgrind were available for Windows I'm sure that would account for most usage, but because most Linux developers don't do much development on Windows, Windows doesn't get mentioned very much. For this reason I think our main target platform of PPC32/MacOS X is still a good one; I think that will expose Valgrind to a large group of new users. - In contrast, I think the overlap between Linux and BSD developers would be much higher, and so the underwhelming responses for BSD show that a BSD port would not achieve very much. Next time: - not very well done - make it clear that we want them to include the existing platforms (eg. x86/Linux) in their answer. - force them to give a CPU and an OS -- many gave only one or the other. - perhaps pre-canned pairings would make sense, plus an "other (please specify)" option: * x86/Linux * x86/Darwin * x86/Windows * AMD64/Linux * AMD64/Windows * PPC/Linux * PPC/MacOS X * IA64/Linux * SPARC/Solaris * PA-RISC/HPUX * Other1 (please specify) * Other2 (please specify) * Other3 (please specify) ============================================================================= Software Issues ============================================================================= The following questions are about software issues, eg. Valgrind's licence, speed, memory usage, ease of use, quality of information produced, robustness, documentation, etc. ----------------------------------------------------------------------------- 10. List up to 3 existing features of Valgrind that you think are good. Give as much detail as necessary. ----------------------------------------------------------------------------- 145 people answered, 34 did not. I grouped the comments about good features into the following broad categories. memcheck/memory checking 118 ease-of-use 40 other 27 accurate information 19 robust/stable 19 GPL 16 speed 13 general praise 11 error messages/output 7 Callgrind/KCachegrind 7 documentation 5 suppressions 5 profiling 4 Addrcheck 3 Massif 3 debugger attachment 3 price 3 ease-of-use: other 3 Cachegrind 2 extensibility 2 memory use 2 Memory checking is clearly the favourite thing here. Ease-of-use is second. Robustness and the GPL license are up there too. Interesting are some that get little mention, eg. price. Breaking down the biggest categories a little, we get the following. memcheck/memory checking: - general praise 41 - leak checking 32 - undefined value checking 20 - invalid access checking 18 - other 7 ease-of-use: - no recompilation 20 - easy to use 20 It's interesting that leak checking is considered the most valuable kind of memory checking; I didn't expect that. The following are the details, grouped into the above categories. I aggregated similar comments, and paraphrased many of the remaining ones. The numbers show how many times the comment was made. For some of the aggregated ones, I include interesting comments after '*' symbols. (And I use this approach for many of the questions that follow, too.) Memcheck/memory checking: - leak checking 32 * Valgrind through this feature has easily doubled my productivity. * precise memory leak checking is impossible to do without a tool like Valgrind * The best memory leak reporting system available anywhere! * Pinpoints memory leaks and errors efficiently and accurately. * Leak detection showing a deep stack trace of the allocating request. * leak origin stacktrace * Backtraces for detailing leaked memory are extremely useful. - memory checking/Memcheck/Memcheck is good 41 * I mostly use memcheck, and it works fine for me. * Everything feature memcheck has is good and useful * It's nice to have my errors handed to me on a silver platter. * Memcheck is indispensable * memcheck - good tool for detailed memory access. * The memcheck tool is great! Sometimes a bit verbose though. The suppress configuration is a bit tendersome to get right. * memcheck: Good, excellent when the new address-space stuff is in. * The memory checker, obviously. * As far as I recall, valgrind let me down only once. * Ability to track and find heap corruption issues is the number one reason I use and recommend this tool. * The promised errors are detected. * it helps finding errors quickly * When mem corruption takes place, the info about where the memory was allocated/deallocated is useful. * catching mem/addr errors as an alternative to Purify * Ability to find bugs * With valgrind/wine I was able to pinpoint exactly an intermittent application problem without even having the sourcecode that the writer, a fellow windows developer, could not find within 2 weeks. - undefined (a.k.a. uninitialised) value errors 16 - laziness of undefined value checking 3 - undefined value errors for system calls 1 - invalid access/buffer overflow checking 18 * although the error message has not always been clear * Use valgrind all the time for this and LOVE IT for that! * It's really helpful when it shows you the place of first wrong memory access and you don\'t have to laborously trace the bug back from the (sometimes very distant) place where the consequnces caused segfault. - invalid free detection 1 - malloc/free vs new/delete mismatch checking 1 - few false positives 5 * Errors generated are genuine ... (true only for addrcheck) Ease-of-use: - no recompilation 20 * infinitely easier to use than Purify and can be used on third-party software. * This alone is such a huge issue. * you do not need to have access to all the sources of the tools you are using... for me the top most quality ... (without that, we could not use it). * really helps us diagnose problems occuring in the field * huge advantage - "easy to use" or similar 20 * debugging is easy with valgrind (when in doubt, valgrind :-)) * transparency/ease of use * Learning curve is quite fast * Quick to get up and running. - no code changes necessary 2 - easy to use (callgrind) 1 * quickly points to the bottleneck(s) of the code - works even if symbols are missing 1 * useful with third-party code Accuracy: - "information is good/accurate"/"quality of information" or similar 19 * Very thorough in the problems it finds * gives excellent results in a meaningful and clear way * gives exactly what I need (good information quality) Robustness/stableness: - robust/stable/reliable 17 * If my program is so buggy that it manages to crash valgrind, valgrind gives almost always a usefull message before crashing * I can't recall ever crashing it... * The most stable software solution, even compared to very expensive commercial products. * pthreads support has issues, but certainly quite good * it mostly works - works for c/c++/gcj 1 - works with nvidia opengl 1 GPL: - GPL/Free software/open source 16 * no license server * can run anywhere * can fix problems in the Valgrind source Speed: - speed 13 * Almost always fast enough (I wouldn't turn down more speed, though :-). * Moderately fast for a memory checker (memcheck) * small slowdown penalty compared to other memory debuggers or profiling tools * I appreciated a lot the speed improvment in the support of multithreaded processes (between v2.2 and v3) General praise: - general praise 11 * the idea of valgrind itself :-) * everything is fine for me, well done * quality * unique product for Linux * I never used any other software like Valgrind, and Valgrind seems perfect to me. It's letting me to save a lot of time. I'm also trusting more my code. * It works, and works well. That's a _lot_. * smartness * I like all of it. ;) * most things are good * And it simply works! * Valgrind is fantastic. Error messages/output: - verbosity 2 * Can't live without this, need lots of detail, but it's also nice to scale it back sometimes. - Plain text output is fantastic. 1 - command line help good 1 - stacktraces good 2 * stacktraces (of user defined length) + time stamps allow to pinpoint where in the program execution a problem was detetcted * backtraces for when memory was freed in the case of a double-free - complete traces for analysis 1 Other tools: - Callgrind/KCachegrind 7 * Kcachegrind is awesome. * With callgrind/cachegrind, I was able to speed up an algorithm by a factor of 2 whose author believed it was already optimized to the max. * Profiling using Callgrind is invaluable. We can't get other profilers to work with dlopen() symbols and shared libraries. * Cachegrind (together with kcachegrind) ist one of the most useful tools on the planet! * call graphs - Addrcheck 3 * I use it in my devopment machine most of the time. * is really a life-saver. i don't use the other tools often. * Catches most of the errors with a very small overhead. With over 4500 hrs of tests to run addrcheck is great. - Massif 3 - Cachegrind 2 - Helgrind 1 * it helped me track down a race that I didn't see staring at the code. Documentation: - documentation good 5 Suppressions: - suppressions good 5 * Easy to track memory errors and suppress them * Suppressions rock! * suppressions - essential to work with third-party libs * Suppression generation is also useful so I can run automated test scripts against my project and filter out problems with GLIBC etc. Which means I get less false positives and my results are actually useful. * One of the most imporatant is "factory made" autoconfigured set of default supressions. Profiling: - the possibility to perform the combination of instruction,syscall, and cache instrumentation. 1 - Accurate profiling data 1 - optimizing code (cache, etc.) is easier 1 - profiler included 1 Price: - cheap 3 * As good as the only competitor known to me but 'infinitely' more affordable. Modular architecture/ability to write new tools: - extensibility 2 Memory use: - memory use 2 * Does not take too much memory. Misc: --db-attach is useful 3 - AMD64 support 2 - --trace-children is good 2 * Ability to trace child-processes is way ahead of the competition - Client requests 2 * allowing a test harness to run tests under valgrind and report issues is a killer feature * my program is an interpreter with its own garbage collection and so does not rely that much on malloc(1) - can be used in automated testing 2 * Commandline interface allows it to be integrated into test scripts. * possibility to integrate into automatic testing (--log-fd, more or less machine-readable output) - that it exists 2 - ability to integrate with custom memory allocators 1 - output to a file descriptor 1 - language independent 1 - fd tracking 1 - nested functions support 1 - good support 1 - lots of command options 1 - code coverage 1 - all basic tools 1 - flexibility 1 - opacity 1 - Code is clean and great. With some effort I am able to understand the code and add/modify to my requirements 1 - don't know 1 - support for bug-reports 1 - Support for multithreaded apps about a specific one 1 - thread tracing. VERY useful when trying to write software employing the most poorly written library on earth, all incarnations of libresolv 1 Conclusions: - memory checking is the main reason people like and use Valgrind; we should not forget this! - People like the ease-of-use, we should not compromise this. Action items: - none Next time: - it's a good, important question - hard to automate -- I don't want to pre-specify categories or have pre-canned answers, I feel like that's biasing the responses too much. - although having three input boxes so the division between multiple responses is clearer would be good! - say "features or attributes" rather than "features" -- people might not realise they can say "speed" because it's not an explicit feature? ----------------------------------------------------------------------------- 11. List up to 3 existing features of Valgrind that you think need improvement. Give as much detail as necessary. ----------------------------------------------------------------------------- 130 people answered, 49 did not. I grouped the comments about things that need improvement into the following broad categories. no answer 49 speed 45 suppressions 23 simulation correctness/robustness 21 error messages/output 20 Memcheck's error detection 18 memory use 15 debugger attachment 11 want Helgrind back 10 Callgrind 10 documentation 10 command-line flags 7 Helgrind didn't work well 6 better portability 4 build process 4 don't know 3 Massif 3 other 7 Of these, speed is clearly the biggest issue. Also, most of the other categories involve comments about different things within that category (eg. the 23 comments about suppressions cover several different aspects) whereas the speed comments are almost all about the same exact thing. The following are the details, grouped into the above categories. I aggregated similar comments, and paraphrased many of the remaining ones. The numbers show how many times the comment was made. For some of the aggregated ones, I include interesting comments after '*' symbols. Speed: - speed [non-specific] 41 - speed (Memcheck) 3 - speed (Callgrind) 1 * but I understand it's hard 5 * definitely most urgent issue * unusable for programs with billions of allocations * cannot test large GUI program properly because of slowness * Athlon64 speed seems much better than on Pentium4 * far too slow for realistic runs for my application * especially threaded programs seem to be really slow * especially Callgrind * startup seems much slower on 64-bit than 32-bit platform Suppressions: - should be able to dump gen'd suppressions to a file 4 - suppressions [non-specific] 3 - want better suppressions for errors in the system/standard libs 3 - easier suppressions 2 - ability to say that callstack is only entries deep and no more 1 - many nVidia drivers errors cannot be suppressed 1 - better syntax 1 - seem to fail sometimes (perhaps a bug?) 1 - C++ names should be demangled 1 - Central repository for common ones (eg. stl ports, common libs) 1 - suppressions for standard libraries (Debian) 1 - The "[Return/N/n/Y/y/C/c]" prompt is unclear 1 - a suppression package (hosted by valgrind, with community contributions) should be included; now one has to google to find them 1 - --gen-suppressions should be copy-n-pastable into a file (perhaps it is these days?) 1 - with --gen-suppressions=yes, show question for each suppression only once, not over and over again 1 - default suppression list - maybe as a command line options, sth like: --suppress-XXX-bugs, which XXX might be: x11, xft, libc, pthread, qt, gtk, etc. Simulation correctness/robustness: - Better instruction set coverage 5 - better/more complete threads support 3 - Use of multiple CPUs when available (e.g. for threaded programs) 1 - Sometimes changes the program behavior 1 - JAVA support would be *really* swell 1 - signal handling 1 - scalability, robustness (much improved in 3.0) 1 - sometimes seg faults 1 - our highly multi-threaded app currently doesn't start with Valgrind 1 - reports garbage with MPI programs 1 - when my application crashes, sometimes it turns Valgrind into a zombie process 1 - can't run MPI under Valgrind 1 - some limits were hard-coded (e.g. the number of rwlocks) 1 - need more thread support, especially backported to the 2.4.1 code; I cannot run on older systems due to lack of thread and gettid() implementation 1 - BoehmGC programs trap SIGSEGV, sometimes doesn't work (esp 2.4.1) 1 Error messages and other output: - undefined value errors -- should pinpoint variable, not just the line 3 - leaks: want to know when they occur 1 - change too often, makes auto-parsing difficult 1 - functions sometimes missing in stack trace, even with -g 1 - somewhat confusing 1 - output formatting is poor; xml/valkyrie helps but more is needed 1 - too verbose, should be able to filter non-app stuff (eg. system files) 1 - stack traces entirely in libraries (eg. ld.so) are irritating 1 - would be nice to see a source code snippet with the stack trace 1 - could use better visualization of results 1 - default stack trace depth of 4 is too small 1 - Purify's error acronyms are nice 1 - filled with irrelevant data, could be augmented with more useful info such as variable names 1 - should come with a tool that can parse the output and easily slap a GUI on top of it, or a summary HTML page, etc. 1 - always outputting process ID 1 - unloading shared objects causes empty stack traces 1 - upon a Valgrind crash, should still produce final output (eg. massif) 1 - "Detection of duplicates in the reports" 1 Memcheck's error detection - undefined value errors difficult to track down, should give info about root cause 5 * "I ABSOLUTELY HATE the uninitialized variable policy" - want stack/static array overrun detection 3 * fat pointers could help - "checking of global variables (C)" 1 - "Ability to find bugs" 1 - sometimes fails to detect uninitialized memory access 1 - spurious undefined value errors with bitfields(?) 1 - parameter checking for all libc functions and OS calls 1 - false positives in code using long doubles 1 - leak checking: there is some global or static object that will not freed when program exit, hope valgrind can detect this 1 - shared memory - 3.0.1 is flagging false positives when the SAME process attaches to the same segment; seems worse than 2.4.1 1 - with libresolv, hard to tell if problems are in the library or the client code 1 - unnecessary leak warnings with STD [STL?] C++ strings 1 Memory usage: - memory usage 12 - memory usage (I understand) 2 - memory usage (cachegrind,callgrind) 1 * much improved in 3.0 * Valgrind in UML often crashes due to use of too much memory (only 256MB available) * especially without addrcheck in 3.0.X * "compressed V bit" idea sounds good * for many malloc calls or large amounts of memory, Valgrind's malloc will fail causing the program to exit. * debug info not loaded in big executables * need to run regression tests in small blocks to get around memory limits * new aspace manager should help? * Big improvements in code in Subversion * Should be able to run well programs that use 20GB or more * Should handle more than 2GB memory usage Debugger attachment: - doesn't work well 4 - Continuing after dbattach seems to fail for me 1 - interaction with GDB is weak (esp. can't continue execution) 1 - debugger invocation on a watchpoint would be useful 1 - often doesn't work: hangs trying to attach 1 - want to run programs under Valgrind within the debugger 1 - hard to attach db 117th time a line is reached; more detailed rules could be nice, a la suppressions 1 - on each error you should be able to investigate in GDB the value of data, etc 1 Helgrind: - want it working again 9 * DIOTA gives better reports * only works when C++ allocators use new and delete directly * Helgrind is critically needed * there is no other good, GPL thread/data race debugger * has problems, but good potential, I look forward to its return * never used it, but sounds useful - many false positives 4 * does not understand atomic increments and decrements * makes it almost unusable - info given often not enough to pinpoint the data race 1 - didn't work for me (don't remember details) 1 - a way to check for race conditions and other shared resources would help debugging multi-threaded apps 1 Callgrind: - better AMD64 support 1 - ability to track internal times regardless of cycles 1 - ability to see function chains that caused cycles 1 - difficulty getting Callgrind to produce results 1 - difficulty running it on one machine, viewing results on another 1 - not ported to 3.0.X 1 - "cycle" concept hard to understand 1 - documentation is improvable 1 - should be released soon after Valgrind releases 1 - call graphs are not very correct 1 Documentation: - examples of non-trivial errors 2 - more info about compilation parameters 1 - better docs about what each error message means 1 - better docs on writing syscall/ioctl wrappers 1 - "how Valgrind works" inadequate 1 - "how to write a tool" inadequate (see PIN docs) 1 - frequent changes make architecture hard to understand; lack of documentation 1 - want a precise description of V bit propagation through FP ops 1 - it was some time before I realised duplicate errors are combined, and that -v showed the per-error counts 1 Flags: - bad defaults 2 - --num-callers should be a bigger value by default like 10 1 - flag names too long 1 - flag names non-intuitive, hard to remember 1 - --help should list all available tools 1 - -v has too many msgs now with the REDIR: etc 1 Portability: - more platforms 3 * "1. Platform support. 2. platform support. 3. Platform support." - support for PPC/MacOS X 1 Building: - can't build 3.0.1 on SuSe7.3, Mandrake10.0, Mandrake10.1 1 - VPATH builds don't work 1 - --program-suffix fails for run-time libraries 1 - installation is a hassle 1 Massif: - nice graphs but not enough control over output 1 - should give info on intermediate memory state (with allocation trace) in textual/html form, not only in graphic form 1 - memory allocated by libraries should be connected to places in application code which invoked them 1 Miscellaneous: - Addrcheck is broken in 3.0.X 2 - Valgrind/Wine integration 1 - Intel compiler support could be better, in particular ifort 1 - --with-gdb would be good 1 - should hack GCC to allow auto-detection of custom allocators 1 - Plugins is probably a good idea; but for me as a user I don't care that different tools exists - I just want to pick the features I need 1 Conclusions: - Speed is by far the main issue, mentioned twice as much as anything else. - People find suppressions hard to use, and errors from system libs are a problem. - Robustness is important, but the comments are mostly spread across a range of different problems. The main exception is that better ISA coverage would be appreciated. - Various comments about Memcheck's error detection, mostly spread across a range of things. The main exception is lazy undefined value reporting which some people dislike (some strongly!) - Reducing memory usage would be good. (Hopefully 3.1.0 will help with this a lot.) - Debugger attachment has various problems. - Ten people want Helgrind back (but another six say it doesn't work well!) - Not too many complaints about documentation, perhaps surprisingly. Action items: - Should do more performance tuning. Now possible with Cachegrind and self-hosting. This will make many people happy. - We should monitor how effectively the aspacem rewrite helps memory usage problems; if there are still problems, we should try the compressed V bits representation. - We should consider how to improve suppressions. Just dumping auto-generated ones to file would be a good start. More flexible and simpler syntax would help too. - We should perhaps try to improve the understandability of error messages. - We should work out if debugger attachment can be made more reliable. - We should keep Helgrind reinstatement as a goal. - We should investigate a strict definedness option; past experience has shown it to not work well but perhaps things could be improved. - We should improve ISA coverage. Next time: - probably the most important question in the whole survey - should combine with Q12, they're very similar - again, hard to automate ----------------------------------------------------------------------------- 12. List up to 3 missing features you would like added to Valgrind. ----------------------------------------------------------------------------- 104 people answered, 75 did not. Summary: other 59 checking of static/stack bounds errors 14 more platforms supported 13 better debugger integration 10 GUIs 10 Helgrind reinstated 5 start/stop monitoring at will 5 none 5 don't know 3 better AMD64 64/32-bit support 3 callgrind in the distro 3 option to ignore errors in certain libraries 3 reference counting support 3 Callgrind for 3.0 2 variable names in error messages 2 XML output 2 show the line(s) of code in error messages 2 option to report undefined value errors eagerly 2 Details: - checking of static/stack bounds errors 14 * This could be done by attaching an "extent" to each pointer and ensuring that indexing and arithmetic does not result in the pointer exeeding the "extent". * this might require compiler changes * Anything to learn from annelid ? * One solution that would be great for us is that an compile option to GCC would allow valgrind to monitor stacks. * Probably impossible without compiler support but would be nice. - more platforms supported 13 * What I really miss is a V.'s port to Solaris. * valgrind for IA64 * Windows support * Native support for FreeBSD. Not a feature, but still a desire :-) * Xen/Linux support * I'd love to be able to track FD-leaks, memory leaks, and uninitialised memory reads in Windows executables run under Wine. I tried the special valgind-for-Wine, with no success. * Valgrind on Intel x86_64 Bits architecture !! * Windows platform support (2000/XP line shall be enough) * Symbian OS support (ARM architecture, non-glibc) * Windows Mobile Support (ARM srchitecture, also non GLIBC) * a Sparc port and an AIX port would be really useful. - better debugger integration 10 * Cool coupling with debugger (ddd) * When an error is detected, freeze the debuggee and attach a debugger (e.g. gdb) so that I can immediately inspect the situation. * tighter integration with or into gdb... * More ways to attach to the debugger. E.g. when malloc() return a pointer that was leaked in a previous run. * Ability to issue queries from gdb command line, such as a valgrind macro to list memory leaks at that point. Maybe you can do that now. * Difficult to attach Totalview debugger. * embed valgrind into gdb; whenever a memory bounds check fails, gdb stops and you can start debugging. * It would be nice to be able to attach a debugger if a certain non-error condition is true. Similar to GDB's conditional breakpoints. * integration with debugger and editor, for example DDD and nedit, where user would be able to stop the valgrind execution on error, examine the variables in debugger, and see exact source code position in editor. I did have this functionality in Parasoft's insure - GUIs 10 * especially allowing access to source code from with the GUI * Valkyrie is not usable yet. * An easy to use GUI front-end for suppression management. * A GUI front-end (or integration into one) with possibility of merging results from several runs, marking reports as fixed, adding annotations to reports. * GUI access to information, - Helgrind reinstated 5 * helgrind is "missing" in 2.4.x upwards, please please make it work, and improve it. * Non-working since late 2.x, I was using it frequently before that. - Start/stop monitoring at will 5 * ONE BIG FEATURE: It would be great to be able to run x86 nativily upto a specific point, then start "grinding"... I would *love* that feature. I often have programs that needs to parse and pre-process intensively before the problematic code is run. * When running applications that start sub-processes, I'd like to tell valgrind which processes to track (or ignore) - none 5 * I don't think there is too much missing if anything from Valgrind. I love it, and can't live without it. - don't know 3 - better AMD64 64/32-bit support 3 - callgrind in the distro 3 - option to ignore errors in certain libraries 3 - reference counting support 3 * Detection of reference counting cycles (such as boost::shared_ptr). * Better tracking (through user extension) of custom resource management code - e.g. for refcounting - callgrind for 3.0 2 - Helgrind working better 2 * less false positives - support for tracking down why a value is uninitialized 2 - show the line(s) of code in error messages 2 - option to report undefined value errors eagerly 2 * PLEASE add the option - Variable names in error messages 2 * Variable names along with Uninitialized errors from mecheck just like - XML output 2 Misc - callgrind should give time percentages 1 - should be able to omit the "=yes" on command-line options 1 - command-line options should have short versions 1 - Logging & offline processing would be nice. 1 - A memory load graph (how much memory was allocated/freed at which time/instruction line) would be good. 1 - I know there is pthread support. Can an MPI-valgrind be settled? it could then detect/report errors on other nodes? 1 - allow to check JNI-code while running java (may be a problem of Suns JDK, not valgrind...) 1 - Annotate existing code to mark data as initialized. 1 - Better report generation. 1 - Number the allocated blocks sequentially, report allocation number in traceback, provide API to get the number for a given block. 1 - ability to query definedness of variables within GDB 1 - loading of debugging symbols external to the program, i.e. from files provided in RH's -debuginfo packages. 1 - Memory tracking. Give every alloc a sequence number. Then, restart valgrind with an option like --track-alloc=1234, and valgrind tells you stuff about the pointer. 1 - I was going to say something about memory usage analysis but I see I need to look at massif. 1 - Signal handler reporting 1 - levels of debug output 1 - JAVA support would be *really* swell. 1 - API for apps to be able to make it possible to redirect valgrind output to debug log files that the apps generate normally. 1 - check for own memory block overrides, e.g. by allocating more space than necessary and watching this dummy bytes for reads and writes. 1 - speed, speed and speed ;-) 1 - Being able to valgrind recent python versions 1 - more details for memory faults & leaks 1 - Shared memory inter-process support, kinda tricky to do though. 1 - checking of mutexes and shared memory handling 1 - "Linking" valgrind to a binary at compilation time so it's completely under valgrind control from the beginning and I could replace some binaries in the production system by "instrumented" valgrind binaries without changing anything else. Three votes for that :-) 1 - get intermediate reports while executable is still running. (controlled by signal, with an API called from the client) 1 - detect when the last reference to a live memory block gets lost 1 - exporting results as reports from kcachegrind. 1 - enable GET/SET_VBITS client requests again in memcheck 3.x 1 - checkpointing, for resuming execution later 1 - Performance analysis (replacement for gprof, etc). 1 - Selectable CPU model. Sometimes it would be nice to be able to run the program as if it was running on CPU without SSE or MMX extensions. 1 - Ability to mark memory areas as read-only, eg to find functions that modify their arguments when they should not. 1 - including scripting-language stack trace as part of stack traces (specifically Lua) 1 - ability to count syncwait time (i.e. the time spent waiting for locks) 1 - Better documentation of VG Macros and inserting commands into source code. Possibly a tutorial explaining how to use Valgrind to debug a custom memory allocation framework. 1 - Easier to use heap profiler - I've never managed to get anything too useful from massif when profiling gtk+ apps 1 - checking of suspicious return addresses 1 - show demangled Ada names in stack traces, for progs compiled with gcc-gnat 1 - better support of shared memory 1 - ability to annotate call stacks with arbitrary info 1 - --gen-suppressions=all - don't bother prompting, just generate the suppression (the prompt interacts badly with --trace-children=yes) 1 - Better alignment control.. Valgrind on x86 can sometimes find align problems which would show on hppa or sparc. Finding more could be nice... 1 - Getting some feature-inspiration from Purify could perhaps improve valgrind even more 1 - When running valgrind in test suites, you need to be able to tell whether there have been any error reported by the tool. 1 Conclusions: - There's no single greatly asked-for feature, which is a good sign. - We'd love to add stack or static checking, but we don't know how. An experimental tool ("Annelid") was written a while back which tried to add bounds to all pointers, but it didn't work very well. Doing full-blown bounds checking without access to source code is very difficult. - As for more platforms, we do hope to add a small number more, eg. PPC32/MacOS X. But the people asking for uncommon platforms are going to be out of luck -- we don't have the resources to maintain many platforms. - As for GUIs, there are already multiple (eg. Alleyoop, Valkyrie). Perhaps they need to be improved, but it seems people aren't aware of them. Action items: - none (some things were covered in the previous question, others were not mentioned enough to force action) Next time: - combine with Q11 ----------------------------------------------------------------------------- 13. Do you have suggestions for new tools that could be built with Valgrind? Give as much detail as necessary. ----------------------------------------------------------------------------- 57 people answered, 122 did not. The most popular suggestions were: no suggestion 10 code coverage tool 9 time profiler 8 function call tracer 3 just improve existing tools 3 resource usage measurer 2 flight recorder/time machine 2 other 30 The detailed responses follow. - no suggestion 10 - code coverage (a la gcov, purecoverage) 9 * look at http://emma.sf.net * with a simple text output - time profiler 8 * attribute time spent to particular loops and inlined functions * quantify-like tool * Something easier to use than KCachegrind with output similar to gprof would be nice. * application profiler * I guess, but the things gprof can't do are already in Cachegrind. - just improve existing tools 3 * concentrate on improving memcheck, and its interaction with gdb. * Don't waste time on polluting your great software with too many features. Strengthen the existing ones to the highest quality and speed level as possible. That's harder than adding new stuff, but definitely worth it. * Batteries are already included... strace, ptrace and gdb are good supplements to valgrind. But I don't think that integration would be an advantage. Valgrind should do what it is good at. And it already does! ;-) - function call tracing 3 * with detailed arguments and return values * used to generate a call-structure chart - resource usage measurer 2 * eg. 100 mutexes created, max 50 in use at once, 1000 locks of mutex X, 200 of those required blocking, 500 instances of class X allocated, 200 in use at once. * How many objects of class xy have been created, what was the peak count of such objects, sizeusage of classes - flight recorder/time machine to allow controlled reverse execution 2 - memory profiler with better per-function attribution 1 - Something to profile amount of data placed on stack 1 - foreign code sandbox 1 - hard-disk usage profiler 1 - branch direction collector, for FDO 1 - automatic regression notifier, as part of test suite 1 - "movie time" [24 frames/second] visualization by cachegrind. 1 - could callgrind be included in the main distrib? 1 - on-the-fly graphical version of massif 1 - API usage validator, using external markup 1 - gui tool that helps with suppression file setup. 1 - Have addrcheck working soon. 1 - A simple GUI frontend 1 - parallel Valgrind that worked well with MPI 1 - a --max-memory option to limit memory use 1 - memory profiler 1 - I'd love a tool which could discover if a function might throw a C++ exception which is never caught. 1 - A memory fragmentation profiler 1 - Floating-point precision tracer 1 - "Ask Igor"-style bug detection 1 - as-it-happens leak detection 1 - maximum memory usage statistics 1 - run-my-program-X%-slower tool 1 - tool more aggressive than addrcheck but less than memcheck 1 - A software testing tool 1 - It did occur to me that V has a better idea about the kernel syscall interface than the kernel does. Not sure what you could do with this though other than beat them around the head with it. 1 - profiling!!!! call graph info 1 - Count or other measure of complexity of allocations and deallocations... to pinpoint where better memory management might be needed 1 - coupling valgrind/ddd, coupling valgrind/eclipse 1 - A GUI interfaced with a GDB GUI (a la ddd, but with a modern interface) 1 Conclusions - Coverage tool is the most popular - Time profiler is almost as popular, but it would be hard to do since Valgrind perturbs execution time so much. It might be possible to estimate execution times using a simple cost model with Cachegrind. - Lots of other suggestions that only came up once, and so are not worth pursuing. Action items: - It might be worth working on a coverage tool. Some work has been done on that front already by various people. Next time: - difficult to analyse, but again hard to improve ============================================================================= Non-Software Issues ============================================================================= The following questions are about non-software issues, eg. Valgrind's website, release frequency, bug handling, mailing lists, publicity, etc. ----------------------------------------------------------------------------- 14. List up to 3 aspects of Valgrind's development that you think are good. Give as much detail as necessary. ----------------------------------------------------------------------------- 66 people answered, 113 people did not. Summary results. mailing list/developers are good/responsive 31 other 12 development speed/activity 9 website 8 frequent releases 7 open source/free software 6 high quality releases 6 don't know/no opinion 6 bug handling 5 public repository 4 stability 4 AMD64 and other arch support 3 easy to build/install 3 general praise 3 documentation 2 transparent development 2 code easy to understand 2 Developer responsiveness is easily the most popular thing. The new website is much preferred over the old one, too. Detailed results. Support: - mailing list is good/active/helpful 14 * Valgrind users mailing lists - thanks for your help! * Most of the questions are answered right away in the Valgrind forum. - developer support/responsiveness 12 * quick response on bug-reports * The fast (sometimes immediate) response of its developers to comments/questions/complains of V.'s users. * Dedicated developers & good community contact, developers are open for suggestions, and accept community patches (I saw few OS projects where main developer(s) are reluctant to change their way, even if problems are obvious). * great support, often from Julian. * valgrind developpers are responding very quickly and efficiently to questions/bugs/... * Impressive responsiveness to problems/bug reports. * Excellent response to user problems; much better customer service than many "commercial" packages. - developers active on mailing list 5 * The fact that developers read the mailing lists and respond is really nice * personal support from the main developers * Quick informative responses from developer to questions on mailing list * Core contributors are very active and helpful on the mailing list. * The quality of help from the community of developers via the mailing list is truly excellent. - bug handling 5 * Bug reporting / fixing is good * good bug reporting system. Information: - website 8 * website overhaul is 10 times better than previous * New website is much improved. * current website is much better * Website is clear and gives good navigation. * the new Web site is very very very clear. * web site is nice, you find all pointers and info you need there. * Very nice web pages. - documentation 2 * I like that detailled technical information is provided. * Documentation etc. very good in general. Development/code: - development speed/activity 9 * It's a live project, constantly improving for the better. * The pace of improvement is beyond impressive. * More tools are appearing every once in a while, that's good. * Ongoing development of tools - open source/free software 6 - public repository 4 * Access to "bleeding edge" via SVN * svn checkouts almost always compile and do useful things. - stability 4 - AMD64 and other arch support 3 * Very glad that you support AMD64 mode now -- thanks! * Willingness to support new architectures (eg. AMD64) - easy to build/install 3 - development process is transparent 2 - code is easy to understand 2 Releases: - frequent releases 7 - high release quality 6 * Every release I have experienced has been of high quality. Very few bugs to speak of. * There are very few releases. If the releases were buggy releases more often would be needed. But they are of a very high quality. * Almost never had any surprise upgrading. (The only surprise was when the --tool= option started to be required as it broke my test suite) Misc: - don't know/no opinion 6 - general praise 3 * No real comment, I think its all great. * Valgrind is a real asset to our development. * No real complaints. - nested functions support 1 - regression tests 1 - It's free. 1 - The improvements in pthread support in version 3 caused valgrind to go taking >> 30 mins to start with our application, to starting almost instantly. (yay for not having that static shadow array of pthread_rwlocks any more)... 1 - There is lot of innovation going on in Valgrind dev. My salutes to the dev team. You guys are doing a great job. I feel its a honour to get to know Valgrind and be able to work with it most of my time. Its really fun. 1 - Not enough people seem to know about valgrind, that's not your fault though. 1 - Right to the point! Often errors can be corrected without using other tools thanks to the quality and precision of the messages. 1 - open interfaces 1 - modularity 1 - binaries/rpms/debs for RedHat, SuSe, Debian 1 - Bringing out new versions with bug fixes is great too. 1 - Not afraid to make important interface changeds 1 Conclusions: - People like the mailing lists a lot, and greatly appreciate developer feedback. - People much prefer the new website. Next time: - useful, again hard to improve ----------------------------------------------------------------------------- 15. List up to 3 aspects of Valgrind's development that you think need improvement. Give as much detail as necessary. ----------------------------------------------------------------------------- 35 people answered, 144 people did not. After aggregating, here are the most common responses. - No answer 144 - "None" 7 - Documentation could be improved 7 - Hard to follow development, should provide more info 4 - Releases should have longer beta/RC period for more testing 3 - Callgrind should be integrated 3 - Helgrind is broken 3 - Releases should be more frequent 2 A non-concise summary follows. I grouped comments that were clearly discussing the same point; they are the ones with numbers after them. - None 7 Documentation: - Better documentation 1 - Quick-start guide out-of-date? 1 - Want examples of how to use tools and when, eg. step-by-step tutorial on sample problems 1 - Want training on writing tools 1 - How to get the most out of Valgrind, detailed output description, what weight to give to different messages 1 Releases: - Infrequent releases 2 - Want release notices on Freshmeat 1 - Should release beta/RCs in the weeks before a release 1 - Should have a longer beta/RC period before releases 1 - Should use more formal and through testing (and collecting community feedback) before every major release 1 Performance: - Speed 1 - Memory usage 1 Callgrind: - Callgrind should be included, to avoid delay on new releases 2 - Better integration with Callgrind 1 Helgrind: - Helgrind broken 3 - Release notes should indicate if Helgrind is still broken 1 Backwards compatibility: - Should avoid changing flags and default values between releases (or keep the old ones as deprecated) 1 Bug lists: - Bugzilla registration is onerous 1 - Bugs often get "forgotten" 1 Mailing lists: - Sometimes questions unanswered on mailing list 1 - Fewer idiots on valgrind-users 1 - Archives are poor 1 Development news: - Should reflect current development status on web-page. 1 - Should give weekly summaries of "what's going on", problem of the 1 week, etc 1 - Hard to follow development, eg. what porting is going on, how is porting being made easier 1 - Development is hard to follow 1 Misc: - Incomplete x86 opcode coverage 1 - Focus more on competing with commercial tools 1 - Better interaction with other projects 1 - Want RPMs from valgrind.org 1 - Low publicity 1 - Website: Callgrind is not listed between the "Variants and Patches" 1 - Want to be able to pay for support -- to get guaranteed 1 bug/problem/question responses, and to influence development 1 - Should list people who can do paid work on Valgrind 1 - Want stack checking for Memcheck 1 - It would be nice to run Valgrind without installing. 1 - The code is hard to understand 1 Overall, not too many comments, which is good. The people who gave no answer presumably don't have any complaints. The comments are spread across many different areas, which seems positive -- ie. there's no single thing we're doing really badly. Conclusions: - Nothing particularly notable - More praise in Q14 than dislikes in Q15, which seems to indicate that we're doing a decent job overall. Next time: - useful, again hard to improve ============================================================================= Project Details ============================================================================= ----------------------------------------------------------------------------- 16. For each project on which you have used Valgrind, we would be interested to know the following information. (Omit any details you want to keep private. Please estimate for any answers you're unsure about. Take more than one line per answer if necessary.) Project name : Website (if one exists) : Brief description : Language(s) : (estimate proportions if > 1, eg. 80% C, 20% Fortran 77) Number of programmers: Size (lines of code) : Number of users your software has : Would you be willing to have your project mentioned on the Valgrind website as one that uses Valgrind? (Please only say "yes" if you have the appropriate authority within your project to do so, or you have asked someone with the appropriate authority.) We will only mention the project name, website, and a brief description. [yes; no] ----------------------------------------------------------------------------- 119 people answered, 60 did not. Info about 136 projects. I categorised them according to the description, from 101 responses. The categorisation was pretty rough -- some of the descriptions were pretty terse -- so take it with a grain of salt. scientific/analysis 22 programming (eg. compilers) 10 telecommunications 8 server 7 libraries 7 graphics 6 audio/video 5 db/info retrieval 4 research 4 air traffic control 3 banking/financial 3 games 3 development tools 3 communications/messaging 3 system (eg. daemons) 3 office/desktop apps 2 chip design/VLSI 2 simulation 1 other 5 Interesting results: the number of telecommunications systems is high; there are none on the valgrind.org projects page, perhaps they don't want their use of Valgrind to be publicised. Office/desktop apps is surprisingly low. The popular ones seem appropriate for Valgrind's heavy C and C++ focus. The language proportions were as follows, from 132 responses. C++ 49.9% C 42.9% F77 1.6% Ada 1.1% Java 1.0% Python 0.9% bash/ksh/shell/scripts 0.5% F90/F95 0.4% Yacc/Bison 0.3% Lex/Flex 0.3% Lua 0.2% Cobol 0.2% R 0.2% TCL 0.1% inForma 0.1% homemade languages 0.1% Javascript 0.1% F95 0.1% Antlr 0.1% JS/XSL/XML/XSLT 0.0% Perl 0.0% other 0.0% C# 0.0% The obvious thing here is that C++ and C together account for 93% of use. In the last survey the result was C 45%, C++ 40%. Number of programmers: I categorised them, discounting all things like "or more", "more in the past", etc. For ranges, I took the average number (eg. 10-20 as 15). For "100s" I used 200. For "about 10" I used 10. There were 120 responses, 115 of which were numeric: 1 27 1.5 1 2 18 3 8 4 6 5 12 6 5 7 6 8 3 10 7 12 1 15 4 18 1 20 8 40 1 50 3 55 1 200 2 2000 1 The ranges are: 1 27 1.5 - 2 19 3 - 4 14 5 - 9 26 10 - 19 13 20 - 39 8 40 - 99 5 100+ 3 The average is 28.1 (10.8 if you omit the 2000 one), the median is 4. There were 5 non-numeric responses: bunches 2 a handful 1 lots 1 unknown 1 Code size (lines of code) -- I stripped all things like "about, approximately", "or more", used the average for ranges (10k--20k --> 15k). The summary ranges of the 103 numeric responses: 1.0M--5.0M 17 500k--999k 12 250k--499k 8 100k--249k 17 50k-- 99k 8 20k-- 49k 18 5k-- 19k 19 2k-- 4k 4 103 + 15 The full numeric responses: 5.0M 1 3.5M 1 3.0M 2 2.5M 1 2.0M 3 1.6M 1 1.4M 1 1.3M 1 1.2M 1 1.0M 5 750k 1 715k 1 536k 1 500k 9 350k 2 319k 1 300k 1 250k 4 220k 1 200k 3 180k 2 150k 1 135k 1 105k 1 100k 8 80k 2 75k 1 70k 1 65k 1 50k 3 42k 1 40k 3 37k 1 36k 1 35k 2 31k 1 30k 2 25k 2 20k 5 15k 3 11k 1 10.2k 1 10k 7 8.9k 1 6.8k 1 6.4k 1 6k 1 5k 3 4k 1 3k 1 2k 2 The 15 non-numeric responses 220 MB code 1 tens of millions 1 multiple millions 2 millions 1 500k + millions in third party code 1 many hundred thousand at least 1 several 100k 1 hundreds of thousands 1 2 Mb text source code 1 128 Kb text source code 1 lots 2 ? 2 Overall, we have a huge range, from 2000 up to multiple millions, and a fairly even spread across that range. Numbers of users: 63 numeric responses, 41 non-numeric. I won't even try to sum/average them. Numeric responses: 50.0M 1 1.0M 2 100k's 1 100k 2 55k 2 30k 1 10k 2 5500 1 5000 1 1000s 2 1000 6 500 2 350 1 300 1 250 1 200 1 150 1 100s 4 100 2 75 1 60 1 50 3 40 1 35 1 25 1 15 1 12 1 10 5 7 1 5 3 4 2 3 1 2 1 1 4 0 2 Non-numeric responses: 22,167,075 Domains, 1,283,102 IP Addresses (Netcraft) 1 several million cellphones :) 1 8 ISPs with a total of 4 million customers. 1 millions 1 in trials but potentially millions 1 very wide spread, impossible to count 1 several hundred thousands 1 1200 active servers 1 200 development companies 1 30 companies world-wide 1 30 large corporations 1 Unspecified number of major corporations 1 thousands 2 several thousand 1 many thousands 1 very many 1 many 3 plenty 1 lots 1 internal 1 not released, currently under development 4 a lot 1 15 operational locations around the world 1 10s for the contractor 1 a few thousand 1 Very hard to say. 1000s 1 ? 9 Regarding the website, people granted permission for 49 projects to be mentioned. Conclusions: - Valgrind is used across a huge range of projects: different types, different sizes (number of developers and lines of code). The one common underlying factor is the programming language: C++ accounts for 50% of use, and C for 43%. Valgrind is inextricably tied to these languages. This is not surprising when you consider that memory checking is by far the biggest use of Valgrind tools. Action items: - add all the projects to the projects page :) Next time: - for the "mention on website" checkbox, should ask that for each project, since some people want one project publicised but one not. - the languages part should be separate and above, like the platforms question (Q3) - for the description, could maybe have some pre-canned categories -- would make automation easier - pre-can the size options, too - for the website, perhaps only get them to list it if they want it mentioned? (maybe also for the project name? although some of the non-public ones were interesting) - would be good to have individual fields, but that makes it hard to enter details for multiple projects... - remove the "(estimate proportions if > 1...)" from the field, maybe put it above. ============================================================================= Other comments ============================================================================= ----------------------------------------------------------------------------- 17. Any comments about the content or structure of this survey? ----------------------------------------------------------------------------- 26 people answered, 153 did not. The detailed responses. - good/fine/ok 5 * It's well structured. - input boxes too small 4 * Little tiny entry boxes suck - none 3 - a good idea 2 * And it has the merit to exist! * good that you do this! - For Q2, should have "less than monthly" or "quarterly" 2 - should have made processing more easily automated 2 - more radio buttons and drop-downs would probably help you tabulate my answers easier ;) - Are you really going to be able to parse the varied formats that people submit answers in? =) - Separate "ok to use name" for each project 1 - you should mention that the configure/installation steps are part of Valgrind and that you want some answers on that too. 1 - One long page, could have been split into 3 pages. 1 - (user gave some details about how he/she is using Valgrind teaching an OS class and a C programming class) 1 - hm, at least I cannot find anything Im missing, maybe a better description why this survey... 1 - (user gave extra detail about q16) 1 - For the "willing to have your project mentionned" question: you will maybe have more positive answers by telling that before doing it, the entry will be sent for review by mail (just in case there is an error/typo/imprecision/...). 1 - Valgrind on Sun64 and HP64 would be great, as problems there don't show up on x86. 1 - Hmm, how about "List up to 3 features of valgrind that you think are BAD"? 1 Conclusions - Very few comments. - People seem to like the survey, appreciate it. Action items: - See "Next time" below Next time: - pretty good question, hard to improve ----------------------------------------------------------------------------- 18. Any other comments about Valgrind? ----------------------------------------------------------------------------- 72 people answered, 107 did not. I split some answers up into more than one part for the categorisation below. Summary: short generic compliments 39 longer generic compliments 16 longer, more specific compliments 11 compliments mentioning similar tools 8 other 8 shortcomings 5 Details: - short generic compliments: good/great tool/product/stuff, etc. 39 * A fantastic tool that has saved me hours of bug fixing!!!! * All I can say is "great product". Thank you very much! * Awesome tool! * Compliments to its developers! * Cool! Very helpful. * Excellent piece of software * Excellent tool, it helps a lot * Great job! * Great piece of software! * Great product * Great stuff !! * Great tool! * Great tool!!! * Great tool, thanks for the AMD64 support. * Great work * Great work guys! * Great work! Keep it up! * It's a fantastic tool! * I think valgrind is great! * Keep up the good work. * Love Valgrind. * Love it * My compliments for this very valuable tool! * Really great tool! * Thank you very much for your work. * Thank you, thank you, thank you for this valuable tool. * Thank you, thank you, thank you! * Thanks for a GREAT tool. * Thanks for a great tool! * Thanks for the existence of Valgrind! * Thanks for this excellent tool!!! * Valgrind is a very good tool with sufficient functionality. * Valgrind is an amazing tool, well done. and thank you. * Valgrind is extremely useful. Thank you. * Valgrind rocks! * Valgrind rules. * Very good tool. Keep up the good work ! * Very nice. * You absolutely rocks! :) - longer generic compliments 16 * I couldn't code without it anymore * It's great. Development would be far more painful & less enjoyable without it. Thanks. * Love it, hard to imagine what life would be like without it. * Valgrind has for several years been a development tool I could not live without! * This has saved me many headaches over time and i'm not sure what i would do without it when i had to trace c/c++ bugs * I'm sure glad it exists, I use it everytime I make a change to the code. I couldnt imagine working without it. * Keep up the good work. Valgrind has saved me from many debugging nightmares. * It has saved me countless hours of bug hunting with my work. Many thanks. * Thanks for providing this great tool, we found a lot of problems in our code and use it regularily before releases to check we didn't cause new issues. * V. is a really great tool. It has saved me many hours in debugging my programs. As a matter of fact, I do not regard any portion of code as reliable unless V. "says" it's so. And most amazing, V. is free. * Thanks for Valgrind. Some bugs I've fixed using tool were inportant and well hidden! * Its seriously the best programming tool I have ever used and as I previously stated its easily facilitated a doubling in my productivity as a programmer. * It rocks. A damn useful tool that I'm really glad I have access to. Valgrind proves that inovation does happen on Linux. * valgrind is just a superb tool, before valgrind, chaos ruled. * The valgrind team is making a huge contribution to the software developers toolkit. * Yeah, without it I wouldn't be able to write the code in the quality its currently is ! Noone would, it saved hundreds of hours of debugging and probably also hundreds of undetected bugs... - longer, more specific compliments 11 * It's a very good product. In a few runs, it helped us to tackle unexpected memory issues very quickly. It's small performance penalty makes it usable even in a testing department without making the testers mad about slowliness :-) * In conjunction with GDB, Valgrind and KCachegrind are quite simply the only C++ debugging and profiling tools I have to have. Excellent work. Thank you! * This sofware really saved my day many times, finding obscure bugs (it even helped finding GCC compiler bug for me) * It has become an integral part of our code release process, and we do not allow code to be released without first running Valgrind on the entire test-suite. * I've even given a couple of short courses about memory debugging, students seem to really like it. Thanks valgrind team ! * Fantastic software. I use it to debug portable code on Linux which then runs flawlessly when compiled for a phone. The problem is that the non-portable code can't be treated in this way - hence the need for a valgrind that supports ARM and can be adapted to work without GLIBC being present. * Please don't compromise on the great innovations you are doing for reasons like backward compatibility. * Thank you for the great work - for our core product, we keep the same source code for the last 12 years. We are able to refactor the code, introduce new features, reimplement old features in more efficient and correct way, also thanks to Valgrind. When we make such changes, we do regression testing with Valgrind, and we can be more sure that the release is OK, before we send it to our customers. * Valgrind is a nice tool, has helped us a lot regarding optimising performance (using callgrind). * The memcheck tool is also nice but not as critical for us (Ada language already has a lot of built-in checks, and gcc/gnat Ada front end has a lot of additional such checks. But still, we use memcheck from time to time, and it has helped us tracking bugs) * I thought valgrind was the greatest debugging tool already for addrcheck, but helgrind finally allowed me to run my (two-threaded) application on a bi-processor - amazing tool ! - compliments mentioning similar tools 8 * Best tool ever! We used to pay a lot for Purify (which still has a few advantages), but the cost and installation hassles has caused us to abandon that. The ease of using Valgrind has made it our tool of choice. * Very nice software. We no longer have to rely on expensive , insuficient, and impractical tools from Parasoft and Rational. * Very useful tool. Allowed us to find issues more easily than some very expensive commercial tools * I love it. We have used other tools like sentinel, purify, memcheck, efence and valgrind is the 'best'. My only regret is that it doesn't run on sparc/itanium/pa risc/alpha so we could use it more on deployed applications. * It's a wonderful tool for bugs that are hard to track. Maybe not as shiny as Purify or Insure++, but very very effective - and free! * Valgrind rocks my world. For my purposes it kicks Purify's butt. I just wish I had it on win32. * We've completely abandoned tools like Insure++ in favor of Valgrind. * Valgrind and all the tools are very comparable to non-free software equivalent (e.g. quantify/purify) and are better on many/most aspects. In particular, for us, quantify/purify was not working very well with gcc/gnat/Ada (and/or was working at some point in time but was very quickly broken by whatever change. purify also does not officially supports Ada, so when there is a bug, there is nothing we can do, only expect someone else using C or C++ encounters a similar bug). - other (shortcomings) 8 * It would be great if you add more functionality from zerofault (see zerofault.com) * ok, maybe it could be faster... and not consume so much memory... * Again, I LOVE valgrind mostly but HATE the unitialized variable policy and would greatly appreciate it if you at least add an option to do the simple easy sane policy of giving error at 1st point at which it is used. You can mail me more on this if you like. * Very good, but still young. And seems over complicated for my needs: if some real-life examples were given of what can be gained from cache profiling... perhaps I would look at it! * The main advantage of purify/quantify is that it is working on a lot of platforms, while valgrind is quite limited on this aspect. - other (misc) 5 * I would like to make valgrind part of my nighly regression testing, but I haven't been able to figure out how to fit valgrind into our testing framework. This isn't really a problem with valgrind. It's an integration issue. * Official comments on how to decline the verb "to valgrind". We decided the appropriate past tense is "valgruntled", as in "I valgruntled the tracker last night". * Would like to see your SVN server use the http protocol as well. Firewalls prevent using svn: This is true for all projects including Valkyrie * I hope that my program will one day be worthy to enter Valhalla. * I'm providing my details below, not because I particularly require any response but so you're aware a real person filled this in :) * I regret not being able to assist more with getting the testcases to run clean or analyze performance characteristics. I plan to improve on this. * I need to spend some time to become more familiar with the other tools in valgrind. * With the 3.0 release of Valgrind, I am now able to use valgrind to debug VPI based C++ code that is being executed from the Modelsim RTL simulator. All prior versions of valgrind would work on stand alone code, but not on the process for this simulator. Conclusions: - There is a lot of gratitude and respect among the user base - Similar commercial tools are losing users to Valgrind Action items: - none (curiosity question) Next time: - not bad, hard to improve ============================================================================= Your Contact Details ============================================================================= ----------------------------------------------------------------------------- Please provide your contact details if you wish. Name: Email: Country: Why are we asking for your name and email address? Partly because it encourages more thoughtful responses if your name is attached. And partly because sometimes we respond to surveys -- for example, sometimes we can point out that a feature that a responder thought was missing is actually implemented. We won't pass your name or email on to anyone outside the Valgrind team. As for your country, we're just curious to see how widely Valgrind is used. ----------------------------------------------------------------------------- For this question: - 133 people gave their name, 46 did not. - 135 people gave their email, 44 did not. - 139 people gave their country, 40 did not. Almost everybody gave either all three pieces of information or none. The email addresses were of the following types. .com/.com.* 63 .COUNTRY 39 .org/.org.* 14 .net 9 .edu/.edu.*/.ac.* 7 .gov 2 .int 1 29 countries were named, as follows. The "(other)" is because one person gave their corporation name. USA 37.5 Germany 18.5 (18x Germany, 1x Scotland & Germany) UK 10.5 (8x UK, 2x England, 1x Scotland & Germany) France 10 Australia 6 Netherlands 6 Belgium 5 Israel 4 Russia 4 (3x Russia, 1x Russian Federation) Sweden 3.5 (3x Sweden, 1x Sweden & USA) Austria 2 Brazil 2 Canada 2 China 2 Czech Republic 2 Denmark 2 India 2 Ireland 2 Italy 2 Norway 2 Poland 2 Spain 2 (1x Spain, 1x Catalunya) Switzerland 2 Turkey 2 Belarus 1 Finland 1 Iceland 1 Slovenia 1 South Africa 1 (other) 1 (1x corporation name) By geographic region: Europe 79.5 (inc. Russia) N. America 39.5 Australia 6 Middle East 6 (inc. Turkey) Asia 4 S. America 2 Africa 1 (other) 1 Conclusion: - Surprisingly few developers outside Europe/USA/Australia. I wonder how much this is caused by Valgrind being entirely in English, and how much it just reflects the amount of Linux development being done in different parts of the world. - Making the name/email optional was a good idea, 75% gave the info, but allowing anonymity probably resulted in more responses than we would have got otherwise, and there didn't appear to be any abuse of the anonymity (eg. multiple responses from one person). Action items: - none (curiosity question) Next time: - pretty good, hard to improve ----------------------------------------------------------------------------- Overall ----------------------------------------------------------------------------- Next time: - all the per-question comments, plus... - make analysis more amenable to automation! (As described in the individual questions) - use bigger text boxes - should have asked what version of Valgrind they're currently using! - could ask what version of GCC they're using (or another compiler)... would let us know how common 2.95 is, for example. - general problem: when people don't answer a text question, is it because they don't have a comment or they're just lazy? Not sure if there's a way to handle that.