----------------------------------------------------------------------------- 2nd Official Valgrind Survey, September 2005: high-level summary ----------------------------------------------------------------------------- The survey, hosted at www.valgrind.org, ran from September 22--October 8, 2005. 179 responses were received. The following are recommended "action items" -- ie. things we should do. Big, important: - Should do more performance tuning. Now possible with Cachegrind and self-hosting. This will make many people happy. - We should try to bring Addrcheck back, or give Memcheck the characteristics that people like about Addrcheck (speed, less memory use) - We should monitor how effectively the aspacem rewrite helps memory usage problems; if there are still problems, we should try the compressed V bits representation. - We should try to bring Helgrind back Big, not as important: - We should consider how to improve suppressions. Just dumping auto-generated ones to file would be a good start. More flexible and simpler syntax would help too. - We should work out if debugger attachment can be made more reliable. - We should perhaps try to improve the understandability of error messages. - We should investigate a strict definedness option; past experience has shown it to not work well but perhaps things could be improved. - We should improve ISA coverage. - It might be worth working on a coverage tool. Some work has been done on that front already by various people. Small: - Should make --leak-check=full as the default for Memcheck. - Should increase --num-callers default from 12 to 20. - Perhaps change what gets shown with -v. - Advertise some of the changes that people haven't realised in the release notes of the next release (eg. --tool no longer mandatory). Other: - add all the projects to the projects page :) - improve the survey for next time! ----------------------------------------------------------------------------- Things to remember ----------------------------------------------------------------------------- These are things that aren't exactly action items, but are good to remember. - Memcheck is the most widely used tool by a long way - x86/Linux is by far the most popular platform - Almost all Valgrind use is on C and C++ programs - People love the ease-of-use (ie. no recompilation required) ----------------------------------------------------------------------------- Short summaries of responses to individual questions ----------------------------------------------------------------------------- Q1. People have been using Valgrind for this long: 37+ months: 7% 25--36 months: 24% 13--24 months: 37% 0--12 months: 31% The average is 21.7 months. ----------------------------------------------------------------------------- Q2. People use it this frequently: hourly 1% daily 13% weekly 47% monthly 38% ----------------------------------------------------------------------------- Q3. They use Valgrind on these hardware platforms: Probably x86 77.8% x86 or AMD64 (unclear) 10.5% Probably AMD64 9.9% Other 1.8% These numbers are rough, the question was poorly done so understanding the answers required some guessing. ----------------------------------------------------------------------------- Q4. The proportion of OS/distro usage with Valgrind is: Fedora Core 17.3% SuSE 16.0% Red Hat Enterprise Linux 15.6% Red Hat (pre-Fedora, up to verion 9.0) 11.2% Gentoo 7.6% Debian 7.2% Linux 7.2% Red Hat (unspecified if before or after Fedora) 4.0% Mandrake 3.4% Ubuntu 2.8% Slackware 2.1% Mandriva 1.7% CentOS 1.1% FreeBSD 0.6% Other 2.4% All the Red Hat versions combined (including Fedora) account for 48.1%. ----------------------------------------------------------------------------- Q5. They obtain Valgrind in the following ways (the percentages sum to more than 100% because many people mentioned more than one source). source from website 78% pre-built (eg. RPM, Debian package) 25% source from CVS/SVN repository 21% already installed 16% other: gentoo ebuild 5% other: FreeBSD ports system <1% ----------------------------------------------------------------------------- Q6. The proportions of tool usage are (1% or more): memcheck 80.7% addrcheck 8.0% callgrind 6.1% massif 1.8% cachegrind 1.8% helgrind 1.4% Each tool was mentioned as being used (even if only 1% of the time) by the following proportion of responders. memcheck 99.4% addrcheck 26.2% callgrind 24.0% massif 16.8% cachegrind 15.0% helgrind 10.6% vprof 1.1% valgui 0.6% ----------------------------------------------------------------------------- Q7. People use the following command-line options always or almost always (10% of users or more): --num-callers 45.6% --memcheck:leak-check=yes/full 41.5% --tool=memcheck 22.4% -v/--verbose 21.1% --memcheck:show-reachable=yes 19.0% --suppressions 15.0% --log-file 15.0% --error-limit=no 13.6% --trace-children=yes 10.9% --db-attach=yes 10.2% ----------------------------------------------------------------------------- Q8. People use the following command-line options sometimes (5% of users or more): --db-attach 32 29.9% --memcheck:leak-check=full 23 21.5% --memcheck:show-reachable=yes 16 15.0% --memcheck:leak-check 13 12.1% -v 11 10.2% --track-fds=yes 10 9.3% --gen-suppressions 9 8.4% --trace-children=yes 7 6.5% --log-file 6 5.6% --num-callers 6 5.6% ----------------------------------------------------------------------------- Q9. If Valgrind was available on all platforms, the usage among the survey participants would be the something like the following (1% or more usage): x86/Linux 46.4% AMD64/Linux 10.7% existing 8.1% IA64/Linux 4.8% x86/Windows 3.9% AMD64 3.2% PPC/MacOS X 2.7% SPARC9/Solaris 2.3% SPARC/Solaris 2.2% Linux 1.6% PPC/Linux 1.4% Windows 1.2% ARM 1.0% "existing" means one of the platforms Valgrind already supports (x86/Linux or AMD64/Linux), but it was unclear which one. Since some people mentioned only a CPU or only an OS, it's worth considering the CPU and OS in isolation also (top 5 in each case): x86 51.9% AMD64 14.1% SPARC 5.7% IA64 5.5% PPC 5.3% Linux 65.3% Windows 5.7% Solaris 5.5% MacOS X 2.8% HPUX 1.8% These percentages are out of less than 100%, because less than 100% of responses mentioned both a CPU and an OS. ----------------------------------------------------------------------------- Q10. The things people like most about Valgrind are (with the number of times the thing was mentioned): memcheck/memory checking 118 - general praise 41 - leak checking 32 - undefined value checking 20 - invalid access checking 18 - other 7 ease-of-use 40 - no recompilation 20 - easy to use 20 other 27 accurate information 19 robust/stable 19 GPL 16 speed 13 general praise 11 ----------------------------------------------------------------------------- Q11. The things people like least about Valgrind are (with the number of times the thing was mentioned) (at least 10 mentions): speed 45 suppressions 23 simulation correctness/robustness 21 error messages/output 20 Memcheck's error detection 18 memory use 15 debugger attachment 11 want Helgrind back 10 Callgrind 10 documentation 10 ----------------------------------------------------------------------------- Q12. The features people want added to Valgrind are (at least 10 mentions): other 59 checking of static/stack bounds errors 14 more platforms supported 13 better debugger integration 10 GUIs 10 ----------------------------------------------------------------------------- Q13. People had the following suggestions for other tools (at least 2 mentions): code coverage tool 9 time profiler 8 function call tracer 3 just improve existing tools 3 resource usage measurer 2 flight recorder/time machine 2 ----------------------------------------------------------------------------- Q14. People liked the following non-software aspects of Valgrind's development (5 mentions or more): mailing list/developers are good/responsive 31 other 12 development speed/activity 9 website 8 frequent releases 7 open source/free software 6 high quality releases 6 don't know/no opinion 6 bug handling 5 ----------------------------------------------------------------------------- Q15. People thought the following non-software aspects could be improved (1 mention or more): none 7 Documentation could be improved 7 Hard to follow development, should provide more info 4 Releases should have longer beta/RC period for more testing 3 Callgrind should be integrated 3 Helgrind is broken 3 Releases should be more frequent 2 ----------------------------------------------------------------------------- Q16. People use Valgrind on the following kinds of project (five or more mentions): scientific/analysis 22 programming (eg. compilers) 10 telecommunications 8 server 7 libraries 7 graphics 6 audio/video 5 The proportion of language use is the following (1% or more): C++ 49.9% C 42.9% F77 1.6% Ada 1.1% Java 1.0% The number of programmers in the mentioned projects were: 1 27 1.5 - 2 19 3 - 4 14 5 - 9 26 10 - 19 13 20 - 39 8 40 - 99 5 100+ 3 Code size of the projects varied greatly, from 2000 lines to "tens of millions", with fairly even representation across the whole range. 49 people gave permission to have their project listed on the Valgrind website. ----------------------------------------------------------------------------- Q17. People had the following comments about the survey (3 or more mentions): it's good 5 input boxes are too small 4 ----------------------------------------------------------------------------- Q18. People had final comments in the following categories: short generic compliments 39 longer generic compliments 16 longer, more specific compliments 11 compliments mentioning similar tools 8 other 8 shortcomings 5 ----------------------------------------------------------------------------- Q19. The responses were from the following regions (with some breakdown): Europe 79.5 (inc. Russia) - Germany 18.5 - UK 10.5 - France 10 - Netherlands 6 - Belgium 5 N. America 39.5 - USA 37.5 - Canada 2 Australia 6 Middle East 6 (inc. Turkey) - Israel 4 Asia 4 S. America 2 Africa 1