-
Notifications
You must be signed in to change notification settings - Fork 0
/
CodeCoverage.html
285 lines (242 loc) · 15.6 KB
/
CodeCoverage.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
<!DOCTYPE html>
<html lang="en" data-content_root="./">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="viewport" content="width=device-width, initial-scale=1" />
<title>Generating Code Coverage</title>
<link rel="stylesheet" type="text/css" href="_static/pygments.css?v=fa44fd50" />
<link rel="stylesheet" type="text/css" href="_static/bootstrap-sphinx.css?v=fadd4351" />
<link rel="stylesheet" type="text/css" href="_static/custom.css?v=77160d70" />
<script src="_static/documentation_options.js?v=a8da1a53"></script>
<script src="_static/doctools.js?v=9bcbadda"></script>
<script src="_static/sphinx_highlight.js?v=dc90522c"></script>
<link rel="index" title="Index" href="genindex.html" />
<link rel="search" title="Search" href="search.html" />
<link rel="next" title="Tools Overview" href="ToolsOverview.html" />
<link rel="prev" title="Design Document Guidelines" href="DesignDocumentGuides.html" />
<script>
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
})(window,document,'script','//www.google-analytics.com/analytics.js','ga');
ga('create', 'UA-59110517-1', 'auto');
ga('send', 'pageview');
</script>
</head><body>
<div id="navbar" class="navbar navbar-default ">
<div class="container">
<div class="navbar-header">
<!-- .btn-navbar is used as the toggle for collapsed navbar content -->
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".nav-collapse">
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
<a class="navbar-brand" href="http://www.mantidproject.org">
</a>
<span class="navbar-text navbar-version pull-left"><b>main</b></span>
</div>
<div class="collapse navbar-collapse nav-collapse">
<ul class="nav navbar-nav">
<li class="divider-vertical"></li>
<li><a href="index.html">Home</a></li>
<li><a href="https://download.mantidproject.org">Download</a></li>
<li><a href="https://docs.mantidproject.org">User Documentation</a></li>
<li><a href="http://www.mantidproject.org/contact">Contact Us</a></li>
</ul>
<form class="navbar-form navbar-right" action="search.html" method="get">
<div class="form-group">
<input type="text" name="q" class="form-control" placeholder="Search" />
</div>
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div>
<p>
<div class="related" role="navigation" aria-label="related navigation">
<h3>Navigation</h3>
<ul>
<li class="nav-item nav-item-0"><a href="index.html">Documentation</a> »</li>
<li class="nav-item nav-item-this"><a href="">Generating Code Coverage</a></li>
</ul>
</div> </p>
</div>
<div class="container">
<div class="row">
<div class="body col-md-12 content" role="main">
<section id="generating-code-coverage">
<h1>Generating Code Coverage<a class="headerlink" href="#generating-code-coverage" title="Link to this heading">¶</a></h1>
<nav class="contents local" id="contents">
<ul class="simple">
<li><p><a class="reference internal" href="#c" id="id1">C++</a></p>
<ul>
<li><p><a class="reference internal" href="#background" id="id2">Background</a></p></li>
<li><p><a class="reference internal" href="#c-specific-notes" id="id3">C++ Specific Notes</a></p></li>
<li><p><a class="reference internal" href="#setup" id="id4">Setup</a></p></li>
<li><p><a class="reference internal" href="#producing-a-coverage-report-automatically" id="id5">Producing a coverage report automatically</a></p></li>
<li><p><a class="reference internal" href="#producing-a-coverage-report-manually" id="id6">Producing a coverage report manually</a></p></li>
<li><p><a class="reference internal" href="#filtering-specific-module-s" id="id7">Filtering specific module(s)</a></p>
<ul>
<li><p><a class="reference internal" href="#examples" id="id8">Examples:</a></p></li>
</ul>
</li>
</ul>
</li>
<li><p><a class="reference internal" href="#python" id="id9">Python</a></p>
<ul>
<li><p><a class="reference internal" href="#ide" id="id10">IDE</a></p></li>
<li><p><a class="reference internal" href="#cli" id="id11">CLI</a></p></li>
</ul>
</li>
</ul>
</nav>
<section id="c">
<h2><a class="toc-backref" href="#id1" role="doc-backlink">C++</a><a class="headerlink" href="#c" title="Link to this heading">¶</a></h2>
<section id="background">
<h3><a class="toc-backref" href="#id2" role="doc-backlink">Background</a><a class="headerlink" href="#background" title="Link to this heading">¶</a></h3>
<p>GCC and LLVM have tooling built in to allow a developer to view the code coverage from their tests.</p>
<p>This will work for native C++ code and any code executed via Python, as the instrumentation is compiled into the binary output. Gcovr will also filter out any non-project code automatically.</p>
</section>
<section id="c-specific-notes">
<h3><a class="toc-backref" href="#id3" role="doc-backlink">C++ Specific Notes</a><a class="headerlink" href="#c-specific-notes" title="Link to this heading">¶</a></h3>
<p>The coverage target(s) do not run the tests automatically. Users must either run the test(s) or file(s) they are interested in manually for data to be produced by the tooling.</p>
</section>
<section id="setup">
<h3><a class="toc-backref" href="#id4" role="doc-backlink">Setup</a><a class="headerlink" href="#setup" title="Link to this heading">¶</a></h3>
<ul>
<li><p>Enable coverage in CMake through the GUI or using</p>
<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span>cmake<span class="w"> </span>/path/to/src<span class="w"> </span>-DCOVERAGE<span class="o">=</span>ON
ninja<span class="w"> </span><span class="c1"># or make</span>
</pre></div>
</div>
</li>
<li><p>Install gcovr</p>
<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span>pip3<span class="w"> </span>install<span class="w"> </span>gcovr
</pre></div>
</div>
</li>
<li><p>Run C++/Python test(s) using ctest or the executable, or similar.</p></li>
</ul>
</section>
<section id="producing-a-coverage-report-automatically">
<h3><a class="toc-backref" href="#id5" role="doc-backlink">Producing a coverage report automatically</a><a class="headerlink" href="#producing-a-coverage-report-automatically" title="Link to this heading">¶</a></h3>
<p>The following targets are available to make/ninja:</p>
<ul class="simple">
<li><p><cite>coverage</cite> : Builds all available coverage reports</p></li>
<li><p><cite>coverage_clean</cite> : Removes all instrumented data and report data</p></li>
<li><p><cite>coverage_cpp</cite> : Builds coverage information on cpp files</p></li>
</ul>
<p>HTML output is written into: <cite><build_dir>/coverage/<lang>/html</cite></p>
<p>Where <cite><lang></cite> is either <cite>cpp</cite> or <cite>python</cite>.</p>
</section>
<section id="producing-a-coverage-report-manually">
<h3><a class="toc-backref" href="#id6" role="doc-backlink">Producing a coverage report manually</a><a class="headerlink" href="#producing-a-coverage-report-manually" title="Link to this heading">¶</a></h3>
<ul>
<li><p>Create a folder for the html files, for example <cite>html</cite></p></li>
<li><p>From your build folder generate a HTML report for the coverage as follows:</p>
<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span>gcovr<span class="w"> </span>-r<span class="w"> </span><src_dir><span class="w"> </span>-j<span class="w"> </span><n_threads><span class="w"> </span>--exclude-throw-branches<span class="w"> </span>--html<span class="w"> </span>--html-details<span class="w"> </span>-o<span class="w"> </span>html/<output>.html<span class="w"> </span>.
</pre></div>
</div>
<p>Where:</p>
<ul class="simple">
<li><p>-r Points to the full path of your source directory. <em>Note: A trailing slash must be present, i.e. /my/src/ instead of /my/src</em></p></li>
<li><p>-j is the number of threads to run whilst generating, if you’re unsure pick a value like 8.</p></li>
<li><p>-o controls the html output directory and file name</p></li>
<li><p>–exclude-throw-branches Gcovr will incorrectly show compiler inserted branches as unused, despite them not appearing in the source code. This filters them out making the branch hit rate sane.</p></li>
<li><p>(Optional) –gcov-executable if using a manually specified compiler use this to specify the appropriate gcov tool, e.g. <cite>gcov-10</cite> for <cite>gcc-10</cite> or <cite>llvm-cov-11</cite> for <cite>llvm-11</cite></p></li>
</ul>
</li>
</ul>
</section>
<section id="filtering-specific-module-s">
<h3><a class="toc-backref" href="#id7" role="doc-backlink">Filtering specific module(s)</a><a class="headerlink" href="#filtering-specific-module-s" title="Link to this heading">¶</a></h3>
<p>The <cite>-f</cite> flag can be appended once or multiple times to show output from a single folder or subset of folders. This works similarly to the regex filter used in unit testing.</p>
<section id="examples">
<h4><a class="toc-backref" href="#id8" role="doc-backlink">Examples:</a><a class="headerlink" href="#examples" title="Link to this heading">¶</a></h4>
<p>To only include, say, the scientific interfaces, it would be the relative path of the source files:</p>
<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span>gcovr<span class="w"> </span>-f<span class="w"> </span>qt/scientific_interfaces<span class="w"> </span>-j<span class="w"> </span><n_threads><span class="w"> </span>--exclude-throw-branches<span class="w"> </span>--html<span class="w"> </span>--html-details<span class="w"> </span>-o<span class="w"> </span>html/<output>.html<span class="w"> </span>.
</pre></div>
</div>
<p>To include only API and Kernel sources:</p>
<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span>gcovr<span class="w"> </span>-f<span class="w"> </span>Framework/Kernel<span class="w"> </span>-f<span class="w"> </span>Framework/API<span class="w"> </span>-j<span class="w"> </span><n_threads><span class="w"> </span>--exclude-throw-branches<span class="w"> </span>--html<span class="w"> </span>--html-details<span class="w"> </span>-o<span class="w"> </span>html/<output>.html<span class="w"> </span>.
</pre></div>
</div>
<p>Additionally, folders can be excluded using the -e flag, this is useful to filter out test coverage. Tests will typically have 100% coverage so tend to add noise.</p>
<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span>gcovr<span class="w"> </span>-r<span class="w"> </span>/path/to/src<span class="w"> </span>-e<span class="w"> </span>Framework/KernelTest<span class="w"> </span>-j<span class="w"> </span><n_threads><span class="w"> </span>--exclude-throw-branches<span class="w"> </span>--html<span class="w"> </span>--html-details<span class="w"> </span>-o<span class="w"> </span>html/<output>.html<span class="w"> </span>.
</pre></div>
</div>
</section>
</section>
</section>
<section id="python">
<h2><a class="toc-backref" href="#id9" role="doc-backlink">Python</a><a class="headerlink" href="#python" title="Link to this heading">¶</a></h2>
<p>Unit tests can also be run to generate coverage too, this requires us to run the test using the Coverage module.</p>
<section id="ide">
<h3><a class="toc-backref" href="#id10" role="doc-backlink">IDE</a><a class="headerlink" href="#ide" title="Link to this heading">¶</a></h3>
<p>(Recommended)</p>
<p>Your IDE may already have an option to run the test with coverage enabled if it can already run the test directly.</p>
<p>To setup unit tests for Pycharm see the <a class="reference internal" href="PyCharm.html#pycharm-ref"><span class="std std-ref">PyCharm</span></a> page.</p>
</section>
<section id="cli">
<h3><a class="toc-backref" href="#id11" role="doc-backlink">CLI</a><a class="headerlink" href="#cli" title="Link to this heading">¶</a></h3>
<p>(Advanced)</p>
<ul>
<li><p>Install Coverage:</p>
<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span>pip3<span class="w"> </span>install<span class="w"> </span>coverage
</pre></div>
</div>
</li>
<li><p>Make a note of the directory or test file you’d like coverage information from. Pytest will search recursively from a given directory.</p></li>
<li><p>cd to build directory and run your test with coverage as follows</p>
<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span><span class="c1"># Where <path> is the directory or file to run</span>
<span class="nv">PYTHONPATH</span><span class="o">=</span>/path/to/build/bin<span class="w"> </span>python3<span class="w"> </span>-m<span class="w"> </span>coverage<span class="w"> </span>run<span class="w"> </span>-m<span class="w"> </span>pytest<span class="w"> </span><path>
</pre></div>
</div>
</li>
<li><p>To limit coverage information to only project files add the <cite>–source</cite> flag:</p>
<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span><span class="c1"># Where /path/to/src/ contains Mantid source code</span>
<span class="c1"># Note this cannot use the home dir, i.e. ~/path/to/src</span>
<span class="nv">PYTHONPATH</span><span class="o">=</span>/path/to/build/bin<span class="w"> </span>python3<span class="w"> </span>-m<span class="w"> </span>coverage<span class="w"> </span>run<span class="w"> </span>--source<span class="o">=</span></path/to/src/><span class="w"> </span>-m<span class="w"> </span>pytest<span class="w"> </span><path>
</pre></div>
</div>
</li>
<li><p>Example: To run coverage for the entire project</p>
<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span><span class="nv">PYTHONPATH</span><span class="o">=</span>/path/to/build/bin<span class="w"> </span>python3<span class="w"> </span>-m<span class="w"> </span>coverage<span class="w"> </span>run<span class="w"> </span>--source<span class="w"> </span>/mantid_src<span class="w"> </span>-m<span class="w"> </span>pytest<span class="w"> </span>/mantid_src
</pre></div>
</div>
</li>
<li><p>Generate out a html report in the directory you ran tests from:</p>
<blockquote>
<div><div class="highlight-shell notranslate"><div class="highlight"><pre><span></span><span class="c1"># This will generate htmlcov/index.html</span>
coverage<span class="w"> </span>html
</pre></div>
</div>
</div></blockquote>
</li>
</ul>
</section>
</section>
</section>
</div>
</div>
</div>
<footer class="footer">
<div class="container">
<ul class="nav navbar-nav" style=" float: right;">
<li>
<a href="DesignDocumentGuides.html" title="Previous Chapter: Design Document Guidelines"><span class="glyphicon glyphicon-chevron-left visible-sm"></span><span class="hidden-sm hidden-tablet">« Design Docume...</span>
</a>
</li>
<li>
<a href="ToolsOverview.html" title="Next Chapter: Tools Overview"><span class="glyphicon glyphicon-chevron-right visible-sm"></span><span class="hidden-sm hidden-tablet">Tools Overview »</span>
</a>
</li>
<li><a href="#">Back to top</a></li>
</ul>
<p>
</p>
</div>
</footer>
</body>
</html>