Decrypted Mind: Articles About Debianurn:uuid:552b9a6e-d068-3ddf-881e-57c16233a1de2020-11-02T00:00:00ZBugzilla integration for KDE Project API2020-11-02T00:00:00ZSandro Knaußurn:uuid:e6fabe10-02f9-3c26-b5a7-22cf1ef252d0<p>The KDE Bugzilla handles a lot of projects and they often match with the repo name, but not always.
For instance we have ancient products and components at Bugzilla, as projects have a lifecycle from playground into Release Service, or Frameworks, sometimes with a change of name.
So you may end up searching Bugzilla quite awhile for the correct product and component to be able to confirm or create bug reports against an application.
Let's have a look at <a href="https://invent.kde.org/frameworks/kpeople">KPeople</a>, and see why the situation is complicated.
You find <a href="https://bugs.kde.org/describecomponents.cgi">two products in KDE Bugzilla</a>: <a href="https://bugs.kde.org/buglist.cgi?order=changeddate%2Cpriority%2Cbug_severity&product=kpeople">kpeople</a> (the repository's name) and on the other hand Frameworks have the scheme of a "frameworks-" prefix: <a href="https://bugs.kde.org/buglist.cgi?order=changeddate%2Cpriority%2Cbug_severity&product=frameworks-kpeople">frameworks-kpeople</a>.
From the data displayed even I as a developer am unable to tell which is the correct product to add new bug reports. Both have bug reports this year that got fixed and the number of bug reports is too low to get a clear picture of which to choose.</p>
<p>This is not only a problem of KDE; it is a general problem in different communities that it is hard for newcomers to find the correct place to search and add new bug reports.</p>
<p>That's why Debian added the bug report information for every package.
This should help users to search the upstream bug reports or create new ones (Bug-Submit and Bug-Database): <a href="https://wiki.debian.org/UpstreamMetadata#Fields">https://wiki.debian.org/UpstreamMetadata#Fields</a></p>
<p>While I was collecting this information for Frameworks and KDE PIM, I wondered why KDE does not have links between each project and Bugzilla.
After some searching and discussions it became obvious, that KDE does not have this information that can be processed.
Okay let's fix this.
The obvious place to reach this information is the <a href="https://invent.kde.org/sysadmin/projects-api">Project API</a> available under <a href="https://projects.kde.org/api/">https://projects.kde.org/api/</a>.
To reach this goal I began adding Bugzilla information to the data source of the Project API named <a href="https://invent.kde.org/sysadmin/repo-metadata">Git Repo Metadata</a>.
Then a merge request later the Project API is able to generate the links to Bugzilla <a href="https://invent.kde.org/sysadmin/projects-api/-/merge_requests/2">invent:sysadmin/projects-api!2</a>.</p>
<p>Where you should search for bugs in kontact? Go to <a href="https://projects.kde.org/api/v1/identifier/kontact">https://projects.kde.org/api/v1/identifier/kontact</a>:</p>
<ul>
<li>bug_database: <a href="https://bugs.kde.org/buglist.cgi?product=kontact&resolution=---">https://bugs.kde.org/buglist.cgi?product=kontact&resolution=---</a></li>
<li>bug_submit: <a href="https://bugs.kde.org/enter_bug.cgi?product=kontact">https://bugs.kde.org/enter_bug.cgi?product=kontact</a></li>
</ul>
<p>After implementing the needed bits I found out that Nicolas Alvarez had the same idea, to store Bugzilla information in Git Repo Metadata <a href="https://invent.kde.org/sysadmin/repo-metadata/-/commit/085d878eaf6cfac541376be753b80829a087519a">invent:sysadmin/repo-metadata@085d878ea</a>.
Fortunately I can say that since June the information are used by Project API.</p>
<p>So now, back to my task to add upstream metadata to Debian packages.
After I filled the needed information to Repo Metadata I created a script to update the links in Debian <a href="https://salsa.debian.org/qt-kde-team/pkg-kde-dev-scripts/-/blob/013b0937ac37b43669c139321ea719b4273fabfa/function_collection/functions_plasma.py#L129">salsa:qt-kde-team/pkg-kde-dev-scripts/function_collection/functions_plasma.py:addMissingBugMetadatafields</a>.
This should hopefully help to always point to the correct Bugzilla links in future.</p>
<p>A random list came to my mind of places that may benefit from the Bugzilla information in Git Metadata Repository:</p>
<ul>
<li>Links from API documentation directly to the Bugzilla.</li>
<li>We can use this information in scripts that adds the new version to Bugzilla (e.g. <a href="https://invent.kde.org/sdk/releaseme/-/blob/master/plasma/plasma-add-bugzilla-versions">invent:sdk/releaseme/plasma/plasma-add-bugzilla-versions</a> has currently a static list of Bugzilla products).</li>
<li>For DrKonqui this information may also useful.
The mapping binary -> bugzilla is currently handcrafted in <a href="https://invent.kde.org/plasma/drkonqi/-/blob/master/src/data/mappings">invent:plasma/drkonqi/src/data/mappings</a>.
This is NOT the low hanging fruit, but maybe someone feels inspired to play with the available data and who is checking each handcrafted entry in that file to still be correct?
The mapping repo -> binary we get by our CI, because when we build the stuff, we know what we built.</li>
</ul>
<p>The truth is, that Nicolas is right that adding the Bugzilla links are a manual task.
But on the other hand, let's add this information once at one place and than we can use it at several places.</p>
<p>Please help adding this information; it is simple a yaml file.
If you see the data missing <a href="https://invent.kde.org/sysadmin/repo-metadata/-/commit/84786036c8f119d607658840824c30596e678d31">example commit</a> and create a <a href="https://invent.kde.org/sysadmin/repo-metadata/-/merge_requests">merge request</a>.
Or you can also give me the data and I'll add it.</p>
Kontact on Debian2017-12-21T00:00:00ZSandro Knaußurn:uuid:b8f8ef4f-818d-369d-a0da-d3ae3842c1c7<p>When coding at Kontact you normally don't have to care a lot about dependencies in between the different KDE Pim packages, because there are great tools available already. <a href="https://kdesrc-build.kde.org/">kdesrc-build</a> finds a solution to build all KDE Pim packages in the correct order. The <a href="https://community.kde.org/KDE_PIM/Docker">KDE Pim docker image</a> gives you an environment with all dependencies preinstalled, so you can start hacking on KDE Pim directly.</p>
<p>While hacking on <code>master</code> is nice, most users are not using <code>master</code> on their computers in daily life. To reach the users, distributions need to compile and ship KDE Pim. I am active within Debian and would like to make the newest version of KDE Pim available to Debian users. Because <a href="https://wiki.qt.io/New_Features_in_Qt_5.5#Deprecated_Functionality">Qt deprecated Qt Webkit within Qt 5.5</a> KDE Pim had to switch from <a href="https://wiki.qt.io/Qt_WebKit">Qt Webkit</a> to <a href="https://wiki.qt.io/Qt_WebEngine">Qt WebEngine</a>. Unfortunately Qt WebEngine wasn't available in Debian, so I had to package Qt WebEngine for Debian before packaging KDE Pim for Debian. Qt WebEngine itself is a beast to package. It was only possible to package Qt WebEngine for the last stable release named "Stretch" in time with the help of others of the Debian Qt/KDE mainatiners especially Scarlett Clark, Dmitry Shachnev and Simon Quigley, and we could only upload it some hours before the deep freeze. So if you have asked yourself why Debian doesn't ship 16.08 within their last stable release, this is the answer. The missing dependency for KDE Pim named Qt WebEngine.
There is a second consequence of the switch: Kontact will only be available for those architectures that are supported by Qt WebEngine. Of <a href="https://buildd.debian.org/status/package.php?p=kontact">19 supported architectures for 16.04</a>, we can only support <a href="https://buildd.debian.org/status/package.php?p=qtwebengine-opensource-src">five architectures</a> in future.</p>
<p>Now after Debian has woken up again from its slumber, we first had to update Qt and KDE Frameworks. After the first attempt at packaging KDE Pim 17.08.0, that was released for experimental, we are now finally reaching the point where we can package and deliver KDE Pim 17.08.3 to Debian unstable. Because Pino Toscano and I had time we started packaging it and stumbled across the issue of having to package 58 source packages, all dependent on each other. Keep in mind all packaging work is not a oneman or twoman show, mostly all in the Qt/KDE Debian mantainers are involved somehow. Either by putting their name under a upload or by being available via IRC, mail and answering questions, making jokes or doing what so ever. <a href="https://jriddell.org/2016/08/05/kontact-build-dependencies/">Jonathan Riddell visualized the dependencies for KDE Pim 16.08 with graphviz</a>. But KDE Pim is a fast moving target, and I wanted to make my own graphs and make them more useful for packaging.</p>
<p><a href="fullgraph.svg"><img src="fullgraph.svg#fullsize" alt=""></a></p>
<p>The dependencies you see on this graph are created out of the Build dependencies within Debian for KDE Pim 17.08. I stripped out every dependency that isn't part of KDE Pim. In contrast to Jonathan, I made the arrows from dependency to package. So the starting point of the arrow is the dependency and it is pointing to the packages that can be built from it. The green colour shows you packages that have no dependency inside KDE Pim. The blue indicates packages with nothing depending on them. But to be honest, neither Jonathan's nor my graph tells me any more than they do you. They are simply too convoluted. The only thing these graphs make apparent is that packaging KDE Pim is a very complex task :D</p>
<p>But fortunately we can simplify the graphs. For packaging, I'm not interested in "every" dependency, but only in "new" ones. That means, if a <code><package></code> depends on <code>a</code>,<code>b</code> and <code>c</code>, and <code>b</code> depends on <code>a</code>, than I know: Okay, I need to package <code>b</code> and <code>c</code> before <code><package></code> and <code>a</code> before <code>b</code>. I would call a an implicit dependency of <code><package></code>. Here again in a dot style syntax:</p>
<pre><code>a -> <package>
b -> <package>
c -> <package>
a -> b
</code></pre>
<p>can be simplified to:</p>
<pre><code>b -> <package>
c -> <package>
a -> b
</code></pre>
<p>With this quite simple rule to strip all implicit dependencies out of the graph we end up with a more useful one:</p>
<p><a href="pim-build-deps-17.08.png"><img src="pim-build-deps-17.08.png#fullsize" alt=""></a></p>
<p>(You can find the dot file and the code to create such a graph at <a href="https://qt-kde-team.pages.debian.net/applications-17.08-build-deps.html">pkg-kde.alioth.debian.org</a>)</p>
<p>At least this is a lot easier to consume and create a package ordering from. But still it looks scary. So I came up with the idea to define tiers, influenced by the tier model in KDE Frameworks. I defined one tier as the maximum set of packages that are independent from each other and only depend on lower tiers:</p>
<p><a href="pim-build-tier-17.08.png"><img src="pim-build-tier-17.08.png#fullsize" alt=""></a></p>
<p>(The dot file and the code to create such a graph you can find at <a href="https://qt-kde-team.pages.debian.net/applications-17.08-build-deps.html">pkg-kde.alioth.debian.org</a>)</p>
<p>Additionally I only show the dependencies, from the last tier to the current one. So a dependency from <code>tier 0 -> tier 1</code> but not from <code>tier 0 -> tier 2</code>. That's why it looks like nothing depends on <code>kdav</code> or <code>ktnef</code>. But the ellipse shape tells you, that in higher tiers something depends on them. The lightblue diamond shaped ones in contrast indicating, nothing depending on them anymore. So here you can see the "hotpath" for dependencies. This shows that the bottleneck is <code>libkdepim->pimcommon</code>. Interestingly this is also, more or less, the border between the former split of kdepimlibs and kdepim during KDE SC 4 times.
I think this is a useful visualization of the dependencies and may be a starting point to define a goal, what the dependencies should look like.</p>
<p>You also may ask yourself why an application needs so much more tiers than complete KDE Frameworks? Well, the third tier of KDE Frameworks is more of a collection for leftovers that don't reach tier 1 or tier 2. See the definition of tier 3 is: <a href="https://community.kde.org/Frameworks/Policies">"Tier 3 Frameworks can depend only on other Tier 3 Frameworks, Tier 2 Frameworks, Tier 1 Frameworks, Qt official frameworks, or other system libraries."</a>. The relevant part is that a framework tier 3 can depend on other tier 3 frameworks. If you use my tier definition in contrast, then you end up with more than ten tiers for KDE Frameworks, too.</p>
<p>After building all of these nice graphs for Debian, I wanted to see if I could create such graphs for KDE Pim directly. As KDE is mostly using the <a href="https://cgit.kde.org/kde-build-metadata.git/">kde-build-metadata.git</a> for documenting dependencies I updated my scripts to create graphs from this data directly:
<a href="kde-build-metadata-17-12-deps.svg"><img src="kde-build-metadata-17-12-deps.svg#fullsize" alt=""></a>
<a href="../kde-build-metadata-17-12-tier.svg"><img src="kde-build-metadata-17-12-tier.svg#fullsize" alt=""></a>
(the code to build the graphs yourselves is available here: <a href="https://web.archive.org/web/20171222041517/https://cgit.kde.org/kde-dev-scripts.git/tree/pim-build-deps-graphs.py">kde-dev-scripts.git/pim-build-deps-graphs.py</a>)</p>
<p>In detail this graph looks different and not just because of the version difference (<code>17.08</code> vs. <code>master</code>). I think we need to update the dependencies data. This also may explain why sometimes kdesrc-build don't manage it to compile complete KDE Pim in the first run.</p>