Merge can now take n inputs #147 (#195)

* Merge can take n inputs #147

* tox compliant

* typo

* test for merging msdfs

* refactoring

* added cli test for merge

* minor change

* typo

* sphinx docs updated

* related to #198

* linted

* updated defaults

* added a TODO

* fixed tests

* new implementation of handling linkml objects

* cleaned up imports

* minor edit to shut mypy up

* removed print

* test edit

* passes locally

* corrected key columns for the merge join

* flake8-ed

* substitute hard code by variables

* default confidence assignment bug fix

* dealing with negation had a bug.

* linted

* replace hard-code to variable

* clean-up

* variables instead of harcoded column names

* if inputs have no confidence so does the output

Author: hrshdhgd

docs/_modules/sssom/util.html

@@ -83,6 +83,7 @@ <h1>Source code for tests.test_cli</h1><div class="highlight"><pre>
     <span class="n">crosstab</span><span class="p">,</span>
     <span class="n">dedupe</span><span class="p">,</span>
     <span class="n">diff</span><span class="p">,</span>
+    <span class="n">merge</span><span class="p">,</span>
     <span class="n">parse</span><span class="p">,</span>
     <span class="n">partition</span><span class="p">,</span>
     <span class="n">ptable</span><span class="p">,</span>
@@ -130,6 +131,7 @@ <h1>Source code for tests.test_cli</h1><div class="highlight"><pre>
         <span class="n">test_cases</span> <span class="o">=</span> <span class="n">get_multiple_input_test_cases</span><span class="p">()</span>
         <span class="bp">self</span><span class="o">.</span><span class="n">run_diff</span><span class="p">(</span><span class="n">runner</span><span class="p">,</span> <span class="n">test_cases</span><span class="p">)</span>
         <span class="bp">self</span><span class="o">.</span><span class="n">run_partition</span><span class="p">(</span><span class="n">runner</span><span class="p">,</span> <span class="n">test_cases</span><span class="p">)</span>
+        <span class="bp">self</span><span class="o">.</span><span class="n">run_merge</span><span class="p">(</span><span class="n">runner</span><span class="p">,</span> <span class="n">test_cases</span><span class="p">)</span>
 
         <span class="bp">self</span><span class="o">.</span><span class="n">assertTrue</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">test_cases</span><span class="p">)</span> <span class="o">&gt;=</span> <span class="mi">2</span><span class="p">)</span></div>
 
@@ -279,6 +281,22 @@ <h1>Source code for tests.test_cli</h1><div class="highlight"><pre>
         <span class="p">]</span>
         <span class="n">result</span> <span class="o">=</span> <span class="n">runner</span><span class="o">.</span><span class="n">invoke</span><span class="p">(</span><span class="n">correlations</span><span class="p">,</span> <span class="n">params</span><span class="p">)</span>
         <span class="bp">self</span><span class="o">.</span><span class="n">run_successful</span><span class="p">(</span><span class="n">result</span><span class="p">,</span> <span class="n">test_case</span><span class="p">)</span>
+        <span class="k">return</span> <span class="n">result</span></div>
+
+<div class="viewcode-block" id="SSSOMCLITestSuite.run_merge"><a class="viewcode-back" href="../../tests.html#tests.test_cli.SSSOMCLITestSuite.run_merge">[docs]</a>    <span class="k">def</span> <span class="nf">run_merge</span><span class="p">(</span>
+        <span class="bp">self</span><span class="p">,</span> <span class="n">runner</span><span class="p">:</span> <span class="n">CliRunner</span><span class="p">,</span> <span class="n">test_cases</span><span class="p">:</span> <span class="n">Mapping</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="n">SSSOMTestCase</span><span class="p">]</span>
+    <span class="p">)</span> <span class="o">-&gt;</span> <span class="n">Result</span><span class="p">:</span>
+        <span class="sd">&quot;&quot;&quot;Run the merge test.&quot;&quot;&quot;</span>
+        <span class="n">params</span> <span class="o">=</span> <span class="p">[]</span>
+        <span class="n">out_file</span> <span class="o">=</span> <span class="kc">None</span>
+        <span class="k">for</span> <span class="n">t</span> <span class="ow">in</span> <span class="n">test_cases</span><span class="o">.</span><span class="n">values</span><span class="p">():</span>
+            <span class="n">params</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">t</span><span class="o">.</span><span class="n">filepath</span><span class="p">)</span>
+            <span class="n">out_file</span> <span class="o">=</span> <span class="n">t</span>
+        <span class="k">if</span> <span class="n">out_file</span><span class="p">:</span>
+            <span class="n">params</span><span class="o">.</span><span class="n">extend</span><span class="p">([</span><span class="s2">&quot;--output&quot;</span><span class="p">,</span> <span class="n">out_file</span><span class="o">.</span><span class="n">get_out_file</span><span class="p">(</span><span class="s2">&quot;tsv&quot;</span><span class="p">)])</span>
+
+        <span class="n">result</span> <span class="o">=</span> <span class="n">runner</span><span class="o">.</span><span class="n">invoke</span><span class="p">(</span><span class="n">merge</span><span class="p">,</span> <span class="n">params</span><span class="p">)</span>
+        <span class="bp">self</span><span class="o">.</span><span class="n">run_successful</span><span class="p">(</span><span class="n">result</span><span class="p">,</span> <span class="n">test_cases</span><span class="p">)</span>
         <span class="k">return</span> <span class="n">result</span></div></div>
 </pre></div>
 

docs/_modules/tests/test_cli.html

@@ -101,7 +101,7 @@ <h1>Source code for tests.test_reconcile</h1><div class="highlight"><pre>
         <span class="n">msdf1</span> <span class="o">=</span> <span class="n">read_sssom_table</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="n">data_dir</span><span class="si">}</span><span class="s2">/basic.tsv&quot;</span><span class="p">)</span>
         <span class="n">msdf2</span> <span class="o">=</span> <span class="n">read_sssom_table</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="n">data_dir</span><span class="si">}</span><span class="s2">/basic2.tsv&quot;</span><span class="p">)</span>
 
-        <span class="n">merged_msdf</span> <span class="o">=</span> <span class="n">merge_msdf</span><span class="p">(</span><span class="n">msdf1</span><span class="o">=</span><span class="n">msdf1</span><span class="p">,</span> <span class="n">msdf2</span><span class="o">=</span><span class="n">msdf2</span><span class="p">)</span>
+        <span class="n">merged_msdf</span> <span class="o">=</span> <span class="n">merge_msdf</span><span class="p">(</span><span class="n">msdf1</span><span class="p">,</span> <span class="n">msdf2</span><span class="p">)</span>
 
         <span class="bp">self</span><span class="o">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="mi">123</span><span class="p">,</span> <span class="nb">len</span><span class="p">(</span><span class="n">merged_msdf</span><span class="o">.</span><span class="n">df</span><span class="p">))</span></div>
 
@@ -110,7 +110,7 @@ <h1>Source code for tests.test_reconcile</h1><div class="highlight"><pre>
         <span class="n">msdf1</span> <span class="o">=</span> <span class="n">read_sssom_table</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="n">data_dir</span><span class="si">}</span><span class="s2">/basic4.tsv&quot;</span><span class="p">)</span>
         <span class="n">msdf2</span> <span class="o">=</span> <span class="n">read_sssom_table</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="n">data_dir</span><span class="si">}</span><span class="s2">/basic5.tsv&quot;</span><span class="p">)</span>
 
-        <span class="n">merged_msdf</span> <span class="o">=</span> <span class="n">merge_msdf</span><span class="p">(</span><span class="n">msdf1</span><span class="o">=</span><span class="n">msdf1</span><span class="p">,</span> <span class="n">msdf2</span><span class="o">=</span><span class="n">msdf2</span><span class="p">,</span> <span class="n">reconcile</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
+        <span class="n">merged_msdf</span> <span class="o">=</span> <span class="n">merge_msdf</span><span class="p">(</span><span class="n">msdf1</span><span class="p">,</span> <span class="n">msdf2</span><span class="p">,</span> <span class="n">reconcile</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
 
         <span class="bp">self</span><span class="o">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="mi">53</span><span class="p">,</span> <span class="nb">len</span><span class="p">(</span><span class="n">msdf1</span><span class="o">.</span><span class="n">df</span><span class="p">))</span>
         <span class="bp">self</span><span class="o">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="mi">53</span><span class="p">,</span> <span class="nb">len</span><span class="p">(</span><span class="n">msdf2</span><span class="o">.</span><span class="n">df</span><span class="p">))</span>

docs/_modules/tests/test_reconcile.html

@@ -5,22 +5,22 @@ td.linenos .special { color: #000000; background-color: #ffffc0; padding-left: 5
 span.linenos.special { color: #000000; background-color: #ffffc0; padding-left: 5px; padding-right: 5px; }
 .highlight .hll { background-color: #ffffcc }
 .highlight { background: #f8f8f8; }
-.highlight .c { color: #408080; font-style: italic } /* Comment */
+.highlight .c { color: #3D7B7B; font-style: italic } /* Comment */
 .highlight .err { border: 1px solid #FF0000 } /* Error */
 .highlight .k { color: #008000; font-weight: bold } /* Keyword */
 .highlight .o { color: #666666 } /* Operator */
-.highlight .ch { color: #408080; font-style: italic } /* Comment.Hashbang */
-.highlight .cm { color: #408080; font-style: italic } /* Comment.Multiline */
-.highlight .cp { color: #BC7A00 } /* Comment.Preproc */
-.highlight .cpf { color: #408080; font-style: italic } /* Comment.PreprocFile */
-.highlight .c1 { color: #408080; font-style: italic } /* Comment.Single */
-.highlight .cs { color: #408080; font-style: italic } /* Comment.Special */
+.highlight .ch { color: #3D7B7B; font-style: italic } /* Comment.Hashbang */
+.highlight .cm { color: #3D7B7B; font-style: italic } /* Comment.Multiline */
+.highlight .cp { color: #9C6500 } /* Comment.Preproc */
+.highlight .cpf { color: #3D7B7B; font-style: italic } /* Comment.PreprocFile */
+.highlight .c1 { color: #3D7B7B; font-style: italic } /* Comment.Single */
+.highlight .cs { color: #3D7B7B; font-style: italic } /* Comment.Special */
 .highlight .gd { color: #A00000 } /* Generic.Deleted */
 .highlight .ge { font-style: italic } /* Generic.Emph */
-.highlight .gr { color: #FF0000 } /* Generic.Error */
+.highlight .gr { color: #E40000 } /* Generic.Error */
 .highlight .gh { color: #000080; font-weight: bold } /* Generic.Heading */
-.highlight .gi { color: #00A000 } /* Generic.Inserted */
-.highlight .go { color: #888888 } /* Generic.Output */
+.highlight .gi { color: #008400 } /* Generic.Inserted */
+.highlight .go { color: #717171 } /* Generic.Output */
 .highlight .gp { color: #000080; font-weight: bold } /* Generic.Prompt */
 .highlight .gs { font-weight: bold } /* Generic.Strong */
 .highlight .gu { color: #800080; font-weight: bold } /* Generic.Subheading */
@@ -33,15 +33,15 @@ span.linenos.special { color: #000000; background-color: #ffffc0; padding-left:
 .highlight .kt { color: #B00040 } /* Keyword.Type */
 .highlight .m { color: #666666 } /* Literal.Number */
 .highlight .s { color: #BA2121 } /* Literal.String */
-.highlight .na { color: #7D9029 } /* Name.Attribute */
+.highlight .na { color: #687822 } /* Name.Attribute */
 .highlight .nb { color: #008000 } /* Name.Builtin */
 .highlight .nc { color: #0000FF; font-weight: bold } /* Name.Class */
 .highlight .no { color: #880000 } /* Name.Constant */
 .highlight .nd { color: #AA22FF } /* Name.Decorator */
-.highlight .ni { color: #999999; font-weight: bold } /* Name.Entity */
-.highlight .ne { color: #D2413A; font-weight: bold } /* Name.Exception */
+.highlight .ni { color: #717171; font-weight: bold } /* Name.Entity */
+.highlight .ne { color: #CB3F38; font-weight: bold } /* Name.Exception */
 .highlight .nf { color: #0000FF } /* Name.Function */
-.highlight .nl { color: #A0A000 } /* Name.Label */
+.highlight .nl { color: #767600 } /* Name.Label */
 .highlight .nn { color: #0000FF; font-weight: bold } /* Name.Namespace */
 .highlight .nt { color: #008000; font-weight: bold } /* Name.Tag */
 .highlight .nv { color: #19177C } /* Name.Variable */
@@ -58,11 +58,11 @@ span.linenos.special { color: #000000; background-color: #ffffc0; padding-left:
 .highlight .dl { color: #BA2121 } /* Literal.String.Delimiter */
 .highlight .sd { color: #BA2121; font-style: italic } /* Literal.String.Doc */
 .highlight .s2 { color: #BA2121 } /* Literal.String.Double */
-.highlight .se { color: #BB6622; font-weight: bold } /* Literal.String.Escape */
+.highlight .se { color: #AA5D1F; font-weight: bold } /* Literal.String.Escape */
 .highlight .sh { color: #BA2121 } /* Literal.String.Heredoc */
-.highlight .si { color: #BB6688; font-weight: bold } /* Literal.String.Interpol */
+.highlight .si { color: #A45A77; font-weight: bold } /* Literal.String.Interpol */
 .highlight .sx { color: #008000 } /* Literal.String.Other */
-.highlight .sr { color: #BB6688 } /* Literal.String.Regex */
+.highlight .sr { color: #A45A77 } /* Literal.String.Regex */
 .highlight .s1 { color: #BA2121 } /* Literal.String.Single */
 .highlight .ss { color: #19177C } /* Literal.String.Symbol */
 .highlight .bp { color: #008000 } /* Name.Builtin.Pseudo */

docs/_static/pygments.css

@@ -155,7 +155,7 @@ <h3>convert<a class="headerlink" href="#sssom-py-convert" title="Permalink to th
 <p>currently only supports conversion to RDF)</p>
 </div>
 <dl class="simple">
-<dt>Example:</dt><dd><p>sssom covert –input my.sssom.tsv –output-format rdfxml –output my.sssom.owl</p>
+<dt>Example:</dt><dd><p>sssom convert my.sssom.tsv –output-format rdfxml –output my.sssom.owl</p>
 </dd>
 </dl>
 <div class="highlight-shell notranslate"><div class="highlight"><pre><span></span>sssom-py convert <span class="o">[</span>OPTIONS<span class="o">]</span> INPUT
@@ -328,13 +328,11 @@ <h3>dosql<a class="headerlink" href="#sssom-py-dosql" title="Permalink to this h
 </section>
 <section id="sssom-py-merge">
 <h3>merge<a class="headerlink" href="#sssom-py-merge" title="Permalink to this headline"></a></h3>
-<p>Merge msdf2 into msdf1.</p>
-<dl class="simple">
-<dt>if reconcile=True, then dedupe(remove redundant lower confidence mappings) and</dt><dd><p>reconcile (if msdf contains a higher confidence _negative_ mapping,
+<p>Merge multiple MappingSetDataFrames into one .</p>
+<p>if reconcile=True, then dedupe(remove redundant lower confidence mappings) and
+reconcile (if msdf contains a higher confidence _negative_ mapping,
 then remove lower confidence positive one. If confidence is the same,
 prefer HumanCurated. If both HumanCurated, prefer negative mapping).</p>
-</dd>
-</dl>
 <div class="highlight-shell notranslate"><div class="highlight"><pre><span></span>sssom-py merge <span class="o">[</span>OPTIONS<span class="o">]</span> <span class="o">[</span>INPUTS<span class="o">]</span>...
 </pre></div>
 </div>