Skip to content

Commit 562515e

Browse files
pollaropollaro
committed
splitting up developer section, network results setting, with some added information spread around
1 parent 00fd4bb commit 562515e

12 files changed

Lines changed: 598 additions & 350 deletions

‎+nla/+net/+result/NetworkTestResult.m‎

Lines changed: 11 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -2,24 +2,22 @@
22
% Network level test results
33
% Class to store all relevant results for a network level test. Each test will create an instance of this to store results
44
%
5+
% :param test_options: Options and inputs for tests to be run (also called input_struct)
6+
% :param number_of_networks: The number of networks in the network atlas
57
% :param test_name: The name of the network test run
68
% :param test_display_name: The name of the network test for display
7-
% :param test_options: Options and inputs for tests to be run (also called input_struct)
8-
% :param ranking_statistic: The statistic to be used in ranking to determine p-value
9-
% :param within_network_pair: Results of the within network pair test. Single sample p-values (except :math:`\chi^2`\ and hypergeometric tests). "legacy_" results use the individual test p-values to rank and determine the final p-value. Multiple ranking strategies available
10-
% :param full_connectome: Results of the full_connectome test. Same format as above.
11-
% :param no_permutations: Results for the non-permuted test. Same format as above.
12-
% :param permutation_results: Results of each permutation. Statistics and p-values. Note: The p-values are for each individual permutation test, not the overall p-value.
9+
% :param test_specific_statistics: The statistics that a specific test produces
10+
% :param ranking_statistic: The statistic used for calculating p-values
1311

1412
properties
1513
test_name = "" % Name of the network test run
16-
test_display_name = "" % Name of the network test for the front-end to display
17-
test_options = struct() % Options selected for the test. Formerly input_struct
18-
ranking_statistic = ""
19-
within_network_pair = false % Results for within-network-pair tests
20-
full_connectome = false % Results for full connectome tests (formerly 'experiment wide')
21-
no_permutations = false % Results for the network tests with no permutations (the 'observed' results)
22-
permutation_results = struct() % Results for each permutation test used to calculate p-values for the test methods
14+
test_display_name = "" % The name of the network test for display
15+
test_options = struct() % Options and inputs for tests to be run (also called input_struct)
16+
ranking_statistic = "" % The statistic used for calculating p-values
17+
within_network_pair = false % Results of the within network pair test. Single sample p-values (except :math:`\chi^2`\ and hypergeometric tests). "legacy_" results use the individual test p-values to rank and determine the final p-value.
18+
full_connectome = false % Results of the full_connectome test. Two sample p-values.
19+
no_permutations = false % Results for the non-permuted test. Single sample p-values (except :math:`\chi^2`\ and hypergeometric tests).
20+
permutation_results = struct() % Results of each permutation. Statistics and p-values. Note: The p-values are for each individual permutation test, not the overall p-value.
2321
end
2422

2523
properties (Access = private)

‎+nla/+net/ResultRank.m‎

Lines changed: 42 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -1,18 +1,19 @@
11
classdef ResultRank < handle
2-
% RESULTRANK Class to take permutation results and rank them. This used to be done incrementally. It"s a little hacky, beware
3-
% This class does a classic permutation based ranking of results. The "observed" (nonpermuted) result is appended to the results
4-
% of all the permuted results. All of these results are sorted and then searched for the observed result. Where this result is located
5-
% in the sorted array ends up being the p-value.
2+
% Ranker to calculate *p*-values from permutation testing
3+
%
4+
% :param permuted_network_results: The NetworkTestResult object from permutation test
5+
% :param number_of_network_pairs: The number of network pairs for the Brain Atlas used
6+
% :return
67

78
properties
8-
nonpermuted_network_results % The results from the network tests (these are the ones being tested)
9-
permuted_network_results % The results from the network tests with all the paramters permuted over and over
9+
nonpermuted_network_results % The results from the network level test (NetworkTestResult object)
10+
permuted_network_results % The network level test results for each permutation
1011
number_of_network_pairs % The number of network pairs in the atlas being used
1112
end
1213

1314
properties (Dependent)
14-
permutations
15-
number_of_networks
15+
permutations
16+
number_of_networks
1617
end
1718

1819
methods
@@ -53,8 +54,18 @@
5354
end
5455

5556
function ranking = uncorrectedRank(obj, test_method, permutation_results, no_permutation_results, ranking_statistic,...
56-
probability, ranking)
57-
57+
probability, ranking)
58+
% Performs ranking of observed result among all results (all permutations plus itself)
59+
%
60+
% :param test_method: The method of the test being ranked (full connectome or within network pair)
61+
% :param permutation_results: The test result for all permutations
62+
% :param no_permutation_results: The observed test result
63+
% :param ranking_statistic: The statistic used in ranking for each test
64+
% :param probability: The name of the *p*-value (single_sample or two_sample)
65+
% :param ranking: The NetworkTestResult object to place the results
66+
% :return: The same NetworkTestResult object with ranking results
67+
68+
5869
for index = 1:numel(no_permutation_results.(strcat("uncorrected_", probability)).v)
5970
combined_probabilities = [...
6071
permutation_results.(strcat((probability), "_permutations")).v(index, :),...
@@ -86,7 +97,17 @@
8697

8798

8899
function ranking = winklerMethodRank(obj, test_method, permutation_results, no_permutation_results, ranking_statistic,...
89-
probability, ranking)
100+
probability, ranking)
101+
% Ranks the observed result using method described by Winkler to correct for FWER
102+
%
103+
% :param test_method: The method of the test being ranked (full connectome or within network pair)
104+
% :param permutation_results: The test result for all permutations
105+
% :param no_permutation_results: The observed test result
106+
% :param ranking_statistic: The statistic used in ranking for each test
107+
% :param probability: The name of the *p*-value (single_sample or two_sample)
108+
% :param ranking: The NetworkTestResult object to place the results
109+
% :return: The same NetworkTestResult object with ranking results
110+
90111
winkler_probability = strcat("winkler_", probability);
91112
max_statistic_array = max(abs(permutation_results.(strcat(ranking_statistic, "_permutations")).v));
92113
for index = 1:numel(no_permutation_results.(strcat("uncorrected_", probability)).v)
@@ -104,7 +125,16 @@
104125
end
105126

106127
function ranking = westfallYoungMethodRank(obj, test_method, permutation_results, no_permutation_results, ranking_statistic,...
107-
probability, ranking)
128+
probability, ranking)
129+
% Ranks the observed result using method described by Westfall and Young to correct for FWER
130+
%
131+
% :param test_method: The method of the test being ranked (full connectome or within network pair)
132+
% :param permutation_results: The test result for all permutations
133+
% :param no_permutation_results: The observed test result
134+
% :param ranking_statistic: The statistic used in ranking for each test
135+
% :param probability: The name of the *p*-value (single_sample or two_sample)
136+
% :param ranking: The NetworkTestResult object to place the results
137+
% :return: The same NetworkTestResult object with ranking results
108138

109139
% sort statistics in ascending order
110140
[sorted_no_permutation_results, sorted_statistic_indexes] = sort(...

‎+nla/NetworkAtlas.m‎

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -3,13 +3,13 @@
33
% Defines ROI positions/information and networks
44
%
55
% :param name: The name of the atlas
6-
% :param net_names: N\ :sub:`nets`\ x 3 matrix. The names of the networks
7-
% :param ROI_key: N\ :sub:`ROIs`\ x 2 matrix. First column is ROI (Region Of Interest) indexes, second column is the network they belong to
8-
% :param ROI_order: N\ :sub:`ROIs`\ x 1 vector. Functional Connectivity data indexes corresponding to ROIs
9-
% :param ROI_pos: N\ :sub:`ROIs`\ x 3 matrix. Centroid positions for each ROI.
10-
% :param net_colors: N\ :sub:`nets`\ x 3 matrix. The color of each network when plotted.
11-
% :param parcels: Optional MATLAB struct field for surface parcellations. Contains two sub-fields ``ctx_l`` and ``ctx_r``. N\ :sub:`verts`\ x 1 vectors. Each element of a vector corresponds to a vertex within the spatial mesh and contains the index of the ROI for that vertex.
12-
% :param space: Optional The mesh that the atlas` ROI locations/parcels are in. Two options - ``TT`` or ``MNI``
6+
% :param net_names: N\ :sub:`nets`\ x 3 matrix. The names of the networks
7+
% :param ROI_key: N\ :sub:`ROIs`\ x 2 matrix. First column is ROI (Region Of Interest) indexes, second column is the network they belong to
8+
% :param ROI_order: N\ :sub:`ROIs`\ x 1 vector. Functional Connectivity data indexes corresponding to ROIs
9+
% :param ROI_pos: N\ :sub:`ROIs`\ x 3 matrix. Centroid positions for each ROI.
10+
% :param net_colors: N\ :sub:`nets`\ x 3 matrix. The color of each network when plotted.
11+
% :param parcels: (Optional) MATLAB struct field for surface parcellations. Contains two sub-fields ``ctx_l`` and ``ctx_r``. N\ :sub:`verts`\ x 1 vectors. Each element of a vector corresponds to a vertex within the spatial mesh and contains the index of the ROI for that vertex.
12+
% :param space: (Optional) The mesh that the atlas` ROI locations/parcels are in. Two options - ``TT`` or ``MNI``
1313

1414
properties (SetAccess = private)
1515
nets % This is the net_names

‎docs/source/edge_level_tests.rst‎

Lines changed: 0 additions & 68 deletions
Original file line numberDiff line numberDiff line change
@@ -77,71 +77,3 @@ Provided Tests
7777
:Permuted p: ``.mat`` file containing N\ :sub:`ROI_pairs`\ x N\ :sub:`permutations`\ of logical values. Observed, thresholded, permuted *p*-values.
7878
:Permuted coeff: ``.mat`` file containing N\ :sub:`ROI_pairs`\ x N\ :sub:`permutations`\ of permuted edge-level coefficients.
7979

80-
Creating additional edge-level tests
81-
-----------------------------------------------
82-
83-
To create an edge-level test, a test class must be added to the codebase. Refer to current tests in ``+nla/+edge/+test`` for examples. Guidelines are listed below
84-
85-
* **Test objects**
86-
87-
All test objects must inherit from ``nla.edge.test.Base`` and be in the ``+nla/+edge/+test`` directory. There are also a few methods and
88-
properties that must be included
89-
90-
* A constant property ``name`` is required.
91-
92-
::
93-
94-
properties (Constant)
95-
name = "Pearson's r"
96-
end
97-
98-
* A ``run`` method is also required.
99-
100-
::
101-
102-
result = run(obj, input_struct)
103-
104-
:input_struct: Also called ``test_options`` in the codebase. Parameters needed for a test. The functional connectivity, network atlas, and other properties are stored here.
105-
106-
.. _requiredInputs:
107-
108-
* A ``requiredInputs`` method.
109-
110-
::
111-
112-
methods (Static)
113-
function inputs = requiredInputs()
114-
inputs = {
115-
nla.inputField.Number('prob_max', 'P <', 0, 0.05, 1),
116-
nla.inputField.NetworkAtlas(),
117-
nla.inputField.Behavior()
118-
}
119-
end
120-
end
121-
122-
This function creates 3 input fields in the GUI. A number ``prob_max`` with range [0, 1] and a default value of 0.05.
123-
A network atlas file, and a behavior file. These are required, meaning that the GUI will not run without these inputs being
124-
fulfilled. These values are all stored in the ``input_struct`` object.
125-
126-
* **Result object**
127-
128-
A result object must be defined for the test edge-level results. If no custom data fields are needed, then the object in ``+nla/+edge/+test/Base.m``
129-
may be used and this step can be skipped.
130-
131-
* A ``output`` method must be included.
132-
133-
::
134-
135-
function output(obj, network_atlas, flags)
136-
137-
:network_atlas: An atlas of the form defined in ``nla.NetworkAtlas``
138-
:flags: Contains flags for the various types of figures to output.
139-
140-
* (Optional) A ``merge`` method to merge blocks of permutation results together. An example can be found in
141-
``+nla/+edge/+result/PermBase.m`` file.
142-
143-
::
144-
145-
merge(obj, results)
146-
147-
:results: Cell array of result objects to merge. The object that calls the method will have the ``result`` merged with it.

‎docs/source/for_developers.rst‎

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,71 @@
1+
For Developers
2+
==============================
3+
4+
NLA allows for adding custom tests to the codebase
5+
6+
Creating additional edge-level tests
7+
-----------------------------------------------
8+
9+
Current tests are located in ``+nla/+edge/+test``. This is where all the current tests are located and also where
10+
any user-created custom tests will need to be saved. All edge-level test objects must inherit from ``nla.edge.BaseTest``
11+
12+
.. mat:module:: edge
13+
14+
.. mat:autoclass:: BaseTest
15+
16+
.. mat:automethod:: run(input_struct)
17+
18+
.. mat:automethod:: inputs = requiredInputs()
19+
20+
21+
* **Result object**
22+
23+
A result object must be defined for the test edge-level results. If no custom data fields are needed, then the object in ``+nla/+edge/+result/Base.m``
24+
may be used and this step can be skipped. If a new result is needed, a permutation result inheriting ``+nla/+edge/+result/PermBase``
25+
must also be created
26+
27+
.. mat:module:: edge.result
28+
29+
.. mat:autoclass:: Base
30+
31+
.. mat:automethod:: output(net_atlas, flags, prob_label)
32+
33+
Creating additional network-level tests
34+
----------------------------------------------------------
35+
36+
Current network-level tests are located in ``+nla/+net/+test``. All custom created tests must be saved in this folder; the
37+
tests present can also be used as templates. All network-level results must fit into the :doc:`NetworkTestResult </network_level_results>` object.
38+
39+
Since these tests are all different with different results and statistics, there is no base test to inherit.
40+
The only requirements are below
41+
42+
* Constant properties required
43+
::
44+
45+
properties (Constant)
46+
name = "students_t"
47+
display_name = "Student's T-test"
48+
statistics = ["t_statistic", "single_sample_t_statistic"]
49+
ranking_statistic = "t_statistic"
50+
end
51+
52+
53+
:name: The name of the test with no special characters (spaces, &, etc)
54+
:display_name: A formal name that will be used for displaying in the GUI. Any string will work
55+
:statistics: All statistics that will be generated by the test. No special characters
56+
:ranking_statistic: The statistic used for ranking and calculating *p*-values. Note: if there is a single sample version of the statisticin addition to a two sample statistic, the GUI will automatically add "single_sample\_" during rankings for non-permuted and within network pair ranking.
57+
58+
* A ``run`` method
59+
60+
::
61+
62+
result = run(obj, test_options, edge_test_results, network_atlas, permutations)
63+
64+
65+
:test_options: Also called ``input_struct`` in edge-level tests. Parameters needed to run the test.
66+
:edge_test_results: The output from the edge-level test.
67+
:network_atlas: A network atlas of the form ``nla.NetworkAtlas``
68+
:permutations: Boolean to determine if the test is being run with permutations (``true``) or without (``false``)
69+
:result: :doc:`NetworkTestResult </network_level_results>`
70+
71+
* ``requiredInputs`` See :ref:`Edge-level tests <requiredInputs>`

‎docs/source/index.rst‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,6 @@ Welcome to NetworkLevelAnalysis's documentation!
1111
:caption: Table of Contents:
1212

1313
preface
14-
overview
1514
methodology
1615
setup
1716
getting_started
@@ -20,6 +19,8 @@ Welcome to NetworkLevelAnalysis's documentation!
2019
network_atlases
2120
edge_level_tests
2221
network_level_tests
22+
network_level_results
23+
for_developers
2324
bibliography
2425

2526

0 commit comments

Comments
 (0)