.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "auto_examples/mass-univariate-statistics/permutation-statistics.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note :ref:`Go to the end ` to download the full example code. .. rst-class:: sphx-glr-example-title .. _sphx_glr_auto_examples_mass-univariate-statistics_permutation-statistics.py: Permutation statistics ====================== .. currentmodule:: eelbrain Eelbrain implents three methods for estimating null-distributions in permutation tests: max-statistic, cluster-mass and TFCE. This example illustrates these using a simple *t*-test. For simulating more complex datasets see other examples in this section. For the sake of speed, the tests here are based on 1000 permutations of the data (``samples=1000``). For precise *p*-values, 10000 permutations (the default) are preferable. .. contents:: Contents :local: .. GENERATED FROM PYTHON SOURCE LINES 20-29 .. code-block:: Python # sphinx_gallery_thumbnail_number = 2 from eelbrain import * import scipy.stats data = datasets.simulate_erp() eeg = data['eeg'] p = plot.TopoButterfly(eeg, t=0.13, head_radius=0.35) .. image-sg:: /auto_examples/mass-univariate-statistics/images/sphx_glr_permutation-statistics_001.png :alt: permutation statistics :srcset: /auto_examples/mass-univariate-statistics/images/sphx_glr_permutation-statistics_001.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 30-35 Max statistic ------------- The max statistic is simply the maximum across the entire *t*-map. The permutation distribution consist of ``max(t)`` for each permutation. This is the default, and is also the fastest test. .. GENERATED FROM PYTHON SOURCE LINES 35-38 .. code-block:: Python result = testnd.TTestOneSample(eeg, samples=1000) .. GENERATED FROM PYTHON SOURCE LINES 39-42 Visualize the permutation distribution. We consider values significant if they are larger than 95% of the permutation distribution, so we look for the 95th percentile: .. GENERATED FROM PYTHON SOURCE LINES 42-50 .. code-block:: Python significance_threshold = scipy.stats.scoreatpercentile(result.permutation_distribution, 95) p = plot.Histogram( result.permutation_distribution, title='Distribution of $max(t)$', xlabel='max(t)', ylabel='Number of permutations') p.add_vline(significance_threshold, color='red', linestyle='--') _ = p.axes[0].text(significance_threshold, 40, ' 95%', color='red') .. image-sg:: /auto_examples/mass-univariate-statistics/images/sphx_glr_permutation-statistics_002.png :alt: Distribution of $max(t)$ :srcset: /auto_examples/mass-univariate-statistics/images/sphx_glr_permutation-statistics_002.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 51-53 In the actual comparison, any data point with a *t*-value that is more extreme than 95% of the permutation distribution is considered signifiacant: .. GENERATED FROM PYTHON SOURCE LINES 53-56 .. code-block:: Python p = plot.TopoButterfly(result, t=0.13, head_radius=0.35) .. image-sg:: /auto_examples/mass-univariate-statistics/images/sphx_glr_permutation-statistics_003.png :alt: permutation statistics :srcset: /auto_examples/mass-univariate-statistics/images/sphx_glr_permutation-statistics_003.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 57-59 Significant regions can be retrieved as :class:`Dataset` of :class:`NDVar` for further analysis (e.g., for defining ROIs). This function will find all contiguous regions in which *p* ≤ .05: .. GENERATED FROM PYTHON SOURCE LINES 59-63 .. code-block:: Python significan_regions = result.find_clusters(0.05, maps=True) significan_regions .. raw:: html
# id tstart tstop duration n_sensors p sig
0 1 0.1 0.165 0.065 26 0 ***
1 4 0.25 0.26 0.01 2 0.001 ***
2 11 0.285 0.29 0.005 1 0.046 *
3 33 0.09 0.16 0.07 11 0 ***
4 41 0.16 0.165 0.005 1 0.046 *
5 45 0.155 0.16 0.005 1 0.001 ***
NDVars: cluster


.. GENERATED FROM PYTHON SOURCE LINES 64-66 .. code-block:: Python p = plot.TopoButterfly(significan_regions[0, 'cluster'], t=0.13, head_radius=0.35) .. image-sg:: /auto_examples/mass-univariate-statistics/images/sphx_glr_permutation-statistics_004.png :alt: permutation statistics :srcset: /auto_examples/mass-univariate-statistics/images/sphx_glr_permutation-statistics_004.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 67-73 Cluster-based tests ------------------- In cluster-based tests, a first steps consists in finding contiguous regions of "meaningful" effect, so-called clusters. In order to find contiguous regions, the algorithm needs to know which channels are neighbors. This information is refered to as the sensor adjacency (i.e., which sensors are connected). The adjacency graph can be visualized to confirm that it is set correctly. .. GENERATED FROM PYTHON SOURCE LINES 73-76 .. code-block:: Python p = plot.SensorMap(eeg, adjacency=True) .. image-sg:: /auto_examples/mass-univariate-statistics/images/sphx_glr_permutation-statistics_005.png :alt: permutation statistics :srcset: /auto_examples/mass-univariate-statistics/images/sphx_glr_permutation-statistics_005.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 77-88 We also need to define what constitutes a "meaningful" effect, i.e., an effect that should be included in the cluster. This can be defined based on the magnitude of the statistic (e.g., a *t*-value). It is commonly set to a *t* value equivalent to an uncorrected *p*-vale of .05. This is specified with the ``pmin=.05`` argument. A summary statistic is then computed for each cluster. In Eelbrain this is the *cluster-mass* statistic: the sum of all values in the cluster (i.e., the sum of the *t* values at all datapoints that are part of the cluster). The clustering procedure is repeated in each permutation, and the largest cluster-mass value is retained, to derive a permutation distribution for cluster-mass values expected under the null hypothesis. Each cluster in the actual data can then be evaluated against this distribution. .. GENERATED FROM PYTHON SOURCE LINES 88-92 .. code-block:: Python result = testnd.TTestOneSample(eeg, samples=1000, pmin=.05) p = plot.TopoButterfly(result, t=0.13, head_radius=0.35) .. image-sg:: /auto_examples/mass-univariate-statistics/images/sphx_glr_permutation-statistics_006.png :alt: permutation statistics :srcset: /auto_examples/mass-univariate-statistics/images/sphx_glr_permutation-statistics_006.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 93-99 Compared to the ``max(t)`` approach above: 1) Clusters tend to be larger, because an uncorrected *p* = .05 (the cluster forming threshold) usually corresponds to a lower *t*-value than a corrected *p* = .05. 2) A second, later cluster emerged, showing that this test can be more powerful when effects are temporally and spatially extended. Because clusters are formed before computing the permutation distribution, there are usually many non-significant clusters (we don not want to list them all here): .. GENERATED FROM PYTHON SOURCE LINES 99-102 .. code-block:: Python result.find_clusters().head() .. raw:: html
# id tstart tstop duration n_sensors v p sig
0 1 -0.05 -0.04 0.01 1 4.6601 1
1 2 0.06 0.065 0.005 1 2.021 1
2 3 0.495 0.505 0.01 1 4.4984 1
3 4 -0.075 -0.07 0.005 1 2.0153 1
4 5 0.01 0.015 0.005 1 2.1638 1
5 6 -0.095 -0.07 0.025 2 14.389 1
6 7 -0.05 -0.04 0.01 2 10.504 1
7 8 -0.1 -0.095 0.005 1 2.2199 1
8 9 0.01 0.015 0.005 2 4.7805 1
9 11 0.185 0.2 0.015 4 18.882 1


.. GENERATED FROM PYTHON SOURCE LINES 103-104 List only significant clusters .. GENERATED FROM PYTHON SOURCE LINES 104-106 .. code-block:: Python result.find_clusters(pmin=0.05) .. raw:: html
# id tstart tstop duration n_sensors v p sig
0 53 0.08 0.18 0.1 19 1027.6 0 ***
1 82 0.235 0.32 0.085 10 220.68 0.021 *
2 200 0.075 0.185 0.11 37 -2329.2 0 ***
3 202 0.215 0.345 0.13 31 -600.26 0 ***


.. GENERATED FROM PYTHON SOURCE LINES 107-111 Threshold-free cluster enhancement (TFCE) ----------------------------------------- TFCE is an image processing algorithm that enhances cluster-like features, but without setting an arbitrary threshold. It thus combines the advantage the max statistic approach (not having to set an arbitrary threshold) with the advantage of the cluster-based test of increased sensitivity for effects that are extended in space and time. .. GENERATED FROM PYTHON SOURCE LINES 111-115 .. code-block:: Python result = testnd.TTestOneSample(eeg, samples=1000, tfce=True) p = plot.TopoButterfly(result, t=0.13, head_radius=0.35) .. image-sg:: /auto_examples/mass-univariate-statistics/images/sphx_glr_permutation-statistics_007.png :alt: permutation statistics :srcset: /auto_examples/mass-univariate-statistics/images/sphx_glr_permutation-statistics_007.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 116-117 Result representation is analogous to the max statistic approach, with the addition that the TFCE map can be visualized: .. GENERATED FROM PYTHON SOURCE LINES 117-119 .. code-block:: Python p = plot.TopoButterfly(result.tfce_map, t=0.13, head_radius=0.35) .. image-sg:: /auto_examples/mass-univariate-statistics/images/sphx_glr_permutation-statistics_008.png :alt: permutation statistics :srcset: /auto_examples/mass-univariate-statistics/images/sphx_glr_permutation-statistics_008.png :class: sphx-glr-single-img .. rst-class:: sphx-glr-timing **Total running time of the script:** (0 minutes 9.747 seconds) .. _sphx_glr_download_auto_examples_mass-univariate-statistics_permutation-statistics.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: permutation-statistics.ipynb ` .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: permutation-statistics.py ` .. container:: sphx-glr-download sphx-glr-download-zip :download:`Download zipped: permutation-statistics.zip ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_