Added sections to readthedocs

README.md

@@ -33,38 +33,7 @@ More detailed information and examples about the specific usage of the additiona
 
 ## Related MITRE Work
 
-### CTI
-
-[Cyber Threat Intelligence repository](https://github.com/mitre/cti) of the ATT&CK catalog expressed in STIX 2.0 JSON.
-This repository also contains [our USAGE document](https://github.com/mitre/cti/blob/master/USAGE.md) which includes
-additional examples of accessing and parsing our dataset in Python.
-
-### ATT&CK
-
-ATT&CK® is a curated knowledge base and model for cyber adversary behavior, reflecting the various phases of
-an adversary’s lifecycle, and the platforms they are known to target.
-ATT&CK is useful for understanding security risk against known adversary behavior,
-for planning security improvements, and verifying defenses work as expected.
-
-<https://attack.mitre.org>
-
-### STIX
-
-Structured Threat Information Expression (STIX<sup>™</sup>) is a language and serialization format used to exchange cyber threat intelligence (CTI).
-
-STIX enables organizations to share CTI with one another in a consistent and machine-readable manner,
-allowing security communities to better understand what computer-based attacks they are most likely to
-see and to anticipate and/or respond to those attacks faster and more effectively.
-
-STIX is designed to improve many capabilities, such as collaborative threat analysis, automated threat exchange, automated detection and response, and more.
-
-<https://oasis-open.github.io/cti-documentation/>
-
-### ATT&CK scripts
-
-One-off scripts and code examples you can use as inspiration for how to work with ATT&CK programmatically. Many of the functionalities found in the mitreattack-python package were originally posted on attack-scripts.
-
-<https://github.com/mitre-attack/attack-scripts>
+Go to [this link](https://mitreattack-python.readthedocs.io/en/latest/related_work.html) for related MITRE work.
 
 
 ## Contributing

docs/attacktoexcel.rst

@@ -0,0 +1,155 @@
+ATT&CK to Excel
+==============================================
+
+ATT&CK to Excel contains a module for converting `ATT&CK STIX data <https://github.com/mitre/cti>`_ to Excel spreadsheets.
+It also provides a means to access ATT&CK data as `Pandas <https://pandas.pydata.org/>`_ DataFrames for data analysis.
+
+Usage:
+-----
+
+Command Line
+-----
+
+Print full usage instructions:
+
+.. code:: bash
+
+    python3 attackToExcel.py -h
+
+
+Example execution:
+
+.. code:: bash
+
+    python3 attackToExcel.py
+
+
+Build a excel files corresponding to a specific domain and version of ATT&CK:
+
+.. code:: bash
+
+    python3 attackToExcel -domain mobile-attack -version v5.0
+
+
+Module
+-----
+
+Example execution targeting a specific domain and version:
+
+.. code-block:: python
+
+    import mitreattack.attackToExcel.attackToExcel as attackToExcel
+
+    attackToExcel.export("mobile-attack", "v5.0", "/path/to/export/folder")
+
+
+Interfaces:
+-----
+
+attackToExcel
+-----
+
+attackToExcel provides the means by which to convert/extract the ATT&CK STIX data to Excel spreadsheets. A brief
+overview of the available methods follows.
+
+
+.. list-table:: Title
+   :widths: 33 33 34
+   :header-rows: 1
+
+   * - method name
+     - arguments
+     - usage
+   * - get_stix_data
+     - `domain`: the domain of ATT&CK to fetch data from <br> `version`: optional parameter indicating which version to fetch data from (such as "v8.1"). If omitted retrieves the most recent version of ATT&CK. <br>`remote`: optional parameter that provides a URL of a remote ATT&CK Workbench instance to grab data from.
+     - Retrieves the ATT&CK STIX data for the specified version and returns it as a MemoryStore object
+   * - build_dataframes
+     - `src`: MemoryStore or other stix2 DataSource object holding domain data<br> `domain`: domain of ATT&CK that `src` corresponds to
+     - Builds a Pandas DataFrame collection as a dictionary, with keys for each type, based on the ATT&CK data provided
+   * - write_excel
+     - `dataframes`: pandas DataFrame dictionary (generated by build_dataframes) <br>  `domain`: domain of ATT&CK that `dataframes` corresponds to <br> `version`: optional parameter indicating which version of ATT&CK is in use <br> `output_dir`: optional parameter specifying output directory
+     - Writes out DataFrame based ATT&CK data to excel files
+   * - export
+     - `domain`: the domain of ATT&CK to download <br> `version`: optional parameter specifying which version of ATT&CK to download <br> `output_dir`: optional parameter specifying output directory
+     - Downloads ATT&CK data from MITRE/CTI and exports it to Excel spreadsheets
+
+stixToDf
+-----
+
+stixToDf provides various methods to process and manipulate the STIX data in order to create `Pandas <https://pandas.pydata.org/>`_ DataFrames for
+processing. A brief overview of these methods follows.
+
+.. list-table:: 
+   :widths: 33 33 34
+   :header-rows: 1
+
+   * - method name
+     - arguments
+     - usage
+   * - techniquesToDf
+     - `src`: MemoryStore or other stix2 DataSource object holding domain data<br> `domain`: domain of ATT&CK that `src` corresponds to
+     - Parses STIX techniques from the provided data and returns corresponding Pandas DataFrames.
+   * - tacticsToDf
+     - `src`: MemoryStore or other stix2 DataSource object holding domain data<br> `domain`: domain of ATT&CK that `src` corresponds to
+     - Parses STIX tactics from the provided data and returns corresponding Pandas DataFrames.
+   * - softwareToDf
+     - `src`: MemoryStore or other stix2 DataSource object holding domain data<br> `domain`: domain of ATT&CK that `src` corresponds to
+     - Parses STIX software from the provided data and returns corresponding Pandas DataFrames.
+   * - groupsToDf
+     - `src`: MemoryStore or other stix2 DataSource object holding domain data<br> `domain`: domain of ATT&CK that `src` corresponds to
+     - Parses STIX groups from the provided data and returns corresponding Pandas DataFrames.
+   * - mitigationsToDf
+     - `src`: MemoryStore or other stix2 DataSource object holding domain data<br> `domain`: domain of ATT&CK that `src` corresponds to
+     - Parses STIX mitigations from the provided data and returns corresponding Pandas DataFrames.
+   * - relationshipsToDf
+     - `src`: MemoryStore or other stix2 DataSource object holding domain data<br> `domain`: domain of ATT&CK that `src` corresponds to
+     - Parses STIX relationships from the provided data and returns corresponding Pandas DataFrames.
+   * - matricesToDf
+     - `src`: MemoryStore or other stix2 DataSource object holding domain data<br> `domain`: domain of ATT&CK that `src` corresponds to
+     - Parses STIX matrices from the provided data and returns a parsed matrix structure of the form `[{matrix, name, description, merge, border}, ...]`
+
+
+Spreadsheet format
+-----
+
+The Excel representation of the ATT&CK dataset includes both master spreadsheets,
+containing all object types, and individual spreadsheets for each object type.
+The individual type spreadsheets break out relationships (e.g procedure examples connecting groups to techniques)
+into separate sheets by relationship type, while the master spreadsheet includes all relationship types in a single sheet.
+Otherwise, the representation is identical.
+
+A citations sheet can be used to look up the in-text citations which appear in some fields.
+For domains that include multiple matrices, such as Mobile ATT&CK, each matrix gets its own named sheet.
+Unlike the STIX dataset, objects that have been revoked or deprecated are not included in the spreadsheets.
+
+Accessing the Pandas DataFrames
+-----
+
+Internally, attackToExcel stores the parsed STIX data as `Pandas <https://pandas.pydata.org/>`_ DataFrames.
+These can be retrieved for use in data analysis.
+
+Example of accessing `Pandas <https://pandas.pydata.org/>`_ DataFrames:
+
+.. code-block:: python
+
+    import mitreattack.attackToExcel.attackToExcel as attackToExcel
+    import mitreattack.attackToExcel.stixToDf as stixToDf
+
+    # download and parse ATT&CK STIX data
+    attackdata = attackToExcel.get_stix_data("enterprise-attack")
+    techniques_data = stixToDf.techniquesToDf(attackdata, "enterprise-attack")
+
+    # show T1102 and sub-techniques of T1102
+    techniques_df = techniques_data["techniques"]
+    print(techniques_df[techniques_df["ID"].str.contains("T1102")]["name"])
+    # 512                                 Web Service
+    # 38     Web Service: Bidirectional Communication
+    # 121             Web Service: Dead Drop Resolver
+    # 323          Web Service: One-Way Communication
+    # Name: name, dtype: object
+
+    # show citation data for LOLBAS Wmic reference
+    citations_df = techniques_data["citations"]
+    print(citations_df[citations_df["reference"].str.contains("LOLBAS Wmic")])
+    #         reference                                           citation                                                url
+    # 1010  LOLBAS Wmic  LOLBAS. (n.d.). Wmic.exe. Retrieved July 31, 2...  https://lolbas-project.github.io/lolbas/Binari...

docs/collections.rst

@@ -0,0 +1,113 @@
+
+Collections
+==============================================
+
+Collections contains modules and scripts for working with ATT&CK collections.
+Collections are sets of ATT&CK STIX objects, grouped for user convienence.
+For more information about ATT&CK collections, see the corresponding
+`ATT&CK documentation <https://github.com/center-for-threat-informed-defense/attack-workbench-frontend/blob/master/docs/collections.md#collections>`_.
+
+Collections Scripts
+------------------------
+
+
+.. list-table:: 
+   :widths: 50 50
+   :header-rows: 1
+
+   * - script
+     - description
+   * - `index_to_markdown <https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/collections/index_to_markdown.py>`_ 
+     - Provides a means by which to convert a `collection index <https://github.com/center-for-threat-informed-defense/attack-workbench-frontend/blob/master/docs/collections.md#collection-indexes>`_ into a human-readable markdown file. More information can be found in the corresponding :ref:`index_to_markdown.py` below.
+   * - `collection_to_index <https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/collections/collection_to_index.py>`_ 
+     - Provides a means by which to convert a `collection <https://github.com/center-for-threat-informed-defense/attack-workbench-frontend/blob/master/docs/collections.md#collections>`_ into a easy-to-share `index file <https://github.com/center-for-threat-informed-defense/attack-workbench-frontend/blob/master/docs/collections.md#collection-indexes>`_. More information can be found in the corresponding :ref:`section<collection_to_index.py>` below.
+   * - `stix_to_collection <https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/collections/stix_to_collection.py>`_ 
+     - Provides a means by which to convert raw stix (in the form of `bundles <https://docs.oasis-open.org/cti/stix/v2.1/cs01/stix-v2.1-cs01.html#_gms872kuzdmg>`_) into a `collection <https://github.com/center-for-threat-informed-defense/attack-workbench-frontend/blob/master/docs/collections.md#collections>`_. More information can be found in the corresponding :ref:`section<stix_to_collection.py>` below.
+
+index_to_markdown.py
+------------------------
+
+``index_to_markdown.py`` provides the ``IndexToMarkdown`` class, which provides a way to transform an existing
+`collection index file <https://github.com/center-for-threat-informed-defense/attack-workbench-frontend/blob/master/docs/collections.md#collection-indexes>`_
+into a markdown file for easy of use and reference.
+The ``IndexToMarkdown`` class is very simple, and provides a single method, ``index_to_markdown``,
+which in turn only requires a single parameter - a dictionary representation of the desired index file to convert to markdown.
+An example of how to use the class, and method, can be found below.
+
+Example Usage
+------------------------
+
+    .. code-block:: python
+
+        import json
+        from mitreattack.collections import IndexToMarkdown
+
+        with open('collection_index.json', 'r') as input_file:
+            with open('collection_index.md', 'w') as output_file:
+                input_index = json.load(input_file)
+                generated_md = IndexToMarkdown.index_to_markdown(input_index)  # Convert index to markdown
+                output_file.write(generated_md)
+        print(generated_md)
+
+
+collection_to_index.py
+------------------------
+
+`collection_to_index.py` provides the `CollectionToIndex` class, which proves a means by which to summarize existing
+`collections <https://github.com/center-for-threat-informed-defense/attack-workbench-frontend/blob/master/docs/collections.md#collections>`_
+into a single `collection index <https://github.com/center-for-threat-informed-defense/attack-workbench-frontend/blob/master/docs/collections.md#collection-indexes>`_ file.
+The `CollectionToIndex` class contains the `generate_index` function, which when provided with a name, description, root url (pointing to where the raw collections are stored),
+and a list of either files, folders, or already loaded bundles in the form of dictionaries, will create a summarizing index.
+
+Example Usage
+---------------
+
+    .. code-block:: python
+
+        import json
+        from mitreattack.collections import CollectionToIndex
+
+        output_indexA = CollectionToIndex.generate_index(name='example', description='example index', 
+                                                        root_url='www.example.com', 
+                                                        files=['/path/to/collection1.json', '/path/to/collection2.json'], 
+                                                        folders=None, sets=None)
+        output_indexB = CollectionToIndex.generate_index(name='example2', description='demonstration index',
+                                                        root_url='www.example.com',
+                                                        files=None, folders=['/path/to/folder/with/collections'], sets=None)
+        with open('path/to/bundle/bundleC.json', 'r') as f:
+            data = json.load(f)
+        output_indexC = CollectionToIndex.generate_index(name='example3', description='exhibit index',
+                                                        root_url='www.example.com',
+                                                        files=None, folders=None, sets=[data])
+        print(output_indexA)
+        print(output_indexB)
+        print(output_indexC)
+
+
+stix_to_collection.py
+------------------------
+
+`stix_to_collection.py` provides the `STIXToCollection` class, which proves a means by which to convert
+existing stix bundles into ones containing a
+`collection <https://github.com/center-for-threat-informed-defense/attack-workbench-frontend/blob/master/docs/collections.md#collections>`_ object.
+The `STIXToCollection` class contains the `stix_to_collection` function, which when provided with a starter bundle,
+a name, a version, and an optional description, will output a modified bundle that contains a summary collection object.
+
+Example Usage
+--------------
+
+.. code-block:: python
+
+    import json
+    from mitreattack.collections import STIXToCollection
+
+    with open('path/to/bundle/bundle2_0.json', 'r') as f:
+        data = json.load(f)
+    output_bundleA = STIXToCollection.stix_to_collection(bundle=data, name='collectionA', version='9.1', description="demo bundle (2.0)")
+
+    with open('path/to/bundle/bundle2_1.json', 'r') as f:
+        data = json.load(f)
+    output_bundleB = STIXToCollection.stix_to_collection(bundle=data, name='collectionB', version='9.0', description="demo bundle (2.1)")
+
+    print(output_bundleA)
+    print(output_bundleB)