Add a new NodeSymbolCollector for debugging or other context

[llaville]

Hello @icanhazstring

My future project based on composer-unused/symbol-parser need a specialized collector that is able to :

• retrieve symbols list depending on strategies available (this feature is already provided by your package)
• identify what nodes are origin of symbols (this feature did not yet exists)

This is the goal of this request :

• can be useful if you want to debug and found where symbols are defined/used by your source code
• may be reused for other usages.

Are you agree to include it by default in your package ?
Awaiting your feedbacks

Pre-requis :

• change visibility of https://github.com/composer-unused/symbol-parser/blob/0.2.2/src/Parser/PHP/ConsumedSymbolCollector.php#L24-L26 from private to protected.

Use Case 1
Retrieve only symbols list : uses the standard ConsumedSymbolCollector

On repo source code, with only one file scanned (AbstractCollector : see $dataSource in example script below), will print something like

>>> Symbols found :
array (
  0 => 'ComposerUnused\\SymbolParser\\Parser\\PHP',
  1 => 'Closure',
  2 => 'PhpParser\\Node',
  3 => 'PhpParser\\NodeVisitorAbstract',
  4 => 'ComposerUnused\\SymbolParser\\Parser\\PHP\\AbstractCollector',
)

Use Case 2
Retrieve symbols list and what nodes defined/used them: replace the standard ConsumedSymbolCollector by new NodeSymbolCollector

On repo source code, will print something like

>>> Symbols found :
array (
  0 => 'ComposerUnused\\SymbolParser\\Parser\\PHP',
  1 => 'Closure',
  2 => 'PhpParser\\Node',
  3 => 'PhpParser\\NodeVisitorAbstract',
  4 => 'ComposerUnused\\SymbolParser\\Parser\\PHP\\AbstractCollector',
)
>>> Nodes handled :
array (
  0 =>
  array (
    'symbols' =>
    array (
      0 => 'ComposerUnused\\SymbolParser\\Parser\\PHP',
    ),
    'strategy' => 'ComposerUnused\\SymbolParser\\Parser\\PHP\\Strategy\\DefineStrategy',
    'nodeType' => 'Stmt_Namespace',
    'nodeAttributes' =>
    array (
      'startLine' => 5,
      'startTokenPos' => 10,
      'startFilePos' => 33,
      'endLine' => 46,
      'endTokenPos' => 239,
      'endFilePos' => 1165,
      'kind' => 1,
    ),
  ),
  1 =>
  array (
    'symbols' =>
    array (
      0 => 'Closure',
    ),
    'strategy' => 'ComposerUnused\\SymbolParser\\Parser\\PHP\\Strategy\\UseStrategy',
    'nodeType' => 'Stmt_Use',
    'nodeAttributes' =>
    array (
      'startLine' => 7,
      'startTokenPos' => 15,
      'startFilePos' => 84,
      'endLine' => 7,
      'endTokenPos' => 18,
      'endFilePos' => 95,
    ),
  ),
  2 =>
  array (
    'symbols' =>
    array (
      0 => 'PhpParser\\Node',
    ),
    'strategy' => 'ComposerUnused\\SymbolParser\\Parser\\PHP\\Strategy\\UseStrategy',
    'nodeType' => 'Stmt_Use',
    'nodeAttributes' =>
    array (
      'startLine' => 8,
      'startTokenPos' => 20,
      'startFilePos' => 97,
      'endLine' => 8,
      'endTokenPos' => 23,
      'endFilePos' => 115,
    ),
  ),
  3 =>
  array (
    'symbols' =>
    array (
      0 => 'PhpParser\\NodeVisitorAbstract',
    ),
    'strategy' => 'ComposerUnused\\SymbolParser\\Parser\\PHP\\Strategy\\UseStrategy',
    'nodeType' => 'Stmt_Use',
    'nodeAttributes' =>
    array (
      'startLine' => 9,
      'startTokenPos' => 25,
      'startFilePos' => 117,
      'endLine' => 9,
      'endTokenPos' => 28,
      'endFilePos' => 150,
    ),
  ),
  4 =>
  array (
    'symbols' =>
    array (
      0 => 'ComposerUnused\\SymbolParser\\Parser\\PHP\\AbstractCollector',
    ),
    'strategy' => 'ComposerUnused\\SymbolParser\\Parser\\PHP\\Strategy\\DefineStrategy',
    'nodeType' => 'Stmt_Class',
    'nodeAttributes' =>
    array (
      'startLine' => 11,
      'startTokenPos' => 30,
      'startFilePos' => 153,
      'endLine' => 46,
      'endTokenPos' => 239,
      'endFilePos' => 1165,
    ),
  ),
)
>>> Nodes not handled :
array (
  0 =>
  array (
    'strategy' => 'ComposerUnused\\SymbolParser\\Parser\\PHP\\Strategy\\DefineStrategy',
    'nodeType' => 'Stmt_Expression',
    'nodeAttributes' =>
    array (
      'startLine' => 39,
      'startTokenPos' => 192,
      'startFilePos' => 911,
      'endLine' => 39,
      'endTokenPos' => 205,
      'endFilePos' => 967,
    ),
  ),
  1 =>
  array (
    'strategy' => 'ComposerUnused\\SymbolParser\\Parser\\PHP\\Strategy\\DefineStrategy',
    'nodeType' => 'Stmt_Expression',
    'nodeAttributes' =>
    array (
      'startLine' => 42,
      'startTokenPos' => 217,
      'startFilePos' => 1059,
      'endLine' => 42,
      'endTokenPos' => 230,
      'endFilePos' => 1124,
    ),
  ),
)

POC example script

<?php

$rootDir = __DIR__;
require_once $rootDir . '/vendor/autoload.php';

use ComposerUnused\SymbolParser\Parser\PHP\ConsumedSymbolCollector;
use ComposerUnused\SymbolParser\Parser\PHP\NodeSymbolCollector;
use ComposerUnused\SymbolParser\Parser\PHP\Strategy\DefineStrategy;
use ComposerUnused\SymbolParser\Parser\PHP\Strategy\UseStrategy;
use ComposerUnused\SymbolParser\Parser\PHP\SymbolCollectorInterface;
use ComposerUnused\SymbolParser\Parser\PHP\SymbolNameParser;
use ComposerUnused\SymbolParser\Parser\PHP\SymbolNameParserInterface;
use PhpParser\NodeVisitor\NameResolver;
use PhpParser\ParserFactory;

class TraceableSymbols
{
    protected SymbolCollectorInterface $symbolCollector;

    public function __construct(SymbolCollectorInterface $symbolCollector)
    {
        $this->symbolCollector = $symbolCollector;
    }

    public function itShouldRetrieveSymbolNames(string $code): array
    {
        $symbolNameParser = $this->getSymbolNameParser();

        return iterator_to_array($symbolNameParser->parseSymbolNames($code));
    }

    protected function getSymbolNameParser(): SymbolNameParserInterface
    {
        return new SymbolNameParser(
            (new ParserFactory())->createForNewestSupportedVersion(),
            new NameResolver(),
            $this->symbolCollector
        );
    }
}

$dataSource = $rootDir . '/src';
$dataSource .= '/Parser/PHP/AbstractCollector.php';

$code = file_get_contents($dataSource);

$strategies = [
    new DefineStrategy(),
    new UseStrategy(),
];

//$symbolCollector = new NodeSymbolCollector($strategies);
$symbolCollector = new ConsumedSymbolCollector($strategies);

$traceableSymbols = new TraceableSymbols($symbolCollector);

$symbols = $traceableSymbols->itShouldRetrieveSymbolNames($code);

printf('>>> Symbols found :' . PHP_EOL);
var_export($symbols);
echo PHP_EOL;

if ($symbolCollector instanceof NodeSymbolCollector) {
    printf('>>> Nodes handled :' . PHP_EOL);
    var_export($symbolCollector->getHandledNodes());
    echo PHP_EOL;
    printf('>>> Nodes not handled :' . PHP_EOL);
    var_export($symbolCollector->getNotHandledNodes());
    echo PHP_EOL;
}

[llaville]

Source code of my NodeSymbolCollector is below :

CAUTION works only if ConsumedSymbolCollector properties visibility are changed from private to protected !

<?php

declare(strict_types=1);

namespace ComposerUnused\SymbolParser\Parser\PHP;

use PhpParser\Node;
use SplObjectStorage;
use function get_class;

/**
 * @author Laurent Laville
 */
final class NodeSymbolCollector extends ConsumedSymbolCollector
{
    private SplObjectStorage $nodeStack;

    public function __construct(array $properties)
    {
        parent::__construct($properties);
        $this->nodeStack = new SplObjectStorage();
    }

    public function getHandledNodes(): array
    {
        $handled = [];
        foreach ($this->nodeStack as $node) {
            [$symbols, $strategy] = $this->nodeStack->getInfo();
            if (empty($symbols)) {
                continue;
            }
            $handled[] = [
                'symbols' => $symbols,
                'strategy' => $strategy,
                'nodeType' => $node->getType(),
                'nodeAttributes' => $node->getAttributes(),
            ];
        }
        return $handled;
    }

    public function getNotHandledNodes(): array
    {
        $handled = [];
        foreach ($this->nodeStack as $node) {
            [$symbols, $strategy] = $this->nodeStack->getInfo();
            if (!empty($symbols)) {
                continue;
            }
            $handled[] = [
                'strategy' => $strategy,
                'nodeType' => $node->getType(),
                'nodeAttributes' => $node->getAttributes(),
            ];
        }
        return $handled;
    }

    public function enterNode(Node $node)
    {
        $symbols = [];

        $this->followIncludes($node);

        foreach ($this->strategies as $strategy) {
            if (!$strategy->canHandle($node)) {
                continue;
            }
            $symbols[] = $collectedSymbols = $strategy->extractSymbolNames($node);

            $this->nodeStack->attach($node, [$collectedSymbols, get_class($strategy)]);
        }

        if (count($symbols) > 0) {
            $this->symbols = array_merge($this->symbols, ...$symbols);
        }

        return null;
    }
}

[icanhazstring]

Love the idea. I didn't spend much time to give some deeper thoughts into it, but will get to it.

What came to mind with the changes you did was some kind of using the symbol-parser to identify used or unused code if your own application. (composer-unused/unused-code, so to say)

Meaning to find dead code for symbols your code base defines but never uses.

[llaville]

Meaning to find dead code for symbols your code base defines but never uses.

This is not really my goal.
I'm still working on a new version of NodeSymbolCollector that will allow more power. BTW, I've identifed a feature that is not yet implemented into the SymbolNameParser, but may be really useful for debuggin purpose.
WDYT about it ?

Here is the patch (adding a new method to return full AST after analyse)

diff --git a/src/Parser/PHP/SymbolNameParser.php b/src/Parser/PHP/SymbolNameParser.php
index 6868ad0..c4d53ea 100644
--- a/src/Parser/PHP/SymbolNameParser.php
+++ b/src/Parser/PHP/SymbolNameParser.php
@@ -18,6 +18,7 @@ final class SymbolNameParser implements SymbolNameParserInterface
 {
     private Parser $parser;
     private NodeTraverser $traverser;
+    private array $ast;
     private SymbolCollectorInterface $visitor;
     private ?ErrorHandler $errorHandler;
     private ?FileIterationInterface $fileIterator = null;
@@ -49,12 +50,17 @@ final class SymbolNameParser implements SymbolNameParserInterface
             $this->visitor->setFileIncludeCallback(Closure::fromCallable([$this, 'handleFileInclude']));
         }

-        $this->traverser->traverse($nodes);
+        $this->ast = $this->traverser->traverse($nodes);

         yield from $this->visitor->getSymbolNames();
         $this->visitor->reset();
     }

+    public function getAst(): array
+    {
+        return $this->ast;
+    }
+
     public function setFileIterator(FileIterationInterface $fileIterator): void
     {
         $this->fileIterator = $fileIterator;

[llaville]

Forget about my previous comment on new method getAst : it's useless !
I've another version of NodeSymbolCollector that I like more

[llaville]

FYI: I've put all my contributions combined into the contribs branch (https://github.com/llaville/symbol-parser/tree/contribs), even if there are not yet validated ! This is just for me to avoid to lose my code.

[llaville]

I'll close this discussion because I've found a better solution that may match all needs (from default to specialized).
Still work on it, but I'll propose it very soon !