Jedi Error

[linjie7674]

When I use repofuse code in my own dataset, I have modified the format of the dataset according to the jsonl file you provided and provided the corresponding code repository. An error is reported when the following code is executed, and most of the extracted crossfile_definition_by_dependency_graph and crossfile_reference_by_dependency_graph are empty.

The following is the error information:

2024-08-30 11:19:27,653 - dependency_graph.graph_generator.jedi_generator - ERROR - Error while extracting call relation for name <Name full_name='src.diffusers.utils.dynamic_modules_utils.get_cached_module_file.copy', description='copy'> in /home/linshaoxin/dong/jie/code/AI4SE-master/AI4SE-master/src/demo/AID-RAG/benchmarks/repoeval/repositories/huggingface_diffusers/src/diffusers/utils/dynamic_modules_utils.py: Error PosixPath('/root/miniconda3/envs/rag_llm/lib/python3.10/site-packages/jedi/third_party/typeshed/stdlib/2and3/shutil.pyi') occurred at:
  File "/root/miniconda3/envs/rag_llm/lib/python3.10/site-packages/dependency_graph/graph_generator/jedi_generator.py", line 229, in _extract_call_relation
    callers = name.goto(follow_imports=True, follow_builtin_imports=True)

  File "/root/miniconda3/envs/rag_llm/lib/python3.10/contextlib.py", line 79, in inner
    return func(*args, **kwds)

  File "/root/miniconda3/envs/rag_llm/lib/python3.10/site-packages/jedi/api/classes.py", line 445, in goto
    names = self._name.goto()

  File "/root/miniconda3/envs/rag_llm/lib/python3.10/site-packages/jedi/inference/names.py", line 206, in goto
    return values.goto(name, name_context=context)

  File "/root/miniconda3/envs/rag_llm/lib/python3.10/site-packages/jedi/inference/base_value.py", line 493, in goto
    return reduce(add, [c.goto(*args, **kwargs) for c in self._set], [])

  File "/root/miniconda3/envs/rag_llm/lib/python3.10/site-packages/jedi/inference/base_value.py", line 493, in <listcomp>
    return reduce(add, [c.goto(*args, **kwargs) for c in self._set], [])

  File "/root/miniconda3/envs/rag_llm/lib/python3.10/site-packages/jedi/inference/base_value.py", line 72, in goto
    names = finder.filter_name(filters, name_or_str)

  File "/root/miniconda3/envs/rag_llm/lib/python3.10/site-packages/jedi/inference/finder.py", line 35, in filter_name
    for filter in filters:

  File "/root/miniconda3/envs/rag_llm/lib/python3.10/site-packages/jedi/inference/base_value.py", line 62, in _get_value_filters
    yield from self.get_filters(origin_scope=origin_scope)

  File "/root/miniconda3/envs/rag_llm/lib/python3.10/site-packages/jedi/inference/gradual/stub_value.py", line 44, in get_filters
    next(filters, None)  # Ignore the first filter and replace it with our own

  File "/root/miniconda3/envs/rag_llm/lib/python3.10/site-packages/jedi/inference/value/module.py", line 63, in get_filters
    ParserTreeFilter(

  File "/root/miniconda3/envs/rag_llm/lib/python3.10/site-packages/jedi/inference/filters.py", line 138, in __init__
    super().__init__(parent_context, node_context)

  File "/root/miniconda3/envs/rag_llm/lib/python3.10/site-packages/jedi/inference/filters.py", line 100, in __init__
    self._parso_cache_node = get_parso_cache_node(

  File "/root/miniconda3/envs/rag_llm/lib/python3.10/site-packages/jedi/parser_utils.py", line 287, in get_parso_cache_node
    return parser_cache[grammar._hashed][path]

2024-08-30 11:19:27,653 - dependency_graph.graph_generator.jedi_generator - ERROR - Error while extracting call relation for name <Name full_name='src.diffusers.utils.dynamic_modules_utils.get_cached_module_file.copy', description='copy'> in /home/linshaoxin/dong/jie/code/AI4SE-master/AI4SE-master/src/demo/AID-RAG/benchmarks/repoeval/repositories/huggingface_diffusers/src/diffusers/utils/dynamic_modules_utils.py: Error PosixPath('/root/miniconda3/envs/rag_llm/lib/python3.10/site-packages/jedi/third_party/typeshed/stdlib/2and3/shutil.pyi') occurred at:
  File "/root/miniconda3/envs/rag_llm/lib/python3.10/site-packages/dependency_graph/graph_generator/jedi_generator.py", line 229, in _extract_call_relation
    callers = name.goto(follow_imports=True, follow_builtin_imports=True)

  File "/root/miniconda3/envs/rag_llm/lib/python3.10/contextlib.py", line 79, in inner
    return func(*args, **kwds)

  File "/root/miniconda3/envs/rag_llm/lib/python3.10/site-packages/jedi/api/classes.py", line 445, in goto
    names = self._name.goto()

  File "/root/miniconda3/envs/rag_llm/lib/python3.10/site-packages/jedi/inference/names.py", line 206, in goto
    return values.goto(name, name_context=context)

  File "/root/miniconda3/envs/rag_llm/lib/python3.10/site-packages/jedi/inference/base_value.py", line 493, in goto
    return reduce(add, [c.goto(*args, **kwargs) for c in self._set], [])

  File "/root/miniconda3/envs/rag_llm/lib/python3.10/site-packages/jedi/inference/base_value.py", line 493, in <listcomp>
    return reduce(add, [c.goto(*args, **kwargs) for c in self._set], [])

  File "/root/miniconda3/envs/rag_llm/lib/python3.10/site-packages/jedi/inference/base_value.py", line 72, in goto
    names = finder.filter_name(filters, name_or_str)

  File "/root/miniconda3/envs/rag_llm/lib/python3.10/site-packages/jedi/inference/finder.py", line 35, in filter_name
    for filter in filters:

  File "/root/miniconda3/envs/rag_llm/lib/python3.10/site-packages/jedi/inference/base_value.py", line 62, in _get_value_filters
    yield from self.get_filters(origin_scope=origin_scope)

  File "/root/miniconda3/envs/rag_llm/lib/python3.10/site-packages/jedi/inference/gradual/stub_value.py", line 44, in get_filters
    next(filters, None)  # Ignore the first filter and replace it with our own

  File "/root/miniconda3/envs/rag_llm/lib/python3.10/site-packages/jedi/inference/value/module.py", line 63, in get_filters
    ParserTreeFilter(

  File "/root/miniconda3/envs/rag_llm/lib/python3.10/site-packages/jedi/inference/filters.py", line 138, in __init__
    super().__init__(parent_context, node_context)

  File "/root/miniconda3/envs/rag_llm/lib/python3.10/site-packages/jedi/inference/filters.py", line 100, in __init__
    self._parso_cache_node = get_parso_cache_node(

  File "/root/miniconda3/envs/rag_llm/lib/python3.10/site-packages/jedi/parser_utils.py", line 287, in get_parso_cache_node
    return parser_cache[grammar._hashed][path]

[FunJim]

I cannot reproduce the issue. Can you please provide your OS information, the python version and the commit id of the repository huggingface/diffusers?

[linjie7674]

I cannot reproduce the issue. Can you please provide your OS information, the python version and the commit id of the repository huggingface/diffusers?

My OS is Ubuntu 20.04.4 LTS, python version is 3.10.14, and the repository huggingface/diffusers I used comes from https://github.com/microsoft/CodeT/blob/main/RepoCoder/repositories/line_and_api_level.zip

[linjie7674]

When I use the CrossCodeEval dataset and the raw data, there is also same problem.

The error information is as follows:

2024-09-05 11:51:10,853 - dependency_graph.graph_generator.jedi_generator - ERROR - Error while extracting d
ef-use relation for name <Name full_name='examples.door_puzzle.ax', description='ax = plt.gca()'> in /home/l
inshaoxin/dong/jie/datasets/crosscodeeval/crosscodeeval_rawdata/yotakayama-STLCCP-edcfad4/examples/door_puzz
le.py: Error PosixPath('/home/linshaoxin/dong/jie/datasets/crosscodeeval/crosscodeeval_rawdata/yotakayama-ST
LCCP-edcfad4/examples/door_puzzle.py') occurred at:                                                     
  File "/root/miniconda3/envs/rag_llm/lib/python3.10/site-packages/dependency_graph/graph_generator/jedi_gen
erator.py", line 358, in _extract_def_use_relation                                                      
    references = script.get_references(name.line, name.column)                                          
                                                                                                        
  File "/root/miniconda3/envs/rag_llm/lib/python3.10/site-packages/jedi/api/helpers.py", line 487, in wrappe
r                                                                                                       
    return func(self, line, column, *args, **kwargs)                                                    
                                                                                                        
  File "/root/miniconda3/envs/rag_llm/lib/python3.10/site-packages/jedi/api/__init__.py", line 429, in get_r
eferences                                                                                               
    return _references(**kwargs)                                                                        
                                                                                                        
  File "/root/miniconda3/envs/rag_llm/lib/python3.10/site-packages/jedi/api/__init__.py", line 423, in _refe
rences                                                                                                  
    names = find_references(self._get_module_context(), tree_name, scope == 'file')                     
                                                                                                        
  File "/root/miniconda3/envs/rag_llm/lib/python3.10/site-packages/jedi/inference/references.py", line 122,
in find_references                                                                                      
    found_names = _find_defining_names(module_context, tree_name)                                       
                                                                                                        
  File "/root/miniconda3/envs/rag_llm/lib/python3.10/site-packages/jedi/inference/references.py", line 63, i
n _find_defining_names                                                                                  
    found_names |= set(_find_global_variables(found_names, tree_name.value))
    
    File "/root/miniconda3/envs/rag_llm/lib/python3.10/site-packages/jedi/inference/references.py", line 63, in _find_defining_names                                                                                  
    found_names |= set(_find_global_variables(found_names, tree_name.value))                            
                                                                                                        
  File "/root/miniconda3/envs/rag_llm/lib/python3.10/site-packages/jedi/inference/references.py", line 108, in _find_global_variables                                                                               
    for global_name in method().get(search_name):                                                       
                                                                                                        
  File "/root/miniconda3/envs/rag_llm/lib/python3.10/site-packages/jedi/inference/context.py", line 330, in get_global_filter                                                                                       
    return GlobalNameFilter(self)                                                                       
                                                                                                        
  File "/root/miniconda3/envs/rag_llm/lib/python3.10/site-packages/jedi/inference/filters.py", line 100, in __init__                                                                                                
    self._parso_cache_node = get_parso_cache_node(                                                      
                                                                                                        
  File "/root/miniconda3/envs/rag_llm/lib/python3.10/site-packages/jedi/parser_utils.py", line 287, in get_parso_cache_node                                                                                         
    return parser_cache[grammar._hashed][path]

[FunJim]

It seems to me the third-party dependencies required for the repository are not installed, causing Jedi to be unable to resolve the symbol and raise an error.

This is normal because we only want to retrieve context at the repository level without including third-party dependencies. The errors are properly handled and should not affect the result.