-
Notifications
You must be signed in to change notification settings - Fork 91
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Deployed 3fd18c8 to 3.6.0 with MkDocs 1.3.0 and mike 1.1.2
- Loading branch information
whitewum
committed
Oct 19, 2023
1 parent
d4333d9
commit 1cb19b6
Showing
4 changed files
with
102 additions
and
6 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -5924,14 +5924,17 @@ | |
|
||
|
||
<h1 id="nebulagraph_importer">NebulaGraph Importer<a class="headerlink" href="#nebulagraph_importer" title="Permanent link">¶</a></h1> | ||
<p>NebulaGraph Importer(简称 Importer)是一款NebulaGraph的 CSV 文件单机导入工具,可以读取并导入多种数据源的 CSV 文件数据。</p> | ||
<p>NebulaGraph Importer(简称 Importer)是一款NebulaGraph的 CSV 文件单机导入工具,可以读取并批量导入多种数据源的 CSV 文件数据,还支持批量更新和删除操作。</p> | ||
<h2 id="_1">功能<a class="headerlink" href="#_1" title="Permanent link">¶</a></h2> | ||
<ul> | ||
<li>支持多种数据源,包括本地、S3、OSS、HDFS、FTP、SFTP。</li> | ||
<li>支持多种数据源,包括本地、S3、OSS、HDFS、FTP、SFTP、GCS。</li> | ||
<li>支持导入 CSV 格式文件的数据。单个文件内可以包含多种 Tag、多种 Edge type 或者二者混合的数据。</li> | ||
<li>支持过滤数据源数据。</li> | ||
<li>支持批量操作,包括导入、更新、删除。</li> | ||
<li>支持同时连接多个 Graph 服务进行导入并且动态负载均衡。</li> | ||
<li>支持失败后重连、重试。</li> | ||
<li>支持多维度显示统计信息,包括导入时间、导入百分比等。统计信息支持打印在 Console 或日志中。</li> | ||
<li>支持 SSL 加密。</li> | ||
</ul> | ||
<h2 id="_2">优势<a class="headerlink" href="#_2" title="Permanent link">¶</a></h2> | ||
<ul> | ||
|
@@ -6077,6 +6080,12 @@ <h3 id="client">Client 配置<a class="headerlink" href="#client" title="Permane | |
<span class="w"> </span><span class="nt">address</span><span class="p">:</span><span class="w"> </span><span class="s">"192.168.1.100:9669,192.168.1.101:9669"</span> | ||
<span class="w"> </span><span class="nt">user</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">root</span> | ||
<span class="w"> </span><span class="nt">password</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">nebula</span> | ||
<span class="w"> </span><span class="nt">ssl</span><span class="p">:</span> | ||
<span class="w"> </span><span class="nt">enable</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">true</span> | ||
<span class="w"> </span><span class="nt">certPath</span><span class="p">:</span><span class="w"> </span><span class="s">"/home/xxx/cert/importer.crt"</span> | ||
<span class="w"> </span><span class="nt">keyPath</span><span class="p">:</span><span class="w"> </span><span class="s">"/home/xxx/cert/importer.key"</span> | ||
<span class="w"> </span><span class="nt">caPath</span><span class="p">:</span><span class="w"> </span><span class="s">"/home/xxx/cert/root.crt"</span> | ||
<span class="w"> </span><span class="nt">insecureSkipVerify</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">false</span> | ||
<span class="w"> </span><span class="nt">concurrencyPerAddress</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">10</span> | ||
<span class="w"> </span><span class="nt">reconnectInitialInterval</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">1s</span> | ||
<span class="w"> </span><span class="nt">retry</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">3</span> | ||
|
@@ -6117,6 +6126,36 @@ <h3 id="client">Client 配置<a class="headerlink" href="#client" title="Permane | |
<td align="left">NebulaGraph用户名对应的密码。</td> | ||
</tr> | ||
<tr> | ||
<td align="left"><code>client.ssl.enable</code></td> | ||
<td align="left"><code>false</code></td> | ||
<td align="left">否</td> | ||
<td align="left">指定是否开启 SSL 认证。</td> | ||
</tr> | ||
<tr> | ||
<td align="left"><code>client.ssl.certPath</code></td> | ||
<td align="left">-</td> | ||
<td align="left">否</td> | ||
<td align="left">指定 SSL 公钥证书的存储路径。</br>开启 SSL 认证后该参数必填。</td> | ||
</tr> | ||
<tr> | ||
<td align="left"><code>client.ssl.keyPath</code></td> | ||
<td align="left">-</td> | ||
<td align="left">否</td> | ||
<td align="left">指定 SSL 密钥的存储路径。</br>开启 SSL 认证后该参数必填。</td> | ||
</tr> | ||
<tr> | ||
<td align="left"><code>client.ssl.caPath</code></td> | ||
<td align="left">-</td> | ||
<td align="left">否</td> | ||
<td align="left">指定 CA 根证书的存储路径。</br>开启 SSL 认证后该参数必填。</td> | ||
</tr> | ||
<tr> | ||
<td align="left"><code>client.ssl.insecureSkipVerify</code></td> | ||
<td align="left"><code>false</code></td> | ||
<td align="left">否</td> | ||
<td align="left">指定是否跳过验证服务端的证书链和主机名。如果设置为<code>true</code>,则接受服务端提供的任何证书链和主机名。</td> | ||
</tr> | ||
<tr> | ||
<td align="left"><code>client.concurrencyPerAddress</code></td> | ||
<td align="left"><code>10</code></td> | ||
<td align="left">否</td> | ||
|
@@ -6237,7 +6276,7 @@ <h3 id="log">Log 配置<a class="headerlink" href="#log" title="Permanent link"> | |
<span class="w"> </span><span class="nt">level</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">INFO</span> | ||
<span class="w"> </span><span class="nt">console</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">true</span> | ||
<span class="w"> </span><span class="nt">files</span><span class="p">:</span> | ||
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">logs/nebula-importer.log</span><span class="w"> </span> | ||
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">logs/nebula-importer.log</span><span class="w"> </span> | ||
</code></pre></div> | ||
<table> | ||
<thead> | ||
|
@@ -6305,14 +6344,44 @@ <h3 id="source">Source 配置<a class="headerlink" href="#source" title="Permane | |
<span class="c1"># - hdfs:</span> | ||
<span class="c1"># address: "127.0.0.1:8020" # 必填。HDFS 服务的地址。</span> | ||
<span class="c1"># user: "hdfs" # 可选。HDFS 服务的用户名。</span> | ||
<span class="c1"># path: "/events/20190918.export.csv" # 必填。HDFS 服务中文件的路径。</span> | ||
<span class="c1"># servicePrincipalName: <Kerberos Service Principal Name> # 可选。启用 Kerberos 认证时,HDFS 服务的 Kerberos 服务实例名称。</span> | ||
<span class="c1"># krb5ConfigFile: <Kerberos config file> # 可选。启用 Kerberos 认证时,HDFS 服务的 Kerberos 配置文件路径,默认为`/etc/krb5.conf`。</span> | ||
<span class="c1"># ccacheFile: <Kerberos ccache file> # 可选。启用 Kerberos 认证时,HDFS 服务的 Kerberos ccache 文件路径。</span> | ||
<span class="c1"># keyTabFile: <Kerberos keytab file> # 可选。启用 Kerberos 认证时,HDFS 服务的 Kerberos keytab 文件路径。</span> | ||
<span class="c1"># password: <Kerberos password> # 可选。启用 Kerberos 认证时,HDFS 服务的 Kerberos 密码。</span> | ||
<span class="c1"># dataTransferProtection: <Kerberos Data Transfer Protection> # 可选。启用 Kerberos 认证时的传输加密类型。可选值为`authentication`、`integrity`、`privacy`。</span> | ||
<span class="c1"># disablePAFXFAST: false # 可选。是否禁止客户端使用预身份验证(PA_FX_FAST)。</span> | ||
<span class="c1"># path: "/events/20190918.export.csv" # 必填。HDFS 服务中文件的路径。也支持通配符文件名,例如:/events/*.export.csv,请确保所有匹配的文件具有相同的架构。</span> | ||
<span class="c1"># - gcs: # Google Cloud Storage</span> | ||
<span class="c1"># bucket: chicago-crime-sample # 必填。GCS 服务中的 bucket 名称。</span> | ||
<span class="c1"># key: stats/000000000000.csv # 必填。GCS 服务中文件的路径。</span> | ||
<span class="c1"># withoutAuthentication: false # 可选。是否匿名访问。默认为 false,即使用凭证访问。</span> | ||
<span class="c1"># # 使用凭证访问时,credentialsFile 和 credentialsJSON 参数二选一即可。</span> | ||
<span class="c1"># credentialsFile: "/path/to/your/credentials/file" # 可选。GCS 服务的凭证文件路径。</span> | ||
<span class="c1"># credentialsJSON: '{ # 可选。GCS 服务的凭证 JSON 内容。</span> | ||
<span class="c1"># "type": "service_account",</span> | ||
<span class="c1"># "project_id": "your-project-id",</span> | ||
<span class="c1"># "private_key_id": "key-id",</span> | ||
<span class="c1"># "private_key": "-----BEGIN PRIVATE KEY-----\nxxxxx\n-----END PRIVATE KEY-----\n",</span> | ||
<span class="c1"># "client_email": "[email protected]",</span> | ||
<span class="c1"># "client_id": "client-id",</span> | ||
<span class="c1"># "auth_uri": "https://accounts.google.com/o/oauth2/auth",</span> | ||
<span class="c1"># "token_uri": "https://oauth2.googleapis.com/token",</span> | ||
<span class="c1"># "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",</span> | ||
<span class="c1"># "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/your-client%40your-project-id.iam.gserviceaccount.com",</span> | ||
<span class="c1"># "universe_domain": "googleapis.com"</span> | ||
<span class="c1"># }'</span> | ||
|
||
<span class="w"> </span><span class="nt">batch</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">256</span> | ||
<span class="w"> </span><span class="nt">csv</span><span class="p">:</span> | ||
<span class="w"> </span><span class="nt">delimiter</span><span class="p">:</span><span class="w"> </span><span class="s">"|"</span> | ||
<span class="w"> </span><span class="nt">withHeader</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">false</span> | ||
<span class="w"> </span><span class="nt">lazyQuotes</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">false</span> | ||
<span class="w"> </span><span class="nt">tags</span><span class="p">:</span> | ||
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">Person</span> | ||
<span class="c1"># mode: INSERT</span> | ||
<span class="c1"># filter: </span> | ||
<span class="c1"># expr: Record[1] == "XXX"</span> | ||
<span class="w"> </span><span class="nt">id</span><span class="p">:</span> | ||
<span class="w"> </span><span class="nt">type</span><span class="p">:</span><span class="w"> </span><span class="s">"STRING"</span> | ||
<span class="w"> </span><span class="nt">function</span><span class="p">:</span><span class="w"> </span><span class="s">"hash"</span> | ||
|
@@ -6351,6 +6420,9 @@ <h3 id="source">Source 配置<a class="headerlink" href="#source" title="Permane | |
<span class="w"> </span><span class="nt">batch</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">256</span> | ||
<span class="w"> </span><span class="nt">edges</span><span class="p">:</span> | ||
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">KNOWS</span><span class="w"> </span><span class="c1"># person_knows_person</span> | ||
<span class="c1"># mode: INSERT</span> | ||
<span class="c1"># filter: </span> | ||
<span class="c1"># expr: Record[1] == "XXX"</span> | ||
<span class="w"> </span><span class="nt">src</span><span class="p">:</span> | ||
<span class="w"> </span><span class="nt">id</span><span class="p">:</span> | ||
<span class="w"> </span><span class="nt">type</span><span class="p">:</span><span class="w"> </span><span class="s">"STRING"</span> | ||
|
@@ -6407,7 +6479,7 @@ <h3 id="source">Source 配置<a class="headerlink" href="#source" title="Permane | |
<td align="left"><code>sources.csv.delimiter</code></td> | ||
<td align="left"><code>,</code></td> | ||
<td align="left">否</td> | ||
<td align="left">CSV 文件的分隔符。仅支持 1 个字符的字符串分隔符。特殊字符做分隔符时应使用对应的转义序列来表示,并包裹在双引号内进行转义。例如,制表符<code>\t</code>应写为<code>"\t"</code>,而不是<code>'\t'</code>;十六进制的<code>0x03</code> (<code>Ctrl+C</code>) 应写为<code>"\x03"</code>或<code>"\u0003"</code>。关于 yaml 格式特殊字符转义的细节请参见<a href="https://yaml.org/spec/1.2.2/#escaped-characters">Escaped Characters</a>。</td> | ||
<td align="left">CSV 文件的分隔符。仅支持 1 个字符的字符串分隔符。使用特殊字符做分隔符时需要进行转义。例如当分隔符为十六进制<code>0x03</code>即<code>Ctrl+C</code>时,转义的写法为:<code>"\x03"</code>或<code>"\u0003"</code>。关于 yaml 格式特殊字符转义的细节请参见<a href="https://yaml.org/spec/1.2.2/#escaped-characters">Escaped Characters</a>。</td> | ||
</tr> | ||
<tr> | ||
<td align="left"><code>sources.csv.withHeader</code></td> | ||
|
@@ -6428,6 +6500,18 @@ <h3 id="source">Source 配置<a class="headerlink" href="#source" title="Permane | |
<td align="left">Tag 名称。</td> | ||
</tr> | ||
<tr> | ||
<td align="left"><code>sources.tags.mode</code></td> | ||
<td align="left"><code>INSERT</code></td> | ||
<td align="left">否</td> | ||
<td align="left">批量操作类型,包括导入、更新和删除。可选值为<code>INSERT</code>、<code>UPDATE</code>和<code>DELETE</code>。</td> | ||
</tr> | ||
<tr> | ||
<td align="left"><code>sources.tags.filter.expr</code></td> | ||
<td align="left">-</td> | ||
<td align="left">否</td> | ||
<td align="left">过滤数据,满足过滤条件的才会导入。支持的比较符为<code>==</code>、<code>!=</code>、<code><</code>、<code>></code>、<code><=</code>和<code>>=</code>。支持的逻辑运算符为<code>not</code>(!)、<code>and</code>(&&)和<code>or</code>(||)。例如<code>(Record[0] == "Mahinda" or Record[0] == "Michael") and Record[3] == "male"</code>。</td> | ||
</tr> | ||
<tr> | ||
<td align="left"><code>sources.tags.id.type</code></td> | ||
<td align="left"><code>STRING</code></td> | ||
<td align="left">否</td> | ||
|
@@ -6506,6 +6590,18 @@ <h3 id="source">Source 配置<a class="headerlink" href="#source" title="Permane | |
<td align="left">Edge type 名称。</td> | ||
</tr> | ||
<tr> | ||
<td align="left"><code>sources.edges.mode</code></td> | ||
<td align="left"><code>INSERT</code></td> | ||
<td align="left">否</td> | ||
<td align="left">批量操作类型,包括导入、更新和删除。可选值为<code>INSERT</code>、<code>UPDATE</code>和<code>DELETE</code>。</td> | ||
</tr> | ||
<tr> | ||
<td align="left"><code>sources.edges.filter.expr</code></td> | ||
<td align="left">-</td> | ||
<td align="left">否</td> | ||
<td align="left">过滤数据,满足过滤条件的才会导入。支持的比较符为<code>==</code>、<code>!=</code>、<code><</code>、<code>></code>、<code><=</code>和<code>>=</code>。支持的逻辑运算符为<code>not</code>(!)、<code>and</code>(&&)和<code>or</code>(||)。例如<code>(Record[0] == "Mahinda" or Record[0] == "Michael") and Record[3] == "male"</code>。</td> | ||
</tr> | ||
<tr> | ||
<td align="left"><code>sources.edges.src.id.type</code></td> | ||
<td align="left"><code>STRING</code></td> | ||
<td align="left">否</td> | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Binary file not shown.
Binary file not shown.