Skip to content

Commit

Permalink
JSON schema importer
Browse files Browse the repository at this point in the history
Add ability to import schema defined in JSON format via file path or directly via JSON string.
This implementation doesn't migrate or update already defined schema elements. Neither it removes existing defined elements.
SchemaInitStrategy is defined in mind to be able to implement migration and more advance schema management, even so current JSON schema importer implementation is fairly simple. This work is made in hopes to simplify schema definition for beginners, speed-up prototypes development based on JanusGraph, and simplify testing.

Signed-off-by: Oleksandr Porunov <[email protected]>
  • Loading branch information
porunov committed Oct 14, 2024
1 parent 9f09767 commit 3adfaae
Show file tree
Hide file tree
Showing 50 changed files with 3,196 additions and 3 deletions.
23 changes: 23 additions & 0 deletions docs/configs/janusgraph-cfg.md
Original file line number Diff line number Diff line change
Expand Up @@ -384,6 +384,29 @@ Schema related configuration options
| schema.default | Configures the DefaultSchemaMaker to be used by this graph. Either one of the following shorthands can be used: <br> - `default` (a blueprints compatible schema maker with MULTI edge labels and SINGLE property keys),<br> - `tp3` (same as default, but has LIST property keys),<br> - `none` (automatic schema creation is disabled)<br> - `ignore-prop` (same as none, but simply ignore unknown properties rather than throw exceptions)<br> - or to the full package and classname of a custom/third-party implementing the interface `org.janusgraph.core.schema.DefaultSchemaMaker` | String | default | MASKABLE |
| schema.logging | Controls whether logging is enabled for schema makers. This only takes effect if you set `schema.default` to `default` or `ignore-prop`. For `default` schema maker, warning messages will be logged before schema types are created automatically. For `ignore-prop` schema maker, warning messages will be logged before unknown properties are ignored. | Boolean | false | MASKABLE |

### schema.init
Configuration options to configure schema initialization options.


| Name | Description | Datatype | Default Value | Mutability |
| ---- | ---- | ---- | ---- | ---- |
| schema.init.schema-drop-before-startup | Drops the whole schema before JanusGraph schema initialization. Notice schema is dropped regardless of chosen initialization strategy (it will be dropped even if `none` schema-init-strategy is selected). | Boolean | false | LOCAL |
| schema.init.schema-init-strategy | Selects the strategy for schema initialization before JanusGraph is started. The full class path which implements `SchemaInitStrategy` interface and has no any constructor parameters must be provided. Also, the following shortcuts exist:<br>- `none` - Skip any schema initialization.<br>- `json` - Use provided JSON file for schema initialization.<br> | String | none | LOCAL |

### schema.init.json
Options for JSON schema initialization strategy.


| Name | Description | Datatype | Default Value | Mutability |
| ---- | ---- | ---- | ---- | ---- |
| schema.init.json.await-index-status-timeout | Timeout for awaiting index status operation defined in milliseconds. If the status await timeouts an exception will be thrown during schema initialization process. | Long | 180000 | LOCAL |
| schema.init.json.file | File path to JSON formated schema definition. | String | (no default value) | LOCAL |
| schema.init.json.force-close-other-instances | Force closes other JanusGraph instances before schema initialization regardless if they are active or not. This is a dangerous operation. This option exists to help people initialize schema who struggle with zombie JanusGraph instances. It's not recommended to be used unless you know what you are doing. Instead of this parameter, it's recommended to check `graph.unique-instance-id` and `graph.replace-instance-if-exists` options to not create zombie instances in the cluster. | Boolean | false | LOCAL |
| schema.init.json.indices-activation | Indices activation type:<br>- `reindex_and_enable_updated_only` - Reindex process will be triggered for any updated index. After this all updated indexes will be enabled.<br>- `reindex_and_enable_non_enabled` - Reindex process will be triggered for any index which is not enabled. After this all indexes will be enabled.<br>- `skip_activation` - Skip reindex process for any updated indexes.<br>- `force_enable_updated_only` - Force enable all updated indexes without running any reindex process (previous data may not be available for such indices).<br>- `force_enable_non_enabled` - Force enable all indexes (including previously created indexes) without running any reindex process (previous data may not be available for such indices).<br> | String | reindex_and_enable_non_enabled | LOCAL |
| schema.init.json.skip-elements | Skip creation of VertexLabel, EdgeLabel, and PropertyKey. | Boolean | false | LOCAL |
| schema.init.json.skip-indices | Skip creation of indices. | Boolean | false | LOCAL |
| schema.init.json.string | JSON formated string of schema definition. This option takes precedence if both `file` and `string` are used. | String | (no default value) | LOCAL |

### storage
Configuration options for the storage backend. Some options are applicable only for certain backends.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@
import com.google.common.collect.Iterables;
import com.google.common.collect.Maps;
import com.google.common.collect.Sets;
import org.apache.commons.configuration2.MapConfiguration;
import org.apache.tinkerpop.gremlin.structure.Edge;
import org.apache.tinkerpop.gremlin.structure.Element;
import org.apache.tinkerpop.gremlin.structure.T;
Expand Down Expand Up @@ -207,7 +208,7 @@ public static Map<TestConfigOption,Object> validateConfigOptions(Object... setti
return options;
}

public void clopen(Object... settings) {
public void setupConfig(Object... settings){
config = getConfiguration();
if (mgmt!=null && mgmt.isOpen()) mgmt.rollback();
if (null != tx && tx.isOpen()) tx.commit();
Expand All @@ -226,6 +227,10 @@ public void clopen(Object... settings) {
if (janusGraphManagement!=null) janusGraphManagement.commit();
modifiableConfiguration.close();
}
}

public void clopen(Object... settings) {
setupConfig(settings);
if (null != graph && null != graph.tx() && graph.tx().isOpen())
graph.tx().commit();
if (null != graph && graph.isOpen())
Expand All @@ -234,6 +239,29 @@ public void clopen(Object... settings) {
open(config);
}

public void clopenNoDelimiterUsage(Map<String, Object> extraConfigs, boolean clearGraph) {
this.config = getConfiguration();
if (null != graph && null != graph.tx() && graph.tx().isOpen())
graph.tx().commit();
if (null != graph && graph.isOpen())
graph.close();
if(clearGraph) {
try {
clearGraph(config);
} catch (BackendException e) {
throw new RuntimeException(e);

Check warning on line 252 in janusgraph-backend-testutils/src/main/java/org/janusgraph/graphdb/JanusGraphBaseTest.java

View check run for this annotation

Codacy Production / Codacy Static Code Analysis

janusgraph-backend-testutils/src/main/java/org/janusgraph/graphdb/JanusGraphBaseTest.java#L252

Avoid throwing raw exception types.
}
}
readConfig = new BasicConfiguration(GraphDatabaseConfiguration.ROOT_NS, config, BasicConfiguration.Restriction.NONE);
Map<String, Object> finalConfig = new HashMap<>();
readConfig.getAll().forEach((pathIdentifier, value) -> finalConfig.put(pathIdentifier.element.toStringWithoutRoot(), value));
finalConfig.putAll(extraConfigs);

graph = (StandardJanusGraph) JanusGraphFactory.open(new MapConfiguration(finalConfig));
features = graph.getConfiguration().getStoreFeatures();
tx = graph.newTransaction();
mgmt = graph.openManagement();
}

public static TestConfigOption option(ConfigOption option, String... umbrella) {
return new TestConfigOption(option,umbrella);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@
import com.google.common.collect.Sets;
import io.github.artsok.RepeatedIfExceptionsTest;
import org.apache.commons.configuration2.MapConfiguration;
import org.apache.commons.io.IOUtils;
import org.apache.commons.lang.NotImplementedException;
import org.apache.tinkerpop.gremlin.process.traversal.P;
import org.apache.tinkerpop.gremlin.process.traversal.TextP;
Expand Down Expand Up @@ -70,13 +71,15 @@
import org.janusgraph.core.schema.ConsistencyModifier;
import org.janusgraph.core.schema.DisableDefaultSchemaMaker;
import org.janusgraph.core.schema.IgnorePropertySchemaMaker;
import org.janusgraph.core.schema.IndicesActivationType;
import org.janusgraph.core.schema.JanusGraphDefaultSchemaMaker;
import org.janusgraph.core.schema.JanusGraphIndex;
import org.janusgraph.core.schema.JanusGraphManagement;
import org.janusgraph.core.schema.JanusGraphSchemaType;
import org.janusgraph.core.schema.Mapping;
import org.janusgraph.core.schema.RelationTypeIndex;
import org.janusgraph.core.schema.SchemaAction;
import org.janusgraph.core.schema.SchemaInitType;
import org.janusgraph.core.schema.SchemaStatus;
import org.janusgraph.core.util.ManagementUtil;
import org.janusgraph.diskstorage.Backend;
Expand Down Expand Up @@ -169,6 +172,9 @@
import org.slf4j.LoggerFactory;

import java.io.File;
import java.io.IOException;
import java.io.InputStream;
import java.nio.charset.StandardCharsets;
import java.nio.file.Path;
import java.time.Duration;
import java.time.Instant;
Expand Down Expand Up @@ -234,6 +240,10 @@
import static org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration.REPEAT_STEP_BATCH_MODE;
import static org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration.REPLACE_INSTANCE_IF_EXISTS;
import static org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration.SCHEMA_CONSTRAINTS;
import static org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration.SCHEMA_DROP_BEFORE_INIT;
import static org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration.SCHEMA_INIT_JSON_INDICES_ACTIVATION_TYPE;
import static org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration.SCHEMA_INIT_JSON_STR;
import static org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration.SCHEMA_INIT_STRATEGY;
import static org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration.SCRIPT_EVAL_ENABLED;
import static org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration.SCRIPT_EVAL_ENGINE;
import static org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration.STORAGE_BACKEND;
Expand Down Expand Up @@ -4521,6 +4531,29 @@ private boolean isSortedByID(VertexList vl) {
return true;
}

@Test
public void testSimpleJsonSchemaImportFromProperty() throws IOException, BackendException {
ClassLoader loader = Thread.currentThread().getContextClassLoader();
InputStream resourceStream = loader.getResourceAsStream("jsonSimpleSchemaExample.json");
String jsonSchemaExample = IOUtils.toString(resourceStream, StandardCharsets.UTF_8.name());

Map<String, Object> extraConfig = new HashMap<>();
extraConfig.put(SCHEMA_INIT_STRATEGY.toStringWithoutRoot(), SchemaInitType.JSON.getConfigName());
extraConfig.put(SCHEMA_DROP_BEFORE_INIT.toStringWithoutRoot(), true);
extraConfig.put(SCHEMA_INIT_JSON_STR.toStringWithoutRoot(), jsonSchemaExample);
extraConfig.put(SCHEMA_INIT_JSON_INDICES_ACTIVATION_TYPE.toStringWithoutRoot(), IndicesActivationType.REINDEX_AND_ENABLE_UPDATED_ONLY.getConfigName());
clopenNoDelimiterUsage(extraConfig, true);

assertEquals(Long.class, mgmt.getPropertyKey("time").dataType());
assertEquals(Double.class, mgmt.getPropertyKey("doubleProp").dataType());
assertEquals(Cardinality.LIST, mgmt.getPropertyKey("longPropCardinalityList").cardinality());
assertEquals("organization", mgmt.getVertexLabel("organization").name());
assertEquals(Multiplicity.SIMPLE, mgmt.getEdgeLabel("connects").multiplicity());
assertTrue(mgmt.getEdgeLabel("connects").isDirected());
assertEquals(Multiplicity.MULTI, mgmt.getEdgeLabel("viewed").multiplicity());
assertTrue(mgmt.getEdgeLabel("viewed").isUnidirected());
}

@Test
public void testEdgesExceedCacheSize() {
// Add a vertex with as many edges as the tx-cache-size. (20000 by default)
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
{
"vertexLabels": [
{
"label": "organization"
},
{
"label": "user"
}
],

"edgeLabels": [
{
"label": "connects",
"multiplicity": "SIMPLE",
"unidirected": false
},
{
"label": "viewed",
"multiplicity": "MULTI",
"unidirected": true
}
],

"propertyKeys": [
{
"key": "time",
"className": "java.lang.Long"
},
{
"key": "doubleProp",
"className": "java.lang.Double"
},
{
"key": "integerProp",
"className": "java.lang.Integer"
},
{
"key": "longPropCardinalityList",
"className": "java.lang.Long",
"cardinality": "LIST"
},
{
"key": "name",
"className": "java.lang.String",
"cardinality": "SINGLE"
}
]
}
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@
import org.apache.tinkerpop.gremlin.structure.Graph;
import org.janusgraph.core.log.LogProcessorFramework;
import org.janusgraph.core.log.TransactionRecovery;
import org.janusgraph.core.schema.SchemaInitializationManager;
import org.janusgraph.diskstorage.Backend;
import org.janusgraph.diskstorage.BackendException;
import org.janusgraph.diskstorage.StandardStoreManager;
Expand Down Expand Up @@ -162,7 +163,7 @@ public static JanusGraph open(ReadConfiguration configuration, String backupName
final JanusGraphManager jgm = JanusGraphManagerUtility.getInstance();
if (null != graphName) {
Preconditions.checkNotNull(jgm, JANUS_GRAPH_MANAGER_EXPECTED_STATE_MSG);
return (JanusGraph) jgm.openGraph(graphName, gName -> new StandardJanusGraph(new GraphDatabaseConfigurationBuilder().build(configuration)));
return (JanusGraph) jgm.openGraph(graphName, gName -> defineSchemaAndStart(configuration));
} else {
if (jgm != null) {
log.warn("You should supply \"graph.graphname\" in your .properties file configuration if you are opening " +
Expand All @@ -173,10 +174,15 @@ public static JanusGraph open(ReadConfiguration configuration, String backupName
"\"graph.graphname\" so these graphs should be accessed dynamically by supplying a .properties file here " +
"or by using the ConfiguredGraphFactory.");
}
return new StandardJanusGraph(new GraphDatabaseConfigurationBuilder().build(configuration));
return defineSchemaAndStart(configuration);
}
}

private static JanusGraph defineSchemaAndStart(ReadConfiguration configuration) {
GraphDatabaseConfiguration graphDatabaseConfiguration = new GraphDatabaseConfigurationBuilder().build(configuration);
return SchemaInitializationManager.initializeSchemaAndStart(graphDatabaseConfiguration);
}

/**
* Return a Set of graph names stored in the {@link JanusGraphManager}
*
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
// Copyright 2024 JanusGraph Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package org.janusgraph.core.schema;

import org.janusgraph.graphdb.configuration.ConfigName;

public enum IndicesActivationType implements ConfigName {
SKIP_ACTIVATION("skip_activation"),
REINDEX_AND_ENABLE_UPDATED_ONLY("reindex_and_enable_updated_only"),
REINDEX_AND_ENABLE_NON_ENABLED("reindex_and_enable_non_enabled"),
FORCE_ENABLE_UPDATED_ONLY("force_enable_updated_only"),
FORCE_ENABLE_NON_ENABLED("force_enable_non_enabled");

private final String configurationOptionName;

IndicesActivationType(String configurationOptionName){
this.configurationOptionName = configurationOptionName;
}

@Override
public String getConfigName() {
return configurationOptionName;
}
}
Loading

0 comments on commit 3adfaae

Please sign in to comment.