Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

!!! FEATURE: High level Neos site import site:importAll #5307

Merged
merged 59 commits into from
Nov 12, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
59 commits
Select commit Hold shift + click to select a range
8a95449
WIP: FEATURE: Site import
bwaidelich Oct 21, 2024
d192496
Extract workspace creation to separate processor
bwaidelich Oct 21, 2024
4d4e98d
Make `SiteImportService` a singleton
bwaidelich Oct 22, 2024
b4461e5
Merge branch '9.0' into feature/4448-site-import
bwaidelich Oct 22, 2024
7b3ce80
Fix `LegacyMigrationService` initialization
bwaidelich Oct 22, 2024
c8b40bd
Re-add `Neos.ContentRepository.Export` behat tests
bwaidelich Oct 22, 2024
191ce88
Rename `Factories` folder to `Factory
bwaidelich Oct 22, 2024
aa8c611
Remove unused namespace imports
bwaidelich Oct 22, 2024
630bc09
Verify tha that Live workspace contains no events prior to importing
mficzel Oct 23, 2024
221096b
Ensure sites are persisted during import
mficzel Oct 23, 2024
918a052
Add site:export command again
mficzel Oct 23, 2024
a37387f
Avoid exporting of contentStreamId
mficzel Oct 23, 2024
4478619
TASK: Make linter happy
mficzel Oct 23, 2024
4a54ace
Add replayall to site:import process
mficzel Oct 23, 2024
9f68937
Rename command `site:import` and `site:export` to `site:importAll` an…
mficzel Oct 23, 2024
cd550d7
Add `site:pruneAll` command that empties the current cr and removes a…
mficzel Oct 23, 2024
e260954
Improve after review
mficzel Oct 23, 2024
361c71e
Extract SitePruningService
mficzel Oct 23, 2024
a05ad66
Remove ProcessorInterface from ProjectionService and introduce separa…
mficzel Oct 24, 2024
bd1ff93
Refactor migration of legacyData and add additional command `cr:expor…
mficzel Oct 24, 2024
14523b2
Rename `cr:migratelegacydata`, `cr:exportlegacydata`, to `site:migra…
mficzel Oct 24, 2024
1c2afb7
Adjust behat tests for legacy export and add sites.feature
mficzel Oct 24, 2024
4423476
Format arguments of site command controller properly
mficzel Oct 24, 2024
a9fe6ce
Adjust after QS Feedback:
mficzel Oct 28, 2024
88aad94
Merge remote-tracking branch 'origin/9.0' into feature/4448-site-import
mhsdesign Nov 2, 2024
f526e73
TASK: Post merge adjustments to make phpstan happy
mhsdesign Nov 2, 2024
0772b6a
TASK: Replace `LiveWorkspaceIsEmptyProcessor` with read model checks
mhsdesign Nov 2, 2024
e57ce53
TASK: Make `RoleAndMetadataPruningProcessor` a simple processor
mhsdesign Nov 2, 2024
ae03530
TASK: Make `ContentRepositoryPruningProcessor` use the `ContentStream…
mhsdesign Nov 2, 2024
cb82c2d
TASK: Turn `SitePruningProcessor` into simple processor without factory
mhsdesign Nov 2, 2024
dff6885
TASK: Turn `SiteExportProcessor` into simple processor without factory
mhsdesign Nov 2, 2024
71c65a2
TASK: Turn `ProjectionCatchupProcessor` and `ProjectionReplayProcesso…
mhsdesign Nov 2, 2024
9ba32fd
TASK: Document and improve `extractSitesFromEventStream`
mhsdesign Nov 2, 2024
9e7c5c1
TASK: Move todo around :)
mhsdesign Nov 2, 2024
9c9ce5b
TASK: Remove redundant documentation
mhsdesign Nov 2, 2024
f674d5e
TASK: Reintroduce resetting of projection state via `ProjectionResetP…
mhsdesign Nov 2, 2024
7e0021d
TASK: Use `Processors::fromArray`
mhsdesign Nov 2, 2024
7fea08b
TASK: Fail early if `resource://` paths are used in export
mhsdesign Nov 2, 2024
c98384d
TASK: Use `Processors::fromArray`
mhsdesign Nov 2, 2024
adf7b38
TASK: Remove automatic schema creation from import and redirect user …
mhsdesign Nov 8, 2024
03aca80
Merge remote-tracking branch 'origin/9.0' into feature/4448-site-import
dlubitz Nov 8, 2024
b701b17
TASK: Post-merge fix of tests
dlubitz Nov 8, 2024
717688d
TASK: Assert that no events exist on the live content stream ... but …
mhsdesign Nov 8, 2024
efb7fbd
TASK: Remove obsolete `I run the event migration for workspace :works…
mhsdesign Nov 8, 2024
ea67195
TASK: Adjust documentation and todos in legacy migration
mhsdesign Nov 8, 2024
210a087
TASK: Remove obsolete `ProjectionReplayProcessor`
mhsdesign Nov 8, 2024
c1d64ba
BUGFIX: Import `Duplicate entry 'onedimension.localhost' for key 'flo…
mhsdesign Nov 10, 2024
49234ff
TASK: Migration, make sure that NodeType does not exist error is thro…
mhsdesign Nov 10, 2024
5f3d350
TASK: Enforce file creation first (to throw possibly errors)
mhsdesign Nov 10, 2024
b6c4cd4
TASK: Dont use `findOneByHost` because it returns not the exact result
mhsdesign Nov 10, 2024
6fc88e0
BUGFIX: Create site with correct online state (inverse condition)
mhsdesign Nov 10, 2024
02e9cde
TASK: Simplify `SiteShape`
mhsdesign Nov 10, 2024
b11bf2e
TASK: Remove `site:migrateLegacyData` (previously `cr:migrateLegacyDa…
mhsdesign Nov 10, 2024
41e472a
BUGFIX: Export correctly export `primary` domain
mhsdesign Nov 10, 2024
aa71347
BUGFIX: `site:importAll` import `copyrightNotice` of assets
mhsdesign Nov 10, 2024
c454117
TASK: Trivial cosmetic changes
mhsdesign Nov 10, 2024
b8670cf
TASK: Make pruning a noop if the workspace does not exist
mhsdesign Nov 10, 2024
17db2eb
TASK: Adjust documentation to reference correct command
mhsdesign Nov 10, 2024
5696f36
TASK: Adjust test to 49234ffb1bd3a5f3391120f8eb0be72c6d831952
mhsdesign Nov 10, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ public function processNode(Node $node, Scope $scope): array
return [
RuleErrorBuilder::message(
'Class needs @api or @internal annotation.'
)->build(),
)->identifier('neos.cr.internal')->build(),
];
}
return [];
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ public function processNode(Node $node, Scope $scope): array
$targetClassName,
$node->name->toString()
)
)->build(),
)->identifier('neos.cr.internal')->build(),
];
}
}
Expand Down

Large diffs are not rendered by default.

Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
@contentrepository
Feature: As a user of the CR I want to export the event stream
Feature: As a user of the CR I want to export the event stream using the EventExportProcessor

Background:
Given using the following content dimensions:
Expand All @@ -12,9 +12,9 @@ Feature: As a user of the CR I want to export the event stream
And using identifier "default", I define a content repository
And I am in content repository "default"
And the command CreateRootWorkspace is executed with payload:
| Key | Value |
| workspaceName | "live" |
| newContentStreamId | "cs-identifier" |
| Key | Value |
| workspaceName | "live" |
| newContentStreamId | "cs-identifier" |
And I am in workspace "live"
And the command CreateRootNodeAggregateWithNode is executed with payload:
| Key | Value |
Expand All @@ -37,7 +37,7 @@ Feature: As a user of the CR I want to export the event stream
When the events are exported
Then I expect the following jsonl:
"""
{"identifier":"random-event-uuid","type":"RootNodeAggregateWithNodeWasCreated","payload":{"workspaceName":"live","contentStreamId":"cs-identifier","nodeAggregateId":"lady-eleonode-rootford","nodeTypeName":"Neos.ContentRepository:Root","coveredDimensionSpacePoints":[{"language":"de"},{"language":"gsw"},{"language":"fr"}],"nodeAggregateClassification":"root"},"metadata":{"commandClass":"Neos\\ContentRepository\\Core\\Feature\\RootNodeCreation\\Command\\CreateRootNodeAggregateWithNode","commandPayload":{"workspaceName":"live","nodeAggregateId":"lady-eleonode-rootford","nodeTypeName":"Neos.ContentRepository:Root","tetheredDescendantNodeAggregateIds":[]},"initiatingUserId":"system","initiatingTimestamp":"random-time"}}
{"identifier":"random-event-uuid","type":"NodeAggregateWithNodeWasCreated","payload":{"workspaceName":"live","contentStreamId":"cs-identifier","nodeAggregateId":"nody-mc-nodeface","nodeTypeName":"Neos.ContentRepository.Testing:Document","originDimensionSpacePoint":{"language":"de"},"succeedingSiblingsForCoverage":[{"dimensionSpacePoint":{"language":"de"},"nodeAggregateId":null},{"dimensionSpacePoint":{"language":"gsw"},"nodeAggregateId":null},{"dimensionSpacePoint":{"language":"fr"},"nodeAggregateId":null}],"parentNodeAggregateId":"lady-eleonode-rootford","nodeName":"child-document","initialPropertyValues":[],"nodeAggregateClassification":"regular","nodeReferences":[]},"metadata":{"initiatingTimestamp":"random-time"}}
{"identifier":"random-event-uuid","type":"RootNodeAggregateWithNodeWasCreated","payload":{"nodeAggregateId":"lady-eleonode-rootford","nodeTypeName":"Neos.ContentRepository:Root","coveredDimensionSpacePoints":[{"language":"de"},{"language":"gsw"},{"language":"fr"}],"nodeAggregateClassification":"root"},"metadata":{"commandClass":"Neos\\ContentRepository\\Core\\Feature\\RootNodeCreation\\Command\\CreateRootNodeAggregateWithNode","commandPayload":{"workspaceName":"live","nodeAggregateId":"lady-eleonode-rootford","nodeTypeName":"Neos.ContentRepository:Root","tetheredDescendantNodeAggregateIds":[]},"initiatingUserId":"system","initiatingTimestamp":"random-time"}}
{"identifier":"random-event-uuid","type":"NodeAggregateWithNodeWasCreated","payload":{"nodeAggregateId":"nody-mc-nodeface","nodeTypeName":"Neos.ContentRepository.Testing:Document","originDimensionSpacePoint":{"language":"de"},"succeedingSiblingsForCoverage":[{"dimensionSpacePoint":{"language":"de"},"nodeAggregateId":null},{"dimensionSpacePoint":{"language":"gsw"},"nodeAggregateId":null},{"dimensionSpacePoint":{"language":"fr"},"nodeAggregateId":null}],"parentNodeAggregateId":"lady-eleonode-rootford","nodeName":"child-document","initialPropertyValues":[],"nodeAggregateClassification":"regular","nodeReferences":[]},"metadata":{"initiatingTimestamp":"random-time"}}

"""
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
@contentrepository
Feature: As a user of the CR I want to export the event stream
Feature: As a user of the CR I want to import events using the EventStoreImportProcessor

Background:
Given using no content dimensions
Expand All @@ -11,56 +11,59 @@ Feature: As a user of the CR I want to export the event stream
"""
And using identifier "default", I define a content repository
And I am in content repository "default"
And the command CreateRootWorkspace is executed with payload:
| Key | Value |
| workspaceName | "live" |
| newContentStreamId | "cs-identifier" |
And I am in workspace "live"

Scenario: Import the event stream into a specific content stream
Then I expect exactly 0 events to be published on stream with prefix "ContentStream:cs-identifier"
Given using the following events.jsonl:
"""
{"identifier":"9f64c281-e5b0-48d9-900b-288a8faf92a9","type":"RootNodeAggregateWithNodeWasCreated","payload":{"workspaceName":"workspace-name","contentStreamId":"cs-imported-identifier","nodeAggregateId":"acme-site-sites","nodeTypeName":"Neos.Neos:Sites","coveredDimensionSpacePoints":[[]],"nodeAggregateClassification":"root"},"metadata":[]}
{"identifier":"1640ebbf-7ffe-4526-b0f4-7575cefabfab","type":"NodeAggregateWithNodeWasCreated","payload":{"workspaceName":"workspace-name","contentStreamId":"cs-imported-identifier","nodeAggregateId":"acme-site","nodeTypeName":"Vendor.Site:HomePage","originDimensionSpacePoint":[],"succeedingSiblingsForCoverage":[{"dimensionSpacePoint":[],"nodeAggregateId":null}],"parentNodeAggregateId":"acme-site-sites","nodeName":"acme-site","initialPropertyValues":{"title":{"value":"My Site","type":"string"},"uriPathSegment":{"value":"my-site","type":"string"}},"nodeAggregateClassification":"regular"},"metadata":[]}
"""
And I import the events.jsonl into "cs-identifier"
And I import the events.jsonl into workspace "live"
Then I expect exactly 3 events to be published on stream with prefix "ContentStream:cs-identifier"
And event at index 0 is of type "ContentStreamWasCreated" with payload:
| Key | Expected |
| contentStreamId | "cs-identifier" |
And event at index 1 is of type "RootNodeAggregateWithNodeWasCreated" with payload:
| Key | Expected |
| workspaceName | "workspace-name" |
| workspaceName | "live" |
| contentStreamId | "cs-identifier" |
| nodeAggregateId | "acme-site-sites" |
| nodeTypeName | "Neos.Neos:Sites" |
And event at index 2 is of type "NodeAggregateWithNodeWasCreated" with payload:
| Key | Expected |
| workspaceName | "workspace-name" |
| workspaceName | "live" |
| contentStreamId | "cs-identifier" |
| nodeAggregateId | "acme-site" |
| nodeTypeName | "Vendor.Site:HomePage" |

Scenario: Import the event stream
Then I expect exactly 0 events to be published on stream with prefix "ContentStream:cs-imported-identifier"
Given using the following events.jsonl:
"""
{"identifier":"9f64c281-e5b0-48d9-900b-288a8faf92a9","type":"RootNodeAggregateWithNodeWasCreated","payload":{"workspaceName":"workspace-name","contentStreamId":"cs-imported-identifier","nodeAggregateId":"acme-site-sites","nodeTypeName":"Neos.Neos:Sites","coveredDimensionSpacePoints":[[]],"nodeAggregateClassification":"root"},"metadata":[]}
{"identifier":"1640ebbf-7ffe-4526-b0f4-7575cefabfab","type":"NodeAggregateWithNodeWasCreated","payload":{"workspaceName":"workspace-name","contentStreamId":"cs-imported-identifier","nodeAggregateId":"acme-site","nodeTypeName":"Vendor.Site:HomePage","originDimensionSpacePoint":[],"succeedingSiblingsForCoverage":[{"dimensionSpacePoint":[],"nodeAggregateId":null}],"parentNodeAggregateId":"acme-site-sites","nodeName":"acme-site","initialPropertyValues":{"title":{"value":"My Site","type":"string"},"uriPathSegment":{"value":"my-site","type":"string"}},"nodeAggregateClassification":"regular"},"metadata":[]}
"""
And I import the events.jsonl
Then I expect exactly 3 events to be published on stream with prefix "ContentStream:cs-imported-identifier"
Then I expect exactly 3 events to be published on stream with prefix "ContentStream:cs-identifier"
And event at index 0 is of type "ContentStreamWasCreated" with payload:
| Key | Expected |
| contentStreamId | "cs-imported-identifier" |
| Key | Expected |
| contentStreamId | "cs-identifier" |
And event at index 1 is of type "RootNodeAggregateWithNodeWasCreated" with payload:
| Key | Expected |
| workspaceName | "workspace-name" |
| contentStreamId | "cs-imported-identifier" |
| nodeAggregateId | "acme-site-sites" |
| nodeTypeName | "Neos.Neos:Sites" |
| Key | Expected |
| workspaceName | "live" |
| contentStreamId | "cs-identifier" |
| nodeAggregateId | "acme-site-sites" |
| nodeTypeName | "Neos.Neos:Sites" |
And event at index 2 is of type "NodeAggregateWithNodeWasCreated" with payload:
| Key | Expected |
| workspaceName | "workspace-name" |
| contentStreamId | "cs-imported-identifier" |
| nodeAggregateId | "acme-site" |
| nodeTypeName | "Vendor.Site:HomePage" |
| Key | Expected |
| workspaceName | "live" |
| contentStreamId | "cs-identifier" |
| nodeAggregateId | "acme-site" |
| nodeTypeName | "Vendor.Site:HomePage" |

Scenario: Import faulty event stream with explicit "ContentStreamWasCreated" does not duplicate content-stream
see issue https://github.com/neos/neos-development-collection/issues/4298
Expand All @@ -73,7 +76,7 @@ Feature: As a user of the CR I want to export the event stream
"""
And I import the events.jsonl

And I expect a MigrationError with the message
And I expect a migration exception with the message
"""
Failed to read events. ContentStreamWasCreated is not expected in imported event stream.
"""
Expand Down
5 changes: 4 additions & 1 deletion Neos.ContentRepository.Export/src/Asset/AssetExporter.php
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
<?php

declare(strict_types=1);

namespace Neos\ContentRepository\Export\Asset;

use League\Flysystem\Filesystem;
Expand All @@ -14,7 +16,8 @@ public function __construct(
private readonly Filesystem $files,
private readonly AssetLoaderInterface $assetLoader,
private readonly ResourceLoaderInterface $resourceLoader,
) {}
) {
}
bwaidelich marked this conversation as resolved.
Show resolved Hide resolved

public function exportAsset(string $assetId): void
{
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -24,10 +24,13 @@ public function __construct(

public static function fromRawEvent(Event $event): self
{
$payload = \json_decode($event->data->value, true, 512, JSON_THROW_ON_ERROR);
// unset content stream id as this is overwritten during import
unset($payload['contentStreamId'], $payload['workspaceName']);
Comment on lines +28 to +29
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

at some point i think we should introduce a utility to manifest this stuff :D we do it at a lot of places to assume the properties are named that way ^^

return new self(
$event->id->value,
$event->type->value,
\json_decode($event->data->value, true, 512, JSON_THROW_ON_ERROR),
$payload,
$event->metadata?->value ?? [],
);
}
Expand Down
63 changes: 0 additions & 63 deletions Neos.ContentRepository.Export/src/ExportService.php

This file was deleted.

39 changes: 0 additions & 39 deletions Neos.ContentRepository.Export/src/ExportServiceFactory.php

This file was deleted.

Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
<?php

declare(strict_types=1);

namespace Neos\ContentRepository\Export\Factory;

use Neos\ContentRepository\Core\Factory\ContentRepositoryServiceFactoryDependencies;
use Neos\ContentRepository\Core\Factory\ContentRepositoryServiceFactoryInterface;
use Neos\ContentRepository\Core\SharedModel\Workspace\ContentStreamId;
use Neos\ContentRepository\Export\Processors\EventExportProcessor;

/**
* @implements ContentRepositoryServiceFactoryInterface<EventExportProcessor>
*/
final readonly class EventExportProcessorFactory implements ContentRepositoryServiceFactoryInterface
{
public function __construct(
private ContentStreamId $contentStreamId,
) {
}

public function build(ContentRepositoryServiceFactoryDependencies $serviceFactoryDependencies): EventExportProcessor
{
return new EventExportProcessor(
$this->contentStreamId,
$serviceFactoryDependencies->eventStore,
);
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
<?php

declare(strict_types=1);

namespace Neos\ContentRepository\Export\Factory;

use Neos\ContentRepository\Core\Factory\ContentRepositoryServiceFactoryDependencies;
use Neos\ContentRepository\Core\Factory\ContentRepositoryServiceFactoryInterface;
use Neos\ContentRepository\Core\SharedModel\Workspace\WorkspaceName;
use Neos\ContentRepository\Export\Processors\EventStoreImportProcessor;

/**
* @implements ContentRepositoryServiceFactoryInterface<EventStoreImportProcessor>
*/
final readonly class EventStoreImportProcessorFactory implements ContentRepositoryServiceFactoryInterface
{
public function __construct(
private WorkspaceName $targetWorkspaceName,
private bool $keepEventIds,
) {
}

public function build(ContentRepositoryServiceFactoryDependencies $serviceFactoryDependencies): EventStoreImportProcessor
{
return new EventStoreImportProcessor(
$this->targetWorkspaceName,
$this->keepEventIds,
$serviceFactoryDependencies->eventStore,
$serviceFactoryDependencies->eventNormalizer,
$serviceFactoryDependencies->contentRepository,
);
}
}
Loading
Loading