Blending ORM and ODM but keeping the Domain Model clean

[webdevilopers]

When trying to use Doctrine ORM Entities with ODM Documents (e.g. MongoDB) most solutions adding extra fields:

• http://doctrine-orm.readthedocs.org/projects/doctrine-mongodb-odm/en/latest/cookbook/blending-orm-and-mongodb-odm.html @jmikola
• http://de.slideshare.net/spf13/blending-mongodb-and-rdbms-for-ecommerce by @jwage
• Add XML mapping example for References doctrine-extensions/DoctrineExtensions#1564 (comment)

For me this feels like "polluting" the Domain Model with extra properties to fit the infrastructure.
So I am currently looking for a way to keep the Domain Models clean:

• ODM double properties for ORM mapping doctrine-extensions/DoctrineExtensions#750 (comment)
• https://twitter.com/webdevilopers/status/720289162142744576

A possible workaround could be a custom mapper:

• https://github.com/akeneo/pim-community-dev/blob/master/src/Akeneo/Bundle/StorageUtilsBundle/Doctrine/MongoDBODM/Types/Entities.php by @nidup

A different solution could be treating referenced entities as Value Objects which at the same time could serve as a historical "query database".

For instance:
Instead of linking an ODM Order Document (still an "Entity" in DDD though) with an ORM Customer you introduce a Customer Value Object that keeps Id and full name. Maybe the delivery and / or invoice address too.

Remember kids:

Implementation issues (such as persistence) are not dealt with in the model, but they must be in the design.” - @ericevans0
https://twitter.com/fromddd/status/725604978564308992

[webdevilopers]

Regarding your mapping suggestion @nidup:

• https://github.com/akeneo/pim-community-dev/blob/master/src/Akeneo/Bundle/StorageUtilsBundle/Doctrine/MongoDBODM/Types/Entities.php

I'm trying the other way around:
Order Document has one Customer Entity.

So I need an extra "Document(s)" Mapping type, right? Do you have a (XML?) mapping example from your use case?

[nidup]

Hi @webdevilopers

Thank you for opening this discussion and for sharing these links!

Pretty sure that @jjanvier will be very interested by this issue too, he very well know the topic and he's preparing a talk about our hybrid storage for a sfpot this week http://www.meetup.com/fr-FR/afsy-sfpot/events/230197553/ ;)

To give a bit more details, we use a hybrid storage in the context of Akeneo Product Information Management system (https://www.akeneo.com/).

Depending on the amount of product data, we can use,

• a full Doctrine ORM storage with EAV implementation for product data
• a hybrid storage with Doctrine ORM for standard entities (family, category, attribute) + MongoDB ODM for the product data

When we implemented the hybrid MongoDB storage, the difficulty was to,

• keep the same business code for full ORM and for hybrid storage ORM + ODM (we use several interfaces for querying and persistence)
• make "transparent" the link between ORM entities and ODM documents for the dev who's working on features

To do so we introduced a StorageUtilsBundle and we configure the storage to be used like this https://github.com/akeneo/pim-community-dev/blob/master/app/config/pim_parameters.yml#L35

Let's take the product MongoDB mapping https://github.com/akeneo/pim-community-dev/blob/master/src/Pim/Bundle/CatalogBundle/Resources/config/model/doctrine/Product.mongodb.yml

This product document is linked to a single family (managed by Doctrine ORM),

        family:
            type: entity
            targetEntity: Pim\Component\Catalog\Model\FamilyInterface

It's also linked to several categories (managed by Doctrine ORM),

        categories:
            notSaved: true
            type: entities
            targetEntity: Pim\Component\Catalog\Model\CategoryInterface
            idsField: categoryIds

These new entity and entities MongoDB types are defined here https://github.com/akeneo/pim-community-dev/tree/master/src/Akeneo/Bundle/StorageUtilsBundle/Doctrine/MongoDBODM/Types

We store entity ids in the MongoDB document and we use MongoDB ODM events to convert these ids into lazy entities,
https://github.com/akeneo/pim-community-dev/tree/master/src/Akeneo/Bundle/StorageUtilsBundle/EventSubscriber/MongoDBODM

To build collection of lazy loaded referenced entities, we use https://github.com/akeneo/pim-community-dev/blob/master/src/Akeneo/Bundle/StorageUtilsBundle/Doctrine/MongoDBODM/Collections/ReferencedCollection.php

We also use the following listener to be able to use an interface resolved as a Doctrine ORM entity in a Doctrine MongoDB ODM mapping (for instance Pim\Component\Catalog\Model\FamilyInterface) https://github.com/akeneo/pim-community-dev/blob/master/src/Akeneo/Bundle/StorageUtilsBundle/EventListener/MongoDBODM/ResolveTargetEntityListener.php

The StorageUtilsBundle is already open source and not coupled to our PIM business code but unfortunately, for now, it's part of our main dev repository.

If after your first tests, you're interested to use it, don't hesitate to ping me, we can extract this bundle in a dedicated repository with a subtree split and register it on packagist to make it easily usable in others projects.

[webdevilopers]

Thanks @nidup for this detailled answer!

Looking at your mapping example:
https://github.com/akeneo/pim-community-dev/blob/master/src/Pim/Bundle/CatalogBundle/Resources/config/model/doctrine/Product.mongodb.yml#L58-64

So you still have an "extra Field / Collection" to keep the IDs that have to be converted in some way?

This is what I was concerned about from a Domain-Driven Design POV. Isn't the Infrastructure / Persistence having a little bit too much impact on the design of the DomainModel?

Coming from twitter discussion w/ @carlosbuenosvinos @eneko here:
https://twitter.com/webdevilopers/status/720289162142744576

[webdevilopers]

My current workaournd looks like this:

I store my Customer via ORM (MySQL). Orders are stored in ODM (MongoDB).
I have an customer property on my ODM Document. There is no extra customerId.
The customer property itself is an embedOne reference. It holds the historical data - I guess you could say "Aggregate" - typically:

• ID
• name
• billingAddress

The namespaces look like this:

• Acme\QualityManagement\DomainModel\Customer\Customer.php (CustomerId, CustomerRepository etc.)
• Acme\QualityManagement\DomainModel\Order\Order.php (OrderId, OrderRepository etc.)
• Acme\QualityManagement\DomainModel\Order\OrderCustomer.php
• Acme\QualityManagement\Infrastructure\Persistence\Doctrine\DoctrineCustomerRepository.php
• Acme\QualityManagement\Infrastructure\Persistence\Doctrine\MongoOrderRepository.php

I think the idea is similar to the "Aggregates and Event Sourcing A+ES" chapter of the red book by @VaughnVernon . With ORM an Event could update a Query (READ) Model that writes the historical data / Aggregate into a database VIEW.

Using embed references in ODM is great to take a final snapshot.

If I ever need more data from the Customer I would inject the CustomerRepository into my Query (READ) Model and get it from the ORM.

By storing the ID inside the reference I can still query for "OrdersByCustomer".

Actually I do have an extra column too but it does not feel like polluting the DomainModel for a persistence purpose.

Though I will still have to use custom mappers or types like @nidup did to convert those IDs into Entities when calling collections on my DomainModel e.g.:

class Customer
{
    private $orders; // some EXTRA-LAZY Doctrine ArrayCollection

    public function getOrder(OrderId $orderId)
    {
        return $this->orders[$orderId];
    }
}

What do you think of this approach?

[jjanvier]

Hello @webdevilopers

Maybe this presentation will interest you. https://speakerdeck.com/jjanvier/products-storage-at-akeneo-pim

It describes how we store our products at Akeneo PIM. At the end of the presentation, you have the most interesting part of the code that are used to link ORM entities to MongoDB documents.

[webdevilopers]

Thanks @jjanvier !

At the bottom line you are using an approach to the DoctrineExtensions by @Atlantic18:
https://github.com/Atlantic18/DoctrineExtensions/blob/master/doc/references.md

But you also support converting collections to Entities?

Still you have an extra Field e.g. categoryIds allthough you have categories already.
Not on the mapping only but also on your Document - the DomainModel - right?

This is what I am mainly concerned about from a DDD POV and that's why I asked @carlosbuenosvinos what he thinks about this approach. Maybe the other contributors @eddmann, @keyvanakbary and @theUniC have an opinion on this.

Technically I agree with your approach and would handle it the same way!

[webdevilopers]

One final way to solve this and keep the DomainModel "clean" is to create a DoctrineCustomer Entity or MongoOrder Document that adds the extra field.

[webdevilopers]

Currently looking at a CQRS post by @martinfowler:
http://martinfowler.com/bliki/CQRS.html

He states:

Developers typically build their own conceptual model which they use to manipulate the core elements of the model. If you're using a Domain Model, then this is usually the conceptual representation of the domain.
You typically also make the persistent storage as close to the conceptual model as you can.

If I regard my Domain Model being a representation I could build my Order Document this way instead:

class Order
{
    private $customerId; // Used to get the related Entity - just the ID or the CustomerId Value Object?
    private $customer; // The related Entity
    private $customerName; // snapshot / aggregate / historical data - bad example since `name` will mostly stay the same :)
    private $billingAddress; // snapshot / aggregate / historical data
}

[nidup]

Hi @webdevilopers!

I'm very interested in the DDD but I don't know it enough for now, I'm actively documenting on this topic :) (so this answer may contain some misuses of the terminology or weird approach 😜 ).

So you still have an "extra Field / Collection" to keep the IDs that have to be converted in some way?
This is what I was concerned about from a Domain-Driven Design POV. Isn't the Infrastructure / Persistence having a little bit too much impact on the design of the DomainModel?

Absolutely, I do agree with you, the categoryIds field is a persistence concern and should not pollute the domain model. When we wrote the original implementation (in 2014, times fly!), we tried to find a very technical solution about how to "build" and "use" this relation between Product Document and Category Entity without taking too much care about the impact on the domain models.

One of the biggest difficulty we had was to build an efficient and abstract way to do complex queries on these objects from the business layer POV. This point also pushed us to pollute the Product model by adding a "normalizedData" field to help us to write MongoDB queries https://github.com/akeneo/pim-community-dev/blob/master/src/Pim/Bundle/CatalogBundle/Resources/config/model/doctrine/Product.mongodb.yml#L78 (practice quite common when using a MongoDB storage to store the data in a way which simplifies the usage).

In a perfect world, imvho, you should not even care of the persistence of your Domain models when defining and coding them because persistence is a pure technical implementation detail. In real life, it's not that easy to decouple these objects from Doctrine ORM, I'm not even sure that's a good idea to try to directly store these domain models as entity/document.

One final way to solve this and keep the DomainModel "clean" is to create a DoctrineCustomer Entity or MongoOrder Document that adds the extra field.

I think this approach is really really interesting, your persistence classes can rely on these Entity/Document for their very specific concerns (fields used to keep a relation between storage, reference collection, etc) keeping a clean DomainModel.

Your business code could define and rely on a "repository" interface allowing to fetch domain models.
The implementation of this interface in infrastructure could internally fetch Doctrine document / entities but return Domain models only (never expose entity/document and taking care to convert / hydrate).

I recently did experiments around Hexagonal Architecture (Ports & Adapters) on a reporting project and achieved to easily decouple Domain from Persistence using a quite similar strategy. A DatabaseAdapter taking care of implementing querying and converting data from storage "objects" into Domain models (I didn't used Doctrine ORM for this project, I used https://github.com/pomm-project in fact only the foundation part + custom code for converting objects).

Thanks again for opening this discussion, I'm looking forward to see other opinions / ideas.

[webdevilopers]

I'm as excited as you are @nidup .

BTW just found this quote here by @vkhorikov:
If you tend to write code like in the sample below, you are on the wrong path <- Persistence Ignorance

• http://enterprisecraftsmanship.com/2014/11/29/entity-framework-6-7-vs-nhibernate-4-ddd-perspective/

;)

Not sure if the approach of a Database related Wrapper like DoctrineCustomer or MongoOrder can be compared to SQL Speaking Objects introduced by @yegor256:
http://www.yegor256.com/2014/12/01/orm-offensive-anti-pattern.html#sql-speaking-objects

Wrapper seem to be an appraoch indeed:
https://twitter.com/electricmaxxx/status/725346621005922304 @dbu @dantleech

[webdevilopers]

BTW the extra IDs are mentioned in the DDD in PHP book too. There they are used as surrogateIds.
These are not used by the Domain. But they are required by Doctrine to be able to persist Value Objects.

The book also suggests to using a Layer SuperType:
http://martinfowler.com/eaaCatalog/layerSupertype.html

Maybe the "polluting extra id" could be moved to an abstract class e.g. "HybridDomainObject" which then can be extended by the Order Domain Model.

[webdevilopers]

Nice article here btw:
http://scabl.blogspot.de/2015/03/aeddd-2.html

Most Java/Spring/JPA projects have separate repository and service layers. Ostensibly, the persistence logic and persistence concerns are encapsulated in the repositories, often called data access objects (DAOs). But in every JPA project I’ve worked on, this encapsulation fails, and the service layer is intimately involved with persistence concerns. It has to be aware of fetching strategies, cascades, and special handling of proxy objects. And it has to manage flushing the persistence cache, and understand which entities need to be reloaded into the cache before continuing after a flush.

In the end, there are many situations where we end up having to make compromising decisions within our domain to get the ORM mapping to work right.

[vkhorikov]

Hi, thanks for the invitation, @webdevilopers

If I understood correctly from the conversation, you suggest creating sub-classes to keep domain classes clean, smth like this: ClassWithPersistenceConcerns extends CleanDomainClass.

IMO, that's a good solution. In fact, that is exactly how ORMs like Hibernate approach the problem: they create sub-classes (proxies) for your domain classes at runtime and use them to handle persistence logic.

The only concern I have about this solution is complexity. Implementing such proxies manually can become a hurdle and one should weight pros and cons of this approach really carefully. It might be that purity you will get out of it doesn't worth the amount of work it requires. But this is worth trying anyway, at least as a research project.

BTW, I agree with your opinion on Ids in domain classes. Here I wrote a post on that: http://enterprisecraftsmanship.com/2016/03/08/link-to-an-aggregate-reference-or-id/

[webdevilopers]

Welcome @vkhorikov !

Are there Events and Event Subscribers in (N)Hibernate too? Do you have any experiences with ORM and ODM (OGM) hybrids in Java or .NET?

I really liked your article on this topic. Good to know I'm not the only one concerned about it! :)

Here is my current PHP approach with Doctrine:

As I described before in my use case I have a Customer Entity which has Document Orders.
The Doctrine Cookbook uses the extra CustomerId we were talking about:
http://doctrine-orm.readthedocs.org/projects/doctrine-mongodb-odm/en/latest/cookbook/blending-orm-and-mongodb-odm.html#event-subscriber

In my example I kept the property on the Order Document named $customer. The expected name for the DomainModel. Instead of being a placeholder I set here the actual ID of the Customer e.g. 969 when persisting.

Now I changed the Event Subscriber from the Cookbook to "convert" / "transform" this field into a Document Collection instead of using the extra field:
https://gist.github.com/webdevilopers/e65f795749c40601276f60a5a3517a85

This works fine so far.

It's just a rapid prototype. I will heavily test it. But it looks promising in the first place.

[webdevilopers]

I've improved the ORM listener and added an ODM listener to get a referenced Entity Supplier which is stored as an Id inside the Order Document with the expected Domain Model property named $supplier.

So far everything seems to work fine without any extra ID.

What do you think? Any bad practice using Doctrine Events this way? Any Hibernate experts here? @thmuch and @brmeyer maybe? :)

[webdevilopers]

Actually it looks like it is very similar in @hibernate:
http://stackoverflow.com/a/11672377/1937050

No wonder since / that @doctrine was heavily inspired by it.

[vkhorikov]

@webdevilopers Yes, the concept of events and event listeners is the same in Hibernate and NHibernate.

The code looks good to me. I don't have experience in PHP, though, so take my words with caution. The thing I think you need to verify is how your solution works with deep class graphs. You need to see if it would load the full object graph at once when you fetch a single entity or would load them lazily, just-in-time, as you traverse the graph.

[webdevilopers]

Definitely @vkhorikov ! Lazy loading looks fine so far.

Currently these listeners run on all ORM Entities and ODM Documents but can be catched.

@doctrine ORM offers a listener that can be mapped to the Entity directly via XML for instance:
http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/events.html#entity-listeners

I'm still looking for an equivalent for ODM Documents:
https://twitter.com/webdevilopers/status/724872011638562816

[webdevilopers]

Just discovered this project by @ElectricMaxxx:
https://github.com/ElectricMaxxx/DoctrineOrmOdmAdapter

Maybe he can give a short comment to our discussion too.

[webdevilopers]

@ElectricMaxxx Can you tell us if your solution
https://github.com/ElectricMaxxx/DoctrineOrmOdmAdapter
uses an extra field too?

Also discussed here:
doctrine-extensions/DoctrineExtensions#750

[ElectricMaxxx]

No. There is no extra Field. My solution tries to connect entitys/documents as One-to-One relation, that live in different doctrines. The reference is a pure configuration on the entity/document very close to the configuration we used to do in every doctrine. And there is a configuration to know the manager on the oposite side. So if a entity with an referenced document is loaded the document will be added as an Proxy and will awake on an get*() call. The information about the loading process comes by an simple event hook. I tried to do it all by hooks, but that fails as every find() in one manager triggers a find() on the other side. So you can see my Solution as a kind of LazyLoading for cross doctrine references. I tried it in production for ORM <-> PHPCR and it works fine. Tests are working too on both. And it should work on each doctrine that behaves like the ORM.

[webdevilopers]

Thanks for the update on your approach.

Though @stof stated that using the same field is not a clean design I prefer this solution from a DDD DomainModel POV - at least if you have to really blend Entities w/ Documents:
doctrine-extensions/DoctrineExtensions#750 (comment)

Personally I use the same workaround @stof mentioned here and create an extra OrderCustomer aggregate:
doctrine-extensions/DoctrineExtensions#750 (comment)

Still tried to check out the Doctrine References Extension but couldn't make the XML mapping working:
Feel free to comment on this separate issue: #9

[webdevilopers]

There is an interesting article by @mathiasverraes:

• http://verraes.net/2013/12/related-entities-vs-child-entities/

In "Persistence" he needs an "annoying" extra-column too:

Before, the Task didn’t know about the Project it belonged to. Now it does, which is a slightly annoying trade-off. Just be aware that we don’t really want a bidirectional relation here. We could give Task a reference to Project instead of just the projectId, and let the ORM work it out.

In our database schema, Task’s primary key is a composite of projectId and taskId. If the ORM or the infrastructure doesn’t deal well with composite ids that are also foreign keys, we could make taskId the only primary key, and make it unique for all Tasks. Again, conceptually, this is a small trade-off, with benefits and drawbacks.

[webdevilopers]

I would love to hear @Ocramius opinion on our approaches e.g. without an extra field and from a DDD POV. Since I've seen him on the Php Friends Of DDD group too:
https://github.com/orgs/PhpFriendsOfDdd/people

He is familiar with @doctrine best practices and he knows its limits well.

Would you lend us an ear Marco?

[VaughnVernon]

Hi guys,

I haven't been following this conversation closely enough, but I will just
chime in based on what I perceive this thread to be about. Sometimes you
need to use persistence specific fields/columns to marry you domain model
to the database. You really shouldn't worry about it too much unless you
have no way to hide these details from clients of the domain model. Is that
your concern?

Unfortunately don't know enough about PHP, but if you can use private
fields and still get Doctrine to map them to/from columns, I suggest just
doing so and moving on. There must be plenty more interesting business
problems to solve that this topic is diverting you away from :)

Best,
Vaughn
On May 8, 2016 2:15 PM, "webDEVILopers" [email protected] wrote:

I would love to hear @Ocramius https://github.com/Ocramius opinion on
our approaches e.g. without an extra field and from a DDD POV. Since I've
seen him on the Php Friends Of DDD group too:
https://github.com/orgs/PhpFriendsOfDdd/people

He is familiar with @doctrine https://github.com/doctrine best
practices and he knows its limits well.

Would you lend us an ear Marco?

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#2 (comment)

[Ocramius]

@webdevilopers fields are not a big deal, in my opinion. As long as the public API stays clean, you are safe from complications.

As @VaughnVernon says, you can just hide it as a detail.

It's all about the greater good

[webdevilopers]

Did @VaughnVernon himself just comment on our issue? :)

Actually I feel a little bit relieved. Indeed there are more interesting "problems"!

"Marry" is a good term here since a marriage can be some kind of compromise sometimes too.

And thank you very much for @Ocramius confirming this approach for @doctrine.

Normally these extra fields are just the IDs of the related Entity e.g. an integer or UUID. And normally this ID would be a Value Object like CustomerId.
From a final DDD POV would you set the integer resp UUID on the extra column or the Value Object?
Guess it depends on the ORM and if it is able to read the VO, right?

[Ocramius]

I'm not worried about the internal state. Again, focus on the public API.

You can either map a VO or just an integer and then convert back/forth to VO inside the API