Composing services

[iamcalledrob]

Hi there. I've been using the "wtfdial" approach for a while, and have really been enjoying it for how well it encourages layering and discourages spaghetti.

One area that's less obvious is how to elegantly compose services together.

For example, imagine we want to add a NewsFeedStoryService:

type NewsFeedStoryService interface {
  // Fetches NewsFeedStories for the user authenticated in ctx
  GetNewsFeedStories(ctx context.Context) ([]*NewsFeedStory, error)
}

An implementation of which "composes" together a bunch of underlying services:

type NewsFeedServiceImplementation struct {
  PostsService wtf.PostsService
  PhotosService wtf.PhotosService
}

func (s *NewsFeedServiceImplementation) GetNewsFeedStories(ctx context.Context) ([]*NewsFeedStory, error) {
  posts, _ := s.PostsService.GetPosts(ctx, wtf.PostsFilter{ ... })
  photos, _ := s.PhotosService.GetPhotos(ctx, wtf.PhotosFilter{ ... })

  results := make([]*NewsFeedStory, 0)
  // Logic to iterate over posts and photos to create stories

  return results, nil
}

I see two ways this could be implemented:

1. As "domain logic", at the module root.
This glue code is canonical logic for how to combine posts and photos into a news feed. The interface would still be needed in order to be able to substitute the real version for a mock version when testing a higher level in the stack, e.g. the http layer.

The main trade-offs I see:

• There's a need for both an interface and a struct at the domain level, so naming isn't obvious.
• This canonical logic mustn't call into other packages to avoid an import cycle, and must have all its project dependencies be specified by domain types. This can be a problem with tests, e.g. perhaps we want to write a more integration-like test that will use sqlite.PostsService plus inmem.PhotosService rather than mocks. These tests would have to be pushed into a different package which moves the test code much further away from the implementation.

2. As an additional "layer" in a separate package
This glue code could also be considered a "layer", inside a different package. Theoretically you could have multiple totally different implementations of the NewsFeedService logic if needed, e.g. for multivariate testing -- but in our scenario we don't need that.

The trade-off here is that this "layer" doesn't obviously fit into the existing structure, e.g. "http", "sqlite", "inmem" etc. It's initially written as a default, canonical implementation.

• You could create package newsfeed and newsfeed.Service{}, but this feels like arbitrary package creation, and doesn't fit into the layered approach.
• You could create package composed and composed.NewsFeedService{}, but this "composed" glue layer is totally arbitrary -- and do you include totally unrelated glue logic there too as you compose more services, e.g. composed.WeatherService{} etc...?

I'm curious if anyone has solved this well, and if there are other more subtle trade-offs?

[heyjorgedev]

@iamcalledrob Struggling with the same kind of decision as you are describing, did you end up deciding what to do?

[iamcalledrob]

I ended up organizing "glue" functions and structs at the domain level, and it's been working well. Behaviour of the glue logic can then be tested using mocked dependencies. Where possible I keep this glue to be a single function rather than a struct. The line of thinking here is that this is "domain logic" that has a single, canonical implementation -- just like the domain types.

Here's a simple real-world example of a glue function that connects two services together:

func AddNewUsersToList(userService foo.UserService, listService foo.ListService) {
    userService.OnNewUser(func(user *foo.User) {
        _ = listService.AddSubscriber(user.Email)
    })
}

Here's a minimal example of a glue function that composes search results from two services, showing how common logic can be used regardless of the underlying services.

func SearchResults(ctx context.Context, query string, userService foo.UserService, photoService foo.PhotoService) (results []any, err error) {
    if len(query) < 3 {
        err = errors.New("query too short")
        return
    }
    users, _ := userService.FindUsers(ctx, foo.UserFilter{ NameContains: query })
    photos, _ := photoService.FindPhotos(ctx, foo.PhotoFilter { HasTag: query })
    for _, u := range users {
        results = append(results, u)
    }
    for _, p := range photos {
        results = append(results, p)
    }
}

Perhaps userService is a sqlite.UserService right now, but later it becomes a redis.UserService etc. And for testing, it'll be a mock.UserService.

Lastly, for more complicated functionality where a common set of dependencies is needed across several related functions, I'm using structs to hold those dependencies For example (this is not real code, but should show the design):

type RegistrationService struct {
    userService foo.UserService
    otpService foo.OtpService
    mailService foo.MailService
}

func (r *RegistrationService) SendOTP(ctx context.Context, email string) error {
    _, err := r.userService.GetUserByEmail(email)
    if err != nil && !errors.Is(err, foo.ErrNotFound) {
        return fmt.Errorf("checking for existence of user by email: %w", err)
    }
    if err == nil {
        return fmt.Errorf("user already exists", err)
    }

    // Function that does some work then in-turn calls in to r.otpService
    // If this work wasn't re-used, CreateAndInsertOTP could be inlined here.
    var otp [6]byte
    otp, err = CreateAndInsertOTP(ctx, r.otpService, email)
    if err != nil {
        return fmt.Errorf("creating and inserting otp: %w", err)
    }

    err = r.mailService.SendMail(ctx, email, "Your otp is " + string(otp))
    if err != nil {
        return fmt.Errorf("sending mail: %w", err)
    }
    
    return nil
}

func (r *RegistrationService) Register(ctx context.Context, otp [6]byte, email string, name string) (*foo.User, error) {
    err := r.otpService.ConsumeOTP(ctx, otp, email)
    if err != nil {
        return nil, fmt.Errorf("consuming otp: %w", err)
    }
    
    user := &foo.User{Name: name, Email: email}
    err = r.userService.InsertUser(ctx, user)
    if err != nil {
        return nil, fmt.Errorf("inserting user: %w", err)
    }

    err = r.mailService.SendMail(ctx, email, "Thanks for registering!")
    if err != nil {
        return nil, fmt.Errorf("sending mail: %w", err)
    }

    return user, nil
}

// CreateAndInsertOTP securely generates an OTP for the provided email and inserts it using the OtpService
func CreateAndInsertOTP(ctx context.Context, otpService foo.OtpService, email string) ([6]byte, error) {
    otp, err := cryptopkg.GenerateSomethingSecurely(6)
    if err != nil {
        return nil, fmt.Errorf("generating secure bytes: %w", err)
    }

    err = otpService.InsertOTP(ctx, otp, email)
    if err != nil {
        return nil, fmt.Errorf("inserting otp: %w", err)
    }

    return otp, nil
}

This "RegistrationService" can then be tested using mocked dependencies, e.g. to ensure that it won't let a user register twice.

There are a few ergonomic trade-offs with this approach which I don't have a great solution to. Mainly, it breaks the pattern that ***Service is an interface which can be mocked and have multiple implementations. You could split this example into type RegistrationService interface { ... } and type RegistrationServiceImpl struct { ... }, but that feels quite un-idiomatic in Go.

In the real world, I haven't found a strong need to mock this logic.

Hope this is helpful!

[itaranto]

In the real world, I haven't found a strong need to mock this logic.

What about in the HTTP handlers unit tests?

[iamcalledrob]

I have created interfaces for this purpose when needed. I just don't start there until the need arises.

[mrh-chain]

Don't you run into circular dependencies here? When you place logic into the domain that needs testing, and those tests imports your mocks, the mocks will import the domain because they... well implement the interfaces in the domain.

[itaranto]

You can use package domain_test.

[MichalMoravik]

Hey!

I think your question is very similar to this already answered one:
#50

Or am I wrong?

[itaranto]

@iamcalledrob I think I fully agree with your findings here.

What I dislike about the "extra-package" approach you mentioned at the beggining, is that domain logic/definitions are now spread over two packages: composed and domain.

I'd prefer putting all domain related things into domain, but as you mentioned in your second comment, you have to come up with a weird names for the "cannonical" service struct like FooServiceImpl which is very ugly and Java-like. Maybe there could be better names for this, but still.

On the other hand, having everything as plain functions can be inconvenient as I cannot group data into structs. I also tend to have two interfaces per "domain entity", like domain.FooService and domain.FooRepository. Usually FooService is a superset of FooRepository:

package domain

type FooService interface {
    FooRepository
    // Some extra methods here
}

type FooRepository interface {
    GetFoo(id string) (*Foo, error)
    // ...
}

Another alternative to FooServiceImpl would be to make it private (fooService) and return a FooService interface in the constructor function. But this could be limiting as in Go you tend to expose structs as structs rather than interfaces.

I'd like to hear your thoughts about this.