Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Generating Migrations file from Models definitions and changes #51

Open
sgaseretto opened this issue Mar 6, 2022 · 9 comments
Open

Comments

@sgaseretto
Copy link

Today you need to specify all the columns from a table in the migrations file and then write them all again in the respective model file. Basically you have to repeat yourself.
Since there isn't yet an official documentation from SearchLight, I was wondering if there is a way to detect the changes from the model definition and generating the migrations file from it, instead of having to repeat yourself two times (also, making the process less prompt to human errors).
This feature will be similar to what Alembic does to generate migrations for SQLAlchemy in python, it sees the changes in a model and generates a migration file only with the changes made to the model.

@FrankUrbach
Copy link
Contributor

You are right. But the nature of SearchLight is to support different databases. If you want create things like that a foundation of what you are searching isn't that hard. But you have to change the files to get the full potential of the migrations (create an index e.g.).
I started something similar in the past. But before we start to implement this I would discuss this with Adrian (@essenciary). To get it working in all databases SearchLight support today this will take a little time to implement it and I don't want waste my time if Adrian don't want to get this in the package.

@sgaseretto
Copy link
Author

Thanks for the rapid response! In my point of view, this feature will be a very nice productivity booster for Genie.jl developers!

@sgaseretto
Copy link
Author

@FrankUrbach this functionality got me thinking all day. Since you tried to implement something like this in the past, how do you recommend me to start it? I was thinking on building something like a new package (SearchLightMigrations) that works on top of SearchLight.Migration to try the idea of generating the migrations files from the model file, but since I'm new to Julia (also I've always used this functionality from other libraries without digging into the library to see how it was implemented) I was wondering what are some suggestions you could give me to start implementing this. Maybe it will start as a separate Package and then, once it becomes more mature (with support for all the DBs supported by SearchLight), it gets merged to SearchLight (or as an alternative while your team implements a better way to do this)

@FrankUrbach
Copy link
Contributor

@sgaseretto the easiest way to do something like that is to use the reflection capabilities of Julia. For each struct you can get the field names and the types of the fields. You can make a dictionary with types you want accept for the database and as value the translation into the string you have to use for the description in the migration file. Then you use a template file where you can file in the description you have generated. But you will stuck by adding indexes or something else as the fields itself because you haven't any information inside a struct which field you want preparing as an in Index field. The rest should be possible to do. If you want start such a package let me know where it is and I can help you if you stuck somewhere.

@sgaseretto
Copy link
Author

You read my mind! That was what I was trying today, building a dict using Model.name.names and Model.types to use them to start generating the migration files. You are right about the indexes, I was thinking on building custom structs like Fields to add if they have to be indexes or things like limit length (when they are strings) like in Django, and then when getting the model struct, see which fields need indexes, but I don't even know yet if something like that is achievable (I'm coming from OOP, so I don't know how this can be done with Julia at the moment, but I believe that probably yes). Another crucial functionality that I'll experiment with is how to know the differences between a current Table and the Model changes, to determine which columns where added and/or deleted when generating new migration files. I'll continue with some experiments and then I'll set up a repo to share it, I believe it will have to be following this instructions.

Also, I believe that at least with the basic functionalities to automatically generate the migrations files will be sufficient, then for some advance configurations, developers will have to manually edit the migrations files. Something similar happens with Alembic, when trying to autogenerate the migrations files, some things are detected by the library when trying to determine the migrations, and some others cannot be detected

@sgaseretto
Copy link
Author

@FrankUrbach do you know if there is a correct way (if posible) to try and test SearchLight in a standalone way with the REPL or in Jupyter, the same way it can be done with Genie? I want to test some things but having to build a whole Genie Project to try somethings feels a little bit awkward sometimes.

@FrankUrbach
Copy link
Contributor

@sgaseretto It isn't necessary to set up a whole Genie project. Make a project add SearchLight and the Adapter you need for your database and try what you want. It should work without Genie. E.g. see the tests of the Postgresql and the MySQL adapters. They work without one line of Genie code.

@sgaseretto
Copy link
Author

Thanks a lot! I'll check them

@sgaseretto
Copy link
Author

I don't know if I'm overcomplicating things, but the only idea I have is to specify in the Model definition if indexes are required or not by defining custom structs like DbId, for example:

  mutable struct StringDbType
      value::String
      limit::Int = 0 # When parsing this, 0 will mean that there is no limit specified
      index::Bool = True
  end

The drawback I see is that then the property that has this "Type" can't be manipulated as String, unless we overload all the operations that can be done to a String for this type... But probably there can be an easier way, but I'm not seeing it, what do you think @FrankUrbach ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants