Initialize modesolver with explicit eigenmode "guess"

[gvnwst]

Problem

When solving for modes in complex dielectric structures, it's common to find a large number of eigenmodes whose eigenvalues are very close together but whose spatial distributions vary significantly.
Typically the user cares about modes whose energy density is localized in a certain region (the waveguide core) and so these wacky modes are irrelevant.

Example case

Take a CMOS photonics processes with alternating silicon nitride and SiO2 in the back-end stack.
These materials support a large number of slab modes.
If the effective indices of the slab modes are near the effective index of the target mode the eigensolver will start just spitting out all sorts of high-order slab modes.
Accounting for the influence of these layers on dispersion and field distribution can be important (for dispersion, mode hybridization, etc) so one cannot simply assume a uniform environment.

Possible solution

Filtering modes is one approach which could help suppress these undesired modes, and is discussed in #1391 without an obvious solution.

Another possibility is to pre-supply an eigenmode whose spatial distribution is similar to that expected by the user.
At the moment the initial guess is a randomly-generated vector.
Exposing the set_initial_vec function inside solver.py to the primary solver interface, and enabling the user to pass custom eigenvector guesses, would be straightforward and allow user-implemented approaches to guide the modesolver to a specific result.

What would be extra-cool is if the user could specify a set of mode guesses. Say I want the q-TE00 mode, then the q-TM10, then the q-TE30 and q-TM30. I could provide initial guesses for each of these four. This could be used two ways:

1. Under the hood, a separate simulation is run for each guess, solving for $n$ modes (so each desired mode gets $n$ "close" possibilities) for a total of $4n$ solved modes.
2. The guesses are used to pre-populate the eigenvector initialization matrix used by the iterative solver (making some assumptions here). After solving for the first mode, the solver then uses the solved effective index and the already existing eigenmode guess to work on the second mode. This is nominally equivalent to initializing 4 simulations solving for $n=1$ modes each, sequentially, using the effective index from each previous simulation for the next. Doing it internally to the eigensolver is probably a lot faster/less memory intensive.
   It's not quite the same as how iterative solvers normally search since the user-specified modes are not NECESSARILY given in terms of decreasing eigenvalue but having the flexibility could be really useful in some cases.

[momchil-flex]

I think exposing the initial vector makes sense and I can see how it can be quite useful in some cases. I think the main thing to consider here in practice is how exactly the user would supply this. We probably shouldn't expect them to provide it directly in the shape of the initial vector as used in the mode solver, and not even in the shape of the E fields that are used (which we can easily reshape to the initial vector), because even that is not that convenient (they have to construct the fields exactly on the mode grid). Probably the best would be to provide a DataArray for the fields and then we just interpolate to the locations needed for the initial vector. There's no fundamental issue with this, just implementation.

I'm assigning a few people here, there's a lot going on right now so we won't be able to get to this fast, but I think it is not a very high effort task and would be good to have, so hopefully we can do it at some point.

[gvnwst]

Agreed on the "formatting initial vector" points. I was thinking it would be pretty straightforward for the user to write a wrapper which takes a spatial distribution and pipes it into tidy3d as a formatted vector... but certainly it would be nicer if it were a supported tool on your end (and I think a user-specified version would end up rewriting some of the grid-interpolation tools which already exist).

An interface which looks like "user provides a continuous function, or a sampled function over a grid, which is subsequently interpolated onto the appropriate mode grid" would be incredible.

Regarding time/effort though, just exposing the initial vector setting function in the near-term would be much appreciated. Leave the interface up to users for the time being, maybe getting feedback here on what works/what doesn't (to justify putting further effort in).

[momchil-flex]

An interface which looks like "user provides a continuous function, or a sampled function over a grid, which is subsequently interpolated onto the appropriate mode grid" would be incredible.

Regarding time/effort though, just exposing the initial vector setting function in the near-term would be much appreciated. Leave the interface up to users for the time being, maybe getting feedback here on what works/what doesn't (to justify putting further effort in).

The thing is, since most things run through the server, we can't really expose the function to the user. The exception would be the local mode solver, but we kind of left this as a bonus feature but for higher accuracy we recommend running even the mode solver through the server. So the initial vector specification needs to be serializable and most likely added to the ModeSpec. Because of this I think there's no low effort way to do this, but on the flip side there is a small difference in effort between getting something done and getting it done well, so when we get to it, we should be able to support just supplying an arbitrary field data and interpolating as needed under the hood.