How to synthesize incrementally/hierarchically

[pbreuer]

I have tried

read_verilog A.v; synth_verilog -top A; write_json A.json
...
read_verilog Z.v; synth_verilog -top Z; write_json Z.json

(edit out all but the top level module in [A-Z].json)

read_verilog Top.v; read_json [A-Z].json; synth_xilinx -top Top; write_json Top.json

and that synthesizes to a finish, but is it right? How do I do this?

Notes: comparing with read_verilog Top.v [A-Z].v; synth_xilinx -top Top; write_json Top.json the modules in the latter output have fewer input and output wires because some can be seen by the synthesis to be not needed in the context in which they are embedded, and the internal variable names are a bit different (locals! I don't think they can collide; can they?), and the "top" attribute has been stripped from the submodules, and that's all I can see as differences. (I would go about 75/25 as a computer scientist that none of that matters, given any slight piece of luck at all, basically because local names don't mean anything in principle and I think those are deliberately constructed to miss other modules' locals, and i don't care about extra dangling wires if we end up with those).

Why? One way needs 153GB memory and 110h cpu time, the other needs 52GB memory and 13h cpu time (for final assembly - up till there the components take about 20GB and an hour or less each). So peak memory requirements are way down and that's the biggie. Also failure/mistake along the way doesn't mean go back to Go and start again from zero.

Any help/scorn, advice where to find 160GB RAM cheaply?

[nakengelhardt]

Unless you use -flatten, each module will be treated separately in the synthesis flow anyway - it should be possible to run synthesis on each module individually to avoid reserving all the working memory needed for the design at once (except for the final write of the complete synthesized design). However, there are at least 2 complications I can see from a brief look at synth_xilinx:

• You need to make sure that you synthesize the correct derived instantiation for parametrized modules with the parameter values actually used by the design (I don't know if there are any checks when loading from json that would ensure this).
• The iopadmap pass needs the whole design hierarchy to insert the I/O buffers in the right places. (If you are using vivado for pnr you can probably skip this with the -noiopad option and rely on vivado to insert them later. Either way, for a first attempt at trying if this works, avoiding this complication is probably best.) At a quick glance this is the only pass I noticed that requires this, but there might be others that I missed.

RTLIL is the best format for saving intermediate state, so the way I would go about experimenting with this is to:

1. load the entire design from verilog, preferably with -defer to avoid having a unused derived module with the default parameter values (read_verilog -defer [A-Z].v Top.v)
2. run hierarchy -top Top
3. using write_rtlil, dump each module to a separate .il file
4. also using write_rtlil, dump the entire design
5. in separate yosys processes, load an individual module from .il and load blackboxes for all other modules in the design with read_rtlil -nooverwrite -lib (in case any of the non-top modules A-Z have submodules)
6. run synth_xilinx -run prepare:map_cells to synthesize the individual module up to the iopadmap call
7. use write_rtlil to write out the synthesized-up-to-iopadmap module
8. in one common yosys process, load all the synthesized-up-to-iopadmap modules and run the iopadmap step
9. dump out each post-iopadmap module to individual .il files again
10. in separate yosys processes, load the individual modules and the blackboxes again and continue the synth_xilinx run (first run the last 2 steps of map_cells, then synth_xilinx -run map_ffs:check)
11. write out the synthesized module to .il again
12. in a common yosys process, load all the synthesized modules again
13. run the final steps synth_xilinx -run check:

I think that should work, but this is off the top of my head and completely untested. Try it on a much smaller example first!

[pbreuer]

@nakengelhardt thank you very much! I have been going through and get to a problem, but my interpretation needs checking. I did this:

1. read_verilog -defer Top.v [A-Z].v
2. hierarchy -top Top
3. select Top; write_rtlil -selected Top.raw-il; select A; write_rtlil -selected A.raw-il; ...
4. select *; write_rtlil Top-all.raw-il; exit
5. new processes: read_rtlil foo.raw-il; read_rtlil -nooverwrite -lib *.raw-il;
6. ---------- synth_xilinx -run prepare:map_cells
7. ---------- select foo; write_rtlil -selected foo.pre-iopadmap-il
8. new process: read_rtlil *.pre-iopadmap-il; synth_xilinx -run map_cells: ; ******** dies *******
9. select Top; write_rtlil -selected Top.post-iopadmap-il; select A; write_rtlil -selected A.post-iopadmap-il; ...

It dies with

4.11. Executing CLKBUFMAP pass (inserting clock buffers).
Removed 0 unused cells and 127813 unused wires.
4.12. Executing HIERARCHY pass (managing design hierarchy).  
4.12.1. Analyzing design hierarchy..
Top module:  \Mux
Used module:     \Synth_Mux_mux256_muxN
Used module:         \Synth_Mux_mux256_myOf
ERROR: Module `\IBUF' referenced in module `\Mux' in cell `$iopadmap$Mux.eta1' is not part of the design.

So it seems I should either not be getting rid of stuff with select, or should be adding it in where needed.

I'd be grateful for the lowdown on what the crash dummy here should really have done!

Thanks again!

[pbreuer]

@nakengelhardt thanks again! Do you mean the full read_verilog command from step 1, or just part of it? I would have thought just part ... and that I need to avoid double load with -nooverwrite. Shrug, but anyway, so step 8 is

8. new process: read_rtlil *.pre-iopadmap-il; read_verilog -nooverwrite -defer *.v; synth_xilinx -run map_cells:

Do I need -defer there? If -noverwrite stops me picking up the stuff I have already done, it should be OK to compile the things from the library that it will pick up as new now, shouldn't it, and don't they need to be compiled?

Also, have I guessed right about the use of select/-selected so far? And should I be using -top in some places?

I'll plug on and see what breaks ... That gets much further in step 8 but still dies with

7.6.2. Continuing TECHMAP pass.
Using template $paramod\$_SDFFE_PP0P_\_TECHMAP_WIREINIT_Q_=1'0 for cells of type $_SDFFE_PP0P_.
No more expansions possible.
<suppressed ~16413 debug messages>
7.7. Executing XILINX_SRL pass (Xilinx shift register extraction).
ERROR: Accessing non existing parameter \IS_R_INVERTED

so ... how has it avoided picking that up? Indeed, it is not present in any of the files written (or read) so far. It must have to come from library input or be generated. Ah ... it's defined here:

/usr/share/yosys/xilinx/cells_sim.v:  parameter [0:0] IS_R_INVERTED = 1'b0;

Should I read -nooverwrite all those files too?

Thanks again

PTB

PS. Same error same place with or without the -defer in the read_verilog in step 8.

[nakengelhardt]

Sorry, I was on my phone last night so wasn't clear. I didn't mean the read_verilog from your design, but the one inside the synth_xilinx step begin that loads the definitions of the target primitive cells, which is these two lines:

        read_verilog -lib -specify +/xilinx/cells_sim.v
        read_verilog -lib +/xilinx/cells_xtra.v

So the whole script would be something like read_rtlil *pre-iopadmap-il; read_verilog -lib -specify +/xilinx/cells_sim.v +/xilinx/cells_xtra.v; synth_xilinx -run map_cells: ;

[pbreuer]

@nakengelhardt That (step 8) went perfectly now. (deep magic! :-). Also the write-out step 9 worked. Continuing ...

8. read_rtlil *.pre-iopadmap-il; read_verilog -lib -specify +/xilinx/cells_sim.v +/xilinx/cells_xtra.v; synth_xilinx -run map_cells: ;
9. select Top; write_rtlil -selected Top.post-iopadmap-il; select A; write_rtlil -selected A.post-iopadmap-il; ...
10. new process (/es ?): read_rtlil Top.post-iopadmap-il; read_rtlil -nooverwrite -lib *.post-iopadmap-il; synth_xilinx -run map_cells:fine ; synth_xilinx -run map_ffs:check;

In 10 I am trying to interpret "continue the synth_xilinx run (first run the last 2 steps of map_cells, then synth_xilinx -run map_ffs:check" but am hampered by not knowing what the last 2 steps of map_cells are or how one specifies them or how one runs them! I think you are referring to the help text for synth_xilinx, perhaps?

That runs and errors:

5.12.1. Analyzing design hierarchy..
Top module: \Mux
ERROR: Module \BUFG' referenced in module \Mux' in cell `$auto$clkbufmap.cc:261:execute$85718' is not part of the design.

so need more library stuff again.

[nakengelhardt]

Oh, right, I didn't read thoroughly what you put in step 8 - this was supposed to be only iopadmap and no call to synth_xilinx at all. If you already ran synth_xilinx -run map_cells: you've finished synthesis of the whole design in the common process at that point (you could just as an intermediate test put a write_json at the end of that and check if the output synthesized correctly). And yes, when I mention the commands in the steps that's the ones printed by the help message of synth_xilinx (although sometimes the help message doesn't print the exact arguments to a command, just something in square or angle brackets, and you might need to look the details up in the log or the source).

My idea was trying to minimize the part run on the whole design to the very minimum, so only
8. read_rtlil *.pre-iopadmap-il; read_verilog -lib -specify +/xilinx/cells_sim.v +/xilinx/cells_xtra.v; iopadmap -bits -outpad OBUF I:O -inpad IBUF O:I -toutpad OBUFT ~T:I:O -tinoutpad IOBUF ~T:O:I:IO A:top; select Top; write_rtlil -selected Top.post-iopadmap-il; select A; write_rtlil -selected A.post-iopadmap-il; ...
Then next a batch of per-module processes would do:
9. read_rtlil foo.post-iopadmap-il; read_rtlil -nooverwrite -lib *.post-iopadmap-il; read_verilog -lib -specify +/xilinx/cells_sim.v +/xilinx/cells_xtra.v; techmap -map +/techmap.v -map +/xilinx/cells_map.v; clean; synth_xilinx -run map_ffs:check; write_rtlil foo.pre-check.il
And finally in one common process again:
10. read_rtlil *.pre-check.il; read_verilog -lib -specify +/xilinx/cells_sim.v +/xilinx/cells_xtra.v; techmap -map +/techmap.v -map +/xilinx/cells_map.v; synth_xilinx -run check: ; write_json synthesized.json

[pbreuer]

@nakengelhardt It worked! Just needed a slight mod in 10 to read_rtlil to use -nooverwrite :

10. read_rtlil Top.pre-check-il; read_rtlil -nooverwrite *.pre-check-il; read_verilog -lib -specify +/xilinx/cells_sim.v +/xilinx/cells_xtra.v; techmap -map +/techmap.v -map +/xilinx/cells_map.v; synth_xilinx -run check: ; write_json Top.synthesized-json

I'll create a makefile and/or shell script and post it here for you to add comments about where you are getting the info from and how you are translating that into intentions, and the intentions into commands, so that somebody else can take that file and adapt it themselves in future for a different target platform than xilinx.

Many, many, many thanks!

PTB

[pbreuer]

@nakengelhardt Makefile is attached. Creative commons licence, you are credited, please add more specifics as you prefer.
Makefile
.
My concern is that this produces a much smaller file than just running read_verilog *.v; synth_xilinx -top Top; write_json ... in the standard way. It appears if I read it in with read_json and list modules with ls to be just the top level module component, not the whole hierarchy, but reading the file myself I DO see the submodules in there with the blackbox attribute set! Here, with sizes in bytes:

 388314060  TACache.json       <- standard
  14337022  TACache.json-new   <- this way

Does this align with what is expected?
Regards

PTB

PS. Yes, removing the blackbox attribute in the submodule descriptions makes the submodules visible when I do read_json Top.json; stat -tech xilinx, but it just lists them and doesn't see them as a hierarchy (except for the one next-to-top submodule). I think this command could be required in yosys but the select command doesn't find anything but Top to select, which is useless:

select A-Z Top; setattr -unset blackbox

What is the proper way to deal with this, if it is indeed a problem? Could connections between submodules be missing?

[pbreuer]

@nakengelhardt I now have a properly working process and makefile. I followed your general idea in using the steps listed in help synth_xilinx. (1) i guessed how far to go monolithically (pass 33, just before the map_cells label in the listed program) and I stop there and write the modules out as .il files. (2) Each is then read in individually and processed to the check label in the listed program and written out again. Then (3) they are all read in at once to one monolithic process for running from check label onwards n the listed program and are written as a single .json file.

Here is a shell script inline, and I'll attach a Makefile. Please do improve it. I likely took step (1) too far into the standard process and could have broken out earlier. (LC count is a half-percent higher this way so at least I am doing something that has an affect! It's not so bad a loss given the speed-up, provided it is correct).

`#! /bin/bash -x
top=WHATEVER

1) monolthic start up to map_cells phase and writes indivdual .ril files.

prog="read_verilog *.v;
read_verilog -lib -specify +/xilinx/cells_sim.v;
read_verilog -lib +/xilinx/cells_xtra.v;
hierarchy -top ${top};
synth_xilinx -run prepare:map_cells;
iopadmap -bits
-outpad OBUF I:O
-inpad IBUF O:I
-toutpad OBUFT ~T:I:O
-tinoutpad IOBUF ~T:O:I:IO
A:top
"
for f in *.v; do
mod=${f%.v}
prog="${prog}; select ${mod}; write_rtlil -selected ${mod}.ril"
done
prog="${prog};"
yosys -p "${prog}"

2) run each module individually through map_cells up to check phase & turning .ril into .il

for f in *.ril; do
mod=${f%.ril}
prog="read_rtlil ${mod}.ril"
prog="${prog}; read_verilog -lib -specify +/xilinx/cells_sim.v"
prog="${prog}; read_verilog -lib -specify +/xilinx/cells_xtra.v"
prog="${prog}; techmap -map +/techmap.v -map +/xilinx/cells_map.v"
prog="${prog}; clean"
prog="${prog}; synth_xilinx -run map_ffs:check"
prog="${prog}; write_rtlil ${mod}.il"
yosys -p "${prog}"
done

3) put together all the .il monolithically producing .json

prog="read_rtlil ${top}.il; read_rtlil -nooverwrite Synth*.il"
prog="${prog}; read_verilog -lib -specify +/xilinx/cells_sim.v"
prog="${prog}; read_verilog -lib -specify +/xilinx/cells_xtra.v"
prog="${prog}; techmap -map +/techmap.v -map +/xilinx/cells_map.v"
prog="${prog}; synth_xilinx -run check: "
prog="${prog}; write_json ${top}.json"
yosys -p "${prog}"
`

Makefile.txt

[pbreuer]

I've marked my own answer as best because the script and/or makefile have worked fine on dozens of examples now. If it can be further improved let me know (I note for example that I am not using "-defer" even though it was recommended, basically because I do not know what it does ... such words as have been said about it unfortunately have meant nothing to me :-(; what the heck is a derived module and why should I care if it is unused or used and what does used or unused mean and how do they matter?).

[nakengelhardt]

I'd have to actually run it myself to see if it needs improving, which I'm afraid I don't have the time for currently, but it looks fine at first glance.

As for derived modules and -defer, this is about parametrized modules: say you have a module with a parameter FOO, and a generate loop that will instantiate a submodule FOO times. The yosys internal representation (RTLIL) has a cell for each instance - and each cell is a concrete object in memory, so it can create the RTLIL for one, two, or five submodules, but it can't represent an abstract number of FOO submodules. So when read_verilog loads a design, it uses the default parameter values to go through the AST and build ("derive") the module representation in memory. Then when hierarchy is run, it looks for the actual values given in each instantiation of the module, and if they are different from the default values, goes back to the AST from the read_verilog frontend and assigns the parameters that are used, and re-derives the module for, say, FOO=1, FOO=2 and FOO=5.

-defer tells the frontend not to build the module representation for the default parameter set. There are two cases where that can be useful:

1. if the module is never used with the default parameters, it saves a tiny bit of memory and time as it doesn't build the internal representation for the default derivation.
2. if the default parameter values lead to invalid logic - say the default is FOO=0, and the generate loop uses FOO-1 as an index somewhere where negative values aren't allowed, and this causes the frontend to exit with an error. The error can be avoided by never going through the code with the default value FOO=0.

So basically, if it's not crashing, you're probably fine not using -defer.

[pbreuer]

@nakengelhardt thank you very much for your expert comment. I think what you have written tells me that lack of -defer will cause explicit failures to be reported in synthesis where it would have synthesized correctly and silently with -defer, aka "type 1 error" (excess reported failures where success is expected, not excess of reported successes were failure is expected, which would be "type 2 errors").

That is possibly too pessimistic at times, but completely fine with me, as you say with "basically, if it's not crashing, you're ... fine not using -defer". I would only care if it missed reporting failure, which it doesn't.

For background, the verilog code is generated so I don't know precisely what it is in it without going to an awful lot of effort to find out. My impression is that occasionally parameterized modules are generated and do appear in the verlog because in yosys' output I see only one instance of FOO reported where I would have expected several, customized differently. On the other hand I do sometimes see FOO_0, FOO_1, etc in the yosys output, conforming to my expectation that several different instances of basically the same component should appear in the final assembly. Perhaps those named instances are produced directly by the generating mechanism from the high level description (in Clash). I don't know. As I said, it would be a lot of trouble and effort for me to go to in order to figure out what's up in any particular case.

As to whether including -defer would have an effect either way, going by your explanation, no, because the generated verilog code must already be correct as it stands with the parameters instantiated as they should be, given that the generating mechanism is producing correct verilog code. If it is not, then it is neither my problem nor yours, so we may as well assume it is correct.

I should therefore use -defer, but I honestly don't mind extra strictness, even if silly in the sense of "that code doesn't appear like that here". Best write stuff that always works, no matter what, and forgetting -defer will help make sure that I do.

Did I remember to write that this procedure lowers synthesis time from 43h to 29h? Peak VMEM use is a little down, from 96GB to 92GB.