How are we supposed to set the clock and timing constraints? We provide a detailed example worked out that contradicts the documentation and fails to allow configuring the clock and timing.

[dsw]

I am trying to set the clock and the Vivado timing constraints.

According to this table: https://github.com/aws/aws-fpga/blob/master/hdk/docs/clock_recipes.csv

the A2 clock group sets the clk_main_a0 clock to 15.625 MHz, which is a period of 64.0 nanoseconds.

I want to request that configuration, however when I try to do that, these flags "--aws_clk_gen --clock_recipe_a=A2" to aws_build_dcp_from_cl.py do not seem to actually work.

Here is the line I run:

nohup time -v nice \
  ./aws_build_dcp_from_cl.py -c PROJECT_NAME \
  --aws_clk_gen --clock_recipe_a=A2 &

and here is the output I get in generated_cl_clocks_aws.xdc

#CL HBM reference clock @100MHz                                                \
                                                                                
create_clock -period 10.000 -name clk_hbm_ref  -waveform {0.000 5.000} [get_ports clk_hbm_ref]                                                                 
                                                                                
# Group A Clocks                                                                
create_clock -period 4  -name clk_main_a0 -waveform {0.000 2}  [get_ports clk_main_a0]                                                                         

A period of 4 nanoseconds means that the clk_main_a0 clock is running at 250 MHz, which is the value from the A1 clock group. The script build/scripts/aws_build_dcp_from_cl.py seems to have this as its default:

parser.add_option("--clock_recipe_a",                                       
                      dest="clock_recipe_a",                                    
                      help="Select Clock Recipe for Clock Group A (A0 | A1 | A2)",                                                                             
                      default="A1")                                             

I would overwrite the entry in generated_cl_clocks_aws.xdc except that it is being generated at compile time.

I have tried changing it in build/constraints/cl_timing_user.xdc but I just get a CRITICAL WARNING telling me to not do it.

Parsing XDC File [/home/ubuntu/aws-fpga/hdk/cl/examples/PROJECT_NAME/build/con\
straints/cl_timing_user.xdc]                                                    
CRITICAL WARNING: [Constraints 18-1056] Clock 'sys_clk' completely overrides cl\
ock 'clk_main_a0'.                                                              
New: create_clock -period 58.000 -name sys_clk [get_ports clk_main_a0], [/home/\
ubuntu/aws-fpga/hdk/cl/examples/PROJECT_NAME/build/constraints/cl_timing_user.\
xdc:22]                                                                         
Previous: create_clock -period 4.000 -name clk_main_a0 -waveform {0.000 2.000} \
[get_ports clk_main_a0], [/home/ubuntu/aws-fpga/hdk/cl/examples/PROJECT_NAME/b\
uild/constraints/generated_cl_clocks_aws.xdc:11]                                
Resolution: Review the constraint files and remove the redundant clock definiti\
on(s). If the clock constraints are not saved in a file, you can first save the\
 constraints to an XDC file and reload the design once the constraints have bee\
n corrected.                                                                    
Finished Parsing XDC File [/home/ubuntu/aws-fpga/hdk/cl/examples/PROJECT_NAME/\
build/constraints/cl_timing_user.xdc]                                           

The .vivado.log file says the following, which seems to be a script that creates the generated_cl_clocks_aws.xdc:

## set clocks_file [open "$CL_DIR/build/constraints/generated_cl_clocks_aws.xdc\
" w]                                                                            
. . .
## puts $clocks_file "# Do not edit this file! It is auto-generated from $argv0\
."                                                                              
. . .                         
## puts $clocks_file "# Group A Clocks"                                         
## puts $clocks_file "create_clock -period $clk_main_a0_period  -name clk_main_\
a0 -waveform {0.000 $clk_main_a0_half_period}  \[get_ports clk_main_a0\]"       
## close $clocks_file                                                           

However it seems to get the period from here (from the .vivado.log):

# source $HDK_SHELL_DIR/build/scripts/aws_gen_clk_constraints.tcl               
## switch $clock_recipe_a {                                                     
. . .
##     "A2" {                                                                   
##         #NOT SUPPORTED# set clk_main_a0_period        64                     
##         #NOT SUPPORTED# set clk_main_a0_half_period   32                     
##         set clk_main_a0_period         4                                     
##         set clk_main_a0_half_period    2                                     
. . .
##     }                                                                        

As you can see, the clk_main_a0_period is 4, which implies 250 MHz, not 15.625 MHz, as, again, is stated here: https://github.com/aws/aws-fpga/blob/master/hdk/docs/clock_recipes.csv

In fact, ALL of the clk_main_a0 periods are the same, for all of the clock groups, A0, A1, and A2 !

What is going on here? How do I set my clock and my timing constraints?

[dsw]

So I tried changing the build/constraints/cl_timing_user.xdc file to use the same name for the clock as the system clock that I am overriding:

create_clock -period 64.000 -name clk_main_a0 -waveform {0.000 32.000} [get_ports clk_main_a0]                                                                 

Doing this gets rid of this critical warning:

CRITICAL WARNING: [Constraints 18-1056] Clock 'sys_clk' completely overrides clock 'clk_main_a0'.

I have a toy processor that I built that just has a register file, an add instruction, and a multiply instruction done by repeated adding. This built and ran successfully with the default clock, which seems to be 250MHz.

I thought I would test my ability to change the clock on this project, as it already builds and runs correctly using the defaults. (Note that even though it builds and runs, I still get two build stages saying 6 critical warnings.)

So I attempted to configure this working project to use a slower clock.

(1) I requested the A2 clock recipe at compile time.

A_RECIPE := A2                                                                  
./aws_build_dcp_from_cl.py -c aws_f2_hello2 \                         
          --aws_clk_gen --clock_recipe_a=${A_RECIPE}

Which the build process seems to acknowledge:

==================================================                              
Running CL builds                                                               
==================================================                              
cl               : aws_f2_hello2                                                
mode             : small_shell                                                  
flow             : BuildAll                                                     
aws_clk_gen      : True                                                         
clock_recipe_a   : A2                                                           

(2) I changed the build/constraints/cl_timing_user.xdc:

# Constrain the main clock port to 64ns                                         
create_clock -period 64.000 -name clk_main_a0 -waveform {0.000 32.000} [get_ports clk_main_a0] 

It built without any more critical warnings than it had given the first time I built it with the defaults when it ran correctly.

(3) I tried to configure it before running.

sudo fpga-describe-clkgen -S 0                                                  
Error: (29) clkgen-ip-not-found                                                 

Not sure what that means. I tried to configure it:

sudo fpga-load-clkgen-recipe -S 0 -a 2                                          
Error: (29) clkgen-ip-not-found                                                 

Not sure how this is different, but it also does not work:

sudo fpga-load-clkgen-dynamic -S 0 -a 2                                         
Error: (29) clkgen-ip-not-found                                                 

(4) Even though that looks like failure, I tried running it anyway, using the same testing code that successfully runs it when I build it using the default clock settings. It does not run correctly, though I did not isolate in exactly what way it fails.

My summary of this clock situation is that your clock configuration system does not work as advertised. I have been working on this with someone who studied chip design in graduate school, so I do not think that this is due to our ignorance.

Again, my toy processor did work and run as expecxted when building on your system using the defaults. I am therefore hopeful that my real CPU will also work if we can just request a different clock. However, after spending many hours on this, we cannot figure out how to select a clock and get the System Verilog to build and run using that clock. Can you please advice on how to proceed?

[mjthimm]

Hi Daniel,

The clock recipe table you linked above is actually the clock recipe table for F1 instances. The f2 branch is the only branch on the aws-fpga repo that contains F2 documentation. The link that you shared in your other issue to our ReadTheDocs (RTD) site is the correct table for F2. The documentation on our RTD site draws solely from the f2 branch.

To your question concerning the AWS clock gen IP, the Error: (29) clkgen-ip-not-found is exactly what it sounds like.

Step 4 of the HDK README says that to be able to use those non-standard clock frequencies, use of the --aws_clk_gen flag is needed. When building your design, this specifies whether or not you want the AWS Clock Gen IP included in it. If it isn't present, those commands will fail as you showed above. Once the --aws_clk_gen flag is present and your recipes are declared for A, B, C, and HBM, those frequencies are made known to Vivado, which will then analyze the design according to the timing requirements of those individual frequencies.

I'm still working on getting you information for the other questions you've asked about manually specifying the frequencies and warnings and will reply here as soon as I have it.

Best,
Max

[dsw]

Above you said:

... to be able to use those non-standard clock frequencies, use of the --aws_clk_gen flag is needed. When building your design, this specifies whether or not you want the AWS Clock Gen IP included in it. If it isn't present, those commands will fail as you showed above. Once the --aws_clk_gen flag is present and your recipes are declared for A, B, C, and HBM, those frequencies are made known to Vivado, which will then analyze the design according to the timing requirements of those individual frequencies.

However if you look at my post, I did pass in the --aws_clk_gen flag:

A_RECIPE := A2                                                                  
./aws_build_dcp_from_cl.py -c aws_f2_hello2 \                         
          --aws_clk_gen --clock_recipe_a=${A_RECIPE}

[dsw]

You said:

The f2 branch is the only branch on the aws-fpga repo that contains F2 documentation. The link that you shared in your other issue to our ReadTheDocs (RTD) site is the correct table for F2.

So from that table, all clock recipes set clk_main_a0 to 250 MHz. Clock recipe A2 sets clk_extra_a1 to 15.625 MHz. So if I want a 15.625 MHz clock it sounds like I would pass --aws_clk_gen --clock_recipe_a=A2 to the build script. Would I also change my customer logic System Verilog to use clk_extra_a1 in place of clk_main_a0 ?

Again, please note that I already tried --aws_clk_gen --clock_recipe_a=A2 and I did not get a result that made sense to me. Is that because I also set a constraint in build/constraints/cl_timing_user.xdc ?

[dsw]

This page: https://github.com/aws/aws-fpga/blob/f2/hdk/docs/Clock_Recipes_User_Guide.md

says:

AWS_CLK_GEN is not required in the CL design for clk_main_a0 recipes. clk_extra_a0/a1/a2 clocks require AWS_CLK_GEN IP instantiated in CL design.

(1) How exactly do I go about instantiating AWS_CLK_GEN in my custom logic?

(2) Note that your columns are named "_a1", "_a2", "_a3", so I think you have an off-by-one error.

[dsw]

I must say that I find this whole clock situation to be wildly confusing. Please help me out by stepping back and viewing your whole system as if you were a beginner to it.

Please provide a full list of all of the steps I need to do to get a 15.625 MHz clock, starting from the default configuration. Below I have attempted an exhaustive list of questions in this direction. Please let me know if there is any other essential aspect I did not ask about.

(1) Do I have to literally copy some System Verilog into my CL directory, such as hdk/common/lib/aws_clk_gen.sv ? This page: https://github.com/aws/aws-fpga/blob/f2/hdk/docs/AWS_CLK_GEN_spec.md says

Usage of the AWS_CLK_GEN IP is fully demonstrated in the cl_mem_perf example. Please refer to that example for more details.

When I look at that example, it does not contain a copy of aws_clk_gen.sv, however it does invoke it from within cl_mem_perf.sv as follows below. Am I to explicitly invoke aws_clk_gen as below verbatim, or perhaps with modified arguments? How modified?

aws_clk_gen                                                                   
  #(                                                                            
    .CLK_GRP_A_EN           (1                        ),                        
    .CLK_GRP_B_EN           (1                        ),                        
    .CLK_GRP_C_EN           (1                        ),                        
    .CLK_HBM_EN             (1                        )                         
  )                                                                             
  AWS_CLK_GEN                                                                   
  (                                                                             
    .i_clk_main_a0          (clk_main_a0              ),                        
    .i_rst_main_n           (sda_xbar_sync_rst_n      ),                        
    .i_clk_hbm_ref          (clk_hbm_ref              ),                        

(2) What other changes do I need to make to my custom logic? Do I need to chose a port other than clk_main_a0 to my custom logic clock, such as clk_extra_a1 ?

(3) What flags do I pass to aws_build_dcp_from_cl.py ? You have yet to say that I should pass any flags other than the ones that I did pass, as stated in my original post in this thread.

(4) Should I make any changes to build/constraints/cl_timing_user.xdc or not?

Again, my toy processor worked great using your default clock, so I think I have everything else done correctly. It would be extremely helpful to me if we could get my bigger CPU to do the same, and I think all that is left to do to get that to work is this clock and timing aspect.

[czfpga]

@dsw

For guidance on using the AWS_CLK_GEN IP, refer to this doc: https://github.com/aws/aws-fpga/blob/f2/hdk/docs/AWS_CLK_GEN_spec.md#introduction specially:

Therefore, F2 Shell provides only two clocks - clk_main_a0 and clk_hbm_ref - to the CL resulting in efficient use of global routing resources. The clk_main_a0 is currently a fixed frequency 250MHz clock (:exclamation: dynamic scaling of frequency as F1 using the SW APIs will be added in a future release). The clk_hbm_ref is a fixed frequency 100MHz clock which can be used by customer as a reference clock for their MMCMs. This scheme provides flexibility for customers to devise their own clocking mechanisms with desired number of clocks.

⚠️ The AWS_CLK_GEN instantiation must be named to AWS_CLK_GEN and located in the CL top module.

Pleas read through the whole document. The Architecture Overview session contain the answers for some of your questions.

The cl_mem_perf contains a complete instantiation example for AWS_CLK_GEN:

  aws_clk_gen
  #(
    .CLK_GRP_A_EN           (1                        ),
    .CLK_GRP_B_EN           (1                        ),
    .CLK_GRP_C_EN           (1                        ),
    .CLK_HBM_EN             (1                        )
  )
  AWS_CLK_GEN
  (
    .i_clk_main_a0          (clk_main_a0              ),
    .i_rst_main_n           (sda_xbar_sync_rst_n      ),
    .i_clk_hbm_ref          (clk_hbm_ref              ),

    .s_axil_ctrl_awaddr     (sda_axil_clkgen.awaddr   ),
    .s_axil_ctrl_awvalid    (sda_axil_clkgen.awvalid  ),
    .s_axil_ctrl_awready    (sda_axil_clkgen.awready  ),
    .s_axil_ctrl_wdata      (sda_axil_clkgen.wdata    ),
    .s_axil_ctrl_wstrb      (sda_axil_clkgen.wstrb    ),
    .s_axil_ctrl_wvalid     (sda_axil_clkgen.wvalid   ),
    .s_axil_ctrl_wready     (sda_axil_clkgen.wready   ),
    .s_axil_ctrl_bresp      (sda_axil_clkgen.bresp    ),
    .s_axil_ctrl_bvalid     (sda_axil_clkgen.bvalid   ),
    .s_axil_ctrl_bready     (sda_axil_clkgen.bready   ),
    .s_axil_ctrl_araddr     (sda_axil_clkgen.araddr   ),
    .s_axil_ctrl_arvalid    (sda_axil_clkgen.arvalid  ),
    .s_axil_ctrl_arready    (sda_axil_clkgen.arready  ),
    .s_axil_ctrl_rdata      (sda_axil_clkgen.rdata    ),
    .s_axil_ctrl_rresp      (sda_axil_clkgen.rresp    ),
    .s_axil_ctrl_rvalid     (sda_axil_clkgen.rvalid   ),
    .s_axil_ctrl_rready     (sda_axil_clkgen.rready   ),

    .o_clk_hbm_ref          (gen_clk_hbm_ref          ),
    .o_clk_main_a0          (gen_clk_main_a0          ),
    .o_clk_extra_a1         (gen_clk_extra_a1         ),
    .o_clk_extra_a2         (gen_clk_extra_a2         ),
    .o_clk_extra_a3         (gen_clk_extra_a3         ),
    .o_clk_extra_b0         (gen_clk_extra_b0         ),
    .o_clk_extra_b1         (gen_clk_extra_b1         ),
    .o_clk_extra_c0         (gen_clk_extra_c0         ),
    .o_clk_extra_c1         (gen_clk_extra_c1         ),
    .o_clk_hbm_axi          (gen_clk_hbm_axi          ),
    .o_cl_rst_hbm_axi_n     (gen_rst_hbm_axi_n        ),
    .o_cl_rst_hbm_ref_n     (gen_rst_hbm_ref_n        ),
    .o_cl_rst_c1_n          (gen_rst_c1_n             ),
    .o_cl_rst_c0_n          (gen_rst_c0_n             ),
    .o_cl_rst_b1_n          (gen_rst_b1_n             ),
    .o_cl_rst_b0_n          (gen_rst_b0_n             ),
    .o_cl_rst_a3_n          (gen_rst_a3_n             ),
    .o_cl_rst_a2_n          (gen_rst_a2_n             ),
    .o_cl_rst_a1_n          (gen_rst_a1_n             ),
    .o_cl_rst_main_n        (gen_rst_main_n           )
  );

For your question:

(4) Should I make any changes to build/constraints/cl_timing_user.xdc or not?

No. You don't need to modify the constraints as the build flow would take care of that. Please follow the document to include the --aws_clk_gen and the relevant --clock_recipe_<> building options and make sure the AWS_CLK_GEN instantiation has the target output clock enabled to match the building options. For example, for --aws_clk_gen --clock_recipe_a=A2, at least the CLK_GRP_A_EN parameter of the AWS_CLK_GEN instance in CL must be set to 1.

[dsw]

Does this shortcut for not using aws_clk_gen work?: The documentation says that clk_hbm_ref provides a 100MHz clock; if my custom logic will run at 100MHz, can I just replace the clk_main_a0 input to my custom logic clock with clk_hbm_ref and run at 100MHz? Does that work or does that cause other subtle problems and therefore not actually work?

[czfpga]

Yes, you can use the 100MHz clock to directly drive your CL logic.

Usage of the AWS_CLOCK_GEN IP in the CL is optional as mentioned in the document. The IP offers the predefined clock frequencies through different clock groups and the flexibility of dynamic clock configuration. If that's not what you need, you don't need to add the IP to your design.

The other option is to use the 100MHz clock as the source clock and instantiate your own MMCM/PLL in the CL and. This option gives you the full control over the generated clock frequencies. The only downside is that you can no longer leverage the AWS clock management APIs to manage the them.

[dsw]

You said:

The other option is to use the 100MHz clock as the source clock and instantiate your own MMCM/PLL in the CL and.

What I mean is, suppose ALL I do is replace the clk_main_a0 input to my custom logic clock with clk_hbm_ref and I do NOTHING else. I do not instantiate my own MMCM etc. Will my chip now run at 100MHz?

If not, what if I also change build/constraints/cl_timing_user.xdc to specify a -period 10.0.

Or is it the case that ANY changes to the clock from 250MHz MUST instantiate some MMCM?

[czfpga]

For this question:

Will my chip now run at 100MHz?

I've responded:

Yes, you can use the 100MHz clock to directly drive your CL logic.

You don't need to specify the clock in .xdc file