Ipynb estimate params by mrzkp · Pull Request #540 · nasa/prog_models

mrzkp · 2023-05-30T18:14:00Z

Jupyter Notebook for estimate_params example.

github-actions · 2023-05-30T18:14:52Z

Thank you for opening this PR. Each PR into dev requires a code review. For the code review, look at the following:

Reviewer (someone other than author) should look for bugs, efficiency, readability, testing, and coverage in examples (if relevant).
Ensure that each PR adding a new feature should include a test verifying that feature.
All errors from static analysis must be resolved.
Review the test coverage reports (if there is a change) - will be added as comment on PR if there is a change
Review the software benchmarking results (if there is a change) - will be added as comment on PR
Any added dependencies are included in requirements.txt, setup.py, and dev_guide.rst (this document)
All warnings from static analysis must be reviewed and resolved - if deemed appropriate.

github-actions · 2023-05-30T18:17:07Z

Benchmarking Results
From:

Test	Time (s)
import main	0.13141459999999994
import thrown object	0.5140094
model initialization	0.12402749999999996
set noise	0.6407368999999998
simulate	0.29586179999999995
simulate with saving	0.9292029999999998
simulate with saving, dt	1.0238760000000005
simulate with printing results, dt	1.2402936000000002
Plot results	15.157017199999999
Metrics	0.03537379999999857
Surrogate Model Generation	3.363017099999997
surrogate sim	1.1037702000000031
surrogate sim, dt	2.994013800000001
To:
Test	Time (s)
---	---
import main	0.12857910000000006
import thrown object	0.5028411000000002
model initialization	0.12143020000000004
set noise	0.6641192000000002
simulate	0.2973857999999998
simulate with saving	0.938256
simulate with saving, dt	1.0532483
simulate with printing results, dt	1.2458516
Plot results	15.034077199999999
Metrics	0.03660229999999842
Surrogate Model Generation	3.3855144000000017
surrogate sim	1.0941775999999983
surrogate sim, dt	2.982657999999997

codecov-commenter · 2023-05-30T18:25:43Z

Codecov Report

Merging #540 (2608db5) into dev (6aff960) will decrease coverage by 0.02%.
Report is 1 commits behind head on dev.
The diff coverage is 50.00%.

@@            Coverage Diff             @@
##              dev     #540      +/-   ##
==========================================
- Coverage   84.50%   84.48%   -0.02%     
==========================================
  Files          45       45              
  Lines        3504     3506       +2     
==========================================
+ Hits         2961     2962       +1     
- Misses        543      544       +1

Files Changed	Coverage Δ
src/prog_models/prognostics_model.py	`85.49% <50.00%> (-0.14%)`	⬇️

kjjarvis

Great job on a first jupyter notebook example!

kjjarvis · 2023-05-30T18:44:31Z

+    "                `runs = [(t, u, z) for t, u, z in zip(times, inputs, outputs)]`\n",
+    "\n",
+    "\n",
+    "Using our optimization function, which runs `calc_error()` as a subroutine (more information about `calc_error()` found in our Calculating Error Example), given each of the runs.\n",


This sentence doesn't make sense to me. Can we reword it?

As discussed in our meeting, let's create a new example with multiple runs to introduce the concept of runs. I have ideas for this, so let me know if you need help.

Then we can reword this to just describe how calc_error is applied to each run independently, without the definition of runs.

kjjarvis · 2023-05-30T18:45:50Z

+    "You can also adjust the metric that is used to estimate parameters by setting the error_method to a different `calc_error()` method.\n",
+    "e.g., m.estimate_params([(times, inputs, outputs)], keys, dt=0.01, error_method='MAX_E')\n",
+    "Default is Mean Squared Error (MSE)\n",
+    "See calc_error method for list of options."


This is a really nice additional note. Should we include a line of code here to illustrate it? (basically pull out line 206 and make it an executable statement?)

kjjarvis · 2023-05-30T18:50:02Z

+   "source": [
+    "Now we can build a model with your best guess in parameters.\n",
+    "\n",
+    "We will use a ThrownObject Model and guess that our thrower is 20 meters tall. However, given our times, inputs and outputs, we can clearly tell this is obviously not true! Let's see if parameter estimation can fix this!"


This comment confused me a bit because we don't have times, inputs, or outputs at this point. Maybe we could reword it? It is an unreasonable height for a human, I think that is enough

kjjarvis · 2023-05-30T18:53:11Z

+    "`estimate_params()` creates a structure of 'runs' by constructing each index of times, inputs, and outputs and placing them into a tuple.\n",
+    "\n",
+    "                `runs = [(t, u, z) for t, u, z in zip(times, inputs, outputs)]`\n",
+    "\n",
+    "\n",
+    "Using our optimization function, which runs `calc_error()` as a subroutine (more information about `calc_error()` found in our Calculating Error Example), given each of the runs.\n",


What is the motivation for including these lines?

Co-authored-by: Katy Jarvis <55932920+kjjarvis@users.noreply.github.com>

github-actions · 2023-05-30T20:05:59Z

Benchmarking Results [Update]
From:

Test	Time (s)
import main	0.12692900000000007
import thrown object	0.49951900000000005
model initialization	0.11861670000000002
set noise	0.6694510999999999
simulate	0.29850429999999983
simulate with saving	0.9151151
simulate with saving, dt	1.0343770999999995
simulate with printing results, dt	1.2469164
Plot results	14.0218132
Metrics	0.03984029999999805
Surrogate Model Generation	3.154052400000001
surrogate sim	0.992927599999998
surrogate sim, dt	2.878385399999999

To:

Test	Time (s)
import main	0.12613110000000005
import thrown object	0.5041228
model initialization	0.11521589999999993
set noise	0.6658248
simulate	0.2980849000000001
simulate with saving	0.9123475999999999
simulate with saving, dt	1.0303399999999998
simulate with printing results, dt	1.2407971999999994
Plot results	14.1012616
Metrics	0.039112599999999276
Surrogate Model Generation	3.2667794000000008
surrogate sim	0.9905574000000001
surrogate sim, dt	2.884912400000001

teubert

This is really great! A great addition.

Just a few comments/suggestions.

teubert · 2023-06-01T17:43:01Z

+    "                `runs = [(t, u, z) for t, u, z in zip(times, inputs, outputs)]`\n",
+    "\n",
+    "\n",
+    "Using our optimization function, which runs `calc_error()` as a subroutine (more information about `calc_error()` found in our Calculating Error Example), given each of the runs.\n",


As discussed in our meeting, let's create a new example with multiple runs to introduce the concept of runs. I have ideas for this, so let me know if you need help.

Then we can reword this to just describe how calc_error is applied to each run independently, without the definition of runs.

github-actions · 2023-06-05T00:17:44Z

Benchmarking Results [Update]
From:

Test	Time (s)
import main	0.13267030000000002
import thrown object	0.5624488999999999
model initialization	0.12396289999999999
set noise	0.6627623999999999
simulate	0.2957882999999999
simulate with saving	0.9329586999999999
simulate with saving, dt	1.0242591
simulate with printing results, dt	1.2345926
Plot results	14.557750500000001
Metrics	0.03680040000000062
Surrogate Model Generation	3.1987537999999986
surrogate sim	1.091985300000001
surrogate sim, dt	2.941084100000001

To:

Test	Time (s)
import main	0.12899729999999998
import thrown object	0.5237215
model initialization	0.1223590000000001
set noise	0.6958458000000003
simulate	0.30500550000000004
simulate with saving	0.9435986000000001
simulate with saving, dt	1.0625382
simulate with printing results, dt	1.3497915999999996
Plot results	14.7202183
Metrics	0.037476099999999235
Surrogate Model Generation	3.3512746
surrogate sim	1.110149800000002
surrogate sim, dt	3.0115874999999974

Co-authored-by: Christopher Teubert <christopher.a.teubert@nasa.gov>

github-actions · 2023-06-07T09:18:42Z

Benchmarking Results [Update]
From:

Test	Time (s)
import main	0.12525790000000003
import thrown object	0.5077932999999999
model initialization	0.12105180000000004
set noise	0.6642322
simulate	0.3105446999999999
simulate with saving	0.9468350000000001
simulate with saving, dt	1.0541715999999997
simulate with printing results, dt	1.2895604
Plot results	14.8193871
Metrics	0.0386930999999997
Surrogate Model Generation	1.6027419000000016
surrogate sim	0.9991164999999995
surrogate sim, dt	2.9271320000000003

To:

Test	Time (s)
import main	0.12642789999999993
import thrown object	0.4992095999999999
model initialization	0.11763520000000005
set noise	0.6742300999999999
simulate	0.30225990000000014
simulate with saving	0.9223899000000002
simulate with saving, dt	1.0275759000000004
simulate with printing results, dt	1.2439065999999999
Plot results	14.9347774
Metrics	0.03962249999999656
Surrogate Model Generation	1.6051256999999985
surrogate sim	1.005881500000001
surrogate sim, dt	2.9225502

github-actions · 2023-06-07T17:46:50Z

Benchmarking Results [Update]
From:

Test	Time (s)
import main	0.13738290000000009
import thrown object	0.4968564
model initialization	0.12484209999999996
set noise	0.6900763000000001
simulate	0.2998441999999999
simulate with saving	0.9456690999999999
simulate with saving, dt	1.0289444999999997
simulate with printing results, dt	1.2421695999999995
Plot results	14.830344400000001
Metrics	0.03643780000000163
Surrogate Model Generation	1.6223934999999976
surrogate sim	1.0958791000000012
surrogate sim, dt	2.9792308999999975

To:

Test	Time (s)
import main	0.1340201000000001
import thrown object	0.4983432000000001
model initialization	0.12341990000000003
set noise	0.6630927
simulate	0.3147264000000001
simulate with saving	0.9753482999999998
simulate with saving, dt	1.0946959999999994
simulate with printing results, dt	1.3079795
Plot results	14.827162899999998
Metrics	0.040068500000000284
Surrogate Model Generation	1.6632789999999993
surrogate sim	1.1283831000000006
surrogate sim, dt	3.0338220000000007

github-actions · 2023-06-07T21:03:48Z

Benchmarking Results [Update]
From:

Test	Time (s)
import main	0.18809049999999994
import thrown object	0.6655873999999999
model initialization	0.18673430000000035
set noise	0.8471829
simulate	0.44303399999999993
simulate with saving	1.3842500999999996
simulate with saving, dt	1.4653251000000003
simulate with printing results, dt	1.9247331999999995
Plot results	23.9124739
Metrics	0.04862099999999714
Surrogate Model Generation	2.4982225999999983
surrogate sim	1.7446450999999996
surrogate sim, dt	4.384621700000004

To:

Test	Time (s)
import main	0.18742949999999992
import thrown object	0.6505394999999998
model initialization	0.18523700000000032
set noise	0.9128593
simulate	0.4492738000000003
simulate with saving	1.4003610000000002
simulate with saving, dt	1.5182539000000004
simulate with printing results, dt	1.9745572999999998
Plot results	23.6432748
Metrics	0.04771989999999704
Surrogate Model Generation	2.4618691999999953
surrogate sim	1.746990700000005
surrogate sim, dt	4.360033299999998

teubert · 2023-06-12T17:38:02Z

+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": []
+  }


Suggested change

{

"attachments": {},

"cell_type": "markdown",

"metadata": {},

"source": []

}

github-actions · 2023-06-12T17:38:11Z

Benchmarking Results [Update]
From:

Test	Time (s)
import main	0.12944640000000007
import thrown object	0.49727940000000004
model initialization	0.12465529999999991
set noise	0.6465977
simulate	0.29983519999999997
simulate with saving	0.9398775000000001
simulate with saving, dt	1.0390199000000004
simulate with printing results, dt	1.2605815000000007
Plot results	15.362323199999999
Metrics	0.0357890999999988
Surrogate Model Generation	1.6226205
surrogate sim	1.0889570999999982
surrogate sim, dt	2.9778656000000012

To:

Test	Time (s)
import main	0.1317520000000001
import thrown object	0.49100220000000006
model initialization	0.12420829999999983
set noise	0.6554557999999999
simulate	0.31355590000000033
simulate with saving	0.9683695999999999
simulate with saving, dt	1.0865255
simulate with printing results, dt	1.3048764999999998
Plot results	14.892852999999999
Metrics	0.038648199999997246
Surrogate Model Generation	1.6650488999999986
surrogate sim	1.117953400000001
surrogate sim, dt	3.0783123000000003

teubert · 2023-06-12T17:57:24Z

Also, let's get rid of the estimate_params.py example. It's replaced by this

github-actions · 2023-06-14T20:19:49Z

Benchmarking Results [Update]
From:

Test	Time (s)
import main	0.1524953
import thrown object	0.5426281
model initialization	0.16059980000000018
set noise	0.7616130999999999
simulate	0.40014740000000026
simulate with saving	1.2371561
simulate with saving, dt	1.3490963999999996
simulate with printing results, dt	1.6961833999999998
Plot results	21.7484148
Metrics	0.04254910000000223
Surrogate Model Generation	2.3823383000000007
surrogate sim	1.8087836999999993
surrogate sim, dt	4.216250299999999

To:

Test	Time (s)
import main	0.16475890000000004
import thrown object	0.5419577
model initialization	0.17616470000000017
set noise	0.8197261
simulate	0.39040529999999984
simulate with saving	1.2697626999999998
simulate with saving, dt	1.3902175000000003
simulate with printing results, dt	1.7852427000000004
Plot results	21.6146009
Metrics	0.04022220000000232
Surrogate Model Generation	2.3723503999999984
surrogate sim	1.7358708000000007
surrogate sim, dt	3.951831300000002

Co-authored-by: Katy Jarvis <55932920+kjjarvis@users.noreply.github.com> update word choice Co-authored-by: Katy Jarvis <55932920+kjjarvis@users.noreply.github.com> update word choice Co-authored-by: Katy Jarvis <55932920+kjjarvis@users.noreply.github.com>

github-actions · 2023-07-13T18:05:01Z

Benchmarking Results [Update]
From:

Test	Time (s)
import main	0.1665329000000002
import thrown object	0.6267655000000001
model initialization	0.23224359999999988
set noise	0.8558059999999998
simulate	0.7316473000000001
simulate with saving	2.1145043
simulate with saving, dt	2.7147145999999998
simulate with printing results, dt	3.368333700000001
Plot results	23.6549534
Metrics	0.052569800000000555
Surrogate Model Generation	3.4079791999999998
surrogate sim	2.3554922000000005
surrogate sim, dt	5.632676099999998

To:

Test	Time (s)
import main	0.16561939999999997
import thrown object	0.6554153999999999
model initialization	0.23515820000000032
set noise	0.8845358000000001
simulate	0.7019302000000001
simulate with saving	2.0579409
simulate with saving, dt	2.682283599999999
simulate with printing results, dt	3.3665918999999995
Plot results	23.1734309
Metrics	0.048312500000001535
Surrogate Model Generation	3.264265799999997
surrogate sim	2.3027767000000026
surrogate sim, dt	5.5073887

github-actions · 2023-07-14T18:01:33Z

Benchmarking Results [Update]
From:

Test	Time (s)
import main	0.11585250000000014
import thrown object	0.5032231
model initialization	0.15662010000000004
set noise	0.6859837000000002
simulate	0.5093831
simulate with saving	1.4167050000000003
simulate with saving, dt	1.8544235999999996
simulate with printing results, dt	2.2997391999999994
Plot results	14.2159684
Metrics	0.042829499999999854
Surrogate Model Generation	2.1999093999999992
surrogate sim	1.3810803000000007
surrogate sim, dt	3.711935199999999

To:

Test	Time (s)
import main	0.12470770000000009
import thrown object	0.5082107
model initialization	0.15659369999999972
set noise	0.6860340000000003
simulate	0.5000398000000001
simulate with saving	1.4012588999999998
simulate with saving, dt	1.8391472999999996
simulate with printing results, dt	2.2814563999999997
Plot results	14.162153899999998
Metrics	0.0423655000000025
Surrogate Model Generation	2.2110567000000003
surrogate sim	1.373380700000002
surrogate sim, dt	3.699323200000002

kjjarvis

Most of these are just small changes, but I'm a bit confused with the example using tolerance. Once we finalize that, this PR will be good to go.

github-actions · 2023-07-17T20:31:11Z

Benchmarking Results [Update]
From:

Test	Time (s)
import main	0.12998299999999996
import thrown object	0.47335400000000005
model initialization	0.1945209000000001
set noise	0.6650418999999999
simulate	0.5784020999999999
simulate with saving	1.6537791
simulate with saving, dt	2.0966006999999998
simulate with printing results, dt	2.7168529
Plot results	19.454953099999997
Metrics	0.049136400000001856
Surrogate Model Generation	3.1016049000000017
surrogate sim	2.2579206999999997
surrogate sim, dt	5.094986600000006

To:

Test	Time (s)
import main	0.13254900000000003
import thrown object	0.4942479000000002
model initialization	0.20455000000000023
set noise	0.6723105999999999
simulate	0.5709550000000001
simulate with saving	1.6993602999999995
simulate with saving, dt	2.1679367999999997
simulate with printing results, dt	2.8385244000000007
Plot results	20.231230599999996
Metrics	0.04222599999999943
Surrogate Model Generation	2.9180398999999966
surrogate sim	2.0875389999999996
surrogate sim, dt	4.659710000000004

kjjarvis

This is looking good, @aqitya. I made just minor suggestions this time, but there are still a handful of outstanding comments from the last review. As soon as those are addressed, I think we'll be ready to approve.

Co-authored-by: Katy Jarvis <55932920+kjjarvis@users.noreply.github.com>

kjjarvis

This looks great. Thanks, @aqitya, for being so thorough!

mrzkp added 7 commits May 26, 2023 10:28

Create param_est.ipynb

ad8c8dd

Update param_est.ipynb

3414f6d

Update param_est.ipynb

5fa76a4

Update param_est.ipynb

611c9c0

Update param_est.ipynb

0adad2f

Update param_est.ipynb

290e739

Merge branch 'dev' into ipynb-estimate_params

b219ce7

mrzkp requested review from kjjarvis and teubert May 30, 2023 18:14

kjjarvis requested changes May 30, 2023

View reviewed changes

Apply suggestions from code review

fe0e86a

Co-authored-by: Katy Jarvis <55932920+kjjarvis@users.noreply.github.com>

teubert requested changes Jun 1, 2023

View reviewed changes

Update param_est.ipynb

7347fa3

Apply suggestions from code review

0e5bb04

Co-authored-by: Christopher Teubert <christopher.a.teubert@nasa.gov>

mrzkp added 2 commits June 7, 2023 09:37

Update param_est.ipynb

5cf1fac

Update param_est.ipynb

760ba8f

More content

c39957b

Merge branch 'dev' into ipynb-estimate_params

f8e4438

teubert reviewed Jun 12, 2023

View reviewed changes

Update param_est.ipynb

e13f729

mrzkp added 6 commits June 30, 2023 08:51

Additional Changes

62899e5

Update param_est.ipynb

9a3714d

Needs review

aea02cb

Update test_estimate_params.py

07f68b8

using Absolute Mean Error

83502da

Update param_est.ipynb

087c5b1

mrzkp requested review from kjjarvis and teubert July 6, 2023 17:31

teubert and others added 5 commits July 7, 2023 11:56

A few recommended changes

e034f7c

clear output

a5ac370

fix typo

79e8b01

Co-authored-by: Katy Jarvis <55932920+kjjarvis@users.noreply.github.com> update word choice Co-authored-by: Katy Jarvis <55932920+kjjarvis@users.noreply.github.com> update word choice Co-authored-by: Katy Jarvis <55932920+kjjarvis@users.noreply.github.com>

Merge pull request #575 from nasa/param_est_ex_recommendations

03c34ea

Merge branch 'dev' into ipynb-estimate_params

d7fb89e

teubert approved these changes Jul 13, 2023

View reviewed changes

Fixed Test

b20d759

kjjarvis requested changes Jul 17, 2023

View reviewed changes

Update param_est.ipynb

f31ae77

Updated param_est example

bac7b34

kjjarvis requested changes Jul 25, 2023

View reviewed changes

mrzkp and others added 2 commits July 26, 2023 10:34

Apply suggestions from code review

d85101e

Co-authored-by: Katy Jarvis <55932920+kjjarvis@users.noreply.github.com>

resolving comments

2608db5

mrzkp requested a review from kjjarvis July 26, 2023 19:07

kjjarvis approved these changes Jul 31, 2023

View reviewed changes

teubert mentioned this pull request Aug 7, 2023

Estimate Parameter ipynb nasa/progpy#72

Merged

Conversation

mrzkp commented May 30, 2023

Uh oh!

github-actions bot commented May 30, 2023

Uh oh!

github-actions bot commented May 30, 2023

Uh oh!

codecov-commenter commented May 30, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

kjjarvis left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kjjarvis May 30, 2023

Choose a reason for hiding this comment

Uh oh!

teubert Jun 1, 2023

Choose a reason for hiding this comment

Uh oh!

kjjarvis May 30, 2023

Choose a reason for hiding this comment

Uh oh!

kjjarvis May 30, 2023

Choose a reason for hiding this comment

Uh oh!

kjjarvis May 30, 2023

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented May 30, 2023

Uh oh!

teubert left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

teubert Jun 1, 2023

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jun 5, 2023

Uh oh!

github-actions bot commented Jun 7, 2023

Uh oh!

github-actions bot commented Jun 7, 2023

Uh oh!

github-actions bot commented Jun 7, 2023

Uh oh!

teubert Jun 12, 2023

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jun 12, 2023

Uh oh!

teubert commented Jun 12, 2023

Uh oh!

github-actions bot commented Jun 14, 2023

Uh oh!

github-actions bot commented Jul 13, 2023

Uh oh!

github-actions bot commented Jul 14, 2023

Uh oh!

kjjarvis left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov-commenter commented May 30, 2023 •

edited

Loading